Developer Interface

Source and Stages

Base abstract classes for defining a pipeline source and processing stages.

class smartpipeline.stage.Source[DataT][source]

Extend this for defining a pipeline source, e.g. Source[MyData] generates Item[MyData] items

property is_stopped: bool

True if the source has called the stop event

property logger: Logger

Specific logger for this object

abstractmethod pop()[source]

Generate items for feeding a pipeline. Must be overridden for properly defining a source. Call Source.stop() when item generation is ended

Return type:

Optional[Item[DataT]]

Returns:

The generated item, if None it is simply ignored (e.g. after calling Source.stop())

stop()[source]

Declare the end item generation, this event will be spread through the pipeline

class smartpipeline.stage.Stage[DataT][source]

Extend this class and override Stage.process() for defining a stage, e.g. Stage[MyData] processes Item[MyData] items

property logger: Logger

Specific logger for this object

on_end()

Called when the stage terminates, useful for executing closing operations (e.g. on files)

It is paired with a successful on_start(), also when the start phase fails after this stage has started. Any exception raised here is fatal for the whole pipeline and is reported to the pipeline caller on termination instead of hanging the execution without feedback.

Return type:

Any

on_start()

Called in the pipeline start phase, when Pipeline.run() or Pipeline.process_async() starts the stage, never by Pipeline.build(). With multiprocessing concurrency the stage in the executor is a copy of the original, so by overriding this method one can initialize variables specifically for the copies; that is mandatory when they are not serializable.

Any exception raised here is fatal for the whole pipeline and is reported to the pipeline caller (on run()/process_async()) instead of hanging the execution without feedback. on_end runs only for stages whose on_start completed.

Return type:

Any

abstractmethod process(item)[source]

Process a single item received by the stage. Must be overridden for properly defining a stage

Return type:

Item[DataT]

Returns:

The same item instance processed and enriched by the stage

class smartpipeline.stage.BatchStage[DataT](size, timeout=1.0)[source]

Extend this class and override BatchStage.process_batch() for defining a batch stage, e.g. BatchStage[MyData] processes Item[MyData] items

property logger: Logger

Specific logger for this object

on_end()

Called when the stage terminates, useful for executing closing operations (e.g. on files)

It is paired with a successful on_start(), also when the start phase fails after this stage has started. Any exception raised here is fatal for the whole pipeline and is reported to the pipeline caller on termination instead of hanging the execution without feedback.

Return type:

Any

on_start()

Called in the pipeline start phase, when Pipeline.run() or Pipeline.process_async() starts the stage, never by Pipeline.build(). With multiprocessing concurrency the stage in the executor is a copy of the original, so by overriding this method one can initialize variables specifically for the copies; that is mandatory when they are not serializable.

Any exception raised here is fatal for the whole pipeline and is reported to the pipeline caller (on run()/process_async()) instead of hanging the execution without feedback. on_end runs only for stages whose on_start completed.

Return type:

Any

abstractmethod process_batch(items)[source]

Process a batch of items received by the stage. Must be overridden for properly defining a batch stage

Return type:

Sequence[Item[DataT]]

Returns:

The same batch with items processed and enriched by the stage; it must contain exactly one item for each input item

property size: int

Get the maximum size of item batches that can be processed together

property timeout: float

Seconds to wait before flushing a batch (calling BatchStage.process_batch() on it)

Data Items

The unit of data flowing through a pipeline.

class smartpipeline.item.Item[DataT](data=<object object>)[source]

An item containing a unit of data to process through a pipeline. Is ia usually generated by the stage.Source subclass and retrieved at the end of a pipeline.Pipeline.

Overloads:
  • ()

  • data (DataT)

The payload type is generic: Item[MyData] carries MyData in Item.data (a TypedDict, a dataclass, a primitive, …). Item() defaults to a Dict[str, Any] payload; a custom payload always requires explicit data, as in Item[MyData](…). Heterogeneous dict literals need an explicit Item[Dict[str, Any]]({…}) for static checking. None is a valid payload (e.g. for Optional types) and it is not replaced by the default one.

add_critical_error(stage, exception)[source]

This is called internally by the error.handling.ErrorManager when the exception is handled. Add a error.exceptions.CriticalError generated in a stage (referenced by its name) for the item

Parameters:

exception (Union[CriticalError, Exception]) – It can be a error.exceptions.CriticalError instance or any exception, which will be encapsulated in a error.exceptions.CriticalError

Return type:

CriticalError

add_soft_error(stage, exception)[source]

This is called internally by the error.handling.ErrorManager when the exception is handled. Add an error.exceptions.SoftError generated in a stage (referenced by its name) for the item.

Parameters:

exception (Union[SoftError, Exception]) – It can be an error.exceptions.SoftError instance or any exception, which will be encapsulated in an error.exceptions.SoftError

Return type:

SoftError

callback()[source]

Call the function set with Item.set_callback()

Return type:

Any

property callback_id: str

Identifier of the callback set with Item.set_callback().

Unlike the function, it travels with the item when a stage worker gets a copy of it, so a pipeline can find the callback it kept in the caller’s process.

critical_errors()[source]

Iter over error.exceptions.CriticalError instances or any un-managed exception eventually generated in some stage processing

Return type:

Generator[CriticalError, None, None]

property data: DataT

Access to the actual data contained in the item

Returns:

The payload, by default a dictionary in which organizing data by fields (recommended to be JSON serializable). Build it with Item() (an empty Dict[str, Any]) or Item[Dict[str, Any]]({…}), and with Item[MyData](…) for a custom payload type

data_snippet(max_size=100)[source]

A short string representation of the Item.data (recommended to override this method)

Parameters:

max_size (int) – Maximum size of the string representation

get_timing(stage_name)[source]

Get the wall-clock time spent by a stage (referenced by its name) for processing the item.

It is not CPU time and excludes queue waiting; on threads it includes GIL waiting.

Return type:

Optional[float]

Returns:

The time in seconds or None if the item has not been processed by the stage

has_critical_errors()[source]

True if the item has raised a error.exceptions.CriticalError or any un-managed exception in some stage processing

Return type:

bool

has_soft_errors()[source]

True if the item has raised an error.exceptions.SoftError in some stage processing

Return type:

bool

property id: Any

Get the unique identifier of the item. It is recommended to override this in order to properly compute it from the Item.data

property metadata: Dict[str, Any]

Get the actual metadata dictionary

set_callback(fun)[source]

Set a function to call after a successful asynchronous execution of a pipeline on the item (through pipeline.Pipeline.process_async())

The function runs in the process which submitted the item: stage workers get a copy of the item without it, and the pipeline calls it back through Item.callback_id.

Return type:

str

Returns:

the Item.callback_id of this item

set_timing(stage_name, seconds)[source]

Set the wall-clock time spent by a stage (referenced by its name) for processing the item.

The duration includes blocking work, retries and, on threads, time spent waiting for the GIL, so it is not CPU time. See the concurrency guide for how to read it.

Return type:

Item[DataT]

soft_errors()[source]

Iter over error.exceptions.SoftError instances eventually generated in some stage processing

Return type:

Generator[SoftError, None, None]

property timed_stages: KeysView[str]

Get the stage names for which the item has a process time

Pipeline

Main class for assembling stages and executing the pipeline, along with its stage specification and lifecycle types.

class smartpipeline.pipeline.Pipeline[DataT](max_queues_size=1000)[source]

Main class for designing the sequence of stages and execute them, e.g. Pipeline[MyData] runs Item[MyData] items through Stage[MyData] stages. Pipeline without parameters defaults to Dict[str, Any] payloads.

append(name, stage, *, concurrency=0, executor='threads', retryable_errors=(), max_retries=0, backoff=0.0, init_args=None, init_kwargs=None)[source]

Append a stage to the pipeline just after the last one appended, or after the source if it is the first stage.

Overloads:
  • name (str), stage (Union[Stage[DataT], BatchStage[DataT]]), *, concurrency (int) = 0, executor (Literal['threads', 'processes']) = 'threads', retryable_errors (Tuple[Type[Exception], ...]) = tuple(), max_retries (int) = 0, backoff (Union[float, int]) = 0.0 → Self

  • name (str), stage (Union[Type[Stage[DataT]], Type[BatchStage[DataT]]]), *, concurrency (int) = 0, executor (Literal['threads', 'processes']) = 'threads', retryable_errors (Tuple[Type[Exception], ...]) = tuple(), max_retries (int) = 0, backoff (Union[float, int]) = 0.0, init_args (Optional[Sequence[Any]]) = None, init_kwargs (Optional[Mapping[str, Any]]) = None → Self

stage can be an instance or a class, with different construction timing:

  • An instance is used as it is. It exists immediately, its name is set by this call, and Pipeline.get_stage() returns it before build(). init_args and init_kwargs are not accepted with an instance.

  • A class is stored and constructed by Pipeline.build(), one by one in append order; its constructor receives init_args and init_kwargs. A class-based stage does not exist until ``build()`` runs: before that, Pipeline.get_stage() returns None for it.

build() constructs each class in order, collecting every failure. No stage worker starts until Pipeline.run() or Pipeline.process_async().

Parameters:
  • name (str) – Name to identify the stage in the pipeline, it must be a non-empty string, it is set in the stage and it must be unique in the pipeline

  • stage (Union[Stage[DataT], BatchStage[DataT], Type[Stage[DataT]], Type[BatchStage[DataT]]]) – Instance of a stage, or a stage class to construct at build time

  • concurrency (int) – Number of concurrent stage executions, if 0 then threads/processes won’t be involved for this stage

  • executor (Literal['threads', 'processes']) – “threads” for using threads, “processes” for using multiprocessing

  • retryable_errors (Tuple[Type[Exception], ...]) – list of exceptions for which the stage applies an exponential backoff strategy. After the final failed attempt, a error.exceptions.RetryError soft error is attached to the item

  • max_retries (int) – maximum number of retries for the stage before attaching a RetryError(SoftError) (default 0)

  • backoff (Union[float, int]) – backoff factor for the exponential backoff strategy used by the stage when it raises one of the exceptions declared in retryable_errors param (default 0.0)

  • init_args (Optional[Sequence[Any]]) – Positional arguments for the stage constructor, only valid with a stage class

  • init_kwargs (Optional[Mapping[str, Any]]) – Keyword arguments for the stage constructor, only valid with a stage class

Raises:
  • PipelineStateError – When the pipeline has already been built

  • ValueError – When the stage or its configuration is not valid

build()[source]

Validate and prepare the pipeline.

Stage classes are constructed and stored, but no stage or worker is started: Pipeline.run() and Pipeline.process_async() start them. Calling build() again on a built pipeline is a no-op.

Raises:
  • ValueError – When no stage has been appended

  • PipelineStateError – When the pipeline is running or closed

Return type:

Self

close(timeout=10.0)[source]

Close the pipeline, idempotently.

Running workers are cancelled, threads are joined, queues are drained, and on_end hooks run for every stage whose on_start completed. Executors and the multiprocessing manager are released. A closed pipeline cannot be built or run again. A second close() racing with another one waits until the first has finished.

Return type:

None

property count: int

Number of items delivered to the caller.

Counts the items yielded by Pipeline.run() and the items delivered by Pipeline.get_item(), including the items which carry soft errors. Items processed one by one with Pipeline.process() are not counted.

Returns:

Count of delivered items

property critical_errors_count: int

Number of critical errors attached to the items delivered to the caller.

Counts the error.exceptions.CriticalError instances carried by the items yielded by Pipeline.run() or delivered by Pipeline.get_item(). Critical items skip downstream stages but are still delivered, so they are included. Items processed one by one with Pipeline.process() are not counted.

Returns:

Count of critical errors on delivered items

get_item(block=True)[source]

Get a single item from the asynchronous execution of the pipeline on single items from Pipeline.process_async()

When called before the first Pipeline.process_async(), it waits for the asynchronous run to start (or for the pipeline to be closed/stopped) instead of raising just because the output queue does not exist yet.

Parameters:

block (bool) – If True wait for the run to start and then for the next processed item

Raises:
  • ValueError – When block is False and no asynchronous run has started

  • PipelineStateError – When the run ends or the pipeline is closed/stopped before an item is available

  • queue.Empty – When we do not block and the queue is empty

  • BaseException – Whatever exception has made the asynchronous runner fail

Return type:

Item[DataT]

get_stage(name)[source]

Get a stage instance by its name.

A class-based stage exists only after Pipeline.build(), so before that this returns None for it; an instance passed to Pipeline.append() is available immediately.

Return type:

Union[Stage[DataT], BatchStage[DataT], None]

property name: str

Pipeline unique name, equivalent to its logger name

process(item)[source]

Process a single item synchronously (no concurrency) through the pipeline.

A built pipeline can process several items one by one until it is closed. It runs the stages directly, with no containers and no queues.

Raises:
  • ValueError – When the pipeline contains concurrent stages

  • PipelineStateError – When the pipeline is running or closed

Return type:

Item[DataT]

process_async(item, callback=None)[source]

Process a single item asynchronously through the pipeline, stages may run concurrently. The call returns immediately, processed items are retrieved with Pipeline.get_item()

Parameters:

callback (Optional[Callable[[Item[DataT]], Any]]) – A function to call after a successful process of the item; it runs in this process, also when stages use workers in other processes

Raises:

PipelineStateError – When the pipeline is closed or its source has stopped

run()[source]

Run the pipeline given a source and a concatenation of stages. Get the sequence of items through iteration

Return type:

Generator[Item[DataT], None, None]

Returns:

Iterator over processed items

Raises:

ValueError – When a source has not been set for the pipeline

set_error_manager(error_manager)[source]

Set the error manager for handling errors from each stage item processing

Raises:

PipelineStateError – When the pipeline has already been built

Return type:

Self

set_source(source)[source]

Set the source of the pipeline: a subclass of stage.Source or any iterable of items (e.g. a generator), wrapped in a private source which stops when exhausted.

A built pipeline without a source can still be given one, as long as it has not started running.

Raises:

PipelineStateError – When the pipeline is running or closed

Return type:

Self

property soft_errors_count: int

Number of soft errors attached to the items delivered to the caller.

Counts the error.exceptions.SoftError instances (including error.exceptions.RetryError) carried by the items yielded by Pipeline.run() or delivered by Pipeline.get_item(). Items processed one by one with Pipeline.process() are not counted.

Returns:

Count of soft errors on delivered items

property state: PipelineState

Current lifecycle state of the pipeline

stop()[source]

Tell the source to stop to generate items and consequently terminate the pipeline after all “remaining” items are processed.

class smartpipeline.pipeline.StageSpec[DataT](name, stage=None, stage_class=None, init_args=(), init_kwargs=dict(), concurrency=0, executor='threads', retryable_errors=(), max_retries=0, backoff=0.0)[source]

Declarative description of a stage collected by Pipeline.append().

A spec holds either an already constructed stage instance or the stage class and the arguments needed to construct it. The pipeline turns specs into stages in Pipeline.build(), in a single deterministic pass.

construct()[source]

Return the stage instance, constructing it when the spec holds a class.

Return type:

Union[Stage[DataT], BatchStage[DataT]]

property retry_manager: RetryManager

The retry configuration of this stage, built once and reused for every item.

class smartpipeline.pipeline.PipelineState(*values)[source]

Lifecycle state of a Pipeline.

A pipeline is built once, can process single items synchronously, and can run its source once. After a run (or an explicit Pipeline.close()) it is closed and cannot be reused.

class smartpipeline.pipeline.PipelineStateError[source]

Raised when an operation is not allowed in the current pipeline state.

Error Handling

Exception classes raised during stage execution and managers for handling and retrying errors.

class smartpipeline.error.exceptions.Error(*args, **kwargs)[source]

Base exception type for stages

add_note()

Exception.add_note(note) – add a note to the exception

get_exception()[source]

Get the original exception (if any) that has generated this error, equivalent to the __cause__ attribute

Return type:

Optional[BaseException]

with_exception(exception)[source]

Set the original exception (if any) that has generated this error, equivalent to explicit exception chaining

Return type:

Error

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

class smartpipeline.error.exceptions.SoftError(*args, **kwargs)[source]

A type of exception which usually only provokes skipping a stage for an item

add_note()

Exception.add_note(note) – add a note to the exception

get_exception()

Get the original exception (if any) that has generated this error, equivalent to the __cause__ attribute

Return type:

Optional[BaseException]

with_exception(exception)

Set the original exception (if any) that has generated this error, equivalent to explicit exception chaining

Return type:

Error

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

class smartpipeline.error.exceptions.CriticalError(*args, **kwargs)[source]

A type of exception which usually provokes skipping the whole pipeline for an item

add_note()

Exception.add_note(note) – add a note to the exception

get_exception()

Get the original exception (if any) that has generated this error, equivalent to the __cause__ attribute

Return type:

Optional[BaseException]

with_exception(exception)

Set the original exception (if any) that has generated this error, equivalent to explicit exception chaining

Return type:

Error

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

class smartpipeline.error.exceptions.RetryError(*args, **kwargs)[source]

A stage processing an input item may raise some Exception for which the stage must try to reprocess the item using an exponential backoff retry strategy. After all attempts fail, a single RetryError is attached to the item and that stage is skipped for it

add_note()

Exception.add_note(note) – add a note to the exception

get_exception()

Get the original exception (if any) that has generated this error, equivalent to the __cause__ attribute

Return type:

Optional[BaseException]

with_exception(exception)

Set the original exception (if any) that has generated this error, equivalent to explicit exception chaining

Return type:

Error

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

class smartpipeline.error.exceptions.BatchOutputError(*args, **kwargs)[source]

A stage.BatchStage returned a number of items different from its input batch.

This is a fatal contract error: the pipeline stops instead of retrying it or attaching a per-item soft/critical error.

add_note()

Exception.add_note(note) – add a note to the exception

get_exception()

Get the original exception (if any) that has generated this error, equivalent to the __cause__ attribute

Return type:

Optional[BaseException]

with_exception(exception)

Set the original exception (if any) that has generated this error, equivalent to explicit exception chaining

Return type:

Error

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

class smartpipeline.error.handling.ErrorManager[source]

Basic error handling of a pipeline, principally manages exceptions.SoftError and exceptions.CriticalError types

check_critical_errors(item)[source]

Check the critical errors registered for an item

Return type:

Optional[BaseException]

handle(error, stage, item)[source]

Manage an error produced by a stage

Parameters:
  • error (Exception) – It can be a generic exception or an error from exceptions explicitly raised by a stage

  • stage (AliveMixin) – Stage which raised the exception during processing

  • item (Item[Any]) – Item which raised the exception when processed

Return type:

Optional[CriticalError]

Returns:

If the handled error results to be critical return the generated exceptions.CriticalError

Raises:

Exception – When a ErrorManager.raise_on_critical_error() has been set and the error is critical

property logger: Logger

Specific logger for this object

no_skip_on_critical_error()[source]

Change default behaviour of the error manager on exceptions.CriticalError: only skip the stage which raises it, like the exceptions.SoftError . Valid only if ErrorManager.raise_on_critical_error() is not set

Return type:

ErrorManager

on_end()

Called when the stage terminates, useful for executing closing operations (e.g. on files)

It is paired with a successful on_start(), also when the start phase fails after this stage has started. Any exception raised here is fatal for the whole pipeline and is reported to the pipeline caller on termination instead of hanging the execution without feedback.

Return type:

Any

on_start()

Called in the pipeline start phase, when Pipeline.run() or Pipeline.process_async() starts the stage, never by Pipeline.build(). With multiprocessing concurrency the stage in the executor is a copy of the original, so by overriding this method one can initialize variables specifically for the copies; that is mandatory when they are not serializable.

Any exception raised here is fatal for the whole pipeline and is reported to the pipeline caller (on run()/process_async()) instead of hanging the execution without feedback. on_end runs only for stages whose on_start completed.

Return type:

Any

raise_on_critical_error()[source]

Set the error manager so that if a exceptions.CriticalError (or any un-managed exception) is encountered, it raises it “externally”, at the pipeline run level, killing the pipeline

Return type:

ErrorManager

class smartpipeline.error.handling.RetryManager(retryable_errors=(), max_retries=0, backoff=0)[source]

This class encapsulate the parameters used to handle the retry strategy in case some kind of error are raise by the stage

Helpers

Built-in helper sources and item subclasses shipped with the library.

class smartpipeline.helpers.LocalFilesSource(dir_path, postfix='')[source]

Generate a special file item for any file found in a directory

property is_stopped: bool

True if the source has called the stop event

property logger: Logger

Specific logger for this object

pop()[source]

Generate items for feeding a pipeline. Must be overridden for properly defining a source. Call Source.stop() when item generation is ended

Return type:

Optional[Item[Dict[str, Any]]]

Returns:

The generated item, if None it is simply ignored (e.g. after calling Source.stop())

stop()

Declare the end item generation, this event will be spread through the pipeline

class smartpipeline.helpers.FilePathItem(path, data=None)[source]

An item with a pointer to a file path

add_critical_error(stage, exception)

This is called internally by the error.handling.ErrorManager when the exception is handled. Add a error.exceptions.CriticalError generated in a stage (referenced by its name) for the item

Parameters:

exception (Union[CriticalError, Exception]) – It can be a error.exceptions.CriticalError instance or any exception, which will be encapsulated in a error.exceptions.CriticalError

Return type:

CriticalError

add_soft_error(stage, exception)

This is called internally by the error.handling.ErrorManager when the exception is handled. Add an error.exceptions.SoftError generated in a stage (referenced by its name) for the item.

Parameters:

exception (Union[SoftError, Exception]) – It can be an error.exceptions.SoftError instance or any exception, which will be encapsulated in an error.exceptions.SoftError

Return type:

SoftError

callback()

Call the function set with Item.set_callback()

Return type:

Any

property callback_id: str

Identifier of the callback set with Item.set_callback().

Unlike the function, it travels with the item when a stage worker gets a copy of it, so a pipeline can find the callback it kept in the caller’s process.

critical_errors()

Iter over error.exceptions.CriticalError instances or any un-managed exception eventually generated in some stage processing

Return type:

Generator[CriticalError, None, None]

property data: Dict[str, Any]

Access to the actual data contained in the item

Returns:

The payload, by default a dictionary in which organizing data by fields (recommended to be JSON serializable). Build it with Item() (an empty Dict[str, Any]) or Item[Dict[str, Any]]({…}), and with Item[MyData](…) for a custom payload type

data_snippet(max_size=100)

A short string representation of the Item.data (recommended to override this method)

Parameters:

max_size (int) – Maximum size of the string representation

get_timing(stage_name)

Get the wall-clock time spent by a stage (referenced by its name) for processing the item.

It is not CPU time and excludes queue waiting; on threads it includes GIL waiting.

Return type:

Optional[float]

Returns:

The time in seconds or None if the item has not been processed by the stage

has_critical_errors()

True if the item has raised a error.exceptions.CriticalError or any un-managed exception in some stage processing

Return type:

bool

has_soft_errors()

True if the item has raised an error.exceptions.SoftError in some stage processing

Return type:

bool

property id: Any

Get the unique identifier of the item. It is recommended to override this in order to properly compute it from the Item.data

property metadata: Dict[str, Any]

Get the actual metadata dictionary

set_callback(fun)

Set a function to call after a successful asynchronous execution of a pipeline on the item (through pipeline.Pipeline.process_async())

The function runs in the process which submitted the item: stage workers get a copy of the item without it, and the pipeline calls it back through Item.callback_id.

Return type:

str

Returns:

the Item.callback_id of this item

set_timing(stage_name, seconds)

Set the wall-clock time spent by a stage (referenced by its name) for processing the item.

The duration includes blocking work, retries and, on threads, time spent waiting for the GIL, so it is not CPU time. See the concurrency guide for how to read it.

Return type:

Item[Dict[str, Any]]

soft_errors()

Iter over error.exceptions.SoftError instances eventually generated in some stage processing

Return type:

Generator[SoftError, None, None]

property timed_stages: KeysView[str]

Get the stage names for which the item has a process time