Developer Interface
Source and Stages
Base abstract classes for defining a pipeline source and processing stages.
- class smartpipeline.stage.Source[DataT][source]
Extend this for defining a pipeline source, e.g. Source[MyData] generates Item[MyData] items
- property is_stopped: bool
True if the source has called the stop event
- property logger: Logger
Specific logger for this object
- abstractmethod pop()[source]
Generate items for feeding a pipeline. Must be overridden for properly defining a source. Call
Source.stop()when item generation is ended- Return type:
Optional[Item[DataT]]- Returns:
The generated item, if None it is simply ignored (e.g. after calling
Source.stop())
- class smartpipeline.stage.Stage[DataT][source]
Extend this class and override
Stage.process()for defining a stage, e.g. Stage[MyData] processes Item[MyData] items- property logger: Logger
Specific logger for this object
- on_end()
Called when the stage terminates, useful for executing closing operations (e.g. on files)
It is paired with a successful
on_start(), also when the start phase fails after this stage has started. Any exception raised here is fatal for the whole pipeline and is reported to the pipeline caller on termination instead of hanging the execution without feedback.- Return type:
Any
- on_start()
Called in the pipeline start phase, when
Pipeline.run()orPipeline.process_async()starts the stage, never byPipeline.build(). With multiprocessing concurrency the stage in the executor is a copy of the original, so by overriding this method one can initialize variables specifically for the copies; that is mandatory when they are not serializable.Any exception raised here is fatal for the whole pipeline and is reported to the pipeline caller (on run()/process_async()) instead of hanging the execution without feedback. on_end runs only for stages whose on_start completed.
- Return type:
Any
- class smartpipeline.stage.BatchStage[DataT](size, timeout=1.0)[source]
Extend this class and override
BatchStage.process_batch()for defining a batch stage, e.g. BatchStage[MyData] processes Item[MyData] items- property logger: Logger
Specific logger for this object
- on_end()
Called when the stage terminates, useful for executing closing operations (e.g. on files)
It is paired with a successful
on_start(), also when the start phase fails after this stage has started. Any exception raised here is fatal for the whole pipeline and is reported to the pipeline caller on termination instead of hanging the execution without feedback.- Return type:
Any
- on_start()
Called in the pipeline start phase, when
Pipeline.run()orPipeline.process_async()starts the stage, never byPipeline.build(). With multiprocessing concurrency the stage in the executor is a copy of the original, so by overriding this method one can initialize variables specifically for the copies; that is mandatory when they are not serializable.Any exception raised here is fatal for the whole pipeline and is reported to the pipeline caller (on run()/process_async()) instead of hanging the execution without feedback. on_end runs only for stages whose on_start completed.
- Return type:
Any
- abstractmethod process_batch(items)[source]
Process a batch of items received by the stage. Must be overridden for properly defining a batch stage
- Return type:
Sequence[Item[DataT]]- Returns:
The same batch with items processed and enriched by the stage; it must contain exactly one item for each input item
- property size: int
Get the maximum size of item batches that can be processed together
- property timeout: float
Seconds to wait before flushing a batch (calling
BatchStage.process_batch()on it)
Data Items
The unit of data flowing through a pipeline.
- class smartpipeline.item.Item[DataT](data=<object object>)[source]
An item containing a unit of data to process through a pipeline. Is ia usually generated by the
stage.Sourcesubclass and retrieved at the end of apipeline.Pipeline.- Overloads:
()
data (
DataT)
The payload type is generic: Item[MyData] carries MyData in
Item.data(a TypedDict, a dataclass, a primitive, …). Item() defaults to a Dict[str, Any] payload; a custom payload always requires explicit data, as in Item[MyData](…). Heterogeneous dict literals need an explicit Item[Dict[str, Any]]({…}) for static checking. None is a valid payload (e.g. for Optional types) and it is not replaced by the default one.- add_critical_error(stage, exception)[source]
This is called internally by the
error.handling.ErrorManagerwhen the exception is handled. Add aerror.exceptions.CriticalErrorgenerated in a stage (referenced by its name) for the item- Parameters:
exception (
Union[CriticalError,Exception]) – It can be aerror.exceptions.CriticalErrorinstance or any exception, which will be encapsulated in aerror.exceptions.CriticalError- Return type:
- add_soft_error(stage, exception)[source]
This is called internally by the
error.handling.ErrorManagerwhen the exception is handled. Add anerror.exceptions.SoftErrorgenerated in a stage (referenced by its name) for the item.- Parameters:
exception (
Union[SoftError,Exception]) – It can be anerror.exceptions.SoftErrorinstance or any exception, which will be encapsulated in anerror.exceptions.SoftError- Return type:
- callback()[source]
Call the function set with
Item.set_callback()- Return type:
Any
- property callback_id: str
Identifier of the callback set with
Item.set_callback().Unlike the function, it travels with the item when a stage worker gets a copy of it, so a pipeline can find the callback it kept in the caller’s process.
- critical_errors()[source]
Iter over
error.exceptions.CriticalErrorinstances or any un-managed exception eventually generated in some stage processing- Return type:
Generator[CriticalError,None,None]
- property data: DataT
Access to the actual data contained in the item
- Returns:
The payload, by default a dictionary in which organizing data by fields (recommended to be JSON serializable). Build it with Item() (an empty Dict[str, Any]) or Item[Dict[str, Any]]({…}), and with Item[MyData](…) for a custom payload type
- data_snippet(max_size=100)[source]
A short string representation of the
Item.data(recommended to override this method)- Parameters:
max_size (
int) – Maximum size of the string representation
- get_timing(stage_name)[source]
Get the wall-clock time spent by a stage (referenced by its name) for processing the item.
It is not CPU time and excludes queue waiting; on threads it includes GIL waiting.
- Return type:
Optional[float]- Returns:
The time in seconds or None if the item has not been processed by the stage
- has_critical_errors()[source]
True if the item has raised a
error.exceptions.CriticalErroror any un-managed exception in some stage processing- Return type:
bool
- has_soft_errors()[source]
True if the item has raised an
error.exceptions.SoftErrorin some stage processing- Return type:
bool
- property id: Any
Get the unique identifier of the item. It is recommended to override this in order to properly compute it from the
Item.data
- property metadata: Dict[str, Any]
Get the actual metadata dictionary
- set_callback(fun)[source]
Set a function to call after a successful asynchronous execution of a pipeline on the item (through
pipeline.Pipeline.process_async())The function runs in the process which submitted the item: stage workers get a copy of the item without it, and the pipeline calls it back through
Item.callback_id.- Return type:
str- Returns:
the
Item.callback_idof this item
- set_timing(stage_name, seconds)[source]
Set the wall-clock time spent by a stage (referenced by its name) for processing the item.
The duration includes blocking work, retries and, on threads, time spent waiting for the GIL, so it is not CPU time. See the concurrency guide for how to read it.
- Return type:
Item[DataT]
- soft_errors()[source]
Iter over
error.exceptions.SoftErrorinstances eventually generated in some stage processing- Return type:
Generator[SoftError,None,None]
- property timed_stages: KeysView[str]
Get the stage names for which the item has a process time
Pipeline
Main class for assembling stages and executing the pipeline, along with its stage specification and lifecycle types.
- class smartpipeline.pipeline.Pipeline[DataT](max_queues_size=1000)[source]
Main class for designing the sequence of stages and execute them, e.g. Pipeline[MyData] runs Item[MyData] items through Stage[MyData] stages. Pipeline without parameters defaults to Dict[str, Any] payloads.
- append(name, stage, *, concurrency=0, executor='threads', retryable_errors=(), max_retries=0, backoff=0.0, init_args=None, init_kwargs=None)[source]
Append a stage to the pipeline just after the last one appended, or after the source if it is the first stage.
- Overloads:
name (
str), stage (Union[Stage[DataT],BatchStage[DataT]]), *, concurrency (int) =0, executor (Literal['threads','processes']) ='threads', retryable_errors (Tuple[Type[Exception],...]) =tuple(), max_retries (int) =0, backoff (Union[float,int]) =0.0→Selfname (
str), stage (Union[Type[Stage[DataT]],Type[BatchStage[DataT]]]), *, concurrency (int) =0, executor (Literal['threads','processes']) ='threads', retryable_errors (Tuple[Type[Exception],...]) =tuple(), max_retries (int) =0, backoff (Union[float,int]) =0.0, init_args (Optional[Sequence[Any]]) =None, init_kwargs (Optional[Mapping[str,Any]]) =None→Self
stagecan be an instance or a class, with different construction timing:An instance is used as it is. It exists immediately, its name is set by this call, and
Pipeline.get_stage()returns it beforebuild().init_argsandinit_kwargsare not accepted with an instance.A class is stored and constructed by
Pipeline.build(), one by one in append order; its constructor receivesinit_argsandinit_kwargs. A class-based stage does not exist until ``build()`` runs: before that,Pipeline.get_stage()returnsNonefor it.
build()constructs each class in order, collecting every failure. No stage worker starts untilPipeline.run()orPipeline.process_async().- Parameters:
name (
str) – Name to identify the stage in the pipeline, it must be a non-empty string, it is set in the stage and it must be unique in the pipelinestage (
Union[Stage[DataT],BatchStage[DataT],Type[Stage[DataT]],Type[BatchStage[DataT]]]) – Instance of a stage, or a stage class to construct at build timeconcurrency (
int) – Number of concurrent stage executions, if 0 then threads/processes won’t be involved for this stageexecutor (
Literal['threads','processes']) – “threads” for using threads, “processes” for using multiprocessingretryable_errors (
Tuple[Type[Exception],...]) – list of exceptions for which the stage applies an exponential backoff strategy. After the final failed attempt, aerror.exceptions.RetryErrorsoft error is attached to the itemmax_retries (
int) – maximum number of retries for the stage before attaching a RetryError(SoftError) (default 0)backoff (
Union[float,int]) – backoff factor for the exponential backoff strategy used by the stage when it raises one of the exceptions declared in retryable_errors param (default 0.0)init_args (
Optional[Sequence[Any]]) – Positional arguments for the stage constructor, only valid with a stage classinit_kwargs (
Optional[Mapping[str,Any]]) – Keyword arguments for the stage constructor, only valid with a stage class
- Raises:
PipelineStateError – When the pipeline has already been built
ValueError – When the stage or its configuration is not valid
- build()[source]
Validate and prepare the pipeline.
Stage classes are constructed and stored, but no stage or worker is started:
Pipeline.run()andPipeline.process_async()start them. Callingbuild()again on a built pipeline is a no-op.- Raises:
ValueError – When no stage has been appended
PipelineStateError – When the pipeline is running or closed
- Return type:
Self
- close(timeout=10.0)[source]
Close the pipeline, idempotently.
Running workers are cancelled, threads are joined, queues are drained, and on_end hooks run for every stage whose on_start completed. Executors and the multiprocessing manager are released. A closed pipeline cannot be built or run again. A second close() racing with another one waits until the first has finished.
- Return type:
None
- property count: int
Number of items delivered to the caller.
Counts the items yielded by
Pipeline.run()and the items delivered byPipeline.get_item(), including the items which carry soft errors. Items processed one by one withPipeline.process()are not counted.- Returns:
Count of delivered items
- property critical_errors_count: int
Number of critical errors attached to the items delivered to the caller.
Counts the
error.exceptions.CriticalErrorinstances carried by the items yielded byPipeline.run()or delivered byPipeline.get_item(). Critical items skip downstream stages but are still delivered, so they are included. Items processed one by one withPipeline.process()are not counted.- Returns:
Count of critical errors on delivered items
- get_item(block=True)[source]
Get a single item from the asynchronous execution of the pipeline on single items from
Pipeline.process_async()When called before the first
Pipeline.process_async(), it waits for the asynchronous run to start (or for the pipeline to be closed/stopped) instead of raising just because the output queue does not exist yet.- Parameters:
block (
bool) – If True wait for the run to start and then for the next processed item- Raises:
ValueError – When block is False and no asynchronous run has started
PipelineStateError – When the run ends or the pipeline is closed/stopped before an item is available
queue.Empty – When we do not block and the queue is empty
BaseException – Whatever exception has made the asynchronous runner fail
- Return type:
Item[DataT]
- get_stage(name)[source]
Get a stage instance by its name.
A class-based stage exists only after
Pipeline.build(), so before that this returns None for it; an instance passed toPipeline.append()is available immediately.- Return type:
Union[Stage[DataT],BatchStage[DataT],None]
- property name: str
Pipeline unique name, equivalent to its logger name
- process(item)[source]
Process a single item synchronously (no concurrency) through the pipeline.
A built pipeline can process several items one by one until it is closed. It runs the stages directly, with no containers and no queues.
- Raises:
ValueError – When the pipeline contains concurrent stages
PipelineStateError – When the pipeline is running or closed
- Return type:
Item[DataT]
- process_async(item, callback=None)[source]
Process a single item asynchronously through the pipeline, stages may run concurrently. The call returns immediately, processed items are retrieved with
Pipeline.get_item()- Parameters:
callback (
Optional[Callable[[Item[DataT]],Any]]) – A function to call after a successful process of the item; it runs in this process, also when stages use workers in other processes- Raises:
PipelineStateError – When the pipeline is closed or its source has stopped
- run()[source]
Run the pipeline given a source and a concatenation of stages. Get the sequence of items through iteration
- Return type:
Generator[Item[DataT],None,None]- Returns:
Iterator over processed items
- Raises:
ValueError – When a source has not been set for the pipeline
- set_error_manager(error_manager)[source]
Set the error manager for handling errors from each stage item processing
- Raises:
PipelineStateError – When the pipeline has already been built
- Return type:
Self
- set_source(source)[source]
Set the source of the pipeline: a subclass of
stage.Sourceor any iterable of items (e.g. a generator), wrapped in a private source which stops when exhausted.A built pipeline without a source can still be given one, as long as it has not started running.
- Raises:
PipelineStateError – When the pipeline is running or closed
- Return type:
Self
- property soft_errors_count: int
Number of soft errors attached to the items delivered to the caller.
Counts the
error.exceptions.SoftErrorinstances (includingerror.exceptions.RetryError) carried by the items yielded byPipeline.run()or delivered byPipeline.get_item(). Items processed one by one withPipeline.process()are not counted.- Returns:
Count of soft errors on delivered items
- property state: PipelineState
Current lifecycle state of the pipeline
- class smartpipeline.pipeline.StageSpec[DataT](name, stage=None, stage_class=None, init_args=(), init_kwargs=dict(), concurrency=0, executor='threads', retryable_errors=(), max_retries=0, backoff=0.0)[source]
Declarative description of a stage collected by
Pipeline.append().A spec holds either an already constructed stage instance or the stage class and the arguments needed to construct it. The pipeline turns specs into stages in
Pipeline.build(), in a single deterministic pass.- construct()[source]
Return the stage instance, constructing it when the spec holds a class.
- Return type:
Union[Stage[DataT],BatchStage[DataT]]
- property retry_manager: RetryManager
The retry configuration of this stage, built once and reused for every item.
- class smartpipeline.pipeline.PipelineState(*values)[source]
Lifecycle state of a
Pipeline.A pipeline is built once, can process single items synchronously, and can run its source once. After a run (or an explicit
Pipeline.close()) it is closed and cannot be reused.
Error Handling
Exception classes raised during stage execution and managers for handling and retrying errors.
- class smartpipeline.error.exceptions.Error(*args, **kwargs)[source]
Base exception type for stages
- add_note()
Exception.add_note(note) – add a note to the exception
- get_exception()[source]
Get the original exception (if any) that has generated this error, equivalent to the __cause__ attribute
- Return type:
Optional[BaseException]
- with_exception(exception)[source]
Set the original exception (if any) that has generated this error, equivalent to explicit exception chaining
- Return type:
- with_traceback()
Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.
- class smartpipeline.error.exceptions.SoftError(*args, **kwargs)[source]
A type of exception which usually only provokes skipping a stage for an item
- add_note()
Exception.add_note(note) – add a note to the exception
- get_exception()
Get the original exception (if any) that has generated this error, equivalent to the __cause__ attribute
- Return type:
Optional[BaseException]
- with_exception(exception)
Set the original exception (if any) that has generated this error, equivalent to explicit exception chaining
- Return type:
- with_traceback()
Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.
- class smartpipeline.error.exceptions.CriticalError(*args, **kwargs)[source]
A type of exception which usually provokes skipping the whole pipeline for an item
- add_note()
Exception.add_note(note) – add a note to the exception
- get_exception()
Get the original exception (if any) that has generated this error, equivalent to the __cause__ attribute
- Return type:
Optional[BaseException]
- with_exception(exception)
Set the original exception (if any) that has generated this error, equivalent to explicit exception chaining
- Return type:
- with_traceback()
Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.
- class smartpipeline.error.exceptions.RetryError(*args, **kwargs)[source]
A stage processing an input item may raise some Exception for which the stage must try to reprocess the item using an exponential backoff retry strategy. After all attempts fail, a single RetryError is attached to the item and that stage is skipped for it
- add_note()
Exception.add_note(note) – add a note to the exception
- get_exception()
Get the original exception (if any) that has generated this error, equivalent to the __cause__ attribute
- Return type:
Optional[BaseException]
- with_exception(exception)
Set the original exception (if any) that has generated this error, equivalent to explicit exception chaining
- Return type:
- with_traceback()
Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.
- class smartpipeline.error.exceptions.BatchOutputError(*args, **kwargs)[source]
A
stage.BatchStagereturned a number of items different from its input batch.This is a fatal contract error: the pipeline stops instead of retrying it or attaching a per-item soft/critical error.
- add_note()
Exception.add_note(note) – add a note to the exception
- get_exception()
Get the original exception (if any) that has generated this error, equivalent to the __cause__ attribute
- Return type:
Optional[BaseException]
- with_exception(exception)
Set the original exception (if any) that has generated this error, equivalent to explicit exception chaining
- Return type:
- with_traceback()
Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.
- class smartpipeline.error.handling.ErrorManager[source]
Basic error handling of a pipeline, principally manages
exceptions.SoftErrorandexceptions.CriticalErrortypes- check_critical_errors(item)[source]
Check the critical errors registered for an item
- Return type:
Optional[BaseException]
- handle(error, stage, item)[source]
Manage an error produced by a stage
- Parameters:
error (
Exception) – It can be a generic exception or an error fromexceptionsexplicitly raised by a stagestage (
AliveMixin) – Stage which raised the exception during processingitem (
Item[Any]) – Item which raised the exception when processed
- Return type:
Optional[CriticalError]- Returns:
If the handled error results to be critical return the generated
exceptions.CriticalError- Raises:
Exception – When a
ErrorManager.raise_on_critical_error()has been set and the error is critical
- property logger: Logger
Specific logger for this object
- no_skip_on_critical_error()[source]
Change default behaviour of the error manager on
exceptions.CriticalError: only skip the stage which raises it, like theexceptions.SoftError. Valid only ifErrorManager.raise_on_critical_error()is not set- Return type:
- on_end()
Called when the stage terminates, useful for executing closing operations (e.g. on files)
It is paired with a successful
on_start(), also when the start phase fails after this stage has started. Any exception raised here is fatal for the whole pipeline and is reported to the pipeline caller on termination instead of hanging the execution without feedback.- Return type:
Any
- on_start()
Called in the pipeline start phase, when
Pipeline.run()orPipeline.process_async()starts the stage, never byPipeline.build(). With multiprocessing concurrency the stage in the executor is a copy of the original, so by overriding this method one can initialize variables specifically for the copies; that is mandatory when they are not serializable.Any exception raised here is fatal for the whole pipeline and is reported to the pipeline caller (on run()/process_async()) instead of hanging the execution without feedback. on_end runs only for stages whose on_start completed.
- Return type:
Any
- raise_on_critical_error()[source]
Set the error manager so that if a
exceptions.CriticalError(or any un-managed exception) is encountered, it raises it “externally”, at the pipeline run level, killing the pipeline- Return type:
Helpers
Built-in helper sources and item subclasses shipped with the library.
- class smartpipeline.helpers.LocalFilesSource(dir_path, postfix='')[source]
Generate a special file item for any file found in a directory
- property is_stopped: bool
True if the source has called the stop event
- property logger: Logger
Specific logger for this object
- pop()[source]
Generate items for feeding a pipeline. Must be overridden for properly defining a source. Call
Source.stop()when item generation is ended- Return type:
Optional[Item[Dict[str,Any]]]- Returns:
The generated item, if None it is simply ignored (e.g. after calling
Source.stop())
- stop()
Declare the end item generation, this event will be spread through the pipeline
- class smartpipeline.helpers.FilePathItem(path, data=None)[source]
An item with a pointer to a file path
- add_critical_error(stage, exception)
This is called internally by the
error.handling.ErrorManagerwhen the exception is handled. Add aerror.exceptions.CriticalErrorgenerated in a stage (referenced by its name) for the item- Parameters:
exception (
Union[CriticalError,Exception]) – It can be aerror.exceptions.CriticalErrorinstance or any exception, which will be encapsulated in aerror.exceptions.CriticalError- Return type:
- add_soft_error(stage, exception)
This is called internally by the
error.handling.ErrorManagerwhen the exception is handled. Add anerror.exceptions.SoftErrorgenerated in a stage (referenced by its name) for the item.- Parameters:
exception (
Union[SoftError,Exception]) – It can be anerror.exceptions.SoftErrorinstance or any exception, which will be encapsulated in anerror.exceptions.SoftError- Return type:
- callback()
Call the function set with
Item.set_callback()- Return type:
Any
- property callback_id: str
Identifier of the callback set with
Item.set_callback().Unlike the function, it travels with the item when a stage worker gets a copy of it, so a pipeline can find the callback it kept in the caller’s process.
- critical_errors()
Iter over
error.exceptions.CriticalErrorinstances or any un-managed exception eventually generated in some stage processing- Return type:
Generator[CriticalError,None,None]
- property data: Dict[str, Any]
Access to the actual data contained in the item
- Returns:
The payload, by default a dictionary in which organizing data by fields (recommended to be JSON serializable). Build it with Item() (an empty Dict[str, Any]) or Item[Dict[str, Any]]({…}), and with Item[MyData](…) for a custom payload type
- data_snippet(max_size=100)
A short string representation of the
Item.data(recommended to override this method)- Parameters:
max_size (
int) – Maximum size of the string representation
- get_timing(stage_name)
Get the wall-clock time spent by a stage (referenced by its name) for processing the item.
It is not CPU time and excludes queue waiting; on threads it includes GIL waiting.
- Return type:
Optional[float]- Returns:
The time in seconds or None if the item has not been processed by the stage
- has_critical_errors()
True if the item has raised a
error.exceptions.CriticalErroror any un-managed exception in some stage processing- Return type:
bool
- has_soft_errors()
True if the item has raised an
error.exceptions.SoftErrorin some stage processing- Return type:
bool
- property id: Any
Get the unique identifier of the item. It is recommended to override this in order to properly compute it from the
Item.data
- property metadata: Dict[str, Any]
Get the actual metadata dictionary
- set_callback(fun)
Set a function to call after a successful asynchronous execution of a pipeline on the item (through
pipeline.Pipeline.process_async())The function runs in the process which submitted the item: stage workers get a copy of the item without it, and the pipeline calls it back through
Item.callback_id.- Return type:
str- Returns:
the
Item.callback_idof this item
- set_timing(stage_name, seconds)
Set the wall-clock time spent by a stage (referenced by its name) for processing the item.
The duration includes blocking work, retries and, on threads, time spent waiting for the GIL, so it is not CPU time. See the concurrency guide for how to read it.
- Return type:
Item[Dict[str,Any]]
- soft_errors()
Iter over
error.exceptions.SoftErrorinstances eventually generated in some stage processing- Return type:
Generator[SoftError,None,None]
- property timed_stages: KeysView[str]
Get the stage names for which the item has a process time