`mt.base.bg_invoke`

Wrapper around threading for running things in background.

Functions 

parallelise(): Embarrasingly parallelises to excecute many jobs with a limited number of threads.
bg_run_proc(): Runs a procedure in a background thread.

mt.base.bg_invoke.parallelise(func, num_jobs, *fn_args, num_threads=None, bg_exception='raise', logger=None, pass_logger=False, **fn_kwargs)

Embarrasingly parallelises to excecute many jobs with a limited number of threads.

Parameters:

func (function) – a function of the form def func(job_id, *args, **kwargs) that can return something
num_jobs (int) – number of jobs to execute, must be positive integer
num_threads (int, optional) – number of threads to be used. If not specified, then 80% of available CPUs are used
bg_exception ({'raise', 'warn'}) – whether to re-rasise a BgException raised by a thread or to suppress all of them and warn instead, in which case logger must not be None
logger (IndentedLoggerAdaptor, optional) – for logging purposes
pass_logger (bool) – whether or not to include logger to func_kwargs if it does not exist in func_kwargs so the function can use the same logger as parallelise()
fn_args (list) – list of postional arguments for the function
fn_kwargs (dict) – list of keyword arguments for the function

Returns:

a list of num_jobs elements, each corresponding to the returning object of the function for a given job id

Return type:

list

Raises:

BgException – an exception raised by a background thread

Notes

use this function instead of joblib if you want to integrate with mt.logg and BgException better

mt.base.bg_invoke.bg_run_proc(proc, *args, **kwargs)

Runs a procedure in a background thread.

Parameters:

proc (function) – procedure (a function returning None) to be executed in a background thread. Any Exception raised during the execution will be logged as a warning
args (list) – positional arguments to be passed as-is to the procedure
kwargs (dict) – keyword arguments to be passed as-is to the procedure

digraph inheritance43e542557f { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "BgInvoke" [URL="#mt.base.bg_invoke.BgInvoke",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Thin wrapper around threading.Thread to run `target(*args, **kwargs)` in background."]; }

is_running(): Returns whether the invocation is still running.

class mt.base.bg_invoke.BgThread(func)

Thin wrapper around threading.Thread to run func(*args, **kwargs) in background.

Once invoked via invoke(), the thread waits until the last function call has finished, then invokes the function with new arguments in background. The user can do other things. They should invoke is_running() to determine when the function invocation is completed. They can also use attribute result to wait and get the result.

If an exception was raised, self.result would contain the exception and other useful information.

Parameters:: func (function) – a function to be wrapped

result

the current result if it has not been consumed. Otherwise blocks until the new result becomes available.

Type:: object

Notes

BgThread differs from BgInvoke in that the function can be invoked many times.

Examples

>>> def slow_count(nbTimes=100000000):
...     cnt = 0
...     for i in range(nbTimes):
...         cnt = cnt+1
...     return cnt
...
>>> from mt.base.bg_invoke import BgThread
>>> from time import sleep
>>> a = BgThread(slow_count)
>>> a.invoke(nbTimes=100000000)
>>> while a.is_running():
...     sleep(0.5)
...
>>> print(a.result)
100000000
>>> a.invoke(nbTimes=100000)
>>> print(a.result)
100000

Inheritance

digraph inheritanceaef057c635 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "BgThread" [URL="#mt.base.bg_invoke.BgThread",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Thin wrapper around threading.Thread to run `func(*args, **kwargs)` in background."]; "Thread" -> "BgThread" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Thread" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class that represents a thread of control."]; }

close(): Waits for the current task to be done and closes the thread.

invoke(*args, **kwargs)

Invokes the function in a graceful way.

This function blocks until the last function arguments have been consumed and then signals the function to be invoked again. It returns immediately.

Parameters:

args (list) – positional arguments of the function
kwargs (dict) – keyword arguments of the function

is_running(): Returns whether or not the function is running.

property result: The result after waiting for the function call to finish.

run(): The wrapping code to be run in the background thread. Do not call.

Exceptions 

BgException: Common base class for all non-exit exceptions.

exception mt.base.bg_invoke.BgException(message, exc_info): Inheritance
digraph inheritanced19810d2a1 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "BgException" [URL="#mt.base.bg_invoke.BgException",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; }

Variables 

bg_proc_manager

mt.base.bg_invoke.bg_proc_manager

A manager to run procedures in background threads.

A procedure is a function that returns None. If it an error is raised while executing the procedure, it will be reported as a warning in the logger (optional).

Parameters:: logger (logging.Logger or equivalent) – logger for catching the exceptions as warnings

<mt.base.bg_invoke.BgProcManager object at 0x7f2f44e2e3d0>

mt.base.bg_invoke

Functions

Classes

Exceptions

Variables

`mt.base.bg_invoke`

Functions 

Classes 

Exceptions 

Variables 