kernel.multiengineclient¶
Module: kernel.multiengineclient¶
Inheritance diagram for IPython.kernel.multiengineclient:
General Classes for IMultiEngine clients.
Classes¶
FullBlockingMultiEngineClient¶
- class IPython.kernel.multiengineclient.FullBlockingMultiEngineClient(smultiengine)¶
Bases: IPython.kernel.multiengineclient.InteractiveMultiEngineClient
A blocking client to the IMultiEngine controller interface.
This class allows users to use a set of engines for a parallel computation through the IMultiEngine interface. In this interface, each engine has a specific id (an int) that is used to refer to the engine, run code on it, etc.
- __init__(smultiengine)¶
- barrier(pendingResults)¶
Synchronize a set of PendingResults.
This method is a synchronization primitive that waits for a set of PendingResult objects to complete. More specifically, barier does the following.
- The `PendingResult`s are sorted by result_id.
- The get_result method is called for each PendingResult sequentially with block=True.
- If a PendingResult gets a result that is an exception, it is trapped and can be re-raised later by calling get_result again.
- The `PendingResult`s are flushed from the controller.
After barrier has been called on a PendingResult, its results can be retrieved by calling get_result again or accesing the r attribute of the instance.
- benchmark(push_size=10000)¶
Run performance benchmarks for the current IPython cluster.
This method tests both the latency of sending command and data to the engines as well as the throughput of sending large objects to the engines using push. The latency is measured by having one or more engines execute the command ‘pass’. The throughput is measure by sending an NumPy array of size push_size to one or more engines.
These benchmarks will vary widely on different hardware and networks and thus can be used to get an idea of the performance characteristics of a particular configuration of an IPython controller and engines.
This function is not testable within our current testing framework.
- clear_pending_results()¶
Clear all pending deferreds/results from the controller.
For each PendingResult that is created by this client, the controller holds on to the result for that PendingResult. This can be a problem if there are a large number of PendingResult objects that are created.
Once the result of the PendingResult has been retrieved, the result is removed from the controller, but if a user doesn’t get a result ( they just ignore the PendingResult) the result is kept forever on the controller. This method allows the user to clear out all un-retrieved results on the controller.
- clear_properties(targets=None, block=None)¶
- clear_queue(targets=None, block=None)¶
Clear out the controller’s queue for an engine.
The controller maintains a queue for each engine. This clear it out.
Parameters: - targets : id or list of ids
The engine to use for the execution
- block : boolean
If False, this method will return the actual result. If False, a PendingResult is returned which can be used to get the result at a later time.
- del_properties(keys, targets=None, block=None)¶
- execute(lines, targets=None, block=None)¶
Execute code on a set of engines.
Parameters: - lines : str
The Python code to execute as a string
- targets : id or list of ids
The engine to use for the execution
- block : boolean
If False, this method will return the actual result. If False, a PendingResult is returned which can be used to get the result at a later time.
- flush()¶
Clear all pending deferreds/results from the controller.
For each PendingResult that is created by this client, the controller holds on to the result for that PendingResult. This can be a problem if there are a large number of PendingResult objects that are created.
Once the result of the PendingResult has been retrieved, the result is removed from the controller, but if a user doesn’t get a result ( they just ignore the PendingResult) the result is kept forever on the controller. This method allows the user to clear out all un-retrieved results on the controller.
- gather(key, dist='b', targets=None, block=None)¶
- Gather a partitioned sequence on a set of engines as a single local seq.
- get_ids()¶
- Returns the ids of currently registered engines.
- get_pending_deferred(deferredID, block)¶
- get_properties(keys=None, targets=None, block=None)¶
- get_result(i=None, targets=None, block=None)¶
Get a previous result.
When code is executed in an engine, a dict is created and returned. This method retrieves that dict for previous commands.
Parameters: - i : int
The number of the result to get
- targets : id or list of ids
The engine to use for the execution
- block : boolean
If False, this method will return the actual result. If False, a PendingResult is returned which can be used to get the result at a later time.
- has_properties(keys, targets=None, block=None)¶
- keys(targets=None, block=None)¶
Get a list of all the variables in an engine’s namespace.
Parameters: - targets : id or list of ids
The engine to use for the execution
- block : boolean
If False, this method will return the actual result. If False, a PendingResult is returned which can be used to get the result at a later time.
- kill(controller=False, targets=None, block=None)¶
Kill the engines and controller.
This method is used to stop the engine and controller by calling reactor.stop.
Parameters: - controller : boolean
If True, kill the engines and controller. If False, just the engines
- targets : id or list of ids
The engine to use for the execution
- block : boolean
If False, this method will return the actual result. If False, a PendingResult is returned which can be used to get the result at a later time.
- map(func, *sequences)¶
A parallel version of Python’s builtin map function.
This method applies a function to sequences of arguments. It follows the same syntax as the builtin map.
This method creates a mapper objects by calling self.mapper with no arguments and then uses that mapper to do the mapping. See the documentation of mapper for more details.
- mapper(dist='b', targets='all', block=None)¶
Create a mapper object that has a map method.
This method returns an object that implements the IMapper interface. This method is a factory that is used to control how the map happens.
Parameters: - dist : str
What decomposition to use, ‘b’ is the only one supported currently
- targets : str, int, sequence of ints
Which engines to use for the map
- block : boolean
Should calls to map block or not
- parallel(dist='b', targets=None, block=None)¶
A decorator that turns a function into a parallel function.
This can be used as:
@parallel() def f(x, y)
...
f(range(10), range(10))
This causes f(0,0), f(1,1), ... to be called in parallel.
Parameters: - dist : str
What decomposition to use, ‘b’ is the only one supported currently
- targets : str, int, sequence of ints
Which engines to use for the map
- block : boolean
Should calls to map block or not
- pull(keys, targets=None, block=None)¶
Pull Python objects by key out of engines namespaces.
Parameters: - keys : str or list of str
The names of the variables to be pulled
- targets : id or list of ids
The engine to use for the execution
- block : boolean
If False, this method will return the actual result. If False, a PendingResult is returned which can be used to get the result at a later time.
- pull_function(keys, targets=None, block=None)¶
Pull a Python function from an engine.
This method is used to pull a Python function from an engine. Closures are not supported.
Parameters: - keys : str or list of str
The names of the functions to be pulled
- targets : id or list of ids
The engine to use for the execution
- block : boolean
If False, this method will return the actual result. If False, a PendingResult is returned which can be used to get the result at a later time.
- pull_serialized(keys, targets=None, block=None)¶
- push(namespace, targets=None, block=None)¶
Push a dictionary of keys and values to engines namespace.
Each engine has a persistent namespace. This method is used to push Python objects into that namespace.
The objects in the namespace must be pickleable.
Parameters: - namespace : dict
A dict that contains Python objects to be injected into the engine persistent namespace.
- targets : id or list of ids
The engine to use for the execution
- block : boolean
If False, this method will return the actual result. If False, a PendingResult is returned which can be used to get the result at a later time.
- push_function(namespace, targets=None, block=None)¶
Push a Python function to an engine.
This method is used to push a Python function to an engine. This method can then be used in code on the engines. Closures are not supported.
Parameters: - namespace : dict
A dict whose values are the functions to be pushed. The keys give that names that the function will appear as in the engines namespace.
- targets : id or list of ids
The engine to use for the execution
- block : boolean
If False, this method will return the actual result. If False, a PendingResult is returned which can be used to get the result at a later time.
- push_serialized(namespace, targets=None, block=None)¶
- queue_status(targets=None, block=None)¶
Get the status of an engines queue.
Parameters: - targets : id or list of ids
The engine to use for the execution
- block : boolean
If False, this method will return the actual result. If False, a PendingResult is returned which can be used to get the result at a later time.
- raw_map(func, seq, dist='b', targets=None, block=None)¶
A parallelized version of Python’s builtin map.
This has a slightly different syntax than the builtin map. This is needed because we need to have keyword arguments and thus can’t use *args to capture all the sequences. Instead, they must be passed in a list or tuple.
raw_map(func, seqs) -> map(func, seqs[0], seqs[1], ...)
Most users will want to use parallel functions or the mapper and map methods for an API that follows that of the builtin map.
- reset(targets=None, block=None)¶
Reset an engine.
This method clears out the namespace of an engine.
Parameters: - targets : id or list of ids
The engine to use for the execution
- block : boolean
If False, this method will return the actual result. If False, a PendingResult is returned which can be used to get the result at a later time.
- run(filename, targets=None, block=None)¶
Run a Python code in a file on the engines.
Parameters: - filename : str
The name of the local file to run
- targets : id or list of ids
The engine to use for the execution
- block : boolean
If False, this method will return the actual result. If False, a PendingResult is returned which can be used to get the result at a later time.
- scatter(key, seq, dist='b', flatten=False, targets=None, block=None)¶
- Partition a Python sequence and send the partitions to a set of engines.
- set_properties(properties, targets=None, block=None)¶
- zip_pull(keys, targets=None, block=None)¶
IPendingResult¶
- class IPython.kernel.multiengineclient.IPendingResult(name, bases=(), attrs=None, __doc__=None, __module__=None)¶
Bases: zope.interface.Interface
A representation of a result that is pending.
This class is similar to Twisted’s Deferred object, but is designed to be used in a synchronous context.
- static __init__(name, bases=(), attrs=None, __doc__=None, __module__=None)¶
InteractiveMultiEngineClient¶
- class IPython.kernel.multiengineclient.InteractiveMultiEngineClient¶
Bases: object
A mixin class that add a few methods to a multiengine client.
The methods in this mixin class are designed for interactive usage.
- __init__()¶
- x.__init__(...) initializes x; see x.__class__.__doc__ for signature
- activate()¶
Make this MultiEngineClient active for parallel magic commands.
IPython has a magic command syntax to work with MultiEngineClient objects. In a given IPython session there is a single active one. While there can be many MultiEngineClient created and used by the user, there is only one active one. The active MultiEngineClient is used whenever the magic commands %px and %autopx are used.
The activate() method is called on a given MultiEngineClient to make it active. Once this has been done, the magic commands can be used.
- findsource_file(f)¶
- findsource_ipython(f)¶
PendingResult¶
- class IPython.kernel.multiengineclient.PendingResult(client, result_id)¶
Bases: object
A representation of a result that is not yet ready.
A user should not create a PendingResult instance by hand.
Methods:
- get_result
- add_callback
Properties:
- r
- __init__(client, result_id)¶
Create a PendingResult with a result_id and a client instance.
The client should implement _getPendingResult(result_id, block).
- add_callback(f, *args, **kwargs)¶
Add a callback that is called with the result.
If the original result is result, adding a callback will cause f(result, *args, **kwargs) to be returned instead. If multiple callbacks are registered, they are chained together: the result of one is passed to the next and so on.
Unlike Twisted’s Deferred object, there is no errback chain. Thus any exception raised will not be caught and handled. User must catch these by hand when calling get_result.
- get_result(default=None, block=True)¶
Get a result that is pending.
This method will connect to an IMultiEngine adapted controller and see if the result is ready. If the action triggers an exception raise it and record it. This method records the result/exception once it is retrieved. Calling get_result again will get this cached result or will re-raise the exception. The .r attribute is a property that calls get_result with block=True.
Parameters: - default
The value to return if the result is not ready.
- block : boolean
Should I block for the result.
Returns: The actual result or the default value.
- r¶