parallel.controller.scheduler

Module: parallel.controller.scheduler

Inheritance diagram for IPython.parallel.controller.scheduler:

The Python scheduler for rich scheduling.

The Pure ZMQ scheduler does not allow routing schemes other than LRU, nor does it check msg_id DAG dependencies. For those, a slightly slower Python Scheduler exists.

Authors:

• Min RK

Class

TaskScheduler

class IPython.parallel.controller.scheduler.TaskScheduler(**kwargs)

Bases: IPython.zmq.session.SessionFactory

Python TaskScheduler object.

This is the simplest object that supports msg_id based DAG dependencies. Only task msg_ids are checked, not msg_ids of jobs submitted via the MUX queue.

__init__(**kwargs)

add_job(idx)

Called after self.targets[idx] just got the job with header. Override with subclasses. The default ordering is simple LRU. The default loads are the number of outstanding jobs.

all_completed

An instance of a Python set.

all_done

An instance of a Python set.

all_failed

An instance of a Python set.

all_ids

An instance of a Python set.

audit_timeouts()

Audit all waiting tasks for expired timeouts.

auditor

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

blacklist

An instance of a Python dict.

classmethod class_config_section()

Get the config class config section

classmethod class_get_help(inst=None)

Get the help string for this class in ReST format.

If inst is given, it’s current trait values will be used in place of class defaults.

classmethod class_get_trait_help(trait, inst=None)

Get the help string for a single trait.

If inst is given, it’s current trait values will be used in place of the class default.

classmethod class_print_help(inst=None)

Get the help string for a single trait and print it.

classmethod class_trait_names(**metadata)

Get a list of all the names of this classes traits.

This method is just like the trait_names() method, but is unbound.

classmethod class_traits(**metadata)

Get a list of all the traits of this class.

This method is just like the traits() method, but is unbound.

The TraitTypes returned don’t know anything about the values that the various HasTrait’s instances are holding.

This follows the same algorithm as traits does and does not allow for any simple way of specifying merely that a metadata name exists, but has any value. This is because get_metadata returns None if a metadata key doesn’t exist.

client_stream

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

clients

An instance of a Python dict.

completed

An instance of a Python dict.

config

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

context

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

depending

An instance of a Python dict.

destinations

An instance of a Python dict.

dispatch_notification(msg)

dispatch register/unregister events.

dispatch_result(raw_msg)

dispatch method for result replies

dispatch_submission(raw_msg)

Dispatch job submission to appropriate handlers.

engine_stream

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

fail_unreachable(msg_id, why=<class 'IPython.parallel.error.ImpossibleDependency'>)

a task has become unreachable, send a reply with an ImpossibleDependency error.

failed

An instance of a Python dict.

finish_job(idx)

Called after self.targets[idx] just finished a job. Override with subclasses.

graph

An instance of a Python dict.

handle_result(idents, parent, raw_msg, success=True)

handle a real task result, either success or failure

handle_stranded_tasks(engine)

Deal with jobs resident in an engine that died.

handle_unmet_dependency(idents, parent)

handle an unmet dependency

hwm

An integer trait.

Longs that are unnecessary (<= sys.maxint) are cast to ints.

ident

A casting version of the byte string trait.

loads

An instance of a Python list.

log

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

logname

A trait for unicode strings.

loop

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

maybe_run(msg_id, raw_msg, targets, after, follow, timeout)

check location dependencies, and run if they are met.

mon_stream

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

notifier_stream

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

on_trait_change(handler, name=None, remove=False)

Setup a handler to be called when a trait changes.

This is used to setup dynamic notifications of trait changes.

Static handlers can be created by creating methods on a HasTraits subclass with the naming convention ‘_[traitname]_changed’. Thus, to create static handler for the trait ‘a’, create the method _a_changed(self, name, old, new) (fewer arguments can be used, see below).

Parameters :

handler : callable

A callable that is called when a trait changes. Its signature can be handler(), handler(name), handler(name, new) or handler(name, old, new).

name : list, str, None

If None, the handler will apply to all traits. If a list of str, handler will apply to all names in the list. If a str, the handler will apply just to that name.

remove : bool

If False (the default), then install the handler. If True then unintall it.

pending

An instance of a Python dict.

resume_receiving()

Resume accepting jobs.

retries

An instance of a Python dict.

save_unmet(msg_id, raw_msg, targets, after, follow, timeout)

Save a message for later submission when its dependencies are met.

scheme

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

scheme_name

An enum that whose value must be in a given sequence.

session

A trait whose value must be an instance of a specified class.

The value can also be an instance of a subclass of the specified class.

start()

stop_receiving()

Stop accepting jobs while there are no engines. Leave them in the ZMQ queue.

submit_task(msg_id, raw_msg, targets, follow, timeout, indices=None)

Submit a task to any of a subset of our targets.

targets

An instance of a Python list.

trait_metadata(traitname, key)

Get metadata values for trait by key.

trait_names(**metadata)

Get a list of all the names of this classes traits.

traits(**metadata)

Get a list of all the traits of this class.

The TraitTypes returned don’t know anything about the values that the various HasTrait’s instances are holding.

This follows the same algorithm as traits does and does not allow for any simple way of specifying merely that a metadata name exists, but has any value. This is because get_metadata returns None if a metadata key doesn’t exist.

update_config(config)

Fire the traits events when the config is updated.

update_graph(dep_id=None, success=True)

dep_id just finished. Update our dependency graph and submit any jobs that just became runable.

Called with dep_id=None to update entire graph for hwm, but without finishing a task.

Functions

IPython.parallel.controller.scheduler.launch_scheduler(in_addr, out_addr, mon_addr, not_addr, config=None, logname='root', log_url=None, loglevel=10, identity='task', in_thread=False)

IPython.parallel.controller.scheduler.leastload(loads)

Always choose the lowest load.

If the lowest load occurs more than once, the first occurance will be used. If loads has LRU ordering, this means the LRU of those with the lowest load is chosen.

IPython.parallel.controller.scheduler.logged(f)

IPython.parallel.controller.scheduler.lru(loads)

Always pick the front of the line.

The content of loads is ignored.

Assumes LRU ordering of loads, with oldest first.

IPython.parallel.controller.scheduler.plainrandom(loads)

Plain random pick.

IPython.parallel.controller.scheduler.twobin(loads)

Pick two at random, use the LRU of the two.

The content of loads is ignored.

Assumes LRU ordering of loads, with oldest first.

IPython.parallel.controller.scheduler.weighted(loads)

Pick two at random using inverse load as weight.

Return the less loaded of the two.