kernel.scripts.ipcluster

Module: kernel.scripts.ipcluster

Inheritance diagram for IPython.kernel.scripts.ipcluster:

Start an IPython cluster = (controller + engines).

Classes

BatchEngineSet

class IPython.kernel.scripts.ipcluster.BatchEngineSet(template_file, queue, **kwargs)

Bases: object

__init__()

default_template = ''

delete_command = ''

handle_error()

job_array_regexp = ''

job_array_template = ''

job_id_regexp = ''

kill()

name = ''

parse_job_id()

queue_regexp = ''

queue_template = ''

start()

submit_command = ''

ControllerLauncher

class IPython.kernel.scripts.ipcluster.ControllerLauncher(extra_args=None)

Bases: IPython.kernel.scripts.ipcluster.ProcessLauncher

__init__()

EngineLauncher

class IPython.kernel.scripts.ipcluster.EngineLauncher(extra_args=None)

Bases: IPython.kernel.scripts.ipcluster.ProcessLauncher

__init__()

LSFEngineSet

class IPython.kernel.scripts.ipcluster.LSFEngineSet(template_file, queue, **kwargs)

Bases: IPython.kernel.scripts.ipcluster.PBSEngineSet

__init__()

bsub_wrapper = '#!/bin/sh\nbsub < $1\n'

default_template = '#!/bin/sh\n#BSUB -J ipengine[1-%d]\neid=$(($LSB_JOBINDEX - 1))\nipengine --logfile=ipengine${eid}.log\n'

delete_command = 'bkill'

job_array_regexp = '#BSUB[ \t]-J+\\w+\\[\\d+-\\d+\\]'

job_array_template = '#BSUB -J ipengine[1-%d]'

name = 'LSF'

queue_regexp = '#BSUB[ \t]+-q[ \t]+\\w+'

queue_template = '#BSUB -q %s'

submit_command = 'bsub'

LauncherProcessProtocol

class IPython.kernel.scripts.ipcluster.LauncherProcessProtocol(process_launcher)

Bases: twisted.internet.protocol.ProcessProtocol

A ProcessProtocol to go with the ProcessLauncher.

__init__()

connectionMade()

errReceived()

outReceived()

processEnded()

LocalEngineSet

class IPython.kernel.scripts.ipcluster.LocalEngineSet(extra_args=None)

Bases: object

__init__()

interrupt_then_kill()

signal()

start()

PBSEngineSet

class IPython.kernel.scripts.ipcluster.PBSEngineSet(template_file, queue, **kwargs)

Bases: IPython.kernel.scripts.ipcluster.BatchEngineSet

__init__()

default_template = '#!/bin/sh\n#PBS -V\n#PBS -t 1-%d\n#PBS -N ipengine\neid=$(($PBS_ARRAYID - 1))\nipengine --logfile=ipengine${eid}.log\n'

delete_command = 'qdel'

job_array_regexp = '#PBS[ \t]+-t[ \t]+\\d+'

job_array_template = '#PBS -t 1-%d'

job_id_regexp = '\\d+'

name = 'PBS'

queue_regexp = '#PBS[ \t]+-q[ \t]+\\w+'

queue_template = '#PBS -q %s'

submit_command = 'qsub'

ProcessLauncher

class IPython.kernel.scripts.ipcluster.ProcessLauncher(cmd_and_args)

Bases: object

Start and stop an external process in an asynchronous manner.

Currently this uses deferreds to notify other parties of process state changes. This is an awkward design and should be moved to using a formal NotificationCenter.

__init__()

fire_start_deferred()

fire_stop_deferred()

get_stop_deferred()

interrupt_then_kill()

running

signal()

Send a signal to the process.

The argument sig can be (‘KILL’,’INT’, etc.) or any signal number.

start()

ProcessStateError

class IPython.kernel.scripts.ipcluster.ProcessStateError

Bases: exceptions.Exception

__init__()

x.__init__(...) initializes x; see x.__class__.__doc__ for signature

SGEEngineSet

class IPython.kernel.scripts.ipcluster.SGEEngineSet(template_file, queue, **kwargs)

Bases: IPython.kernel.scripts.ipcluster.PBSEngineSet

__init__()

default_template = '#$ -V\n#$ -S /bin/sh\n#$ -t 1-%d\n#$ -N ipengine\neid=$(($SGE_TASK_ID - 1))\nipengine --logfile=ipengine${eid}.log\n'

job_array_regexp = '#\\$[ \t]+-t[ \t]+\\d+'

job_array_template = '#$ -t 1-%d'

name = 'SGE'

queue_regexp = '#\\$[ \t]+-q[ \t]+\\w+'

queue_template = '#$ -q %s'

SSHEngineSet

class IPython.kernel.scripts.ipcluster.SSHEngineSet(engine_hosts, sshx=None, copyenvs=None, ipengine='ipengine')

Bases: object

__init__()

Start a controller on localhost and engines using ssh.

The engine_hosts argument is a dict with hostnames as keys and the number of engine (int) as values. sshx is the name of a local file that will be used to run remote commands. This file is used to setup the environment properly.

engine_killer_template = "#!/bin/sh\nps -fu `whoami` | grep '[i]pengine' | awk '{print $2}' | xargs kill -TERM\n"

kill()

sshx_template_prefix = '#!/bin/sh\n'

sshx_template_suffix = '"$@" > /dev/null 2>&1 &\necho $!\n'

start()

UnknownStatus

class IPython.kernel.scripts.ipcluster.UnknownStatus

Bases: exceptions.Exception

__init__()

x.__init__(...) initializes x; see x.__class__.__doc__ for signature

Functions

IPython.kernel.scripts.ipcluster.check_reuse()

Check to see if we should try to resuse FURL files.

IPython.kernel.scripts.ipcluster.check_security()

Check to see if we should run with SSL support.

IPython.kernel.scripts.ipcluster.escape_strings()

IPython.kernel.scripts.ipcluster.get_args()

IPython.kernel.scripts.ipcluster.main()

IPython.kernel.scripts.ipcluster.main_local()

IPython.kernel.scripts.ipcluster.main_lsf()

IPython.kernel.scripts.ipcluster.main_mpi()

IPython.kernel.scripts.ipcluster.main_pbs()

IPython.kernel.scripts.ipcluster.main_sge()

IPython.kernel.scripts.ipcluster.main_ssh()

Start a controller on localhost and engines using ssh.

Your clusterfile should look like:

send_furl = False # True, if you want 
engines = {
    'engine_host1' : engine_count, 
    'engine_host2' : engine_count2
}