genutils

Module: genutils

Inheritance diagram for IPython.genutils:

General purpose utilities.

This is a grab-bag of stuff I find useful in most programs I write. Some of these things are also convenient when working at the command line.

Classes

Error

class IPython.genutils.Error

Bases: exceptions.Exception

Base class for exceptions in this module.

__init__()

x.__init__(...) initializes x; see x.__class__.__doc__ for signature

EvalDict

class IPython.genutils.EvalDict

Emulate a dict which evaluates its contents in the caller’s frame.

Usage: >>> number = 19

>>> text = "python"

>>> print "%(text.capitalize())s %(number/9.0).1f rules!" % EvalDict()
Python 2.1 rules!

HomeDirError

class IPython.genutils.HomeDirError

Bases: IPython.genutils.Error

__init__()

x.__init__(...) initializes x; see x.__class__.__doc__ for signature

IOStream

class IPython.genutils.IOStream(stream, fallback)

__init__()

close()

write()

IOTerm

class IPython.genutils.IOTerm(cin=None, cout=None, cerr=None)

Term holds the file or file-like objects for handling I/O operations.

These are normally just sys.stdin, sys.stdout and sys.stderr but for Windows they can can replaced to allow editing the strings before they are displayed.

__init__()

LSString

class IPython.genutils.LSString

Bases: str

String derivative with a special access attributes.

These are normal strings, but with the special attributes:

.l (or .list) : value as list (split on newlines). .n (or .nlstr): original value (the string itself). .s (or .spstr): value as whitespace-separated string. .p (or .paths): list of path objects

Any values which require transformations are computed only once and cached.

Such strings are very useful to efficiently interact with the shell, which typically only understands whitespace-separated options for commands.

__init__()

x.__init__(...) initializes x; see x.__class__.__doc__ for signature

get_list()

get_nlstr()

get_paths()

get_spstr()

l

list

n

nlstr

p

paths

s

spstr

NLprinter

class IPython.genutils.NLprinter

Print an arbitrarily nested list, indicating index numbers.

An instance of this class called nlprint is available and callable as a function.

nlprint(list,indent=’ ‘,sep=’: ‘) -> prints indenting each level by ‘indent’ and using ‘sep’ to separate the index from the value.

__init__()

NotGiven

class IPython.genutils.NotGiven

SList

class IPython.genutils.SList

Bases: list

List derivative with a special access attributes.

These are normal lists, but with the special attributes:

.l (or .list) : value as list (the list itself). .n (or .nlstr): value as a string, joined on newlines. .s (or .spstr): value as a string, joined on spaces. .p (or .paths): list of path objects

Any values which require transformations are computed only once and cached.

__init__()

x.__init__(...) initializes x; see x.__class__.__doc__ for signature

fields()

Collect whitespace-separated fields from string list

Allows quick awk-like usage of string lists.

Example data (in var a, created by ‘a = !ls -l’)::

-rwxrwxrwx

1 ville None 18 Dec 14 2006 ChangeLog

drwxrwxrwx+ 6 ville None 0 Oct 24 18:05 IPython

a.fields(0) is [‘-rwxrwxrwx’, ‘drwxrwxrwx+’] a.fields(1,0) is [‘1 -rwxrwxrwx’, ‘6 drwxrwxrwx+’] (note the joining by space). a.fields(-1) is [‘ChangeLog’, ‘IPython’]

IndexErrors are ignored.

Without args, fields() just split()’s the strings.

get_list()

get_nlstr()

get_paths()

get_spstr()

grep()

Return all strings matching ‘pattern’ (a regex or callable)

This is case-insensitive. If prune is true, return all items NOT matching the pattern.

If field is specified, the match must occur in the specified whitespace-separated field.

Examples:

a.grep( lambda x: x.startswith('C') )
a.grep('Cha.*log', prune=1)
a.grep('chm', field=-1)

l

list

n

nlstr

p

paths

s

sort()

sort by specified fields (see fields())

Example::

a.sort(1, nums = True)

Sorts a by second field, in numerical order (so that 21 > 3)

spstr

SystemExec

class IPython.genutils.SystemExec(verbose=0, debug=0, header='', split=0)

Access the system and getoutput functions through a stateful interface.

Note: here we refer to the system and getoutput functions from this library, not the ones from the standard python library.

This class offers the system and getoutput functions as methods, but the verbose, debug and header parameters can be set for the instance (at creation time or later) so that they don’t need to be specified on each call.

For efficiency reasons, there’s no way to override the parameters on a per-call basis other than by setting instance attributes. If you need local overrides, it’s best to directly call system() or getoutput().

The following names are provided as alternate options:

• xsys: alias to system
• bq: alias to getoutput

An instance can then be created as: >>> sysexec = SystemExec(verbose=1,debug=0,header=’Calling: ‘)

__init__()

Specify the instance’s values for verbose, debug and header.

bq()

Stateful interface to getoutput().

getoutput()

Stateful interface to getoutput().

getoutputerror()

Stateful interface to getoutputerror().

shell()

Stateful interface to shell(), with the same keyword parameters.

system()

Stateful interface to system(), with the same keyword parameters.

xsys()

Stateful interface to system(), with the same keyword parameters.

Functions

IPython.genutils.abbrev_cwd()

Return abbreviated version of cwd, e.g. d:mydir

IPython.genutils.all_belong()

Check whether a list of items ALL appear in a given list of options.

Returns a single 1 or 0 value.

IPython.genutils.arg_split()

Split a command line’s arguments in a shell-like manner.

This is a modified version of the standard library’s shlex.split() function, but with a default of posix=False for splitting, so that quotes in inputs are respected.

IPython.genutils.ask_yes_no()

Asks a question and returns a boolean (y/n) answer.

If default is given (one of ‘y’,’n’), it is used if the user input is empty. Otherwise the question is repeated until an answer is given.

An EOF is treated as the default answer. If there is no default, an exception is raised to prevent infinite loops.

Valid answers are: y/yes/n/no (match is not case sensitive).

IPython.genutils.belong()

Check whether a list of items appear in a given list of options.

Returns a list of 1 and 0, one for each candidate given.

IPython.genutils.chop()

Chop a sequence into chunks of the given size.

IPython.genutils.debugx()

Print the value of an expression from the caller’s frame.

Takes an expression, evaluates it in the caller’s frame and prints both the given expression and the resulting value (as well as a debug mark indicating the name of the calling function. The input must be of a form suitable for eval().

An optional message can be passed, which will be prepended to the printed expr->value pair.

IPython.genutils.dgrep()

Return grep() on dir()+dir(__builtins__).

A very common use of grep() when working interactively.

IPython.genutils.dhook_wrap()

Wrap a function call in a sys.displayhook controller.

Returns a wrapper around func which calls func, with all its arguments and keywords unmodified, using the default sys.displayhook. Since IPython modifies sys.displayhook, it breaks the behavior of certain systems that rely on the default behavior, notably doctest.

IPython.genutils.dir2()

dir2(obj) -> list of strings

Extended version of the Python builtin dir(), which does a few extra checks, and supports common objects with unusual internals that confuse dir(), such as Traits and PyCrust.

This version is guaranteed to return only a list of true strings, whereas dir() returns anything that objects inject into themselves, even if they are later not really valid for attribute access (many extension libraries have such bugs).

IPython.genutils.doctest_reload()

Properly reload doctest to reuse it interactively.

This routine:

• imports doctest but does NOT reload it (see below).
• resets its global ‘master’ attribute to None, so that multiple uses of

the module interactively don’t produce cumulative reports.

• Monkeypatches its core test runner method to protect it from IPython’s

modified displayhook. Doctest expects the default displayhook behavior deep down, so our modification breaks it completely. For this reason, a hard monkeypatch seems like a reasonable solution rather than asking users to manually use a different doctest runner when under IPython.

Notes

This function used to reload doctest, but this has been disabled because reloading doctest unconditionally can cause massive breakage of other doctest-dependent modules already in memory, such as those for IPython’s own testing system. The name wasn’t changed to avoid breaking people’s code, but the reload call isn’t actually made anymore.

IPython.genutils.error()

Equivalent to warn(msg,level=3).

IPython.genutils.esc_quotes()

Return the input string with single and double quotes escaped out

IPython.genutils.fatal()

Equivalent to warn(msg,exit_val=exit_val,level=4).

IPython.genutils.file_read()

Read a file and close it. Returns the file source.

IPython.genutils.file_readlines()

Read a file and close it. Returns the file source using readlines().

IPython.genutils.filefind()

Return the given filename either in the current directory, if it exists, or in a specified list of directories.

~ expansion is done on all file and directory names.

Upon an unsuccessful search, raise an IOError exception.

IPython.genutils.flag_calls()

Wrap a function to detect and flag when it gets called.

This is a decorator which takes a function and wraps it in a function with a ‘called’ attribute. wrapper.called is initialized to False.

The wrapper.called attribute is set to False right before each call to the wrapped function, so if the call fails it remains False. After the call completes, wrapper.called is set to True and the output is returned.

Testing for truth in wrapper.called allows you to determine if a call to func() was attempted and succeeded.

IPython.genutils.flatten()

Flatten a list of lists (NOT recursive, only works for 2d lists).

IPython.genutils.get_class_members()

IPython.genutils.get_home_dir()

Return the closest possible equivalent to a ‘home’ directory.

We first try $HOME. Absent that, on NT it’s $HOMEDRIVE$HOMEPATH.

Currently only Posix and NT are implemented, a HomeDirError exception is raised for all other OSes.

IPython.genutils.get_ipython_dir()

Get the IPython directory for this platform and user.

This uses the logic in get_home_dir to find the home directory and the adds either .ipython or _ipython to the end of the path.

IPython.genutils.get_log_dir()

Get the IPython log directory.

If the log directory does not exist, it is created.

IPython.genutils.get_pager_cmd()

Return a pager command.

Makes some attempts at finding an OS-correct one.

IPython.genutils.get_pager_start()

Return the string for paging files with an offset.

This is the ‘+N’ argument which less and more (under Unix) accept.

IPython.genutils.get_py_filename()

Return a valid python filename in the current directory.

If the given name is not a file, it adds ‘.py’ and searches again. Raises IOError with an informative message if the file isn’t found.

IPython.genutils.get_security_dir()

Get the IPython security directory.

This directory is the default location for all security related files, including SSL/TLS certificates and FURL files.

If the directory does not exist, it is created with 0700 permissions. If it exists, permissions are set to 0700.

IPython.genutils.get_slice()

Get a slice of a sequence with variable step. Specify start,stop,step.

IPython.genutils.getattr_list()

getattr_list(obj,alist[, default]) -> attribute list.

Get a list of named attributes for an object. When a default argument is given, it is returned when the attribute doesn’t exist; without it, an exception is raised in that case.

Note that alist can be given as a string, which will be automatically split into a list on whitespace. If given as a list, it must be a list of strings (the variable names themselves), not of variables.

IPython.genutils.getoutput()

Dummy substitute for perl’s backquotes.

Executes a command and returns the output.

Accepts the same arguments as system(), plus:

• split(0): if true, the output is returned as a list split on newlines.

Note: a stateful version of this function is available through the SystemExec class.

This is pretty much deprecated and rarely used, genutils.getoutputerror may be what you need.

IPython.genutils.getoutputerror()

Return (standard output,standard error) of executing cmd in a shell.

Accepts the same arguments as system(), plus:

• split(0): if true, each of stdout/err is returned as a list split on

newlines.

Note: a stateful version of this function is available through the SystemExec class.

IPython.genutils.grep()

Simple minded grep-like function. grep(pat,list) returns occurrences of pat in list, None on failure.

It only does simple string matching, with no support for regexps. Use the option case=0 for case-insensitive matching.

IPython.genutils.idgrep()

Case-insensitive dgrep()

IPython.genutils.igrep()

Synonym for case-insensitive grep.

IPython.genutils.import_fail_info()

Inform load failure for a module.

IPython.genutils.indent()

Indent a string a given number of spaces or tabstops.

indent(str,nspaces=4,ntabs=0) -> indent str by ntabs+nspaces.

IPython.genutils.info()

Equivalent to warn(msg,level=1).

IPython.genutils.list2dict()

Takes a list of (key,value) pairs and turns it into a dict.

IPython.genutils.list2dict2()

Takes a list and turns it into a dict. Much slower than list2dict, but more versatile. This version can take lists with sublists of arbitrary length (including sclars).

IPython.genutils.list_strings()

Always return a list of strings, given a string or list of strings as input.

IPython.genutils.make_quoted_expr()

Return string s in appropriate quotes, using raw string if possible.

XXX - example removed because it caused encoding errors in documentation generation. We need a new example that doesn’t contain invalid chars.

Note the use of raw string and padding at the end to allow trailing backslash.

IPython.genutils.map_method()

map_method(method,object_list,*args,**kw) -> list

Return a list of the results of applying the methods to the items of the argument sequence(s). If more than one sequence is given, the method is called with an argument list consisting of the corresponding item of each sequence. All sequences must be of the same length.

Keyword arguments are passed verbatim to all objects called.

This is Python code, so it’s not nearly as fast as the builtin map().

IPython.genutils.marquee()

Return the input string centered in a ‘marquee’.

IPython.genutils.mutex_opts()

Check for presence of mutually exclusive keys in a dict.

Call: mutex_opts(dict,[[op1a,op1b],[op2a,op2b]...]

IPython.genutils.native_line_ends()

Convert (in-place) a file to line-ends native to the current OS.

If the optional backup argument is given as false, no backup of the original file is left.

IPython.genutils.num_cpus()

Return the effective number of CPUs in the system as an integer.

This cross-platform function makes an attempt at finding the total number of available CPUs in the system, as returned by various underlying system and python calls.

If it can’t find a sensible answer, it returns 1 (though an error may make it return a large positive number that’s actually incorrect).

IPython.genutils.optstr2types()

Convert a string of option names to a dict of type mappings.

optstr2types(str) -> {None:’string_opts’,int:’int_opts’,float:’float_opts’}

This is used to get the types of all the options in a string formatted with the conventions of DPyGetOpt. The ‘type’ None is used for options which are strings (they need no further conversion). This function’s main use is to get a typemap for use with read_dict().

IPython.genutils.page()

Print a string, piping through a pager after a certain length.

The screen_lines parameter specifies the number of usable lines of your terminal screen (total lines minus lines you need to reserve to show other information).

If you set screen_lines to a number <=0, page() will try to auto-determine your screen size and will only use up to (screen_size+screen_lines) for printing, paging after that. That is, if you want auto-detection but need to reserve the bottom 3 lines of the screen, use screen_lines = -3, and for auto-detection without any lines reserved simply use screen_lines = 0.

If a string won’t fit in the allowed lines, it is sent through the specified pager command. If none given, look for PAGER in the environment, and ultimately default to less.

If no system pager works, the string is sent through a ‘dumb pager’ written in python, very simplistic.

IPython.genutils.page_dumb()

Very dumb ‘pager’ in Python, for when nothing else works.

Only moves forward, same interface as page(), except for pager_cmd and mode.

IPython.genutils.page_file()

Page a file, using an optional pager command and starting line.

IPython.genutils.popkey()

Return dct[key] and delete dct[key].

If default is given, return it if dct[key] doesn’t exist, otherwise raise KeyError.

IPython.genutils.print_lsstring()

Prettier (non-repr-like) and more informative printer for LSString

IPython.genutils.print_slist()

Prettier (non-repr-like) and more informative printer for SList

IPython.genutils.process_cmdline()

Process command-line options and arguments.

Arguments:

• argv: list of arguments, typically sys.argv.
• names: list of option names. See DPyGetOpt docs for details on options

syntax.

• defaults: dict of default values.
• usage: optional usage notice to print if a wrong argument is passed.

Return a dict of options and a list of free arguments.

IPython.genutils.qw()

Similar to Perl’s qw() operator, but with some more options.

qw(words,flat=0,sep=’ ‘,maxsplit=-1) -> words.split(sep,maxsplit)

words can also be a list itself, and with flat=1, the output will be recursively flattened.

Examples:

>>> qw('1 2')
['1', '2']

>>> qw(['a b','1 2',['m n','p q']])
[['a', 'b'], ['1', '2'], [['m', 'n'], ['p', 'q']]]

>>> qw(['a b','1 2',['m n','p q']],flat=1)
['a', 'b', '1', '2', 'm', 'n', 'p', 'q']

IPython.genutils.qw_lol()

qw_lol(‘a b’) -> [[‘a’,’b’]], otherwise it’s just a call to qw().

We need this to make sure the modules_some keys always end up as a list of lists.

IPython.genutils.qwflat()

Calls qw(words) in flat mode. It’s just a convenient shorthand.

IPython.genutils.raw_input_ext()

Similar to raw_input(), but accepts extended lines if input ends with .

IPython.genutils.raw_input_multi()

Take multiple lines of input.

A list with each line of input as a separate element is returned when a termination string is entered (defaults to a single ‘.’). Input can also terminate via EOF (^D in Unix, ^Z-RET in Windows).

Lines of input which end in are joined into single entries (and a secondary continuation prompt is issued as long as the user terminates lines with ). This allows entering very long strings which are still meant to be treated as single entities.

IPython.genutils.read_dict()

Read a dictionary of key=value pairs from an input file, optionally performing conversions on the resulting values.

read_dict(filename,type_conv,**opt) -> dict

Only one value per line is accepted, the format should be

# optional comments are ignored key valuen

Args:

• type_conv: A dictionary specifying which keys need to be converted to

which types. By default all keys are read as strings. This dictionary should have as its keys valid conversion functions for strings (int,long,float,complex, or your own). The value for each key (converter) should be a whitespace separated string containing the names of all the entries in the file to be converted using that function. For keys to be left alone, use None as the conversion function (only needed with purge=1, see below).

• opt: dictionary with extra options as below (default in parens)

  purge(0): if set to 1, all keys not listed in type_conv are purged out of the dictionary to be returned. If purge is going to be used, the set of keys to be left as strings also has to be explicitly specified using the (non-existent) conversion function None.

  fs(None): field separator. This is the key/value separator to be used when parsing the file. The None default means any whitespace [behavior of string.split()].

  strip(0): if 1, strip string values of leading/trailinig whitespace.

  warn(1): warning level if requested keys are not found in file.

  • 0: silently ignore.
  • 1: inform but proceed.
  • 2: raise KeyError exception.

  no_empty(0): if 1, remove keys with whitespace strings as a value.

  unique([]): list of keys (or space separated string) which can’t be repeated. If one such key is found in the file, each new instance overwrites the previous one. For keys not listed here, the behavior is to make a list of all appearances.

Example:

If the input file test.ini contains (we put it in a string to keep the test self-contained):

>>> test_ini = '''\
... i 3
... x 4.5
... y 5.5
... s hi ho'''

Then we can use it as follows: >>> type_conv={int:’i’,float:’x’,None:’s’}

>>> d = read_dict(test_ini)

>>> sorted(d.items())
[('i', '3'), ('s', 'hi ho'), ('x', '4.5'), ('y', '5.5')]

>>> d = read_dict(test_ini,type_conv)

>>> sorted(d.items())
[('i', 3), ('s', 'hi ho'), ('x', 4.5), ('y', '5.5')]

>>> d = read_dict(test_ini,type_conv,purge=True)

>>> sorted(d.items())
[('i', 3), ('s', 'hi ho'), ('x', 4.5)]

IPython.genutils.setattr_list()

Set a list of attributes for an object taken from a namespace.

setattr_list(obj,alist,nspace) -> sets in obj all the attributes listed in alist with their values taken from nspace, which must be a dict (something like locals() will often do) If nspace isn’t given, locals() of the caller is used, so in most cases you can omit it.

Note that alist can be given as a string, which will be automatically split into a list on whitespace. If given as a list, it must be a list of strings (the variable names themselves), not of variables.

IPython.genutils.shell()

Execute a command in the system shell, always return None.

Options:

• verbose (0): print the command to be executed.
• debug (0): only print, do not actually execute.
• header (‘’): Header to print on screen prior to the executed command (it

is only prepended to the command, no newlines are added).

Note: this is similar to genutils.system(), but it returns None so it can be conveniently used in interactive loops without getting the return value (typically 0) printed many times.

IPython.genutils.snip_print()

Print a string snipping the midsection to fit in width.

print_full: mode control:

• 0: only snip long strings
• 1: send to page() directly.
• 2: snip long strings and ask for full length viewing with page()

Return 1 if snipping was necessary, 0 otherwise.

IPython.genutils.sort_compare()

Sort and compare two lists.

By default it does it in place, thus modifying the lists. Use inplace = 0 to avoid that (at the cost of temporary copy creation).

IPython.genutils.system()

Execute a system command, return its exit status.

Options:

• verbose (0): print the command to be executed.
• debug (0): only print, do not actually execute.
• header (‘’): Header to print on screen prior to the executed command (it

is only prepended to the command, no newlines are added).

Note: a stateful version of this function is available through the SystemExec class.

IPython.genutils.target_outdated()

Determine whether a target is out of date.

target_outdated(target,deps) -> 1/0

deps: list of filenames which MUST exist. target: single filename which may or may not exist.

If target doesn’t exist or is older than any file listed in deps, return true, otherwise return false.

IPython.genutils.target_update()

Update a target with a given command given a list of dependencies.

target_update(target,deps,cmd) -> runs cmd if target is outdated.

This is just a wrapper around target_outdated() which calls the given command if target is outdated.

IPython.genutils.timing()

timing(func,*args,**kw) -> t_total

Execute a function once, return the elapsed total CPU time in seconds. This is just the first value in timings_out().

IPython.genutils.timings()

Execute a function reps times, return a tuple with the elapsed total CPU time in seconds and the time per call. These are just the first two values in timings_out().

IPython.genutils.timings_out()

Execute a function reps times, return a tuple with the elapsed total CPU time in seconds, the time per call and the function’s output.

Under Unix, the return value is the sum of user+system time consumed by the process, computed via the resource module. This prevents problems related to the wraparound effect which the time.clock() function has.

Under Windows the return value is in wall clock seconds. See the documentation for the time module for more details.

IPython.genutils.uniq_stable()

uniq_stable(elems) -> list

Return from an iterable, a list of all the unique elements in the input, but maintaining the order in which they first appear.

A naive solution to this problem which just makes a dictionary with the elements as keys fails to respect the stability condition, since dictionaries are unsorted by nature.

Note: All elements in the input must be valid dictionary keys for this routine to work, as it internally uses a dictionary for efficiency reasons.

IPython.genutils.unquote_ends()

Remove a single pair of quotes from the endpoints of a string.

IPython.genutils.warn()

Standard warning printer. Gives formatting consistency.

Output is sent to Term.cerr (sys.stderr by default).

Options:

-level(2): allows finer control:

0 -> Do nothing, dummy function. 1 -> Print message. 2 -> Print ‘WARNING:’ + message. (Default level). 3 -> Print ‘ERROR:’ + message. 4 -> Print ‘FATAL ERROR:’ + message and trigger a sys.exit(exit_val).

-exit_val (1): exit value returned by sys.exit() for a level 4 warning. Ignored for all other levels.

IPython.genutils.with_obj()

Set multiple attributes for an object, similar to Pascal’s with.

Example: with_obj(jim,

born = 1960, haircolour = ‘Brown’, eyecolour = ‘Green’)

Credit: Greg Ewing, in http://mail.python.org/pipermail/python-list/2001-May/040703.html.

NOTE: up until IPython 0.7.2, this was called simply ‘with’, but ‘with’ has become a keyword for Python 2.5, so we had to rename it.

IPython.genutils.wrap_deprecated()