core.inputsplitter

Module: core.inputsplitter

Inheritance diagram for IPython.core.inputsplitter:

Analysis of text input into executable blocks.

The main class in this module, InputSplitter, is designed to break input from either interactive, line-by-line environments or block-based ones, into standalone blocks that can be executed by Python as ‘single’ statements (thus triggering sys.displayhook).

A companion, IPythonInputSplitter, provides the same functionality but with full support for the extended IPython syntax (magics, system calls, etc).

For more details, see the class docstring below.

Syntax Transformations

One of the main jobs of the code in this file is to apply all syntax transformations that make up ‘the IPython language’, i.e. magics, shell escapes, etc. All transformations should be implemented as fully stateless entities, that simply take one line as their input and return a line. Internally for implementation purposes they may be a normal function or a callable object, but the only input they receive will be a single line and they should only return a line, without holding any data-dependent state between calls.

As an example, the EscapedTransformer is a class so we can more clearly group together the functionality of dispatching to individual functions based on the starting escape character, but the only method for public use is its call method.

ToDo

• Should we make push() actually raise an exception once push_accepts_more() returns False?
• Naming cleanups. The tr_* names aren’t the most elegant, though now they are at least just attributes of a class so not really very exposed.
• Think about the best way to support dynamic things: automagic, autocall, macros, etc.
• Think of a better heuristic for the application of the transforms in IPythonInputSplitter.push() than looking at the buffer ending in ‘:’. Idea: track indentation change events (indent, dedent, nothing) and apply them only if the indentation went up, but not otherwise.
• Think of the cleanest way for supporting user-specified transformations (the user prefilters we had before).

Authors

• Fernando Perez
• Brian Granger

Classes

EscapedTransformer

class IPython.core.inputsplitter.EscapedTransformer

Bases: object

Class to transform lines that are explicitly escaped out.

__init__()

IPythonInputSplitter

class IPython.core.inputsplitter.IPythonInputSplitter(input_mode=None)

Bases: IPython.core.inputsplitter.InputSplitter

An input splitter that recognizes all of IPython’s special syntax.

__init__(input_mode=None)

code = None

encoding = ''

indent_spaces = 0

input_mode = 'line'

push(lines)

Push one or more lines of IPython input.

push_accepts_more()

Return whether a block of interactive input can accept more input.

This method is meant to be used by line-oriented frontends, who need to guess whether a block is complete or not based solely on prior and current input lines. The InputSplitter considers it has a complete interactive block and will not accept more input only when either a SyntaxError is raised, or all of the following are true:

1. The input compiles to a complete statement.
2. The indentation level is flush-left (because if we are indented, like inside a function definition or for loop, we need to keep reading new input).
3. There is one extra line consisting only of whitespace.

Because of condition #3, this method should be used only by line-oriented frontends, since it means that intermediate blank lines are not allowed in function definitions (or any other indented block).

If the current input produces a syntax error, this method immediately returns False but does not raise the syntax error exception, as typically clients will want to send invalid syntax to an execution backend which might convert the invalid syntax into valid Python via one of the dynamic IPython mechanisms.

reset()

Reset the input buffer and associated state.

source = ''

source_raw = ''

source_raw_reset()

Return input and raw source and perform a full reset.

source_reset()

Return the input source and perform a full reset.

InputSplitter

class IPython.core.inputsplitter.InputSplitter(input_mode=None)

Bases: object

An object that can accumulate lines of Python source before execution.

This object is designed to be fed python source line-by-line, using

push(). It will return on each push whether the currently pushed code could be executed already. In addition, it provides a method called push_accepts_more() that can be used to query whether more input can be pushed into a single interactive block.

This is a simple example of how an interactive terminal-based client can use this tool:

isp = InputSplitter()
while isp.push_accepts_more():
    indent = ' '*isp.indent_spaces
    prompt = '>>> ' + indent
    line = indent + raw_input(prompt)
    isp.push(line)
print 'Input source was:

‘, isp.source_reset(),

__init__(input_mode=None)

Create a new InputSplitter instance.

Parameters :

input_mode : str

One of [‘line’, ‘cell’]; default is ‘line’.

The input_mode parameter controls how new inputs are used when fed via :

the :meth:`push` method: :

- ‘line’: meant for line-oriented clients, inputs are appended one at a :

time to the internal buffer and the whole buffer is compiled.

- ‘cell’: meant for clients that can edit multi-line ‘cells’ of text at :

a time. A cell can contain one or more blocks that can be compile in ‘single’ mode by Python. In this mode, each new input new input completely replaces all prior inputs. Cell mode is thus equivalent to prepending a full reset() to every push() call.

code = None

encoding = ''

indent_spaces = 0

input_mode = 'line'

push(lines)

Push one or more lines of input.

This stores the given lines and returns a status code indicating whether the code forms a complete Python block or not.

Any exceptions generated in compilation are swallowed, but if an exception was produced, the method returns True.

Parameters :

lines : string

One or more lines of Python input.

Returns :

is_complete : boolean

True if the current input source (the result of the current input

plus prior inputs) forms a complete Python execution block. Note that :

this value is also stored as a private attribute (_is_complete), so it :

can be queried at any time. :

push_accepts_more()

Return whether a block of interactive input can accept more input.

This method is meant to be used by line-oriented frontends, who need to guess whether a block is complete or not based solely on prior and current input lines. The InputSplitter considers it has a complete interactive block and will not accept more input only when either a SyntaxError is raised, or all of the following are true:

1. The input compiles to a complete statement.
2. The indentation level is flush-left (because if we are indented, like inside a function definition or for loop, we need to keep reading new input).
3. There is one extra line consisting only of whitespace.

Because of condition #3, this method should be used only by line-oriented frontends, since it means that intermediate blank lines are not allowed in function definitions (or any other indented block).

If the current input produces a syntax error, this method immediately returns False but does not raise the syntax error exception, as typically clients will want to send invalid syntax to an execution backend which might convert the invalid syntax into valid Python via one of the dynamic IPython mechanisms.

reset()

Reset the input buffer and associated state.

source = ''

source_reset()

Return the input source and perform a full reset.

LineInfo

class IPython.core.inputsplitter.LineInfo(line)

Bases: object

A single line of input and associated info.

This is a utility class that mostly wraps the output of split_user_input() into a convenient object to be passed around during input transformations.

Includes the following as properties:

line

The original, raw line

lspace

Any early whitespace before actual text starts.

esc

The initial esc character (or characters, for double-char escapes like ‘??’ or ‘!!’).

fpart

The ‘function part’, which is basically the maximal initial sequence of valid python identifiers and the ‘.’ character. This is what is checked for alias and magic transformations, used for auto-calling, etc.

rest

Everything else on the line.

__init__(line)

Functions

IPython.core.inputsplitter.get_input_encoding()

Return the default standard input encoding.

If sys.stdin has no encoding, ‘ascii’ is returned.

IPython.core.inputsplitter.has_comment(src)

Indicate whether an input line has (i.e. ends in, or is) a comment.

This uses tokenize, so it can distinguish comments from # inside strings.

Parameters :

src : string

A single line input string.

Returns :

Boolean: True if source has a comment. :

IPython.core.inputsplitter.num_ini_spaces(s)

Return the number of initial spaces in a string.

Note that tabs are counted as a single space. For now, we do not support mixing of tabs and spaces in the user’s input.

Parameters :s : string Returns :n : int

IPython.core.inputsplitter.remove_comments(src)

Remove all comments from input source.

Note: comments are NOT recognized inside of strings!

Parameters :

src : string

A single or multiline input string.

Returns :

String with all Python comments removed. :

IPython.core.inputsplitter.split_user_input(line)

Split user input into early whitespace, esc-char, function part and rest.

This is currently handles lines with ‘=’ in them in a very inconsistent manner.

Examples

>>> split_user_input('x=1')
('', '', 'x=1', '')
>>> split_user_input('?')
('', '?', '', '')
>>> split_user_input('??')
('', '??', '', '')
>>> split_user_input(' ?')
(' ', '?', '', '')
>>> split_user_input(' ??')
(' ', '??', '', '')
>>> split_user_input('??x')
('', '??', 'x', '')
>>> split_user_input('?x=1')
('', '', '?x=1', '')
>>> split_user_input('!ls')
('', '!', 'ls', '')
>>> split_user_input('  !ls')
('  ', '!', 'ls', '')
>>> split_user_input('!!ls')
('', '!!', 'ls', '')
>>> split_user_input('  !!ls')
('  ', '!!', 'ls', '')
>>> split_user_input(',ls')
('', ',', 'ls', '')
>>> split_user_input(';ls')
('', ';', 'ls', '')
>>> split_user_input('  ;ls')
('  ', ';', 'ls', '')
>>> split_user_input('f.g(x)')
('', '', 'f.g(x)', '')
>>> split_user_input('f.g (x)')
('', '', 'f.g', '(x)')
>>> split_user_input('?%hist')
('', '?', '%hist', '')
>>> split_user_input('?x*')
('', '?', 'x*', '')

IPython.core.inputsplitter.transform_assign_magic(line)

Handle the a = %who syntax.

IPython.core.inputsplitter.transform_assign_system(line)

Handle the files = !ls syntax.

IPython.core.inputsplitter.transform_classic_prompt(line)

Handle inputs that start with ‘>>> ‘ syntax.

IPython.core.inputsplitter.transform_help_end(line)

Translate lines with ?/?? at the end

IPython.core.inputsplitter.transform_ipy_prompt(line)

Handle inputs that start classic IPython prompt syntax.