printing – Graph Printing and Symbolic Print Statement

Guide

Printing during execution

Intermediate values in a computation cannot be printed inthe normal python way with the print statement, because Theano has no statements.Instead there is the Print Op.

  1. >>> from theano import tensor as T, function, printing
  2. >>> x = T.dvector()
  3. >>> hello_world_op = printing.Print('hello world')
  4. >>> printed_x = hello_world_op(x)
  5. >>> f = function([x], printed_x)
  6. >>> r = f([1, 2, 3])
  7. hello world __str__ = [ 1.  2.  3.]

If you print more than one thing in a function like f, they will notnecessarily be printed in the order that you think. The order might even dependon which graph optimizations are applied. Strictly speaking, the order ofprinting is not completely defined by the interface –the only hard rule is that if the input of some print output a isultimately used as an input to some other print input b (so that b depends on a),then a will print before b.

Printing graphs

Theano provides two functions (theano.pp() andtheano.printing.debugprint()) to print a graph to the terminal before or aftercompilation. These two functions print expression graphs in different ways:pp() is more compact and math-like, debugprint() is more verbose.Theano also provides theano.printing.pydotprint() that creates a png image of the function.

  1. >>> from theano import pp, tensor as T
  2. >>> x = T.dscalar('x')
  3. >>> y = x ** 2
  4. >>> gy = T.grad(y, x)
  5. >>> pp(gy)  # print out the gradient prior to optimization
  6. '((fill((x ** TensorConstant{2}), TensorConstant{1.0}) * TensorConstant{2}) * (x ** (TensorConstant{2} - TensorConstant{1})))'
  7. >>> f = function([x], gy)
  8. >>> pp(f.maker.fgraph.outputs[0])
  9. '(TensorConstant{2.0} * x)'

The parameter in T.dscalar(‘x’) in the first line is the name of this variablein the graph. This name is used when printing the graph to make it more readable.If no name is provided the variable x is printed as its type as returned byx.type(). In this example - <TensorType(float64, scalar)>.

The name parameter can be any string. There are no naming restrictions:in particular, you can have many variables with the same name.As a convention, we generally give variables a string name that is similar to the name of the variable in local scope, butyou might want to break this convention to include an object instance, or aniteration number or other kinds of information in the name.

Note

To make graphs legible, pp() hides some Ops that are actually in the graph. For example,automatic DimShuffles are not shown.

  1. >>> theano.printing.debugprint(f.maker.fgraph.outputs[0])  
  2. Elemwise{mul,no_inplace} [id A] ''
  3.  |TensorConstant{2.0} [id B]
  4.  |x [id C]

Each line printed represents a Variable in the graph.The line |x [id C] means the variable named x with debugprint identifier[id C] is an input of the Elemwise. If you accidentally have two variables called x inyour graph, their different debugprint identifier will be your clue.

The line |TensorConstant{2.0} [id B] means that there is a constant 2.0with this debugprint identifier.

The line Elemwise{mul,no_inplace} [id A] '' is indented less thanthe other ones, because it means there is a variable computed by multiplyingthe other (more indented) ones together.

The | symbol are just there to help read big graph. The grouptogether inputs to a node.

Sometimes, you’ll see a Variable but not the inputs underneath. That canhappen when that Variable has already been printed. Where else has it beenprinted? Look for debugprint identifier using the Find feature of your texteditor.

  1. >>> theano.printing.debugprint(gy)  
  2. Elemwise{mul} [id A] ''
  3.  |Elemwise{mul} [id B] ''
  4.  | |Elemwise{second,no_inplace} [id C] ''
  5.  | | |Elemwise{pow,no_inplace} [id D] ''
  6.  | | | |x [id E]
  7.  | | | |TensorConstant{2} [id F]
  8.  | | |TensorConstant{1.0} [id G]
  9.  | |TensorConstant{2} [id F]
  10.  |Elemwise{pow} [id H] ''
  11.    |x [id E]
  12.    |Elemwise{sub} [id I] ''
  13.      |TensorConstant{2} [id F]
  14.      |InplaceDimShuffle{} [id J] ''
  15.        |TensorConstant{1} [id K]
  1. >>> theano.printing.debugprint(gy, depth=2)  
  2. Elemwise{mul} [id A] ''
  3.  |Elemwise{mul} [id B] ''
  4.  |Elemwise{pow} [id C] ''

If the depth parameter is provided, it limits the number of levels that areshown.

  • The function theano.printing.pydotprint() will print a compiled theano function to a png file. In the image, Apply nodes (the applications of ops) are shown as ellipses and variables are shown as boxes.The number at the end of each label indicates graph position.Boxes and ovals have their own set of positions, so you can have apply #1 and also avariable #1.The numbers in the boxes (Apply nodes) are actually their position in therun-time execution order of the graph.Green ovals are inputs to the graph and blue ovals are outputs.

If your graph uses shared variables, those sharedvariables will appear as inputs. Future versions of the pydotprint()may distinguish these inplicit inputs from explicit inputs.

If you give updates arguments when creating your function, these are added asextra inputs and outputs to the graph.Future versions of pydotprint() may distinguish theseimplicit inputs and outputs from explicit inputs and outputs.

Reference

  • class theano.printing.Print(Op)[source]

  • This identity-like Op has the side effect of printing a message followed by its inputswhen it runs. Default behaviour is to print the str representation. Optionally, onecan pass a list of the input member functions to execute, or attributes to print.

Parameters:

  1. - **message** (_string_) – prepend this to the output
  2. - **attrs** (_list of strings_) – list of input node attributes or member functions to print.Functions areidentified through callable(), executed and their return value printed.

Parameters:x (a Variable) – any symbolic variableReturns:symbolic identity(x)

When you use the return-value from this function in a theano function,running the function will print the value that x takes in the graph.

  • theano.printing.debugprint(obj, depth=-1, print_type=False, file=None, ids='CHAR', stop_on_name=False, done=None, print_storage=False, print_clients=False, used_ids=None)[source]
  • Print a computation graph as text to stdout or a file.

Parameters:

  • obj (Variable, Apply, or Function instance) – symbolic thing to print
  • depth (integer) – print graph to this depth (-1 for unlimited)
  • print_type (boolean) – whether to print the type of printed objects
  • file (None, 'str', or file-like object) – print to this file (‘str’ means to return a string)
  • ids (str) – How do we print the identifier of the variableid - print the python id valueint - print integer characterCHAR - print capital character“” - don’t print an identifier
  • stop_on_name – When True, if a node in the graph has a name,we don’t print anything below it.
  • done (None or dict) – A dict where we store the ids of printed node.Useful to have multiple call to debugprint share the same ids.
  • print_storage (bool) – If True, this will print the storage mapfor Theano functions. Combined with allow_gc=False, after theexecution of a Theano function, we see the intermediate result.
  • print_clients (bool) – If True, this will print for Apply node thathave more then 1 clients its clients. This help find who usean Apply node.
  • used_ids (dict or None) – the id to use for some object, but maybe we onlyrefered to it yet.Returns: string if file == ‘str’, else file arg

Each line printed represents a Variable in the graph.The indentation of lines corresponds to its depth in the symbolic graph.The first part of the text identifies whether it is an input(if a name or type is printed) or the output of some Apply (in which casethe Op is printed).The second part of the text is an identifier of the Variable.If print_type is True, we add a part containing the type of the Variable

If a Variable is encountered multiple times in the depth-first search,it is only printed recursively the first time. Later, just the Variableidentifier is printed.

If an Apply has multiple outputs, then a ‘.N’ suffix will be appendedto the Apply’s identifier, to indicate which output a line corresponds to.

  • theano.printing.pydotprint(fct, outfile=None, compact=True, format='png', with_ids=False, high_contrast=True, cond_highlight=None, colorCodes=None, max_label_size=70, scan_graphs=False, var_with_name_simple=False, print_output_file=True, return_image=False)[source]
  • Print to a file the graph of a compiled theano function’s ops. Supportsall pydot output formats, including png and svg.

Parameters:

  • fct – a compiled Theano function, a Variable, an Apply ora list of Variable.
  • outfile – the output file where to put the graph.
  • compact – if True, will remove intermediate var that don’t have name.
  • format – the file format of the output.
  • with_ids – Print the toposort index of the node in the node name.and an index number in the variable ellipse.
  • high_contrast – if true, the color that describes the respectivenode is filled with its corresponding color, instead of coloringthe border
  • colorCodes – dictionary with names of ops as keys and colors asvalues
  • cond_highlight – Highlights a lazy if by sorrounding each of the 3possible categories of ops with a border. The categoriesare: ops that are on the left branch, ops that are on theright branch, ops that are on both branchesAs an alternative you can provide the node that representsthe lazy if
  • scan_graphs – if true it will plot the inner graph of each scan opin files with the same name as the name given for the mainfile to which the name of the scan op is concatenated andthe index in the toposort of the scan.This index can be printed with the option with_ids.
  • var_with_name_simple – If true and a variable have a name,we will print only the variable name.Otherwise, we concatenate the type to the var name.
  • return_image – If True, it will create the image and return it.Useful to display the image in ipython notebook.
  1. import theano
  2. v = theano.tensor.vector()
  3. from IPython.display import SVG
  4. SVG(theano.printing.pydotprint(v*2, return_image=True,
  5.                                format='svg'))

In the graph, ellipses are Apply Nodes (the execution of an op)and boxes are variables. If variables have names they are used astext (if multiple vars have the same name, they will be merged inthe graph). Otherwise, if the variable is constant, we print itsvalue and finally we print the type + a unique number to preventmultiple vars from being merged. We print the op of the apply inthe Apply box with a number that represents the toposort order ofapplication of those Apply. If an Apply has more than 1 input, welabel each edge between an input and the Apply node with theinput’s index.

  • Variable color code::
    • Cyan boxes are SharedVariable, inputs and/or outputs) of the graph,
    • Green boxes are inputs variables to the graph,
    • Blue boxes are outputs variables of the graph,
    • Grey boxes are variables that are not outputs and are not used,
  • Default apply node code::
    • Red ellipses are transfers from/to the gpu
    • Yellow are scan node
    • Brown are shape node
    • Magenta are IfElse node
    • Dark pink are elemwise node
    • Purple are subtensor
    • Orange are alloc node For edges, they are black by default. If a node returns a viewof an input, we put the corresponding input edge in blue. If itreturns a destroyed input, we put the corresponding edge in red.

Note

Since October 20th, 2014, this print the inner function of allscan separately after the top level debugprint output.