Theano in an Hour: Symbolic Graphs, Automatic Gradients and a Compiler That Rewrites Your Math

Lamblin, speaking at the 2016 Deep Learning School, describes Theano as a mathematical symbolic expression compiler. Users write expressions in NumPy-like syntax, from basic operations such as addition and max up to whole layers or networks. Those expressions can then be manipulated through substitution, cloning and replacement.

The graph can be traversed to perform automatic differentiation, optimized for numerical stability and speed, and handed to a runtime that computes output values. Theano also ships tools for inspecting and debugging both its own code and the user's. The tutorial is accompanied by an IPython notebook on GitHub containing all the snippets from the slides.

The project is more than eight years old. It began with a few contributors at MILA's predecessor lab and now has users and contributors worldwide, driving research papers and industrial prototypes. Lamblin lists libraries that build on it: Blocks, Keras and Lasagne for higher-level layers and training algorithms, PyMC3 for probabilistic programming, and Platoon and Theano-MPI for multi-GPU and multi-machine training.

The workflow has three steps: define an expression, compile a function, then execute it on values. Inputs are symbolic variables with a declared type, such as a vector or matrix of floats or integers. The number of dimensions must be known in advance, but shapes and memory layout are not fixed, so mini-batch sizes can vary between calls.

Shared variables are the other kind of input. They are symbolic but also hold a value that persists across function calls and is shared between functions, which makes them the natural home for model parameters such as a weight matrix and a bias. From these inputs the user builds intermediate variables, in the example a sigmoid of the product of X and W plus the bias, and then a squared-error cost.

Lamblin shows how to visualize the result with a helper such as `pydot` printing. Variables appear as boxes, while apply nodes represent the operations connecting them. Inputs and shared variables have no ancestors, and unnamed intermediate variables can be hidden to simplify the picture.

For backpropagation, Lamblin recalls the chain rule and notes that the full Jacobian is usually a bad idea to compute explicitly. What is needed is an expression that takes the gradient of the cost with respect to an output and returns the gradient with respect to the input, effectively a vector-Jacobian product, sometimes called the L operator.

Almost every Theano operation implements a method returning that as a symbolic expression. Calling `theano.grad` backpropagates from the cost through the graph, accumulating where a variable is used more than once. The resulting gradients are ordinary graph variables, as if the user had written them by hand with dot products and sigmoids.

Since the graph simply grows, the next step can be gradient-descent update expressions, for example scaling the gradients by a constant 0.1. The extended graph contains many small inserted operations, which is why the optimization phase matters.

`theano.function` takes the inputs you want to feed and the outputs you want back. You do not need values for every declared input, and a function can cover only part of the graph. A predict function from X to the output ignores Y and never computes gradients, and an error function can start directly from the prediction.

Shared variables are implicit inputs to every function and are fetched automatically. A monitoring function can return both output and cost, which requires feeding Y as well.

For training, you pass a list of updates, each a pair of a shared variable and the expression for its new value. These act as implicit outputs: once all outputs are computed, the updates take effect. Printing the bias before and after a call shows that it changed.

During compilation, the chosen subgraph is rewritten. Duplicated computations run once, unnecessary ones such as X divided by X become a constant, and numerically unstable forms are replaced. Lamblin cites log of one plus x underflowing for small x, and log of softmax becoming a more stable log-softmax operation. Memory-saving in-place and destructive operations are inserted, and the transfer to the GPU also happens at this stage.

By default most optimizations apply, but they can be reduced globally or per function for faster iteration. Comparing the original and optimized predict graphs shows a single GEMM call to an optimized BLAS routine and an in-place sigmoid. The optimized update graph is much smaller and fuses element-wise operations so the array is traversed once, which helps with automatically generated gradients.

Because the order of element-wise operations cannot be known in advance, Theano generates code on the fly, producing a C++ or CUDA module that is compiled and imported back into Python. The runtime can use a C++ executor that avoids switching between the Python interpreter and the engine. Lamblin also mentions a `debugprint` text view of the graph for detail.

On GPUs, a new backend, planned for the 0.9 release, supports more data types and easier handling of GPU arrays from Python. A configuration flag selects the device, after which shared variables live in GPU memory by default and CPU ops are replaced. He recommends float32 since most GPUs are slow at double precision, and notes the device flag must be set before Theano is imported.

Theano graphs are directed acyclic graphs, so loops are not allowed. If the iteration count is known, you can unroll with a Python loop, but that fails for dynamic lengths such as sequence generation. `scan` solves this by wrapping an inner step function in a single node that handles indices, slicing sequences and feeding outputs back.

That structure lets Theano define a gradient for the node, which is another scan going backwards and applying backpropagation through time. It can also be moved to the GPU, with the inner function recompiled there.

Debugging is trickier because an error at execution time is not obviously tied to where the expression was defined. Theano offers informative errors, compilation modes that check for NaN values, and test values that compute each new variable on sample data as the graph is built, which helps catch shape mismatches. It can be extended with Python wrappers, C or CUDA ops, and custom optimizations.

Recent additions include the new GPU backend, faster 2D and 3D convolutions, a faster optimization phase, avoidance of repeated recompilation, interactive graph visualization and conditional pdb breakpoints. Planned work includes more GPU operations, completing the basic RNN ops, better 3D convolutions and more data parallelism.

Moving to the hands-on notebooks, Lamblin builds logistic regression on the MNIST digits, which are packaged with the repository. The input is a matrix for mini-batches, the weights and bias are shared variables initialized to zeros, and the model is an affine map followed by softmax. The prediction is the argmax over classes for each example.

The loss is the mean negative log-likelihood, built from basic blocks by taking the log of the probability at the target index. There is no gradient-descent object, so the update rule is whatever you write, and adding momentum means adding shared variables for velocity. A separate compiled function computes misclassification rate for validation and test without updating anything.

The training loop includes early stopping and saves the best model by storing the parameter values rather than pickling Python objects, which is more robust and easier to port to other frameworks. In the live run, training ended after 96 epochs. Visualizing the learned filters shows plausible digit shapes, but the validation and test errors remain far above human level, which Lamblin takes as a sign the model is too simple.

For the next example Lamblin switches to a convolutional network on MNIST, defining small layer classes that hold parameters and expressions. Layers cover logistic regression, convolution with pooling and a fully connected hidden layer. Convolution weights use the Glorot and Bengio 2010 initialization, and the bias is one number per channel, so the layer can handle images of varying size. He notes frameworks such as Keras, Blocks and Lasagne provide these in practice.

The training graph stacks two convolutional layers, flattens the features into a fully connected layer and ends with the logistic regression output. The parameter list is the concatenation of every layer's parameters, so one call to `grad` gives all updates. A full run takes about five minutes, so he shows a previous result along with the first-layer filters and activations.

The last example is an LSTM predicting the next character. The step function contains the LSTM equations and is passed to `theano.scan`, along with masks because mini-batch sequences are padded to the longest in the batch. The cost is categorical cross-entropy over the sequence, with the mask excluding positions after the end. Data comes from Fuel, a MILA tool that reads plain text, groups sequences of similar length, shuffles and pads.

The model is trained with plain SGD and backpropagation through time. A separate function generates one character at a time, normalizing probabilities in higher precision because float32 sums can miss one. Seeded with "the meaning of life is," early output is gibberish, with odd characters even from other scripts, but it improves slowly. Lamblin suggests exercises such as better optimizers, different nonlinearities, new initializations or different seed text.

In Q&A, an audience member asks for a live debugging example. Lamblin deliberately gives the logistic regression a wrong shape. The graph builds fine, but executing it raises an error reporting a shape mismatch in the dot product, along with the apply node and input shapes, though not where it was defined.

Recompiling with the optimizer disabled makes the error include a back-trace of where the node was created, pointing to the line with the dot product. He also mentions breakpoints as another technique and offers to share pointers.

Another questioner wants to distribute models without requiring users to install Python and compilers. Lamblin says that is difficult, since Theano is deeply intertwined with Python: memory management is done by Python, and intermediate CPU values are NumPy arrays. His suggestion is a Docker container, which he says works reasonably well even with NVIDIA GPUs. He adds that help disentangling Theano from the Python runtime would be welcome but is a large project.