Learning TensorFlow Hands-On: From a Mystery Line to Handwritten Digits

Sherry Moore of the Google Brain team opens the Deep Learning School tutorial by promising a short overview of TensorFlow, how Google uses it, and then a hands-on session building models for the most classic machine learning problems. She thanks her colleague Wolf for verifying the installation on every platform and points the audience to her product manager for feature requests.

TensorFlow is a machine learning library developed at Google and open sourced the previous November. She says it quickly became the most popular machine learning library on GitHub, with over 32,000 stars, over 14,000 forks and contributions from about 400 individual developers. It was designed for machine learning, but its flexible data flow infrastructure suits any application that can fit that model of firing when data is ready.

The team built it working alongside researchers; Moore says she sat next to the inventor of AlexNet when she joined, and researchers told them what worked for them. As a Google team they also wanted a path from research to prototyping to production without throwing away code, so researchers can write something and productionize it right away.

To explain the core abstractions, she asks the audience what you need to build a neural network: neurons, which operate on data. In TensorFlow, data is held in tensors, which are multi-dimensional arrays similar to NumPy's ndarray. The graph is a composition of nodes connected by inputs and outputs; a node fires when all its inputs are available, and tensors flow through the network.

Moore argues that good infrastructure has to be modular so that each piece can be upgraded or modified as long as the APIs stay consistent, which lets everyone work in parallel. In TensorFlow, front-end libraries written in languages such as Python or C++ construct the graph, which is sent to the core execution system, the runtime. She invites people to contribute front ends in their own favorite language.

Depending on where the application runs, the runtime sends kernels to the matching device: a CPU, a GPU, a phone or a TPU. The audience will run it on laptops that day, Google runs it in its data centers, and others run it on iOS and Android. She would love to see it on a Raspberry Pi, imagining a security camera that uses machine learning to capture high-resolution pictures only when things move.

At Google, she says, TensorFlow is used everywhere: recognizing pictures with Inception, voice search, and Smart Reply. According to statistics gathered in February, over 10% of responses sent on mobile were generated by Smart Reply, and she jokes that in early training the top answer was always "I love you." It is also used for games, art and Deep Dream.

Google has published models including Inception, image captioning, a language model on a billion words, ResNet on CIFAR-10 and sequence to sequence. The day's labs use the core TensorFlow APIs, but there are higher-level libraries such as Keras, Slim, Pretty Tensor and TF Learn. She encourages people to build their own libraries if none fit and to send contributions.

The first lab is linear regression, framed as the familiar guessing game: someone gives you pairs of numbers and you try to infer the equation. Moore sets up a mystery line with a hidden weight and bias, and given enough x and y samples the model should be able to guess them. She wrote the lab originally for a high school audience.

She states the recipe that comes up for any model: input data, an inference graph (also called a forward graph) that produces logits, training operations where you define a loss and an optimizer, and finally running the graph. Training is just minimizing the loss. She tells the audience she will quiz them later and hand out mystery prizes, including TensorFlow stickers.

The lab deliberately uses the core APIs rather than higher-level ones like Keras or contrib learn. Her analogy is giving the audience raw lobster and Kobe beef rather than a packaged Whole Foods meal, so they can assemble whatever they want. The first cell plots the generated input data, showing for example y near 0.32 when x is 0.2.

Variables are the stateful nodes where TensorFlow holds a network's weights and biases, and they are what gradients get applied to during training. Moore shows that variables created without a name get default names like "variable six," and recommends giving them meaningful names. Rerunning notebook cells without restarting keeps growing the default graph, so she restarts to avoid confusion.

She suggests keeping the API documentation open and inspecting what you have built, including the operations, their names, inputs, outputs and attributes. Printing the graph is how you see the neural network you just constructed, and it is very useful when debugging.

To train, you create a session, which is the handle to the runtime, and call run to send it your graph. She stresses this differs from imperative libraries where execution happens as you type: in TensorFlow you construct a graph and then ask the runtime to run it. After training, a plot compares the initial guess, the target values and the trained values, and the fit is a "great big success."

Moore urges everyone to experiment, since the worst that can happen in a notebook is clearing everything. Exploring the documentation to find other optimizers such as Adadelta, Adam and momentum is one exercise, and an audience member shows how tab completion reveals them. The audience then votes to move to digit recognition.

The second lab uses MNIST, the collection of handwritten digits. The network sees an image and guesses a digit, the ground truth says whether it was right, and training uses the difference. Moore says this lab adds four new concepts: placeholders, saving checkpoints, loading checkpoints and running evaluation.

Placeholders let you feed any data into the network at run time, a feature researchers requested because they wanted to feed data during training. The lab defines one placeholder for images and one for labels, so the graph is built once and reused for training, inference and evaluation. She also mentions TensorBoard, which produced the graph visualization shown in her slides, and which helps catch accidentally enormous graphs created by calling something repeatedly.

The network has two hidden layers plus a linear layer that outputs logits. Training uses a loss function and gradient descent as before, plus a global step variable stored with each checkpoint so you know how long a given checkpoint has been trained. A saver then writes the network's state, so interrupted training can resume instead of restarting.

Moore explains why checkpoints matter: when she first trained Inception it took about six days, and later with replicas about two and a half days, so you cannot afford to restart from scratch. In the lab she collects the loss in an array and plots it with matplotlib to see it going down. At Google they run many training jobs and glance at such curves to see which is doing best.

She says that cutting and pasting researcher code into a file is often literally the algorithm, which is then published and passed to data scientists and production teams who productionize it. The checkpoint files hold all the weights and biases, which can later be loaded for image recognition, email replies or art. In the evaluation step the lab loads the checkpoint, recalls the remembered logits and placeholders, and feeds in evaluation images.

Predictions are mostly right, though the first example (ground truth five, predicted three) is ambiguous to the eye. Moore explains that drawing your own digits often fails because MNIST values are between zero and one and the image size and scale must match training data. Just as a baby cannot recognize what it has never seen, a captioning tool would say "men talking on a cell phone" for unfamiliar scenes.

For exercises she suggests saving checkpoints every 100 steps and evaluating early ones, which should perform poorly. At Google they run evaluation in parallel with training, for example every 10 minutes for Inception, and she says getting to 78.6% was state of the art at the time. She recommends learning to run evaluation on the complete validation set and closes with the message that TensorFlow is designed for research, prototyping and production.

An attendee asks whether models built with a Python front end can be used from C++. Moore says yes, since only the graph construction front end differs, though the C++ API is not as complete and many training libraries are missing. For inference, the label_image example in C++ loads a checkpoint and runs the model, and a Python version called classify_image exists too.

Several questions are about platforms and roadmap: Windows support, TPU availability, integration with Mesos and HDFS for distributed TensorFlow, and ARM processors such as the TX1. Moore says the team is actively working on such things, notes that the early team was about ten people, and defers timeline questions to her product manager since she cannot commit to dates. TPUs were not available to outsiders at the time, as she understood it.

On data and mobile, she points to TensorFlow Poet and a retraining example that uses a downloaded set of flower pictures, which you can swap for your own images. The same model can run on laptops or Android, but she would not recommend training Inception on a phone because of memory and compute limits; inference is fine. The model is a graph plus values, with no special format.

Regarding TensorFlow Serving, she says the model can be written in Python or C++ and that any C++-only exporter requirement was probably older. She mentions some other front-end languages such as Go and invites contributions for more. Finally, an inference binary is possible and is how label_image works, with checkpoints converted into constants for speed.