Data Movement, Not Math: How to Make Deep Learning Fit on Robots and Phones

Sze opens by noting that the compute needed to train deep networks has grown exponentially, by more than 300,000 times in a few years according to OpenAI's data. She says the environmental cost can be severe: the carbon footprint of training neural networks is orders of magnitude greater than flying across North America or an average human life.

She then makes the case for moving computation from the cloud to the device. Poor communication infrastructure, privacy for sensitive data such as health records, and latency for robots and self-driving cars all argue for running models locally. A car at highway speed cannot afford to send an obstacle detection to the cloud and wait for instructions back.

The difficulty is power. A self-driving car has been reported to burn over 2,000 watts just on computation, generating heat that often needs water cooling. Handheld devices are limited to under a watt because of battery size and heat, while the embedded platforms used for these tasks draw over 10 watts.

In the past, engineers waited for transistors to get smaller, faster and more efficient. Sze says Moore's Law and Dennard scaling have both slowed, so the trend in efficiency has flattened over the last decade. Because networks keep demanding more compute, the field has turned to specialized hardware, and she notes a record number of startups building it.

To design efficient hardware, she says, ask where the energy goes. The answer is data movement rather than computation: a 32-bit read from a small 8 KB on-chip SRAM costs about 5 picojoules, which matches or exceeds a 32-bit floating-point multiply. Reading from off-chip DRAM costs about 640 picojoules, on an exponential scale.

The applications make this worse. Autonomous navigation needs semantic understanding of every pixel, perhaps at HD resolution with image pyramids that multiply the data by one or two orders of magnitude, plus geometric understanding that builds ever larger 3D maps. Deep networks doing this need hundreds of millions of operations and weights, two to three orders of magnitude more complexity than the video compression on a phone.

Sze reviews the basics for alignment. Each layer computes a weighted sum followed by a nonlinearity such as ReLU, and the multiply-and-accumulate operation accounts for over 90% of computation. She runs through fully connected, convolutional, recurrent and attention layers, noting that attention relies on matrix multiplies and that stacks of them are called transformers.

Convolutional networks are expensive because of added dimensions: multiple input channels, many filters per layer, and batches of images. In AlexNet, channel counts range from 3 to 192, filters from 96 to 384, and batch sizes from 1 to 256. Over the years models have also gotten deeper, with more weights and MACs, and so more demanding of hardware.

The parallelism is good news because all the MACs can run at once. The bottleneck is memory: each MAC needs a weight, an input activation and a partial sum, then writes an updated partial sum, giving four memory accesses per computation. In the worst case each DRAM access costs two orders of magnitude more than the MAC, so AlexNet's 700 million MACs could imply three billion DRAM accesses.

Neural networks offer many reuse opportunities. Convolution slides a filter across an image, so weights and activations are used in many MACs, activations are reused across multiple filters, and weights are reused across a batch of images. To exploit this, hardware builds a memory hierarchy: under 1 KB beside each processing element, sharing between neighboring elements at about 2x the energy, a 100 to 500 KB global buffer at about 6x, and DRAM at 200x.

The catch is that the local memory is tiny while networks have millions of weights, so large work must be broken into chunks, much like deciding what fits in your backpack. Sze describes weight-stationary designs, as in Google's TPU and the NVIDIA deep learning accelerator, output-stationary designs and input-stationary designs. Each keeps one data type local and minimizes only that type's movement.

Her group's row-stationary dataflow instead has each processing element handle a row of convolution, mixing weights, activations and partial sums, to balance movement across all data types. Comparisons show the other dataflows each win on one type of energy but lose on others. The takeaway is to optimize overall data movement, not a single data type.

Anything multiplied by zero is zero, so hardware can skip those multiplications and the memory reads that go with them, cutting power by almost 50%. Runs of zeros can also be compressed with run-length encoding, cutting data movement by up to 2x. ReLU naturally produces zeros, and pruning creates more by zeroing weights.

Her group put these ideas in a chip called Eyeriss, with 168 processing elements in a 14-by-12 array and a 100 KB shared buffer, plus compression for off-chip traffic. The row-stationary dataflow cuts global buffer accesses by 100x and off-chip accesses by over 1,000x. Compared with a mobile GPU, it uses about an order of magnitude less energy.

Thinking as a systems builder, Sze plots object-detection accuracy against energy per pixel. Neural networks like AlexNet roughly doubled accuracy over hand-engineered HOG features, but they cost two to three orders of magnitude more energy. HOG on specialized hardware was even more efficient than video compression, meaning understanding a pixel cost less than compressing it.

She asks the audience whether they would accept a phone that recognizes twice as accurately but dies 300 times faster. Nobody would, she says, and even with specialized hardware neural nets remain far less efficient than video compression. The field therefore needs lower energy without sacrificing accuracy.

Many algorithm techniques help: pruning, efficient architectures that decompose 3D filters, and reduced precision down to eight bits or binary. But most of this work targets the number of MACs and weights because they are easy to count, which is not what the system cares about. What matters is energy and latency, and these depend on where data comes from, so not all weights or all MACs are equal.

Sze's group built a tool that takes a network's weights, layer shapes and sparsity and estimates energy per layer, split between data movement and computation. For GoogLeNet it showed that weights account for only 22% of energy, with much more going into moving input and output feature maps. Armed with this, they developed energy-aware pruning, which prunes the highest-energy layers first, achieving a 3.7x energy reduction on AlexNet versus 2x for magnitude-based pruning at equal accuracy.

Latency has the same problem. Data from Google's mobile vision team shows layers with equal MAC counts can differ by 2x in latency, and equal-latency layers can vary 3x in MACs. With Google, her group developed NetAdapt, which takes a pretrained network and a latency or energy budget, proposes modifications, measures them empirically on the target phone, and iterates.

For example, it might shrink the channels of one layer to move from 100 ms to 80 ms, keep the best proposal, then tighten the budget. For the same accuracy NetAdapt came out up to 1.7x faster than handcrafted networks. The lesson is to put the hardware in the design loop with real measurements.

Applying the approach to monocular depth estimation, a harder dense-output task, the FastDepth work achieved a 10x speed-up on an NVIDIA Jetson while maintaining accuracy. A live demo ran at about 40 frames per second on an iPhone.

A hardware designer cannot know which algorithmic tricks users will apply, so the hardware must be flexible enough to turn pruning, compact architectures and low precision into real gains. Many accelerators use an array of MAC units that amortizes memory reads across the array, but that depends on a number of channels, feature map size and batch size that compact networks do not provide.

With depth-wise layers, which have only one channel, only a small part of a large array can do useful work even with thousands of multipliers. Scaling the array then fails to speed things up. Sze says what is needed is a flexible dataflow such as row-stationary, plus an on-chip network that can serve different amounts of reuse.

Highly reusable big networks want multicast, reading once and broadcasting to many engines at low bandwidth. Compact networks want unicast, with high-bandwidth unique data to each element. An all-to-all network would cover both but is too expensive and does not scale, so her group's second-generation Eyeriss uses a hierarchical mesh: all-to-all at the lowest level and a mesh above it. It supports many layer shapes, dense and sparse data and scales with more processing elements, giving about an order of magnitude gain in both speed and energy efficiency.

Super-resolution generates a high-resolution image or video from a low-resolution one, useful for saving transmission bandwidth and for filling ever larger screens with content that is still 1080p. It is computationally expensive, since state-of-the-art methods use deep networks on inputs of around 200 by 200 pixels, which is far from HD.

Sze draws on her video compression background. A compressed video is a structured description of redundancy between frames, saying which pixels in one frame match pixels in the next. Her FAST framework runs super-resolution on only some frames and transfers the results to the others using that structure, so transferring over N frames gives an N-fold speed-up.

On a broad video-standards data set, transferring over four frames gives a 4x speed-up with unchanged PSNR, and 16 frames gives 16x with a slight quality drop. Visually, text stays sharper than with interpolation, which she says gets the quality of per-frame super-resolution at the cost of simple interpolation.

Beyond neural nets, Sze describes visual-inertial odometry, which fuses camera and IMU data so a robot or AR headset knows where it is. Her group built Navion, a chip under 5 mm on a side that does complete visual-inertial odometry on chip. It reduces energy by two to three orders of magnitude compared with mobile or desktop CPUs, using under 25 milliwatts and about a megabyte of storage by keeping data on chip, compressing frames and exploiting sparsity.

For exploration, a robot can compute the Shannon mutual information of its occupancy map to decide where scanning will reveal the most new information. The bottleneck is delivering map data to many parallel cores, so her team splits the map into memory banks arranged so no bank is accessed twice at once. This lets a 200 by 200 meter map at 0.1 meter resolution be evaluated in under a second, a 100x speed-up over a CPU at one tenth the power on an FPGA.

Sze closes with health applications. Eye-movement measurements can quantitatively track neurodegenerative diseases such as Alzheimer's and Parkinson's, but normally need expensive high-speed cameras and a specialist. Her group showed that algorithms on a consumer camera, such as an iPhone, give reaction-time distributions about the same as a Phantom camera, enabling at-home measurement.

She also shows that sensors burn power too. By turning a time-of-flight depth sensor on only 11% of the time and estimating depth in between from the RGB camera, a cheap algorithm on an ARM Cortex processor held mean relative error to about 0.7%. Her closing message is that efficient computing extends AI beyond the cloud, and that it requires cross-layer design of hardware and algorithms.