Bigger Models, Cleaner Splits and Human Baselines: Andrew Ng's Practical Recipe for Applied Deep Learning

Ng opens by saying he dropped the idea of a visionary talk and chose instead to share lessons from running Baidu's AI team, a group of about a thousand people working on vision, speech and language. His aim is to give attendees patterns they can use at work the following Monday. He adds that parts of the machine learning workflow, such as how we think about bias and variance, are changing in the deep learning era.

To explain why long-known neural network ideas are working only now, he plots performance against the amount of data. Traditional algorithms such as SVMs and logistic regression tend to plateau, as if they did not know what to do with all the data. As the internet, mobile and IoT pushed society to the right of that curve, small, medium and then large networks kept improving, and only a large network can absorb that much data.

Ng cautions that in the small-data regime the ordering of algorithms is unclear, because it depends on who engineers features more diligently. The trend is clear when data is large. Since reaching top performance needs both a big network and lots of data, he says it now makes sense to put an AI team and a computer systems team side by side, because high-performance computing knowledge is too specialised for a researcher to pick up alongside machine learning.

Ng offers a mental map of the field in four buckets. These are general densely connected models, one-dimensional sequence models such as RNNs, LSTMs and GRUs, image models built on convolutional nets, and an "other" bucket containing unsupervised learning and reinforcement learning. Almost all current industry value, he says, comes from the first three. He is excited about the fourth as the possible future of AI, particularly unsupervised learning, and encourages research there.

The second major trend is the rise of end-to-end learning for rich outputs. Machine learning used to output a single number or label, but now systems can output image captions, transcripts from audio, translated sentences, synthesised images and even audio, as in the recent WaveNet paper. Ng says these capabilities were not buildable three or four years earlier.

He explains the idea with speech recognition. Traditional systems moved from audio to hand-engineered features and phonemes before producing a transcript, while the end-to-end approach lets a neural net map audio directly to text. Putting that into a production speech system was controversial at first, and Ng recalls being shouted at by a linguist after calling phonemes a fantasy of linguists. He says the community has since come around.

The weak point of end-to-end learning, Ng says, is that it needs a great deal of labeled input-output data. He uses the example of estimating a child's age from a hand X-ray. A non-end-to-end approach would use deep learning to find the bones, measure their lengths and apply a simple formula, which lets engineers inject known knowledge about bone length and age. He thinks a pure image-to-age model would struggle because there are too few annotated X-rays.

He adds that for suitably appealing inputs and outputs you can nearly always train a model and publish a paper, but that does not make it a good idea. Self-driving cars show the point: most teams use a pipeline that detects cars and pedestrians, plans a trajectory and then computes steering. Going directly from image to steering might work with enough data, and Ng says he would not claim it will fail, but he is cautious today because nobody has enough data. He warns that cherry-picked examples can make weak ideas look good in demos.

When a trained model underperforms, Ng says, the skill lies in choosing among options such as more data, longer training, regularisation, a bigger model or a new architecture. He recommends measuring three numbers: human-level error, training error and dev error. With human error at 1%, training error at 5% and dev error at 6%, the large gap to human level signals high bias, so train a bigger model or longer. With training error at 2% and dev error at 6%, you have a variance problem and should add regularisation, early stopping or more data.

He turns this into a flowchart. First ask whether training error is high, and if so fix bias with a bigger network, longer training or a new architecture. Once training error is acceptable, ask whether dev error is high and if so fix variance with more data, regularisation or a new architecture. Ng stresses that the new architecture is the hard option, while the simpler moves in the flowchart drive a lot of progress.

Ng argues deep learning has changed this picture. Because you can nearly always train a bigger model or add more data, you more often have a way out of whatever problem you face. People therefore talk less about the bias-variance trade-off, since the coupling between the two is weaker and you can reduce one without hurting the other. Training a bigger network was far harder to do with logistic regression, where you had to come up with more features.

A bigger model strains systems, and more data has led to heavy investment in labeling and in automatic data synthesis. Ng describes it as a small but growing area where hand engineering is needed to produce training data for a big network. For OCR, you can paste random English words in random fonts onto random internet images. He notes that this only works after a lot of tuning of blur and contrast, and that a student of his spent months with little progress before getting the parameters right and building one of the best OCR systems of the time.

Speech works similarly: take clean audio and add recorded background noise such as car noise, which simply superimposes the waveforms and has a dramatic effect on performance. He also mentions Stanford students synthesising ungrammatical sentences for grammar correction, and video games as a source of nearly unlimited data for reinforcement learning. As a caution he cites cars from Grand Theft Auto, where twenty cars look fine to a person but form a very impoverished dataset for a learning algorithm.

For company practice, Ng strongly recommends a unified data warehouse. At Baidu, he says, data belongs to the company rather than to a team, and user data goes into one logical warehouse even if it is physically distributed. The debate should be about access rights and privacy, not about whether to bring the data together, because gathering data from many organisations is slow and painful.

The old 70/30 random split came from an era of small academic datasets, Ng says, whereas in production the training and test distributions often differ. His example is a speech-enabled rear-view mirror in China, where Baidu had around 50,000 hours of varied speech data but only about 10 hours recorded in the actual in-car scenario. In the deep learning era, he notes, it is increasingly common to pour all the data into one model rather than build separate ones, as long as the model is big enough.

His advice is to make dev and test sets come from the same distribution, so the 10 in-car hours would be split into about five hours each. The dev set is effectively the problem specification, and if a team tunes against it for months only to find the test set different, much work is wasted. Hence the walk north to Oakland analogy. He also says that once the test set is set you should not change it, even if you collect or synthesise more training data.

He proposes carving a small "train-dev" slice, such as 20 hours, out of the training data. This gives five numbers: human level, training, train-dev, dev and test error. The gaps separate the sources of error: human to training error is bias, training to train-dev is variance, train-dev to dev is train-test mismatch, and dev to test is overfitting the dev set. He notes academia has not studied mismatch much, though he says it matters to production builders, and he also stresses that bias and variance sound simple but take a long time to master.

For mismatch, the fixes he lists are to get more data similar to the test set, or to synthesise or augment training data to look more like it. If dev error is fine but test error is high, you have overfit the dev set and need more dev data. Ng says people who deeply understand bias and variance can drive very rapid progress, even though it is less glamorous than new architectures.

Ng points out that the whole analysis benchmarks against human-level performance, which has become a common theme in applied deep learning. Across many projects, progress is rapid until a system reaches human level and then becomes much harder. He gives reasons, helped by audience guesses: there is a theoretical limit on performance, which he calls the optimal or Bayes error rate, set by noisy audio or blurry images, and humans are already good at many such tasks.

Below human level, he says, you have better ways to make progress. You can get labels from humans, do error analysis by examining what the algorithm got wrong, and more easily estimate bias and variance. If training error is 8% and dev error 10%, you cannot tell what to fix, but humans at 7.5% suggest a variance problem and humans at 1% suggest a bias problem. Past human level, you lose that proxy, though you can still look for subsets, such as one kind of accented speech, where the system remains worse than people.

To define human-level error, Ng poses a quiz about reading medical images, with options from a typical person to a team of expert doctors who debate each case. His answer is the team of experts, at 0.5% error, because the goal is to estimate the optimal error rate and so get the best estimate of bias. In practice, he says, data labeling will use cheaper tiers of annotators, and papers often claim to beat the human level using a looser definition. He adds that comparing to humans matters more now simply because systems are getting close to human level.

Ng notes that Silicon Valley has established workflows for designing desktop and mobile apps, such as wireframes and visual design, but lacks good processes for AI products. It is unclear how a product manager should specify the accuracy needed for a cat detector or the definition of a self-driving car. He says he sees new processes being invented, and he is often asked what AI can actually do.

His first rule, which he admits is imperfect, is that a task a typical person can do in under a second of thought has a good chance of being automated with deep learning. Examples include judging whether a face is smiling and transcribing a short audio clip. The rule works less well for language, where humans take longer. Baidu's product managers deliberately look for such tasks, and a student noted that if a human can label in under a second, you can collect lots of data.

The second rule is that deep learning is valuable for predicting the next outcome in a sequence of events that happen over and over. He cites predicting whether a user will click an ad, probably the most lucrative application today, and estimating food-delivery times from many past orders. He admits that these rules are far from perfect, but finds them useful for communicating with product people.

The question Ng gets asked most is how to build a career in machine learning, and he admits he has no great answer. After a course, he says, a common path is to work on a project alone, and he respects Kaggle as a place to learn. He then describes the reliable PhD-student process: read a lot of papers and replicate results. In his experience, after somewhere between 20 and 50 papers people start having their own ideas to push the state of the art.

He pairs this with a warning about working on AI. It is not people in airy rooms discussing the future of humanity at whiteboards, and most of the work is dirty work: downloading and cleaning data, tuning parameters, debugging stack traces, optimising databases and hacking GPU kernels. People stall if they refuse to do it, but they also stall if they only do the dirty work. The most reliable formula he has seen is to do both, with the dirty work for the team and papers in parallel.

Ng tells the audience that studying instead of watching TV next weekend brings no short-term rewards, since the boss and peers will not notice. But doing it weekend after weekend for a year makes you really good, and he says everyone he knows who became great did the same, including late nights tuning hyperparameters. He closes by comparing AI to electricity, saying people with these skills are in a position to transform many industries, and that the path from AI work to helping people in areas like transportation, healthcare and logistics is getting clearer.