A Single GPU Is Enough: Jeremy Howard on Teaching Deep Learning

Jeremy Howard is the founder of fast.ai, a research group built to make deep learning usable by people who do not have a Google-sized cluster or a three-year graduate degree. Lex Fridman introduces him as a distinguished research scientist at the University of San Francisco, a former president of Kaggle, and one of the first people he names when someone asks how to start. The conversation does not open on neural nets. It opens on a high-school assignment.

The assignment was to study a question: is there a better musical scale than the usual twelve-interval one? Howard wrote a BASIC program on a Commodore 64 that searched other scale sizes, looking for tunings whose harmonies sat closer to exact ratios. A perfect fifth, he wanted, should really be three-to-two, not the approximation you get in twelve-tone equal temperament. That, he says, was the first program he still remembers: handing a search to a machine.

Music has run through his life. He has played saxophone, clarinet, piano, guitar, drums — “whatever.” The favorite is saxophone, especially baritone, or bass if it were not so awkward to haul around. RSI in his fingers later forced him to cut back anything that used his hands, and he talks about returning to it someday with a small, unhidden regret. Fridman likes the pairing of music and code: a brain that drives both with the same creative machinery.

Asked for favorite languages, Howard does not name Python. He names early Microsoft Access. Visual Basic for Applications, he grants, was not a good language. The environment around it was another matter: tables, relationships, queries, forms, event handlers, and calculations wired together graphically, aimed at useful little applications rather than massive scale. He has not seen anything as complete since. Airtable, he says, is a small subset of that idea, loved for good reason.

The through-line is data, not syntax. He has always wanted to take some data, do something with it, and put it back into the world. Relational databases still interest him; programming on top of them now usually means a server, an ORM, and a pile of awkward joints. He mentions Don Syme’s F# work as one of the few attempts to make a database show up in the type system, with tab completion for fields and tables. AppleScript, in those early years, scratched a similar itch: making machines talk to each other.

After Access, the environment he most loved was Delphi — Object Pascal, from Anders Hejlsberg, who had done Turbo Pascal and later did .NET and TypeScript. Delphi was a compiled, fast language that felt as easy as Visual Basic. He is not sure anything quite like it exists now. If you took C# or Java, dropped the virtual machine, and compiled a small binary, you would be close; he hopes Swift, with SwiftUI and cross-platform work, might get there. There is even a Free Pascal project, Lazarus, trying to recreate Delphi. Pascal itself, he adds, is not a nice language. If the question is languages he loves, the answer is J.

Howard sketches three families. One line runs from Alonzo Church’s lambda calculus into Lisp and Scheme. Another is the imperative, object-oriented path through ALGOL, Simula, C, and C++. The third, rarely mentioned, is array-oriented languages, which start with Ken Iverson’s paper *Notation as a Tool for Thought* — a math paper, not a programming one. Iverson wanted notation more flexible, more expressive, and more strictly defined than ordinary mathematics, which he considered awful. That notation became a language. In the late 1950s, good names were still available, so he called it APL: A Programming Language.

Ken and his son later built J on everything they had learned from APL. Howard calls it the most expressive, composable, beautifully designed language he has seen. You generally do not write loops. The whole language is an extreme version of the broadcasting NumPy users already know: a lot happens in one line, and it looks like math. Because one screen of code often holds a whole program, you can keep it in your head and explain it. APL split. J is open source, a niche of enthusiasts. K is not niche: some of the world’s richest hedge funds run it. The entire system is small enough to sit in CPU L3 cache, and it wins every data-processing speed benchmark Howard has seen. The license, he says, is on the order of a hundred thousand dollars per CPU.

If J and K are so strong, why Perl and Python? Pragmatism. He wants to finish things, and that means going where the libraries are. He built the email company FastMail in the late 1990s, and the entire backend was Perl — including a monitoring system and a web framework he wrote himself, because those wheels did not exist yet. Perl, he says, already won: it was everywhere. Then Larry Wall stopped putting the time in, and a project that started with strong leadership could not stay successful without it. Python is less elegant in almost every way, but it has the data-science libraries, so he uses it. He is clear that this is “the best we have,” and that it is not good enough.

What he wants is a language that is, in Chris Lattner’s phrase, infinitely hackable: he, his students, and his collaborators should be able to look at and change everything from the top of the stack to the metal. Python is the opposite. It is so slow that below a certain layer everything is C, and at that point the debugger, the profiler, and the build system all change. That is not hackable. Deep learning, he says, is still primitive; most things do not work very well even when they beat the alternatives. Every domain he inspects — speech, NLP, classification — is full of obviously stupid choices waiting to be fixed. The language is what lets you jump in and experiment, or stops you.

The cost is concrete. For recurrent nets and language, the loop that walks through words has to live in CUDA C, so the heart of the most important algorithm is a kernel researchers cannot innovate on. Convolutional nets, the default architecture for much of the field, almost certainly ought to be sparse; maybe two people use sparse convolutions, because you would have to rewrite the CUDA C layer. Nobody is at fault, Howard says. Nobody has gotten around to it, and it is hard. GPU programming, in any language, still means synchronizing thousands of threads, thinking about grids, blocks, and warps. Optimizing one algorithm that way can be a year’s work for a specialist.

The projects that might fix this sit on compiler research: MLIR, which Lattner is leading, plus Tensor Comprehensions, TILE, TVM, and Halide. Halide split the problem into two domain-specific languages — one for the tensor math you want, one for how to compile it, tile it, and parallelize it — and showed you could cut optimized GPU code by about ten times without losing performance. The hoped-for stack is Swift writing those DSLs in something as compact as J, Swift layers above that, SwiftUI on top. For now it still bottoms out on CUDA and NVIDIA. If AMD ever gets the software right, or if devices from companies such as Graphcore show up, MLIR backends could retarget. The market needs that competition: NVIDIA, he says, overcharges for enterprise cards because nobody else has done the software properly.

TPUs are worse. Google, Howard says, made an explicit choice to make them almost unprogrammable, on the theory that too much IP lives inside them. You cannot program the memory directly; you cannot even look at the code running on the machine that has the TPU. Everything goes through a virtual machine. What remains is cookie-cutter composition of high-level blocks — tedious, awkward, and in his view unnecessary.

fast.ai’s origin is tied to Enlitic, which Howard describes as the first company focused on deep learning for medicine. The number that got him moving was a roughly ten-times shortage of doctors, especially in the developing world, and an estimate that filling the gap at ordinary training speed would take about three hundred years. If models could take on some of the analytics — diagnosis and treatment planning — you might not need every practitioner to be trained to the current peak. The opportunity he cares about is not replacing radiologists in Boston. It is triage on a device in India, China, Indonesia, or Africa, so a health worker with a month of training can get a high-quality read on malaria or tuberculosis and know whether to send the patient to a hospital.

The shortage is specific. Outside South Africa, he says, the entire African continent had five pediatric radiologists; most countries had none. Even if you can get an image made, the person looking at it is a nurse at best. In India and China, almost no X-rays are read by a trained professional, because there are not enough of them. An algorithm that flagged the riskiest five percent for a human would change what medicine can do in places that are building hospitals and buying equipment but will not have the expertise for a long time. He rejects the frame of automating people out. There is already a shortage; making someone ten times more productive does not change unit economics by deleting them, and it ignores the things people still do better than machines. Triage is the useful version: if the device is 99 percent sure there is nothing there, send the patient home.

Why has this been slow? It is new. He started Enlitic around 2014. At RSNA, the world’s largest radiology meeting, he told anyone who would listen that he was doing deep learning; nobody knew what he meant, and nobody was interested. The regulatory frame, the education system, and the idea of what a doctor is were all built for a different world. A small cohort who are both physicians and deep-learning practitioners is only now coming out of PhD programs in Boston and Cambridge. Some doctors have finished the fast.ai course, published, and started journal clubs at the American College of Radiology. It will be a long process. Hospital lawyers, not the statute itself, are often the bottleneck. HIPAA, he notes, does not stand for privacy; it stands for portability, and it was written with gray areas so data could be used thoughtfully. Lawyers see a gray area and refuse anything that might get them sued. The benefit of releasing a hundred thousand de-identified records is abstract — a paper, a thousand extra diagnoses, no named survivor. A privacy failure has a name.

His privacy answer is to do more with less data. Vendors, especially Google and IBM, are incented to prove they need more data, more compute, and more brilliant staff, so you have to trust them. Jeff Dean has said the goal is a thousand times more computation and fewer people. fast.ai’s goal is the opposite: use the people, data, and computers you already have. Organizations often already have enough for state-of-the-art results if they know transfer learning. Recommender cold start is real; music apps used to ask which artists you like. Buying marketing files from a broker to skip five minutes of questions, he thinks, does not justify the privacy externality. Medicine is different: the hospital has your images and records, and you own them. He helped a startup, doc.ai, whose app could pull data from Sutter Health, LabCorp, and Walgreens onto a phone so the patient decides what to share.

Before starting fast.ai he spent a year asking where deep learning’s biggest short-term fruit would be. Kaggle had shown him the field had crossed a threshold: in every area he looked, it was rapidly becoming the state of the art, and twenty years with neural nets had told him that once that happens it happens almost everywhere. He picked medicine, and he is glad Enlitic opened that world, but he could not do everything. It took him a long time even to learn what problems clinicians solve, what data they have, and who holds it. The better move, if the aim is impact, is to put the tools in the hands of people who already have the domain and the data. Most of those people cannot code and do not have three years for a graduate degree. His own background is applied — McKinsey, a decade of consulting — so he takes domain experts seriously. That was the mission: make deep learning quick, easy, and effective for them.

Fridman asks about theory versus practice. Howard does not soften it: most research in the field is a waste of time. Science has a publication problem. Researchers work on what peers already recognize, which means they all work on the same thing, with no incentive to do what is practically useful. The result is minor advances on overstudied questions. Transfer learning, if it got better, would be world-changing — more people doing world-class work with less data and less compute — and almost nobody works on it. Active learning, how to get more from humans in the loop, is his favorite topic and equally unfashionable. Inside companies that have to label data, everyone reinvents it: they notice two classes cause all the mistakes, then notice the model could tell them which examples are hard. Academia has no reason to care about that result.

He has only really written one paper, and he did not write it. His colleague Sebastian Ruder wrote it; Howard did the research. It introduced successful transfer learning to NLP, the algorithm called ULMFiT, and he built it for the course: he would only teach practical NLP, he thought that meant transfer learning, and he could not find examples, so he did it. A prototype of a couple of days smashed state of the art on an important dataset in a field he knew almost nothing about. Ruder offered to write it up; it landed at ACL. Once you show something is possible, he says, people pile in — GPT, GPT-2, BERT followed. Junior researchers still have to pick the safe option, a slight improvement on what everyone is already doing. Nobody, in his view, does anything interesting by picking the safe option. He later helped start a company, Platform.ai, around active learning, and found there was almost no literature to borrow.

Howard teaches two courses a year at USF: Practical Deep Learning for Coders, and a research-level Cutting-Edge course. While he builds them he keeps a large office, big enough for about thirty people, and invites any student who wants to sit with him. It is usually full: twenty or thirty people with nothing to do but study deep learning. During one of those stretches someone mentioned DAWNBench, a Stanford competition to train a model as fast as possible, and as cheaply as possible, on CIFAR-10 and ImageNet. There were ten days left. He almost skipped it, then realized the contest was the same material he was already teaching. He and about four students rented an AWS box with eight GPUs.

He had always tried to avoid multi-GPU work, because accessibility means a single GPU a normal person can afford, not a data center. Eight rented GPUs felt like a fair exception. Within a few days they had CIFAR-10, 32-by-32 images, down to something like ten minutes, and they were on top of both the time and the cost boards — ahead of Google and Intel. They tried ImageNet next, 1.3 million images, aiming for under twelve hours, and got to about ten. Google posted around five hours on a TPU pod. The team kept going. A simple idea helped: why start at 224 or 256 pixels? Train first at 64-by-64, then give the same model a couple of epochs at 224. It is like showing someone a blurry dog, then a sharp one; they already know what a dog is. The accuracy bar was 93 percent, a bit above what a standard ResNet-50 could hit at the time. A distributed version a couple of months later put them at the top of the leaderboard in 18 minutes.

A few years earlier his friend Pete Warden had turned a backyard granny flat into an ImageNet training room and, after a year of work, got the run down to about ten days. That gap is the motive for fast.ai. Howard still thinks multi-GPU and especially multi-machine training are largely a waste of time because they slow iteration. ImageNet itself is often a waste: most people do not train on 1.3 million images, and smaller subsets give the same relative answers. He released ImageNette, a French-spelled easy subset, and ImageWoof, a hard subset of dog breeds. Both train on a single GPU in about ten minutes, and the results transfer to ImageNet nearly all the time. He keeps meeting people who decided not to learn deep learning because they believed only Google could do useful work. None of the big breakthroughs of the last twenty years, he says, required multiple GPUs — not batch norm, not dropout, not the original GANs.

His lab has gone further and shown you often do not need GANs at all: with transfer learning they get GAN-level results in a couple of hours on a single GPU, using loss functions that skip the adversarial game. A student, Jason Antic, used the method in DeOldify to colorize old black-and-white movies on a home GPU in a lounge room, still among the world’s best results. Fridman wonders whether cheap microphones could be fused the way extra phone lenses are. Howard says computational photography already proved the point — Night Sight, fake bokeh, multiple cameras — and the same super-resolution approach should work for audio; nobody has done it yet, and it would be a couple of months of work. They planned to release an audio library to push that along.

The DAWNBench learning-rate tricks came from Leslie Smith, a researcher who, like Howard, cares about training networks quickly and accurately. Smith found super-convergence: certain networks, with certain settings, could be trained ten times faster by using a learning rate ten times higher. The paper did not land in the usual venues. It was not a fashionable academic topic, and deep learning is not treated as an experimental science. In physics, Howard says, you can publish that a particle did something the theory cannot explain and let sixty years of people work on the why. Leslie could not publish “this trained ten times faster than it should have and I do not know why.” Reviewers wanted an explanation. Howard reads more unpublished papers than published ones for exactly this reason. Super-convergence also generalized better: fewer epochs means the model sees the data less often and accuracy goes up.

Smith later figured out a large part of the mechanism; Howard’s group added small tweaks. The main trick is to start at a very low learning rate and increase it slowly, so early steps are tiny and later ones are larger than anyone thought possible. With a few other tricks they can get super-convergence reliably. On DAWNBench they used learning rates far above what people expected to work. The last year of optimizer research, he says, has shown that weight decay, learning rate, and Adam’s epsilon interact in strange ways, and that different parts of a model should train at different rates — discriminative learning rates, especially important for transfer learning. The idea of a single learning rate is already disappearing. If you understand gradients and how they change, you can set each parameter. Fewer knobs means you do not need a deep-learning specialist to train the model, which means domain experts can do more of the work.

Google’s alternative, running thousands of times more compute so that one of many models happens to work, he calls insane. If you understand how models learn, you jump to the good one. Cheaper, more accessible, and it frees human time for interpretation, data gathering, and finding errors. Looking at data is a core part of the course. Before they train, they look. After the first fine-tune of an ImageNet model — five minutes — they study misclassified images, a confusion matrix, and then Google the things the model is mixing up. Models tell you which features matter and which groups you are failing, which makes you a domain expert faster. They also catch leakage. If the top feature is customer ID, you go ask the people who mint IDs and hear that a one is appended when an application is accepted. Use the model, he says, to debug the data.

On cloud hardware, Google’s TPUs and the best NVIDIA GPUs are in the same neighborhood — maybe the TPU is 30 percent faster — but GPUs are far more programmable, so that is where he would spend research time. Getting a GPU is easier than it was: AWS, GCP, and others will rent you one. AWS is still too much AMI-hunting and installing. GCP, at the time of the conversation, had a fast.ai and PyTorch image with the course and Jupyter already running. Better still were one-click platforms such as Salamander, which they owned, and Paperspace, which pop a notebook with the course material preinstalled. When the courses started, the biggest dropout was lesson one, because people could not get an AWS box running. GCP’s $300 of free credit helped; one-click hosts helped more.

The teaching stack moved from Theano and Keras, to TensorFlow and Keras, to PyTorch, then to PyTorch plus the fastai library. Theano and TensorFlow asked you to define a static computational graph first: declare everything the model will do, then later run it on data. You could not debug it, step through it, or poke it in a notebook. PyTorch was not the first to reject that, but it was the first strong challenge: just write ordinary Python, and the library will figure out the GPU. That leap mattered for research and for teaching, and it was necessary for something like DAWNBench, where you have to try ideas quickly. PyTorch is still unfriendly to newcomers — you write the training loop and manage gradients — and it wastes researchers on boilerplate. So they built a layered API: three lines of code to train a state-of-the-art net at the top, and you can drop down, API by API, toward the machine. People have won competitions and written papers with it.

Python still blocks them on recurrent nets that become impractical if you actually change the slow part. The latest course and some research had started to use Swift. He thinks they are about three years from that being practical, and he is willing to wait. A nascent fastai vision library was already running on Swift for TensorFlow. Python TensorFlow, in his view, will not cut it: they copied the interactive surface people liked in PyTorch with syntactic sugar called Eager, without changing the runtime, so a step can be ten times slower. He reads TensorFlow’s source often and does not like what he finds. tf.data looks to him like a mess forced by TPU constraints, a debt you do not pay down without starting over. Swift for TensorFlow, sitting on MLIR and LLVM, could be a different beast. New students, today, should still start with fastai and PyTorch, because they will get the concepts and the actual state-of-the-art techniques fastest. Switching libraries later is a couple of days if you understand the foundations. Swift will not be a beginner’s home for years: no data-science community, and a numeric-computing culture he thinks has made poor performance choices — changing now that Lattner is at Google. The whole lesson-one notebook already runs in Swift in Python mode.

Finishing both fast.ai courses takes anywhere from two months to two years. A very competent coder can do about seventy hours per course. Plenty of people take a year off, enter Kaggle competitions, read Ian Goodfellow’s book, and feel competent at the end. Domain experts often have pedestrian coding skills, so part of the year is simply writing more. The usual bottleneck is coding, not math. People with a heavy classical-statistics background can struggle for the opposite reason: they want fewer parameters and to stare at individual coefficients, which is the wrong intuition. Coders who know no statistics tend to do best. The courses are free. There is no revenue. They had recently won a COGx award for best AI course.

Teaching, he says, was as necessary for his own fluency as for the mission. Watching domain experts from many backgrounds get state-of-the-art results in their own fields convinced him of something he had wanted to believe: anyone can do it. The snobbery that only certain people can code, or are smart enough for AI, is, in his word, bullshit. The differentiator between people who succeed and people who fail is tenacity. A lot of people give up. The ones who do not, pretty much all succeed, even the ones he privately worried were not getting it yet. He now has evidence, seen again and again, not just a preference.

Advice for beginners: train lots of models. Print the inputs, print the outputs, change the inputs, watch the outputs, run experiments until you have intuition. Inference on someone else’s model is not the start; fine-tuning is, because then the model is in your domain. It takes about five minutes on data you care about. Lesson two teaches you to script Google Image Search, build a dataset from scratch, and put a web app online. In class he built a teddy-bear versus grizzly versus brown-bear classifier at basically 100 percent accuracy; scraping took four minutes. Notebook widgets help clean the set and inspect errors. The “share your work” thread had more than a thousand replies. One student beat the best academic paper on Devanagari characters after lesson one. Another classified Trinidad and Tobago hummingbirds, the national bird. Over years, keep training in your domain. The world does not need another person making a slight evolutionary improvement on a crowded benchmark. It needs people who can diagnose malaria, study media bias in language, or find problem areas in fisheries. Even transfer-learning research, he says, needs a problem you actually understand. Studying self-driving without having been in a car is how he would describe a large share of robotics people in that field.

What does it take to build a successful startup? The same thing it takes to become a successful practitioner: not giving up. You can run out of money or time, so keep costs extremely low, save enough to buy yourself a runway, and work on something you understand and care about. The classic failure he sees in this field is a deep-learning PhD trying to commercialize the dissertation. That is a waste of time, because the topic was chosen as an engineering or math exercise, not as a problem someone has. If you have actually been a recruiter, spent your days sifting resumes, know what you were looking for, and can try a model for a few minutes to see whether it matches your own judgment, you are on a real path.

Take venture capital as late as possible, preferably never. His first two companies were self-funded, and he thinks that was the right kind of scary. VC is scarier: people who do this for a living stand behind you shouting grow, and they do not care if you fail, only if you fail to grow fast enough. Building with friends, at a pace that made sense, got them to something large enough that they never had to work again and small enough that no VC would have been impressed. That was enough. In 1999 he started FastMail and Optimal Decisions at the same time, with two different friends. FastMail cost about $70 a month for the server. When disk ran out he put a payment button on the front page: more than 10 megabytes of space was $10 a year. From then on they made money. Optimal Decisions sold million-dollar work, so they sold scoping projects — prototypes that were not free — at $50,000 to $100,000, covering costs and making the client feel the work was valuable. Both companies were profitable at six months.

With VCs he found it hard to do what he thought the company needed rather than what would make the investors happy. They always say not to optimize for their mood; then they get upset if you do not. Optimizing for an exit can be fine, but not at VC scale, which wants a thousand-x. A lifestyle sale at ten million dollars, he thinks, means you have made it. Building something you would happily do forever is a good outcome. Building something you sell in three years is also a good outcome.

Howard uses Anki heavily. Spaced repetition goes back to Ebbinghaus, about a hundred and fifty years ago, who wrote random letter sequences on cards and measured what he remembered a day or a week later. Memory fell on a curve. If he revised just as he was about to forget, the decline slowed; another revision later slowed it again. He derived a roughly optimal schedule. Anki automates that: it asks whether you remembered, and if you did not, it brings the card back much sooner. You cannot fool yourself. If you are not learning, the pile grows, so you have to treat your brain well — mnemonics, stories, context. Piotr Wozniak’s SuperMemo is the modern research lineage; Michael Nielsen had recently written about using it for papers. Howard himself uses Anki for almost nothing except Chinese. Chinese is a conscious decision: he wants to keep it even when he is not in China and cannot practice. Programming languages and papers get the opposite treatment. He tries not to memorize them. He pulls out the important concept, understands it, and either folds it into the library and his habits or throws it away. What he cares about, he uses, so he remembers it.

For twenty-five years he has committed at least half of every day to learning or practicing something new. Colleagues have always hated it, because it looks as if he is not working on what he is supposed to be working on. It also means he does everything faster, because he has practiced a lot. The people he started learning Chinese with were all gone within twelve months; he was still doing it ten years later. Every Chinese card is hung on a story designed to be memorable — funny, disgusting, sexy, or tied to someone he cares about. He also lets himself fail. Medical problems stopped the cards for about three years, and there were other pauses of a few months. Coming back to 18,000 due cards, you either quit or decide to do this every day for the next two years. The surprise was that after three years the Chinese was still in there. Relearning was much faster than learning the first time. Fridman recognizes the same thing with a guitar left idle for a year.

Asked for the next big breakthrough in AI, Howard refuses the question. He does not think it can be predicted. The tools already on the table can solve a lot of unsolved social problems; he hopes many people will learn the toolkit and use it. New technological breakthroughs are not the bottleneck. Human-level intelligence: he does not know, has no way to know, and does not understand why people make predictions with no data. There are more urgent questions. Labor-force displacement is one. The econometric shrug — other technologies did not cause mass unemployment, therefore this one will not — he finds frivolous. He already sees a hollowed-out middle class, and students leaving school with a worse financial future than their parents, which he says has not happened in the last few hundred years. Anxiety turns into despair and even violence. Andrew Yang was running on the issue; Howard is desperately concerned. Every data scientist using deep learning, he says, holds a high-leverage tool. They have a duty to think about humans in the loop, runaway feedback, an appeals process for people the algorithm hits, and explaining the system’s constraints to whoever deploys it. Data scientists often treat those as someone else’s job. He thinks that is wrong. Fridman tells him he is in the right seat to teach them otherwise — and to send them toward literature and history.