Andrew Ng: The Hardest Part of AI Is Rarely the Algorithm

Lex Fridman opens by asking Andrew Ng when he first fell in love with computers. Ng grew up in Hong Kong and Singapore and started coding at five or six, copying BASIC programs out of books so he could play crude shoot-em-up games on his own machine. As a teenager his father, a doctor, handed him books on expert systems and neural networks. The spark was not prestige. It was the discovery that typing could turn into a game, and that a computer might one day look intelligent.

The motive that stuck arrived in a high-school internship in Singapore. He photocopied, did office chores, and treated the shredder as the highlight of the job. The teenager in him thought: if software could automate this, he could go do something else. A lot of his later work, he says, still sits on that theme. Machine learning, as he now describes it, is a way to write algorithms that automate what people already do.

The same instinct showed up in the classroom. Teaching Stanford's machine-learning course to about 400 students a year, he found himself filming the same lecture and telling the same jokes in the same room. Why not reuse last year's video and spend the recovered hours on deeper relationships with students? That question led to the first open online courses that later became Coursera. When the first course went live, about 100,000 people signed up before the videos were even finished. After dinner with friends he would ask himself whether to go home or go to the office. The imagined faces of those learners sent him into a tiny studio, where he adjusted a Logitech webcam, a Wacom tablet, and a lapel mic, and often recorded until 2 or 3 a.m.

Fridman asks what it felt like to go home alone at 1 or 2 a.m. knowing the videos would reach thousands, then millions. Ng says it was humbling, and that he was not thinking about how he looked. From the early days he told the whole team that the number one priority was to do what is best for learners. In the studio the only question on his mind was how to make a concept as clear as possible. Instructors are tempted to teach their own papers so students will cite them. That principle, he says, killed the temptation.

People sometimes asked why he spent so much time explaining gradient descent. His answer was that a solid foundation would serve a longer career better than a tour of his latest research. The popularity of the course also revealed something the narrow AI community had underestimated: interest was not limited to people who go to NeurIPS. Developers in Russia, India, China, South America, and everywhere else wanted in. He can easily imagine more than half of developers someday working on machine learning in some form, including the infrastructure around the model, the way most software engineers now have some grasp of the cloud.

None of that was an overnight success. He first posted Stanford lectures on YouTube, then built four or five forgotten websites that let the team learn what worked. One feature he was proud of let two people log into the same browser so both would get credit for watching together. They rolled it out at a San Francisco school, Sacred Heart Cathedral, to about twenty users. Zero people used it. Online learners wanted to pause and rewind alone, not study as a tagged pair. A design that looked brilliant on paper, and was not even that hard to build, produced no impact at all.

Fridman notes that biologists, chemists, physicists, and mechanical engineers now sit on large datasets they never planned to program against. Ng reaches for a historical analogy. Once only a small slice of humanity could read and write, and it was easy to argue that a few authors and a few public readers would be enough. Widespread literacy raised the bandwidth of human communication, including writing for an audience of one, as in an email. Computing, he argues, is still in the scarce-scribe phase. Coders mostly have to write for large audiences because so few people can write at all.

He would like the owner of a mom-and-pop store to customize the TV display for this week's special. That is human-to-computer communication catching up with human-to-human writing. Talking to people in ordinary jobs, he used to lack a crisp reason they should learn to code. Machine learning and data science changed the pitch. The number of people with a concrete daily use for data science, he suspects, may be larger than the number with a concrete use for traditional software engineering. A shopkeeper who can analyze sales and customers gets value immediately, perhaps more than from writing software. Recruiters, accountants, and factory workers could use a bit of data science at work the same day they learn it.

Fridman then asks why, in a world of PowerPoint, Ng still walks on stage with a marker and a whiteboard. For math, building an equation one piece at a time while talking improves understanding. Writing is slow, which is a genuine downside if you need a long sentence. That slowness is also the point. It forces him to reduce everything to a few basics. A great talk, he says, is not one packed with content. It is one that clearly says a few simple ideas, and the whiteboard enforces that minimalism.

Pieter Abbeel, now one of the world's leading roboticists and reinforcement-learning researchers, was Ng's first PhD student. Ng is explicit about the debt: his long-term professional success sits on early work that Abbeel was critical to. Abbeel's thesis used reinforcement learning to fly helicopters, including the inverted aerobatics that still look astonishing in old Stanford videos from around 2008. What most people never see is how hard the research was. Abbeel, Adam Coates, Morgan Quigley, and Ng spent years on versions of the helicopter that simply did not work.

The worst bottleneck was not the learning algorithm. It was figuring out where the helicopter was while it flew upside down. One GPS unit failed because the antenna pointed at the ground. Two units, one facing up and one down, failed because the downward unit could not synchronize during a fast flip. Quigley spent about a year on specialized hardware to interpret GPS signals. It did not work. Ng remembers sitting in his office with Abbeel, looking at the latest failure, and saying, "darn it, what now?" The fix, with Coates's help, was cameras on the ground. Once an external system solved localization, the team could finally focus on reinforcement learning and control.

At the time there were many theoretical RL papers and almost no physical applications, which is why the helicopter became famous. Fridman asks what kept Ng on such dirty applied work when theory was the prestige path. Ng likes theory, but he enjoys it more when he can see a line to helping someone. He once asked a mathematics professor why he did what he did. The professor, stars in his eyes, said he was discovering truth and beauty in the universe. Ng respects that completely. He is simply more motivated when the work of his team helps people. The world needs both types. He is one of them, and he was willing to burn a year of failed GPS hardware to get a flying system.

Fridman asks what Ng saw in deep learning before it was fashionable. Ng splits the early years into what they got wrong and what they got right. The miss was how central unsupervised learning would be. Early Google Brain poured effort into it. Around 2005, after NIPS, he and Geoff Hinton sat in a cafeteria outside the conference. Hinton pulled out a napkin and sketched an argument that convinced a generation. The brain has about 10^14 synaptic connections. A human life is on the order of 10^9 seconds. If each weight is only one bit, you need to learn 10^5 bits per second. Ng had a one-year-old daughter. "I think I'm a very loving parent," he says, but he is not going to give her 25 labels a second. Most of what we know, the argument went, cannot arrive through supervised labels.

He still thinks unsupervised learning matters. He also thinks that bet was the wrong near-term path. The part they got right was scale. Adam Coates, then in Ng's Stanford group, ran experiments and drew a chart: model and data size on one axis, performance on the other, the curve heading up and to the right. That single figure gave Ng the conviction to pitch a project at Google with Sebastian Thrun. It became Google Brain. The idea that bigger datasets would yield better performance was simple and, at the time, controversial. Well-meaning senior friends tried to talk him out of it. This was not a new architecture, they said, just scaling, a bad career move. If you want a breakthrough, Ng replies, you sometimes have to have conviction and do the thing before it is popular.

Fridman brings the old fight into the present. Some people now say datasets are large enough and the frontier is sample efficiency and better learning mechanisms. Ng's answer is that both matter, and it depends on the problem. On some datasets we are near Bayes error or past human level, so more data hits a ceiling. On many others we are still far from either, and bigger data with existing neural nets, without a new architecture, will keep moving the needle. Recent language-model breakthroughs mixed novel architecture with scale. Looking at GPT-2 and BERT, he says scale was a large part of the story, and the quietly neglected part is the weakly supervised quality of the web data those models were trained on.

Code has mature version control. Data does not. Ng remembers emailing C++ files around before anyone used CVS or Git. He thinks the industry is still that immature about managing data, especially messy data. Manufacturing makes the gap impossible to ignore. On visual inspection, his team will show a plastic part to one inspector who is sure it is an unacceptable defect. A second inspector, equally sure, calls it a tiny mark that should ship. Sometimes the same person, shown the same part in the afternoon, confidently reverses the morning's call.

If one person cannot agree with himself across a few hours, what is a learning algorithm supposed to do with the label? Consumer internet companies with a billion users can average the noise away. Other industries do not have big data. They have small data: maybe a hundred examples of a defect. If ten of those labels are wrong, that is 10 percent of the dataset. Cleaning it up, deciding who labels, how disagreements are resolved, and how much human effort to spend is not an algorithm paper. It is process. At Stanford he increasingly tells students to find their own end-of-term project and their own data, rather than download a clean public set. The clean set is easier. The messy one is where you meet the real world.

Ng was then running three efforts at once. DeepLearning.AI builds courses so individuals can break into the field. Landing AI helps established companies adopt AI. AI Fund is a studio that creates companies from scratch. Asked how a beginner should start, he points to the Deep Learning Specialization on Coursera, which walks from neural nets through convnets, sequence models, and attention, with programming exercises in Python. The math bar is deliberately low: high-school math and matrix multiplication are enough; calculus helps intuition but is not required. The official length is about sixteen weeks, though some people finish in under a month. Anyone for whom payment is a hardship, he stresses, should apply for financial aid and take it free.

What he is proudest of in the specialization is the practical know-how: how to tell if you are overfitting, when to collect more data, and when not to bother. He still sees engineers spend six months gathering data because more data always feels valuable. A few tests might have shown, months earlier, that more data would not fix this problem, and that time should have gone to the architecture or another idea. He compares it to learning to code. Syntax matters, but the larger skill is how you group lines into functions and abstractions. At Carnegie Mellon he had a friend who "debugged" C++ by deleting every line with a syntax error. That is a joke in ordinary programming. In machine learning, debugging really is a different discipline.

You cannot binary-search a neural net or single-step it in a debugger. The questions are why it does not work yet, and whether you can expect it to work. People who can answer those questions systematically are easily 10x, maybe 100x, faster. The menu is familiar: change the architecture, add data, add regularization, try a different optimizer, try a different kind of data. What is hard about deep learning, as with mathematics, is not one operation. It is a stack of ideas. Miss one and you lose the prerequisites for what comes later. The course tries to break each concept into a unit small enough that the later steps still have a floor under them.

Fridman observes that many students' first "aha" with neural nets comes from deep reinforcement learning: a network that knows nothing, explores, and learns. Ng still teaches RL in a Stanford class and remains fond of the field. If he is trying to teach the most useful techniques for today, though, he shrinks the time on it. It is not what is working in production. A group of senior machine-learning researchers once stood around asking for the first real deployed RL application. There were emerging examples. There were not many great ones, at least outside games. He has a team looking at RL for some robot control tasks, but as a share of impact it is still small.

The teaching tension is real: the tool that ignites passion is not always the tool that deserves the most hours. He would rather use RL as a vivid picture of learning from scratch, then spend the core of a course on supervised learning and real datasets. Even inside machine learning, he argues, it would be sad if everyone learned only deep learning. Some days his manufacturing team uses neural nets. Some days they sit with PCA residuals. Sometimes they use a probabilistic graphical model. Those tools have huge industrial impact and far less academic chatter. Diversity of methods is how you find the right instrument for the job.

Asked for the most beautiful idea in deep learning, he does not pick scale. If his only job were academic research with unlimited budget and no short-term pressure, he would spend all his time on unsupervised learning. The branch that currently excites him is self-supervision. Grab unlabeled images, rotate each by a random multiple of 90 degrees, and train a net to predict the original orientation. Labels are free and infinite because you made them. Researchers then transfer the hidden representation to other tasks, and it works surprisingly well. Variants include Jigsaw, which shuffles a 3-by-3 grid and asks which of the 9-factorial permutations it came from, and word2vec-style missing-word prediction. OpenAI, Facebook, Google Brain, DeepMind, and others were all in that stream. He thinks it is one piece of the puzzle, getting close to real impact first in computer vision and video, after a decade in which sparse coding and independent component analysis lost attention to how well supervised learning was doing.

Fridman asks for a daily schedule for someone learning deep learning. Ng's answer is regularity. His team sends the newsletter *The Batch* every Wednesday, so readers already know when to spend a few minutes catching up. He himself studies on Saturday and Sunday. He does not wake up on Saturday and decide whether he feels like reading. It is just what he does. Brushing his teeth takes two slightly annoying minutes, but as a habit it costs no cognitive load. Having to decide every morning would be the exhausting part. He even eats the same thing each day, one less decision.

Fridman adds his own guitar rule: at least five ridiculous minutes a day, which over a year or two produces results that do not look ridiculous. Ng says bursts and all-nighters do not compound because you can only do them a limited number of times. Reading research papers is the same. The power is not reading two papers. It is a few papers a week for a year, after which you have read about a hundred and actually learned a lot. Coursework is the efficient way to start a field; his Stanford PhD students who jump straight into research are often told to spend the first couple of years taking classes. After the courses run out, the next step is small projects, then papers and blog posts. Start tiny, at the individual level and at the company level.

When he wants to learn something deeply he still takes handwritten notes. He recently took a Coursera course on clinical trials with a Moleskine at his desk. Handwriting is slower, so you cannot transcribe. You have to recode the lecture into fewer words, and that deeper processing is what produces long-term memory. Typing is better than nothing; taking a class with no notes is worse. Compared with handwriting, typing makes people more likely to copy verbatim and retain less. The same instinct shows up in how he edits videos: every word has to fight for its life, because a wasted minute, multiplied by millions of viewers, is a huge loss of other people's time.

On careers, Ng refuses a single path. A top-50 PhD program can be a great experience. So can a job on a top team. If the goal is a professorship at a leading university, the doctorate is required. If the goal is to found a technical company, going and building it is also a good option. What affects your day more than academia versus industry, or this logo versus that one, is who you interact with. Walk into a giant building and the sign above the door is not the point. The point is the ten people, the thirty people, you talk to every day. We become more like the people around us. Great peers make you faster.

His practical advice after an offer is blunt. Ask who your manager is, who your peers are, and who you will talk to. If a company says there is a rotation system and they will figure it out after you join, he finds that a worrying answer, because you may land on a team doing work that does not excite you. Finding those people is a hard search, in the same family as finding friends or a partner, which is why you should insist on the question while you still have leverage. He gave an hour-long career talk in Stanford's CS230, including job-search tactics, and the video is public.

AI Fund exists because, running Baidu's AI group, Ng had two jobs: support existing businesses, and systematically start new lines, including a self-driving team and a smart-speaker effort announced before Amazon's Echo. The second job was the fun part. The fund is a startup studio that brings in founders, sometimes with an existing idea, and tries to make company-building repeatable. Customer obsession is the filter: in the long run only the customer votes. They iterate on how many customer calls count as validation, how to tell whether a one- or two-year-old algorithm will survive production, and how a team picks up domain knowledge in education, health, or manufacturing. The deeper offering is against loneliness. Most founders do not know when to hire a first VP of engineering, how to do B2B sales, or how to market without lighting money on ads. An ecosystem at the hard decision points raises the success rate. They also kill ideas on purpose. One business case and revenue plan were fine; the product would have made people watch more of a narrow kind of video. Ng looked at it and said he did not want to do it.

Landing AI's job is to help companies that already exist. Ng calls AI a general-purpose technology that will transform every industry. Software and the internet are already a long way through that change; even firms outside the top handful have some machine-learning capability, or are getting there. Look at manufacturing, agriculture, healthcare, logistics, and transportation, and the opportunities are huge with very few teams on them. A McKinsey estimate he cites is $13 trillion of global economic growth, against a U.S. GDP of about $19 trillion; other estimates say $16 trillion. The number that matters to him is the structure: a large share of that impact lands outside the software internet sector.

In factories, people still stand at the line and inspect smartphone parts for scratches and dents with the naked eye. A camera plus deep learning can raise yield, quality, and throughput at once. The practical problems are not the ones in papers. Datasets are tiny. Lights go on and off. A factory burned and was rebuilt, wiping the training environment. A tree outside a window was cut down and the lighting changed, and the model fell over. Academia is still weak when the test distribution drifts far from the training set, even with work on domain adaptation and transfer learning. The learning model may be 5 percent of the lines of code, or less. The rest is software engineering: reliability, maintenance, and interfaces to the existing plant.

He has heard the same conversation too many times. The ML person says the algorithm does well on the test set. The business person says thank you, but the model sucks, it does not work. They are not talking about the same object. Machine learning, he says, is automation on steroids. Automate visual inspection and you may have to redesign the surrounding jobs: if the model flags a defect, do you scrap it, rework it, or send it to a human? In a tech company, a downed speech system sets off alarms and a leader can point at twenty engineers. A factory does not have twenty ML engineers on call. Maintenance, audit, and incident response all have to be invented, often without shared terminology yet.

The first step for a company, laid out in his public *AI Transformation Playbook*, is to start small. He has seen far more failures from starting too big than from starting too small. Google Brain itself was controversial: people thought deep learning had been tried and had failed. His first internal customer was Google's speech team, not search or ads. The second was Maps, reading house numbers from Street View. Only after those two wins did he have the serious conversation with ads. The first GPU server sat under someone's desk and taught them how multiple users share a pool of GPUs, a lesson that later scaled. Small projects buy trust, and they teach the organization what the technology actually does.

Fridman asks whether, beyond applied automation, Ng shares the older dream of human-level or superhuman intelligence. Ng would love humanity to build AGI. Whether that takes 100 years, 500, or 5,000, he finds hard to estimate. On existential risk he says he does worry about the long-term fate of humanity. He also worries about overpopulation on Mars, just not today. After a talk, someone cut a video accusing him of being heartless toward dying children on Mars. He apologizes to the future viewer: he cares, he just does not know how to work on that productively now. Alignment debates and the paperclip problem, in his view, are a huge distraction from harder problems we can actually address.

Self-driving cars are his example. People treat the trolley problem as the central ethics puzzle. How often, he asks, do you face that dilemma while driving? Autonomous cars will hit it about as often as we do. The real failure is a big white truck across the road, when the right action is to brake, and the system does not. Bias is a live issue. So is wealth inequality. The internet accelerates concentration of power because data and compute centralize, and winner-take-most dynamics are spreading industry by industry, the way Uber and Lyft hit taxis. Tremendous wealth is being created; the open questions are how it is shared and how to help people whose jobs are displaced. Education is part of the answer and may not be enough.

He worries about teams, perhaps accidentally, making noise about distant problems while ignoring concrete ones. It is more fun, when a regulator says your product is causing harm and the fix may hit revenue, to talk about promising not to wipe out humanity through this interface. Looking back, he has made so many mistakes that every insight arrives with the thought that he should have had it five or ten years earlier. Sometimes he reads a book and wishes he had read it a decade before, then discovers it had not been written yet. Discovery keeps finding things that look obvious in hindsight and always took too long. Asked for pride and happiness, he names two: his daughter Nelva, with whom he can never spend enough time, and helping other people. The meaning of life, as he puts it, is helping others toward their dreams and making humanity more capable. Fridman closes with Ng's own test: if what you are working on succeeded beyond your wildest dreams, would you have significantly helped other people? If not, keep searching.