Craftsmanship, First Principles and the Road to Self-Driving: Jim Keller on Working with Elon Musk

Keller opens by describing what he sees as Elon Musk's great insight. Most people are constrained by what they already have: they know how a thing works and make small tweaks to it. Musk instead asks what he actually wants, for example which configuration of atoms, and then figures out how to build the machine that makes it. Keller says almost nobody thinks this way.

Fridman, who notes that Keller was one of the key early people on Autopilot's hardware side, asks whether vehicle autonomy can follow the kind of exponential improvement Musk expects. Keller answers that the computer Tesla needed to build was straightforward. Whether it must be two, five or ten times faster is, to him, a matter of time or price in the short run.

He adds that you do not have to be especially smart to drive a car, so it is not a super hard problem. The big safety issue, he says, is attention, and attention is something computers are very good at. That framing sets up the disagreement that runs through the rest of the conversation.

Fridman pushes back, saying humans take their vision system for granted. The task is not just detecting objects and estimating distance from video: it is understanding the whole scene, filling in gaps and inferring cars that are occluded. When someone cuts you off, he adds, your brain builds theories about why they did it, and a driver arguably needs mental models of pedestrians, cyclists and other cars.

Keller replies that if you need that narrative to drive, then current Autopilot systems cannot do it. But if cars are ballistic objects with probabilistic changes in speed and direction, and roads are fixed and mappable, then you can place every object and calculate every trajectory thoroughly. Autonomous systems will remember the pothole that humans keep forgetting, and they are happy to maximize the givens.

He compares this to a human driving to work for the 50th time and not remembering the trip, because the person is effectively on autopilot. Cars, he says, have no theories about why they got cut off and so never stop paying attention. On merging into traffic, Keller says people would be surprised how simple the calculation is, while Fridman says he would be surprised if it were. They leave it there.

Keller offers a rule of thumb: progress disappoints in the short run and surprises in the long run. In ten years he expects self-driving to be taken for granted, like GPS, a satellite system that once seemed like a big deal and is now in everything. Fridman holds that systems involving human behavior are more complicated than we give them credit for, and Keller jokes that this is just what a human would say.

Fridman hopes that, whoever is right, machines will be so much better at things like attention that the overall safety picture improves. Keller calls himself a big believer in safety and points to existing features such as cruise control that stops you hitting people, and lane keeping. If you look at the Pareto distribution of accidents, he says, knocking off something like 80 percent of them is super doable.

Asked about the intense scrutiny autonomous vehicles get from media and the public, Keller describes talking to both American and European regulators. He worried that rules would bake in specific technologies, since a literal reading of the brake rules implies hydraulic brakes. The regulators told him they cared about scenarios: head-on and offset crashes, not hitting pedestrians, not leaving the road, not running red lights.

They also had data on which scenarios injure or kill the most people, and Keller says the conversations were about what the right next step was. He adds that Musk is interested in autonomy for freeing people's time and attention as well as for safety. Keller finds it philosophically correct that people expect autonomous systems to be safer than humans and scrutinize every accident.

Asked what is special about designing a chip for cars, Keller says the first challenge is that the machine learning team's algorithms change quickly, so an accelerator may turn out to be the wrong one. His rule of thumb is that a really good general-purpose computer scores one, GPU designers deliver about five times more for the same silicon because parallelism is given rather than discovered, and special accelerators add another two to five times on top.

That gain comes from knowing the math is always 8-bit integers into 32-bit accumulators, a narrow subset of mathematical possibilities. Teams still try to keep the hardware programmable, because a new algorithm could be so much more effective that you would have been better off on a GPU. Keller describes this as a creative tension between wanting specialization's benefits and avoiding being over-specialized.

The second challenge is cost. Keller notes that some recent startups put a server in the trunk to replace the driver, with a ten or twenty thousand dollar budget. Musk's constraint was to put a computer in every car, whether or not people buy autonomous driving, and hitting that target forced complicated system-design choices such as how many camera ports to provide and how to handle safety.

Keller compares the work to a violin maker. Stradivarius and the musicians are incredible, he says, but the maker picked the wood, sanded it, glued it and waited for the right day to apply the finish. Most engineering, even for a genius craftsman with the best techniques, is that kind of careful work, and he says smart humans, and in fact all humans, like doing it. He mentions that he dug ditches in college and got really good at it.

He points to an expression, complex mastery behavior: learning is fun, rote simple work is not, but complicated steps you are good at are satisfying. Building a resource-constrained automotive computer is therefore craftsmen's work, with thoughtful trade-offs rather than new mathematics. Once you pick an architecture and look inside, he adds, you see adders, multipliers, memories, transistors and wires.

Keller extends the point to factory work. He recalls a day when an assembly line was shut down and some tough workers were bummed because they had reorganized the line and were about to set a record. Attaching trim to a moving car in 25 places in a minute and a half is, he says, unbelievably complicated, and he jokes that it may be harder than driving. Fridman and Keller banter over whether building a machine that drives is harder than driving, and Keller notes the machine is the hard part.

Asked what he took from working with Musk, Keller says he liked how Musk thinks. You believe you understand the first principles of something, then you talk to Musk and realize you have not scratched the surface, because he holds that whatever you have built is a local maximum. Keller recalls a friend who invented a better electric motor, and says he told him that visiting superintelligent aliens would probably not be asking who built the motor.

Keller says Musk is good at taking everything apart and looking at it without assumptions, including constraints. He recalls Tesla employees gathering to watch SpaceX land its rockets, where around five hundred people came and some cried. Getting rid of your assumptions is painful, Keller says, but people look back glad they went through it. He adds that he reads books, since authors condense twenty years of passion into 200 pages, and that after reading 20 management books he had read more than the other VPs and half the ideas worked first time.

On the closing question of whether autonomy can be solved in years rather than a century, Keller answers definitely. His analogy is speech recognition, which failed for a long time with frequency-domain analysis and then worked well once deep learning arrived, after several iterations. Data, computation and algorithmic understanding keep improving, and he says radar already helps keep cars from running into things.

Fridman repeats that adding humans changes the problem from ballistics to something more complicated, and Keller agrees he could be proven wrong. He observes that cars respond slowly in physical terms, while human beings operate about half a second behind reality, which nobody fully understands. He expects pleasant surprises given the rate of improvement in computing, software and hardware.