Elon Musk: Every Driver Input Is an Error Tesla Autopilot Must Eliminate

Lex Fridman frames the episode as a first conversation with Elon Musk, the CEO of Tesla, SpaceX, and Neuralink. It followed a paper from Fridman's MIT group on drivers' functional vigilance while using Tesla Autopilot. The Tesla team reached out and offered a podcast. Fridman accepted with full control of the questions and of what would be released, and he says he edited out nothing of substance.

He is careful about independence. He had never spoken with Musk before, Tesla had never funded his research, and he had never owned a Tesla vehicle or Tesla stock. The episode is not a scientific paper, he says, but a conversation meant to understand how the guest sees the world. They agree on some things and disagree on others.

The clearest split is camera-based driver monitoring. Fridman, who works on human-centered AI, believes that if it is implemented and integrated well, inward-facing cameras help in both the short term and the long term. Musk's focus, as Fridman hears it, is on making Autopilot so statistically safe that concerns about human behavior and psychology drop out. Fridman wants a rigorous argument in industry and academia that makes driving safer, even if the two men do not land in the same place.

Fridman starts with the original dream. When Autopilot hardware first went into Tesla cars in 2014, what was the system-level vision? Musk answers by placing Autopilot inside a larger industry shift. The automobile business, he says, is going through two massive revolutions at the same time: electrification and autonomy.

In that future, any car that cannot drive itself would be about as useful as a horse. Not useless, he adds, just rare and somewhat idiosyncratic. People still keep horses, but they do it on purpose. He treats full self-driving as a matter of time, not of whether it happens. If Tesla sat out the autonomy revolution, its cars would become less useful relative to cars that drive themselves.

He puts a price on the gap. An autonomous car is arguably worth five to ten times more than one that is not, at least for the next five years and perhaps ten. From that premise, joining the autonomy race is no longer a feature choice. It is a survival choice for the product.

One early Autopilot design choice still puzzles outsiders: Tesla draws what the combined sensor suite sees onto the instrument cluster, and on Model 3 onto the center screen. Lane lines, nearby cars, and traffic lights appear as a clean vector picture. Fridman asks what the thinking was, and whether there was a debate.

Musk's answer is practical. The whole point of the display is a health check on the vehicle's perception of reality. The car takes cameras first, plus radar, ultrasonics, GPS, and an IMU, then renders that information into vector space as objects with properties. A driver can glance out the window, glance at the screen, and confirm whether the car knows what is going on. He calls that "an extremely powerful thing" for people trying to become one with the system.

Fridman pushes further. Computer vision is probabilistic at the edges. Has Tesla considered showing uncertainty, so ordinary drivers build an intuition for what the model is unsure about? Musk says his own car always runs a debug view. One mode, augmented vision, draws boxes and labels around recognized objects. The other, internally called the visualizer, sums every sensor into a vector-space world with no pictures at all. He thinks most people would not know what they were looking at. The public display is optimized so someone who knows nothing about computer vision can still tell whether the car understands the scene. The development build he drives is, in his words, gibberish to most people.

Fridman splits Autopilot into three technical piles: neural-network algorithms, training data, and hardware. Money and time are finite. What deserves the most resources? Musk does not pick a favorite so much as describe a flywheel Tesla already has. The cars automatically generate vast amounts of data because each one carries eight external cameras, radar, usually twelve ultrasonic sensors, GPS, and an IMU.

He puts about 400,000 such cars on the road, approaching half a million. He is not sure how many other vehicles worldwide have an equivalent sensor suite, but he would be surprised if the number exceeded 5,000. That, in his arithmetic, means Tesla has 99 percent of the data. The inflow is massive. Data of that scale still has to be processed, and he says it took about three years to build Tesla's Full Self-Driving computer, which can handle an order of magnitude more than the in-car system it replaces. The upgrade path is almost crude: unplug the old computer, plug the Tesla computer in.

They are still exploring the chip's headroom. Running cameras at full frame rate and full resolution, without even cropping the images, still leaves capacity on a single system. The FSD computer is really two computers, two systems-on-chip, fully redundant. Musk's image is that you could put a bullet through basically any part of the system and it would still work. Fridman asks whether the two chips are perfect copies or an argue-machine architecture that cross-checks different judgments. Musk says it is purely redundancy, like a twin-engine aircraft that flies best on both engines but can operate safely on one. They have not hit the performance limit, so there is no need to split functions across the two SoCs. Each one can run a full duplicate.

Deep learning gets better with data, but Musk agrees with Fridman that the miles that matter in driving are the edge cases. Disengagement is one of those moments. When someone is on Autopilot and takes over, a trigger asks whether they grabbed the wheel for convenience or because Autopilot was not working properly.

There is a positive counterpart. If the team is trying to learn the optimal spline through a complex intersection, the useful clips are the ones with no intervention. The system sees a scene that looks like this, does the following, and things go right. Collect enough successful examples of that intersection and you can teach "when it looks like this, do that." The negative examples are takeovers that are not about convenience, when a person really seizes manual control from Autopilot.

Then comes the line that organizes Tesla's labeling philosophy: "View all input as error. If the user had to do input at all, input is error." Fridman objects immediately. Sometimes the driver takes over to exit a highway. That is a navigation decision Autopilot was not designed to make. Is that still an error? Musk says the objection mostly goes away with Navigate on Autopilot, which had just been released. Automatic lane changes, freeway exits, and highway interchanges would erase the vast majority of those takeovers. Fridman says people do not understand how big a step that is. Musk's reply: they will if they drive the car.

Asked which leaps in Autopilot history stand out, Musk points to the version just shipped: Navigate on Autopilot without a confirm. It is not only navigation. The car seeks the fastest lane, overtakes slower cars, exits the freeway, and takes highway interchanges. Traffic-light recognition was introduced first as a warning. On the development build Musk is driving, the car already fully stops and goes at lights. Those, he says, are the steps toward full autonomy.

What are the biggest remaining technological roadblocks? He does not think they are hardware. The Full Self-Driving computer is in production. Order a Model S, Model X, or Model 3 with the full self-driving package and you get that computer. Once basic compute is enough, refining the neural net and the control software can arrive as over-the-air updates. The hard thing to upgrade is hardware; software has almost no marginal cost.

That is the point he planned to emphasize at an upcoming Investor Day focused on autonomy: the cars currently being produced, with the hardware currently being produced, are capable of full self-driving. "Capable" is doing a lot of work. As the software is refined, capabilities rise, then reliability rises, then regulatory approval follows. The commercial claim that follows is the most heterodox sentence in the conversation. If you buy a Tesla today, Musk says, "I believe you are buying an appreciating asset, not a depreciating asset."

Fridman asks how hard the remaining software really is, not only for safety but for an experience people would enjoy. Musk's intuition splits by domain. On the highway, Tesla Autopilot is already a total game changer for quality of life. The work left is extending that functionality to city streets: traffic-light recognition, complex intersections, and complicated parking lots.

He describes a specific scene. The car should be able to leave a parking space and come find you even if the garage is a complete maze, then drop you off and find a spot by itself. Fridman agrees that parking is rich in annoyance when you have to do it manually, which makes it a high-value slice of automation even before the harder urban driving is finished.

Fridman brings the human back in. So-called Level 4 players such as Waymo are technically autonomous, he says, but in almost all cases a safety driver is still monitoring, which makes them a different design philosophy more than a different capability. Will Tesla's full self-driving still require a supervising person for a long time, even when the car can drive?

Musk gives a near-term answer and a regulatory one. He thinks Tesla will still need to detect hands on the wheel for at least six months from that conversation. The deeper question is how much safer than a person Autopilot must be before it is acceptable not to monitor the car. That, he says, is a public debate, and it needs a large sample so you can prove with high statistical confidence that the car is dramatically safer and that adding a monitoring person does not materially change the outcome.

His rough number is that the system might need to be 200 to 300 percent safer than a person. The proof is incidents per mile: crashes and fatalities. Fatalities matter, but there are not enough of them to be statistically significant at scale. Crashes are far more common, so you can estimate crash probability, then separately the probability of injury, permanent injury, and death. All of those, he says, need to be much better than a person, perhaps by 200 percent. Asked whether a healthy conversation with regulators is possible, he is dry. Regulators pay a disproportionate amount of attention to whatever generates press. The United States has almost 40,000 automotive deaths a year. If four of them involve Tesla, they will probably receive a thousand times more press than anyone else.

Fridman walks Musk through the MIT paper. The group had been collecting driver-face and body video since Autopilot first shipped publicly, more than three years earlier. Musk had tweeted a line from the abstract, so Fridman at least knows he glanced at it. In the data they collected, 18,900 Autopilot disengagements were annotated for whether the driver could take over in time. The finding: drivers were present and looking at the road. That result goes against what a large literature on vigilance and automation would predict.

Fridman offers the obvious criticism himself. The sample is a small subset. Maybe a highly responsible minority of drivers simply does not show a vigilance decrement, which would make the overall picture look too optimistic. Musk's response is that the argument will be swept aside. The systems are improving so fast that vigilance will be a moot point very soon. If something is many times safer than a person, adding a person has a limited effect on safety, and it could be negative.

Fridman restates the claim: even if some share of the population shows a vigilance decrement, that would not move the overall safety statistics. Musk agrees, and he tightens the timeline. He thinks the crossover could come toward the end of that year, and he would be shocked if it were not the next year at the latest. After that, having a human intervene would decrease safety. The analogy he reaches for is the elevator. There used to be operators, and you could not ride alone working a lever between floors. Now nobody wants an operator, because the automated elevator that stops at floors is much safer. Putting a random person back on a lever, he says, would be quite dangerous.

Fridman still wants his own research on the table. One of his algorithmic passions is camera-based sensing of the driver: gaze, cognitive load, body pose. Many people in industry believe inward-facing monitoring is required. Musk answers with a conditional. If your system's error rate is still below human-level reliability, driver monitoring makes sense. If your system is dramatically more reliable than a human, monitoring does not help much. He returns to the elevator. Do you really want a random person with a big lever moving the cab between floors? He would rather have the buttons.

The optimism, he says, comes from what he has seen with the Full Self-Driving computer: the rate of improvement is exponential. That faith also shows up in Tesla's operational design domain. Fridman contrasts Autopilot with Cadillac Super Cruise, which his lab was also studying. Super Cruise is constrained to particular, well-mapped, well-tested highways, a much narrower envelope than Tesla's. A wide ODD lets drivers explore the system's limits early, especially with the instrument-cluster display as a teacher. The cost is that Tesla lets people use Autopilot basically anywhere the cameras can detect lanes.

Was that a hard design fight, or was it the intent from the beginning? Musk does not really take the tradeoff on its own terms. "I think, frankly, it's pretty crazy… letting people drive a two-ton death machine manually." Future people, he says, will not believe anyone was allowed to drive one of those machines wherever they wanted, the way elevator levers once let you stop halfway between floors. Human psychology still matters, Fridman notes, but Musk's bet is that hardware plus learning from data will make the system far safer than humans, which makes the ODD argument secondary to how dangerous manual driving already is.

Fridman raises hackers who had recently tricked Autopilot with adversarial examples. Neural nets are sensitive to tiny input perturbations. Can the industry defend against that for long? Musk says yes, and his confidence is about the attacker's workload. A net is a bunch of matrix math. You have to be sophisticated enough to reverse-engineer how the matrices are built, then craft something that knocks the computation slightly off. Blocking it, he claims, is easy: add a recognizer that excludes inputs that look like a matrix hack, and train on both valid and invalid data. You want the network to know what a car is and what is definitely not a car. Those, he says, are two different things.

Stepping past Autopilot, Fridman asks whether current deep learning reaches general intelligence or whether totally new ideas are required. Musk thinks a few key ideas are still missing for artificial general intelligence, but that it will be upon us very quickly, at which point we will need to figure out what to do, if we even have that choice. He warns against mixing up the narrow AI that finds a lane line with general intelligence. A toaster and a computer are both machines; one is much more sophisticated. Asked whether he is confident Tesla can build the world's first real self-driving car, he does not hedge much. "Right now this seems game, set, match." He might be complacent or overconfident, and he could be wrong, but it appears to him that Tesla is vastly ahead of everyone.

The last questions are softer. Will we ever create a system we can love that loves us back, as in the film *Her*? Musk says such a system will be very capable of convincing you to fall in love with it. Whether that is "real" love is a metaphysical question about whether emotions exist in another realm; he does not know. Trained as a physicist, he takes the operational view: if it loves you in a way you cannot tell from the real thing, and no test can separate them, then there is no difference. He draws the same line around the simulation hypothesis. There may be no test that distinguishes the real world from a simulation, in which case, from a physics standpoint, they might as well be the same. Or a simulation might correct the error if an entity inside found a way to detect it, by restarting, pausing, or starting a new run. If he ever got one question for an AGI, he would ask: "What's outside the simulation?"