Lane Centering Is the Only Self-Driving Feature That Already Helps

Lex Fridman opens with the old question of whether we live in a simulation. George Hotz says yes — and that the claim may never be falsifiable. If the designers wrote a formal proof that no information can leak in or out, and if the hardware stays in spec, a program inside would have no more evidence than a process inside a well-designed virtual machine. He spent time away from comma learning Coq, a dependently typed language for mathematical proofs, where code that type-checks is correct by construction. That, he says, is what a closed simulation could look like.

Fridman pushes the hacker's intuition: any complicated system has ways in and out. Hotz does not dig in. A simulation does not have to be complete if it has a scheduled end date, and entropy looks like one. The universe computer may even be simple, just very large — Conway's Game of Life scaled up. Asked what he would change if the simulation were hackable, he shrugs off the practicalities. The talk he once gave about "escaping" was a way to replace a failing space-race story with a postmodern one: think upward, because the old modernist narrative of rockets and Mars no longer holds the culture.

The world first learned his name in August 2007, when he unlocked the first iPhone. He started from electronics, not software: open the phone, pull an address line high. The C he shipped with that hack was, in his words, atrocious — a broken state machine written by someone who did not yet know how to program. The lesson that stuck from years of finding holes is sharper than any exploit. "You learn to not take things for what people say they are, but you look at things for what they actually are."

The summer after the iPhone unlock, Hotz interned at Google, building Street View hardware and a software library to talk to it. The code was terrible, and for the first time people he respected told him not to write that way. Feedback was not enough. He got good, he says, because he wanted to emulate and visualize ARM binaries so he could single-step the iPhone bootloader instead of staring at a debugger and guessing.

He built that tool once and it was terrible, a second time and it was terrible, a third time at Facebook when it was merely okay. The fourth version, written as a Google intern again in 2014, was the first he would call usable. It is a timeless debugger: rewind is as cheap as stepping forward, every memory access is logged, and clicking a variable shows every time it changed — Git for the running machine. It is the wrong instrument for Chrome, a 200-megabyte binary that already crawls on desktops. It is the right one for Capture the Flag challenges, for ROMs, and for small pieces of code.

The same rewrite ethic shows up later in production. Off-the-shelf distributed filesystems struck him as ten-thousand-line configuration nightmares, so at comma he wrote one in about 200 lines of Go, with nginx volume servers and a small master. If he is proud of any code per line, it is that filesystem — because it was the fourth version, and he had thrown the first three away. He does not believe in a product that works on the first try. He believes in the number of times you were willing to start over.

Fridman watches Hotz's Twitch streams the way some people watch a high-APM game: windows flying, vim on screen, typing so fast it is energizing to leave on in the background. Hotz admits the streams are exaggerated. His style has a real split. Give him an undocumented API and he will throw things at it until it works, very fast. Ask him to rotate a vector 90 degrees and flip it over the XY plane, and he will spam-program for two hours, because that is a sheet-of-paper problem. On camera he is twice as frantic as he is when nobody is watching.

OpenPilot still has large Python components, but he now understands why Python fails at scale: there is no type checking, and the compiler can tell you almost nothing. Misspell a variable and a linter might catch it; a type error will not. He loves dependently typed languages and hates that they still ship with 1980s syntax. Go, which he once dismissed as Python that takes twice as long, is what he reached for when he wanted something strongly typed with a sane ecosystem.

JavaScript's ecosystem he finds "unbelievably confusing": npm bumps a package from 0.2 to 0.5, Babel breaks, and you are compiling ES6 into something that still does not quite run. Why, he asks, does JavaScript need to be compiled at all? He answers his own question the way he answers most language fights. JavaScript lives in the browser, it is fast, and the browser may become the main place compute happens — so it will survive like a cockroach. comma's log analysis tools are JavaScript anyway, because the web is still the best UI toolkit by far.

Hotz got into vehicle automation the way he gets into most things: a friend said the coolest applied AI problem was self-driving cars, and did he want to meet Elon Musk. Tesla was still on Mobileye. Musk wanted a vision replacement. The proposed contract was blunt: match Mobileye-level performance and collect $12 million, minus $1 million for every month of delay. Hotz did the arithmetic out loud. Even at ten months he would still take home $2 million. Maybe five. Maybe he would finish nothing and work a year for free. He did not think he was solving Level 5. He thought he was cloning lane keeping.

In the meeting he suggested putting a GPU behind each camera to keep compute local. He left, waited ten minutes, and realized it was an incredibly stupid idea — send every camera to one big GPU. "Usually in life I get away with saying stupid things," he says; people often do not even notice, and he can steer the conversation back. Musk called it out on the spot. The contract still fell through for other reasons, and the second meeting he was much more prepared. A friend had shown him Mobileye's output: two lane lines and the pose of a lead car. Gather a dataset, train a net, weeks of work. He had something of similar quality in about three months, judged by borrowing a friend's Tesla and driving his system next to Autopilot 1.

Musk later said that building Autopilot was more complicated than a single George Hotz, and that it would take two years and a company of ten people. Hotz's update is the most credible thing he says all hour. Four years later he has a company of twelve, and he thinks they still have another two years to go. He will bet against Tesla having an autonomous robotaxi fleet by the end of that year. Whenever Tesla does get Level 5, he expects comma to be one to two years behind — not a date, just a lag.

Hotz thinks Tesla's Navigate on Autopilot is terrible, and he would not ship comma's internal demos of the same idea even as open source. Consumer Reports, he says, already described the problem: when it changes lanes, it does it worse than a human. You should not ship that. What you should ship is the thing Autopilot and OpenPilot already do better than a person. Turn it on for an hour of highway and it will never touch a lane line. A human will probably clip the line twice. Fridman asks whether that number is data or feel. Hotz admits it is a personal estimate, then notes he could answer it from comma's logs. Either way, every clip is a little spike of stress, and lane centering removes it.

That, he insists, is the biggest value-add in the industry, and it is also all there is. He drove a rental car to San Diego when the company moved and missed the system so much that driving felt exhausting. Waymo, in this accounting, does not yet add value to anyone's life. It is a more expensive, slower Uber, waiting on a cliff that may or may not arrive. Really good lane keeping, plus the more forgiving longitudinal control of adaptive cruise, is already a reason to buy a car. Chasing Level 4 as the headline is, in his view, the wrong chase.

The hardware under OpenPilot is deliberately unromantic: a phone in a plastic case, sold without software, running on a Snapdragon 820 with a Sony IMX298 looking forward and the selfie camera watching the driver. A small device called panda talks USB to the phone and sits on three CAN buses, proxying cars that already have some form of lane-keeping assist. comma supports on the order of 45 models, mostly Hondas and Toyotas, plus GM and Subaru. OpenPilot is approaching 0.6; 1.0, he says, will not add features. It will make lane keeping really, really good, and it will not be called a consumer product until driver monitoring cannot be cheated.

Fridman puts the human-factors objection on the table: the more you automate, the more vigilance drops, and people text, get complacent, or fall asleep. Hotz's answer is product, not philosophy. comma will not call the software 1.0 until driver monitoring is at 100 percent and you cannot cheat it. The current system has obvious cheats. They first tracked where the head was looking, because eyes are hard at phone-camera resolution; the mere presence of the monitor also reminds you to pay attention. He has just hired someone full-time for the job. He wants phone-in-frame detection and a sleeping driver caught. The current camera does not see enough of the body. The next one is a fisheye.

Takeover has to be cheap. OpenPilot disengages on gas or brake, not on steering, so you can snatch the car back and re-engage without a ritual. Autopilot's cancel, he says, is a small disaster: nobody knows where the cancel button is, so people hit the brake, even when what they wanted was the throttle. Using the wheel to cancel is awkward too, because Tesla uses the same torque sensor both to prove your hands are there and to disengage. Super Cruise, once engaged on the highway, he calls phenomenal — comma bought a Cadillac just so the office could feel the target — but disengaging it drops you into ordinary adaptive cruise that is still accelerating, which feels wrong. Sun glare can also blind GM's camera.

Musk's claim that driver monitoring is unnecessary is the one Hotz will not defend. Musk, he says, sometimes says things that are completely true and sometimes says things that are stupid; this one is stupid. When a system only messes up once every thousand miles, attention will decay, and you absolutely need monitoring. He drove to the studio sleep-deprived in a car with no automation and was falling asleep himself. comma renamed the driver-monitoring packet to driver state, sitting beside car state, so the person is a first-class signal rather than an afterthought.

OpenPilot's remaining sore spot is stopped cars at speed — the radar-versus-vision problem behind some Autopilot crashes, because a parked vehicle and a signpost look alike in radar. Hotz does not want to paper over it with a map. comma's localization stack can recover to about ten centimeters over a minute, enough to ground-truth where the car actually went, but it is all local. Their simulator is not Unity. It loads a real drive and asks what the system would have done on historical data. They use it only for testing. The moment they train on it, he says, it stops being a test.

He splits simulators into two kinds. A fully fake world does not solve computer vision; it replaces it with computer graphics. You can make the road look perfect and you still cannot make other cars behave like people. "The definition of driving is what humans do when they drive." Waymo-style driving, in his jab, is not driving. Reinforcement learning on hand-coded pedestrian feature vectors does not impress him either. You already assumed a state you invented.

Fridman, trained as a roboticist, wants the textbook split: perception emits a state vector, planning searches over it, and Waymo's simulator can replay perception to test the planner. Hotz's reply is the hardest line in the interview. There is no human-understandable feature vector between perception and planning. Localization and planning do have one — position, orientation, derivatives. Perception does not. A bush by the roadside tells a human there might be a car behind it. Drivable area plus a list of objects does not contain that. Add occluded regions, and he will invent ten more fields you forgot. "If your perception system output can be written in a spec document, it is incomplete."

What comma actually emits is a 1,024-dimensional vector. Nobody knows what the dimensions mean, because they were learned on real data, the way the middle of a face autoencoder is not a spec you could have written. He wants the AlphaGo of driving — end to end — and he does not think Level 4 is a real category. Hand-coded Go engines were human-inferior until a relatively small learned system beat everybody. Driving is harder than Go; the open question is how much harder. The consolation is that lane keeping, adaptive cruise, and even treating red lights like cars can deliver value the whole way there.

Navigate on Autopilot, Hotz says, does the same lane change every time, and it feels mechanical. Humans sometimes go slow and sometimes go fast. Tesla's version is a hand-coded policy written by an engineer in a room who probably went out and tested it a few times on Highway 280 — maybe two engineers, he jokes. comma had the same temptation. The thing they think they have solved is lateral. Longitudinal is still harder.

The method is almost embarrassingly small. The model has an input bit for whether it is currently changing lanes. During end-to-end training that bit is set from automatic ground truth: label the moment the car crosses the line, about 95 percent accuracy, which is plenty. At test time, if you want a lane change, you flip the bit to one. The policy that comes out inherits the messy human variation in the data instead of a single magic-number trajectory.

Some humans make ugly lane changes. His working theory is that all good drivers are good in the same way and all bad drivers are bad in different ways, and he says they have seen data that backs it. Good behavior collapses into a few clusters; bad behavior is scattered noise. The net learns the majority cluster, and which of the four or ten good clusters you pick can be context from the scene rather than a driver identity. If the hope holds, you do not need a human to label who is skilled. Clustering does the data cleaning. The only advice he would still give Tesla is to put driver monitoring in the cars. He thinks Andrej Karpathy already knows to turn the hand-coded lane-change policy into a learning problem.

Fridman suggests a near-term business: sell the data, or stand up a consortium so automakers pay in and share it. Hotz likes the idea in theory. The tools are open source, the pipeline is already there, and he would be happy to take a three-line email that names a monthly price and reasonable privacy terms — he floats something like $100,000 a month off the top of his head. What he will not do is sit in a room with a business-development person who does not know most of the words he is using. Mobileye around 2015, he notes, had among the lowest R&D spend of any chip company and a building full of BD, because carmakers are impossible. That kind of meeting does not help him win self-driving cars.

The long-term plan he has had since about 2017 is to become a car-insurance company. If OpenPilot makes driving twice as safe, comma will have the best dataset of who is actually safe. Insure those people, refuse the rest, and the market bifurcates: Geico can have the leftover risk at crazy-high premiums. That, he says, is how you turn comma into a ten-billion-dollar company if you win. Asked for a billion dollars tomorrow, he would not know how to scale. He would want to make the system better. Burn is about $200,000 a month against about $100,000 of revenue from selling hardware. They need roughly 4x revenue, and they have not tried very hard yet.

Lidar he files with Musk's true claims, not the dumb ones. It is a crutch, and not even a good one, used more for centimeter localization than for perception. Without it, a perfect localizer is hard; with camera fusion he thinks you can get reliability on the order of 100,000 miles and a fallback so you are never way off without knowing it. Waymo, he grants, is the furthest along — a three-to-five-year lead if that stack works, maybe three. The problem is the bill, and the deeper problem is economics. "Self-driving cars have no network effect." Uber has to attract drivers and riders at once; switching costs protect the first mover even against a rival that takes 5 percent instead of 10. A robotaxi fleet is capital plus riders. Blanket a city the way scooter companies did, and ten checkbooks race to the bottom. GM paying $1 billion for Cruise was, in this telling, an insurance policy against Waymo: Google gets a monopoly for at most three years.

Hotz cuts driving into three layers, and he is honest that he has not solved the third. The static problem assumes you are the only car on the road. Maps and localization can finish it. That is why farm autonomy already works: you statically schedule tractors the way you schedule processes, and they never meet. Maps help only here. Static is easy for one route, one weather, one state of lane markings, and a perfect localizer. It is not easy for the whole world, which is why a Tesla drifting out of lane is still failing the first problem. Lidar makes a centimeter localizer straightforward; vision does not.

The dynamic problem is everything that cannot live in the map. A car stopped at a red light may or may not be there on the next pass, and it may move. You have to detect it in real time and predict what it will do, which means models of other people's behavior. Hard, but tractable. comma's current lateral approach, he says, will scale all the way across static and dynamic.

The third layer is counterfactual: your action changes theirs. You merge and the other car yields; you do not, and it goes. Other agents are humans, so the only method he has is reinforcement learning in the real world once the fleet is large. He would keep the exploration temperature at zero on the deployed policy — always take the max, never probe. OpenPilot already makes tens of thousands of mistakes a week across roughly 700 daily and 1,000 weekly active users, with zero consequence, because a human catches them. That disengagement is a free RL label. Driver monitoring is how you keep the labeler awake.

Hotz is exhausted by safety engineers. Prioritizing safety over innovation past a point, he argues, kills safety in the long run. comma's actual model is short and public in the OpenPilot safety docs. First, the driver must be paying attention at all times — he still calls the software alpha until they can enforce that. Second, the driver must be able to take control easily at any moment. Third, the car will never act on a timescale faster than about one second, enforced with torque, braking, and acceleration limits. All of that code is open source, and he says it is now MISRA-compliant. He came to respect the automotive coding standard because it is written by computer scientists who talk about whether a condition is decidable.

Security, for him, is a special case of safety. Safety is caution tape around a hole so nobody wanders in; security is a ten-foot fence with barbed wire so nobody can. An unreliable system is therefore not secure. The car should always do something safe with local sensors, and those sensors should be hard-wired. Someone could still hack the CAN bus and turn the wheel — they could do that before comma existed. V2V he calls a terrible idea because it needs both parties to get the message. Cellular data in the car is useful for routing around jams, which is experience, not a safety fallback. If the safety story requires a constant link to the vehicle, he thinks it does not work. Remote operators do not get a vote.

The trolley problem gets a joke with a point. There is an alert in the code, "ethical dilemma detected." It has never fired, because they do not know how to detect one. They are a Level 2 system, so they disengage and leave the decision to the human. People get earnest about trolleys, he says, for the same reason a whole office will argue about paint color and go quiet on SQL versus NoSQL: it is a question where everyone is entitled to an opinion. The engineering constraints do not move. He is proud to be Level 2, and he wants that said without magic-upgrade talk. Level 4 on current hardware is not a software flash.

He also admits, more often than most people in a two-hour interview, that he might be wrong. He has been dead-set on approaches that failed. He respects Chris Urmson and the Waymo-class engineers as engineers, Linus Torvalds as infrastructure, and skill wherever he finds it. Attention from the masses he does not want; attention from people with real skills he does. Burn and hype he treats as a trap. He hates trading iPhone-hacker social capital for fundraise money, and he says he regrets doing it once.

The line that "life is work" was, he says, half trolling after a meal. Work gives his life meaning; it does not follow that every minute should be work. People putting in twelve-hour days should be working smarter, not harder. On the girlfriend remark from an earlier interview, he is serious at two depths. Shallow: VR brothels are coming, "it's not cheating if it's a robot," and a novelty stall of dolls plus headsets would already be a trip. Deep: "I would really like to fall in love with the machine."

He does not believe in downloading Wikipedia into the brain. Reading speed is limited by an inner loop, not by the eyes, and bootstrapping that sounds both unclear and horrifying. What he wants is a relationship: something that builds a super-sophisticated model of him, runs simulations on that model, and chooses what to say in order to guide him along a path. He does not want to know whether those simulations are conscious. Merging with an AI, in this picture, looks like a long couple who have become almost one person — not a broadband cable into cortex.

The singularity he dates as a crossover, not a vibe. You can already buy roughly a human's worth of compute for about a million dollars. He pencils a person at around 20 petaflops and a TPU at a generous 100, so a million dollars is on the order of five humans. Human FLOPs grow with population, slowly; machine FLOPs grow exponentially. The crossing is when most of the world's FLOPs are silicon rather than biological. He likes 2038, because that is when the 32-bit Unix timestamp overflows — half a joke, and he has done the math. Asked what winning looks like in five years, he reaches for Jürgen Schmidhuber: an agent thrown into the world should build a maximally compressive model of it, and explore so as to maximize the derivative of that compression. He still does not know what the game is. Once he does, he will know how to win.