OpenAI’s $5 Million Bet: A 60-Person Team Wants GPT-4V to Give Cars Common Sense

Two days after OpenAI’s first developer day, the company was not done making news. Its startup fund invested $5 million in Ghost Autonomy, a six-year-old self-driving startup that has already raised more than $200 million, to bring large multimodal models into cars. Nathan Labenz, hosting Cognitive Revolution, frames the deal as a bet on a software-primary system built on commodity hardware, aimed at long-term partnerships with carmakers rather than a robotaxi fleet of Ghost’s own.

Self-driving cars are already on the road, Labenz notes, but they are not ready for prime time at scale for a mix of technical and social reasons. The technical bottleneck that Ghost wants to attack is the long tail of unusual scenes that human drivers handle with common sense. On top of the low-level, high-frequency components the company has already built, the plan is to let a large multimodal model—complete with world knowledge—make the high-level decisions and tell those specialist systems what to do.

John Hayes, Ghost’s founder and CEO, says the partnership is the culmination of a chase that started in March, when OpenAI announced GPT-4 with multimodal capability that was not quite ready to use. What he saw was a reset in how vehicle autonomy, and most of robotics, would be built. AI, in his view, is “destroying whole areas of computer science.” Tesla’s recent moves acknowledge that reality, he says, but Ghost is the first to put a multimodal LLM directly on the driving path—and he thinks everyone will be doing it soon.

Hayes is a serial founder, not a career roboticist. His first California job was at a virtual-world startup. He spent a couple of years at Yahoo on talent search, then cofounded Pure Storage and stayed for a decade. The thread that connects storage to self-driving is a formula, not a domain. Around 2009, flash was already taking over consumer machines: the new MacBook Air, $100 laptops, and every phone. Hayes’s team bet that flash would take over the enterprise too. They moved consumer hardware into the data center, spent heavily on software to make it reliable, and sold a box that cost about the same as the incumbent storage system while running ten times faster, with better reliability and far lower energy use.

In 2017 he tried to apply the same formula to autonomy. Living in Mountain View, he had been surrounded by test cars for nearly a decade and noticed two things. The first was utopian talk that people would simply stop owning cars. The data, he says, do not support that: taxis are about 1% of trips, and public transit adds another 1%. Families with young children keep car seats and everyday clutter in the vehicle. A car is an extension of the home, and that habit is not disappearing. The second was cost. Lidar, custom computers, and survey-grade maps produced a product so expensive that the companies chasing robotaxis were, in his view, missing 98% of the market with no easy way to scale down.

Ghost therefore bet on two trends he still calls almost undeniable. Mobile cameras and mobile chips would keep getting better, because the only thing a phone launch still sells is a better picture. And AI would clobber the specialized algorithms that autonomy had spent years hand-building. The founders of Waymo, Cruise, Nuro, and several companies that no longer exist all came out of the DARPA Urban Challenge of 2004 to 2008. That contest assumed a map of the city, lidar to find car-sized moving objects, and cameras mainly to read traffic lights, because image recognition in 2008 was not good enough to do more. Ten years later, Hayes says, it was clearly not that simple—and the industry was not going to finish the job without a technology reset.

The classic autonomy stack is an elaborated version of that DARPA design. You start with a map that encodes the rules of the road and the behavior you expect from other people. Serious teams use lidar to localize against a 3D scan of every building and tree, then run perception to place other actors on the same map. Prediction, often at about 10 hertz, tries to guess where those actors will go over the next second, because a vehicle has momentum and cannot reverse a decision instantly. Planning then searches for a path. The failure mode is famous: if you contemplate every sudden lane change a nearby car could make, the only safe answer is to sit still. Prediction and planning feed each other, and the search can conclude that motion is never justified.

Waymo, originally Google’s self-driving project, is the granddaddy of this approach. Hayes recalls a start around 2011 and a scale-up around 2013. Cruise solidified itself as number two from 2015 with essentially the same stack. Trucking companies such as Aurora and Kodiak keep the layered design but stretch perception and prediction hundreds of meters ahead, because a loaded truck is a high-momentum machine. A more radical camp, including the UK startup Wayve, throws the stages out and trains one end-to-end model from sensors to a trajectory. Tesla, Hayes says, has walked through several of these ideas in series: an early FSD Beta that skipped prediction, a later classic solver with more models in the loop, and then another reboot toward end-to-end.

Labenz sketches a 2-by-2: expensive specialized sensors versus phone cameras on one axis, modular engineering versus end-to-end learning on the other. Hayes simplifies it into two philosophies of knowing that a system works. One is inference: if every part is reliable, the system is reliable, which is how automotive safety cases are written. The other is measurement: stop rationalizing the parts and watch what happens on the road. Aurora cannot drive a million trucks, so it has to reason from components. Tesla has on the order of a million cars and thousands—maybe low tens of thousands—of trusted drivers, so it can ship, measure, and find the real problems statistically. How much data you hold decides which philosophy you can afford.

The SAE ladder has five levels; in practice Hayes sees about two and a half that matter. Consumer cars live in Level 2: adaptive cruise, automatic emergency braking, and auto-steer, with a human who must keep watching. Level 3 is the first time the car can say it has the situation and the driver may read a book until a chime. Mercedes brought the first Level 3 product to market as traffic-jam assist: under 40 miles per hour, surrounding lanes filled, mapped roads, compatible weather. BMW and Acura in Japan have announced similar products. Level 4 adds a minimal risk condition, or MRC: when the system cannot handle a scene, it falls back to a basic safe action, usually stopping in place in a city. Level 5, a car that handles any scenario, “may never exist.” Robotaxis are Level 4 as a business. Riders Labenz has asked describe them as better than Uber; you watch for a minute and then relax. Waymo launched in Arizona because the weather is dry, the heat keeps pedestrians off the street, and early Cruise runs were at night. San Francisco now has only Waymo, which still avoids the steepest hills and the most crowded hours. The industry’s soft spot is what Hayes calls an uncanny valley: a system that works 99% of the time trains people to stop watching, and the remaining 1% arrives with no one ready. That dynamic, he suggests, may have played a part in Cruise’s trouble.

Ghost’s hardware list is short on purpose: consumer cameras, consumer chips, and a radar. Radar is already on about 30% of new cars, fully solid-state, and uncontroversial after decades in production. Lidar’s prized trick is a direct range measurement. Ghost replaces it with stereo vision—two cameras on one computer, with AI producing real distances. Traditional stereo either needs custom chips or dies on calibration: pixels are a micron wide, and a micron of distortion breaks the match. Ghost’s stereo runs at 30 frames a second against lidar’s typical 10 hertz, and a 12-megapixel camera is a thousand vertical scan lines against 64 on an expensive lidar. Commodity sensors, stacked this way, out-resolve the specialty box.

The software bet is speed. Most stacks decide on a cycle of about 100 milliseconds. Ghost pushed that to roughly 30 milliseconds, about ten times faster, which is a meter of travel on a freeway. If the call is wrong, the cost is a meter, and another decision arrives immediately. Over a few hundred milliseconds a vehicle with momentum cannot change acceleration much, so the tree of futures stays small and the heavy prediction-and-planning machinery can be thrown out. That, Hayes says, is the question he has been chasing since founding the company: how do you design an architecture that does not require Waymo’s thousands of engineers when you have about 60?

Discipline around that choice is strict. The system is fully symmetric: the same software stack records data and runs inference, on the same hardware, so there are no special survey vehicles. The loop from finding a problem on a drive to shipping a fix is kept short; the company does multiple software releases a day. Ghost also refuses to ask the driver for settings. Other assist systems want a cruise speed and a following distance. Ghost reads the traffic with stereo and radar and chooses both, sliding on its own between stop-and-go, thick flow, and thin flow. The car is supposed to feel as if it is floating in the current rather than slavishly executing a number you typed. Ghost does not build cars. The business is to be the supplier that gives GM, Ford, and their peers an answer to Tesla’s FSD.

Edge cases still do not go away. A lane model that looked solid degraded wherever there were trees; the culprits were shadows, time of day, and glare, because the model had assumed a kind of illumination that objects off the road can disturb. They have also found roads with literal spills of white paint. Each surprise means more data, more training, more planning logic. Urban driving is the wall every team has been hitting for a decade, and Hayes’s diagnosis is blunt: the long tail is common sense, and common sense cannot be enumerated as rules. That is why the company adopted a barbell: small, fast models in the car whose training sets one person can still intuit, and, on the other end, models that have already absorbed nearly all human text and images so nobody has to guess what is salient in a scene.

The break came with GPT-4V. Ghost first reached for it as a labeler, a way to ask whether a scene had trees or a bridge without training a special recognizer. Then the team noticed that the model already had driving common sense. A deceptively simple question became the test: am I on this freeway right now? GPS is unreliable around on-ramps, so the visual field is the better source of truth. An open-source multimodal model, LLaVA, got it right about 60% of the time out of the box. Fine-tuning on a few hundred street scenes pushed that to about 98%. Construction zones with a flagman produced sensible answers too. The path was no longer a pile of purpose-built detectors. You could show the model the images, annotate what sensors could measure, and ask what the car should do.

The new architecture splits driving in two. A high-speed system in the car runs at 30 hertz and owns safety: collision avoidance, lane following, anything that lives in milliseconds. Above it sits a much slower model trained on a wide slice of human knowledge, looking at the same sensors and issuing instructions at the maneuver level. “The car becomes an API,” Hayes says, instructed by an LLM. Adding a new signal used to mean a dataset, a recognizer, and a planning-code edit, forever. The inversion is to open the scene completely, ask the model, and have it explain the answer in terms of visual elements. Labenz tried a version of this himself in Salem, Massachusetts: a temporary October parking sign taped to a lamppost, stripes of tape across it. ChatGPT read the sign, then inferred witch-trial tourism and seasonal traffic. That leap is what people call hallucination. Hayes calls it the best part of the models, because “the hallucination is where all the common sense applies.”

Maps do not disappear; they change jobs. Licensed map facts go into the prompt as free context. The map is also a memory key: images from prior drives, and the inferences drawn from them, are stored against the road so GPT-4V is not called—at 10 to 30 seconds a request—for facts that have not changed. If a wiper is sitting in the current frame, yesterday’s picture is the better input. Long-term memory is the map. Short-term memory is signs and other observations kept as text, expired as the car moves, and fed back into the prompt, which makes the whole thing look like an agent with two timescales of memory. Latency does not worry Hayes much. GPT-3.5 got ten times faster in a year; over a five-year automaker planning cycle he expects models to get 100 to 1,000 times faster and cheaper. The design default is to call the largest model. Only the simplest questions, such as whether an empty country road has anything around the car, go to a cheap specialist.

The product he wants is a car that explains itself in plain language. Today’s rentals beep and flash a symbol you have never seen. Screens deleted buttons and replaced them with deeper menus, so discoverability got worse. Ghost’s current behavior is staged: while you are driving, it is a 100% conventional car and does not fight you; if you let go, the default is to keep following the road; you can say “take the next left,” or hand it a destination, and cabin cameras help guess whether you intend to take over. Next is conversation. The car already knows you are at a gas station, a charger, or on an unpaved road, so buttons and menus can go away. Why OpenAI? Because an advanced multimodal API you can call today is, Hayes says, the only game in town. Google’s Gemini has been promised and delayed; Anthropic’s multimodal plans are unclear. Enterprises will keep eating documents. Consumers, limited by a phone keyboard, will eat cameras, video, and context. Driving is a subset of that same shift.

Labenz calls himself an accelerationist on self-driving. Driving is already dangerous, the safety numbers from Cruise and Tesla look at least as good as humans in the domains where they operate, and society still meets the technology with a shrug or a shutdown. Most consumers, Hayes replies, have never experienced any form of autonomy, so their views are formed in a vacuum of media stories. People who have ridden in the systems tend to turn bullish. Labenz borrowed a neighbor’s Tesla and drove his grandmother on an eight-hour round trip. A Midwest summer thunderstorm dumped on them and the car handed control back; a few other moments were odd. He never felt unsafe, and he would rather have it driving.

Tesla owners, Hayes says, have already split. One group learns the system’s tells—this turn, this weather, this bridge—and unconsciously engages and disengages, keeping the feature on more than 90% of the time. The other group gets scared once and never touches it again. The industry has not figured out how to onboard people so they learn the limits instead of bouncing off them. Regulators are not magicians either. Automotive rules are usually performance standards written years, even a decade, after a product ships, because nobody has an existence proof when it is new. Cars sold today with auto-steer have no dedicated governing regulation, and Hayes thinks that is correct: there is still no formula that says one approach is clearly better.

The conversation turns to the Cruise incident from about a month earlier. A human-driven car hit a Cruise robotaxi in San Francisco; a woman pedestrian was thrown in front of the Cruise vehicle; after braking, the car executed a minimal-risk pull-over and dragged her about 20 feet. California pulled the operating permit, and Cruise then shut the rest of its national fleet on its own. Labenz wanted someone to stand up and say the activity is already safer, the crash is terrible, they will fix it, and they will not close the doors. Hayes is colder: Cruise lost on government relations and media at every level, in an incident that should have been explainable in public as rare. The engineering failure underneath is that the pull-over was a programmed end state. Cruise had already taken heat for stopping in place and blocking emergency vehicles, so the MRC logic tried to get out of traffic—and chose the wrong safe action. Deciding when to freeze, when to pull over, and when to wait is a common-sense problem that conventional programming may not be able to finish. Cruise, he says, faces a binary: fight to win cities back, or shut down. Both are hard.

Labenz tries to light a US-China fuse: trim the trees, paint the signs, do not insist on ten-times-safer-than-human before anyone may deploy. Hayes answers with unromantic numbers. China adds 20 million first-generation drivers to its roads every year, a flood of effective teenagers that makes a spike in road deaths a political risk. Only two companies produce electric cars by the million, Tesla and BYD, and BYD is Chinese. American automakers are consumed by that EV fight—minerals, factories, retraining, how to get batteries into best-selling trucks. At industry conferences the split is “1% AV and 99% EV.” US regulators follow consumers. The technology race, Hayes argues, does not start until people want the product, and that demand has to be built with rides people can actually take. He had been hoping Waymo would expand city to city with free first trips. Robotaxi companies are still widening their domains even without a Travis Kalanick figure, and he wants that groundswell more than a slogan about beating China.

Zoomed out past cars, Hayes sees AI retooling computer science itself. The bottleneck to putting it everywhere is not chips so much as people who actually understand it. Generative models changed the audience: someone who cannot write Python can now touch a system and get a response. The creativity that unlocks, he says, has no known ceiling. Good results still take a person willing to write many prompts and think hard, but as models improve that skill matters less. Gemini will ship, new versions of Claude will arrive, and the models will converge in function the way CPUs did. Once that happens, generative AI mediates APIs and people alike, and “the totality of human knowledge” becomes a commodity with fierce competition. The last decade’s “no-code” pitch is finally real: say what you want in ordinary language and have a computer do something that used to be inaccessible.

The rest of the conversation leaves a practical reading list. Judge an AI system by where its errors come from, not only by the error rate; try GPT-4V on a messy road sign or a construction zone and watch common sense fill gaps a recognizer never would. When you look at a hardware startup, look at the supply chain it is riding—Ghost’s bet is the phone industry’s yearly camera and chip gains, while lidar companies have to carry their own cost curves. If you build software, notice that Gmail settings, Salesforce configs, and car menus are the same bloated surface: the replacement is a system that already has context, so you do not have to hunt a menu. And do not outsource your view of self-driving to a news cycle. If your city has Waymo, take a ride. If a Tesla FSD demo is on offer, take it. Hayes’s claim is consistent from the robotaxi discussion through the end: people who have experienced the systems become more optimistic, and that experience is the ground on which regulation and demand actually move.