Four Joints, Eighty Thousand Pounds: Boris Sofman on Cozmo and Waymo

Asked for his favorite robots in science fiction, Sofman names WALL-E and R2-D2 first: characters that convey intent, emotion, and attachment with no language at all. His second answer is the Terminator series, and not for the killing machines. He likes the mismatch in which Arnold Schwarzenegger's older T-800 is out-specced by the new model and still holds the scene. The two answers sound like taste. They are actually an engineering brief.

Lex points out that a Terminator-like humanoid looks achievable, if ruinously expensive, while Cozmo already does what WALL-E does with almost no visual vocabulary. Sofman says leaving the human form was intentional. Copying a body loads you with joints, actuators, sensors, and encoders, then sets an expectation that strengths will match a person's. AI does not work that way. It is stronger in some places and weaker in others, and a non-human form lets you change the rules.

Fewer degrees of freedom are not a handicap. Stick figures communicate almost arbitrary emotion with two dots and a line, and extra dimensionality becomes a crutch in which teams get lost. Animators Sofman worked with trained on brutal drills, such as a ball that can only jump, because aggressive limits force deeper motion. WALL-E was constrained on purpose even though an animated film can do anything. Cozmo's one-line brief was the same idea in hardware: bring a Pixar character into the real world.

Sofman and his two co-founders started Anki as PhD students at Carnegie Mellon's Robotics Institute, closer to the applied end of the field than to theory for its own sake. Outside research, factories, and the military, consumer robots barely existed; iRobot was the rare exception. They wanted a bottoms-up company that could hit a space ripe for impact and then climb. Entertainment looked empty: movies and video games were rich, physical toys were not, and a chasm sat in between.

The timing was a hardware accident they chose to exploit. Smartphones had crashed the cost of cameras, motors, memory, microphones, and microcontrollers by orders of magnitude, and the 2007 iPhone made it obvious that a phone could be both interface and brain. Complexity could move into software while the physical parts stayed cheap. Drive, Anki's first product, was a racing video game made physical: a virtual state matched the track, and cars could have characters, weapons, and specials. What surprised them was amplification. The same gag that feels small on a screen hits much harder when it happens on a table.

That physical multiplier is why they accelerated toward Cozmo, a character with enough awareness of the room to feel alive. Sofman describes Drive as a video-game engine for the real world and Cozmo as the same engine with an animation and interaction layer on top. Toys also forgive failure. A car can leave the track; Cozmo can flub a bit. If the thing you are optimizing is fun, you have room to explore, and room to explore is what most robotics products never get.

On Drive, Sofman built planners that made optimal passing moves and learned quickly that optimal is dull. Players want chaos, clumsiness, and the feeling of getting better, so the team started adding noise so the AI would play like a human who is still learning. "Perfection is actually not fun," he says. The same instinct moved to Cozmo and went further. The team had been terrified of robot errors. They found the opposite: a character that notices a mistake and reacts well earns empathy; a bare machine that errs is just stupid.

Consumer price made the mechanical budget non-negotiable. You cannot ship a thousand-dollar toy unless you deliver a thousand-dollar experience, so cost of goods had to sit well under $100. Cozmo landed under $50 end-to-end, packaged and delivered, at under $200 retail. The robot itself has four degrees of freedom: two treads, a head that nods, a lift that rises, plus sound and a low-resolution screen. Testers and customers said the same sentence anyway: they had to let him win, because they did not want him upset.

The team composition was the other constraint. An early Logitech engineering lead who had shipped on the order of a billion consumer units ran mechanical; software was told to eat cheap noisy motors and sensors. A game-design faction wanted Cozmo to be a pile of mechanics. Sofman and a co-founder bet on character, arguing you will never beat virtual games on game-ness, while a physical character is an unfair advantage. They hired Carlos Baena, a nine-year Pixar veteran who had worked on WALL-E, and built an emotion library at the extremes of happy, sad, angry, surprised, confused, and scared, then parameterized it with randomness so combinations would not repeat.

A behavior engine read real-world context and sampled responses. First sighting of the day was a spike, like a toddler at the door, not a two-hour high. Left alone, Cozmo got bored and nudged things off tables. Doubling how often he glanced over for approval raised playtime about 40 percent. Eye contact, almost invisible as a film device, turned out to be universal from children through adults. Dogs were a better reference than movie blocking. The games, Sofman insists, were there to create situations. Without a win, a loss, or a surprise, emotion looks random; with context, it looks smart.

Lex asks the *Her* question: how smart does a future Cozmo need to be before it can be a friend, and how many years away is that. Sofman splits the problem. Pet-like companionship arrives first, because aiming at a human and missing drops you into the uncanny valley. A non-human character has more room to land a good experience. That is also why Cozmo had no voice, and why *The Sims* skipped speech: a voice narrows age, gender, and taste, while silence lets an eight-year-old and a forty-year-old hear different things.

The same character layer changed what people would tolerate. Most users will not accept Alexa or a Google speaker starting a chat unprompted; it feels like being listened to. A cat bumping your leg, or Cozmo roaming over, does not. Europe was supposed to be the privacy nightmare: cameras, microphones, children, Germany and France. Sofman says they prepared for a beating and got almost nothing. Journalists treated vision and hearing as obvious. Of course he has to see and hear, they said, or he would not be alive. Charisma, in other words, communicated intent, and intent disarmed the fear.

Speech output, he argues, is still a bug for a product. State of the art is not good enough, so talking shatters the fiction the same way a near-human face does. Speech *understanding* is useful; talking back is a much higher bar. You can cheat by leaning into limits, a toddler learning English, a character whose flat affect is the joke, but it is a dangerous extra degree of freedom. Friendship without general intelligence is possible if you set a domain and meet it. His worry in 2021 was simpler: he did not see anyone picking up Cozmo's three-legged stool of hardware, AI, and character with the same aggression on the character leg.

Anki shut down in 2019 after a brutal year of near-death financing. Commercially, Sofman can point to a lot that worked: millions of units, annual revenue approaching $100 million, number-one rankings in categories, strong reviews. The business underneath was a gift product. About 85 percent of volume landed in the fourth quarter. The company paid people, factories, and ads all year, sold in November and December, and got paid by retailers in December and January. Success made the swings worse. Tens of millions went out for inventory, marketing, and payroll, then everyone waited.

Hardware had also fallen out of fashion. Sofman cites Fitbit and GoPro sliding from roughly $10 billion toward $1 billion in revenue, with hardware firms often valued at about one times sales. Anki was stuck in the middle: trying to climb out of entertainment, unable to drop the business it was valued and financed on, and without a clean path to profitability at those price points. A financing round collapsed at the last second. There was no way through to the next holiday. They sold some assets and wound down.

He had been transparent with more than 200 people through days-of-cash moments, credit-line chicken, and bridge-loan terms. The announcement was not a shock. It was still the moment he broke down on an all-hands and someone else had to take the room. Afterward the office stayed until two or three in the morning, drinking and telling stories. Many called it the best job of their lives. He still has not opened a thick stack of letters from kids and parents collected at the end; maybe a five-year reunion, he says. A majority of the AI and robotics group later landed at Waymo, which wanted exactly that background.

Sofman joined Waymo in 2019, about two and a half years before this conversation. The company, spun out of Google's self-driving project after the DARPA challenges, builds what it calls the Waymo Driver: sensors, compute, software, and infrastructure that are not tied to one chassis, the way a human license is not tied to one make. Waymo One moves people and already ran a fully driverless ride-hail service in the Phoenix area. Waymo Via moves goods. Underneath, he says, the overlap is larger than outsiders expect: see the world, decide, and prove a safety level that supports autonomy at scale.

Via's near-term product is Level 4 on Class 8 trucks with 53-foot trailers, starting with long-haul freeway stretches and transfer hubs at the edge of metro areas so the last few miles can stay manual. Dallas is the first ground-up driverless truck hub, with routes toward Houston, Phoenix, and Austin, and a longer ambition of stitching toward the Port of Los Angeles. L4, as he uses it, means nobody behind the wheel inside a defined domain, not every road and weather. L5 is not worth planning for. The sequence of expanding domains is the actual strategy, because getting to market teaches you things that R&D cannot.

The economic case sits on a driver shortage already over 80,000 and headed toward hundreds of thousands, rising e-commerce demand, fatigue, and hours-of-service rules that dictate warehouse geography. U.S. trucking is about a $900 billion industry, highly fragmented, with the average operation often a one-to-five-truck family shop and some fleets seeing attrition around 95 percent. Autonomy at first plugs into that mess. Over years it can change where goods sit, how far a single "driver" can go, and how ports, warehouses, and hubs handshake. An 80,000-pound truck also has a different body: longer sensing horizon, trailer-cabin coupling, jackknife and rollover limits that shift with load. Waymo built a test track with giant training wheels on the trailer and pushed past the limit until it actually rolled, then wrote that high-dimensional boundary into software.

A long-haul Waymo truck carries more sensors than the cars, loosely on the order of twice as many: dozens of cameras, radar, and multiple lidars. Cars keep a main pod on the roof. Trucks move two main pods out to where the driver mirrors sit. A center mount would look into a trailer taller than the cabin and leave a wedge of occlusion that software would have to invent around. Side pods see flush with the trailer, kill most of the blind spot except directly aft, and let the car perception architecture transfer with retraining rather than a new invention.

Redundancy is not a slogan. At scale every sensor fails, and failure is not only silence: fouled data, a cut wire, a dead slice of compute. Overlapping lidar, cameras, and radar make inconsistency detectable. The hardware generation Sofman can name in public is the fifth, shared with the Jaguar vehicles after the Chrysler Pacifica era. He notes that swapping sensors is expensive because models have to be retrained, so they try not to churn the stack for sport.

Lidar is Waymo's in-house advantage and the primary near-range sensor: dense points, precise location, and consistency. Consistency means a person, a car, or an animal still looks like a 3D bounce whether it is noon or night, glare or not. Range is where lidar drops first. Trucks therefore go camera-and-radar primary at long distance. Cameras still return signal past a kilometer, but noise rises, classification gets harder, and a two-meter localization error at that range is the difference between a shoulder and your lane. Radar's gift is direct speed, plus better penetration in weather. Sofman no longer wants a lidar problem, a camera problem, or a radar problem. He wants a fusion problem.

Late fusion runs separate camera and lidar pipelines and merges at semantics, classification, and tracking. Sofman says the evidence is tilting toward early fusion, because late merge cannot exploit complementary weaknesses. Rain is the clean example: lidar reflections, camera noise, no single modality is happy, but the union can filter toward a real signal. As the stack becomes large-scale and data-driven, you want the best extractable cue from each modality *before* you make a constraining decision that is hard to unwind downstream.

Machine learning, he says, is massive and still growing. Early AV work after the Grand Challenge used structured models, heuristics, and search, and those methods can look deceptively strong until the long tail. Perception was revolutionized first. Behavior is following, not always as one end-to-end net, but as more modules that improve when you pour in data. Once you have driven tens of millions of miles and must expand domains without regressing old ones, a human-tuned robotics stack stops scaling.

Trucks reuse the car labeling and ML plumbing, and almost every truck model starts from the latest car model as a pretrained prior. The transfer goes both ways. On a behavior-prediction model for other agents, 85 percent of the training data still came from cars, much of it from surface streets. Freeways create the opposite problem: interesting events and even human crashes are sparse, so you cannot drive your way through every permutation. Structured testing, augmentation, and simulation have to sample an infinite space on purpose.

Waymo simulates about a thousand miles for every mile it drives. Twenty million on-road miles imply on the order of twenty billion in simulation. The uses stack: regression shields in development; replay of a bad scene with a proposed fix; large historical batches that say whether a metric moved, because physical driving is the least efficient test; and validation, taking disengagements, rolling them forward, and asking whether anything would have happened. Usually the answer is no. You can also inject virtual agents, jitter positions, and stress the scene.

"The hardest problem in autonomous driving is actually the evaluation problem," Sofman says, "not the actual autonomy problem." A great demo can look driverless and still be years from driverless, because the blocking events live at once per 100,000 or 500,000 miles. You can patch the thing you saw. You cannot easily know you did not create five things you have not seen. Those were painful lessons in Waymo's history, and they are why evaluation now sits at the center of how the company develops.

A human truck driver sees a serious freeway event about every 1.3 million miles and a really serious injury about every 28 million. A Turing test fails early: thirty minutes and fifteen miles will not sample that tail. Some rules you can prove structurally, such as never exceeding the speed limit, or you can inject every hardware fault instead of waiting. Behavior is infinite. The portfolio looks like aerospace: structured argument, statistical sampling, correlated proxy metrics, because you do not get a million launches. Harder than a shuttle, though, because you cannot freeze the software. You have to keep changing it after you are already driverless, in an environment you do not control, and still know you did not get worse.

Lex asks Sofman to argue both sides of Waymo versus Tesla, the two companies building robots at this scale. For Waymo, the claim is focus: sensors, compute, infrastructure, stack, and safety framework aimed at Level 4 from the beginning. That sounds bureaucratic until you notice the fifth-generation hardware and a full swap of hardware and software between the Phoenix driverless system and the current one. What first crossed a human-relative safety bar was not what you would use to scale exponentially. Those lessons, he says, cannot be shortcut. Waymo is also the only company in the conversation that had already shipped a fully driverless public service.

An L2 truck that helps a human could ship with far fewer sensors and still add value quickly. Jumping from that machine to L4 would then be a hardware and compute bet, not just a software one. Tesla's counter, which Sofman is willing to steelman, is data. A huge real-world fleet, with paying customers acting as safety drivers, is a superpower if you can filter to what matters. He will not argue that cameras cannot eventually solve driving; humans do. He will argue timeline and risk. Perception's long tail is already a research frontier. Putting that on top of behavior, decision-making, and the safety case makes a problem that is "almost insurmountably hard" harder still.

Waymo itself uses more human-driven data than outsiders assume, because those traces are some of the best training signal for autonomy. Lex's closing frame is that nobody has answered how hard driving really is, including the silent pedestrian glance that says go. Sofman agrees the two companies are running different experiments on the same unsolved question. A polished trial loop is as misleading in perception as it is in a ride-along. Full driverless, not excellent assistance, is a different bar.

Once Phoenix went driverless, Sofman says, iteration slowed. Fear of regression makes every release an expensive validation. Healthcare is the only other industry he analogizes, and even surgery has a bounded, known risk. Driving already kills people and is not banned; the AV bar is still "beyond reproach," not merely better than a human, because one wrong crash can set the field back a decade. Lex brings Cozmo back: a robot that communicates intent might be allowed to take risks people understand, the way they understand a yellow light. Sofman draws a line. This is not a private choice. Other people share the road. The ethical and regulatory bar is higher.

He still thinks the long view looks like the internet or the mobile phone: odd, in retrospect, that none of this was there. Cities could spread as commuting pain changes; parking, which he cites as something like 30 percent of rush-hour urban traffic in one statistic, could shrink. He claims on the order of 95 percent of accidents are attention and other failures automation is strong at. Logistics becomes less grindy. He is an optimist about net creation and wants local pain taken seriously anyway.

On jobs, trucking is a shortage industry, the average driver age drifting toward 50, and the worst miles are the 300-day long hauls away from family, with fatigue and health costs stacked on. Autonomy should eat those stretches first and leave short, urban, sleep-at-home work for humans longest. Drivers paid by the mile feel unpaid waiting as a wound; better logistics would raise pay and cut empty trucks. Zooming out, he does not hand-wave meaning. Remote assistance is a new job. At an Amazon warehouse using Kiva robots, he was told throughput jumped by more than 2x while headcount on the floor roughly held, because people moved to final pick. Growth is not automatically kind. Bridges, retraining, and pride in a craft are part of the work, not a later subsidy.

Amazon's Astro, a $1,500-class home robot with a screen, looks to Sofman like a search for a killer app. Anki was cautious about jumping to fancier hardware that would raise the value bar above what today's cost structure can hit. Mobility is an unused advantage only if you use it for something neighbors would actually buy. He had "signs" Amazon was too interested in Cozmo years earlier. Character, he repeats, is the leg technology companies underbuild. iRobot survived because a vacuum is a known annoyance and a known category; he cites on the order of 15 percent of that market. Cozmo had to convince a child and a parent at once, and around 2013 Anki had to stop talking about how advanced the tech was and start talking about why a family of four would want it. Nest could say "smarter thermostat." Cozmo had no such anchor.

Asked why robotics companies die, he does not reach for a tragic-genius story. Either the technology is hunting for an application, or price and function are mismatched. The field combines more disciplines than most companies, so it is expensive, and consumer scale means millions of units, not ten thousand enthusiasts. Tesla Bot, in his view, is research. Few problems require a humanoid; wheels beat legs; extra joints add cost. Building a humanoid is often the pursuit of a humanoid, not of a job. A factory should ask whether it is optimizing strength, mobility, or versatility before it copies a body. Boston Dynamics is "incredible technology" still on a long hunt for an application. The more interesting line, which he shares with Waymo's goods problem, is that as perception, control, and planning deepen, more of the world starts looking like a robotics problem, including a truck redesigned for autonomy instead of a Daimler Freightliner with a kit on it.

His CMU years ran from undergrad through a PhD on UPI, a DARPA-funded off-road vehicle with shoulder-high wheels that had to hit a waypoint ten kilometers through forest, ditches, and rock. Robotics, unlike a narrow CS niche, forces breadth and collaboration; it is a tool across industries, not an industry. Startup advice is blunter. Co-founders matter on the darkest days. School taught him that effort maps to output; a startup taught him that fifty investors can still not see it, a manufacturing defect can end you, and the market often wins over team and hustle. Waymo can spend at a pre-revenue scale almost nothing else can, because even a discounted multi-trillion-dollar freight-and-ride market pays for the bet. Anki could not. For students: overlap passion with a growing need, collect a versatile tool such as machine learning, join a growing surface area, and notice when work has quietly become a life you cannot unwind. He has been on the obsessive end of that spectrum and is trying to climb back toward family and old friendships.