From Intelligent Copilots to Aurora: Sterling Anderson on Shared Control and Building a Self-Driving Company

Anderson, who headed the Tesla Autopilot team before co-founding Aurora, starts with his PhD work at MIT in 2007, when the DARPA Urban Challenges were underway. His group wanted to address safety problems in human driving earlier than full self-driving would arrive, and the result was a system they called the intelligent copilot.

The key departure from traditional planning and control was to design in terms of constraints rather than paths. The team used the topology of the environment, computing sets of homotopies, meaning continuously deformable routes through obstacles, to give the human a set of intuitive options. The vehicle only had to stay inside a bounded tube in state space, defined by spatial limits like road edges and dynamic limits like tire friction and side-slip angle.

The system simulated the vehicle's trajectory forward using model predictive control and used a threat metric, such as wheel side slip, to decide how much authority to take. If the best achievable trajectory came close to the car's limits, control would shift to the automation and then hand back. In a simulation with a deliberately poorly tuned driver model, the car without the copilot left the road, while the copilot-assisted car stayed on it and tracked the driver's intended path better than the copilot alone understood it, never using more than half of the available control authority.

A major worry for the team was that changing how the car responds to steering input would break the driver's mental model. They tested coupling human and computer control through a planetary gear so the driver feels a torque opposite to the car's motion. Anderson says the answer depended on experience: seasoned drivers expect the wheel to pull left as the car turns right, while less experienced drivers found it confusing.

The group then tried the system on unmanned ground vehicles under a DARPA program led by Gill Pratt. About 20 test subjects drove a course through a remote console with no line of sight, competing for prizes with penalties for hitting or brushing barrels. The interface was degraded with dropouts and delays, and the copilot was switched on or off without the drivers knowing which.

With the copilot engaged, collisions fell by about 72% and speed rose by roughly 20 to 30 percent, though Anderson says he was blanking on the exact figure. Drivers also reported a statistically significant gain of about 12% in how much control they felt. The copilot was in fact taking 43% of the control on average, which he reads as drivers feeling the car did what they wanted rather than exactly what they commanded.

After the research, Anderson and a collaborator tried to commercialize shared control through early companies, but the industry was in a different place. Few automakers thought self-driving would affect their business, and even ride-sharing was still viewed as unproven. He then led the Model X program at Tesla and took over Autopilot before co-founding Aurora in December 2016.

His co-founders are Chris Urmson, who formerly led Google's self-driving car group, and a Carnegie Mellon professor with deep applied machine learning expertise who helped found Uber's self-driving team. Anderson says the timing was right because automakers now accept that self-driving, ride-sharing and electrification are three vectors that will change the industry. Deep learning had also advanced, low-power GPUs and TPUs had arrived, and sensors such as high-resolution radar and lidar had improved.

Aurora has offices in Palo Alto and Pittsburgh and operates vehicle fleets in both places. A few weeks before the talk it announced partnerships with Volkswagen Group and Hyundai Motor Company. Anderson says the business model was deliberately non-threatening to others: Aurora's strength is self-driving systems, not building vehicles, so it plays its position and works with those who can play theirs, which he thinks is the fastest, broadest and safest route to market.

He closes the prepared remarks by pointing to the stakes rather than reciting statistics. Vehicles sit unused most of the time, and he cites an estimate that between 6 and 15 million people in the United States lack the transportation they need because they are elderly, disabled or otherwise limited. He calls it both a tremendous technical challenge and an opportunity for engineers and non-engineers alike.

Asked about lidar, Anderson declines to discuss Tesla beyond public information, but says Aurora believes the fastest and safest route to market uses multiple sensing modalities, including lidar. On economics, he argues the business case for fleets closes even with today's sensor costs. His advice is to start at the top of the market with the most capable system possible, then bring costs down over time, noting that lidar contains no "unobtainium" and that automotive suppliers are exceptional at driving costs down.

Responding to a question about car enthusiasts, he says he is one himself and hates sitting in traffic in Boston or San Francisco. He expects personal driving to become more of a sport or leisure activity, comparing it with riding a horse: if you want to do it, go to a racetrack or a mountain road. He sees room for both.

On commercialization, he stresses that Aurora is partnering and developing vehicles rather than broadly commercializing yet, and that this is done in close partnership with automakers who know their cars, customers and distribution networks. He is not convinced that an open source model would reach the market fastest. He expects a handful of successful self-driving stacks, far fewer than the number of companies today.

Asked for the main technological bottleneck, he names forecasting the intent and future behavior of other actors, both in response to one another and to the self-driving car's own decisions. It spans perception, prediction and more. Aurora interleaves modern machine learning techniques, for example projecting the behavior learned for its own vehicle onto others, assuming nominal behavior while guarding against off-nominal behavior. He stresses that this is not a solved problem and is the sort of work that dominates the long tail of development.

On the risk of malicious actors, Anderson calls security a very real, constant game of cat and mouse. He says he would not pretend to have a plan that solves it once and for all. Aurora tries to use best practices in the fundamental architecture so that key parts of the system are less exposed, then keeps working at it.

Asked what self-driving opens up, he says automakers can differentiate the user experience and offer new services such as augmented reality or location services. The interior can change too. If self-driving systems are validated to reduce collisions, much of the extra mass added for passive safety, such as roll cages, crumple zones and airbags, could eventually become less necessary.

On how to decide a feature is safe to launch, he says it is a statistical evaluation in every case. The set of scenarios is unbounded, so you can never characterize everything. What you can do is establish with reasonable confidence that you perform better than the average human driver, and at that point the opportunity to launch is real. He also says he worries about job displacement, which will happen as with every such technology, and that the transition for people in affected transportation work should be discussed now.

On partnering with both Volkswagen Group and Hyundai Motor Company, he says Aurora's mission is to enable as much of the industry as possible, and both shared its vision and a wish to make a difference at scale. On vehicle-to-vehicle and vehicle-to-infrastructure communication, he says it can only help but has historically scaled poorly and arrived slowly, so Aurora builds without assuming it. He adds that coordinating fleets, where electrification, ride-sharing and autonomy meet, is where much of the value will come from.