Vision or Lidar, Human or Machine: Lex Fridman's 2019 Map of the Self-Driving Race

Fridman begins with the mission. Autonomy is about access to mobility for people who can't drive because of age or location, about turning dead time in traffic into productive time, and above all about preventing crashes. He cites a counter that someone dies in a car crash every 23 seconds, and says that this is what he thinks about when he works on deep learning papers.

In 2018, he says, Waymo reached 10 million miles driven autonomously in October. On the semi-autonomous side, Tesla Autopilot passed one billion miles. He calls that probably the largest deployment of neural networks making life-critical decisions many times a second, and he stresses that the primary sensor is a camera and that all of those miles are also data.

March 2018 also brought two fatalities. An Uber vehicle in Tempe, Arizona, hit and killed a pedestrian, and a Tesla on Autopilot struck a divider in Mountain View, California, killing its driver, the third Autopilot fatality. Fridman argues that these events get disproportionate attention, so that successful vehicles will have to take risks, and if the public doesn't understand the mission, failed risks could set the field back.

Answering a question, he notes that manually driven cars see roughly one fatality per 80 to 100 million miles. Dividing Tesla's one billion miles by three fatalities looks three times safer, but he says that comparison is apples and oranges, because Autopilot is used mostly on highways, by particular drivers, in a modern car. Three events are also too few to draw statistical conclusions, though that doesn't stop the media from reacting to a single one.

Fridman describes the state of autonomous taxi services as real passengers on public roads with big caveats. Most pilots are small, slow, confined to limited communities, and almost always have a safety driver. He lists Voyage in Florida villages, Optimus Ride in Boston's Union Point, Drive.ai in Texas, May Mobility in Detroit, Waymo One in Phoenix, and Nuro delivering groceries with no occupant. He adds Uber's restarted testing in Pittsburgh, Aptiv's testing in several cities, Aurora and Cruise.

On scale, he says that MIT has one autonomous car that can give rides on certain roads, but one vehicle is not a service. For discussion he sets a threshold of 10,000 vehicles, which would be around a quarter of New York City's 46,000 active Uber drivers turning into passengers. In his words, "Scale is everything here."

Automakers have made dates public. Tesla moved its prediction from 2018 to 2019, Nissan, Honda and Toyota said 2020 under constraints, and Hyundai, Volvo, BMW, Ford (at scale) and Chrysler said 2021, with Daimler saying the early 2020s. Fridman sees these as driven partly by the instinct to look cutting edge, and contrasts them with cautious voices such as Carl Iagnemma, who compares autonomy to an elevator that still has a help button, and Gill Pratt, who says removing the human is far away.

He frames the spectrum as Elon Musk versus Rodney Brooks. Musk expects full autonomy in 2019 and said in 2017 that in about ten years it would be unusual to build cars that aren't fully autonomous. Brooks puts full autonomy beyond 2050, with major cities banning manual driving in some regions in the 2030s and most US cities doing so around 2045. Fridman adds that cars stay on the road for 10 to 20 years, so any shift would be slow.

After a lighthearted video of himself playing guitar in MIT's autonomous car, Fridman gives his main claim for the next 10 to 15 years. Autonomous cars will not be adopted because they are safer, because they won't be safe enough to be the deciding factor. They also won't be faster, since cautious driving is slower than the assertive way people drive today, and there are no clear near-term economics that make them much cheaper.

He compares this to Uber and Lyft, which took over taxi service because of the human experience. Autonomy will spread only if it offers an experience people enjoy, such as natural-language interaction, sensing the driver's attention, and handing control back and forth in a personalized way. He says many people in the field treat that as an afterthought, while he believes you should make the car "fun as hell" and solve autonomy jointly.

On levels, Fridman says there are really only two: human-centered autonomy, where the human is responsible, and full autonomy, where the car is responsible legally, experientially and algorithmically. Full autonomy therefore excludes teleoperation. It also excludes any rule that gives a driver 10 seconds to take over, because the car must be able to find a safe harbor and pull over if the driver doesn't respond.

Fridman lists ways fully autonomous vehicles might be deployed. They include zero-occupancy last-mile delivery, highway trucking with platooning on well-mapped routes, defined urban taxi routes with set pickup and drop-off points, and closed communities such as those Voyage and Optimus Ride serve. A further idea is empty vehicles that drive themselves to the customer, who then takes over.

On the semi-autonomous side, he describes high autonomy on the highway, on-ramp to off-ramp, where the driver remains responsible, and low-autonomy driver assistance such as Super Cruise and Volvo's S90, which handle some of the miles. He then covers out-of-the-box ideas: connected vehicles that remove traffic lights, layers of tunnels under cities where cars act as mini trains, and flying cars, including Uber Elevate.

Fridman ran a public poll with about 3,000 responses on who would be first to deploy 10,000 fully autonomous cars on public roads without a safety driver. Tesla got 57 percent, Waymo 21 percent, someone else 14 percent, and 8 percent said no one in 50 years. He points out that in 1998 the search leaders were Ask Jeeves, Infoseek and Excite before Google disrupted them, so the field is wide open.

Fridman goes back to the DARPA challenges. After nobody finished in 2004 and four cars finished in 2005, many said the problem was solved, and the 2007 Urban Challenge, won by CMU, strengthened that view. He compares this with Rutherford's remark that the rest of science is stamp collecting. The underlying beliefs, he says, are that driving is easy and formalizable, that humans are bad drivers, and that decades of human-factors research show people can't stay vigilant while monitoring a system.

He asks what happens if those beliefs are wrong. Driving involves subtle nonverbal communication between vehicles and pedestrians, decisions in 100 to 200 milliseconds in snow, rain and poor visibility, and game-theoretic assertiveness. A car must sometimes take a risk, expecting a pedestrian to slow down, and algorithms today don't account for how the car's own behavior changes the pedestrian's. He adds that formalizing an objective function can have unintended consequences, as in the CoastRunners boat that earns more reward by hitting the wall.

Humans, he says, are bad at judging what is hard and what is easy. We carry roughly 540 million years of visual-system evolution, and perceiving and understanding a scene in a split second, along with controlling a body, is far beyond current robots. He shows a human soccer player against a robot and DARPA Robotics Challenge humanoids to make the point.

Fridman argues that the claim that humans and automation don't mix isn't obvious. Most of the supporting experiments are in labs, where an undergraduate watching a screen tunes out. When your own life is on the road, the psychology may differ. His group instrumented 22 Teslas and observed drivers over two years.

In that data, drivers used Autopilot for 34 percent of miles, and across about 26,000 transfers of control they were always vigilant. He found no moment where a driver responded too late to a critical situation. He cautions that 22 cars are 0.1 percent or less of the Autopilot fleet, so this is only an inkling, but it suggests a system could handle perhaps 90 percent of the driving and ask for help when needed.

He then contrasts the engineering burdens. Full autonomy has to solve mapping, localization, perception, control, planning, safe harbor, external communication and teleoperation perfectly. Shared autonomy has to solve the human side, sensing what the driver is doing, communicating, and learning the driver over time. Smartphone use keeps rising whether driving is manual or autonomous, so managing driver attention is, in his words, a fundamental machine learning and human-robot interaction problem.

Fridman presents two approaches. The vision approach uses cameras and deep learning on huge data. Cameras give the highest-resolution, texture-rich information, are cheap, and match a world built for human eyes, but without a ton of data they aren't accurate enough for roughly 99.99999 percent reliability. The lidar approach maps particular roads thoroughly and uses the most accurate sensors. It is consistent, reliable and explainable, but it is expensive and is mostly not deep-learning-based, because few cars carry lidar to collect data.

He walks through the sensors. Radar is cheap, works in extreme weather and is vital for obstacle avoidance, but has low resolution, like football's offensive line that does the work and gets no credit. Lidar gives accurate 3D depth and 360-degree coverage but is expensive and struggles with fog and rain. Cameras are rich and cheap but noisy, weak at depth and poor in extreme conditions, and ultrasonics help at close range. Sensor fusion combines them, and he notes that a camera, radar and ultrasonic suite like Tesla's is comparable to lidar on his comparison plot.

The question, Fridman says, is who catches failures. In the camera-based semi-autonomous approach, the human is the fail-safe. In full autonomy, as at Waymo, the fail-safe is lidar and highly accurate maps. He also observes that Tesla's move from Nvidia Drive PX2 to its own chip echoes Google's path from GPUs to TPUs.

While people debate, Fridman notes, data is accumulating. The Ford F-150 is still America's most popular car, manually driven cars still generate data, and every automaker is adding semi-autonomous features. The two paths he sums up are vision versus lidar, L2 versus L4, and semi-autonomous versus fully autonomous, with Tesla at one billion miles and Waymo at 10 million.

Answering a question, he says traffic-light detection is largely solved with cameras. The hard case is when the light is green and a drunk, drugged, drowsy or distracted driver, the "four Ds", approaches a pedestrian trying to cross, and the car has to decide what to do. Different objects look different to different sensors, such as a flying bag, which is why fusion matters.

He closes by saying that ImageNet, COCO and games like Go and chess are toy problems compared with putting algorithms into cars where they can save lives. For him, AI that touches people in a real way is the defining problem of the 21st century, and it brings psychology, philosophy and sociology together with perception and robotics.