From Udacity's Self-Driving Car Course to a Startup for Retirement Communities: Oliver Cameron on Building Voyage

Cameron says his talk is about how to start a self-driving car startup, and that Voyage's path was an "insane zigzag." He describes himself as someone who learns by building rather than in the classroom, drawn to software because you can make something from nothing and have millions of people use it. After building a lot as a teenager he started a company and went through Y Combinator.

The pivotal moment came when he heard about Sebastian Thrun putting a Stanford AI class online, which was controversial around 2011. He took Thrun's artificial intelligence for robotics class in 2013 and says his head exploded; he got addicted to the MOOC format and joined Udacity, where he spent four years on machine learning, robotics, and eventually self-driving car curricula.

At Udacity he learned from two company builders: Vishal Makhijani, whom he calls an operator extraordinary, and Thrun, who founded the Google self-driving project. Udacity's focus, as he describes it, is on the idea that talent is everywhere and that his job is to find ridiculously talented people and put them to work on the hardest problems.

Around 2016, three years in, Udacity wanted to teach things other places weren't, at the right time before an industry takes off. Google was then the main effort in self-driving, and Cameron's team believed the field needed more talent to move faster. They built a 12-month curriculum with industry partners, because the knowledge of building these cars was trapped in industry rather than academia.

The 12-month course took intermediate software engineers into perception, prediction, planning, localization, and controls, because Cameron saw Udacity students as best suited to being generalists who contribute across the stack. Projects included a kidnapped-vehicle localization exercise and a model predictive controller. Thrun, he says, likened working with him to getting basketball lessons from Michael Jordan.

Internally there was skepticism, since Udacity judged content by the number of open jobs and in 2016 self-driving jobs were essentially at Google. Yet the reaction was instant, and today over 14,000 students have succeeded. Some are building a self-driving truck in India or a perception engine in South Korea, and others have gone to companies such as Cruise, Zoox, Waymo, and Argo.

To make the curriculum special, the team also bought a Lincoln MKZ, a drive-by-wire vehicle, and fitted cameras and lidars on a mount made from Home Depot parts. The milestone was to drive Mountain View to San Francisco, 32 miles along El Camino Real with about 130 traffic lights, with zero disengagements, within six months. Testing began in a parking lot at Shoreline.

They hit the 32 miles in the middle of the night, which did not count, and then achieved it in daytime after about four months. For Cameron it brought home that the industry was ready, in the way someone in a bedroom can build and launch software almost overnight.

Cameron says the team ran open-source challenges because the best approaches would come from collaborating with students around the world, as even Udacity staff were not leading experts. The one he focuses on asked teams to predict steering angle from a single camera frame, in a period when end-to-end deep learning papers were all the rage.

More than a hundred teams self-organized and submitted very different approaches over about four months. The winning model's author later went on to lead the self-driving team at Yandex. On a winding road toward Half Moon Bay its predictions matched the actual steering closely.

On the car, however, the camera-only approach had its faults. A car steered by such a model performs differently in the real world than it does on a desk with pre-recorded frames, and students later added corrections for this.

After the curriculum, the car, and the challenges, Cameron felt he had to build something himself and gathered the same team to start Voyage, pitching it to Khosla Ventures as a new kind of taxi service. They did not want to follow the usual formula, because Google has an enormous engineering pipeline, billions in the bank, and brand recognition that is hard to match.

His motivation is that transportation is broken: over a million people die on the roads each year, vehicles that seat eight usually carry one person, and he cites a statistic that only about 7% of a typical vehicle's energy goes to moving what is actually in the car. It is also expensive, which makes it inaccessible.

Why now? He says sensors have reached the resolution, range, and reliability needed, unlike the single-channel lasers and spinning Velodyne HDL-64 of the 2007 DARPA era. Compute has caught up with GPUs, and talent is spread around the world. Computer vision he rates as yellow, steadily moving toward green.

Cameron believes the path is ride-sharing without a human driver, since a network with human drivers is limited by driver miles and would never replace car ownership or fix fatalities. He quotes his lead investor Vinod Khosla, who says a market entry strategy is often different from the market disruption strategy.

Voyage chose retirement communities for four reasons, Cameron says. They are slower, with speed limits and traffic that he likens to "listening to a podcast at 0.75x." Residents have real transportation challenges: many shouldn't drive but have no alternative, some put off surgeries, and others fear losing mobility to vision problems.

The third is a clear path to customers, since ride-sharing in public cities is a brutal race to the bottom. The fourth is a central authority, since these places are typically run by private companies, which means faster deployment. They also tend to be in places with good weather for self-driving cars, such as Arizona and Florida.

Voyage's first community is The Villages, with over 125,000 residents and over 750 miles of road. It gives Voyage an exclusive license to operate an autonomous vehicle service, and in exchange Voyage gives the community equity. Seniors are often affluent, and Cameron says there are 2,500-plus such communities, with the senior market growing to over a hundred million by 2060.

He describes the landscape as a map in which large players focus on big cities, using their ability to deploy thousands of cars and take long timelines. Voyage, which "fights for survival every single day," instead focuses on self-contained communities.

Voyage deliberately optimizes its sensors for performance rather than cost. At the top of the car is a 128-channel lidar that sees 300 meters in 360 degrees, with additional lidars for blind spots, giving about 12.6 million points per second. Cameron says a demo that focuses only on perception, prediction, planning, and controls falls apart on edge cases, so the team works on the whole stack aiming at a truly driverless vehicle.

The company also partners for simulation, mapping, and operations rather than building teams for each. Cameron's example of an unsolved perception problem is foliage: reports say Cruise cars slow for bushes and that a Voyage car was troubled by a two-foot bush protruding into the lane. Voyage started working on this in early 2018.

Map-based detection subtracts what is in the map, so grown foliage becomes a phantom object. Neural networks trained on 3D scans, such as PIXOR, VoxelNet, and PointPillars, don't use the map as a prior, and they also separate two people standing next to each other, which a traditional clustering algorithm merges into one. Better perception makes prediction, tracking, and navigation downstream safer.

He also follows reinforcement learning and imitation learning, including Wayve's work toward a car powered by reinforcement learning, and the ecosystem of tools and 3D radar vendors that keeps pitching Voyage.

After two years on Voyage and four before at Udacity, Cameron says his lessons are not technical, and came from painful moments. First, don't be intimidated: self-driving began in academia, so people assumed a PhD was required, but driverless cars need people from all backgrounds and industries.

Second, understand your limitations. As the company grew from one person to 44, he says you cannot do everything, so you should build a team who can do what you did ten times better, and he admits he didn't do it early enough. Third, be proactive rather than reactive, and step aside when experts arrive.

Fourth, hire the best, meaning those who match the company as well as being the best in their field, without sacrificing culture under pressure. Finally, always be curious: knowledge should be shared across the company, because someone with an unusual background might do something transformative with it.

Asked how Voyage picked retirement communities, Cameron says the founders initially planned to replicate El Camino everywhere, but Thrun cautioned against it. In 2009 Thrun had told Google leadership including Larry Page that retirement communities might be the best way to deploy, but got pushback because Google was aiming at Level 5. He told the founders to visit such communities, and they did.

On accessibility, Voyage currently focuses on active adult communities, whose residents can get in and out of a car on their own but may avoid driving at night because of vision. He believes the senior market is large enough to own, with tailored products later, possibly human help for the first and last mile or robots that help people into cars.

On sensors, he says the suite is never final and is tied to generations of vehicles. The first generation was a Ford Fusion with a single HDL-64 lidar, cameras, and radar. In the second, at the speeds in their communities, radar wasn't particularly useful, so it was dropped, and a third generation may focus on cost.

On architecture, he recommends ROS, whose plug-and-play nodes let a steering-angle network replace a rules-based planner and perception engine during experiments, with ROS 2 aimed at production. On insurance, Voyage works with Intact Insurance and sends data from its cars to show its detection and planning are accurate. He expects future rates to vary with environment complexity.

Cameron expected seniors to be slow adopters, and his own grandmother hates self-driving cars. But seniors lag on things like Facebook because those replace well-defined habits, such as calling someone or mailing a photo. A self-driving car looks and feels like the car they are used to, apart from a button to press to go.

They also don't care much that it is autonomous. His oldest passenger, aged 93, remembers moving by horse and cart, so after seeing flight, cars, and subways, a self-driving car is just something that moves them.

Asked what is missing for Level 4, Cameron says perception is the holy grail: with perfect perception, self-driving is solved. False positives are acceptable today but false negatives must go to zero, since a missed object makes everything downstream chaotic. Voyage bands multiple networks together, since VoxelNet is good with pedestrians while PIXOR, from a bird's-eye view, is not, and adds traditional computer vision for some regions.

On hurricanes and weather, he doesn't yet have an answer to mass evacuation, but notes that each vehicle has a cellular link to a nearby remote operator who monitors operation and can bring it to a safe stop in sudden rain. Some companies are pitching microclimate forecasting with webhooks to warn of incoming weather.