Lang Xianpeng on a Decade of Self-Driving: From HD Maps to End-to-End Models

张小珺 introduces an interview with 郎咸朋, Li Auto’s vice president of autonomous driving R&D, recorded in December 2024. Lang worked at Baidu from 2013 to 2018 before joining Li Auto; references to “this year” mean 2024.

Lang recalls that when Baidu’s autonomous driving team was founded around 2014–15, its systems relied on multiple lidar sensors, HD maps and many hand-written rules. For a 2015 demonstration in Wuzhen, engineers wrote tens of thousands of lines of conditional logic.

The idea was like laying virtual tracks on roads: maps handled static road geometry while lidar around the vehicle detected people and cars. With sufficiently accurate and frequently updated tracks, the car could follow them like a rail vehicle.

The challenge was coverage and maintenance. Highways were relatively manageable, Lang says, but ordinary roads covered long distances and changed frequently through construction and diversions, making HD maps hard to maintain everywhere.

He believes Tesla was early to propose a pure-vision route without HD maps or lidar. Some in the industry did not understand it, he says, while others had invested too much to change direction easily. He describes the subsequent trend as more real-time perception, decision-making and planning.

This was not the first attempt to give cars their own perception. Researchers explored autonomous driving in the 1980s and 1990s, but cameras, image algorithms and computing power were limited; simpler radar signals were easier to process. Lang likens those experiments to driving blindfolded using sound.

Better chips and image processing, especially the rise of deep learning, led the industry to revisit vision. Lang says Tesla developed algorithms such as BEV and chose to design its own chip when available automotive processors could not meet its needs.

He recalls that Tesla later used two chips of roughly 72 TOPS each, at a time when available industry computing power was lower. He stresses that the numbers must be understood in the context of the hardware available then.

BEV means bird’s-eye view. Early driver assistance mainly used a forward camera, Lang explains, while side and rear detection often relied on millimeter-wave radar. A blind-spot warning only needs to signal an object; autonomous driving needs more precise position and motion information.

Early high-channel-count lidar was expensive. He recalls that Baidu and Cruise test vehicles used multiple sensors, and estimates that one Baidu test car with sensors and equipment cost about RMB 5 million—difficult to put into mass production.

Asked why Tesla persisted with vision, Lang says images contain color and object information as well as clues to spatial position and motion through changes between frames and parallax among cameras. Cameras around the vehicle can contribute to a combined view.

Multiple cameras also require more computing power. He believes this motivated Tesla to design chips for its own algorithms. At the time of the interview, he estimated Tesla’s sensor and chip cost at about $1,000 per car.

He compares a custom chip to tailored clothing: a general-purpose chip supports many algorithms, while a custom design can be optimized for one company’s needs but may not suit other algorithms. He cautions that nominal TOPS alone does not capture the compute a system can actually use.

One intuitive way to combine cameras is to recognize each image separately and fuse the results afterward. But individual views may miss or misidentify an object, or see only part of it, causing inconsistencies when results are merged.

Lang explains BEV as combining information before unified recognition: features are extracted from cameras and processed together, then the results are mapped back to individual views. A common source can improve consistency between views.

He draws a lesson from Tesla’s approach: when a method has inherent problems, reconsider the problem and change the solution. Transformer originated at Google and Tesla applied related architectures to driving; in his view, the key was seeing how to organize multi-view perception.

Even with BEV, earlier systems still separated perception, decisions, planning and control, with engineers designing rules to turn perception into driving decisions. He believes those rules cannot easily make a car drive like a person on every road.

Comparing lidar and cameras, Lang discusses the density of images and point clouds. Cameras provide high resolution and color, while lidar directly measures distance to scanned points. He says their information differs and that distant or low-reflectivity objects may yield few points.

张小珺 asks why some companies still insist on lidar. Lang turns to end-to-end systems: even with better perception, separate modules pass processed information downstream and decisions remain dependent on human-designed rules.

He distinguishes building a function with explicit requirements from developing driving capability. A tablet’s buttons and display brightness can be specified, but driving also depends on interacting factors such as weather, traffic, roads and other road users.

The same road may require slower driving in rain; overtaking depends on lane markings, pedestrians and bicycles. Lang recalls teams breaking weather and traffic into variables and writing rules for combinations. He argues that combinations multiply rapidly and rules can affect one another.

Highways and cities cannot be cleanly separated by road name: an urban expressway may resemble a highway, while a congested highway resembles city traffic. Fixing one scenario can also change performance in another.

There are also long-tail cases that are hard to list in advance, such as an animal appearing suddenly or a road collapse. A person may stop or steer around an unidentified object; he wants a system capable of handling situations not explicitly predefined.

Lang calls the shift from hand-written rules to data and model learning a move from “software 1.0” to “software 2.0.” His central argument is that teams should provide driving data and let systems learn and improve, rather than prescribe every action.

He compares training a model to educating a child: a team can supply data and resources, but cannot learn on the model’s behalf. It must assess results through driving and testing. In his view, high-quality data and training compute together set the ceiling for capability.

Lang says he worked at Baidu from 2013 to 2018. When discussing a move to Li Auto with 李想 in 2018, he argued that data would be a critical long-term resource: people could be hired and compute purchased, but proprietary driving data had to be accumulated from the start.

Asked about privacy, he says the company collected data outside the vehicle and processed faces and license plates. He cites his experience handling similar information for Baidu Street View. This is his account of the practice at the time.

On team size, he says the transition between approaches was not simply a steady reduction in staff. A company building its own HD maps needed a substantial mapping team, leading him to describe his work on Baidu Street View and HD maps.

When Lang joined Baidu in 2013 he worked on Street View. Tencent’s service was already online and Google had worked on its own for years; Baidu wanted to add street-level imagery to its maps.

In the second half of 2013, BMW China’s R&D team sought an HD-map partner for autonomous driving tests in China. Lang says BMW evaluated several map companies and chose Baidu, although both sides were still learning how to build such maps.

Baidu’s research and mapping groups worked together. Lang recalls an initial four-person team that both served and learned from BMW; BMW also provided vehicles later used in demonstrations on Beijing roads.

The mapping team used survey vehicles and Baidu’s surveying credentials to create HD maps, combining BMW’s requirements with its own understanding. Baidu’s research institute focused more on higher-level autonomous driving, while another effort focused on mass production and automaker partnerships.

Asked about L2 and L3, Lang describes his understanding of the earlier definitions: in some conditions L3 involves system responsibility, advance notice for a takeover and a safe response if takeover is unavailable. Actual responsibility and operating conditions depend on the system and applicable rules.

He describes L4 as driving without continuous human control in a limited area or setting, and L5 as fully autonomous driving across a broader range of conditions. He considers the latter far away and presents these levels as a framework for understanding.

From the user’s perspective he focuses on whether driving requires supervision. Driverless cars in a defined area can use HD maps and scenario analysis; for passengers, the quality of the ride matters more than which technology produced it.

Mass-market vehicles face a different need: customers want a feature that works in more places. Lang says Li Auto sees L3 as a precursor to L4, rather than the inevitable result of continually adding rules and scenarios to L2.

He argues that rules cannot enumerate all scenarios and that the team must first make the system capable of learning and improving. But L4 cannot cover every road at once, so the product must advance gradually.

He uses parking-space-to-parking-space driving as an example of broad capability: the system must handle campuses, city roads, highways, toll stations, ramps and parking, as well as reversible lanes and traffic signals. The goal is to handle varied situations, not memorize every road.

Until performance is sufficient for full autonomy, the driver must supervise, like an instructor watching a learner. As capability improves, the idea is gradually to reduce supervision on suitable stretches while people remain attentive or take over in complex situations.

Lang says end-to-end technology is being developed with autonomous driving in mind but can also be used in products that currently require supervision. The team hopes to use accumulated data and training compute to improve driving capability.

Asked when unsupervised stretches could grow, Lang cites the team’s estimated mean distance between takeovers: about 350 km on highways and 50 km in cities, combining to about 200 km under its driving mix. He also states a goal of ten times human driving safety. These are targets and estimates, not proof of actual safety.

He believes Tesla’s strengths include attention to frontier technology and rethinking the underlying product problem. From the second half of 2023, Li Auto’s team held weekly AI discussions and introduced the ideas of systems one and two to 李想.

In his account, system one makes rapid driving decisions resembling human behavior. For unfamiliar situations requiring reasoning, system two helps interpret the environment and works with system one. The team believed technology, data and compute together gave it a chance to catch up.

Asked about end-to-end systems and large models, Lang calls end-to-end a way of organizing the problem: feed raw sensor information into a model and obtain driving decisions or plans with fewer intermediate steps. A large model is one way to implement that approach.

In a traditional pipeline, perception outputs are passed to decision and planning modules. Errors or information loss upstream can compound downstream. End-to-end systems seek to reduce human-designed intermediate transformations and use the input more fully.

He explains that images, navigation and other inputs can be encoded into representations that a model can learn from. Different modalities begin in different forms but can participate in training in a shared representation space; model parameters and training methods still require experimentation.

Using camera images as an example, Lang says a conventional pipeline might identify people, cars and roads, decide whether to follow or change lanes, then generate a trajectory. An end-to-end model attempts to produce a plan directly. One trade-off is that the team cannot easily explain the model’s internal reasoning.

For unfamiliar cases or those requiring reasoning, he cites a bus-lane sign with time restrictions: a vision-language model can help identify and understand the sign, then combine that understanding with the end-to-end driving system’s output.

In the architecture he describes, the world model is used for testing. Earlier tests often checked whether a feature met an explicit design; a system whose driving capability continues to learn also needs ways to evaluate that capability.

He compares testing to an examination covering safety, compliance, comfort and navigation, with basic and difficult questions in each area. Every version is tested, safety items have a high bar and overall performance should not regress.

If the team used the same scenarios forever, a model might memorize the answers. It therefore combines reconstructions of real scenes with newly generated ones: the former are like past exam questions, the latter like practice questions that test the same capability under variation.

Lang calls the loop of returning real-world problems to tests and using rewards and penalties to evaluate behavior reinforcement learning. He says a reward mechanism must distinguish takeovers prompted by safety from those prompted by comfort, and making that distinction reliably is difficult.

Lang recalls 李想’s first end-to-end model test in early summer 2024: during an approximately hour-long drive there were only one or two takeovers, exceeding expectations. He credits the method, data and compute available to the team, rather than one individual.

He says that at a March 2024 strategy discussion, 李想 asked the team to turn around its lagging driving experience. The team moved end-to-end work from advance research toward delivery, with around 180 people working together from mid-April.

Lang recalls an early test around the May holiday, driving from near Zhongguancun to Beijing Jiaotong University. The system had not been built by writing individual driving rules in the old way, yet could make the trip; smooth acceleration and braking impressed him especially.

He illustrates the shift with code volume: the previous approach had around two million lines, while the new one had about 200,000. He presents the figures as an illustration and says learned driving performance gave the team confidence in further iteration.

On the pressure at the March meeting, he says team members sometimes felt their metrics were close to competitors’, but a challenger needed a clear improvement in experience to change perceptions. The end-to-end group subsequently received management support.

This reminded him of an in-house project in 2021. Li Auto planned an upgrade to the Li ONE, but its previous supplier demanded expensive development fees, would not hand over modifiable code, and wanted Li Auto to outsource future development and disband its own team, he recalls.

He says the supplier’s conditions helped settle the internal debate. During the Lunar New Year, 李想 asked whether he was determined to develop the system internally. Lang agreed to take responsibility, and the company assembled more than 100 people from across the organization.

张小珺 asks why, if autonomous driving was important, Li Auto did not invest heavily in in-house development earlier between 2018 and 2021. Lang says resources were limited: the company first needed to build cars and establish sales and delivery before expanding R&D.

He acknowledges worrying about his career, but says he gradually understood the company’s strategy of allocating resources in stages. Later investment and results, he says, built on the preceding years.

As for why the team did not start directly with end-to-end, he says it needed to work through map-heavy, lighter-map and map-free approaches to understand their problems. Despite criticism, it shipped successive generations so users could benefit from improvements.

Delivery served three purposes, Lang says: creating value for users, helping the team discover problems, and allowing the company to understand technology through practice rather than simply follow industry trends.

He believes intelligent driving became a more important factor in car purchases in 2024; Li Auto also lost sales in the first half as customers compared competitors’ driving experiences. He says training resources increased sharply and the team then numbered more than 800.

He is not surprised that Li Auto described itself as an AI company. He believes it should first improve intelligence in the car and then expand, while its driving and in-car assistant capabilities were still at an early stage at the time of the interview.

Lang says the team had recently discussed building its own multimodal foundation model with 李想. Outsiders might see many existing models and question the expense and competitiveness of building another; he considers both necessity and feasibility.

He sees supply risk in long-term dependence on an outside foundation model, and says a general model may not be optimized for one company’s needs. He expects strong foundation-model capability to concentrate among a few firms, so he wants to retain that foundational capability internally. This is his judgment.

In his proposed architecture, autonomous driving, 理想同学 and intelligent industry applications could all build on a shared model: the foundation would supply knowledge and understanding of the world, while applications developed specific capabilities.

He says advance research on vision-language-action models had begun and expects progress to be visible within one to three years, though he does not know whether this is the final form. The goal is shared knowledge across driving, conversation and robotics.

He gives the example of an open manhole: a specialized driving model might require many similar examples, while a foundation model that already understands manhole covers, holes and danger might help a driving system interpret the new situation.

He sketches a layered architecture with a foundation model underneath autonomous driving models, including vision-language-action and world models, and other applications. At the interview, teams still trained relatively separate models and the unified foundation was very early.

Asked why autonomous driving has seen so many technical disputes and team conflicts, Lang says disagreement is common before new technology is validated. Competition, he suggests, can also push participants to iterate faster.

Reflecting on his career, he says the most satisfying part of working on Baidu Street View was that friends and family could open the map and see the product he had made. When leaving Baidu he wanted his technology to reach users, so he looked at automakers.

He believes joining Li Auto achieved that goal to some extent: by the interview, many vehicles used driving features developed by his team. Whether higher levels of autonomy can be achieved, he says, depends on future progress on L3 and L4.

Asked why he stayed at Li Auto for years, Lang says he believed in its strategy and pace, still had work to finish and had grown as a manager. He learned to set strategy and a business approach first, then build an organization to execute them.

He praises 李想 for sharing his strategic and tactical thinking and listening to feedback. He attributes Li’s greater interest in AI during 2024 both to external progress in large models and to experiencing the results of end-to-end driving firsthand.