En 30 secondes
Tesla’s 2021 AI Day traced a driving system from eight camera feeds to a shared spatial representation, then through planning and control. Tesla says FSD (Supervised) requires an attentive driver ready to take over. The team explained why temporal memory, automatic labeling and simulation help address occlusion and rare cases. It also presented the D1-based Dojo training design, while saying that large-scale compiler work remained unresolved, and introduced a proposed humanoid robot.
- 1Single-image accuracy does not guarantee accurate 3D geometryKarpathy shows how small depth errors can distort projected road boundaries.
- 2Time and distance memory preserve different informationThe team keeps features for occluded cars and road markings seen before a long wait.
- 3Multiple trips can label more of a road sceneOffline reconstruction aligns camera observations to generate training labels.
- 4Dojo’s training tile combines 25 D1 diesThe team describes integrating power, cooling and chip connections around the tile.
- 5Dojo’s large-scale compiler problem was not yet solvedAn engineer says prototype networks had run but scaling still needed work.
À retenir
- Small depth errors can make otherwise plausible image predictions inaccurate in three dimensions.
- Time- and distance-based memory retain different driving context.
- Offline labeling can combine multiple trips and later frames to reconstruct scenes.
- Dojo’s 2021 design used 25 D1 dies per training tile; scaling software remained work in progress.
- The Tesla Bot shown on stage was a performer, with a prototype planned for later.
Lire le texte remanié, chapitre par chapitre
Before the presentation
The broadcast begins with a lengthy pre-event sequence.
The broadcast plays music and pre-event footage for approximately 47 minutes.
From camera images to a spatial view
Karpathy explains the shift from single-frame predictions to multi-camera, temporal processing.
Elon Musk opens Tesla AI Day 2021 by inviting people working in AI hardware and software to join the company. He says the presentation will show work beyond electric cars.
Vision team lead Andrej Karpathy explains that eight cameras provide images which the system turns into a three-dimensional representation of lanes, curbs, vehicles, signs and their positions.
He compares the task with building an artificial visual system: the team must design how camera information flows through a neural network.
Karpathy traces the earlier system, which processed individual images to support lane keeping and following a vehicle. A shared network extracted features and separate branches handled tasks such as object and traffic-light detection.
He calls this shared architecture a HydraNet. Reusing a common feature extractor reduces repeated work, while individual task branches can be developed and tuned separately.
Single-image predictions were not enough for navigating a parking lot. Tesla initially stitched curb detections together across cameras and time with a hand-tuned tracker.
Karpathy says tuning that tracker was difficult, and predictions that looked accurate in images could project poorly into three dimensions when depth estimates were slightly wrong.
An object can also cross several camera views, leaving no single camera with a complete picture. The team therefore wanted to combine all camera images and predict directly in the shared spatial representation.
That requires both a model able to transform images into a bird’s-eye view and training labels in that output space. Karpathy first explains the model and returns to data later.
A fixed projection is difficult because slopes and occlusion change which image region contains useful evidence. The team uses an attention-based transformer to combine features from the cameras.
Camera mounting varies slightly between vehicles, so the system adjusts images to a common virtual camera using calibration information. Karpathy says this improved consistency.

He shows a comparison of road-edge and lane predictions before and after multi-camera processing, plus examples of vehicles crossing camera boundaries.
The next challenge is time: a car may be temporarily hidden, and a turn marking may have been seen before a long wait at an intersection. A single instant cannot provide all that context.
Tesla’s approach stores features over time and distance traveled. A time-based memory can help track a briefly hidden car; distance-based memory can retain a road marking while the vehicle waits.
Karpathy describes a spatial recurrent network that updates a local representation as the car moves. The model can retain earlier information when a view is blocked and update it when the scene is clear.
He shows examples in which video context helps maintain detections through occlusion and improves estimates of distance and speed compared with single-frame predictions.
The resulting pipeline combines calibrated camera images, shared features, spatial fusion, temporal memory and task-specific outputs. Karpathy notes that the architecture was still evolving in 2021.
Planning paths around other road users
Elluswamy describes search, optimization and a learned planning experiment.
Ashok Elluswamy introduces planning and control: use the perceived scene to choose steering and acceleration while balancing safety, comfort and efficiency.
He says city driving creates many possible paths and interactions. The team first searches among broad maneuvers and then optimizes a smoother path within the selected corridor.

In a lane-change example, the planner compares choices that brake sharply or risk missing a turn. It searches many candidates and selects a path according to safety, comfort and reaching the turn.
Elluswamy says planning must account for other road users too. In a narrow street example the system considers whether an oncoming driver has space and is likely to yield.
After choosing a corridor, continuous optimization adjusts the trajectory using factors such as obstacle clearance, travel time and smooth acceleration.
For less structured scenes, the team explores learning-based planning. Elluswamy uses a parking problem to show how a basic search can waste work when its guidance does not account for obstacles.
He compares the search with a neural-network-guided method in a toy example. The latter uses scene context to guide exploration, reducing the number of explored nodes in that demonstration.
The proposed architecture combines perception, learned and explicit planning, and final steering and acceleration commands. The team then turns to the data needed to train it.
How Tesla labels driving data
The team combines human labeling, offline reconstruction and automatic labels.
Karpathy says a network’s architecture alone is insufficient: its many parameters need large, clean and diverse training datasets.
He says Tesla brought labeling in-house after finding an external process too slow for its needs. At the time, the company had built a labeling organization of more than a thousand people.
Instead of labeling each camera image separately, annotators can label reconstructed three-dimensional space and project the result back into multiple views.
Karpathy says humans are useful for semantic judgments while computers can assist with geometry and tracking. The team combines both for larger datasets.
Elluswamy describes automatic labeling: video and vehicle-sensor clips are processed offline with neural networks and geometry algorithms to produce training labels.
For road surfaces, the system estimates height and semantic features, then checks the reconstruction against observations from multiple cameras and times.

Different trips through the same location can be aligned to label areas no single trip covered. He says the purpose is to label the recorded clips rather than maintain a permanent high-definition map.
Offline processing can use later frames as well as earlier ones. That hindsight helps estimate motion and connect object tracks through occlusion.
The team shows reconstructed road geometry and moving objects, with the goal of producing many labeled clips for multi-camera video models.
For rare poor-visibility examples, Elluswamy says the fleet supplied relevant clips and automatic labeling produced thousands of examples faster than manual labeling alone.
Simulation and deployment
Synthetic scenes fill data gaps while the team manages the compute required for training and car deployment.
He introduces simulation using a rendered driving scene. Synthetic scenes begin with known object positions, so labels such as depth, motion and segmentation can be generated automatically.
Simulation helps with rare events, crowded scenes that are expensive to label, and situations in which other traffic responds to the car’s actions.
The simulator models camera effects such as noise, blur and optical distortion. The team also uses rendering, varied assets and procedurally generated roads to make training scenes diverse.
Elluswamy describes recreating a real failure in simulation and varying it, allowing the team to test changes against related cases rather than only the original clip.
Milan Kovac explains the infrastructure behind labeling and model deployment. He says large offline workloads used thousands of GPUs and many CPU cores, alongside vehicle computers in simulation.
When moving a network into the car, the team must meet latency and frame-rate constraints. Kovac describes a compiler that maps model operations to the dedicated onboard hardware.
He says Tesla runs large numbers of evaluations on software changes and uses tools to compare model outputs across video clips.
Kovac says training demand has grown beyond the existing GPU cluster, setting up the presentation of a custom training computer called Dojo.
The planned Dojo training computer
Venkataramanan describes D1 chips, training tiles and the planned software stack.
Dojo lead Ganesh Venkataramanan says the goal is to train larger models quickly while controlling power and cost. He identifies bandwidth and latency, not only raw compute, as scaling challenges.
He describes arranging compute nodes and memory in a two-dimensional network, with a compiler attempting to keep related work close together to reduce communication.
The team designed a training node around fast local memory, matrix and vector operations, and a programmable processor. The design aims to balance computation with data movement.
Venkataramanan presents the D1 chip, saying Tesla designed it for machine-learning training and high-bandwidth communication between chips.
A training tile combines 25 D1 dies. He explains that power delivery, cooling and high-bandwidth connections were integrated around the tile to support scaling.

He says the team had recently run a neural network on an early functional tile. Larger trays and cabinets were presented as the path to a planned exaflop-scale system.
Dojo’s planned software stack partitions hardware for different jobs and maps models across nodes. Venkataramanan describes compiler, driver, profiling and debugging components.
He gives Tesla’s performance and efficiency targets for Dojo and says the first cabinets were still being assembled.
The proposed Tesla Bot
Musk introduces a humanoid robot concept and distinguishes the performer from a working machine.
A costumed performer appears before Musk introduces the proposed Tesla Bot. Musk explicitly says the stage figure is not a functioning robot and hopes for a prototype in 2022.
He proposes applying cameras, onboard computing, batteries and actuators to a humanoid machine intended for repetitive or dangerous work.
Musk speculates that robots could eventually make physical work optional and says the proposed machine did not yet work at the time of the presentation.
Questions on scaling, safety and simulation
The team answers questions about Dojo, driving systems, manufacturing and robot design.
In questions, Musk says fully open-sourcing the system would raise the question of how to fund development, while suggesting other manufacturers could eventually license it.
Asked about scaling Dojo’s compiler across nodes, an engineer says the problem was not yet fully solved, though the team had run networks on prototype hardware and had performance models.
Musk says the practical test for Dojo is whether the software team would prefer it to the existing GPU cluster, and emphasizes faster training iterations.
Simulation lead Ian Glo asks whether the platform can eventually serve other parts of Tesla. He says more vehicle systems and the proposed robot could be incorporated over time.
Asked about language and vision, Musk says driving depends heavily on visual understanding but audio can matter for emergency vehicles and communication.
On adapting to different countries, the team says it uses video from many places but focused development on US roads to avoid adding multiple sets of road rules at once.
The team discusses geometric reconstruction accuracy and says cameras are the main source, with other sensors sometimes used for benchmarking.
Asked about manufacturing automation, Musk says Tesla uses a mix of automated and manual processes and discusses large vehicle-body castings.
A Dojo engineer describes modular liquid cooling and power delivery. Musk says the team drew on Tesla’s experience in vehicle power electronics and cooling.
Asked about larger networks and inference costs, the team notes that Dojo can train large offline labeling models as well as models intended for the car.
The engineers discuss combining learned perception and planning with explicit physics and optimization, and say an end-to-end approach could avoid decoding some intermediate representations.
Asked about machine learning in factories, Musk says he generally prefers simpler methods where they work, because machine learning can be difficult to deploy.
The simulation team describes combining conventional graphics, ray tracing and experimental neural rendering. It says synthetic data fills gaps alongside real-world footage.
Musk says real video remains the dominant dataset and describes generating variations around unusual events to test and train the system.
Asked about the humanoid robot, he says its initial aim is to handle repetitive, dangerous or boring tasks; other applications remain possibilities.
The team says larger models can improve performance when sufficient data is available, but deploying a model in a car requires managing latency. Larger offline models can instead support labeling.
On robot hands, Musk says the initial design would try five fingers and use tools, while noting that the design would evolve.
A questioner asks how to build public confidence in driving-system safety. The team describes ongoing work to improve perception, while presenting its expectations for future progress.
Asked how the planner handles uncertainty about other drivers, Elluswamy says it considers multiple possible behaviors and can slow or yield when the situation is unclear.
The team says proposed changes can run in shadow mode to compare what a network would have done with a human driver’s actual action.
Asked which component would help most, the team emphasizes completing surround-video processing and recruiting people to work on the many remaining tasks.
On continuous learning, Karpathy says the team generally trains and evaluates a stable model before deploying it, rather than continually changing each car’s model on the road.
Asked about AI alignment, Musk says more capable systems warrant concern and describes Tesla’s immediate aim as useful AI for driving and basic robot tasks.
The last question concerns camera sensors. Musk expects future camera hardware but expresses confidence in the existing system’s potential; the team closes the event.
Sur la même chaîne
24:44EN
59:56EN
23:21EN
1:09:55EN
27:27EN
1:05:20EN