Why Tesla AI Day Was the Most Impressive Real-World AI Effort Lex Fridman Has Seen

Fridman opens by saying Tesla AI Day presented the most amazing real-world AI and engineering effort he has ever seen. His reason is not primarily the Tesla Bot. He has long believed that autonomous driving and general real-world robotics perception and planning are much harder than people assume, and that the required scale of effort was out of reach in the near term.

What changed his view was seeing the pieces together in one place. They include the neural network architecture and pipeline, the in-car Autopilot computer, the Dojo hardware for training, data and annotation, simulation for rare edge cases, and the generalization of all this beyond cars to a humanoid robot.

Fridman lists the neural network ideas as steps or leaps beyond the state of the art. The first is predicting in vector space instead of image space, which he calls an obvious idea that is very difficult to execute, since the world is three-dimensional and not a 2D projection. The second is fusing camera data before detection, at the multi-scale feature level, rather than detecting per sensor and combining only the decisions.

Third is using video context to model time as well as space. Each frame concatenates positional encodings, multi-camera features and ego kinematics, using a spatial recurrent network that builds a 2D grid around the car with a recurrent unit in each cell. That allows a map in the space of RNN features and potentially planning in it. He notes Andrej Karpathy also mentioned moving fusion earlier in the network, toward fuller end-to-end driving.

The final idea concerns planning, where Fridman understands neural networks have been used least. Since optimal planning in action space is intractable, heuristics are needed, and the presentation proposed learning them, as neural networks guided tree search in AlphaZero and MuZero. That prunes the search for a plan that avoids local optima and gets close to the global optimum. He praises the presentation for not dumbing anything down and calls the combination brilliant.

To predict in vector space, you must label in vector space, so Tesla built in-house tools and hired an in-house team of annotators. They label in vector space and project the result into image space, which saves work and puts annotation in the same space as prediction. Auto-labeling, Fridman says, is the key to the whole approach.

He singles out the use of clips combining video, IMU, GPS and odometry from multiple vehicles at the same place and time to generate labels for the static world and for moving objects and their kinematics. The clips effectively annotate each other, so the approach grows stronger as the fleet grows and catches more edge cases.

Simulation covers rare edge cases that will not show up often even in a very large dataset. It is also used to annotate ultra-complex scenes, such as one with around a hundred pedestrians, where accurate labeling of real data is practically impossible. He concludes that the architecture and annotation work is a big leap.

On the compute side, Fridman mentions the neural network compiler that optimizes latency and testing and debugging tools that compare candidate networks before deployment, which he likens to developer tools for neural networks. He recalls that close to 10,000 GPUs are being used to retrain the network continually, and that it is fully retrained end to end every week or two, though he says he does not remember the exact number.

Dojo is still under development: the Autopilot computer does inference in the car, while Dojo is the data-center machine for training. He describes a training tile built from Tesla's own D1 chips with very fast I/O, designed to connect in arbitrary numbers, and a large cluster that he recalls as exaflop-scale. He thinks it could become a contender for the world's most powerful neural network training computer.

The bigger picture, he says, is the iterative data engine: auto-label plus manually label edge cases, collect data, retrain, deploy, and repeat. He still thinks nobody knows how hard autonomous driving is, and that driver sensing and human-robot interaction remain open problems, but believes this loop has no ceiling.

Fridman gives two more reasons the effort is so notable. Dojo could be offered as AI training as a service, directly taking on AWS and Google Cloud, since nothing ties it to Autopilot. He points to the simplicity with which Tesla described deploying PyTorch across its nodes and says it could serve any machine learning problem that needs scale.

The third reason is that the architecture and data engine apply well beyond roads: in homes, factories and robots, any form with cameras and actuators, including the humanoid Tesla Bot. As someone who loves robotics, he finds the bot exciting. His lifelong dream is a robot that becomes a friend and companion, not only a servant for boring or dangerous tasks.

He thinks both problems will be solved in parallel. If successful, the Tesla Bot may address perception, movement and object manipulation, while he hopes to help with human-robot interaction and friendship. He ends by saying the whole picture paints an exciting future.