Jensen Huang on the AI Computing Revolution and Human Expertise

Joel Hellermark asks when Jensen Huang became convinced of deep learning's potential. Huang recalls that, around the 2012 ImageNet competition, several groups of researchers approached NVIDIA for access to the newly released GeForce GTX 580 and help programming GPUs for deep neural networks.

At the time, Huang says, neural networks were a questionable research area in many people's eyes. What struck him was how effective the approach was. He began asking how far it could scale and what other problems it could solve.

Huang says the layered structure of deep neural networks and the effectiveness of backpropagation suggested considerable room to scale. NVIDIA saw deep learning not only as an algorithm for problems that are hard to specify, but also as a new method of developing software.

He describes a model as a general function approximator that can learn across different problem sizes and dimensions. Extrapolating from that insight convinced NVIDIA that computing itself might need to change. Andrej Karpathy later called the approach Software 2.0.

Looking back, Huang sees GPU-accelerated computing and deep learning as a reinvention of computers after decades of conventional computing. Hellermark recalls the early use of GPUs by Ilya Sutskever and colleagues, then asks where model architectures are heading.

Huang points to transformers and researchers' ability to express many kinds of data in a form they can learn from: vision, audio and text. He expects multimodal models to improve performance by combining information from different sources.

As an illustration, he imagines a vision system that has seen horses but never zebras. Combining a horse image with the description of black and white stripes could help it form an idea of a zebra without having seen one. In his account, the combination extends what the system can recognize.

Different sensor modalities can also make perception more robust. Huang cites combining cameras, radar and lidar, and gives a simpler example: the words ‘What is that?’ become clearer when accompanied by a pointing gesture.

Hellermark asks whether models can learn physics from multimodal training data. Huang thinks language can describe many physical effects, just as someone might infer something about the color red through comparisons in words. But he doubts words alone convey all the sensations and subtleties of experience.

For systems that must predict physical phenomena, Huang argues for grounding AI in physics. He draws a parallel with reinforcement learning from human feedback and proposes physical feedback from simulation for robots.

Huang describes NVIDIA Omniverse as a system for simulating physics and creating a digital twin of an embodied robot. The simulated environment could provide feedback to the robot's model. He compares physical grounding for robots with alignment intended to make chatbots safer.

Asked whether progress will require radical breakthroughs or can continue through scaling existing architectures, Huang says he does not know the science well enough to give a definite answer. His intuition is that symbolic reasoning and explicitly stated rules may augment models trained through extensive learning.

He notes that people receive many rules through language rather than learning each one by trial and error. In his view, intelligence may combine rules, reinforcement learning and experience; he leaves open exactly how that combination should work.

Hellermark asks what role people will have if AI becomes much more intelligent than humans, and describes using AI to help researchers run many experiments. Huang admits he does not know, but compares the prospect to working alongside people whose particular abilities already exceed his own.

Huang expects access to intelligence to broaden, comparing it with farming and the wider availability of power and energy. Those changes expanded what society could do. He nevertheless expects deep domain expertise and passion to retain value, with AI helping people explore more possibilities.

For most of Huang's career, he says, the vast majority of people could not program computers in languages such as Python, C++ or Java. With systems such as ChatGPT, people can instead give instructions in their own language and even ask a system to write software.

Huang hopes this will help bridge the digital divide and empower more people to use computers as tools for creating value. He points to children producing programs and images by prompting, despite not having learned conventional programming.

Hellermark asks how NVIDIA uses AI to build more advanced computing hardware. Huang says modern chips are so complex that the company uses AI to explore hundreds of thousands of design permutations and identify useful trade-offs.

People still decide which trade-offs matter. A critical path might be optimized for speed, while a component repeated thousands of times might demand energy efficiency. AI can then search combinations and corner cases; Huang says some designs in the Hopper generation were found this way.

Hellermark asks how application companies can build defensible businesses as models become more capable and widely available. Huang answers that deep understanding of a domain and of customers' problems has not lost its value.

Even if an AI can pass a bar exam, Huang says, applying expertise to a particular industry requires learning its fundamentals, the problem at hand and the relationships between people. Customer needs are often poorly expressed, so understanding their context remains a social as well as a technical challenge.

Huang describes three ways AI can affect NVIDIA: improving how it builds products, changing the products themselves and augmenting the company. Chip and software design are examples of the first. For the second, he points to graphics systems that use AI to infer missing pixels.

He says NVIDIA trains models in the background to predict additional pixels and distributes improved algorithms. In his account, this changes how a GPU produces images and improves efficiency. He then imagines an AI-assisted organization that helps employees find information and detect signals in demand or supply chains.

Asked for a contrarian view, Huang says discussion already ranges from extreme enthusiasm to extreme concern. He expects powerful technology to create economic and social disruption, requiring retraining as some jobs change, while also creating occupations that did not previously exist.

As a contemporary example, he points to prompt engineering and predicts it could become a significant field. He also observes that AI systems can help write prompts for other AI systems.

Huang welcomes the safety discussion and says investment in AI capabilities should be matched by investment in safety. He compares the work to self-driving cars, where resources must go both to making a car drive and to making it safe.

For language models, he mentions guardrails, alignment, reinforcement learning from human feedback and vector databases that can supply facts to reduce hallucination. He expects a surrounding collection of AI techniques, methods and practices to turn core models into useful chatbots.