Moore's Law Is Slowing. Nvidia's CTO Still Wants 10× a Year

Sonya Huang introduces Michael Kagan as one of the legends of the semiconductor industry: formerly chief architect at Intel, then co-founder and CTO of Mellanox, which Nvidia acquired for $7 billion in March 2019. In the years since, she says, he has been a major driver of Nvidia's position as the AI compute platform, largely because interconnect is how you push chips past Moore's law. The AI race, in her framing, is a silicon race — squeeze the most intelligence out of each unit of silicon — and the frontier has moved from packing transistors onto one die to wiring thousands and hundreds of thousands of chips into a single fabric.

Pat Grady sits down with the CTO of what was then the world's most valuable company and goes straight to a house argument. Sequoia partner Sean, he says, likes to make the case about every six months that Nvidia would not be Nvidia without Mellanox, the company Kagan co-founded some 25 years ago and has stayed with through the acquisition. Kagan does not treat that as flattery. Computing demand is growing exponentially, people keep estimating linearly, and the exponential itself has accelerated.

Moore's law, Kagan reminds them, was twice every other year, and as physics it is "not quite running anymore." The break came around 2010 and 2011, when the GPU stopped being only a graphics processing unit and became a general processing unit — the first time AI workloads ran on it, taking advantage of programmability and parallelism. From then, the coefficient changed. Models started doubling in size and capacity every three months, which he translates as 10× or 16× a year of performance growth versus the old 2× every other year.

You cannot feed that by iterating on a single component. Innovation has to happen at a much higher scale, and that is where the network comes in. He splits the problem into scale-up and scale-out. In CPU days, scale-up meant more transistors, out-of-order execution, then multi-core. In the GPU world the building block is the GPU, and when one piece of silicon is not enough, Nvidia's answer is NVLink. What they call a GPU today is a rock-size machine. You need a forklift to lift it. Order a GPU on Amazon, he jokes, and do not wonder when a huge rack shows up. People think chip. It is a system.

That building block is not just silicon and wires. A software layer exposes CUDA as the API, which is what lets them scale, with some simplification, from one GPU all the way to 72 while keeping the same software interface. Those 72 GPUs are actually 36 computers, two GPUs each, wired together and presented as a single GPU. Once the block is as big as power, cost, and efficiency allow, you scale out: connect many of them and split the application across the machines. Before the Mellanox deal, NVLink scale-up stopped at a single node. Getting connectivity outside that node is not plugging in a cable. It is software and network technology that make multiple nodes work as one machine, and that is the first place Mellanox showed up.

Scale-out is the second Mellanox problem. If a task takes one GPU one second, you split it into a thousand pieces, send each to a different GPU, and finish in a millisecond — but only if you can communicate the split, consolidate the results, and do that across many iterations. Communication and computation take turns. You want as many pieces as possible because that is the speedup, yet if communication blocks, you waste time and energy. Fast networks buy you small pieces. Tuned applications hide communication behind computation. If a transfer runs long, everybody waits.

So the network needs more than hero numbers, the peak gigabits per second on a slide. Sending a bit from A to B is physics; Nvidia is a little better, Kagan says, but that is not the big advantage. The requirement is that no matter who is talking to whom, the latency distribution is very narrow. Do it thousands of times in the same time, versus a wide spread on other products, and the machine stays efficient. Accommodate jitter inside the compute phase and you may only split the job across 10 GPUs instead of a thousand. Inherently, the network determines cluster performance.

Nvidia therefore architects the data center as a single unit of computing: 100,000 GPUs, multiple compute chips, two to five network chips, software and hardware designed at that grain. The other face of the machine is the customer. It has to serve multiple tenants and run an operating system. That is BlueField, the DPU, a computing platform for the data center OS. On a conventional computer the CPU runs both the OS and the application; you want to maximize cycles on the app, and you want to isolate infrastructure computing from application computing. Side-channel attacks such as Meltdown, almost a decade earlier, are much harder when those jobs do not share a CPU. Kagan is not objective, he admits, but he also does not think Mellanox's networking business could have grown this fast alone. Nvidia's Ethernet business, leaving aside NVLink and InfiniBand, is in his telling the fastest-growing Ethernet business ever.

Grady asks what breaks as you go to 100,000 GPUs, maybe a million, and how software designs around it. Kagan calls it multi-stage. The unobvious point for many engineers is that a component that works 99.999 percent of the time is fine in a single box. A 100,000-GPU machine is millions of components. The chance that everything works is zero. Something is definitely broken, and you have to design hardware and software to keep going — performance, power efficiency, service — anyway. That challenge starts at a few tens of thousands, well before a million.

Sometimes a single job runs on the entire data center. Hardware and low-level software then have to give the application and the scheduler hooks to place different parts of the job efficiently. A network at that scale is a different story from a general-purpose data center network, which he compares to the internet: loosely coupled microservices that look like one service from the outside. Here you are running one application on 100,000 machines. Grady asks if that is training-specific. Kagan says it is true for everything; it depends on scale. Inference is another topic, but the hooks have to be there either way.

As long as the machine fits a building, on the order of 100,000 GPUs, the constraint is power — now talked about in gigawatts. For many reasons you still want to split work across data centers kilometers or a continent apart, and that introduces the speed of light. Latency variance between parts of the machine becomes huge, and congestion across that difference is what deteriorates networks. Old telco practice put a box with huge buffers at the edge as a shock absorber. Huge buffers are not good, Kagan says; bigger is not better, and those buffers are jitter. If one workload spans distant sites, every machine has to know whether it is doing short or long communication and adjust patterns so you do not need the buffers. Nvidia's answer is Spectrum-X, an Ethernet switch at the data-center edge that feeds telemetry so endpoints can adjust for congestion.

Huang wants the shape of the workload: backprop versus forward pass, whether demand is shifting from pre-training to inference, and whether those jobs will even share data centers. Kagan starts with training. It has a forward pass that is inference, a back-propagation pass that adjusts weights, and, for data-parallel training, a third phase that consolidates weight updates across model copies. Until recently that stack was the main driver of compute. Until maybe two years earlier — ages in AI — inference was mostly perception: show a picture, that is a dog, one path, done.

Generative models made inference recursive. After the prompt, every new token sends you through the entire machine again, so one answer is many inferences. Reasoning is another multiplier: a simple "what time is it" is cheap; a hard question means comparing multiple paths, and every path is inference. Inside a single inference he splits two phases. Prefill processes the prompt and relevant context; it is compute-heavy, not especially memory-heavy. Decode emits the answer token by token and is memory-intensive. Put it together and inference demand is not less than training. It is more. Two reasons: each inference now does far more work than perception ever did, and you train a model once and infer many times. ChatGPT trained once; something like a billion customers pound that same model, and now they are making videos. His wife, since she discovered it, talks to ChatGPT more than to him. Best friend.

Will inference live on the same machines? You can infer on a phone, so there will be much smaller installations. At data-center scale, programmability is more viable than a zoo of specialized hardware, because every custom instance has a cost and a drawback. Nvidia has announced a GPU SKU optimized for prefill; it can still decode, and a decode GPU can still prefill, so a shift in mix can be absorbed. The interface stays CUDA. That unified programming model, he says, is what made Nvidia Nvidia before Mellanox.

Grady reaches for an analogy. Chips hit quantum mechanics; racks hit power. Is there a law of data-center scaling where communication overhead kills you if the building gets too big? Kagan's practical limit is how much energy you can consume in a given volume. Heat has to leave. Nvidia has moved pretty much entirely to liquid cooling so it can build denser compute than air would allow. The last big sites, xAI-scale, are 100 or 150 megawatts. Now the talk is gigawatt data centers, and people talk about 10 gigawatts. Asked whether they are sending data centers to space, he laughs and names a more earthly bottleneck: how fast concrete gets stable.

Before Mellanox he spent 16 years at Intel and became chief architect, so Huang asks about the recently announced Nvidia–Intel partnership. Computing changed over the last decade-plus. Nvidia began as accelerated computing for video games and evolved into AI, a new way of processing data. A von Neumann machine is "explaining somebody what to do." He can explain many things. He cannot explain how to distinguish a cat from a dog. Those problems need acceleration, and general-purpose computing is not going away. The partnership is about fusing accelerated computing with x86, still the dominant architecture there. Short-term, it gives both companies channels into markets that were otherwise hard to serve. The culture underneath, which the episode also used as a cold open, is win-win: not a bigger piece of an existing pie, a bigger pie. Nvidia's success is customer success and ecosystem success, not the failure of the competition.

When Mellanox joined in 2019, Grady notes, the combined market cap was about $100 billion. Today it is about $4.5 trillion, 45× in six years. How did that change the culture? Kagan remembers Jensen Huang in Israel, and a slide where he said 1 + 1 would be 10. He was off by a factor of four. The cultures were similar enough to fuse, and he was the only founder who remained after the other left a few months post-deal. What he thought about in the shower was making the acquisition succeed: Nvidia had paid $7 billion for a company he had founded, mixed feelings and all. About 85 or 90 percent of the original Israeli employees stayed; Nvidia more than doubled headcount in Israel and announced a new campus. Jensen kept emphasizing networking as critical to Nvidia's success. Kagan thinks the deal is considered among the most successful mergers in the history of technology. Looking back, the move was right.

Science fiction, for Kagan, is making history into an experimental science. Physics can try something and see if it works; time in history only goes one way. A good enough world simulator — Nvidia's Earth 2 climate simulator is the example — lets you test what today's choices do to global warming 50 years out. Further on, once models understand physics, they can teach it. Theoretical physics observes phenomena and generalizes a law. AI is good at observing, processing data, and generalizing. It might surface laws of physics we do not even imagine now.

Huang names the slope: Moore's law was 2× every two years; Huang-plus-Kagan's law is what? Kagan puts it in the range of 10×, or a few orders of magnitude a year. For two or three years Nvidia has accelerated product introduction from every other year to every year, each wave an order of magnitude higher — not chip-level scores, the machine you can build, the single unit of computing. How long can that last? He does not know. They will do their best to maintain it and probably even accelerate. It is all about the exponent. Logarithmic plots make Moore-style curves look linear, which is the wrong way to look. Nobody predicted that the least-used function on the 2007 iPhone, 17 years later, would be the phone, with e-commerce, texting, news, mail, and identity living on the same slab. They are building the platform for whatever comes next, which cannot be imagined from here.

Her last question is the optimistic view, five or fifteen years out. Steve Jobs called the computer the bicycle of the mind. AI, Kagan says, is probably a spaceship. There are things he wants to do and does not have the time or resources for; with AI he will. That does not mean twice as much. Maybe ten times as much — and he will want to do a hundred times what he wants today. Ask any project leader whether they have enough people. Give them a resource twice as efficient and they will do four times more work and want ten times more. Electricity changed the world that way. London still shows the old gas-lamp infrastructure, and no one imagined a civilization that cannot live without the grid. The same, he says, with AI. A new world.