MI300 Leads Inference by 2x: AMD's CTO on Winning Chip Competition Without a Walled Garden

Sarah Guo and Elad Gil welcome Mark Papermaster, AMD's chief technology officer, onto No Priors to talk GPUs and the competition that has been driving the industry. His timing, he says, was lucky. Fresh out of electrical and computer engineering at the University of Texas, he joined IBM just as CMOS — the process every chip uses today — was coming into production. He got onto IBM's first CMOS projects, designed some of the first parts, and put his hands on nearly every facet of chip design. Later he ran microprocessor development across PowerPC, which meant working with Apple and Motorola, and the "big iron" chips in IBM's mainframes and large RISC servers.

Steve Jobs then hired him to run iPhone and iPod engineering. After a couple of years he took the AMD role in fall 2011 as both CTO and the person running technology and engineering, right as Moore's Law was starting to slow and the industry needed a lot of new invention. AMD itself is more than fifty years old. It began as a second-source company on key components and x86 microprocessors. When Lisa Su, now CEO, and Papermaster were brought in just over a decade ago, the mandate was to get AMD back to strong competitiveness. They started with CPUs, then spread across the portfolio. In February 2022 AMD acquired Xilinx; it later bought Pensando for networking interconnect as workloads scale out.

The company he describes today is much broader than the PC-and-gaming shop he inherited. AMD CPUs and GPUs sit under the world's largest supercomputers. A massive installed base of EPYC x86 servers runs in the cloud. The chips are underneath Xbox, PlayStation, and many add-in gaming boards. Xilinx fills out embedded, alongside embedded x86. That is the map. The question of the hour is how it got from a turnaround in PCs to a flagship that Meta and Microsoft were willing to stand on stage and buy.

What first caught AMD's attention in AI is the same story everyone in the industry knows. GPUs became pivotal once the application space opened, and Geoffrey Hinton's work showed they could drastically improve image recognition and natural language processing. AMD saw the opportunity immediately. The question was the course. From about 2012 through 2017, nearly all revenue still sat on PCs and gaming. The building blocks had to be leadership products that would attract people onto the AMD platform for high-performance work. First they had to rebuild the CPU roadmap. Zen arrived in 2017 as Ryzen in PCs and EPYC in servers, the revenue ramp started, and the portfolio began to extend.

In parallel they had already "called the ball" on heterogeneous computing, even before Su joined. AMD's earlier acquisition of ATI had brought GPUs into the house, and that combination is one of the reasons Papermaster took the job: it was really the only company with a very strong CPU portfolio and a very strong GPU portfolio. The industry, he thought, needed serial, scalar CPU work and the massive parallelization of a GPU in one place. They have been shipping CPUs and GPUs combined for PC applications longer than anyone, starting in 2011 with APUs — accelerated processor units. For big-data work they started in HPC, the national-lab and oil-exploration flavor, winning government bids that led to supercomputer designs running AMD CPUs and AMD GPUs. That hardware-and-software effort ran for years. It culminated on December 6, 2023, with the flagship MI300: one variant a beast for high-performance compute, another optimized to take AI training and inference head-on.

At launch, Meta and Microsoft made public purchase commitments. The workload Papermaster is most focused on is still the bulk of today's AI spend: building up the accuracy of large-language-model training and inference, the ChatGPT- and Bard-class systems that ingest vast data with an ultimate goal of AGI-type capability. MI300 was designed as the halo product that could take on the industry leader in that arena. He says it has: competitive on training, leading on inference, with over 2x on FP16 vLLM, a metric generally everyone can run.

They did that on purpose. They built efficient engines for the math that training and inference need, and they brought the memory required for more efficient computing — more work at less power and less rack space than the competition. In a data center, electricity and floor area are the bill. Large-model inference is hungry for memory bandwidth; if memory cannot feed the engines, the compute sits idle. Stacking high-bandwidth memory against the compute chiplets shortens the data move. The halo-then-spread motion is the same playbook they used when Zen rebuilt the CPU business: plant the flag at the top, then fill in the rest of the line.

Guo puts the rest of the contest on the table: overall performance, efficiency, and the software platforms — CUDA versus ROCm. Why would a developer move? Papermaster says the arena is multifaceted, which is why so many startups struggle. A fact he wants in the air: the bulk of inference today still runs on general-purpose CPUs. That is everyday AI applications, not huge LLM serving. Large-language-model work is almost all on GPUs because that is the software and developer ecosystem. AMD has been competitive and gaining share on CPUs, generation after generation on schedules it laid out. For GPUs it took until now to have world-class hardware and world-class software.

Because it is a GPU, deployment should be easy. The chips keep GPU programming semantics, so lower-level code is not a foreign language. They support the key frameworks: PyTorch — AMD is a founding member of the PyTorch Foundation and one of two qualified hardware offerings — plus ONNX and TensorFlow. On Hugging Face, they partnered with Clem Delangue's team. As thousands of open-source language models are released, Hugging Face tests them on AMD Instinct GPUs equally with NVIDIA, with regression testing literally every night. Early customers close the loop. Elad's portfolio company Lamini has been running LLM services on AMD racks and talking to real users; Papermaster wants that feedback more than any internal suite, because the goal is a seamless deploy. Gil's gloss is that the hunger from software and ML teams for competition tells you how constrained the environment had been.

Papermaster goes further: no competition means a stagnant industry. Look at CPUs before AMD returned to leadership — incremental improvements, not much else. The pull and partnership, he says, have been tremendous. In return AMD will keep shipping competitive product. Guo asks about the philosophy of keeping a stack as large as ROCm open source. It is near and dear. Collaboration is the culture. AMD's CPU compiler is LLVM; LLVM sits under the GPU compilers too. More than the compiler, they opened the entire ROCm enabling stack, a huge piece of winning the large supercomputing installations. Xilinx had the same philosophy, so the 2022 combination only deepened the commitment. They do not want to lock anyone in with a proprietary walled-garden stack. They want to win with the best solution, give customers choice, and win on merit, generation in and generation out.

Gil zooms out to the AI-compute cloud. Beside Azure, AWS, and GCP, names like Baseten, Together, Modal, and Replicate have emerged, offering different tooling and endpoints, and in part riding a GPU shortage. How does that market look in three or four years if constraints fall away? Papermaster says the supply constraint will go away. AMD is ramping and shipping Instinct on plan. The more important fact is that the market itself is expanding at a breathtaking rate. Today's generative LLM apps started hyperscaler-based because training and a good deal of that inference need massive clusters. Now application after application is taking off nonlinearly. People are learning to fine-tune, to train smaller models that only have to serve their business, not answer every question in the world. That produces a huge variety of cluster sizes and configurations.

Compute keeps sinking: from hyperscalers to what he calls tier-two data centers, then to the edge. Put inference on the factory floor, at the source of the data, when you need low latency. Then end-user devices. AMD has been shipping AI inference accelerators on its PCs throughout 2023 and at CES 2024 already announced the next generation of AI-accelerated PCs. The Xilinx embedded line is seeing a lot of pull for bespoke inference in a plethora of applications. Tailored compute installations are how you service ballooning demand. Gil asks what is actually binding GPU supply — packaging, TSMC capacity, power into data centers. Papermaster's pandemic story is the template: a run on PCs and x86 servers, substrate shortages, scramble, then they secured more substrate capacity and leaned on a decades-deep TSMC partnership. Get ahead of the signals and shortages stay contained.

AI demand is a massive increase, fabs are responding, and you cannot think only in wafers. AMD and its GPU competitor both use advanced packaging. He holds up MI300: a set of chiplets, some doing CPU work, some IO and memory control; on the HPC variant they drop in a CPU in the same integration; high-bandwidth memory around the engines, connected laterally and, on MI300, vertically. It is a complex chain, but AMD has been fabless for coming on eighteen years and he tips his hat to the supply-chain team. The industry, he thinks, will move beyond those shortages. Power is the constraint that remains. Every major operator is looking for sources of power. For the people who design the engines that consume it, that puts energy efficiency at the very top of each generation.

Guo reminds him he once said the end of Moore's Law was part of why he wanted AMD. He was attracted to a company with a history of disruption, to the CPU-plus-GPU combination, and to chiplets, which were only early exploration then and which the engineering team rallied around. The easy picture of Moore's Law is that the foundry did most of the heavy lifting: a new node shrank devices, gave more performance, used less power, at the same cost. Slowing does not mean devices stop improving. It means they cost more, power does not fall as much, and you still pack more transistors — but you need more invention. He calls that holistic design.

Holistic design has layers. Match the engine to the job: CPU, GPU, or a dedicated engine such as the super-low-power AI acceleration in PCs and embedded devices. Combine chiplets so each function sits on the best technology node for that function. Then keep going through packaging, interconnect, and the software stack. Optimization is the full circle from transistor to application. MI300 is the exhibit: small chips for CPU, IO, and memory, high-bandwidth memory feeding the engines, lateral and vertical stacking. AMD does not need to own the entire system or the end application. Collaboration is the culture; partner deeply and optimize the solution end to end.

The strategic layer arrived suddenly for everyone else. Guo asks how AMD thinks about TSMC's near-monopoly and fab security in a harder geopolitical climate. Countries now run on chip designs that power essential systems, so continuity of supply is national security, and AMD builds that into strategy with partners. They have supported fab expansion: TSMC in Arizona, Samsung in Texas, plus facilities in Europe and other parts of Asia. Packaging needs the same geographic diversity — chips on carriers, interconnect, the whole ecosystem. Software can ship an MVP and go viral. Expanding a semiconductor supply chain takes years of prep, and you have to start now. The industry was built as a global web of expertise pockets. With more volatile politics and economics, spreading manufacturing is imperative, and he says it is well underway.

Gil runs a list of new hardware — Vision Pro, Rabbit, Humane, Figure — and asks what predicts success or failure. Papermaster starts with technology. Chip design is part of why the explosion is even possible: more compute, shrunk down, drawing so little power that audiovisual devices can exist at all. Earlier headsets were too heavy, too big, and not enough compute. If the lag from a photon on the head-mounted screen to finished processing is too high, you get physically ill watching a movie or playing a game. Meta Quest and Vision Pro are happening because the industry chewed through that power-and-compute problem. Technology is only the enabler. What he learned at Apple is that successful devices serve a need. They give you a capability you love, not an incremental "I can do this a little better than something else." They have to create a new category. It is too early to call the current crop, but the frame is clear.

AI on PCs, he thinks, is close to making PCs a new category. Imagine not speaking English, watching this podcast live, hitting a live-translation button, and hearing it in your language with no perceptible delay. That is one of a myriad. Guo notes that in the cloud era, load moved to servers and latency stopped being a primary design consideration. Now models are slow, people chain them, and on-device work is back. Papermaster agrees. Run the heavy, efficient compute in the optimized data center, but write the algorithm so the pieces that need instant response sit at the edge or on the device. His test case is a highly autonomous vehicle that loses its cloud signal and simply stops. You would not stand for that.

What does he want engineers, founders, and executives to know about AMD in 2024? It is a huge year. They spent years building hardware and software for AI and have now AI-enabled the entire portfolio: cloud, edge, PCs, embedded, and gaming devices that upscale with AI. 2024 is the deployment year. The bedrock is there. Partners are there. He admits they are often unknown in AI; everyone knows the competitor. They want to be known over the course of the year, on results and on value, as the company that brought AI across that breadth of applications — massive LLM training and inference in the cloud, and equally the rest of the compute space. Microsoft is talking about cloud-to-client enablement; many ISVs are asking the same question Guo asked: how do I write the app so the best experience taps both the cloud and the device in your hand or on your laptop. He calls it a transformational year, and AMD wants to be in the middle of it.