A Wafer-Scale Chip So AI Stops Waiting on Amdahl

The cold open is Feldman, not a product demo. This was not about making money or moving the ball a little. The Cerebras founders wanted to put their shoulders to an industry and see if they could transform it, with a solution that was "not 10 times better but 50 or 100 times better." Nathan Labenz then sets the stakes. Algorithms, data, and compute are the three ingredients of AI; the show has talked a lot about the first two. This is the first step into hardware.

The guest is Andrew Feldman, CEO and co-founder of Cerebras Systems, makers of the world's largest chip and an inductee into the Computer History Museum. For years AI has run on GPUs because they were the best way to parallelize at scale. Nvidia, Labenz notes, is now a $750 billion company, more than doubled just in 2023. Those GPUs were not designed for AI. Wiring them together, and shuffling the data, has been a significant share of large training runs. GPT-4 is rumored to have cost some $100 million or more; next-generation systems could stretch into the billions.

That makes compute a matter of industrial policy and a possible regulatory choke point. Feldman and his colleagues, all chip-industry veterans, saw the deep-learning workload coming and set out to build from the ground up. What they sell is not only the biggest chip ever made but a full multi-million-dollar computer that tries to erase cluster complexity by scaling the silicon itself. Nvidia, Google, Meta, and Microsoft are all designing AI chips too. Labenz's view is that a market growing this fast has room for more than one winner.

The record everyone knows is 2.5 trillion transistors. Feldman reaches for cars. Some are good for soccer practice and the grocery store, some haul lumber and bricks, some are fun on Saturday. Cerebras wanted a chip optimized for one thing. AI, he says, is a huge amount of relatively simple compute plus a tremendous amount of communication. Build a very big chip and you keep that information on die. Communication on chip is fast and power-efficient, on the order of thousands of times faster and thousands of times more energy-efficient than leaving the chip, crossing a motherboard, maybe hitting a switch, and arriving at another chip.

Labenz asks him to start even lower. A chip, Feldman says, is an electric circuit on a piece of silicon: transistors embedded in silica, which is how most of the digital world is powered. The diversity is easy to miss. There are milliwatt sensors, chips that run dishwashers and coffee makers, chips throughout a car, and then the phones, laptops, and cloud boxes people already think of. Chips are where compute lives. Designing 2.5 trillion of those transistors is really hard. It takes years, hundreds of millions of dollars, and a taste for brutal engineering.

Cerebras is a systems company, not a design house that stops at the GDSII file. They build the chips, the motherboards, the enclosures, the power, the cooling, and sometimes clusters of the resulting machines, including some of the largest AI supercomputers ever built. The stack runs from silicon through the compiler and management software. Feldman is on his fifth startup; each time the pattern has been dedicated chips, boards, systems, and system software aimed at a hard problem.

In late 2015 the founders sat down and asked which problem they wanted to solve. AI was the one that interested them. What does that work actually want from the machine? Their answer was a dedicated computer, not a repurposed graphics processor, in which every decision — size, shape of the core, number of cores — optimized for AI and not for a pile of other jobs. Labenz hears foresight: the models of 2015 were much smaller. Feldman denies having any. Technical vision belongs to the co-founders. His skill is picking them, and world-class employees.

What he did see was a historical rhyme. Cell-phone compute was a new challenge; neither Intel nor AMD captured it, and ARM took 100 percent. AI looked like the same kind of sea change. They probably underestimated its size. Serial founders, he says, work with the same people again and again. CTO Gary Lauterbach was his co-founder at the last company. Michael James, J.P. Fraker, and Sean Lee were technical leaders there. Of the first 20 or 30 employees, almost everyone had worked with them before, some going back to 1996.

On the whiteboard they wrote two things: they wanted to work together again, and they wanted to do something important. Not a little better — 50 or 100 times better. They socialized that idea with the best people they knew, then with investors. They made eight pitches and got eight term sheets, funding quickly with what Feldman calls the best in the business, plus super-angels to keep them from egregious mistakes. Architecture came first: point A to Z, which pieces need invention, which are solid engineering. They like to start with the hardest piece, which here was a chip orders of magnitude faster than a GPU, especially on memory bandwidth.

Labenz had been reading Chris Miller's *Chip War* and came away with a picture of extreme specialization and single points of failure — one Dutch company making the tools that etch every advanced chip. Going off anyone else's roadmap, he asks, what was actually hardest? Feldman answers with medieval maps: known land, then a dragon, and under it the words "here be dragons." They were only interested in sailing there. People had tried and failed to build big chips. There was a literature on why you would fail. Cerebras solved those documented problems in about six months for $20 million. Nobody had failed further up the mountain.

He likens it to sitting at the base of Everest before it had been summited. Climbers come down and say halfway up is brutal. You go to the top, come back, and tell them that was not the hard part. At every stage they had to invent tools, manufacturing equipment, cooling, and components; there is no catalog for pioneering work. Vendors got excited anyway. One supplier's engineer, not a Cerebras employee, told them that what they were doing was why he became an engineer. There was no easy bit.

After the gang reassembled, they spent those six months on a software simulator that they believed would behave like the chip, and began experimenting with trade-offs while early physical design started. Chip architecture, Feldman says, is a fixed piece of real estate. You allocate some of it to compute, some to memory, some to communication. Big cores or lots of little ones, centralized memory or a private store per core, a few fat pipes or many small ones: each workload would answer differently. AI is not one big shared-cache problem. It is lots of little calculations that move from layer to layer. Supercomputers historically wanted 64-bit double precision; AI does not, so they do not have it. GPUs still spend silicon on graphics problems Cerebras simply omits.

Labenz offers a poor man's account of GPU memory pain in training: you have to keep parameters and backprop gradients around until a batch of updates lands, then you can free space. Feldman says it is a good poor man's account, and names two problems that haunt GPUs. First, parameter counts now overflow a single card, so you have to shard weights across many GPUs. Second, a big layer's matrix multiply is itself too large, so you break it into pieces, spread those over hundreds or thousands of cards, and wait for the last one. That is Amdahl's law. You care desperately how far away in time each shard is.

Look at the Llama or GPT-4 papers, he says, and the acknowledgments often credit dozens of people just for distributed compute — not the model, the shatter-and-place work. On a Cerebras chip that placement takes minutes. The architecture has memory for trillions of parameters and 850,000 cores, enough that even the largest layer of the largest network fits. You never have to break things up. Labenz hears a closer match to the mental picture of a neural net: layers, data moving, parameters sitting still, instead of an incomprehensible distributed mess.

History, in Feldman's telling, split in 2015 and 2016, when communication looked like the hard part. Cerebras built a bigger chip so you communicate less. Nvidia bought Mellanox, a specialist in Ethernet and InfiniBand, to tie lots of little chips together. Tying little chips together is messy, and performance is almost always sublinear, the way adding a second person never cuts a job to half X and adding 50 people never cuts it to a fiftieth. Coordination is what management exists for. Cerebras holds activations on chip, streams data and parameters in, streams gradients out, and gives every core its own memory. Contention — the supermarket checkout when every core lunges at off-chip DRAM at once — never forms.

Labenz is struck that the company started before Transformers and is still, in Feldman's claim, the fastest on Earth at them. How narrowly tailored is the chip? If a recurrent-network breakthrough arrived, would that hurt? Feldman calls this one of the fundamental questions in computer architecture: what to make general, what to specialize. Nvidia put custom circuitry for 3x3 convolutions in the V100 and A100. Great for computer vision, extremely inefficient for language. That is the price of specialization. Cerebras aimed one level down. Graph nets, vision, and large language models all decompose to sparse linear algebra. They optimized the engine for that, and it turned out to be a good choice.

Sparse means a matrix full of zeros. A GPU multiplies everything anyway. Multiplying by zero takes time and power and produces no new information; it is walking in place in a race. Cerebras's machine was not designed for graphics, which is a dense problem. When a core hits a zero it skips the multiply, because it already knows the answer. As sparsity shows up in weights, activations, or structure, you stop doing useless work. Every watt and every cycle can point at finishing the training run, the autoregressive decode, or the classification.

A stack of recent papers, including Cerebras's own and a Neural Magic paper on inference, show you can sometimes round weights to zero and lose nothing. Pair that with a sparse engine and you train much faster, with less compute and a fraction of the energy. That is the research program: shrink the absurd compute these models demand without losing accuracy. Asked how a skip-on-zero is physically instantiated, Feldman switches to freight. If your logistics only move pallets, you wrap boxes, forklift them, and ship empties across the country because you cannot see inside. Huge memory bandwidth is the ability to inspect each box, throw out the empty ones, and move vectors or even scalars instead of pallets. Fine-grained control is what you buy when bandwidth is extraordinary.

Once you have a chip, you still need a wafer, a fab, and a package, at every step off standard spec. The process is photolithography, not engraving. If you want aggressive geometry — packing an extraordinary number of transistors per square millimeter — from about the 7-nanometer node onward there are two choices: TSMC in Taiwan or Samsung in Seoul. In 2016 Cerebras was at 16 nanometers. One advantage of not being in your thirties, Feldman says, is relationships. They flew to Taipei, sat with TSMC leadership, and described a chip 56 times larger than anyone had ever done, using the existing process steps, with the ideas already patented.

TSMC is a company with tens of billions in revenue. Cerebras was "30 dudes in a PowerPoint." In the meeting, they greenlit the project. First silicon worked. The manufacturing trick is the reason chips are called chips. A 300-millimeter wafer enters the fab. The stepper flashes cookie-cutter dies across it, then the wafer is usually singulated — cut along those lines, the way leftover dough is lifted away after a cookie cutter. Cerebras asked why you would break Humpty Dumpty and then pay for expensive switching to pretend it was whole. Their patents let them keep TSMC's stepper flow and communicate across the scribe line, the extra dough that is normally cut away.

On video, Feldman holds an Nvidia A100 next to the Cerebras wafer-scale part and calls the GPU a "poor sad little small thing." A chip 56 times bigger, with trillions more transistors, creates a new class of problems: power, cooling, data feed. They invented delivery for both. They were among the first to put water cooling on AI-dedicated processors. Water is an excellent coolant and air a weak one; lower temperatures raise reliability. Hundreds of GPUs' worth of compute in one area could be kept cool and fed with huge amounts of data. That is how they get to the performance numbers.

Labenz wonders about a second convergence: while Cerebras spent seven or eight years on giant chips, the rest of the field built a software stack to manage GPU-cluster complexity. Feldman says the software landscape is both simpler and more complicated than 2015. Then you had Caffe, Theano, rudimentary TensorFlow, and no PyTorch. The world has settled on PyTorch, and Chris Lattner's MLIR gives an intermediate representation everyone can compile to. Models and datasets, meanwhile, have gotten absurd. Dozens of GPUs do not get it done; you need thousands or tens of thousands. Cerebras replaces hundreds, so the next move was clusters of their own machines.

In November 2022 they announced Andromeda: 16 machines, 13.6 million cores. The largest supercomputer on Earth, Feldman notes, has about eight million cores, albeit bigger ones. With that they could move fast. Five or six weeks before this conversation they released seven GPT models trained to the Chinchilla point, Apache 2 licensed, into the open-source community. He positions that against OpenAI keeping GPT-4 closed and Meta putting a restrictive license on Llama. If you want to make money on Cerebras's models, please do. They have worked on clusters past 192 machines; for today, he says, there are not even workloads that need that much, though projects using enormous numbers of these systems will be announced over the next six or eight months.

They do not sell chips. They sell the whole computer, on-premise or in their cloud. Customers sit in Japan, North America, and Europe, in the military, government, large enterprise, and startups. Younger practitioners want the cloud. Big pharma and energy often keep proprietary data on-premise; GlaxoSmithKline and Total are among the names he offers. A developer still writes PyTorch either way. On GPUs you spend a lot of time on distributed compute, and you can still render graphics. On Cerebras you do not think about distribution, you cannot do graphics, and you are likely tens or hundreds of times faster, so research iteration accelerates. Ten people on a GPU cluster each get a sliver and drink coffee waiting. Ten people on one Cerebras machine run sequentially, each job finishing in a tiny fraction of the time.

Will giant chips show up in a Tesla or a phone? No. The edge is storage-light and training is profoundly data-intensive, so the big iron stays in the data center and phones keep small, efficient inference, the way x86 servers and ARM handsets already divide work. Labenz had imagined a memory-rich edge chip doing big-model loops for robots. Feldman splits inference in two. Classification — child, dog, pile of leaves — is fast on a very small chip after a great deal of training, which is why Elon brags about data-center compute. Autoregressive inference, ChatGPT-style, is so hard that Microsoft and OpenAI cap free use. Most of what robots do is classification. Managed inference from Cerebras is "not yet, but it's around the corner."

Favorite products first. Feldman is still awed by ChatGPT, likes Replit for code generation, and uses Jasper for brand voice inside the company. The applications he is proudest of are not consumer. Cerebras was part of Argonne National Laboratory work that won the top paper at the last Supercomputing conference, predicting COVID mutations from viral DNA — work that may underpin vaccines covering more variants. GlaxoSmithKline is doing epigenomics on their machines. A lot of the influence, he thinks, will come from work people never see.

Would he get a Neuralink implant if a million heads were already wearing one and the clinical picture looked good? He knows Elon, and thinks Musk has identified a real problem: input and output. The brain takes in, through the eyes, vastly more data than anyone can deliver to even a big chip — hundreds or thousands of orders of magnitude, in his phrasing. Babies are inundated by looking, touching, tasting. Large language models, he says, show that it is not about parameter count so much as data and data quality. Labenz adds the language bottleneck: it takes a lot of words to describe a picture, while images, smells, and tactile sensation are richer. Feldman calls that an accurate formulation of the I/O problem. He is not sure he would leap in. He is sort of happy with his human limitations.

Labenz Googles 100 trillion synaptic connections against 2.5 trillion transistors and asks why everyone is so sure these substrates feel nothing. Feldman is "confident I have no idea." Underneath is a lot of addition and multiplication. We know extraordinarily complicated things can be built on extremely simple foundations. We might not recognize machine self-awareness if we saw it. Economists used to say humans are rational and self-interested; that is false as a mechanism and useful as a rule of thumb. He expects the same with machines: we will see familiar behaviors, misattribute them to feelings, and the rule of thumb will still be reasonable. There are people wiser than a chip-builder to have on the show for the rest.

Hopes and fears, last. Every powerful technology can be used for grand evil. Facial recognition can persecute minorities and identify terrorists. Compare technologies to actual reality, not an idealized version: humans are terrible drivers, and the goal of self-driving is to kill a lot fewer people. Fifty or seventy years of effort have not cured human bias; first models are biased too, but it is easier to correct an algorithm than a person, and algorithms stay corrected. The web began with pornography and the dark web, and the benefits caught up years later. Thoughtful phishing, scamming, and oppression are real and large. On balance he thinks this technology will fundamentally improve how we live, work, and play, because there is no important technology that cannot be used for evil.