At 100,000 Accelerators, Something Fails Every Hour

Sonya Huang opens Training Data by placing Amin Vahdat in the middle of what she calls the biggest CapEx buildout in human history. Google alone is expected to spend more than $200 billion this year, most of it on data centers. Vahdat was named head of Google's AI infrastructure at the end of last year. He does not argue with the scale. He has never seen anything like it at Google, or, he thinks, in the history of humanity, for the pace of the transformation. The number he then puts on the table sets the tone for the rest of the hour: stitch 100,000 accelerators into a synchronous job and something will fail multiple times a day, and in some configurations multiple times an hour.

Outsiders see an arms race in FLOPS and handsome photographs of fiber. The people who run these halls live in a long tail of faults, in power, and in a tree of optics. Each chip, Vahdat says, is a wonder of nature at the bleeding edge of what can be manufactured, and it is not even one chip: packages of two, four, eight or more chiplets, plus HBM, network links, maybe co-packaged optics. That is not a criticism. It is a count of the places something can break. At 100,000 of them, something will.

An AI data center and a conventional one share more than people assume: concrete, an enclosure, electrical and mechanical yards, cooling, row after row of distributed power, a huge network stitching compute together, and a fair amount of storage. The difference is the planning horizon. A classic hall is a 25-to-30-year building investment whose servers last about six years, so the design has to swallow four or five generations of mixed hardware. You might put networking, storage, and some GPUs or TPUs in the same envelope. The building is supposed to stay generic.

An AI hall is often purpose-built, co-designed with the hardware that will go in it, sometimes to the point of saying this building will not hold much storage. A storage rack might draw 10 to 40 kilowatts. Set that next to a CPU or GPU rack already in the hundreds of kilowatts, with people talking about megawatts in the next two years, and the architecture splits. Thirty storage racks in a row versus one or two AI racks is a different size, a different power plan, and a different network. Storage, especially spinning disk, needs tiny bandwidth compared with an AI rack. Make the building fully fungible over 30 years and you will probably make it too big and overbuilt. The AI choice is to sacrifice that generality for deep co-optimization of cooling, power distribution, and networking.

Huang mentions a Vera Rubin cluster Google delivered to Ineffable Intelligence, a Sequoia portfolio company, and the photographs of fiber leaving the racks. Vahdat agrees it is a thing of beauty, at least for people in the trade. The cabling has a fractal quality that made one of Google's most popular social posts, more for the picture of the fiber than for the customer, though he is careful to praise the Ineffable team. Beauty is in the eye of the beholder. For this audience, the racks are a monument.

How do you hold yourself accountable in a buildout this large? Huang had heard Vahdat call FLOPS a vanity metric. Chip-centric numbers, he says, are theoretical: under some conditions, this is the maximum a chip can deliver. What they actually care about is performance delivered by a workload. That is rarely decided by a single chip's FLOPS, HBM, or SRAM. It is how two, eight, a thousand, ten thousand or more chips compose, plus the CPUs that feed them data and the network that ties them together. FLOPS utilization is one window onto that. The word Google uses is goodput.

Throughput is work you are doing. Goodput is whether that work is delivering an answer. Training, serving, and agentic jobs are often synchronous: many components must coordinate at microsecond and millisecond grain. One of a hundred thousand stops and the whole job may stop, because everyone is counting on everyone else. Then you have to find which unit failed, restore a checkpoint, restart. Worst case, especially on the inference side, you start from the beginning. Time spent failing, recovering, and waiting to diagnose is still work. It is not helping you finish. Vahdat's paper-and-pencil analogy: if you err at step three and go back to step one, you are still writing. The time to the actual answer is what matters.

If there were a common failure reason they would have fixed it. What remains is a long tail of constant discovery. A new product at the bleeding edge will hit you, often in the high-speed interconnect, sometimes in hardware. A lot of the pain is software: a compiler bug, a runtime bug, a model issue, an operating system issue. Perfect, fully reliable silicon still loses if the stack above it is wrong. The telemetry problem is finding a needle in a haystack continuously, across seconds and minutes, and for some jobs across hours, days, even weeks. Nvidia, he notes, is an incredible whole-systems company and ships a strong reference stack; most customers use it, and many specialize. TPUs have a reference stack too. Most people lean on it. Many will change it.

Huang asks whether Vahdat has really told his teams that Google must roughly double serving capacity every six months. He clarifies: the target is effective available capacity, which on the serving side means token-generation capability. That is not the same as doubling FLOPS. Hardware has some inherent flop rate. Doubling what that hardware can emit in tokens can come as much or more from software, a model optimization here, a runtime change there, dozens or hundreds of small landings stacked on top of one another. The rate of capacity improvement, he says, is incredible.

She wants an empirical split: silicon versus models versus other software, maybe as intelligence per watt. He does not have an exact breakdown. In Google's experience, most of the gains in intelligence per watt come from the model side and the software system, because they make the hardware you already have get used. Hardware is still stunning. Year-over-year performance improvement of 2x or more is absolutely possible, a multiplier, not free, that lifts everything above it. Google's internal version of the metric is goodput per watt. Watts belong in the denominator. That sentence is the setup for the rest of the power discussion.

Google started custom silicon in 2013, against the grain. Conventional wisdom said you do not build a dedicated accelerator for a single workload: Moore's Law is still doubling performance every 18 or 24 months, and you get to keep C++, Java, Python, the standard programming models. Huang calls it the bitter lesson of chips; Vahdat agrees, specialization never wins, except that a handful of applications would have taken an unimaginable amount of general-purpose CPU. Even inside the company, plenty of people were not sure. The bet worked. The first chip was inference, mainly translation and speech. The second was training. Around the time that second chip arrived, transformers were invented, which he calls a major moment that completely shifted the program. Recommenders ran well, ads followed, and the GenAI wave pushed the systems larger still. Scope and impact kept expanding.

Two years ago the question was whether 2026 should have one chip or two. A single part could do inference and training quite well. Two parts could specialize. The analysis produced TPU 8i for inference and TPU 8t for training. Inference and serving were taking off; a chip significantly faster for serving, covering perhaps 30 to 60 percent of the market over its life, started to make sense. If inference had stayed 2 or 5 percent, even a 2x specialized chip might have lost to one uniform general part. The more you specialize, the faster and more efficient the hardware, and the less flexible. If the workload vanishes in a month or three, even a huge spike is too narrow a window to intercept. It has to be durable, and you have to be able to project what specialization will buy.

A detail that is easy to miss: 8i and 8t can each do the other's job. If one had zero training performance and the other zero inference, Google would have to forecast the mix, exactly, over a six-year hardware life. Leftover capacity on either side can still pick up the slack. How far you specialize and how much fungibility you keep are the same problem. Huang asks the provocative follow-up: if the market is transformers, why not burn the transformer into the chip? Matrix multiplies, vector ops, and softmax, the linear-algebra primitives, are already roughly baked in. The next level is your particular model: how many layers, how you walk them, the exact shapes. Some companies are thinking about that. He finds it interesting. Customers, he adds, do not treat TPU and GPU as fully interchangeable. GPUs are more general-purpose; Google Cloud sells a lot of them and uses them internally. The company wants to give people a choice and then the option that actually fits.

The case for co-design is impedance mismatch. If you abstract every layer so the job can run on any cloud, any hardware, any topology, you can move overnight when capacity appears. You also leave performance on the table: 10 percent, 20 percent, maybe 2x at each boundary, multiplying into a large end-to-end gap in intelligence per watt. The case against is lock-in. Huang asks whether that is why OpenAI and Anthropic reportedly landed on different model architectures; Vahdat will not speculate. What he will describe is Google. Working shoulder to shoulder with DeepMind is, he says, one of the most fun and gratifying parts of the job. Consumer services and Cloud extend the same loop, but DeepMind is the deep partnership.

Hardware runs a five- or six-stage pipeline spanning years: parts in production, parts back from the manufacturer and in bring-up, parts about to tape out, parts in design, parts in concept. On the production end, the two teams squeeze delivered goodput per watt together because they can see the model and the silicon and everything between. On the eve of tape-out is the interesting part. DeepMind may say that hardware support for a math op would make end-to-end training or serving much faster. Engineers and researchers pack into a room for a few days, a week, two weeks. The usual answer is they cannot quite do what was asked, but they can do this other thing, and if the model moves a little they get 98 percent, or 90. Sometimes they delay tape-out a week or two because the benefit is worth it. Doing that across a company boundary is possible and much harder. Vahdat talks to Koray and Demis multiple times a week. They are using Gemini to design hardware for future Geminis.

Hardware plans two to five years out; model people do not, by default, think on that clock. Google Research, DeepMind, and the invention of the transformer grew up in overlapping rooms, so a generation of researchers already knows they can influence silicon and that silicon does not ship on a two-week changelist. A 0.5 percent tweak on a chip about to tape out will not bring them running. A real opportunity will, and both sides will try to get it in. Predicting the most common workload five years out sounds impossible. Vahdat's counter is that at a medium level of detail, the TPU architecture has not really changed since v1. A CPU instruction set is still loads, stores, adds, branches; the software on top turned over many times. TPU primitives are huge matrix-multiply units, a SparseCore for vectors and scatter-gather, remote load-store over the ICI network. Those fundamentals have carried many generations of models and even of neural-net structure. They are writing that story up in detail.

The biggest workload change of the past year, Huang suggests, is the long-horizon agent, and Vahdat agrees. Two effects dominate. First, the loop is no longer human-to-accelerator. You type in a browser, wait, read, think, follow up, a gap of seconds to tens of seconds. An agent has no person in the loop to rate-limit it. The response comes back, it parses, reasons a bit, and fires the next request, and interaction time drops into milliseconds. Second, that parsing and reasoning probably runs on a CPU, which then has to decide what context to fetch before the next prompt: local DRAM, someone else's DRAM, SSD, or HDD farther away. A huge amount of orchestration appears. Demand for accelerated compute is going up. Demand for traditional CPU, networking, and storage is going through the roof at the same time.

Does that mean more CPU racks beside the GPUs and TPUs? That is the specialization question again. Mix the halls and you cannot take TPU density and network to the extreme; a TPU rack is denser and hungrier for bandwidth than a CPU rack, and the building design changes. Keep the fleet uniform: TPUs in one building, CPUs next door, hard drives in a third. Now you need serious networking between buildings, and once you leave a hall the reliability and cost of the network jump. Latency moves from microseconds toward hundreds of microseconds, plus queuing. Huang says the job is hard. Vahdat says it is fun.

Google has been a networking shop for a long time. Fifteen or sixteen years ago it was among the first to bring wavelength-division multiplexing inside the data center, multiple signals on one fiber, and it uses that for all rack-to-rack communication. At the same time it introduced optical circuit switching. A packet switch takes a packet into the electrical domain, reads the header, looks up a port, and forwards, billions or trillions of times a second. An optical circuit switch does not touch the bits. It decides which input port's light should hit which output port. The first implementation used MEMS: micro-electrical motors steering 3D mirrors in a box of 128 or 256 ports, each incoming fiber reflected to a programmed destination, the angle changeable in software.

They did it for two reasons. One was locality: a search compute cluster and its storage cluster talked a lot, so the mirrors built a purely optical shortcut between those groups of racks. The other was to expand or shrink the network spine without a human moving fiber, a controller rewriting the topology. In the TPU era the same kit got a new job. TPUs interconnect in a torus. When a rack fails, light can be redirected to a spare without moving any fiber, in milliseconds. That is goodput engineering, not a networking flex. Huang asks why keep fiber at all, why not free-space optics everywhere. Attenuation and bandwidth would drop dramatically, Vahdat says, and aiming across a very large building in 3D without fiber is challenging to possibly impossible. They have talked about it. It is still mostly fiber, until the light hits the tiny chips in the circuit switch.

Asked for the biggest constraint, Vahdat refuses a single winner. They are all hard, and they shift. Fundamentally, though, power is the one. Everything else looks solvable given time. Power is the binding, long-term issue. Abundant clean energy, nuclear perhaps, would solve a lot; when it arrives at scale is still unknown. In practice Google's preferred model is always to be utility-connected. You do not call PG&E and ask for a gigawatt tomorrow. You give years of notice and plan together. He is pointed about a responsibility that is easy to skip: when the utility builds or upgrades transmission and substations for Google, Google covers those costs so they do not land on other customers' rates.

Timing mismatches are real. Suppose you need a gigawatt in 2028 and the utility can deliver a gigawatt in 2029 and 700 megawatts in 2028. You can wait, or you can generate some of the missing 300 megawatts yourself, solar, batteries as backup, other sources, and perhaps keep that local plant even after the full gigawatt arrives so you can push power back to the grid. The hottest two weeks of the year, when residential demand spikes, are a moment to give rather than take. Full self-build is not the preference. To hold a gigawatt at 99.99 percent-plus reliability you would build two gigawatts, one-plus-one redundancy, expensive, ideally clean, and close to the hall. Partnering with the grid is statistical multiplexing over a larger base: you give when they need it, you take when you need it, everybody wins, residences included. Behind-the-meter generation happens in rare cases, and even then under the assumption that a grid hookup is coming.

How big should a data center be? An art, and a source of significant debate. Ten or fifteen years ago Google argued about putting everything in one gigawatt building, which then felt like the edge of the imaginable. Single point of failure is an obvious worry on a 30-year clock. Training workloads pull the other way: bigger is better, network distance as short as possible. Power availability pulls back again. There is no site in the country or the world that can take Google's total requirement. So the answer is layered. Edge or in-country sites might be tens of megawatts; a partnership with an ISP might be a single rack; a training cluster might approach a gigawatt; other sites hundreds of megawatts. They have models and simulators. The optimum still depends.

Huang's lifecycle guess is partly right: newest, biggest clusters train the frontier model, older ones recycle into inference. Demand for inference is too large to live only on retired training clusters. Training also tends, in a given year, to concentrate on one continent or even one portion of a continent, which leaves other continents short of serving capacity, so you have to build serving clusters anyway. Those are not necessarily cheaper per megawatt. Serving wants compute, network, and storage close together, which fights specialization. Training is a big, dense, uniform deployment. Serving is a mix, spread across the planet, with multiple versions of a model, so the clusters are smaller and less vertically integrated.

A line of Vahdat's that traveled widely was that seven- and eight-year-old TPUs are still at 100 percent utilization. He did not mean it as a slogan. Older TPUs, and GPUs too, still see significant use. They do get replaced. Depreciation is about six years; newer generations are more power-efficient, so once they are fully depreciated an upgrade makes sense. What you pull is not a chip but a system, a pod of perhaps 9,600 chips and 152 racks, swapped for whatever TPU 12 or 13 or 14 looks like. The new footprint will not perfectly match the hole. They cannot plan that many generations ahead, so decom and refill is hard, physical work, done as fast as they can.

Google developed JAX, likes it, uses it extensively, and could have said that if you want TPUs you use JAX. Many customers like PyTorch. So there is TorchTPU, and unmodified models can run. Vahdat's analogy is IP in the late 1970s and early 1980s. Competing protocols lost because IP was an open standard, the narrow waist of the hourglass: any software on top, any hardware underneath, anyone who brought IP could plug into a router port and join the internet. That is why the network exploded. You can plug in something highly specialized if you want, or something you think is better. Google still wants open standards, and ideally open source around them. That is also why it supports CPUs, GPUs, and other accelerators. The buildout, Huang says, is too big to be any one vendor's walled garden. He agrees. It has to be a choice.

AI is already changing the hardware job, not only the software one. Software engineering's use of the tools is well documented. Hardware engineers, by the token counts he looked at that day, are using AI as much as the software engineers. Time from design kickoff to tape-out is shrinking. Bring-up is shrinking. Data-center design has changed too. Gigawatt building versus 100- or 200-megawatt campus used to be a detailed, spreadsheet-driven, human process. It still is, in part, but a lot of AI now gathers the information and puts it in front of the people who decide. Not quite reasoning models replacing judgment. Humans stay in the loop. And yes, they are using Gemini to design the hardware future Geminis will run on.

Huang asks whether people running the numbers on orbital compute means Earth is truly bound. Google is pursuing it, Vahdat says, and they have called it a moonshot with no tongue in cheek. The physics is why. In space you get about 40 percent more solar power because there is no atmospheric attenuation, a 1.4x. In sun-synchronous orbit the panels see sunlight 98 to 100 percent of the time, against 28 to 35 percent on land, another 3 to 4x in hours. Multiply those and you can more or less drop batteries from the equation, still carbon-free. That is a rare solution that works at the level of energy constraint. The troubles are equally clear. Cooling in space is harder, not easier. Reliability is worse because repair is harder; perhaps you send a robot. Redundancy becomes your friend. Free-space optics stops being a debate and becomes a requirement, lasers on receivers, calibrated in real time. He sees no fundamental showstopper.

Ten years is right at the edge of the cone of uncertainty. Some people are already thinking about the 2036 machine; the error bars are enormous. The trend is integration. The VR200 racks for Ineffable were beautiful with fiber; 2036 will be more integrated, less fiber. He will not say none, but perhaps a small bundle leaving the rack. Racks will be manufactured centrally, 72 or 144 or 288 accelerators, probably 576 or 1,152, deeply integrated. A single rack at multiple megawatts. You wheel it in, plug water, power, and fiber, three lines, and you are off to the races, like an appliance. Huang asks if it will be a big alien orb in space. By 2036 he would not rule out launching the rack, having a station's robotic arm catch it, and plugging it into the right module. Fun to think about. They get to live through this, he says at the close, and they get to define it.