Compute Scaled a Millionfold. Bandwidth Only 40x. Fractile Wants to Fix That Ratio

Walter Goodwin founded Fractile in London in the summer of 2022. The company now has about 150 people, and it builds very fast inference chips for the world's largest models. Speed was the bet from day one. Foundation models trained on the internet were arriving, and a few people were already arguing that more compute had to be poured in at test time—the AlphaGo pattern, a capable network that becomes superhuman when you roll it out at scale, brought to language. The four-year pursuit has been chips that run huge models much faster than today's hardware, in a way that still scales beyond the current frontier and to extraordinarily long contexts.

On No Priors, host Sarah Guo asks where that puts Fractile in the GPU and accelerator landscape. Goodwin's sharpest framing comes later, but it is the thesis of the episode. Workloads move on a software clock. Frontier labs put out a new model about every two weeks. In chips, he says, the equivalent of a frontier model is structural speed: if you can carve out a three-to-six-month advantage, you will be winning all of those deployments.

Goodwin calls today's menu of AI ASICs an extraordinary zoo. Look across it and many of the chips are relatively identikit. That is a structural industry property, especially among hyperscaler efforts. Google kicked this off with the TPU more than ten years ago. Meta has MTIA, Microsoft has Maia, OpenAI now has Jalapeno. Those proprietary designs are developed and delivered with a relatively small number of ASIC houses—backend delivery partners of which Broadcom, a $2 trillion company, is the largest.

The seeming embarrassment of riches hides a lot of similarity. All of these chips use HBM, the same DRAM that sits on NVIDIA and AMD GPUs. They make the same kinds of bets on tensor cores for matrix multiplications. They use the same advanced packaging at TSMC. What is still scarce, he says, are efforts that go all the way up and down the silicon stack to build fundamentally new capabilities. Not many teams have decided to own architecture, front-end design, physical design, process technology, and the foundry relationships that have you flying back and forth to Taiwan or Korea every week. That last mile still sits with a relatively small number of companies.

Guo asks about the common handoff. If you are Google designing a TPU, some people on the team understand the workload well enough to invent something like a tensor core: a designated circuit that is exceptionally good at matrix multiplies, because those dominate the flop count. Architects then hand a circuit-level description to front-end designers. Mechanistically that still looks like writing code. The code goes to a player like Broadcom, which synthesizes it, places it, and turns it into GDS2—a bitmap that tells TSMC where every metal layer and transistor goes. Analog IP for chip-to-chip connectivity and the physical placement specific to a three-nanometer or five-nanometer node are, today, generally outsourced.

Fractile pulled that whole chain in-house. Goodwin reaches for a management cliché that he thinks actually fits: a lot of industries replace a waterfall with something more agile. The team spans deep workload understanding—sometimes running ahead to ask where they would change a model architecture to match their bet, then advocating that to customers—plus front-end designers, physical design, back-end implementation, and advanced packaging. Headcount is not enormous. About 150 people, skinny in every sector, but a closed loop. In AI chips more than any prior chip play, you are forever chasing the tail of the workload. You have to place bets correctly, with skill and luck, and then strike fast. A handoff to a partner leaves you at the mercy of how that partner behaves.

The most important technical bets, Goodwin says, started with direction. Four and a half years ago the company was already an inference story, but he spent the next two years educating the world about what inference meant. Some inference chips on the market, including one that recently went public, were training chips until about 18 months ago. People avoided the idea because "marginal cost" sounds small. What it actually means is the cost you pay every time you deploy. Fractile bet on a deployment era and on speed, and that informed the architecture.

For the first two years they worked, a little like Groq or Cerebras, on an SRAM-based chip. SRAM is super-high-bandwidth memory that lives on the same piece of silicon as the logic, so weights and KV cache can move extremely fast. That can drive thousands of tokens per second on language models. Toward the end of 2023 and through 2024, he started to worry about scalability. Two things grow in AI: parameters, and context length. Context length was the one that made them nervous. SRAM capacity is small. Long context does not fit.

The team turned toward memory vendors and logic foundry partners, in what he calls skunkworks projects, to get extremely high bandwidth to higher-capacity DRAM. The result is a platform he says will ramp in the second half of next year. It is meant to unite the scalable, lower-cost DRAM you get on a GPU or TPU with the speed advantages of a Groq or Cerebras chip. Later in the conversation he puts a number on the new capability: 25 times more bandwidth per chip than an HBM-based chip.

Why that much bandwidth? Inference means loading weights and state many thousands of times per second; bandwidth is speed. At data-center scale, serving thousands of users, the economics collapse to cost per gigabyte of memory. Today's fast-inference chips have super-high bandwidth and incredibly low capacity. In practice they are not running long-context attention; they still flip back to a GPU for that. That is the tantalizing mismatch. Guo asks where speed actually moves the dial. Goodwin reaches for Henry Ford: if you had asked people what they wanted, they would have said faster horses. A snappier chatbot is the faster horse of fast inference. The elevator on capability is taking a multi-trillion-parameter model and running very long-running agents at many thousands of tokens per second.

A chip company, Goodwin says, has to span a duality. You must be exceptionally good at the workloads coming down the pipe under the hardware lottery: models trained on HBM GPUs or XPUs, today's autoregressive MoE transformers, run faster and more efficiently. New capabilities also let you pull the model landscape toward you. With 25 times more bandwidth per chip than HBM, Fractile has been exploring what he calls scaling laws for bandwidth. The usual scaling laws are flop scaling laws: more flops at train time and test time, better performance. Some bandwidth laws are already visible in the known landscape of ideas.

Mixture-of-experts is the first. For iso-intelligence, you save a ton of flops by going from 1-in-16 sparse to 1-in-128 or 1-in-256. The headwind is serving. On today's HBM GPUs and XPUs those models become prohibitive: bandwidth-bottlenecked, really low MFU, ugly flop utilization. Model architecture wants to go sparser; memory bandwidth is dragging it. Attention is similar. Some forms are less bandwidth-hungry and more flop-hungry for a given level of intelligence. When bandwidth is no longer the bottleneck, you can pick the cheaper-flop version.

He puts the imbalance in one pair of numbers. Flops have scaled about a millionfold in the last 20 years. Memory bandwidth has gone up about 40x in the same period. Lift the bandwidth dimension and you conserve the other: fewer flops for the same intelligence, which is a multiplying factor on global throughput. That is also why he is excited about hard problems. Once inference is faster, a model can think a lot more before each experiment and buy fewer wall-clock trials with reasoning tokens. Compute gets cheaper; wall-clock time gets more expensive. The exchange keeps tilting.

What you want at any moment, Goodwin says, is a chip that precisely targets the workload you care about, in your hand, in volume. Six months later you will want something else. Models arrive every two weeks. Sanity comes from what is common: new LLMs desperately want orders of magnitude more memory bandwidth; they tend to be autoregressive and low-batch, with a trade-off between throughput cost and speed. Open-source Chinese models churn the exact attention mechanism every couple of weeks, plus MoE sparsity and sparsity on attention itself. Bigger shocks are possible. Physical and financial reality still apply brakes. Fab cycle times are three to five months even on a super-hot lot. Once the chip is back, you need a three-to-five-year amortization window for it to be a sensible financial decision. Data-center power, install latency, and financing the silicon do not go away.

He believes two slightly conflicting things at once. Compressing the overall design cycle is hugely valuable: more shots on goal, forever ready to declare a flagship and ramp it. He does not believe we end up shipping a fundamentally new architecture every few weeks. A 12-to-18-month volume ramp still needs longevity. The value is in shortening the gap between an observation and realizing that bet in volume. Elad Gil had asked a top-three semiconductor CEO—off the air—how soon chief-architect intent becomes a fully usable GDS2 file, and heard ten years. Goodwin's heuristic for the past couple of years: question the logical assumption, divide the timescale by four, then subtract a bit. End-to-end prototyping in the next few years, yes. A new chip every few weeks, no.

Amdahl's law is why he insists on a full-stack team. If AI crushes 12 months of front-end design and physical design, tape-out, and verification stay on the old clock, the chain is stuck. Place-and-route still runs for days as conventional algorithms on NP-hard problems. Fuzzy approximate placement could help teams iterate; final sign-off will still go through Cadence and Synopsys DRC and LVS checks, because those rules were ground out with TSMC and other foundries over decades. A guest on Dwarkesh Patel's podcast had put the same logic in extreme form: think for a hundred years before you fire the experiment, then another hundred years on the results, because the experiment in the middle is expensive and takes wall-clock time. Stronger intelligence makes it almost a moral compulsion, Goodwin says, to think harder before every trial you launch.

Guo asks what a large AI player or hyperscaler buys in five years: NVIDIA and AMD, the new accelerator class, or the internal efforts at least four of these players already have. Goodwin's first layer is supply. Everyone deploying at scale is trying to stand up as many separate platforms as they can. Compute is existential, so diversity of supply stays a need. There is a joke that first-party efforts exist primarily to reduce the price people pay NVIDIA. He thinks there is some truth to it. Those chips are architecturally quite similar to GPUs. The bet is not a capability the other chips cannot enable. It is gamesmanship, aggregate capacity, and a bit of control: build some here, buy some from AMD, maybe get a better NVIDIA price.

Chips that actually give new capabilities are a different market. Anyone who wants to deploy at the frontier, he argues, needs some way to run these models orders of magnitude faster. Frontier labs live in a window of premium intelligence; otherwise everyone would just use Kimi models. Pressure from open source makes the rest of the stack—the best weights and the fastest deployment, so you can do the most reasoning in the shortest time—more important, not less. Speed-over-everything becomes a real segment. Fractile tries to look like a frontier lab's own team: people who think deeply about workloads, and who will tell a partner how to change the model architecture to fit the chip.

Why does that not just roll up inside the labs? Asymmetric risk. Suppose lab one goes all-in on proprietary silicon. Lab two finds a computational breakthrough that is far more efficient at the same intelligence, but it only runs on the chip lab two chose. Lab one could be dead in the nine months before it can deploy enough of a matching chip to catch a 5x efficiency jump. Frontier players need to deploy the same platforms as one another. They are playing a different game from a chip vendor. Going all-in on a deeply differentiated hardware bet is, in his words, an irrational and very dangerous move for anybody playing at the frontier.

He is honest about the gap to NVIDIA. Crack open an NVIDIA system and you find six to nine custom chips, all built by NVIDIA, working together. Fractile is trying to build a single chip. The ambition is to become productive enough to field its own responses in that same way: at any moment, a rolling frontier of aligned bets, ready to trigger a ramp. That machine is the three-to-six-month wedge. Build it, and you keep winning the deployments.

The operational test Goodwin keeps returning to is economic, not peak tokens. High-bandwidth, low-capacity chips look fast until long context shows up and the job flips back to a GPU for attention. At data-center scale the bill collapses to cost per gigabyte of the memory you actually employ. Evaluate the memory cost first, then the latency. A 25x bandwidth story only holds if capacity is cheap enough to keep the whole sequence on the fast path.

The same logic applies on the model side. MoE sparsity and the form of attention are hardware co-variables, not just architecture choices. 1-in-128 or 1-in-256 sparsity is ugly to serve on HBM; a different bandwidth structure might flip that. For any long-chain engineering project, his divide-by-four rule is the schedule check: question the assumption, then cut the estimate. The window he cares about is not a year. If you can structurally take three to six months off the path from observation to volume, you win the deployment—whether you are a chip company or a team shipping an AI product.