The 100x lives between the layers: Dylan Patel on co-design and a 60x year

That 60x figure comes from Dylan Patel, founder of SemiAnalysis, sitting with Sequoia’s Shaun Maguire and Sonya Huang in his own office for an episode of Training Data. The firm now has about 90 people and, through a project called InferenceX, runs automated inference tests every day across roughly 15 chip types. Rumors put SemiAnalysis past $100 million of annual revenue. Patel’s reply is a shrug: “It’s as accurate as the information is. You never know.”

Five years ago, Maguire notes, semiconductors were not sexy in the West even as they remained so in the East. Patel went long anyway, building what Maguire calls probably the premier research shop in the space, spanning process details, supply chain, and the bigger picture. There are also rumors of a venture fund. Patel does not confirm one. He is more interested in the curve his team now publishes in public: cost and power against interactivity, refreshed as the software under the models moves.

The conversation that follows runs from how he prices a living benchmark, to why Google will pay $11 an hour to rent GPUs from xAI even though it builds TPUs, to how much of 2040’s incremental compute he expects in orbit. The through-line is accounting. Patel treats chips, models, and dollars as one problem, and he has been doing that since long before anyone paid him for it.

Patel did not grow up inside Silicon Valley. His parents ran a motel; the family lived in it, then ran a gas station across the street. He jokes that the first neural network he trained was watching who walked in and sliding a stepstool under the right cigarettes before they asked — Camels for one demographic, menthols for another — because he was too short to reach the rack and, strictly speaking, too young to sell them. Waiting for the order meant an extra trip with the stool. Being ready meant a cleaner sale.

Hardware arrived as a broken present. He asked for an Xbox 360 for Christmas after the April announcement, got it, then watched it die of the famous red ring of death just as a cousin from Alabama — another motel kid — was coming for spring break. He opened the box, shorted the temperature sensor after every gentler trick failed, and the machine came back. Pandora’s box, he says, was open. By twelve he was moderating Android, Apple, and PC-building forums, reading Intel, NVIDIA, and AMD threads the way other kids read sports, always with an economic tinge from the family business. When the internet loved AMD on price-performance, he would still argue NVIDIA: smaller die, better performance, better watts, better margin. He also played a lot of StarCraft II and, at one point, sat grandmaster on the North American ladder.

School was degrees that had nothing to do with semiconductors, then two years as a quant at a small risk firm. He says he made the company well over $10 million of risk-free revenue by exploiting a market glitch; someone else took the credit, and even after the bonus was later right-sized, the social contract was gone. In early 2020 his grandmother, who had lived with the family in the motel, developed dementia, forgot who he was, fell down stairs, and died. COVID, a brother in Nashville, and a doxxing of his anonymous account piled on. He froze for three weeks, then posted two real essays under his own name on his 24th birthday. SemiAnalysis was a blog, not a newsletter, and the traction was immediate because the work was no longer shitposting.

What followed is stranger. He bought a truck, a bed-mounted tent, and an air mattress, and spent months circling U.S. national parks — weekdays in motels negotiated down to about $30 a night, weekends with semiconductor textbooks on a trail, all of it pre-Starlink. He has not had a fixed address since mid-2020. He now flies to 40-plus conferences a year, from 20,000-person NeurIPS to a 300-person Japanese chemical meeting where almost no one spoke English. SPIE lithography talks were 90 percent incomprehensible the first year, half the second, 75 percent the third, and still not finished. The day before this taping he learned of a chemical with $100 million of sales that every chip requires and that he had never heard of. The office culture matches the forum years: engineers yell that a technology is the coolest thing on earth, ex-fund people ask what it costs, and Patel’s management line is “you don’t wrestle with a pig because the pig enjoys it.” His first real industry relationship was also an internet fight. In 2020 he baited Naveen Rao, later of MosaicML, from a public thread into DMs and a call. Rao’s new company is trying to innovate on silicon, software abstractions, and models at once, including analog compute and energy-based models. Patel thinks it will not work quickly. Sequoia invested anyway.

Sonya Huang asks whether inference becomes the biggest market on Earth, and beyond it. Patel’s version is slightly different: the use of tokens, and the value those tokens create, will be the biggest market, much bigger than oil, and many percentage points of GDP. Open or closed, inference is the thing. What he built to measure it is InferenceX, which he now treats as an industry standard, and it exists because point-in-time reports were dead on arrival.

Models land every week — Chinese labs, and on the day of the taping, Mythos 5 and Fable. PyTorch, vLLM, and SGLang update on the order of twice a week, plus new drivers and new decoding tricks. Performance therefore changes continuously, which is why equivalent-quality cost has fallen about 60x in a year. A quarterly PDF cannot keep up. InferenceX is a living run: CoreWeave, Crusoe, Nebius, Oracle, Microsoft, Amazon, Google, and OpenAI donate compute; the SGLang and vLLM teams, plus newer private forks, help tune; NVIDIA, AMD, Google, and Amazon are in the collaboration, with TPUs and Trainium being added. More than $50 million of hardware is already donated. With TPU and Trainium it should clear $100 million. About 15 chip types run every day, covering the best open models from Moonshot, Alibaba, and several other Chinese labs plus U.S. open weights such as GPT-OSAS and Nemotron. Results and configurations are public.

The artifact that matters is a Pareto curve of throughput against interactivity. Patel’s complaint about older bake-offs is that vendors compared their own optimum to someone else’s off-peak point — “if I drove a Porsche versus some race car driver, obviously I’d drive it slower.” InferenceX publishes open-source containers at the optimal points along the curve, so anyone can download near-peak configs. One end is a batch of 100 users at 10 tokens per second each, a thousand tokens a second off one slice of hardware, cheap and slow. The other is a single user at 250 to 500 tokens per second, roughly 4x faster and 4x more expensive. Batch document jobs want the cheap end and can wait overnight. Claude Code’s fast mode, OpenAI’s priority queue, and SemiAnalysis’s own usage sit at the expensive end, because the person in the loop is costly and a faster feedback loop is worth the markup. Today’s AI infrastructure is still mostly one-size-fits-all. Patel expects the curve to split into different products.

Cerebras, in this frame, is an innovative company with a real market in very fast tokens — SemiAnalysis almost only uses fast mode — but SRAM architectures like Cerebras and Groq struggle when the model is huge and the context is long. A 10-trillion-parameter model with a million-token context, he says, does not obviously fit. Labs still mint most of their dollars on the best model, and people migrated in volume to Fable and Mythos even though those tiers cost more. Asked about token volume versus dollar volume, Patel is blunt: who cares about tokens. Nobody brags about Camry unit sales when the F-150 prints the profit.

Maguire offers a split of the last three years: most of the intelligence-per-watt gains from hardware, some from models, and a question about leftover juice in kernels. Patel’s answer is immediate. “Shaun, I completely disagree with you.” Hopper to Blackwell is about a 30x on the most optimized DeepSeek deployment, which is real and visible on InferenceX, but the model layer moved more. Three years ago the bar was GPT-4. Now a small open Qwen-class model — about 27 billion parameters total, two billion active — is far better. The hardware gain is sizable. The model gain is larger. The prize is the layer in between.

DeepSeek is the public exhibit. Every expert shape in V3 was cut for Hopper; V4’s shapes are cut for Blackwell and Huawei’s chip. TPUs are, in Patel’s words, objectively an amazing chip: they run DeepMind and do a great deal of Anthropic’s pre-training. They still “suck at running DeepSeek,” while shining on other models that NVIDIA does not love. Matrix-multiply unit size, attention structure, expert shapes, network I/O, and collectives are fused with the silicon. Pull a co-designed model onto last generation’s hardware and it is “really not that great.” Gemini 3 was tuned for TPU v6e; the next Gemini is tuned for v7. Anthropic trains heavily on TPUs and infers a lot on Trainium and GPUs. OpenAI’s earlier models leaned Hopper; newer ones lean Blackwell. Optimize each layer by 2x and you have 8x. Co-design them and, he says, you can get 100x.

Maguire asks whether China has simply done this better. Patel does not fully buy it. The West, he says, just does not publish. GPT-4o never disclosed its sparsity or parameter shapes, yet it was roughly the same size as DeepSeek V3 — slightly smaller — and shipped a bit earlier. What looks like a unique trick is often a trick someone else has not peeled open. That same frame revises the CUDA story. Huang observes that in the last three to six months, frontier labs stopped treating custom kernels for a second chip as a religious objection: Claude and Codex write a lot of that work, and there are on the order of ten model companies that need deep programmability, not ten thousand. Patel agrees halfway. Coding models do commoditize software, so the CUDA software moat is at least partly unbundled. What people still call CUDA lock-in is mostly that DeepSeek, Kimi, Zhipu, Alibaba, Tencent, and Xiaomi built GPU-shaped models. Run them on TPUs and they often do not run well. Inference APIs and companies doing RL customization of open weights are stuck buying NVIDIA for that reason, not because they love writing kernels.

The reverse is available to Google if it ever open-sources truly strong models that hate GPUs; then users would rent TPUs. Gemma exists. Whether Google will bet the ecosystem on it is another question. Small teams should keep using vLLM, SGLang, and PyTorch. The big labs already do not: OpenAI forked PyTorch long ago. Their move is to pick the cheapest capable hardware and co-design model plus infra all the way down, with AI writing the software. Patel refuses to crown NVIDIA or TPU. Two years out he can imagine Google making 10-plus million TPUs a year and NVIDIA many tens of millions of GPUs, both of them $100-billion-plus businesses — NVIDIA larger still — and he is careful to say this is not a revenue forecast. NVLink with switches ties 72 GPUs; Google’s ICI has no switch and can connect 8,000 chips at very high bandwidth if traffic can hop through neighbors. OpenAI’s models are much more sparse; Anthropic’s are still sparse but denser. Which fabric is “better” depends on the model, and the model changes every year. His end-state bet is that everyone will build an ASIC and everyone will keep a bucket of general-purpose compute, because labs do not even know next year’s architecture. Specialized chips race toward a local minimum. Google already has three TPU design programs — one with Broadcom, one with MediaTek, and a third architecture he will not describe — which is itself an admission that the minimum can move.

Asked for the bottlenecks he is watching, Patel starts with memory as technology, not as a supply-chain shortage. The NAND cell is about 25 years old. The DRAM cell is about 40. Nothing fundamental has replaced either. The last five years of HBM have mostly been more stacks, run faster. What he finds actually interesting in the next few years is stacking memory directly on the compute die so bandwidth explodes. There are companies and proofs of concept in that direction. Capacity and bandwidth have otherwise improved slowly, which is why every lab is still fighting arithmetic intensity and communication.

The second bottleneck is power density. For at least two decades, you could glance at a data-center or desktop chip and guess its watts: about one watt per square millimeter. A 100 mm² chip was roughly 100 watts. Flagship NVIDIA and TPU silicon still sit on that line even as packages hit 1,400 watts; Rubin is headed for 2,000, Rubin Ultra around 4,000, almost entirely by pouring more silicon area. The development work that excites him is pumping more than one watt into each square millimeter. Thermal and electrical interference make that a hard problem, which is why the industry stayed at one. If it lands, you need less silicon for the same compute.

On intelligence per watt, he first jokes that a 30-year-old TI-84 calculator already beats a human brain at arithmetic. For general intelligence, InferenceX tracks power as well as cost against interactivity. Cost at a given benchmark level is down about 60x annually; intelligence per watt is closer to 40x, because some of the savings are not power. The curve has not stopped. Versus the brain, chips are still many orders of magnitude behind, and Patel is glad it does not matter much: it is easier to feed a computer than a person who gets sick, has food preferences, and sleeps. Energy as a “wall” gets a similarly earthy rebuttal. The United States can make millions of truck diesel engines; convert them on the line into generator sets, park them beside a data center, and staff the site with people pulled out of auto shops. Ugly, he admits, and not something he could do himself — but not an immovable law of physics.

This year’s new data-center compute is about 20 gigawatts even after delays; next year, more than 30. That is not enough to close the gap Patel cares about. From Opus 4.5 to Fable and Mythos, the world’s installed compute did not double in seven or eight months, while the set of valuable tasks a model can do jumped by several times. Anthropic, in his telling, can pay up because the unit economics work: Q2 already net-income profitable excluding stock-based compensation, Q3 possibly profitable even including it, and Opus 4.8 API tokens north of 80 percent gross margin. Rent extra GPUs, TPUs, or Trainium at a painful rate, sell tokens immediately, and if you started at 75 percent gross margin and doubled compute cost you still have 50 percent. NOI still goes up.

Price is the honest signal. Google has TPUs and still rents GPUs from xAI at $11 an hour. Trainium leased to Anthropic and OpenAI has shaken out under $10 billion per gigawatt-year; GPUs, before the latest squeeze, usually ran $12 to $13 billion; Google’s rental of SpaceX GPUs hit $25 million per megawatt-year, or $25 billion per gigawatt. Colocation that used to clear at $60 per kilowatt per month now transacts at $120 to $160, $200 when the tenant’s credit is weak, and about $80 in India because the grid and the network are shaky. Quality diverges too. Google will put 1.5 gigawatts of hardware in a 1-gigawatt building and slosh load to keep the feed full; it will also take 2 gigawatts from a utility that can only guarantee 1, with three days a year of curtailed power covered by batteries and on-site generation. CoreWeave’s GPU clusters, on SemiAnalysis’s tests, beat Amazon, Google, and Microsoft on performance and reliability. Google still sells capacity six months before the hall exists and takes that paper to the debt market to pay the purchase order. SpaceX sells machines that are already running. A balance sheet, Patel says, is how you raise revenue per megawatt.

Huang worries about leverage after Crusoe said a customer had halted a build. Patel needles her: high leverage and high growth is a small equity stub with huge upside, and she is an equity investor, not a lender. The tide turns only if model-created economic work stops outrunning new compute. Over the last six months it has not. He will not call the loop recursive self-improvement, but models are already writing infrastructure and pulling the next release forward, a “pseudo-recursive” cycle. Capital is the constraint that still scares him. Google sits on a large SpaceX stake — at one point on the order of 10 percent from Larry Page’s early check, still a position Patel ballparks around 5 percent — with something like $100 billion it could sell after lockup, plus enormous gross profit, and it still did an offering. Meta announced a raise and the stock tanked. Every large buyer, he thinks, will keep raising debt or equity until model progress actually stops.

Neoclouds exist, in his 2023 “Amazon Cloud Crisis” telling, because the hyperscalers’ old strengths became liabilities. Amazon’s Nitro NICs, custom SSDs, and Graviton cores were brilliant for CPU tenancy and cost per core. Nobody in AI rents a single GPU in an eight-GPU server or a 72-GPU rack; they take the whole rack, for years, so isolation hardware mostly hurts performance. Microsoft’s data-center org was fine at predictable build rates and fell over when the forecast doubled, then had to buy neocloud capacity. Incentives finish the story. Nobody at a hyperscaler gets rich for delivering six months early. Crusoe’s team is hyper-levered equity: late means they do not get rich. Jensen Huang, Patel argues, is not a bystander. He “absolutely hates a world where all the hyperscalers have all the power,” so he bankrolls random labs, tells the world to invest, and likes Chinese labs, because a three-lab, three-cloud oligopoly is a world in which NVIDIA is screwed. A GPU sold to Crusoe and a GPU sold to Google are the same price today. Five years on, Crusoe still being alive means TPU and Trainium are weaker. Throw bait in the water; the best fish live. Thinking Machines’ Tinker, a product less than six months old with a few hundred million dollars of ARR, is the kind of messy success he thinks Huang is fishing for. Space data centers, for the record, stay under 1 percent of compute through 2030 in Patel’s view, even as OpenAI and Anthropic alone exceed 100 gigawatts that year; by 2040 the industry is in terawatts and more than half of incremental compute may be in orbit.

Maguire asks which semi talking point makes Patel lose his temper. Memory as a bottleneck is true and therefore boring. The one that actually infuriates him is “AI has no ROI,” which he hears as a denial that models are getting better. People point at a benchmark stuck at 90 percent and call a plateau. That benchmark is saturated; look at the new ones. Capabilities, he says, have been up and to the right the entire time. He has been studying this industry for 18 years, since the forum days, and he is still learning a $100-million chemical the day before a podcast. Knowledge in semiconductors now depreciates faster than most people can restock it.

On a ten-year view he is excited about space, asteroid mining, and the SpaceX vision — “not investment advice” — and treats co-packaged optics as a when, not an if: sometime between 2027 and 2030, everyone already knows it arrives. The bets that thrill him are the ones that try to move several layers at once, the Naveen Rao shape of company, even if they will not work on a two-year clock. Maguire closes by noting the decade of unglamorous work underneath the last few years of being in the right place. Patel thanks him, and the accounting lesson left on the table is practical rather than cosmic.

If you run open models, go look at InferenceX before you decide the model is slow; the gap between a random config and the Pareto front is often several times. Put the job on the interactivity curve: overnight document batches should not pay 4x for latency nobody feels, and a coding agent whose human is expensive should price fast mode as a labor save. When you evaluate a stack, ask whether this model and this chip were co-designed for each other. Moving lab A’s weights onto lab B’s silicon and watching performance collapse is the normal case, not a scandal.