A 10B Model Trained Across Three Continents — and an H200 for $1.49 an Hour

Nathan Labenz has been looking for the right guests on distributed training for a while, and he opens with a complaint that applies far beyond this startup: a positive vision of the future is currently the scarcest resource in AI, including among frontier-lab leaders. Vincent Weisser and Johannes Hagemann, founders of Prime Intellect, say their mission is to make compute and intelligence "too cheap to meter" and widely accessible. Zoom forward a few years, Vincent argues, and the ugliest outcome is not that the systems fail to work. It is that superpowerful intelligence exists and is available only to a few states, tech giants, and individuals. Done right, the same technology could put agents, better medicine, and better education in the hands of every creator; done wrong, the intelligence age becomes harder to enter, not easier.

On timelines, Vincent puts high probability on the pictures drawn by Sam Altman, Dario Amodei, Demis Hassabis, and Leopold Aschenbrenner, and he notes that Kurzweil- and Bostrom-era forecasts have held up better than their reputations. Superhuman coding, math, and even autonomous software organizations may arrive well before reliable humanoid robots. A 95% robust self-driving car is close to useless, and an investment agent that blows up one percent of the time cannot be trusted with anyone's capital. The only ten-year bottleneck he takes seriously is computational irreducibility — Stephen Wolfram's point that not everything can be simulated. Curing all disease is much harder than writing a lot of code, even if surprisingly large parts of reality, including biology, can be approximated.

Nathan pushes on daily life: if this works, is anyone still working a job, and would even Bill Gates hand malaria eradication to a superintelligence? Vincent points to financially independent people who keep working, and to Nick Bostrom's *Deep Utopia* on meaning after material problems are solved. Human labor would still command a premium in restaurants, theaters, and childcare, where people want a trusted person rather than a robot. Ordinary knowledge work becomes curation and direction of an army of agents working for free. The best founders already hire people smarter than themselves on every dimension; "execution is cheap, ideas are worth everything," which he thinks inverts today's reality and could produce billions of startups, movies, and autonomous creations with humans still setting the aims.

After the break, Nathan asks whether they keep a p(doom). Vincent calls the concept close to a mind virus: he has watched too many people say "20 percent" because a famous person said 20 percent, then fail to unfold any causal chain. Putting sharp numbers on unknowable billion-year scenarios, he argues, has already licensed policies with the opposite of their intended effect. Nathan's own range is 5 to 95 percent, or 10 to 90 — nobody has talked him out of worrying, and nobody has talked him into the beach. Vincent is more alarmed by the second-order effects of the worry itself, including calls for a one-world government and well-intentioned overregulation that puts people on autopilot under the state.

The failure mode he actually names is full centralization of superintelligence by a few nation-states and big-tech labs, disenfranchising large parts of humanity. He says he knows behind-the-scenes stories from AGI labs that worry him, and that some people who fought for alignment and policy have been co-opted by labs with their own power-seeking goals. Nathan analogizes concentration to purifying a plant into a drug: the leaf is fine; huffing the 100 percent extract is how you get into trouble. He is sympathetic to a world where everyone rises at once and checks everyone else, even while he cannot yet see a stable equilibrium if AI is changing everything, everywhere, all at once.

Prime Intellect is a Delaware C-corporation with an Ethereum-like design: a foundation, grants, fully open source, and a long-term public utility anyone can fork, improve, and build agents on. Step one is a global compute market with an API and CLI, so a user can request one H100 or a thousand and find the cheapest place to run. Step two is software for distributed training and synthetic data generation, so idle GPUs anywhere can join a fault-tolerant run and drive down the cost of intelligence. Step three is collaboratively training leading open models that contributors get paid to improve — they mention continued work on DeepSeek's R1 as one target. Step four, still ahead, is a permissionless peer-to-peer protocol for compute and intelligence sitting on top of that market.

A year earlier, H100s were so scarce that the labs who paid extra got priority, and startups signed two- or three-year contracts worth hundreds of millions just to lock a thousand GPUs; on-demand rental barely existed. Production caught up, Blackwell began to ship, and on Prime Intellect you can now rent a thousand H100s on demand — something impossible three to six months before the taping — while an H200 lists at $1.49 an hour. Vincent's analogy is precise: they do not buy hundreds of millions in GPUs, so they are not a hotel. They are trying to be Airbnb, a marketplace sitting on top of other marketplaces, from hyperscalers and independent data centers to billionaires who overbought H100s and startups parking unused reservations. NVIDIA, he argues, keeps the supply scattered on purpose: Google and Amazon also make competing chips, so allocations flow to Lambda Labs, CoreWeave, and other smaller, more aligned clouds across hundreds of sites.

Nathan asks where this sits between a fenced trillion-dollar data center and a Bitcoin-like network no government can kill. Johannes answers that even Stargate will not be one cluster: energy limits you to multiple roughly 500-megawatt sites over the next few years, not a 10-gigawatt campus, and the big labs already train across data centers on 40 to 100 Gbps private links. INTELLECT-1 already trained across Europe and Asia on the public internet. Vincent, citing physicist David Deutsch, calls compute regulation as naive and dangerous as regulating math or code. It pushes capacity toward Malaysia, Singapore, and other lightly regulated, cheap-power regions — the same script Bitcoin followed when China banned mining and hash rate moved to the United States, Iceland, and parts of Africa.

Both founders left Europe, and Vincent treats the EU AI Act as a historic own-goal: zero practical safety benefit, a dead weight on small startups, and an AI community that "didn't even have a pulse" and is now gone. He groups California's SB 1047 with it. Nathan had come around to the final version — publish a safety plan, accept scrutiny — as reasonably light-touch, but Vincent still wants closed models under *more* scrutiny than open ones, because an API endpoint is a black box you cannot inspect. Open weights, like the internet and blockchains, can be jailbroken, patched, and battle-tested in public. Nathan's reservation, which the episode never really answers, is that you cannot take an open model back, and DeepSeek's R1 already showed that midsize open models can be post-trained into much stronger reasoners.

Nathan calls Metagene-1 one of the best defense-favoring open-source projects he has seen. With the Nucleic Acid Observatory, SecureBio, and researchers at UCL, Prime Intellect helped train a metagenomic foundation model for early pandemic detection in wastewater — the same channel that first flagged COVID. Vincent says it took on the order of $20,000 to $30,000 of compute to reach a frontier detection model, with more distributed training planned as data grows. The category he wants multiplied is high-upside, low-misuse science models; AlphaFold is the template, and even a hard-nosed utilitarian ledger, he argues, says society is undersupplying them.

The design choice Nathan highlights is a 512-token context window, far shorter than any viral genome, so the model cannot emit a full pathogen. It can read a fragment of a wastewater sample and flag anomalies — myopic by construction, in the sense of doing one local job extremely well. Nathan's phrase is "unilateral provision of a global public good": people who need the detector get everything, and people who want to abuse it find no entry point, because the limitation is baked into the artifact. Vincent frames the broader strategy as differential acceleration of defenses — detection, monitoring, and vaccines in place before the next outbreak, which he argues society lacked during COVID.

Next on that track are denser pathogen monitoring at city and airport scale, a virtual-cell foundation model, and an autonomous scientist with humans in the loop. Vincent's blunt biosafety claim is that COVID most likely came from nation-state gain-of-function research gone wrong, not from an intelligent system; the missing piece was antibodies and defenses. He wants open, widely used scientific systems because, in his view, the only realistic path both to aligned superintelligence and to pandemic-proof science is many participants with guardrails, not a single closed lab. Nathan is less sure the same "cannot be abused" property will survive once the artifact is a general-purpose scientist rather than a 512-token detector.

Johannes walks through why distributed training is hard even inside one cluster. Data parallelism copies the full model, trains on different batches, and syncs gradients — the least communication. Tensor parallelism slices weights and communicates every layer, so it is the hungriest for bandwidth. Pipeline parallelism ships activations between stages of layers, sitting in between. The trap is easy to miss: a gradient is the same size as the model, so a 100-billion-parameter step produces 100 billion numbers per replica. Training also stores optimizer state — full-precision copies, momentum, and variance — which is why INTELLECT-1's per-node memory ceiling was about 10 billion parameters.

Their workhorse is DeepMind's DiLoCo, for distributed low-communication training. Each island of GPUs trains locally for hundreds of steps, then syncs. What travels is a *pseudo-gradient*: the difference between weights at the start and end of the local run, not the true per-step gradient, and that delta turns out to compress unusually well. DeepMind showed the method at 400 million parameters; Prime Intellect pushed it to 1 billion, then to 10 billion with INTELLECT-1, without a centralized baseline because a matched run would have been too expensive for a small startup. Johannes is careful to call this an empirical result in the lineage of local SGD and federated learning, not a theorem.

Counterintuitively, DiLoCo is weaker at the start of training and catches up later. Early on, every replica's gradient points the same way, so merging adds little signal; later, on a flatter loss surface, the disagreement is the information. For INTELLECT-1 they sent those pseudo-gradients in 8-bit instead of 32-bit (a 4× cut) and synced every 100 steps (another 100×), a 400× reduction — enough to train 10 billion parameters over global internet links. Three limits remain: 100-billion-scale still needs more quantization; newcomers need at least an A100 or H100 node; and the method was efficient to about 16 workers before extra replicas overfit in different directions and drown the aggregated signal.

Nathan hopes sequence-level mixture-of-experts routing could split science from literature, or ship an open model with a "virologist" expert simply removed. Johannes is not bullish. Routing by human domains is a worse inductive bias than letting token-level MoE learn its own routes, which empirically are not semantic. DeepSeek V3 activates about 37 billion of 671 billion parameters per forward pass, roughly 6 percent, but that does not yield an interpretable safety switch. Nathan finds the miss unfortunate; it would have been an elegant way to square open weights with dangerous knowledge.

Vincent's punchline is that the R1-style inference-time compute paradigm is almost coincidentally perfect for a decentralized setting. Pretraining backpropagates every token. Reinforcement learning samples many rollouts and updates only on the especially good and especially bad ones, so communication falls off a cliff. Chain-of-thought is readable text, which you can inspect, edit, and load with safety checks — unlike the hidden traces of o1 and o3 — even if the model sometimes switches into Chinese or emits fragments that do not map onto human reasoning. Johannes notes you can generate those reasoning traces on H100 and H200 nodes, and even distilled small models can help produce the synthetic data.

DeepSeek's R1 paper, unusually for that lab, left out infrastructure details: how long they trained, what the interconnect looked like, how many generations per problem. In the broader literature Johannes cites, a typical pattern is hundreds of rollouts — on the order of 256 questions — and then one backward pass, which can mean minutes of forward compute per update, or hours if the rollout phase is long. Nathan adds that the ratio is not fixed. If items are so hard you succeed one time in a thousand, the reinforcement signal is too sparse. Both the R1 and Kimi papers describe algorithms that keep problem difficulty in the band where the model still learns without sampling a thousand times for one success.

Other wide-area ideas include SWARM parallelism, from Max Ryabinin, now at Together AI: pipeline-parallel stages over the WAN, syncing only the last activation. Latency kills it. A hundred milliseconds is nothing if you all-reduce every hundred steps, and everything if you ship tiny tensors constantly, and million-token sequences for o1-style models make activations grow with context. Johannes is more interested in offloading AdamW's optimizer state — full-precision parameter copies, momentum, and variance — to CPU memory, which can put a 100-billion-parameter model on a single H100 node; 650-billion-parameter DeepSeek-scale still will not fit. After this conversation was recorded, Google released Streaming DiLoCo, evidence they have not hit a wall. Anthropic's Jack Clark wrote in *Import AI* that we may soon live in "a world of models trained continuously in the Invisible Global compute sea."

Fault tolerance on GPUs is still primitive. Train across 100,000 chips and a node dies every few hours; many frameworks crash the whole job and restart from the last checkpoint. Prime Intellect open-sourced a framework called Prime that keeps data-parallel ranks alive when workers drop and lets nodes join mid-run — engineering they learned the hard way when INTELLECT-1 stopped for hours on unanticipated edge cases. A post by Yi Tay after leaving Google for Reka described a 100× spread in how reliable different GPU vendors were. Johannes expects GPUs to become more CPU-like commodities as software and hardware catch up. They are not there yet, and a globally distributed run needs more than vanilla checkpoint-and-pray.

Vincent does not think the path to AGI on this stack requires a trillion-parameter dense model. R1 suggests reinforcement and synthetic data might get there with fewer active parameters, and every frontier lab is already trying to train across sites. They need it, he notes dryly, to sprinkle data centers through enough congressional districts to get the buildout funded. Microsoft is spending about $80 billion a year on data centers; OpenAI has floated $100 billion-plus commitments; governments, he says, are building six- and seven-figure GPU clusters that never show up in blogs. TikTok at one point put more than 600,000 GPUs onto the market when it did not need them. Compute, in this picture, shows up wherever energy is cheap and regulation is light — including, in the limit, floating sites on wave power.

Nathan notes NVIDIA at $3.6 trillion versus AMD at $200 billion, down a third over the prior year, an 18× gap, and asks if hardware abstraction will crush those margins. Vincent's read is that two things can be true: the market is exploding toward trillions, so NVIDIA, TSMC, ASML, and AMD can all become historically valuable even as NVIDIA's near-total claim on AI profits fades. Margins may compress while revenue grows 10 to 100×. AMD is a clear number two that fumbled software; Google TPUs and custom silicon at Amazon and Apple are the more serious long-term competition, because those firms can force their own adoption. Specialized transformer ASICs still sit near zero share and will gain some. Prime Intellect's software is meant to help commoditize the stack until, in a perfectly efficient economy, returns compress toward the cost of capital.

They are in touch with Emad Mostaque's Intelligent Internet effort — similar vibe, different path. Prime Intellect is obsessed with peer-to-peer compute and with making decentralized training actually work, shipping in small open increments. INTELLECT-1 contributors so far gave compute in kind, including Hugging Face and Dylan Patel of SemiAnalysis. The end state they describe is a tokenized, permissionless public utility, governed by a nonprofit foundation, that Vitalik Buterin cannot halt any more than he can halt Ethereum. In the intelligence age, Vincent wants people to own a piece of a superintelligent system that produces value — a harder asset, in his view, than fiat or even Bitcoin — so superintelligence does not get lost to platform monopolies the way the late internet did.