The night Flo Crivello slept five hours: GPT-4 never hit a wall

Nathan Labenz opens The Cognitive Revolution at the end of a dizzying week in AI, voice already failing, with returning guest Flo Crivello, founder and CEO of Lindy AI. The week’s document is Leopold Aschenbrenner’s situational-awareness manuscript: AGI near, superintelligence not far behind, and a U.S.–China race that Leopold hopes America can win by enough of a lead to buy safety time, then offer China a benefit-sharing deal it cannot refuse. Labenz says he cannot find a major hole in that sketch. Frontier-lab leaders are talking about AGI on a two-to-five-year timeline, and the U.S. government is already starting to “feel the AGI.”

He still finds an AI Cold War so scary, and so tragic, that he wants an alternative. Trust between Washington and Beijing looks almost impossible, and he would not want to live in Xi’s China, but he would rather take chances trying to work with the CCP than hope to solve safety in a super-short window under winner-take-all pressure. A Chinese firm has just released a Sora-like video model, which suggests Chinese research is not necessarily far behind. Meanwhile he guesses GPT-5-class systems will still sit in a sweet spot — powerful enough to be useful, still safe enough to deploy, perhaps not safe enough to open-source — and he is already warming to regulation such as California’s SB 1047.

Flo is most of the way through the manuscript and losing sleep over it. It is not the first time he has literally lost sleep over AGI; last night he slept five hours, and he does not recommend reading the piece at bedtime. The argument, he says, is cogently laid out. He does not understand how people fail to freak out. “If you understand what’s going on, you should freak out.” Labenz agrees, then parks a second question — whether the thesis is worth promoting, even apart from its accuracy — and asks to build the case until listeners are at least a little alarmed.

The line people repeat, Labenz says, is that nothing better than GPT-4 has appeared in 18-plus months, so scaling has stalled. Flo’s first rebuttal is historical: people have been saying deep learning was hitting a wall for ten to 20 years; it is a meme. His second is empirical. Models at GPT-4-level performance, sometimes greater, now run at about a hundredth of the cost and 10 to 100 times the speed. Gemini 1.5 Flash, he argues, is greatly under-discussed: cheap, fast, and “insanely good.” Inference wins are not always training wins, but a 100x jump at inference usually means something also moved in the training pipeline.

Context windows tell the same story. Eighteen months ago they were 4,000 tokens; today they are a million. Mamba-style architectures are starting to ship and, from what Flo gathers, working. He has no insider information, but he feels GPT-5 will land in Q3 or Q4 of 2024 and take people by surprise. Labenz offers a concrete job: he asked a model to write a character sketch of him from the last 250 emails he sent, about 250,000 tokens. Anthropic tops out at 200,000. Flash has a million. One call, barely prompted, finished in about 45 seconds and cost under 20 cents — 1 to 2 percent of original GPT-4 prices, and a hundred times the original context window.

On original GPT-4 he would have chunked the mail, summarized summaries, and watched signal die at each rung. GPT-4o’s multimodality is equally under-discussed, mostly because few people have used it. Greg Brockman posted a generated chalkboard that read “probability of text, pixels, audio,” implying one architecture handling those modalities in a shared space instead of a transcription-to-text-to-DALL-E pipeline. Interruptions are smooth enough that the model appears to mix its own audio with the user’s and stop when other sound takes over. Flo calls it the bitter lesson again: do not engineer a threshold; train the thing end to end. The multimodality surprised him. The reasoning did not. He has already moved GPT-4o off most of his workflows.

Flo has updated in two ways, both in Aschenbrenner’s book. First: if AGI does not arrive by 2030 or 2032, it may not arrive by 2040. The next five years are a window of one-time gains. Data will run out, or at least force a solution. Dollars will hit a ceiling around a trillion-dollar training run. Around the model sit the low-hanging “unhobbling” fruits of cognitive architecture. Together he sees three to six orders of magnitude still ahead. Grab those and still miss AGI, and what remains is Moore’s law — slow, on the order of half a magnitude per decade — plus architectural work that is not super fast either.

Second: takeoff speed. He used to be undecided between a day and ten years, and he used to keep AGI and ASI apart. He is now convinced of something like one to four years, and he no longer decouples them. If you have AGI, you have ASI, because you can automate AI research, pick up “1,000 extra bandwidth,” and ride exponential improvement. Labenz has been fighting that conclusion in his own head, partly because he does not want a China arms-race meme to become self-fulfilling. He also thinks the *declaration* of AGI may track OpenAI’s Microsoft contract as much as capability: when the board says AGI is achieved, OpenAI need not hand over the IP, and 2025 is a year they could choose to say it.

John Schulman’s recent podcast, less viral than Aschenbrenner’s essay, sketched the next unhobbling: collect mid- and longer-horizon project-execution data, and teach models to hit obstacles, backtrack, and try another route. Labenz would be shocked if Scale and other partners were not already watching knowledge workers think out loud. He has tried Devin and GitHub workspace. Workspace starts from a GitHub issue and loses non-coders immediately. Devin feels like the future: you state a task, it goes to work, and you can drop a message mid-run; it interrupts, absorbs, re-plans, and continues. It still feels like a coding intern who needs coaching. One more turn, he thinks, and it competes with an employable, non-elite coder. Flo has not used Devin — he watches Lindy agents all day — but he puts AI coding agents at 100 percent in the next few years.

After a sponsor break, Flo names a psychological glitch. Suppose he shows Labenz compelling evidence that someone will come to the house tomorrow and kill the family, then adds that a shark tornado is also on the way. The second claim is so cartoonish that people throw out the first. AI discourse, he thinks, is bundled the same way. Eliezer Yudkowsky-style claims about compiling matter and taking over the universe sit in the same bag as a boring one: the AI engineer is coming, and all knowledge work gets automated in five or ten years. No practitioner disputes that, Flo says, and that is exactly why the public never hears it. Attention has moved on to the sci-fi half.

Risk splits into two categories. AGI-goes-rogue is one; Flo understands the skepticism even though he disagrees. Misuse is the other, and he does not understand how there can be any doubt. People will do evil with extremely powerful technology that is being opened to everyone and “gleefully” open-sourced. Dario Amodei, he says, put the current protection well: the world has evil psychopaths, but the overlap with capable people is small. Education socializes you, and opportunity cost rises — you can be a trader, an engineer, a founder, an investor. When an educated person goes wrong, it is a Unabomber-level story people remember for decades. That protection is disappearing.

The upside is real. A 17-year-old will have the same go-to-market power as Coca-Cola, 10,000 employees in a bedroom, an industrial MrBeast. The corollary is that some people will use the same stack for harm. Bio and cyber are the obvious classes. Civilization already runs on computer systems we barely understand — banking, the grid, the internet. Superhuman hacking is about to be handed to everyone. Flo’s concrete nightmare is waking up to a fog of war: no telecom, no banking, no electricity, transport down, cities starving, and the grid coming back only to be knocked down again for weeks. “AGI can defend us” fails twice, he says. Offense and defense need not rise together, and even if they did, Bank of America and PG&E will adopt slower than hackers in Russia, perhaps by years. Labenz adds a new paper from former guest Daniel Kang’s group: LLM agents finding and exploiting zero-days that sit past the models’ training cutoff.

Labenz does not want a U.S.–China AI arms race, and he has been hunting a story in which either we miss drop-in high-end knowledge workers by the late 2020s or we stall before an intelligence explosion. He cannot find a good one. Weaknesses look unhobbled on a clear path. Nearby, neural tools keep compounding: AlphaFold 3; models that learn particle-level dynamics from quantum-mechanical simulations orders of magnitude faster than the raw sims, then predict crystallization as an emergent phase change; automated cloud labs that run physical experiments through APIs. Even a system that is not beyond Einstein would sit on a parallelizable tool array. Bio data and simulation traces could even fold into the core pretraining mix, alongside text, pixels, and audio.

Asked for a coherent stall story, Flo’s first reflex is data headwinds: we will run out of data, it may be tractable, he could be wrong. He also thinks the question is the wrong one. Climate, nuclear war, and pandemic prep do not wait for certainty beyond a shadow of doubt. Experts converge, expected value says spend. In AI, the most-cited people are ringing the alarm. Deniers used to hide behind a lack of research consensus; that consensus has flipped, and they have not updated. Treat them, he says, the way climate deniers are treated: politely ignore them, and move the conversation to “what now.” California’s bill, whatever one thinks of the specifics, is the right kind of conversation. Labenz’s p(doom) answer is 10 to 90 percent, or 5 to 95: both ends are enough to act.

A trillion-dollar cluster feels to Labenz like an upper bound that assumes nothing clever happens — and security alone would push people to split that plant across sites. Flo’s colder point is that China is not required. Google, Anthropic, and OpenAI have been racing for a long time. Sit out geopolitics and economics still pulls: even $10 trillion to automate all knowledge work is cheap, and markets will find the money. Timeline anchoring works like the shark tornado. A friend once found Flo’s 15-to-20-year worry wild because “people are saying five”; now Flo says five, and still thinks 15 to 20 years is tomorrow. Civilizations already spend on risks that far out. Even a 2035–2040 date is a reason to plan now.

Take China off the board, Labenz thinks, and cooler heads might prevail. Live players number three to seven; they mostly know each other, live nearby, have worked together — Mistral’s team is largely ex-DeepMind. The U.S. government might then be a moderating force instead of a National Security State racing for decisive advantage, solving alignment in months, and offering Beijing a deal. Flo only half agrees. China has private companies too. Government, in his picture, is a coopting force, not a driving one. Baidu or Tencent will race like American labs. The state’s moment comes later, when the DOD notices the thing is powerful and asks the nerds for the keys before it gets out of hand.

On what to do now, Labenz reviews SB 1047: make sure models will not cause mass casualties or hundreds of millions in damage, and if you cannot be reasonably confident, deploy with safeguards — which means not open-sourcing. Today that confidence still exists; the bill is about a future in which it does not. Interpretability, sparse autoencoders, and a Chinese fine-tuning-suppression trick that traps a generator in a local minimum so banned capabilities are harder to tune back in than to train from scratch all look like pieces. None of it feels like a plan. Flo would chain himself to a gate today if there were one, and the gate would be Meta’s.

Step one, he says, is to stop open-sourcing the models, period. The benefits are small relative to the cost, because we do not know what is inside. Microsoft’s MedPrompt paper found that GPT-4, with a laboriously engineered prompt and no medical fine-tune, beat models specialized for medical use. These objects are a dense jungle; years of hacking still leave GPT-4 opaque. A small tweak can jump capabilities. You may already have released an AGI-capable or super-hacker model and not know it, six months from someone finding the right prompt. Fine-tuning can be offered through an API, as OpenAI does. Auditing weights is not possible in the way people imagine; the thing that *is* trivial is stripping safeguards. Hugging Face is full of uncensored families. That is step one.

Step two is a Manhattan Project for alignment: tens of billions of dollars, kitchen-sink spending, perhaps a mandate that every dollar or FLOP of training is matched by a dollar or FLOP of alignment. He does not expect it. If the California bill cannot pass, larger bills will not. He forecasts an AI 9/11, and prays it is not too bad; seriousness arrives after. Labenz asks how interpretability researchers keep access to advanced models if weights never leave secure servers. Flo’s analogy is the original Manhattan Project: vetted people, classified secrets, structured access. Mission drift is the fear — a government project that becomes “stay ahead of China.” Flo does not take the turn lightly. He is a libertarian who hates the state. MIRI spent ten or 15 years looking for an alignment path that did not require regulation, then gave up. Two longtime libertarians arguing for the last resort is, in this conversation, the signal.

Reeled back to the present, Flo describes himself as a humble founder of B2B SaaS. In Aschenbrenner’s language, Lindy is unhobbling: layers around the model, not the model itself, useful this quarter and still aligned with the march of the systems. The bet is that those layers make GPT-3 roughly as capable as GPT-4, and GPT-4 roughly as capable as GPT-5. Long-term memory and RAG retrieve by situation. Continuous learning uses in-context learning sealed by the user: give Lindy a task, give feedback, watch it ask for feedback, and let it improve. Tool use, computer use, and UI use sit on the same stack. He buckets the company as three jobs: the agent’s black box minus the LLM (memory, planning, a critic, verification, recursive goal decomposition), the inputs (text, images, audio — Lindy can join meetings), and the outputs (APIs today; talking in Zoom, placing calls, and using a computer later).

Labenz calls himself an adoption accelerationist and a hyperscaling pauser, and someone has challenged whether that is coherent. Fine-tunes and scaffolding unlock real value; he does not want six more orders of raw scale. OpenAI’s work with Harvey — continued pretraining plus post-training on a GPT-4 base, apparently well under 10 percent extra FLOPs — produces a custom model people strongly prefer, probably worse at everything else. A hundred narrow specialists sound like surplus without the risk of another magnitude jump. Zuckerberg’s TikTok-cycle comment still nags: scaffolding shows you what to bake into the next model, then the scaffolding dies. Flo thinks most of what Lindy builds survives a five-year horizon, maybe 80 percent rather than 100. Continuous-learning RAG looks obsolete in an infinite-context world until you price it: killing RAG with a forever-growing window is “literally 100 times more expensive.” The death of RAG, he says, has been greatly exaggerated.

He reaches for Android versus iOS. Android on garbage-collected Java was supposed not to matter because computers would be infinite; for years the phone was sluggish while iOS stayed butter-smooth, a 100-versus-500-millisecond gap people actually felt. Two-to-five-x, or in the RAG case 100-to-2,000-x, is the kind of gap Lindy is willing to live on. Labenz still expects to advocate for a pause after GPT-5: one more generation in the sweet spot, then dial the doctor and the lawyer into reality instead of racing magnitudes. Flo calls it brinksmanship, and borrows an Eliezer image: laundry detergent that produces gold faster the more you pile on, until it blows up the Earth. He is comfortable with two or three more orders of magnitude, not six or ten. The danger is that temptation never runs out. “Just one more order of magnitude, bro.” Regulatory gears are slow, and GPT-5 is coming regardless, so the conversation about the generation after next has to happen now.

On OpenAI’s leadership, Labenz has ridden a winding road: clueless, then surprisingly planned, then again unsure, with repeated exoduses that look a little like a purge, even as Sam Altman still seems to engage the ideas Labenz cares about. Flo reaches for Uber. Hyperscaling chaos is hard to imagine from outside; Uber was adding about 800 people a month. OpenAI turned a quasi-obscure research lab into a company generating billions of dollars in 18 months. Of course Altman is a ruthless businessman. The Super Alignment pledge to spend 20 percent of compute was, Flo thinks, meant when it was announced; then chaos got in the way. That is the point: these firms are structurally incapable of self-regulating. They need an external forcing function, and Altman has if anything invited it. Flo does not think the safety team was purged, except perhaps Ilya over the coup. They left because OpenAI did not make good on the promise. Leadership worries about safety and still, rationally, puts FLOPs on capability.

Who still matters among foundation-model builders? Flo thinks the field is set: Anthropic, OpenAI, Google, Meta. Mistral came close and does not put itself in that category. The Mamba people surprised him and could matter, though the incumbents can replicate that research and surely have models in training. Asked whether big tech will metastasize into every industry — Labenz’s “big tech singularity,” prompted by a Google shipping-planning API that claims to double a container line’s profits — Flo is unimpressed. Google is deliberate about where it expands. Search was always an AI problem; Android exists so Google can be the default engine. There is no chance the company turns itself into a logistics firm. Google Fiber was the expansionist test, and they pulled out. He reads the API as a sign of how fast AI is entering every cranny of the economy, not as Google becoming a shipping company.

Labenz still worries about a Facebook-ads pattern: a platform so good at allocating demand that it pins every seller’s CAC just under margin and takes the rest. If Google prices a profit-doubling API at current profits, shippers become serfs under an implied cutoff threat. Flo grants the monopoly logic and denies the premise. Network effects made Facebook dominant; AI will not have one winner, the labs hate each other, and a price war is already crashing token costs. He used to fear becoming an OpenAI serf. Now he has Gemini, Llama, Claude, and Mistral in the running. One shipping API that doubles profits can take half the industry; two race to the bottom and the shippers keep more. Flexport, he jokes, is going to do it. Any last thought? “AGI is coming and it is time to freak out.” Labenz signs off hoping, against hope, that an AGI or ASI arms race is not the natural attractor it currently looks like.