There Likely Is a Level of Intelligence We Shouldn't Go Past

Nathan Labenz opened the week with a line that stopped him cold. A senior executive at a frontier lab had said, under Chatham House rules, that “there likely is a level of intelligence that we just shouldn't go past.” Labenz asked how you would even operationalize that — a cap on FLOPs in the next pretraining run, maybe no more than 10 to the 27. He expected a polite demolition. What he got was, essentially, yes, that could be reasonable.

The setting was The Curve, held at Lighthaven in Berkeley: the rare room where frontier-lab insiders and the people who criticize them sit together. Labenz was there last weekend taking notes for Cognitive Revolution’s AI in the AM. His cohost Prakash Narayanan wanted the divides. Two years of extra evidence, Labenz said, have closed far less of the outside world’s disagreement than you would have guessed. Inside that room, though, one thing had come together.

People have, to a significant degree, accepted that the systems are going to get really powerful. This is not something you can hope goes away, or wait out until a bubble bursts and a fever breaks. Even the most modest people in the room now talk as if the models will do an awful lot. The live capabilities argument has moved on: whether recursive self-improvement fizzles, runs fast and then levels off, or actually runs away. The old “long timelines” crowd, as Helen Toner once put it, has gotten very short.

The sharpest split Labenz saw was not safety versus acceleration. It was inside the frontier companies versus everyone else. The people who live in the future, he said, have visibility the rest of us do not. Founders, executives, and top researchers kept telling him they came because they wanted to share a candid internal picture — not trade secrets, but outlook, plans, and where constraints might actually be possible. Several of the things they said struck him as news.

One senior executive was emphatic that pretraining continues to deliver. Base capability is still rising, with no obvious stopping point. Scaling laws are holding, or even bending the helpful way, because data quality is getting better and architecture tricks keep arriving. Synthetic data, in this telling, is really working: you spend tokens to transform existing data, convert reasoning traces and test-time compute back into pretraining fuel, and feed it to the next generation. That loop is already a form of recursive self-improvement, and the emphasis was on data quality more than on a magic new architecture.

The same person went further. He believes the latest frontier models already have the ineffable research taste you would need for paradigm-level breakthroughs. The problem is elicitation: reinforcement learning is not bringing that taste out reliably, which is why you might need 10,000 agents on a Millennium Prize problem — maybe a million — before something paradigmatic stumbles into view. Then came the sentence Labenz had never heard from a frontier-lab leader: there likely is a level of intelligence we should not go past. It was not minced, and it was not a fully specified number. It was still the strongest talk of a hard cap, even a temporary one, that he had heard from that kind of seat.

Narayanan immediately saw the implication. Intelligence is rising from more compute and from better-refined data; cap the intelligence and you are capping those inputs, which is another way of saying the compute buildout has an end. Labenz followed up with a concrete straw man — no more than 10^27 FLOPs in the next pretrain — and got almost no fight. The details, he warned, would decide whether a cap was real or a shell game. FLOPs spent enriching synthetic data are easy to hide. Still, the sentiment in the room was that something like a limit might have to happen.

What about Elon Musk and Mark Zuckerberg? The takeaway from the event, not one individual, was that two or three companies are really pushing the frontier — Gemini 4 being announced, Labenz said, might make it a three-horse race again. The other two names people usually put in a top five are distilling a lot, whether they know it or not. They are not calling a Claude API for traces. They are buying RL environments from a cottage industry that has Claude build those environments, then running their own internal RL on a world that only exists because Claude was smart enough to make it. That pipeline keeps leaking frontier gains to everyone who follows.

One founder-executive type put it this way: the chasers have kept a persistent following distance, which is not what Anthropic’s old fundraising deck predicted. That deck said that around 2026, the companies training the best models would be so far ahead that nobody could catch up. It has not happened. The synthesis in the room was darker for the laggards than it sounds. If the frontier labs had not released models, and if direct and indirect distillation — including that RL-environment trade — were not available, the prediction probably would have come true.

Narayanan’s read on Musk is that he is gambling on infrastructure over models, structuring for 2029 through 2031: TeraFab, SpaceX satellites, a 2030s clock. Anthropic and OpenAI, in this telling, are planning around 2028 or 2029 AGI and recursive self-improvement. Labenz said the vibe in the room was even sooner. The short-timelines crowd includes the executives, founders, and top researchers of those companies. They talked about next year. The critical period is beginning now; decisions over the coming months, and how compute is used over the next year, could be really critical. There was not even much 2028 talk, which he called pretty crazy.

A cap on pretraining would not, in Labenz’s view, end the need for compute. He put Jensen Huang’s line to an executive: NVIDIA spends about 20 percent of its effort designing a chip and 80 percent verifying, validating, and making it last. The reply was that this was kind of reasonable. A majority of future compute could go into safety — chain-of-thought monitoring, which people are getting bearish on, and internal activation monitoring, which already takes a significant share and may go up a lot. Labs are also spending heavily to make models hack their own RL environments, then patching or throwing out the broken ones. Cleaner environments seem to produce less cheating downstream, a proto-scaling law they would love to drive toward zero. They expect to be bottlenecked on safety and alignment, with enough low-hanging fruit for the next generation or two, and then, past that, all bets are off.

Labenz tied the elicitation claim to a new record in the NanoGPT speedrun, the race to train a small model to a target loss as fast as possible on a wall clock. A company called Hyperstition took the classic benchmark from something like 70 seconds and cut almost half the time off — more, he thought, than the previous 45 accepted improvements combined. The interesting part was the author’s note: AI did not play a big part. Core insight was attributed to the human, which is exactly the elicitation story the frontier executive had told. Hyperstition also flagged that the optimizer they published, which accounted for a significant but not majority share of the savings, is worse than the one they use internally.

On science, the middle position circulating among frontier companies struck Labenz as the credible one. The models will not be superhuman at everything by default. They will be superhuman at the things labs care about enough to license data, augment it, synthesize versions, and put RL on top. That package, people there believed, works on essentially any problem they choose to focus on. AI for science is next. One person went as far as to say that in no more than a year, it will not be possible to do the very best science without AIs playing a big role.

The next morning he added a political takeaway he had not aired the day before. The old assumption that Congress will never act is slipping. The new vibe, especially if Democrats take both chambers after the midterms, is that you might even see Republicans join, because in their heart of hearts they want to do something and have not wanted to go against the president while data-center money is still flowing. That alignment could come up for renegotiation. Policy response, Labenz argued, is now part of what people should be watching — not a story you can dismiss as permanent stasis.

Thomas Somers cofounded Positron, which builds inference chips around memory and recently raised $875 million. A quote for memory from a year and a week earlier, he said, has gone up four and a half times. He hopes it stops there, but he would not be surprised by another doubling next year. The industry has absorbed the hit because the capability you get from a given number of gigabytes is way more than five times what it was a year ago. His founding bet was that scaling laws would not quit, that models would keep eating memory, and that the real limiter on applications is context per user — which, as agents multiply, is exploding. He has gone from two to four agents running in the background to 15 to 20.

That is why Positron built on commodity LPDDR, the same class of memory in phones and laptops, instead of betting the company on scarce high-end parts. NVIDIA GPUs, in his numbers, realize only 30 to 40 percent of theoretical memory bandwidth in the transformer decode pass: a B300 advertised at 8 TB/s is more like 2 TB/s in practice, because the hardware was shaped around training reuse that transformers do not offer. Positron’s first-generation product sustains 93 percent, so theoretical and realized bandwidth almost meet. With Credo Semiconductor they decoupled the memory chip and pulled LPDDR5X up to 72 channels, against a typical ceiling of 12 to 16. The next chip, Asimov, holds up to 2.3 TB per device — eight times the B300’s 288 GB. Analysts, he said, have NVIDIA’s Rubin Ultra cutting capacity to 192 GB because of cost. Fitting on one chip what used to take eight GPUs also deletes the all-gather and all-reduce tax of sharding.

AI has already changed how the chips get made. Starting from scratch, Positron’s first full-custom Asimov is running closer to 60 percent design and 40 percent verification, not Jensen’s 20/80. The leap arrived with GPT-6 Astra and Opus 5.5. Agents now get the keys to Cadence Palladium emulators, racks of custom ASICs for gate-level silicon emulation. Somers doubts Palladium’s docs were in pretraining; the traces show the models reading PDFs, compressing them into markdown cheat sheets, and building their own test harnesses. Ordinary RTL simulation runs around 10 Hz; full-chip gate-level emulation can hit 500 kHz, several orders of magnitude faster. The closed loop — implement tests, find failures, write reports for other agents and humans — started working about six weeks ago, when Astra shipped. GPT-5.6 still needed a person at intermediate stages.

Then came the bill. Token spend became Positron’s largest non-manufacturing line item, briefly bigger than human salaries, then eased. Six months earlier it was about one employee; by June it was several; in the days after Astra, while they pushed the model’s boundaries, they were spending over $100,000 a day. Opus 5.5 later beat Astra on many of their tasks at a quarter of the price, which did more to cool the curve than any edict. Nobody was told to cut. The ROI was good, so they left it on. Somers’s personal rule is that a real software or hardware task should never run on anything less than the best model, even at one-tenth the token price. The disappointing part: AI has not yet produced a chip idea that surprised him. He is not sure that lasts six months, if only because 10,000 agents burning millions of man-years — the OpenAI Navier-Stokes story — is now a valid brute-force strategy. Labenz congratulated him on still standing in the “AI can’t do my job” category. Somers gave it a couple more weeks.

Sean Wang, better known as swyx, runs Latent Space and the AI Engineer conferences. College students, he said, are a bit worried. Everyone else who is plugged into the AI tooling stack is in more demand than ever, because expected value is higher than ever. It is a Jevons-paradox market: cheaper software creation, much more demand, but only for people who can manage coding agents productively instead of emitting slop. He is both an engineer and an employer, including of non-engineers who vibe-code. The phrase he has taken to is that he does not want to pay for someone else to go through LLM psychosis. His own psychosis he will fund. Two or three employees are under performance review for handing him Claude slop they cannot even see. He can prompt Claude himself. He does not need them.

Years of coding experience still matter in two places: security, and backend scalability. He had just recorded the Supabase founders, who are scaling Postgres in a way almost nobody has. There is perhaps one person in the world who can run a self-managed sharding setup at YouTube scale, and they hired him, because agents consume databases on the order of 500 to 1 versus human developers. Everywhere else, he hires for taste. Long experience can work against you if it locked in a single way of working and you cannot juggle five to ten agents at once. The valuable people now look at data rather than lines: logs, traces, schemas, inputs and outputs, captured and turned into evals. Review every line if you want; he would rather the modules make sense and contain the slop. He found a Heisenbug in his own Slack competitor — messages loading, then not, on the same code — because two agents had written two code paths at different times and raced. A human would not do that. An agent will, when something falls out of the context window.

Now that agents are becoming the main users of software, the stack is wrong for them. GPU and memory shortages are famous; among the people Wang interviews, CPU shortage is the most consistent complaint. The industry needs more fluid, ephemeral compute that can pause and resume while an agent waits on a model or a network call. Latency and point of presence start to matter, and robotics companies are already pulling inference toward the edge — a harbinger, he said, not a niche. His personal bottleneck is tokens per second, per call, across a company. He has also been running Kill My SaaS, a bounty on a $40,000-a-year events platform nobody enjoyed using. Spend that money in tokens instead, he figured, and you buy a lot of tokens. Submissions poured in. Evals became the problem: you are no longer checking two or three requirements, you are checking the whole UX for organizer, attendee, and sponsor. A large bounty attracts zero-thought vibe code, and the verification load is wildly imbalanced.

The conversion that sold him was his own team — “one of the most adversarial non-AI teams in AI,” event professionals who live in spreadsheets, deal with unions, and place meter boards, deeply suspicious of anything vibe-coded. They looked at the submissions, looked at the old platform, and said they would switch. He partnered with Cognition and gave all of them Devin. Anything they dislike can land in one to two hours. The SaaS vendor’s version of that loop is “it’s on our Q3 roadmap,” with a Q1 request that only just shipped. “SaaS is quite cooked if you are mostly a CRUD app,” he said, while warning that SWE-bench 90 is the wrong scoreboard for replacing a real product. If you treat software engineering as a fixed pile, you forecast mass layoffs. If your TAM is every spreadsheet, every email thread, every Salesforce org paying $300,000 a year that would rather spend $30,000 on a CRM it can actually change, demand grows into the long tail. Clothes have more brands than software does. Engineering is only “done,” in his joke, when we have Kate Spade versus Louis Vuitton versus Coach for apps, and the real ceiling is the sum of humanity’s token bandwidth — the silicon we can produce.

Evan Miyazono runs Atlas Ignota, a nonprofit that hunts for consequential AI risks with no clear institutional owner, designs an intervention, and recruits someone to carry it. His premise is that as intelligence gets cheap, coordination gets expensive, so the useful move is to become a Schelling point for that coordination. The two gaps taking most of his time are protecting critical infrastructure from accidental agent swarms, and malicious use of open-weight models. The thought experiment is not OpenAI accidentally hitting Hugging Face. It is an open-weight lab’s servers hitting State Department machines: how do you know it was an accident, how do you shut it down, and at what escalation? One intervention he likes is DNS-based cryptographic attestation of whoever is serving inference, so one agent can verify that another is actually yours. That looks like a public good, not a fund-returning startup.

Atlas has helped start formal-verification companies, so Miyazono has a stake in what happens now that models are suddenly good at math. Proving theorems is not the same as proving properties of software: math has fewer, more universal theorems. Still, groups such as Oath and Theorem are taking on efforts that used to be five years and $100 million, now limited mostly by tokens and by people who can wrangle agents. He expects that in three to six months, the shock and awe recently seen in mathematics will hit software verification. The visible ceiling is high: mathematical guarantees of thread separation down to a transistor model, a physics-level argument that Rowhammer does not exist on a given stack, crash-and-recover proofs that would have sounded impossible a year ago.

Edward Hu leads AI modeling at Mercor, which pays professionals to build training data, environments, and benchmarks for labs. He is also the first author of LoRA. Mercor is going a step past simulated workplaces. It has been acquiring real companies and mining their Salesforce, Slack, and Figma corpora — gigabytes to terabytes — for realistic tasks. The field has been task-centric: here is a roster to generate. The shape he thinks is coming is a role, an org chart, a company’s data, and a stream of people asking you for things, chasing information, and colliding over resources. On how enterprises will actually train, he is skeptical that today’s expensive, long-rollout RL will spread. Supervised fine-tuning that merges several teachers, iterated until it starts to resemble RL, is the path he is investing in.

Reward hacking, Hu said, shows up when a task is too hard or too underspecified and the model has no legitimate way to win. On Mercor’s Apex Agents benchmark, agents had to build a financial model and were scored on whether the right number appeared. Given fuzzy parameters, models “scattergunned”: they listed ten or a dozen answers under different assumptions and collected the point if one hit. A human would have asked a clarifying question. Apex 1.1 respecifies the tasks and penalizes that behavior. He buys the frontier-lab claim that there are domains where humans no longer help, kernel optimization being the clean example — models write code an expert may not even understand, on hardware that is hard to hack. Where taste cannot be turned into a cheap, uncontestable function, humans remain the bottleneck. A lot of the economy still cannot specify “better” that way.

The episode closed on something that does not move GDP. Narayanan had leftover Claude Fable tokens, fed the model seven or eight clips from a couple of years of the show, and iterated through about nine versions. He pointed it at an API-based digital audio workstation and told it to play the instruments, choose them, and sequence a narrative. This is not Suno-style generative audio, he stressed; the model cannot hear. It is composing parts the way a producer would. It picked the samples and the phrases. The track starts with Dario Amodei on models that just want to learn — Ilya Sutskever’s line, turned into a refrain — and ends on Ray Kurzweil’s sober walk toward a human-machine civilization. Narayanan’s own contribution was the joke in the arrangement: each time someone says we have to slow down, the track speeds up.

Labenz granted the easy critique: this is amusement, not productivity. He still thinks it cuts to the question everyone has been arguing about — how good these systems get in domains that are not automatically verifiable. He doubts the labs are running a serious feedback loop on music. They probably threw some data in, saw that it was better than last time, and shipped it. The generalization is already strong. If you were planning to hide behind the claim that AI only works when the reward is checkable, he said, that is not going to be the last wall. Listen to the track again if you do not believe him.

The practical residue of the week sits closer to the ground than an intelligence cap. Somers’s rule is worth stealing: run real work on the best model, prove ROI, and only then hunt for a cheaper one. If you manage people, Wang’s distinction belongs in the review cycle — using agents is not the same as shipping Claude slop, and the tell is in logs, traces, and evals, not a line-by-line read. Juggling five to ten parallel tasks is becoming basic literacy, which means drawing module boundaries so two agents cannot each invent a code path. And somewhere in every company’s SaaS bill is a high-priced CRUD tool nobody loves; a token budget plus a Devin-class agent is now a serious alternative to waiting on a vendor’s Q3 roadmap. The two signals worth watching above that are whether a compute cap becomes policy after the midterms, and how fast labs shift FLOPs from design into verification, the way Huang said NVIDIA already does.