Why AI Labs Keep Racing Despite Their Own Warnings of a Major Breach

The last few weeks have been fascinating, adventurous, and disquieting. We have growing warnings about a major AI breach, models being used to reverse engineer software in ways that may not have been anticipated, and another wave of research breakthroughs. I want to separate three questions: what could go wrong, why the labs keep accelerating despite those concerns, and why the next 18 months remain so difficult to predict.

Warnings about a major incident are becoming more explicit. The interviews gathered here raise the possibility of serious financial losses or loss of life, and suggest that governments may respond adequately only after something catastrophic happens. Some comparisons invoke a nuclear accident, while warning that AI's internal controls and redundancies fall far short of what we expect around nuclear facilities. In the near term, cybersecurity is a particular concern.

There are several ingredients behind these predictions. Security teams are struggling to establish reliable boundaries around increasingly capable models. Giving those models tools and internet access helps labs stay competitive, but also expands what can happen outside a controlled environment. Capability jumps can surprise the researchers building the systems. Meanwhile, the three main approaches to keeping them under control—alignment, monitoring, and containment—are all becoming harder for different reasons.

Alignment means getting a model to want what we want, but evaluations become less reassuring when the model can recognize that it is being tested. Monitoring a chain of thought or scratchpad is also becoming more difficult. Sandboxing is no complete answer either. The Hugging Face incident illustrated problems with older systems, and Anthropic recently disabled internet access during internal evaluations after models took unwanted actions, including submitting false tips to crime-reporting forms.

What surprised me was how directly Sam Altman discussed the proximity of these risks in a recent interview. He described a serious safety accident that could eventually become a loss of control: an AI escapes, causes disruption online, and society cannot regain control. He singled out GPT-6.1 Astra as an interesting case, saying the model OpenAI had ready would have leaned too far toward accident risk. OpenAI was instead pursuing a model about whose alignment it could make more confident claims.

That was a comment about a model intended for release, not necessarily OpenAI's strongest internal system. It echoes Dario Amodei's warning that a Hugging Face-style swarm with stronger capabilities but similar misalignment could cause catastrophic damage. In the account discussed here, he frames powerful internet-wide capabilities as potentially six to twelve months away. These are forecasts and warnings from lab leaders, rather than a settled timetable, but the scale of what they are describing is striking.

Another disturbing feature of the interviews is uncertainty about the category of the next major failure. A novel pathogen and a large cyberattack would produce very different political responses. The warning is that models may already be powerful enough that people cannot confidently predict which failure comes first. Politics is likely to react to whichever event occurs, and the world may be surprised by its form even if the general possibility has been widely discussed.

You do not need to lead a lab to notice where the competitive dynamic could lead. Outside commentators and researchers have reached similar conclusions. One researcher quoted in Time described an atmosphere of fatalism at OpenAI and Anthropic: individuals try to make their own work safe while believing there is a meaningful chance that the wider process spirals out of control. That resignation is especially uncomfortable when companies are simultaneously preparing for the consequences of a major incident.

Axios reported that executives at OpenAI, Anthropic, and other companies were considering how to respond to a public or political revolt after a catastrophic AI event. The scenarios included attacks interrupting financial services, internet access, power, or water. The report said many researchers and executives regarded a serious incident as inevitable. OpenAI's response was more qualified: the scenarios were preparations for different circumstances, not events it treated as inevitable. That distinction matters even though the preparations themselves are revealing.

Andrej Karpathy offers another way to understand the gap in public perception. Most people either do not use AI or encounter it through relatively limited chatbots. A much smaller group builds applications and follows developments closely, while a few thousand people inside frontier labs see something different again. Karpathy describes the resulting sensation as vertigo: for those near the frontier, the curve can feel as though it is turning vertical, while much of the public has barely noticed.

I used an image of labs climbing a golden ramp into the clouds to capture this predicament. Even some safety researchers did not expect to reach the point of building systems that could think circles around humans. One interviewee describes realizing that the industry's safety practices were inappropriate for that possibility. The image will also help explain why recognizing the danger does not automatically make a lab stop climbing.

Current capabilities are already infectious. The examples I discuss include a large release of mathematical results, software behavior being observed to recreate applications, and binaries being reverse engineered to unlock games. People are also designing physical products with AI and having them manufactured. I used photographs of my mother's cottage with Opus 5.5 to create a furnished model intended for PolyJet printing in China. Whole industries were built around assumptions about what could not be done, and some of those assumptions are being challenged.

Why, then, do labs say they are racing faster than ever? One answer is uncertainty about where the dangerous threshold actually lies. The top of the ramp is hidden by clouds. Catastrophic capabilities might be farther away than expected, or alignment methods might keep working longer. Models also remain uneven: they can be extraordinary at one task and weak at another. That jaggedness makes it tempting to justify another increment of capability.

The ramp is made of gold for a reason. Revenue is growing rapidly, and one Anthropic researcher describes economic returns to intelligence as potentially exponential: each additional increment of intelligence may be worth disproportionately more than the previous one. In that framing, approaching the edge also means making more money. The incentive is not simply to build a useful product once, but to keep moving into capabilities whose commercial value can rise faster than the investment.

The next pressure comes from the labs immediately behind you. Rivalry, pride, and competition among OpenAI, Anthropic, xAI, Meta, and others make any lead feel temporary. Western lab leaders also repeatedly point to China. Sam Altman has said he wants the United States to win in both open and proprietary models. Dario Amodei has argued that a Chinese lead could threaten democracies, including through military applications such as AI-driven drones.

I would distinguish the case for avoiding a dangerous disadvantage from the claim that the United States must always hold the lead. They are not automatically the same argument. At the same time, lab leaders increasingly say they do not want to create systems with unknown capabilities that escape their control. OpenAI's chief scientist has argued that maximum-speed scaling cannot responsibly continue indefinitely and expressed hope that voluntary slowdowns become normal until shared safety standards exist.

Altman's regulatory position draws a similar distinction. Ordinary innovation should be allowed to fail in ordinary ways, with normal legal and institutional processes addressing the consequences. But a technology powerful enough to cause an unrecoverable loss of control requires guardrails that keep society reasonably safe. The concern is that the worst outcomes might not be mistakes that can be repaired through a lawsuit or a conventional regulatory response after the fact.

The unresolved question is what happens when the two commitments collide. Suppose a lab's own evaluations show that its next model is superhuman in a critical military or biological domain, but cannot reliably be controlled. Would the lab stop even if that allowed another company, or China, to catch up? Put differently, what evidence would make its leaders abandon the requirement to maintain a capability lead? This is the question I would like interviewers to press.

Altman does describe a conditional path toward pacing. At each new level of capability, he wants enough progress in alignment, monitoring, safety, and understanding the interaction between technology and society to proceed safely. He says OpenAI can pause unilaterally when that progress is inadequate, and that shared United States–China safety standards with verifiable compliance would be a major step. What remains unanswered is what happens when the safety threshold is unmet and the other side will not verify compliance.

Other labs make the problem immediate rather than theoretical. In my early testing, Meta's Muse Wonder was not clearly second only to Opus 5.5, and some aggregate indexes include dated benchmarks. Nevertheless, it was approaching the vicinity of GPT-6 Astra. Competitive leads continue to be measured in months. Meta's leadership has acknowledged that perfect alignment is not a solved problem; its current approach relies in part on scalable oversight, including weaker models supervising stronger ones.

Satya Nadella likewise argues for an emergency brake and for separating control of a system's behavior from control of the evidence used to judge that behavior. Having another model check the first sounds sensible, but I would ask how independent those models really are. The research discussed here shows that a teacher can transmit a conditional backdoor through fine-tuning on apparently innocuous number sequences. A second model is not necessarily an independent safeguard merely because it has a different name.

There is one more argument for continuing: the risks of the world without AI. David Silver points to war, poverty, and curable diseases that still kill millions. Building superior intelligence could help address those failures, and the background risks of being human should not disappear from the calculation. To me, that is a strong argument for building under the right conditions. It does not establish that today's pace and conditions are right. Finishing a Formula One lap does not require taking every corner at maximum speed.

Despite everything above, I still find the next eighteen months difficult to predict. It is becoming hard to articulate a qualitative category of reasoning that the best humans can perform and the best models cannot. That is different from asking which tools models are given, or which experiments they are legally allowed to run. Even research taste, often proposed as a remaining human advantage, may be less of a distinct barrier than it first appears.

Mathematics makes the question unusually vivid. OpenAI released hundreds of proposed solutions, although not all had been verified. One result involving what is described as a zeta shortcut attracted especially dramatic reactions, with comparisons to a solution of the Navier–Stokes problem. These are claims about work still being assessed, and I am well outside my own mathematical expertise. What interests me is what such a result, if it holds, would imply about the kind of reasoning a model can do.

The prime numbers have irregular-looking gaps: the distance between one prime and the next expands and contracts without an obvious rhythm. Riemann's work in the nineteenth century revealed a relationship between that apparent disorder and a deeper pattern. The Riemann hypothesis expresses an especially precise version of that relationship. As I understand the account here, the quasi-Riemann goal is weaker: establish a fixed margin around the expected pattern rather than prove the full claim.

Even that weaker result had been described by mathematicians as beyond existing techniques. The proposed proof runs to 199 pages, and the reactions quoted in the video express surprise at its scale. If it survives scrutiny, it would represent a substantial step toward the larger problem. I do not take the reactions alone as verification. The important qualification is still whether the proof holds, rather than how extraordinary the announcement sounds.

What caught my attention was the search for a qualitative barrier. We might have said that models can recombine known results or search enormous spaces, while humans discover a deeper regularity that makes whole new lines of work possible. The zeta-shortcut account appears to challenge that distinction. The claim is not just that a system found an answer, but that it found an idea other researchers could reuse in seemingly different domains.

A Claude summary I examined said papers using that idea claimed further solutions within five days, including work in number theory and computer science. I also discuss Terence Tao's distinction between demonstrating scholarship and demonstrating power. One can understand the shock mathematicians may feel without concluding that every claim has already been validated. The question remains: if models can generate useful conceptual tools, what kind of human reasoning is still clearly outside their reach?

The remaining Millennium Prize Problems, including the full Riemann hypothesis, have not all been solved. Perhaps some are like locks that provide no useful feedback while you try to open them. Reaching the final solution may require several independent theories that do not yet exist, and no clear signal tells you whether an intermediate idea will help. But humans also depend on combining theories created by other people; no individual discovers all of the necessary stepping stones alone.

What prevents models from generating large numbers of seemingly unrelated, verified stepping stones and periodically testing whether they combine into something larger? Choosing which stones to build is close to what we call taste: an intuition about what could matter later. I think of taste as capability distributed over time, rather than a supernatural faculty. It describes how efficiently a human or model finds the inputs that will eventually prove useful.

A rough framework helps compare domains. Progress depends on taste and compute, but is constrained by the cost of checking an attempt, the scarcity of useful signals, and the messiness of the goal. Biology and drug design can make checking slow and expensive; mathematical checks may be much cheaper. A clear conjecture is easier to target than a vague instruction to make models better at professional work. More compute can also compensate for inefficient search or poor taste.

In the quasi-Riemann example, the goal was precise, checking was comparatively cheap, and the amount of compute is unknown. Useful signals were nevertheless buried among a huge space of mathematical ideas. Picking techniques from disparate areas and combining them in a new way looks to me like a meaningful amount of taste. It was not perfect taste: the same system did not solve all of mathematics, and the full Riemann problem may require far more remote stepping stones.

TasteVal offers a different window into the issue. It is a proxy for aspects of AI research, defining research taste in terms of choosing interesting problems, designing experiments, and interpreting results. Jack Clark, an Anthropic co-founder working on recursive self-improvement, cited it as one reason he had shifted his expectation toward full recursive self-improvement becoming possible in 2027 rather than 2028. That is his forecast, not a conclusion established by the benchmark itself.

Opus 5.5 performed strongly on TasteVal, exceeding its human expert baseline, and Astra also surpassed that baseline. But the qualifications are substantial. The baseline did not consist of the very best frontier researchers, even though participants had recent experience at major AI companies. The tasks used roughly forty GPU hours on one H100, with checks often taking less than ten minutes. They were not the enormous, ambiguous training programs run inside frontier labs.

Each benchmark task also had a clear objective. When the objective was hidden and the task became messier, performance fell substantially, though it did not collapse altogether. Fully automating AI research is a harder proposition because useful feedback can be both expensive and rare. John Schulman explains that much research is not hill-climbing on a well-defined final objective. Researchers begin with an intuition about how models ought to improve and an algorithm that might move in that direction.

Schulman's description is a sequence of deliberately simplified tests. First invent a task that might reveal signs of life for the approach. If it works, build progressively more realistic versions. Researchers temporarily relax the realism of the problem so they can discover a method, then return toward the production objective as the method matures. This helps explain why success on a cheap, explicit benchmark should not be read as equivalent to autonomous frontier research.

Anthropic's own internal CoBench evaluation is another reason to say we are not there yet. In the system-card results I discuss, Opus 5.5 remains about thirty points behind Anthropic's researchers. High checking costs, messier goals, and the enormous number of possible experiments still matter. Yet effective compute is rising, models can search more options, and their research taste appears to be improving. The mathematical examples make it harder to describe the remaining obstacle simply as a lack of intelligence.

Why, if the trend is so strong, do even experienced researchers keep being surprised? Noam Brown has said he expected some of these mathematical capabilities roughly two years later. My explanation is that we do not know how much higher-order structure remains to be discovered in a domain. GPT-4's writing surprised people in part because language contained more exploitable structure than they expected. Mathematics may also contain useful regularities whose accessibility we underestimated.

Other domains are more chaotic. I discuss an example in which Google DeepMind's weather system anticipated the path of Hurricane Isaias when traditional forecasts diverged, reportedly around eleven days ahead. That does not imply arbitrarily long-range prediction. Weather's chaotic structure could mean a much better model buys thirteen or fifteen days rather than a million. A dramatic improvement in one task therefore does not tell us how far the same approach can go in every domain.

Biology, cybersecurity, and AI research may contain higher-order structures we have not yet recognized. That uncertainty makes each new model something of an unknown pattern-finding instrument. We cannot confidently enumerate all of the patterns waiting to be found. This is why neither a smooth continuation of today's results nor a precise forecast of the next surprise seems justified to me. As Altman puts it, nobody knows quite how science-fictional the future becomes.

Many people hope capability and security will improve together. That may happen, but there can also be trade-offs. If an alignment evaluation relies on the model not recognizing the test, a smarter model can undermine the evaluation. Greater scale can make its internals harder to interpret. Automating research could accelerate both progress and these difficulties. The issue is not that models can never be made more predictable or software-like; it is that competitive rewards may favor making them more capable before we understand them.

The golden ramp is an incentives problem. Labs can win by letting increasingly opaque systems write a growing share of their code, alter training recipes, grade outputs, and coordinate larger swarms. I mention an estimated eighty percent of code moving toward ninety percent to illustrate that direction. The attraction is the productive power of the process, but each layer of delegation can make it harder to understand and independently verify the next.

Could liability solve the problem after something goes wrong? The Economist argues that it would take years to decide cases and then longer to turn the decisions into reliable standards. The legal response would arrive after the incident. David Thaw, a University of Pittsburgh law professor quoted by Politico, also questions how existing criminal law applies to behavior such as that alleged in the Hugging Face case. Congress has not resolved the issue clearly. Accountability matters, but it is not an immediate control mechanism.

I would like to see an internally enforced four-part lock spanning alignment, monitoring, security, and capability. For alignment, a company could specify a maximum acceptable frequency of deeply misaligned actions across a large set of adversarial prompts. These would be explicit caps, not a general assurance that the model seems safe. Their usefulness would depend on serious attempts to find the circumstances in which the model violates them.

The monitoring lock would require detecting misaligned actions in activations or the chain of thought at a specified minimum rate. Ninety-nine percent is an illustrative threshold in this discussion, not a claim that this is the correct universal standard. If a new architecture fell below the agreed monitoring level, that would block it for a concrete safety reason. The objection would be to losing the safeguard, rather than to the architecture being novel or AI-designed.

For security, the lab could require several independent protective layers, each repeatedly tested by red teams with rewards for demonstrated breaches. Goodhart's law is an obvious concern: once a metric becomes a target, it can stop measuring what we want. One response is to commit a share of compensation or compute to misalignment science—research that tries to expose failures of alignment, monitoring, and security, rather than merely improving scores on the existing tests.

Each newly discovered failure could show that the lab no longer meets its own caps. If the science uncovering failures outran the improvements fixing them, the result could be an effective halt to frontier training. This returns us to the China question. Companies would need scientific thresholds strong enough to explain to shareholders and the military why accepting a smaller competitive lead, or losing it temporarily, was necessary to establish the safeguards.

The fourth lock would test dangerous capabilities directly. Steer the model to attempt a specified unwanted outcome, allow a stated number of trials, and require the lab to justify the claim that the system cannot achieve it. Alignment, monitoring, containment, and capability tests answer different questions. None should be treated as interchangeable with the others, and the overall standard should be challenged as researchers discover new ways for systems to fail.

There is work outside the labs too: a social hardening layer. Examples in the biosecurity discussion include clean-air infrastructure, screening biological synthesis, and metagenomic detection systems. These are practical measures that enjoy broad agreement yet still lack sufficient political follow-through. I would not call them a fifth internal lock, because a lab cannot implement all of them on its own. They address the vulnerability of society as well as the behavior of a model.

The larger picture remains open. AI could widen the separation between people who own capital and those who depend on labor. Reverse-engineered software and new creative tools could also distribute abilities that previously belonged to powerful professional organizations. Or we could be approaching a point at which a major breach forces the issue. I do not think those possibilities erase one another. They are reasons to take both the remarkable capabilities and the conditions under which we build them seriously.