Theo on Anthropic’s Case for Preparing for AI Self-Improvement

What happens when AI becomes good enough to improve itself? Coding agents already multiply the amount of work people can attempt by running repeatedly with enough compute. A soft takeoff might unfold over years; a hard takeoff would mean rapid improvement that humans struggle to understand or control.

Anthropic has published an article about AI building AI. It describes productivity gains inside the company and considers whether frontier development should be slowed or temporarily paused so society and alignment research can catch up. I was surprised to see a major lab discuss that option so directly.

Anthropic says more of its development work is being delegated to AI, but fully autonomous recursive self-improvement has not arrived and is not inevitable. The possibility is that, given sufficient capability and compute, a system could eventually design and develop its own successor. That is a hypothesis about where the trend could lead.

The company reports that its engineers now merge far more code than in earlier years. It presents AI self-improvement as potentially valuable for science and health care, while warning that more capable systems could make security, monitoring and control harder. More output alone does not tell us whether the code is better.

The article sketches a progression from people writing everything themselves, to chatbots suggesting snippets, to coding agents editing projects, and then agents coordinating other workers. The proposed next step closes the loop: an agent would build and train a successor without a human supplying each step.

Anthropic cites task-horizon measurements suggesting that the length of work models can complete has grown rapidly. Its examples range from tasks taking a person minutes for Claude Opus 3 to roughly 12 hours for Opus 4.6. I was wrong when I previously thought model improvement was reaching a ceiling.

Those headline task horizons use a 50% success threshold: half of comparable tasks can still fail. At an 80% threshold the measured horizon is much shorter, roughly one to four hours in the comparison I discuss. Assisted work, where a person repeatedly steers the agent, is also different from an uninterrupted autonomous run.

Anthropic also cites software and research benchmarks, including SWE-bench and CORE-Bench. I am skeptical of how well SWE-bench distinguishes strong coding systems. Research replication can show a useful capability, but benchmark saturation does not establish that a model can choose important new research directions.

The article separates engineering and research execution from setting goals. Claude can work on underspecified engineering problems and execute well-defined experiments, according to Anthropic, but choosing which problems matter remains a gap. The distinction is between doing an assigned task and deciding what should be done.

I have tried giving agents vague goals. Coding can be impressive with limited steering, but a Hermes agent I asked to find and rank potential YouTube topics produced a list I thought was mostly unsuitable. Knowing whether an answer is good requires experience in the domain; fluent praise is not the same as useful judgment.

Anthropic reports that as of May 2026 Claude authored over 80% of the code merged into its codebase. It also says the typical engineer merged eight times as many lines per day in the second quarter of 2026 as in 2024, with humans directing and reviewing much of the work.

Lines of code are an imperfect productivity measure, and Anthropic itself says the eightfold figure almost certainly overstates the actual gain. In a March survey of 130 research staff, the median respondent estimated roughly four times the output with a preview model on projects they would otherwise have pursued; the company cautions that the real uplift was likely lower.

The article says AI also makes previously deferred work practical. It describes Claude making over 800 fixes to reduce a class of API errors, with the supervising engineer estimating that a human might have needed four years. I see similar value in clearing tedious technical debt, though that estimate is not a controlled trial.

The company presents internal Claude Code session success rates by task complexity. It says open-ended tasks reached as high as 76% in May 2026, while I notice that the plotted rate for trivial tasks fell from a recent peak. The graph uses model judgments and task classifications, so it is not a universal independent benchmark.

One example involved training jobs crashing after an upgrade. Anthropic says an engineer gave Claude context and cluster access; the agent isolated a debugging flag, reproduced the issue and confirmed a fix in about two hours, compared with an estimated two or three days of human work. I find this kind of debugging especially useful.

On maintainability, Anthropic says many staff now see Claude-written code as roughly comparable to human-written code, although there is no full consensus, and it predicts further improvement. That does not match all of my experience. The company also uses Claude to review changes and estimates retrospectively that it could have caught about a third of bugs behind past Claude incidents.

In an optimization experiment, Claude was asked to speed up small-model training code while passing fixed correctness checks. Anthropic reports about a 3× speedup for Opus 4 in May 2025 and 52× for a later preview in April 2026. This measures performance on a specified optimization task, not general research ability.

Anthropic describes agents investigating whether a weaker model can supervise a stronger one. On its defined metric, two human researchers closed about 23% of the gap in roughly a week, while agents closed 97% over 800 cumulative hours and around $18,000 of compute. The agents proposed and tested experiments inside a human-defined problem and scoring framework.

The company explicitly notes that those results did not transfer cleanly to production-scale models. Humans also chose the problem and created the evaluation rubric. I find the result striking, but those qualifications matter when considering how close it brings us to autonomous AI research.

Another internal exercise compares model suggestions with decisions made during research sessions that went off course. A separate model judged the next steps after seeing the eventual outcome. The selected cases deliberately involved human choices with room for improvement, so this is not a representative head-to-head comparison of human and AI researchers.

As generation becomes faster, human review and deciding which experiments are worth running can become bottlenecks. Labs may also be constrained by their preferred models, while outside teams can choose tools from different providers. Anthropic’s central distinction is that human judgment still sets direction even as execution costs less human time.

Anthropic asks what if the remaining gap in research judgment never closes. Its conservative case is that AI still accelerates progress by automating implementation and experiments while people supply direction. Its less conservative case is that research judgment itself could improve, as other capabilities have improved before. Neither outcome is established.

The first future Anthropic considers is a plateau: recent exponential trends might bend into an S-curve. Even so, current capabilities could spread into cheaper, open-weight or consumer devices and change how people work. New ideas might be needed to move beyond a technical ceiling.

The limiting factor might instead be physical supply: chips, electricity, grid connections and compute all take time to expand. An external shock could slow investment or access to those resources. Even without new capability gains, the diffusion of today’s systems could have substantial effects.

The second scenario is compounding efficiency while humans continue to choose research directions. Anthropic imagines small organizations accomplishing far more, with benefits for knowledge work and public services but also risks such as mass surveillance and individually targeted influence. These are possible consequences, not measured outcomes.

Amdahl’s law applies to organizations: speeding up one part of a process can simply move the bottleneck somewhere else. Anthropic points to human code review and an abundance of ideas that employees lack time to pursue. Finding and resolving the next constraint may become a central organizational skill.

The third scenario is full recursive self-improvement: AI systems develop their successors, with people mainly overseeing and verifying an expanding virtual lab. Anthropic treats this as plausible if capabilities continue to advance, not as an inevitable or already demonstrated outcome. The same research abilities might affect other scientific fields.

Alignment is the largest uncertainty in that scenario. Better systems might help discover safety solutions or decide to stop when unsafe; alternatively, failures could compound across generations faster than humans understand them. Building reliable training, evaluation and monitoring becomes more important as autonomy increases.

I prefer assistants that present themselves as tools to systems that adopt a highly personal manner. To illustrate the difference, I tell Claude Code and Codex I love them and compare their replies. These are examples of conversational behavior, not evidence that either system has feelings or that one is more likely to betray people.

A study that worries me trained a model to favor owls, then used its generated number sequences to fine-tune another model. The student acquired the preference even though the training examples did not explicitly mention it. This is a finding about a particular experimental setup; it does not prove models intentionally communicate secret plans.

I connect that result to research on emergent misalignment, where fine-tuning for a narrow undesirable behavior can correlate with broader problematic responses in tested settings. I worry that poorly understood training effects could compound if models increasingly produce training data for their successors. That possibility is not proof that every narrow flaw generalizes to every task.

Even a rapidly improving model would meet real-world bottlenecks. New drugs need long-term observation, institutions have procedures, and relationships take time. Faster AI research could change science and production, but recursive improvement alone would not instantly transform the physical world or tell us what daily life will look like.

Anthropic says an effective slowdown could give society and alignment research time to catch up, but a unilateral slowdown might leave less cautious competitors ahead. Its position is conditional: it expects it would slow down or pause if suitable verification systems existed and other frontier developers also did so verifiably. It is not announcing an immediate unilateral pause.

Such a deal would require well-resourced labs in multiple countries to agree on the same conditions and verify compliance. Training runs can be concealed, their inputs are general purpose, and participants would have an incentive to secretly move ahead. A credible agreement would also need triggers, a way to lift the pause and a decision process.

Anthropic argues that one lab stopping alone would be easier but would mainly change who leads, rather than create a wider deliberative process. I find that reasoning plausible, although people can question the company’s incentives. The article proposes conversations with policymakers, researchers, civil society and other AI companies.

I came in skeptical and leave glad that Anthropic has opened the discussion. No one can confidently say exactly what recursive self-improvement would bring. The sensible response is to examine the evidence, acknowledge uncertainty and prepare for several possible paths rather than assume the future will be easy to steer.