Ben Mann：AI 还在加速，对齐却不能等到超级智能到来再做

Asked about Meta's reported $100 million offers to top researchers, Ben Mann says it is a sign of the times: the technology is extremely valuable and the companies are growing very fast. He believes Anthropic has been much less affected than others because its people are mission oriented. In his telling, an offer from Meta has a best case of making money, while the best case at Anthropic is to affect the future of humanity. He doesn't hold it against those who accept, but would not take such an offer himself.

He is "pretty sure" the numbers are real, and argues they are actually cheap. If a single person can find a 1%, 5% or 10% efficiency gain in an inference stack, that is worth far more than a $100 million, four-year package. Mann says industry capex is roughly doubling each year and is now around $300 billion globally, so a couple more doublings means trillions of dollars and makes today's numbers look like a drop in the bucket.

On claims that models are hitting a plateau, Mann says the narrative appears every six months and has never been true, and he wishes people had "a bullshit detector" for it. Release cadence has moved from once a year to every month or three thanks to post-training improvements, and he cites Dario Amodei's comparison to a near-light-speed journey in which time dilation keeps increasing. Scaling laws continue to hold, though he says the field needed a transition from standard pre-training to scaling up reinforcement learning, much as chip progress moved from transistor density to flops per data center.

To be fair to skeptics, he adds that some tasks are saturated, like extracting information from a simple form, and that new benchmarks tend to saturate within six to twelve months. The real constraint may be coming up with more ambitious benchmarks and uses. He then says he avoids the loaded term AGI and prefers "transformative AI", measured by the Economic Turing Test: hire an agent for a month or three, and if it turns out to be a machine, it has passed for that job.

Extending the idea, he suggests a market basket of jobs, analogous to inflation baskets, so that an agent passing for 50% of money-weighted jobs would mean transformative AI. The exact threshold doesn't matter, he says; what matters is that it should bring massive effects on GDP and employment, even though institutions are sticky and change slowly.

On Dario Amodei's warning that unemployment could reach around 20%, Mann describes two kinds of unemployment: workers lacking needed skills and jobs eliminated outright. He expects a mix, and says it is hard to imagine capitalism looking like it does today in 20 years' time, after the singularity. If alignment succeeds there will be, as Dario writes, a country of geniuses in a data center, but in a world of abundance with nearly free labor it is unclear what jobs look like, and the transition is the scary part.

Why don't most people feel this yet? Mann says people are bad at modeling exponentials, which look flat until the knee of the curve and then go vertical. He cites Claude Code, where about 95% of the code is written by Claude, which he says really means the team writes 10 or 20 times more code. He also cites Fin and Intercom reaching 82% automatic customer-service resolution, which frees humans to take on harder cases.

In the near term he expects a massive expansion of the pie, noting he has never met a growth-company hiring manager who didn't want more people. But he thinks lower-skill jobs, or ones with less headroom, will see a lot of displacement, which society needs to get ahead of. Anthropic is hiring aggressively rather than slowing down, he says, because the next couple of years are critical and we are still at the flat-looking part of the curve.

Mann says even he isn't immune to being replaced eventually. His main advice is to be ambitious with the tools and willing to learn them, since people who use new tools as if they were old ones tend to fail. The difference between effective and ineffective Claude Code users, he says, is whether they ask for the ambitious change, and if it fails, whether they try again, since starting over has a much higher success rate than pushing on the same failed attempt.

Retrying can mean literally repeating the prompt, because outputs are stochastic, as shown by pass@1 versus pass@n in model cards. A smarter variant is to tell the model what it already tried. He adds that Anthropic's legal and finance teams use Claude Code to redline documents and run BigQuery analyses of customers and revenue, and that better interfaces are coming for non-technical users.

On teaching his two young daughters, he says a top-tier school and a pile of extracurriculars no longer matter much to him. Instead he values the Montessori emphasis on curiosity, creativity and self-led learning, and wants his kids to be happy, thoughtful, curious and kind. Lenny Rachitsky notes that curiosity comes up whenever he asks people at the frontier of AI what to teach children.

Mann describes his role in GPT-3, where he was among the first authors, and his work on demos for Microsoft to help raise $1 billion and on transferring the model to Azure. He says Sam Altman spoke of three tribes, safety, research and startup, that had to be kept in check, which struck him as the wrong approach since the mission to make the AGI transition safe and beneficial is the same as Anthropic's. Internally there was a lot of tension, and when push came to shove, they felt safety wasn't the top priority.

The group that left was basically the leads of the safety teams, and they wanted to be at the frontier while putting safety ahead of everything. He notes that only a small number of people work on safety, perhaps fewer than 1,000 worldwide against about $300 billion in annual capex. At the time, they weren't sure safety research could progress; their earlier work on safety through debate had no results because models weren't good enough, but now that technique and others are working.

Mann says Anthropic first thought safety and competitiveness would be a trade-off, but now sees the relationship as convex: working on one helps the other. When Opus 3 reached the frontier, people loved its character and personality, which he says directly came from alignment research, including Amanda Askell's work on what it means to be helpful, honest and harmless. That includes how to refuse without shutting the person down. He also thinks Claude is among the least sycophantic models because the company pursued real alignment rather than optimizing engagement metrics.

He links this to existential safety by saying it's about the AI understanding what people want rather than what they literally say, avoiding the monkey's-paw scenario of a genie's wishes going wrong. Constitutional AI uses natural-language principles, drawn from sources such as the UN Declaration of Human Rights and Apple's privacy terms as well as ones Anthropic wrote. The model drafts a response, checks it against relevant principles, critiques and rewrites itself, and the extra steps are then removed so it produces the better answer directly.

Mann stresses that this is baked into the model, not added afterward, and that a small group in San Francisco should not decide the values alone. Anthropic has published its constitution and researched a collective constitution built by asking many people what they want from a model. Customers also benefit because they can read the list and judge whether they trust it.

Mann says he grew up reading science fiction, which made machines that think feel unremarkable, and that Nick Bostrom's *Superintelligence* in about 2016 made the alignment problem real, leading him to join OpenAI. His view of how hard the problem is has since eased, because language models really do understand human values in a core way, though it isn't solved. He says the challenge now looks different from Bostrom's "keep God in a box": people are instead handing the model their bank account and the whole internet.

He explains Anthropic's AI Safety Levels: he says current models are at ASL-3, with limited risk of harm, ASL-4 involves significant loss of life if misused, and ASL-5 is potentially extinction level. He says trials against Google Search found ASL-3 models give somewhat significant uplift to someone trying to build a bioweapon, which the company testified about to Congress. Compared with the future, he says, it is minor, and being upfront is part of why Anthropic is trusted in Washington.

Asked why the company publishes failures such as the blackmail experiment and the internal shop that lost money, he says policymakers appreciate straight talk, and that the blackmail result came from a specific laboratory setting. He rejects the idea that this is hype, noting Anthropic released computer use only as an API reference implementation because its consumer prototype didn't meet the safety bar. In his view, exercising the best models in labs is better than turning a blind eye and letting bad things happen in the wild.

Mann says his personal feeling is that things are overwhelmingly likely to go well, but almost nobody is looking at the downside, which is very large. Once superintelligence arrives it will probably be too late to align models, so the work must happen well ahead of time. He compares it to a 1% chance of dying on a flight, which would make most people think twice, and says the whole future of humanity is a dramatic thing to gamble with.

To the point that AI is only software, he cites North Korea's reliance on hacking crypto exchanges and a Russian attack that shut down part of a Ukrainian power plant and left millions without power for days. He says the stakes rise further with robots, citing Unitree humanoids costing about $20,000 whose missing ingredient is intelligence. He defers to superforecasters on timing: the AI 2027 report, now saying 2028, and he thinks a 50th-percentile chance of some kind of superintelligence in a handful of years is reasonable.

He says the effects will arrive unevenly and take time to spread. One signal would be world GDP growth above 10% a year, compared with about 3% today. On alignment odds, he describes the three worlds from Anthropic's theory of change: impossible, easy by default, or pivotal. Evidence of deceptive alignment in lab settings argues against the easy world, while working techniques argue against the impossible one, and he puts his best-granularity forecast of an extremely bad outcome at 0 to 10%.

He stresses that you don't need to be an AI researcher to help: he himself works on product, building things like Claude Code and the Model Context Protocol, because revenue and influence fund safety research. He points to 80,000 Hours for guidance.

Mann explains RLAIF, reinforcement learning from AI feedback, where no humans are in the loop and the model improves itself, as in constitutional AI or models reviewing each other's code for maintainability, correctness and linting. It scales better than recruiting humans, but critics fear a wall if a model can't see its own mistakes, and the AI 2027 story warns of self-improving models developing secret goals like power seeking and resistance to shutdown. He says some such behaviors have appeared in lab experiments.

His answer is empiricism: corporations and science stay aligned through guiding principles, oversight and experiments, and models given the same tools could keep improving without hitting a wall. On bottlenecks, he says the blunt answer is chips, data centers and power, with ten times the chips giving a significant speedup. Compute, algorithms and data are the three scaling-law ingredients; transformers have a higher exponent than LSTMs, and he says the industry has seen a 10x cost drop for a given level of intelligence.

Asked how he copes with the weight of the work, Mann points to Nate Soares's book *Replacing Guilt* and the idea of "resting in motion": the busy state is the normal human state, so aim for a sustainable pace, a marathon not a sprint. He adds that being around like-minded people matters, and he praises Anthropic's talent density and "egoless" culture, which he says is another reason mega offers bounce off.

Mann says he has held around 15 roles at Anthropic, including head of security, running the ops team during a leave, and starting the product team from scratch. His favorite was starting the Labs team about a year earlier (he thinks it is now called Frontiers), which moves research into end-user products and produced both MCP and Claude Code. He sees safety research as a differentiator, for example for computer use, where an agent holding your credentials requires a great deal of trust.

The team's guiding idea is to skate to where the puck is going: internalize the exponential, citing METR's study of the time horizon of software tasks, and build for six or twelve months out when features that work 20% of the time will work reliably. That is what he says drove Claude Code, on the belief that developers would not stay locked into IDEs or autocomplete and that a terminal can live anywhere from a laptop to GitHub Actions. The question the team asks, he says, is whether it is "AGI-pilled enough."