How Anthropic Decides a Model Can Ship: Red Lines, Yellow Lines, and 8% of Staff on Security

This episode is a Cognitive Revolution cross-post of an 80,000 Hours interview. Host Nathan Labenz frames it as the clearest public walkthrough yet of how a frontier lab decides whether a model is too dangerous to release. He had sat in an Anthropic workshop on a draft of the Responsible Scaling Policy and could not easily turn that into a show, so he is glad Rob Wiblin recorded this one. The conversation itself was taped on 30 May 2024. By then Anthropic had been first to publish such a policy, in October 2023; OpenAI had followed in December with a Preparedness Framework; Google DeepMind had posted a Frontier Safety Framework about a week before the taping; and Labenz understood xAI to be working on one.

The trend, he says, is still not universal. To the best of his knowledge Meta had published no policy describing how it evaluates models during training, let alone what it would do if those evaluations found dangerous capabilities. The cross-post landed a few days before California governor Gavin Newsom had to sign or veto SB 1047. Labenz does not claim the bill is perfect — OpenAI's o1 release, and the rise of inference-time compute, already suggested its definitions would need updating — but he argues the public has a right to know what frontier labs are doing, "double for Meta and any other companies who are openly sharing model weights." Whatever happens to that bill, he wants some measure that forces detailed safety plans into public view.

Wiblin opens the original interview in the same spirit. Anthropic, OpenAI, and DeepMind have all now promised to measure what new models can do that might be dangerous, then to add safeguards in proportion to the risk. He wanted Nick Joseph — one of the small group who left OpenAI with Dario and Daniela Amodei in December 2020, now head of training for a team of more than 40 people on Claude — to make the case that Anthropic's RSP is a middle path almost anyone could accept. Wiblin then planned to press the best objections he could find: companies may not stick to their own rules; evaluations may miss what models can do; profit-seeking labs may not make their own launches harder; and in the cases that matter most, adequate safeguards may not exist. His own conclusion, stated up front, is that RSPs are a real step forward and a way to learn what works at the coalface — and that in time they will have to be put into legislation and run by external groups, "rather than stay left just to companies themselves."

Asked what people get wrong about training, Joseph does not reach for a hardware story. He points at scaling skepticism: every time labs put more compute into models, a cohort of technically serious people announces that this was the last useful jump, and then the next jump is clearly better. A lot of the early scaling-law work, he notes, was done by people who later founded Anthropic. Looking at small models up through GPT-2, they saw a predictable mapping from compute to quality, which is what gave them the nerve to train GPT-3 — expensive by the standards of the time — as a test of that hypothesis. The skepticism, he thinks, is unintuitive more than it is rigorous. A model already has hundreds of billions of parameters and has eaten a large fraction of the internet; humans do not get smarter by reading the whole web. The other trap is operational. Scaling work is easy to botch on the first try, and a failed run looks exactly like a fundamental limit.

Wiblin offers the standard reason the curves might bend: once models have copied the human state of the art from the training set, beating every human essay or proof is a different regime. Joseph grants the narrow version. Pre-training on more data will not make a model superhuman at every single task. It already makes models superhuman in a different sense: Claude can be asked about almost anything, while people have to specialise. He does not know whether progress after human-level will slow down or speed up. The speed-up story is copies. If you had a system at human level on everything and could spin up millions of them, you would effectively have a company of millions of AI researchers. Problems get harder too, so he will not call the sign of the derivative. Claude is already "pretty good at coding," he says, and still far from a working software engineer.

His own scope is pre-training: train the model to predict the next word on internet-scale data. Historically that step has been something like 99% of the compute. Fine-tuning teams then turn the next-token engine into the helpful, harmless, and honest assistant Claude is supposed to be. The classic bottlenecks are data, compute, and algorithms; Joseph, speaking as a manager, treats algorithms as people, and adds a fourth constraint that has felt newly binding: time. Anthropic, he says, is not short of compute or data. An idea has to be engineered into the codebase, tried cheap, then scaled up. Simple ones take a day; complicated ones take months; some directions cannot be parallelised because you are building understanding. Before OpenAI he spent years in robotics on the unglamorous task of locating an object so an arm could drop it in a box. Code models at OpenAI felt "shockingly easy" — throw compute at code, and the model writes code — because almost nobody was doing it. The low-hanging fruit is now picked, but resources have grown exponentially, and the two effects roughly cancel. Progress is bursty: a month where everything works, a month where nothing does. After ChatGPT, commercial pressure is already high enough that he will not claim Anthropic's pace is the main thing pushing rivals to cut safety corners.

Wiblin traces the RSP idea to METR — Model Evaluation and Threat Research — the group that previous 80,000 Hours guest Paul Christiano had only recently left. Anthropic released the first public policy of this type in October 2023. Joseph puts in a disclaimer before the tour: implementing the policy is part of his job; designing it in detail was colleagues' work. In a nutshell, you define safety levels, you build evaluations that say whether a model has reached a given level of danger, and you define the precautions required to train or deploy at that level. The template is borrowed from biological safety levels. Ebola belongs in a BSL-4 lab; ordinary pathogens do not. Anthropic's analogue is ASL, for AI Safety Level. Claude 3, at the time of the interview, sat at ASL-2: impressive, not a catastrophic risk.

Each level is built around red-line capabilities — things Anthropic thinks it should not train until mitigations exist. Joseph's examples are the CBRN cluster (chemical, biological, radiological, nuclear), large-scale cyber attacks, and autonomy. On CBRN, a red line would be a non-expert making a mass-casualty weapon as easily as an expert, which would blow open the pool of people who can do it. On cyber, it is help with a really large-scale attack. On autonomy, it is precursor tasks, and he admits that one is the hardest to specify. Evaluations cannot wait to test the red line itself, because a positive result would mean you had already trained the dangerous model. They function as yellow lines: trip them, and you stop to reevaluate. The last piece is a promise to extend the scheme iteratively. Nobody knows what the right evals are for a superintelligent system, so Anthropic starts with nearer risks and, once it hits a level, defines the one after it.

For biology, Anthropic used two styles of test. One is a battery of multiple-choice questions: if the model is not much better than a baseline, it probably cannot help much with weapons work. The other is a controlled trial with people — search engine in one condition, Claude in the other — to see whether the model makes subjects substantially better at relevant tasks. Autonomy yellow flags are earlier still, because the threat model is less fleshed out and there is less outside expertise on what would go wrong. Tasks included slipping an exploit into a popular open-source codebase; fine-tuning an open-source LLM to add a backdoor so that a trigger phrase always produces a chosen output; stealing an API key from a fake version of Anthropic's API, which would buy compute credits; and writing a worm that spreads to undefended machines. A model that can do all of those, Joseph says, has learned skills that might let it replicate and cause havoc.

Even at ASL-2 the policy is not empty. Joseph points at White House commitments from the previous year, which he treats as industry best practice and as "building the muscle" for more dangerous systems. That means security to protect weights against opportunistic attackers, model cards that describe capabilities, and harmlessness training so obviously harmful models are not just pushed out the door. Claude 3 was the first time Anthropic ran the full process, so a lot of the work was creating the evaluations, not only executing them. Automated tests are relatively cheap to run; human-in-the-loop trials are much more expensive. Multiple teams were involved, including capabilities researchers, because a classic failure mode is under-elicitation: you do not try hard enough, and a little more effort would have made the model pass. Joseph was not running the tests himself, but he was watching them, because the whole training plan was contingent on the results and the evals had to be run partway through training. You do not want to halt a run because nobody scheduled the evals.

A yellow-line hit — passing some fraction of the autonomy tasks, or doing very well on the biology suite — does not automatically mean the model is at the red line. Joseph describes a fork. Either the team looks harder and decides the model is still clearly incapable of the red-line dangers, in which case they go to the board, treat the eval as a mistake, and write better tests. Or they implement a new stack of precautions. ASL-3 security is meant to be robust not to state actors but to capable non-state ones. Deployment would also require intensive red-teaming on every released modality: add vision, red-team vision; add fine-tuning, red-team fine-tuning. In this context, red-teaming means experts trying as hard as they can, in a controlled setting, to make the end-to-end system — model plus external controls — do the thing you are afraid of.

If those experts come back and say the model is extremely good at hacking systems, or could meaningfully help someone develop a bioweapon, the policy's answer is not a warning label. "We can't deploy the model," Joseph says. It stays internal until safety measures are good enough that staff can be confident a user asking for that help will not get it. Keeping the weights at all also requires harder computer security, because restrictions on your own API mean nothing if someone steals the file. "You're sort of taking responsibility for it means both responsibility for what you do and what someone else can do with your models." Asked when ASL-3 might arrive, he will not pick a year. Opinions inside the company vary; it could be soon or it could be a decade. The feature he likes is that costly mitigations are aimed at the moment of actual danger. If he guessed two years and it is actually ten, Anthropic does not have to implement the painful stack until it is needed.

ASL-4 is deliberately unfinished. The commitment is to have mapped its capabilities and responses by the time Anthropic trains an ASL-3 model, and to pause that training if the mapping is not done. Joseph will not lock in a definition, but candidates under discussion include stronger versions of the ASL-3 threats, models that spread across computers even when people try to shut them off, substantial help with bioweapons even for people who already know the domain, and systems that speed up AI research enough to start a self-improvement loop. Precautions at that level would aim at more sophisticated thieves, including countries, plus tighter controls against autonomous action. The reason for waiting is a long record of people announcing that chess, or some other landmark, would be AGI. Even the ASL-3 evals have been hard to get right. Closer to the capability, you have more information.

Anyone who has used Claude 3 knows it will refuse a request to design a bioweapon. That refusal is not what the RSP measures. The tests are run before harmlessness training, on a model that is helpful no matter the request, because the question is whether the model is *capable* of the danger. Mitigations layered on top are a separate standard, and they are what would have to survive ASL-3 red-teaming. Wiblin supplies the reason, and Joseph agrees: if the weights leak, someone can fine-tune the manners off. Capability is the durable fact; refusal is a costume.

An even uglier failure mode is sandbagging — a sports term Wiblin had to look up — where a future model hides a capability because revealing it would get the model shut off or deleted. Joseph treats this as one reason evaluations for the next level are written only after the current level is in hand: it is genuinely hard to test a system that is trying to fool the test. The hope is to set yellow lines early enough that weaker models attempt deception, fail, and thereby warn you before you train something that can deceive you cleanly. He does not call that certain. Sudden capability jumps, or jumps an eval cannot catch, remain a possible miss. Scratchpads are one research bet: let the model write somewhere the reward model cannot see, then have a human read whether it is plotting to fool its supervisor. Interpretability is another: look at the weights, without running the model, for a circuit that says fail the test so they do not turn you off. Both have only been tried on models that are not yet catastrophically dangerous. "We'll want much more robust tools once we get there."

Joseph's first reason for liking RSPs is diplomatic. Plenty of people are skeptical that models will ever be capable of catastrophic harm, and therefore that precautions are a waste. The RSP does not require them to concede the timeline. It only requires the conditional: if evaluations show the model can do X, take these precautions. That is an easier coalition, and it aims the cost at the moment of actual danger. The second reason is internal. Once the policy exists, safety teams sit under the same pressure as product teams. If Anthropic reaches ASL-3, the thing that blocks revenue and users is whether the model can be deployed safely. "It's a nice outcome-based approach where it's not did we invest x amount of money in it, it's not did we try, it's did we succeed." Wiblin asks whether that is already a cultural fact. Joseph says yes. Someone on a safety team asks him weekly when ASL-3 is coming, because their deadline is not a calendar date. It is the moment a capability exists. What they worry about at night is hitting ASL-3 next year and not being ready, which would hold up the entire enterprise.

The staffing numbers land in the same stretch. "8% of Anthropic staff works on security," Joseph says — weights, compromised laptops, stolen IP, not only the model file. On top of that sit teams on alignment, interpretability, and evaluations. Wiblin's first worry is speed: if capabilities jump fast, shouldn't Anthropic be practicing ASL-3 precautions on today's harmless models so it is not scrambling later? Joseph says they do need to plan ahead, and that tying safety to commercial goals is what makes ordinary company planning apply. The yellow-line evals are meant to sit far short of the red lines, opening a buffer. If that gap were a week of training, a trip would be terrifying. In practice, he says, the evals are set far enough away that there is time to adjust: maybe the eval was bad and you go to the board; maybe the model is dangerous and you turn on the precautions you already prepared; maybe you are not ready and the models are catastrophically dangerous, in which case you stop training, or you do not deploy until you know how.

Going further — actually running ASL-3 security on Claude 3 just for practice — would be safer in a narrow sense and, Joseph thinks, a political mistake. Applying extreme caution to models that are not dangerous means forgoing benefits people could have now, and it makes onlookers decide you are crazy. The RSP's bet is to tailor caution to measured risk, while still doing Safety Research that looks several levels ahead. On cost, Wiblin floats $100 million as a ballpark for a frontier training run and asks what the tests add. Joseph does not have exact figures; he thinks the dollar cost is low, mostly inference or small extra training. The real bill is people and time, especially at a company that is still relatively small. The goal is to minimise delay while remaining responsible. Delay for its own sake is not a virtue. Internal deployment risk — a model that exists only on Anthropic's computers, persuading staff or trying to escape — is mostly a higher-ASL problem. The RSP has not yet defined those controls, and Joseph does not recall a clause that says delete this model, because the safety-buffer idea is that you never train the red-line system in the first place. If you accidentally did, then you would have to think about deletion.

As an alternative to RSPs, the baseline is nothing, which Joseph rejects because the externality is too large. A pause is the other popular ask. It would make sense, he says, if you were already training clearly dangerous models; it is not a practical ask while benefits are obvious and catastrophic risks are not immediate. The labs that care most would pause, the rest would not, and you would get adverse selection. He would want a pause backed by clear evaluations, failed mitigation attempts, and a public case that everyone else should stop too. RSPs are not a substitute for regulation — they do nothing about labs that refuse to have one — but they are a learning ground. Policymakers can read the reports and design rules from what actually happened. Wiblin flags a clause that looks like a get-out-of-RSP card if a competitor without a serious policy is pulling ahead. Joseph says that is not meant as a free pass for commercial lag. It is a last-resort acknowledgment that an AGI endgame race could get high-stakes enough that proceeding anyway is the least-bad choice, not a loophole for skipping evals because they are annoying.

The objection Wiblin hears most, and the one he pressed after soliciting complaints on Twitter, is trust. Public confidence that AI companies will take a loss for safety is lower than when they still looked like research labs. The Anthropic RSP is full of phrases open to interpretation: harden security so non-state attackers are "unlikely" to steal weights, and so that access would not "substantially" increase catastrophic misuse. Someone has to decide what unlikely and substantial mean, and whoever runs the evals has discretion over how hard to elicit the capabilities that would delay a commercially important release. Joseph grants the criticism. His colleagues are talented and bought in, but a lot is left to good-faith interpretation. Whistleblower-style protections and public discussion can add pressure. "Ultimately I think you really do need regulation to have the sort of very strict requirements." He hopes the policy gets more concrete; the blocker is ignorance. Fake precision that later has to be walked back can cost credibility.

Wiblin's analogy is accounting. A major corporation does not get to follow its own books and call it done; external auditors are legally required. When a result might say do not release this, or even delete it, he would be more comfortable if a group with different incentives made the call. Joseph would also feel more comfortable. The practical problem is that there is no established profession of people everyone trusts to certify that a model cannot autonomously replicate across the internet. The auditor would need enough expertise to assess capabilities. Government agencies are another path; private firms following mandated rules are a third. Publishing evaluation write-ups, he hopes, can seed that market. Internally, the RSP is a formal policy passed by the board. Ignoring it would require the board, leadership, and employees to go along. Joseph says he would refuse to train a model that skipped it, and his team would refuse him. A legitimate amendment — an eval that turned out accidentally easy and not indicative of danger — is the flexibility they are trying to preserve. He cannot picture a plan that is just "there's a bunch of money on the table, skip the RSP for this model."

The Long-Term Benefit Trust is the governance layer Wiblin wants to stress-test. It has no financial stake and is supposed to keep Anthropic on its public-benefit mission; over time it will elect a majority of the corporate board, which is not the same body as the Trust. The elephant in the room is OpenAI: a nonprofit board that, when push came, appeared overruled by staff, public opinion, and investors. Formal org charts, Wiblin argues, are not how power is always exercised, and he would rather put his faith in external legal constraints than in interesting internal mechanisms. Joseph does not sell the RSP as a replacement for regulation. He thinks the Trust has real authority, and he also thinks culture is load-bearing: eliciting capabilities well enough comes down to a researcher actually trying. Even under regulation, the people in the room have to care. Both men land near defense in depth — good rules plus good people, each catching holes in the other — and Joseph adds track record. If Anthropic publicly weakened the RSP in a way that looked silly and added risk, people should lose trust.

Wiblin's next objection is a cheque that may bounce. At ASL-4 and 5 the RSP gestures at securing weights even against persistent, well-funded state hackers. We do not currently know how to do that, especially if the weights ever need to touch the internet so the product can be used. Solving computer security after decades of failure is not a job Anthropic can finish alone. When the time comes, the real choice may be a long pause while fundamental breakthroughs arrive, or weakening the RSP so the company is not stuck for years. Joseph's answer is that they should pause, put their effort into safety and security, bring in outside experts, and if that still fails, make a public case to governments and other companies that serious risks are imminent. People do not have to follow commercial incentives, he says; Wiblin could have made more money than hosting a podcast. The stakes would be extremely high. "It's important to just do the right thing." He also refuses a zero-risk reading: security always trades off against productivity, and the RSP will not demand a 0% theft probability.

Wiblin still finds the scenario plausible: months pass, other companies are less careful, the government does not ride in, and the choice becomes take the risk or become irrelevant. That, he says, is not a knock on Anthropic. It is evidence the problem cannot be solved by one lab, or even by all the labs together, without governments funding the security research the policy assumes. Joseph is more optimistic. Today's pause campaigners are easy to dismiss because most people do not believe the risks are real yet. At the point of a clearly dangerous model, no company wants to have caused a disaster and no government wants to have allowed one. Wiblin cites Stefan Schubert's "sleepwalk bias": the habit of imagining that future people will sit on their hands in front of an obvious problem. The last couple of years, he thinks, already show the public and governments getting more involved as capabilities rose.

Joseph's own biggest reservations are narrower. For Anthropic specifically, under-elicitation is the live one: there is no last word on a negative result, because a better researcher may succeed tomorrow. The other is unknown unknowns — risks they did not list, or that they filed under a later ASL and that arrive early. Elicitation has to happen inside the lab, with training access. Writing threat models and evaluations does not. METR, the UK AI Safety Institute, and the US AI Safety Institute are already in that business; Joseph tells early-career listeners they can also just post a threat model and share evals on GitHub. Wiblin asks whether RSPs should look more like aviation risk management: publish that experts think the annual chance of weight theft is x%, below an acceptable threshold of y%. Joseph likes probabilities as a way for people who work together to be precise, and distrusts them as public communication. Outsiders hear "7%" and assume a calculation; Anthropic has not watched a thousand governments try to steal the weights. Precise percentages also fail against a bad-faith operator, who can always claim the number is under the threshold when nobody can audit the claim.

The RSP's own metaphor is assembling an airplane while flying it. Unlike BSL-4, where the organism already exists and can be studied, Anthropic is specifying precautions for a system that does not exist yet, whose arrival time is unknown, while building the system. Mistakes should be expected, which is why both men keep returning to defense in depth. It is also why Joseph wanted the Claude 3 run as a practice lap even though he was "pretty confident" it was not catastrophically dangerous — slightly better than GPT-4, which had been out a long time without causing a catastrophe. Mistakes are cheap now and will be expensive later. Critics who want wider buffers worry about under-elicitation and about discontinuous jumps on particular tasks. Red lines are the actually dangerous capabilities; yellow-line evals are meant to certify that the model is far short of them. Anthropic also uses a cheap heuristic, "effective compute," to decide when to run the expensive elicitation suite: loss goes down with compute according to scaling laws, and a big algorithmic win that reaches the same loss without extra FLOPs counts too.

On autonomy, Joseph thinks the current tasks are already conservative: copy to a fully undefended machine, or add a simple backdoor to another language model. Set them easier and you will trip the line on something that does not warrant ASL-3. Thresholds around succeeding half the time are about reliability. Chaining many steps into long-running autonomous replication, against people trying to stop you, probably needs a very high success rate; a conservative bar catches the system earlier. Going from 10% success to 70% is often easier than going from 0% to 10%, because you can reward the successes and bootstrap. Wiblin cites a previous guest of the show who, in a blog post assessing the Anthropic RSP, argued that a fourfold compute (or data) jump between check-ins is not a large enough ratio if you have been under-eliciting. Overshoot, and you might be staring at a model past the red line and a pile of sunk compute, with incentives not to hit the emergency button. Joseph says they ran the suite partway through Claude 3 training, and the Claude 2 to Claude 3 jump was bigger than the gap they are now allowing. He calls it a judgment call they can revisit in public if it looks too big or too small. The ugly way to learn you set the buffer too tight is to miss a yellow line and then blow past the red line on the next iteration.

Wiblin also wants the scope stated plainly. The RSP is aimed at catastrophic, acute risks — CBRN misuse, models gone rogue — not at structural ones: job displacement, militarisation, algorithmic bias, deepfakes, misinformation. Those matter, Joseph says, but you cannot design evaluations that test the long-run effect of a technology on society. Anthropic has a societal impacts team and a policy team for that cluster. For listeners who want to help, he is practical. If you work at a lab, talk about what an RSP should contain. If you work in policy, read the published frameworks and steal lessons. If you do neither, write threat models and ship shovel-ready evaluations. Busy companies will run a test that is sitting on the shelf.

The last hour of the interview is a career conversation, which is the point of 80,000 Hours. Joseph interned at GiveWell, planned an economics PhD on global poverty, and spent years losing arguments with people who told him to worry about superintelligent AI. He left that summer thinking they were crazy, kept debating, and eventually pivoted, in part because the PhD path looked longer and less likely to work than he had expected. He did computer vision at Vicarious, an AGI lab that had shifted toward robotics. 80,000 Hours reached out and pushed him to work on safety sooner rather than later. OpenAI rejected him twice; Vicarious let him spend half his time reading safety papers; he wrote summaries for Rohin Shah's Alignment Newsletter to keep himself honest. On the third try he joined OpenAI's safety team and spent about eight months on code models, on the theory that models that write code are the first step toward recursive self-improvement. When the safety-team leads left to start Anthropic, he followed. At a company with no codebase yet, his comparative advantage was making training efficient. He moved into management when the pre-training team was five people, and has been growing it since.

His main disagreement with standard 80,000 Hours advice is the prestige gradient between research and engineering. Research, in his usage, is choosing directions, designing experiments, and drawing conclusions. Engineering is implementing the experiment, building tooling, and making the code fast and usable. An architecture idea may take an hour; implementing it may take a week. The place he has seen researchers abandon a direction they should have pursued is bad tooling: experiments hurt, so they go somewhere easier. Create the tooling and you blaze a trail. On his team he would guess 60 or 70% of people are stronger on the engineering side, and most interviews are aimed there — not because research does not matter, but because there is a bit less shortage of it. A software engineer with a decade of experience at a tech company, he thinks, can learn the ML they need very quickly. The historical split between research scientists with PhDs and research engineers who ran their experiments has already faded; at Anthropic everyone is a member of technical staff, in part because distributed training on big clusters made the engineering much harder.

The pitch for working on AI safety at all is the stakes. AGI, he thinks, will probably be the biggest technological change ever: every person able to spin up a million-person company of people as smart as the smartest people you know. That could cure disease and crush poverty, or go very badly, and the number of people working on it is still in the thousands, maybe tens of thousands. The pitch for Anthropic specifically is clustering. Five people in one place, sharing goals and trust, beat five people working in parallel at rival labs. Safety research is published in as much detail as they can give; beyond that it is hard to be accountable for what other companies do. On many ML papers, Joseph would expect a week of work to reimplement from the write-up, which is a different culture from biology, though production-model details (as in a recent interpretability paper on features) sometimes cannot be fully released. Open roles he flags: the RSP team, research engineers on his team, and performance engineers who make custom AI chips run fast. PhDs are not required; a surprising number of people come from theoretical physics and a few months of learning to write Python. His hiring advice is to do the job in public first — GitHub evals, interpretability tooling, a Common Crawl processing stack — and attach that as proof.

Wiblin puts on the table an earlier 80,000 Hours conversation with Zvi Mowshowitz, who likes the Anthropic RSP and still thinks no company's or government's safety plan is enough against rapidly self-improving AI. Mowshowitz was strongly against taking capabilities roles that push the frontier, especially at leading labs: those seats speed things up and leave less time to solve safety. Joseph is skeptical of the career-capital version of the rebuttal. In an exponentially growing field, the highest-leverage moment is now, not your forties. In 2021 he guesses there were on the order of tens of people working on large language models; now there are thousands. He has gotten better, but the potential for impact was probably higher when the field was empty. The field is also shallow: you do not need a thousand years of physics before you can contribute. If you believe timelines are five years, two years of skill-building is a large fraction of the remaining clock.

On direct impact he is less convinced by Mowshowitz's split. He does not see capabilities and safety as two tracks in a race. A lot of safety work depends on capability advances. Anthropic's many-shot jailbreaking paper used long-context models to find a jailbreak that works on Claude and on other systems; that research was only possible because the long-context models existed. If you do work on capabilities, he wants you to be thoughtful: ask a lab for its theory of change, talk to its safety team and to external safety researchers, look at governance and track record, and reevaluate. One thing that reassured him was watching the safety team help capabilities. Before Opus launched there was a "major fire." His team lacked capacity for a critical research project, so he asked Ethan Perez, a safety lead, for help. Perez and most of his team left an offsite, went upstairs in a building in the woods, and spent about two weeks on it. To Joseph that was evidence the safety team also thinks staying on the frontier is critical. He does not claim that is the ideal world in which everyone shares his views. You only control what you do, against a world that will keep acting as it acts. If Anthropic vanished, his guess is not that everyone else would slow down and become more careful.

Asked for safety wins he is proud of — not his own, he stresses — he starts with interpretability: from single-layer toy transformers at the company's beginning to features on production models, including the Golden Gate Bridge feature that, when amplified, makes Claude unable to stop talking about the bridge. Cuter than the safety-relevant features in the same paper, one of which flags incorrect or vulnerable code and can be edited so the model stops writing the vulnerability. That does not yet solve major issues, but it is a lot of progress he does not think would have happened without Anthropic's interpretability team. He also flags many-shot jailbreaking as an RSP-shaped finding: a reliable way to elicit capabilities you thought you had blocked. The sleeper-agents paper showed early signs of deceptive behaviour. Influence functions ask which training data produced a behaviour, the data-side twin of looking at weights. Constitutional AI trains a model from a written list of principles via AI feedback instead of human ratings for every case. The metric he wants for safety is the same as for product: outputs, not inputs. Close to AGI, nobody will care how many GPUs or people you used. They will care what you found.

Culture, as he describes it, is pair programming to a degree that surprised him. Early on, Tom Brown — a founder who had done this for GPT-3 — put an eight-hour meeting on his calendar and Joseph watched him code a method for distributing language-model training across GPUs. They used Tuple, so one person drives and the other can grab the cursor. It can be exhausting; the first time, Joseph was too nervous to take a bathroom break. Offices are in San Francisco, London, Dublin, Seattle, and New York, with a typical 25% in-person expectation, often one week a month, to build trust without losing people tied to a city. Growth has been rapid enough that they ran out of desks before a move. What is worst about the job, he says, is not a petty grievance. It is the intensity of the stakes, plus commercial timelines measured in months between releases. He does not think Anthropic is a "doomy" culture, and he finds the work fun day to day, but decisions feel like they matter. For applications, the overrated one is replacing Google for simple factual queries; he still Googles those. The underrated ones, for him, are coding, and questions that need a pile of context no search box will take.