The Q* Letter Never Happened: Helen Toner on OpenAI, Adaptation Buffers, and AI at War

Helen Toner is now director of strategy and foundational research grants at Georgetown’s Center for Security and Emerging Technology and the author of a new Substack, Rising Tide. To the public she is still the OpenAI director who helped fire Sam Altman in November 2023, then watched him return days later. Nathan Labenz, host of Cognitive Revolution, treats that episode as a burden she will always carry, then spends most of the hour on the work she actually wants to do: adaptation buffers, governance building blocks, and AI in military decision-making.

She joined the board in 2021, but she had known the company since it was founded in San Francisco around 2015 and 2016, when she was already in the city and turning toward AI policy. At the time it felt late. AlexNet had landed in 2012, and by 2015 the deep-learning revolution was in full swing. Further waves of attention arrived around 2018 and 2019, then ChatGPT in 2022. Looking back, she no longer feels she came to the party late. She had been working the problem full-time for five years before the board seat.

Founding a company to build AGI was, she reminds listeners, against the grain of polite society and of “impolite machine-learning society.” DeepMind was roughly the only serious research shop talking that way. By 2021 OpenAI already had GPT-3 in the API and a billion-dollar Microsoft investment, so it could have filled a board with conventional stars. Toner thinks she was invited because she had AI-policy experience when few people did, time in China, national-security background, and something rarer still: she had taken the AGI mission seriously for years.

Timeline language has drifted underneath her. In 2016, “short timelines” meant it was plausible enough to prepare for very advanced systems in a few decades, within a lifetime. Today the same label often means superintelligence before the 2020s are over, a claim she is much more uncertain about. She still refuses a camp. The consistent view is that the chance is high enough to warrant a lot of thought and preparation, which is different from betting it arrives in three or five years. Labenz’s asteroid analogy, once a way to justify a small watch, now feels to him like an object already incoming.

People keep waiting for a buried scandal. Toner says there isn’t one. As a director she still has confidentiality duties that former employees do not, live litigation can be mined for small inconsistencies, and many details sit inside private conversations that would drag in people who do not need to be involved. The most detailed interview she has given, she says, was last year’s TED AI Show. Even if she could dump everything, it would not flip the overall picture. “Boring detail,” not a second secret.

The one new fact she does add is about Q*. She stayed quiet so she would not get ahead of OpenAI’s reasoning work, later released as o1 and o3. Media reports that a reasoning breakthrough, and an employee letter about it, caused the board to fire Altman were, in her words, “totally false.” The board knew that research was underway. It never received a breakthrough letter. It did not make the decision based on a letter from employees. “That whole Reuters story was totally false.”

Labenz then describes a company that sounds almost split. In one stretch of days, researchers published a paper on obfuscated reward hacking that he ranks among the clearest warnings of how AI can go wrong, while the policy shop sent the White House a hawkish comment on China. Altman has said it is “our values or their values,” with no third way, and asked for training rights that would sweep aside ordinary property claims. Toner finds the contrast confusing, and more pronounced over the last year or two. Her guess is simply that employees have room to tweet and publish, and that different artifacts go through different processes. She hopes the technical staff are watching how contradictory the policy messages have become.

She will not diagnose a “hero quest” culture from the board table. Directors, she notes, are not immersed in the staff the way teammates are. Labenz worries that elite-performance coaching, detachment, and guru language look like an out-of-domain import from sports: keep shooting, trust the process. Toner reframes the out-of-distribution risk differently. Society often lets builders ignore regulation and downstream use, which is fine for a slightly better airplane wing. If technical progress outruns society’s ability to adapt, the people who stayed in that decoupled technical lane may be almost the only ones who can still affect the outcome.

What should someone do who is inside a frontier lab and uneasy? Toner starts with policymakers, because most whistleblowing law is built for illegal conduct. The SEC knows how to take a fraud tip. Many AI worries are not crimes, so the line for protected disclosure is mushy. The cleaner design, she says, is to pair whistleblower protection with a disclosure or process duty: a safety and security plan, published or filed with the government. Then the test is whether the company stuck to its own plan, not whether an employee “feels worried.” That is fairer to trade secrets and clearer for staff.

The user of that system is a technical person, not a lawyer, possibly scared and short on time. If eligibility and next steps are vague, the mechanism might as well not exist. For people still inside, she repeats a point she recently heard and liked: they may be at the most powerful moment of their careers, because they are working to replace themselves. Tech labor’s bargaining power is already falling. Sitting tight until things get “more serious” can mean sitting tight until you have less leverage, not more.

She also doubts the cinematic crisis will arrive on schedule. The more likely path is boiling-frog: this is worrying, that line was a bit dishonest, this risk feels a bit large. “If you’re only ever going to do something if there is some big moment,” she says, be realistic that you may never do something. Labor organizing and AI policy have barely met. She has not drafted a union platform, but she expects those worlds to collide and wants to see what leverage looks like when it represents a broad set of interests rather than a hotline.

On what the public actually needs to know, she likes sharing capability and risk test results and the processes used to keep systems safe, without a government checklist of required techniques. A joint piece by Daniel Kokotajlo and Dean Ball on publishing model specs, what a system is trained to do, also makes sense to her. The point is to shrink the information gap between developers and everyone else. Compute thresholds, in her version, are not a claim that models above 10^26 FLOPs are automatically dangerous. They are a way to put extra scrutiny on the newest, most capable systems, where unknown unknowns are largest. The U.S. reporting threshold is on wobbly footing at Commerce; the EU AI Act’s code of practice, aimed at models above 10^25, is under pressure to be watered down.

Adaptation buffers are the idea Toner wants to travel. Society, she thinks, has a decent record of digesting technologies that first look like the sky falling: printing press, telephone, television. Successor-species questions may be different. Misuse is where she thinks the policy world overstates novelty and reaches for a lockdown. The instinct is that an open model which helps with bioweapons or sophisticated cyber operations is simply too dangerous to spread, so the answer must be nonproliferation: keep access inside a tiny, controlled set of people.

That instinct collides with a dual dynamic. Training the next frontier model keeps getting more expensive in compute and in the need for a top research team. Every time a capability is reached, though, the first copy is costly and later copies get cheaper. DeepSeek did not push the cutting edge. It matched what some U.S. labs had done, a month or two earlier on reasoning models and more like six to nine months on the base model, at a lower price. People can fight about the exact dollar figure. The direction is the point.

Nuclear nonproliferation works, she notes, because bombs still need a lot of highly enriched uranium and specialized plants. Imagine that requirement falling every year until the IAEA has to inspect farmhouse soil, because a few acres theoretically hold enough uranium for a tiny bomb. That regime would not survive. AI looks like the version where the “uranium” gets cheaper. Trying to lock it down forces the policy to become more invasive over time. The alternative is to spend the window between first expensive demo and cheap ubiquity on resilience: vaccine manufacturing, wastewater monitoring, test kits, and the analog investments on the cyber side.

She pushes back on treating this as only a technical offense-defense race. Whether AI helps hackers or defenders more, vaccine designers or bioweapon designers, is one slice. A lot of bioterror prevention is who the FBI is watching. A lot of cyber defense is dissemination: ready-to-ship tools in the hands of people who run water plants, the grid, and chemical facilities and who are not AI-sophisticated. Even if AI eventually favors defenders, the lag from model release to a teenager’s attack is much shorter than the lag to thousands of operators cleaning legacy IT/OT codebases. Thinking of that interval as a transition, not a permanent nightmare, makes the response saner. She is not sure the buffer will be enough if progress is extremely fast. She still thinks it is healthier than pretending the technology can be bottled.

Asked for a regulatory proposal she likes, Toner is blunt: she has not seen a comprehensive regime for frontier risk that satisfies her. The problem is large, poorly specified, and timed to an unknown clock. What she likes are building blocks. Dean Ball’s regulatory-market idea, in which government accredits private regulators and developers earn a liability shield by keeping them satisfied, is better than nothing and designed for states rather than a deadlocked Congress. Transparency, measurement science, interpretability, alignment research, and more technical talent inside government do not solve the thing, but they leave more actors able to act later. She doubts any of this fully counters the financial pressure to race.

Iterative deployment, OpenAI’s old idea of shipping along the way so the world can adapt, now looks fragile. Ilya Sutskever’s new company says it will release nothing until superintelligence. Altman, launching GPT-4.5, told users the model was compute-hungry and might come offline if people did not love it; OpenAI later said it would deprecate it from the API. Labenz floats a relative speed limit: you may only train a model some multiple larger than what you have already deployed. Toner likes iterative deployment if you also have criteria for when iteration is no longer safe. She is pessimistic about input slowdowns, pauses of N months, or rules that need federal law. Her preference is conditional: do not proceed until you hit a stated level of understanding or risk mitigation, the kind of if-then many labs have already volunteered.

The deeper barrier is not only disbelief. A friend who has worked the Hill for years told her the House is less functional than at any time since the Civil War, independent of AI. Some people are drafting a Patriot Act analog to pull off the shelf after a crisis; she thinks that preparation is rational, even if getting good law through afterward is still a lift. Three Mile Island is her warning to developers. Post-accident nuclear rules were, in her telling, heavier than the comparative risk and helped freeze an industry. Without earlier guardrails, AI is on track for a knee-jerk overreaction, and it may already be too late to prevent that politically. Which incident lights the public up is unpredictable. After GPT-4, Kevin Roose’s Sydney chat caught fire even though people who read the transcript thought he had led the model there. Voice cloning that lets a user phone someone as Trump or Taylor Swift is vivid in the same way, whether or not it is the risk insiders rank first.

Then there is China, the sentence that can still unite a broken Congress. Toner rejects the starting point that Washington and Beijing should be friends unless proven otherwise. International-relations default, she says, is hostility between an established power and a rising one; the U.S.-British handoff was the rare exception. The U.S. spent decades trying to usher China into a rules-based order as a “responsible stakeholder,” climaxing with WTO entry, then watched that project collapse after Xi Jinping took power in 2012. The awkward twist is that Washington is itself sliding back toward might-makes-right. Labenz notes that Altman and Dario Amodei both used to call an AI arms race with China one of the worst outcomes, and now talk about sprinting, even unipolarity, while Chinese labs open-source. Toner’s reply is cold: open-sourcing is rational for a follower; it is much harder once you lead. The CEO hawkish turn is “the path of least resistance.” The one point of agreement in a messy debate is “we have to beat China,” and companies can cash that line for funding, defense contracts, lighter regulation, and liability shields.

Toner’s latest CSET paper, led by colleague Amelia Probasco, is about AI for military decision-making. Probasco is a Rhodes Scholar who served in the Navy operating Aegis missile-defense systems. That biography is the first correction the paper offers: strictly speaking, autonomous weapons already exist. Aegis, conceived in the 1960s and 1970s and fielded in the 1980s, automatically identifies incoming missiles and shoots them down. It does not use deep learning. Debates that get stuck on killer robots miss a much larger category already being sold.

Decision-support systems help commanders decide. On the simple end, computer vision can compute viewsheds from a photograph, showing what a sniper could see. Other tools do battlefield medical triage, or anomaly detection on ship tracks, a task that has improved sharply over the last decade or two. On the complex end, Palantir and Scale AI have advertised LLM “battle buddies.” Early marketing videos stacked a long list of functions; later campaigns looked more restricted. Some of those functions are ordinary: a natural-language query against a documented database, “where is my nearest friendly unit” or “how many missiles of this type do they have.”

The same demos then pivot to “generate three non-escalatory courses of action.” Toner wants to know how an LLM knows what is escalatory, who evaluates the consequences, and why that task is being mixed with a database lookup. Other clips ask the model to read a foreign government’s writings and infer intent. The paper’s purpose is to say that decision support can reduce fog of war and still go badly, and that demos currently blur those cases on purpose. Labenz, an enthusiast in other contexts, says he would not take an AI buddy into combat until scheming and reliability were clearly solved. Toner agrees the China race talk is decoupled from the systems we actually have.

They offer three tests. Scope: is the system tightly bound and testable for a task, or a sprawling chat interface for whatever a commander types? Data: does the training distribution match a novel battlefield where an adversary is actively trying to fool you? Human-machine interaction: does the interface help the operator see what the system cannot do, and avoid overtrust? Military UI, she notes, has a history of being undervalued and then contributing to unwanted outcomes. Those three questions, she implies, travel well beyond the Pentagon. They are how you should look at any AI product you are about to buy.

Labenz’s nightmare is “AlphaGo for the army”: self-play in simulation until tactics outrun human commanders, speed included, a path he analogizes to Skynet. Toner does not buy the near-term version. A paper by Dan Hendrycks, Eric Schmidt, and Scale AI’s Alexandr Wang on mutually assured AI malfunction gets half a nod. The useful core is that racing to superintelligence and announcing world-rule invites sabotage, and that wrecking a project may be easier than finishing one, which can stabilize behavior and puncture the fantasy of a clean sprint to victory. It does not have Cold War MAD’s clarity. We knew what nukes did. We do not know what superintelligence looks like or how much it is worth strategically.

Simulating war well enough to hill-climb past humans looks intractable to her. You would need the dynamics of a single battle, the theater-level movement of forces, the global economy and politics, and public attitude, while an adversary throws a wrench into your assumptions. People are building such simulations; they can help in limited ways. They are not a board game with a perfect score function. Unknown unknowns dominate.

She borrows a contrast from David Chapman between rationality and reasonableness, the latter being how you actually cook breakfast: start, look, adjust, rather than write a ten-step plan. Technical cultures, Chapman argues, treat the messy world as a noisy approximation of chess, Go, or StarCraft. Toner says the order is backward. Those clean games are extremely special cases of the real world. Reinforcement learning is good at the special cases. Today’s models still struggle with practical troubleshooting and mid-course correction, which is most of war and most of life.

Asked for a stable equilibrium, she declines. She does not know what the nation-state looks like in a superintelligence world, or democracy, or the Chinese Communist Party. More questions than answers is, she says, the honest position on this timeline, short or long. Labenz thanks her for staying in the arena after OpenAI and for still trying to make the future survivable. That is where they leave it: not with a doctrine, but with a former director who will talk about Q* now that the models are public, and who wants the next fight to be about buffers, interfaces, and the difference between a demo and a decision.