Crunch Time: Ajeya Cotra's Playbook for the Months Before Superintelligence

This Cognitive Revolution episode is a cross-post from the 80,000 Hours podcast: Rob Wiblin in conversation with Ajeya Cotra. Host Nathan Labenz introduces her as someone AI insiders have long treated as one of the most rigorous thinkers about the future, a judgment he says she recently validated by coming in third of more than 400 participants in the AI Digest 2025 forecasting survey. He was proud to land in the top 5 percent at number 23. Cotra has done AI research and strategy since 2018, led technical AI safety grantmaking at Open Philanthropy (now Coefficient Giving) in 2024, and, Nathan says, now works on risk assessment at METR.

The phrase she uses for the next phase is crunch time: a potentially short window in which AI is powerful enough to dramatically accelerate AI R&D, but not yet totally beyond human control. Nathan warns that even accelerationists may get future shock from the conversation, because Cotra thinks it is quite plausible there are no insurmountable bottlenecks to widespread, compounding automation. If that is right, the world of 2050 could look as different from today as today would look to hunter-gatherers of 10,000 years ago. Aside from transparency and early warning, he reports, frontier developers are converging on a strategy of using each generation of AIs to align, understand, and control their successors.

Nathan signed the Future of Life Institute's October 2025 petition calling for a ban on superintelligence, not because he thinks that recursive plan is forever destined to fail, but because he worries we do not yet understand AIs well enough to bet the outcome. He still agrees with Cotra's practical advice: almost regardless of the work you do, adopt AI as aggressively as possible, both to keep an accurate picture of the situation and because you will not keep up without it. He would go as far as to say that even Pause AI campaigners ought to be using AI extensively. Two weeks after the original episode, on March 5, Cotra posted *I Underestimated AI Capabilities Again* on her Substack Planned Obsolescence, reporting that predictions she made in January 2026 were already being met. Anthropic's Mythos model, he adds, has reportedly found zero-day exploits in every major operating system and every major web browser. Crunch time, he argues, is arguably here now.

Wiblin opens by replaying how much of their interview from two and a half years earlier has since become ordinary. Cotra had flagged METR's evaluations of autonomous capabilities, now widely read in policy circles; probes that monitor and shut down dangerous conversations, now among the more useful outputs of mechanistic interpretability; chain-of-thought monitoring; growing situational awareness and deceptive alignment; the finding that training models not to misbehave can teach them to hide it better; and the expectation that models would get schemier and more sycophantic once reinforcement learning came back into the mix. He is not claiming she invented those ideas, only that she was ahead of the curve — which is why he wants the next set.

Cotra's working theory is that disagreements about almost everything in AI now rest on how much people expect AGI to speed up science, infrastructure, and manufacturing. She has been watching the word itself get more mainstream and more diluted. At a DealBook panel in New York last year, the moderator asked whether it was more likely than not that by 2030 we would have AGI defined as AIs that can do everything humans can do. Seven or eight hands went up, not including hers — her timelines are somewhat longer. A few questions later he asked whether AI would create more jobs than it destroyed over the following ten years. Eight of ten hands went up, again not including hers.

She was confused: how can you believe AI will do absolutely everything the best human experts can do in five years, and also believe employment will be a net positive over the next ten? When she poked people later, they backed off immediately. What does AGI really mean? We already have it; people keep moving the goalposts; we keep shipping cool products and the public will not accept the label. The old Singularity-era definition was an extreme thing. Venture capitalists, she says, have an instinct to call GPT-5 AGI, or something much milder. That creates a situation in which people feel they have a lot of evidence that AGI is not a very big deal, because they already have it, or had it two years ago, and look around: nothing much is changing.

Whether or not AGI arrives in the next few years, a lot of people are starting not to care about that question. They still expect the next 25 or 50 years to play out like the last 25 or 50: a moderate amount of technological change, with AGI as the engine of that continued mild improvement. Cotra thinks there is a pretty good chance that by 2050 the world will look as different from today as today does from the hunter-gatherer era — 10,000 years of progress, driven by AI automating all intellectual activity. On the far extreme is the worldview in *If Anyone Builds It, Everyone Dies*: a sudden crack of extreme superintelligence, humans like cats or ants beside it, and immediate physical-world impacts, classically nanotechnology and near-light-speed probes. AI futurists who worry about existential risk, she says, are united by expecting some version of that extreme progress in the coming decades.

Her modal picture is more specific. In the early 2030s she expects what Ryan Greenblatt calls top-human-expert-dominating AI: a system that does remote, computer-based work better than any human expert, from virology to software engineering. By then weaker systems will already have changed the world. After that, things can go much faster, because those AIs can use human physical labor to build robotic actuators and start closing the loop of making more of themselves. Whenever she checks on robotics, it is taking off for the same reasons as cognitive AI: large models, lots of data, imitation, scale. Tom Davidson's Forethought essay *Three Types of Intelligence Explosion* is the piece she recommends. Software that improves architectures and data is only one loop. AIs also run on chips printed in factories, on machines built by other machines, down to raw materials. Full-stack closure, not just the software stack, is the explosion.

The correlation between expected speedup and expected disaster is, she says, very strong. Reasonable accelerationists tend to think the default path is painfully slow and gradual, and that we should cut some red tape to get to a slightly more reasonable pace. People worried about existential risk think the default is explosive: society overturned on all dimensions in a year, five years, six months, or a week, and they want to slow it to something like ten years. Meanwhile the accelerationists think capturing the benefits will take 50 or 100 years, and they want to speed that up to 35. The policy prescriptions point in opposite directions. The destination both camps might actually want — a transformation that takes ten or twenty years — could be the same.

Among sensible people who have engaged the question, peak impact on economic growth ranges from 0.3 percentage points on current rates to a thousand percent a year or higher. That is a hundredfold to ten-thousandfold disagreement among people who have already talked, shared reasons, and not changed their minds. Cotra has spent the last couple of years having sincere conversations across that spectrum. She thinks the camps lean on two different outside views. The slower party points to 100 or 150 years of roughly 2 percent growth in frontier economies. Electricity, washing machines, television, radio, computers, and the internet all happened inside that band and do not show up as an uptick in the statistics. Mainstream economists like the stylized fact that new technology is the engine that *sustains* 2 percent; without it, growth would have slowed. AI, on this view, is the next thing that lets us keep chugging along.

Around that outside view sits a cloud of folk wisdom: things are always harder and slower than you think. Hofstadter's law says it always takes longer than you think, even when you take Hofstadter's law into account. Wiblin's favorite is the programmer's credo: "We do these things not because they are easy, but because we thought they would be easy." If you write down a clean story for how things go super fast, you have inevitably failed to account for bottlenecks and drag. The faster camp leans on a much longer economic history. Assign reasonable GDP numbers to the last 10,000 years and you see acceleration: growth was not always 2 percent; in 3000 BC it may have been something like 0.1 percent. Both camps agree the Industrial Revolution happened and lifted growth from well below 1 percent to about 2. The faster models say that was a feedback loop: more people, more ideas, more food, a larger population, repeat. If AIs and robots can replace the entire cognitive-and-physical package and close the loop of making more AIs and robots, 2 percent is not a law of the universe.

Why no convergence even after long argument? She is partial to the crazier side and will not claim a perfectly balanced account, but she has noticed that the slower worldview comes with a built-in error theory of the other camp: there is always a promised revolution, and the prophets have always been wrong. Someone could have made the same argument about television or computers. That prior makes object-level stories about robots bootstrapping robots feel illegitimate, because that style of thinking is biased toward ignoring friction. The faster camp has the opposite error theory: everyone blanket-assumes bottlenecks, and when you inspect the specific bottlenecks they might knock a thousand-percent peak down a bit, but they are not reasons to think the ceiling is 2 percent or even 10. The disagreement is decision-relevant. If the thousand-percent people became 0.3-percent people, they would probably change what they work on, and vice versa.

Sharing theoretical arguments has not been persuasive. Cotra wants empirics as early as possible. One step is characterizing whether, how, and why AI is speeding up software and AI R&D. METR ran an uplift randomized controlled trial, the first of its kind at this quality, splitting software developers into a group allowed to use AI and a group forbidden from it, then timing how quickly they cleared tasks. In that study, AI *slowed people down*. She does not expect that result to remain true, but she is glad the data is being collected, and glad people are starting to cross-check automated benchmark scores against in-context, real-world speedups. She wants more of every kind of evidence: big uplift RCTs, companies running internal trials on their own product rollouts, even limited self-reports.

The high-level formula is to look at the places where adoption has penetrated most and measure speedup in actual output variables. A solar-panel plant that had really adopted AI, and a record of how much faster or better it could manufacture, would be the sort of thing she wants. Chip manufacturing is probably the hardest manufacturing there is, so you might get an earlier heads-up from something more straightforward, but she would like monitoring across many kinds of plants. The thing she ultimately cares about is the AI stack: chip design, chip manufacturing, the equipment that manufactures chips, and software. The moment AI-futurist stories say things go much faster coincides with AI fully automating the process of making more AI. Earliest power users anywhere still matter, because insight transfers.

In late 2023 she put out a pair of requests for proposals at Open Philanthropy. One asked for difficult, realistic benchmarks for AI agents — not multiple-choice tests for chatbots, but real tasks: book a flight, write software, run the tests, iterate until it works. That arm got a lot of academic interest. They funded a cluster of benchmarks, including Cybench, a cyber-offense suite now used in standard evaluations. The companion arm asked for types of evidence other than benchmarks: surveys, RCTs, the messy measurements they had been discussing. It got much less interest. She reads that as a field habit: everyone agrees benchmarks overestimate real-world performance, and still treats them as the serious research object because they are clean and contained.

One thing she is excited about from the second RFP is the Forecasting Research Institute's LEAP panel — Longitudinal Experts on AI Panel — which asks 100 or 200 AI experts, economists, and superforecasters granular questions at six-month, one-year, and five-year horizons. Some items are benchmark scores. Others are things like whether companies will report slowing hiring because of AI, or whether an AI can plan a real-world event. Having people make subjective predictions, explain how those connect to longer-run worldviews, and then check who is right over time may be the most flexible tool available. The hard part is finding indicators that are early enough to act on if the worried camp is right, and valid enough that the other side cannot dismiss them as unrealistic.

One way this goes wrong is if leading companies see internally that AI is speeding them up enormously and decide not to tell anyone — and not to release the products. In the *AI 2027* scenario, the company far enough ahead keeps its best systems internal and releases only what is slightly better than competitors' public models. Wiblin would rather the rest of the world, and at least the government, get six months or a year of heads-up. Cotra has been thinking about transparency requirements that would force companies to release clues. There is a spectrum of evidence. The easiest to test and least informative is benchmark results, which companies already publish in model cards at product launch — Claude Opus 4's hacking and software-engineering scores, GPT-5's equivalent. In her ideal world they would also publish their highest internal benchmark scores on a calendar cadence, every three months, because danger can come from purely internal deployment. An agent good enough at AI R&D can make other capabilities, and therefore other risks, arrive faster than anyone expected.

Then there is information nobody currently reports. CEOs sometimes brag that 90 percent of internal lines of code are written by AIs. She wants systematic versions of those metrics, and a better one: what fraction of pull requests to the internal codebase were mostly written by AI and mostly reviewed by AI, with humans largely off both sides. Watching that number climb would be a joint signal of capability and of how much deference the company is giving the systems. If things are going to go crazy fast, AIs have to be doing most things, including management, approval, and review; if humans have to do that work, speed is capped. She also wants the most concerning misalignment incidents from real internal use — a model that lied about something important and covered up the logs. Benchmarks alone will not sound the alarm. They saturate, they have an S-curve, the next generation of harder tests is built, and even 100 percent on today's suites would not convince her a model could take over the world. The ultimate real-world measure is observed productivity: discovering insights faster than before. That is a late signal, and a clear one.

Companies already tolerate transparency more than other regulatory instruments, but the more informative numbers are fraught. Algorithmic-insight rate is competitive and embarrassing if it is too slow; investors lose heart. Safety incidents are worse. A third-party aggregator that published anonymized industry averages would help only so much, because there are few enough labs that people could guess. Wiblin notes that governments routinely demand commercially sensitive information — he once sat at Australia's Productivity Commission, which had rarely used but extraordinary subpoena powers. Cotra is skeptical that government-only disclosure is enough. AI safety research is being built while the plane is flying. It is not a box-checking exercise an understaffed technical office can do. The set of indicators people even want will keep shifting, and it will go better if there is a robust external scientific conversation about what would trigger alarm.

If she imagines sounding the alarm on an intelligence explosion, she pictures a society-wide conversation, closer to COVID, or to the weeks of public argument after Joe Biden's disastrous debate performance that led to him leaving the ticket. A small group entrusted with authority would have found that very hard to force. You want common knowledge, attention, and a chance for technical experts who currently think this is science fiction to jump in. It would be very powerful if someone like Arvind Narayanan, known for skepticism of these stories, looked at the data, changed his mind, and said this is happening and it is dangerous. Sending numbers only to governments is still better than sending them nowhere. She would not pick an all-or-nothing fight for the whole package, but getting the highest-value items is high on her list, and it is the strategy safety legislation has landed on: New York's RAISE Act and California's SB 53 are both transparency-oriented, with whistleblower protections as a shared plank. Rumors at San Francisco tech parties do not move DC, London, or Brussels toward costly action the way official, unrefuted, salient numbers would.

Suppose society does get a heads-up that an intelligence explosion is starting. A crucial variable is how good the systems are at everything *besides* AI R&D. The alarm has sounded: AI has fully or almost fully automated R&D at the leading lab, perhaps at all of them, and those labs are going much faster than in the human-driven era. Progress you thought would take ten, twenty, or thirty years might now take a year, two years, or six months. The systems may not yet be that dangerous, but the world is about to move very quickly from that point to godlike abilities. What she wants, if we gain confidence we are at the starting line, is to redirect as much of that AI labor as we can from further AI R&D to work that could protect us from future generations — takeover risk and a wide range of other problems. It is still not in the narrow selfish interest of whichever company is in the lead, because slowing down unilaterally lets someone behind catch up. A clear picture of six, twelve, or eighteen months until radical superintelligence might be a window to coordinate.

People who are not very worried about AI sometimes point to history: destructive technology has been offset by safety-increasing technology, and on balance things have gotten safer since 1900. Can we pull off the same trick in crunch time? A lot of people who *are* worried dismiss the plan as flying by the seat of your pants — expecting the thing creating the problem to solve the problem. Cotra thinks humanity has repeatedly used general-purpose technologies that way. Cars created carjackings and drive-by shootings; police also have cars. Computers enable hacking; they also enable automated monitoring and vulnerability discovery. You could not imagine a police force that did not use computers. The basic principle is sound: if you are worried about problems a technology creates, one of the first things on your mind should be how to use that technology to solve them. This window is unusually narrow. Cars did not bolt an acceleration button for the whole tech tree to the steering wheel. AI does. You have to monitor how long you have, and not blow through it.

Wiblin suggests the plan is especially unattractive if the early systems are misaligned, because then you are asking the AI to align itself. Cotra does not agree that misalignment bites only on alignment research. A hardened misaligned system that wanted the option of a bioweapon later would have as much incentive to do a bad job on biodefense as on alignment, and incentives to undermine AI-for-epistemics (a more rational civilization might notice the misalignment) and defensive efforts generally. The canonical “AI for AI safety” plan does *not* assume we know we can trust them. Figuring out a setup that uses control techniques, alignment techniques, and interpretability until we feel good about relying on their outputs is a crucial step, because it either bottlenecks progress — checking everything, slowing down — or it does not, and "we hand the AIs the power to take over."

The work she wants that labor pointed at starts with alignment: ensuring this generation, the next they help create, and the one after that, stay motivated to help humans, honest, and steerable. That is the foundation. Then broader societal defenses. If powerful AI will flood critical systems — weapons, the power grid — with newly discovered cyber vulnerabilities, use those same AIs to find and patch them before bad actors do. Biodefense: her colleague Andrew has an ambitious plan to rapidly scale detection of novel pathogens, medical countermeasures, PPE, and clean rooms; if the AIs are good at that research, and robots can automate manufacturing, the timeline compresses. More speculative, but still a kind of defense: using AIs to make collective decision-making wiser, better at finding truth and compromise policies people can trust, including reducing the chance of a US-China war and, more mundanely, reversing some of the degradation of political discourse after fifteen years of social media. Forethought's list of “grand challenges” — value lock-in, power grabs, space-resource sharing, epistemic disruption, Malthusian competition among AIs — she would lump, from today's vantage, under AI for coordination, negotiation, and truth-seeking. If you look at public communications from OpenAI, Anthropic, and Google DeepMind, this element is already in their stated safety plans: as AIs get better, they will be incorporated into safety more and more.

For the approach to make sense, there has to be a window *before* AIs are uncontrollably powerful, when they are already capable enough to change the game for safety research, and when that window lasts at least six months or a year even without a crazy slowdown. You have to be able to notice as you approach it. If hitting some generality threshold turns into crazy superintelligence in days or weeks, the plan does not work; you would probably not notice in time. There can also be unlucky orderings of capabilities. Imagine AIs that are savants at AI R&D — maybe only at making future generations more sample-efficient — and not good at anything else, including nearby safety research. You could have six months of crunch time with systems you cannot use for defense. In the worst version, a narrow search, almost like AlphaFold at proteins, hits an architecture or training strategy that then goes foom. You are counting down three months left, and these are not the AIs you can use for anything useful.

Companies are working harder to make models good at coding and AI research than at anything else, and those are more concrete, measurable problems than moral philosophy. Cotra already sees a specialized skill profile: AI helps her friends who do ML research all day much more than it helps her with weird thinking, podcasts, emails, and grant decisions. Fortunately, a big chunk of AI safety *looks like* ML research, and the friends getting large speedups include safety people doing control and alignment — among the first things you would want help with. AI for epistemics, moral philosophy, negotiation, and policy design may just not be that good. Another pattern: AIs will be much better at work with tight feedback loops, which is one reason they are so good at coding — a hard-to-fake signal of whether the code ran. Idea generation is easier to train than executing a one-year plan with thousands of humans and robots. A white paper can look neat; running the factory that makes ten billion widgets is another skill.

Why wait for the explosion at all, if useful AI labor should be used as soon as it exists? Two reasons, she says. At crunch time the default trajectory might be twelve months to extremely powerful, uncontrollable superintelligence, which changes the calculus toward short-term work. And by definition the systems are then good at further AI R&D, which you can try to point at filling out the skill profile — scaffolding and fine-tuning a biodefense model or a moral-philosophy model you do not have now. The plan is compatible with pausing at the brink; she would hope we do, because twelve months is not enough. She wants to make the pause less binary. Default is almost 100 percent of AI labor into making better AIs, more AIs, more chips. A stop is 0 percent. There is a spectrum in between, and the frame also answers what you *do* during a pause: the protective work, with those AIs around to do it. You might even go a little further on capabilities, especially if you tilt them toward biodefense, while the systems are still not uncontrollable. In an ideal world, a clear early warning would make her vote to stretch a one-year takeoff into ten, fifteen, or twenty years — slowly inching up, not pause-then-unpause into a jump.

If this is the companies' technical-alignment plan, why does she think it is most likely to have failed? Not, on her views, primarily because alignment turned out to be incredibly hard — though that is possible. The most likely failure is that they just did not actually do a big redirection. They say this is the plan, but they have no quantitative claims about what fraction of AI labor, or human labor, goes to safety versus further acceleration. They will face tremendous pressure from competitors to stay ahead. Unless they have much more robust commitments than they have now, a lab with 100,000 really smart human equivalents might have a hundred of them working on AI safety: more than before, not much compared with how fast things are going. Legal mandates, compute caps, or a coordinated 50 percent split run into the same problem: this is not a box-checking exercise. What counts as safety research, and who audits the team leads? It would take extremely technically deep regulators that do not really exist right now. Anything that requires a large corporation to be discontinuous faces big headwinds. She would rather they smoothly increase the share of internal inference going to safety as the AIs get better, so the final jump is not huge.

For Open Philanthropy the near-term work looks a lot like what she wants from society: track how useful AIs already are for the organization's work and its grantees. Push themselves and grantees to automate, and notice when AI gets good at what Forethought, Redwood Research, Apollo, or the policy shops do. Crunch time is not a reason to wait to do anything; it is the prediction that a lot of previously hard-to-automate work becomes easier. If AI is really good at math research, maybe technical grantmaking should shift toward more mathy bets because they become more tractable. Right now 80 percent plus of grant money goes to salaries for humans. In a few years, AIs may be better than most human grantees, and the money should mostly buy API credits or GPU time. They already pay for ChatGPT Pro and API credits at scale; she wants that fraction salient, endorsed, and climbing in line with capabilities, so that if crunch time starts in six years, inference is already a large share of spending.

Wiblin can imagine dithering rather than committing billions — terrified of going too early, looking foolish, spending $10 billion on a false alarm. Cotra thinks that is a very natural institutional failure, and not only from fear. Open Philanthropy's process is a junior person assembling materials, a manager, sometimes two, three, or four layers of cascade. You would not trust a random junior person to spend a billion dollars on a suddenly automatable strain of work; you might need a different process. Two worlds are possible. In the easy one, Open Phil itself is largely automated by then — junior hiring slowed, program associates are AIs, a thousand people on the AI team instead of 45 — and conviction is visceral. In the concerning one there is jaggedness: AI is extremely good at math, at certain technical safety, at some manufacturing useful for a PPE play, and not good at Open Phil's own jobs. The organization has no gut sense that it is time, and it is still the right move to pour money into a few automated verticals.

The buried lead for an external group is access. In *AI 2027*-style stories, the leader keeps the best systems internal and releases considerably worse ones. Or they will sell, but the price is steep because the opportunity cost is training the next model. The second problem is more straightforward to hedge: hold a portion of the portfolio exposed to compute prices. In the extreme, own GPUs, rent them out in peacetime, redirect them in crunch time, and cut deals to get the latest models onto those chips. Less extremely, hold Nvidia and other AI-exposed public stocks so a GPU price spike helps on the investment side. At the *beginning* of crunch time she bets things are still relatively commercial and open, leading companies within about a month of one another, spiky in different ways — what econ 101 and the present day would predict, unlike OpenAI's earlier lead. Super-exponential feedback is the force that reopens gaps: unlike exponential growth, where ratios stay fixed, a leader who hits a milestone first gets more and more powerful relative to laggards. Whether the leader then keeps capabilities secret depends on the competitive landscape. Close competitors and investors demanding quarterly earnings force access outward. A distant lead, and the chance of superintelligence that rivals nation-states, pulls toward secrecy — an evil strategy, she says, but "definitely in the water." The mundane competitor for AI labor is simply making the products and media people most want to pay for. Society does not spend a huge fraction of GDP on biodefense, cyber defense, or moral philosophy today. AI accelerates what people pay for, and this is not high on the list.

Even with enormous compute, you can only go so fast. LLMs emit one token after another; twice the compute does not complete an answer twice as fast without limit. Physical defenses — manufacturing PPE, scaling medical countermeasures — have calendar time in them. Social and policy work does too. AIs might be very helpful at figuring out a mutually beneficial, enforceable US-China agreement; human decision-making still probably requires officials to convene, talk, ratify, and feel good about it. Deep theoretical problems can be parallelized — many instances brainstorming, then evaluating one another — but the right solution often involves multiple dependent leaps. A hundred AIs in parallel still wait on the first insight before they can search for the second, and you may need three or four steps in.

So what has to be done in advance? In general, think about what the AIs of that time will be comparatively *disadvantaged* at. They will understand the situation better than we do now, think faster, move faster. What we can contribute are things that inherently take a long lead time. That includes physical infrastructure, like the bio setup her colleague Andrew is building out. It also includes social consensus. It takes years for an idea to become an accessible concept in society's toolkit: that maybe the United States and China should try a treaty that lets AI progress somewhat slower than it otherwise would, and spend a lot of AI compute solving the problems. Until that is in the water, people will not think to have the AIs go down that path and fill in the details.

Other organizations, especially governments, should be planning for the same adoption problem. Random pieces of red tape make it harder for governments to adopt AIs than for anyone in industry. We might end up with regulatees in fast cars and regulators on horses and buggies. More broadly, if your company works on defenses, AI safety, moral philosophy, or related goods and is not already going maximally hard on adopting AI for its own use case, it is probably worth having a team whose job is to notice the moment the tools become actually useful. Wiblin's list of tomorrow-actions sits in the same place: inventory the core steps of your work and test what share AI can already do; if you run a team, assign someone to watch adoption the way Cotra wants Open Philanthropy to; start the long-lead stockpiles, facilities, and ideas years before you need them.

Wiblin turns to the career since their last interview. In 2023 she was doing general AI research and strategy at Open Philanthropy; in 2024 she led technical grantmaking; toward the end of that year she took four months off. She had been at the organization more than six years before she led her first grant — she joined in 2016, and the first grant she actually ran was around the FTX collapse in 2022. From the outside, a philanthropy that gives away money, her work had been strange: heady topics, long reports on LessWrong. She always vaguely thought she should dip into grantmaking, the core product, and kept getting drawn into deeper intellectual projects instead. What pushed her in was the crash. Hundreds of people had been promised FTX Foundation grants that were not going through, or might be clawed back. Open Philanthropy put out an emergency call, and she had takes on technical research plus the organization needed surge capacity. In something like six weeks she made about 50 grants, after having made none.

She liked elements of it and hated how little you could dig in, especially in an emergency. She wanted technical AI safety grantmaking to have more inside-view justification for the research directions than “this person is a cool researcher who cares about safety” — a reasonable heuristic that she felt had driven a lot of 2015–2022 decisions. The portfolio was somewhat orphaned; people who had run it had left, some for the FTX Foundation. She tried to take it on in a novel way: form inside views about which technical directions actually addressed takeover risk. For her own research on timelines and threat models, months of effort get her to the point where she can have a long, intelligent back-and-forth with critics both more doomy and less. She wanted the grant program at that kind of reflective equilibrium, not just “outside view, we should support a range of things.” Open Philanthropy as a whole does not aspire to that confidence on most grants: it takes too long, money does not go out the door, and you still end up uncertain and spreading bets. She has sympathy for that, and still thinks doing the homework changes the details — including co-creating better versions of researchers' ideas.

She ended up in a barbell. Renewals and inbound asks got a canonical Open Philanthropy process and heuristics: aligned with reducing takeover risk, good track record, relatively fast. A second book of grants she would hold to her own standard. That second book became a late-2023 to mid-2024 bet on AI-agent capability benchmarks and other evidence of real-world impact — the RFPs discussed earlier. It was by far the narrowest and most deeply justified technical RFP Open Philanthropy had put out: opinionated about properties of a great agent benchmark, an application form designed to elicit the information she thought mattered, and a lot of “even if you think your tasks are hard enough, they are probably not.” They made $25 million of grants through that call and another $2–3 million through the companion RFP on RCTs and surveys. A skeptic could argue she could have put in less effort and funded twice the volume across ten areas.

Burnout arrived as the role collided with a management change. Around the time she switched into grantmaking, Holden Karnofsky, who had run the AI team and been her manager, stepped away. Their working relationship had involved a lot of arguing about the substance. Remaining leadership was stretched thinner and had less fluency with the AI material. She wrote a memo on understanding-oriented safety grantmaking and wanted object-level argument toward a shared organizational view. What she got was more transactional: you can do it that way if you want, with some skepticism. She felt lonely. She is not very entrepreneurial; she has a high need to be plugged into the central brain of the organization she is part of. Hiring under her to build a team did not fix it. The vision was nebulous, she was not good at hiring and management, and her perfectionism — a long history of trying to get writers to write up her ideas, then doing it faster herself — showed up with grantmakers too. She may have moved through the benchmarks RFP faster alone. In September she stepped away after eight years of full-time work. The sabbatical mixed ordinary life, a new group house, a briefly successful exercise habit, unpublished writing, helping stand up the first CURVE conference of AI skeptics and safety people, and a lot of reflection. She learned that without a path to influence the center, she will meddle anyway, and feel sad if she cannot.

On sabbatical she also asked what she had liked about effective altruism as an ecosystem and a mentality. EA, she reminds listeners, is the idea that you should think explicitly, seriously, and quantitatively about how to do the most good with a career or a donation, because paths can differ by orders of magnitude — green-technology research versus turning off your lights. The ethos is that taking helping seriously looks like the research you would do if your spouse had cancer, not like doing what instinctively feels good. She fell head-first into that rabbit hole at 13; it has been more than half her life. Three things really hit. First, EAs challenged themselves to care about beings very different from them in time and space: even “vanilla” global poverty, where most rich-country giving still goes to other rich-country people; animals; future generations thousands or millions of years out; AIs, if they can be conscious. Second, they were nerdy and methodologically inventive about which charities beat which, with a culture of open debate and admitting mistakes. GiveWell put a mistakes page on its website. Third, a high-integrity package that does not obviously follow from caring more: GiveWell refused donation matching because matching is usually a scam — the big donor would have given anyway.

The moral circle is still there. The intellectual depth is still there in other parts of the ecosystem, especially technical questions about controlling early transformative systems, which always pulled her heart even though she worked at a grantmaker. Wiblin jokes that she was a more natural grant *recipient*. If she had graduated in 2022 instead of 2016, she probably would have done MATS and joined an AI safety group rather than GiveWell. After Karnofsky left, demand for that depth evaporated a bit at leadership; if she were starting over she probably would not have applied to Open Philanthropy. The third pillar, the almost comically high integrity, was eroding. When EA's main method was convincing smart people to donate differently, unusual honesty was a valuable asset: early GiveWell donors liked the mistakes page because it meant they were not being sold something. Once you have attracted funders and talent and are trying to achieve things in the world, including politics, extreme transparency gets hard. Donors want privacy. Campaigns do not want opponents to know the strategy. The world's most impactful organizations are not consistently incredibly transparent.

EA told her she should care only about the goals — helping others as much as possible, not conforming to an aesthetic. She thinks she was kidding herself about how much of her motivation was pillar one, altruism, versus pillars two and three, intellectual depth and having absolutely nothing to hide. Those latter two became smaller features of pursuing EA goals in her career. The environment around Open Philanthropy also got more hostile as AI became a large industry: commercial interests, rival ideologies, people who would sit down and ask how to give the organization a terrible day. Even before the 2023 AI-policy heat, Open Philanthropy had compromised on wild early transparency ambitions — publishing grants it decided *not* to make, blog posts about the strengths and weaknesses of its first program-officer hires. She still wants that in her heart, and it is absolutely insane. Adversarial pressure makes a funder of a large ecosystem more risk-averse than many of its grantees. Her instinct is to say more and fight back; it is harder from Open Philanthropy's position.

Critics have long said EA has things in common with religious movements. She thinks the analogy fails in an important way: EA aspires to be, and largely succeeds at being, more truth-seeking than the world's religions. It is not a bad analogy in others. For people deeply involved, it provides a map of the good life, a vision of what it means to be good, intersecting with politics — gestation crates for pigs, for example — while also offering local community the way a church does. Wiblin experiences the professional-organization side more; she thinks it is a mistake to collapse EA into three cause areas, factory farms plus global poverty plus AI safety. It is a way of looking at the world, and you can take an EA approach to more parochial questions, including US policy. From her personal point of view, EA was not enough like a religion. She benefits from structure and emotional reinforcement, and her corner of the community's ideal is to have an impactful job, do it well, and work a lot of hours. She would have liked a more spiritual angle — the existential register of her colleague Joe Carlsmith's blog, grappling with a world that might be utopian or dystopian in a decade or two. An EA church on Sundays, someone thoughtful speaking and leading discussion, would have enriched her life and probably made her higher-impact. The professional community is deliberately not structured that way: if you are doing great safety research, they do not want to care whether you believe the deepest teachings. At 20 she had disdain for spirituality and unrealistic hopes that everyone good would get on board and solve poverty and factory farming. Doing the work taught her that everything is hard and slow, and Google Docs do not automatically connect to higher aspirations. Demand for a separate thing that reorients her to the bigger picture has gone up with age, the opposite of Wiblin, who wants to check out and be with family.

Toward the end of the sabbatical she planned to start a Substack and write about EA and AI, without a super-strong impact case — maybe, after that journey, she had more room to make a career decision on a basis other than impact. She stayed because Open Philanthropy was searching for a new director to lead global catastrophic risk work, the seat Karnofsky had left in 2023. Both top candidates looked good to her, and she thought someone new could use help from a person who was not running a program area and could help them get context and strategy. It was a chance to see if she could get the plugged-in feeling she had been missing. The director of GCRs is Emily Oehlsen, who is also president of Open Philanthropy. Cotra spent most of 2025 helping her understand what had been funded, the AI worldview, and sub-team strategies. She works extremely well with Oehlsen. Karnofsky had been a research PI: interested in her project, lots of bandwidth, not “I am doing this larger thing, help with this piece.” Oehlsen runs a more integrated ship. Cotra works more than she did just before the sabbatical, and it feels less hard.

She is talking to Open Philanthropy, still a top candidate, and to two technical orgs: Redwood Research, known for pioneering the AI-control agenda, and METR, which she thinks of as trying to be the world's early-warning system for an intelligence explosion. Both missions are close to her heart and narrower than Open Philanthropy, which would let her go deep in a way that would probably be more satisfying. The cliché a 20-year-old version of her would roll her eyes at: your extremely local environment, the literal person you report to, and whether you talk to people all day or work alone, can make a transformative difference. EA's declining transparency and the shaky bits of utilitarian foundations bother her more when the working environment is locally hard. She had been underrating the mundane and the micro. She was in the middle of a work trial at METR while they were recording, paying attention to how the rhythm and the people felt. A leadership change should be a trigger to rethink your role even if you do not leave. She is glad she took the four months rather than quitting outright, glad she came back to help Oehlsen, and glad she is now doing a proper search instead of healing in an unfocused way.

Should 80,000 Hours talk more about effective altruism, or is it right that the show almost never does now that it is more AI-focused? Cotra thinks it is correct that you do not need the whole EA package to worry about misaligned takeover, misuse, or disruption, and that there should be a thriving “AI is going to be a big deal” ecosystem that does not take EA as a premise. She also thinks EA thinking still has a lot to add. It will mostly be EAs who think seriously about whether AIs themselves are moral patients, with rights weighed against safety. It will mostly be EAs who take seriously the chance that disruption is so large we lock in a set of values for millions or billions of years. Even among people who accept that AI will disrupt everything in ten or twenty years, the ones thinking hardest about the most intense disruptions will be disproportionately EAs, because the culture challenges you to do far-seeing, rigorous speculation anyway. Digital sentience is a clean example: not particularly lucrative, not yet respected, not easy to make progress on, not the career your parents are proud of. A healthy EA community, in her picture, is an engine that incubates cause areas when they are speculative and methodologically unfirm, then matures them until they can stand on their own — the way it incubated worrying about AI takeover. Space governance and value lock-in are candidates for the same treatment. Some versions of lock-in go through an overtly scary power grab; others look more like social media plus-plus, superintelligent curation that lets each person stay inside a bubble and never change their mind, with power still distributed.

Wiblin notices that the distinctive niche she is pointing to is not primarily altruism. It is a research instinct: willingness to sit in the uncomfortable space between making things up and waiting for firm measurements, trying to make informed speculation rather than a good story or a refusal to predict. Even tame GiveWell cost-effectiveness analyses have to compare doubling a very low income to a risk of death using surveys and messy studies, then spell out the judgment. Redwood Research has the same habit. Inside EA, everyone looks at the person more speculative than them and sees castles on sand, and at the person less speculative and sees the streetlight effect. For people with that rare mentality, the implication is to take jobs other people will not fill. There is even a case that EA, as one piece of the world's response to crazy AI change, should be heavily indexed on research again. The community once had a glut of people who wanted to be researchers, then a push — including from 80,000 Hours — toward operations and policy. That was a good move at the time. If EA's comparative advantage is being the weird truth-teller in speculative territory, some of the people now in ops and policy who in their hearts want that work should consider going back.