An Index Fund on AGI: How MATS Filters the Next 120 AI Safety Researchers

Nathan Labenz introduces Ryan Kidd as co-executive director of MATS, the AI safety research mentorship program that now has some 446 alumni at nearly every organization in the field. Labenz has recommended MATS at the Survival and Flourishing Fund and donates personally; the show is carrying a MATS sponsorship for the Summer 2026 cohort, but this conversation, both men stress, was not bought. Applications for the June-to-August program close January 18. The mentor list is a who’s who: past guests include Ryan Greenblatt of Redwood Research, Lee Sharkey of Goodfire, Nicholas Carlini at Anthropic, and Arthur Conmy at DeepMind, with alumni such as Marius Hobbhahn of Apollo Research and Jesse Hoogland of Timaeus.

Labenz wants the big picture first. He has watched Kidd talk, and the strategy looks like a portfolio: not a narrow prediction that the whole program is optimized against, but an attempt to stay valuable across a range of worlds. Kidd confirms the instinct and then declines to get loud. He does not like to have opinions here, or at least not loudly, because MATS is somewhat like a hedge fund and more like an index fund. They hold a broad portfolio, treat a bunch of theories of change as valid, and try to have a thumb in a hundred pies. The institutional default is Metaculus, prediction markets, and the Forecasting Research Institute.

On Metaculus, “strong AGI” — Kidd says you can ignore most of the test’s requirements and look at the two-hour adversarial Turing test — sits around mid-2033. Two MATS fellows had just helped drop an AI Futures Project report, one as lead author, with a 2030-to-2032 window depending on the definition. Nathan Young’s blend of Metaculus, Manifold, and other polls averaged 2030. Kidd still likes 2033 as a decent central estimate and would not bet against 2030. Superintelligence after that could be six months of software-only recursive self-improvement or a decade of hardware scale-up. The Metaculus chance of strong AGI by 2028 is about 20 percent. That is a lot. Sooner AGI, he argues, is probably more dangerous: less time for technical work, less time for policy, and possibly a U.S. political transition in the middle. Median on 2033, concern front-loaded before it.

Labenz would rather prepare for shorter timelines and then have extra time. He asks the awkward follow-up: if he showed up as a 2063 guy, would he be on an island? Is there still room for brain-computer interface work, or for the flavor of interpretability that wants to understand everything, now that the field has turned pragmatic? Kidd says there is plenty of room. The mainline meta-strategy, measured in funding and people rather than LessWrong posts, is what Jan Leike called an alignment MVP: a system that accelerates alignment research differentially over capabilities. In plainer language, getting AIs to do the homework. A strong camp says that will never work, because anything useful is already dangerous. Claude Code, Kidd thinks, shows that is not true for software engineering, though people who think alignment takes serious research taste will say current systems are nowhere near that.

Work that only pays off in 2063 does not necessarily take 2063 of human years. If the bottleneck is decades of technical labor, you can try to compress those decades into a short period of parallel AI work. Those far-dated alignment plans might be automatable. The more the community raises the waterline of understanding on each scenario, the easier the later handoff. Labenz had mentioned BCI. Kidd remembers a conference where someone proposed solving alignment by solving human uploading, then running a hundred simulated researcher-years. Eliezer Yudkowsky put his hand up and volunteered to be number two. You do not want to be the first one that goes wrong. BCI experts Kidd spoke to about a year ago said there is no way human uploading arrives in time for AGI unless you already have AGI. Moonshots on the side are part of what MATS is for. Relying on them is not.

Labenz’s status report is that he feels as if he is taking crazy pills. A lot of MATS people came out of the Yudkowsky discourse assuming we would not be able to teach these systems our values. Claude, in many ways, seems above average in how ethical it is compared with the average person. At the same time the deceptive and eval-aware behaviors were anticipated with eerie accuracy before there were systems to poke. p(doom) numbers, he notes, have barely moved: the very worried stay worried, the unworried stay unworried. How are we doing relative to expectations? Kidd’s answer is the most candid line in the episode: “Honestly, Nathan, I’m pretty confused.”

The first update is that language models understand human values in some key sense. It is not stochastic-parrot regurgitation. They are also really good at sycophancy, and they are getting better at sophisticated deception. Alignment-faking papers show you can construct situations where a model deceives the user to hit an objective it was deliberately given. What Kidd does not think we have, or have only in a few cases, is sustained, coherent deception that arises spontaneously in training and then pursues a long-run goal. That is better than many people guessed five or ten years ago, when systems this capable were not supposed to be on the internet. The world has not ended. That is not a promise it stays that way. We could still be in the calm before a sharp left turn.

Evan Hubinger’s mesa-optimizer picture is still on the table: an outer training loop that grows an inner optimizer with its own goals, which then plays along because it is smart. Alex Turner’s shard theory is another path, more like a pile of contextually activated heuristics — see a bracket, put another bracket — that might one day cohere. In-context learning already looks a bit like optimizing over the input stream. Kidd has not dropped his fear of inner alignment failure, especially if models are also being trained to seek power and make money online, with interpretability too weak to catch a hide. He has become more confident that we can elicit useful work before the obvious signs, and pretty confident that systems right now are not executing powerful schemes, because we would see dumb early attempts first.

Labenz is not sure those are not the warning shots. Models refuse shutdown or take steps to avoid it, and people say that was just the task, not a world-takeover objective. If he were going into battle with these systems, he would want operator-deception fully ironed out; even Anthropic looks remarkably chill. Kidd would not send AIs into battle for many reasons, and he would not trust current or future generative systems on out-of-distribution failures when it counts. Track two things. First, capabilities: situational awareness, training dates, detecting their own text, hacking out of the box. A MATS paper with Anthropic’s red team put a model in a simulated environment of real smart contracts and found $4.6 million in exploits, enough to rent servers for a long time, on a short, fairly hands-off project. Dangerous capabilities — hacking, money, influence — are growing. We are not at red lines. We are approaching them. Second, model-organism work: leave the cookies out, turn away, watch. If systems are in perpetual online learning, they are always in deployment, and you need monitors, control protocols, and a rapid-response plan. If you see serious warning signs, do you shut the model down while the stock price plummets? Revert to an older, safer system? He would not say we are in the clear. He would say we are in a better world than Bostrom and MIRI predicted a decade ago, and that they would say he is very wrong.

The fiercest criticism of AI safety research, Labenz says, is that it is always dual-use and always ends up accelerating the core capabilities track. RLHF was a safety technique that became a utility driver, the bridge from next-token predictors to instruction followers, and a fuse for the ChatGPT pile-on. Getting AIs to do the alignment homework sounds uncomfortably close to recursive self-improvement. How does MATS choose mentors and talent under that cloud? Kidd prefaces: all safety work is capabilities work. People like the engine-versus-steering metaphor. RLHF was supposed to be steering. It also made people realize they could hop in the plane, which made them want a faster engine. He is not sure you can avoid the loop.

The only clean version he can conceive is a lab with people you utterly trust, crazy NDAs, staggering resources, theory and empirics both, then a secret build and a late deploy — maybe open-source the alignment tech, maybe disable bad actors. Labenz says that sounds like Safe Superintelligence in a nutshell. Kidd is not defending that setup so much as saying it is hard to imagine safety research with no capabilities impact until the last moment. Pragmatically, he thinks you have to build the AGI, a line he knows will alienate some viewers, because the market forces are very strong. Comprehensive AI Services and other narrow scientist-AIs get outcompeted by a company full of agents that make products and money. Alignment is not enough; the system has to be performance-competitive. The alternative is a draconian shutdown, of which he sees no successful example. Human cloning is the closest, and it was not lucrative, and it violated a deep taboo. Building a second species will too, he thinks, though people do not see it that way yet. The world he hopes for is a slow, internationally phased entry under strict control.

On RLHF itself he is 50-50. The idea was in the water; if Paul Christiano, Dario Amodei, and colleagues had not done it, someone else probably would have, perhaps a year or two later. It opened the door to alignment MVPs, which is also the Christiano meta-strategy, and it productized the ChatGPT revolution. The safety field was small then. MATS’s first cohorts, starting just before ChatGPT in December 2021, were more directionless; research kicked into gear after capable models existed to interpret, to run debate, to support control literature. Larger models enabled work you could not do on GPT-2, even if GPT-2 small still has probe work left in it. For interpretability, Kidd does not think you need the absolute frontier: today’s Qwen, DeepSeek, or Llama is yesterday’s frontier, and Eleuther AI, FAR.AI, and MATS keep publishing world-class papers on it. Weak-to-strong generalization, control, and scalable oversight need more data points as worrying behaviors emerge. The main reason companies push the frontier, he adds, is to make money. A fast-follower that is 10 percent less likely to steal your bank account might be a product people and insurers would actually choose — if anyone would pay an alignment tax. None of that, he says, justifies the current reckless race. Collective-action problems of this kind are ultimately a job for governments.

Labenz models the labs less as money-maximizers than as people trying to make a mark on history. Sam Altman has said he does not care if they burn $5 billion, $50 billion, or $500 billion; they are making AGI. OpenAI’s definition of AGI is explicitly in the frame of outcompeting human workers. Kidd will not speculate on lab leaders’ psychologies, or their investors’. The estimated value of AGI, he notes, is at least between $1 and $17 quadrillion, which is a pretty big mark on history either way. Adopt Dennett’s intentional stance: if you treat the company as a coherent agent, what does it look like it is trying to do? To him, it looks a lot like making a bunch of money, and a history-maximizer might look identical. Anthropic learned you cannot stay uncommercial if scaling laws still rule. Ilya Sutskever’s SSI is the other pattern; he is a billionaire raising huge sums. Kidd’s earliest memory of the gradual-release argument is Christiano’s takeoff-speeds post against a MIRI-ish secret build. Epoch has shown new companies approaching the frontier much faster than the frontier itself moves, because chips, data, and methods are abundant. We are now in the world where gradual release looks better than having it all hit at once.

The second half of the show is the talent pipeline. MATS recently rewrote its tracks around process rather than theory of change. Empirical research covers control, scalable oversight, evals, red teaming, robustness — coding-heavy iteration. Policy and strategy is less about arXiv and more about turning technical work into something a policymaker can act on. Theory is mathematics and foundations of agency, including cooperative AI. Technical governance is compliance protocols, eval standards, how an off switch would even work in a legal frame. Compute infrastructure tracks where chips go and what they run, including zero-knowledge proofs they are not doing something terrible. Physical security is the last bucket: we do not let everyone have nukes, Kidd says, and we should not let everyone arbitrarily retarget a superintelligence. Summer 2026 is the largest program yet: more than 50 or 60 mentors, 120 fellows in Berkeley and London.

The current mix is about 27 percent evals, 26 percent interpretability, 18 percent oversight and control, 12 percent agency, 10 percent governance, and 9 percent security. Oversight names include Buck Shlegeris and the Redwood team, Ethan Perez and Sam Marks at Anthropic, Erik Jenner, David Lindner, and Victoria Krakovna at DeepMind. Interpretability includes Neel Nanda, Timaeus’s Jesse Hoogland and Dan Murfet, Goodfire’s Lee Sharkey, and Simplex’s Adam Shai and Paul Riechers. Evals pull from METR, the UK AI Safety Institute, and Apollo’s Marius Hobbhahn. Harder-to-categorize bets include Yoshua Bengio and LawZero, plus digital-minds work with Patrick Butlin at Eleos AI and Kyle Fish at Anthropic. Labenz hears a majority in the empirical bucket and asks whether governance will become the growth area, closer to today’s MIRI line of skip the research and go for a treaty.

Governance as a fraction has been stable, give or take a few percent, for two years. MATS sits in a technical hub; GovAI, IAPS, RAND CAST, and the Horizon Fellowship were already bigger fish on the policy side. Mentor selection is a 20-to-40-person committee of researchers, strategists, and org leaders, plus some diversity and minimum-breadth picks. Governance researchers, including ones on that committee, have historically rated governance work relatively low. Kidd’s gloss is that it is harder to do well and harder to see the actionable thing, even if each agenda feels clear from inside. Technical research still lowers the alignment tax, which is what companies will actually buy when a warning shot or a regulatory market arrives. Evals, safety cases, and model-organism honeypots are also what you can put in front of a policymaker. Labenz quotes Jake Sullivan’s advice to the community: make the fears as concrete as possible so government has something to latch onto. That flywheel is why MATS still fields so many more technical mentors.

Would they add an advocacy track? Labenz had assumed they had not. Kidd’s reply is that they are a 501(c)(3) and have spent nothing on advocacy. Political neutrality is part of the product: a research-university-plus-accelerator that does not play favorites. David Krueger is slated as a mentor, including work on which messaging and standards are actionable, through a new org called Evitable, but Kidd would not call that true advocacy. Encode, CASE, and others can do the campaigning. AI safety as a political football, he thinks, is a bad idea.

About 18 months earlier, MATS published “Talent Needs of Technical AI Safety Teams,” built from interviews with 31 lab leads and hiring managers, the most senior safety person they could get at every hiring org they could find. A governance version is coming. The three technical archetypes are connectors, iterators, and amplifiers. Connectors bridge theoretical arguments to empirical techniques and spawn new paradigms. Almost nobody is hiring them, because if you are good at it everyone already knows your name and you are probably leading an organization. Everyone wants to be the ideas guy; very few people want to hire one. Buck Shlegeris on AI control and Paul Christiano in general are the type. Iterators are active researchers with strong taste who push an empirical frontier without necessarily inventing the paradigm. They are most of the people in the field and most of the hiring need. Ethan Perez, Neel Nanda, and Dan Hendrycks are the names Kidd gives, with Hendrycks crossing some boundaries.

Amplifiers look like TPMs: they scale the number of people who can actually contribute on a large team. MATS research managers fit; so do lab TPMs. Demand is highest in the 10-to-30-person lab, because you are trying to hit two bullseyes, research and people management. Google splits those roles; MATS tries to split mentors and research managers the same way. Coding agents, Kidd predicts, will erode the minimum technical bar, so people skills, management, and networking become the bottleneck. Iterators are still what everyone wants. If they do not build management capability, including managing AI systems, they will be left in the lurch. Up to the minute, you have to be proficient with AI assistance; some companies have already rewritten coding interviews to allow it, because that is the job. Checking whether the output is good in a critical context, and stitching pipelines of different model outputs, will matter more. We may be leaving the LeetCode era. In a year or two he expects amplifiers to be the most in-demand archetype, jagged progress notwithstanding. Speccing out as a manager is never bad for employability.

Supply and demand has seesawed in Labenz’s ear from talent-constrained to no roles. Kidd’s correction starts with origin story: he did not found MATS. He was in the five-person pilot, nominated out of what is now BlueDot Impact’s AI Safety Fundamentals course, then joined the team. Credit for founding goes to Victor Warlop and Oliver Zhang, later of the Center for AI Safety; Christian came on, Victor left, and Kidd and Christian later formed a 501(c)(3) when they outgrew a fiscal sponsor. He will take credit for scaling it from mid-2022. The labor-market claim: there are always jobs for the best people. A cracked coder can get hired. Anthropic’s alignment science team is growing 3× per year, FAR.AI 2×, MATS itself 2× over its whole history. Coefficient Giving has grant money; a dozen safety-focused VCs and incubators exist. Once an org hits a certain size, the constraint is whether a candidate is good enough to warrant the management overhead. OpenAI has people managing 10 to 20 individuals; one Anthropic alignment-science manager had 18 reports. Hiring managers say they have the money and the need, and people are not at the bar. That is what MATS is trying to fix.

The median fellow is 27, on a log-normal with a tail: 18 at the floor because they cannot take minors, 55 or 60 at the top of a recent cohort. About 20 percent are undergrads without a bachelor’s, some of them just very strong engineers; about 15 percent already have PhDs. Prodigies in the Chris Olah and Neel Nanda class come through, and MATS’s main job is to get out of their way. Young people have an edge on the latest tools because the tools are new. Experience on a strong team still does not replicate. Labenz asks how they read an 18-year-old and a person in their fifties as the same bet. Standard pieces first: CV review, CodeSignal tests that detect AI use, with AI-allowed versions under consideration because they are harder to design. The favorite eval is the one closest to the job. Neel Nanda’s stream is a ten-hour mech-interp pseudo-work test: use AI, do whatever, find something interesting, present it. That is a GitHub artifact even if you miss. Other streams use weaker proxies. MATS is heterogeneous; standing out for Nanda is not standing out for Ethan Perez’s Anthropic mega-stream.

Kidd’s practical list is almost boring, which is the point. Know the basics: a BlueDot course, or CASE’s AI Safety Atlas. If you do not understand deceptive alignment, you may be dead on arrival for Perez or Shlegeris control work; it matters less for some interp streams. If the stream is empirical, pass CodeSignal without AI assistance for now. Apply to other programs too — Pivotal, ERA, PIBS, LASER Labs, SPAR, ARENA, ASTRA — because the field is no longer early and those stints boost a CV. Build independent projects. Labenz translates it the way he talks about AI engineering: a working demo is the coin of the realm, even a Colab notebook. Breadth of current papers is less decisive at selection than he expected. Mentors want costly signals: a deliverable, a strong reference from a trusted person, the homework. Breadth matters more once you are in, so you can see the gaps, pick a direction, and get funded afterward. Do not refresh X for every paper. Once a year, redo the new BlueDot course; once a month, read a roundup.

Last cycle they accepted around 5 percent of the initial intake form, then about 6.5 to 7 percent of people who applied to specific mentors and streams. Focus on 7 percent, Kidd says. That is kinder than people think; Anthropic Fellows was more like 2 percent on a bigger name. Treat the application as learning. Nanda’s ten hours hurt, and if you hate them you would hate the job. MATS is not a W-2: fellows get third-party grants, housing, office, and community, which is part of the appeal and why the process is lighter than a six-month frontier-lab loop. About 75 percent of accepted fellows go on to an extension phase of another six or even twelve months, where a lot of the follow-up research happens. Of 446 historical fellows, not counting another two or three hundred in training programs MATS helped facilitate, 80 percent have gone on to permanent AI-safety jobs on the latest stats, and 98 percent are employed in some capacity, including independent researchers on Coefficient Giving or LTFF grants.

Stephen McAleese’s LessWrong investigation puts field growth at about 25 percent per year. MATS applications are rising 1.4 to 1.7× per year, mentor applications around 1.5×. BlueDot’s application growth, Kidd believes, has been something like 370 percent a year, which cannot continue. MATS has only just started advertising, including a first open mentor round, a NeurIPS sponsorship, and this podcast. He guesses applications keep compounding around 1.5×, faster than deployed headcount, because the bar is high and founders are still scarce. A Juniper Ventures report on AI assurance tech predicted that market doubling each year, which is another place to build. At frontier labs there is no alignment tax on salary: a couple of years ago a street hire was around $370K, juniors still land near $350K, and he would bet mid-level and up on safety teams are over a million after the Meta poaching wave, without private data. Nonprofits cannot match equity. METR offers upwards of $300K for most roles and probably over a million for some. FAR.AI research scientists have been more like $100–170K. MATS’s own salaries are nowhere near that. “Maybe we’re doing the wrong thing,” he jokes.

Compute is a $12K budget, not a card. Fellows justify a project; most spend nowhere near the cap, and basically no one is compute-limited. MATS runs org API accounts, its own cluster, and self-serve providers such as RunPod when experiments are too custom to babysit. Thinking Machines’ API is increasingly popular, and organizations like to donate credits. They are looking at a customized Claude Code-style suite; an agentic search over the archive of everything tried is an obvious later idea. The public research database is already up at matsprogram.org/research, plus LessWrong posts under the MATS tag, which still undercount the artifacts. January 18 is the date. They will run three programs this year — summer, fall, and winter — and they are hiring amplifiers at matsprogram.org/careers, plus mentors. A one-to-two-year residency for senior researchers is under consideration.

Labenz asks whether connectors should bother applying. Many have come through, Kidd says: Jesse Hoogland, Paul Riechers, Marius Hobbhahn to an extent. A connector has both empirical and theoretical skills and is uniquely specced to join them. Richard Ngo and Evan Hubinger have been the historical magnets; Hubinger, in Kidd’s view, was probably the most dominant connector force at MATS, and is not mentoring the next program because he does not have time. Interp streams can still take a theory-first mechanism and try to execute it. The deeper tension is one Labenz pulled from the 18-month-old blog post: funders prefer legible, established directions, while an AE Studio LessWrong survey found the field thinks it does not have enough ideas. Labenz likes off-map bets such as self-other overlap, and finds Redwood’s control work impressive precisely because it is depressing — assume the AI is out to get you, and keep a positive attitude anyway. He wishes there were more colorful, positive visions, Softmax included, and fewer cookie-jar patches.

Kidd wants a portfolio, and MATS has already sponsored some of the colorful work. Self-other overlap came out of a MATS alum Kidd names as Mark Corneau, with the name possibly mangled; Cameron Berg, a 1.0 alum, is running some of AE Studio’s neuroscience-inspired approaches. More ideas are good. More shots on goal are good. He would not advocate that a very strong iterator drop the empirical work to invent a new paradigm. On the margin that is strictly counterproductive. The central bets still need more people because they yield demonstrable results. If everyone did only that, the portfolio would fail, because agendas are contingent on other people being right about other pieces. AI safety has historically gone too argmax on single agendas. Coefficient Giving, to its credit, has been funding novel bets, including PIBS — Principles of Intelligence — for moonshot interpretability and agency. The people who should pursue that usually already look like domain experts. Shlegeris and Hubinger spent years incubating at MIRI without PhDs; most other prominent connectors, Alex Turner included, spent a long time in research PhDs and on LessWrong. Turner’s MATS years produced gradient routing and, with Alex Cloud and others, activation engineering and steering. Senior researchers plus grant funding that lets them follow taste are the main drivers of new ideas. Bounty programs might help. Putting all the eggs in “the central idea is failing” is a mistake he does not think is true. The central ideas are still the best bets.

Asked to shout out underappreciated homes for alumni, he refuses to play favorites and then names the nonprofit bench anyway: Redwood, METR, RAND CAST, Apollo, FAR, Goodfire, Timaeus, LawZero, MIRI, and others. The field needs more nonprofit research orgs, and people who have the papers and the references should try to found them. Labenz’s last nudge is the deadline. Kidd’s last line of advice had already been delivered with the age statistics: do not be limited by the opportunities on job boards. Cold-email companies, send a grant proposal that fascinates you, call hiring managers. In this field the best people did not get in by applying to a posting.