The intern who wrote GELU now wants humanity's last exam

Nathan Labenz opens with the Google Scholar ranking. Dan Hendrycks's most-cited paper is still a 2016 note on an activation function, written in the first week of his first internship, before he felt licensed to work on safety. He pronounces it GELU, "like Jell-O," and says creators get to decide. The formula is *x* times the cumulative distribution function of a normal, a smoothed stand-in for ReLU's hard corner. In the same paper he also named SiLU—*x* times a sigmoid—which, with GELU, he says is now the default activation in frontier Transformers: GPT, Grok, Llama, and the rest.

The résumé around that paper has gotten much heavier. Hendrycks is executive director of the Center for AI Safety, an advisor to Elon Musk's xAI, and, as Labenz notes, a technical force behind California's SB 1047. He also orchestrated the 2023 statement signed by the CEOs of OpenAI, Google DeepMind, and Anthropic: mitigating the risk of extinction from AI "should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." This conversation is his attempt to give the full worldview, from activations to geopolitics, without pretending the methods will definitely work when it counts.

ReLU is *x* times an indicator: a flat line glued to a rising line, with a kink at zero. Hendrycks's intern intuition, drawn from decades of machine learning, was that sharp transitions are usually worse than smooth, probabilistic ones. A step function's natural smoother is a CDF or a sigmoid, so he replaced the indicator and shipped what worked. He offers a signal-processing analogy—smooth filters beat sharp ones in audio, even if they cost more compute—and then immediately shrugs. Deep learning, in his view, is performance-driven. Theory is optional.

He treats activation functions as detectors. A neuron anti-correlated with "whisker" could be a plane, a chair, almost anything; the negative signal is noise, so you throw it out and keep the positive. ReLU does that with a cliff. GELU does it with a curve. Why the curve wins is not something he claims to know, even in hindsight, and he does not think the field should wait for that explanation. For a lot of his later papers, he says, it is still unclear what the method is actually doing. That is ordinary life with complex systems: inspection has limits, and you ship what moves the number.

Labenz asks about Kolmogorov–Arnold networks, the Max Tegmark–group work on learnable activations. Hendrycks is cold. Since 2017, attempts to beat the Transformer have piled up in what he calls a silent graveyard. When he started in 2016, LayerNorm, the Transformer, and a zoo of activations were still landing; GELU and SiLU more or less closed the activation line. State-space models exist, he notes, but industry barely uses them. Low-hanging architectural fruit has been picked. Any remaining gains will be high-hanging, and he will assume a tantalizing new architecture is nothing until it proves otherwise.

Deep learning, as he decomposes it, is algorithms, data, and compute; algorithms split into architecture and loss. On architecture he expects little. Mixture-of-experts, the one change he credits over seven years, maybe buys 30% efficiency—real, but not in the same league as 10× compute or 10× tokens. Plot open-weight models' benchmark scores against training compute and he gets roughly a 95% correlation, despite different recipes and post-training. Lab "secret sauce," he argues, is doing much less of the heavy lifting than compute and data.

The main algorithmic "advance" of the past seven years, in this telling, is humanity repeatedly learning to feed more data. BERT trained on a few gigabytes of text; RoBERTa used about 10× more; the field has been 10×-ing since, with Chinchilla as another reminder. Public text is now nearly exhausted. Whether synthetic data continues the trend is an open empirical question, not a slogan. He does expect some future change—maybe next-day prediction instead of next-token, better data, better objectives—but he does not expect many of them, and he does not expect them to rival raw resources.

Labenz offers a shrink-wrap metaphor: reality as a weird high-dimensional shape, the model as plastic pulled tighter around it. Hendrycks likes the multidimensionality. Humans, he notes, already have a strange capability profile: we talk constantly, yet transcribed speech is maybe half grammatical; we are decent at social modeling and terrible at arithmetic; vision is fantastic. Models should be expected to look equally weird—silver medals at the IMO while failing to count the *r*s in "strawberry." Attractive mathematical properties, equivariance, neurosymbolic aesthetics: computer vision spent years imposing those, and they were not necessary. A rising tide of scale fills in gaps. Look at the number.

The only analogy that has not led him astray is complex systems: cats, brains, economies, ecosystems. You cannot reason your way through the parts, because feedback loops produce qualitative surprises. He points listeners to a lecture on complex systems at aisafetybook.com, the textbook he helped write. Mechanistic staircases from neurons up, like particle physics to psychology, are theoretically possible and practically a bad way to get a working model of the mind.

He is equally skeptical that video or embodiment will punch through a data wall. Around CLIP, studies found that image understanding barely improved text representations. A gigabyte YouTube file transcribes to less than a kilobyte, and the talk is not a carefully edited paper. File size is not intellectual content. Helen Keller got very smart without vision; high-level reasoning does not obviously require a mind's eye. Sora-style world models may help physical tasks, he grants, but he doubts they will lift GRE Physics when the problem is already in text. Embodiment looks more like text helping robots avoid fire than robots teaching language models to think.

MMLU has 57 subjects—history, calculus, professional law, graduate physics, arithmetic—at a span of difficulties. Hendrycks built it because linguistic-understanding benchmarks were dying every few months. Models already had syntax, grammar, and basic English; "common sense" (light a match, pile of leaves, fire) had become a collective academic lie to deny. He had been making an ethics-understanding set, watched models get common sense, and reached for harder ethical material: tort and criminal law. Roughly 2,000 legal questions later, the obvious next move was to throw in every kind of human exam. The team copy-pasted for a month. Out came the dataset.

NLP researchers initially hated it. They thought it would bribe models to memorize the web instead of learning linguistics. Hendrycks's bet was that the field was already in a knowledge-and-exams paradigm, not a linguistics one. When they noticed models were worse at STEM, they collected MATH: about 12,500 competition problems. He now ranks the usage order as MMLU first, LMSYS Arena second, MATH third. All three, he expects, will be beaten in a year or two.

Other benchmarks he planted early. Machiavelli drops LLM agents into choose-your-own-adventure games and scores whether they lie, screw people over, or treat others as means when reward is on the line. Text adventures, he argues, get you most of the morally salient decisions without rendering Grand Theft Auto. Agent competence was not there when they built it; he now does less of that two-years-early research because it is riskier. The design philosophy still stands: measure propensities before you need them.

As of the taping, frontier MMLU sits in the high 80s and has been stuck there since GPT-4. Hendrycks's explanation is blunt: models have not become generally smarter. They format answers better, and LMSYS picks that up; MATH has moved on synthetic math data. But he knows of no model trained above 10^26 FLOP, while GPT-4 was around 10^25. A 10× larger model might reach the 90s and saturate the test. After that, MMLU becomes an efficiency benchmark, not a frontier guide.

Calibration numbers: aggregating 95th-percentile scores across MMLU's exams—bar, AP Calculus, graduate tests—lands around 90%. Individual Mechanical Turk workers get about 30 to 40%. Labenz recommends the site "Are you smarter than an LLM," where you discover how much economics and history you have forgotten. Musk's description, which Hendrycks endorses, is that MMLU is basically an undergraduate knowledge-and-skill test, with some graduate items mixed in. Models, he guesses, already beat any individual human across the whole suite.

Contamination is a weaker story than critics want. If labs had memorized the answers, scores would not still track difficulty: professional law remains hard, as it was two years ago. Scale AI built variant arithmetic items and caught a few models cheating; leading labs looked clean. Training on MATH's 7,500-example training split he considers reasonable—recollecting that set would cost millions—and it mostly teaches the boxed-answer format. François Chollet-style worries that models are running a catalog of stored procedures, he says, underspecify the critique. Michael Jordan said every game felt like practice because he had already done the move. A large, stable skill catalog may just be what a multipurpose system is. Students paying $20 a month to cheat with ChatGPT is, for him, the practical proof.

ARC-AGI's claim that progress toward AGI has stalled gets the same reply he had after GPT-3: people said he was overreacting, then there was a quiet year that was really Nvidia lead times and the wait to stand up a 10× cluster. He expects the next generation within six months. A data wall is possible. Until someone trains the 10× model and it fails, declaring a wall is too early. Reasoning, like speech recognition before it, is waiting on a reliability threshold. Five hard questions nailed in a row will end the vibe-check debate; he does not think reasoning has crossed that line, and he thinks it might within a year if the scale-ups work. He does not think a wholly new paradigm is required.

MMLU will saturate. Hendrycks's successor is Humanity's Last Exam: a call for experts, PhD students and above, to contribute one or two genuinely hard questions from their own research. Accepted items put the author on a paper he expects to exceed a thousand co-authors, with a prize pool of about $500,000 to $1 million—thousands of dollars for the best questions, hundreds for very good ones. The aim is questions harder than Olympiads, on esoteric topics with few papers and lots of unwritten relations, targeting current-model accuracy of 0 to 10%. GPQA, the closest mental model, is now around 55 to 60% with only about 500 items, so run-to-run noise is plus or minus 5%. He wants more sample size, more difficulty, and a private holdout, built with Scale, to catch cheating.

Not every benchmark should be hill-climbed. He is building a virology wet-lab test: show the model a photo of a Harvard biology PhD student's bench, state a goal in text, and ask what to do next with the Petri dish. Expert-level scores there would be a weapons-proliferation problem; that number needs to stay low. "Go make money online" as an agent benchmark, he notes, has an obvious fastest path: scams. Chatbots are still mostly better search. Agents open a legal can of worms. Criminal law turns on intent and knowledge; the U.S. bioterrorism statute talks about *knowingly* aiding terrorism. A model that walks someone through a bioweapon did not "know," and neither did creators who were not monitoring. Nobody is in trouble. Courts and statutes will be slow.

His working recommendation, including in draft advice to xAI, is to make industry treat "reasonable care" as a standard: models should not foreseeably cause the kinds of harms that land you in court—theft, not mere offense. Inherit ethics from cosmopolitan law instead of letting a handful of tech aristocrats invent morality for the world. Informed consent, not hiding things from operators, is another legal idea he wants imported. Machiavelli-style propensity tests and these dual-use evals are, in his phrase, seeds planted before the agent era; some of that research only pays off two years later.

Hendrycks's second-most-cited paper is a 2019 benchmark for image-classifier robustness under degradations. Vision robustness has been nearly hopeless: maybe a percent or two a year on toy tasks. Superhuman Go systems still fall to attacks no human would fall for. In a recent paper, "Safety Washing," he and colleagues asked which safety-flavored metrics actually track compute. Jailbreak robustness does not. Default scaling does not close the hole. You need an extra intervention.

Circuit breakers, with Andy Zou and Zico Kolter, is that intervention. Collect representations associated with expert-level virology (or other weaponization topics), then train the model to yank those representations in a random direction when they fire, while preserving ordinary behavior. Reliability, he says, went from about 90% to 99% with minimal capability cost—already in production-plausible territory for API models. They ran a bounty on 25 models, three of them circuit-broken, against several hundred anonymous attackers. Gemini, Claude, GPT-4, Mistral, Cohere and the rest fell quickly. After roughly three days and on the order of 20,000 attempts, the circuit-breaker models had not been jailbroken.

He puts two honest footnotes on the win. The eval is single-turn, not a long back-and-forth; a multi-turn contest is future work. And he would not be surprised if the models fell tomorrow. Tens of thousands of failures by competent people is still a good sign. For multimodal agents, circuit breakers will not make a dog/cat classifier robust, but they can make the system cease to function—refuse—when adversarial pixels try to elicit bioweapon or cyberattack instructions. Vision looked like a decade of despair. LLMs look up.

Labenz had filed representation engineering under mechanistic interpretability. Hendrycks does not. Interpretability, broadly, is understanding inner workings. Mechanistic interpretability is bottom-up: neurons, copy-paste heads, circuits glued into an explanation of math reasoning. Representation engineering is top-down. The unit of analysis is the representation—the population of weights and activations. Find a concept (honesty, harm, emotion), read it like mind-reading, then bend it like a crude form of mind control. Output-level methods such as RLHF and DPO constrain what is said. Representation-level control constrains the internals, which he thinks buys extra robustness.

The psychology analogy is the one he keeps using. To understand a person you do not start at particle physics, climb through chemistry and endocrinology, and hope to arrive at jealousy. Psychology has its own ontology. Mechanistic staircases between levels are possible—biochemistry exists—but they have not been a productive way to handle complex systems. Drug trials still require giving the drug to people. Philosophy of science, for him, favors functional explanations over mechanistic ones: the flower turns toward the sun *to get energy*. The chemical cascade is real and not what you needed.

Sparse autoencoders, which landed after the representation-engineering paper, look to him like a migration toward the same productive layer: extracted representations, not wiring diagrams. He recalls telling Nat Friedman it no longer looked like Anthropic was after mechanisms. The approaches are complementary. Top-down, you might have the wrong human concept; bottom-up, you might mislabel a feature or drown in geometrically packed, not-quite-orthogonal directions. He is happy they moved toward something that works. He would not call it mechanistic interpretability anymore.

Circuit breakers are a good fit for models behind an API. Fine-tuning removes them easily; refusal can die in as few as ten examples. Tamper-resistant training tries to close that. Simulate adversaries who strip the breakers, then train breakers that survive those attacks, inner loop nested in outer loop, in the spirit of adversarial training. Hendrycks was surprised for months that this worked at all, and that it generalized to unforeseen attackers. The hope is durable guardrails on open-weight models: the benefits without some of the weaponization risk.

He does not think about the method as shaping a smooth loss-landscape minimum, the way a Chinese-group paper he calls Sophon frames fine-tuning resistance. Late-2010s geometry of saddles and sharp minima did not, in his experience, pay rent. He would rather steal tricks that have worked: exponential moving averages, Nesterov-style momentum, a curriculum from weak to strong adversaries, more adversaries in the mix. The high-level loop—make the adversary's next fine-tune have higher loss—will probably stay. The exact terms will change.

The blocker for Meta putting this on Llama is not primarily compute overhead. It is the capability tax. Circuit breakers are nearly free. Tamper-resistant breakers look like adversarially trained vision models: a noticeable hit, not a ruined model, but a substantial peg. Not production yet. Many papers still have to be written. If the tradeoff comes down, labs could ship open weights with guardrails that are actually hard to peel off. Until then, he wants more groups on reliability and fewer on proofs under assumptions.

Biologist Michael Levin, a past guest, argues that in an adversarial world, systems that are too interpretable get parasitized, and biological complexity is partly defense. Hendrycks agrees with the "don't look under the hood" half: emergence is the point of a car, a Go program specified as MCTS, not as assembly. He then adds cybernetics' good regulator theorem: to control a system you need a model of the *whole* system, which is why theory of mind evolved. Complex adaptive systems harness stressors. Sitting-duck, fully transparent systems are easier to exploit. Security through obscurity has a bad reputation and, in practice, buys a lot of security. Closing the blinds on a new algorithm is obscurity. It still helps.

Mark Zuckerberg and Yann LeCun's line is that open-source software is the most secure software: many eyeballs, shallow bugs, overnight patches. Hendrycks says that logic fails for neural nets. Traditional software is written line by line; you can ship a Tuesday fix. Nobody ships "Llama 3.1, Tuesday update, patched three MMLU items." Label every neuron in a brain—whisker at 15 degrees, upper-left airplane wheel—and you still will not know what the person does in a new situation without running them. Understanding carpenters, plumbers, and IRS agents does not yield a model of the economy. "The system is its own best explanation. What it does is what it is." Open weights hand the adversary the full object, gradients included. Tamper resistance gets much harder.

He is, for now, pro open source. The line is expert-level virology. If a released model can walk *anybody* through a bioweapon, that is proliferation of a weapon of mass destruction, and society cannot bear it. Tamper-resistant training is the hoped-for way out. Until then, future releases need a cost-benefit test; today's models, he thinks, still net positive. Provably safe AI he is pessimistic about. Corner cases will be found by accident, not derived from a platonic ideal of the net. Nuclear plants are not proven never to fail. They stack preventive and protective measures until residual risk is negligible. That, not certainty, is the standard he wants.

Weaponization defense, in the picture he actually uses, is four imperfect pieces: an input filter for off-sounding requests, circuit breakers, output monitoring of users who keep asking weaponization questions, and know-your-customer so a banned account cannot reincarnate a second later. Together they can drive risk toward negligible. He is wary of the temperament that wants mathematical guarantees: the same people, he notes, bounced from evasion networks to neural tangent kernels to implicit bias in gradient descent because those topics fit a skill set, not because they were the most promising. He gravitates to unprestigious work—dataset collection, policy, field-building—and leaves a topic once it gets crowded. The neglected gaps, he thinks, are worldviews and strategies: a low-regret national-competitiveness platform, GPU verification for international governance, legal people in the room on agent ethics. Technical papers are harder and more oversupplied.

On xAI he is careful not to speak for Elon Musk. The draft risk framework he is imitating is the Defense Department's 2001 split into personnel, operational, and institutional risk, rewritten as machine-learning-system, operational, and societal risk. Musk, in this reading, is less moved by instrumental convergence—the textbook treats that as an open argument—and more by public epistemics during a fast transition. Grok being less censored, X summarizing news more objectively, forecasting bots that are 100× to 100,000× cheaper than prediction markets and in the same accuracy ballpark: those are societal-risk mitigations, not a paperclip of "truth maxing." Competitors, he says, refuse anything that might offend someone. xAI's current bar is legality—no copyright, no defamation—with reasonable care as the standard he personally finds attractive. Truth as a single closed-ended objective he will not endorse; Musk has also mentioned civilizational happiness and human autonomy.

On China, he thinks full coordination is not tractable without a catastrophe or a near-miss that institutions can see, because scenarios do not move them. Channel the energy into lower-regret options: a Western alliance rather than a U.S.–China package; subsidies for domestic chip plants so a Taiwan shock—forecasters and his bot both put invasion this decade around 30%—does not produce a Western compute collapse while China's foundries keep running. Export controls, in short timelines, look reasonable; the backfire mode is that they pushed China to chip independence early, so after a TSMC outage the West falls behind. Chip geofencing and registries are more contentious and possibly useful. The two-sentence extinction statement, he says, was mostly sequencing: who would sign with zero friends, who needed two, who needed seven. Blast-emailing everyone at once would have failed. The good regulator theorem, applied to people. He will not say who signed last.