AGI Has One Problem Left: Spend a Million Times More on a Token

The episode opens on the claim Magic’s founder will spend the rest of the hour defending. What remains to be solved, Eric Steinberger says, is general-domain long-horizon reliability, and he thinks you need inference-time compute to get it. When you prove a new theorem, write a large program, or draft an essay of real complexity, you do not emit tokens one by one. You sit with some of them. Finding a way to spend not 1x or 10x but a million times the resources on those tokens, in a productive way, is “probably the last big problem.”

Host Sonya Huang of Sequoia Capital’s Training Data introduces him as a researcher-founder with an unusual origin story: a Vienna-born wunderkind who caught Noam Brown’s attention while still in high school and is now building an AI software engineer. Steinberger’s version of age fourteen is less glamorous. He calls it a midlife crisis. He spent a year sampling physics, math, biology, and medicine, looking for something worth doing, and then bumped into the idea of AI. If you could build a computer system that did all the other valuable things, you would not have to choose. Decision paralysis resolved. He says he could see the next thirty years unfold, and he likes predictability.

Math was what his brain gravitated toward. AI was what looked useful. Seventeen-dimensional spheres, he jokes, are a poor career if usefulness to humanity is the test. The choice was already clear a decade ago; it just was not close. Now it is close and clear. He did not learn to program out of curiosity about computers. He learned because he wanted to solve AI, and for a long time he was bad at it. The first paper took two months to understand.

The break was a giant email to Johannes, a DeepMind researcher who had been one of David Silver’s PhD students — Silver being a co-founder of the AlphaGo effort. Steinberger asked him to spend a year bashing his work every two weeks, a compressed mini-PhD, and listed ten ideas for beating the algorithm in Johannes’s dissertation. Johannes agreed, held him to adult standards, and was not nice just because the mentee was in high school. When the project finished, Steinberger had just graduated. Noam Brown, who had published Deep Counterfactual Regret Minimization, reached out because Steinberger’s Single Deep Counterfactual Regret Minimization beat it by a little. Brown had written his paper in about three months; the high schooler had spent a couple of years. They worked together for two years. Brown was then at FAIR, Meta’s AI lab, and Steinberger became a part-time researcher there in his first university semester. Brainstorming with Brown, he says, was ping-pong: a problem that might have been a six-month research bet would be discussed on a call and done.

What makes Brown great, in Steinberger’s telling, is meta-level taste: picking the right problems, grinding until the result compounds, and making bets that are not obviously right when he makes them, because he makes them early. He is also simply smart and, during his PhD, a 100-hour-week worker. Huang then stacks the rest of the plate. While collaborating with Brown at FAIR, Steinberger started ClimateScience, a climate-education NGO that became, in her words, incredibly popular. That is when he dropped out. After three months of university, research, and the nonprofit at once, his conclusion was: “I can do two things. I cannot do three.” Elon can, he says; maybe he will learn in ten years. At the time he could not.

He thinks starting a charity should carry the same status as starting a company. ClimateScience was run like a startup: people worked insanely hard, objectives were clear, the product was software. The exception was that no money came in and no money went out. About a thousand volunteers and twenty hardcore engineers, and he says there was almost no transfer from that experience to running Magic. He handed the nonprofit over because his premise had been wrong. He would never have started anything else if he had thought AGI was this close. Climate and other issues matter to him, but none of them is his unique calling. Once it was no longer twenty years away, there was nothing to hesitate about.

Internally they joke about an AGI to-do list he has carried around for years. A 2017 version, presented at a conference while he was still in high school, was wrong and he knew it. One item stayed stubborn: how to make models reason in the general domain. He does not like blue-sky research in the sense of staring at a wall for the right question. He wants the question first, then the answer, because an intense project needs a direction to run. Language models, especially code models, made that item look solvable. The proposal is almost embarrassingly simple: take LLMs and make them do the RL stuff. Reinforcement learning has worked for a decade in restricted domains. If you can make something work in twenty restricted domains, and you have something else that works in a general domain, combining them gives you both axes — the top-right corner of the matrix. Details of algorithms and efficiency remain. The direction, he thought, was finally clear enough to found a company.

Huang asks for advice for people in his shoes. He can only speak for his own profile: he knew at fourteen exactly what he wanted, had no Plan B, and treated every other path as below the neutral line. Burning the boats early, he says, lets you do at nineteen what other people postpone a decade. He was a terrible programmer for a long time, but a merely decent teenage researcher is already enough to earn a mentor who will bash you for a year. Go for the end goal immediately. Doing a PhD because you think you need the credential is, in his words, all nonsense. The other tactic that actually works is the five-page email. He now receives plenty of two-paragraph notes and finally understands why people think cold outreach fails: two paragraphs fail. “Here is how I am going to beat your algorithm, please help me,” at five pages, has gotten a serious reply every time he has asked. Great people will not coach ten students. They will coach one or two. You have to win that seat.

He also refuses to stay shy of the stack. Magic interviews researchers who cannot say how they would implement an LLM, which he finds insane. Learn top-down: here is the problem, what do I need, what do I need, all the way to kernels and hardware. He will never be the best kernel programmer or hardware designer; he needs to understand enough to do better work at the top. Then read everything. Bill Gates once said that once you have learned enough things, they start to resemble one another and the curve flattens. Steinberger took that as an order to read every paper he saw. Speed comes. You build a database in your head. Several of Magic’s ideas that work well now, he says, depended on some random prior result he would never have invented in tandem. One good idea is often enabled by three other people’s ideas. Without inventory, you cannot compose.

There was a formative near-quit with Johannes. They had a mediocre version of the algorithm, both disliked it, and the mentor was close to giving up — maybe this was not going to work, maybe it was not worth publishing. Steinberger’s response was essentially: watch me. A month or two later they had it. He remembers a walk home asking whether he could actually do this, concluding there was no other option, and sitting down to program while still sad. The next day was fine. He says he has not felt that pitch of doubt since, which turned out to be useful. Be brutally honest with yourself about what you suck at, or you will never get good at it. Betting on strengths is fine only if the necessary conditions are not missing. Engineering ability, for a researcher who trains giant models, is one of those conditions.

Asked for his superpower, he credits synthesis first: this paper plus that paper plus a modification. After noticing that, he tried to get better at leaps — alien ideas with no reference. The Transformer is his example of mostly synthesis. Attention existed, stacking blocks existed, residual updates came from ResNet; you had to drop recurrence and add a few new pieces, including normalization tricks, for it to work. Most good ideas are synthesis with a leap inside the best ones. Magic’s own recipe, he admits, is still mostly synthesis: efficient long-context LLMs, throw RL on them, make it all run. The people he admires most cluster around that skill. Noam Shazeer, a Transformer co-author and Character.AI founder, is “the best at synthesis,” full-stack with no real weakness, able to implement and run the whole system, and unconstrained about the future. Steinberger cites a YouTube talk where Shazeer flipped through completions from 100 million, 300 million, 1 billion, and 5 billion parameter models and asked what if you made it bigger — scaling laws as a demo, while other labs later made it scientific. Among deep-learning elders, Geoffrey Hinton deserves the most credit for grinding through the years when tiny models were mocked. And then there is the army of Noams: Brown, during his PhD, soloed a problem several labs were attacking with ten people and famous names, and repeatedly posted 100x gains in RL data and compute efficiency. Steinberger’s running joke is that you should name your kid Noam. Every well-known Noam in the research community, he claims, is great.

Huang brings him back to the unsolved list. A lot of it, he says, is solved: multimodal, long context, reasonably smart models, costs that are no longer crazy. You have to be a reality denier not to see what is coming. The remainder is still general-domain long-horizon reliability, paid for with test-time compute. People in the LLM world have only recently internalized what the RL world treated as obvious: more thinking time makes the system stronger. Maybe you can get away without it, which would be insane; even then it would still help a lot. He would rather spend a billion dollars on a pretraining run and put serious money into inference than spend ten billion on pretraining. There is no public successful recipe yet. Best-of-N sampling with N large enough is “not terrible,” but he does not think the final idea is on the table. Even if research froze, he thinks the field would still eventually hit something everyone would call AGI — maybe not Riemann-hypothesis-hard, but pretty good. He is more curious about the ceiling if you actually do all the things.

Magic’s product-side mission is to automate software engineering. It took them a while to learn how to train super-giant models, which he calls an interesting engineering challenge. A model that is a great software engineer should be able to do everyone’s job at Magic, including the researchers’. That is the point: recursive improvement of both the models and alignment, no longer bottlenecked by human resources. There are not that many Noam Shazeers in the world; if you had one in a computer you could spin up a million of them, and maybe alignment gets easier. He flags the statement as wildly simplified and idealistic. The picture he actually wants is cloning next week’s work into a cluster and pressing the money button. Software engineering is the lever that creates everything else — Huang supplies the phrase “MVP of AGI,” and he agrees. They would not train a Sora. Video generation is fantastic and, from an AGI perspective, uninteresting if models will soon copy themselves.

Among companies chasing an AI software engineer, Magic is the rare one training its own models. Steinberger’s answer is not that he enjoys training runs. To build the best model you have to build the model, and if the API vendors solve the fundamental problems, what are you? You might as well start three years later. It took Magic about as long to learn large-model training as it took OpenAI to go from GPT-3 to GPT-4 — two years — and it is a massive financial investment people have trusted them with. If it works, the return is fantastic; if it does not, the GPUs ran and the money is gone. On the business side he is harsher: value will accrue at the AGI layer and the hardware layer, never at the application layer. There is no incentive to offer an API if that API creates a hundred-billion-dollar company. You will build that company internally, and if OpenAI does not, someone else will. Chat interfaces, he says, can be built with YC pre-seed money. Netflix, Airbnb, and Uber are much harder. Make the API twice as expensive, launch your own product, and undercut everyone. Not owning the model in a domain that will mint huge revenue for a single company is, in his view, a terrible place to sit.

Huang presses the 800-pound-gorilla problem: some labs raise more, and some incumbents can spend more than a hundred billion dollars of revenue a year. Steinberger reframes it. The question is how much it costs to build AGI, not how much you can raise. Extra hardware helps, but a hundred times more compute does not get you there substantially sooner if the right ideas fit in a smaller budget. He is betting the cost is not $100 billion. If he is wrong, they will not have $100 billion, and that is the end of the story. If they reach AGI with sufficient rather than infinite compute, additional capital becomes an even playing field: ship, show ROI, raise more. For investors, sufficient early funding might even have better ROI than infinite funding. On team size, he admits scaling people was an early weakness. Five researchers were right at the beginning; now he would say closer to twenty on the models, and he does not see the research org exceeding the tens. If you can utilize 200 people, he says, you are a better CEO than he is. Magic is smaller partly because it does not build a video model. If doubling the team doubled speed, he would do it any day.

Huang remembers a 2022 pitch, when marketing and email assistants were the rage, in which Steinberger was already talking about AI that feels like a colleague. He now draws three tiers: useless; an assistant you have to micromanage; and something that manages you, more like a colleague. The layer where the model is exactly even with you barely exists — a thin point you pass through. Once the model is more competent than you are, your job is to say what you want accomplished and answer clarification questions. You should never have to tell it “here is a bug.” He is not claiming that is version one of every product. He is claiming it has to be the goal. The feeling he wants is talking to his best engineer: a discussion in which the other person is almost always right, then the code gets written, reviewed by someone else, and it works.

Everything else is Magic’s problem, not the user’s. Which tools the agent uses, whether it runs locally or in the cloud, whether it needs a VM or a browser — “doesn’t matter.” Customers care about their problems getting solved. He is stubborn about not launching anything that is not that, even while admitting they will probably have to ship checkpoints. The end state he keeps returning to is talking to the computer, going to lunch, and coming back to find it has built AGI. How a company gets there is up to the company. On timing he is careful in a characteristic way: people who know him should multiply his timelines by one and a half or two. Even then he thinks it is a very small number of years, “definitely less than 10,” and “way less.”

Huang cites SWE-agent’s 14% on SWE-bench. He does not care about 14%, and he does not even know whether 80 or 90 is good enough. He thinks you need 99. At 96 he still would not trust his computer; he does not want to review the code. A product that requires you to review and understand the code is a different tier from one that does not. When you do not want to review, you are not talking about 95. You are talking about 99-something, whatever his developers already accomplish, plus some. Self-driving is the analogy, with one difference: a crash there kills people, a crash here means you read code, so you can launch earlier. The standard does not drop. The last nines are the hard ones. Benchmarks, he notes, keep falling faster than prediction markets expected. He does not see that stopping.

A lightning round closes the conversation before the announcement Sequoia actually invested on. Favorite AI app, not Magic: the invisible ones, spam filters and recommendation algorithms, the systems that keep ordinary life working. If you took them away, life would be awful; most AGI-flavored apps still would not hurt as much. Most influential paper: DeepStack, a neural-net-plus-imperfect-information-game paper that is no longer relevant to the frontier. It was the first paper he tried to understand deeply, reimplement, and then Wikipedia-hole through. He had no idea how to do research until that grind. What he is most excited about over one, five, and ten years is not a model release. It is how society integrates with the technology — impact beyond another browser tab that speeds you up a few percent. He is not, he says, the intrinsically curious researcher. He cares about the outcome.

Huang reminds him that last time they recorded, they could not talk about the thing that got Sequoia’s own AI researchers excited enough to invest: Magic’s long-context eval. Magic has now decided to announce where it stands on context — a 100-million-token window, in the episode description — and to open-source the eval rather than just talk about token counts. They had been running a hashes eval internally for a while, frustrated by needle-in-a-haystack. They used it in fundraising, to guide architecture, and to steer research. Open-sourcing it lets other labs compare. Needle-in-a-haystack, Steinberger argues, tests whether you can find a weird thing in a giant pool of not-weird stuff. All you need is a little backpack: walk from the start of the context to the end, pick up the odd object, and return it. Because the needle is weird, the model has an implicit prior that makes it more memorable, so it never has to remember the whole window. Some models look excellent at long context under that test when they are not.

HashHop goes the opposite direction. Replace everything with randomly generated hashes — no semantic information at all. If you hide a needle in a pool of hashes, you really do have to know the whole thing. Then they add hops: find this item, then use it to find the next, and you can keep going. Those two dimensions, how much you must remember and how many hops you can chain, are what he thinks you actually want to quantify in a general-purpose context eval. Domain-specific tests, especially code, still matter, and Magic runs those internally. They open-sourced this eval and not others because it measures the thing people mean when they say “context window.” If the model is not forced to remember the whole window, is the window really that big?