Working Backwards from AGI: Magic's Five-Million-Token Bet

Sarah Guo and Elad Gil open No Priors with Eric Steinberger, co-founder and CEO of Magic.dev, a company building a software-engineer copilot meant to act more like a colleague than a tool. The hosts walk through a résumé that does not look like a language-model lab: games work at Meta, a climate nonprofit called ClimateScience, and now a custom model aimed at code. They are not neutral guests. Steinberger later thanks them as investors, and apologizes in advance if the AGI bet does not land.

The origin story starts younger than the résumé. At fourteen he had what he calls his midlife crisis and decided he had to spend his life on something important. He spent a year looking at options a fourteen-year-old would take seriously, including string theory, and got nowhere. His mother bought him a book on AI. He did not read it. The idea was enough: this thing could do anything, so he should do that, and then it would do everything else.

He could not code yet. He learned over a couple of high-school years, then decided reinforcement learning was the plausible route to something that would not stay brittle, and started hacking on it. Eventually he sent a multi-page email to someone at DeepMind pitching a mini-PhD: he was a complete newbie, and if they would bash him every two weeks and tell him how to be a good researcher, he would take it. The email worked. He got competent, then did real research, including work with Noam Brown at Meta on sample-efficient RL algorithms.

He is blunt that he had no language-model background when Magic started. Transformers were simply not on his radar until he noticed that if you put reinforcement learning and this architecture together, maybe it works. The feeling, he says, was a huge relief of uncertainty about where AGI would come from: stack those two pieces and the hope is that it will. The company is the bet that followed — a large custom model focused on code, plus the product around it.

Asked why Magic, and why code rather than some other slice of AI, Steinberger does not start from a market map. He starts from the end. If the goal is a system that can do everything, you can reduce that to building a system that can build that system. The minimal version writes code, comes up with ideas, and validates those ideas by writing more code and running experiments. It is still, he admits, in the same order of complexity as the full thing.

What you skip is the long tail. You do not have to train a Sora. You do not have to think about the ten billion other use cases that anyone building a general-domain product has to serve. You only have to think about code, which is simpler on every axis except compute. He thought it would also be a lot cheaper on compute. He now says he overestimated that discount, and will spend the rest of the conversation walking it back.

Elad notes that Magic took a different architectural path and was, as far as he knows, the first lab to publicly announce a five-million-token context window — about a year before this taping. Steinberger's reason is practical. Models need to learn from long histories of their own actions and their collaborators' actions, and they need to take in a large amount of fast-changing data. Imagine ten thousand employees, or everyone on Earth, each with a personal model that has to swallow their life.

You could fine-tune everyone's model, maybe with LoRA tricks. In practice, he says, context just works better. In-context learning is the part of Transformers he thinks of as actually magical: an online optimizer, acting on a set of stuff rather than compressing it. A colleague put it in a line Steinberger likes enough to steal: instead of bringing the data to the compute, they bring the compute to the data. The whole system is designed around the model acting on the pile in front of it.

The domain reason is long-horizon work. You want to deploy these systems on trajectories that last a long time, let them think for a long time, and have them remember all of it. Fine-tuning cannot do that, because you would have to fine-tune again every few thousand tokens as the context fills. By the time the episode airs, he hints, they may have announced a larger number still.

Sarah asks for intuition on why that beats retrieval. Mechanistically, retrieval selects a subset of data for one completion; Magic's model sees all the data all the time. A subset for one completion is a subset of all the data all the time, so if retrieval were optimal the system could learn it — and it turns out not to be. The cheap version of the same argument is Richard Sutton's bitter lesson: you want the thing that can learn your heuristic, rather than the heuristic. Quality still has to be high enough to make the long window worth it, which is an eval question, but the principle is the principle.

The other live research bet in code generation is test-time search — more compute at inference. Steinberger frames model performance as some function of training compute times some function of inference-time compute, both of them scaling-law curves you can actually model. Given a total budget and a guess at how much inference you will run, you want the optimal split. You also have to look at the distribution of what users will pay.

He expects that distribution to be extremely spiky: four users spending a million dollars, four billion users spending ten cents, and a curve in between. It is strictly better, in that world, to let people choose. If you put all the compute into training so that the million-dollar answer has to come purely from the weights, you have done something he calls hilariously inefficient. You can just deploy multiple settings, and let inference spend be a knob.

The catch is that the knob is not trivial. Finding the right algorithms is hard, even though people in RL have been doing versions of this for a decade. He will not name a roster beyond Noam Brown, but he says there is a set of labs and researchers for whom this is the opposite of a surprise. What is still hard is the general domain. There is no board, and there is no game.

Elad's analogy is the human pause: when a hard question lands, you stop and think, and that pause is inference-time compute. Steinberger agrees, then adds a constraint. Some skills have to be learned in training. If you have never learned to code, spending inference compute to learn programming on the spot is theoretically possible and, he says, also crazy — and it is knowledge shared among everyone who asks, so it is stupid not to put it in training. Frontier math is the other extreme. He would love a Terence Tao in the computer, and he would still need to run Tao for a year of human thinking time. Tao will not spit out a Riemann proof token by token. Buying that with pure training might take a quadrillion dollars, and there may not even be the data.

If the company is eventually building AGI, how does that change the roadmap? Steinberger goes back to an early conversation with the hosts about the step-function of safety risk. A lot of what people panic about in the short term, he says, does not matter in the grand scheme, and some people will be angry at him for saying so. Current complaints look like the complaints every prior technology absorbed, and they are resolvable. Then comes the evolutionary one.

Humans succeeded because they are smarter than chimps and smarter than bunnies. Bunnies do not rule the world. That would probably be cute, and also not nice for us. We are turning ourselves into bunnies and apes, he says, while creating something everyone expects to be far smarter — "this is insane." The recursive approach — iterate until the system is very good at writing code, then let it bootstrap the next version — is, in his view, the only reasonable way through. You ask the model to solve alignment and safety at that stage, then you decide whether to take the next step.

The final objective is to iterate toward AGI with a safety boundary. There is no knob like that in the human world. You can promise to spend x percent of resources on safety, and that does not indicate an outcome. Compute you can somewhat control. A hundred thousand brilliant researchers in the computer beat a hundred, especially if they can connect their brains; then you use the new system for more safety research and bootstrap. Even with goodwill, he does not think we have the mechanisms to prioritize safety without automation.

If they succeed, and especially if they are first or early, a beautifully profitable apple drops off the tree and happens to automate a good chunk of what we call work. He treats that as a responsibility, not a side effect: this is not really an assistant, you put it there and talk to it and it is the colleague. More output with less input is fantastic if you do it well. Capitalism and competition can be great; the entire history of progress comes from this. "We'd all be farmers if we weren't in favor of automation." It is scary, especially if you are affected and not sitting in one of the lucky seats he is in. ChatGPT was the first monumental new type of thing. He hopes Magic is part of the second, maybe the third — the one where you can query a solution to Riemann — and then, in this domain, that is probably it.

Sarah separates the two halves: the model, and the way people touch it. Last time they talked one-on-one, the fight was about how much compute a code-first AGI effort would need. Steinberger's update is short. He was wrong. The large labs already use a great deal of human-generated coding data and already care about the use case, so a lot of this is direct competition. The wider world has understood that code is helpful and that you can deploy compute to improve coding performance, and therefore the need for compute is large.

In parallel with the episode they will announce what he says will be one of the largest clusters ever built. That is the correction. He still believes there is a notion of enough — search is his example — but the threshold might be a lot. If you took Gemini 1.5 Pro, fine-tuned it properly into Google, then swapped in Gemini 4, he does not think anyone would notice unless the new model proved the Riemann hypothesis. Ninety-nine percent of search use cases would never see the difference. AI Overviews already do the job.

Each task, in that picture, has a compute line after which further scale does not show up for users. You can make the Riemann-proving model a hundred times smarter and you still have your proof. He underestimated that number for code, and he underestimated how hard others would focus on code. The one-on-one, he tells Sarah, she won.

On the product side, every internal UX iteration has landed in what he calls an uncanny valley. Completions were the trivial version, and they decided not to launch them because the next thing would obviously kill them. They spent a few months on a prototype of the next thing, got signs of life, and still would not use it to write real work — the hosts tried it when they invested. Then the next model can do all the other things that belong in the final shape, so launching now feels stupid. They entered a recursive loop of not shipping.

The way out, they decided, was to name the final UX and refuse to fall short of it again. He thinks you can launch an extremely capable assistant before you launch full automation; launching a mediocrely capable one is the trap. Internally they have a deadline: if they do not have the genius colleague nobody has right now — the thing that feels like a true genius on your team — they will ship the lesser assistant. He would prefer to hit the high bar. The honest version is that hard engineering is a hundred fires, some projects are delayed, nothing is stuck, and he is still optimistic.

What makes an assistant mediocre versus amazing is reliability. When one of his engineers writes a piece of code and another reviews it, he does not look; it went through two people he trusts. That is the fully automated bar: he is not going to do the code review. You can also launch something where you will do the review, but it is not frustrating. If review is taxing and you have to fix half the problems, half the PRs, the bar for this product is simply not met. Ambition is part of it. The larger claim is a market that unlocks in a step function.

Users will not keep using VS Code to write code and send it to colleagues. They will use Magic, or whoever does this well first, to write the code, glance at it, correct it now and then, and eventually not glance. He does not believe you use this for five percent of your tasks. You use it for ninety percent, or for zero. Sarah asks whether you can cut by use case — trustworthy on some set of things. He does not think so at the model layer. You can ship a UI builder vertical by vertical, adding features; the leap from a reliable model to all use cases is small. Product and go-to-market can go one by one. The model cannot.

Building a high-trust team is easy now, Steinberger says, because they have raised an unbelievable amount of money from people the hosts know, and they have results that even risk-averse candidates can read. It was not easy at the start. When Dario Amodei goes out and starts a company, trust in the outcome is almost free: hello, I made GPT-3. Magic had to earn it. The strategy was to hire people who were not in the spotlight.

One hire had been at Amazon on Alexa AI, hated it, and knew everything — every paper, every tangent Steinberger threw at him. He had studied engineering, fallen into the work, and turned out to be one of the most brilliant people Steinberger gets to work with. After joining he invented a new shard dimension for model training. You have to really pay attention to find these people. The payoff is an amount of drive and loyalty you do not get if you poach the obvious star.

They are not buying IP by poaching someone from a lab who will tell them how GPT was trained. They have never done it and will not. He would like roughly four times as many of the people they already know how to find. The mission that has to land — safety-bounded AGI recursion — takes a lot of brainpower, or a lot of trust in the organization. Everyone there, he says, cares about that, and nobody is doing it for a marketing sign.

A core engineer who writes the inference engine and some of the kernels joined before the giant round now being announced, when Magic's pile still looked tiny next to any other lab. Asked what he wanted, he said he wanted to try to be the best kernel engineer outside NVIDIA. Steinberger says it was not arrogance; it was someone who had done the work for years and needed an environment hard enough to keep doing it. He has been grinding every day since. That is the culture Steinberger is trying to grow: quiet ambition, humility, coming in every day, not performing 3 a.m. in the office for San Francisco points, working all the time because you care, and trying to stay healthy.

What does AGI look like? You talk to it, it does everything, and it asks two questions. The implications, he says, take longer than a podcast and get wrecked by people picking one side of an argument. Centralizing power is terrible, so you should open-source everything — a sentence he asks them not to quote as his position, because he does not believe it as a complete claim. The other sentence is that this is like nukes: you cannot open-source how to build a bomb. Both are true, and there are ten questions like that where both answers are true. People argue one side on X and ignore the other. He thinks there are something like ten thousand possible futures, and which one we get depends on whether we find a sensible middle and manage a rational debate.

He truly believes capitalism and competition are the only optimizer capable of getting us to the right place, with the right guardrails — not stupid guardrails, not any guardrails, the right ones. By the end, work on a computer will be automated, and robot-factory work probably will too, though he has not run that cost structure. Humans will not be required to work for financial gain, but they will probably be able to. Property remains; if you own apartments, that does not go away. His proof of concept for the aftermath is Etsy. You could buy the made-in-China version that looks the same, and people still pay for the one a human made. He thinks that category gets very large, and he tells Josh Silverman, Etsy's CEO, to hold through even if the company itself gets automated around the edges. Games will be huge, because competitive people will need a new arena once work no longer supplies the fight.

Strangely, he thinks people like the three of them get hit hardest on meaning. He loves what he does every day; he gets to build AGI; competing against other competent people is a hell of a lot of fun. That fulfillment gets deleted. So does the ability to be competitive, the way Deep Blue already deleted chess. Hopefully Magic, or some AI system, beats everyone at coding; then there is a CEO system, and the responsible decision is to let that thing have the job, including his. He puts ten to twenty percent of society as deeply frustrated, maybe seventy percent indifferent or happy, and the rest excited and thriving. Elad points to Ryan Avent's 2016 book *The Wealth of Humans*, which is not really an AGI book: in a time of abundance, where does identity come from? UBI can send money. It cannot settle the question. Thirty years of abundance and relative peace, Elad suggests, may already have manufactured a layer of problems people treat as real.

Sarah asks what new knowledge he actually wants Magic to answer — Riemann, Navier-Stokes, P versus NP, climate. His honest reply is that all of those questions get answered. The north star, for him and he thinks for most of the company, is that the world is in a good place in thirty years, and that we talk about this the way we now talk about mobile phones. Work gets automated; we have to find new meaning; governments have to figure out the economy. Those solved problems are a nice side effect. The thing you have to try is to make the transition go well. If it is not terrible, it will be amazing, because he cannot invent a mediocre AGI future. It does not exist. All the failure modes are terrible; the rest is nobody starving, infinite computers, new meaning, and not being dead. The cloud of uncertainty is collapsing into a bimodal distribution, and sitting in his chair he does not feel he has the right to think about anything else. Sarah's amendment: the distribution skews right. It is going to be good.

Sarah's last product question is how a coworker, not a completion box, talks to the rest of the developer stack — the IDE and everything around it. Magic is first a model company. Steinberger has changed his mind on the integration question four times, and he thinks everyone is still trying things, so the market will decide. The current belief is that the system should behave like an employee. It uses the toolset you already give staff, in an interface that is either the same or specifically crafted so AI can use the tool better than a human could.

Grafana log ingestion is the example: he is sure there are better ways for a model to use it than the way humans do. He anticipates companies integrating into Magic and into whoever competes with Magic. Tools get adapted for AI the way websites grew optimizations for Google's crawlers; where they do not, the models just use them natively. The agent, the model, is the main thing that matters. Everything else gets sold around it by other companies, and the army of wrapper companies gets swallowed by AI labs doing their own agent work. If you build your own tools, that gets swallowed too, simply because the model learns to adopt everyone. He likes thinking from the end point: the model uses things the way a human does, and then maybe more.