Reliability is the bottleneck. Autonomy is what would make AI feel alive.

Hosts Sarah Guo and Elad Gil open on a reminder that still startles in late 2023: a year earlier, OpenAI was about a hundred people. ChatGPT has since seized consumer attention, and the lab’s co-founder and chief scientist Ilya Sutskever is there to talk about limits, AGI, and superalignment. They start where his own conviction started, before AlexNet, when nothing in deep learning really worked.

Sutskever calls that period the dark ages. With no successes to point to, the field split into camps: good old-fashioned knowledge representation, Bayesian nonparametrics, graphical models, and neural networks. The last group was marginalized because you could not prove math theorems about nets. “If you can’t prove theorems about something, it means that your research isn’t good.” He gravitated to them anyway. They felt like small brains. Who cares about theorems, he thought, if you are training little brains that might one day do something.

AlexNet, in his telling, was three things landing at once. GPUs had just entered machine learning as toys people wanted to play with, and they fit nets unusually well. He had internalized a then-contrarian claim: the nets of the day failed because they were too small. A thousand neurons on a high-resolution image cannot do much, no matter how clever the learning rule; a brain sees quickly because it is a large network whose neurons do not have much time. The third piece was know-how. Gradient descent was old; the belief that you could not train these nets was older. Collaborator Alex Krizhevsky turned out to love GPUs and was among the first to write genuinely fast code for them, squeezing two cards until the result was unprecedented. The size target was not a magic number. It was as large as the available compute would go.

The intuition, he says, follows if you accept that an artificial neuron is not that different from a biological one. Whatever animals do with brains, a similar-sized net might do if trained. Training itself he describes as solving a “neural equation”: each data point is an equation, each parameter a variable, gradient descent the solver, and the dataset must be large enough to constrain the large net. Most researchers of the time could not have trained such a system even if they wanted to. The details all had to be right.

Asked how OpenAI’s purpose evolved, Sutskever does not hedge. “The goal did not evolve over time. The tactic evolved over time.” From day one the aim was to make sure AGI—autonomous systems that can do most of the jobs, activities, and tasks people do—benefits all of humanity. The first plan was to open-source a lot of technology and operate as a nonprofit. That seemed the obvious match of structure to mission.

What changed was compute. OpenAI was among the earliest groups, he says, to realize that real progress would take a lot of it, with an appetite that now looks endless. A nonprofit could not build a large cluster. Hence the odd “capped profit” company, to his knowledge the only one in the world: investors put money in, but even if the firm does incredibly well they do not get more than a multiplier on the original stake. If AGI can do every human task, it might unemploy everyone. He does not know that it will; he says it is not impossible. A company that built such a thing, in that logic, should not be incentivized to make infinite profits. He also flags the soft spot: several AI companies will compete, with unforeseen effects on the argument.

The research agenda moved the same way. Year one was conventional machine learning in what he now calls the Stone Age of 2016 and 2017: academic, paper-sized discoveries, scientific recognition. More than half his twenty years in AI were spent in that frame. The first large-project attempt was Dota 2, driven by Jakub Pachocki and Greg Brockman, training a net to play as well as the best humans. It worked, and it was the wrong formula: networks too small, a narrow game. The second realization arrived gradually through generative models, next-word prediction, and compression. When the Transformer paper landed they pivoted immediately, built GPT-1, saw signs of life, then GPT-2, then GPT-3. GPT-3, he says, opened everyone else’s eyes. The industry collapsed onto one formula: train a larger Transformer on more data.

Gil’s own wake-up was the GPT-2 to GPT-3 jump. Asked for the most surprising emergent behavior, Sutskever says he is too close to pick a capability. If he has to choose one surprise, it is that the whole thing works at all. He lived for years in a world where neural nets did not work; now they do amazing things. A more personal answer: when he speaks to a model, he feels understood. Gil quotes the Clarke line about advanced technology and magic. Sutskever says he is fully in that camp. Internally, big bets mix top-down ideas they are not 100% sure of with bottom-up exploration. Scale keeps improving the current recipe, but different ideas improve by different amounts, so they want to scale the best thing, not merely a bigger thing.

What is improving most as they scale? He points to the public models. Compare GPT-3, GPT-3.5, ChatGPT, GPT-4, then GPT-4 with vision, and the obvious change is reliability. They used to be only partly there; now they are mostly there, with gaps. They will do more tasks, and they will have deeper insight into “the true nature of the human world.” That insight, he says, will keep deepening.

Guo puts the product tension on the table: huge models dazzle, inference is expensive, teams fine-tune smaller ones and worry they are giving up reasoning. Sutskever’s reply is the episode’s most portable sentence. The main thing you lose when you switch to a smaller model is reliability. At this point, reliability is the biggest bottleneck to these models being truly useful. He defines it operationally. If a question is no harder than ones the model already gets right, you should have high confidence it will keep succeeding. You ask twenty good questions about a historical episode and do not want a gross mistake on the twenty-first. You upload financial documents, want analysis you will act on, and because the model does not succeed every time, you cannot trust it any of those times. He analogizes to self-driving: mostly-good is not good enough. The model case is less extreme. The logic is the same.

Application builders, he agrees, should use the smallest model that works. That is self-evident. Larger, better models will still unlock unprecedentedly valuable jobs. His example is legal advice you can actually trust: extremely valuable, perhaps only a much bigger model can deliver it, and that value can justify the compute. Research at 7B, 13B, and 34B is not wasted. In the medium term there will be an ecosystem. Plenty of people will be happy with the best 7B. Plenty of applications will not be. Not every task will pay for a large model.

Open source, he says, is complicated. Near term, the demand is real: many people want to be the final decider of how a model is used and which cases it supports, rather than handing that to a hosted closed model. Quite a few companies will choose open weights. Long run he is less sure. Today’s models are not very competent. Their popular impact is entertainment and simple questions, images, a conversation. That is far from completing a large, complicated task.

Then comes the line he wants people to keep. “Intelligence is power.” Imagine a model that could autonomously start and build a large tech company, or do science, or deliver big science projects. Open-sourcing systems of that power would have hard-to-predict consequences. The argument that is straightforward for current models becomes much less clear-cut. Figuring out that boundary, he says, is itself an urgent research project.

His working picture is a lag. Closed models are more capable, so they can be studied first. You live with a generation, decide its capability is no big deal, and about a couple of years later the open models catch up. A day may come when closed models start to feel “a little too drastic,” and some other approach is needed. On near-term scaling limits he is blunt: the most immediate ceiling is data, this is well known, and some research is required. Without details, he says the data limit can be overcome and progress will continue.

Gil raises the standard objection: the human brain has specialized regions for vision, language, empathy, personality. Can a single Transformer cover everything AGI needs? Sutskever splits the answer. First, stop asking whether an architecture is “enough.” Ask what it costs. Nobody doubts Transformers can do amazing things; a modification might buy compute efficiency. Different ideas also get better by different amounts as you scale, so you scale the best one.

Second, the brain is less specialized than the map of named regions suggests. Some young children with severe epilepsy have half the brain removed; because the surgery happens so early, they grow up as pretty functional adults, with the same regions somehow compressed onto one hemisphere. Efficiency is probably lost. The regions rearrange. A cruel ferret experiment from thirty or forty years ago attached the optic nerve to auditory cortex. After a few days of learning, recordings found neurons in that auditory tissue that looked a lot like visual cortex. The cortex of humans and animals, he says, is extremely uniform.

That is all you need: one big uniform architecture. Gil notes that biology is lazy in general, reusing the same twenty amino acids and the same tissue tricks. Sutskever reads the analogies as a technological green light. Artificial neurons and biological neurons, a unified brain and a unified net: “from a technological point of view, we are very much on the right track.”

When should we start calling these systems digital life? Sutskever’s answer is autonomy, which is also why they are not that useful yet. You cannot ask one to do your homework or your taxes and walk away. As reliability rises, usefulness rises, and they will look more like artificial life, which is when the trepidation should rise too. Gil objects that biological life is usually defined by reproduction. Sutskever says definitions are for our convenience. Technology already reproduces: look at twenty-five years of cell-phone evolution, or a century of cars, trees of copied ideas running through human minds. Reproduction is present. Autonomy is not. An autonomously reproducing, highly capable AI he calls pretty dramatic and quite scary.

Superalignment, he says, depends on where you think AI is headed. Progress has been stunning; even if it slows a little, five years, let alone ten, is a different world. It does not seem implausible to have data centers much smarter than people, and smarter here means deeper insight into the same subjects and faster learning, not just more memory. If such superintelligent, possibly living machines are built at all, the bare minimum he can articulate is that they hold warm and positive feelings toward people. Friendship is optional. Being prosocial is not. The project is to start the science now: how to control such systems, how to imprint a strong desire to be nice and kind, knowing there will be many of them and the world will be complicated.

He does not call success guaranteed. He thinks it is possible, and that the demand for a humanity-loving superintelligence will grow as people feel the year-over-year jump: last year’s helpful model put to shame, then systems that start to do science, then a software engineer that is really quite good. Few people work on the scientific problem yet; the models are just becoming powerful enough to study it productively, and he teases exciting research to share soon. His self-test is simple. Look at what you have experienced. Is it slowing down? Will it slow down next year? What needs to be done will keep getting clearer.

Are we in an acceleration phase? Yes, he says, with forces on both pedals. Costs, scale, finite data, and the management complexity of huge engineering projects push toward slower. Investment, interest from engineers and scientists, the existence proof of biological evolution, and a research culture with unusual “give” push toward faster. Quantum physics, as he understands it, asks years of graduate school before you can contribute; AI lets people come in, get up to speed, and add something quickly. That friendliness to smart newcomers is itself an accelerating force. Maybe engineering complexity will throttle the rate. Maybe the pushing forces keep it this fast for a few more years. If at all.