The Only Real Freedom Is How AGI Is Born

Lex Fridman opens with a biographical fact that still startles: in high school and just after, Greg Brockman drafted a chemistry textbook that ran from the structure of the atom through quantum mechanics. A friend told him it would never be published, because he did not have a Ph.D. So he built a website to promote the ideas, and on that site he discovered programming. Mathematics, chemistry, and code, in his telling, are three versions of the same act — think hard, write the conclusion in an obscure form, and put it in humanity’s library.

The difference is leverage. A proof might be read by five people in a field; a program can be run by anyone on the planet, and it scales. That is the digital world’s hold on him: a single person with an idea can affect the whole Earth, something that is almost impossible if you are moving physical atoms. He once expected to become a mathematician. Iteration speed pulled him the other way.

Fridman asks whether minds are ultimately just math, just information processing, or whether biology still hides some other magic. Brockman is willing to treat humans as information-processing systems, and he thinks the description explains a lot. The internet’s meaning is not the cables. It is that one person can instantly reach any other person, and retrieve almost any piece of knowledge the species has accumulated. That, he says, is a rewiring of human capability.

Push the same lens one layer out and society, the economy, and a company all look like machines larger than any individual. People inside a firm pursue their own goals and think hard about the right thing to do, yet the company as a whole does something no one separately planned. We like to think of ourselves as the smartest and most powerful things on the planet, Brockman says, but there are systems bigger than us that we all feed. He reaches for Isaac Asimov’s *Foundation* and psychohistory: with trillions of beings, you might predict what the huge macro-being will do, almost independent of what any individual wants.

Brockman’s second angle is technological determinism, and he thinks about it constantly at OpenAI. General intelligence will arrive at some point; the live question is how to steer it better rather than worse. As a scientist or inventor, what impact can you actually have? The telephone, he notes, was invented by two people on the same day. If Einstein had not been born, someone else would have found relativity — perhaps twenty years later, but humanity would not have missed the truth forever.

Everyone is standing on the same giants’ shoulders. You cannot hope to create something no one else ever would. You might change the timeline a little. Moore’s Law, he says, is the eerie version of the same fact: an entire industry set its clock to an exponential for fifty years. If you really want to make a difference, the only real degree of freedom you have is to set the initial conditions under which a technology is born.

The internet is his working example. Plenty of teams were building similar networks; the one that won was created by people who believed anyone should be able to plug in, with an academic mindset of openness and connection. For the next forty years the internet basically grew in that shape. The direction may be shifting now, but those first years determined decades of progress. Fridman adds Wikipedia: what would the internet look like if it had run ads? It is one of the most important public resources online, and it works surprisingly well because its creators set the initial conditions and then it carried itself forward.

Fridman looks ahead: suppose you have built a system that can pass a Turing test in natural language. What is the first question you ask it? Brockman does not hesitate. If you have built something powerful enough to shape humanity’s future, the first question you should ask is how to make sure it plays out well. Fridman presses: you would not ask a colleague, you would not ask Ilya? Brockman says those conversations have already happened, with Ilya Sutskever and everyone at OpenAI. You want as many perspectives as you can. The system’s answer is one input, not a verdict you defer to.

He analogizes the moment to the years just after nuclear weapons. The most important question in the world became what the world order would look like, and how the species would survive. With AGI the question is slightly different: you still have to avoid the negative effects, but you also have a positive side that people miss. A system as capable as a human, and far more scalable, is exactly what you want reading the scientific literature, hunting for disease cures, creating material abundance, and inventing biodegradable robots that turn ocean debris into harmless molecules.

Fridman observes that most possible AGI trajectories may be positive, yet public discussion — Sam Harris, even Elon Musk — dwells on the dark ones, as if catastrophe were more fun to think about. Brockman splits the problem in two. First, it is simply hard to picture a world with a new technology. In 1950, describing Uber would have required describing the internet, GPS, and a phone in every pocket, and even then the world would be hard to inhabit in the mind. Every disruptive technology has shown this, and AGI is more disruptive than all of them. Positive visions are hard to draw because they depend on too many intermediate steps that have not happened yet.

Second, destruction is easier than creation, in the physical world and in the world of ideas. To build something you need every piece to be right; to wreck it you need only one piece to go wrong. A lot of people’s thinking dead-ends the moment they see the negative story. Brockman still has some hope: the positive vision can be talked about, and simply saying that the positive half exists gets a reaction. People realize they have been staring at only one side of the picture. That, he says, is a key part of how OpenAI thinks about AGI — including the apparent contradiction of warning about risks while trying to build the system.

Fridman asks how hard it is, in Brockman’s intuition, to keep AGI on a positive track rather than slipping into existential threat. Brockman answers by pointing at OpenAI’s structure. The organization has three main arms: Capabilities, which does the technical work and pushes what systems can do; Safety, which works on technical mechanisms so that the systems built are aligned with human values; and Policy, which handles governance and the question of whose values. Technical safety is the one people talk about most. Dystopian AI movies are mostly stories of technical safety failing.

A lot of people look at the technical safety problem and declare it intractable. How are you supposed to write down what humans want? Can you even write down what *you* want? They stop there. Brockman’s rebuttal is that we have already built systems that learn things humans cannot specify. The rules for recognizing a cat or a dog in an image are intractable to write down, and yet models learn them. OpenAI’s safety work, still in early proof-of-concept, suggests you can learn human preferences from data. The team, he says, has had some encouraging updates.

How do we align humans, he asks. A human baby can grow up to be an evil person or a great one, and a lot of that is learning from data — feedback, positive examples, the long pass through childhood. The only example we have of a general intelligence that learns values from data is us. We should not be surprised if similar techniques are how we solve value alignment for machines. Fridman, gesturing at Yuval Noah Harari’s *Sapiens*, wonders whether good and evil are just collective fictions we agree to hold. Brockman does not need the grand frame. He folds the question into policy.

Policy, he thinks, is the piece people talk about far less than they should. Imagine the best case: technical safety is solved, the system does whatever the operator wants. Then the most important question becomes who the operator is, what they want, and how that will affect everyone else. You do not have to go existential to feel the difficulty. Look at how differently countries and cultures conceive of how society should operate. His concrete hope is a world where the United States, China, Russia, and hundreds of other countries can still exist as they see fit, while extremely powerful systems run alongside humans and the result is that existence is more meaningful — people happier, wealthier, more fulfilled. How to design that world, once you actually have the system, is not obvious.

Fridman quotes Oscar Wilde — we are all in the gutter, but some of us are looking at the stars — and asks how a mission like OpenAI’s came about. Brockman starts with the landscape. For sixty or seventy years people asked what would be possible if you could automate human intellectual labor. Science fiction supplied dystopias, and films like *Her* a more utopian sketch. Computers and the internet already outstripped what anyone predicted. If you can build AGI, he says, it will be the most transformative technology humans will ever create. The question then is whether there is a path.

After two AI winters, the field almost stopped daring to dream. Talking about AGI became nearly taboo. Brockman thinks people took the wrong lesson from history. When the perceptron was released in 1959, the *New York Times* wrote that it would one day recognize faces, call names, and instantly translate speech. Contemporaries looked at the system, said it could do none of that, and spent a decade discrediting the whole direction. They succeeded; funding dried up. In the 1980s there was a resurgence that Brockman had always heard was due to the invention of backpropagation. Reading the articles from the time, he found the causality was reversed: the democratization of computing power suddenly meant you could run larger neural networks, people got excited, and then backpropagation was invented. Those “larger” nets had about twenty neurons. Of course they did not get great results.

Not until 2012 did the simplest, most natural approach people had come up with in the 1950s — in some ways even the 1940s, with the McCulloch–Pitts neuron — become the best way of solving problems. Brockman lists three properties of deep learning worth paying attention to. Generality: a small set of tools, deep nets plus some reinforcement learning, solves speech recognition, machine translation, game playing. Competence: forty years of ordinary computer vision research, replaced by a deep net, works better. Scalability: larger networks, more compute, more data, and it keeps working better — shown time and time again. Those three together feel like essential parts of building a general intelligence. Scaling what exists will not automatically yield AGI; reasoning is an obvious missing piece. But for the first time it feels as if there is a paradigm that gives hope.

Once you believe AGI might be possible, perhaps within a lifetime and maybe sooner than people expect, you stop thinking about yourself so much. You start thinking about how to make the world go well, how to build an organization, gather people and resources, and make sure that if you succeed the world is one you actually want to live in. OpenAI was formed in 2015 on that high-level picture. Then everything ran in reverse from a normal company. Usually you start small, ship a product, find product-market fit, raise, hire, scale — and then the giants notice you and try to kill you. OpenAI did it in the opposite order. At a dinner in July 2015 the core question was whether it was already too late to start a lab. AI was tipping from an academic pursuit to an industrial one; the best people were inside big-company research labs; no amount of resources OpenAI could gather would look like much next to the tech giants; and you needed a critical mass of five to ten people, not two founders and a product. Their conclusion was restrained: it was not obviously impossible, so it seemed worth trying.

In 2019 OpenAI formed OpenAI LP, a capped-profit company that now carries the OpenAI name, while the original nonprofit still exists. Brockman says the whole entity is trying to accomplish one mission: ensuring that artificial general intelligence benefits everyone. The main way is to build general intelligence themselves and distribute the benefits. They are also fine if someone else does it, as long as the upside is not locked inside one company or one set of people. Those ideas are baked into the charter and into the LP’s structure. If they succeed, investors can get a return — but the return is capped. AGI would create orders of magnitude more value than any existing company, and that surplus is legally titled to the nonprofit to fulfill the mission.

About two years before this conversation they realized that building AGI would take far more money than a nonprofit could raise — many billions of dollars, not hundreds of millions. They looked at every legal structure on the market; none was quite right. That should not be surprising, Brockman says: if you are going to do an unprecedented technology, you probably need an unprecedented structure. They spent about a year compiling the OpenAI Charter. The first line item says they expect to marshal huge amounts of resources, but they will minimize conflicts of interest with the mission. Aligning on what they stand for was the necessary first step toward a company that could raise the money without betraying the point.

The charter embeds two paths at once: we are primarily trying to build AGI ourselves, and we are also okay if someone else does. That is a weird thing for a company, and it was the core tension in designing the LP. You need a shot at being a primary actor, which means building an organization, raising massive resources, and signing up for a long stretch of pain and risk — the startup mindset, competitive on purpose. The true mission, though, is not for OpenAI to build AGI. It is for AGI to go well for humanity. Going a hundred percent in either direction is the wrong answer. One thing always in the back of Brockman’s mind: do not talk as if OpenAI’s goal is simply to build AGI. The goal includes safety, and the goal is not to be the ones who build it.

Fridman asks a blunt question: why can’t a for-profit be good for the world? Google makes most of its money from ads; that does not mean it cannot do good. Profit, in his view, is fuel; charter, culture, and people determine impact. Brockman treats it as one of the oldest debates in human society, then offers an empirical observation: it is hard to list the world’s most impactful nonprofits and easy to list the most impactful for-profits. Today’s world is organized so that for-profits can be self-sustaining and roll on their own momentum. When the guardrails are wrong, you get logging companies walking into the rainforest. How you get positive benefits out of a for-profit, he says, is actually very similar to how you get positive benefits out of an AGI: a system more powerful than any human, superhuman on many axes, somewhat autonomous, and you have to set the guardrails. Nonprofits are pure, but not enough happens. For-profits, in some ways, have too much happening. OpenAI is picking a road in between. In the success case the cap is a small fraction of astronomical value, so investors and employees look roughly like a very successful startup — and the fiduciary duty runs to the charter, even when that comes at the expense of stakeholders.

The charter says to avoid enabling AGI to unduly concentrate power. Fridman asks why OpenAI itself would not keep all the power of any system it builds. Structurally, Brockman says, the power to dictate OpenAI’s actions ultimately rests with the board of the nonprofit, which has a duty to fulfill the nonprofit’s mission. That is how the pieces are threaded together. Day to day is another matter. The board calls shots at a high level; the people who actually have the keys to the technical kingdom are the employees. How any company’s values get actualized is the same here: you need people who believe in the mission and the charter, and who will take actions that may be worse for them but better for the charter. That has to be preserved over time, and it is part of how they hire.

At a giant, a line employee has a hard time walking up to the CEO and saying the company is doing this wrong. Google employees did organize collective action around Project Maven, so other firms may have their own mechanisms. At OpenAI, Brockman says, it is extremely easy for anyone to pull him or Sam Altman aside, and people do it all the time. That low threshold, in a small company, is one of the things he thinks is distinctive.

Another line in the charter is the late-stage switch from competition to collaboration. Assuming the technical side can be done, Brockman sees two key problems in deployment. The first is the run-up to the first AGI. Look at self-driving cars: it is a competitive race, and what always happens in a competitive race is huge pressure to get rid of safety. The scenario he worries about is multiple teams realizing they can get there, then choosing the fast path because the slower, more guaranteed-safe path would lose.

OpenAI’s commitment is not to generate that race. If the race is already being run and someone else is further ahead, they will not try to leapfrog. They will work with the leader and help them succeed, as long as what they are trying to do fulfills the mission. That commitment cannot be unilateral. Other players who are serious about building AGI should make similar promises. If everyone believes AGI should benefit everyone, it should not matter which company builds it. What everyone should worry about is racing so hard that something goes wrong.

Fridman asks what role government should play, from research through late-stage development. Brockman says government has to be in the room: this is technology that will shape how the world operates. OpenAI has done congressional testimony and talked with lawmakers. The message right now is that it is not the time for regulation, it is the time for measurement. The main policy recommendation is that government — the way it already does with bodies like NIST — spend time figuring out where the technology is, how fast it is moving, and become literate about what to expect.

He splits the problem. For today’s narrow applications, the right regulators already exist. Self-driving cars belong with NHTSA; these systems are performing jobs humans already do, and we already have ways of thinking about standards and safety. Empower those regulators now. For AGI there will be a later point with better answers, again starting with measurement and then thinking about rules. It is very easy to smother a growing field, and that is something to avoid — but the way to avoid it is not to race ahead while leaving other stakeholders out.

In February 2019 OpenAI released the GPT-2 paper and not the full model, citing abuse concerns. Brockman zooms out: language modeling is clearly on a trajectory where scaling yields qualitatively better performance. GPT-2 was a scale-up of a model released the previous June; run larger, it suddenly wrote coherent prose, which they had not seen before. Next comes 10x, 100x, 1,000x, and no one knows what you get. The June model they judged as an academic artifact whose positive value far outweighed possible harm. GPT-2’s capabilities were substantive enough that a line had to be drawn somewhere between the two, and safety had to start being taken seriously.

Internally the decision could have gone either way. After they announced they would not release the model, some people said it was obvious they should have, and others said it was obvious they should not. That split, Brockman argues, almost definitely means holding it back was correct: if it is not obvious whether something is beneficial, you should probably default to caution. Future models — possibly sooner than people expect, because scaling does not take that long — you definitely will not want to release into the wild. GPT-2 is a test case for a deeper question: how does a society go from having no concept of responsible disclosure, where the mere idea of withholding a model for safety is unfamiliar, to a world where you have a powerful model and at least think about it? The security community spent years building that norm. AI, he says, has none of it yet. Potential harms include biased training data and generated fake news or abusive content; the upsides include sci-fi authors using the model as a creative tool. Both are easy to imagine. The point of the drill is the process.

Fridman sketches two futures for a world of smarter fake news and networked bots. In one, humans always invent systems that tell robot from person. In the other, we accept that we are swimming in a sea of synthetic text. Brockman cites the meme of a robot arm clicking “I’m not a robot.” Trying to distinguish machine from human, he says, is ultimately a losing battle. Captchas used to be a simple image with a few artifacts, because OCR was terrible; today he can barely complete them himself. Captchas were a moment-in-time trick. As AI gets stronger, the tasks that are easy for humans and impossible for machines get rarer.

Hope is not gone, but the direction has to change. Authenticate the source of content rather than the content itself — Social Security numbers, real-world identity tied to digital identity. The cost is privacy and anonymity. If the only content you can trust is content you can trace, how does anonymity survive? Reputation networks might be one answer; he does not pretend the answer is obvious. Today he can still read a tweet and feel that a real human wrote it. That feeling will fail. Researchers used statistics to show that millions of FCC comments on net neutrality were auto-generated. What do you do when the statistics no longer work, and the most persuasive arguments are written by AI? That, he says, is not science fiction anymore.

Fridman flips the question: why is interacting with AI itself a problem? Why can’t ideas come from models trained on human data? Brockman draws a hard line at deception. A Turing-test setup in which an AI pretends to be a human and fools you feels wrong. People need to feel in control of their environment and to know whether they are talking to an AI or a person. The other side he is fine with. He comes back to *Her*: the film starts by asking how meaningful virtual relationships can be, then pulls you in until every emotional button is pressed the same way as if a real human were on the other end of the phone. A funny joke sometimes does not care who wrote it. The reason to build AI at all, he says, is to enhance human lives, let people do more, and feel more fulfilled. If systems can do that, sign him up.

Fridman asks how far unsupervised language modeling can go toward a Turing-test conversation. Brockman says the Turing test, in its real form, is not about language. It is about reasoning. To really pass, he should be able to teach calculus to whoever is on the other side, have them understand it, and solve new calculus problems. Language models do not do that layer; you need some way of plugging in reasoning. Whether that takes a sequence of radical new ideas or a slight reshaping of existing systems is an open question. The models have already gone much further than many people expected. GPT-2 may not understand fire underwater, but it also knows that smoke comes from flame, learned from static text, with no body and no physical experience.

Will scaling GPT-2 far enough make reasoning emerge from the same mechanism? Unlikely, he thinks. The type signature is a little wrong. Humans spend a variable amount of compute to get better answers — think a bit harder, get a better answer. That signature is not encoded in a language model, which spends its “evolutionary history” getting very good at prediction and then, at runtime, does one forward pass. There might be small tweaks: generate a whole sequence of thoughts and keep only the last bit, the part you converge toward. At the very least he would expect changes like that. The other missing piece is out-of-distribution generalization: encountering something you have not experienced and still refining your mental model. That feels tied to whatever reasoning is. It might be a small tweak. It might take decades of new ideas.

Fridman brings up a Richard Sutton blog post: the biggest lesson of seventy years of AI research is that general methods that leverage computation ultimately win. Brockman found the reaction interesting. A lot of people read it as “compute is all that matters,” felt threatened, and he does not think that reading is true. Algorithmic ideas have been very important; to build AGI you want to push computational scale *and* human ingenuity as far as they will go. The useful question is what kind of ideas to strive for. If you can find a scalable idea — more compute, more data, and it gets better — that is the holy grail. That is why he is excited about deep learning as a path to AGI: the most successful systems, scaled up, work better.

Fridman puts the emotional objection on the table. A thirteen-year-old in Kansas may have a single GTX 1080, or no GPU at all. If scale is what matters, how can that person contribute, and should compute be democratized as much as algorithms? Brockman cuts the space of progress in two. One portion, increasingly large, does require massive compute; that is why OpenAI is built to push scale and cluster size. Another portion does not. The ideas that really shine should work much better when scaled, but you can discover them without massive resources. GANs and VAEs are the recent examples; in practice people came up with them without huge clusters, even if the first GAN images were pretty terrible. Ian Goodfellow was smart enough to know the path would work. There is a personal tradeoff, which Brockman illustrates with Harvard systems professor Matt Welsh: after tenure, Welsh went to Google for a summer and did not go back, because as a researcher the best he could hope for was that Google or Yahoo would take his proof of concept and make it actually work at scale. You can be the person who invents the GAN and never generates the coolest images. Both sides have value. Sometimes small-scale work already shows promise, as the original GPT did in June 2018. Sometimes, with a very big asterisk, behaviors emerge that are qualitatively different from anything seen at small scale, and the original inventor looks at them and says they did not think it could do that.

That asterisk is the Dota story. PPO was created by OpenAI researcher John Schulman. For Dota they ran it at massive scale, with some engineering tweaks, and got long-term planning on a time horizon they had thought was not possible. Schulman looked at it and said he did not think it could do that. Three orders of magnitude more scale was the difference. OpenAI picked Dota because it was a step toward the real world relative to chess or Go: continuous time, a huge action space, a forty-five-minute game, many units, a lot of messiness previous games had not captured. Every hardcoded Dota bot was terrible; the game is too complex to write by hand. It looked like a good place to push the state of the art in reinforcement learning.

They started with one-versus-one and used self-play: two agents that know nothing battle, one discovers something a little good, both now know it, and they get better without bound. They beat the world champions. The skill curve was a clean exponential; the human iteration loop of scaling and fixing bugs yielded very steady progress. Then they scaled to five-versus-five, the basketball version, with the coordination a team sport requires, and pushed the same self-play idea to professional level. Brockman compares the agents to insect intelligence. An insect lives a long time in its environment, bakes a huge amount of experience into its body, will not learn calculus, and navigates that environment extremely well, including things it has never seen. The Dota bots are like that: they play against humans, who never existed in their training environment and have totally different styles, and they handle it. That does not emerge from PPO at small scale. The run used on the order of a hundred thousand CPU cores and hundreds of GPUs, pouring something like hundreds of years of experience into the bot every real day.

At The International they went on stage against some of the best teams in the world and lost both games, but each went twenty-five or thirty minutes, back and forth. That, to Brockman, already showed professional level; luck could have gone the other way. The tournament date was fixed, so they pushed as far as they could beforehand. Two weeks later they had a bot with about an 80 percent win rate against the version that played at TI. Treat the match as a snapshot, not an end state. They would soon announce a finals against the world champions — likely before this podcast landed — as the last competitive milestone. The project’s goal was never really beating humans at Dota. It was pushing reinforcement learning. That, he says, they had already done.

Asked where deep learning and RL go next, Brockman insists on ideas plus scale, not scale alone. OpenAI runs a project lifecycle. A few people start on a small-scale idea — language was like that for a long time, really one person pushing. Signs of life appear, as with the original GPT, and then you put more people and compute behind it. The end state looks like Dota or robotics: a team of ten or fifteen, engineering and machine-learning science fused, results that would have been impossible otherwise. A cycle is typically a year or two; the organization was a little over three years old. A new Reasoning Team, kicked off with Ilya Sutskever, was just starting, aimed at how to get neural networks to reason, and expected to be long-term. The tests that would impress him are logic, especially mathematical logic, and dual problems that capture the same core: programming, even security analysis of code, plus out-of-distribution generalization. Fridman jokes that it would be nice if the team proved P equals NP. Brockman says that would be very, very exciting — especially if it turned out P really did equal NP. For 2019 he did not expect good reasoning results yet. Language modeling was the thing on the ramp, a technique that already worked, ready to scale.

Do we live in a simulation? Brockman sorts questions into two bins: those that yield materially different, testable predictions about the world, and those that are fun to speculate about. The simulation hypothesis, for him, is closer to asking whether there is a flying teapot between Mars and Jupiter. Maybe, but it is hard to know what that would mean for his life. Simulation as engineering is another matter. OpenAI’s robot hand Dactyl was trained in simulation with the Dota system and transferred to a physical robot. People look at the Dota work and say it is just a game, how will you ever escape to the real world? The answer is they already did it with a physical hand no one can program. Applied with the right techniques, simulation goes a lot further than you think. Whether a being inside the simulation wakes up and has consciousness is a much harder question.

Fridman asks whether creating general intelligence requires consciousness, a body, or neither. Brockman gives the un-grand answer first. Look at what already works. A lot of people would have said you need real-world experience, a body, grounding — how else would you know about smoke and fire? GPT-2 shows you can go much further than that reasoning predicts. So do we need consciousness or a body? Probably not, at least for the systems they have. Those systems already feel general in some dimensions, not as competent or as fast-learning as AGI would be, but proto-AGI in a way, and they need none of those things.

Then the grand half. If the neural nets are already conscious, would we know? Animals seem to sit on a continuum: his cat is conscious in some way, just not as conscious as a human. You could imagine a consciousness meter that gives a small reading on a cat and a much bigger one on a person. Point it at a neural net. If you are training a massive simulation, do the nets feel pain? It becomes hard to be sure the answer is no, and hard to think about what it would mean if the answer were yes. One possible story: humans have consciousness because it is a convenient computational shortcut. A being that needs to avoid pain and eat food in order to survive might be most cheaply implemented as a conscious being. If that is true, we should expect really competent reinforcement-learning agents to have some kind of consciousness too. That is a big if, and there are arguments the other way. Fridman says the idea that even GPT-2 has some degree of consciousness may not be as crazy as it sounds — more like how we already talk about the intelligence of a dog, a cat, and a human.

Last question, asked lightly: will we ever fall in love with an AI, as in *Her*, or will an AI fall in love with a human? Brockman says he hopes so. Fridman says there is no better way to end than on love.