The World's Only Capped-Profit Company

Sarah Guo and Elad Gil open 2024 with a clip show. The hosts at No Priors spent 2023 interviewing a large slice of the AI field and, this week, they are sending listeners back through a handful of favorite moments, with full episodes linked in the description. First up is Ilya Sutskever, co-founder of OpenAI. They talked with him, the hosts note, before the board asked Sam Altman to step down and then brought him back, so the clip does not touch that drama. What it does touch is the strange corporate shape that drama later made famous.

The hosts only allude to the later chaos: on 17 November 2023 the OpenAI board fired Altman, and five days later he was back. Sutskever first stood with the board, then signed the staff letter asking for Altman's return, which is why he became the hardest person in the story to read. This interview was taped before any of that, and it still explains the design that made the company so odd. From the beginning, Sutskever says, the goal was artificial general intelligence that benefits all of humanity, meaning autonomous systems that can actually do most of the jobs, activities, and tasks people do.

The first plan was to open-source a lot of technology and run as a nonprofit. The goal was public-interest; the organization matched. What changed was compute. OpenAI was among the earliest, he says, to realize that real progress in AI requires a lot of it, and that the appetite is truly endless. A nonprofit could not build a large cluster. So they converted into an unusual structure called capped profit. Investors put money in, but even if the company does incredibly well they do not get more than some multiplier on the original investment. To his knowledge, they are the only capped-profit company in the world.

The argument for the cap is the same as the AGI definition. If the technology could become capable enough to do every task people do, it might unemploy everyone. He does not know that it will. He also does not think it is impossible. If that is the case, the company that builds the technology should not be incentivized to make infinite profits. He leaves an honest gap: there will be multiple AI companies, competition will have unforeseen implications for the argument, and he does not know if it will literally play out this way. That was the thinking.

Alyssa Henry, former CEO of Square, is the most concrete voice in the reel. She spent years around small merchants: card readers, loans, payroll, the unglamorous stack of a shop. What excites her about the last year of tools is not a new demo. It is how much more broadly applicable expert assistance has become. The consumer side unlocked first, she says, and that is what showed people what the technology could do once it was integrated into a specific domain.

Talk to small-business owners and most will tell you they know they should be doing marketing. Reach customers better and you drive more business. Then they describe the actual evening: work all day, take care of family, and at 8 p.m. the choice is to chill for a minute or spend the next three hours making an image, writing campaign copy, and assembling the email. They know they should. It is too hard, it takes too much time, and they are not experts. Henry's line is the whole product thesis: they got into this because they love cupcakes, not because they like writing email marketing.

That gap was massive white space. If the thing were easy enough, accessible enough, and cheap enough, they would say yes, that would be huge. It was not. Hiring a marketing consultant was too expensive, the ROI was not there, and a decade of SaaS marketing tools quietly assumed the user had time and skill, which is exactly what the owner does not have. Henry's judgment is that the current wave is exciting because it unlocks previously unaddressed demand. A useful test of an AI product follows: does it serve people who could already hire an expert, or people who were locked out? The second market is the one that was empty.

Mustafa Suleyman, co-founder of DeepMind and later co-founder and CEO of Inflection AI, is asked how his team defines intelligence and emotional intelligence so they have measurable benchmarks while they build. He points back to Shane Legg, the person he credits as the core driver of the language around artificial general intelligence. Legg had worked on definitions of intelligence for his PhD with Marcus Hutter. Suleyman found that work inspiring. It was the turning point that made him believe the essence of human intelligence could be distilled into an algorithmic construct.

For the thesis, Legg put together something like 80 definitions of intelligence and aggregated them into one formulation: intelligence is the ability to perform well across a wide range of problems. That gave DeepMind an engineering measurement. They could constantly ask whether they were producing an algorithm that was inherently general, that could do many things well at the same time. The host asks if that is still the working definition. Suleyman says no. He has changed. It is a good definition, and in a weird way it over-rotated the entire field onto one aspect of intelligence, which is generality.

OpenAI, then Anthropic and others, took up a default mantra: all that matters is whether a single agent can do everything, whether it can be multimodal, whether it can translate, generate speech, recognize speech, and so on. Suleyman offers another definition he thinks is valuable: the ability to direct attention or processing power to the salient features of an environment given some context. What you want is to take raw processing horsepower and point it in the right way at the right time. A certain tone may be more appropriate. A certain expert model may be more suitable. You may actually need to go use a tool.

The key element that will unlock the field, he says, is the router in the middle of a series of specialized systems. Some of those systems will not even look like AI. They might be traditional software, databases, tools. The router, or central brain, is the decision-maker, and it does not necessarily need to be the largest language model we have. After a year of scaling slogans, this is a DeepMind veteran's vote for triage over a single giant generalist: the hospital receptionist who sends you to orthopedics or the ER does not have to be the best surgeon in the building.

Reid Hoffman, Inflection AI investor and LinkedIn co-founder, takes the labor question. Listeners already agree, he says, that AI sits somewhere between the largest tech transformation of our lifetime and perhaps the largest in human history. He describes it as the steam engine of the mind. The steam engine gave physical superpowers of construction, transport, and manufacturing. This one will give mental superpowers. That is the implication for humanity he was gesturing at in the book *Impromptu*. It is also true that in some places there will be substitution, replacement of work. That is the macro picture.

On timing, everyone tends to over-predict how quickly everything will change next year. That is not going to happen. Then they under-predict the ten- and twenty-year transitions. Doomsayers come out first for every technology. The printing press was described as degrading human cognitive capabilities and spreading misinformation. Electricity got the same treatment. You can go back and find the end of the world in each of these things. Hoffman has seen a few cycles. He does not deny replacement. He wants the attention on amplification.

Technologists default to full autonomy. Self-driving cars are the case where he thinks that is right: you do not need an amplification loop, you need effective logistics and safety, and you want to save the 40,000 deaths currently attached to human-driven vehicles. A whole bunch of other work will actually be better with people plus AI. "That plus is a thing to focus on." He wrote *Impromptu* the way he did because he had not seen enough creation, theorizing, and design in that symbiotic loop. Writing, coding, medicine, and teaching are plus problems. Driving, in his view, is not.

Daphne Koller of Insitro is there for the economics of biotech discovery, but the clip the hosts keep is about architecture. In most fields, she says, there is a swing of a pendulum. Early work in probabilistic graphical models, a literature she helped define, pulled artificial intelligence into machine learning and numerical data and away from purely symbolic AI. Then deep learning shoved that aside. There was so much power in pattern recognition from raw images and text that interpretable representations could wait.

She thinks the pendulum is starting to swing back. You need both: the huge pattern recognition of deep learning, and the ability to reason about causality, plus enough interpretability that you can convey why the model did what it did, which matters if the audience is a doctor. The powerful paradigm, in her view, is a synthesis of the two disciplines. Insitro sits exactly where that synthesis is not optional. Drug discovery cannot live on uninterpretable pattern matching alone, and it cannot go back to symbolic systems that never saw the data.

Noam Shazeer, the celebrated Google engineer who co-founded Character.AI, has had his head down in language. Language is a problem that can do anything. He wants the model good enough that he can ask how you cure cancer and it invents a solution. He ignored the other modalities while everyone got excited about early deep-learning wins in images. His reason is a density joke: an image is worth a thousand words, but it is a million pixels, so text is a thousand times as dense. He is, he says, a big text nerd.

He is still excited to see the other modalities take off. They will be super useful for products people want to use. He thinks a lot of the core intelligence is going to come from the text models anyway. Put next to Koller, the two clips are a split bet on 2023's technical center of gravity. She wants deep nets married to causal, explainable machinery. He wants the intelligence to keep growing out of language, with pictures and speech as the product skin.

Arthur Mensch, co-founder and CEO of Mistral, is there to explain why the company's mission is to keep AI open. Models can output any kind of text, and in many cases you do not want that. When you build an application you have to think about guardrails on output and potentially on input: filter what you deem invalid or illegal on the way in and on the way out. Mistral's answer is a modular architecture the application maker can use. Provide the raw model, unaltered, so you have not banned regions of its output space. Then propose filters on top that detect what you do not want, pornography or hateful speech for a chatbot.

You do not want to ban those things from the raw model, because if you want to use the model for moderation it has to know about the bad content. Assuming the model should be well-behaved is, he thinks, a wrong assumption. The right assumption is that the model should know everything, and then you put modules on top that moderate and guardrail. That empowers the application maker. Mistral treats good guardrail modules as part of the platform, and it wants healthy competition among startups in that layer. You do not get that by trusting a couple of companies to do their own safety.

You get it by requiring application makers to comply with rules: chatbots should not output hateful speech, so they have to find a good guardrailing solution, and now there is economic interest in providing the best one. That is how Mensch thinks the ecosystem should work, and how Mistral is building the platform. Jensen Huang, co-founder and CEO of Nvidia, closes the reel on a different kind of split. A very large part of the company is designed to build very complicated computers perfectly. That organization is an invention-and-refinement shop. Then there is a whole bunch of skunk works, trying to invent things ten years out that they are not exactly sure will work, with a lot of adaptation and pivoting.

Nvidia, he says, actually has two different ways of working. One is organic and shape-shifting: if a particular investment is not working out, they give up and move the resources. That is the agile part. The other is not rigid, but it is really refined. "These two systems have to work side by side." The hosts thank listeners, point back to the full episodes, and send people to no-priors.com for transcripts. The eight voices do not agree. Sutskever is betting on a capped giant, Suleyman on a router, Mensch on open weights plus filters, Huang on running a factory and a scout team at once. 2024 would keep stress-testing all four.