Kevin Scott Seized Microsoft's GPU Budget, Bet on OpenAI, and Still Didn't See ChatGPT Coming

Kevin Scott calls the path from a farming community in central Virginia to Microsoft CTO an unlikely journey he never could have planned. His parents did not attend college; his father and grandfather were construction workers. What he did have, as a teenager in the early 1980s, was an obsession with personal computers and enough scraped-together money for a Radio Shack Color Computer — chiclet keys, hooked to a 13-inch television, programs stored on a cassette recorder. He stayed obsessed, and at each step he tried to work on the most interesting thing someone was willing to let him touch.

The first vision was to become a computer science professor. He won one of two slots from his high school at a regional Governor's School covering four or five rural counties, and Dr. Tom Morgan, the computer science teacher there, opened a world he found thrilling. He spent the next decade and a half as a compiler-optimization, architecture, and programming-languages person. Around age thirty he got disillusioned. The impact a professor can actually have, he told Sarah Guo and Elad Gil, is inspiring students who go on to do greater things than you ever will. That felt like the greatest impact, and the least appreciated part of the job in the 2000s.

He did not know what was next. Academic friends were joining a startup called Google, which he dismissed as a box you typed keywords into that returned ten results. Then he noticed who worked there: Urs Hölzle, Jeff Dean, Alan Eustace — compiler people whose papers he had read and whose conferences he had attended. He sent a resume. Google put what felt like every compiler person in the company on his interview panel, which he still describes as the best interviewing day of his life. The company had just opened its first office outside Mountain View; he and his wife wanted New York more than Mountain View, so he became roughly the tenth person in Google New York.

The new mission arrived after he got there. Google was hiring brilliant people with almost no assignment, and many of them chose intellectually interesting work that would never move the needle — the same impact problem he had left academia to escape. He picked an unsexy problem, solved it with serious computer science, and the team won a Google Founders Award. Then he became a manager so he could help more engineers line their work up with something both interesting and meaningful. Guo notes that early Google put academics in charge of important pieces of the company in a way the industry almost never repeated. Scott says sitting at OpenAI now gives him early-Google nostalgia: similar size, similar energy, even string theorists hired into computer-science roles again.

Scott had been at Microsoft a little over six years when they recorded, and he says it was obvious almost immediately that AI would be extremely important to the company's future. Microsoft already understood that before he arrived. The problem was focus. There was no shortage of IQ or capital, but the investment was peanut-buttered across a pile of projects. The catalyst, for him, was technical: transfer learning was starting to work.

The statistical machine learning he had cut his teeth on at Google in the early 2000s was siloed. You had a domain of data, a model architecture, a deployment and measurement plan, and it all stayed inside one use case. Transfer learning meant training on one set of data and using the system for multiple purposes. He saw a hint of that in DeepMind's reinforcement-learning work transferring across games. The real excitement arrived when it started working for language — ELMo, then BERT, RoBERTa, Microsoft's own Turing models, and a stack of related efforts. Suddenly there were so many language applications you could imagine building *if* the systems kept getting better.

The team looked for one kind of evidence: that the stuff would continue to improve. As soon as they found it, they went all in. That included the OpenAI partnership, and it included a more internal move. At one point Scott seized the entire GPU budget for the company and announced that those resources would no longer be spread around. Capital-intensive bets would go only where there was strong, evidence-based conviction that adding scale would actually pay off. Elad Gil recalled a dinner five years earlier, before GPUs were the talk of the technical town, when Scott's most pressing issue was already how to spend the GPU budget that year and how to distribute the machines. It has not gotten easier.

Why partner rather than only build? Scott gives two reasons. First, if transfer learning holds, these systems become a platform whose cost you amortize across many applications. Microsoft runs a hyperscale cloud, and he treated it as almost existential that the infrastructure, networks, and software frameworks serve ambitions beyond Microsoft's own. That meant high-ambition external partners, and OpenAI was clearly the most ambitious lab in the field — still is, he says, with breathtaking goals. Second, the two sides shared a vision that the models were evolving into platforms. Alignment did not make the partnership easy. He jokes that there is a conservation law of stress. It did mean the stress was in service of a common goal, which is what makes a hard partnership work.

Gil presses on timing. Microsoft's first significant investment landed around GPT-2, before GPT-3, when the step-function between the two was not obvious. Scott refuses to rewrite history. There was a huge diversity of opinion inside the company on the wisdom of the deal. Satya Nadella talks about no-regrets investing: bets with multiple ways to win, where you still learn something if you lose. The worst case here was spending a lot of capital on computing infrastructure and learning how to build AI training environments at very high scale. You would have to believe something strange about the future of AI, Scott says, to think you would not need advanced computing.

They also had evidence — some of it from OpenAI that he cannot discuss — that scale-up was working. The public clue he is willing to cite is OpenAI's famous compute paper: from 2012, when people figured out how to train with GPUs, through about 2018, state-of-the-art training consumed roughly ten times more compute every year. That gave him "super super high confidence" they would never reach the point of saying they had enough compute. The first machine Microsoft called an AI supercomputer was started in 2019, deployed at the end of that year, and became the environment GPT-3 was trained on. Smaller stamps of the same design now serve lots of people inside the company and partners on Azure — a compute grid of that size, with that kind of network, that simply was not possible to rent before.

NVIDIA has been the compute and networking partner since it bought Mellanox. Generation over generation the systems get better price-performance, and Microsoft works closely with NVIDIA on what the next GPUs need, because it has a clear sense of where model architectures are going. Hopper was being deployed at scale at the time of the taping, including features such as 8-bit floating-point arithmetic that the two companies had been planning for a long while.

Scott rejects the idea that open and closed models are a binary choice. Microsoft still sees no reason to stop building bigger models, but its own deployments — Bing Chat, Microsoft 365 Copilot, GitHub Copilot — already use a portfolio, mixing models for cost and performance as well as for precision and quality. He is genuinely excited by open-source technical innovation. The big question mark is how the community handles responsible AI and safety. The day before the taping he had been playing with Databricks' 12-billion-parameter Dolly 2.0, which runs nicely on a single machine. He is, he admits, still enough of a dork to love things that fit on one box.

For business customers, Microsoft started from an assumption: AI will be a platform, and the most useful way to use it is building tools that assist people with jobs, not fully autonomous scenarios. The first product was GitHub Copilot. You say in natural language what you want a piece of code to do; it emits the code; you scrutinize it the way you would a pairing partner's suggestion. They then noticed that the Copilot pattern is generic, and they built a bunch of them. Scott describes the Copilot stack as something like an OSI diagram: user-interface patterns and an emerging plugin ecosystem at the top; an orchestration layer, including open-source tools such as LangChain and Microsoft's own open-sourced Semantic Kernel; prompt construction and a "meta prompt" of standing instructions so the model conforms to the application; retrieval-augmented generation, or RAG, which Microsoft was doing before the acronym existed; and a safety apparatus that filters on the way down to the model and on the way back up, sometimes with multiple round trips before a response reaches the user.

Enterprises add extra questions: data privacy, where data flows, which plugins can be enabled. Microsoft will build some of this. The community will build a tremendous amount of it, because there has never been a platform where one company ships all the useful pieces. "That's just nonsense," Scott says. Immediately before the interview he had been in a review with Microsoft Research, watching researchers pivot onto AI-centered work. It felt like MSR in 2001, when he interned there and the PC was racing toward ubiquity: bright people with the tiniest glimpse of a future, reorienting everything around it.

The biggest surprise, even for him, was how much GPT-4 and ChatGPT catalyzed. ChatGPT, he notes, was a ten-month-old model with a little RLHF on top and, by admission, not a beautiful user interface — a way to get practice on a handful of things before the GPT-4 launch. Nobody really knew it would blow up. At taping it was only five months old, which he says people keep forgetting. The pace is disconcerting even for someone in one of the better seats. It is often harder, he adds, for machine-learning teams than for a brand-new entrepreneur, because it is a different way of doing the work, and even people inside Microsoft who have had time to think still struggle to adjust.

Guo asks how a company with Microsoft's portfolio should organize adoption, and what he tells friends running other large companies. Some things have changed, Scott says, and some have not changed at all. The confusion many people get wrong is simple: models are not products, and infrastructure is not a product. You do need to understand quickly what this new platform can do. That does not let you skip the hard work of figuring out what a good product that uses it looks like.

The place he tells people to look for the most interesting products is the phase change from impossible to hard — something you literally could not do before the technology existed, which is now merely difficult. Things that went from impossible to easy are probably not interesting. His frivolous example is the smartphone. The iPhone launched in 2007; the App Store followed a year later. The first apps were the ones that had gone from impossible to easy: fart apps, a "woo button" that played a Ric Flair yell. Those were not businesses. What made the smartphone platform were the hard, non-obvious things the platform builders themselves did not imagine. Your phone is no longer interesting because of SMS, a browser, and a mail client. It is interesting because of TikTok, Instagram, WhatsApp, and DoorDash.

The same warning applies to incumbents. The job is not to sprinkle LLM fairy dust on existing products and ship a boring incremental feature. Incremental things may be fine. The really interesting things are non-obvious and very not incremental. That is the management problem: you have a building full of smart people who can see everything that is now possible. The challenge is to steer them toward the hard, meaningful, non-obvious work, and away from incremental ideas that burn GPU cycles and product IQ you will need for the things that actually matter.

Scott's 2020 book, *Reprogramming the American Dream*, was not written for people like the hosts. The premise is the world he grew up in: a construction-worker father, a construction-worker grandfather, a maternal grandfather who repaired appliances and had farmed. The neighbors were smart, entrepreneurial, ingenious people using the best tools they could get their hands on to create opportunity and solve problems for their communities. He believes the platform vision of AI — cheaper and more accessible all the time — is for them too. Don't be daunted or scared, he wanted readers to hear. Plug it into the things you already do, and maybe the benefits of the platform get distributed more evenly.

He uses his own first project as the measuring stick. In 2003 and 2004 at Google, with a graduate degree and real mathematical sophistication, he built a machine-learning classifier. That meant stacks of technical papers, *The Elements of Statistical Learning* cover to cover, and six months of writing code. A high-school student, he says, could now do the whole project in four hours on a weekend. The aperture of who can use the tools just keeps getting bigger. He had been anxious while writing that the manuscript would be stale by press time. The real problem was worse: the book hit the presses the week the pandemic shut everything down. Amazon was delivering only essentials; every bookstore in the country was closed. You literally could not buy it.

What surprises him is how much of the argument is still true, even stronger. The platform is more powerful and more accessible; the unit economics are better; what you can do per token of inference is going up. GPUs are expensive, but optimization work keeps squeezing more out of the compute you have, and the compute itself is getting cheaper. What the public dialogue is missing, he argues, is the opportunity side. Responsible AI and regulation matter, and they fill the newspapers. There should also be room for Sal Khan's TED talk. Khan's problem is a perfect fit for AI: the two-sigma problem, from Benjamin Bloom's study, that a tutored student performs above 98 percent of students in a normal classroom, with reduced variance. If you believe that, then every learner in the world deserves high-quality individualized instruction at no cost. The only realistic way to realize it is something like AI. That, Scott says, is why he gets up every morning and deals with people yelling "give me my GPUs" for the fifth year in a row.

Optimism is not a middle finger to responsible AI or to people who care about regulation. You can hold both. What he will not accept is starting from gloom. "There is no historical precedent where you get all of these beneficial things by starting from pessimism first," he says. "Pessimism doesn't get you to optimistic outcomes." Guo adds that the missing pieces of the public conversation are global education equity and health equity — cheaper, personalized teaching and physician-assistant-level help — and that the golden-age story is the one that keeps getting lost.

Gil puts the question in parental terms. Two years earlier he would have told his kids to study computer science as the language of the future; now he is not sure. Scott declines a twenty-year forecast. Rewind twenty years, he says, and nobody would have predicted this present. Some things still look sturdy. His daughter wants to be a surgeon, and he thinks that is a pretty good job. There is a cognitive exponential underway, not a robotics one. Jobs that change a physical system — nurses, surgeons, physical therapists, people who work in nursing homes — will probably be needed in much larger numbers, especially with a rapidly aging population. AI will have substantial productivity effects. Maybe those effects are just enough to make room for the net-new demand the healthcare system will have to absorb.

The United States, he thinks, apportions less dignity to jobs like his father's than it should. He lived in Germany for a while, where becoming a machinist is a career parents celebrate. Electricians, machinists, solar-installation technicians, and a long list of related trades will be needed, not least because the entire power-generation and distribution system will have to be rebuilt in our children's lifetimes. Creative work, even with better tools for coding and making podcasts, will probably be needed more rather than less. Humans are extraordinarily good at putting humans at the center of their stories. You could already make *Machines' Gambit* instead of *The Queen's Gambit*, a show about computers playing chess among themselves because they are better than any person. Nobody wants to watch that. Superhuman Formula One drivers on a closed track, forklift strongman contests — same problem. We care what people are trying to express. Guo's punchline: that is why people watch *The Real Housewives of Dubai*. Scott does not want to paint too rosy a picture. Every major platform shift brings disruption. What history also shows is a surprising degree of need for human occupation. Industrial Revolution predictions of four-hour workweeks and lives of leisure were, in his word, bullcrap. A lot of that is because people actually like doing things.

Looking twelve months ahead, he expects the first great year of foundation-model deployments: lots of companies launching, lots of ideas getting tried, every big tech company shipping something substantial. It will touch Microsoft's entire product portfolio. The way you interact with the software will be substantially different by the end of the calendar year than it was on the way in, and he thinks that will be true for everyone. The competitive map between big tech and small tech will shift; small companies will be able to drive wedges and get footholds. One thing he and Sam Altman have talked about a lot: he suspects the next trillion-dollar company gets founded this year. It will not be obvious which one, but the industry is long overdue. Technically he expects projects like RedPajama, and others like it, to make real progress on more capable open-source models, hopefully with safety solutions to match. Multimodal models that can take visual inputs and reason over them will unlock another wave, the way GPT-4 did. The theme of the year, he says, will be progress and activity almost too much to track. He is going to need a Copilot just to make sure he is not missing the next important thing.

He wants to end on regulation. The week they recorded, Altman was testifying before the Senate Judiciary Committee. Scott thinks more of those conversations are a good thing, and that the technology community needs to agree on sensible practices before the rules even exist. Treating regulation as a pain is the wrong reflex. Desire for it is a signal that the work matters, because nobody tries to regulate frivolous things. Electricity is his analogy: let the technology flourish, but build generators that do not let people disintegrate themselves on the electrodes, coordinate the grid, and make sure power coming into a house does not burn it down. Most of the thinking should be about deployments. The largest foundation models already sit behind layers of safeguards, and each application and each deployment needs another layer. Startups and the open-source community all have a part. As Microsoft's CTO he competes fiercely with OpenAI, Anthropic, Google, and the rest. As Kevin, citizen of the world, he is not worried those particular competitors are going to do something unsafe.