Zico Kolter on AI Safety, Scaling and the Limits of Open Models

AI-generated misinformation may accelerate a loss of trust in evidence, says Zico Kolter. He argues that people increasingly reject material that conflicts with what they already believe, making the erosion of shared facts a more serious danger than everyone believing every fake.

Harry Stebbings introduces Kolter, a Carnegie Mellon professor and newly appointed head of its machine learning department. Kolter has also recently joined OpenAI's board; the episode description identifies him as a Bosch chief expert and chief technical adviser to AI safety startup Gray Swan. Those roles are relevant to his views on the industry and safety.

Kolter begins with a simplified account of large language models. They learn mathematical relationships from large amounts of text by predicting the next word, then generate responses by repeatedly making similar predictions. AI is broader than language models, but these systems are the focus of the discussion.

Some critics dismiss next-word prediction as incapable of intelligence. Kolter disagrees: he considers the coherent, capable responses that emerge from this process demonstrably intelligent and calls the discovery one of the most notable scientific findings of recent decades.

Stebbings asks whether AI developers have exhausted available data. Kolter distinguishes high-quality, publicly accessible text, much of which has already been used, from the much larger universe of possible data. There is only so much material comparable to Wikipedia, but text is not the whole story.

He estimates public models have trained on the order of tens of terabytes of text. Video, audio, private material and other sensory data are much more abundant, though it remains uncertain how effectively models can use them. Access, processing and computation all impose constraints.

Video is vastly larger than a transcript of the same conversation, making it expensive to process. Kolter nevertheless expects visual, temporal and other sensory information to matter for intelligence. He says researchers have not yet learned how to exploit those modalities well or generalize between them.

Even if the available data were fixed, Kolter says models need not immediately plateau. Larger models can sometimes improve when trained repeatedly on the same dataset, and today's methods do not necessarily extract all the information already present in their inputs.

As an illustration, he describes an image-classification experiment in which synthetic images generated from the original data provide a small improvement. The mechanism shows that training can extract additional information; it does not mean synthetic data automatically solves the problem. How best to extract the available information remains an open research question.

On the expected shift toward smaller models, Kolter says the eventual balance is unclear. He personally prefers the largest available models for varied daily work because they perform better. Smaller specialized models may become economical for well-defined tasks repeated many times.

Stebbings cites the argument that improvements are becoming harder to notice. Kolter thinks familiar questions and saturated benchmarks can obscure progress: a small model may already answer a standard question adequately, while newer models make a noticeable difference on harder coding and lecture-related tasks.

As models become cheaper and more numerous, Kolter expects some consolidation but cannot predict the market's final shape. Training a model from scratch merely to show that a company can do so may cease to be economically sensible when good models are already available.

He is unconvinced that compute scaling has reached a hard plateau. The scaling laws he has seen suggest further gains, although they become more expensive and other methods may deliver improvements more efficiently. Training cost and inference cost make the economics as important as the technical possibility.

Stebbings asks whether model companies are pursuing AGI or profitable products. Kolter says the aims need not be mutually exclusive, while noting that people use the term AGI in very different ways.

His own demanding definition is a system as useful as a close collaborator across a year-long project, potentially in virtual form. Applied to his own work, it would perform most of his professional functions rather than merely help with a single isolated task.

Kolter believes such a system may arrive in his lifetime but emphasizes enormous uncertainty. His rough range is four to fifty years, and he does not personally favor the short end; nonetheless, he says the pace of progress makes it difficult to dismiss that possibility.

He agrees that societies can adapt to technological change without minimizing the scale of disruption. In his view, the most successful companies will not simply replace workers and keep doing the same things; they will learn how people can guide the systems and use their workforce to make better use of the technology.

Current AI products do not yet give workforces an ideal path to their full capabilities, he says. Large enterprises also worry about moving data and training on proprietary information, although many already use cloud services to store sensitive data.

Kolter says indiscriminately training on all internal data is problematic because employees have different access rights and model training can expose information. Retrieval-augmented generation offers another approach: retrieve relevant material a user is authorized to see and place it in the model's context to answer a question.

He distinguishes answering a question with supplied information from retraining a model on that information. He says some API arrangements do not use the submitted data to retrain the model; the actual treatment of data depends on the service and its terms. He sees misunderstanding of this distinction as one obstacle to adoption.

Kolter is frustrated by confidence at both extremes: that AGI is certainly imminent, or that current approaches could never lead there. His own beliefs have changed in response to observed progress, and he wants others to update their views as evidence accumulates.

On misinformation and deepfakes, Kolter worries especially that people will cease believing evidence at all. AI did not invent propaganda or distrust, he says, but it can accelerate them and make it harder to establish a common account of events.

He suggests that reliance on trusted associates predates modern recordings and that shared visual evidence is historically recent. Stebbings asks whether retreating to personal trust could impede knowledge. Kolter agrees there are serious costs but expects communities that value science to continue producing knowledge, as they did before video.

Stebbings argues established media brands may become more valuable as trusted sources. Kolter notes that people already disagree about which institutions deserve trust. The challenge of evaluating sources will persist even if some audiences rely more heavily on familiar organizations.

Asked whether platforms should police content, Kolter says moderation already involves balancing unlawful content and room for differing viewpoints. AI intensifies an existing problem rather than creating misinformation from scratch, so he hopes existing social, economic and government structures can still provide useful guidance.

Kolter stresses that regulation is not his own field. He sees a role for government in making technology beneficial, but worries that rules framed around rapidly changing technical details can become obsolete quickly or misunderstand how systems are developed.

He thinks regulating downstream uses may often be more tractable, with existing laws potentially adapted to the volume and speed of AI-generated material. At the same time, he does not think existing law alone resolves every issue and acknowledges that effective regulation of a fast-moving technology is difficult.

Kolter's principal current safety concern is that models cannot reliably follow a developer's specifications. Instructions from users or untrusted material can override intended behavior, a family of problems discussed under names including prompt injection and jailbreaking.

He contrasts this with the expectation that conventional software will follow its programmed rules. A chatbot giving a prohibited answer may be relatively limited in consequence, but a model embedded in a larger system can encounter third-party data that attempts to redirect its actions.

As agents read external material and act on the world, Kolter compares the problem to a known security vulnerability without a complete patch. Researchers are making progress, he says, but dependable control remains unresolved and magnifies other risks.

He is particularly concerned about cyber risks and says models can already help analyze code for vulnerabilities. Provider instructions may seek to prohibit harmful uses, but if those safeguards can be bypassed, capabilities remain harder to contain. His assessment is a concern about systems at the time of the interview, not proof of every hypothetical attack.

The next question is what harmful capabilities become accessible. Kolter describes cyberattacks as a clearer near-term concern to him than biological threats, while emphasizing that other specialists take chemical and biological risks seriously and that he does not dismiss them.

The danger is not limited to an autonomous system acting on its own. In his scenario, a sufficiently capable model could lower the expertise required for malicious acts and put dangerous capabilities within reach of many more people. He argues that developers should assess those potential capabilities seriously.

Stebbings raises a comparison between AI and nuclear weapons. Kolter prefers an analogy to nuclear technology more broadly: the same general field can produce destructive weapons and useful low-carbon power. AI likewise has many beneficial applications, making a weapons-only comparison misleading in his view.

Kolter supports open models and prefers the term open weight when the model can be run locally but other elements are not open in the traditional sense. Open-weight releases have made his academic research possible and, he argues, have become an important part of the AI ecosystem.

At the time of the interview, he says he would not be especially nervous about open models with the capabilities of contemporary leading closed systems. He considers the then-current Llama model close, though not quite equivalent, and notes that it had not caused an observed catastrophe; this is a time-specific judgment rather than a guarantee.

That support has a limit. Kolter expects capabilities may eventually reach a point at which releasing model weights for anyone to modify and specialize deserves caution. His example is a hypothetical system that could reliably find exploitable vulnerabilities across virtually any software.

Such a capability might also help defenders, but he argues that patching software across the world is harder and slower than finding vulnerabilities. For that reason, he would not automatically release weights for a model with that demonstrated capability.

He welcomes the then-common pattern in which the most capable systems appeared first as closed models and open-weight equivalents followed later. In his view, that gap can provide time to study capabilities in a more controlled setting before considering broader release. He acknowledges that the sequence was not inevitable.

Kolter says most safety work should address practical problems already present in systems, such as reliable instruction following and misuse. He is also glad researchers consider more speculative scenarios involving powerful autonomous systems, even though he regards some as far-fetched and outside his main area.

To bridge near-term and existential concerns, he describes a hypothetical future in which AI agents are integrated deeply into software and critical infrastructure. A correlated failure affecting power systems could have catastrophic consequences whether it came from an ordinary defect or deliberate behavior by an autonomous system.

The CrowdStrike incident provides a contemporary reminder, he says, that shared failures can propagate widely. Because researchers do not fully understand model internals, failure modes or attack vectors, deployment in critical infrastructure warrants particular care. He also regards biological risk as potentially serious, while noting it is not his primary area.

Stebbings points to voice cloning as a challenge to voice-based authentication. Kolter agrees that existing systems and assumptions are already being disrupted and will need to adapt as AI capabilities evolve.

Despite the risks, Kolter describes himself as optimistic. He wants useful AI tools, potentially including AGI, to serve human purposes. His motivation for safety research is practical: if people are to benefit from these systems, they must be safe enough to use.

In the quick-fire round, Kolter says he once thought clever model architectures were central to progress. He now believes architecture matters less than he previously assumed and that other architectures might work if given enough development effort. This is his interpretation, not a settled finding.

He also changed his mind about data. He had expected valuable data to require heavy manual curation and labeling; modern systems showed him that large amounts of existing internet data could produce remarkable results. He still sees value in curation.

Kolter says OpenAI board chair Bret Taylor contacted him shortly before he became CMU department head. He joined to bring expertise in AI and safety to the board; he confirms there are four board meetings a year, while describing his role as providing technical perspective.

Asked whether China is two years behind the United States in AI, he avoids endorsing a specific gap. He recognizes competition between countries but argues that AI safety needs international cooperation and shared standards because its consequences cross national borders.

His final complaint is that questions about what architecture replaces the Transformer overemphasize model structure. He would rather focus attention on the data used and the capabilities systems produce, while acknowledging that his view makes for a provocative sound bite.