Miles Brundage on Leaving OpenAI and Preparing for AGI

Matthew Berman reviews an October 2024 post by Miles Brundage, a departing OpenAI policy researcher, who wrote that neither frontier labs nor the world were ready for AGI. Brundage had spent six years at OpenAI, most recently as senior adviser for AGI readiness, after earlier policy research work at Oxford.

Brundage says he had committed his career to the broad goal of ensuring that AGI benefits humanity. Advising OpenAI executives and its board on readiness was, in his words, essentially his dream job, making the decision to leave difficult.

He says he wants more time for issues that span the industry, more freedom to publish, and greater independence. At the time, he planned either to start a nonprofit or join an existing one, while still describing OpenAI as an exciting place to work for people suited to its culture.

Brundage recounts his OpenAI roles, from policy research scientist to head of policy research and senior adviser for AGI readiness. He concluded that he could influence AI development more effectively from outside the industry and pursue important topics that his existing responsibilities had left little time to address.

Berman interprets this in light of OpenAI’s product focus and other departures. Brundage’s own account is narrower: he acknowledges that some industry publication constraints are reasonable and helped write policies, but says the constraints had become too great for him. Berman’s speculation about other employees’ reasons is not evidence of their individual motivations.

Brundage also wants to be perceived as more impartial. Employment at a company affects how others receive a researcher’s statements, and he believes independence can make it easier to examine the organization and the wider industry critically.

He distinguishes OpenAI’s readiness to steward increasingly powerful systems from the world’s readiness to manage them. He says he has already advised executives and the board about gaps in the former and expects to be more effective externally on the latter, though working outside a company can mean less access to internal information.

Brundage’s assessment is that neither OpenAI nor any other frontier lab was ready for AGI, and society was not ready either. He says he did not regard this as controversial among OpenAI’s leaders. Crucially, he distinguishes readiness at that moment from the separate question of whether institutions might be ready when it becomes necessary.

Berman reviews an earlier Brundage assessment that assigned grades to aspects of AI governance. These were the researcher’s judgments from roughly a year before the video, rather than standardized measurements or an up-to-date report card.

The earlier assessment gave shared understanding of AI’s upsides and downsides a B-minus and regulatory infrastructure a C-plus, while calling for more technical work. Brundage gave legitimacy a D-plus under a demanding standard and saw a need for greater public participation and institutional trust.

Societal resilience received an F in that historical assessment. Berman connects this to deepfakes and the difficulty of maintaining provenance or watermarking across changing media and providers. Brundage also gave differential technological development a D-plus; Berman interprets this as concern that market incentives alone may not adequately support safety.

Berman notes that AGI has no single universally accepted definition and describes a five-level framework associated with OpenAI: conversational systems, reasoning systems, agents, systems capable of innovation, and systems that can perform organizational work. He locates 2024 models near the first two categories, a judgment rather than an independent certification.

He describes the later levels as increasingly autonomous agents, systems that can innovate or improve themselves, and organizational AI. Berman connects self-improvement to the possibility of an intelligence explosion, but the stages and any transition between them are a conceptual framework, not a proven trajectory or schedule.

Despite leaving, Brundage says working at OpenAI can be highly impactful for people with the appropriate skills and mission. He urges employees to take seriously how their actions and statements shape the company’s culture and may create lasting consequences as it handles more advanced capabilities.

For his next step, Brundage describes a likely mix of AI policy research and advocacy, with the details still to be determined. He invites potential collaborators to get in touch rather than claiming to have already established a new organization.

Brundage regrets the polarization of AI policy into camps that emphasize either benefits or risks. Berman interprets this as a critique of the acceleration-versus-pause binary and says he himself is optimistic about AI while recognizing that things can go wrong.

Berman places himself closer to the accelerationist side and argues that AI could greatly benefit humanity. Brundage’s position, as presented in the post, is that the benefits of AI and AGI are not automatic: governments, companies, nonprofits and civil society must make deliberate choices informed by public discussion.

That responsibility, Brundage argues, extends beyond mitigating risks to distributing benefits fairly, much as societies have had to consider access to electricity and medicine. Berman agrees that equitable access matters, especially if the most capable systems become costly.

Berman recalls California Governor Gavin Newsom’s veto of SB 1047 and explains why he opposed regulation tied to fixed thresholds. He worries that technological change can outpace a rule focused on a model’s original training scale. The vetoed bill used compute and cost criteria, including provisions for certain fine-tuned models; Berman’s reference to a parameter-count threshold should not be read as the bill’s definition.

As an example, Berman says a smaller model might achieve stronger performance by using more compute at inference time. Brundage likewise argues that public analysis of compute governance has lagged industry developments and that scaling inference, not only training, can improve performance. This does not establish that every training-based rule would be rendered ineffective.

Brundage calls for stronger funding for the US AI Safety Institute and the Bureau of Industry and Security, including capacity to understand what happens to advanced chips after export. He also favors opportunities under existing legal authorities and attention to implementation of legislation already approved, including the EU AI Act.

Berman expresses skepticism about the EU AI Act and its effect on innovation. He says he has not read much of Brundage’s wider policy work, so his description is an interpretation of the specific post, not a complete account of Brundage’s regulatory proposals.

Brundage argues that private and social incentives may diverge, so companies and governments may not give safety enough attention by default. He also sees a problem of credible commitments: competitors can suspect others will cut corners while finding it hard to verify the actual safety of a system.

Berman explains the resulting challenge: how can an organization demonstrate safety without revealing sensitive or potentially misusable intellectual property? Brundage compares this to the difficulty of verifying arms-control commitments while protecting confidential information.

Brundage speculates that AI-enabled economic growth in coming years could eventually make early retirement at a high standard of living feasible. He simultaneously worries that automation of remotely performed tasks could disrupt opportunities for people who urgently want work. Both the timeline and the distribution of any gains are uncertain.

He sees reducing the obligation to work for a living as a potential long-term reason to build AI and AGI. Berman finds the possibility exciting while acknowledging that a society in which people need not work would raise further questions he does not explore in detail.

Brundage predicts that the gap between free and paid AI capabilities may widen as companies devote more test-time compute to customers willing to pay. He points to a brief 2024 period when GPT-4o was available to both free and paid users with differing rate limits. The widening gap is his forecast, not a guaranteed outcome.

Berman expects increasingly capable low-cost or open-weight models, citing Meta, but recognizes that running additional inference requires hardware and electricity. Access to model weights does not eliminate the cost of operating them; the ultimate economics and distribution of access remain open questions.

Brundage considers much of the AI-for-good field well intentioned but insufficiently supported by technical expertise in civil society and government. He also argues that computing hardware is a useful governance focus because it is measurable and produced through a concentrated supply chain.

He worries that the government offices overseeing exported compute lack resources and that policy debate lags technical developments. Berman returns to the point that inference scaling complicates an approach focused solely on training.

Brundage calls for a broader AI strategy that weighs the benefits and drawbacks of decentralized development. He asks what policies make sense with or without international agreements, and what options lie between today’s largely market-led approach and nationalized or internationalized research.

He also asks how to have a serious discussion about the appropriate pace of progress beyond a simple acceleration-versus-pause split. Berman presents these as research and policy questions, not as a settled prescription.

Finally, Brundage says OpenAI offered funding, API credits and early model access for his prospective work, with no associated pre-publication review. He had not decided whether to accept, because even helpful support could affect actual or perceived independence. Berman welcomes the offer and closes by inviting reactions.