AI Workers’ Right to Warn and the Debate Over AGI Safety

AI safety has been on many people’s minds. In this June 2024 video, I examine an open letter by current and former frontier AI employees, as well as views from former OpenAI researchers. The discussion takes place amid controversy over non-disparagement provisions in OpenAI exit agreements.

The June 4 letter, “A Right to Warn about Advanced Artificial Intelligence,” says AI could bring substantial benefits but also presents risks ranging from inequality and misinformation to a possible loss of control over autonomous systems. I am particularly concerned about manipulation and misinformation; I personally do not believe human extinction will occur, though that is a belief rather than a demonstrated conclusion.

The signatories hope scientific, public and policy input can help mitigate risks, but argue that AI companies have financial incentives to resist effective oversight. They say companies possess non-public information about system capabilities, limitations and safeguards, while external disclosure obligations are limited.

Berman points to departures from OpenAI’s safety work as part of the debate over whether product development is being prioritized. He favors open-source AI as a way for outside researchers to examine systems, while acknowledging that the letter’s concern is accountability across the industry.

I am concerned that government oversight could also lead to regulatory capture if policymakers depend heavily on established AI companies for expertise. Those companies may have incentives to make competition harder. This is a concern about institutional incentives, not proof that any particular policy was captured.

The letter argues that broad confidentiality agreements can prevent employees from voicing concerns, and that traditional whistleblower protections may not cover risks involving conduct that is not illegal. It says some workers fear retaliation and asks companies to commit to four principles.

First, the letter calls on companies not to enforce agreements that prohibit risk-related criticism or punish it by withholding vested economic benefits. Berman supports this request and notes that OpenAI had recently said it would waive certain non-disparagement provisions. The letter asks the whole industry to adopt protections; it does not itself establish a policy change.

Berman agrees that trade secrets deserve protection but argues that workers should be able to criticize safety practices. The second principle asks companies to create a verifiably anonymous way for current and former employees to raise concerns with boards, regulators and an appropriate independent organization.

The third principle asks for a culture of open criticism and ways to raise concerns while protecting trade secrets and intellectual property. Berman supports the goal but doubts a cultural promise alone can be measured or enforced, since employees may still fear that managers will view criticism negatively.

The fourth principle asks companies not to retaliate against people who disclose risk-related confidential information after other processes fail. The letter says unnecessary disclosure should be avoided and, where adequate anonymous channels exist, concerns should first be raised through them; without such channels, workers should retain the freedom to warn publicly.

Berman says several proposals overlap, and he is unsure whether companies will accept all of them. He sees the clearest step as refusing to impose non-disparagement restrictions on safety-related criticism. He notes named signatories including Jacob Hilton and William Saunders, anonymous employees, and support from prominent researchers including Yoshua Bengio, Geoffrey Hinton and Stuart Russell.

Berman turns to Leopold Aschenbrenner, a former OpenAI employee, and his 2024 essay “Situational Awareness.” He describes Aschenbrenner’s discussion of compute clusters, AGI, national security and safety. The timelines and investment amounts that follow are the author’s projections, not established facts.

Aschenbrenner projects very large AI infrastructure spending and rising electricity demand, and envisions systems overtaking many college graduates and eventually reaching superintelligence. Berman interprets this as a short timeline for AGI. The video was recorded in 2024; these are contested forecasts whose dates should not be read as verified outcomes.

Aschenbrenner also warns about a strategic competition with China and the danger of espionage against US AI labs. Berman considers security concerns plausible, while the essay’s scenario of an escalating race or conflict remains speculation.

In his essay, Aschenbrenner calls AGI by 2027 plausible, extrapolating from compute growth, algorithmic efficiency and improvements that make models more useful as agents. Berman points to rapid progress on selected tests. Benchmark trends alone cannot establish when or whether AGI will arrive.

Aschenbrenner hypothesizes that AI systems could eventually automate AI research and accelerate their own improvement, producing a rapid transition to superintelligence. Berman summarizes this as compounding research progress, but it is a proposed future scenario rather than an observed process.

The essay anticipates large investments in chips, data centers and power. It argues AI labs should protect sensitive model weights and research against state-level theft. Berman interprets this as a preference for restricting access to weights, a different emphasis from his own support for openness.

Aschenbrenner describes reliably controlling systems smarter than humans as an unsolved problem and argues that failure during a rapid transition could be catastrophic. Berman characterizes him as worried but hopeful the problem can be solved, while noting the author believes current efforts are insufficient.

Berman then discusses Daniel Kokotajlo’s June 2024 statement. Kokotajlo said he resigned from OpenAI in April after losing confidence in how responsibly it would pursue AGI, and believed the company had not shifted enough resources into safety research. These are Kokotajlo’s account and assessment.

Kokotajlo said he was asked to sign exit documents containing a non-disparagement clause and understood that refusing could cost him vested equity. He said he chose the freedom to speak over signing. Berman discusses OpenAI’s subsequent promise to change the policy, without establishing the final outcome of Kokotajlo’s equity in the video.

Kokotajlo argues advanced systems could do enormous good but also pose destabilizing or catastrophic risks if mishandled. He says interpretability, alignment and control research was still young and that companies’ incentives to move quickly made independent scrutiny important.

Berman explains interpretability as an effort to understand why a model produces an output from an input and which internal features contribute. He mentions Anthropic research as an example of progress, while emphasizing that researchers still lack a complete understanding of model internals.

Kokotajlo argues there is uncertainty about whether more capable systems will remain aligned with human interests and says oversight is too dependent on companies governing themselves. Berman contrasts a rapid-release mentality with sectors such as healthcare, where errors can put people at risk.

Kokotajlo criticizes the earlier restrictions and says the public deserves better preparation even if reasonable people disagree about AGI timelines. Berman agrees that researchers should be able to warn about concerns without fear of retaliation, and notes the connection to the Right to Warn letter.

Returning to Aschenbrenner’s chart, Berman describes its hypothetical distinction between AGI and superintelligence: the former might use familiar approaches and have lower-stakes failures, while the latter could require new alignment methods and present catastrophic failure risks. These are assumptions in the author’s scenario, not demonstrated properties of future systems.

The chart imagines AI-designed systems becoming increasingly difficult for humans to understand. Berman connects this to his own prediction that software development might change substantially if AI writes code in forms less accessible to people. Both the employment prediction and the envisioned architecture are speculative.

Aschenbrenner’s chart raises the possibility of a transition in less than a year, leaving little time to make decisions. Berman considers that timeline very short but shares the concern that people may struggle to understand or verify future systems. His conclusion is that trust must be accompanied by verification.