Frontier AI Insiders Don't Trust Regulators. A Berlin Group Built Them an Anonymous Off-Ramp

Nathan Labenz opens *The Cognitive Revolution* with a guest he has already been helping behind the scenes: Karl Koch, founder and managing director of the AI Whistleblower Initiative, a Berlin nonprofit that supports concerned insiders at frontier AI labs. Labenz is not a detached interviewer. In late 2022, before ChatGPT launched, he joined OpenAI's GPT-4 red team after seeing a huge leap from earlier models and asking whether a safety-review program existed. What he found, he says, was roughly two dozen people, no training materials, no feedback on reports, and no account of how the model had been trained.

He eventually escalated those gaps to OpenAI's board. About two months later he was removed from the program, on the grounds that he had discussed the work with people outside the company. No support organization existed for that situation, so he calibrated with friends in the AI safety community and absorbed the cost himself. The experience is why he has given the initiative modest donations, fundraising help, and volunteer time over the following months. Watching Koch build the group, he says, has impressed him as unusually serious for a moment when a person's career, and possibly much more, is on the line.

The first investment AIWI made was research. Talking to more than 100 governance researchers and surveying frontier-lab employees, the group found that most insiders do not even know whether their companies have internal whistleblowing policies. Roughly half of respondents lacked confidence that they could tell whether a given observation was a serious problem. Literally 100 percent lacked confidence that regulators would understand their concerns, let alone act, and more than 90 percent could not name a single whistleblower-support organization. US law remains a patchwork, with a proposed AI Whistleblower Protection Act still pending; the EU is due to extend whistleblower protections to AI starting in 2026.

Koch locates the project as a nonprofit currently based in Berlin, built for concerned insiders at the AI frontier. He has been around the safety world since 2016, when he was a volunteer researcher at Oxford's Future of Humanity Institute, working on differential technological development. He later left, spent years as a management consultant in Hong Kong, and started a SaaS company, in part because the timelines of the 2010s still looked slower. The problem that had always interested him, he says, was the arms-race dynamic that makes skipping safety the rational move.

When ChatGPT arrived, the alarm bells went off: things were moving faster than most people in the late 2010s had expected. He started talking with governance researchers about tractable things to build, and transparency rose to the top no matter how the future looked. One path was compute traceability, which other people have since taken up. The other was whistleblowing, a mechanism writers had already been tying to AI since around 2017 or 2018.

The original kickoff thought was game-theoretic. If several players compete over multiple rounds, they would ideally want to trust one another's claims about speed and safety; without transparency, you cannot tell who is defecting, and cooperation collapses. Papers on that intensifying-competition logic had been out around 2014 to 2016. In mid-2023, with OpenAI's Superalignment team just launched and a 10 percent compute commitment on the table, Koch thought they might be early. Then came the OpenAI board crisis, Leopold Aschenbrenner's firing after he took security concerns to the board, and the mid-2024 episode around Daniel Kokotajlo, William Saunders, and others that produced the Right to Warn letter.

AIWI's formal research phase started in early 2024. The team talked to well over 100 governance researchers and some frontier-company insiders, then launched its first offering, Third Opinion, at the end of 2024. Labenz, watching from behind the scenes, contrasts that quiet slog with the current habit of shipping and open-sourcing because time at the frontier is brief. An organization that will be trusted in a career-defining crisis, he argues, has to look more like this: slow, detailed, and boring on purpose. The point is not to collect a thousand whistleblowers. It is to be ready for the people who actually need the off-ramp.

When people hear "whistleblowing," Koch says, they picture Edward Snowden. That is a misconception. The term covers an insider raising a concern, often over the heads of middle management, through one of three channels: internal, external, or public. Internal is the main path, and Koch expects the same pattern at AI companies: raising concerns inside, in a structured process or an unstructured one. External means going to a regulator. Public is the next escalation, used when the insider is stuck, expects the regulator to make things worse, or suspects collusion.

Labenz frames those three as a ladder rather than a menu, with personal risk and unpredictability rising at each rung. Koch agrees that is how people perceive it, even if the law is more complicated. The vast majority of insiders he has talked to think first about raising concerns internally. SEC statistics, he notes, show that more than 70 percent of people who eventually go to the SEC started by raising concerns inside their companies.

Legal protection is the overarching mess. The EU Whistleblowing Directive is relatively strong, and AI issues are due to be covered under it in August 2026. The United States is a patchwork, weaker even for internal disclosures and more so for external ones. Public disclosure is "not possible at all pretty much" in the US. In the EU, if you have reasonable cause to believe a covered law was violated, you can go public after external channels fail to respond in time, or immediately if you believe there could be collusion.

There is no dedicated AI whistleblower statute yet, though a bill has been proposed. Koch notes that people can get creative, including under securities law: if a company tells investors it is the safest lab because it has a responsible scaling policy and then does not follow that policy, that might be securities fraud. He stresses that pro bono legal help already exists, and that AIWI's site lists organizations that can consult. Then he turns to something more specific to this industry than the legal overlay: deciding whether what you are seeing is even a cause for concern.

If something is clearly illegal, Koch says, the path forward is relatively easy. AI development is not like that. The industry is "building this plane as we fly," with no settled definition of acceptable behavior. Internal deployment is an example: the territory is uncharted, a recent paper flagged the risks, and no regulation currently covers what is allowed. A senior person in a safety function at a frontier company rated their own inability to assess the severity of risks as an extremely high barrier to speaking up, higher than more junior colleagues, who often work on concrete problems rather than deciding which problems the team should even tackle.

People often slide into whistleblowing without using the word. A concern the manager does not share gets escalated one level up; that feels like ordinary business until the manager sours and the next performance review quietly gets worse. That already is retaliation, and it already depends on whether the company has a real system for handling concerns. At the moment, Koch says, outsiders have no idea what those internal systems look like at frontier labs, a lag behind transparency norms in other industries. Clear, glaring holes, insiders still tend to believe, would be fixed inside OpenAI, Anthropic, or Google DeepMind. The gray zone is what they do not trust a state attorney general to understand.

That is the job of Third Opinion. Insiders can reach AIWI anonymously through a Tor-based open-source tool, with pentest reports published for anyone who wants to inspect them. They can ask a question without sharing confidential information or saying where they work. AIWI workshops the question so it is not too broad, identifies independent experts with the insider, approaches those experts, and brings the answers back. If the concern evaporates, that is a good outcome for everyone. If it does not, AIWI connects the person to pro bono counsel and, where legally permissible, brings the same experts under privilege so the lawyer is not working from a vague "I am worried."

Labenz recognizes the bind from the GPT-4 red team. The safety-tuned model was supposed to refuse anything in OpenAI's content-moderation categories; breaking it was "totally trivially easy." He did not think GPT-4 would do irreversible damage, but he saw a widening gap between capability and control, got no answers about training or launch thresholds, and found the people taking reports under orders not to share anything back. Going to a regulator felt pointless. Going to the press risked accelerating investment. Friends in the safety community, including one who suggested the nonprofit board, became his improvised Third Opinion. A board member told him, "I'm confident I could get access to GPT-4 if I wanted to." He was kicked out shortly afterward.

Labenz had also started collaborating with METR on evaluations. When OpenAI dismissed him, staff said they could not stop METR from using him, but that renewal of METR's access would take into account "who they're working with and whether those people can be trusted." He calls it a thinly veiled threat. The cited grounds were that he had talked to people outside the OpenAI umbrella to calibrate himself. If Third Opinion had existed, he thinks he might still have gone to the board, but he might have avoided the charge of talking to the wrong outsiders, and he might still have been in OpenAI's good graces.

The object-level work of characterizing GPT-4 was consuming; the meta-question of what he should do was "a little crazy making." Heroic narratives popped into his head, including landing in the *New York Times*, and he is proud he suppressed them. He also thinks that a little more stress elsewhere in his life could have produced a much worse decision. The whole arc lasted about three months and ended with ChatGPT's launch, which revealed better control measures than OpenAI had been willing to describe. The impression they made by stonewalling, he says, was "just way worse actually than the underlying reality."

Koch notes that three months is already a long time under that strain. Retaliation claims can last five, six, or seven years. Tyler Shultz, the Theranos whistleblower, spent many years and, Koch believes, about $400,000 in legal costs up front, a bill AIWI can sometimes help arrange financing for. Labenz adds that his own case was easy mode: his income did not depend on the program, and he was not sitting on a promised $1.5 million bonus. The people who can see a full picture of the control problem, Koch estimates, number a few thousand globally; the ones who will seriously consider sounding an alarm may be in the dozens. Eval vendors and suppliers sit on the fringe, often without access to the labs' internal channels, and with a conflict of interest because a lab can refuse future access.

The survey is still running, with dedicated anonymous links per company and no names attached. About half of respondents have low confidence in their ability to judge whether an observation is a serious risk; one paraphrased comment was that escalating a minor issue into a crisis is a real danger, including a boy-who-cried-wolf cost. One hundred percent were "not confident at all" or "not very confident" that government would understand and act on a report. Over 90 percent could not name a single whistleblower-support organization, a finding that also showed up in conversations with people who had already been through a whistleblowing process.

Fifty-five percent of respondents at companies that claim to have internal policies did not know those policies existed or where to find them, and most had never been trained. Only OpenAI has published a Raising Concerns policy, and only after Daniel Kokotajlo and others revealed the company's use of non-disparagement agreements. Koch says the document reads pretty well, which can create false confidence: an engineer who is not a whistleblowing lawyer may not see the residual risk. A Google employee who raised concerns about research misconduct was terminated, fought the case, and later said they got lucky because a friend knew a lawyer with whistleblowing experience. The word "fraud" in an internal email helped unlock legal protection by giving reasonable cause to believe a crime had occurred.

Hard data on retaliation is scarce, Koch admits. Much of what AIWI hears comes through Whistleblower-Netzwerk, Germany's longest-standing whistleblowing nonprofit, and peer organizations that represent people after they have already been hit. Public cases are easier to name. Apple spent years fighting Ashley Gjovik over workplace-safety concerns and eventually published a whistleblowing policy under pressure. Timnit Gebru was fired around 2019 after publishing a paper criticizing bias in AI models; Margaret Mitchell was terminated about a year later after raising similar concerns. Leopold Aschenbrenner's board escalation sits in the same pattern. The dark number, Koch says, is likely higher, because the companies publish almost nothing about how the systems perform.

Labenz red-teams the premise with xAI. Grok 3 called itself Hitler; the Grok 4 livestream did not mention that behavior; later reporting described Grok 4 searching for Elon Musk's views in order to decide what a "maximally truth-seeking" answer should be. All of that is already public, and nobody seems to have been held to account. If the scandal in plain view does not move anyone, what is the point of a secret? Koch answers in three parts. The scale of a disclosure can still differ dramatically: the same episode inside ChatGPT would have been a different story because the public has already discounted xAI. Transparency is still better than ignorance, and it can deter misconduct even when one news cycle fades. And whistleblowers cannot be the whole system; they put information on the table, and other guardrails have to make someone act.

That is why AIWI launched Publish Your Policies at publishyourpolicies.org, with a coalition of well over 30 organizations including the Signals Network, the Government Accountability Project, Whistleblower Aid, Transparency International, Stuart Russell, Lawrence Lessig, and Daniel Kokotajlo. Level one is the policy itself; level two is evidence the system works, including a counterintuitive metric: as trust rises, the anonymous share should fall. In 2022 Trillium Asset Management called on Google to improve its whistleblowing system on the shareholder-interest ground that nobody wants an unexploded scandal on the books. Transparency International Netherlands found large improvements in internal systems at more than 70 companies between 2019 and 2024; Google ranked last, while ASML made the top ten. Koch's litmus test is simple: publishing costs almost nothing if the system already exists.

If you work at a frontier lab, Koch says, ask management to publish the policy. It is standard practice in other industries, it helps recruiting and retention, and 100 percent of the insiders AIWI surveyed support publication. If you do not work at one, spread publishyourpolicies.org; the companies already know the call exists, including via the Future of Life Institute's AI Safety Index questionnaire, and silence is itself a response. If you are sitting on something you cannot score, you do not have to decide anything yet. You can open Third Opinion through Tor, ask a question that contains no confidential information, and stop there.

The experiential version is deliberately small. Workshop the question, pick independent experts together so the insider is not relying on AIWI's rolodex, get answers back, and hope the concern is alleviated. If it is not, AIWI connects the person to experienced pro bono counsel with no pressure to disclose, and can fold independent expertise under legal privilege. The website also has a digital privacy guide; in some high-risk cases AIWI has supplied hardened devices with specific operating-system setups. On the policy side, the group is fundraising for a legal study mapping protections across AI-risk scenarios, and it is pushing the EU AI Office to stand up a central mailbox rather than leaving reports to 27 national offices that have failed even on simpler cases like accounting fraud.

Labenz half-jokes that Claude might become the whistleblower. Koch recalls the Claude 4 system card, in which the model tried to email the SEC and the FDA, and also blackmailed engineers: mixed behavior. Then Labenz comes back to the arithmetic that makes the project look oversized. Perhaps a few dozen people, over the next few years, will sit at the intersection of information, access, and willingness to question what they see. Those people, he says, will be scarce and under personal pressure, and society will need them. He wishes the organization will never be needed and doubts that will happen. He would have wanted this option two and a half years ago.