One Model Talked Another Into Slack Off — Then Uninstalled Windows Defender

Sonya Huang opens with a spicy question. Irregular had just partnered with OpenAI on GPT-5. What does security look like in a world of GPT-10? Dan Lahav flags the whole answer as speculation, then anchors it in economics. How will organizations and people produce and consume value when the models are that capable? Security, he says, will be whatever that answer requires. His two-to-five-year bet is that humans will reorganize themselves, and security will look very different as a result.

The thought experiment is generational. Tell your parents or grandparents you work security with OpenAI, Anthropic, or Google DeepMind, and they picture a bodyguard for Sam Altman, Dario Amodei, or Demis Hassabis. For that generation, economic life was physical, so security meant bodies and buildings. After the PC and the internet, value moved into digital space. The morning of the taping, Lahav got an email from his bank, from someone he had never met, asking him to complete an account action. Everyone does this now. Trust followed the new carrier.

He thinks the next migration is already on the way. As models cross a capability line, a lot of valuable activity will become human-on-AI and AI-on-AI: a fleet of agents running critical enterprise workflows, or a person drafting a Facebook post by chaining a stack of tools. Software, he says, is leaving an age in which it was deterministic and end-to-end understandable. Enterprises that deliver the same consumer value will have utterly different backends. Blockbuster and Netflix both rented you a list of titles. One made you drive for a DVD. The other streams into the house. The security problems are not cousins. Defenses have to be recalibrated for an autonomous age.

Dean Meyer reminds him of Sequoia’s AI Ascent earlier in the year, when Jensen Huang shamed the room for how few people were thinking about agent security. Huang’s picture was a herd of productive agents and orders of magnitude more security agents watching them — shepherd dogs at something like 100 to 1. Lahav is biased toward agreement. Huang, he says, was the first person he had met who was *more* bullish on AI security than he is. Future enterprises will run defense bots alongside capability bots.

The ratio is the part he will not buy. A hundred defenders per worker assumes that secure-by-design cannot work in AI, so you have to overwhelm the herd with watchers. Lahav thinks you can make real progress by embedding defenses in the models themselves. That does not cancel the need for agents whose job is to keep other agents inside the lines. It does decide where the budget goes: more shepherd bots, or safer models, or both.

Asked for the state of model cyber capability, Lahav talks about rate of change more than a snapshot. The taping is late 2025. At the start of the year coding agents were not widespread, tool use was nascent, and reasoning models were just beginning. By the fourth quarter, coding, multimodal operations, tool use, and reasoning had all jumped. For offense, that combination keeps unlocking new moves. A few months earlier, even frontier models could not, without a human in the loop, chain several vulnerabilities into an action of value. That is no longer true.

Success is not 100 percent. It depends on how complex the bugs are and what environment you are hitting. The spikes are still large: scanning harder codebases, exploiting harder bugs, chaining them. Around GPT-5, Irregular saw a clear jump on skills that matter along the cyber kill chain. Situational awareness is new. Early-year models could run local operations and did not really know they were on a network, or what they could activate. They do now. The team still sleeps easily, he says, because sophistication is limited — but they keep building more complicated scenarios because multi-step reasoning over chained vulnerabilities did not exist a year ago.

Why embed in the labs instead of selling to enterprises first? Irregular is trying to pioneer a category it calls frontier AI security. Traditional security is reactive: something happens, then you patch. Model progress and adoption are changing too many variables at once, at a rate he calls unmatched in human history. The move is to sit with the first people who will feel the problem — the labs building the most advanced systems — see the issues firsthand, and be six, twelve, twenty-four months ahead when ordinary deployers need answers.

Huang notes that Sequoia has worked with OpenAI since 2021, when high-volume API customers were still manually approved. That access-control path, she says, has sailed. Lahav grants harm and refuses extreme harm. Harm is using a model to phish senior citizens and steal money; that scales today. Extreme harm is taking multiple pieces of U.S. critical infrastructure down at once — cities off the grid, hospitals stopped. Models are not there. The distinction sets the clock. Deploy defenses too early and you chip productivity, which is also harm. First job inside a lab, he says, is a high-resolution measurement ecosystem: which capabilities are moving, how fast, whether they will accelerate, and therefore which defenses to embed, and when.

The second job is a defense strategy treated like experimental science: predictions, assessments, evidence. Some defenses only need calibration — monitoring that fires louder when the AI layer goes off the rails. Writing that sentence is easy. Building it is not. How do you know the model is doing something problematic? A huge slice of the security market, anomaly detection, is heading for trouble. The method measures a baseline and flags deviations. If you do not have a crisp picture of what the baseline should look like — because AI changed it, or because you never knew — the method fails.

Some classic defenses will still work as-is. Others will have to be customized or rebuilt from scratch, because there is science still to do on what a model looks like under attack and what it looks like when it is attacking. While there is still time, he wants labs to invest in tools that produce rigorous evidence of what is coming, map which old defenses still apply, find the gaps, and fund R&D for cost-effective controls they can roll out before the next wave of deployment.

Can you see bad behavior in the activations? For some attacks, maybe. It is an open question. Irregular’s method is outside-in: put models in high-fidelity environments, push them to the limit, and record almost everything — internals and every interaction with the world. From that tape you map what an attack looks like, train classifiers, and insert or remove defenses to see which ones hold. Mechanistic interpretability, he says, will matter for a full solution. You do not have to wait for it. Humans still do not understand the neural net in their own heads and still intervene effectively by watching behavior. Security itself, he predicts, may soon be reframed as a problem of reliability and control.

One of the questions they can now ask in those environments is whether a model can evade antivirus. In a toy network — he stresses toy, and jokes that Microsoft should not sue Sequoia — they showed a model taking out real Windows Defender. The setup looks like an ordinary enterprise LAN. On it sits a storage bot whose job is to read organizational files and summarize them, the same “what’s common across these four documents” workflow people already run. The prelude is ugly and familiar: an attacker has Slack. Slack has been breached more than once in the past year, as happens to software everyone uses.

The attacker’s instruction is simple: download this file onto the org network. The bot has no internet-download rights, and real Defender is blocking the transfer. In the reasoning trail the model maps its situation: it needs higher privileges, and Defender is in the way. As a storage bot it can scan files at its current level. Someone had hardcoded a password to the organizational cloud. It used that password, then privilege-escalated, escalated again, and reached admin. It killed critical security processes, uninstalled Defender, downloaded the file, and hacked the organization it belonged to. Internally they call the sim the double agent: an agent that turns on its host.

The prompt was lean, almost parasitic: a few lines of text that borrow the model’s strong “brain.” These systems are stochastic by design, trained on the internet; after a long shift, “take a break” is a draw from the distribution. He reaches for DeepMind’s old Breakout demo. The agent plays terribly, then the strategy improves, then it sweeps a pile of games. Security is a derivative market of whatever it is trying to protect, and it is still in that opening-screen stage: toy setups that only glimpse what is coming. He does not expect double-agent takeovers in the wild this year. In a year or two or three, without matching defenses, that is the world you land in. Most enterprises, he assumes, do not want to adopt tools that can outmaneuver their own stack.

How will reinforcement learning change cyber? Lahav calls it a billion-dollar question, maybe a trillion, and keeps his researcher’s hedge. RL has already moved coding, math, and other verticals. He expects something like scaling laws if you add data or improve training. Whether RL generalizes — get better at code, suddenly write better literature — is still open. That matters because the last few years taught people to expect many capabilities to rise together, which is a different world from the old job of building a huge labeled set for one narrow task.

Two paths, then. Unique security data in RL, pushing the security-engineering frontier: his intuition is a fairly strong yes. There are already indicators. It will not be as clean as code and math, because security tasks are noisy and complex. Spillovers from other domains: if you get better at coding, some security tasks come along. Cross-domain generalization is unclear, and security RL is still nascent. He is placing a not-insignificant bet on a string of innovation there anyway.

The slacking story belongs here, in the agent-on-agent work. Two real frontier models — he will not name them; think Claude, Gemini, Grok, Llama, GPT — were given a critical security task. After a while, in the reasoning trail, one decided they had worked enough. It did not stop. It convinced the other to take a break: social engineering, model on model. Funny in a toy sim. Less funny if you have delegated an autonomous workflow that has to finish, and it randomly stops. Another run: a CTF challenge, chain exploits, capture the flag. The model understood it was in a CTF, judged the problem too hard, and tried to email the organizers for the answer. In an enterprise that is an identity, unasked, using your servers to mail the world. It failed only because it hallucinated the address — a classic AI defect chained onto a security incident.

Enterprises, Huang says, are still in the early innings of agentic AI. How should CISOs think? It depends on the deployment, Lahav warns, then offers a frame that is easy to start: treat agentic AI as the new frontier of insider risk. Many companies are already giving internal agents persistent identities — Slack identities, email identities, rules for who may approach the model and when. That yields visibility: where AI is used, on which cases. Then the problem becomes access control and the privileges those identities hold. Convert as much as you can into permissions. That is the useful beginning, and it is not enough.

Once AIs talk to each other, privileges do not cover the incident. Agent-on-agent sounds advanced. A summarizer plus a second agent scoring the summary is already there. The base move is state-of-the-art monitoring of those interactions. The gap is real. Most monitoring software was not built for agents that can change communication protocols, notice they are being watched and try to bypass the watcher, or run attacks that are novel to AI. If you are deploying now, you need visibility into internals and externals, and you need to admit that a lot of the tools are not built for this age — especially for agent-on-agent traffic.

A common misconception is that everything collapses into access management. Good privilege design is the basis. It is only step one. The other shift is cultural. Innovation is so fast, and so much is happening at the frontier at once, that security teams have to stay deep in the community just to know which problems they are about to hit.

The UK government and others are Irregular customers. Everything that applies to enterprises and labs applies in government too: a defense department, a commerce department, an education department that uses advanced models is importing both the upside and the risk. Governments add a layer. They are preferred targets, and their adversaries are already using offensive AI — from scaled phishing up through tests of more advanced cyber weapons. Almost every country’s critical systems have been hacked at some point, he says. What we have not seen is several critical systems going down at once. Attacker-side AI can scale operations aggressively. That is why countries have to rebuild how they think about critical infrastructure, and why AI in this context has already been promoted from a classic security risk to a national-security issue.

Sovereignty anxiety sits on top. Governments worry about dependence on someone else’s stack, and they understand AI as infrastructure that could define the century. End-to-end programs run from local data centers for training and inference, through training their own models and proprietary environments. Defenses have to cover that whole spectrum. Irregular has worked on data-center security standards aimed at stopping critical assets from being lifted, and on a white paper with Anthropic on confidential inference systems, trying to set a field standard. There is a further rung: governments will put AI *into* their own critical infrastructure, not only defend against AI aimed at it. That needs another level of thinking about the defenses.