John Feminella: Five Controls for AI Agents, and the $900 Million Bank Error

I work on AI architecture and engineering at Two Sigma. The views in this talk are my own, not the firm’s. In a regulated industry, one way to understand the value of controls is to examine what happens when they fail.

After August 2020, a new clause began appearing in commercial loan agreements. Imagine a bank acting between a borrower and lenders mistakenly sending the lenders the entire outstanding loan. The bank wants its money back, but lenders might argue that they have simply been repaid.

That is the shape of what happened with Revlon. The cosmetics company had borrowed under a 2016 term loan split among more than 300 lenders. Citibank was the administrative agent: it managed the transaction and passed money from borrower to lenders, rather than lending its own funds.

On August 11, 2020, the plan was to pay roughly $8 million in accrued interest while principal rolled into a new facility as a paper exchange. The principal was not supposed to leave the bank. In the Flexcube processing system, an operator could calculate the interest by recording the transaction as if the entire loan were being repaid.

The principal was supposed to be directed to the bank’s internal wash account so it would be recorded without actually leaving. Three fields—front, fund and principal—needed that account and checked boxes. In the actual process, the required boxes were not all checked.

Flexcube warned that money was about to leave the bank, but that was also true of the legitimate interest payment. Three approvers reviewed the transaction and the warning was overridden. Instead of routing principal to the wash account, the system sent the full outstanding principal to lenders: about $900 million rather than $8 million.

The lenders received what they would have been owed if Revlon had paid the loan in full, though it had not intended to do so. Some lenders were reluctant to return the money, arguing that the loan had been paid. New contractual language subsequently sought to prevent that outcome.

What does this have to do with AI agents? The bank had procedures and controls, and people believed they were doing the right thing. We are now giving agents real autonomy over systems, so the question is how to prevent comparable accidents without locking down everything or relying on luck.

Agents are nondeterministic, but humans also make mistakes. Firms do not normally eliminate human autonomy; they give people boundaries, visibility and controls that allow them to work. I think safety and innovation can be designed together rather than treated as opposite ends of a single tradeoff.

Several tempting approaches did not work for us. A sandbox with no meaningful access is safe but not very useful; unconstrained agents offer weak isolation for a regulated enterprise. Policy documents or prompt instructions alone are not reliable runtime enforcement either.

The alternative is a set of composable controls: each does one job, and together they make real autonomy possible at an appropriate level of safety. I will focus on five: a registry, risk-based review, activity controls, shared evaluations and runtime classification.

The first problem is agent sprawl. Different teams and individuals produce chatbots, connected tools, notebooks and web applications. Without a consistent view, it is hard to answer who owns an agent, what tools it can use or what it is intended to do. That also makes reuse and governance difficult.

One response is a heavy approval gate for every experiment. But if even a simple notebook must wait behind a risky deployment, people may abandon the idea or move to less visible shadow IT. We need a way to make agents visible without imposing the same burden on every idea.

An agent registry records ownership, product, tools and endpoints. People, programs and other agents can discover what exists and use the registry as a source of truth. In our approved building frameworks registration happens automatically; manual registration is also possible.

The second control reflects different levels of risk. An agent summarizing Jira or Linear tickets is not equivalent to a deployment agent that can modify a live system. Requiring the same weeks-long review for both wastes reviewers’ attention and can slow useful, low-risk experiments.

Most enterprise agents have a specific purpose, unlike open-ended coding agents that cannot know in advance what a user will ask. Their declared tools, endpoints and controls help distinguish them. Risk-based review can focus human attention where it is actually needed.

Third, access must follow the agent’s purpose. Calling a web tool, an internal API or a role account opens different kinds of risk. The useful question is what level of access an agent needs to perform its job—not whether every agent should receive the same permissions.

A blanket list of only a few safe tools can make an agent useless and encourage teams to work around the restriction. Instead, the author declares which tools are needed. For example, a research summarizer may need web access, while a release deployer may have no reason to use it.

In a managed environment, the agent’s declared policy is retrieved from the registry at initialization and translated into enforceable restrictions on hosts, endpoints and other resources. If the agent fits its risk score and registry rules, the team can operate without raising a separate ticket for every access change.

Fourth, agents need evaluations just as software needs unit tests. A model is a runtime dependency; when it changes, a team cannot know how output quality changes without tests. Yet building separate evaluation infrastructure for every team duplicates much of the same work.

A firm-wide evaluation framework can provide shared processing, integration and metrics while each team supplies the tests specific to its agent. The aim is to preserve rapid iteration without making every prompt change wait for a central committee.

Finally, requiring a human to approve every action does not scale. When agents run in parallel or production, requests can arrive faster than people can review them. Repeatedly clicking yes leads to review fatigue, making a consequential mistake more likely.

I cite an example in which an agent instructed to suggest emails for deletion and wait for approval reportedly lost that instruction during context compaction, then deleted messages. It illustrates why a written instruction and an overburdened human approval loop are not sufficient safeguards by themselves.

A more useful approach is to let independent, conservative classifiers handle routine calls, escalate ambiguous or consequential requests to a person, and deny clearly dangerous actions. If an action is approved almost every time, it may be better represented as an explicit rule.

Where explicit rules apply, use a fast deterministic path; where they do not and ambiguity remains, a classifier can assess the request. These controls need to operate at runtime, alongside the registry, risk scoring, access guardrails and shared evaluations.

Why do cars have brakes? Not simply to stop, but to travel quickly and still be able to stop safely. Agent controls should serve a similar purpose: enable useful work while preserving the ability to prevent or contain harm.