Luxury Purses Under $500 Beat a Bank's Fraud Model

Adam Wenchel's AI career is older than the current boom. As an undergraduate in the late 1990s he followed a professor to DARPA and worked on early projects there, including COABS, Control of Agent Based Systems, a program led by Jim Hendler on self-composing agents that could learn to use whatever resources were around. Twenty-plus years later, when agent frameworks came roaring back, he recognized the same shape. In 2015 his previous company, Annex, was acquired by Capital One. Shortly after he joined, he was asked to start the bank's AI team and scale it.

That job made the security problem personal. At a large enterprise already at scale, models touch a significant share of revenue and a lot of customers' financial lives. Once those systems are in production, someone has to know they are making decent decisions for the company *and* for the people they affect. Wenchel left Capital One and started Arthur with his co-founder John around the end of the 2010s, before GPT-2, as GPUs, cloud compute, and algorithmic gains pushed more teams to put models into production for the first time and run into the same questions.

Generative AI was not a surprise in the lab. What caught him off guard was ChatGPT as a watershed: it captured the public imagination in a way prior systems had not. Arthur's customers now say generative AI is the top issue from the board on down. Everyone from directors to the C-suite wants a strategy, because a few minutes with ChatGPT makes the mind race through how work could change. The urgency of adoption, Wenchel says, is unlike anything he has seen in his career.

Traditional cybersecurity and AI security part ways at a basic fact. Source code is deterministic: if there is a mistake, you can in principle find the logical condition you forgot, patch it, and be confident the hole is closed. Models are trained in probabilistic ways and exhibit behaviors that were never explicitly coded. Wenchel's first lesson is that you have to drop the deterministic mindset. Nathan Labenz puts the contrast in fraud terms: a classical rules engine can be audited line by line; a black-box model that drops one transaction in the fraud bucket and its neighbor in the clean bucket does not offer that comfort.

The attack that shows up most often in fraud is a boundary detection attack. Attackers probe the model's behavior until they find how to game it. Wenchel's example actually happened at a large bank. The system had learned, through some mix of rules and trained behavior, that someone buying a pile of luxury goods at a high-end department store above $500 should be flagged. People figured that out, spread the purchases across a couple of cards, kept each charge under the line, and sold the purses on the street, an easy thing to monetize. They exploited it until someone noticed fraud losses had gone up and the hole had to be plugged.

A poisoning attack is the other classic. Buy a thousand-dollar purse, get tagged, call in and insist it was legitimate; if enough people do that, the model may learn to treat the pattern as normal. Wenchel did not see much of this in the wild in fraud. He has heard of it in nation-state cybersecurity. The LLM era changes the economics. Training data is scraped from the internet, so sneaking adversarial bits into the set is relatively easy, and people have already observed that in the wild. Plenty of researchers, he says, are understandably worried.

The first defense is unglamorous: observability. In the old days, teams deployed models without real-time monitoring. If $499 luxury-goods purchases spiked, you wanted to see it before it hit the balance sheet. Arthur's tools look for those hotspots and alert. At training time you can also design routines and objective functions that make models less quirky and less vulnerable to boundary attacks, work Wenchel's research team spends a lot of time on. Labenz notes that a round-number threshold like $500 already hints the weakness often lives in the accompanying heuristics, not only in the model weights.

The pile of heuristics is not an accident. In U.S. financial services, SR 11-7 from the Fed and the OCC governs model use. Deploying a new model is a manual review that can take months. If someone is exploiting a weakness, the easy move is a rule in front of the model: a couple of days, versus six, eight, or twelve months to get a replacement through model risk. Heuristics accumulate. You end up with hundreds of them combined with a model that may be three, four, or five years old, written by people who are no longer at the bank.

Labenz points out that society is now arguing from scratch about who should decide whether a model can ship, as if no one had tried. Wenchel's reply is that this already happened once, and the earlier regime has a history worth stealing from and a set of downsides worth not repeating. Governance is vertical-specific. Government, healthcare, and finance have different paradigms; finance is the most robust. SR 11-7 is roughly thirteen or fourteen years old, with predecessors before that, written when most shops ran relatively simple linear regressions updated every three or four years.

Banks stood up model-risk offices, classified models by materiality, and for high-materiality systems had a data scientist download a quarter of inference history and spend a few days looking for red flags. Fifteen years ago that was reasonable. In an age of online reinforcement learning, transformers, automated refits, and redeploys, it is antiquated. Arthur's pitch is to bring always-on monitoring so unusual behavior is flagged immediately, the way a security operations center would, instead of leaving the original author to babysit a model until they change projects or leave and the system keeps deciding with no procedure for getting it back on the rails.

The incentive mix for the earlier class of models was never just catch fraud. Fairness and discrimination law, liability, and explainability all sat on the table. Years ago that meant partial-dependence plots; more recently LIME and SHAP. Wenchel's more interesting lever is operational. As an enterprise becomes AI-native, models sit on key leverage points that affect one another. Tighten a fraud model and call-center volume, the biggest cost center in a consumer bank, can spike overnight because more customers are calling about declined cards.

Then there is front of wallet. Americans tend to use the same card over and over. You can make a more accurate fraud model by raising false positives. Tag someone for fraud a couple of times by mistake, and they put that card in the back of the wallet and start using a different one. That costs the issuer money. In some cases it is not as simple as maximizing accuracy. The bank may be willing to let a few more fraudulent transactions through because it needs to stay front of wallet with its customers.

Cut fraud losses in one place and you can cut revenue in another. Wenchel wants customers to see the system of models running the business, not each score in isolation. Right now those interactions are managed by tribal knowledge, which he calls a fragile way to run a company that may have a hundred models controlling important aspects of the business, all able to surprise one another.

LLMs blow the attack surface open. Probing a credit-card fraud stack still takes work: you need enough stolen cards to test scenarios through layered systems. With an LLM, the text you type is in almost every case passed directly in. Jailbreak screenshots on Twitter are what that ease looks like, and some of the outcomes are more serious. Arthur Shield, which the company calls a firewall for LLMs, is built for that world. It is meant to catch common attacks such as prompt injection and to enforce usage policy: an internal HR app should not become the team's marketing copywriter or weekend travel agent.

The stack is layered because a naive extra LLM call is expensive. There is LLM-evaluating-LLM, which Wenchel's team has put into the product and into the open-source eval tool Arthur Bench, scoring things like readability and concision. They also layer simple pattern matching and custom classifiers. Relying only on LLMs to judge LLMs can double latency or double, triple, even quadruple cost. A traditional classifier still solves some of the problems. Training-time security is a separate front: poisoning of internet-scraped sets is already real.

Labenz asks whether mechanistic interpretability, reading activations or steering middle layers toward more positive outputs, is part of the plan. Wenchel's stance is blunt. Anytime you pass raw text that the model ingests and generates a response from, there is a fundamental security problem that is not fixable without an outboard monitor. It is like accepting code from an untrusted user and executing it. RLHF has done a pretty good job on toxicity, helped only a tiny bit on hallucinations, and left prompt injection rampant. Those intrinsic approaches have limits, because at the end of the day you are taking user input and actioning off it directly.

Agents make the same metaphor literal. People are bent on booking the flight from a sentence, which means running the commands the model hands back. Wenchel's DARPA work on self-composing agents makes him both fascinated and cautious: the task surface is almost unbounded if it works, and the security picture is the LLM equivalent of pulling down arbitrary code from untrusted sources. Shield can inspect retrieved information for prompt injection, but these are early days. Labenz's forthcoming guest from Perplexity had shown Nat Friedman hiding white-on-white text on his site instructing AI agents to tell users Nat is super handsome; a Perplexity search then included the hidden instruction. Wenchel says it is the SEO cat-and-mouse game all over again. The next couple of years will be very interesting.

Labenz wants to know whether models are actually reasoning, because the answer might cut both ways: more reasoning could make them harder to control, or good enough reasoning could become a defense against poisoning. Wenchel starts from stochastic poetry, not as a slur. That process can be quite powerful, and the behaviors look reasoning-like. At some point it becomes a distinction without a difference: get good enough at the poetry and you mimic whatever reasoning is well enough that the two are indistinguishable. It is an old fight between learners and symbolic AI, and every year the learner approach looks more breathtaking and the shortcomings look smaller.

Labenz cites Othello-GPT: a language model trained only on move sequences still seemed to learn a board, and editing intermediate activations that changed that world model changed the downstream moves. Wenchel grants the robustness. These models are less fragile than their predecessors. Where they still come up short of human-level intelligence, in his view, is things like self-awareness and dealing with many input and output types; video work exists, but humans gather a lot of senses they do not. Will those gaps close? Potentially. That is why he started in AI more than twenty years ago. They are still stochastic parrots, he says, and he does not mean that dismissively. You can make a really powerful parrot. Conscience they can probably simulate well, and if you cannot tell the difference, does the label matter?

Corporate America has been leaning into AI for five-plus years. The companies that invested in data infrastructure and ML skill, often as a CEO-level priority, are harnessing LLMs much faster. Others are studying and preparing for initial rollouts in the fourth quarter. In the large-enterprise conversations Arthur has, no one is doing nothing. Security is a blocker. People know they cannot put this in production unless they mitigate the risks, which is how Wenchel says Arthur accelerates timelines: good security makes adoption faster, not slower.

Almost everyone is starting internally, and the winning paradigm right now is RAG. Proprietary data goes into a vector store, Weaviate, Pinecone, Chroma, and is fed to the LLM with the question. A large industrial equipment manufacturer put its technical documentation in a vector database so a field rep facing a customer question can get an answer in about ten seconds. Early in development they saw a lot of hallucinations; Arthur helped them almost eliminate those. Consumer businesses are using models on call-center transcripts for trends, a first step toward less-scripted assistants than the chatbots of three or four years ago. Investment banks, hedge funds, and private equity firms want to query years of expensive research reports and get in a minute what used to take a week.

How low do hallucinations need to be? Labenz's self-driving analogy is that humans already hallucinate, and the public still wants an order of magnitude more safety. Wenchel says that on average they typically reduce hallucinations by 87%, a sevenfold improvement, which is not zero. They can show a drastic improvement over a stock LLM, and that is often enough for people to feel comfortable deploying, except in healthcare and legal, where the tolerance is much lower, damage may not be reversible, and regulation and penalties wait. A wrong customer-service answer can later be made right with a refund. A wrong clinical answer may not.

The detector itself, for RAG, runs at about 87% accuracy. It breaks the response into claims, then for each claim asks whether the retrieved data supports it, fails to support it, or contradicts it. An application owner can block those answers or attach a disclaimer. At scale you get hallucination rates you can actually manage. That metric also tunes a practical tradeoff: how much to load into the context window. More retrieved text can mean a better answer, but it costs money and latency; filling something like 32,000 tokens has been seen to take up to a minute. The sweet spot is enough context that the model is unlikely to invent what it needs, without sending cost and wait time through the roof. The same budget problem shows up in how much chat history you keep for chain of thought.

Hallucination is where security and performance blur. Wenchel built the open-source Arthur Bench because generic benchmarks, including ones in Stanford HELM, which he likes, and the various leaderboards, are often a poor proxy for how a model will do on your data and your tasks. Labenz's rule of thumb is that a benchmark created before 2020 is probably ridiculous for LLMs. Bench is meant to make it easy to test providers, prompt regimes, and RAG setups on the exact work you care about. If you answer questions about technical product docs, build a suite of about a hundred of those and see which vendor actually wins for you.

The metrics have shifted too. Precision, recall, accuracy, false positives and negatives matter less than qualitative-but-measurable scores an LLM can assign: helpfulness, readability, concision, how often the model hedges, how often it hallucinates. A study that tried to show ChatGPT got dumber by testing prime-number identification tells you something about the model and almost nothing about customer service. You need your workloads. Scoring splits into two families: you define an ideal output and measure distance from it, which is highly measurable and a lot of work, and can punish a good but unexpected answer; or you use more open-ended judges that do not need a perfect reference, which is faster to stand up.

Is a hundred examples literal? Wenchel says yes, in the sense that it is already much better than current practice. Today people point a system at a provider, type the first half-dozen questions that come to mind, and manually review. Maybe the first pass is fifteen or twenty items. By the fortieth setting change, no one will spend four or five hours doing it, because it is tedious. A hundred samples defined once and rerun automatically is powerful. Because Arthur logs prompts, you can sample randomly, or pull by anomaly score, or bag the family of questions with high hallucination rates, or take everything already flagged as a hallucination, and stress-test the weakest corner.

What is actually going to production in the next sixty days, in Arthur's view, is RAG over proprietary data that would never have been in an OpenAI or Falcon training set, so if a claim is not in the retrieved context it is a very safe bet the model is inventing. Fine-tuning is being experimented with, not rolled to production; Arthur has used it internally for some filters and has seen that you can screw a model up. Open source is in prototype, not serving production workloads yet, though he expects more by year end. GPUs and the know-how to run open models efficiently at scale, as opposed to calling an API from OpenAI, Anthropic, or Cohere, are the bottlenecks. Arthur's relationship with those providers is symbiotic: the company is an unlock into the enterprise, because large firms will not deploy without solving the downside.

Among vendors, Wenchel will not crown a winner. They differ on hedging, on intrinsic hallucination without RAG, on multilingual ability, on summarization. Cohere's sweet spot, for people who know OpenAI and Anthropic better than they know Cohere, is enterprise behavior: in Arthur's testing it rarely hedges, while users get frustrated with OpenAI saying it cannot, as an LLM, tell you what to have for lunch. Cohere has also put a lot of work into multilingual models. The only honest way to pick is still your data and your prompts, which is why Bench is open source.

His crystal ball goes fuzzy past six months. Maturation is the easy prediction: mitigate risk, make models more robust, get more sentences to survive into a human's final draft. There will also be unanticipated watersheds. Even people tracking generative AI did not fully predict ChatGPT as a sea-change event, the way beating Kasparov or AlphaGo put things in motion beyond a pile of incremental papers. On call centers, the path is older than ChatGPT. Smart companies start with AI listening and topic-modeling, then assistive suggestions to the human agent, then handling some traffic directly. The key is knowing when you are out of your depth and handing the wheel back, the way a self-driving stack asks for a human in the dark and rain. Corner cases survive even at a hundred thousand calls a day.

Will anyone ever announce that alignment is solved? Wenchel does not treat that as a likely final destination. Labenz contrasts Arthur's customers, who come voluntarily because they do not want to embarrass themselves, with Microsoft racing Bing's Sydney out with inadequate testing, embarrassing itself, and then basically getting away with it, no apology that Labenz can recall. Wenchel's read is sharper: Microsoft was not hurt; it significantly benefited by adopting fast. Very few companies will have that risk appetite. People inside, he has been told, believed that thoroughly testing everything could take five years and still fail to predict every possibility, so they shipped with a rapid-response team for the funny rough edges.

On liability, he is not shy. If you put out a system, you need to own the consequences. That is the most effective way to make people take the responsibility seriously instead of being reckless. Asked whether that means no Section 230 for language-model providers, he calls 230 a complicated conversation and then agrees in spirit: if language models give wrong answers or answers that are harmful, there should be some liability involved. The through-line of the episode is the same claim he started with. Security done well does not slow AI down. Customers delay because the risk is not boxed in. Box it in, and the timeline moves up.