Three Engineers, Three Weeks: The Internal LLM Tool Half of Stripe Uses Every Week

Emily Glassberg Sands is Stripe's head of information, a span that includes data science, growth, machine learning infrastructure, business applications, and corporate technology. She joined in 2021 after David Singleton, the CTO, reached out to ask her to lead data science. She did not know a ton about Stripe then, but she knew millions of businesses were collecting payments on it, which meant interesting data on those firms and on a large swath of the economy. Stripe, she thought, was in a position not only to help companies run more effectively but to learn which interventions actually improve long-term success, and in some cases to take those actions itself.

She now wears two hats. The first is enabling effective use of data across Stripe, from internal decision-making to data-powered products, including the unglamorous work of ML infrastructure and cleaner data plus a set of new generative bets shipping to users. The second is the self-serve business: the huge number of SMBs and startups that come to the website, pick an integration, and grow into invoicing, subscriptions, billing, tax, or revenue tools as the model demands. Stripe has used traditional ML for years in fraud and risk. First and foremost, though, it is building financial infrastructure for the internet, not an AI lab.

When LLMs broke through the zeitgeist, Stripe had the same "what the heck are we going to do" moment as a lot of companies listening to this podcast. Sands remembers looking at technical breakthroughs and product launches with awe and a little overwhelm. The opportunity to serve users was obvious. What it was, exactly, and how to get it off the ground quickly and safely, was not. The story starts with three engineers who hacked together, in three weeks, an internal beta called LLM Explorer: a ChatGPT-like interface for Stripe's 7,000 employees, so they could figure out how to apply the models to their own work.

Patrick and John Collison have spent years protecting a culture of bottom-up experimentation. Sands's team assumed experimentation would happen anyway, so they wanted it to happen well and safely. Elad Gil's map of enterprise LLM adoption has three buckets: external products, internal tools, and vendors such as Intercom or Zendesk adding AI on their own. Most companies, he says, start by imagining the product, then collapse into an internal tool so people can get their hands on it. Stripe followed that pattern on purpose. LLM Explorer is a front end over multiple back-end models. It began with GPT-3.5 and GPT-4 and now serves more than half a dozen. Security features such as stripping PII and rehydrating it were in from the start.

They spun up a Slack channel and a hackathon to build momentum. They did not need them. Within days a third of Stripe was using the tool. Logs showed people doing all sorts of things, which created a chance to share inside the product instead of in side Slack threads. Shortly after launch they added presets: save and share a prompt, search, upvote, see what bubbles up. Overnight there were about 300 reusable interaction patterns. Thousands of Stripes still run copy through the Stripe style guide, whether they are product marketers writing the website, sales development reps writing a cold email, or an executive preparing a talk track. Weekly active users sit at almost 3,000, just shy of half the company, engineers plus a ton of salespeople and marketers. The technology itself, she argues, can create the community, so people are not reinventing the same prompt in private.

Getting from internal prototyping to production-grade systems took a second mechanism. Applied ML accelerators ring-fence one-to-two pizza teams, more than one of them, to seed new AI bets. LLM Explorer itself came out of an accelerator. Pulling three engineers off Radar to go build an explorer is otherwise unrealistic. Funding runs out of Singleton's CTO office: a one-pizza team, six months of relatively durable budget, a rough charter and milestones, and an expectation that the team will learn and iterate. Other accelerators work the applied side, including support, user-facing and internal ops agents, asking whether a homegrown solution could span several internal applications even though third-party tools exist to buy.

The staffing model is almost the opposite of a hiring spree. The vast majority of people who join an accelerator have been at Stripe for many years. They rotate on, and it is likely to become their permanent home, though that is up to them. The company treats the structure as a growth path for internal talent. At the time of the conversation there were four such accelerators, with an open question of whether the right number is six, eight, or ten, in parallel with helping vertical product teams adopt LLMs directly.

In user-facing applications, Stripe is finding LLMs useful in two ways: automating the writing of code, and accelerating information retrieval. Radar Assistant and Sigma Assistant, both in beta and rolling out more broadly, are the examples. Radar was one of Stripe's first non-payments products, an ML system that identifies and blocks fraudulent transactions under the hood without user direction. Radar for Fraud Teams sits on top and lets users write custom rules, for example blocking IP addresses in a country where you have no customers. Those rules used to be coded by hand. Radar Assistant lets people describe them in natural language.

Speed matters in fighting fraud. You have to work faster than the fraudsters. With the assistant, a whole range of people, from fraud analysts to less technical folks, can implement rules quickly without going through a developer. Guo points out the pattern travels. Many decision engines are already a mix of heuristics and machine learning. The ability to take an explicit natural-language policy and have it work with less engineering help applies to underwriting and other choices, not only fraud. Sands adds a sharper cut. For some customers this widens which employees can use custom Radar rules. For a lot of small businesses it is the first time they can use those solutions at all. They got on Stripe with payment links and hosted invoices. They do not have developers on hand. They would not dream of coding fraud rules. Plain English is what puts the tool in reach, and that democratizing effect, she says, is aligned with Stripe's founding ethos.

Sigma Assistant uses the same trick in a different context. Sigma is Stripe's SQL-based reporting product. For most users the data in it is all of their revenue: which customers, where, buying what, for how much, who retains, who churns, the facts behind a lot of firm decisions. The assistant means employees do not have to speak SQL. They ask questions in natural language. Beta users are already ranging from "how much revenue did we generate in December" to "what types of customers tend to be most delayed with their payments." A broader rollout was planned for later that year.

Stepping back three to five years, Sands thinks the question is bigger than what a fintech can ship in a year. What does it mean to apply generative AI to the economy at large? You could start with payments optimization. Stripe already does front-end and back-end work there. Incremental model improvements still drive meaningful basis points of uplift quarter over quarter. It does not feel crazy to her that a foundation model built on financial data could outperform traditional approaches by 100 or 200 bips. Then it gets more interesting: can Stripe become more of an economic operating system for its users? Dashboards of insights and recommendations, an API for customer-level predictions, pricing, recommendations, and discounting on autopilot. Stripe already abstracts payments, refunds, and disputes. Higher-order tasks, the value of a user, the right price, geo strategy, could be next. At a macro level, CPI and small-business indexes are useful and lagging. Real-time payments data might speed insight and response.

A year into the exploration, the operating model is still iterative. It is not one pizza team. It is four, maybe six or eight or ten, plus support for vertical teams. The north star she names is consistent: non-technical users doing things they could not do before, and technical users moving an order of magnitude faster, mostly by automating code and speeding retrieval. That work is coming both out of existing vertical teams and out of the accelerators.

The explorer was quickly turned into an internal API so Stripes could build production systems programmatically and safely. Sixty applications now sit on it, a bunch internal and several external, and those seeds became the product initiatives the company is investing in more heavily. Default models exist by use case, but teams keep agency on cost, latency, and performance. The most expensive applications are billed back to local teams, and the central group sits with them on unit economics, quality, scale, and latency. Centralizing enterprise agreements and infrastructure bought economies of scale. It is not prescriptive. On buy versus build, they stitch together external tools where the market is good, including Flyte and Weights & Biases for training-style tracking, and they build where the need is unique. The charge-level experimentation platform is internal, because latency and reliability requirements at that grain are too high for what the market offered.

Asked for the biggest whitespace in fintech AI, putting Stripe aside, Sands offers three directions. Merchant identity, who the business is, what they sell, whether they meet a card network or BIN sponsor's rules in a complicated regulatory environment, is a hard and valuable problem, important not only for go-to-market but for credit, supportability, and network access. Integrations are another: Stripe already has no-code products such as payment links, and LLMs can write code, but building a robust, user-specific financial integration without a bench of payments engineers still has a lot of work in it as the decisions get more complicated. The third is a layer on top of payments data that does not only make payments work better but helps the business itself grow. Stripe's incentives, she notes, are aligned. If users grow, Stripe grows, and the economy grows.

That data layer is not only a vision. Radar is the obvious example. The optimized checkout suite is the one an end user might notice: a bundle of front-end payments optimizations, including dynamically presenting payment methods in the order most relevant for that customer. Little things, she says, that add up in conversion and merchant revenue. Less visible is a host of back-end optimizations. Stripe uses ML to tune authorization requests to issuers, identifying retry messaging and routing combinations that recover a big chunk of false declines, about 10 percent, billions of dollars globally. False declines are good transactions rejected as suspicious: missed revenue and an annoyed customer.

Similar tricks show up across products. For recurring charges in billing, smart dunning identifies the optimal day and time to retry a payment that failed, for example after insufficient funds, and reduces declines by about 30 percent. Radar considers a thousand characteristics of a transaction and decides in less than 100 milliseconds whether each of the billions of legitimate payments on Stripe can go through. Conversion, authorization, fraud, and cost optimization are all places where scale of data creates a better end-user experience, more revenue for the business, and a larger internet economy.

Stripe does get a rich view of where the economy is headed and uses it to guide internal decisions. Whether a version of that, a CPI or small-business index six months earlier, can go to users is an open, early question, part of the economic-operating-system idea at the macro layer as well as the micro. It does not drive headcount whiplash. Adding 10 people or 1,000 is much less about the macro on average and much more about where Stripe sees a chance to serve users. The current wave of AI startups is the example. They have fundamentally different needs than prior waves, which begs the question of where to invest now to get ahead of demand.

The through line in Sands's career, academia then industry, is using data to understand how individuals and firms make decisions and to make those decisions higher quality. In college she met a playwright in New York who told her that less than a fifth of productions on U.S. stages were written by women, and asked for help understanding why. Excellent playwrights donated four never-before-seen scripts. Sands sent them to hundreds of theaters, varying only the pen name, Mary Walker or Michael Walker. The same script, when purportedly written by a woman, was less likely to be produced. More important, the theater community cared. They wanted the best plays on stage. The study spurred awareness and change. Today half of productions on U.S. stages are written by women. That experience is why she pursued a Ph.D. in economics, why she left academia for Coursera when it had 40 people, and why Stripe's habit of acting as a beneficent player in its ecosystem appealed to her.

She spent about eight years at Coursera, growing from an individual contributor to leading the end-to-end data team. The first products were the obvious ones: personalized discovery, a personalized learning experience, more scalable teaching. Relatively quickly the team moved toward the labor market: using education data to help learners and companies measure and close skills gaps and get people into jobs that fit. People get education for two reasons, she says. They develop skills, and they get rewarded for those skills in the labor market. AI can customize elementary learning and put high-quality instruction in pockets of the world that would not otherwise have it. The second job, signaling that learning, building a credential, matters just as much. On World Economic Forum work she pushed to make skills, with data, more the currency of the labor market. Coursera's enterprise product moved that way. The fairest version, she thinks, is skill measurement, not a substitute for better teaching but a recognition of what individuals actually need from education.

Stripe has grown up with waves of builders: marketplaces and social platforms, the young DoorDash, Instacart, Postmates, Twilio, then SaaS, now AI. Sands names four differences in this latest wave. AI startups have substantial compute costs out of the gate, which pressures them to build monetization engines faster. Demand is global from day one, digital art and music and other borderless products that they want across borders immediately. Consumer-facing AI leans hard on subscriptions. And because they monetize earlier, lean teams have to operate financially like real businesses sooner than they are ready, which is why Stripe is seeing adoption of its revenue and financial automation suite. Gil compares it to 1970s four-year vesting, when companies went public in four years, and to early internet firms such as Yahoo and eBay that became profitable within a few years. Rapid monetization, he suggests, is what product-market fit plus pent-up demand looks like.

Stripe's own numbers back the feel of a spike. The last year saw a massive jump in generative AI companies on the platform, from two-person teams most listeners have never heard of to hyperscaling startups with millions of users, such as Otter.ai and Midjourney. Looking at Forbes's list of the top 50 AI companies from the previous year, over half were using Stripe. Oftentimes a bunch of top startups are not really monetizing yet. What is striking here is the share that are monetizing early and fast, at the foundation layer, OpenAI and Mistral, and at the application layer, Moonbeam for writing, Runway for video editing. That, she says, is pretty remarkable.