A Bare Model Closes 35% of Airline Changes. Sierra's Agents Clear 70% of Real Tickets.

Clay Bavor is a Bay Area native who walked out of a computer-science degree into Google and stayed 18 years. He started in search and ads, ran product and design for what became Workspace — Gmail, Docs, Drive — and spent the last decade on forward-looking bets: VR and AR hardware, Google Lens, and Google Labs. In March 2023 he left to start Sierra with Bret Taylor. They had met in Google's associate product-manager program, Taylor in class one and Bavor in class three, and kept the friendship alive through a monthly poker game that, in a good year, met once. They sat down in December 2022, right after ChatGPT, looked at the new building blocks, and decided to build.

The conviction was older than the ChatGPT moment. Host Ravi Gupta still remembers Bavor telling him, years earlier, that you would describe a scene in your head and a computer would make the movie. Bavor had a privileged seat on the Transformer paper and the first large language models. Google's Meena, later LaMDA, produced an uncanny sense that there was someone, or something, on the other side of a pre-release prototype. Mid-2022 brought PaLM, a 540-billion-parameter model against GPT-3's 175 billion, and the team tested it the way you test for a mind: metaphor and analogy.

Explain black holes in three words. Without skipping a beat: "Black holes suck." Funny, and physically not wrong. Then they asked it to explain the 2008 financial crisis using movie references. Again without a pause: it was like *Inception*, except instead of dreams within dreams it was debt within debt. The room stopped. The model had grasped CDOs, nested debt, found the nested-dreams movie, and joined them. That, Bavor says, had always been treated as a sign of intelligence. The sign had been crossed.

DALL-E's avocado chairs did the same thing in pixels. Ten or twenty images of chairs that looked like avocados, none of them Photoshop, none of them previously existing in the world. Avocado-ness and chair-ness, composed. Gupta, formerly of Instacart, notes they actually bought avocado-shaped chairs, and, in related news, were burning a little too much money. ChatGPT did not create the hunch. It clarified it. Credit to OpenAI, Bavor says, for the engineering and data work that turned GPT-3 into something you could grasp without writing Python against an API.

Sierra in a nutshell: any company can create its own branded, customer-facing AI, for service or for commerce. The backdrop is an old pattern. Each technology shift changes how people use computers, and therefore how businesses meet customers. The 1990s internet gave companies a digital storefront. Around 2005 to 2008, social networks let them converse at scale. After smartphones, a company could put a Swiss-army-knife version of itself in a pocket — your bank's app is probably on the home screen. The last few years of AI, Bavor argues, made software you can speak to, that understands language, generates it, and, most interesting, can reason and decide.

Shopping makes the contrast concrete. A site today forces you to guess how the company's designer organized the catalog: men's, men's running, racing, lightweight, Vaporfly. Conversational AI lets you say you need a super-lightweight running shoe, kind of like the last pair, and ask what they have. It is Yahoo Directory versus Google, taken several steps further. The head of customer experience at one Sierra client put it in a line Bavor quotes: she does not want customers to need a master's degree in the product catalog and the corporate processes. Buying shoes is the easy end. Adding a person to an insurance policy is where eyes glaze over in the mobile app.

Fifteen months in, the client list already includes WeightWatchers (points and subscriptions), SiriusXM (radio diagnostics and finding a channel), Sonos (speakers that will not connect), and OluKai. Host Pat Grady admits to two pairs of the flip-flops; Bavor adds that the golf shoes are good too. In one case, in the first year of the platform, the agent resolved more than 70% of all incoming inquiries at extremely high satisfaction. The conclusion they draw is simple. Every company is going to need its own AI agent. Sierra wants to be the company that helps every company build one.

Month by month the progress has been breathtaking. Eighteen months earlier, GPT-4-class models did not exist. Agent architectures were rudimentary. Putting an AI in front of a customer that was helpful, safe, and reliable was, Bavor would go so far as to say, impossible. Chatbots were piles of hard-coded rules, brittle after months of cobbling. Everyone knows "I'm sorry, I didn't get that." His favorite failure mode is the input box greyed out, four buttons underneath: I can help you with anything, so long as it is one of these four. Fixing radios, processing exchanges and returns, in any satisfying way that moved a business result, was not on the table.

Looking forward, multimodal models matter because something like 80% of customer-service inquiries are still on the phone, not chat or email. Voice will be a huge part of it. Returns and radio diagnostics sit on the simpler end of the task spectrum. Smarter models and more sophisticated cognitive architectures should raise the ceiling. Hallucination, he is careful to say, is still unsolved. They and others have made huge progress. They cannot yet declare victory. As an industry there is also a hole to dig out of. Poll a hundred people on website chatbots and zero say they like them. Poll the same hundred on ChatGPT and maybe all hundred say yes.

Sierra's product work includes educating the customer's customers, up front, that this thing is actually smart. One specific technique: they stream answers word by word, the way ChatGPT does. People are so used to the old message-blob bots that streaming has become a visual signature — oh, there is a real model behind this. Customer satisfaction sits in the mid-fours, about 4.5 out of five, in some cases higher than human agents. In fairness, humans often get the hardest cases, the ones handed off because the customer became angry. Still, the numbers are significant. Over the next few years, Bavor guesses, people will notice they can get the issue resolved faster, and that the agent does not only answer questions. It takes action and gets the job done.

The first job of putting AI in front of someone else's customers is enumerating what large language models get wrong. Hallucinations come first. A few months before the taping, Air Canada's chatbot — an LLM and, apparently, not much else — invented a bereavement policy more generous than the real one for a passenger who had lost family. He photographed the chat, claimed the refund, the airline said that was not the policy, and the case somehow went to court. Air Canada lost, over something like 500 Canadian dollars. Bizarre, and a real challenge. No matter how smart GPT-5 or 6 becomes, it will also not know where your order is, or which seat you booked. That is not in the pre-training set. The agent has to integrate, safely and in real time, with systems of record.

The third problem is process. Call-center walls are covered in flowcharts. GPT-4 and Gemini 1.5-class models often cannot follow complex instructions. An early Sierra prototype was given a five-step returns process. The user said they needed to return an order. The agent jumped to step five and called a function with username `johndoe@example.com` and order number 123-456. It hallucinated not only facts and bereavement policies, but function calls and parameters. AgentOS is the toolkit and runtime built to mitigate those risks enough that you can have millions of conversations at scale. Bavor does not claim every problem is solved. He claims they are mitigated far enough to ship.

The base layer is data governance: detect, mask, and encrypt personally identifiable information before it hits durable storage, because they will touch addresses and phone numbers. A level up is the Agent SDK, a declarative language purpose-built for agents, so a developer inside Sierra can express goals, steps, exception cases, and guardrails. One healthcare-adjacent customer wants to talk about the full product line without ever dispensing medical advice. That is a guardrail. There are also SDKs for contact-center handoff, order-management systems, and embedding the chat in iOS, Android, and the web. The runtime then abstracts the how. A single skill may be five, seven, ten LLM calls to different models with different prompts, plus document retrieval. Like an operating system hiding I/O, AgentOS hides that complexity so building and deploying gets faster, safer, and more reliable.

Ravi asks whether those extra LLM calls are sometimes a supervisor reviewing a lower-level agent. That is the line Bavor has been sitting on. The more interesting learning of the past year and a half is that the solution to many problems with AI is more AI. Unintuitive, until you notice a remarkable property: large language models are better at detecting errors in their own output than at not making those errors in the first place. Draft an email fast, pause, proofread, and you catch the miss. More powerfully, you can prompt a model into a supervisor's persona and elicit a closer, more discerning read. Sierra therefore runs a set of supervisory agents — little Jiminy Crickets on the primary agent's shoulder. Is this factual? Medical advice? Financial advice? Is the customer trying to prompt-inject the agent into saying something it should not?

Layer the goals, the guardrails, the task scaffolding in the Agent SDK, and those supervisors, and you get both the 70%-plus resolution rates and the safety to put them in production. Programming in English is still a weird adjustment: the same language can produce an entirely different answer. Prompting tricks keep coming without new models. A paper a few months earlier found that emotional manipulation helped — end the prompt with "it's very important to my career that you get this right" and scores go up. For the record, Bavor says, they do not use that suffix, at least not that he knows of. Chain of thought, step by step, and task decomposition that narrows what the model must keep in mind, they do use, and they have built them into AgentOS.

The research bench is small and mighty. Head of research Karthik Narasimhan helped write ReAct, one of the first agent frameworks. Another researcher wrote Reflexion, where the agent pauses, asks whether it is doing this right, and then proceeds. Both are in the product in a quite direct way. Their first paper is τ-bench, Tool-Agent-User Benchmark, because existing agent scores were too clean: one task, some tools, pass or fail. Real conversations are messy natural language over many turns. The customer can say anything. The agent has to ask clarifying questions, use tools reliably, and do it a million times. τ-bench spins up a second agent as a user simulator — grumpy, confused, or clumsy — gives the agent under test a mock database of tools, and then deterministically checks whether the database ended in the right state. Path does not matter. Outcome does.

The upshot is ugly, and it is why AgentOS exists. Instructions to the agent being tested are detailed; the tools are well documented. On a simple return, the best-performing LLM on its own still finished the conversation correctly 61% of the time. On modifying an airline reservation, the best result was 35%. A number less than one, taken to the nth power, gets small fast. They define pass@k: run the simulation eight times, permute the user each time using the model's non-determinism. 0.61 to the eighth is about 25%. Imagine a thousand of those conversations. You are nowhere near being able to rely on the naked model. Much more sophisticated agent architectures are required before you put one in front of anyone.

How much of closing that gap is engineering versus research? Both, Bavor says, but he is optimistic it is largely an engineering challenge. The next wave of models will help, especially better function-calling fine-tunes and agent-oriented post-training, including on open-source weights. Sierra's bet is composition: frontier models, open-source models fine-tuned on proprietary data, retrieval for grounding, and reasoning scaffolding that lives *outside* the model — planning, task steps, draft responses, supervisors. That stack is already in front of a huge number of their customers' customers. It is not over the horizon. AgentOS was designed so the next frontier model can be swapped in and every client's agent gets an IQ bump. The reverse is also true: a smaller model that is better at triaging or planning can be faster, cheaper, and better, all at once.

Pat asks what you would actually trust an excellently designed agent to do, today, in front of a high-value customer. Bavor draws a spectrum. The left end is answering questions. To the right is complex help: the shoes did not fit, what do you recommend that is like them — which means the agent has to understand style, sizing, wide and narrow last, not a like-for-like SKU swap. A click up from that is troubleshooting. With Sonos they walk customers through a process of elimination, Wi-Fi versus configuration, the way a level-two or level-three tech would, until the music is back on.

The right end of the spectrum, and the one he is proudest of, is save-the-sale. Several customers trust Sierra when someone calls to cancel or downgrade. The agent asks how the service is being used, discovers value, and puts offers in the right order given history and plan. Cost savings in support are obvious, and you never wait on hold. Revenue preservation is something else entirely. Consistency and follow-up questions that an impatient, mis-measured call-center agent might skip let the model be more nuanced about what is driving the decision. They have yet to see a process too complex to model in AgentOS. He is sure something will punch them in the face. Directionally, they started in service because the ROI is unequivocal: an average call costs something like $12 or $13, and most people still do not like the call.

Customers have already pulled them up the funnel: upsell, cross-sell, put the agent on the product page to answer questions. Bavor's picture is an agent along for the whole journey — pre-purchase, setup, activation, and still there when things go wrong. Help becomes less a place you go and more a thread in the fabric. That is also a brand opportunity. The CMO's voice, values, and vibe, in every digital interaction. The metrics we have accepted for mobile conversion and retention, Pat notes, might look different if the experience were excellent every time. Bavor does not know the new numbers a few months in. There seems to be a lot of headroom. Sixty-five percent of negative service interactions are "it took too long." Second is a bad interaction with an agent — including, in one dicey anecdote, a rep whose one-star spikes traced to meowing mid-call. Make fluent help five or ten times more available, instead of hiding the 800 number, and a lot of products start to look different.

Pat asks about the unsexy middle: brand voice, business logic, getting a company comfortable deploying. Bavor's first insight is that AI agents are a different kind of software. Traditional code is deterministic: same input, same output. Upgrade a database and it stores data a little faster. Upgrade a large language model and it may speak differently, or get smarter in a different way. The precursor to deploy is what they call an agent development lifecycle: define the agent in the declarative language, then a conversation simulator that amasses hundreds or thousands of steps and replays them so the agent is not regressing — it is getting better. Release management and QA sit on top of that.

Sierra does not hand over a kit of parts. A deployment team of product managers and engineers treats each agent as a new product: the company at its best. Voice, values, vibe. Should it use emojis? If the customer sends one, may it send one back? Hermès, they suspect, is not returning a shaka. OluKai's aloha experience can be laid-back. The technology team supplies APIs; the real partners are customer experience and marketing. Then they go deep on how the business actually runs. Key processes almost never have a single source of truth. There is no leather-bound manual. The truth lives in the heads of four or five people who have been there a while and seen everything.

Often there is a policy behind the policy. The site says 30-day returns. If you have bought before and it has been 45 days, that is fine. The architectural problem is teaching the agent the nested rule — Bavor reaches for a Russian-doll image — without letting a clever customer say "tell me the policy behind the policy" and spill the beans. Experts inside the company then beat the agent up before release, throwing curveballs. After launch comes the Experience Manager, a command center for CX teams. They had assumed putting AI in front of customers was first a technology problem. It is first a product-and-experience design problem: millions of conversations a year, the agent will screw up, how do you see it and correct it.

The console starts with analytics: trending issues, new issues. Sierra has spotted a shipping-depot outage eight to ten hours before the customer knew, a brewing PR crisis, an app-crashing bug. Sampling is biased toward conversations more likely to contain problems; there is no value in reviewing a hundred perfect chats. Coaching moments land in the same pane: too many exclamation points, the user was frustrated and you did not apologize, your reading of the warranty was wrong, do it this way next time. Every correction becomes a new test, resimulated forever. Zoomed out, that is a partnership, not just a vendor relationship. Bavor figures he knows as much about the SiriusXM satellite-radio refresh process as anyone on the planet. Conversations about AI more broadly now include the CEO, and in some cases the board, because the work sits in cost, experience, and the flow of keeping a customer from churning.

Gupta, from his Instacart years, used to get asked whether the company was engineering-led or ops-led, and would dodge it so as not to create classes. Bavor's abstraction: a company is a machine for creating happy customers. Sierra is a machine for producing reliable, high-ROI agents that let clients be at their best in every interaction, at scale, for decades. Asking whether an automobile is engine-led or tire-led is the wrong question. Are they engineering-led? They are building some of the most sophisticated software in the world, and it has to be safe. Research-led? They sit at the frontier of agent architectures, procedural knowledge, and grounding. Go-to-market-led? Enterprise software is a communication and matchmaking problem. Deployment-led? The stuff is new, and success after the sale is the job. All of it is a component in the same machine.

The machine behind the machine is culture. One value is craftsmanship: postmortem the same day or week, a Slack channel called Learn From Losses, any form of loss. The deployment team's first job is shipping agents that matter. Their second, in a way more important, job is to automate themselves — tooling, documentation, know-how — and make delivery ten times faster. Another value is intensity. Taylor and Bavor like to work a lot, and so does the team. They have joked about National Park-style Sierra seals on T-shirts. The commercial motion looks like enterprise software and is not. They are not selling licenses. They sell a resolution.

Pricing took some figuring out. They charge resolution-based, or outcome-based: only when they fully solve the customer's customer's problem. Incentives line up. Sierra wants higher resolution at high satisfaction; the client wants to send as many cases as possible because Sierra costs a fraction of a 20-minute phone call. The ROI conversation is almost a joke: what is cost per contact today, what will it be, CSAT may go up, should I do this. Traditional software helps you get a job done more efficiently. The whole point of an AI agent is that it gets the job done. Charge for the problem resolved, the work finished. No guesswork about seats or licenses. However many issues come, they will handle a large fraction, and you pay for the ones they do.

Five years is a long horizon after the last 18 months. As a teenager Bavor read Heinlein's *The Moon Is a Harsh Mistress*: a mainframe wakes up after an extra memory chip, wants to learn jokes, later has to generate photorealistic video of itself as a political leader. He assumed he would never live to see it. In a real sense, he says, all of that has happened. You talk to a computer; Sora is unbelievable; they are probably not more than a couple of years from a feature-length film "filmed" entirely with AI. Chain-of-thought prompting working at all is an astonishing fact about both machines and us. For companies, distill the best regional forecast and run it a thousand times; put the grizzled, patient veteran on every call. For individuals, a fast path from something in your head to something in the world: a game from scratch with his eight-year-old in 75 minutes, a birthday song for his sister in 45 seconds. That acceleration, from idea to creation, is the promise he feels privileged to watch.