Half of Production AI Is RAG. Fine-Tuning Almost Disappeared.

On No Priors, Elad Gil and Sarah Guo sit down with Ankur Goyal, cofounder and CEO of Braintrust, shortly after the company raised $36 million from Andreessen Horowitz and others. Braintrust is an end-to-end platform for evals, observability, and prompt development, used by companies that were among the first to ship AI in real products: Notion, Airtable, Instacart, Zapier, and Vercel. Ankur had been vice president of engineering at SingleStore and, before that, founder of Impira, a document-extraction company Figma acquired. The customer list is why the conversation matters: he is describing production stacks, not Twitter arguments.

He dates his AI work to what he now calls ancient history. Impira started in 2017, when shipping anything that worked was already hard, so the team built internal tooling to evaluate outputs, collect real user data, and feed that data back into better evals. A few years later Figma acquired the company, and Ankur watched the same problems reappear. His team built pretty much the same tooling again. Impira was pre-LLM; Figma was post-LLM. The problems did not care.

That repetition is what made him start Braintrust. Problems that existed before language models, he argues, are likely to exist in LLM-land for a long time, and having built the same toolkit twice made the need feel obvious rather than speculative. He took the early idea to people such as Brian and Mike at Zapier and Simon at Notion. After years of user interviews, he says he had never seen anything resonate like the desire for a serious eval solution. The first prototype was, in his words, pretty crappy. People used it anyway, and a little over a year of complaints and ideas turned it into something he now thinks is powerful.

Sarah Guo remembers the earliest conversations as a plan that might even have been open source. Then the first customer calls arrived, and people said the opposite: do not open-source this. They wanted Braintrust to exist for a long time, and they wanted to be able to pay for it. In developer tools, that kind of pull is rare. Many of those early customers had already built an internal version of Braintrust before they ever talked to Ankur, which is how they learned the job was harder than it looked.

Evals, he says, "sound really easy — oh, it's just a for loop." You print the results and glance at them. In practice, the faster you can run evals and read results that get complicated once agents enter the picture, the faster you can iterate. Doing that well is a hard product problem, and the pioneers of AI engineering had learned it the expensive way. Zapier's Brian pushed a second argument: AI would be a pervasive technology across the whole company, not a project one engineer babysits with a single team. He pointed Ankur at Vercel's docs and said new engineers read them and learn the right way to build web applications. Braintrust, in that framing, could be the same kind of standard for AI.

The pain Braintrust is built around is uncertainty about quality. A team gets excited, builds a prototype that works on a few examples, ships it to users, and discovers it does not actually work. Crossing from that demo to something that systematically works is the gap. Ankur's pitch is that evals demystify the process: write them in Braintrust, turn the crank, and grind output quality up, instead of living with the anxiety that a shipped feature may never get good. Elad notes that swapping GPT-4 for Claude, or changing a prompt, is exactly when you need to see which user outcomes got better and which got worse.

Sarah asks the question everyone wants answered from a company that sees live systems: how much of the Twitter stack — fine-tuning, RAG, agents — is real? RAG is unambiguous. Ankur says people are doing it, it is simple and obvious, and probably around 50% of the production use cases Braintrust sees involve RAG of some sort. Fine-tuning is a different story. He thinks people treat it as an outcome when it is only a technique. The outcome they actually want is automatic optimization of a workload, and fine-tuning is one very difficult way to get there.

Braintrust re-benchmarks fine-tuning on customer workloads every two to three months. There was a window after GPT-3.5 fine-tuning launched, when GPT-4 access was scarce, when it was the only lever some teams had to raise quality. Running GPT-4o is now extremely cheap. Since that shift, Ankur says almost all of his customers — if not all of them — have moved off fine-tuned models onto instruction-tuned models and are seeing strong performance. The irony is that when he and Sarah were first sketching Braintrust, fine-tuning was one of the first features they expected to build. Nobody is really doing it.

The distinction, for listeners who blur the terms, is like the difference between writing Python and building an FPGA. Instruction tuning means modifying the prompt to include examples of how the model should behave. You are still collecting guiding data, but you feed it into a much shallower process. Fine-tuning actually modifies or supplements model weights. Because it sits at a lower level, it is slower, more expensive, and easy to injure: a fine-tune can make the model worse on real-world use cases. That is why it is so much tougher to get right, and why prompt-side examples have won.

Open-source models, he says, are close to a watershed and not there yet. Anthropic had that moment when Claude 3 shipped, and Claude 3.5 Sonnet really took off. Llama 3.1 feels near the same line, but practical adoption is still very limited even as interest is higher than ever. Ankur is a developer who hates paying per token, then he looks at customer bills: in some cases the API cost is negligible, and where it is expensive the ROI is high. Customers optimize for the best user experience and the fastest iteration speed for developers. Until open source moves one of those two needles, broad production adoption will stay hard.

Elad points out that Ankur spent years as VP of engineering at SingleStore, a company known for a ferocious database team, and asks what data infrastructure the AI world is missing. The shift Ankur describes is almost insulting to the old stack. Enterprises hoarded semi-useful data in warehouses, and a whole pre-LLM industry — he names DataRobot — existed to train models on that proprietary structured data. The non-intuitive fact about LLMs is that something trained on the internet outperforms what an enterprise can produce from its own warehouse.

That changes how anyone should think about the value of data. Hoarding claims history or transaction history might not actually be that useful. The real question is how to construct a model that reasons well about the problems you care about, and Ankur does not think the path looks like ETL into a warehouse running on Amazon. A lot of the data customers store in Braintrust through logs never makes it to a warehouse at all. People do not care, he says, because they would not know what to do with it if they put it there.

Warehouses were designed for ad hoc exploration of structured data with SQL. Neither of those things is the AI workload: you are dealing with huge amounts of text, and you are not exploring it with queries. The most advanced companies Ankur sees use embeddings and the models themselves to sift data — for example, to find customer-support tickets that are not well represented in an eval set or a fine-tuning set. He thinks LLMs, and embeddings specifically, will become core to how people query data, replacing traditional algebraic relational indexes.

The vector-database debate, in his view, is mostly silly. Relational databases are perfectly capable of adding HNSW indexes. What will actually be disrupted is the OLAP workload. You cannot slap semantic search onto the architecture of a traditional data warehouse, and that is a deeper change than anything happening on the OLTP side. The storage layer, the cleaning step, and labeling all still exist, but they have to be rebuilt around a different question: what will you do with the data once you have it.

This is Ankur's third startup, and some of what he carried from MemSQL is a hard technical interview. He and cofounder Manu spent two or three days on the questions. A Braintrust frontend interview includes writing some C++, which loses a lot of candidates. He treats that as a useful signal: they want people willing to jump around in parts of the stack they do not already know. The other carry-over is an obsessive relationship with customers, made repeatable by being deliberate about which customers to prioritize. If Zapier and Notion have similar use cases, their requests are a good proxy for the next company like them. The following wave of AI builders, he says, now wants to engineer products like the ones they already admire — most of which run Braintrust — so the best practices are baked into the product.

Sarah has watched the same technical journey repeat. Early on, teams investigated fine-tuning or an open-source model, then realized they should just use GPT-4. They built internal tools, then realized their focus should be the product. In early Braintrust calls people said they did not need it; three months later they called back, and the delay was always roughly the same. Companies now a year or eighteen months in, Ankur says, usually have one or two mature north-star products and are trying to get them to the next stage. The most consistent move he has seen is walking back from the illusion that totally free-form agents will solve all of their problems.

Two or three months before the taping, many pioneering companies went way down the agent rabbit hole and realized it was the wrong approach. Performance is hard to control, error rates are high, and they compound quickly. Most of those companies have since tried a different architecture: control flow managed deterministically by their own code, with LLM calls made throughout the product rather than shoved into the while-loop of an agent. Ankur does not have a settled name for it yet. He calls it something like pervasive AI engineering throughout a product.

Who should staff that work has flipped. Many early LLM adopters had no ML staff when ChatGPT launched, which let them move fast from a blank slate. Classical machine-learning people and data scientists often resisted at first — the models were not good at their problems, or maybe the whole thing was a scam. Ankur went through the same shock watching Impira's document-extraction technology become irrelevant. He thinks it was largely emotional: GPT-3 was snarky, and it was also better at everything than anything you could train, which disrupts identity if you have spent a career on AI. The people who came around now form AI platform teams, mixing product engineers with a few statistics or data-science folks. For an enterprise starting today, he would hire smart product engineers first, get far with a good UI and basic AI engineering, and only add rigor, fine-tuning, or open-source cost-cutting after product-market fit. Vercel's v0, he notes, is now serious engineering, but it started simple.

Ankur's line, which Elad quotes back to him, is that Python is the language of machine learning and TypeScript is the language of AI. The data from Braintrust's customers supports it. A vast majority use TypeScript, including teams that used to write Python as their primary language. One reason is sociological: TypeScript is the language of product engineering, and product engineers are driving most of the AI innovation in the world Ankur sells into, so they are dragging the ecosystem with them. The other reason is technical. TypeScript's type system lets you launder the crazy stuff that comes out of a model into a well-defined structure the rest of the software can use. Python's type system is still immature. Uncertain shapes of data, he argues, are the entire point of the language choice.

The biggest tooling shift of the past six months, in his view, is people dropping frameworks. Early on, teams treated AI like a new Ruby on Rails: new kinds of applications that would need new frameworks. They have walked that back. AI is now a core part of software engineering, spreading through the codebase, not trapped inside whatever a single framework can express. On the vendor side, customers are consolidating, and Ankur says AWS is the force behind it. Bedrock plus Anthropic — especially Claude 3 and 3.5 — gave AWS its mojo back. He has talked to companies whose AI vendor list is literally OpenAI, AWS, and Braintrust. Do not underestimate the hyperscalers on infrastructure.

Elad asks why a CEO still spends so much time writing code. Ankur's answer is part market, part personality. Buyers of enterprise software now care about UI polish in a way Notion trained them to, which is a long way from the checklist-driven IT purchases many VCs grew up watching. For him, coding is simply the fastest, most efficient, and most pleasurable way to participate, and he has failed for more than a decade to quit. So Braintrust was engineered to support that habit. One of the first hires was Albert, a former investor and investment banker who covers selling, marketing, ops, and recruiting — the work that ate half or more of Ankur's days at Impira. Sarah notes that Jensen Huang made a similar point on the show: architect the company around the CEO, rather than forcing the same template every time. Sales comp, she adds, is not the place to invent.

Two other edges they were not firm enough about at Impira: everyone in the office, and being comfortable with interrupt-driven work. At Braintrust, if a customer complains, hates a piece of UI, or has an idea, the team almost always fixes it immediately. That pace makes many engineers uncomfortable and is exactly what the right engineers have wanted their whole careers, so they handpick for it. Go-to-market followed what Ankur calls the Elad school: a list of about 50 people leading the way in AI, recruited as investors or customers. Some invested while saying they had already built an internal version; many of those people later became users anyway. Fifty companies is a small TAM on paper, but they are influential, and they led to many more customers. Elad's warning is the usual failure mode: founders either target everyone from the Fortune 500 to tiny startups, or they niche down into a segment that is not worth pursuing.

Ankur says he asks himself every month how Braintrust should evolve, and surprisingly little of the answer changes. The company started by solving evals. Then Airtable used the evals product as observability: creating an "experiment" every day and dumping live logs into it. Once you see that, the next product is obvious. In AI, the point of observability is to collect data into datasets you can eval, and then maybe fine-tune; evals remain the important part. Customers next asked why they had to leave Braintrust and go back to an IDE that knew nothing about their evals or logs. They wanted to edit prompts in the product, reproduce live failures, save, and auto-deploy to production. That last part, Ankur admits, scared the crap out of him as an old-school engineer. It is what people wanted, so they built it. He mentions Martin, a recently converted daily active user, who now spends about half his day tinkering with prompts in Braintrust.

The shape he wants is a hybrid of GitHub and an AI workshop: create prompts in the product, write more advanced pieces in Python and TypeScript, stitch them together, and run the whole path through evals and observability. The ambition is a universal developer platform for AI. On quality, having lived through the pre-LLM era, he thinks the anxieties are the same. At Impira, people said all documents would be perfectly extracted within six months. LLMs are amazing, and document processing is still not a totally solved problem. People take whatever technology they have and push it to its extreme. AutoGPT, in his view, was a productive experiment in pushing past what the models can reasonably do. AI is inherently non-deterministic, so evals will still be there. The objects being evaluated will just get more complex.

AI is already evaluating itself. The analogy is a math homework assignment: it is much easier to validate a proof than to generate one, and the same principle holds for models. A frontier model looking at its own work, or another model's, can assess it accurately. Ankur estimates that probably more than half of the evals people run in Braintrust are already LLM-based. As GPT-4-quality inference gets cheaper, teams are starting to run those judges on logs. You can write LLM and code-based evaluators and execute them automatically on a fraction of production traffic. Sometimes that even lets you evaluate things no employee is allowed to look at. The model can read PII, score whether the use case is working, and report back. That unlock, he thinks, is what people will be doing for at least the next year.