Most of this data has never been in a database: Chroma's wartime CTO on RAG

Nathan Labenz opens this October 2023 episode of The Cognitive Revolution by bringing back Anton Troynikov, cofounder and CTO of Chroma. Their first conversation was episode 5, released in early March, before GPT-4. In the eight months since, retrieval-augmented generation overtook agents as the hottest application trend: companies spun up document-backed bots for labor saving, and nearly as many deployed a vector database for the first time. Labenz had treated the earlier taping as a tutorial on embeddings and latent space; this time he wanted everything that had changed, including for Chroma as a business.

He came in wondering whether Chroma and other vector-database startups would struggle to grow, given how incumbent database vendors were racing to ship their own vector stores. Troynikov, he says, did a good job of reminding him of one of his own maxims: there is a good chance everyone is still thinking too small. The observation that stuck was that most of the data stored in Chroma has never been in a database before. That suggests plenty of growth at the infrastructure layer, at least for a while, even as lots of incumbents get into the game.

The company's plan is to pull more of the retrieval stack into the database until using it is as simple as dumping text into a box, not unlike how people already use chatbots. Labenz frames Troynikov as a wartime CTO and a genuine intellectual leader who still doubts LLM reasoning more than he does, and he suggests sending the episode to a software or AI engineer. When the interview starts, the first question is the new word in Troynikov's Twitter bio.

Labenz notes that Troynikov now has "wartime" next to a pirate flag in his Twitter handle. The flag was already there; wartime is more recent. Troynikov says part of it is branding, but the mental shift is real: Chroma has a concrete objective and is taking on what he thinks of as the right level of risk for a startup. That is a different place to be than exploratory mode, when you are still figuring out what people want.

The mandate is public enough to say out loud. Build a horizontally scalable system on Chroma's vector search and storage engine, then deliver it as a cloud service — the number-one thing users are asking for. The team has grown since March, he says; they know the shape, the quality bar, and the differentiation. What remains is execution risk: how well they can actually ship it.

Differentiation is still muddy, Labenz admits, because he cannot yet tell vector-database vendors apart. Troynikov treats that as an artifact of the adoption curve. His estimate is that 99% of the people who will one day build an AI-in-the-loop application have not touched it at all. Chroma's users run from hackathon first-timers, who need to be told you can inject information into a model, to production teams asking how to write human feedback and evaluations back into the data layer. There are far fewer teams at that later end, plus a middle step where a successful experiment has to scale the right way.

The easy on-ramp is deliberate: `pip install chromadb`, or `chroma run` on the CLI, with sensible defaults and no need to choose an embedding function — throw text at it and it works, which is why Chroma is the default in many frameworks. A single node with a lot of RAM already runs tens to hundreds of thousands of collections with millions of vectors each. The next problem is architectural. Most vector products were built as search indexes for pre-AI use cases such as Pinterest-style similar images: mostly static data, rare deletes, one shared index. An application database is the opposite — an index per user space, independently updated. Chroma Cloud, he says, will hide that behind elastic scaling and charge per query plus storage, so users never think about pods or shards.

After a sponsor break, Labenz asks what people actually use for embeddings. Troynikov says most users are happy with open-source models; a sentence transformer is straightforward to run locally and often close enough to OpenAI's Ada. Ada still wins when data already lives on Azure, because the endpoint avoids egress and the cost of running your own model at scale. The deeper preference he is seeing is not just price. Teams want to keep as much of the data and the RAG loop in house as they can — not only embeddings, but replacing a GPT-4 call with Mistral or Llama 2 fine-tuned for the task, so they do not send data over the network. He calls that an inflection against the "one model to rule them all" story, and notes that the language-model call is the expensive part anyway.

Labenz's own first tip is not to skimp on how many records you retrieve: use the context window. Troynikov pushes back. It has been folklore, and recent research has shown it more empirically, that distracting information in the context window measurably destroys overall application performance, in a way that is hard to attribute. You want every relevant result and none of the accidents. That depends on chunking that is semantically meaningful for the task, which depends on the embedding model; the pieces are interconnected. Labenz has seen Claude 2 go off the rails at a full 100K tokens and behave at 50K. His Athena app went from two or three chunks to ten, because he trusts GPT-4 or Claude 2 to pick the right passage more than he trusts retrieval to surface only three.

Troynikov would step further back. Get a reliable measure of effectiveness first, from two places: offline evaluation of results, and human feedback inside the application. Only then turn the old knobs. His rule of thumb is as much relevant information as you can fit, and as little irrelevant information as you can get out. Without that measure, he implies, later tuning is guesswork.

Labenz has been experimenting with an adapter between the user's query and the inner-product math the database does. Troynikov says it has been shown that an affine transform — squeezing, stretching, and rotating space — is enough to map one embedding space onto another. What it really does is expand some dimensions of the vector and constrict others. You can recompute every stored embedding, or you can compute the forward transform and apply it only to the query. That second path is the interesting one: the same store can serve different applications and even different users, each with a learned transform, without modifying a single stored vector. Feedback from every app sitting on the database could also back-propagate into a shared embedding-model fine-tune. Chroma intends to offer both.

In practice, Labenz is not fine-tuning anything yet. He uses the language model to transform text to text: generate what the answer might look like, with placeholders, then search with that hypothetical document. The technique is HyDE. Looking up a client's kids' birthdays works poorly as a question and much better as a statement like "kid A was born on this date." Troynikov notes that general-purpose embedding models are already trained so that queries land near answers; HyDE is manually supplying a property the model may already have, at the cost of an extra call per query. If it is cost-effective, use it. In the long run, he thinks fine-tuning the embedding space itself is cheaper.

Birthdays also point at a split he wants developers to take seriously. Numerical facts, names, and events often belong in a SQL table the model can query directly if it knows the schema, rather than in a lossy embedding search. Embeddings are for unstructured data. Chroma already uses a relational store for documents and metadata, which is how keyword filters work out of the box. Zooming out, the problem is not which representation to pick. It is making sure the model has the information it needs to finish the task, behind one interface, so the application developer is not wiring HyDE and three databases together by hand.

The metaphor he likes is that AI developers today have the Postgres storage engine and are writing the query planner, the SQL interpreter, and the block-storage algorithms themselves. Time spent on that plumbing is time not spent on the product. Chroma wants a unified data interface for the AI application, with adapters to other sources as needed, and more intelligence dropped into the data layer — including a local model that treats "given this query, find the right data" as a decision problem of the kind already used in reinforcement learning. SQL generation against tables the user points at is the obvious next step.

Labenz's business worry is the one every AI sub-market is having: how much value accrues to startups versus incumbents. Nobody, he thinks, rebuilds Salesforce before Salesforce ships its AI layer, and you will pry deployed databases out of mid-career developers' hands only with great difficulty. Troynikov's reply is the observation Labenz already previewed. In many cases, data that goes into Chroma has never been in a database. It sat in document stores for humans to read. Models can now reason about large quantities of natural language, whereas computers could previously reason only about structured tables. He does not need to convince people to rip out the database they already have. He needs to tell them they can finally work with the rest.

Salesforce comes up because he has been reading *Softwar*, the Larry Ellison biography. Marc Benioff was an Oracle executive; Ellison was an early Salesforce investor. What Salesforce did was unify an organization's data around the customer funnel — the earliest point customer information enters the company — and then build a platform of applications on top. Troynikov sees Chroma following a similar model, with the caveat that this is speculation. The company's current job is to earn a seat at that table by delivering the product the market wants. He does not think Chroma is in competition with a lot of incumbent stores; they are a complement, the way Salesforce and Oracle work together, as long as the interface lets people build applications on top.

Today the new inventory is documents, because GPT is a general-purpose text model. Tomorrow — he means this almost literally, "more like a week from now" — there will be models that interpret images and sound, including PowerPoint slides people want distilled into something actionable. In sheer megabytes, he thinks that pile rivals what lives in relational tables. Labenz's own Athena work is already on the boring side of that wave: an internal tool, built with chatbase.co, over about 400 policy documents for a thousand executive assistants. They used to get hundreds of questions a day; a significant share is now answered immediately by the model. Retrieval and model quality, Troynikov says, are a virtuous cycle, which is why he is glad Chroma is focused on this component rather than on training the next foundation model.

He sorts organizations into three kinds. First, legacy businesses adapting existing processes to AI; some will succeed and outcompete the ones that fail. Second, companies building tooling that could not exist without the model. Third, and still emerging, businesses that serve the same customers as incumbents but are built from the ground up around AI-first processes — the first few that pull it off, he thinks, will be wildly successful. Real estate is the vertical that comes to mind: a vast human-powered document-processing operation over heterogeneous formats. The filter is distribution. Go where you can get distribution fast enough that AI efficiencies actually matter. He would not try to compete with Coca-Cola on the basis that you use AI and they do not.

Growth, Troynikov says, still comes mostly from the early part of the adoption curve, but users graduate, and Chroma already has production enterprises, fast-growing startups, and hackathon first-timers on the same product. He is proudest of a use he did not know about at first: the Voyager paper from Jim Fan's group at NVIDIA, a Minecraft-playing agent with no training run. Looking at the codebase, he found Chroma as the memory engine. Skills go in, get retrieved when needed, and can be deleted when you do not want a behavior. That, for him, is the beauty of retrieval as a complete AI system compared with weights you cannot edit.

He also rejects the idea that agents and RAG were sequential fashions. They are not orthogonal. Agents use retrieval, and making retrieval work better will make agents work better out of the box. He is already talking to agent-developer companies about that. "Good retrieval is a significant fraction of what we need to make good agents work."

Labenz had sent notes about fine-tuning failing to implant facts. Troynikov's rough mental model is that fine-tuning is the style or manner in which you want a task accomplished — what to pay attention to, how to synthesize a conclusion — while retrieval is the actual data. The latest retrieval-augmented pre-training and fine-tuning papers, he says, show the knobs are complementary: fine-tuning helps the model use what retrieval brings back. The scaling curves are still real. If you do not have enough tokens to overfit the facts, you will get more from a general-purpose model plus retrieval than from tuning. That feeds the larger market question of one "Death Star" model versus many smaller task-specific ones, which also decides whether compute lives in an Azure data center or on your network.

How much knowledge can you squeeze out of a model while keeping the raw general ability? His current picture is that you arrive at something that emulates reasoning by showing it concrete examples of reasoning; there is no abstract corpus of reasoning to train on. Nobody yet knows how to keep the abstract part without the facts. He calls Anthropic's recent circuit work a tremendous breakthrough, still on a very small scale, and he is not convinced it is "just an engineering problem from here." Representation engineering from Dan Hendrycks, Zico Kolter, Andy Zou and colleagues is the other line he has been advocating. The ideal, for him, is a 140-IQ machine that knows nothing about the world until you tell it, then reasons with exactly that information, inspectable down to the neuron.

Labenz asks how far stored and retrieved objects will slip from human-readable text. Troynikov hears a safety question first, and he likes the control it implies. Commercial interest and AI safety align here: a commercial deployment has no interest in a system it cannot predict. Models might learn steganography, or several models might collaborate through a shared store, but that is a much harder ask than hiding knowledge in weights. If the stored object is supposed to be human-interpretable text and he does not understand it, his instinct is to delete it.

The question Labenz actually meant is more modest. Instead of embedding text, retrieving the document, and stuffing it into a context window, why not store vectors and inject numbers straight into the model? Troynikov says 100%. Returning documents, decoding an embedding, and re-encoding it into another space is a silly, lossy pathway. Pass the vector into the space you care about. He does not think that breaks interpretability, because you still know where every vector came from and which inputs and outputs it corresponds to.

Controllability still matters. One gate is decodability: if an embedding leaves the realm of what you can decode, treat it as garbage, delete it, or forbid that region of latent space. He also wants to drop the idea that a vector of numbers is automatically something we do not understand. Nobody writes airliner instructions in English; we know how the plane will fly because the tools keep up. Chroma's first product, in company history, was a latent-space visualizer. Coming full circle, they need to build that tooling back into the product, because you cannot eyeball the top rows of this database the way you can in SQL. The falling barrier in AI, he argues, is that an average person talking in natural language can already do what used to count as advanced research — a combination of power and accessibility he compares to the early web, and to the Cambrian explosions that follow.

Back on the practical RAG problems Labenz is living with, Troynikov's first product step is to plug Chroma into the evaluation tools partners are already building. Evaluations are useless unless you can act on them. Chroma is developing a feedback endpoint: connect LangSmith or another eval product, press a button, and automatically adapt the embedding space from those scores. Next, subsample the user's data and pick an embedding model before the whole corpus is embedded, using both the geometry Chroma already understands — densities and distributions in latent space — and evals from similar tasks. Labenz's Athena chat, with a thumbs-up or thumbs-down on each retrieved snippet, is exactly the signal they want.

How much labeled retrieval do you need? At minimum, enough to construct that affine transform densely, which is a function of embedding dimensionality. A higher-dimensional space may be more representative; a lower-dimensional one is easier to fine-tune with less data. Even Ada-2 embeddings are not used at full rank: a given corpus sits on a manifold, and Chroma can project down, fine-tune there, and keep the full-dimensional copy as the data shifts. The projection is itself a learned transform. They are hiring a head of research for this class of problem, including the longer-term idea of pushing embeddings directly into the inference layer.

There is also a pile of convenience work that changes quality. If twelve documents lie next to one another and retrieval returns all of them, you have captured nothing new; maximal marginal relevance is the usual heuristic for not repeating yourself. Models can do better: a *conditional* summary, given what the user is actually trying to accomplish, that collapses the redundant set into one data point. Labenz has been thinking about a cheaper Claude Instant filter between "retrieve ten" and "stuff the context," to save GPT-4 tokens and latency. Troynikov's line is that you should never see that layer, any more than you think about the query plan Postgres generated for a web app. Lightweight open-source models, running on the same inference hardware as the embedding function, are the natural fit.

Owning that loop is also how you own the data flywheel: a model that keeps adapting to your business rather than to a general-purpose average. For most people, Troynikov agrees with Labenz's advice. Start capturing feedback from day one. Online fine-tuning can wait; most teams will never have enough data to do it continuously. The data still has to be in the log.

The line Troynikov keeps returning to is the episode's cold open. People are conditioned to think of data as a static thing sitting somewhere at an instance in time. He thinks of it as a loop, almost a control-system loop: data is an engine, interacting with the outside world through computation, constantly adapting and improving itself. Capture the affordances for that loop even if you are not fine-tuning today.

Around the cloud launch he wants Chroma Datasets: English Wikipedia, Bloomberg, real-time news — corpora everyone currently embeds on their own. Hit an endpoint, pay a few cents, get the document or the vector. Once many applications query the same public collection, Chroma knows which model the data is being fed to and what task it is for, and can fine-tune the representation for that use instead of serving a generic embedding. He calls those living datasets. The embedding space you want for Bloomberg if you are writing news is not the one you want for demographics, and probably not the one you want for a dating app, high-end or otherwise.

Keeping the actual rows fresh is a different problem from the search-index world. Updates and inserts are rare in a web-scale index; in an application they happen all the time, interactively. A crop of streaming and sync tools is growing to pour external events into the store, and Chroma has been talking with Unstructured and others and plans to support them. Labenz's live headache on Athena is more like journaling: should chat history live alongside canonical client profiles? If a profile is updated, do you keep the old version? He finds himself newly impressed by the elegance of human memory.

Near term, Troynikov would use heuristics Chroma already has: put timestamps in metadata and filter by date. Longer term, he wants time as a dimension in the embedding space itself, with statements about the past decaying, and the retriever weighting that axis when it needs a history. Project time out if you want everything; slice it if you want a moment. The embedding model should understand time, though not every fact belongs in the vector store — some still want a SQL table. A Max Tegmark-group paper finding that language models represent latitude, longitude, and a timeline did not surprise him. Contrastive pretraining only encodes similarity and difference; at the top of the hierarchy there is no third reference point, so directions can float, while structure is forced among nearer points in lower layers. That is also, for him, why the highest-order concepts show up in the middle of the network.

Labenz wants a number he can quote when someone says the approximate nearest-neighbor step ate a result. Troynikov's answer is that almost all of the miss is the representation. The tradeoff between recall and speed for ANN is well understood and fully tunable; the graphs for each algorithmic implementation are public. When your results differ from those graphs, it is the representation's fault. On default settings he would guess something like one in a million, with the usual caveat that it depends on the data distribution. A pathological HNSW graph stuffed with the same entry repeated as unique rows will corrupt the index. Latency is dominated by model inference anyway, so trading a little more search time for recall is cheap. At small scale you can skip the index entirely; Chroma already switches to a brute-force structure automatically.

On partnerships he is comfortable working widely: LLM API labs, because better models make retrieval more useful and vice versa; application frameworks, for the same reason; eval platforms, via the feedback endpoint; ETL for unstructured data. Chroma's remit, he says, is well scoped, and the company is out of the first exploratory phase of the product.

Will OpenAI build its own database? They have hired retrieval people, and teasers around developer day imply at least a lightweight internal system. He thinks they are unlikely to release something for developers like what Chroma is building, and that the nuts and bolts of a retrieval engine are not on their main line, even if retrieval itself matters for AGI. OpenAI has been doing enterprise deals early and going in with retrieval vendors; Troynikov reads that as a sign they want customers to pay for GPT, not to own the store. Google has already highlighted Chroma as a launch partner for the PaLM 2 public API, which he attributes in part to how easy the developer tooling is. Anthropic has come up less. He has been playing with Claude 2's long context and PDF ingestion, and sees the lab as more of a research test bed. Training-data retrieval looks like the old search-index use case, where people currently use FAISS, which is hard to scale — a gap he expects Chroma to fill over time.

Labenz asks whether Troynikov has updated his p(doom) after a slide that had been making the rounds at the AI Engineer world. Troynikov finds the question almost funny: there are no consequences for saying whatever number you want, so people say fun stuff. He has stopped arguing about doom. The question he actually cares about is whether this technical path can produce a reasoning system at all.

The experiment started from chess. When GPT played badly from random positions, some people said it was emulating bad chess players — a claim Troynikov found insanely strong, because it implied the model knew the rules, knew optimal play, *and* could condition on a weak opponent. He looked for the weakest version of that claim: Conway's Game of Life. Asked for the rules, GPT recites them. Given a word problem — n cells alive, k dead, the center cell alive, what is the next state of the center — it scores 100%. That is at least one internal representation of the rule. Presented with the same information as a grid of all 512 state transitions, it recites the rules and then gets the answer wrong, even when each cell is its own token.

Right versus wrong was no longer the point. If the model has rules, they should apply across representations. He fed it random 10×10 boards, asked for a single iteration, and measured consistency: does it do the same thing with the same local transition, and does that match the rules it executed on the 512 individual cases? Inside the grid representation it is mostly consistent. It does not apply the rules stated in one representation to the other. That, for him, points at no world model — one piece of evidence, one angle of attack, not a finished argument. A further detail: how random a transition is depends on how often that state appears statistically. Rarer states, more random outputs. He reads that as another strike against.

The practical moral is not that the technology is useless. It already produces value in domains that run on human interpretation rather than exact results. The real power, if it arrives, is a general-purpose computing machine, and he does not think we are there yet. Labenz's quick reaction is that this sounds like an incomplete world model rather than none, and that the live question is how general "general" is. Troynikov tells him to look at the thread. He has stopped arguing about doom and started asking more practical questions.