A Hundred Million Tokens in Context Would Cost $10 Million. Tengyu Ma Runs the RAG Math

Sarah Guo introduces Tengyu Ma as an assistant professor of computer science at Stanford and the co-founder and CEO of Voyage AI, which trains embeddings and rerankers for retrieval systems. His research, she notes, is unusually wide: theory, reinforcement learning, embeddings, transformers, optimizers. Ma says the common thread is that most of his papers have some theoretical thinking in them. The work runs from mathematical proofs of deep learning systems through practical language models, and lately it has concentrated on two bets.

The first is training efficiency. We are running out of data and compute, he argues, so we have to use both much better. The second is reasoning, which he treats as important and, in a sense, risky: nobody yet knows how fast the hard reasoning problems can be solved. Early on he worked on optimization for matrix completion. He then moved to sentence and vector embeddings, including a paper — from before transformers — that simply averaged word embeddings and applied a few PCA transformations, and saw a large jump in quality. Contrastive learning came next, first on images, then as the method that now trains embedding models, along with an attempt to explain why it works.

The optimizer paper from last year is Sophia. Ma's group found a new optimizer that improved pretraining efficiency by 2×. Sarah's reaction is that Adam is very old at this point, and Ma agrees: it is about ten years old. In those ten years, countless papers claimed to beat it in various settings, and Adam remained the default for training large language models. That stubbornness is why he put students on the problem around 2018 or 2019. One paper came out after several years and several failed projects and ideas.

The more expensive evidence arrived from industry. A Facebook friend used Sophia in large-scale multimodal training — Ma does not know the exact size, but assumes more than 100 billion parameters — and saw a 1.6× improvement in training efficiency at that scale. He translates it into a budget: a $16 million run versus a $10 million run. An optimizer-level change, in other words, moves real money.

Ma started Voyage last year on leave from Stanford. He chose Stanford in part because the industry connection is stronger than at many universities, and he says entrepreneurship was always somewhere in the career plan. The timing, not the ambition, was the question. Technologies felt mature enough that commercialization finally made sense. He dug up lecture slides from seven years ago, when he started teaching, including a session with Chris Ré on applied machine learning. The pipeline had seven steps: define the problem, collect data, choose a loss, train, iterate. It was a slog.

In the foundation-model era, someone else trains the base model. You throw a prompt at it, add retrieval-augmented generation, and that is mostly the stack. Applying AI in industry, he says, is much, much easier than it was seven years ago, which is why he felt it was time to commercialize. Guo recognizes the same premise in her own fund: somebody does the bulk of the work in a more general way, so the application layer gets much cheaper. You only do the last few steps.

Voyage builds two components, embeddings and rerankers, aimed at retrieval quality. The reason is what customers kept saying. Implementing RAG is not hard: you connect the pieces and the system is up quickly. The bottleneck is the quality of the answer, and that quality is almost entirely bottlenecked by retrieval. If the language model sees highly relevant documents, it can synthesize a good answer. Even Llama 70B can do that well.

Used as a black box, a language model knows nothing about a company's private information. RAG first retrieves knowledge from inside the firm, then hands that knowledge to the model so it can generate without hallucinating as freely. Once the model sees the relevant passages, it uses them as an anchor to refine the answer, and the hallucination rate drops dramatically. Operationally, you vectorize the knowledge base — documents, video, code, everything — store those vectors as indices in a vector database, and search with the query's vector.

The customer map is wide. Voyage has a chemistry company using RAG on its own documents and product descriptions, plus finance, legal, code retrieval, and code generation. Ma thinks it applies to almost any case, including personal ones: a RAG system on a phone so you can reach your own history, or a laptop search that does not demand the exact filename. Semantic search, in that setting, means you describe the contents and the system finds the file. Guo notes that the average enterprise technology leader had barely heard the term a year earlier. The architecture is young and already in an argument.

Last year's argument was RAG versus fine-tuning, and Ma thinks it has mostly settled. RAG is much easier; fine-tuning often needs a lot of data before you see a result, and hallucinations can survive the tune. The live debate is RAG versus agent chaining, or versus long context — putting every proprietary document into a series of instructed models, or into one window, instead of vectorizing anything. He wants to take long context first, and he wants to debate the future version, because long context is not practical yet.

Dumping a billion tokens of private data into a transformer, at today's prices, is "just impossible" — on the order of five to ten magnitudes more expensive, depending on the corpus. Caching activations from the internal operations of those documents would cut the bill a lot, but the intermediate computations for the whole context still have to live in GPU or CPU memory, and the theoretical gap versus RAG remains. People will say everything gets cheaper. Embeddings are neural nets too; GPUs get cheaper and the networks get smaller, so RAG cheapens in parallel. His prediction is that RAG stays much cheaper than long context. The relative cost structure does not flip.

Guo puts a number on the window: Gemini 1.5 Pro's million tokens is roughly five books, or 25,000 to 30,000 lines of code, and not much video or audio. Many companies, Ma says, hold 100 million tokens — a 100× gap. That is the difference between a bill that sounds like $100,000 and one that sounds like $10 million. Ten million is unacceptable; a hundred thousand is livable. From first principles he treats context as short-term memory and RAG as long-term memory. Why haul the entire library into working memory to answer one question? The efficient move is to retrieve a subset every time. Computer architecture already learned this: disk, memory, and CPU cache are a hierarchy because that is how you get efficiency.

Embeddings and rerankers are, in his phrasing, the models that manage a language model's long-term memory. The technology will evolve; in two years Voyage or someone else may ship a new memory design that extends embedding models rather than replacing them. Agent chaining, meanwhile, is "a bit less well defined" and mostly orthogonal. You still use embeddings in the chain, you still do iterative retrieval, you still call a large model. He would rephrase agent chaining as an iterative, multi-step retrieval-augmented system. Guo notes that the original motive for chaining was the same efficiency motive as RAG. Ma agrees — and warns that if you let a very large model manage the knowledge system, you give the efficiency back. Knowledge should sit with a small model, and at that point embeddings may be the right tool. Iterative retrieval is useful now because embedding models still have headroom and are not clever enough; as they get smarter, one or two rounds should suffice, and extra hops will matter less.

Assume RAG is the dominant architecture for large proprietary data that has to be reliable. How do you improve it? One path is prompting — tell the model to abstain if the retrieved documents have nothing relevant. Models follow instructions so well now that you barely need that trick. The bottleneck, in Ma's view and in what users report, is retrieval. You can attack it with software engineering: chunking, metadata, iterative retrieval, heuristics. Or you can attack the networks themselves, the embeddings and the rerankers. Voyage chose the networks. That work is data-heavy, trained on trillions of tokens and then specialized, and it is the kind of lifting a vendor should do rather than every end user.

His longer bet is that cleverer networks thin out the software layer on top. Chunking is already less necessary: relatively long-context embedding models, in the 10K to 16K range, can take a 50-page PDF whole, so splitting documents into 5K or 12K pieces matters less. The same pattern should hit other glue. Today people turn images into text descriptions and then embed the text. When embedding models are multimodal and smarter, that conversion step should disappear.

Domain models are how Voyage spends a scarce parameter budget. They train a general-purpose embedding model from scratch, then continue training on domain data: two trillion tokens of code for a code embedder, one trillion legal tokens for a legal embedder. No customer private data goes into those models, so anyone can use them. They excel in one domain and barely move in others. Latency is the constraint. A query must be embedded at inference time and compared in the vector database, and application budgets are maybe one second, often 200 milliseconds, sometimes 50. That makes more than about 10 billion parameters unrealistic. With limited weights, customization is how you put them on the right task. Domain fine-tuning has shown 5 to 20 percent gains: 15 to 20 percent on code, where there is more data and the job needs algorithmic understanding, and 5 to 15 percent on legal, where the baseline is already stronger. Fine-tuning further on one company's private data adds another 10 to 20 percent on top. Absolute headroom still matters. Start at 50 percent accuracy and you have 50 points to gain; start at 90 and you have 10.

Vector dimension is the underrated latency knob. Searching a 100-dimension embedding is much faster than searching a 1,000-dimension one. Voyage's embeddings are about 3× to 4× smaller in dimension than some competitors, which is part of how they promise low latency. When should a builder start investing in this? Ma's default is day one, once a prototype exists. Profile latency and quality. Evaluate retrieval on its own — how often the returned documents are actually relevant; the industry has standard methods — and evaluate the end-to-end answers. Compare the two, see the bottleneck, then swap components: a stronger embedding, a reranker, even a different language model. Do not start by piling on prompt tricks.

Asked to forecast the stack as GPT-class, Claude, Mixtral, and Llama models keep improving, Ma predicts simpler systems — a biased view, he admits, and also what Voyage is building toward. The ideal is three components, maybe four: a large model, a vector database, an embedding model, and perhaps a reranker to refine what came back. Connect them. Do not worry about chunking, modality conversion, or cleaning formats, because the networks absorb most of that. Seven years ago you had to hand a language model a very clean format. Talk to GPT-4 today and you can have typos, weird layouts, even dump a JSON file at it. Embeddings, he thinks, will travel the same road. The future of AI, in this telling, is a very simple software-engineering layer on top of a few strong neural components.

Founding a company, he found, is very different from research in big tech and only a little closer to running a university lab — where he is already CEO, CTO, CFO, and HR, just at a smaller scale. The most useful advice came from an angel investor: read the books, even the ones experienced founders find basic. He has only read a few chapters of Elad Gil's book on scaling from ten people to a thousand, because Voyage is about ten people now. Talking to angels and to lead investors, including Guo, is how he cuts unforced errors. For him the game is reducing the number of mistakes and correcting them fast. Fixing an error a week after you make it, versus a month later, is a 4× efficiency difference. Guo notes that even the founder lesson comes out as a multiple, which is consistent with the research style.

On academia's role in an age of scaling, Ma is willing to be unfashionable. If the only question is how to scale the system, the incentives are wrong and the capital is not there. He cites Sam Altman's argument that a nonprofit cannot raise enough to scale, and he mostly agrees, which is why university labs struggle to do large-scale research. They should work on different questions, on a smaller scale, with a three-to-five-year horizon, rather than helping OpenAI improve GPT-5 on the current plot. That is why the lab still works on optimizers: Adam is ten years old, and a 5× to 10× efficiency gain in five years would change the landscape — $100 million of GPT-5 training becoming $10 million. Reasoning is the other thread, because it is unclear whether scaling laws get you to the Riemann hypothesis or any hard conjecture. Superhuman math, he says, is hard to believe if the only data is Common Crawl. "If you train on just the Common Crawl data on the web, can you be a good mathematician? It's kind of very hard to believe." Guo calls that an open-minded place to stop: there is still something to figure out.