From RAG to Reasoning: Why the Future of AI Is a System, Not a Model

Speaking one day after OpenAI released o1, Douwe Kiela says the release is exciting because it pushes the field toward test-time compute and toward thinking about systems rather than models. In his reading, o1 compresses chain-of-thought ideas that had circulated for a while into the model through reinforcement learning, turning the model into a more complicated system. He finds it encouraging that this approach works so well for reasoning, while his own team has pursued similar ideas on the retrieval side. The broader lesson is to address the entire system rather than a single next-token predictor.

Kiela cautions that not every future model will follow the same recipe, because test-time thinking increases latency and deployments face different constraints. He notes that o1 looked much stronger on some benchmarks such as math and law, while older models could still be better or faster on other tasks. The release also raises product questions: if some models eventually take hours to answer complex queries, interfaces and user expectations will have to change. Even for humans, he adds, harder problems naturally take longer.

Asked how he evaluates a new model, Kiela says he relies mostly on benchmarks and on a strong team whose judgment he trusts. He follows releases closely because they validate directions his company has been exploring, especially in retrieval. He suspects some o1-style ideas may already exist in other models without being disclosed, but he credits OpenAI for executing well. The point is less novelty than the composition of ideas into a working system.

Kiela says Contextual AI began from a common enterprise frustration: companies were excited about generative AI, but the technology was not ready for prime time, especially in enterprises. He and his co-founders knew RAG would solve part of the problem, but they also believed they could do much better than the original RAG formulation. The original vision behind RAG was more ambitious than what ended up in the paper, and Contextual AI is an attempt to build that larger vision. This distinguishes the company from foundation-model labs such as OpenAI and Anthropic.

The first principle is systems over models. Kiela estimates that a model may be only 10–20% of the larger system an enterprise actually needs, and enterprises are buying the system rather than the model. The second principle is that Contextual AI does not pursue AGI, which he sees as fundamentally a consumer product because no one knows exactly what a consumer will want. In an enterprise, by contrast, the desired behavior is often known and should be specialized rather than general; he cites a bank example where EU rules make general-purpose AI risky for performance reviews.

Contextual AI therefore specializes the parts end to end and focuses on high-value, knowledge-intensive use cases where integration pays off. Kiela says this runs against the usual startup playbook of selling one narrow layer in a stack, because customers often get dragged into assembling many vendors themselves. By controlling retrieval, reranking, generation, post-training, alignment and fine-tuning, the company aims for a compounding advantage in solving the whole problem. That vertical integration is the product.

On the enterprise side, Kiela sees many compelling demos but far fewer production deployments. Demos are often built on roughly 20 PDFs and tuned directly against the test set, so they fail when expanded to 10,000 PDFs. The failure is not only in machine learning; risk, compliance, security and operations also become stumbling blocks. Production, not demo quality, is the bar Contextual AI targets.

Kiela jokes that RAG could have used a catchier name, but the idea came from a long-standing interest in grounding. His PhD focused on grounding language in perceptual information, such as using pictures of cats to better understand the word “cat.” At FAIR, he worked with Ethan Perez on grounding in Wikipedia and built an early RAG prototype that worked, aided by Facebook AI Image Similarity Search, an early vector database archetype. The hard part was back-propagating into retrieval, though most later practitioners would use off-the-shelf retrievers instead.

When Perez had to return to university for visa or university reasons, Kiela teamed up with Patrick Lewis and Sebastian Riedel, who were working on open-domain question answering, the killer use case for RAG. He remembers the collaboration fondly as part of FAIR's golden era. The paper's eventual influence was not obvious at publication, and Kiela is skeptical of anyone who claims otherwise. Scientific history tends to be rewritten around great papers after the fact.

He applies the same skepticism to the Transformer paper. At FAIR, he says, researchers were underwhelmed because they already had RNNs with attention, and cutting recurrence for parallelism seemed less revolutionary than it now sounds. The Transformer's initial machine-translation gains were small and finicky, and similar work using convolutions appeared around the same time without the same credit. Kiela argues that Transformers succeeded because they were optimal for GPUs, not because GPUs followed Transformers, and that the attention mechanism deserves much of the credit.

Kiela describes alignment as central to making systems maximally useful. RLHF was the secret sauce that made ChatGPT work, because supervised fine-tuning alone optimizes next-token prediction and cannot fully capture human preferences at the sequence level. RLHF uses a reward model to propagate sequence-level rewards, but training that reward model is expensive and the model is not used in actual generation. It also depends on preference data, which becomes slower and more expensive as use cases become more specialized.

DPO removed the need for a separate reward model, and Contextual AI's work with Kawin Ethayarajh on KTO removed the need for paired preference data by optimizing directly on feedback, drawing on Kahneman-Tversky prospect theory. Later work with Carl addressed under-specification: a preference pair may say which answer is better without explaining why. CLaRE contrasts revisions so the signal includes the fix, and anchored preference optimization (APO) accounts for how strong the model is relative to the preference data. If the model is better than the data, Kiela says, you want to learn the ranking, not treat the preferred answer as ground truth.

In practice, Contextual AI has its own data annotation team and learns directly from customer thumbs-up and thumbs-down feedback using these algorithms. That direct feedback loop would be difficult with standard RLHF. Kiela adds that much alignment work still happens on the core model during post-training, which is where a lot of the magic occurs after pretraining. The goal is to make the pretrained model good at the specific job the customer needs.

He illustrates specialization with finance: a system does not need to know quantum mechanics, Shakespeare or Japanese haiku if it needs to be excellent at one financial task. Alignment and post-training provide that focus. The combination of an integrated system, custom alignment and specialization is what helps customers clear the production bar and see real ROI.

Kiela's wish list for off-the-shelf components starts with extraction. Language models are only as good as the context they receive, and good retrieval depends on correct extraction from messy sources such as PDFs. He says existing open extraction systems were not good enough, so Contextual AI had to build its own models for that stage. Much of the remaining work is unglamorous engineering needed to make the pipeline generalizable.

Evaluation is the other major gap, especially for enterprises that need to understand deployment risk and real accuracy. Kiela says there is still no standard evaluation approach enterprises can rely on, and he would like to see one emerge. Contextual AI may propose its own ideas, but he views the area as underexplored. A good product process starts by specifying what success means, then hill-climbing in a prototype before productionizing.

The bespoke nature of evaluation often reflects a deeper problem: many customers do not yet know what they want. Kiela's team spends significant time defining success with the customer before building. He worries that many organizations treat evaluation as a spreadsheet with about 50 examples, which has high variance and is not principled. Serious evaluation has to be designed, not improvised.

Still, Kiela sees common architecture across customers. They need to extract information from tens or hundreds of thousands of documents without failing, put it into a retrieval mechanism, contextualize the language model, and tune the components together. Modern deployments use a mixture of retrievers rather than a single retriever. The components are relatively clear, but specialization and knob-tuning between them create much of the value.

Kiela says his job requires changing his mind quickly, and several recent shifts stand out. Synthetic data working so well was unexpected, and agentic workflows with tool use now look more feasible than he would have guessed a year earlier. He also admits he was too dismissive of early chain-of-thought work, which he saw as a gimmick with flawed evaluations. Even if the parts seemed unremarkable, combining chain-of-thought rollouts with reinforcement learning or human/language-model fixes produces a more powerful model at the cost of test-time efficiency.

He is skeptical of the claim that the industry is running out of tokens. Society produces huge amounts of data every day, and the real issue is the trade-off between quality and quantity: lower-quality data requires more of it. For text, there is plenty to train on; for multimodal data, especially video, the surface has barely been scratched. He also criticizes a prominent paper arguing that models cannot learn from their own synthetic data as unrealistic and damaging to funding for promising work.

On reasoning, Kiela pushes back on the idea that models only recently became capable of it. They have been reasoning for a while, and further gains can come from data—for example, high-quality step-by-step reasoning traces from mathematics PhD students. He also describes a fun metalinguistic dataset with Tristan at Stanford, inspired by Douglas Hofstadter's “I Am a Strange Loop,” that tests self-reference and statements like “there are X words in this sentence.” If systems scored highly on that, he would be impressed, because it would show a kind of meta-level reasoning beyond solving math problems.

Kiela remembers FAIR in 2016 as a free, academic environment, though GPUs were always scarce. His early work included emergent communication among multi-agent systems and multimodal grounding, topics he expects to return in force. FAIR became harder to keep free as it grew, but it produced world-changing artifacts such as PyTorch. Kiela thinks FAIR and Mark Zuckerberg do not get enough credit for betting on open source.

He sees Zuckerberg's open-source strategy as pragmatic: if language models become a new platform, Meta does not want to depend on someone else's platform. Giving away PyTorch, React and Llama also improved Meta's reputation and hiring, turning it from an “evil empire” into a cooler player in AI. Kiela does not treat open source as charity; it is platform strategy with real competitive benefits.

Multi-agent systems are also less futuristic than the label suggests. Kiela argues that systems already comprise many models, and synthetic data is itself a form of one agent training another. The interesting research question is the optimization and communication topology: which agents talk to which, and whether they are fully connected or sequential. Different topologies produce different dialects, much like organizational culture, and humans may eventually be agents in those systems too.

On Hugging Face, Kiela calls it the central place for publishing models and a company that exists partly by the grace of open-source efforts such as Meta's. He notes that GitHub and PyTorch had model hubs but did not resource them properly, leaving Hugging Face to become the default. It is an amazing company, he says, but its position depends on the open-source ecosystem it serves.

Kiela tries to be pragmatic about fundraising. He sees many AI companies raising more than they need at valuations they will struggle to justify, especially if the hype cycle cools. Contextual AI plans to come out of that period as a winner, which means not training its own large base models and instead allocating capital to strong people and close customer work. The company can be more capital-efficient because it builds on open and existing models.

He warns that the market still expects a turnkey AI product that can absorb any company's data and work immediately. That is not where the technology is, so Contextual AI combines long-term productization with short-term white-glove support. His analogy is selling a Tesla Roadster: a high-end sports car that is hard to drive and comes with mechanics and engineers to tune it. Meanwhile, the company is building the assembly line for the more turnkey Model S.

On scaling, Kiela argues that the products coming off the assembly line are systems, not just models. Scaling laws matter up to a point, but then post-training, chain-of-thought and distillation back into the model provide many other directions for improvement. In the quickfire round he calls agents both overhyped and underhyped: they do not work reliably yet, but they are showing signs of life. His biggest surprise in building Contextual AI was how fragile infrastructure is, with GPU and node failures requiring constant replacement.

On open versus closed models, Kiela describes a triangle: frontier closed models at the top, open models at the bottom, and the most interesting trade-offs in the middle where cost, usefulness and price meet. He expects a mix, with strong post-training on open models reaching that sweet spot. Outside his own space, he is most excited by AI entertainment and video generation, which is moving faster than he expected and may lead to personalized movies or infinite episodes. If he were not running Contextual AI, he would still be drawn to transforming work, but he finds traditional enterprises such as JPMorgan compelling because of the massive problems their AI leaders can solve.