Building RAG: from document chunks to agents that search

Diyi Yang asks students to build an agent around a question they care about. Before choosing a project, identify the task, research question, available data or APIs, agent framework and evaluation. Topics can range from retrieval and tools to memory, safety and multi-agent systems.

A demonstration that succeeds once is not enough. Yang asks students to define success, data sources and grading, then evaluate across multiple tasks or users. Possible methods include model-based judging with rubrics, human ratings and feasibility checks; comparisons and failure analysis matter too.

Suggested project questions include agent cost and latency, adaptation to new tools or interfaces, prompt-injection robustness, tool misuse, human-agent teamwork and reward hacking. Possible applications include a syllabus-to-calendar assistant, research discovery or a coding agent specialized to a particular codebase.

Concrete examples include extracting deadlines from syllabi, optimizing course schedules and discovering research papers. A narrow coding agent and an evaluation of generated figures or artifacts are other possible projects.

Check that the data, compute, agent framework and evaluation are actually available. Yang recommends starting early, obtaining data first and formulating a well-defined question. The results should teach the reader something, not merely demonstrate a working system.

Start early and work throughout the term, Yang says. A project should state its question clearly and communicate what the results teach, including cases where the system fails.

Retrieval-augmented generation, or RAG, supplies relevant information to a model at the moment it needs it. Yang frames the lesson around what to index, how to search, where the pipeline breaks and how retrieval fits into an agent.

For example, a model asked when office hours are least crowded might confidently guess Wednesday. Given queue logs, it can instead cite the observed counts and recommend the quieter Thursday session. Both answers may be fluent; only the second is grounded in checkable evidence.

RAG treats knowledge as something to fetch dynamically instead of requiring a model to memorize everything. When facts change, an operator can update the document index rather than changing model weights. Returned sources can also give people a way to verify the answer.

Earlier open-domain question-answering systems separated a retriever from a reader. The retriever narrowed millions of Wikipedia articles to a small candidate set; the reader examined those passages and extracted an answer. Both stages matter: the reader cannot use evidence that never reaches it.

Formally, a retriever selects the top K passages for a question from a large collection. Earlier systems often used fixed TF-IDF retrieval with a neural reading-comprehension model; a modern language model can serve as the reader without task-specific training.

In the original RAG formulation, a query encoder maps the question to a vector, which is compared with an indexed set of document vectors. The highest-ranked documents condition a generator, and the model combines probabilities across candidate documents to produce an answer.

Yang describes joint training of the generator and query encoder while the large document index remains fixed. This is the original paper’s architecture, rather than a requirement that every modern application train its components together.

The process can be factored into retrieving a document given the query and generating an answer given both. In RAG-Sequence, a document conditions an entire answer; RAG-Token allows the contribution of retrieved documents to vary across generated tokens.

RAG-Sequence and RAG-Token differ in how the model uses retrieved passages over an answer. The first associates a whole sequence with a passage when computing its likelihood; the second can combine evidence differently at each token.

The training objective rewards the model for assigning probability to the correct answer. The original dense passage retriever uses separate BERT-based encoders for queries and documents and scores their vectors with an inner product.

Document vectors can be computed offline and stored in the index. At query time, only the question needs encoding before a maximum-inner-product search retrieves candidates. A generator such as BART receives the question and passage as text and writes the answer token by token.

The passage encoder and the question encoder are separate. Since the collection is much larger than the stream of current questions, offline passage encoding is an important part of keeping retrieval practical.

Retrieved passages become text supplied to a generator. Yang notes that the same idea transfers to modern language models through prompts, even when an application does not reproduce the original training setup.

Yang shows early results in question answering and human evaluations of factuality and specificity. She cautions that these are historical results and current models score higher; the point is that retrieved content can improve grounding.

Yang contrasts earlier neural readers and generators in human evaluations. Retrieved evidence helps explain why a response can be more specific or factual, but the reader still has to use the evidence correctly.

RAG systems have become more modular. Before retrieval, they may route, rewrite or expand a query. After retrieval, they can combine, rerank or summarize results; more complex workflows can repeat retrieval and reading instead of executing a single fixed pass.

For example, query rewriting can make an ambiguous question easier to search, while reranking reorders candidates after an initial search. The modules can be combined into different processing patterns rather than one mandatory pipeline.

A student asks about speed, memory and cost, not just answer accuracy. Yang says preprocessing document representations offline can lower query-time work, while models that jointly attend to each query and document cannot precompute the entire pair.

A student asks whether frontier models still need explicit RAG. Yang says it remains useful for private company databases or information outside a model’s training data. Some products already provide general web search, but application-specific data may still need its own retrieval system.

Yang also answers a question about BERT: it illustrates the original encoder architecture, but the field has moved on and many other representations are possible. The retrieval principle does not depend on a particular historical encoder.

Indexing begins by choosing retrieval units and representations. A contextual word embedding can distinguish a financial bank from another meaning of bank; sentence and passage encoders represent larger units. Multimodal and graph encoders can represent images, audio, nodes or relationships as well.

Representations need not be restricted to text. Yang gives CLIP as an example of bringing image and text representations into a shared space, and mentions graph encoders that represent nodes or their relationships.

Once units are embedded, systems need a similarity measure such as cosine similarity or an inner product. Chunking is equally important: an embedding cannot recover context that the chosen text unit does not contain.

Fixed-size chunks might contain 100 to 500 tokens. Small chunks may lack context, while large chunks can mix several topics into one vector. Overlapping neighboring chunks by roughly 10–20% can preserve information that crosses a boundary.

Other choices include splitting by sentence, paragraph or heading, adding metadata and summaries, or using a language model to identify meaningful boundaries. These change the information available in each retrieval unit.

Yang illustrates the consequences with a course-grading example, cautioning that the deadline shown on her slide is not current. Fixed windows, overlapping windows and heading-based chunks would each give the retriever different pieces of the same source.

Consider a query about ACME’s second-quarter 2023 revenue growth. The correct passage may report a percentage but say only “the company,” omitting the name and period because they appeared earlier. By itself, that chunk is harder for both keyword and vector search to match.

Contextual retrieval asks a language model to read the document and supply a short description situating each chunk, then prepends that description before indexing. The enriched text can be indexed both as an embedding and with BM25, giving semantic and exact-term matching access to the missing context.

Generating context costs an additional model call per chunk. Yang presents reported examples where contextual embeddings reduced failed retrieval at 20 from 5.7% to 3.7%, and a hybrid setup from 5% to 2.9%; those are results from the illustrated evaluation, not guarantees for every corpus.

The prompt for contextual retrieval gives the model both the whole document and the selected chunk, asking for a short description that locates the chunk in that document. The generated description is prepended to the original passage before indexing.

Late chunking addresses the same loss of context differently. Instead of embedding chunks independently, an embedding model first processes the longer document. Token representations are then pooled into chunks, allowing each chunk vector to carry information from its wider context.

In the results Yang shows, late chunking helps especially when chunks are small: a tiny isolated piece otherwise has little context, whereas a larger piece already carries more. The gain depends on the task and document setup.

RAPTOR builds a hierarchy for questions whose answers are spread across a document. It embeds and groups chunks, summarizes them, embeds the summaries and repeats to create a tree. Retrieval can select detailed leaves or higher-level summaries depending on the question.

In the tree, a detailed question can match a leaf, while a broad question can match a summary higher up. Yang uses questions about the theme and ending of Cinderella to illustrate why a single local passage may not contain the whole answer.

GraphRAG takes a cross-document view. At indexing time, a model extracts entities and relationships, which are connected into a graph and grouped into communities. Community summaries can support broad questions; partial answers from relevant communities can then be combined.

The graph approach starts by extracting entities and their relationships from chunks, then finding communities in the resulting network. At query time, community-level summaries can help synthesize themes distributed across many sources.

A student asks whether richer structures hurt very specific questions. Yang explains that contextual chunks, trees and graphs solve progressively broader information needs, but sophisticated indexing has a cost. Simple, fast approaches remain attractive when they cover most questions.

Yang gives public perception of a topic as a question that might require material from ten documents, rather than one exact passage. The purpose of the more elaborate structures is to answer that broader question, with extra indexing and processing cost.

Selecting graph communities is a design choice: a system can estimate relevance and aggregate answers from several groups. Yang says the models used for this work need to be capable but not necessarily frontier-sized; quality, latency and cost should be compared.

When evaluating the language model used in a retrieval pipeline, Yang recommends comparing retrieval quality against the model’s cost and speed. A smaller capable model can be a reasonable choice for an indexing or summarization step.

Another student asks about errors compounding through the pipeline. Yang agrees that a mistaken retrieval can corrupt the final answer. Reflection and critique can add checks, but cascading mistakes remain a practical concern.

Exact vector search compares a query with every indexed vector, which can be expensive across millions of documents. Approximate nearest-neighbor methods sacrifice some recall to search much faster. Clustering and vector quantization are among the available approaches.

One approximate-search method clusters vectors and searches selected nearby clusters. Quantization offers another way to reduce storage or computation by representing vectors with shorter codes.

HNSW organizes vectors into a layered navigable graph. All vectors appear at the bottom, with progressively fewer at higher layers. Search starts at a sparse upper layer, moves toward nearer candidates and descends to denser layers for a more local search.

The index is built by inserting vectors, choosing their top layer randomly and linking neighbors across layers. The graph permits a greedy traversal without scanning the entire corpus; Yang presents it as a practical speed–recall tradeoff rather than exact search.

BM25 is a longstanding keyword retrieval baseline. It combines term frequency with inverse document frequency and adjusts for document length. Terms that distinguish a document from the rest of the collection carry more weight than terms that appear nearly everywhere.

A term’s contribution saturates rather than growing indefinitely with repetition, and length normalization accounts for the fact that long documents naturally contain more words. These design choices make BM25 useful for exact names, codes and numbers.

Inverse document frequency reflects how many documents contain a query word. In the form shown, a term appearing in more than half the documents can have a negative contribution; the aim is to distinguish the query’s informative terms from very common ones.

Dense passage retrieval, or DPR, uses separate neural encoders to place questions and passages in a shared vector space. It scores their inner product, allowing semantically related wording to match even when the exact words differ. Passage representations can be prepared offline.

The term-frequency saturation parameter controls how much repeated mentions add before the score flattens. A small value makes a term’s presence versus absence more important; a larger value gives repeated appearances more weight.

At query time, DPR encodes the question and compares its vector with passage vectors in the index. This moves much of the document-side work offline, though encoding and searching each new query still costs time.

DPR is trained with contrastive loss: increase the score of a relevant question–passage pair relative to negatives. Negatives can come from other examples in the batch or from BM25 results that match the question’s words but do not contain its answer.

A BM25 hard negative is particularly instructive because it shares the query’s words but lacks the answer. Contrasting it with a genuinely relevant passage pushes the model toward more than surface-level overlap.

Hard negatives teach the encoder not to rely only on word overlap. Yang reviews retrieval measures: hit at K asks whether a relevant item appears among the top results; recall at K measures the share of relevant items retrieved.

Yang also discusses a failed-retrieval measure as the complement to finding relevant evidence in the top results. Teams should choose metrics according to whether they care about finding any answer, recovering all relevant material or ordering the best evidence first.

NDCG weights relevant results according to their rank and normalizes against an ideal ordering. Mean reciprocal rank averages the reciprocal position of the first relevant result across queries. Different metrics answer different questions about a ranked list.

In results shown from DPR research, dense retrieval outperformed BM25 on four of five datasets, and the lecture highlights gains with relatively small numbers of labeled examples. Yang treats these as results from the cited experiments, not a universal rule for all corpora.

The displayed DPR experiments compare training on individual datasets and on a combination of datasets. Yang notes that adding training examples improved the reported accuracy, and that even a modest labeled set helped in those particular comparisons.

The two methods fail differently: BM25 can miss paraphrases, while dense search can miss exact product IDs, codes or numbers. Hybrid retrieval runs both and combines their rankings, so each can recover candidates the other misses.

Reciprocal rank fusion merges rank positions rather than directly adding raw scores that may be on incompatible scales. It can combine rankings from multiple retrievers without tuning each corpus’s scores. A student asks how stored passage representations are found; Yang explains the index maps back to the offline-computed embeddings.

Yang distinguishes bi-encoders from cross-encoders. A bi-encoder processes a query and passage separately, making it possible to precompute the many document representations. A cross-encoder processes a query and candidate passage together, enabling detailed attention across the pair but requiring query-time computation for each pair.

With a cross-encoder, the query and passage can attend to one another before the relevance score is computed. That interaction carries a cost: a new query creates a new pair, so the fully joint representation cannot be prepared for every possible query in advance.

The class compares architectures by asking what can be precomputed, how much interaction is preserved and what happens to latency. Joint processing can improve detailed matching, but it is expensive across a large collection; it is often applied to a smaller candidate set as a reranker.

ColBERT takes a middle path called late interaction. Query and document tokens are encoded independently, preserving a vector for each token. At scoring time, each query token finds its most similar document token; the maximum similarities are summed across query tokens.

ColBERT’s MaxSim operation finds the best document-token match for each query token and adds those similarities. It keeps more token-level detail than a single-vector representation while retaining separately computable document encodings.

Because document token representations are independent of a later query, they can be computed offline. Yang shows published comparisons where ColBERT performs well, while emphasizing that storage, latency and the characteristics of the application all affect the choice.

Conventional retrieval pipelines execute a fixed procedure. Agentic RAG makes retrieval an action: the system can decide whether to answer directly, what to search for and whether the retrieved evidence is sufficient to stop.

Those decisions are distinct: a simple factual request might need no search, while another might require a reformulated query and several retrieval rounds. The agent must also judge whether it has enough support to answer.

Self-RAG trains a generator to retrieve on demand and critique its own outputs using special reflection tokens. It can generate candidate segments from retrieved passages, assess relevance and support, then rank the results. The approach requires a training pipeline and critique-labeled data.

In the Self-RAG example, a retrieval marker prompts the system to obtain evidence when needed. It creates candidate answer segments for retrieved passages and uses relevance and support judgments to decide which candidate to keep.

Corrective RAG, or CRAG, instead places a separate evaluator between retriever and generator. It grades retrieved documents: retain and refine good evidence, search again when evidence is poor, or combine actions when uncertain. This can be added without retraining the generator itself.

CRAG’s evaluator maps confidence into a corrective action. It can refine useful retrieved material, discard poor material and search again, or use both approaches for uncertain evidence; the generator itself does not have to be changed.

Search-R1 learns a search-and-reason loop using reinforcement learning rather than requiring a hand-written plan. The model can think, issue a query, read returned information and search again; Yang shows an example involving three refined searches.

The illustrated Search-R1 trajectory alternates reasoning and retrieval, changing its query as new information arrives. The goal is to learn the process rather than specify every step in advance.

During a training rollout, emitted queries invoke a search engine and returned passages enter the model’s context. The reward described in the lecture is based on the final answer’s correctness. Yang presents the reported comparison with other search approaches as evidence for the method.

RAG can fail at several layers. Retrieval may miss the right evidence, use a stale index or access material it should not. A reader may overlook evidence in a long context or face conflicting sources; repeated retrieval and generation can compound earlier errors.

At the reading stage, long contexts can lead a model to overlook a passage in the middle, and two retrieved sources may disagree. At the loop level, a bad early search or interpretation can influence subsequent queries and compound the error.

Search results can also contain prompt injection: an attacker may place instructions in retrieved material to influence the agent. Yang closes by emphasizing that retrieval is a useful design principle, but the full pipeline must be evaluated for these failure modes.