Retrieval-Augmented Generation: External Knowledge, Embeddings and Search

I want to explain retrieval-augmented generation, or RAG, and why it helps applications use external information. Pinecone sponsors this video and is my exclusive channel partner; its vector database is one product used in the workflow I describe, not a prerequisite for RAG.

People often assume fine-tuning is the straightforward way to add documents to a model. I suggest retrieval is frequently a better fit for supplying changing information. My “nine out of ten” estimate is a personal rule of thumb, not a measured result; fine-tuning can do more than change tone and can also affect knowledge and task performance.

RAG retrieves relevant outside material and adds it to the context used to answer a question. It can give an application access to stored information or conversation history, although the model’s weights are not automatically updated and reliable long-term memory requires an actual storage and retrieval design.

A model cannot automatically know everything that happened after its training cutoff. You can put new information directly in a prompt, but sending a large collection repeatedly can be inefficient; retrieval is one way to choose a smaller set of relevant material.

The context window is a token budget for a model’s input and output, with exact limits and accounting depending on the model and interface. I cite roughly 8,000 tokens for the original Llama 3 and 128,000 for GPT-4o as examples from 2024, not current limits or limits for every variant.

Even when the window is large, repeatedly sending extensive context may incur cost and latency. Retrieval also has indexing and search costs, so whether it saves money or time depends on the workload and implementation.

Imagine a customer-service assistant that needs to use previous conversations. Sending the entire history on every turn would eventually consume much of the available context, while only some previous exchanges may matter for the current question. A retrieval system can select relevant records, subject to privacy, retention and access controls.

The same issue arises with private company documents. A general model may not have been trained on them, and putting every document into every prompt is impractical at scale. An authorized retrieval layer can find relevant passages without sending the full collection each time.

Consider a new Tesla earnings report that postdates a model’s training information. The application can index the report, retrieve passages relevant to a question and include them with the question in the prompt. The document must actually be available and indexed; for a quarterly report, the relevant filing may be a 10-Q rather than the annual 10-K mentioned in the example.

With reports from many companies, retrieval aims to select information about the company and period requested rather than repeatedly adding every filing. Retrieval quality and document freshness matter: the closest-looking passage is not always the right or most recent source.

I show Pinecone’s example of a question about a Volvo XC60 manual. The illustration contrasts an unsupported answer with one generated using retrieved documentation. An unsupported response may hallucinate, but it need not always do so; consult the manual for the exact vehicle and model year when dealing with a safety-related feature.

To prepare the illustrated system, documents are sent through an embedding model and stored with metadata in a vector database. In practical implementations, documents are often divided into passages, and the original text or a reference to it must remain available for inclusion in the prompt.

An embedding is a numerical vector representing text. The two-dimensional diagram is a simplification of a higher-dimensional space; an embedding model and similarity metric determine what counts as nearby, and closeness is an imperfect proxy for relevance or truth.

At query time, the system embeds the user’s question, searches for similar indexed passages and sends the selected context with the original question to the language model. The model can then use the passages to formulate an answer, though the retriever or generator can still make mistakes.

The manual example illustrates how grounding might reduce unsupported responses; it does not prove RAG eliminates hallucination or guarantees accurate instructions. Retrieved passages may be incomplete, misleading or outdated, and the model can misinterpret them.

I then show a multi-step example involving Apple Remote. A direct model answer in the illustration is iPod, while the search-based workflow looks up the remote, identifies Front Row and searches for another control method, arriving at keyboard function keys. The steps illustrate a particular example, not a guarantee of general agent reliability.

The reasoning and search steps can be organized in different ways, including one agent making multiple tool calls; each step need not be a separate agent. Such workflows can retrieve and combine sources iteratively, while still requiring verification of the result.

Pinecone stores and searches high-dimensional vectors at scale, and I describe it as fast. Those are sponsor-related product claims: actual latency, cost and capacity depend on configuration and workload. A vector database is one option rather than a requirement for RAG.

I compare vector similarity search with ordinary search. Many search systems use lexical matching, structured queries or hybrid methods as well as embeddings; proximity in vector space is not how all search works and does not itself guarantee relevance.

A basic implementation creates embeddings for documents, stores them with the relevant data, embeds a query and retrieves matching context. Pinecone offers one hosted implementation; teams still need to choose models, handle document updates and permissions, and evaluate retrieval and answers for their use case.

I close by offering to make a more detailed tutorial on preparing documents, embeddings and a Pinecone account, and thank Pinecone for sponsoring the video. The lesson is the retrieval-and-generation workflow, with its actual quality dependent on sources and evaluation.