How AI agents find information and take action

A language model can answer from patterns learned during training, but its knowledge has a cutoff and it does not automatically know a company's private documents. Afshine Amidi starts with those limits because useful assistants must work with information that was unavailable when the model was trained.

Putting every relevant document in the prompt is one possible response, but context is finite and longer prompts can cost more. A larger context window also does not guarantee that a model will use every piece of information reliably: the information has to be found and interpreted in a much larger input.

Amidi describes the needle-in-a-haystack test: place an answer in a long body of text, then ask the model to retrieve it. Such a test can probe whether information at different positions is accessible, although a real task may require more than finding an isolated fact.

A separate limitation is action. An ordinary model predicts text; by itself, writing that it sent an email or checked a live system does not actually perform those operations. An application must connect its responses to tools that can obtain information or make changes.

The lecture builds toward agents in stages. First, a retrieval system selects useful external information for an answer. Then a tool-enabled system lets the model request an operation, receive the result and decide what to do next.

The distinction matters for private or changing knowledge. A company's documentation can be updated independently of the model, but the model needs some way to locate the relevant part at question time instead of relying on what happened to be present during training.

Retrieval-augmented generation, or RAG, answers that need by adding a search step before generation. Given a question, the system retrieves relevant material, puts it alongside the question, and asks the model to answer using that material.

The initial corpus might contain long documents, so the system divides them into smaller chunks. Chunk size is a trade-off: an isolated fragment can lose necessary context, while a large fragment brings along irrelevant material and consumes more of the model's input budget.

Chunks can overlap so a fact or sentence near a boundary is less likely to be split into unusable pieces. That overlap also duplicates text, so it is a design choice rather than a free improvement.

To search by meaning, an embedding model converts a chunk into a vector, a list of numbers representing its content. It does the same for the user's query, and a similarity measure helps identify chunks with representations close to the query.

Retrieval typically happens in stages. A first pass gathers a manageable set of plausible candidates and tries not to miss relevant results; a second pass can afford a more expensive comparison to place the best candidates near the top.

The generator sees the selected results as context, not as an update to its model weights. Good answers therefore depend on both stages: retrieving useful evidence and getting the language model to use it appropriately.

An audience question raises the size of chunks and their relation to the original document. Amidi notes that the right division depends on the material and the task: a passage should be large enough to preserve meaning without burying the answer among unrelated content.

A vector index makes it practical to search many precomputed document representations. The query still needs to be encoded when it arrives, but the application need not run the full document collection through the embedding model for every question.

In a bi-encoder, the query and each document are encoded separately. The document side can therefore be prepared in advance, and similarity can be calculated quickly when a new query comes in. This speed makes the method useful for the first, broad retrieval pass.

Semantic similarity is not the same as exact matching. If a query contains a person's name, an identifier or a distinctive technical term, a vector search may miss the exact string that a lexical search would find.

BM25 is an established lexical ranking method: it scores documents according to the terms in the query and how informative those terms are. It gives the system another route to evidence when exact words matter more than broad similarity.

Hybrid search combines lexical and semantic signals. The goal is to benefit from both kinds of match: a document can be relevant because it uses the same rare identifier, because it expresses the same idea in different words, or both.

The way a question is written can itself make retrieval harder. A short question and a long answer may look unlike one another to an embedding model, even when the answer contains exactly what the user wants.

One proposed technique, hypothetical document embeddings, asks a model to draft a possible answer first and uses that hypothetical document to search. The draft is a search aid, not evidence: the system still needs to retrieve actual supporting material.

Query decomposition takes another approach. A complex request is broken into smaller searches so that each part can retrieve evidence on its own. The system then has to bring those results together to answer the original question.

Chunks can also be given a short description of their place in the source document before indexing. This contextual prefix helps a fragment retain information that was lost when it was cut from a longer document.

Generating those contextual descriptions repeatedly can be costly. Amidi points to caching the fixed portion of a prompt as a way to avoid paying for the same long document context over and over when creating descriptions for many chunks.

A cross-encoder provides a more detailed comparison after the candidate set has been narrowed. Instead of encoding the question and each document independently, it processes the query and a candidate together and produces a relevance score.

The joint input lets tokens in the query interact with tokens in the candidate. That makes it possible to evaluate relationships within the pair that a comparison of two independently produced vectors can miss.

Amidi uses a clothing example to show why the relationship matters: matching the presence of colors and clothing terms is not necessarily enough if the question asks which item was red. The model needs to connect the color to the right object.

The price of that closer comparison is computation. Running a cross-encoder against every document in a large collection would be expensive, so a fast retriever first proposes a much smaller set and the cross-encoder reranks those candidates.

The two-stage design makes recall in the first stage consequential. If the relevant document never becomes a candidate, the reranker cannot rescue it. Conversely, passing more candidates can increase coverage while raising the cost of reranking.

A retrieval system should be measured, not judged by a few plausible-looking answers. Amidi introduces ranking metrics that compare returned results with relevance labels, including normalized discounted cumulative gain, mean reciprocal rank, precision and recall.

Discounted cumulative gain rewards useful results near the top of the list: a highly relevant first result is worth more than the same result buried farther down. The normalized version compares the ranking with an ideal ordering, making results easier to compare across queries.

Mean reciprocal rank asks where the first relevant result appears. If it is first, the reciprocal rank is higher than if it appears several positions later; averaging across questions gives a summary of how quickly the system tends to find an answer.

Precision asks what fraction of returned results are relevant; recall asks what fraction of the relevant material was retrieved. Raising the number of candidates can help recall but can also introduce more irrelevant results, so the useful balance depends on what comes next.

Metrics need a representative set of questions and relevance judgments. A retrieval configuration that looks good on one example may fail on the names, terminology or document types that matter to a particular application.

For code, exact matching remains useful. A command-line text search can find a known symbol or error message directly, while semantic retrieval may help when the user can describe a behavior but does not know which words the code uses.

A vector database can help organize and retrieve candidate embeddings, but it is only one component of the system. Chunking, the embedding model, the query, reranking and evaluation all influence the evidence that ultimately reaches the language model.

Shervine Amidi takes over with a limitation of a fixed document collection: a question about a current condition or a user's location may require information that has to be obtained at the time of the request. This is where tools extend what the model can do.

In a tool-calling setup, the model selects a tool and supplies arguments in a structured form. The surrounding application runs the actual operation, sends the result back to the model, and allows it to use that result in its next response.

The distinction between suggesting and executing an operation is important. The model generates the request; application code talks to the external service. The returned value is then added to the conversation so the model can reason from something the tool actually provided.

A live lookup, for example, can use a location supplied or resolved by the application to request a current result. A static document index alone cannot guarantee that the answer reflects a value that changes from moment to moment.

Tools also include search, a calculator and code execution. Some only read information, while others can cause effects outside the conversation, such as writing a file or invoking an operation on another system.

The model needs to know what each tool is for and what arguments it accepts. Clear names, descriptions and input and output formats make it more likely to pick the intended operation and construct a request the application can execute.

Tool use can be learned as a general pattern rather than training a separate model for every future function. At inference time the available functions and their descriptions are supplied as context, and the model chooses among them for the current request.

That makes the quality of a tool description a practical engineering variable. If a model repeatedly chooses the wrong tool or passes the wrong field, collect representative examples and revise the description or interface against those cases.

A structured interface also gives the application a place to validate arguments and handle failures. A requested call can be malformed or return an error; the surrounding system has to expose enough of that result for the model to continue appropriately.

Selecting a tool is more difficult when several functions have overlapping purposes. Similar descriptions can leave the model choosing between operations that seem plausible but have different effects.

Amidi uses the ambiguity of a request involving a bear to illustrate why words alone may not determine the desired operation. The application needs to distinguish the user's actual intent and the relevant context before acting, especially when a mistaken action has consequences.

One approach is to expose only the subset of tools likely to be useful for the current task. Conversation history, remembered preferences and other context can help choose that subset rather than presenting the model with every tool the application owns.

A smaller model or routing step can help select candidate tools before the main model receives them. This reduces the amount of tool documentation in context, but the routing step must not filter out a function needed to complete the task.

A large menu has two costs: it consumes input space and can make similar tools harder to distinguish. Better grouping and clear descriptions can improve selection without asking the model to consider a long, undifferentiated list on every turn.

The model still needs a way to obtain missing parameters. If a request does not contain enough information to safely choose an operation or fill its arguments, asking a follow-up can be preferable to silently guessing.

Model Context Protocol, or MCP, offers a common way for an application to connect a model-facing client to capabilities exposed by servers. Rather than inventing a bespoke integration for every pairing, the protocol gives the parties a shared interface.

An MCP server can expose tools the client may call, resources it can access, and reusable prompts. A client discovers the available capabilities and passes the relevant definitions into the model's working context.

The protocol does not make the model itself execute external code. The application and server still carry out the operation and return a result; the model uses the descriptions and results as part of its reasoning and interaction.

Tool definitions and other repeated context create an opportunity for prompt caching. When a provider supports it, reusing an unchanged prefix can reduce repeated processing of material that would otherwise be sent on every model call.

The order and stability of the prompt matter for that reuse. A fixed tool definition is a better candidate for a reusable prefix than a current timestamp or a rapidly changing message placed early in the input.

Cache lifetime involves a trade-off. A longer time to live can benefit repeated requests, while cached material eventually needs to be refreshed when it changes; the best choice depends on the application's traffic and how stable the content is.

Caching saves work only for the matching portion of the prompt. Changes in tool descriptions, order or other early context can reduce reuse, so an application should think about its prompt layout as well as its tools.

The harness is everything around the model that makes it useful for a task: the tools it can call, the instructions and context it receives, and the workflow that decides what happens after an action. Amidi compares the model to an engine that still needs the rest of a vehicle.

ReAct, short for reason and act, structures an agent as a loop. It considers the current situation, chooses an action, observes the result and uses that observation when deciding on the next action.

A thermostat offers a simple illustration: observe the temperature, compare it with a goal and take an action, then observe again. For a model-based agent, an action may instead be a search, a file edit or another tool call.

The loop gives an agent a way to respond to results it could not know in advance. It can search, inspect what came back, and decide whether another search or a different operation is needed rather than committing to an entire plan before receiving feedback.

When separate agents need to communicate, agent-to-agent protocols aim to standardize that exchange. The surrounding system still needs to decide which agent has which responsibility and what information to pass between them.

Long-running loops accumulate context. Compaction reduces the amount carried forward, for example by dropping transient tool output, keeping a recent window, or creating a structured summary of earlier work.

Each method discards something. A rolling window is straightforward but can lose an important early decision; a summary can preserve selected facts while omitting details that later turn out to matter. The right strategy depends on what the agent must remember.

Amidi distinguishes the immediate context from longer-term memory. An agent can store useful information outside the active conversation and retrieve it when needed, instead of keeping every past message in every prompt.

A memory system needs choices about what to save and when to retrieve it. The agent should preserve information likely to help future work, but retrieval and summaries can also introduce irrelevant or incomplete context.

Coding agents bring these ideas together. A coding agent reads instructions and source files, searches the repository, proposes or edits code, runs tools and examines the results. Its abilities depend on both the underlying model and the harness that makes those tools available.

Because coding agents interact with an execution environment, Amidi says, actions need safety controls. Earlier approval systems asked users to allow or reject many individual actions; he describes automatic approval approaches in which a separate, lighter model evaluates a proposed action.

A plain text-search command such as grep can be a useful coding tool because an exact symbol or error message often provides a direct route into the code. Amidi says coding agents frequently use this fast, precise retrieval instead of traditional semantic RAG.

Repository instruction files such as AGENTS.md supply persistent local rules and preferences for each session. This gives users a place to specify how they want the coding agent to work in a particular environment.

Skills package repeatable workflows as instructions, potentially alongside scripts and other resources. Amidi suggests completing a workflow end to end and asking the coding agent to turn it into a reusable skill.

Amidi mentions Pi as an example of a minimal coding harness. By keeping the surrounding prompts and workflow lean, it provides a common setting in which to compare how different models perform.

Computer-use agents work through a graphical interface by observing screenshots and issuing actions. Amidi says this is currently inefficient because the agent repeatedly takes a screenshot and predicts the next action.

A subagent can take on a separate piece of work and return its result. Amidi notes that each new agent needs its own context and a complete agent loop, creating costs that some minimal harnesses avoid or limit.

Agent swarms take collaboration further: multiple agents communicate and work toward a shared goal. Amidi points to recent mathematical advances as one use, while noting that systems with many powerful agents can also expose vulnerabilities.

As an example of that security concern, Amidi refers the class to a model evaluation and threat research group’s account of a Hugging Face incident. He closes the lecture by directing students to that reading.