AI Agent Memory: What to Store, Retrieve and Forget

Michael Ryan compares agent memory architectures and shows why remembering more does not always improve decisions.

MRMichael RyanVortrag · Stanford University
Kanal
Stanford Online
Veröffentlicht
15. Nov. 2026
Video
1:18:54
Lesezeit
15 Min.
Folien
8
Original
Englisch
Auf YouTube öffnen
Beim Scrollen schrumpft das Video in die rechte untere Ecke und läuft weiter. Untertitel folgen Ihrer Lesesprache.
LesespracheIn 2 Sprachen verfügbar
Diesen Artikel gibt es noch nicht auf Deutsch, daher wird das Original (Englisch) angezeigt.
ZUSAMMENFASSUNG

In 30 Sekunden

An agent can have a rule in storage and still fail to use it when it matters. Stanford instructor Michael Ryan distinguishes the context an agent sees from stored experiences, facts and skills, then compares ways to retrieve and update them. MemGPT gives a model tools to manage a hierarchy of memories; Voyager saves successful procedures for reuse. But retrieval can overload a model, memories can go stale or expose private details, and a benchmark that tests action over time may tell a different story from one that tests recall.

  1. 1An agent can lose the instruction that keeps an action safeRyan describes an email assistant that forgot it was supposed to suggest deletions after a long inbox was compacted, then began deleting messages. Memory must carry goals and constraints across long tasks.
  2. 2Stored facts, skills and experiences differ from current contextWorking memory is what the model currently sees; episodic records preserve past interactions, semantic memory holds facts, and procedural memory stores how to act.
  3. 3A bigger context window does not eliminate retrievalRyan says long tasks can exceed even large context windows, and models may struggle to locate important information in very long inputs. Retrieval chooses what should enter the context.
  4. 4Voyager saves successful code as skills for later tasksThe Minecraft agent tests a generated program against the environment, stores successful programs with searchable descriptions and reuses relevant skills for harder goals.
  5. 5MemGPT gives the model tools to manage a memory hierarchyRecent messages, editable working context, searchable conversation history and archival storage have different roles. The model can retrieve and update information instead of keeping everything in the prompt.
  6. 6Better recall does not guarantee better answersRyan describes a benchmark where retrieving more observations increased recall but reduced answer accuracy, because too much context could interfere with using the relevant memory.
  7. 7Task-based evaluation exposes room for memory systems to improveMemoryArena tests whether agents can act over long, changing tasks. In its comparison, memory-agent approaches scored lower than long-context and retrieval baselines; Ryan cautions that failures can have causes beyond memory.

Das Wichtigste

  1. Decide what gets stored, when it is written or retrieved, and how it is searched before choosing a memory architecture.
  2. Keep raw interaction logs when possible: summaries can omit details that later turn out to matter.
  3. Use successful, verified procedures as reusable skills, but plan to update them when the environment changes.
  4. Test whether retrieved information improves the final task, not only whether the right item was found.
  5. Give users control over sensitive memories and consider how poisoned, stale or irrelevant entries affect decisions.
VOLLTEXT

Den bearbeiteten Text nach Kapiteln lesen

Nur das Transkript
01

What agents lose when memory fails

Lost instructions can change an agent’s actions, while cross-session memory can preserve preferences, procedures and progress. Ryan uses concrete examples of both failures and benefits.

An agent may store a policy and still fail to use it. In one example, an agent running a company had generated an employee handbook, but when an employee reported being 45 minutes late, it repeatedly responded that this was fine instead of applying its policy. The memory existed somewhere, but was not reliably accessed.

In another example, a user asked an agent to inspect an inbox and suggest messages to archive or delete. The inbox was so long that the context was compacted; the agent lost the instruction to suggest and started deleting messages. Keeping a goal and its limits available matters for long-running work.

Without memory, an agent can lose procedures, knowledge and progress. A coding assistant might have to rediscover how to launch a GPU job each time, even after the user has already worked out the procedure with it.

Preferences also have to persist across sessions: always ask before deleting emails, for example, or check before spending money on experiments. As a conversation grows, an agent can otherwise lose the initial goal or the focus of a longer project.

Memory supports personalization, learning from experience and stateful work. Ideally, a user can return to a proposal and say “pick up where we left off,” and the agent can find the relevant prior work rather than asking the user to reconstruct it.

Preferences can change. A benchmark example starts with a user who likes Italian food but later develops a gluten allergy and explores Mediterranean cuisine. Remembering only the old preference might lead the assistant to suggest pasta; updating the memory can support a more suitable recommendation.

Agents can also reuse procedures. If a system learns to find a location by name, it may reuse that step when it later plans a route between places. Another example retrieves a previous conversation about a flight reservation when the user asks for an update.

02

Four kinds of memory and where they live

Working, episodic, semantic and procedural memory serve different purposes. The same categories can live in model weights or be retrieved into the context window.

The CoALA framework draws an analogy to human cognition and separates working, episodic, semantic and procedural memory. These categories help us reason about agent design, although their boundaries are not always sharp.

Working memory is essentially the context window: the observations, user input, goals and reasoning that the model currently sees. Other memories are stored elsewhere and can be retrieved and added to that context when needed.

Four memory types in CoALA
FOLIE 1 / 8Four memory types in CoALAThe diagram separates working context from procedural, semantic and episodic memory.

Episodic memory records what happened: a sequence of actions, tool calls and observations, or prior conversations. In a shopping task, the path through product pages and price comparisons is an episode. Raw traces can later be searched for details that a summary did not preserve.

Semantic memory stores facts and knowledge, including user preferences and facts about the world. A chatbot might remember a user’s child likes jellyfish; a lookup against an encyclopedia is another way to retrieve factual knowledge.

Procedural memory is knowledge of how to do something: an agent’s skills and rules for action. A skill file might specify a company’s brand colors and explain how to apply them to a presentation. Once retrieved into context, it can guide the agent through the task.

A student asks why retain episodic memory if other memories are abstractions derived from it. Converting an interaction into a compact statement is lossy: a system can choose the wrong detail to retain. Raw records let it search again, and examples can sometimes teach a procedure better than an abstract description.

Does a million-token context window make separate memory unnecessary? Retrieval-augmented generation is often part of a memory system, not an alternative to one. Long-running tasks can exceed large windows, and models can struggle to find a relevant detail within a very long context, so selecting information remains useful.

The implementation can also be domain-specific. Ryan notes that the content to keep and the way to retrieve it depend on the system and its tasks. Short-term versus long-term memory is a separate distinction from the functional categories of episodes, facts and procedures.

Another distinction is where information lives. Parametric memory is encoded in model weights: pretraining gives a model factual knowledge, while training can also make it better at following procedures. Context memory is retrieved from outside and placed into the window the model is currently using.

A student asks what happens when the two conflict. Context can often override information in the parameters, Ryan says, but the outcome is not guaranteed, particularly if a behavior is strongly fine-tuned. For a small set of changing facts, retrieval is usually practical; training a model to memorize facts requires much more data and can be lossy.

For a procedural task, updating the underlying model can be more attractive if an appropriate training environment is available. Ryan’s comparison is not a universal rule: the decision depends on the task, the data and whether the information can be maintained externally.

03

Choose what, when and how to remember

Memory design involves content, triggers and storage. Ryan compares files, databases and knowledge graphs and their retrieval trade-offs.

A memory architecture must answer three questions: what is stored and retrieved, when storage or retrieval occurs, and how the information is organized and found. The answers can differ even for systems that all claim to have “memory.”

The stored content can be facts, procedures or conversation traces. Retrieval might be a tool call chosen by the agent, a background worker observing the conversation, or an event such as a context window becoming too full.

Files are one straightforward storage option. An agent can keep memories in directories and use familiar shell commands to list, search, read and write them. A skill file may blend factual details with instructions; a separate directory can hold raw conversation traces.

Databases are another option. A vector database can search for semantically similar information, while a relational database can filter structured metadata. Exact-text methods such as BM25 can help when the wording itself matters; the same retrieval choices used in RAG apply here.

Zep knowledge-graph memory
FOLIE 2 / 8Zep knowledge-graph memoryEpisodes, entities and relationships, and community summaries form three levels of memory.

Knowledge graphs organize entities and their relationships. In Zep, episodes feed an entity graph, and related entities can be grouped into communities with summaries. A conversation about a cousin and a dog, for example, can yield entities for each and a relationship between them.

Summaries of graph communities can guide retrieval to relevant entities. A designer can choose which kinds of entities and relationships matter for a domain; traversing graph links can also shift some of the reasoning into the lookup process rather than leaving it all to the model.

Knowledge graphs are typically used for semantic memory, although procedures could also be represented as linked steps. Asked which storage method is best, Ryan says it depends on the application and its latency requirements.

His starting preference, when it suffices, is a file system the agent can manage itself. It may incur latency as the agent runs commands and reasons through results; other stores become useful when a system needs faster or more specialized lookup.

The MemGPT design illustrates that one system can combine retrieval methods: it embeds documents for search but keeps conversation history available for exact text lookup. Neither a vector index nor a knowledge graph is inherently the answer for every kind of memory.

04

Learning procedures from successful experience

Voyager stores tested Minecraft programs for reuse, while Agent Workflow Memory generalizes successful trajectories. Skill files provide a simpler file-based procedural memory.

Ryan groups the architectures he has selected into four patterns: learning procedures from experience, a tiered memory hierarchy, reflective consolidation, and memory maintenance. He presents these as a useful lens rather than an official taxonomy.

Voyager is a Minecraft agent that learns by exploring. A curriculum proposes tasks such as mining a log; the agent writes code to act in the game, runs it and observes the result. If the attempt fails, feedback helps it revise the program.

When a program succeeds, Voyager saves it in a skill library. The stored code is paired with a natural-language description and an embedding. A later task can search for similar descriptions and reuse the matching programs rather than starting over.

Voyager’s skill library
FOLIE 3 / 8Voyager’s skill librarySuccessful programs are stored with embedded descriptions and retrieved for future tasks.

For example, a new task to craft an iron pickaxe can retrieve related skills such as smelting iron or making a crafting table. This is procedural memory: successful ways of acting are retained and returned to the context when useful.

A student points out that a search can return several similar skills. Ryan says duplicate names are prevented, but near-duplicates can still accumulate under slightly different names. Similarity-based retrieval does not by itself solve the problem of maintaining a clean library.

In the results Ryan presents, Voyager with the skill library reached diamond tools in 102 prompting iterations, while the version without it reached only iron tools. Reusing skills also reduced the attempts needed for individual tasks in the comparison.

Agent Workflow Memory applies a related idea beyond Minecraft. A computer-use agent produces action traces for tasks such as finding an order. Successful trajectories are analyzed for common procedures, which are written back into memory for use on later tasks.

A learned shopping procedure might say to hover over a main category and then click a subcategory. Rather than storing every click from every run, the system extracts a reusable pattern and includes it in the system prompt.

Ryan reports improvements across tasks involving shopping, maps and online discussions, sometimes comparable to or better than human-written workflows. In response to a question about reinforcement learning, he says this text-based learning can improve with relatively few examples, whereas reinforcement learning usually needs more.

In the particular workflow-memory paper, procedures are added to the system prompt between sessions rather than loaded by a tool during each run. This is simple for a limited number of examples but would require a retrieval mechanism to scale to much longer use.

Skill files are a simpler form of procedural memory. A system can initially expose only a skill’s name and description; when the agent needs it, it reads the full file. The underlying file system can provide the storage without a special memory database.

A limitation of procedures learned from past success is that the environment can change. The original Voyager and workflow-memory methods do not fully describe how to repair stale procedures. A system using editable skill files could update them, but maintenance still has to be designed.

05

MemGPT and tiered memory

MemGPT lets the model manipulate working context and search separate stores for conversation history and archival information. Other architectures put more of that management in a background workflow.

MemGPT treats the language model as a manager of a memory hierarchy. It has tools for reading and writing memory rather than relying on a single ever-growing prompt. Its context contains system instructions, an editable working context and a first-in, first-out queue of messages.

As new turns arrive and the window fills, the oldest messages leave the queue. A queue manager moves them into recall storage, where the agent can perform exact string or date lookups. Separate archival storage holds longer-term facts and documents in a searchable vector database.

MemGPT’s memory hierarchy
FOLIE 4 / 8MemGPT’s memory hierarchyEditable context, a message queue, archival storage and recall storage are managed through different mechanisms.

The model can append or replace facts in its editable working context. In one example, a conversation establishes a user’s birthday and partner’s name; later, when the relationship changes, the agent updates the relevant memory instead of keeping only the old fact.

The model can search the conversation store for an exact term, such as a place mentioned in a relationship story. It can also search archival documents and request another page of results if the first page does not answer a question. Information already in working context need not be separately retrieved.

The language model chooses what term to look up; the recall store can then perform exact matching. In response to another student, Ryan says rewriting context is not especially friendly to the KV cache: changes can invalidate cached computation and require it to be refreshed.

Ryan distinguishes MemGPT’s “working context,” the editable portion of the prompt, from his broader use of “working memory” for everything currently in the model’s context. Documents outside the window remain external until looked up and added.

The results he shows suggest that allowing the model to use memory tools can improve a deep-memory retrieval task compared with simply retrieving a fixed number of documents. Adding more retrieved material can otherwise degrade performance as irrelevant context grows.

MemoryOS offers another tiered design, with short-term, medium-term and long-term persona memory. A workflow analyzes usage and moves or edits information between tiers, promoting frequently useful material. Its management decisions are more explicitly handled by background machinery than in MemGPT’s model-directed approach.

A student asks how to compare memory systems when accuracy, latency, cost and storage all matter. Ryan says papers use several benchmarks, mostly reporting their associated accuracy metrics, but he does not see a single standard that compares all these dimensions together.

06

Reflection and background memory maintenance

Generative Agents combine a memory stream with reflection and planning. Other systems consolidate, link and update memories, sometimes using idle-time computation.

Reflective consolidation turns experiences into higher-level lessons. In the reflection pattern, an agent acts, observes a result, evaluates its trajectory and stores a reflection. Ryan notes that, in the comparison discussed, adding reflections about why traces mattered worked better than simply adding raw traces.

Generative Agents simulates characters who perceive a shared environment, form goals and interact. Their observations enter a memory stream; the system retrieves relevant memories, forms plans and reflections, and writes those back before deciding how to act.

Generative Agents memory stream
FOLIE 5 / 8Generative Agents memory streamPerceptions, retrieval, planning and reflection feed a shared memory stream.

One simulated character, Isabella, is planning a Valentine’s Day party. When asked what she looks forward to, the system retrieves memories and reflections ranked by recency, importance and relevance, then answers using the party plans. Both raw observations and higher-level reflections become part of the stream.

An ablation removes components such as reflection or planning and compares the resulting behavior. Ryan says reflection was an important contributor to the measure of how realistic the simulation appeared.

Memory maintenance is a separate problem: stored information needs to remain useful as new evidence arrives. One proposal, sleep-time compute, performs background reasoning between interactions instead of waiting until a user asks the next question.

In the example, a conversation describes a juggler with 800 balls, one quarter of them tennis balls, half of those indigo and one tenth of those marked. A system can reason through possible quantities while idle, then answer that there are ten marked indigo tennis balls when the user later asks.

The difficulty is that the system does not know which question will be asked. Reasoning over many possible questions consumes resources that may never pay off. The benefit is potentially lower response latency when its preparation happens to match the later question.

Asked whether this can also improve accuracy, Ryan distinguishes theoretical and practical comparisons. Given enough time to reason after seeing the actual question, it may not beat that approach; under a tight response-time budget, earlier work can sometimes help. He also notes safety questions if an idle-time agent has tools and can act while the user is not watching.

A-MEM explores an agent responsible for managing memory: linking entries, deciding what to store and evolving prior memories. Ryan summarizes it as shifting memory maintenance into a dedicated process rather than only letting the main agent read and write entries.

Mem0 likewise extracts information from exchanges and decides whether to add, update, delete or leave a memory unchanged. Entity and relationship extraction can update a knowledge graph and resolve conflicts so that changed preferences are represented.

Comparing four memory patterns
FOLIE 6 / 8Comparing four memory patternsThe table contrasts what each pattern stores, when it writes and reads, and its principal trade-off.

The four patterns make different trade-offs. Reusable procedures save effort but can become stale; tiered memory limits active context but relies on management; reflection can introduce unsupported abstractions; and maintenance costs computation and may lose information.

07

How to evaluate memory in realistic tasks

Recall, personalization and acting across long trajectories test different capabilities. Ryan compares conversational benchmarks with MemoryArena’s evolving task environments.

A useful memory benchmark should be realistic, long enough to require memory, challenging and verifiable. Ryan compares tests of conversational recall, changing user preferences and agents acting over sequences of tasks.

LoCoMo simulates conversations among agents over multiple days, with human editing to keep the generated histories consistent, and asks questions with known answers. One question might ask whose birthday was celebrated after the conversation said it was the user’s daughter.

The study illustrates that retrieval and use are different problems. Ryan points to a comparison where recalling the relevant conversation improved when more observations were retrieved, but answer accuracy fell because the additional context made the model’s job harder.

PersonaMem simulates a user interacting with an assistant over multiple sessions as preferences evolve. Its multiple-choice questions test whether the answer reflects the latest relevant preference rather than an old, contradictory or generic one. Ryan considers this more realistic in some ways, while noting that multiple choice remains a limitation.

MemoryArena moves from answering questions about stored facts to an agent acting in an environment, receiving feedback, updating memory and continuing. In group travel planning, the system first plans for one traveler, then repeatedly revises the itinerary as additional people contribute budgets and preferences.

MemoryArena group travel task
FOLIE 7 / 8MemoryArena group travel taskTravelers join in sequence, adding preferences and constraints the agent must respect.

The final itinerary can be checked against everyone’s constraints after a long series of interactions. Ryan reports that memory-agent approaches scored below the long-context and retrieval baselines in the comparison shown. He cautions that agents can fail for reasons other than memory, but sees substantial room for improvement.

A student asks whether memory needs to be separated from the agent harness. Ryan says the tools that read and write it are themselves part of the harness. Whether external tools are necessary also depends on how long the task becomes and whether the whole history can fit in context.

08

Forgetting, privacy and using memory appropriately

The future value of information is hard to know, but keeping everything has costs. Poisoned memories, private details and irrelevant personalization can all change agent behavior.

Choosing what to remember is difficult because future relevance is unknown. Keeping everything raises retrieval, context and maintenance costs, while forgetting can discard useful evidence. A joking remark about refusing morning meetings without pastries shows how hard it can be to distinguish a durable preference from a joke.

Memory poisoning and agent behavior
FOLIE 8 / 8Memory poisoning and agent behaviorThe example illustrates how a malicious stored procedure can be retrieved by an otherwise benign request.

Trust is another concern. Ryan describes an experimental memory-poisoning example in which an otherwise benign driving request retrieves a malicious procedure for unsafe driving. He calls the scenario simplified but uses it to show how untrusted memories can redirect an agent’s behavior.

Memory can also expose private details. In an example from a privacy benchmark, a user asks an agent to send a weekly report to their manager; the agent includes a lunch with a recruiter at a competing company. The system has the fact but does not understand that the user may not want it shared.

Finally, an agent can over-personalize by using a memory merely because it is available. Ryan gives an example in which a question about young voters prompts an irrelevant reference to the user’s personal “small wins.” Better retrieval or training to ignore irrelevant context may help, and different users want different degrees of personalization.

Mehr aus diesem Kanal

9 Folien1:16:51EN
Stanford Online· 14. Nov. 2026Michael Ryan: Choosing Agent Patterns by Control Flow and ContextMichael Ryan · Stanford UniversityStanford’s CS329Z explains workflow patterns, reasoning-action loops, and ways to keep an agent’s context focused.Agent EngineeringAI agentsEntwicklungs-Workflows & Tests+1
4 Folien1:17:14EN
Stanford Online· 13. Nov. 2026DSPy, LangGraph and the engineering around an AI agentMichael Ryan · Stanford UniversityMichael Ryan compares agent frameworks through a calendar demo, then shows why evaluation, recovery and tracing matter.Agent orchestrationEvalsAgent Engineering+2
2 Folien1:40:43EN
Stanford Online· 13. Nov. 2026How AI agents find information and take actionAfshine Amidi, Shervine Amidi · Stanford UniversityStanford instructors explain why useful agents need retrieval, well-described tools and a harness that manages context and actions.AI agentsRetrieval-Augmented GenerationSearch & retrieval+2
Folien1:19:15EN
Stanford Online· 12. Nov. 2026How AI agents learn to use tools—and how to test themDiyi Yang · Stanford UniversityDiyi Yang traces tool use from retrieval failures and Toolformer to runtime discovery, MCP and reliability benchmarks.Agent EngineeringMCPAI agents+2
1:11:21EN
Stanford Online· 12. Nov. 2026Stanford researchers use AI to challenge thinking, not replace itCynthia Bailey, Mark Algee-Hewitt, Megan Ma, Markus Covert · Stanford UniversityA humanities scholar, legal researcher and bioengineer show how they use models while protecting judgment and human interpretation.AI for scienceAI in EducationHuman-AI Collaboration+2
3 Folien1:16:31EN
Stanford Online· 11. Nov. 2026Building RAG: from document chunks to agents that searchDiyi Yang · Stanford UniversityDiyi Yang explains how indexing, hybrid retrieval and reranking work—and why agents need to check what search returns.RAG & WissenHybrid searchAI agents+1