Give an AI agent two kinds of long-term memory

Annie Wang asks what a multimodal agent should remember after a session ends. A styling assistant illustrates the goal: use a customer’s wish list and prior purchases to suggest clothes for an upcoming dinner.

The context held during a conversation resembles short-term memory: it helps while the session runs, but does not automatically carry over. The design question is which information is worth retaining for a future session.

Wang distinguishes exact facts from semantic understanding. A dated event or purchase can be stored as a structured record with an ID; a preference such as choosing understated clothes is a pattern that may emerge across conversations.

For exact facts, she recommends a structured database such as PostgreSQL, MySQL or Cloud SQL. A calendar question needs the actual event, not a semantically similar record.

For less structured understanding, a vector store can retrieve related memories by meaning. For example, it might find style preferences relevant to a dinner even if the question does not match the wording of the stored memory.

A vector store is a category rather than one product: vector search can be used inside PostgreSQL or through services such as AlloyDB. Wang also describes Agent Engine Memory Bank as a managed option that can extract useful information from past sessions and update memories over time.

Putting every fact into similarity search creates a problem when an answer must be exact. A question about a purchase should query the database, whereas a question about usual dinner attire can use semantic recall.

Raw video frames and audio waveforms are separate from agent memory. The application may choose to retain media for replay, audit, debugging or training, but the memory layer generally needs the useful meaning extracted from the signals rather than every raw frame or sound.

At the beginning of a session, the agent can load structured facts such as upcoming occasions and wish-list items, then retrieve relevant semantic memories such as preferences and comfort constraints. Together, the two sources give it context for an answer.

Wang recommends saving explicit factual changes promptly: when a user asks to add a blazer to a wish list, the agent calls a tool to write that change to the database.

Long-term preferences need more judgment. A single remark about liking yellow today should not automatically become a permanent style preference; repeated behavior or a user’s explanation can provide a stronger signal.

Memory extraction can happen during a conversation or after it. Extracting during shorter intervals can produce cleaner memories and preserve information if a user leaves early, and Memory Bank supports mid-session generation.

For a live voice agent, model-based extraction is slower than a simple database write and should not block audio. Wang recommends running extraction asynchronously, either after a turn or after the session, outside the latency-sensitive path.

In her demonstration, structured facts are written immediately and semantic understanding is extracted after the session. The next session loads both again: read context, write changes and carry useful information forward.