MemGPT: Managing LLM Memory Beyond a Fixed Context Window

In October 2023, Matthew Berman reviews the UC Berkeley paper “MemGPT: Towards LLMs as Operating Systems,” demonstrates the early code and introduces authors Charles Packer and Vivian Fang. The problem is how a conversational agent can retain and use information that exceeds its model’s fixed context window.

The paper considers long-running conversations and document analysis. Berman compares the proposed virtual context to operating-system memory: a limited main context resembles RAM, while external storage resembles disk. MemGPT moves selected information between them; it does not literally give the underlying model an unlimited context window or guarantee perfect recall.

Berman explains that expanding a conventional Transformer context can be expensive because standard self-attention has quadratic scaling, and that long-context models can struggle to use information in the middle. These observations motivate a retrieval and memory-management system, although performance depends on the architecture and workload rather than a universal law about every model.

MemGPT provides the language model with function calls for reading and updating memory. A parser handles events, including user messages and documents, and the model can issue several memory operations during one task before responding. The paper describes this as self-directed management under explicit system instructions, not independent learning of model weights.

The main context has a fixed token budget and contains system instructions, recent conversation and editable working context. External storage can be much larger but remains constrained by physical resources, retrieval quality and cost. System instructions and prior messages also consume part of the context budget, leaving less room for fresh user content.

Berman contrasts this with recursive summarization, where old material is repeatedly compressed to fit. Summaries can lose details over successive rounds. He uses repeated video compression as an analogy: the ability to retrieve stored history may preserve information that a summary omitted, though retrieval can also miss or return incorrect information.

The paper’s main-context hierarchy pins instructions, keeps a recent conversation queue and allows the model to edit working context through functions. Berman shows an example in which the agent records a birthday and favorite cake from a conversation, then replies to the user. It infers the birthday from the message date, illustrating both useful memory selection and the possibility of an incorrect inference.

The paper separates recall storage, containing processed conversation history, from archival storage, a general read-write store for facts, preferences and documents. Its experiments use databases and search by time, text and embeddings to bring relevant items back into the limited main context. These are external system components, not memories automatically embedded into the model’s parameters.

Explicit instructions explain the memory hierarchy and available function schemas. When a context warning arrives, the agent can move or edit information; it can also correct a stored preference when a user says they prefer romantic comedies to horror. Berman notices that one example mentions a game absent from the visible conversation and raises the possibility of hallucination.

The researchers evaluate multi-session dialogue for consistency and engagement. A deep-memory retrieval task asks a narrow question about a prior conversation, such as an artist mentioned earlier. MemGPT can search recall storage and use a retrieved passage to answer; the paper compares its results with fixed-context baselines under specified experimental conditions.

A separate conversation-opener evaluation tests whether an agent draws on earlier information when greeting a user. Berman shows examples involving a person studying for the LSAT and their preference for coffee or tea. Such examples illustrate personalization; they do not establish that every reference is accurate or welcome.

For document analysis, Berman explains why a collection of long company documents may exceed the context available in 2023. He discusses research showing uneven retrieval across positions within long inputs, comparing it to a classroom memory exercise. The analogy is illustrative and does not establish that language models and human memory work the same way.

The paper’s displayed document question-answering and nested key-value retrieval charts compare MemGPT with GPT-3.5 and GPT-4 baselines as the input grows. In these experiments, the MemGPT curves remain comparatively stable while some baseline results decline. This is evidence for the tested tasks and settings, not a claim of constant accuracy on any number of documents.

There are trade-offs: the system instructions consume some of the underlying model’s limited context, and repeated model calls and retrieval add cost and latency. The paper also reports that GPT-4 function-calling models made fewer function errors than tested GPT-3.5 and Llama 2 variants. Those findings describe the 2023 models and experiments, not all later open models.

Berman is interested in connecting MemGPT to multi-agent tools such as AutoGen, but describes that as a possible future integration. External storage can extend accessible information; it does not create literally unlimited, error-free agent memory.

In the historical installation demonstration, Berman opens the project’s repository, clones it, creates and activates a Python 3.10 Conda environment, installs requirements and supplies an OpenAI API key through an environment variable. These are the steps he showed for an early version, and the project’s current setup may differ.

He demonstrates loading text documents into archival storage and computing embeddings for example SEC filings. The displayed estimate is 12 cents for three documents and he says embedding took about six minutes. Those figures belong to his demonstration and are not a general cost or speed promise.

His first question does not explicitly direct the system to use archival memory. He then asks for Lyft and Uber revenue based on that memory, and the interface shows separate archival searches and a generated answer. The generated figures require checking against the original filings before use; retrieval and fluent responses do not guarantee factual correctness.

Berman starts a second query asking how often a word appears in the filings, but stops when he worries that the repeated calls could become expensive. He says the early project is evolving and hopes for open-model support. A user can also adjust the persona or instructions governing when the agent searches memory.

At the end, Charles Packer and Vivian Fang introduce themselves as members of the MemGPT team. Packer explains that a limited-context model can be instructed to notice a memory warning and save relevant facts to a persistent store before the conversation moves on. This extends what the system can retrieve later without altering the fixed underlying window.

For the near term, the authors describe supporting more workflows, including document embedding search and potential AutoGen integration. Longer term, they want to reduce reliance on GPT-4, whether by improving other models or tuning open models for the language-model layer. These were research plans at the time of recording, not guarantees about later releases.