AI Memory in 2023: Long Context, Retrieval and Recurrent Memory

In this April 2023 video, Matthew Berman considers how AI systems can retain and use information over longer periods. He sees limited context as an obstacle for assistants and autonomous agents, and compares external memory retrieval with a research proposal for processing much longer sequences. Neither paper establishes that AI memory or AGI is solved.

Berman argues that access to relevant prior information could help an assistant reason and adapt. At the time, he cites roughly 4,000-token context for GPT-3.5 and 8,000- and 32,000-token GPT-4 versions as examples of the limits developers faced. These figures describe the 2023 setting, not current product limits.

A game character with an AI agent would need access to past events to behave consistently over time, Berman suggests. Without some way to preserve and retrieve earlier experiences, interactions can be constrained by the information that fits in the current prompt.

Coding is another example: a repository may be too large to place in a single context window, so a developer must select relevant files or provide them in pieces. A larger usable context could reduce some of that work, but merely fitting a codebase in a window does not guarantee that a model understands it correctly.

Berman also considers a book published after a model's training. A user can provide relevant material as context or use other methods such as retrieval or fine-tuning. Adding information to a prompt gives the model temporary access to it; it is different from updating the model's learned weights or giving it durable memory.

He recalls the 2023 Generative Agents paper: its simulated characters store experiences, retrieve relevant records for a current situation and synthesize higher-level reflections. The researchers tested 25 agents in a small virtual town; the architecture generates plausible behavior in that simulation, not human-equivalent lifelong memory.

Berman compares reflection to remembering broad patterns from older events rather than every detail. The paper's actual mechanism is a designed system of stored natural-language observations, retrieval and generated reflections. It is an analogy to human memory rather than a demonstration that agents remember as people do.

The second paper, Scaling Transformer to 1M tokens and beyond with RMT, passes learned memory representations between segments of a long sequence. In its original 2023 report, the authors augmented BERT and evaluated task-specific synthetic retrieval and reasoning across up to 2,048,000 tokens; that is not a general-purpose two-million-token ChatGPT prompt.

The researchers reported constant GPU memory in their particular inference setup and computation that scales approximately linearly with the number of fixed-size segments. Berman notes possible quality degradation and long processing times. Passing compressed memory between segments also means retaining all details or answering arbitrary questions over the entire input is not guaranteed.

Berman summarizes the available approaches as choosing what to put in the prompt, retrieving or compressing external memories, and investigating recurrent architectures. He calls the research promising but not ready for production, and hopes further work will improve memory for useful agents. His belief that such progress is needed for AGI is a hypothesis, not a result demonstrated by these papers.