Richmond Alake on Agent Memory, Context Engineering and the Cost of Recall

Berman opens the stream with a striking image and greets viewers from around the world. He revisits messages between Sam Altman and Mira Murati associated with the 2023 OpenAI leadership crisis, including Murati’s assessment that the situation was “directionally very bad.” Berman imagines the anxiety of waiting for a board decision; his description of the physical setting and participants’ reactions is speculation, not established by the messages alone.

Berman finds Altman’s brief replies relatable and reads further messages about pressure for an update. He says Altman appeared to handle the exchange calmly. He then introduces Oracle’s Richmond Alake, whose work on agent memory impressed him, and invites audience questions.

Berman briefly notes the release of GPT-Realtime-2 and says he hopes to cover voice later. OpenAI announced the voice model on May 7, 2026. At the end of this stream he says he has not yet tested it, so there is no voice demonstration here.

Alake explains that his developer-experience work at Oracle focuses on reaching AI developers. Once the participants resolve audio and screen-sharing issues, Berman starts with Anthropic’s “dreaming” feature for Claude Managed Agents.

Berman reads that dreaming was introduced as a research preview: it reviews previous sessions to identify patterns and improve an agent’s memory. The displayed announcement describes a scheduled process, with a choice between automatic memory updates and reviewing proposed changes.

Alake cautions that he has not deeply examined the feature. His interpretation is that a background process consolidates or reinforces useful memories, resolves conflicts and possibly forgets less useful information, so later sessions can retrieve information more effectively. Calling this “dreaming” is an analogy, not evidence that the system dreams as a person does.

He points to Letta’s sleep-time compute research as an earlier example of doing memory-related processing outside an active interaction. Berman wonders whether background consolidation could reduce work during peak use and yield more precise answers with fewer tokens. Alake agrees it is a plausible cost-saving direction; the two are speculating, not presenting measurements of Anthropic’s actual operating costs.

Alake also mentions Hindsight, developed by Vectorize, and conversations about its use with Oracle AI Database. He then describes a ChatGPT interface for marking cited memory sources relevant or irrelevant. He does not know its internal implementation, but sees user feedback as a possible way to improve retrieval and personalization.

The pair recall the paper “Generative Agents: Interactive Simulacra of Human Behavior,” which describes agents that observe, retrieve memories, reflect and plan in a simulated town. Berman recalls 1,000 agents and a birthday party. The 2023 paper shown on screen instead describes 25 agents and a Valentine’s Day party; a separate later study examined 1,000 simulated people.

Berman asks whether human feedback will remain necessary. Alake predicts it will help near-term systems but that memory management may become more automatic; he notes that developers can already build systems without a human grading each memory. This is a forecast, and removing human review also changes the opportunities to catch incorrect or unwanted memories.

Alake invokes OpenAI’s earlier sycophancy rollback as an example of why personalization can go wrong. He describes this as a memory problem, although that characterization should not be taken as a definitive explanation of the incident. He points viewers to an agent-memory course and stresses that memory is not yet a solved problem.

A viewer asks whether “dreaming” merely adds a pass over logs because transformers lack persistent state, and whether memory and computation should instead be fused. Alake agrees that many present systems bolt on external storage without changing the model’s weights. He is unsure whether that is sufficient long term and says continual learning investigates updating internal state over time.

The speakers discuss Dario Amodei’s view that significant economic value might come from systems built around existing models. Berman compares the model to an engine that needs a car around it: tools, memory and a harness turn raw capability into useful work. Both say they are uncertain whether future progress requires continually changing weights.

An audience member asks about low-rank adaptation, or LoRA, as a possible way to update models more cheaply. Alake compares it with earlier machine-learning feature stores and says his team is exploring techniques for changing models, while acknowledging the cost and latency of adapting large models in production. This is research direction rather than a demonstrated production result.

Alake shares his screen and introduces an Oracle developer hub with notebooks and a finance-agent example. He says the Oracle Agent Memory Python package aims to provide reusable memory building blocks for developers who do not want to implement every detail themselves. He is an Oracle employee discussing his employer’s tools.

Alake traces a rough progression from chat interfaces to retrieval-augmented generation, where external domain data is supplied to the model, and then to tool-using workflows. He distinguishes automation of defined steps from greater autonomy in choosing actions, arguing that many customers actually need the former.

In his retrieval diagram, data is ingested, cleaned, chunked and represented in different forms, including structured records, embeddings and graphs. A user query can retrieve and rerank relevant material before it is provided to an LLM. The design aims to bring task-specific information into the answer rather than relying solely on the model’s parameters.

Alake argues that maintaining many separate databases can add operational complexity, and recommends a consolidated system supporting several retrieval strategies. He proposes Oracle AI Database as one option. That is his architectural and vendor perspective; the right choice depends on the workload and existing systems.

Berman asks how to combine Slack, email, videos, project tools and internal notes so agents can answer questions and proactively find patterns. Alake says connectors or a custom ingestion pipeline can bring sources together, after which semantic, relational and graph retrieval offer different ways to surface information. Building ingestion and choosing the right retrieval method remain engineering tasks.

Alake describes an agent loop: gather context, reason, decide on an action, use a tool or stop, and repeat based on observations. As interactions accumulate, the finite context window needs attention. He calls for managing the signal-to-noise ratio instead of indiscriminately adding every prior message.

Context engineering includes tracking how much of the window is used, compacting or summarizing earlier material, offloading information and retrieving it when needed. Alake notes that producing a concise summary is not the hard part by itself: the challenge is preserving the important signal so future work remains reliable.

Berman returns to the viewer’s criticism that these techniques work around a lack of persistent model state. He agrees it is a limitation but sees substantial practical value in improving the scaffolding available now. Alake uses human and biological analogies to explain why engineers may borrow ideas while still exploring different implementations.

Asked how an agent should forget without creating retrieval noise or drift, Alake suggests scoring memory units using recency, relevance and importance. Different applications can weight those factors differently. He speculates that humans may never forget information; that is not an established description of human memory, which remains an active field of research.

A viewer compares agent infrastructure to CPU caches and storage. Berman and Alake find the analogy helpful: short-term context and persistent storage have different roles, as in the memory hierarchy discussed in MemGPT. The analogy is explanatory, not an assertion that human brains or LLMs work identically to computers.

Alake says long-horizon tasks such as debugging and research require an agent to iterate without losing essential context or allowing cost and latency to grow uncontrollably. Memory aims to support adaptation and continuity across those steps, but the system must still be evaluated for reliability.

Another viewer asks whether a model could manage its own context rather than relying on humans to curate it. Alake expects agents to refine their own harnesses and cites work on meta-harnesses; Berman notes that some viewers already report experimenting with such systems. Their expectations about how widely this will work are predictions.

Alake groups agent memory by duration and function. Information in a context window serves as short-term working material, while files and databases can persist information between sessions. Shared memory can coordinate multiple agents. These categories are useful abstractions, not strict equivalents of biological memory.

For procedural memory, he points to skills files that describe how an agent should carry out a task, and to tool descriptions retrieved only when relevant. Episodic memory can record time-associated interactions; semantic memory captures facts and relationships. Retrieving tools on demand can help avoid placing every tool description in every prompt.

Alake explains why he uses human-memory comparisons when teaching nontechnical audiences. He also points to neuroscience’s influence on neural-network research and to flight as an example of engineering inspired by nature. A viewer challenges whether the analogy copies surface features rather than deeper principles; the speakers treat it as a useful question, not a settled scientific claim.

He distinguishes context engineering, which manages the information presented to a model at a particular moment, from memory engineering, which also creates, consolidates, revises, augments or forgets information outside the window. Both are parts of an agent harness, and the policy for retaining and retrieving information determines what the agent can use later.

Alake presents an Oracle notebook comparing a naive agent that appends the conversation history with an agent using engineered memory. He says he ran 100 turns, but the chart actually displayed in the stream shows 80 conversation turns. In that chart, estimated input tokens grow for the accumulating-history baseline while the retrieved-memory line remains roughly stable.

He asks whether the shorter context preserves useful answers and shows a model-as-judge comparison. The displayed chart reports 52 preferences for the engineered approach, 12 for the naive approach and 16 ties over 80 turns. These are judged preferences for that setup, not independently verified accuracy or evidence of unlimited context; Alake explicitly says he did not test 1,000 turns.

A further trade-off is KV caching. When a prompt prefix remains unchanged, a provider can reuse cached key and value computations rather than recomputing them; changing or compacting earlier context can disrupt that reuse. Token reduction alone therefore does not determine total latency or cost.

Alake describes a second notebook chart: the append-only baseline can benefit from a stable cache, a memory-engineered version has higher latency in his example, and a hybrid design comes closer to the baseline. He says he plans to write more about the experiment. The comparison depends on the implementation, retrieval overhead and provider caching behavior.

Berman questions whether everyone will build a custom harness. Alake argues that coding assistants lower the barrier and that different workflows need different memory choices. He begins distinguishing deep research, conversational assistants and workflows, but the discussion shifts to an audience question before he fully develops the three categories.

A viewer asks whether pre-filling a KV cache with consolidated memory reduces retrieval misses compared with loading memories on demand. Alake relates the question to cache-augmented generation, in which reusable context is precomputed, but says he does not know the answer and would need to experiment. Embeddings used for retrieval and a model’s KV cache are distinct representations; pre-filling alone does not prove better recall.

Alake concludes that developers can experiment with memory and harness design and invites feedback on Oracle’s package and educational resources. Berman thanks him and shares a course. After the guest leaves, Berman says he intended to try GPT-Realtime-2 but needs to test it first, and closes the stream.