ARC-AGI, Agent Memory and Interactive Worlds: Forward Future Live

Matthew Berman opens the August 22, 2025 Forward Future Live with co-host Nick Wentz and a slate of AI news. Their guests are ARC Prize Foundation president Greg Kamradt, Letta co-founder Charles Packer, and Skywork AI multimodal director Alan, who will discuss benchmarks, agent memory and interactive world models respectively.

The hosts begin with DeepSeek V3.1, a hybrid model offering both thinking and non-thinking modes. Berman says slower inference can help on hard tasks, but he prefers speed for most of his own work and uses GPT-5 Pro only occasionally. Wentz also usually favors the fast mode and asks viewers when extra thinking is useful.

Wentz identifies improved agent skills alongside hybrid inference as a major theme in frontier models. Both hosts say the right choice between faster and more deliberate responses depends on the use case; viewers offer their own examples.

They discuss a contemporaneous comparison claiming 66% on SWE-bench Verified and API prices of $0.56 per million input tokens and $1.68 per million output tokens. The quoted figures and relative prices are historical, model-provider figures and depend on benchmark setup and pricing conditions. Wentz says inexpensive models could be attractive if the numbers hold in practice.

Berman calls the model an appealing value proposition, including its downloadable weights, while observing that developers needing the strongest coding performance may choose another model. The hosts then turn to reports that many Grok conversations appeared in search results.

They worry that sharing features expose sensitive conversations without clear enough warnings. The reported Grok issue concerned chats users had shared via links, not every private chat. Berman compares it to ChatGPT, but its separate search-discoverability feature required users to opt in after choosing to share; his suggestion of automatic opt-in should not be taken as established fact.

Wentz says he often shares personal details with AI assistants because that context can improve answers, which makes privacy expectations particularly important. The hosts want clear warnings about when a shared conversation may be indexed.

Next they discuss Meta Superintelligence Labs and a reorganization under Alexandr Wang. They note reported changes in research leadership and Yann LeCun’s skepticism that large language models alone will reach human-level AI. Their suggestion of internal friction is speculation, not an established account of the organization’s internal relationships.

Kamradt describes the nonprofit ARC Prize Foundation’s goal of advancing openly available AGI research. It uses competitions, paper prizes and ARC-AGI benchmarks to encourage research in public. He calls the benchmark an intentionally opinionated way of measuring intelligence.

Kamradt says the foundation runs open competitions and awards paper prizes to elicit useful research. He stresses that ARC’s conception of intelligence is deliberately specific rather than a universally agreed industry definition.

He traces that view to François Chollet’s 2019 work: intelligence, in this framing, concerns efficiently acquiring new skills. A system can be excellent at chess or Go without demonstrating broad adaptation to unfamiliar domains. Humans serve as the foundation’s practical reference point for general intelligence.

ARC-AGI-1 and 2 show examples of a small visual transformation and then ask the test taker to apply the learned rule. ARC-AGI-3 extends the concept to interactive games: the agent enters an unfamiliar environment, experiments, discovers rules and tries to achieve the goal.

In the interactive version, the test taker must discover an unfamiliar environment’s rules through action and feedback. Kamradt contrasts that process with learning one narrow skill from large quantities of domain-specific data.

Kamradt says the foundation seeks tasks people can solve and current systems struggle with. He proposes a deliberately demanding behavioral definition of AGI based on whether humans can still find tasks they can do and AI cannot. That is his framework, not a universally accepted definition or a conclusion a single benchmark can settle.

Sharing an ARC-AGI-2 puzzle, Kamradt points out colored shapes and lines in the examples. The task is to infer how color and direction determine the output, then fill in a new grid. Models receive the data as JSON and may choose how to represent it internally.

The puzzle has a learning side and an application side. Kamradt says the system must infer the relationship between input and output examples and then demonstrate the same transformation on a new grid.

Each question tests a different small skill, Kamradt says, instead of repeatedly asking for the same transformation. He says ARC-AGI-2 contains more than 1,300 tasks and that multiple humans solved each task during testing. He reports a then-leading AI score of about 16%, which is a snapshot of the benchmark, not a timeless ranking.

Kamradt says the foundation recruited around 400 people and recalls that an average person solved roughly 70% of the tasks assigned to them, while groups collectively solved all tasks. He is uncertain of the exact panel size. These are his recollections of a particular evaluation, not a direct comparison of each person with every model on every task.

Asked what is missing in frontier models, Kamradt points to continuous learning, test-time adaptation and sample-efficient generalization. Context can provide new information, he says, but out-of-domain tasks remain hard. He does not claim to know the complete architectural answer.

Although ARC-AGI-2 had not been saturated, the team started ARC-AGI-3 early because benchmark development takes time. In August 2025 Kamradt expected a 2026 launch without committing to a date. He says finding problems that people can solve and AI cannot is becoming more difficult.

Kamradt says creating and maintaining a benchmark takes considerable time and funding. Starting early was intended to keep pace with model progress, rather than to suggest ARC-AGI-2 was already solved.

The new benchmark uses interactive reasoning rather than a single static question and answer. Kamradt compares the format with terminal and customer-service environments: an agent acts, sees feedback and adapts. In the ARC games, the test taker must work out the rules and find a way out.

Berman recalls the team removing game names to avoid giving hints. Kamradt adds that English names would privilege English speakers and that the aim is to minimize task-specific preparation. Humans and AI should receive the same available information and face the same standard.

Removing descriptive game names also reduces language-specific advantages. Kamradt says someone who does not speak English can still be generally intelligent, and the test should avoid confusing language familiarity with the ability to learn.

Kamradt demonstrates LS20. Available actions are displayed, but the player initially has to discover which object moves, how symbols relate to the exit, and what the diminishing purple markers mean. Experimentation reveals a limited energy budget and the need to replenish it.

A later level adds a color transformation and forces the player to manage movement and energy. Berman admits he became stuck; Kamradt says the team intends to include levels only if multiple ordinary computer users can solve them without prior instructions.

As the demonstration continues, a key changes color, and the player has to learn the new transformation while keeping enough energy to move. Kamradt says public testing will determine which games and levels qualify.

The exact agent instructions were still to be refined. Kamradt expects a minimal prompt explaining the goal and permitted actions, with matching information for human testers. He wants the comparison to be as level as possible.

Interactive tasks add an efficiency dimension. Kamradt plots the number of actions against level progression and proposes comparing AI action counts with human baselines. Actions spent exploring and actions used to execute a solution both count, so reaching the goal by brute force is distinguishable from efficient learning.

The proposed graph puts action count on the horizontal axis and level progression on the vertical axis. Kamradt says the red curve represents an average human run, and the team planned further human testing to establish its baseline.

Wentz asks whether prioritizing efficiency might penalize creativity. Kamradt says creativity matters, but this particular benchmark targets efficient generalization in a narrow domain. A portfolio of evaluations is needed to assess a person or AI system more broadly.

The foundation ran a 30-day preview competition. On the displayed private evaluation leaderboard, Tufa Labs’ StochasticGoose completed 18 levels and scored 12.58%, using 255,964 actions. Kamradt cautions that the scoring was still in flux and contrasts the action count with his rough human estimate; the setups and totals should not be mistaken for a controlled universal ratio.

The displayed leaderboard includes a private set of three unreleased games. The winning entry’s score is a preview figure, and Kamradt says the final scoring mechanism may change; its progress and action count are both important context.

In a replay, the winning agent tries many actions and learns which ones change the screen. Kamradt calls it an impressive capability demonstration, while arguing that its large action count is not sample-efficient intelligence. He says learning carries across levels, although he does not believe it transfers between games.

Berman asks whether broad scaling or specialized techniques will ultimately solve the benchmark. Kamradt expects general approaches to matter, but hopes the winning work will reveal several useful techniques that researchers can publish and apply elsewhere. He does not claim that a benchmark-winning agent would automatically be an autonomous employee or AGI.

Kamradt hopes the methods behind a strong future submission will be published openly. Other researchers could then extract useful techniques and apply them elsewhere, rather than treating the benchmark solution itself as a complete general-purpose system.

After a lighthearted look at Kamradt’s weightlifting, the hosts return to news. Berman says he would pay for a reliable voice assistant that lets him handle email, meetings and reservations while driving. Current conversational voice features are useful, Wentz says, but taking dependable actions is another step.

Steven Zhang responded to Berman’s post with a quickly built Work From Car demo. In the video, the voice interface reads an email, sends a short reply and checks a calendar after account permissions are granted. The hosts like the rapid prototype, though Wentz says he would not yet trust it with his personal information.

Berman says connectors to email and other services make the desired experience feel close, but trust and the interaction layer still need work. A compelling demonstration does not establish that the assistant is reliable enough to manage sensitive accounts unattended.

Wentz discusses a developer’s preference for Cursor over Claude Code and the broader competition in coding agents. Editors built on frontier models need their own advantages in speed, context and integrations as the model providers release competing tools. Berman recalls building a startup on Twilio and later finding himself competing with a provider.

Wentz argues that an AI editor needs more than access to the same model offered by its supplier. He points to speed, integrations and context as potential differentiators, while acknowledging that developers also have different interface preferences.

The hosts discuss ChatGPT study mode and flashcards. Wentz sees an adoption and retention strategy: as memory and personalization grow, switching could become harder. Berman likes the idea of guiding a student toward an answer rather than simply giving one, while treating the strategic interpretation as a hypothesis.

They review a then-current LM Arena leaderboard showing Gemini 2.5 Pro near GPT-5. Berman explains that users choose between anonymized answers based on preference, which can reflect style as well as correctness. Wentz considers the preferences informative but agrees that leaderboard positions and evaluation methods have limits.

Berman cites Sébastien Bubeck’s report that GPT-5 Pro helped with an open problem in convex optimization, and an OpenAI announcement about engineered Yamanaka-factor variants. He and Wentz are excited by the possibility of AI-assisted research, but neither independently verifies the mathematical result during the show.

The hosts cite OpenAI’s announcement about modified Yamanaka factors as another example of AI-assisted science. The reported work concerns protein design and laboratory evaluation, not a demonstrated cure or a finished clinical therapy.

Berman connects future scientific and mathematical assistance to possible recursive improvement. That is a proposed path, not proof that an autonomous self-improvement loop exists. The protein work was a collaboration with Retro Biosciences, involving experimental testing; improved cell-reprogramming markers do not by themselves establish a treatment for human aging.

Charles Packer joins to explain MemGPT and Letta. He describes tool-based memory, where a model can decide to create or edit a stored memory after a conversation, as central to the original MemGPT idea. He connects it to a broader vision of an operating system around language models.

Packer says the question was how to build a system in which a powerful model could retain experience and improve over a long period instead of eventually losing context or derailing. The decade-long agent he imagines is a research goal, not an existing demonstrated capability.

Berman asks what inspired the work and how a research paper led to a company. Packer recalls researchers with different backgrounds becoming excited by GPT-4 and experimenting with persistent memory through newly available function calls.

He and Berkeley collaborators Sarah Wooders and Kevin Lin became excited about GPT-4 and experimented with attaching a memory bank through tool calls. Packer says the early approach worked surprisingly well. Letta grew from the ambition to find methods beyond MemGPT that could support meaningful long-term improvement.

Packer says present-day coding agents and simple memory edits should not yet be described as meaningfully self-improving. His aspirational example is a coding assistant that stays with an engineer throughout a career and grows more useful over time; his expectation that this will arrive soon is a forecast.

He compares a language model to a CPU: powerful computation still needs an operating system and a hierarchy for moving information in and out of limited working memory. Retrieval-augmented generation is one simple example, but a human-written retrieval algorithm decides what enters the context.

Packer’s vision is for the model to manage more of that context itself through tools, moving information among layers of memory. Better tool use could make this orchestration more flexible, though whether it reliably produces long-term learning remains an open engineering question.

Berman raises the risk of memory lock-in: a company that holds years of personal interactions may make switching providers difficult. He asks whether users need an open standard or a separate memory store that can be accessed by different models.

Berman asks whether memory might live independently of the model provider so a user can take it to another system. Packer agrees there should be an open form of memory and imagines agents that accumulate experience over much longer periods.

Packer argues for an open version of memory that people can control and move. He expects users to accumulate valuable personal context across many model generations, and worries about losing it when changing services. His discussion is an argument for portability, not proof that all providers already prevent export.

Packer argues that retaining an agent’s state separately from a provider matters for companies as well as consumers: a business could be stranded if it depends on a provider that later disappears. He expects accumulated memory to grow more valuable as the underlying models are replaced over time; those long-lived agents remain a vision.

Packer says Letta’s core technology and REST API are open source under Apache 2.0, while its free web playground is not open source. He describes portable, locally storable memories as an intended way to avoid lock-in.

Packer says Letta focuses on long-running agents and cites Bilt as a customer using agents to build transaction-based profiles for recommendations. He describes many agents per customer and millions of customers; the scale is his company’s account, not an independent audit.

Berman challenges the phrase “infinite context window.” Packer agrees that nothing in computing is literally infinite: the underlying model has a finite context limit. Letta instead combines a sliding context, compaction and tools to search stored conversation history.

For example, a system might choose a 30,000-token working window even when a model accepts much more. When the window fills, Letta summarizes older material while retaining a portion of recent context. Packer characterizes the summary as lossy inside the window, but says the agent can search the external database for retained history.

Packer compares searching an agent’s stored history to searching messages in a chat application. The model’s immediate context remains bounded even if tools can retrieve older records; retrieval quality and stored-data availability still matter.

Berman asks whether larger model context windows still help. Packer compares them to increased computer RAM: larger capacity raises the ceiling for applications. He particularly expects multimodal inputs such as images and video to consume many tokens, while routine agent workflows may benefit from selective context management.

A larger context window may be especially helpful when image or video data consumes many tokens, Packer says. For a sequence of support-ticket actions, he expects careful selection and summarization of relevant information to remain useful.

Packer says a hypothetical perfect, nondestructive compaction system might allow a smaller working window for better focus or lower cost. This is an idealized goal, not a guarantee about current compaction. Larger windows can also reduce how often an agent must compact its history.

Berman asks whether current model architectures can reach AGI without updating their internal learning. Packer believes present models plus the right scaffolding could yield self-improving intelligence, while acknowledging that it is a hard research problem. He treats changes to weights and external memory as complementary options rather than insisting on one mechanism.

A long-lived agent could switch underlying models many times, Packer argues, making external memory especially valuable as a durable record. It might also periodically use that record to fine-tune a model. The argument is about a possible future architecture, not evidence that such an agent already persists for decades.

Packer suggests that stored experience might bootstrap periodic fine-tuning, even if an agent eventually replaces the underlying model. His emphasis on external memory follows from the possibility of swapping models without discarding the agent’s accumulated state.

Packer introduces sleep-time compute: instead of using computation only when a user asks a question, an agent can process prior interactions or prepare useful context in idle periods. He imagines anticipating needs and consolidating memory after a conversation. More computation can help some tasks, but extra thinking does not guarantee better results.

Berman suggests background work could move computation to less busy periods and reduce latency when the user returns. Those benefits depend on infrastructure, workload and costs. Packer says Letta already has a sleep-time agent that updates a shared memory asynchronously while another agent handles the conversation.

Berman suggests the workload could be scheduled outside peak periods, potentially improving user-facing latency and cost. These are potential system-design benefits rather than guarantees that running background inference is always cheaper.

Packer gives a concrete example: a conversational agent can respond to an introduction while a separate process records the person’s name, rather than making the user wait for a memory tool call. He says the Bilt recommendation deployment also uses sleep-time compute; he presents this as the company’s deployment, not an independent performance evaluation.

The background process can create what Packer calls learned context: a memory block refined through many inference cycles. He sees long-term personal context as valuable and argues that users should be able to download and transfer it instead of being locked into a single provider. Berman agrees and thanks him.

Packer argues that memory derived from years of interactions could become a valuable asset in its own right. That is why, in his view, the user should be able to move it across services and model generations.

Berman returns to news with Elon Musk’s suggestion that Grok 5 might be true AGI. He warns viewers to distinguish the claim from hype and notes Musk’s history of ambitious timelines. A future product name or a benchmark score does not establish AGI.

Berman says the Grok 5 label itself says little about the model’s eventual capabilities. He praises the pace at which xAI built its models while cautioning that product predictions and timelines are uncertain.

The post compared Grok 4 and GPT-5 on Vending-Bench, citing approximately $4,600 versus $3,500 in net worth and higher unit sales. Berman describes it as a real vending machine, but the original Vending-Bench is a simulated vending business. These historical simulation figures should not be represented as real-world store revenue.

Berman shows Figure Robotics footage of a humanoid crossing uneven debris. Figure’s Corey Lynch attributes the whole-body controller to simulation-to-real reinforcement learning. The clip illustrates a particular demonstrated traversal, not a full assessment of reliability across conditions.

A Boston Dynamics Atlas clip shows a robot handling items while people alter its surroundings. Berman notes its ability to respond in the demonstration and expresses enthusiasm about household robots. A selected demo does not establish that general-purpose home deployment is ready.

He also points to Goodfire research on detecting rare, unwanted behaviors after model post-training. He does not work through the technical method on air. The topic is finding failures that may be missed before deployment, not a claim that all such failures can be eliminated.

Skywork AI multimodal director Alan joins to discuss Matrix-Game 2. Berman compares interactive generated environments with Google’s Genie 3. Alan says his team also works on image, multimodal reasoning and generative media, with an aim of connecting research to products.

Alan describes himself as a researcher focused on multimodality, generative music and spatial intelligence. He says Skywork’s projects include models for image understanding and generation as well as multimodal reasoning.

Alan attributes the recent growth of world models partly to more compute and better video-generation base models. Adding action controls to those models enables an interactive experience. This is his explanation of the research trend rather than a single established cause.

Asked whether these systems are already games, Alan says current keyboard-and-mouse movement is still rather boring. He sees potential for more engaging interactions among users and non-player characters, but calls the present stage an early concept. Berman asks what might be possible in the next year and treats Musk’s game-development timelines cautiously.

Berman asks about predictions of fully AI-generated AAA games. Alan suggests simpler game-like experiences may appear first, while emphasizing that existing interactive video models have substantial limitations in memory and control.

Alan speculates that simple open-world or first-person experiences might become playable soon, while noting major limitations in memory and real-time generation. He describes the current model as retaining hundreds or thousands of frames, corresponding to minutes rather than persistent worlds. That is a technical limitation, not a promise of a commercial game.

He explains that the autoregressive diffusion system generates new frames conditioned on prior ones. A longer cache could extend history, while cross-time retrieval is a possible direction for recovering earlier views and maintaining consistency when users revisit overlapping areas.

On compute, Alan says Matrix-Game 2 has 1.8 billion parameters and can run on a single H100, with trade-offs in frame rate and resolution. The project documentation also lists tested A100 and H100 setups and a GPU memory requirement; hardware needs and performance depend on configuration.

Alan confirms that the project releases model weights and describes openness as part of Skywork’s approach. Sharing research components can lower barriers and generate community feedback, he says, while the company builds products on top. Berman welcomes the ability to inspect and run the model independently.

Alan says releasing components and weights can help researchers build on the work and provide diverse feedback. He describes product development on top of shared research as part of the company’s business strategy.

Looking ahead, Alan wants larger models, better resolution and frame rates, and more precise controls. He also imagines multiple users and voice interaction with other players or non-player characters. These are research directions, not features demonstrated as complete in the interview.

As an example of speech interaction, Alan imagines one person asking for a cabin and another planting a tree nearby. Berman asks how multiple players would inhabit the same generated world. Alan says maintaining consistency when their fields of view overlap is a challenge even if they share a core model.

In a possible multiplayer architecture, Alan says the core model could be shared but overlapping views would have to stay consistent. He acknowledges technical obstacles and does not present a completed multiplayer implementation.

Berman presses on persistence: a player might leave and return the next day expecting earlier changes to remain. After clarifying the question, Alan suggests using saved historical frames as memory to condition later generation. He presents this as an approach to explore, not a demonstrated solution for a durable shared world.

For Skywork, Alan frames Matrix-Game as part of a broader interactive AI strategy. He gives firefighter training as a possible future simulation application and says the team is working on inference cost, longer sequences and consistency. Scaling to millions of users remains an early-stage ambition.

Alan describes reducing inference cost and extending consistent sequences as prerequisites for a scalable player-driven experience. The firefighter-training example is a proposed use of simulation, not an established deployment.

Alan describes Skywork’s productivity suite for documents, slides and websites, alongside its music and video creation products. He says the common goal is to lower the barrier to creation. Berman notes that Matrix-Game 2 weights and code can be found on Hugging Face and GitHub, then thanks Alan and the other guests.

Berman closes by thanking the guests and viewers and mentioning the Forward Future newsletter. The stream’s historical product claims and predictions reflect the discussion on August 22, 2025; technical demonstrations and proposed future capabilities should be read in that context.