En 30 segundos
A voice agent that resends every word on every turn can become expensive and slow without retaining anything after a session ends. Google Cloud engineer Laxmi Harikumar and Scribant developer Miles Libbey combine a short rolling window for immediate intent with a separate memory bank for persistent facts. In their short demonstration, the 40-word window reduced classifier input by about 86% compared with the full speech stream; a rolling summary saved about 70%. The numbers describe that demonstration, not a general benchmark.
- 1Proposed actions do not reach external services until approvedSpeech-to-text feeds parallel summary and action extraction, followed by date resolution. An execution gate presents cards before changing Calendar or other services.
- 2Raw history, simple retrieval and knowledge graphs have different costsHarikumar says full history grows prompts, while simple text retrieval may return conflicting facts; a custom knowledge graph requires additional engineering.
- 3Store and update durable facts outside the live promptThe presenters demonstrate asynchronous fact extraction and retrieval from Memory Bank, including user preferences that can carry into a later session.
- 4Send the classifier a bounded window of recent speechThey replace roughly ten transcript lines with about 40 words, with the length adjustable to the application.
- 5The short demo shows different savings for summary and windowThe presenters report about 70% fewer tokens for the rolling summary and 86% for the 40-word window against the full speech stream; this is one example.
Conclusiones
- Keep recent words for immediate intent and retrieve persistent facts separately instead of sending the entire transcript on every turn.
- Resolve relative dates deterministically and require approval before proposed actions touch external services.
- Measure token use for raw speech, summaries and the rolling window; the right window length depends on the speech.
- In the presenters’ short example, a 40-word window saved about 86% of classifier tokens versus the full stream.
Lee el texto editado por capítulos
A voice-to-action agent for messy thoughts
Scribant turns spoken thoughts into notes and proposed actions. The demo shows a meeting and other tasks emerging from a live transcript.
A voice-to-action agent can turn a stream of thoughts into useful actions, but a continuously running conversation creates two problems: a growing prompt and no durable memory between sessions. In this demonstration, the team adds a rolling context window and a separate store for longer-lived facts.
Miles Libbey built Scribant after long commutes left him with voice notes full of ideas that were tedious to sort. He wanted an agent to synthesize notes and route proposed tasks to services such as Google Drive, Calendar and Jira.
Laxmi Harikumar gives it a mix of errands, a meeting and a design-document deadline. The live transcript appears beside a summary, while the application sorts possible actions into separate buckets. Libbey says the agent is designed to listen and work in the background rather than interrupt.

Harikumar selects the proposed meeting event and shows that it appears on the calendar. The experience separates the messy original speech from the structured action the user decides to take.
The pipeline keeps action execution behind approval
Transcription feeds an agent pipeline and a deterministic date resolver. An approval gate separates proposed actions from writes to external services.
The pipeline begins with Google Cloud Speech-to-Text, which transcribes the audio. Two specialist agents then work in parallel: one maintains a summary and another extracts things such as events, tasks, searches and development tickets.

A deterministic date resolver turns a relative phrase such as next Wednesday at three into a calendar date. The pipeline presents proposal cards and remains read-only until the user approves an action; it does not write to Calendar or a development workspace merely because a task was mentioned.
Why raw history is not a durable memory
The presenters compare full-history prompting, simple retrieval and knowledge graphs. Each presents a different trade-off in latency, consistency or engineering overhead.
That design works within a session, but repeatedly stuffing the growing chat history into a prompt consumes many tokens on each turn. The presenters set out to preserve useful context across sessions while keeping the live agent responsive.
Harikumar compares three ways to give an agent context. Appending raw history is simple but can grow rapidly when the agent processes speech every two seconds. A basic retrieval system fetches matching chunks, saving tokens, but can return both an old and a new address without deciding which is current.
A custom knowledge graph can represent entities and relationships, but she says developers must also build extraction, maintain schemas and implement updates when facts change. Their alternative uses Vertex AI Memory Bank to manage those memory operations separately from the main execution loop.
Extract and retrieve persistent user facts
The demonstration creates a memory bank, stores facts such as preferences and inspects them in the console. These facts can be retrieved in a later session.
Harikumar says the memory service runs asynchronously and handles extraction, deduplication, conflict resolution and fact updates. The team creates a memory bank and finds its identifier in the cloud console before adding stored facts.
Ingestion saves facts likely to matter later, such as a trip to India or a preference for tea. The team also builds an API to test ingestion and retrieval: storing a fact is only useful if the application can fetch it when a later question calls for it.
The console displays saved facts with resource names and creation and update times. In the example, a vegetarian catering requirement and a tea preference appear as compact entries rather than a full conversation transcript.

The second change concerns the immediate context. Libbey had been sending about ten lines to the intent classifier so it would have enough context. With persistent facts available separately, the team instead sends a rolling window of approximately 40 words.
Compact the immediate context and measure it
A rolling window limits what the intent classifier receives. A dashboard compares the raw stream, a rolling summary and the bounded window in a short live test.
Forty words is a starting choice, not a fixed rule. The window should be adjusted for the application and the length of users’ sentences so the classifier sees recent information needed to decide whether speech implies a task, calendar entry or question.
Compaction limits irrelevant material and helps keep the prompt within its bounds and latency down. Harikumar distinguishes a semantic summary from rule-based pruning; for example, an agent may need a tool’s final result without carrying every intermediate result forward.
To see what the change is doing, they build a telemetry dashboard showing tokens in the transcript, summary and rolling window. This makes it easier to see which part of a streaming agent consumes the context budget.
They test the agent with a short conversation about a prior trip to India, a possible visit to Spain and an interest in beaches and camping. The application identifies a query and actions while the dashboard compares the versions of the context.
For roughly 200 spoken words, the rolling summarizer produces about 92 tokens. Against sending the full speech stream to the intent classifier, the presenters report around 70% token savings for the summary and about 86% for the 40-word window in this short demonstration.

The presenters also compare roughly 1,400 raw tokens with about 24 tokens of distilled memory in their example. Inspecting the bank shows facts including a visit to India, an interest in Spain, beaches and camping, and making tea.
Inspect and manage what was remembered
The demo stores travel plans and preferences for later use. The presenters say memories can also be deleted.
Those facts can be recalled in a later session instead of being reconstructed from the entire stream each time. Harikumar says users can delete memories and keep them for as long as they choose.
The proposed architecture combines a bounded live context with persistent memory: use the short window for the current intent and retrieve saved facts when relevant. The presenters frame their token dashboard as a way to monitor the cost of a continuous speech agent.
Más de este canal
16:12EN
55:11EN
13:01EN
23:57EN
41:04EN
17:27EN