Jesse Hall: Build Voice Agents for Conversations, Not Leaderboards

Most agents we use day to day work in the background. They write code or do research; we give them a task, walk away, and return later. An extra ten seconds, ten minutes, or sometimes even ten hours may be acceptable. A real-time agent faces a different expectation.

Over the past year, you may have encountered a real-time voice agent when calling a pharmacy, airline, or car dealership. At some point, you probably did something very human: you spoke over it, interrupted it, or confused it. The conversation fell apart, stopped feeling natural, and you pressed zero to reach a person.

The stakes are higher in financial services. A caller might schedule a payment and then change the amount or date halfway through the request. If the agent fails to keep up, the result can be more than an awkward conversation: it can be an incorrect financial action. We need agents that handle these changes naturally.

When people think about a good voice agent, latency is usually the first concern. We want it to be fast, and latency does have a finite budget. But speed is only part of the equation. The real goal is to make the agent as human-like as possible within that budget.

Voice is a natural interface: we learn to speak before we learn to read. LiveKit provides an open-source real-time framework and infrastructure for agents that hear, see, and speak to the person on the other end. There are other interaction modes, including video and data streams, but today we are focusing on the challenges of voice.

A traditional voice agent uses a cascaded pipeline: speech-to-text, a language model, and text-to-speech work together. Newer speech-to-speech models can replace that pipeline, although they are not yet the right choice for every use case. With hundreds of models and dozens of providers, choosing a stack quickly becomes complicated.

Leaderboards score individual models and components, but we are shipping a conversation. Suppose a caller says, “I want to pay on the 3rd,” and speech recognition hears “the 30th.” The LLM plans around the 30th and passes that date into the system; it cannot know the transcript was wrong. Those errors flow downstream. Component leaderboards also tend to use clean audio, while real callers may be on speakerphone in traffic with children in the back seat.

The useful question is which stack to ship for a particular use case. Even the measurements we use to compare providers can be misleading. A TTS dashboard might report a very fast 210-millisecond time to first byte, yet we have tested popular providers that send as much as three-quarters of a second of silence afterward. A byte is not audible audio. Measure the time until the user actually hears something.

The same problem appears with language models. Time to first token is a text metric, and TTS cannot start speaking just because one token has arrived. It generally needs a complete sentence. A model that generates tokens quickly is not necessarily the model that produces the first speakable sentence fastest, so time to first sentence is the more useful measure.

Voice agents also have to do things: look up transactions, check payment status, and submit disputes. Backend operations can take long enough that the delay becomes obvious. We cannot hide forty seconds or more of latency. Instead, ask what a human representative would do while the system is working.

A human would not simply go silent. They might say, “Give me a second, I’m looking that up,” explain that the system is slow, or ask another question while the lookup runs. That ability to multitask is part of natural conversation, and our agents should be able to do it too.

The code example adds an asynchronous update that hands conversational control back to the agent while a tool call is running. The conversation and the backend tool no longer have to share the same thread of work. The agent can keep talking while the query continues independently in the background.

In the hotel-booking demonstration, the caller supplies dates, the number of guests, room preferences, and contact details. The agent accepts interruptions and keeps moving through the booking. Rather than collapsing when the caller speaks over it, it maintains a natural exchange.

Behind that conversation, tool calls save each piece of information to a real backend. The agent does not wait for every save to finish before asking the next question. If a tool returns an error, it can verify or correct the information. If the problem persists, it can hand the call to a human. This is how our agents should operate.

Another essential ability is knowing when to speak and when to keep listening: turn detection. LiveKit’s built-in detector considers more than the transcript. It also listens to the audio’s rhythm, pitch, and intonation to decide whether the person has finished speaking.

In evaluations against our hotel agent and other use cases, a leading alternative interrupted callers 9.9% of the time, compared with 4.5% for our detector. Those numbers describe our own scenarios. The evaluations are open source, so teams can run them against their own datasets and see what results they get.

Imagine that I want to pay $555.23 and pause for two seconds after saying “five hundred.” A detector that only sees “500” in the transcript may conclude that I am done. A detector that hears my inflection can recognize that the amount is unfinished and wait for me to continue.

The latency window begins with the caller’s last syllable and ends with the agent’s first audible response. The turn detector determines when the caller is done, the language model can prepare an answer from a provisional transcript, and TTS starts generating audio as soon as possible. Every chosen model, function call, and handoff has to fit within that finite window.

At roughly one and a half seconds between turns, the conversation can still hold together, but it starts to feel unnatural. At around 500–600 milliseconds, the exchange feels more human-like. The question is therefore how much natural behavior we can fit into the latency window, rather than how to make everything faster without limit.

To choose models, test an entire conversation using a complete working agent. The hotel receptionist from the demonstration has a real backend; it is a functioning, open-source agent rather than a mockup. We put simulated callers against that agent to evaluate what actually happens.

Simulated callers can pursue goals, change their minds, interrupt, mumble, and bring background noise into the call. Text-based simulations test business logic. Audio-based simulations also exercise speech recognition, turn-taking, and the complete voice pipeline, bringing the test closer to a real call.

Each scenario should specify what the caller wants, what actions the agent should take, and the expected outcome. For the hotel, did the correct reservation reach the database? For a financial workflow, was the correct amount charged with the proper authorization? We score those outcomes against the scenario.

Completion alone is insufficient, because an agent can reach an apparent result through the wrong steps. Did it take ten steps when it should have needed two? Did it hallucinate a function call? We must evaluate the route it took and the whole conversation, as well as the final result.

Keep the agent and scenarios fixed, then swap the models and run the tests again. Every model should see identical input, byte for byte. Repeating that loop builds a personalized benchmark for a particular agent and use case. That is how a team can discover which model combination works best for its own situation.

These tests can produce surprises. A model that gets the correct result fastest may be the right choice even if it is not at the top of a general leaderboard. The benchmarks, simulations, and methodology are open source, so teams can try them with their own agents.

In the hotel-agent benchmark shown on the slide, Gemma 4 31B on LiveKit Inference reached the first sentence in 381 milliseconds. The same model through OpenRouter took 844 milliseconds; Gemini 2.5 Flash took 932, GPT-4.1 took 1,097, and GPT-5.5 took 1,645. These are results for this hotel scenario. LiveKit optimizes inference serving for voice workloads while remaining model agnostic, so teams can test whichever models suit their applications.

A voice agent is more than a chatbot with a speaker attached. It is a real-time system that needs to respond like a person. Set a latency budget based on what the caller hears, measure time to first audio and first sentence, confirm critical amounts and dates, verify identity before accessing an account, and keep listening while slow tools run.

Run complete conversations repeatedly through simulations. Check the final state, privacy safeguards, unnecessary tool calls, and turn-taking. Convert every failure into a regression scenario and iterate. Start with one workflow or failure you care about: run it, listen to the conversation, and check what happened in the backend. The goal is an agent that can handle being interrupted or confused and still keep the conversation natural. Go build that.