Giving AI a Nervous System: Voice, Touch, and the Post-Keyboard Interface

Russ, founder and CEO of LiveKit, joins the hosts to discuss how LiveKit powers applications such as ChatGPT voice and why voice may become a primary interface to AI. He describes using ChatGPT Advanced Voice while driving on Bay Area highways with a single AirPod—kept legal and safe—to ask for lessons on frontier technology, including quantum theory, quantum entanglement, quantum computers, and even how lightning works. The example is casual, but it shows voice as an always-available tutor that can turn a commute into a seminar.

Russ says the liberating part is being able to ask any 'dumb' question without the social pressure of a college auditorium where everyone else seems to understand. The model has broad world knowledge and, crucially, does not judge the learner. That combination makes voice a natural fit for exploration, not just command-and-control.

On the device, the LiveKit SDK captures camera and microphone input and sends the user's speech over LiveKit's edge network. That network is a global mesh of servers that routes audio to an 'agent' on the backend; Russ compares the agent framework to an application server in a web app. In ChatGPT's case, OpenAI builds an agent with LiveKit's framework to receive the incoming audio.

In the older voice mode, the agent ran a pipeline: speech-to-text, then text into the LLM, then streaming tokens back into speech, then audio over LiveKit to the client. With Advanced Voice, the path changes: audio travels over LiveKit to the agent and then through a realtime API WebSocket to the GPU machine, where it goes directly into GPT-4o as audio. GPT-4o has a joint embedding for text and speech, performs inference on audio embeddings, and generates speech that returns through LiveKit to the device.

LiveKit began as an open-source project for connecting humans to humans through video conferencing and live streaming, but voice mode turned it toward human-to-machine links. Russ argues that if foundational-model companies are building the 'brain' of AGI, LiveKit is building the nervous system that carries sensory data in and results out. The metaphor explains why the company cares about cameras, microphones, speakers, and realtime transport rather than only model quality.

Russ frames humans as tool builders and AGI as 'tool builders building tool builders'—a synthetic human-like computer. If that computer behaves like a human, the argument goes, the interface should resemble human I/O: cameras as eyes, microphones as ears, and speakers as mouths. The keyboard and mouse are useful abstractions, but they are not how people naturally communicate with one another.

Russ is unsure what the office of the future looks like—whether work stays at a desk or happens through agents on a phone—but expects the nature of work to change drastically. He points to Jarvis-style interfaces and creative tools that become more voice-based, interactive, and multimodal. In that vision, humans act more like orchestrators or maestros, directing transformations on assets rather than doing all the mechanical shaping themselves.

The copilot-versus-agent debate will likely resolve as a hybrid, much like human workplaces. Some AI coworkers will own an entire business function and operate autonomously; other interactions will be close pairing sessions, design meetings, or multi-stakeholder collaborations. The difference is that more of those collaborators will be AI rather than humans.

Voice will not swallow every interaction, because people still text, read, and type into boxes. Text, voice, and computer vision will coexist, with video generation farther out. A restaurant example makes the point: a familiar order can be handled perfectly by voice, but a new cuisine may call for a hybrid interface with generated UI and a few taps rather than listening to an entire menu read aloud.

Russ has long watched the industry oscillate toward thin clients: devices powerful enough to render rich UI, yet users still face thousands of apps with inconsistent designs. At his previous company, he worked on pixel streaming mobile apps to pursue that thin-client dream. ChatGPT suggested a different endpoint: a single, familiar interface that could front many applications.

When people are not talking face to face, they often communicate through chat—texting, WhatsApp, Slack, Twitter—with a text box and a stream of messages. Russ understands complaints that AI chat interfaces feel monotonous, but argues that chat is already the native human remote interface. It could become the thin client: one UI that combines text, voice, and on-the-fly generated UI for any application.

Today's interactions are mostly mode-based: voice in the car or on the couch, text in a crowded space or inside VS Code. Russ wants more fluid blending. He recalls pair programming at Twitter for about a year and a half, an experience that mixed looking at a screen, asking questions, and letting the partner take the keyboard—a multimodal dance that AI applications have not yet replicated well.

Russ separates voice AI into emergent and existing-dollar use cases. Emergent examples include Gemini Live, Character.AI, Perplexity, and other voice interfaces that let people look up information, learn, or use the model as a therapist; these companies are pushing the interaction envelope. Anthropic's computer-use API is another sign of rapid capability exploration.

The lower-hanging fruit is telephony, where voice is already native and massive. Call centers, IVR phone trees, and any experience that starts with dialing a number represent existing spend and a margin-optimization game. Companies such as Sierra and Parloa are integrating AI into that space, targeting a telephone system that has existed for more than 50 years and carries billions of calls every month.

Latency is not the main reason voice automation has not fully taken off. Customer support already has systems that work at scale, so swapping the engine risks NPS and satisfaction; models still hallucinate and need evaluations and bulletproofing. Because the AI is imperfect, companies often need humans on standby to correct the agent or take over, which limits the fantasy of shutting off the contact center entirely.

The latency story has nonetheless improved dramatically. Early 2023 conversational latency was roughly four seconds; with GPT-4o's realtime API and LiveKit, Russ cites an average of about 320 milliseconds, close to the roughly 300 milliseconds of human conversational turn-taking. Faster can even be worse: a Cerebras-powered assistant around 100 milliseconds responded so quickly that it interrupted users, and people asked for it to slow down.

A phone-support agent must not only talk but update backend systems: move a record through Salesforce, mark a ticket, or use some bespoke internal tool. Those systems are often custom or obscure, so the ideal is for the AI to operate them directly. That is where computer-use APIs become relevant, though Russ says the industry is not there yet.

LiveKit has been building infrastructure for that future: cloud or headless Chrome instances that agents can control through a Playwright interface, loading pages, clicking buttons, and filling forms. At an AGI House hackathon, the team demonstrated a beta service where an agent built with LiveKit's framework connects to one of these browser instances and does work. Other companies build similar infrastructure, but LiveKit's twist is what happens when the agent gets stuck.

When the agent needs a password, cannot decide which item to choose, or otherwise stalls, it can stream the browser as video to a human client application. The human clicks on video pixels, but those events are replayed back to the cloud session, creating a shared interactive browser that the agent also operates on. The human can unblock or nudge the agent and then let it continue.

Russ describes this as giving AI the ability to touch. LiveKit had already helped AI see, hear, and speak; browser control adds manipulation of applications, much as a person touches a phone screen through touch events. He distinguishes this from humanoid robots touching physical objects, but the underlying idea is the same: extend the senses available to an AI agent.

Russ's recent attention has gone to telephony because of the sheer workload, including insurance eligibility lookups where hospitals call insurers to confirm coverage for procedures. Millions of these calls happen, and AI systems are beginning to automate them—sometimes with AI calling humans rather than humans calling AI. Longer term, he was waiting for a fully multimodal model like GPT-4o that can take and produce any combination of speech and text, and he is watching alternative architectures and on-device voice models.

The device/cloud split will matter greatly in robotics. Russ notes that humanoid robots such as Figure may do planning and reasoning in the cloud, while reflex actions, kinematics, and execution of plans run on the device. If a robot is walking and a car approaches, waiting for a cloud round trip before stepping back onto the sidewalk is not viable.

Russ uses a human analogy: no person carries all the world's information in their head, so people do 'cloud inference' by looking things up on a phone or calling support when a router reboot fails. A local model will not know the latest Linux spec or how to resolve every obscure issue. Some inference will therefore always happen in the cloud, while other inference stays on the device.

In an ideal world with no resource constraints, a device would parallel-path local and cloud inference and let the fastest correct responder win. Real systems also need records of what the model said, heard, or saw for legal reasons, training data, and error analysis, which pushes data to the cloud. Privacy-sensitive use cases may stay purely local, but even Apple Intelligence sends some work to a very secure cloud.

Russ calls Transformers overhyped and spiking neural networks underhyped or under-researched: he is drawn to the more brain-like, analog design for audio and video signals, while noting it is harder to train than Transformers. He has also changed his mind about moats. Because underlying models change so fast, LiveKit no longer assumes that unique data assets create a durable advantage; the better strategy is to be deeply embedded with customers and build quickly. Application-layer moats may look like traditional software—hard to build and hard to distribute—rather than one clever insight.

A second surprise is penetration speed: Russ expected AI to spread faster and deeper, yet he regularly meets people who have heard of ChatGPT—reported at more than 200 million weekly actives—but do not use it or contemporary AI in their work. Asked which AI startup excites him besides LiveKit, he names Tesla. He built a voice interface at an xAI hackathon that could control a Tesla through its official API, opening doors and honking the horn, then rode in his co-founder's car as it self-drove at 1 a.m. from San Francisco to Menlo Park. He calls the experience magical and a sci-fi-level demonstration of technology.

If he had to start an AI application tomorrow, Russ would build a video game with a novel way of interacting with NPCs. He imagines open worlds filled with lifelike, dynamic characters whose stories branch like an infinite choose-your-own-adventure, with voice as a default input. That leads him to revise his underhyped answer: voice AI in video games may be one of the most exciting near-future applications.

For listeners, LiveKit's work is mostly open source and available on GitHub at github.com/livekit, along with the company website and X account. Russ directs people there to follow the repos, tools, and projects. The company's open-source footprint is central to how it reaches developers.