Five billion minutes of calls: why Dialpad trained its own business LLM

Nathan Labenz opens 2024 by admitting that AI’s impact on knowledge work has moved a little less quickly than he expected. At retail prices, he calculates, a billion dollars buys only one or two GPT-4 API calls for each of the world’s eight billion people—a tiny toehold in global knowledge work. Even a hundredfold jump would still be one GPT-4 call per person per day. GPT-3.5-class models are effectively commoditized, but GPT-4 is different. In the West, he says, only Anthropic and now Google with Gemini are really in the ballpark, with Baidu’s Ernie 4.0 a worthy contender he hopes to cover separately.

He ticks through the other bottlenecks. Language models are weird, and the know-how to put them into automated workflows is still scarce; companies that would love to automate tasks nobody enjoys have to wait if they do not know someone who can execute. Agents have not matured as fast as he thought, partly because of GPU shortages. Decomposing work into subtasks helps, but stitching the chain back together makes it hard to know what information to pass at each step. Fine-tuning will help shape behavior, and he expects state-space models such as Mamba to deliver more coherent agents—if anyone can find the long-episode data those architectures will need.

That data, he suspects, lives inside the software platforms where people already work, which capture records and implicitly define the action space for a human or a hypothetical AI employee. That is why Dialpad interested him: a communications company with more than 30,000 business customers and a proprietary set of more than five billion minutes of sales and service calls, now used to train its own transcription and language models. Dan O’Connell, Dialpad’s Chief AI and Strategy Officer, is not expecting AI employees quite as fast as Labenz is. Given last year’s predictions, Labenz concedes O’Connell may well be right—and still biases himself toward shorter timelines, because those are the scenarios in which scouting the technology matters most.

Labenz did his homework: a free trial, a few calls to himself, a look at G2. He describes Dialpad as an almost-everything hub for the work sales and customer-service teams are actually doing. O’Connell smiles at the accuracy. The easy frame is a communications platform: voice, video, and messaging, on any device, anywhere, for internal team chat and for external sales and support. AI is built through that stack—capture a conversation, transcribe it, then ship features for assistance, automation, and insights.

On revenue they put themselves north of $200 million ARR, with about 30,000 customers split roughly a third each across SMB, mid-market, and enterprise. The logo list runs from Stripe, Uber, and HubSpot down to a two- or three-person law firm. These are cloud believers. Plenty of companies still run a PBX in a closet; Dialpad’s buyers already wanted managed communications in the cloud, which makes them quicker to try transcription and sentiment than to litigate wiretap law. Serving enterprise configuration and small-business simplicity at once is hard. Labenz wonders whether natural-language interfaces will let complicated platforms hide seventeen menus. O’Connell agrees, and he is especially excited about putting an LLM on top of analytics: conversational search over what he calls the last offline data set—the conversations themselves.

The fastest-growing slice is communications and AI for sales and support, but the vast majority of those 30,000 customers are also deployed for internal communications. Friendly competitors on the UC side are Microsoft Teams, Zoom, and RingCentral; on sales and service, Five9, 8x8, and Talkdesk. One consolidated product creates upsell paths from a sales org to the rest of the company. Power users in a contact center can live in the app, pulling Zendesk, HubSpot, or Salesforce into the same window. Dialpad wants to be the destination for the call or the meeting, then pull context from systems of record. That same capture layer is opening adjacent plays, such as guiding recruiters through a consistent interview and writing answers into Greenhouse.

The announcement Labenz reads from the blog is blunt: DialpadGPT, trained on five billion minutes of business calls and online interactions, billed as the world’s first business-focused LLM. In his head that is about 500 billion tokens at a hundred conversational words a minute, roughly five percent of GPT-4 training scale. O’Connell starts the origin story earlier. In 2016 he founded TalkIQ to build one of the first real-time speech-recognition engines. The industry pattern was to record a thirty-minute conversation, send the file to a third party, and wait another thirty minutes for a transcript. For sales and support, streaming was the prize: route the call from what the customer is asking, map sentiment live, guide the agent, track an agenda.

They started from open source, including Kaldi, and fine-tuned for long-form phone conversation—very different from asking Google or Siri the weather. A startup still needs distribution. Dialpad’s founding team had come out of Google and built Google Voice. O’Connell calls the combination peanut butter and jelly: TalkIQ could understand conversations, Dialpad could power them. He has been there five and a half years. Customers opt in to share data. The corpus is anonymized, stripped of personal identifiable information, and mostly long-form sales and support speech. When ChatGPT enamored the world, the training set and the distribution were already sitting there.

Foundation models still had problems: capacity, token limits, latency, cost. A transcript had to be broken up—six times, in his telling, later sixteen—before you could even summarize, and information degraded in the handoffs. Those are engineering problems that get solved over time. Dialpad did not think it could wait. So it trained its own model two ways: fine-tune strong open-source software with a team that already knew conversational neuroscience, linguistics, and labeling, and also build from the ground up, ultimately a smaller model specialized for industries and use cases. Labenz’s 500-billion-token triangulation goes unchallenged. The data is opted-in platform traffic, not a scrape of the public web.

They train in house, on their own A100s, with bare-metal operations—the same posture they take on the telephony platform. Many communications startups treat Twilio as a CPaaS and never build the pipes. O’Connell thinks the best businesses verticalize their stack, the way some companies now build silicon and the way Tesla does in cars. Owning the stack buys pace of innovation and cost advantage at scale, which is how you bring a better product to market faster. He is careful: he is not sure they would make every past decision if they were starting a company today. The proprietary data, the know-how, and the expert team are why this path worked.

They still plug partners into places: OpenAI for some features, Google Vertex and its Bison model for others. The vast majority is in house. Labenz is surprised, and a little worried for copycats. Plenty of companies will try Llama 2, tell themselves they are controlling their destiny, and get bogged down in cluster management and the “artisanal process” of shaping model behavior, only to watch OpenAI ship the next version before Llama 2 really works. O’Connell does not pretend there is a trick. The team has been in these technologies for more than ten years: over fifty people on AI, eighteen PhDs, sixteen patents, conference awards for models in development. Access to the data set is the other half. A decision six years ago to focus Dialpad on AI is what made the OpenAI drama two weeks before the taping a curiosity rather than an outage. Summarization was not going down because a vendor might cease to exist overnight.

As a technologist he wants to control his own destiny. Do not outsource the most important parts of the business. Right now he believes AI is the biggest opportunity, and doing it yourselves can be a better, more cost-effective experience—with a bunch of landmines. Speech recognition is the foundation he refuses to treat as a commodity. Accents, words that do not show up in a dictionary, proximity to the microphone: the problem is not solved. Investors who wave at a leaderboard should ask which training set and which network the number was measured on. Dialpad’s accuracy is on its own codecs and telephony network, not something you can drop onto T-Mobile. Transcription accuracy is garbage-in, garbage-out for every assist, automate, and insight feature that sits on top.

Asked how to trade off peak quality, latency, and cost, O’Connell offers Dialpad’s matrix. Number one is latency. The interesting features happen in the conversation, in milliseconds. Agent assist pulls from a knowledge base and puts a card on screen in real time. If that card is three seconds late it has lost its utility; at ten seconds there is no point building the feature. Number two is getting those real-time features into as many hands as possible, which is a capacity question the large models still struggle with. Number three is cost. If you are really providing value, you can recover the price, and margins tend to improve as you optimize. Dialpad has raised about half a billion dollars from firms including Andreessen Horowitz, GV, and ICONIQ, so cost is not the blocker. An earlier, capital-constrained startup might put cost first.

Quality is a threshold, not a knob they turn after the fact. They set bars for useful output, then dogfood: support, sales, and every employee use the models. If Dialpad would not pay for the feature internally, it will not charge customers. The loop that companies skip is feedback. Label anything AI-generated. Let users mark a recommendation, a summary, or an action item as good or bad. They ship on a bi-weekly cadence, retune, and redeploy. Labenz’s own fine-tuning work has shifted for a different reason: he used to stay on OpenAI because nothing else matched; now GPT-3.5-class fine-tunes look matchable on Mixtral, but OpenAI’s pricing and managed hosting still win, especially autoscaling. O’Connell notes that a communications platform has its own load shape. Meetings spike on the hour and the half hour, not at 11:17. Autoscaling is part of owning the stack.

The most-loved features start with real-time transcription. Summaries sit on top, and for support and sales they label the purpose and outcome of the call, turning unstructured speech into structured data that used to require people in a contact center reviewing recordings and tagging by hand. Agent assist is personal for O’Connell: his first job was in a contact center, and he was terrified of the first conversation because he did not know what the person would ask. Being able to pull the answer at the moment it is needed is the point. The last highlight is inferred customer satisfaction. Surveys capture the people who loved it or hated it. The model infers CSAT from the conversation itself, on every call, with no change in customer behavior and no new software, at something like a hundred times the survey volume.

Labenz asks the question every company should steal: of the work the AI does today, how much was a human already doing, and how much simply was not happening? Very few of these calls would have been transcribed without AI; some might have been summarized. O’Connell splits it. Quality assurance used to mean sitting next to someone, listening, and scoring friendliness against a rubric. A lot of work either left no record or was brutally time-intensive. Doctors’ offices and legal settings had handwritten notes. The applications they focus on are mundane tasks ripe for AI: transcription near human levels, often quoted in the 90-percent-plus band, then summarization and structure that either fully automate the task or augment it enough to free people. That is the first wave—work that was not getting done, or was getting done with a headset and a clipboard.

They start with a custom speech model for every customer, so product names and invented acronyms transcribe correctly. The destination O’Connell wants is a model per employee. Sarah, he notes, can be spelled several ways, and misspelling someone’s name in every transcript and summary is a fast way to make them hate the product. They have not solved per-person models, and he does not think anyone has at that granularity. A model for the customer’s customers is another unsolved layer. Labenz offers a small-scale trick that will not work at Dialpad volume: when cleaning a podcast transcript, put a proper-noun list at the top of the prompt. Company names snap into place for about a dollar on Claude. Fine-tuning a customer’s recognizer still needs enough audio and ground truth; he does not have the hour count on hand. For a language they do not support, they used to cut 15-second clips and pay labelers. Now they can plug in Whisper, Speechmatics, Deepgram, or Google, pick whoever is most accurate on their codec, and let opted-in usage become the training set. In a year or two they take it in house, cutting cost and raising accuracy. Low-demand languages may stay on pass-through forever.

Content generation is earlier, and O’Connell is cooler on it. The obvious communications play is auto-drafting the email that accompanies a post-call summary. He uses ChatGPT for blog-post drafts and sees utility as a starting point. For a two-sentence follow-up, users may spend as long customizing the draft as they would writing it. Labenz says that is a function of generation quality: fine-tune GPT-4 on a body of business correspondence and routine mail might clear a high acceptance rate. O’Connell is not bought in, and he would not make auto-email the billed wow-factor feature. Microsoft Teams Copilot and Zoom AI Companion are leaning that way. He is not personally convinced, especially if you then need every user to fine-tune an email model they do not understand. What does excite him is next-best-action. Because they sit on the conversation and the CRM, they can see which language led to a better business outcome and, the next time a similar call happens, tell a different rep to do that. Generation, in that frame, is a recommended step, not a ghostwritten letter.

Labenz pushes further. Dialpad both captures what sales and support people do and looks like a bounded environment in which to deploy and evaluate an agent—clearer than “go use a computer.” His Bezos-style test is what the customer wants: immediate response, 24/7, pause and resume at convenience, maybe switch modalities. ChatGPT already offers those things. A small restaurant at lunch rush does not even want to pick up the phone. He thinks a lot of sales and customer service ends up mediated by AI, not because he is racing to cut cost, but because the customer would choose it. O’Connell splits the vision. Digital deflection and next-generation LLM chatbots for support: he believes wholeheartedly. About 80 percent of people already start brand support on a digital channel. Automated outbound sales: he does not. At dinner with Marc Andreessen, the idea of two bots negotiating—each person with a personal agent—came up. O’Connell does not buy it. People like to buy from people. Even on support, once you escalate to a phone, he is not sure customers want a voice bot, however good, especially when they are frustrated.

He does not think indistinguishable voice agents show up in 2024. Demos are impressive; you can still tell. ChatGPT’s voice mode handles interruptions and long pauses poorly—Labenz found the model talking over him while he was still thinking. Labenz also does not want AI deceiving people; even a passing voice should be labeled. His worry is the other direction: if it is not 2024 but it is 2025, it is still close. O’Connell’s hesitancy is a trough-of-disillusionment instinct. Self-driving is a decade in. If you had asked him ten years ago, it would have sounded solved. Waymo cars cruise San Francisco and still are not “the problem is done.” Human language and everyday work workflows are the same kind of last mile. He hopes he is wrong. He has a sinking feeling it is harder than the demos.

Labenz hopes he is wrong the other way. Two years earlier, in December 2021, there was still no public instruction-following model. At Waymark he had just moved GPT-3 fine-tunes from Curie to DaVinci scale and gotten a merely useful script writer. Two years later the scripts are good, image understanding can pick from a user’s library, and text-to-speech has crossed the bar of whether a company that owns TV stations would put it on air. Most of what they broadcast is now AI voice. Those capabilities jumped from perhaps 20 or 30 percent of human to 80 or 90 percent, and he still sees low-hanging fruit. The standard, he argues, should be what humans actually do, not imagined perfection. Waymark’s users were not great writers, which is why they used lightweight software; the AI now beats the old human scripts, and a human editing the AI is a bit better still. Self-driving operators publish numbers that look safer than people, and Tesla Full Self-Driving already feels safer than some humans he has ridden with. The funny Chevrolet site bots that agree everything they say is a binding offer are an argument for limits, not for delay. Do not let the agent sign contracts or quote a final price. Do let it work, because human agents also miss handbook follow-ups.

O’Connell grants the yes: virtual agents exist today, digital deflection and accuracy are measurable, and contact centers already use bots to handle more complex, more natural workflows with fewer people. Marketing content generation is both opportunity and risk—the writers’ strike put that anxiety on the table. He agrees AI employees will show up; he thinks the last mile is harder and the timeline longer than Labenz does. Labenz’s closing reliability argument is that the economy does not need something much smarter than GPT-4 for most roles. It needs faster, a little cheaper, and more reliable. Transformers are episodic. They have amnesia between sessions, a giant pile of weights, a short context, and hacky RAG in between. They do not get told once that they did it wrong and then actually listen. He is betting on architectures with integrated memory, including the state-space models he has been watching. O’Connell agrees long-term memory is the next step. Enterprise records are partial or missing; a communications platform that records every conversation—if you want the record—can personalize from the last seven calls. That is the bet. Split a thirty-minute transcript sixteen ways, summarize each chunk, then summarize the summaries, and you feel the gap versus one request over the whole context. They cover a lot of ground, promise to bookmark the disagreement, and leave sales, support, and marketing as the three ripest fields.