A Custom Voice Used to Cost $10,000. Ten Minutes of Audio Now Clones It.

Nathan Labenz opened with a demonstration that used to be a boutique production job. Two years earlier, he said, a custom voice from a leading provider meant hours of recorded audio and a ten-thousand-dollar check. Using Play.ht, he cloned his own voice from about ten minutes of speech, at minimal cost, and in less than an hour could hear himself reading copy. He later sent Felfel the first interview question in that clone — the first time, despite years of shopping voice APIs for his company Waymark, that he had heard a machine speak as him.

Labenz is not a casual user. Waymark makes TV commercials for local and small advertisers, and he has spent hours testing every serious text-to-speech API on the market. The constraint is cultural as much as technical: no matter how small the budget, clients do not want ads that sound like they were made by a robot. He told Felfel that Play.ht’s “ultra realistic” voices are among the best products available today on both sound quality and ease of use.

The same intro previewed the dual-use problem that would dominate the back half of the conversation. Deepfakes, Labenz noted, are one of the few AI harms society actually predicted, and they are already here. Creators on TikTok have begun telling families to set shared passwords that impersonators would not know. Felfel, he said, has a multi-pronged safety approach, but builders can only do so much — and the technology is also, in Labenz’s phrase, easy, quick, and genuinely fun to make things with.

Labenz’s first impression of the product was that it wrapped big-tech text-to-speech APIs — Google, Amazon, maybe IBM or Microsoft — and resold them through a friendlier interface. Felfel said that was essentially right. The company started on third-party APIs, and IBM Watson was first, ahead of AWS Polly and Azure. He and his co-founder, both software engineers, had spent a couple of years trying about ten ideas that all failed, then took a break.

The spark was personal. Felfel kept two Audible subscriptions and wanted articles to work the same way books did. Medium was booming with writers, he was reading it daily, and there was no good way to listen while running. Most voices on the market were too robotic to consume for an hour; IBM’s were still Siri-like, but tolerable. He built a Chrome extension for himself. Someone found it in the Chrome Web Store, posted it to Product Hunt, and users poured in asking for mobile apps.

Consumer monetization stalled. People wanted to listen more and pay less, and the founders did not want to fill the product with audio ads they themselves found annoying. The useful signal was in who showed up: writers and publishers who already had audiences, asking whether their readers could listen too. Play.ht pivoted to a B2B “audio articles” product — the listen button now common on publication pages. Growth was real but smaller than hoped. Felfel’s diagnosis was blunt: the job was vitamins versus painkillers. Writing the article is the painkiller; adding audio is a nice-to-have nobody searches for.

Under the audio-articles surface, Play.ht had built an editor that exposed the full power of the cloud APIs, including visual SSML controls. Customers used it for more than articles. When Covid hit, learning-and-development usage rose. The ceiling was still quality. Generic AWS and Azure voices were fast and “good enough” for IVR-style jobs, but they were still robotic, and they were not built for the use cases Play.ht wanted to serve.

High-end work made the wall obvious. Game studios, Felfel said, use text-to-speech only as scratch audio while a game is in development; production still hires voice actors for feeling and expression. Big publications that care about brand — he mentioned outreach to places like the *New York Times* — rejected the quality as not good enough for their readers. Labenz’s ad buyers were making the same point from the other side of the table. The traditional TTS market of L&D and phone trees was not the prize. The prize was the human voice market: voice-over actors, games, media production.

That left a binary choice: sell the company and do something else, or solve quality. They chose to train their own models. Five months after launching the in-house voices, those models were already serving about 60 percent of usage, with millions of conversions a month. The older APIs did not shrink; they kept growing, mainly because they cover roughly 130 languages and accents while Play.ht’s own model still spoke only English. Users, Felfel added, do not care whether the audio comes from a house model or an API. They have a job to be done: good-quality audio, fast enough to ship.

Becoming an AI company was not the original instinct so much as a delayed one. Felfel and his co-founder had always wanted to train and deploy their own models; using AWS felt like the unnatural path. The problem was that off-the-shelf open-source systems — FastSpeech, Tacotron, and the rest — topped out at roughly API quality. Paying for an in-house ML team to match Polly made no sense. For two years they called the author of almost every paper and open-source TTS project and tried the code. Nothing was a breakthrough.

Early 2022 changed the bet. DALL-E, GPT-3, and diffusion models had shifted vision and language toward self-supervision, and Felfel argued speech should follow. Classic TTS was supervised on, say, twenty studio hours of one speaker, so the model learned *Nathan’s* voice, not how humans speak. A new accent or emotion meant new labeled recordings; cloning was nearly impossible. Self-supervision let them train on hundreds of thousands of speakers — Mozilla’s Common Voice, public-domain audiobooks, audio from the internet — without needing studio-clean labels.

The first model was only about 300 million parameters, medium by today’s standards. Quality came from the data. Then the team saw something they had not planned: the same voice, given “oh no,” came out low and sad; given “oh my god,” it lifted into excitement. Training pairs had taught the model that those phrases live in different emotional contexts, so it extracted style from text alone. Generation, though, was unusable at first — about twenty minutes of compute for one minute of audio. Six months of performance work brought it near real time, with a streaming API expected within weeks of the taping.

Architecturally it is a Transformer pipeline, not a single trick: text goes to mel spectrograms, a compressed audio representation, then a vocoder turns those into waveforms. Felfel compared the idea to Microsoft’s VALL-E, trained on some 40,000 hours, and Google’s AudioLM. Two hard problems followed. Language models are non-deterministic, but speech needs stable pronunciations for company names and acronyms. And there was no off-the-shelf stack for training large voice models, so they built data loading, clipping, and multi-GPU training from scratch. Platform voices are not random samples: the base model can emit on the order of a hundred thousand voices, but Play.ht hires actors for one to four hours of audio and fine-tunes copies for peak quality.

The new voices unlocked work the company had taken a risky bet to reach. Felfel said South Park Studios was making a new episode that would use one of Play.ht’s voices for a character — “not being used in protection,” as the ASR put it, but in actual shows. Play.ht also powers the audio on Reid Hoffman’s *Chats with GPT*. Gaming is the more structural shift: protagonists get actors, NPCs get text bubbles, because hiring a hundred voice actors to record every line does not pencil, and free conversation with those characters is impossible even with unlimited budget. Multiple game studios, he said, want fully dynamic experiences; a well-known streaming service wants the same for podcast listeners.

Creators arrived in two patterns. Non-native English YouTubers use British and American voices as if they had hired accented staff. Others clone themselves so a script can ship in minutes instead of two studio trips a week. Intensity surprised the team: some users told Felfel they spend twelve hours a day in the editor, building full audiobooks and podcasts, picking characters and designing how dialogue hands off. Play.ht had expected a quick-replace tool. People were treating it as a craft surface. One user’s extra minute of latency, multiplied across a twelve-hour session, is a ruined feedback loop — which is why performance work never stopped.

Complaints flipped in a way that first confused the team. Users no longer said the voice sounded like a robot; they said a word’s stress was wrong, a pause was missing, a line needed more excitement. The reference class had become human actors. Labenz’s picture of the next product was not SSML tags but a director on set: nobody hands an actor markup, a director says speak it like a grandfather seeing a grandchild for the first time. Felfel agreed. Creators would describe the speaker, generate, iterate — the image-generation loop applied to voice. A newer, larger in-house model was being trained to add emphasis, pauses, and styles (narrative, commercial, instructional, meditation) while keeping the same speaker.

Non-determinism is a split: audiobook users want drop-in-and-convert reliability; other creators want several takes, the way image users pick among samples. Those choices are training signal. The team is freezing phoneme-level pronunciations for acronyms and odd company names while leaving room for human-like variation elsewhere. Voice also has knobs images lack: how tightly a clone should match the original, how much accent to keep, even a gender-adjacent slider that can make a male voice a bit more feminine without turning it female. Cross-language cloning was an accident that worked. Upload a French speaker and you get English with a French accent; thirty seconds of Japanese-accented speech yields an English voice that sounds like a Japanese person speaking English. Internally, Japanese and Portuguese voices fine-tuned on about ten hours of audio already beat AWS, because the base model already knew how to talk like a human. A from-scratch multilingual model is next, aimed at keeping one designed voice consistent across languages for dubbing.

Compute, Felfel said, is not the inference bill. Users pay for generation, the model is optimized, and their stack can load and unload clones on the fly so a single GPU holds hundreds of voice models. The expensive line is training and, even more, the small experiments around it — weeks of Japanese and Portuguese fine-tunes, phoneme work, control features. The large model they were finishing had cost hundreds of thousands of dollars, not the millions of a frontier LLM, still a lot for a startup. Three years of customers and “decent revenues” were covering it so far.

Labenz pressed on a plateau unique to this domain: once you match a human voice actor, what would superhuman even mean? If quality levels off, do the models become a commodity, even open source? Felfel split the question. Base models will see diminishing returns, the way speech recognition clustered around 95 percent accuracy. In a year or two, an open-source or API model of similar quality is likely. His defense is not a forever-better foundation model. It is vertical models on private data you cannot scrape: there is no public dump of 10,000 or 50,000 hours of customer-support or sales calls, and Play.ht is already partnering with companies that hold that audio.

Enterprises add another lock. Production studios want actor clones for post-production and cannot risk a leak, which points to on-prem or private deployments. Felfel offered a parallel: GitHub Copilot had been on the market about two and a half years, and almost no FAANG-scale company used it, despite the time savings, because they feared code would leak. Application layer is the other moat. An Amazon team used Play.ht even though AWS Polly is a sister product, because they did not want to hire a developer to build a solution. If OpenAI shipped a voice API tomorrow, he argued, companies would still buy a UI that lets them start working. Labenz noted OpenAI is already playing both sides of that stack: API prices around two dollars per million tokens, functionally free for many jobs, and seven-figure contracts with companies that have data they will not mix.

Cloning is already a wow feature and, Felfel agreed, on a path to controversy. Users have sent clips to spouses who did not realize the voice was AI. Internally, Play.ht hooked ChatGPT to its voices and had the system make outbound sales calls — their human SDRs’ scripts were “hey, are you interested in this product,” and the team wanted to see if generation could do better. Nobody on the other end figured out it was an AI voice. They considered posting samples on Twitter, then stopped. The technology was already good enough to be dangerous.

The danger is not theoretical. People are already using other services to make fake TikTok ads in Joe Rogan’s voice. Felfel had just read about a family defrauded by a caller impersonating a son in jail who needed money. Before opening cloning, Play.ht spent a couple of months on calls with every applicant, asking what they wanted the feature for — YouTubers and podcasters, mostly — then kept a strict manual review that did not scale. They opened self-serve only after adding harsh moderation on the studio and API, accepting false positives: better to block a user by mistake than let someone be harmed.

They also shipped a public, free classifier on the website that tries to tell human audio from AI audio, hoping social networks, banks, and voice-login systems will pick it up. It catches more of Play.ht’s own output than third-party TTS, Felfel admitted, and it is a first version of a hard problem: Transformer vocoders leave traces, but a universal detector is extremely difficult. They rejected a celebrity-voice blacklist. Cloning Joe Rogan is bad; the worse case is cloning an ordinary person to drain a bank account, and you cannot put all of humanity in a database. The practical layer is watching what users *write* into the generator. Labenz, who has red-teamed language models, offered the analogy: asking for someone’s mother’s maiden name is itself a signal. Misses get fed back so the moderator improves.

Society, Felfel argued, is not equally ready across media. Photoshop has trained people for a decade to doubt images. A call from your brother does not prompt “is this real?” as the first thought. Over time, every medium’s default reaction will become suspicion, including lip-synced video of public figures saying things they never said. He paraphrased Sam Altman: technology may need to move just slowly enough for people to adapt. His own fear sits on the legal side. Stability AI was being sued; he hoped regulation would not force a winter onto responsible markets while less careful ones kept shipping. Internet platforms, he noted, scaled in part because they were not treated as publishers of every user post — a contested bargain, but one without which Facebook and Twitter would have died in court.

In the rapid-fire close, Felfel’s daily stack was practical. ChatGPT summarizes long emails and writes code, including translating Python to JavaScript; he iterates in the chat, then pastes into an editor. Copilot remains useful when starting something new, as completion rather than as a colleague. Grammarly still cleans up email, and he called it one of the biggest AI tools from before ChatGPT, even if fewer people brand it that way. Chat-based coding, he said, feels like working with an intern: you give a task, it comes back, you push another round. A year earlier he would not have believed he would work that way.

Asked whether he would take the million-and-first Neuralink implant if it offered thought-to-text, he said he might try it *before* the million, provided it was safe. Typing is a bandwidth bottleneck between thought and the machine; a working implant would widen that pipe and, he thinks, let people who have it pull ahead. Zooming out on the decade, he hoped society would get time to adapt and that the tools would make people more productive. What he does not want is a legal chill that slows the companies trying to be careful while everyone else races. The line he kept returning to is the one that frames the whole episode: this is the first time the human voice has been automated, and trust rules have to be rewritten on that fact.