Li Mu: Adding Ten Million Hours of Speech to a Text LLM, Open-Sourcing Higgs Audio

What happens if you add ten million hours of speech while training a text LLM? Li Mu opens with that question, then plays a few clips the model generated: a multi-speaker audiobook with a soundtrack, a flirtatious back-and-forth, and a parody in Teacher Ma's voice that roasts speech models as worse than Bilibili's remix section, only to get clapped back with 'if you're bad, train more.'

He says that last year they kept their eyes on text LLMs: make the IQ high enough, follow instructions, play games with you, help write copy—in other words, read and write. This year they want the model to listen and speak as well. Speech is a relatively old field in AI, and he is not a speech expert. The novice idea is simple: don't train a separate speech model; add a huge amount of speech data while training the text LLM, and let scale work miracles.

The goal is to keep the text model's IQ from dropping as much as possible, while also picking up the ability to communicate in speech. This episode shares the work in progress, the technology behind it, and the pitfalls they hit, and he hopes the open-source model is useful. First up is architecture: how to add speech into a text language model.

He starts by reviewing how a text language model works. At heart, given the words so far, it predicts what comes next. Ask 'What is the capital of China?' and the model first appends 'Bei', then, after the question plus 'Bei', predicts 'jing'. In real use you usually don't let it continue the text freely, because that is too hard to control.

More often, an interaction is split into three parts. The system tells the model what to do—answer a question, or switch to chatting or writing a passage. The user gives the actual task, the question to ask or the novel to write. Then comes the model's reply. You can stack more user turns and replies into a multi-turn conversation.

Getting the model to support speech in and speech out is actually just as simple: give it a new task. The system command can be 'convert the following text into speech', the user provides the text to convert, and you want the model to output the matching speech data in its reply. You don't need a separate interface; it is still the same conversation format.

Why reshape a language model this way? Speech recognition and speech generation used to be separate models, doing only text-to-speech or speech-to-text. They have talked about Whisper: it is also based on Transformer, but it only does one job. The downside of folding everything into a text model is obvious: to gain speech ability, the model will be larger than a pure speech model; and adding speech data to the same text model will likely lower its IQ.

The upside is just as clear. He gives a simplified professional recording scene: the director tells the actors they are about to record Xiao Ming and Xiao Hong arguing; Xiao Ming is impatient, Xiao Hong is shy; only then come the lines, possibly with stage directions and sound effects. A professional actor does not just read the sentences—they have to fit the character and the scene to pull off the performance.

Last-generation TTS models struggled to understand a brief that complicated. With a language model behind it, it is much more likely to grasp that you are asking it to do something complex—the way people have always used text models: give a long brief, spell out A through G, and ask it to make that. Training tries hard to make the model follow human instructions. If the text ability carries over, he also wants speech output to support that kind of instruction. That is the first point: understanding complex instructions.

The second point is that plain text-to-speech is no longer enough. You might not just generate a clip of speech; you might want the model to write a song, sing it, and have the accompaniment keep up. The understanding side is similar: the simplest job is pulling spoken words out of the audio; a step further is analyzing what is going on in it.

People can tell, from sound alone and without looking at the picture, what kind of person it is, male or female, roughly how old, what the speaking situation is—whether they are arguing, talking, or teaching a class. From ambient sound you can even guess indoor or outdoor. Speech is full of signal; humans understand the whole context. With a text language model, you can do this more complex understanding and reasoning.

A step more complex is the voice chat people have already played with: going back and forth with the model. He wants the model not to reply mechanically turn by turn, but to understand your current mood and express emotion in return. Latency has to be low enough too: not wait a second or two after you finish a sentence, but interact face-to-face, on the order of a few tenths of a second.

All of these tasks can be unpacked into system, user, then a reply, and handled uniformly by a language model. One relatively fixed, simple recipe, for every problem. Then you add more data and more compute, and let the scaling law work miracles. That is why they chose this model that looks more complicated.

Next come two how-tos. First, how to get the model to understand and produce the entire speech or audio signal. Second, what kind of data you have to construct at training time so that speech signals and text signals fuse, and the model understands what this stretch of speech is actually saying.

He first reviews how text is represented in a language model: with tokens, or word pieces. A Chinese character or an English stem can each be treated as a token. A passage of text becomes a sequence of tokens; that sequence is what the language model sees, and it then outputs the next token.

Tokens are discrete. There is a vocabulary that tells you there are tens of thousands of them. The reason is that a language model's output is a softmax, which is multi-class classification: each step, pick one item from the vocabulary as the output. Speech is awkward because it is a continuous signal. How do you represent it as discrete tokens?

The simplest approach: slice one second of speech into 100-millisecond chunks, which gives you 10 pieces. Then assume that, at a fine enough grain, the world's sounds keep repeating—Chinese only has so many syllables—so you pick some representative common fragments as templates, and represent each small chunk with the nearest template.

He uses 45 such fragments as templates. One second of speech comes in, gets cut into 10 small chunks, and each chunk finds the closest of the 45 templates. That second becomes a sequence of length 10, and each position is just an index—say the first 100 milliseconds used template 5. That way, the language model can handle speech the same way it handles text, on both input and output.

Of course it is not that simple; speech is not easy to represent. He gives a set of numbers: one hour of speech stored as 128kbps MP3 is about 60MB. 128kbps is only a middling level—for music you might want 192 or even 320, or lossless. So this is already a so-so MP3.

If you use tokens, assume a vocabulary of 64,000, already close to a practical dictionary size; and 24 tokens per second of audio, which is also a common practice. Each token only needs to store its index, which is log2(64K). So one second of audio is about 384 bits, and one hour compresses to about 0.16MB—another 375x compression on top of 128kbps MP3.

Compress it that far and the quality is bound to turn to mush. Getting the tokenizer right is a critical technique. He asks: when you are throwing away almost everything, should you preferentially keep the acoustic signal—pitch, for example—so you can restore it at generation time; or try to keep the semantics, meaning what this passage is actually saying?

Their finding: you should preferentially keep the semantics. A little acoustics is enough; keep some of the speaking style and restore the rest later by other means. Semantics, though, vary endlessly: the same voice says different things in different scenes. Pack as much semantic information as you can into the semantic tokens, so that speech tokens and text tokens link up early, and the model can switch fluently between text and speech.

This episode will not go too deep on speech tokens. If you are interested, look at the code they released, and the comparisons with other people's tokenizers. The point is: you have to represent and compress the speech signal well, or the audio you generate or process later will not satisfy you.

Once sound is turned into tokens and fed into the model, the next question is how to understand and generate. Speech-to-text or text-to-speech is, at heart, a conversion between modalities: getting the language model to map the speech representation and the text representation of the same thing onto each other. Since they are already using a text LLM, which is strong at text, you can try to map speech semantics back onto text as much as possible, so you can use the model's large bag of abilities in text space.

Someone might ask: for a real-time voice assistant, it sounds like I say a sentence and the model replies with a sentence—can't you just work in speech space? With today's technology, you still go back to text space. Persona, what to do and not to do, is controlled in writing. In live interaction you are also likely operating in text space: ask what the weather is today, and the model may first search in text, then map the result back to speech. The core is connecting text and speech, not fencing off a separate speech world.

Simply put, the model has to understand that the text tokens and the speech tokens for a concept are just two expressions of the same meaning. With that goal, you can talk about what data you need and how to represent it. As for data, more is always better.

Did you guess whether they used Bilibili or YouTube data? The answer is no. Not because the data is bad—that data should be pretty good—but the sites said crawling is not allowed. To stay as clear of copyright issues as they can, they either buy, if the price is not too high, or look for audio files on pages that allow scraping.

Data grabbed this way can of course be very bad; you might have to throw away 90% to keep a little that is usable. To put together ten million hours of training data, you may first have to scrape about a hundred million hours. Only after you have the data do you get to how to feed it into model training.

He gives a conversation example. At training time it is expressed like this: at the system level you write the acoustic features of this audio, what they are talking about semantically, who is there, and how those people speak—a description of the scene. The user input is the text to be spoken; the model outputs the matching audio. The task: given the scene description and the text to generate, output real speech that fits the scene.

How do you get those labels? If you are a student doing research, he recommends handing the speech to OpenAI's GPT or Google's Gemini for labeling. They cannot do that, for two reasons. First, those companies explicitly forbid using model outputs to train another model. Second, the cost is too high: what they have to process is not ten million hours at all, but perhaps over a hundred million hours dumped in; they cannot afford that API bill.

Their approach: with the same model architecture, train a separate speech-understanding model. At generation time you tell it the scene and what the user is supposed to say, and it outputs speech. Now reverse it: the user gives you a clip of speech, and you analyze the scene, who is there, what they are like, what they are saying, what the mood was, and output all of that. Equivalently, swap the generator's inputs and outputs.

He uses an analogy: if you want to teach disciples fist-and-kick kung fu and cannot teach it all at once, you first teach two disciples, one who can punch and one who can kick, and have them spar every day. As they fight they improve together, and in the end both learn fists and kicks. That brings the story full circle. The trained model's code is released in the boson-ai/higgs-audio repo on GitHub.

The repo has links. If you do not want to install, you can play on the playground, including Hugging Face's playground: it has to run on a GPU, but each person seems to get three free minutes a day, and clicking it calls Hugging Face's own GPU. He plays a demo line: 'The sun rises in the east and sets in the west.' This channel, of course, still wants you to get your hands dirty, so he jumps to install.

If you already have GPU PyTorch, just follow the instructions to install. If not, he recommends the Docker image NVIDIA provides: locally you only need the NVIDIA driver and Docker, and you skip the hassle of installing CUDA and PyTorch. They are still using that 3090 from three years ago.

First git clone the repo, then run the PyTorch Docker, mapping your home directory in. Download and extract take a long time. Once you are in the cloned directory, install directly—another long wait. He complains that people write messy dependencies when they publish packages, all tangled up with each other.

Once it is installed he goes back to the README and copies a sample to run. The example uses belinda's voice, temperature 0.3. Want it more stable, lower the temperature; want richer expression, raise it. The text to generate is the same sentence as in the playground. Copy it, run it, and you wait again for the model to download; if you have a GPU it will run on GPU by default.

You can listen to the generated audio. Look closely and what it is doing is this: under the example voice prompt, it finds belinda's text and speech, uses the text as the previous-turn user input and the speech as the previous-turn system output; you then give a new stretch of text, and it can output speech that matches this person's voice. Every speech clip shown earlier in this episode was generated this way.

The homepage has something even wilder: taking one person's voice from one language into another. In the demo, English asks 'Any good places that I should go?', and the same voice chats in Chinese about going to Seoul next week and how long to stay. By this point you can probably guess why this model's voice cloning is so strong—even though voices like these mostly come from videos on sites like Bilibili or YouTube, the model has not seen them at scale, because training never used those videos.

What they crawled is pure audio, and the statistical distribution is quite different from video data. There are two reasons. First, having seen that much data, they have more or less seen every kind of voice. Second, look at how the data is constructed: when doing voice cloning, they turn the entire clip used for cloning into the previous-turn user input and system output.

In the middle of the training data there is a stretch of continuity, made of many turns. Given the previous few turns of conversation, the next turn should be continuous in meaning and in scene. So the model can pull from the cloning reference you provide the whole scene, the speaker's characteristics, and some semantics, then match and continue them onto the new text you give.

That also suggests: when you want to generate text for a particular scene, it is better to find speech of that person talking in a similar scene—arguing, relaxing, laughing out loud. It is the same for people: if you provide a voice of someone speaking in dead earnest, then ask to imitate that voice doing a talk show, it is definitely harder.

If you cannot find a perfectly matching reference, there is a fallback: edit the system message. In the system, describe the voice scene you need cloned, and hope the model combines the voice you provided with the system instruction to synthesize what you want. In his view, that is how prompt engineering should work for this kind of multimodal speech-text LLM.

That is the end of this episode. This is a new direction. They wanted to share interesting problems they ran into while building the company, the work they have done, and what they have learned, and they hope it helps. Finally, he welcomes everyone to try the model they open-sourced, and hopes it brings you some joy.