A Million Bytes and 256 Symbols: Meta's MEGABYTE Wants to Throw the Tokenizer Away

GPT-3's vocabulary still contains a Reddit username. The speaker's posting style was distinctive enough that byte-pair encoding treated the handle as a frequent string and locked it into a list of more than 50,000 tokens. The model now "knows" that string forever. Other rare tokens, Labenz notes, famous models seemingly cannot be made to say at all. When Andrej Karpathy forwarded Meta's MEGABYTE paper, he used that kind of weirdness as Exhibit A and summed it up in one line: everyone should hope that tokenization could be thrown away.

In May 2023, Meta AI researcher Lili Yu and colleagues released *MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers*. A month later she joined Nathan Labenz on The Cognitive Revolution to walk through the architecture from first principles. Tokenization, Labenz reminds listeners, is the hack that lets today's models survive: inputs are chopped into a fixed vocabulary before they ever become numbers, and every prediction is chosen from that same set. Image "tokens" are a different flavor of the same idea, little squares inside a larger picture. The reason it exists is ugly but practical. Sequence length is expensive, GPUs are scarce, and without some first-pass compression the compute bill runs away.

A few numbers from the conversation are worth keeping in view. Flatten a 600-by-600 image and you have a million bytes, a length current transformers simply cannot support. A byte has only 256 possible values, two to the eighth, eight zeros and ones. A million bytes of text is longer, for practical purposes, than Claude's 100,000-token context; a million bytes of audio is about a minute of music. MEGABYTE's pitch is that if you can model at that scale, you no longer need the tokenizer, and the same byte stream can be text, images, or sound.

Before GPT-3 or GPT-4 reads a word, it cuts the string into pieces from a vocabulary of more than 50,000 words, word parts, numbers, and symbols. The model never sees letters. It only sees those chunks. The industry standardized on this because packing frequent strings into one token shortens the sequence and cheapens the compute. Yu is blunt about why people did it: nothing is free, and you cannot just swallow a raw input this long.

The costs are already familiar in text. Spaces are a chronic headache. Prefix a prompt with a random jumble of letters and the tokenizer can emit something the model then continues as nonsense. Change domain and the vocabulary goes out of distribution: take a weaker model into biology, chemistry, or a language it was not built for, and fine-tuning struggles because the tokenizer itself is the mismatch. The rare-token problem sits on top of that. Yu says those are the headaches people already feel. Multimodal work makes them worse.

"Text tokenizer gives you a headache," she says, "but the image tokenizer can actually give you a nightmare." The difference is loss. Tokenize a string and detokenize it and you get the original back. Run an image through something like VQGAN and you get a slightly different picture: colors off, details gone, fingers mixed up. Audio is lossy in the same way. Diffusion models can eat pixels and skip the tokenizer, but they are expensive. Autoregressive image generation has to face the million-byte sequence. Tokenizers were introduced, she argues, for one reason: existing architectures could not handle raw length. MEGABYTE is meant to kill both birds — remove the tokenizer, and supply an efficient multiscale transformer that can support it.

Labenz describes the diagram as a fork: a global model in the middle, smaller local models branching off it. Yu accepts that picture with a correction. This is not a set of agents chatting in high-dimensional space. Everything is bytes, everything is causal, and the local models follow one another even when their predictions can be computed in parallel. In the paper's figure, a string of 16 bytes is grouped into four patches of four. A patch embedder, a simple concatenation, compresses each group of four bytes into one vector. The global model then sees a sequence of four instead of 16.

Shorter sequences save money twice. Attention memory scales with the square of length, so that pressure eases. Yu stresses that attention is not the real bill. In GPT-3, attention is only two or three percent of compute; the feed-forward layers are the expensive part, and they scale with sequence length. With a patch size of eight, the global transformer needs one-eighth the flops of a naive transformer, which is budget you can spend on a bigger, more powerful global model. After the global pass, outputs fan out to local models. Each local model owns its patch and predicts byte by byte. The local models do not talk to each other, so they can run in parallel.

Inference is more sequential than training. Prompt patches the user already provided can be processed in parallel; anything the model must generate still has to proceed patch by patch, and byte by byte inside a patch. If you only need the last patch's output, Yu confirms, you can ignore the earlier local heads after the global model has run. The byte vocabulary itself is tiny. Two hundred fifty-six symbols map into an embedding of dimension 32, a module so small the team barely optimized it. Position embeddings are split: one set tells the global model which patch it is in, another tells the local model which byte sits inside the patch. The embedding matrix for the bytes themselves is shared.

Both the global and local models are intact dense transformers. Yu says that is by design. Dense transformers are the architecture that has been tested at large model, data, and compute scale; if someone later finds a better attention mechanism, you should be able to swap it in like a package. During training, every byte gets a prediction, so 16 bytes produce 16 losses that are then averaged. Labenz had been under-weighting that fact. Yu pushes it further: encoder models such as BERT only put loss on the masked 15 percent of tokens in a forward pass, while a decoder-only model gets a loss on every token. She personally thinks that dense learning signal is a big reason decoder-only models became large, powerful, and uniform. Predict every byte, and the data is used far more completely.

The results come in two families. First is a compute-matched bake-off. Same dataset, same GPU hours, three models: a naive transformer, Perceiver AR, and MEGABYTE. Yu explains why the match matters. Bolt a local model onto a global one for free and of course it gets better; that is not an interesting claim. The real question is opportunity cost. If you add a global model, you have to make the local model smaller because the compute budget is fixed. Run the three architectures on the same food and MEGABYTE, at the byte level, leaves the other two behind.

The second comparison is against published models on text benchmarks such as PG-19. Here the team is honest about an awkward gap: many papers never report how many GPUs they used or how long they trained. The only shared constant is the dataset. Meta trained until validation loss stopped falling, then compared that number with whatever other papers reported. The conclusion is that MEGABYTE is comparable to standard token-level models, while still collecting the tokenizer-free advantages. Yu is careful not to overclaim. Byte-level MEGABYTE is, they believe, the best of the byte-level setups they ran. Whether byte prediction as a language-modeling task can fully replace a tokenizer is the question they are working on now, including rescoring byte outputs as if they had been delivered in token form.

Images and audio show what "just read the bytes" is for. On ImageNet 64 they match the then-state of the art. They also ran 256-pixel and roughly 600-pixel images; flatten the larger ones and you are back at a million bytes. Audio is even more telling. Sixteen-bit audio has 65,536 possible values per sample, so earlier work either classified down to 256 bins or built hierarchical decoders because a 65,000-way softmax was too expensive. MEGABYTE ignores the file's official encoding. It reads the file as a byte stream and models that. Yu says that is the direction the team is excited about: in the future, model the raw format of any file, any input, any modality, without caring whether it is image, audio, or video.

Patch size is still mostly a heuristic. For text they pick eight, close to the roughly four bytes in a typical BPE token, but more compute-efficient. For images they follow the image-tokenizer habit of 8-by-8 or 16-by-16 patches. Both extremes collapse back into an ordinary transformer: a patch of one, or a patch equal to the whole sequence. The encouraging result is that the model is fairly insensitive to patch size. A wide range of values works, even if a more systematic study is still needed, especially if one later wants local models as strong as a 2,000-token transformer sitting under only eight or 16 patches.

Online reaction to the paper was almost uniformly glowing, but one sober question stuck with Labenz. If you localize information early into patches, will the model still show the in-context, few-shot behavior people associate with vanilla transformers? Yu answers on two levels. Patch boundaries do cost something. The team tried a convolutional layer and saw only a slight improvement. Her bet is that with enough data the model should learn invariance across those boundaries, and that scaling is the real answer. In-context learning she is less worried about, because it should show up in perplexity: if the model predicts the next byte well, in-context learning should not be a problem.

Labenz also likes the modularity. A global model that understands everything could, in principle, grow swappable local heads, different hands on the same brain. Yu agrees. Video-language models already do something similar, bolting image understanding and even image generation onto a pretrained text model. The architecture, she says, is quite similar in essence.

The test that would actually change adoption is scale. Yu is blunt: nobody cares if you train a model under a billion parameters on fewer than 100 billion tokens. The team is trying to train with compute similar to LLaMA and match LLaMA's performance without a tokenizer. The methods are open-sourced. If that works, she thinks wide adoption could come remarkably fast. Labenz notes the paper went up on May 19 and they are recording about a month later; given the size of Meta's clusters, the answer should not take forever.

Two domains she wants to watch closely are math and code. Long integers get smashed into two- and three-digit tokens, so a model is doing arithmetic through a dirty window. On code, a colleague's in-house tokenizer produced about 30 percent fewer tokens than GPT-2's BPE, which means the GPT-2 vocabulary is 20 to 30 percent less efficient on code, and extra spaces only make the mess worse. Labenz connects that to reports that later OpenAI models mixed a lot of code into early training, perhaps because code's logical structure helps every task. Yu adds Meta's Galactica lesson: a science model trained on arXiv needed three or four special tokenization rules for LaTeX and related markup. Users barely feel it. Developers have to maintain the tangle. A byte-level model deletes that layer.

Zooming out, Labenz asks what it feels like inside Meta AI during a run of high-profile releases. Yu answers from her own corner: she also works on multimodal language models, and MEGABYTE came out of that bet. The team believes the future belongs to mixed-modality, unified models. The world currently has ChatGPT for text and Midjourney for images; why not one model that does everything?

Safety is not ignored. Meta has responsible-AI teams, and data, bias, and the open-source process all go through strict review. Every open-source model has to pass bias filters and a stack of questions. Researchers complain every day that using data or releasing a model is hard, she says, but that is the world they have to live in. Her own view is that responsible AI should not be left to a handful of companies. "We should open source it and everybody help debug, help develop more safe model."

That stance sits inside a split Labenz sees forming. OpenAI publishes far less now; if OpenAI had invented MEGABYTE, he doubts it would be public. Researchers who want open science go to places like Meta. People who want to ship a world-changing product, or who think the work is too dangerous to share, go elsewhere. Yu agrees the polarization is real and, once the industry can prototype and productize quickly, somewhat unavoidable. She also flags a quieter risk that has not hit Meta yet: as research and product work look more alike, GPUs flow toward customer-facing teams, and a pure research group can lose compute. So far, she says, so good.

Near the end Labenz asks about the US-China overlay, given that Yu grew up in China, finished university there, and now publishes from the middle of both debates. She refuses the two-country frame. A year or two earlier, she notes, Israel and Japan had rules under which an AI model could train on any data. She can imagine startups incorporating there for that freedom. "Go beyond the US and China," she says: every country could become the place that allows the widest training. Personally she has not felt a threat. The environment has been quite friendly. Labenz calls that the best news he heard in the episode.

Labenz spends part of the introduction telling listeners to pause and look at figure one. It takes about a minute: patches in, a global model in the center, local models forking out. Figure two has the formulas. The paper and the code are public. Yu's own advice is in that same spirit. The architecture is easier to hold in your head as a picture than as a stack of abstractions about agents talking to agents.

The tokenizer problems the two of them walk through are not abstract. Long integers sliced into two- and three-digit tokens, code indentation eaten by BPE, foreign-language or scientific vocabularies sitting out of distribution — those are the reasons a model cannot add, cannot indent, and charges different API prices by language. Yu's comparison of an in-house code tokenizer against GPT-2 BPE, a 30 percent gap, is a concrete way to see the tax. A byte-level model at LLaMA scale would not magically solve reasoning, but it would delete that particular layer of mess.

The pass/fail criterion they keep returning to is equally concrete. Train with compute similar to LLaMA. Match LLaMA. Need no tokenizer. If that works, the road to a unified multimodal model gets wider, because the same byte stream can be an image, a waveform, or a document. If it fails, the questions about patch boundaries and in-context learning will have an answer too. Either way, Yu says, people already do not care about sub-billion-parameter, sub-100-billion-token demos. The next paper is the one that matters.