From $500,000 to $50,000: MosaicML Made Training Your Own Model a Priced Product

Nathan Labenz opens with a scoreboard. MosaicML was first to offer GPT-3-quality models for $500,000 in September 2022, first to train Stable Diffusion from scratch for under $160,000 in January, then did it again for under $50,000 in April. On LMSYS, a leaderboard of blind human head-to-heads, MPT-7B-Chat sat ninth in the world at recording; strip out OpenAI, Anthropic, Google, and Llama-derived models and it was the top open-source chat model you could fine-tune and use commercially. The other release, MPT-7B-StoryWriter-65k+, shocked people who had not yet treated long context as the default.

Abhi Venigalla, a research scientist, and Jonathan Frankle, Mosaic’s chief scientist, describe the company as an end-to-end shop: custom language and diffusion models for people who have valuable proprietary data and want to own the result. They started with training from scratch, then added public checkpoints so a customer can continue from a model that already speaks, then added inference because every trained model eventually has to answer requests. The point, Venigalla says, is to go from private data to a private model to a private endpoint without handing the crown jewels to someone else’s API.

Customers span sub-billion-parameter models, single-digit billions, and well beyond. Replit trained a 3-billion-parameter code model that, on some tasks, beat the first Codex—small punching above its weight. Data is the constraint that actually matters. Some clients arrive with tens of billions of tokens from support logs; others show up with several hundred billion, or more than a trillion. That will not fit in a handful of input-output pairs, so the work is continued pre-training and a careful mix so the model keeps general world knowledge while learning the house dialect.

Frankle puts a fine point on Labenz’s first question. “Who’s training these models? Everybody. The question you should ask is who has interesting proprietary data. Everybody.” The roster runs from tiny startups whose model *is* the product, through AI-first shops like Replit, to the kind of company that, if he named it, you would call big and boring. Those firms have amazing data; training is how you activate it. In the old days people said data was the moat. Then a year of giant training runs made it fashionable to say the model was the moat. Mosaic’s job, Frankle says, is making machine-learning training boring again so the moat can sit back on the data.

Asked what customers actually do with the models, Frankle gives the boring answer that turns out to be the real one: extraction and summarization. A new court opinion lands at a hundred-plus pages and someone needs to know what it says this afternoon. A LangChain-plus-vector-database stack has already retrieved a pile of relevant documents; the model’s job is to pull the passage that matters or compress it into something a person can use. Chat is in the mix, as a product or as a UI, but it is nowhere near the majority. The model does not have to be perfectly right. It has to point a human decision-maker in the right direction. Nobody, he notes, is about to make an important medical call off a generated summary.

A lot of that work was never done at all in the pre-LLM era, because nobody could staff it. Venigalla uses Mosaic’s own open-source users as the example: the human ceiling is a README and an FAQ. You cannot sit a support engineer next to every person with a question. Models make a previously unscalable interaction cheap enough to try. Labenz sees the same pattern in task automation. Outbound recruiting is the canonical case—almost no company has the people to reach every candidate it would like to reach. Early adopters get a window of advantage; then the channel finds a new equilibrium.

Why Mosaic instead of OpenAI’s fine-tuning API? Venigalla splits the buyers. Startups building something the public APIs cannot quite do—a custom coding model, an odd language mix—do not want to hire a scarce research team and rent the cluster themselves. Enterprises that already shipped a first version on an API then hit an optimization problem: can a smaller custom model serve the same traffic cheaper? Mosaic prices training and serving as compute. Fine-tuned models on OpenAI, he says, cost about 6x the base rate; Mosaic charges the GPU-hour whether you are running the base weights or the tuned ones.

Frankle packages the enterprise case as three C’s, and admits the alliteration is convenient. Customization: you cannot stuff several hundred billion tokens into a few input-output pairs, and you may want more code or less of someone else’s web crawl than the public mix contains; Mosaic now has a menu. With the big APIs you have no idea what is in the soup, and you hope the thing you worry about simply is not there. Control: training can run in the customer’s cloud VPC, or on-prem, so the data never leaves; the customer owns the weights and can serve them elsewhere or open-source them. Cost: a domain-specific 3-billion-parameter model is often cheaper than trying to fine-tune a rumored 1.6-trillion-parameter giant on more than a handful of pairs.

Labenz is surprised so many established companies already look this far along, given that some workplaces are still banning ChatGPT while others threaten to fire anyone who has not learned it. Venigalla says the ChatGPT explosion changed the inbound. Startups came first; after the world saw what was possible, enterprises wanted an in-house ChatGPT they could trust—control over the data going in, recency, and a system internal staff could actually use, against a backdrop of confidential text leaking into public chat. They may not be far along the journey yet. They already know they will need something proprietary.

Would a Wendy’s-style brand model even know what a Big Mac is? Frankle has not seen customers pruning Wikipedia out of the weights. The usual recipe is a mix of open-source data and internal text, because a knowledge base alone is a thin world and often is not even written in clear prose. He also rejects Labenz’s framing. “Do you use GPT-4 or do you train your own model?” is a false dichotomy. Customers use GPT-4 where it is great and allowed; they cannot use it on sensitive data or on internal processes it was never trained on. Different tools, different jobs. Labenz’s own mantra is that nothing in AI right now is either-or.

On the curriculum, Frankle gets picky. He hates the word fine-tuning. Arrive asking to fine-tune on 200 billion tokens and “there’s nothing fine about that”—it is as hardcore as pre-training, if not more so. If you branch from a Mosaic checkpoint you still mix in open-source data. Train only on the slice you care about and you get catastrophic forgetting: the model loses generic language the way a person who only reads professional books forgets how to chat. The question is old. BERT-era papers from a long three years ago already asked how to continue training properly; one is titled *Don’t Stop Pre-training*, which Frankle says tells you the whole story.

Labenz has lived both sides on OpenAI’s fine-tuning API. A tightly specified format and length can lock in with a few dozen examples. Teaching the model to write in his voice, with the tokens he can actually muster, only half-works and still hallucinates. He also sees pre-training and late-stage tricks blending: few-shot structure is being folded back into the pre-training run, and instruction-tuning losses now look like ordinary next-token matching on a different data format. Venigalla calls it the great convergence—no new classification head, just the original objective on new sequences. Frankle’s bottom line is that the best questions here have not been answered rigorously. Mosaic and its customers are learning every day.

Labenz asks about a gap in his own map: language models in the life sciences. Mosaic worked with Stanford’s CRFM team on BioMedLM, trained on PubMed papers so you can ask diagnostic-style questions in biomedical language. Google’s Med-PaLM work is the parallel, putting models head-to-head with physicians as an augmenting tool. A second direction Mosaic has not pushed as hard this year is sequences that are not English at all—protein and amino-acid chains—where a Transformer plus a long context is a natural fit. Multilingual customers are another surprise. People want bilingual models for language pairs Frankle would never have put on a five-language shortlist, and that puts ugly pressure on the tokenizer. Labenz notes that a single Hindi character can explode into eight tokens.

How raw can hundreds of billions of internal tokens be before they become a problem? Frankle’s first approximation: the model is what you put into it. Train on web data full of ads and the model starts inserting ads into sentences. Tokens are not all equal, but you have to do a lot of cleaning before quality differences show up, which is why Wikipedia and code punch above their weight. Venigalla’s answer to “will it invent products we do not sell?” is long context. Stuff a June 2023 product list into the prompt and the model attends to that more faithfully than to whatever it memorized, which is more reliable than retraining every time the catalog moves.

Most enterprise apps still need a polish pass—instruction tuning or RLHF—so people can talk to the model naturally instead of prompting it like a completion engine. A common path: start from a base checkpoint, deploy it on Mosaic’s inference playground, share it with internal users, then iterate a fine-tune. Academic benchmarks are saturating; Mosaic is pushing interactive and automated evaluation. Red-teaming is still rare among clients. Frankle will not announce a product, but he says evaluation, including red-teaming, is part of being a one-stop shop from data through training to inference, and people should keep their eyes open.

Why pay Mosaic to serve a model you could spin up yourself? Venigalla says serving one customer is easy and serving many is not. You need batching, lower-precision weights, multiprocessing, autoscaling, and a lot of tuning. For a 30-billion-parameter model, few existing stacks hit the latency people expect from a hosted API. Mosaic’s job is to collapse that infrastructure. Labenz has studied the public price list somewhat obsessively: MPT-7B-Instruct is five hundredths of a cent per 1,000 tokens. Put differently, 20,000 tokens—about 40 pages—costs one cent.

Where does that cent go? Mosaic will deploy on any cloud, on GPUs you already have, or on capacity it rents, with Oracle as the current preferred partner. Inference servers run all the time, unlike a training job that eventually finishes. An A100 might rent for about $2 an hour; the provider’s own cost can be lower because of scale. That hourly number already packages a GPU amortized over roughly three years plus the electricity in the rack. Mosaic then helps an enterprise pick a card that is not overkill—an A10 can be enough—and translates a requests-per-second target into a 24/7 fleet. One cent on a single call is just the monthly thousands of dollars, smeared across the month’s tokens.

There are two products. A starter series is a shared, pay-per-token service for Mosaic and small open-source models. Custom weights cannot live on that shared pool without cold-start pain, so those customers move to an enterprise path: dedicated capacity, priced per hour, with Mosaic doing the capacity planning. Placement is transparent. Workloads run wherever the customer’s security and existing contracts already live; renting through Mosaic is optional. Frankle will not guess Oracle or AWS’s split between electricity and silicon. Electricity, he guesses, is the smaller slice, and he is happy to introduce Labenz to Oracle. Demand, meanwhile, is high enough that the research team is making hard choices between more papers and more customers.

Labenz tries to recap the price history and quotes Stable Diffusion at “like 150k.” Frankle stops him: “Less than 50k. Gotta get that number.” Labenz later checked. The extra drop was announced in the few days between when he prepped and when they recorded. The field’s deflation does not wait for the guest notes.

StoryWriter’s 65,000-token window, which can go further still, is built on ALiBi. Attention is still quadratic; nothing about that math changed. What changed is the positional scheme. Venigalla’s nuance is that the quadratic slice is smaller than people think. At a 2k training context, the attention dot-product can be less than 10% of the network’s work. Stretching context from 2k to 65k made work per token about 4x for StoryWriter—more expensive, not ridiculous. At inference you can technically go to an “infinite” context; practically, ALiBi extrapolates about 2x the length it was trained on before quality falls off.

The infinite-matrix worry is a misunderstanding of what is being stored. ALiBi replaces learned positional embeddings with a fixed slope, set at initialization and not trained, that you stretch at inference the way you stretch a continuous ramp from −1 to 1 across however many tokens you want. Train with that ramp across 2,000 positions, serve at 4,000, and you just pull it farther. On the 65k model there is no real hard cap: change `max_seq` in the config, instantiate, and the bias rebuilds for 130k. The two limits Frankle actually respects are GPU memory and that empirical 2x extrapolation wall.

Labenz still wants the finite-parameters-to-infinite-sequence trick unpacked. Without any position information, a Transformer is a bag of words: the next token can attend to everything and has no idea what came first, and you can already stretch that as far as memory allows because no extra weights appear. ALiBi only adds a non-learned bias you can resize. The deeper reason, Frankle says, is parameter reuse. The same attention weights apply at every position, the way a convolution can be resolution-agnostic because it is the same kernel sliding over the image. You need RAM for activations. You do not need a new parameter for each token.

Reading is fast; writing is slow. You can stuff 65k tokens into the input and the forward pass is relatively quick; generating every subsequent token still takes time, and that is the optimization target for the coming weeks. Frankle’s last word is a warning disguised as a roadmap. Mosaic has shipped a lot in a couple of weeks, and the cadence is not slowing down. MPT-7B and Stable Diffusion for less than $50,000 are “the boring baselines that we’re going to be crushing.” For an industry that just cut a training bill by more than 3x in three months, it sounds like a boast. It also sounds like a status report.