There will not be one model. Fireworks is betting on hundreds of small experts

Lin Qiao, co-founder and CEO of Fireworks, joins Jacob Effron on Unsupervised Learning with an unusually clear product thesis. She was one of the creators of PyTorch and spent seven years at Meta; Fireworks has raised more than $75 million and, in her framing, exists to deliver the best quality at the lowest latency and cost. Effron opens with the question every inference company is getting in late 2024: OpenAI’s o1 and the rest of the test-time-compute wave look like they will change the job. Does that break Fireworks’ roadmap?

Qiao says the company had that world in mind from the start. Inference, as she describes it, is not “API in, API out, done.” The system she wants is a logical-reasoning stack that can reach hundreds of small expert models, not a frozen endpoint behind one name. When a user sends a query, the interesting work is routing, composing, and checking — not picking a single champion model and hoping it generalizes.

She counts three hard limits on any one model. First, models are probabilistic, which is a poor fit when the product must be factual and always truthful; hallucination has to be controlled by the system, not wished away inside the weights. Second, real applications are multimodal. The conversation they are having is audio plus vision, and even a text-only product still splits into classification, summarization, multi-turn chat, and tool calling, each of which wants a slightly different expert. Third, knowledge is capped by finite training data, while much of the useful world lives behind public or private APIs that a pretrained model cannot see.

The name she uses for the alternative is a compound AI system: multiple models across modalities, plus APIs, databases, and knowledge bases, working together. Berkeley coined the term; Databricks is telling a similar story, which she takes as a sign that the category is real. Design splits into two schools. Imperative systems, like an ETL pipeline, let you dictate every step. Declarative systems, like SQL, let you state the what and leave the how to the engine. Drawing on PyTorch, Fireworks leans declarative while insisting on full debuggability, and it already ships the low-level blocks — language, transcription, translation, speech, vision over PDFs and screenshots, embeddings, image generation, with video on the way.

Asked why generative applications are still rarer than the hype, Qiao does not blame missing GPUs. She blames the nature of training. A trainer has to pick, from thousands of possible problems, the subset that matters, then spend the most money and the most careful data work there — and quietly abandon the rest. The resulting model is “really, really, really good at doing some things and then really bad at doing certain things.” That is not a bug to be scaled away. It is the process.

Shrink the problem and a small model can push quality in a way a generalist cannot. That is why she thinks the future is hundreds of small expert models, and why she is bullish on open source: base weights give builders control, and a growing set of labs now specialize in post-training and fine-tuning, then give the specialist back to the community. Enterprises, she says, are moving the same way. They want steerability and control, not a black-box oracle that is merely large.

Effron sketches two futures: Fireworks as a library that orchestrates a dozen strong open models for a customer, versus telling that customer to fine-tune or even pretrain. Qiao’s answer is that she believes in customization, but it is not a one-minute job. The hard part is the path from a prompt that works, to a prompt that has grown unmanageable, to weights that have absorbed the policy. That path is the next argument she wants to make precise.

The practical sequence she sees is almost always the same. Teams start with prompt engineering because it is immediate: change the text, see the result, test whether the model is even steerable. Then the system prompt grows. She has watched it reach thousands of lines, at which point nobody knows what to edit and a small change can scramble everything else. Managing that blob, she says, is a product problem of its own, and Fireworks is building something specifically to make that customization loop easier.

That stall is also the signal to fine-tune. By then the team has already proved the model can follow the instructions; those instructions are data. Absorbing a long system prompt into the weights makes the model faster and cheaper without giving up quality. She maps the same moment onto the company lifecycle: it tends to arrive as a product moves from pre-product-market-fit exploration into post-PMF scaling. The organic motion is prompt, then prompt-management pain, then fine-tune. Skipping the first two steps is how people fine-tune on a guess.

Should enterprises pretrain at all? Qiao’s answer is blunt. Pretraining is consolidating into the hyperscalers. A few companies will still do it because it is core to the business; most should ask where their differentiation actually lives. Pretraining is expensive, and the ROI is hard to justify when post-training on a strong base model is both cheaper and more agile. The scarce resource is no longer “can we train from scratch.” It is “can we steer a model that already knows most of the world.”

From Fireworks’ seat — hundreds of models, a wide customer mix — Qiao’s classification is simple. The generative use cases getting real adoption are human-in-the-loop automation, not human-out-of-the-loop automation. Her hypothesis is that a generative system has to be human-debuggable, understandable, maintainable, and operable. If people cannot evaluate it, cannot influence it, and cannot run it in production, it will not get adopted, no matter how impressive the demo.

The list she gives is mostly assistants sitting next to a professional. Scribing tools for doctors. Education helpers for teachers, students, and language learners. Coding assistants, the most competitive category, where Fireworks works closely with Cursor, Sourcegraph, and others. Medical support in a market she says has a shortage of more than a million nurses in North America and a lot of patients who need attention. In call centers, the version that works is an assistant that makes a human agent faster and better, not a bot that replaces the agent. Early digital SDRs are starting to land, but she still puts them in the same “help the person” bucket.

On the model side she sees a lot of convergence around Llama variants, which she reads as a verdict on quality: a strong base, good instruction following, and a good fine-tuning target, with Meta’s backing as extra reassurance for enterprises. Evals, meanwhile, often start as “vibe-based,” which she finds unsurprising in early product work. Teams then discover they have to invest. A/B tests remain the ultimate judge of product impact, but they are slow. Sourcegraph, one customer, skips evals entirely and ships to developers, who vote with usage. Everyone else, she argues, needs a growing eval set, because the expert-model train is moving every week and products themselves harden from open-ended toys into specialized features that each want their own specialist model.

Fireworks’ own f1 is the thesis made concrete. Customers call it as if it were a new model. Underneath, Qiao says, it is a logical-reasoning inference system: multiple models talking to one another, plus reasoning steps implemented in the platform. Quality control gets harder once models converse, because the information they pass is sensitive. She puts the complexity above building a database management system, and latency and cost become first-class once there are many steps. The original single-model-as-a-service layer was only the building block toward that stack.

After f1 launched, the waiting list was mostly people building agents, which means function calling. In her definition that is not “call one tool.” The model has to hold a long multi-turn context, choose among as many as hundreds of tools, and emit a one-shot plan that mixes parallel and sequential calls. Her example is a chart of the top three cloud providers’ stock prices: a search to find who the top three are, three parallel price lookups, then a charting call. Getting the model to know *when* to call a customer’s tool is a tuning problem they have been investing in for about a year, and she thinks they shipped the first function-calling model ahead of adoption.

Why build that instead of waiting for open source? Because the hundreds of expert models will come from the community, but composability lives in the orchestration layer, and each small expert is itself a tool. That layer is strategically too important to wait out. Reasoning, she adds, will not have a single winner either. One path is a strong base model inspecting itself with chain of thought, tree of thought, and backtracking. Another, which she is especially excited about, is reasoning in latent space rather than in words — the way people can think without talking. Fireworks does not want to bet on one flavor. It wants to integrate all of them, and when f1 goes generally available she wants developers to get plugins so they can build their own f1s.

On hardware, Qiao’s claim is that developers who understand low-level optimization are scarce, and the cadence has compressed from a new chip every three years to a new skill every year from each vendor. There is no best hardware, even for a single model, because the bottleneck depends on the workload mix. Fireworks already runs AMD as well as Nvidia, and it will route mixed access patterns inside one workload to different chips. The point of the platform, she says, is to absorb that complexity so builders can stay on the product.

Will hyperscalers then win plain open-source inference on price? She treats that as the wrong frame. Every hyperscaler “want[s] to be Apple” and build a vertically integrated stack because they can. Their real advantage is solving problems that take enormous money and headcount: data centers, power, racks, lighting up large-scale storage and compute. Compound inference is a different kind of hard. At Meta she did not see a thousand-person inference team. Once you have a horizontally scalable, production-quality system, “the rest is just let it scale.” The coming reasoning systems add complexity that you cannot clear by throwing people and budget at it, which is exactly where she thinks a specialist company should live.

Local models get a cooler hearing. Desktop offload can save real money — she cites Zoom moving compute off the cloud — but phones are another story. Power, thermals, and ratings are watched closely, so the models that can actually ship on mobile sit in the 1B to 10B range and are severely limited. Privacy is a weaker argument than it sounds, because most personal data already lives in the cloud rather than on local disk. The interesting split is desktop versus cloud for consumer apps, not a coming world where every phone runs a frontier model.

On open source she is still close to Meta. They co-design Llama Stack, an attempt to standardize the tool layer around Llama the way Android standardized a plug-in ecosystem, and she says Llama 4 is coming with continued investment. The industry-wide question is when pretraining ROI fades. Her version of the wall is data: the same internet crawl, exhausted synthetic text, exhausted multimedia-plus-text combinations. She thinks the industry is already on a soft wall, not a hard one, which is why spend is sliding from pretraining to post-training to inference. Fireworks will stay compatible with imperative agent toolkits such as LangChain, will not become a cheap GPU cloud, and is happy that Databricks is also talking about compound systems. Berkeley named the category. She wants to be a key player in it.

Effron notes that Qiao started Fireworks in September 2022, a few months before ChatGPT. At the time, she says, there was still an active debate about whether AI was “here,” because applications looked fragmented and the data did not seem to be there. Her timing bet was simpler: hyperscalers usually run three to five years ahead of the rest of the industry, Meta was already massively AI-powered, and PyTorch adoption was the tell that the wave was coming. The original customer in her head was the company that still had to hire a machine-learning team, curate data, and train from scratch. Generative foundation models broke that requirement.

The shock, in her telling, was accessibility, not magic. A foundation model absorbed most of the knowledge; a team can use it as-is or fine-tune on a few thousand samples. You either skip the ML org entirely and let a product team build, or you keep a small one. That is why the adoption curve went vertical, and why Fireworks could laser-focus on inference: every generative model is a PyTorch model, and her team had already operated large, complex PyTorch systems under high production traffic. From Meta she also watched the mobile-first transition produce the data that then powered AI — a pattern she thinks the whole industry followed.

The research she watches outside day-to-day Fireworks sits in two buckets. One is model-system co-design, treating quality, latency, and cost as a three-dimensional optimization instead of letting researchers chase quality while systems people chase speed. The other is disruption: “Transformer as a technology is overdue for disruption,” plus how agents will communicate, including thinking in latent space. To keep from exhausting themselves chasing every model drop, they anchored on a trend she does not expect to reverse — specialization and customization — and they close the loop with a “one-size-fits-one” optimizer that takes a workload and an objective and emits a deployment config, and maybe an adjusted model.

In the quickfire, the overhyped idea is the perception that generative AI is magic and a recipe for every problem. The thing she changed her mind on is go-to-market: she expected startups, then digital natives, then traditional enterprises in sequence; instead all three showed up at once, sales cycles shortened, and procurement itself is being rewritten. Startups want low-level abstractions they can assemble; traditional enterprises want higher-level ones so they can ignore details. Fireworks offers both, because it needs the lowest layer internally anyway. A hundred-billion-dollar training run only makes sense to her on a fundamentally different architecture. On applications she is still most excited by the unfinished agentic world, with early digital SDRs as a proof point, and she points builders to the self-serve playground at fireworks.ai.