The Cheapest Inference Provider Is the Most Dangerous

Nathan Labenz opens by admitting he does not know this business well. Fireworks AI sells inference compute in a market where GPUs are a hot commodity and NVIDIA's stock is the scoreboard, yet people also whisper that inference is a low-margin grind. His guests are Lin Qiao and Dmytro Ivchenko, cofounders who spent about five years on PyTorch at Meta and are now trying to make generative models fast, stable, and cheap enough to ship.

Qiao does not dodge the commodity question. Reselling hardware, she says, is a low-margin business. When the company was still brainstorming, GPU shortage made arbitrage look clever, and they walked away because it would not sustain a long-term company. She has also watched competitors post "artificially manufactured" prices in a crowded field. Those providers, she argues, cannot build a durable business; at minimum, developers should be cautious, because the stack disappears when the funding runs out.

What Fireworks will compete on is a different trio: extremely low latency, high quality, and low total cost of ownership. Consumer and prosumer apps have to feel hyper-interactive or they are not viable products, and customers already come to Fireworks after failing to get stable latency from their own stack or from OpenAI and Anthropic. Low TCO matters because this is not the old CPU world of fat gross margins. GPUs are expensive, power hungry, and so hot that air cooling often fails — operators turn to liquid cooling or immersion cooling, which means putting the GPU in oil. That operating cost is why many generative products never become businesses. Qiao's punchline is that low TCO is a byproduct of high performance, not a race to the bottom on list price.

Labenz recognizes the accounting gap from his own apps. Engineers who roll their own infrastructure rarely put their salaries into the cost model, so the spreadsheet looks much cheaper than the real number. Fireworks currently runs on AWS, GCP, and Oracle OCI rather than its own racks, optimizing for product velocity on battle-tested clouds while aspiring to ride the best hardware in the industry — NVIDIA first, with AMD, Intel, and custom ASICs on the map.

Dmytro walks the hardware spectrum before the Meta story. NVIDIA still dominates for two reasons: the chips, and the developer stack. An H100 offers about two petaflops; Blackwell aims to double that, with superchips talked up toward ten petaflops, plus faster memory bandwidth and lower-latency interconnect. AMD's MI300 already posts better hardware specs on paper and is chasing CUDA with an opening ROCm stack. Intel's Gaudi 3, in his telling, sits between H100 and B100/B200. Groq is the specialized foil: it pulls HBM off the board and puts memory on-die so tokens stream out of a demo at startling speed, at the cost of fewer FLOPs, expensive on-die SRAM, and a setup that can run into the millions of dollars per model. Cool demos, he says; still mostly edge cases against programmable GPUs.

Both founders, and a large share of the early team, came out of Meta's PyTorch org. Hyperscalers already mix CPUs, GPUs, and ASICs; Meta has said it deployed more than 600,000 H100-class GPUs in house, which Qiao translates as the electricity of 600,000 American households, or roughly twice San Francisco. That fleet was not a weekend project. It took more than five years to go from basic CPU machine learning to deep learning on GPUs and ASICs, and the software lessons are the ones Fireworks is selling.

In 2018 Meta had at least three frameworks: Caffe2 on mobile, ONNX for server production, and a mandate to unify research and production as PyTorch. The idealistic plan was literally named Zipper — stitch the easy PyTorch front end to the fast Caffe2 back end. It failed miserably. The two codebases had never been designed to work together, and the integration overhead exceeded writing a new framework. They kept the PyTorch front end and rewrote the back end as TorchScript, an interpreter that would run the model efficiently. The product rule that survived is still Fireworks' rule: hold the line on ease of use, and swallow the complexity in the backend. TorchScript still asked developers to annotate scriptable code, so two years later the team started TorchCompile to automate the annotations away. That effort became PyTorch 2.0, which shipped after they left.

The second expensive lesson was that a framework is not a library you swap into an existing service. Innovation at that layer demanded new infrastructure: new data loaders, a new transformation path, a new training loop, a new distributed inference service. After five years they had rolled a full PyTorch platform that was serving more than 50 trillion inferences a day across 50 data centers. Fireworks exists to shrink that five-year journey into five weeks, or five days, for everyone else.

Labenz tries the product and finds a playground that feels like OpenAI: pick a serverless model, pay per token, copy the snippet. Quality is the harder pitch, because the best models are still proprietary and the closed labs are cutting prices too. Fireworks lists models up to 16 billion parameters at 20 cents per million tokens. Anthropic's Haiku sits around 25 cents on input. Nathan pauses on the history: original GPT-3 was 6 cents per thousand tokens, which is $60 per million. In two years the bill fell by a factor of about 300, for models that beat that GPT-3, and Haiku is already in spitting distance of Fireworks' small-model tier.

Qiao's competitive map does not include other inference vendors. "Our real competition is actually OpenAI and Anthropic." She expects open and closed models in the same size bucket to converge, because a small network has an upper bound on the knowledge it can absorb, and the industry is also converging on how much data it can legally train on. The week they recorded, Mistral dropped a mixture-of-experts model with eight experts of about 22 billion parameters, and Fireworks enabled it the same day. Llama 3 was weeks away; Google kept shipping Gemma; Chinese labs' Qwen and Yi models were getting better. If quality converges, she will bet on open source, because developers fine-tune, re-release, and compound. Fireworks is built on that loop.

Open models are not there yet without fine-tuning, and fine-tuning is slower than prompt engineering. Qiao's answer is to make the loop as close to prompting as possible, because enterprise problems are not AGI. Nobody is asking a model to solve a hundred hard reasoning tasks at once. They want intent routing to agents, catalog classification, reranking of retrieved results, structured extraction from receipts and medical bills, or a paraphrased sales follow-up. Aligning a model to one narrow objective is much easier than aligning it to a hundred. Enterprises already have the data; she has seen fine-tunes beat GPT-4 on the customer's own task. Labenz's company Waymark is the exhibit: its core script-writing job runs on fine-tuned GPT-3.5. GPT-4 was too slow, and the business rules would not fit in a prompt; cost was secondary. The fine-tune set is in the hundreds of examples, not thousands. For a narrow task, that was enough.

The fine-tuning API is deliberately OpenAI-compatible, so a team can prototype on OpenAI and migrate when it needs deeper tuning, lower latency, or lower TCO. Under the hood Fireworks leans on LoRA, the low-rank adaptation trick that freezes pretrained weights and injects two small trainable matrices into each Transformer layer. Qiao works an example: a 2,000-by-5,000 weight matrix holds 10 million parameters; rank-1 factors are 2,000 plus 5,000, or 7,000 parameters — more than a thousand-fold cut, so the feedback loop is fast. Serving is the second trick. The naive way to deploy 100 LoRA adapters is to merge each with the base model and keep 100 full copies in memory. Fireworks was among the first to share one base model as a trunk, with adapters hanging off it like branches. Same workload, roughly 100 times less memory cost.

Dmytro tries to cut through the technique-of-the-week noise. For Transformers, two operations dominate: matrix multiplication, and attention, which is a fussy back-to-back batched matmul. Text generation then splits those ops into two flavors. Prefill — chewing the prompt — is mostly compute-bound. Decode — emitting tokens one at a time — is mostly memory-bandwidth-bound. Typical apps sit around a 10:1 input-to-output ratio, with prompts of a thousand to ten thousand tokens. Prefill batches can run into the tens of thousands of tokens; decode batches more often sit somewhere between 16 and 200, because a long prompt already consumes activation memory. Both phases load weights from HBM toward registers. Only decode is actually waiting on that load.

KV caching is the optimization that makes decode cheap enough to ship. Prefill is still quadratic attention over the prompt; after that, each new token attends to cached keys and values instead of recomputing the past, so generation becomes linear. The cache still has to live in HBM for fast access. Multi-query and grouped-query attention shrink it by cutting the number of KV heads, commonly by a factor of four to eight, and sharding the model shrinks it further. You can spill the cache to CPU memory, Dmytro says, but that path is a complexity swamp. The NVIDIA software ladder runs from TensorRT-LLM on Triton Inference Server at the rigid, almost-ready-to-serve top, down through cuBLAS and cuDNN, into CUTLASS and the newer CuTe API, and finally PTX. On Hopper, tensor cores are launched asynchronously from CUDA cores, which themselves were launched asynchronously from the CPU — two levels of asynchrony. The result, he says, is that high-quality public H100 kernels are scarce compared with the A100 generation.

OpenAI's Triton is the readable escape hatch: write Python, lower through MLIR, land on PTX. Structure makes kernels maintainable; there is still no free lunch. Unless the op is pointwise, you inspect the generated instructions, the pipelining, and the tweaks, or you do not get cutting-edge speed. The reason the work never ends is new hardware, especially new number formats. Training and inference moved from FP32 to FP16, then to FP8; Blackwell is pushing FP4. Recent 1.58-bit papers train with weights in {−1, 0, 1}, which turns multiplication into addition. Each precision change ships a new instruction mix. FlashAttention 2, written for Ampere in FP16, is the cautionary tale: the same kernel on an H100 delivers about half of what Hopper can do, because it never uses Hopper's instructions.

Sharding is the other permanent tax. Tensor parallelism splits weights across GPUs, can shrink time-to-first-token, and still costs throughput because of communication. Pipeline parallelism chops the model by layers and only helps when the model will not fit; it does not help latency. Data parallelism splits the input, is comparatively cheap, and complicates attention because tokens still need to see one another. Dmytro's rule is blunt: do not shard until you have to. Cross-GPU traffic is orders of magnitude more expensive than moving data from a GPU to its own registers, and Megatron-style row/column splits exist mainly to keep that all-reduce from eating the win. A 7B model that fits on one GPU may still be sharded if the prompt is long and the latency budget is tight. That problem, he says, is not going away.

Look up from the kernel and the application is just as jagged. Latency might mean time to first token, time to the first 30 tokens for a voice stream that cannot start speaking until it has a buffer, or end-to-end completion. RAG workloads drive input-to-output ratios well past 10:1; a classifier may emit a single token; a coding model may generate a flood. Context length, repeated system prompts, and where the customer sits on the latency-cost curve all change the right deployment. Labenz is writing a talk for business leaders whose first slide is "learn to think in Pareto curves," from token latency all the way to false positives. He asks whether Fireworks will eventually just show those curves and let a team pick a point.

Qiao reaches for databases. Twenty-five years ago, SQL had a stable pattern — select, where, group by — and none of the early systems were smart, so an entire career called DBA appeared to twist knobs. Those people made a lot of money, because a tuned database was faster and cheaper, which is the same latency-versus-TCO conversation. Then query optimizers started watching the workload, building indexes, partitioning columns, and skipping sequential scans. Hire fewer DBAs; the system tunes itself. That is the vision she wants for inference. Today Fireworks still asks what you care about and deploys a configuration. Over time the stack should watch live traffic and get lower-latency and higher-quality without another human in the loop.

The product line already has three tiers: developer, business (serverless, pay as you go), and enterprise, where the customer arrives with a budget, a latency target, and a known mix of retrieval versus generation. Enterprise traffic is always skewed — a few heavy hitters and a long tail. Fireworks' advice is to treat the tail as one deployment bucket and hyper-optimize each heavy hitter. Labenz sketches three apps in one company: a voice surface that must get the first 30 tokens out immediately, a background job that should be as cheap as possible, and a chatbot in the middle. Those three, Qiao says, should not share a config. Long-context processing and heavy generation are different machines even before quality choices — instruction-heavy prompts versus a fine-tune that deletes the instructions, versus a RAG pack that constrains hallucination — enter the picture.

Labenz is surprised they train models at all, given the inference stack's complexity and Qiao's claim that small models are converging. She says the north star is not serving isolated checkpoints. It is an inference stack for knowledge extraction. An LLM's knowledge is bounded by its training corpus, its cutoff date, and whatever cannot be crawled. A lot of the rest lives in other foundation models — image, video, audio — and behind APIs: search, weather, stocks, maps, docs, spreadsheets, calendars. A single enterprise can expose hundreds of internal API surfaces, plus key-value stores, document stores, production logs, and internal search. Function calling is the layer that ties those sources together.

That is why Fireworks trained its own function-calling models, released two open-weight versions on Hugging Face, and has a third on the way. The application she wants is a personal admin that learns what you like, finishes work before you ask, and puts reminders in front of you. Those products only exist if the serving tier can pull knowledge from models and from tools. On SSMs and Mamba, Labenz's hobbyhorse, she is cooler: customers have barely asked. Pure Mamba models have not matched Mistral or GPT-class quality; AI21 Labs' Jamba hybrid is still unproven on real tasks. Long context is theoretically the SSM pitch, but the industry still leans on needle-in-a-haystack tests that she does not consider comprehensive. Dmytro is confident they could serve the architecture quickly. Quality, not kernels, is the gate.

A week after the main taping, Fireworks announced it would be the exclusive inference provider for Stability AI's Stable Diffusion 3. Dmytro frames the split of labor simply: Stability has researchers who train models and a history of monetizing them; Fireworks has production engineers who can bring those models up cheaply and reliably. The architectural news is that SD3 moved onto Transformers. The old ResNet-and-convolution profile is gone, so a large share of the text-model kernel and sharding work transfers. Image generation still has no decode phase. It is prefill all the way: start from noise, run a forward pass per denoising step, and map the Transformer output back to pixels. SD3's quality preset is about 50 steps; the turbo variant is about four, trading fidelity for speed. The tokenized input size is fixed after the encoders, which makes optimization more straightforward than a variable-length prompt.

Labenz tried the model the morning of the bonus tape and found himself on Stability's domain, not Fireworks' playground. The detour is deliberate. Stability is sensitive to misuse and trained its own input-sanitization models, so it keeps that hop. Image generation also weights quality and cost over the last millisecond of latency, so the extra route is acceptable. Quantization does not copy from text. Language models need to preserve high activation spikes, because those spikes change the next-token choice. On an image, a clipped spike is less lethal; overall precision across the canvas matters more, because glitches show up as visible artifacts. You cannot paste an FP8 recipe from Llama onto SD3 and trust the pixels.

The dirty work, Qiao said at the top of the show, is the point of the company: application developers should not have to care. Behind that sentence sits 600,000 H100s, a five-year PyTorch platform, Hopper kernels that almost no one has published, and a bet that inference will grow an optimizer the way databases did. Pick a vendor for latency you can live with, count the people who would otherwise babysit GPUs, and leave yourself an OpenAI-shaped exit if today's cheapest bill is being paid by a venture round.