Open and Closed Will Tie at the Same Size: PyTorch's Former Lead on Fast Inference

Lin Qiao thought swapping Meta's other frameworks for PyTorch would be a library change. How hard could that be? Six months, they guessed. It became a five-year rebuild of data loading, distributed inference, and training, because none of that existed on PyTorch yet. When she left, the stack was sustaining more than five trillion inferences a day. Fireworks, the company she founded in 2022, exists to compress that kind of time-to-market from five years to five weeks, or even five days.

Sonya Huang introduces her as an AI-infrastructure heavyweight: the person who led PyTorch, the backbone of the global machine-learning ecosystem, with co-founders including Dima from that same project. Fireworks is a SaaS platform for generative-AI inference and high-quality fine-tuning, especially on a small-model stack. The pitch is three numbers: low latency for real-time apps, low cost so a business can actually grow, and automated customization so an enterprise gets quality that is tailored rather than generic.

The judgment she puts on the table later in the hour is the one the episode is built around. Open-source and closed-source quality, she predicts, will converge inside the same parameter bucket — 7B to 70B, even within 100B — until the gap is nearly gone. The remaining work is fitting a model to a specific workload. That is exactly the job small models and a thick open-source community are good at.

PyTorch, in her analogy, is a programming language for digital brains. Researchers can create those brains quickly. The brains do not think fast enough. That was the job she took on as head of PyTorch. When she started, Meta was running three stacks: Caffe2 for mobile, ONNX for server-side production, PyTorch for research. Three teams, three times the maintenance. The mission was to collapse them into one.

The first attempt was the obvious stitch: keep PyTorch's simple front end, bolt on Caffe2's back end, zip them together. The two frameworks had never been designed to work as one. Integration was harder than writing a framework from scratch. Plan B was to keep the beautiful front end and rebuild the back end. That became TorchScript, PyTorch 1.0. Then came the migration they had treated as a library swap. Data loading, distributed inference, scaling training — each layer had to be built on PyTorch from the ground up. Five years later the inference and training stack was new, and the volume was in the trillions per day.

The lesson she wants to productize is not the heroics. It is the calendar. If the industry keeps repeating a five-year rebuild every time a research tool has to carry production, generative AI stays a hyperscaler hobby. Fireworks's stated mission is to hand other people the production stack in weeks.

Most inference platforms stay framework-agnostic. Fireworks does not. The reason, Qiao says, is a funnel she watched for years. PyTorch began as a researcher tool and took over the top of model creation. Applied teams almost never rewrite a research model into another framework for production, so the models flow down the funnel in PyTorch. New models, especially generative ones, are born in PyTorch and served in PyTorch. Wherever the researchers are, the next production environment is. Supporting other frameworks would be a distraction.

What researchers like is simpler than a feature list. "Simplicity scales." The PyTorch lesson at Meta and in the community was a relentless focus on making the user experience simpler while hiding more complexity in the back end. She copies that into Fireworks. Against open-source vLLM and TensorRT-LLM — projects she likes, with her heart still in open source — the differentiation is blunt. Off the shelf, Fireworks is faster. Those projects are libraries. Fireworks is a system that auto-tunes to a developer or enterprise workload for both speed and quality, which a library cannot do.

The cost of that simplicity is that Fireworks absorbs the ugly parts: handwritten CUDA kernels, distributed inference across nodes, disaggregated inference that chops a model into pieces and scales the pieces differently, semantic caching that skips recomputation when content matches, and quantization at several layers — KV cache, weights, even communication across GPUs — each with its own performance gain and quality tradeoff. Pat Grady names the pattern conservation of complexity, after Larry Tesler's line from Xerox PARC: the complexity of a task is neither created nor destroyed, only assigned. Fireworks's business is to take the customer's share so the customer can spend cognitive bandwidth on the application.

The path to Fireworks, she says, is consistent. Last year almost everyone started on OpenAI. Startups with a product idea wanted the most powerful model while they hunted for product-market fit. Once they had it and tried to scale, two problems showed up. Most generative apps are B2C — consumers, individuals, developers — and people will not wait half a minute for an answer. Latency is product viability, not a nice-to-have. Cost is the other hand: if the product works, it scales fast, and losing money at small scale means bankruptcy at large scale. Latency and total cost of ownership are the two reasons customers migrate off OpenAI.

Grady reminds her of an observation she made a year earlier: training scales with the number of researchers, inference with the number of customers, and there will always be more customers than researchers. Inference is the bigger business, and the migration sits exactly on that turn. Who shows up surprised her. The plan was a three-step march — technically aggressive startups, then digital-native companies, then conservative enterprises after they had watched. Reality arrived all at once. Health insurers, healthcare companies, and banks came inbound. She had prepared a business-value pitch for traditional enterprises. They pulled her into low-level technical detail. Early contacts were often CTOs. She reads it as a shift toward innovation-driven transformation rather than a CIO-led cost conversation.

Two reasons, in her telling. Leaders treat this wave like cloud-first or mobile-first: the industry map will be redrawn, startups are growing fast, incumbents who do not move become irrelevant, and incumbents are racing one another. Second, generative AI broke the old gate. Traditional AI meant training from scratch, no foundation model, data and compute concentrated in hyperscalers. After foundation models, training from scratch is a waste: the crawlable internet is shared, architectures look alike, and the move is fine-tuning on a small, high-quality dataset. Small data, small models. That is what makes the technology affordable for everyone, which is why even banks are jumping in.

Huang asks how the open-closed gap will move. Qiao's answer is specific: inside the same size bucket, 7B to 70B, even within 100B, quality will converge. Grady asks whether closed source will always sit six months ahead. She says there is no lag — neck and neck at the same size — and leaves a marker: come back to this podcast in a couple of years. Closed labs, she argues, can only bet on a few models. OpenAI's language models might be three, or one if you notice that architecture and training data define a model and those inputs look similar. Anthropic, Mistral, and the rest have the same constraint: concentrate on a segment. That is the best ROI they can buy.

Open source runs a different dynamic. A strong base such as Llama 3 gets released, and thousands of researchers build on it. Fine-tuning an open model is not magically easier; the community is simply thicker. Llama 3 follows instructions well, so aligning it to a specific problem is easier. Fireworks has invested heavily in function calling, and she says fine-tuning a function-calling model on Llama 3 is much easier than it was on Llama 2 or mixed bases. Instruction following, logical reasoning, and other base skills are now strong enough that warping a small open model into a high-performance specialist is the point.

Grady compares infrastructure history: twenty years ago open source was a fast follower, Red Hat the canonical example; later Mongo and Confluent showed it could lead. Will models ever have an open-source leader? Qiao's answer is structural. Closed labs fire at a few models. Open source lets a thousand flowers blossom — her phrase, twice. Enterprises have messy, specific problems. A thousand flowers means some existing solution is already sitting in your problem space, and then you customize for the workload: better quality, lower latency, lower cost. Huang asks whether OpenAI's regular price cuts keep her up at night. Qiao notices that OpenAI itself is going smaller and cheaper, which is evidence of the same convergence. She does not think OpenAI will do heavy per-customer customization. Its mission is AGI, a different problem. Enterprise-shaped tasks are the gap small models plus customization are built to fill.

Can customers match OpenAI quality on the Fireworks stack? It depends on the domain. Some tasks do not need fine-tuning at all. Code generation, coding copilots, transcription, translation, and OCR are already "phenomenal" off the shelf. Tasks with business logic do need it, because every company defines "good" differently. Classification is the clean example: a marketplace deciding furniture versus a dish is a standard that lives inside the company. Summarization looks general until an insurer wants a specific template. Those are the jobs that require a loop through data and training.

The loop is longer than the word "fine-tune" suggests. Collect data, instrument traces, label, then pick an algorithm: supervised fine-tuning, DPO, or a slew of preference methods that never mark absolute good, only "I prefer this over that." Then choose LoRA-style parameter-efficient tuning or full-weight tuning. Some tasks need hyperparameter search, not just weight updates. Most app developers have not touched AI. After a round of tuning, leftover failures have to be sorted: more data and another cycle, or a product that was never specified. She gives a concrete case. An assistant that auto-generates while you type: the cursor is in a table cell. Does auto-generate extend the cell, add rows, or do nothing? That is a product-manager question. A better model will not answer it.

Fireworks wants a clean split. It will take the grunt work — which tuning method, automatic labeling, automatic collection from production — and hide it behind a simple API. How the product should respond stays with the customer. She says they have been building in that direction and hope to announce the product soon. Grady's gloss is that this is conservation of complexity again: free people to think from the customer back instead of from the tech out.

The north star, in her words, is "simple API access to the totality of knowledge." The platform already has more than a hundred models: language, image, audio, video, embeddings, multimodal. Glue all the foundation models together and knowledge is still bounded. Training data has a start and an end. The crawlable internet is finite. A great deal of knowledge sits behind public APIs you cannot see in real time, and behind private enterprise APIs nobody outside the company will ever get. The layer that reaches the rest is a blend of many models and those APIs. The vehicle is a function-calling model: it knows which APIs exist and why you would call them, and it routes precisely, to models and to non-model tools. Think of OpenAI's mixture-of-experts as a router over a few huge specialists. Fireworks wants a mixture of hundreds of small, agile experts, each high quality on a narrow problem. Where do the experts live? Her ambition is blunt: on Fireworks, curated by the platform. A next-generation function-calling model, with "multiple layers of breakthroughs," is coming with demos and examples.

Will Nvidia have competition? Lucrative markets invite it, and industries dislike monopolies. The question is when, and she thinks soon. On general-purpose GPUs, AMD is coming up. In a specific AI segment where the model shape has stabilized and the problem is well defined, custom ASICs get a role. Are returns to scale on the frontier slowing? Both things at once, she says. Applicability will plateau and the industry will lean into customization. Do not treat the plateau as a wall. At Meta, ranking-and-recommendation models were supposed to be settled; a few years later a burst of model innovation in that "stable" space pushed Meta's S-curve up. Generative AI will get a new architecture too. By her clock, that revolution is already overdue.

In the lightning round she uses Fathom for meeting notes and summaries; the sales team trains on it; she calls a notetaker a game changer. 2024 will not have a single best model. LMSYS Arena has a new one every week, they chew on each other, and quality keeps rising, which is good for the industry. The organization she admires most is Meta, for continuously open-sourcing Llama and shrinking the quality gap — decentralizing power from hyperscalers to anyone who wants to build. On agents she is bullish: they will blossom this year. Huang closes with a point from Sequoia partner Roelof Botha. In internet and mobile, the people who drove down the cost of running the stack unlocked the application layer. Qiao, she says, is doing the same thing to AI's production cost curve.