Plus Five for Format, Plus Ten if It Compiles

When Ksenia Se, host of Inference by Turing Post, opens with the episode’s headline question — when will we train a model once and then let it learn forever — Predibase CEO and co-founder Dev Rishi does not treat it as a future tense problem. That world, he says, is already here, though it is spread very unevenly. Most customers still take a model on which someone else did 99% of the heavy lifting and apply a last-mile 1% of customization. The shift he is watching is from that borrowed, static checkpoint toward a pipeline that keeps improving the model while it sits in production.

A handful of Predibase’s early customers have already stood up those pipelines, and that is the product direction Rishi is most excited to keep building. In his telling, the default enterprise pattern is still a frozen model someone else trained. The cutting edge is a loop: stop treating deploy as the end of training, and start treating production traffic as the next training set. He frames the next era as customers operating a system that learns after it ships, not a one-time fine-tune they are afraid to touch.

Predibase, which builds fine-tuning, post-training, and inference infrastructure, shipped what Rishi says was the first end-to-end reinforcement fine-tuning platform a couple of months before this conversation. Asked whether RFT is a fundamental unlock or just another tuning trick, he starts with the mechanics rather than the marketing. Traditional supervised fine-tuning wants large labeled datasets, with an input and a target output for every row. RFT can start from something like a dozen examples, plus a customer-written reward function instead of all those labels.

Those reward functions are rubrics that grade a model’s output so the model can adapt toward whatever gets points. His running example is teaching a model to write code: plus five if the formatting is correct, plus ten if the code compiles, plus twenty if the unit tests pass. The scores stack, and the model learns to chase the conditions that can actually be measured. Soft quality — whether a paragraph of marketing copy “feels right” — is much harder to write down as a rubric, and he does not pretend RFT is equally good there.

He compresses the whole method into a slogan: if you can measure it, you can improve it. That line is both the promise and the filter. Tasks with crisp acceptance tests — did it compile, did the answer match, did the JSON validate — are in-distribution for RFT. Tasks that amount to taste are a weaker fit, which is why he treats RFT as a sharp tool for narrow, checkable work rather than a universal replacement for every other tuning method.

Rishi does not claim reinforcement fine-tuning will retire supervised fine-tuning. Today it is one of several techniques that help customers start tuning, not the only hammer in the toolbox. What he thinks will actually change the workflow is not the one-shot job. Right now RFT is still a training process you run and then freeze. Where he sees it going is inside a continuous feedback loop, so models get better online after they are already serving users. That, he says, is the paradigm shift.

Has anyone closed that loop yet? Only the cutting-edge end of the customer base, and even then it is early. He is working with a couple of healthcare companies building copilots and assistants for patients. They sit on a lot of real conversation data and run two feedback channels at once: larger models judging whether a conversation went well, and clinicians labeling a subset of those conversations. Instead of waiting months to accumulate a labeling round, they can start from a handful of conversations and feed that into an RFT loop. The change in cadence, from months to days, is the part he treats as qualitative, not just faster.

Two product bets at Predibase are meant to make that pattern less heroic. The first is almost boring: automatically collect prompts and responses from any production deployment, because data has always been the bottleneck and live traffic is the most honest training set. The second is learning from small nudges rather than giant labeled corpora — direct preference optimization and similar methods already sit in the platform, and the research side is working on using RFT with user-feedback data. Rishi says the company will publish more in the coming months on what feedback users actually offer in real agent applications, and how to turn that into a continuous stream. The first release will show that small amounts of feedback move performance, and that more feedback scales the gain.

Rishi’s view of agents is unsentimental: the industry is in very early innings, and a lot of what people are shipping is brittle. He defines an agentic workflow with two requirements. First, a chain of multiple LLM calls — one-shot document classification does not count; a conversation that explains a medical diagnosis and books a follow-up does, because it takes several turns. Second, tool calling, so the agent can invoke external functions on the user’s behalf. If either piece is missing, he would not call the system an agentic workflow.

Brittleness, in his account, is a math problem hiding inside a demo. Builders often have a compelling walkthrough as long as the user stays on the golden path. Step off it, and the model hits a pile of edge cases. If each call is only 90% accurate and the workflow needs five calls, the user experience is already under 50%. That last mile of quality, he argues, is what decides whether agents are actually usable. For anyone buying an agent, the implication is blunt: do not stop at single-step accuracy; multiply it out across the chain.

He has seen a version of this movie. He was a product manager on Google Assistant, which he describes as one of the first AI agents — built with classical NLP, not generative models. The industry-wide bridge, as he sees it, is getting from flashy demos to robust agentic workloads. Ksenia notes that he is a product-first founder in a research-heavy field. His path ran through a computer science undergrad and master’s, some early CS research, then what he jokes was selling out to product because it sat at the intersection of research, engineering, and design. Even at Google he worked closely with research teams, which is why he is comfortable in a domain where the underlying techniques are still moving. The pace, though, is unlike cloud or mobile: he expects a breakthrough roughly every week, and he recalls the stress of planning Predibase’s RFT launch while watching whether OpenAI, Anthropic, Mistral, DeepSeek, Google, Amazon, or Meta would drop something that sucked the oxygen out of the room.

Predibase did not start as an LLM company. Its original mission was to democratize deep learning — interfaces and infrastructure so people could train their own models. Then, in late 2022, OpenAI did the democratizing with large pretrained models that could hold a conversation in one shot. Rishi says the interesting part is that Predibase had already been hosting a smaller version of that workflow: in early 2022, the most popular feature was picking a pretrained model such as BERT and fine-tuning it on your data. ChatGPT did not invent the job. It changed who showed up to do it, and what they expected the product to be.

Before that break, the user had to be an NLP engineer who understood BERT, T5, or a vision transformer. Today, he says, some of the most prolific AI engineers started a year or two ago. The company treated the moment as a focusing, even if it felt like a full pivot: at the beginning of 2023 it bet the future of production LLMs would be specialized and customized, built the tuning and post-training stack for that, and later in the year went hard at inference once serving became obviously central. The persona, the use cases, and the user journey all shifted. The underlying bet — adapt a pretrained model to your data — did not.

Asked whether enterprises will run more small models on narrow tasks, Rishi’s first move is to grow the whole pie: by 2027 he expects more use cases of every kind. The mix, though, tilts. His favorite customer quote is that generalized intelligence is great, but “I don’t need my point of sales system to recite French poetry.” Checkr, which handles employee background checks, needs to extract criminal codes, violations, and prior history at very high quality. It does not need the model to write Python or verse. In enterprise, he thinks the high-value pattern is a prescriptive specialized agent that reliably solves a defined series of tasks. General-purpose company chatbots will exist and demo well, but they are not where he sees the money concentrating. Consumer products are different — versatility is actually useful there.

One thing that was not obvious in 2023 and is obvious in 2025, he says, is that we are not going to live in a world where one model rules it all. Open-source, closed commercial, and different parameter sizes will mix, and people will pick the best tool for the task, as they do with any other software. Predibase’s thesis is that tasks needing narrow AI will grow even faster than the tasks general models already dominate. The company’s north star is helping customers develop specialized AI, tune it, and run performant production deployments; it is not religious about supervised fine-tuning remaining the best method. RFT was the big new technique of 2025, and he expects more methods, plus expansion into multimodal, vision, and voice as customers ask. Planning runs through 2026, but he quotes the famous boxing line that everyone has a plan until they get punched in the face — in AI, the punch arrives weekly.

Inference, Rishi says, is an onion: easy at first, harder as you peel. In the crawl stage the bottleneck is not brilliant software engineering so much as GPUs. A single replica of the largest models can take eight or sixteen H100s, which means procurement, autoscaling, and standing up an inference server. Predibase open-sourced LoRAX, the inference stack it runs internally, so a capable engineer can get prototype serving working. That is not the same job as production. Plenty of smart teams can feed an early app. Far fewer want to live with what comes next.

Production means 99.9% or 99.999% SLAs, fault tolerance, blue-green updates, multi-region replication, and squeezing value from every token because GPUs are expensive. Total cost of ownership becomes the metric. The platform includes software-defined tricks such as TurboLoRA, which he says can double model throughput — the kind of optimization that only shows its worth once a model is feeding a business-critical application. Monitoring, SLAs, and high-performance serving are not prototype problems. They are what you discover when “it works on my GPU” stops being an acceptable bar.

He agrees that base-model inference is heading toward commoditization: there is little reason one company’s DeepSeek endpoint should be dramatically better than another’s. The interesting trend, internally branded “intelligent inference” — he jokes they will work on the name — is an inference pipeline hooked to a post-training stack so models get better over time. That join, he argues, is the biggest thing a single platform can unlock. Open-source fine-tuning with Ludwig and serving with LoRAX are good enough to start experiments, then step onto a managed Predibase with GPUs included. The stack for getting a model running is no longer the main gap.

What is still missing is a resilient way to evaluate. LLM outputs are sometimes objective and sometimes irreducibly subjective — a good summary is not the same kind of number as classification accuracy. Ksenia points to the Leaderboard Illusion and the failure of crowdsourced eval; Rishi says most teams still build in-house. He sees three patterns: historical holdout data, which is the cleanest when you have it; LLM-as-judge with larger grader models, which is the most common; and shipping the product to collect directional feedback. A lot of evaluation is in-house, he says, “and we should not underestimate how much of it is vibes today.” Getting eval right is a prerequisite for the continuous-improvement loop. Open-source reasoning models help because systems like DeepSeek R1 expose reasoning tokens, so you can score the steps rather than only the final answer. Closed models that arrive as a new persona force teams to rebuild their taste from scratch.

In 2023, Rishi reminds the host, the best open-source model was GPT-J, a long way from GPT-3.5. Today DeepSeek R1 and V3, Qwen 3, and Llama 4 are on par with leading commercial models and sometimes ahead on benchmarks. That is about six months ahead of his own calendar: he thought open source might tie commercial models in 2025 and beat them only by year-end. The rate of open-source innovation, he says, has been remarkable. For companies that once treated open weights as a research curiosity, the production gap has closed faster than the 2023 talking points allowed.

He almost never talks about AGI, and when asked, he puts it far from the world he actually sees. That world is not artificial general intelligence so much as practical specialized intelligence. If AGI means passing a Turing test, he suggests we may already be in the ballpark; he does not spend time on Terminator scenarios. He does spend time on what generalized capability does to a Fortune 200 such as Marsh McLennan, or to Checkr, after decades of process-driven productivity. Those businesses, he thinks, are headed for a step-function change. Five-, ten-, and twenty-year philosophy is hard when the field cannot predict eighteen months. Nathan Labenz, in the Cognitive Revolution introduction, had already flagged the same phrase — practical specialized intelligence — as a possible, if partial, alternative to an AGI land grab.

What concerns him is not “is generative AI a bubble?” — a question he says he never gets from a Fortune 200 CIO who is actually shipping generative apps. On the ground, fifty cents or a dollar per million tokens versus the return is not even a question in most cases; when it is, the issue is picking the right use case. The failure mode he does worry about is a hype bubble around far-fetched, French-poetry-style multi-agent demos, followed by a crash of disillusionment that tars the high-value work the models can already do. From 2012 to 2022, he argues, AI outside the top 1% of organizations overpromised and underdelivered: YouTube-style recommenders never actually showed up in small and medium businesses. This wave is different, unless the industry imports the old paradigm of overclaiming.

The trend that most excites him is a change in how software gets built. The old sequence was perfect, then test, then dogfood, then ship. A mentor once told him that if you ship something you are not at least a little embarrassed by, you have waited too long. In AI he now sees teams putting out 60% solutions on purpose: they want to test product-market fit and to start collecting the data that will improve the model. That startup habit, he notes, is spreading from 20- and 50-person companies into larger organizations adopting generative AI. The model is no longer a finished artifact you unveil. It is something you put in front of users so it can keep learning.

Taken together, the filters he offered across the hour are blunt enough to use on Monday. RFT is worth trying when a task’s success can be written as a rubric — format, execution, field-level hits — and a few dozen examples can start the flywheel; if the rubric will not write, supervised fine-tuning or a different task is the honest move. Agent pitches should be judged on end-to-end success, not single-step accuracy, and on what happens off the golden path. Inference prototypes can live on open-source LoRAX and Ludwig; production needs SLAs, failover, multi-region, and a per-token cost that will not blow up. Evaluation should start before launch, combining holdout data, model graders, and product feedback, while steadily shrinking the share that is only vibes.

His closing book recommendation is unrelated to machine learning: *The Happiness Advantage* by Harvard psychologist Shawn Achor. The research punchline is that the causal arrow often runs backward — happiness makes success more likely, rather than success guaranteeing happiness. The book covers both the performance effect of a positive outlook and the practical ways to put yourself in a happier position without waiting on external wins. He hopes Predibase is that kind of company. Generative AI is noisy, fast, and competitive, and well-funded players, Predibase included, have a lot of cards on the table. Teams, he says, do their best work when they are excited about the future they are running toward, not when they operate mainly from concern.