Everyone Talks About RLHF. Almost Nobody Is Doing It.

LLMs are easy to try and hard to master, Labenz says in his introduction. A starter prompt can look decent in minutes. For a company that wants the model in a product, or wants to automate an internal process, most of the work is still ahead: capturing and studying data, collecting user feedback, hunting failure modes, monitoring performance, and running experiments that actually compare one setup to another. In his experience the split is more extreme than the usual 80/20. It may be 90/10. It may be 95/5.

Raza Habib has a PhD in machine learning from University College London and interned at Google AI. He now runs Humanloop, a Y Combinator company whose job is to close that gap — from “here is a smart API” to a product someone will trust. Labenz discloses that he is a Humanloop customer, advising Athena in the executive-assistant business; there is no sponsorship attached to the interview. He simply wanted to hear what Habib has learned from watching a lot of companies try, succeed, and fail at the same stretch of road.

Habib’s first surprise is who is calling. Messages arrive from CTOs at companies you would recognize: don’t tell anyone, but two or three strong engineers and I have been on this for three months, and we think it might change everything — can you help? HubSpot co-founder Dharmesh Shah has been in the room with his engineers, trying to make the stuff work. Labenz notes that many of these leaders have not written much code in years; the tools are now low enough that they can get back in. Larger companies tend to add LLM features to existing products or automate internal processes. Startups almost always put the model in the product they sell to someone else.

The three recurring pain points are prototyping, evaluation, and customization. First, language models are stochastic and need instructions, so teams have to mold a messy business problem into a format the model can actually solve — a concept Labenz says people who have only tried ChatGPT badly underestimate. Second, the tasks are more subjective than classical ML: if you generate an email, marketing copy, or a job description, what does “good” even look like, and how do you watch it at scale? Third, models make things up and do not know company-private data, so they have to be customized to a context, a company, and a user.

The use-case list is already a jumble. Early on, after GPT-3, most people built writing assistants — Sudowrite, Copy.ai, Jasper — to get past the blank page. Legal is a large bucket: contract review, summarization, question answering, extraction from documents. Sales teams want meeting transcripts summarized and pushed into a CRM. In medicine, several companies are attacking insurance paperwork and form-filling for doctors. Code ranges from natural language to SQL, so a business analyst can query a database, to Cursor, which is trying to rebuild an IDE as if an LLM had been there on day one. Habib’s phrase for the moment is a genuine Cambrian explosion.

Labenz draws an axis. At one end sit GitHub Copilot and ChatGPT: open-ended, real-time co-pilots. At the other sits Copy.ai: a few bullets, a button, a finished artifact, almost no user control. Habib refuses the binary. What you are looking at is a still-unmapped space of user experiences for language models, and the breakouts succeeded because they cracked a UX insight as well as a modeling one. Startups tend to ship a more opinionated wrapper for people who are not AI-savvy. Enterprises often want something more on rails — first-line ticket replies, for example — so the user never has to think about the model at all.

ChatGPT is so much more accessible than OpenAI’s original playground, he argues, because the interface is fault-tolerant by design. In a conversation, if the first answer is not quite right, you do not immediately give up; you expected to take a few turns. The playground was a text box. You pasted something, hit enter, and a lot of people bounced: this is crap, it does not work. GPT-3.5 and GPT-4 are genuinely better models, Habib is careful to say. Part of why chat took over anyway is that it gives people a chance to correct the system and try again. Copilot lives at a different extreme. Latency in a coding flow is brutal; waiting on a large model in a chat pane is a context switch. A model that sits on your shoulder, sees the file, and occasionally suggests a line turns out to be the right shape.

Humanloop’s own toolset leans toward people trying to make a particular feature reliable rather than a fully free-form chat. The playground is multi-provider, with chat and completion modes, and it can run the same prompt template across ten or twenty inputs at once — a sales email, say, tested on a handful of real cases, not the one example currently on screen. Labenz uses a teaching version of the same trick: drop Kobe Bryant, Santa Claus, and Joe Biden into one template and watch it flex. Teams then graduate that prompt to an API endpoint. The hard part, Habib says, is going from “quite good” to “I trust this in production,” which first requires knowing whether it is working at all.

Labenz finds one number in the GPT-4 technical report almost insane: users preferred GPT-4 to GPT-3.5 only 70/30, despite a jump from the bottom of the bar exam to the top decile. That gap says evaluation is hard, and that people are probably seeing patterns that are not there. Habib agrees it is especially messy for subjective work. If you are a writing assistant, the right answer is whatever the user says is right. Code that has to run, or SQL that has to query, is easier to score. Summarization and question answering sit in the uncomfortable middle. Humanloop therefore logs both quantitative metrics and human feedback, keyed to a session ID on every generate call.

Three kinds of feedback actually get used. Explicit votes — thumbs up, thumbs down — help, but most people never touch them, so the sample is skewed toward extremes. Implicit behavioral signals are richer: did they send the sales email, copy the marketing copy, or mash regenerate, which is a fairly loud “the first one failed”? Textual corrections, the edits a user makes before using the output, are the most honest error data — useful for manual review and as fine-tuning fuel. Binary preference collection is common internally; Anthropic and OpenAI train on it. In production it is a bad experience except in places like AI Dungeon, where choosing among generated paths is the product. Labenz wonders about trading free access for labeled feedback, an ads-model successor. Habib’s suspicion is that implicit signals already give you most of that trade, so few products bother to make it explicit.

Once monitoring is in place, model switches stop being vibes. Several customers moved from text-davinci-002 to GPT-3.5 or GPT-4 and watched user-satisfaction charts jump almost overnight. Teams without tooling could only say it *seemed* better. Labenz’s own reason for picking Humanloop is related: a thousand executive assistants, a need for full history, and OpenAI’s 30-day delete policy pointed the wrong way. The daily loop is then mechanical. Filter failure cases, reopen them in the playground, form a hypothesis — wrong retrieval, bad prompt, model not capable — change one thing, redeploy, and check the charts or an A/B test. How fast that find-fix-verify cycle turns, Habib says, is close to whether you succeed.

He also argues that time spent *playing* in an interactive environment beats staring at leaderboard numbers. The quantitative scores still matter; play gives you something they do not. His exhibit is Noam Chomsky’s *New York Times* op-ed listing things language models would never be able to do. The concrete examples, not paraphrases of them, all ran in a playground. “How hard would it have been to just try?” Habib asks. Labenz’s reaction is a single word: preach.

Most Humanloop customers are not fine-tuning yet. The ones who are tend to be further along, because early on prompt engineering still has so much juice and a much faster loop. Habib’s contrarian prediction is that fine-tuning will get a lot more common soon, driven by cost and task-specific performance: a smaller model that is as good or better on *your* job. Phind, a YC developer-search product he describes as Google married to Stack Overflow, already runs a heavy fine-tune-and-feedback cycle and, in his view, now has a world-class model for that task. Two practical things had been holding the rest of the market back. OpenAI’s public fine-tunes were still the original davinci, orders of magnitude more expensive than turbo and weaker. And people worried the next base model would make their work obsolete. Both of those constraints, in April 2023, looked like they were starting to flip.

Open-source is part of the flip, slowly. OPT and BLOOM were, Habib says honestly, significantly behind closed models. Llama is competitive but not commercially licensed. Language models will not have a Stable Diffusion moment as fast as images did: they are harder to train, bigger, and the small versions are not good enough to live on a phone. He does not think the wait for GPT-3-level open alternatives is long. If you want the biggest, baddest model you will still go to OpenAI or Anthropic for a while. You do not always need that, especially after a fine-tune. A separate slice of the world will not let data leave their servers at all; for them a custom fine-tune is the only option. Humanloop’s bet is to own the data, evaluation, and feedback infrastructure, partner for the actual training jobs, and already make GPT-3 fine-tunes relatively painless.

Then the line the episode is remembered for. RLHF, Habib says, is like sex in high school: everyone is talking about it, almost no one is actually doing it. The InstructGPT and Anthropic papers show a large gap; a lot of that gap is usability — following a natural-language instruction — not a jump in base capability. The pipeline is long: preference data, a reward model, then RL, each stage a new place to go wrong, with the InstructGPT appendix full of details you have to get right. His researcher’s instinct was that every time they removed an RL component, the system got a little more robust. Mode collapse and “dark matter” capability loss are real, especially for creative writing customers who prefer base models because RLHF flattened diversity. For most product tasks, supervised fine-tuning is enough. A founder he had just spoken to handwritten 50 examples, fine-tuned davinci-002, and got a pretty reliable model. Fifty examples is not an RLHF-sized project.

His complexity ladder is therefore blunt. Prompt engineering is fast, cheap, and capped: you will never get a 10x latency or cost win from it. Supervised fine-tuning is harder and can deliver those wins. RLHF can add more, if the problem is big enough to justify the mess. Pick the simplest tool that does the job.

In April 2023 agents are all over Twitter. Humanloop’s production customers say the same thing, over and over: we tried hard, it works about 80 percent of the time, and that is not good enough to ship. Habib’s Tools feature, for now, only feeds the model extra information — Google, Wikipedia, a private FAQ or database, stuffed into context. Letting the model *take actions*, GPT-3-plugins style, is still off, because they have not found it reliable enough. He is excited about that future. He is not pretending it is here.

RL on the agent, he suspects, will not fix the core math. Start with a module that is 85 or 90 percent accurate, chain the steps, and you are looking at 0.9 to the n. As n grows, success goes to zero. The system is open-loop: no feedback as it goes, so errors compound. The real need is a healing mechanism — another model, perhaps, that looks at a step, says that bit is wrong, try again. Without course correction he does not see how agents become reliable. Labenz notes that two agent-platform CEOs have already been on the show; neither is at an inflection in usage. Habib is still optimistic on the calendar, for a slightly self-mocking reason: six or seven years in machine learning, he has been too pessimistic every time, and has learned that progress keeps beating his forecasts. Labenz’s entrepreneur’s prior runs the other way — he always underestimates how long change takes — except for raw AI capability, the one exception in his career. They land on the same gap Humanloop was founded to sit in: research-solved and product-usable are different dates.

Labenz has asked a lot of plugged-in people which AI products they actually use. The list is usually ChatGPT plus Copilot. Habib’s is longer, and it is still short. Mem, a self-organizing notes app that most users do not even think of as an LLM product. Phind, and Perplexity’s Bird SQL Twitter search while it lasted. Cursor, which he uses all the time. Fathom, a Zoom notetaker he hits about four times a day. GPT-3.5 and Claude dropped straight into Slack, which gets used more than a destination chat tab because that is where work already happens. Why so few killer apps? Because this wave of products only started being built four or five months ago. Looking around and asking where they all are, he thinks, is premature.

His analogy, which Labenz immediately files in a personal collection, is early television: people pointed a camera at a stage play. Cuts, camera language, montage — the native grammar — came later. AI products are still in the stage-play phase. On strategy he splits the board. In established categories, HubSpot, Salesforce, and the big contract-review vendors will add summarization and extraction without much trouble; they have distribution and feedback volume. Entirely new categories — an IDE rebuilt from scratch, a language-learning app that only makes sense now — are where startups can win. Incumbents that do not adopt still die. If you are legal tech or a CRM and you skip this, someone else will ship a better experience. The speed of the current wave is already unusual. Microsoft has been faster to add LLM features than anyone; Replit is the other name he grants, with a smaller surface to cover. OpenAI’s partnership with Bain looks, to him, like a smart refusal to do enterprise sales in-house while claiming to be focused on AGI.

The closing hypothetical is Neuralink: a million people already implanted, safety data roughly as good as a COVID vaccine, thought-to-text working — would he get one? It depends on the political context, the same way it does for AI. He already has a prosthesis, his phone, and he is glued to it; the implant is only one step further. He does not want an advertiser model, or Facebook or Google, with a pipe into his brain. Hopes, zoomed out: accelerate science, including the AlphaFold-shaped kind of win; make personalized tutoring cheap enough to scale; free people from drudge work; make software cheaper to build for small markets that never justified a company. Fears, this side of AGI: concentrated political power, and autonomous systems screening CVs and then scaling whatever bias was already there.

Asked for the thing that actually predicts success, Habib does not reach for a model name. Teams that do not get there tend not to have a robust evaluation system, so as they change prompts they are flying in the dark — they cannot tell if the last edit helped or hurt. The second principle is a fast iteration cycle: try something, see how it did, go again, whether the change is a fine-tune, a prompt chain, or a UX rethink. Humanloop, he says, has probably watched more attempts to get LLM apps into production than anyone except maybe OpenAI, and those two habits are what he would put on a wall.

The maturity path is copyable. Start in a playground with a handful of test cases and answer one question: is this even feasible? If yes, wire it into a product. Larger companies often put a Streamlit app in front of hundreds or thousands of internal examples first. Startups often skip that step; once they are satisfied it works 80 or 90 percent of the time, they just deploy and iterate on production data. You do not need a data lake. A few hundred interactions a day — not many, given how often people touch a model in several places — is already enough signal to drive decisions. Put the three feedback types in the product tomorrow. Filter failures once a week. Form a hypothesis, change one thing, check the charts. That loop, turning, is already ahead of most teams still tuning prompts by feel.