OpenAI Cut Fine-Tuning 90 Percent, and Founders Crossed Llama 2 Off the List

Nathan Labenz hosts *The Cognitive Revolution* and runs Waymark, a company that uses OpenAI models every day to generate television ads for clients. That mix of critic and customer is the point of this August 2023 episode. He and co-host Erik Torenberg are not asking whether fine-tuning is interesting. They are asking whether, 24 hours after OpenAI’s GPT-3.5 Turbo fine-tuning announcement, it is still worth doing the work yourself.

Labenz’s frame is “everything everywhere all at once.” He will not say open source does not matter, or that nobody will fine-tune Llama 2. He will say that if you look at Llama 2’s impact so far, this might be the biggest piece of it: it seems to have accelerated OpenAI’s fine-tuning offering, which had been the weakest part of the product for about a year. Other surfaces moved. Retail fine-tuning stayed stuck on original GPT-3.

Until this release, a retail customer could fine-tune only the original GPT-3 generation. That model was not even instruction-tuned. You still had to do classic prompt engineering: make the task look like autocomplete, or stuff in a couple of examples and hope the model would show rather than tell. Fine-tuned inference was 6x the base price, plus a separate charge for the training run itself. The best retail option, original Davinci, cost 12 cents per thousand tokens.

Waymark’s unit of work is a generated commercial, which Labenz rounds to about a thousand tokens, or on the order of 10 cents at the old price. The ads often sit inside multi-thousand-dollar campaigns shown to a client, so 10 cents was feasible. A thousand generations a day is still a hundred dollars a day, a few thousand a month, money that starts to show up. The new stack is GPT-3.5 Turbo. Base inference is 0.15 cents per thousand tokens. OpenAI still puts an 8x multiple on the fine-tuned model, which rounds to a little more than one cent per thousand. Relative to 12 cents, that is a 90 percent drop.

People online, he says, gasped that it is eight times more expensive than the base model. That is the wrong comparison. If you are already succeeding with the base model, you are probably padding the prompt with instructions and examples that fine-tuning lets you drop, so some of the 8x comes back as shorter context. For everyone who was not getting the accuracy or consistency they needed from a few-shot 3.5 setup, the release is the difference between “does not work” and “likely works.” OpenAI’s own claim is that a fine-tuned 3.5 Turbo beats GPT-4 in many cases. It is now, Labenz says, the best thing the public can fine-tune.

Buried in the same launch is a completions model, davinci-002, marked legacy rather than deprecated. For Waymark, that is a drop-in: the old input-output pairs do not have to be rewritten as chat messages. Labenz is not sure the model is specified clearly, but he thinks it is probably 3.5-class, cheaper, and faster, which might even unlock generating two ads at once. OpenAI is pushing the chat modality for new work. Supporting the old format for the foreseeable future is how you let existing customers upgrade without a rewrite.

GPT-3.5 Turbo’s base model is stronger than Llama 2’s, Labenz says, though on a high-end battery like MMLU the open model is already in the same band, a real achievement. Product quality is another story. A paper had just counted Llama 2’s false refusals. Ask where you can buy a can of Coke and it may apologize for being unable to help you acquire illegal substances. OpenAI makes a product. When that kind of failure shows up, users complain and the company iterates. Meta published, hit its paper deadline, and never much cared whether the model refused the wrong thing.

That gap is why he crossed “explore fine-tuning Llama 2” off the list the day of the announcement. Llama 2 is clearly better than original GPT-3, so the experiment had been sitting there as a way to cut Waymark’s bill and maybe lift quality. Fine-tuning the weights is only the first hurdle. If you want the model in an application, you have to serve it, scale it, and keep it up. The managed-inference market is popping up around that need. Relative to OpenAI, those products still feel immature.

He went shopping. MosaicML, a former *Cognitive Revolution* guest whose company Databricks had just bought, has an inference product. When Labenz asked, they admitted they did not have autoscaling: you deploy one instance, and if traffic backs up it is on you to spin up a second. Hugging Face Inference Endpoints do autoscale, but slowly. A second instance takes a couple of minutes, a third another couple, and scale-down is just as laggy. Waymark does not have huge baseline traffic. It does have a failure mode: a room of a hundred cable-company people told to hit generate at once. Hugging Face could not quite absorb that spike. Mosaic would not do it out of the box. He had been hearing good things about Baseten, which Jon Noronha flagged on the Gamma episode, and had not yet verified it. The pattern is a safari through buried documentation.

OpenAI’s last-generation fine-tuning, even on a much bigger parameter count, just worked. No serious rate-limit drama at Waymark’s scale. The new API is 24 hours old, so this is not a review, but if it matches the old operational feel it will be both better language and less ops. Then the squeeze: suppose self-hosting takes you from one cent per call to 0.2 cents. At 10,000 calls a day that is $100 on OpenAI versus $20 on your own cluster, $80 saved. Most organizations, he thinks, will not clear the hurdle. Developer hours add up fast, everyone already has a full to-do list, and a better open-source model will land in two seconds, which means you chase your tail. Stay on OpenAI and you ride the June checkpoint to a September one. Function calling is coming to fine-tuning and nobody else has it at their level. GPT-4 fine-tuning sits behind that, further still.

The remaining objection is data. OpenAI led the announcement with a reminder: fine-tune data does not train their models or anyone else’s. You still have to believe them. Labenz thinks they are loudly aware that customers do not want their data co-mingled, that policies have been updated, and that the company is emphasizing the point for a reason. His least-hot take on the whole release is that moats exist. They are not insurmountable, and they do not freeze competition out of every use case. They are high enough that a lot of people, including him, crossed Llama 2 off the list yesterday.

OpenAI is scanning fine-tune datasets with a GPT-4-powered moderation tool, looking for things like a spear-phishing bot. Labenz doubts it will hold, or at least thinks it will be pretty easy to get around. Naive, egregious datasets will get caught. Clever ones will not. The company now *encourages* instructions in the fine-tune, a reversal from the old “just show examples” guidance, because 3.5 is instruction-trained. Instructions also make the intended task legible to the moderator. Hide the intent in a pile of examples, or otherwise disguise it, and he guesses people will slip through.

That may not matter as much as it would have a year ago. OpenAI’s loss function is no longer only “minimize harm on our platform.” Llama 2 is a true open-source, ungovernable alternative. Someone who sneaks past the filter can also just go do Llama. Catching what they can is still worth doing; unlocking a qualitatively new harmful capability is not what this 3.5 step does. He reads the 3.5-before-4 sequence the same way. GPT-4 finished training three months before ChatGPT launched. Bing is already fine-tuning GPT-4, and Cursor, the GPT-4-native coding environment, had early access. The GPT-4 fine-tune product is probably mostly ready. Ship 3.5 first, let the filters work, see what people do, because a 3.5-class open model already exists. Jumping straight to GPT-4 fine-tuning might actually unlock something new for the people you are trying not to help.

A separate narrative had been circulating: ChatGPT traffic is in decline, and AI has been proven good for scamming, spamming, and shaming. Labenz opened the article. No mention of code, education, or medicine. Website visits are a red herring. The models are being rolled into products that travel with the user. He goes to ChatGPT less himself because he also uses Perplexity, Claude, and the code models in his editor. At Athena they buy GPT-4 through HumanLoop, another former guest, on a company account, paying for tokens instead of $20-per-seat ChatGPT. He would bet the unpublished tokens-served graph at every leading lab is still climbing. Dario Amodei, who had kept a low profile until the head-of-state meetings forced some media, said Anthropic’s usage is basically exponential. They have not even made commercialization the main focus, and the number keeps going up. He can only imagine the curve at organizations that *are* trying.

Labenz’s own morning was the 10,000x version of the same point. He was evaluating AI-safety charities, which means mechanistic-interpretability papers and GitHub libraries he has to understand before he gets on the phone. GPT-4 gets the methods section and an instruction to explain it as if he were a new graduate student. Ad-hoc expert time is easily a couple of hundred dollars an hour. What he gets for two to five cents would cost $200-plus from a hired grad student or Ph.D. The claim that nobody can find a use case is, to him, farcical.

Torenberg asks for one or two near-term cases each in code, education, and healthcare. Labenz starts with code because it is what he already does all day, which is also why the show has done less of it. GPT-4 is a very good coder and follows directions extremely well. Replit’s Amjad Masad and Michele Catasta have noted that product managers have won hackathons, because PMs are good at specifying exactly what they want and the models are good at turning that into working code. Clear instructions get pretty good code back in most cases.

His own favorite move is coding by analogy. Paste working code from the web or a previous project, say what it does, then ask for a version that differs in a specific way. GPT-4’s training cutoff means it may not know today’s API or library. It is excellent at using a live example as inspiration. Cursor and Replit are pushing true pair-programmer loops: file structure, config, environment, then hierarchical work into each file. You can get pretty far on a lot of projects. It is a skill. It is not killer on first use. It does not take long to become high-value, including at debugging.

The more expertise you bring, the more the model gives back. A real newbie can go off the rails, get confused, confuse the model, and spiral. Labenz has been teaching Waymark’s executive assistants how not to let that happen. Andrej Karpathy is the ceiling case. Rusty in C, conceptually precise, he used GPT-4 to write the low-level details of CPU inference for Llama 2, then took the code to Twitter, where people suggested micro-changes. Mid-size 7B models started running on CPU at a decent clip. Nobody will want that on a laptop all day, but the idle CPU in the world is an unbelievable pile of compute, and a year of GPU-shortage talk looks different if 7B becomes acceptable on it. Karpathy was already a 10-to-100x developer. Add another factor and you can start to see the 10,000x coder people allude to.

On education he points to the Khan Academy episode with Shawn Jansepar. Khanmigo looks like ordinary ChatGPT and is tuned to be Socratic: it will not just hand you the answer, it coaches, tries to make the problem relevant, and has checks for whether the session is going off the rails. Through elementary school and most of high school it has pretty good command. AP Physics and anything higher still slips. The simple fact is the ratio. Teacher to student is now basically one to one, which has never really been possible. Princes had tutors. Everyone else had a classroom. That scale, even before the tutoring itself gets smarter, is almost certainly a game changer.

What he wants next is smaller experiments. Today’s education RCTs are drowning in noise: train the teachers, wait a month for a standardized test, hope the kid was there that day. Bring the tests down one or two orders of magnitude. Which way of explaining fractions to a fourth grader actually works, and what is second best, and does it correlate with how they learned the last thing? Learning styles in a classroom are mostly lip service. A one-to-one tutor can deliver the version that helps you. He would be surprised if the new data did not unlock something, even if the theories are still to be proven.

Medicine, for him, is Google’s multimodal Med-PaLM M: an AI doctor on demand in your pocket, 24/7, pennies on the dollar, now with images. Take a picture of something on your skin, or pull the clinic x-ray from your record for a second opinion. That is getting close. Google will not take an Elon approach and deploy first. It sounds like another generation or two before they get serious. The model still sees images at 256 by 256, a bottleneck the team knows is not final form; they are proving theories, not maxing the system. Labenz’s four-month-old, Charlie, had a sacral dimple. The pediatrician sees them all the time and still sends you for an ultrasound because of a tiny chance of something serious. The family dragged themselves across town on a Saturday. Charlie was fine. If the model only helps people decide when they really need care, the time, money, and peace of mind are already a health outcome. Medicaid experiments often fail to move short-horizon objective health and still leave people financially and mentally better off because the decisions got cheaper.

The medical establishment has been more receptive than he expected, cautious but not hostile. Med-PaLM M’s radiology reports are preferred to the human radiologist’s 40 percent of the time. Still losing, not by much, at perhaps one ten-thousandth the cost, and probably with prompt tricks still on the table. Do radiologists unionize, or do they keep saying they are overburdened and this is great? Self-driving looks roughly as safe as humans and still is not enough; the bar may be an order of magnitude, a 90 percent risk cut, because a failure is spectacular. Medicine, at this stage, is advice, not surgery, so you have a buffer to double-check. At the current pace he finds it hard to imagine more than two years before the work is “done-ish,” even if you can always go superhuman on genotypes humans cannot read. Google’s interesting regulatory answer was that no law exactly blocks them; they would like a law they can definitely follow. Meanwhile, he says, it is time to start talking about what the clinical trial looks like.