Start With Ten Gold Examples: A Three-Step Playbook for AI Automation

On August 20, 2024, OpenAI opened GPT-4o fine-tuning. The next day Nathan Labenz used the news as an excuse to air a method he says he has been sharpening since the summer of 2021, when GPT-3 fine-tuning first became available. He is not coy about the pitch. The framework is one of the few things that has “stood the test of time” in a landscape that otherwise eats its young, and he tells the audience he is “genuinely not holding anything back.”

The talk is a studio version of a keynote he had just given at the Adapta Summit in São Paulo, to more than 4,000 Brazilian business owners who wanted practical utility rather than another AI scout report. Labenz is host of The Cognitive Revolution, a GPT-4 red-teamer, and an operator at Waymark, the small-business video company that still has a case-study page on OpenAI’s site for its use of GPT-3 fine-tuning. The same principles, he says, apply whether you are automating a back-office process or bolting a “done for you by AI” experience onto existing software. Each new generation of models has mainly changed how ambitious those goals can be. GPT-4o fine-tuning just raised the glass ceiling again.

He leads with two numbers that are already in the wild. Earlier in 2024, Klarna announced an OpenAI-powered assistant that handled 2.3 million customer conversations in its first month live — two-thirds of the company’s service chats, the work of 700 full-time people, 35 languages, twenty-four hours a day. Klarna estimated the system would make the company $40 million more profitable in 2024 alone. It was not doing everything; one-third of chats still sat with humans. Two-thirds, at that scale, is already a serious business.

Labenz starts with two deliberately plain definitions. Work is the transformation of inputs into outputs. Intelligence is the ability to do that work *without precise instructions* — the clause that separates a mind from a deterministic machine. MNIST, the oldest dataset in machine learning, makes the distinction tactile. Humans glance at tiny handwritten digits and almost never miss. Ask Claude 3.5 Sonnet, then his daily driver, to write old-fashioned rules (“a line across the top might be a seven”) and the model itself warns you that this is a machine-learning job. The function it produces scores 14 percent.

A second check with Perplexity, hunting for the best fully engineered algorithm in the literature, turns up a clever trick: zip-compress the images and predict from file size. That reaches about 80 percent — wrong one time in five, still unusable for mail. Even a small neural net, trained the ordinary way, can hit 99.7 percent, right at human level. Digits on envelopes are not a toy. Millions of pieces of handwritten mail move through the United States every day, and the computers that sort them are not running if-then trees.

The first takeaway is therefore negative. If traditional code can do the job, use traditional code: it is faster, cheaper, and more predictable. Models cost more, run slower, and make stranger mistakes; spend them only where intelligence is actually required. That does not mean the remaining work is low-value. On routine tasks where we know what good looks like and can show examples, the best systems are closing in on human experts. A Google study Labenz had covered on the show built a diagnostic chatbot, from a foundation model plus several rounds of fine-tuning, that outperformed human doctors as judged by other doctors — strictly better across guess counts and accuracy thresholds. Google has not shipped it. The paper, he thinks, is a credible sign of what is coming.

Most people still meet AI in a chat window. That includes the two headline examples: Google’s diagnostic bot and Klarna’s support assistant both present to the user as conversation. Chat is magical because it is real-time and omnivorous, but the human stays pilot. You are constantly deciding whether the model is helping or whether you would be faster alone. Almost everyone who has used these systems in depth has had both experiences. Automation is a different goal. You want to delegate a slice of work on a consistent basis and stop inspecting every output.

At Athena, the executive-assistant company Labenz works with, they call that move AI delegation, and they plan to train assistants on the same framework. The shape is a structured workflow — Zapier, another no-code tool, or custom code — designed once, validated, then left running in the background. In the middle sit agents: on-the-fly delegation, a little project handed over with “come back when you are done.” Today that still means trust but verify. Most people, he says, are not having great results. Agent companies themselves keep repeating his first rule: anything you can do with code, do with code; use the model only where you must understand the user’s intent, then map that onto a deterministic program. Phone trees that become natural-language conversations are the near-term prize.

He is willing to be held to a prediction. So much scaffolding already exists that the next generation of models — better at reasoning, multi-step planning, and reflection — should make small-scale ad-hoc agents work. Leading labs, he assumes, already have more advanced systems internally. It would surprise him if that leap did not arrive in the near future. Until then, the heart of the talk is not agents. It is the slower craft of choosing a task, documenting it, and raising performance until you can actually look away.

The method has three steps: choose the work, understand it, optimize performance. The first is a filter. You want inputs that become outputs where intelligence is required and a fully explicit program would fail. Then size the work as a *task*, not a *job* — the most common way Labenz sees business owners overshoot. METR’s then-new benchmark, which buckets problems by how long a human takes, matches the intuition: models often handle a couple of minutes, and success falls off once the work stretches past a day. Prefer work that is slow and expensive, because that is where the payoff lives, and repetitive, because a one-off belongs in a chat window, not a pipeline.

Context has to be explicit. Models have read the internet and know nothing about your company, your project, or your team. Whatever a new hire would need, the model needs in writing, often more, because it does not pick things up at the water cooler. Gold-standard examples are a special case of that context: the team must agree on what good looks like. Labenz has walked into organizations where nobody can describe the work, several people do it differently, and they still disagree after you surface the gap. Skip beauty-in-the-eye-of-the-beholder problems. Start low-risk, with fast feedback, so early mistakes do not compound in a blind spot. Automate the un-fun parts and keep the work you enjoy. Last, and hardest: pick something the model can actually do.

He would rather call the systems alien intelligence than artificial, because they violate human performance intuitions. Translation is already superhuman for day-to-day work — faster, cheaper, any major language to any other — even if literary translation still wants a person. Programming sits in the middle: consistently good, sometimes above the median contest entrant, still behind human experts on frontier problems. Common-sense spatial reasoning is a known weakness; the internet does not contain many paragraphs about stacking an apple on a book. Malcolm Gladwell’s 10,000-hour expert does not exist yet. Labenz puts his own time at two to six thousand hours over three years and still calls that the expert class. Ethan Mollick’s rule of thumb is the one he endorses for practitioners: spend ten hours talking to a new model before you claim to know its temperament.

His scorecard is a set of asymmetries. Breadth is superhuman: MMLU scores above 90 percent across dozens of domains, a feat almost no person can match. Depth still belongs to specialists — the one-boat river-cruise operator in Manaus who knew Amazon fish, habitats, and water chemistry better than any model, even though Claude 3.5 had been able to name the operator from a trip-planning chat. Insight, the Eureka moment, remains a human edge; IMO silver-medal systems and ARC-AGI solvers get there by generating huge piles of candidates, not by a flash of structure. Memory is still more useful in a person, though Labenz bets architectures plus infinite disk will eventually reverse that. Speed, cost, availability, and scale go to the model: often ten times faster, ten percent or less of the cost including setup, always on, a thousand jobs as a thousand instances. Sanity-check any vendor who tells you a very different story.

Choosing the task can take a couple of hours if you already have judgment. Labenz quotes Tyler Cowen on how long it takes to read a book — Cowen answers with his age — to make the point that the hours are already spent. Understanding the work is social, not technical. Ask a support agent how tickets get handled and you will hear “we get a ticket, we respond, we close.” People who do routine work do not spend much time in the meta. The phrase that unlocks them is the same famous prompt that helps language models: “Let’s break it down step by step.” Then the flow appears: find the account, prioritize, hunt for documentation, escalate to product if missing, write the reply, close.

Look at that diagram and mark the black boxes — the places intelligence is truly required. Finding an account from an email is a database query. Prioritizing a ticket is not. Ask “how do you think about it?” and a dump of considerations comes out: current backlog, contractual SLAs, free versus paid versus VIP, quality of the report, known issues, how angry the customer is, whether others will be hit. Every company is different, so you have to ask each one. Then document inputs, outputs, and, most of all, the reasoning in the middle — chain of thought, in model language. People make that judgment in seconds and almost never write it down, which is one reason models are still weak at reasoning: we do not emit the data exhaust of our own minds.

The checkpoint is specific. If you cannot produce ten high-quality examples with reasoning, you will not automate the work successfully. That is one of the most important gates on the path; miss it and you are in real trouble. Most tasks can gather ten without heroics — sometimes you only need to backfill reasoning onto historical tickets — but people often list five factors, go do the job for a while, and discover six through ten. Do not redesign the process on day one. Win a hated slice of the current flow, get invited back, and only then notice that ticket prioritization exists because humans cannot answer everything at once. An AI that can write every reply in parallel might make the classification step superfluous. That is a second project.

When several candidates compete, run expected value and then discount hard by chance of failure. Answering investor questions can look high-value per hour and still be a bad first target: “good” is vague, context is last month’s numbers, and the relationship changes the tone. Customer tickets are lower unit value, higher volume, and a clearer gold standard. Labenz’s usual advice is to start there, and to put the risk conversation on the table early if you are the person selling the work.

OpenAI’s own chart splits optimization into what the model needs to know and how it needs to act. Labenz relabels those as information and behavior, then sequences three techniques. Prompting comes first, and he uses the same blueprint every time: role (“you are a customer-service agent”), task (“assign a priority”), then instructions that dump every consideration from the interviews — paragraphs or pages; for frontier models, more is better. Format is next, and the non-negotiable line is AAA: analysis before answer, always. Put the answer first and you get post-hoc justification, “one of the most common ways” he sees people leave performance on the table. Attach the gold examples. With ten you can run ten tests, each using nine as few-shot and hiding the tenth. Often, in 2024, that is already enough. If it is not even close, you probably chose a bad task.

Retrieval-augmented generation is for everything that will not fit in the prompt or must be fresh. Some of it is deterministic: email in, account row out, or a public weather API. Unstructured policy lives in a vector store. His bias is to over-supply context. The model knows nothing about your business; extra tokens are cheap, missing facts force it to guess. Fine-tuning is the heavy move: change the weights so you own a specialist. The trade-off is brutal and useful — the custom model gets better at this job and worse at everything else — so the rule of thumb is one fine-tune per task. GPT-4o mini, then in the top five on LMSYS, cost $3 per million tokens to train and $0.30 per million input at inference, with the same rate limits as the public models. Gemini Flash, Claude Haiku in Bedrock preview, and Llama 3 were all open for the same pattern. Ten examples cost almost nothing and come back in minutes.

After the first run you land in one of four places. Nowhere close means the task is too big; decompose and try again. Close but not good enough means 10x the data, not more epochs — extra passes over a tiny set overfit until the model parrots. People rarely write the next ninety examples by hand; they generate a couple of hundred, keep the hundred that work, and optionally repair failures. A thousand examples is usually the ceiling for a single task; past that, decompose. When only edge cases remain, pick about five typical misses, write the correct chain of thought in contrast to the wrong one, and patch. Plan three core rounds — 10, 100, 1,000 — plus a few patch rounds, so calendars and client expectations survive contact with reality. Experts hop around this map as an art. Beginners should still de-risk the black box first, before anyone builds Zapier scaffolding that becomes stranded inventory.

Several operating rules sit on top. Maximize performance before you squeeze cost or latency; projects die because quality missed the bar, almost never because GPT-4o mini was too expensive. Compare the model to humans, not to perfection — every business he has seen has typos, misclassifications, and bad days — the same public nervousness that greets robotaxis already safer than people. Each extra “nine” of accuracy costs about 10x: 90 percent from ten examples, high nineties from a hundred, 99.9 percent nearly impossible (MNIST itself tops out at 99.7). Do not go blind after launch; sample and alert. And do not over-build. US electrification took on the order of 60 years; this stack turns over in a few. Bank the win and wait for the next model to obsolete some of your cleverness, the way GPT-4o mini expanded the tasks you can even fine-tune.

Waymark used to require customers to arrive with an idea and type the script themselves. People froze. The company now automates that stretch end to end, and users create about ten times more content. Athena’s onboarding is even cleaner as a unit of work: every new client sits for a one-hour recorded call. Converting that tape into a briefing for the assigned executive assistant used to take four hours and about $100. The pipeline now transcribes, then feeds a carefully written prompt — Labenz does not think they ever needed a fine-tune — and returns a better document for under a dollar, more than 99 percent cheaper. The client still talks to a human for an hour. The assistant still gets a running start. Only the expensive middle disappeared.

The failure modes are familiar. Engineers forget to price their own time and will spend expensive hours to save pennies of inference, a trade that is almost never worth it unless the user is waiting. Waymark’s old image-picking stack — captions, OCR, an aesthetic model, then a language model — got wiped out by multimodal systems; the complexity was a liability. When hiring help, watch the vibe. A green-flag adviser names the hard black box, insists on examples, and budgets several review rounds. A red flag says the work is easy and starts in Zapier. Labenz quotes his father: the other guy’s job is always easier. In an enterprise, add data permissions, privacy, and SLAs. If OpenAI is down for an hour and your contract promises a one-hour response, you have an incident. More friction makes a wrong first task much more expensive, so spend even longer on selection.

No-code is his default for internal tools. n8n can even be self-hosted; he does not particularly recommend the hassle, but the option exists. Inputs, outputs, and logic are the same language as the rest of the talk, with a model call sitting where intelligence is required. Custom software still has a place if you are shipping a product. For something that lives inside the company, no-code is faster, cheaper, and easier for a non-developer to tweak. Future-planning, not future-proofing, is the last slide. Electrification took generations; this is years. At Waymark the architecture has gotten simpler with each generation. Better to get five or ten workflows to good-enough, cash the checks, and ride rolling upgrades than to chase the last nine on a single problem the next model will swallow.

The action list he leaves behind is short enough to start the next morning. Write down three things in the business that are slow, expensive, repetitive, and preferably disliked. Pick one where you can assemble ten gold examples with the reasoning written out; if you cannot, switch. Draft the blueprint prompt — role, task, instructions, analysis before answer, examples — and run the ten leave-one-out tests. If it is not good enough, grow the set from 10 to 100 to 1,000, then patch five edge cases. When it matches humans, ship it and go automate the next thing. If you cannot produce those ten examples, Labenz’s harsher line is the one to keep: this job will not automate, so change the job.