Prompt Engineering Became Programming, and Riley Goodside Is Fine With That

Riley Goodside, still Scale AI’s first staff prompt engineer, comes back to The Cognitive Revolution quieter than the text-davinci-002 era that made his feed famous. The biggest reason is ordinary: in June 2023 he and his partner had a daughter, Felicity, and work plus a newborn left less time for extracurricular posting. The job he once treated as a campaign to break into the industry is done. Scale’s customers include much of that industry, which makes the irreverent material he was known for harder to publish.

The deeper reason is the craft itself. In 2022 you had to know stupid things, he says — do not end a GPT-3 prompt with a space — before you could get a good completion. Chat models are friendly enough that many people get by without ever hearing the word tokenization. They still hit gaps, but ignorance is survivable. That is why the tip-and-trick posts dried up. Folk prompts that fit in a tweet, “let’s think step by step” among them, are less relevant, and the ones that still work are no longer mandatory.

What replaced them is heavier. Prompt engineering has not stopped flourishing; it has moved into code, retrieval, wrappers, and tools such as DSPy. Riley’s teaser, which the rest of the hour unpacks, is that the work now looks like programming. Nobody wants to do code review on X. Labenz, speaking as an intensive user and as someone watching Scale’s customers, had joked that nothing happened after GPT-4. Riley does not buy the stall story. The progress is just harder to name in a screenshot.

Asked what has actually mattered since GPT-4 went public, Riley points at REPLs: ChatGPT’s Code Interpreter, now Advanced Data Analysis, and projects such as Julius AI that give the model an interpreter to live in. There is still alpha, he thinks, in that capability. We are in a regime where language models would benefit from knowing how to use language models — from having the situational sense a user already has about what they are good for.

GPT-3 would not say “I don’t know.” You had to remind it that inventing an answer was optional. That kind of knowledge is getting easier to skip and is not gone. Tokenization traps are the classic leftover. Pre-training on an internet that did not yet contain chatbot dialogues meant the fictional character of a chat assistant had to be painted on in post-training. The modern internet is full of those dialogues. People filter the ones worth sharing; the best of them will be fed back in. Recursive use — code that calls models, models that understand that code — is the lift he expects next.

Tool use is where that hope meets a rigid API. Riley would rather fine-tune than trust a long prompt, because real-world endpoints have semantics a model forgets: the shape of a JSON argument, the difference between two similar calls. Few-shot and long context can collect the data you later tune on. If the generation has to be structurally exact, a handful of examples already helps. Labenz’s worry is the opposite failure: start from a general base, tune it on your internal tools, and watch it reach for those tools when it should not.

Riley’s usual progression is stubbornly unglamorous. Start with an instruction prompt. Add k-shot examples until no more will fit. Only then worry about how you select those shots — embeddings are optional; any method that works is fine — and only after that wander into retrieval. Fine-tuning comes last, both because it costs more and because the earlier steps are not throwaway. A wasteful long prompt still produces high-quality answers you can recycle as shots or as tune data.

The step people skip, he says, is decomposition. At every stage ask whether a piece of the job can be pulled out and given to a smaller model, or to something that is not a model at all. Dig into the data. Use your intuition on a single broken example. Annotate. Think about the distribution of realistic inputs, and especially about the boundary cases where the output is ambiguous. You want every real input to look like an interpolation of the demonstrations you already wrote.

Sometimes the pipeline is the product. Riley cites a Microsoft system that beat a medically fine-tuned PaLM variant with a complicated prompt-engineering stack on a non-specialist model. That is not the answer anyone wants to hear — a massive inference graph — but it is how a lot of real problems end. At the limit, k-shot selection makes the task trivial: if you can retrieve the problem you are about to solve, the model copies the answer. Treat selection as pre-computation, and ask which bits of the reasoning should be explicit at which step.

Labenz has been walking the same ladder for an upcoming episode on automation, and he keeps hitting the same hole. A customer-service ticket and the reply that went out are not a demonstration of the task. The internal notes field is usually empty. The actual thought process — I am doing this because I noticed that — lives in someone’s head and has to be interviewed out. On GPT-3.5, input-output pairs were not enough. Explicit reasoning traces were the difference between a model that sort of imitated the house style and one that grokked the job.

Riley agrees, and he generalizes it. Natural text sometimes reads like a recording of conscious thought. Calculation is a violation: when a paragraph casually states what 57 times 125 is, somebody stopped, used a calculator, and resumed. Years of experience get collapsed the same way. If you let a model write the longhand — think through the digits, name the intermediate checks — it is more likely to be correct. Not perfect. Better. Getting that kind of thought onto the page is rarely wasted, and synthetic chain-of-thought is one way to do it.

He draws a distinction people miss. Chain of thought the *same* model generated, filtered to answers that landed correctly, teaches that model its own idiosyncratic path: when it zigs this way it scores. Human traces, or another model’s traces, do not capture those private deductions. You can still flip fields — every English-to-French pair is also French-to-English — and build a synthetic pipeline. The Walnut-53 paper is his concrete case. Researchers built a cipher the base model could not read, tuned it through bridge tasks (repeat the encoded task in plaintext, then finish in cipher), and slipped unsafe demonstrations past a monitoring model that could not decode the payload. Decomposition, again: teach the missing skill, then compose.

Labenz wants the in-house chatbot that is “just like ChatGPT, but it can hit our APIs,” without overfitting so hard that every casual question becomes a tool call. Riley’s first advice is to avoid the problem if you can. Wait for the next base model. Fine-tuning is not free, and one of the prices is general skill. A recent paper, he says, showed that tuning on any new factual information makes a model more likely to hallucinate in general. The mental model he keeps is simple. Pre-training taught the network a deep instinct for what belongs in the real-world corpus. Tune it on something outside that body and you are telling it that the weird stuff is sometimes allowed. Ignore that instinct often enough and it invents.

The classic mitigation is to mix general chat back in — you do not have the pre-training set, so you find a cousin. On open weights you can also mix the weights themselves: tune on the narrow task, evaluate on that task and on the general benchmarks you care about, and search for a blend that holds both scores. A trick Riley has only heard anecdotally, not run himself, is to tune the *base* Llama rather than the instruct model, take the delta, and apply that delta to the instruct checkpoint — a crude way of sliding new knowledge under the RL polish instead of scratching it.

Which model should you tune first? The one that wins on your task. Poetry and code do not have to share a winner. Leaderboards are a shortlist, not a verdict, which is why Scale’s SEAL boards split coding from precise instruction following and build the sets from role-play scenarios rather than exam questions. Language-by-language rankings rearrange themselves. A model that writes Spanish technical docs and then snaps back into English will never show that on a public board and will ruin your week. Cost inflates just as fast as quality when you add consensus pipelines. After that, Riley sees the remaining spread among frontier models as mostly post-training. The platonic-representation work Labenz cites — bigger models’ internals looking more alike — fits a view of these systems as a compression of text. Gains keep coming from demonstrations, not from another jump in parameter count, and Riley does not treat that as a roadblock. The space of things you could train on is still wide open.

If you spend the elbow grease — generations, retrieval, fine-tuning, iteration — how close is a *task* to human level, and what does that have to do with AGI? Riley dislikes the usual definition, the one that sounds like OpenAI’s “vast majority of economically valuable labor.” The target moves. Before the industrial revolution a huge share of people farmed; in the United States that share is now a couple of percent. Translation and transcription are already pushing people into other work. Freeze the list at 2024 and you still have not captured what people picture: a system that is itself the fountain of new things to do. He thinks we are a long way from talking coherently about machines that have finished philosophy.

The practical test is narrower. Language models can do tasks, not jobs. If you can get the work down to a repeatable unit, inputs and outputs laid out as text on a spreadsheet, you may be able to automate it. Novices ask for a business strategy or a startup idea. He cannot imagine a model giving a good one. Humor is the archetype of the failure: it ages, it competes, it has to surprise. World-class jokes from the 1930s do not land now. The less a job looks like humor — the more there is a standard of care, a best practice, a consensus about what good is — the better your odds. Without that consensus you are in trouble.

He will not pretend there are no sparks. ChatGPT once connected a throwaway remark about the Python library `aers` to a Dutch insult Nathan had heard in a YouTube video, a link that almost certainly was not sitting in a transcribed blog post. Google’s FunSearch pushed the state of the art on packing-style math problems by generating programs, scoring them with an objective function, and iterating. The catch, Labenz says, is that most things people want — a joke, a social post — do not come with that score. Paying 10× or 100× for tokens is fine if you can find the one good sample.

Riley’s advice for the “I will pay for a hundred generations” case is to stop scoring and start comparing. Generate several answers, even pass them to a long-context model and ask which is best. Better: pairwise wins, losses, and ties, then an Elo ranking. How well a model does is happenstance; more attempts are the cheapest intelligence you can buy. He recalls a paper that put the exchange rate at roughly forty GPT-3.5 calls for one GPT-4 call. You can keep generating with the stronger model, rank, and take the top.

Wherever your domain has a real indicator, use it. Unit tests and execution for code. For translation, go English to French and back and ask whether the English survived. Most of the work of making these systems good, he says, is ad hoc quality filters so that you are not stuffing bad demonstrations into the prompt or the tune set. Retrieval that merely dumps similar chunks still leaves the model doing novel work at inference time. If you are going to fine-tune, the demonstration has to look like the real input: the question plus the chunks, then the answer those chunks support. Recasting documents into synthetic questions gives you a place to run those checks before production.

Labenz recognizes the move from the summer of 2022, when text-davinci-002 fine-tuning was briefly available and the models were weak enough that you had to grind. The core techniques — synthetic transforms, backfills, generated demonstrations — are the same. Off-the-shelf systems are now good enough that people get tricked into thinking a prompt tweak will finish the job. They are still not good enough, Riley and Labenz agree, for most of the production work companies actually want to automate.

Is he still a staff prompt engineer, or has he been rebranded an AI engineer? The title has not changed. Whether anyone will hold it in a couple of years is anyone’s guess. What is clear, Riley says, is that the early-2023 “job of the future” story did not happen. Most firms, especially outside tech, do not hire a full-time prompt person. Demand is concentrated inside the AI industry itself: people preparing training data, filtering it, doing quality control. At Scale, and at the labs, those applications are abundant. He never believed everyone would do this for a living.

The analogy he used last time was typing. Typists used to be a job; everyone still types and nobody calls themselves a typist. He is not sure the analogy still fits. The 2022 tricks are less mandatory. What is flourishing is the code-heavy layer — retrieval, DSPy-style optimizers, wrappers — often under other names. Back in 2020 people told him prompt engineering would not last because whatever there was to know, you could teach the model. That is not literally how anything works today. Metaphorically it is. The best use of a trick is to produce better training data and retire the need for anyone to know the trick.

He pictures a forest fire. Prompt engineering is the flame on the perimeter. Any particular tree does not stay burning long. Labenz’s version is a black hole: the models suck the method in and nothing escapes. Both sketches imply the same next half-step. Multi-step agent traces, the places today’s workflows get stuck and retry the same action, get folded back into post-training. Riley thinks that picture is about right, and that entire avenues of text — DNA, forgotten formats, a DeepMind setup that distilled Stockfish’s win probabilities into an ordinary language model — are still sitting there waiting for someone to lift them into the corpus.

On the jailbreak side, Labenz asks whether Pliny-style attacks will be trained out or whether we will always need a layer on top. Riley splits the question. There is growing evidence you cannot do everything in the weights. Anthropic’s many-shot jailbreak — hundreds of malicious prompt-response pairs stuffed into a long context — overpowered fine-tuning on polite refusals. The same paper found a cheap fix: a suffix reminding the model it is Claude and that it is aligned. He would not trust a tuning-only future. Token-repetition attacks that regurgitate pre-training data are the blunt example. A filter that says “the user is repeating the same token” is easy. Teaching the model not to spill is hard, and not worth it.

That answer leans on structured access, which leaves the open-weight question uglier. Once the weights are out, people do what they want. Riley does not know a method that keeps an open model and also holds the line, short of scrubbing concepts from pre-training — a tactic that became less popular after Llama 2 because of what it costs. Labenz offers two research bets still in the lab: Sophon, a Chinese-lab idea that trains an image generator to output a null transform on a restricted domain with a smooth-bottomed loss so the gradient vanishes and fine-tuning out of the well is hard; and Dan Hendrycks’s short-circuiting work, an internal classifier that is still a system around the model. Neither is an answer at language-model scale. Riley is hearing of Sophon for the first time.

The last two questions bounce off. State-space models excite him as an outsider who once did massive multivariate time series; he will not pretend to have an architectural take on Mamba. ARC pleases him because it looks like Raven’s matrices — wallpaper with a missing tile, no vocabulary required — a bet that generalizable intelligence might be a simple task rather than an SAT of accumulated facts. He is unsure the bet is real. The thing that worked was making models know everything. Scale is hiring prompt engineers, especially people who can do impressive things in security and red teaming. How do you stand out from another résumé that loves ChatGPT? The avenue that worked for him, he admits with a grain of salt, is still the same one: let X sift the ideas and let the good ones bubble up.