From Autocomplete to Alien Optimizer: Riley Goodside on Three Eras of Prompting

Goodside describes himself as a data scientist for most of his career, best known as the only data scientist at OkCupid from 2011 to 2015. The most advanced ML he used there was gradient boosting and random forests, and the hard problems were often about pricing. He later worked on time-series problems at startups and ran data science at Grindr in 2021.

He was intrigued by the GPT-2 announcements but assumed training such models was a supercomputer problem, so he did not dig in. His first real contact with GPT-3 was through AI Dungeon, where early users coaxed it into odd behaviors and even found proto-examples of prompt injection. What really caught his attention was GitHub Copilot and Codex, after he left Grindr and took a sabbatical.

He began tweeting Playground examples and following engineers working on Copilot and at OpenAI, knowing he was out of his depth on architecture and wanting to be an advanced user. One factor in his success, he jokes, was making his window narrower so his screenshots were legible on phones, in a consistent green-and-white style that showed what was prompt and what was completion. Tweeting led to an invitation to a party from Nat Friedman, where he met Alexandr Wang, and eventually to Scale.

Goodside says many of his best tweets were demos rather than rigorous evaluations. He knew what the model could do and arranged impressive tasks around that, avoiding things like reversing a word, because models see tokens of about four characters rather than letters. He likens this to arranging a ballet over lava: he knows exactly which rocks are solid.

His early demos showed that the model could follow very long, intricate prompts, with pages of tasks referring to one another. He says OpenAI's Boris Power told him they had not trained for that: they tuned on simple prompts like ten ideas for an ice cream shop, and instruction following generalized to unseen lengths. That convinced Goodside he was noticing real capabilities, especially ones at the fringe of reliability that models would later master.

He then craved structured output, since the GPT-3 API is effectively one string in and one string out, so any structure must be parsed by you. His favorite trick is what Power called the format trick: end your instructions with "use this format," give a template with angle-bracket placeholders for the changing parts, and the model understands exactly what to produce. He started exploring variations of this in summer 2022.

Goodside says you must understand the narrative of how models are made. The first era is pre-training: a neural network produces a probability distribution over the next token, much like phone autocomplete. Do this well enough and abilities emerge, such as completing two plus two equals four, or translating after seeing French and English labels.

The GPT-3 paper framed this as few-shot or in-context learning, where examples in the prompt let the model get the gist. Goodside prefers a framing from Reynolds and McDonald, in which a pre-trained model is a superposition of all possible documents and each added word excludes some. After ten correct examples, the odds that the eleventh answer is wrong are low, so the document space has been constrained to correct answers.

That view explains a surprising result: zero-shot translation can beat ten-shot if you flatter the model. Writing that a masterful French translator flawlessly translates the sentence establishes a fiction in which only good translations occur. He adds that this predictive view also explains how a model will invent a plausible Oxford English Dictionary definition for a made-up word, or explain why it is a squirrel if asked, since it simply continues the premise.

Goodside says he joined the conversation around April 2022 when text-davinci-002 already existed, which belongs to the second era, instruction tuning, whose key search term is InstructGPT. A pre-trained model asked for the capital of Germany might just ask for the capital of Spain, since it assumes it is reading a list of questions. People had to use tricks like Q and A formats to get answers.

Fine-tuning had two uses: mimicry of a style, and training on thousands of correct examples of a task so it behaves as if prompted with them. OpenAI extended the second idea to everything. Contractors enumerated tasks people might want from a chatbot, wrote typical prompts and then wrote correct completions, and Scale contractors did some of this work. Tuning on that corpus made models assume the document is instructions followed by their execution.

Goodside describes this model as an interpolation of human demonstrations. Because human demonstration is expensive, the next step was to automate it: let the model produce answers and, if humans agree they are good, add them to the pile. That process, combined with tuning from Codex, led to text-davinci-002, which he notes descends from the code-davinci models.

The third era arrived with text-davinci-003, released the day before ChatGPT at the end of November 2022. Goodside says it was the first RLHF-tuned completion model OpenAI offered, though many people wrongly thought the earlier one was. In RLHF, the model generates answers, humans rank them, a preference model learns to rank like a human, and that preference model then scores generations automatically.

This helped with false-premise questions, which he used to treat as the archetype of what GPT-3 could not do. Douglas Hofstadter and David Bender had published questions in The Economist in June 2022, such as when the Golden Gate Bridge was transported across Egypt, to show the hollowness of GPT-3's understanding. With ChatGPT, Goodside says, all but one of them were answered sensibly, the exception being a question about the world record for walking across the English Channel, which still tempts the model to hallucinate.

In Goodside's view, an RLHF model is no longer well described as an interpolation of human text. There is a hill to climb, and in principle it could become superhuman in the way AlphaGo is at Go, because it is optimizing to satisfy a preference model. It is still text prediction, but on a very alien body of text, and it should not sample answers from the internet's distribution, such as claiming a random gender or country, but behave as if aware of what it is.

Host Nathan Labenz asks whether instruction tuning and RLHF are qualitatively different, and about mode collapse. Goodside notes that one of his tweets, showing that GPT-3 was bad at describing letter shapes and kept giving a box with an X, was cited in Janus's LessWrong post on mysteries of mode collapse. The stronger example is that text-davinci-002 picked 97 about 20% of the time when asked for a random number from 1 to 100, whereas the pre-trained distribution was more uniform with a slight bias toward 42.

His speculative explanation is that the reward model learned imperfectly that every number is equally good and slightly favored 97, and the language model was smart enough to exploit it. Later RLHF models show fewer vivid cases but the general pattern remains. The model learns that there is a correct answer and that it should put all its probability on the best one, which can misgeneralize.

The practical symptom is in creative writing: ask for product descriptions for thousands of products and the phrasing is repetitive in a way pre-trained models of 2019 and 2020 would not be. Labenz suggests this means something may be lost in sculpting models for the highest-rated output. Goodside agrees that this is a real drawback.

Asked how to think about these systems intuitively, Goodside says he now spends less time on text formatting and more on how pieces fit together. His advice is to treat each capability the model does well as a Lego brick and compose them. LLM-powered search is an example: Perplexity-style products apply a model to parse search results, and problems like mixing up two people with the same name are an enumerable set that will be fixed one by one.

He recommends LangChain as a grab bag of techniques and describes Scale's Spellbook, a platform for deploying prompts as APIs with parameters and for comparing prompts and models. That helps teams decide when to switch to a cheaper model, since prices differ hugely between GPT-4, GPT-3.5 Turbo and open models, and sometimes a fine-tuned T5 is enough.

The archetype of what works well is any problem that in 2017 would have required picking an ML architecture and collecting a dataset, such as sentiment classification or entity extraction. His example is extracting US cell carriers from tweets despite abbreviations, misspellings and sub-brand handles. Now you write a page of instructions, a list of handles and a few edge-case examples, and when a new carrier appears you add one line instead of retraining.

Labenz compares prompting to briefing an intern, then brings up a hackathon project called GPT is all you need for back end, in which the model plays an API that returns JSON for invented methods, with no code or database. Goodside calls it a vivid example of using an imaginary computer. He cautions that context limits mean any state must be resent in each prompt and the model will hallucinate changes, and he sees the format trick as another way of defining an API with an input and output.

On GPT-4, Goodside says much of what he called tap dancing across lava in 2022 is now normal: long instructions are followed. Longer context opens up Q&A on GitHub repositories and whole pull requests as context, and he had been curious about representing multi-file input to generate whole Python packages. He thinks more software will be written by people who did not think of themselves as programmers, and says he can muddle through TypeScript with ChatGPT's help.

He is especially interested in multimodality, noting the demos of explaining memes and solving an engineering exam problem from a photograph in French. He says it could also enable training pipelines using synthetic photos with embedded text. Labenz adds that approaches like BLIP-2 inject image embeddings into the language model, and cites Kosmos-1 and PaLM-E as other examples.

Goodside says OpenAI's edge is eroding, since it used to have the best models people could actually access. Claude and Bard now seem to be in the same general range, with differences in speed and inference cost. He expects rapid progress now that LLaMA weights are out, quoting Simon Willison that large language models are having their Stable Diffusion moment.

To compare models, he builds a minimal softball version of the task, such as escalating a customer support inquiry, and then creates progressively harder variations to see where each model fails. Prompts must be tuned per model, especially for chat APIs, where few-shot examples become pre-populated assistant messages or are stuffed in one user message. Chat models can also break format on policy-violating requests, replying with a refusal instead of JSON, and one trick that helps is adding a user message afterward saying that was great, exactly what I needed.

He suggests using a cheap classifier to decide if an input needs a bigger model, and caching responses to common questions. On sampling, he likes temperature zero for reproducibility when studying behavior but higher temperature with top-p when he wants creative diversity and to fight mode collapse. For consensus voting across multiple generations, he wants diverse approaches, so it does not help if every sample collapses to the same answer.

Asked about the economic impact, Goodside says intelligence will get cheaper, which will shift what humans do, though he calls himself an armchair economist on the net effect. He sees paralegal-style work as most automatable: people reading reams of documents to find the relevant case or summarizing reports. He thinks demand for skilled people may hold since they become more productive, while less skilled clerical and temp work, like copying from PDFs into structured text, will be hit harder.

Personally, he says he has acclimated to the right level of skepticism after years with hallucinating models. The risk is that people assume that output with academic-looking footnotes is reliable. For someone who fact-checks, though, it is valuable for explaining tax code or finding the JavaScript equivalent of a Python library, and for niche questions that would suit a Stack Overflow expert.

He uses Copilot most often, then GPT-3.5 Turbo or text-davinci-003 for code, and it works best on libraries well represented before 2021, such as SQLite. It also saves him from SEO content farms for lookups like Unicode characters. Labenz recommends GPT-4 as a second opinion on medical questions rather than a doctor, such as preparing before an appointment, and Goodside adds that it is good at well-settled science, like explaining alpha-2 adrenergic receptors.

Goodside, who is helping build a red-teaming capability at Scale, defines red teaming as adversarial use of models to find how they break so mitigations can be developed. Jailbreaks such as Dan-style fictional scenarios are the best-known example. Beyond toxic output, companies also need rules about soliciting or divulging personal information and about encouraging self-harm.

He considers the concerns real, citing scenarios in the GPT-4 technical report such as ordering custom chemicals at home, and spam generation. Pre-trained models created a new category of misuse risk for anyone deploying them. When he first tweeted about prompt injection in September, he thought he was making a public service announcement about safe quoting of inputs, not flagging a high-severity problem. Bing's exploration of its shadow self with Kevin Roose shows how hard fixes are.

Still, he thinks models are getting safer because they can be tuned to follow more subtle rules, and RLHF is not only about getting models to behave but about making them follow directions and be more capable. When Labenz proposes pausing scale-up in favor of interpretability, Goodside replies that the two are hard to decouple: better alignment lets you deploy a model with more responsibility, and it is hard to pause either. He wants alignment research accelerated as much as possible.

In the closing questions, he singles out text-to-video, mentioning Runway's Gen-1 and Gen-2, and says he would not want to be an early Neuralink adopter. His personal estimate is that AGI arrives within the next decade, with an adjustment period when AI is human-level but not yet rapidly improving itself. He hopes early near-human AI will help make better progress on aligning later models.