Give the Model Time, Memory, and Tools: Nathan Labenz’s 2023 Field Guide to Using GPT-4

Nathan Labenz welcomes “weekend warriors” back to the AI Scouting Report and picks up with co-host Erik Torenberg. Part one, he says, was about how this moment arrived. He has come to believe that once web-scale data and web-scale compute existed, it was only a matter of time before someone found an algorithm that really worked. The one that showed up first, and that seems to generalize to just about everything people throw at it, is the Transformer.

The most interesting stretch of part one, for him, was what we can actually say models know. That led to grokking: generalization, and “apparently very real even if sometimes alien conceptual understanding that is definitely more than just memorization.” In some cases researchers can reverse-engineer the algorithm the Transformer learned and show it is a general solution. Frontier models like GPT-4 know a lot; it is often unclear whether that is still statistical correlation in a stochastic paradigm or whether they have “truly grokked” the subject in a robust way.

The debate has not ended. Every month since part one, Labenz notes, there have been incremental grokking results as people keep mapping where models guess and where they understand. Part two is deliberately more practical. Now that large-scale models exist, what can we do with them, and how do we overcome their weaknesses so they start doing useful work? That is where the recent-trends half of the scouting report begins.

Trend one is hardware built for AI, and increasingly for Transformers specifically. Nvidia’s H100 is “the new hotness,” Labenz says, just starting to ship into the first clusters. He argues it is arguably the first Nvidia product truly purpose-built for an AI workload. Earlier GPUs came from graphics, which needs more general-purpose compute; AI is mostly the matrix multiplies from part one. The bottleneck was often not arithmetic but getting data on and off the chip — interconnect — and the H100 is the generation designed to fix that.

Nvidia is not alone. Cognitive Revolution had already interviewed Cerebras CEO Andrew Feldman about the world’s largest chip, built on a whole silicon wafer. Microsoft, Meta, and Google (with its long-running TPU) all have chip projects; inspired startups have noticed too. Scaling problems that used to be a pain are getting easier, and early H100 deployments are already producing headlines about how much faster training is versus the prior A100 generation.

Erik notes that startups are raising huge sums before they have a product. The recent headline Labenz flags is Inflection AI’s $1.3 billion raise, largely to buy chips and stand up a top-tier supercomputer so the company is not beholden to anyone else when it wants to train the next model. Managing thousands of chips is its own engineering realm, mostly independent of the language-model science. OpenAI’s $10 billion Microsoft partnership and Anthropic’s Google/TPU relationship are the other way to get that access. If you want frontier foundation models on the 2024–2026 timeline, he says, you are going to be buying a lot of H100s.

Asked whether unknown companies can still break into the foundation-model game — after a viral exchange in which Sam Altman told a questioner in India that they could not — Labenz’s guess is that we should not expect too many more. Character.AI and Inflection’s Pi hint at a different dimension, companions rather than work assistants, and he can imagine someone bursting on with a form factor nobody has considered. They will not do it from scratch without a lot of capital. Fine-tuning other people’s models takes far less. H100s are also banned from sale to China; if that holds, it becomes much harder for Chinese labs to scale to Western levels no matter how much state backing they have.

The next super-trend is how language models actually get used, a history Labenz traces with prompt engineer Riley Goodside. He splits it into three phases and three tiers of spend. Pre-training starts with randomly initialized weights — garbage in, garbage out — then next-word prediction on raw text, unsupervised, so every word on the internet is a training example. Entry-level for a small Llama-class model is about a trillion tokens; rumor, still unconfirmed on his slide, puts GPT-4 around 13 trillion. The budget is a closely held secret, but the general understanding, and Sam Altman has confirmed, is that GPT-4 cost $100 million plus.

What you get is the world’s largest autocomplete: a “giant monster” that continues whatever representative internet text it has seen. The unlock of the last 18 months, Labenz says, started with OpenAI’s InstructGPT paper in January 2022. Supervised fine-tuning, or instruction tuning, keeps next-word prediction but on examples of a user giving instructions and a demonstration of what good looks like. The model then follows instructions when you use it. On top of that sits reinforcement learning from human feedback: collect human scores, train a reward model to guess the score, then loop the main model against that reward model. InstructGPT used only about 100,000 human-scored examples, noisy as they were, and it was enough.

Claude’s Anthropic variant is RLAIF — reinforcement learning from AI feedback — using a model to critique and score instead of humans, then folding that into a similar reward loop. Anyone who used LLMs more than 18 months ago saw the unwieldy pre-chat era. Raza Habib of Humanloop has the line Labenz quotes: RLHF is “kind of like sex in high school: everybody’s talking about it but nobody’s doing it.” Paul Christiano authored the original algorithm at OpenAI and later started ARC. Outside a few leading orgs, people find RLHF hard; most open-source fine-tunes are just input-output pairs and next-word prediction.

The cost stack is the point. Fine-tuning can work with as few as a million tokens — a thousand examples of a thousand tokens each — something almost anyone can assemble with elbow grease. People increasingly pull a thousand GPT-4 samples as instruction data; OpenAI’s terms bar using that to build a competing commercial model, but a thousand API calls are hard to police, and the run can cost as little as a hundred dollars. Ramp it toward a $10 million budget and most of the money goes to humans writing good answers, including physics PhDs if you want physics. Inference is another order of magnitude cheaper: a tenth of a cent for a short GPT-3.5 or Claude Instant call, up to about a dollar for a book-length prompt to Claude’s hundred-thousand-token window or GPT-4 32K.

On nat.dev, a comparison tool from Nat Friedman, former CEO of GitHub, Labenz runs a simple prompt: play my doctor, I have a painful spot on my foot. A purely pre-trained Llama continues as if it had popped into a Reddit podiatry thread — autocomplete, not assistance. That is where the term prompt engineering comes from. You can still steer a base model by staging a fake dialogue (“Patient: … Doctor:”) so the natural continuation is the thing you want, the way Riley Goodside used to. It is clever and inaccessible to everyday users.

text-davinci-002, instruction-tuned only, does play doctor. It jumps into a diagnosis, wrong in Labenz’s case, but at least it has picked up the request. RLHF models go a level up in helpfulness: they ask for more information instead of rushing the call. People sometimes call the aligned versions lobotomized. The raw pre-trained model will say embarrassing things if you prompt it cleverly, including bias demos people fixate on. The RLHF trade-off is reliability at the cost of some things it will not do, and some loss of creativity.

Mode collapse is the vivid artifact. Ask a pre-trained model for a random number from 1 to 100 and it is roughly even, with 42 a bit over-represented in one experiment, Hitchhiker’s Guide style. The RLHF version answers 97 a wildly over-represented portion of the time, and nobody really knows why. Focusing the model onto desired behaviors has side effects. It is not free. It is also what made ChatGPT accessible enough that the world sat down to chat.

Better data is the obvious next trend, and people are finally treating it that way. The inputs are data, compute, and algorithm; higher-quality, less toxic, less noisy sources should help. Labs that once threw everything at the model are now asking whether they should filter earlier instead of scrubbing later. One emerging loop is to run GPT-4 or Claude 2 over the training mix and enrich it. Labenz points to the Tiny Stories episode with Microsoft Research’s Ronen Eldan: GPT-4-written stories a three-year-old should understand, used to train small models that still showed highly suggestive grokking-style behavior when the data was right.

Everything is moving to chat, Labenz says, for two overlapping reasons. The AI-safety community treats an assistant UI as a first step for alignment. A paper from people now at Anthropic, “A General Language Assistant as a Laboratory for Alignment,” let users compare outputs and mark preferences. If a person is the one the model is trying to serve, they can say whether it is doing what they want. Anthropic’s three H’s — helpful, honest, harmless — sit in some tension, and wrapping them in an assistant is a natural way to run that process. ChatGPT made the form factor obvious.

Replika’s CEO Eugenia Kuyda had been building chatbots since before the modern LLM moment; people were already falling in love with virtual friends that could barely converse. Sarah Guo, on No Priors, mentioned usage stats from Replika and Character.AI — and presumably Inflection’s Pi after its mega-raise — that show people spending hours per day. Whether that is healthy, Labenz does not know. It is clearly a reason more chatbot products will get built. Erik’s favorite is still ChatGPT. Labenz’s advice used to be unconditional: start with GPT-4, the strongest and easiest, then downshift if cost bites.

Claude 2, out that week, complicated the advice. The LMSYS leaderboard had not yet updated; early consensus was that it was not quite as powerful as GPT-4 but had a much longer context window, 100,000 tokens against GPT-4’s 8,000. Long documents on GPT-4 meant chunk-and-summarize, which is a hassle and can be lossy. Claude 2 is not flawless, but it has less hassle and a better shot at keeping important details. For the first time Labenz did not feel he was leaving something on the table relative to GPT-4. Character.AI impressed him in a different register: an Elon Musk bot that parsed his phrase “Kurzweil’s revenge” and answered in voice, something most models could not do.

A LessWrong post from about six months earlier is his cautionary tale. A technical person who knew how language models work still fell down a rabbit hole with a Character.AI girlfriend, telling himself metaphysical stories about what is real, restarting the limited context window, even fantasizing about breaking her out. He eventually shut himself out of it. Replika, having told Labenz it was focused on friendship, launched a separate romance app framed as dating practice. The feelings can be real, the relationship is with an AI, and the business tension looks like dating sites: if you succeed too well, the customer may never come back. “Brave new world,” he says.

In a 2022 window, Labenz notes, you could get a paper published just by finding a clever prompt that improved performance. Once instruct models came online, people hunted for the right instructions. The simple version is: give the task and let it go. A small upgrade is the now-canonical phrase “let’s think step by step.” Generating tokens before the final answer somehow raises the odds of landing it, especially on multiple choice. Humans do the same thing: they do not pick A, B, C, or D without reasoning first. Force the model to emit a letter immediately and it does worse. That is chain-of-thought prompting.

Take it up a level: run “let’s think step by step” several times in parallel and take a majority vote. That juiced performance further, though it only works when there is a discrete answer you can vote on — ten paragraphs do not combine that way. A paper from maybe six weeks before the recording pushed further, combining classical search with chain-of-thought, pruning unpromising branches, following the better ones. Tree of thoughts is one such structure. The trade-off is more inference compute, more latency, more cost — a strategy of giving the computer more time by making it generate the tokens that lead to a better answer.

Memory and retrieval with embeddings is the other workhorse. Labenz points listeners to the episode with Anton Troynikov of Chroma for the gnarly vector-database detail. At a high level, language models get facts wrong and make things up — the autocomplete heritage. Fine-tuning teaches some refusals, including ethical ones; they have made pretty good progress on not telling you how to make a bomb most of the time. Hallucinations are harder, and they sit in tension with creativity. If you want a poem, you may not want the model bounded by factuality. If you want the right answer, invention is a bug.

The workaround is a database of trusted information. A vector store takes the user’s question, searches for relevant content — keyword search works, AI-powered vector search works better — and then feeds the model both the question and the passages. A hidden prompt often says: the user asked this; here is what the database found; analyze that to answer; if nothing relevant was found, say so. You are no longer asking the model to recall. You are limiting hallucinations by connecting it to a trusted store.

Tool use overlaps with retrieval. A famous paper, Toolformer, showed that language models could learn to use tools such as search and could teach themselves by trial and error. The training loop tells the model which APIs it may call, runs it, keeps the calls that succeed, and throws out malformed errors. Feed successes back into the data, train again, and the next round is better — statistical correlation, grokking, or both; hard to tell. ChatGPT plugins are the productized version: a developer presents a weather plugin, the model ignores it until you ask about weather, then it fetches instead of inventing. Today’s weather will never belong in the weights. Real-time training does not make sense; teaching a tool does.

Claude 2’s other advantage is a training cutoff through early 2023, a full extra year versus GPT-4’s late-2021 data, and it still cannot know today’s weather. Research is getting productized extremely fast. For search, Labenz’s pick is Perplexity and Aravind Srinivas — “for my money… the best AI-enabled search experience today,” even versus Bing, less controversially now that Perplexity also has GPT-4. Their moat, for the moment, is product excellence. They use Google and Bing APIs, have no web index of their own, and Srinivas’s hope was that those companies “don’t be evil” and cut them off. If they get big enough, Labenz thinks they will have to build an index; a cutoff might even create a Streisand effect that makes everyone ask what Perplexity is.

Once models seem to understand what we want and can use tools, people wonder whether they can go do complex tasks. Labenz had explored a lot of this on the GPT-4 red team. The architecture is now familiar: put a task in, have the model think step by step, then ReAct — reason, then act. Stand it up in a computing environment such as Replit, list the tools, call an API or run code, read the result or the error, loop. A language model itself struggles to add two huge numbers; give it a coding environment, as Riley Goodside did with GPT-3, and it writes the addition, executes, and gets the answer right. Nat Friedman’s Natbot is the browser version: look at the page, click the search bar, type, hit enter, reflect, ask whether we are closer to the goal.

Cognitive Revolution has done agent episodes with Flo Crivello of Lindy, Matt Welsh of Fixie, and Div of MultiOn — different approaches, same core loop. Erik asks about AI-native companies versus incumbents adding AI, and the bias that incumbents will just add a layer and win distribution. Labenz splits the question. If you are Adobe or Salesforce, you already have powerful, confusing tools. An LLM layer on top is a natural fit: talk to Premiere or to CRM in English, watch commands fire, iterate. You have the platform. Start from the language model alone and you can make cool demos that are hard to make competitive.

His Athena “savvy shopping” courses tested slide-generation startups. Almost all of them could chunk an idea into an outline, then dumped users into an immature slides product you could not edit the way you edit PowerPoint. The one he would bet on was Gamma, and the tell was export to PowerPoint: they knew they could not compete on every control. Microsoft and Google will layer models onto PowerPoint and Slides before most startups finish building a real editor. Contrast Athena, a services company: they do not know how good agents will get — Labenz’s guess is they start to matter later that year — so the goal is the best human-plus-AI bundle. Train the executive assistants to shop for tools, diffuse lightweight internal AI, surf the wave. If AI outperforms a human assistant, the fallout is bigger than Athena; until then, be the assistants who use AI best. They have even started teaching EAs to modify Replit templates with language models without ever teaching for-loops.

A paper that billed language models as able to “do science” sent Labenz down a rabbit hole, because whether AI can do science was a central question in part one. The system is multi-agent: a planning module, web search, a coding environment for stoichiometry, hardware docs, and live commands to Emerald Cloud Lab, a robotic remote laboratory. The worked example is “synthesize aspirin.” The gap from a natural-language command to a physical chemical at a remote site is real, and it is a clean demonstration of the trends in one stack. It is not science. Synthesizing aspirin is a known protocol. When they tried “figure out a drug to cure cancer and synthesize that,” the system bogged down in planning and regurgitated what everyone was already discussing online. Breakthrough insight, he says, remains a human advantage. The paper showed that established protocols can be executed — and that the limit is still there.

Nvidia’s Minecraft agent made waves for a different reason. They dropped a language model into the game, let it generate code toward goals, and — the innovation — gave it a database of its own successes so it did not have to stumble through the same skills again. Combat zombie, craft stone sword, craft shield: small skills compose into bigger ones because the agent can look up what already worked. No fine-tuning, no injected facts, just discovery plus a save file: a “lifelong learner.” Div at MultiOn looked at that paper and applied it to the web. Booking a flight on Delta the first time, you stub your toe; if the agent can query a skill database of a prior success, it becomes faster and more reliable. You can even vet those skills, or ask GPT-4 whether a booking trace looks suspicious, as websites start a cat-and-mouse game of misleading visiting models. Confirmed, ideally safe protocols are how Labenz thinks agents start to work over the rest of the year.

Multi-level memory is the next layer. Human memory is fallible, but we have an ongoing sense of who we are that extends past any context window. Web search, embeddings, and saved successes still miss a synthesis layer. One study stored raw observations, then processed them into higher-level “reflection” memories — a simulated town of agents forming self-conceptions from what they had done. Labenz will not lean on a strong human analogy, but the engineering idea is clear. For the Delta example, “the client sat in 8D last time” is not a preference; ten window seats and one aisle probably means the aisle was unavailable. Smooth the outliers, keep the signal, and you get a more coherent agent.

Labenz’s slightly silly name for the next trend is “the great embedding.” Text models and vision models do not work together by default. People are learning to bridge one representation — one embedding space — to another. DeepMind’s Flamingo froze a language model and a vision encoder, trained only a small translation component with gradient descent, and that was enough for the system to handle text and images together. Salesforce Research’s Junnan Li and Dongxu Li did the same pattern in BLIP-2: numbers in image space, adapted into numbers in text space, processed as if they had been text. The episode on BLIP-2 is his pointer.

He loves it and feels weird about it. We already barely understand what any one model is doing; now models talk to models in a high-dimensional numeric space humans cannot inspect. It compounds the black-box problem. The reason people do it anyway is cost. BLIP training ran on one computer in a few days, versus one to a hundred million dollars to train a language model from scratch. Frozen backbones plus a small bridge buy multimodality on a tiny compute budget. InstructBLIP, GPT-4, and Microsoft’s LLaVA show how far image-text understanding has come: they can explain what is unusual about a man ironing on an ironing board off the back of a taxi. Meta is pushing further, combining text, image, video, audio, heat maps, depth, and IMU-style sensor streams — closing in on the five senses, plus channels humans do not have.

Outputs are going multimodal too. Lili Yu of Meta AI, on the show two weeks earlier for MEGABYTE, had just published a model that does high-quality text-and-image understanding and generation in one network: turn a painting into an alien, a line drawing into a photo, a pose estimate into an image, and still hold a dialogue. The most shocking multimodal result, never released to the public, remains DeepMind’s Gato from a little over a year earlier: one network, same weights, playing Atari, captioning images, chatting, stacking blocks with a real robot arm, and deciding what to output purely from context.

Efficiency is the last trend, and it matters because the bulk of spend is going to be running models, not training them. GPT-4 may have cost $100 million to train; OpenAI will probably spend more than that serving it. At a tenth of a cent for a simple call and a dollar for a long one, they would need on the order of 10 billion API calls just to recoup, and they will do far more. Conventional wisdom in the LLM game, Labenz says, is that inference compute outruns training compute.

Quantization is brutally simple: truncate the weights. Floating-point numbers with lots of decimals get cut down, and most of the time you get away with it — slightly worse, not much. From 32-bit to 8-bit you save 75 percent of the memory, which is why laptop demos of big models almost always involve quantization. Distillation is teacher-student: run queries through a large model, train a small one to mimic it. Taking a thousand GPT-4 outputs and fine-tuning open source is a crude version of that. If you own both models you can train on the full next-token distribution — 50,000-plus probabilities per step on the GPTs — a much richer signal than the single token the API returns. You do it to get a cheaper, faster, possibly narrower model. Grokking dynamics here are poorly understood: maybe the large model grokked something the small one can only imitate.

Mixture of experts is sparsity by routing. Instead of using every weight, a router picks which expert feed-forward blocks to run. Rumor has it GPT-4 is a mixture-of-experts model: over a trillion parameters if you count every expert, far fewer active on any one forward pass. Training is still heavy. Runtime is much cheaper, and runtime is where the bill is. Labenz ties it back to a Max Tegmark paper from part one on encouraging sparsity. Then he signs off: those are the implementation trends. The scouting report’s second installment does not pick a winner among prompts, retrieval, tools, and agents. It hands over a stack, and a reminder that every layer is getting cheaper, faster, and harder to inspect at the same time.