The GPT-4 Red-Teamer Who Got Unstuck by a Mediocre Outline

On December 20, 2023, The Cognitive Revolution ran a crossover that was not originally its own. Dan Shipper, founder and CEO of Every, had started a show called How Do You Use ChatGPT, inviting heavy users to walk through their actual chats. Early guests included Gumroad founder Sahil Lavingia, writer Nat Eliason, and Notion researcher Linus Lee. This episode, which Cognitive Revolution rebroadcast, puts host Nathan Labenz in the guest chair.

Labenz is unusually well placed to talk about models. He founded Waymark, an AI video company, co-hosts Cognitive Revolution with Erik Torenberg, and was one of OpenAI's external red-teamers before GPT-4 launched — one of the people, in Shipper's phrasing, trying to figure out how to make the model do bad things. Coming into the conversation, though, he is frank. He had used ChatGPT mostly for unfamiliar tasks, where he needed orientation and a first step. For his own writing, he had never found it super helpful.

That is the setup: a person whose job was to find GPT-4's failure modes sits across from someone whose job is to teach people how to use it. Labenz calls the exchange both extremely practical and a real exchange of ideas, and says it inspired him to try more of this for himself. Near the end, when Shipper asks what he is most excited about next, Labenz mentions the new Mamba architecture and state space models. He also reports that he used Shipper's recommendations to develop the strategy, devices, and overall structure for a coming episode, in a way he found legitimately helpful.

Shipper introduces Labenz with three hats: founder of Waymark, host of Cognitive Revolution, and GPT-4 red-teamer. The last one is the unusual one. Red-teaming, borrowed from security, means playing the adversary before a product ships. For GPT-4 it meant trying to induce dangerous outputs so those uses could be found before hundreds of millions of users found them. Shipper notes that Labenz had recently posted a Twitter thread about that experience.

Every bills itself as the best business writing on the internet. Besides the newsletter, the company builds small AI tools, and Shipper writes every day; his methods come from that daily practice. The new show's format is the method: sit with one person and ask how they actually use ChatGPT, rather than debating trends. Labenz's Cognitive Revolution, by contrast, tries each week to assemble a picture of how AI will transform work, life, and society. The two formats meet on a claim Shipper has kept seeing in Labenz's work: that one of AI's values is helping people offload cognitive labor.

Shipper puts the claim on the table. In the industrial revolution, machines took physical labor. AI, Labenz has argued, will augment some cognitive labor and offload some of it. Is that true, and is it good? Labenz does not start with good or bad. He splits "AI doing work" into modes, and says this episode will mostly live in the first: copilot mode. You go through life and work as usual. In a specific situation — especially once you are used to the tool — you realize the model can help, switch over for a second or a minute, then proceed. You remain the agent pursuing the goal.

Delegation mode is different. The goal, Labenz says, is to get output consistent enough that you do not review every instance. Then you can shift work you no longer have to do: save time and money on things that used to be a pain, or scale things that currently do not scale. Almost everyone, he thinks, has work that would be valuable if done but that they simply do not have time for. Copilot is intelligence augmentation. Delegation is labor substitution. Both can be transformative, and they are not the same picture.

What is still missing is the middle: ad hoc delegation. You are going along and want to hand the model a bigger subtask on the fly. Agents, he says, still cannot do much of a significant task, so people are shoehorned into real-time help or a carefully validated workflow. He expects that gap to close over the next year as agents begin to work. Asked whether offloading cognitive work is good, he holds both sides at once. A post-scarcity world is not crazy, and — quoting Sam Altman — he would not rule out "lights out for all of us." The condition is that humans stay in control of the overall dynamic.

He is thrilled by his own productivity and by cheap access to expertise for people with far less means, including medical advice they cannot currently take a day off to get. He also keeps a healthy fear. Shipper likes the refusal to collapse into camps. He asked whether offloading cognition is good because the fear scenario dominates for many people, while the hopeful camp often wears rose-colored glasses. Finding real use cases, he argues, is part of making AI a force for creativity rather than a story that only replaces people. AI has also revealed how much drudgery sits inside highly valued knowledge work. Once the repetitive pieces of writing and running a business become visible, he hands them to the model and stops thinking about them.

For Labenz, coding is the drudgery that comes to mind first. He is not a full-time programmer; he dips in and out, and he does not remember syntax well. After a gap he can confuse JavaScript with Python. Having the model type even relatively simple code is often a multiple-x speedup, and often a quality improvement over what he would have written alone. It also lowers the cost of getting into "code mode" at all — the focusing step he compares to birdwatchers getting their eyes on. Reading code, he says, is a lot easier than writing it. The output is "super-Nathan quality," if not superhuman quality.

Shipper generalizes: much of what current text models do is summarization, and much of creative work is summarization in disguise. Programming is summarizing what you found on Google into the exact form your use case needs. Writing is summarizing books, conversations, and borrowed ideas, then deciding the order. Once you see that class of work, you stop doing it by hand. Labenz has used AI for classic summarization on the podcast — timestamp outlines of discussion topics — but has not done as much as he should to repurpose episodes across platforms. When something goes out in his own name, he basically does not use ChatGPT at all. Neutral voice of the show is fine. Synthesis in Nathan's voice is not there yet, and that is the plan of attack he wants to leave with.

Shipper answers the writing question by rejecting delegation. The failure mode is "just go do this whole thing." As a copilot on microtasks, it works. One example: he was writing about Sam Bankman-Fried, FTX's collapse, and whether utilitarianism and effective altruism contributed. He had studied philosophy and read a lot of Peter Singer, but he had not written a three-or-four-sentence summary of utilitarianism in a while. Ordinarily that would have meant three hours of rereading. ChatGPT gave him a summary in the context he needed. He did not use it wholesale; he tweaked it into his voice. That is the pattern: the model does the microtask, and none of what he publishes is wholesale ChatGPT.

He often starts even earlier, recording himself on a walk, spewing associations, then having the model transcribe, summarize, and pull out article ideas. When he has an idea, he starts with a really messy document full of quotes, sentences, and scraps. He reaches a point of "I don't even know where to start with this, this is crazy," pastes the entire document, and asks it to put the mess into an outline. The outlines are really basic. What the model is good at, he says, is pointing out the obvious solution you missed because you were too close to the problem. Of course the article sets up a problem and then talks about the solution. That is a format he has written a thousand times. When you are being precious about a "special" piece, the obvious structure is the one you refuse to see.

The other gift is helping you put into words what you are going for, then generating options until one of them is the thing. He will ask for fifty metaphors; forty-nine will be trash; one will be amazing or will push him toward the one he actually writes. Labenz has had luck talking a letter through on a walk, giving detailed verbal revisions because the app's transcription understands him well, then finishing at the desk. He also uses ChatGPT at the other end: "critique this" on an email or a podcast intro. Every has skilled human editors, and Shipper still asks the model what it thinks of an intro at 2 a.m. the night before a deadline. Labenz's caution is the opposite failure: autocomplete that derails you when you are on the verge of the thought you actually meant. If the core ideas are not down yet, he has struggled to get anything good. The tip he is taking is to use the model on ordering and iterating once those core ideas exist.

They open one of Labenz's historical chats. He is AI advisor at Athena, founded by a friend named Jonathan, one of the founders of Thumbtack. Athena hires executive assistants in the Philippines at premium wages, aiming to give ambitious founders and executives the kind of transformative assistant Jonathan once had. The live question is what AI means for that work. Labenz has trained assistants on AI and built prototype demos. Athena Chat is an in-house ChatGPT, built on an open-source project, meant to hold a long-lived client profile the assistant can query and update — preferences, history, background documents — as a hub that evolves over time.

Despite education, assistants still need coaching on how to prompt. So he wanted a prompt-coach module: intercept what the human assistant typed, run it through a meta-prompt, and check best practices. Are you telling the model what role to play and what job to do? Are you specifying a response format? Are you letting it think step by step before a final answer? The most common way people shoot themselves in the foot, he says, is prompting so the model cannot scratch through the problem. Anything that forces an immediate answer with no room to reason is bad. He sees it even in academic publications.

The hangover is from an earlier era. The first public instruction model was OpenAI's text-davinci-002 in January 2022. Before that, you did not say "write me a haiku"; you wrote "a haiku by Author Name:" and hoped it would continue the pattern. Academic benchmarks were built as question-answer, question-answer, then a blank for the answer, scored on few-shot prompts, all before chain-of-thought was widely understood. Bring that scaffold to GPT-4 and it will respect the implicit structure and jump straight to a multiple-choice letter or a number. Quality drops versus just giving the single question and letting it spell out reasoning, which it now does by default. He had recently seen Bard give an answer before the explanation; once a wrong answer is on the page, the explanation is an excuse for it. His mnemonic for the assistants is AAA: analysis before answer, always.

In copilot mode he does not use elaborate custom instructions. He started naive: I am working on this React app and I am a bit lost, can you explain the structure? The open-source project used React, Redux, slice toolkits, and sagas — five things he did not know. Without the tutorial he would have been off searching "what is Saga JS?" The model gave him a rundown, then he noticed a pattern: if you can give the model something in a format it natively showed you, it will work with that structure well. He asked it to write the command to print the file tree, including installing a package he did not have. That is the classic developer stall: three nested problems deep, conceptually clear, time dying on installs. Then he pasted the file names — no code yet — and the semantic names were enough for the model to guess what the app was.

Athena Chat is a ChatGPT-like environment with client profiles, history, models, and function calling in the background. He wanted a prompt-coach module in the lower-right corner: run prompts through the meta-prompt, show color-coded warnings. He tried copying a sidebar module himself; it showed up in the wrong place. He explained the failure, got code and instructions, then — this was right as vision landed in ChatGPT — pasted screenshots of where it was versus where he wanted it. Over two to three hours, never having written a line of React, the model wrote the code, CSS, and style-pack syntax. He would guess the unassisted version would have been two to three days, an order of magnitude, mostly frustration at work a million people have already done.

Shipper names what he finds striking in the transcript: a dance. At the start Labenz does not know React or where to put the module; the model does not know who he is, what he is trying to accomplish, or the specifics of the project. He did not say "just go do this." He asked how a React project works. Mutual understanding increased, and they accomplished the thing together. Later they are only dialing in padding and centering. Labenz is polite to his AIs, and he can already see a future where this episodic relationship — a new chat knows nothing; you can continue only up to a limit — gives way to history that follows you across tasks. Shipper agrees: custom instructions are a step, and memory will make the same model far more useful.

Labenz asks how much to put in custom instructions. Would he write that Nathan is a React novice who does not know how to install anything? Shipper has extensive ones and a lot of opinions. The first field is what ChatGPT should know about you. There is enough about him and Every on the internet that the model already knows his name; not having to re-explain who he is is useful. When he said he wanted to build a course, custom instructions made the model treat him as a writer and entrepreneur building one. With them off, it asked what course he wanted to take.

He puts in serious relationships — his sister, her husband, her son, his girlfriend — and people at Every, so he can say "Kate" without a gloss. More interesting are the things he is trying to work on. He writes that he has a fear of rejecting people that makes him too agreeable, and that he is a little too opportunistic and would like to be more strategic. ChatGPT becomes the thing that, in the moment, reminds him of insights he already had about himself, and that pulls him back to his goals. Labenz's own use is still mostly episodic and cold-start — an unfamiliar framework, a patent application — and he treats bringing the model closer to his core work as a New Year's resolution. At Waymark he already works intimately with language models inside the product, which is a different mode from Shipper's interactive dance.

The next chat is diagrams for a provisional patent on Waymark. Small-business users paste a website URL; the product scrapes content, builds a synthetic profile, and takes a short purpose prompt such as a video for Saturday's sale. A language model writes a script; computer vision picks images from the library. Pre-AI, Waymark was an easy template library. Now the AI makes the content. Labenz compares users to Mr. Burns at the art museum: "I'm no art critic, but I know what I hate." They wait thirty seconds, watch a video featuring their business, and either proceed or say not that, give me another. Like most software companies they will never prosecute the patents; they just do not want to be hassled. He word-vomited the architecture and asked the model for syntax that could diagram it.

They compared Mermaid, Graphviz, and others, then generated markup he pasted into a renderer. After several refinement rounds the model got confused — too much syntax in working memory — so he started a fresh chat with only the best diagram and asked for localized edits. Color coding appeared: green for what the user does, then the scrape, then a fork for images. The diagram clarified which steps can run in parallel and which depend on which; even the technology team, he says, may not have been able to draw it. Aesthetic scores have to finish before you can select best images. Maintainable text syntax is also easier for language models to understand than a rendered picture. He mentions DiagramGPT as a natural-language-to-Mermaid tool, and they discuss what should live inside ChatGPT versus a dedicated editor. GPTs cannot yet host custom editor experiences, so for now the loop is model writes syntax, another app renders.

The copilot also left work. After ten years of swearing he would not replace his car until the replacement was self-driving, three kids later he bought a used minivan. Dealer sites are a jungle of trim levels — SE, SXT, AVP — that encode sunroofs, rear screens, and blind-spot detection. He used Perplexity, which he treats as a complement to ChatGPT and a more direct rival to Google: accurate answers to concrete questions, not a brainstorming partner. He asked what cheap, not-super-old minivans had, and when USB charging arrived in cars; he is old enough to remember the cigarette-lighter adapter and did not want that era. He has had Perplexity CEO Aravind Srinivas on Cognitive Revolution twice. He now uses "perplexity" as a standard: not always rock-solid ground truth, usually right when he fact-checked, good enough to act on. He would not make life-and-death decisions without more checking. He sent his wife that the car should have a backup camera, per Perplexity. ChatGPT's browse, in his tests, was slower and less complete on the first pass. Specialization, he argues, can beat a general model in the domain it actually attempts.

Asked what he is excited about, Labenz takes the field's leaders at their word. Listen to Sam Altman and to Anthropic's Dario Amodei and you are looking at systems that could be superhuman in substantial ways within a couple of years. People talk past each other, he says, because you can be superhuman in consequential ways without being omnipotent or infallible. He expects AI to land in that wide space. A recent Google DeepMind result on differential diagnosis was striking. About a year earlier, language models first hit passing level on medical licensing tests — discountable as a structured exam. Then expert-level on the test. Then multimodality that can read X-rays and tissue slides, with radiologists still ahead but by a narrow margin he recalls as something like 60–40.

The new study takes case reports from medical journals: confusing, instructive cases, not routine colds. AI's differential diagnosis beat humans with access to AI, and human-alone was last, by a significant margin. The paper's modest conclusion was that doctors need better interfaces. Labenz thinks that buries the lead. Another lesson is that the AI was getting it right about twice as often as the human clinician, in the ballpark of 60% versus 30%. We rarely measure human performance and treat a licensed doctor as the standard. The best doctor in the world might beat the 60%; most people cannot access that person. He sees a world of radical, cheap access to expertise — a blow for equality of opportunity that will also scramble wages and become political. He hopes we get to enjoy the fruits.

Transformers are not the end of history, and neither is ChatGPT's no-memory chat. In the last week or two, state space models have been in the headlines as a possible successor, alternative, or replacement, with better long-term memory, scaling, speed, and throughput. He strongly suspects a mixture: brains are not one repeated unit, and AIs will be composites of architectures with different information-processing strengths. The jump from GPT-2 to GPT-4 in four years was shocking; he thinks the next few years will bring at least as much change. That is the Mamba episode he cannot stop thinking about — and, as he said at the top, the one whose strategy and structure he built with Shipper's paste-the-mess method. Prompting technique has a short half-life. Architecture is what decides what the next round of apps can look like. He voted with a real show.