They Used Humans as Stand-In AIs. Now 250,000 People Run Elicit.

Nathan Labenz records from a hospital maternity ward, where he and his wife Amy have just welcomed their third child. The guests he has wanted since launching *The Cognitive Revolution* are Andreas Stuhlmüller and Jungwon Byun, co-founders of Ought, a product-driven research lab that builds mechanisms for delegating open-ended thinking to machine-learning systems. Their flagship product, Elicit, is an AI research assistant aimed first at literature review: you ask a research question and get back a table of papers with extracted details and summaries. Labenz has used it on medical questions in his own life and on the petty-but-real problem of retrieving a paper he remembers only by its conceptual punch line.

In early 2023 he had tried most of the comparable tools on the market, including Bing, and come away empty more often than not. Elicit, he says, is one of the few that actually adds value to a research workflow, which is no small claim given how hard those tasks are. The longer-term reason he wanted the pair on the show is methodological. Ought has been pushing a process-focused approach—decomposing tasks and training high-quality reasoning steps rather than scoring only final answers—for years before the rest of the industry started talking about process supervision.

That bet is baked into a mission statement they wrote about six years earlier and have not had to rewrite. Ought exists to automate and scale open-ended reasoning so that future improvements in machine learning help as much with thinking and reflection as they do with tasks that have clear short-term outcomes. Labenz wants to know why, if the models are so fluent, they still help least on the work that requires the most thought—and what the company has been doing about it since before GPT-2 existed.

Labenz quotes a line from Ought's site that still feels under-appreciated: it is especially unclear that machine learning will help in matters that require substantial thought. Stuhlmüller says they wrote it six years ago and it is still roughly true. As he frames it, machine learning can imitate a behavior—write text the way a human writes—or it can optimize a reward, as when a Go program is rewarded for winning. Essentially every case where AI has substantially exceeded human capability, he argues, has been a reinforcement-learning case, because only a reward signal produces unforeseen creative moves.

Reinforcement learning, though, wants a well-defined task and a clear objective. Winning at Go qualifies. Helping someone think more clearly, arrive at truer beliefs, or make a better decision does not. The tempting substitute is a proxy: does the output look good—persuasive, impressive, helpful-seeming? That is not the same as being robustly helpful or arriving at a really good decision. Right now the models are still mostly in an imitation regime, so they can write, brainstorm, and code in ways that are genuinely useful. If the goal is to exceed the best humans at thinking the way AlphaGo exceeded the best humans at Go, he says, there is still work to do.

Labenz asks whether the binding constraints are sparse rewards and unreliable human evaluators. Those are real, Stuhlmüller replies, but the deeper problem is that the task itself is undefined. You can ask a model what decision to make, hear a list of considerations, and judge whether it sounds good. That loop gives you some resolution as a reader. It does not, on its own, produce a system that is better at the decision than you are. The most natural paradigm, he says, simply does not solve the problem of exceeding human capability at reasoning in a way that is actually helpful.

When Ought started, GPT-2 did not exist, and AGI timelines of 2100 still counted as the aggressive view. To study how future systems might help with thinking, the team used humans as the closest available stand-in for AI. Participants received a tiny context window and a narrow task, details Byun says have generalized uncomfortably well to actual language models. The best-known setup was relay programming: a contest-style puzzle, each person getting about a minute to make progress and pass notes to the next, without loading the whole problem into one head.

The question underneath was fundamental: how composable is thinking? Do you need a global mental model, or can people who never see the big picture still move a problem forward? There was a supervision reason to insist on blindness. If a future system can read an entire book and you cannot, you cannot evaluate its work. Break the job into pieces a human can spot-check, and oversight stays possible. Stuhlmüller would rather have that transparency even if each step is a bit less efficient than an end-to-end leap.

The experiments produced two hard lessons that now look like a preview of language-model pipelines. First, errors: if each subtask has even a 10% chance of going wrong, twenty steps make a failure almost certain, so you have to eliminate errors faster than you introduce them—and the correction mechanism itself has to be built from the same one-minute pieces. Second, state. In a movie-review task, people who could see only fragments of a summary got derailed by one argument and lost the overall point. Context was too small, and nobody had a smart way to pass state along. Byun says the same gap remains: every language-model call still starts from scratch.

Ought's product philosophy is interpretability by construction: make models follow processes humans can understand, rather than optimizing an opaque pile of floats for a final metric. Stuhlmüller contrasts ordinary machine learning—set every matrix number so the last score goes up, and ignore what internal parts mean—with ordinary programming, where functions need independent meanings you can evaluate out of context. Spaghetti code is what you get otherwise. The obvious objection is the bitter lesson: end-to-end scale wins, and hand-decomposing tasks will not. His reply is that language models can help with the decomposition itself, so the process need not stay manual. Even if end-to-end is more powerful in principle, they think transparency is worth pushing hard for.

Byun draws a sharper line against after-the-fact interpretability. Ask a model why it answered as it did and it will rationalize, inventing reasons that need not match the computation. Interpretability by construction trains the system on human-legible processes up front, so the steps you inspect are the steps it actually ran. In Elicit that shows up as a trustworthiness checklist—sample size, funding conflicts, which variables were controlled—run automatically over every paper. The tempting move is to collapse those fields into a single score from citation counts and journal prestige. They refuse, because scores are easy to hack and a checklist keeps the detail. If they later add a score, it will sit on top of the checklist, and a user will always be able to expand it back.

Labenz's own use of the product is less grand and more revealing. He would not make Elicit his doctor, the same disclaimer he offers for GPT-4, but he will take a second opinion, and he has used it to surface papers he remembered only as a ten- or twelve-word conceptual fragment. That retrieval trick, he says, already puts Elicit ahead of almost every competitor he has tried. The checklist and the table are not a demo of clever generation. They are a bet that researchers will update on evidence only if they can see why a paper deserves the weight.

He wonders whether 32,000-token windows and chain-of-thought are converging on the same idea: spell out the steps in one prompt and grade the trace. Stuhlmüller grants partial convergence, then flags a result in which replacing the written steps with literal dots still improves the answer, because latent computation happens in those tokens. You want the written reasoning to be causally responsible for the final answer. That may not matter for today's low-stakes uses; it will matter if the model is deciding whether to invest or whom to hire. Byun's contrast is a one-pager on long COVID versus a one-pager that lists the databases searched, papers read, ranking dimensions, trustworthiness checks, and comparison data. The first you can only distrust. The second you can audit.

Where does the extra causality of a compositional pipeline come from? Not from a different model for every step, Stuhlmüller says; the same model can run every subtask. It comes from removing information. A worker asked only whether a webpage answers a question is not told that a medication decision hangs on the result, so it cannot tilt relevance to please the larger story. Stuff everything into one prompt and the influences tangle; you can no longer say which fact drove which judgment. Labenz, who had been thinking mostly about the upsides of long, coherent context, takes the point: once prior papers in a linear stream can leak into the next extraction, you no longer know what the answer depended on.

Task size follows a simple rule, and it is close to Eric Drexler's Comprehensive AI Services picture with a smaller grain. Drexler, as Stuhlmüller reads him, is comfortable with fairly large services whose internals can still be optimized end-to-end; Ought wants even smaller, independently inspectable components, a direction Drexler's later Open Agency Model also moves toward. Make tasks as large as possible while humans—optionally with AI assistance—can still robustly evaluate them, and do not cross that line. Evaluation is noisier than people think. Labenz cites the GPT-4 technical report: the blind win rate over GPT-3.5 is only about 70–30, not 99–1.

Composition is what makes Elicit's checks cheap. The system first finds relevant passages, then answers from those passages, and links the answer back to the source, so a reviewer checks a claim plus a few sentences instead of rereading a paper. Retrieval is a stack of approximations to an ideal Stuhlmüller describes as running the most expensive model, paragraph by paragraph, over every paper ever written. In practice they cheap-filter on the order of a thousand candidates with embeddings, which he calls cached language-model computation, rerank with a cross-encoder, then spend the expensive model on a handful of documents. There is no clean optimum, only a budget.

That architecture is also their answer to the copilot-versus-assistant debate. A copilot has to be fast, and speed caps how much thinking it can do; Stuhlmüller would be surprised if every task people care about fit in that box. Byun, speaking as a user, has almost no patience for "I am a helpful assistant" throat-clearing or for iterating on a mediocre draft—she wants the best answer handed over. Labenz adds that even a fully automated endgame benefits from millions of users marking where the model breaks, akin to Google robotics work he heard from Kirthana, where robots call for help and a remote operator's joystick traces become training data. Ought's short-term bet is still to study what the human actually does, break it down, and train the model on those steps so helpfulness is designed in rather than hoped for.

Byun describes a three-phase roadmap and is explicit that literature review is a starting block, not a forever product. Phase one, discover what is known, is about freeing information trapped in PDFs and books and turning it into tables you can query; a new List of Entities workflow then extracts cross-paper concepts—datasets, benchmarks, drug effects, interventions, open research directions—each linked to sources. Phase two, discover what is unknown, would find gaps and cruxes and help researchers design experiments, with causal modeling as the technical bet, going deep rather than wide. Phase three, decide in the face of the unknowable, switches the user from research producer to decision-maker, using models to surface values and unresolvable assumptions on questions that have no right answer.

Who they serve is a function of the task, not the logo on the building. About 60% of users are academic researchers; the rest include think tanks, government, finance, consulting, clinicians, and small-business owners, and some biomedical systematic-review teams have folded Elicit into their official process. A Coca-Cola researcher studying how to cut sugar counts; building an internal search engine over corporate documents does not. Workflows are meant to open over time—from presets, to editable parameters, to editable steps, to user-built pipelines—and some researchers already stitch a hidden Tasks page into homemade loops. Labenz's one-line summary is "Zapier for reasoning."

Stuhlmüller is not trying to grow an endless catalog of widgets. The reusable pieces are few—search, answer given results, summarize, rank, cluster, filter—and the power is in composing them, which he likens to functional programming with natural language, the way a handful of operators such as `map`, `fold`, and `filter` can build a whole program. Domain-specific wrinkles, like whether a study was double-blind, can come from the model, from users, or from checklists that differ in content but not in shape. Across empirical fields the questions that generalize are blunt: what did they do, what did they find, and what did they not find. If he had to pick one magic question, it would be the last, because limitations bound a result better than the authors' own summary of what they did.

For production they landed on Flan-T5 XXL, an 11-billion-parameter model, after trying OpenAI, Cohere, Galactica, GPT-J, and others; with roughly 250,000 users, self-hosting beat the API bill. Fine-tunes cover two tasks—question-specific summaries and paper Q&A—using a Constitutional AI-style loop in which a large model judges pairs on a rubric of concise, truthful, and useful, then a reward model trains Flan-T5, work their engineer Charlie has led. GPT-4 is a significant step up and still hallucinates: ask who Andreas Stuhlmüller is and it fabricates universities. He has taken to telling labs to stop making bigger models and make them hallucinate less; Byun adds that the models cannot cluster a space of ideas well, cannot admit they are stuck, and—after RL—are too overconfident to use for ranking, with log probabilities no longer even exposed.

At least 10,000 people use Elicit more than once a week, and Ought, still a nonprofit, has hit the ceiling of what philanthropic funding can support as the product has grown. The founders want it to scale into high-stakes decisions and are actively considering a commercial vehicle so the organization can be self-sustaining. Labenz hears a Wolfram-like aesthetic of explicit, trustworthy processes; Stuhlmüller grants the shared taste and distinguishes the strategy. They would rather own the infrastructure for composing a few operators than ship a gajillion widgets.

On AI safety, Byun's honest range of outcomes is enormous; she will not pretend to a forecast. What she does want is for safety not to happen only inside the largest labs. Consumers should start demanding process supervision and transparency from the applications they use, putting pressure on the supply side. At her previous company, Upstart, machine learning sat inside heavily regulated consumer credit: the good news was that marketing could not make claims without evidence; the bad news was that regulation always lagged innovation. She wants the regulatory clock to keep pace, and she wants the tech industry to tell the stories of when regulation actually protected people.

Stuhlmüller is colder about the staffing. Relatively few people work on alignment at all, and almost nobody works on using AI to improve collective wisdom and decision-making—the bet Ought is making. Frontier labs now list process supervision among their core alignment bets, which he takes as a good sign, but the dollars still look nothing like the spend on scaling models. His wish list is prosaic: debuggers, visualizers, and compilers for compositional language-model programs. On optimistic days he thinks society may eventually limit very large training runs, and that AI itself could help us see whether and where that limit belongs. Labenz likes a high-end cap that would not inconvenience PhD students or startups, but would give the public a heads-up before someone goes a hundred times past GPT-4.

Byun finds a sliver of historical luck: unlike James Watt or Gutenberg, today's lab leaders at least know they are unleashing something large, and they have treated safety as a first-day concern. Her hoped-for decade is that tools like Elicit open new research methods, and that AI writes decision-relevant, rigorous one-page memos for people in high-stakes government roles—something she wished for after friends described COVID decisions made on chaotic email threads. Stuhlmüller's optimistic picture is that most public fights are disagreements about evidence, not incompatible values: whether AI is 90% doom or 10%, whether universal basic income is wise. If AI helps everyone see the totality of the evidence, factional combat can give way to coordination. In a short lightning round, he recommends his open-source Emacs plugin `gpt.el`; she has been writing Python with ChatGPT and Deepnote despite little programming background; both would take a Neuralink-style thought-to-text implant without hesitation.