The Right Size of an AI Task Is Whatever You Can Check

Nathan Labenz, a small investor in the company and a heavy user for structured literature reviews, welcomes back Andreas Stuhlmüller and Jungwon Byun, co-founders of Elicit. In the year since their last appearance the product has launched in rapid succession, they have raised a $9 million seed, and subscriptions reached $1 million in annual recurring revenue in four months. The latest release is notebooks — not a garnish, Jungwon says, but a reimagining they have tried to make work four times. The vision underneath is unchanged: apply AI to complex reasoning at scale, from climate to chronic fatigue to institutional decisions.

She names two failures in research as it is actually done. Publications in most fields grow exponentially; a scientist can barely stay on top of this year’s output, let alone the historical pile, and feels she is drowning in papers she will never read with the rigor she wants. The second failure is process. People start on Google Scholar, open a tab, follow a citation, and eight hours later cannot remember the original question. At multinational industrial labs with offices on several continents, finding whether the company itself already did the work often means talking to someone who happened to know someone who was around a decade ago.

Elicit’s three words for the alternative are systematic, transparent, and unbounded. Systematic means putting the researcher in the seat of metacognition: write down what a good paper looks like, what a relevant result is, what success is for this project, then apply those criteria across every relevant paper rather than across whatever one person can read tonight. Transparent means a default log of starting papers, filters, screens, follow-up queries, and how the final set was reached — shareable and auditable. Unbounded is the attack on the PDF. A published file is static and immediately outdated; nobody can reproduce it, extend it, or tweak one assumption and rerun. Notebooks are meant to be a live document you can keep adding steps to.

Labenz walks through his own loop. Ask a research question, get a sandbox of papers, then add columns — population size, whether the null was rejected, or a custom one. Studying curriculum learning, he asked for the largest model used in each paper, kept only the big ones on the theory that they behave differently, and pulled that subset into a deeper table. Making a private filter into an explicit column is the step most people skip. Jungwon adds that a later researcher should be able to pick up the notebook, decide they also care about smaller models or a different architecture, and continue without finding the corpus from scratch. Column-to-column compounding is already in a prototype and sitting in the weekly ship queue.

Labenz wants the weeds on reliability. Task decomposition helps performance in the literature, but builders live on a Pareto frontier: one mega-prompt is a lottery, and slicing too fine explodes token count, shreds context, and slows the product. Stuhlmüller has been decomposing since GPT-2, when even “does this claim imply this other claim” was hard. Then the question was the largest chunk a model could reliably do. Now, he says, the question is what is easy to supervise — how big or small the pieces must be so a human, or another model, can check whether the work was correct.

Long context does not retire that question. In a million-token window, a good retrieved answer is easy to verify; a miss is not, unless you reread the million tokens. The same logic applies if models supervise models: you still want the supervision problem to be easy. Their example is a paper on what they call factored verification of Elicit summaries. They did not break down generating the summary. They broke down verifying it: split the summary into independent claims, then ask whether each claim is clearly substantiated by the context. That catch list included subtle failures — a slight exaggeration of a paper’s findings, or a sentence that implies two independent results are related because it reads more smoothly.

He analogizes to software. You write modular code so your future self and other people can understand and check it. Humans already use clear interfaces for that reason. The most important decomposition principle, he says, is the same: design interfaces that make the work easy to inspect. Supervisability comes first; token cost is a secondary variable. They are interested in runtime checker models, because a core Elicit criterion is helping users turn extra budget into more accurate answers, but the verification work so far has mostly produced training data for a distilled, lower-hallucination model rather than a second pass in the live session.

At Waymark, Labenz’s video company, fine-tuned GPT-3.5 started iterating faster than the team’s intuition. Structured evals became mandatory. Hard prohibitions — his “Ten Commandments” — can go to a model; an overall quality score still needs a human watching the video. Jungwon says Elicit’s experience rhymes: defining the task is most of the battle. Their core users are serious researchers writing systematic reviews. When the team recently worked on search, the ideal was simple. Look at a published review, take the papers it actually cited, and ask whether Elicit would have found exactly those papers rather than nearby ones that look relevant on a skim. The metric is ordinary search machinery such as NDCG. The interesting work is understanding the user’s problem.

Summarization can lean on models once “good” is encoded as rules: no even-subtle hallucinations, no exaggeration, no implied link between unrelated findings. Extraction uses automated evals against human gold standards that domain experts have sat with. Jungwon’s principle is to prefer gold that already existed because someone was trying to get the best answer, not because someone was building an AI test set. Users have even submitted “correct” labels in which Elicit then found errors. In side-by-side tests on predefined extraction columns, she says, Elicit has consistently beaten trained research staff. On that slice, she is willing to call the product superhuman today.

Labenz has learned the same lesson all year: many domains never measured human performance before AI arrived, and people assume the human baseline is excellent when it often is not. On hallucinations they were strict. Stuhlmüller does not remember the exact figures on air, but the order of magnitude was from about 1.5 hallucinations per summary on the largest models down to about 0.5, counting slight exaggerations and slight inaccuracies. Fine-grained evals in the product also push users to evaluate finely. “Relevant papers for Nathan” is a mushy task. “What was the largest model in this paper?” is a task they can score.

The principle, Stuhlmüller says, is always let people use the best models available, which means not being tied to one compute provider. Combinations from different vendors are often stronger than any single model. At recording time the stack included fine-tuned Flan-T5, fine-tuned GPT-3.5, GPT-4, GPT-4V for tables, and work-in-progress Claude 3. Next time they talk, he expects the list to be different. Getting good at swapping models quickly is the capability that matters. They are not in the business of doing machine-learning work for its own sake; they are in the business of deploying systems that give the most accurate useful answers, and if the right prompts on off-the-shelf models will do, that is better because it ships faster.

A blog post claimed a new feature every week; the average landed at 1.4 weeks, with misses around bigger launches. Labenz wonders how that cadence survives a reliability bar at which a slight hallucination is a defect. Jungwon says they stagger. Accuracy-heavy features sit in research for a long time and then land across several weeks, interleaved with infrastructure and pure product work. Stuhlmüller adds a harder reason to ship fast: if you want to be at the frontier of accuracy, a new model comes out every week, and a slow product is simply not as accurate as it could be. Decomposition helps launches too. Shipping a new predefined column for “statistical technique used” is much easier than fine-tuning a model on all scientific papers and then evaluating it.

The capability threshold that moved most in the past year is tables. P-values, whether a hypothesis was refuted — those answers often live only in a paper’s tables. Before multimodal models the problem was basically impossible; even now it takes a lot of parsing engineering to make it fast. A second threshold is one-or-two-hop questions. Epoch, which tracks compute, parameters, and dataset sizes in AI, rarely finds a paper that says “we used this many FLOPs.” It finds “we trained this architecture on this GPU for two days,” and someone has to reason. As models get smarter, Elicit can take questions that sit a step or two off the literal text.

Elicit uses chain of thought for essentially every question. CSV export even includes extra columns with the reasoning, which Stuhlmüller recommends reading; it is often instructive. Transformers spend a fixed amount of compute per token. Some questions cannot be answered without extra steps no matter how much you fine-tune — arbitrary arithmetic is the clean example — and pause tokens or other architectural tweaks are still research. For Epoch-style questions he would add ingredient columns (architecture, hardware, duration) rather than only asking for the final FLOPs number, because the reasoning can still need a human.

Labenz finds the literature on chain of thought confusing: it helps on Big-Bench and theory-of-mind when the old prompt jumped straight to the answer, yet counterfactual studies sometimes show the final answer barely depends on the chain. Stuhlmüller’s reply is that nobody, including him, understands what is going on inside the models, and anyone who claims to know exactly how chain of thought helps is probably wrong. What he will say is tactical. People still write the obvious bad prompt: model, say yes or no, then justify. That is terrible, because the model must answer with no reasoning and then lock itself into a possibly wrong avenue. Labenz remembers first-release Bard shipping that way and being stunned that a top research lab lost something in translation.

Jungwon’s practical fix is a backout. Allow “I don’t know, and here is why,” or “it is not mentioned; here is where we looked and where we expected to see it.” In a lot of their settings the answer is simply not in the passage. Forcing an answer is what produces hallucinations; a graceful miss is still useful to the user. Stuhlmüller folds it back into interface design. One interface demands yes or no. Another returns data with two branches — contains the answer, or does not — and different helpful outputs in each. How you deploy the model, not only how you train it, decides whether it is useful. He would also like models that can say “wait, I meant no,” instead of post-hoc justification.

Data-set size, they have found, looks like Labenz’s experience at Waymark: a few hundred to a few thousand examples is generally enough for fine-tuning. Larger sets matter more for evaluation. They have used a Constitutional AI-style preference model to pick the better of two summaries, with a constitution that said the summary should be accurate and concise, again on a few hundred to a thousand human judgments. The stack is supervised instruction-tuning first, sometimes an RLAIF layer after. The ideal case is no fine-tune at all, so they can deploy faster as new foundation models appear.

Labenz relays a question from Epoch’s Tamay Besiroglu: what would a scaling law for scaffolding look like? Fine-tuning and surrounding structure seem like substitutes in Elicit’s stack, maybe complements in others. Stuhlmüller takes chain of thought as the most basic scaffold — people underestimated GPT-4 when they forbade it — and then names debate and amplification as scaffolds that did not work well at GPT-3 and work better now. The shape of that landscape is not well understood. Meanwhile other people are bolting planning frameworks onto models that could barely use tools a year ago. A sudden model jump could make a lot of half-working agents start working, even if Elicit’s own structured product will not look qualitatively different from one release to the next.

Elicit, Jungwon says, tries not to build only for the present. The moment the product will matter most is closer to AGI, when people need these systems for impactful decisions. Internally they talk about an “exascale engine.” Users today run models over tens to hundreds of papers. The destination is reasoning across tens or hundreds of thousands of papers, with models supervising models, without stuffing everything into one huge black box of opaque weights. Stuhlmüller puts the fork in societal language. How do we want to turn compute into more work? One answer is train larger and larger models and hope, maybe with a little interpretability. They are working toward another: more compute will lead to better, more correct answers, but through more transparent architectures, if the infrastructure can be built.

Take a query such as randomized controlled trials on creatine for cognition after 2020. The system parses the natural-language question into a search intent plus filters, then hands an API-shaped call to a hybrid semantic and lexical engine. About five hundred strongest candidates come back. A sequence of increasingly expensive rerankers follows — MonoT5, cross-encoders, heuristics on recency and citation count — and even at summary time the model can still drop a paper that is not relevant. The whole pipeline has to finish while the user stares at a browser tab.

That speed constraint, Stuhlmüller argues, is the wrong bottleneck for the users who matter. People spend nine months on a systematic review. They would wait an hour, and pay on the order of a hundred dollars, for a much better pass. Today they cannot, because Elicit is tied to the browser session; close the tab and the work dies. The infrastructure they are building decouples execution from the session: tell it to spend more compute, close the laptop, come back to a progress bar, as if you had outsourced the job to a human research assistant. Labenz has been banging this drum in the app ecosystem. Speed and cost are overrated; success on high-value cases is not.

Labenz puts the safety question bluntly. A language model that helps someone build a novel pandemic agent is a standard worry; Elicit may be the tool that would help most, because the critical paper is probably already in the corpus. Stuhlmüller splits long-term safety from misuse. Misuse has not been a big concern at current scale, but it will matter as models do more work. Borderline use cases — defending against a pandemic versus creating one — would require an account review, or a version of the product that simply does not support them. He points to a recent paper by Dan Hendrycks and others on unlearning the worst knowledge, the Weapons of Mass Destruction Proxy benchmark: you should not be able to ask an unreviewed account to make a weapon and get a recipe.

Factored evaluation, Jungwon notes, can watch the research log for social harm as well as for hallucination, and users already care about the impact of their own work. Labenz red-teams every new product. Most app developers put in almost no safeguards, and fine-tuning often seems to wipe whatever guardrails the base model had. Inflection’s Pi is the exception he has never really jailbroken. It is good at noticing when the user is becoming deranged — “whoa buddy, you sound pretty worked up” — and talking the person down. Detecting that a person seems unwell may be easier than detecting that content is inherently bad, because so much knowledge is dual-use.

Jungwon treats that as another case where making the product better is the defense. Elicit already wants the intent behind a query, because intent is how you return relevant papers. In 99.99 percent of cases that capability just delivers a better experience. In the rare case, the same capability is what lets them notice someone trying to do something sketchy. Expanding a messy question into a structured one — a trend Labenz also saw at Exa and in Anthropic’s meta-prompt tools — is both a product improvement and a natural gate before the system spends a lot of compute.

Biomedicine remains a top domain because it will convert budget into accuracy and already runs batch workflows — systematic reviews, meta-analyses — that generalize. The user mix has evened out. Early on, biomed plus machine learning were about half the users; engineering (mechanical, chemical, environmental) and the humanities have grown fast. The status quo being replaced is often a person hired to slog through thousands of papers, work so rote that even the people paid to do it do not want it. Teams will spend thousands to tens of thousands of dollars on a project; some individuals will spend hundreds because the value is obvious. The levers are how many papers you consider, how shallow or database-grade the extraction is, and how deep you go on any one paper.

For Labenz’s own sparse topics — mixture of experts, curriculum learning, the rare paper at their intersection — Stuhlmüller says the next spend is a pivot of the table: rows as concepts, papers filling downstream cells, so you can see types of curriculum learning and the evidence for each. A basic “list of concepts” workflow already exists, more comprehensive and slower than default search, and he is often surprised how well it works. Habit formation worries him less than it would in a consumer app. Weekly launches keep even periodic users aware of the company; priority stays on high-value batch work with no real alternative. An “AI cable bundle” of many apps under one subscription, Labenz’s side idea, gets a fintech-shaped shrug from Jungwon: it depends who owns the customer.

The nonprofit-to-company story, she says, is older than OpenAI. Biotech and university tech-transfer offices have long done basic research without commercial constraints, then stood up a separate vehicle to scale. They took external advice, including from OpenAI and Anthropic early last year, used nonprofit lawyers, left the former nonprofit under an independent board, and ran independent valuations so the nonprofit was clearly better off. The push came from philanthropic donors and the board. Elicit is now a public benefit corporation. Mission and money converge: to succeed commercially the product has to be accurate, trustworthy, and good at complex reasoning. All employees moved to the commercial side; the nonprofit is the board.

Hiring is the constraint. The team is twelve people. Stuhlmüller most wants senior software engineers — Python and TypeScript, front end and back end, people who have built modular systems for a while, not a bootcamp and not necessarily an ML background. The job is making reliable systems out of unreliable parts, and the talent density already includes people from Google, Stripe, and Square. Jungwon wants product designers for evaluative interfaces, product marketing, and a data scientist to marry offline evals with live feedback. She closes where the cold open began. They have a large vision of transforming research, and there may not be a lot of time left before the tool has to be useful for high-stakes decisions in times of chaos.