Trust at Scale: Elicit's Founders on Reasoning Languages, World Models and Automating the Company

The founders restate their mission: to radically improve the quality of reasoning, especially for high-stakes decisions. Since the last episode two years earlier, Elicit has moved from paper search and summarization to systematic literature reviews with a fixed search, screen, extract and write flow, and now to flexible research agents that run for long stretches. Stuhlmüller describes redoing computational experiments from an old workshop paper in the cloud, finishing in about ten minutes.

They illustrate why process still matters with a recent test. Asked to analyze about 100 toxicology papers, several agents, including Claude, ChatGPT and Elicit's own, were then asked how many they had actually read and replied that they had not done it. The founders attribute this to models being trained on outcomes rather than process, and note that if you only looked at the final analysis, you would never catch it.

Elicit's answer is a custom programming language that research agents write to orchestrate other agent calls: screen these papers, then extract data from them. The process runs as written, so you know it did what it claimed. Byun says rebuilding Elicit on this agentic infrastructure began around March or April 2025, and the goal is to keep transparency and systematicity at scale without losing the raw power of frontier models. She frames the guarantee as the same process applied to 10,000 documents, drugs or targets.

Byun describes the customers as a funnel. The largest group is academics and individuals who want a fast, robust and well-cited synthesis of the literature; Elicit requires multiple citations from vetted sources for every claim, avoiding references to papers that do not exist. Next come researchers running systematic reviews who need to apply one inclusion process across all papers, extract data from charts and figures, and weight findings by quality.

The fastest-growing motion is life sciences, where Elicit now formally works with seven of the top 20 companies across the development life cycle. Early-stage biologists and toxicologists iterate on experiments and run tournament-style rankings over thousands of genes and targets. Commercial and medical teams, after phase two and three trials, must justify launch strategy, target populations and pricing to regulators and payers with every claim backed by evidence.

Host Nathan Labenz recounts how doctors told him there was "no evidence" for options that lacked gold-standard randomized trials, and asks how wide the net for evidence should be. Byun says citation counts, journal impact factor and personal familiarity are lossy proxies, noting that one foundational CRISPR paper appeared in a lower-tier journal. Language models let Elicit look at actual methodology and let researchers set their own standards, such as requiring randomized trials or a minimum sample size, depending on the domain.

As the research agent pulls in web sources like company filings, Byun says it still has a fuzzy sense of which sources are better and the team wants to structure this further. The next step is expressing confidence per claim: breaking insights into claims and signaling whether each is nearly certain or contested by conflicting results. The aim is that confidence be usable rather than binary.

On calibration, Stuhlmüller says token probabilities are now useless, but verbalized probabilities from current models are better than older token-level ones. The real problem is instability: ask how likely a trial is to fail, then mention the base rate or thin prior research, and the model shifts readily. Unlike an expert who says they already considered that, the models do not seem to hold a coherent world model behind their numbers.

Byun predicts that within a few years generation will largely be a solved problem, with AI taking the first pass at everything and humans evaluating whether it was right and trustworthy. That creates a skills transition: documenting failure modes, codifying best practices, collecting good examples and building better verifiers. She argues that infrastructure for articulating what good looks like is badly needed.

Labenz raises the worry from a recent recursive self-improvement event that safety plans lean heavily on chain-of-thought monitoring. Byun recalls that the team built an early language-model observability tool, the Interactive Composition Explorer, around 2021, and learned that replaying every step is inefficient. Better checks are independent ones, such as sensitivity analyses and logical consistency tests.

Stuhlmüller adds that you can check either the process or the outcome, but an outcome should then carry a certificate of reasoning, like a formal proof in mathematics, listing how conclusions change with different inputs and what was examined. He also notes that tool calls are visible reasoning artifacts: if a model never opened the methodology section of a paper, you can see that. He expects tasks like "is there any paper refuting this claim" to stay hard to trust, because the proof is in the process.

Labenz asks how far fuzzy tasks can be decomposed into easy-to-verify pieces and whether something emergent is lost. Stuhlmüller explains why Elicit cares: models trained with verifiable rewards are excellent at coding and math but weak at fuzzy tasks, and he finds them surprisingly useless for planning company strategy even with access to his email and Slack. He says the models do not seem to build a coherent picture of what is going on.

He distinguishes two situations. Creating a reward signal for training is rough, because models will optimize hard against any imperfect signal. Checking an already trained model is easier, since you can test whether claims are internally consistent and whether different decompositions reach the same conclusion. Elicit does not post-train foundation models, and still does some fine-tuning, which he says is now mostly for efficiency at scale rather than for new behaviors.

Stuhlmüller motivates world models with a friend's cancer case, where a systematic review still left about 5,000 relevant papers. A giant context window would not reason coherently over that, and a Karpathy-style LLM wiki of markdown files is a good start but too flexible. He wants a representation that can answer prediction, intervention and counterfactual questions, such as what happens under one therapy versus another, which sounds like graphical or structured probabilistic models.

The goal is the best of both worlds: language models flexible enough to transform representations, with answers that stay internally consistent as new papers add symptoms and treatment strategies. This would let Elicit spend enormous compute turning large evidence bases into something that can answer questions for projects far larger than one query. Stuhlmüller describes it as continual learning that humans can inspect and understand.

He is deliberately not prescriptive about form. A cancer mechanism might be a graph of events, while company planning might be a spreadsheet of user numbers and margins plus a tech tree, and the challenge is propagating information between such lenses. He calls it an ongoing research problem. His practical starting point is ordinary SQL tables fed from sources like Mixpanel and Amplitude.

Labenz asks whether planning should give way to rapid building and shipping now that coding is cheap, and whether pharma should do many more trials rather than choose better ones. The founders caution that software cost was only one bottleneck: user attention is finite, bad impressions are costly, and teams can get stuck on a local maximum. In their view cheaper engineering has so far mostly changed things at the margins, like bug fixes and small admin features, while core capabilities still need careful intent.

On build versus buy, Byun says pharma companies should build only their core comparative advantage, and that heavily regulated, interface-heavy workflows like systematic review are not where a pharma company wins. On trials, she says why not both: digital twins and simulated trial effects may allow more bets in areas like rare disease, while a higher bar fits drugs with other toxicity concerns. Stuhlmüller adds that patients can join only one or two trials, and a company has finite resources, so choosing well remains essential.

Byun gives a concrete example: for an executive hire whose requirements had shifted during the search, she wrote a rubric, then gathered interview notes, email threads and feedback into a Claude project. She had it fill in evidence piece by piece for roughly 20 dimensions, such as consistently hitting goals or building a great team, before asking for ratings and a synthesis. It checked her own process and avoided recency bias, and the candidate said it was the most comprehensive professional validation they had ever received.

Stuhlmüller's example, which he calls more mundane, is weekly planning: working backward from monthly and yearly goals, calendar blocks and dependencies, a constraint-satisfaction problem he thinks models can help with. He runs an automated version that he dislikes and an interactive one that walks him through the plan, so he remains in the outer loop for now. He muses that eventually the model might be the outer loop and the human the inner one.

Elicit's company goal is that over year-end vacation the business keeps running, and software engineering is the first function they are automating. The system, called the line, starts when someone reacts with a line emoji to a feature request in Slack or when customer support feeds it in. It specs, implements, records a test video, reviews, and merges to dev and prod, merging roughly 30 to 50 issues per week fully automatically.

One founder does not expect the whole company to run unattended by year-end, since high-level steering will still be needed. The main obstacle is that models are not calibrated about when to call a human: if the model's claim that a feature is automatically reviewable is right only 80% of the time, that would break production too often. The founder points to the gap between 50% and 80% success curves on METR's long-task graph and wishes for an ultra-reliable mode that guarantees certain classes of errors never happen.

Labenz asks whether companies will pay ever-rising token costs. Byun says Elicit's offering displaces services spend, so the barrier is more about price anchoring than dollars, and pharma is disciplined about ROI. Stuhlmüller says he spends about $2,000 a week on tokens, probably among the top five at Elicit, and could perhaps double or triple it but not much more.

He does not use fast mode because the marginal return is too low for most tasks. He expects one smart orchestrator agent to dispatch many cheaper agents, a pattern already used in the Elicit app. His own spend comes from an elaborate setup built on Pi that orchestrates agents, from background automation for calendar, journal and email triage, and from cross-checking models against each other, which he says often improves results and doubles or triples cost for a quarter of his use cases.

On model convergence, he keeps being surprised by how similar models are, but says small differences still matter. Elicit has no model picker because tasks like screening and extraction use different models. When they compared Claude Opus 4.5 with Gemini 3 Pro, Opus was better at extraction accuracy while Gemini beat it by at least 5% on the share of claims directly supported by evidence.

Asked about exposing Elicit to other agents, Stuhlmüller notes there is already an MCP server and an API documented at docs.elicit.com, and that he runs systematic reviews himself through the API. On AI for science, the founders reject the idea that someone will simply win: the field spans cloud labs, protein models, single-cell dynamics and multi-year pharma plans. They do not know whether today's top 20 pharma companies will transform or be replaced by companies not yet known.

Labenz asks whether deeper integration of modalities, as in image models, will make process supervision harder for science. Stuhlmüller says continuous representations might work well for something like protein dynamics, but he would not bet on neuralese, reasoning in weight space, taking over language. The key property of discretization is error correction: a slightly smudged word can still be recognized, so errors do not compound over long chains, much as digital computers beat analog ones.

Asked how the species is doing on reasoning, Stuhlmüller says he gets better answers to questions he cares about, but that does not indicate what matters most. He thinks AI has transformed basically nothing yet outside coding, and that the big decisions by governments and large organizations are still ahead. Models optimized to look good and persuade could worsen epistemics if nobody makes truth-seeking an explicit priority.

He also sees a possible basin of attraction: models optimized for truth-seeking could help people forecast the results of their actions and then prioritize further epistemic interventions. He says we are still before a point of no return in either direction. He closes with a reminder, echoing the METR study in which engineers felt faster but were slightly slower, to introspect on whether models are actually improving or narrowing your decisions.