Metaculus Flagged Covid in January 2020. Now It Is Putting AI on the Forecasting Desk

Nathan Labenz opens with two numbers that still sit uneasily next to each other. Metaculus has hosted some of the most cited AGI-timeline questions for years, and fewer than 1,500 people have ever registered their own forecasts on them. He also reminds listeners that he once entered one of Philip Tetlock's original Superforecasting tournaments, about fifteen years ago, after Tyler Cowen posted an invitation. He did reasonably well and was never named a superforecaster. The craft, he has long thought, could help most organizations make better decisions. It has not become a mainstream technology, in part because doing it well is painstaking work and hard to scale into the messy, contextual situations where decisions actually get made.

The obvious next move, on a show about AI, is to ask whether models can do the labor. Labenz says several recent papers, including work involving Tetlock, already show AI forecasts that are competitive with the best individual humans, though they still fall somewhat short of a human crowd. His guest is Deger Turan, a few months into the CEO job at Metaculus. Turan had been at the AI Objectives Institute, a nonprofit lab working on sociotechnical alignment: whether a group of real people, in a real deployment, perceive a model as better aligned with their goals. The through-line of his career is collective intelligence — how to fold many perspectives into something like a shared world model.

Even a high-fidelity map of what people want, he argues, does not tell you whether action X will make the outcome better. That gap is what pulled him toward forecasting. In 2016, while still in school, he worked with Dan Jurafsky at Stanford on the flood of comments in the FCC's net-neutrality debate — about two million pieces of raw text, which felt enormous at the time. Form letters were not automatically illegitimate; bot spam was a different problem; a clumsy but genuine opinion was a third. Every time the team narrowed the metric, Goodhart's law showed up. The crusade he has been on since is to get past "how happy are you, one to seven" and leave room for a qualitative answer the speaker may not yet be able to name.

Labenz brings the Goodhart problem into the present. GPT-4o mini had just climbed ahead of Claude 3.5 Sonnet on the LMSYS arena, while heavy users still treated Sonnet as the stronger model. Turan refuses the generic "which model is better" question: better is content-specific, and even inside Metaculus he finds Claude more useful for question writing than for fact retrieval. Usefulness, he says, is almost always anecdotal. Accuracy on a named tournament is a much narrower thing. The ACE-style competitions are interesting for how accurate the forecasts became. They do not answer how an end user is supposed to act.

That distinction is the agenda he wants for Metaculus's next two quarters. Wisdom of the crowd works — and, he adds, so does aggregating bots trained with language models. The case studies are already there. What is missing is a forecasting deployment that asks what levers a decision maker actually holds, and where the world models start to diverge. He wants to find inflection points, extract schools of thought among forecasters, and get much more serious about conditional questions: if we pull lever X, does the future get better?

Labenz offers a recent, almost too-simple illustration. Reporting suggested the sitting president was not sold on the idea that someone else had a better chance until polling data made the gap hard to ignore. Turan calls that a simplified case that was obvious to almost everyone except the decision makers, for longer than it should have been. The cases he actually wants are more sophisticated: suppose you have $50 million to allocate across microplastics, climate risk, and AI risk. The honest version of that question requires a world model, and the simplest tool is a conditional forecast. Conditional questions are hard to think in, which is why he wants LLM- or discourse-driven tools that lower the barrier.

After a sponsor break, Labenz asks the product question: is Metaculus a prediction market, a forecasting platform, or something closer to Polymarket and Manifold? Turan will not call it a market. Metaculus aggregates forecasts; it is a forecasting platform and an aggregation engine whose goal is "epistemic security" in how we picture the future. He is happy with "Wikipedia about the future." There is no monetary incentive on the question itself, and that changes the behavior. On a market you buy and sell contracts on an outcome. If the current price is 60% and you also think 60%, you have no reason to add the information — it can only get worse from there.

Metaculus instead scores people over time on calibration and accuracy. The community prediction keeps each forecaster's most recent estimate, then weights by recency; the Metaculus prediction, especially on paid client work, also weights track record. Binary questions become a weighted mean of probabilities. Numeric and date questions become a weighted mixture of full distributions. Copying the community prediction will do pretty well in a tournament, Turan says, but that is free-riding. The honest move is to do your homework first and only then look at the crowd, so you still add independent information.

Money is not absent. Strong track records get people hired as pro forecasters on narrower questions for federal agencies, large retailers, hedge funds, and think tanks — questions that never hit the public site. The platform also pays for reasoning, not just a number. Turan was in Taiwan talking about whether the electricity grid would be challenged in a potential invasion when the forecast jumped from 30% to 40% with no comments underneath. A government-side counterpart told him, in substance, that without knowing why, the move could not enter a decision. Reasoning transparency is now one of Metaculus's core projects.

Turan is less interested in another accuracy leaderboard than in moments when an institution or a person could act. In epidemiology, he says, Metaculus outperformed expert panels on covid, informed hospital resource allocation and public-health decisions, and beat baseline models on vaccine uptake. During the mpox outbreak the community produced a view quickly. In January 2020, conventional wisdom still held that covid would not be significant; Metaculus was already predicting that more than 100,000 people would eventually be infected. Some readers took that as an early warning. He remembers a post about someone making a pile of investment decisions because Metaculus said so.

The comment he likes best is more concrete. Before Russia's invasion of Ukraine, a Metaculus user in the country wrote that they had moved from Kyiv to Lviv on February 13, "entirely thanks to this prediction thread and the Metaculus estimates." That, for Turan, is the instrumental case. The numbers have been cited in The Economist, Our World in Data, Bloomberg, Fortune, and Forbes. Scott Alexander's Astral Codex Ten recaps — including a "who predicted 2023 best" comparison of Metaculus, markets, and superforecasters — are a reference anchor he is happy to point people toward. The open question is how to turn that track record into repeatable action, including on short-horizon problems like homelessness in San Francisco, where an extra $5 million might build two houses or fund a lobbying theory about housing-market externalities.

Labenz has been coming back for years to Metaculus's weak-AGI and strong-AGI timeline questions, each with four resolution criteria and a date distribution to draw. They were informative for a long time. As the dates get closer, he feels a split between what actually matters and the letter of the question. The weak-AGI item still requires an expert interrogation in which the interrogator cannot tell who is the AI. He does not think current systems are close to that bar, and he thinks the reason is a product choice: if you wanted to fool an expert, the first thing you would do is make the model say "I don't know" much more often and narrow its range of answers. Breadth of knowledge is currently a tell.

Turan is not sentimental about the operationalization, and he is not eager to retire a Schelling point either. Both questions date to 2020; you will not always write a question that is optimally informative years later. In 2020, he argues, they were useful enough that Metaculus probably moved the Overton window on how seriously people took AI's impact. He quotes Jaron Lanier, a line he once put at the front of his thesis: the Turing test cuts both ways — if you can converse with a simulated person, how far have you let your sense of personhood degrade to make the illusion work? Passing a closed-box interrogation is a low bar. A system that merely learns to say "I don't know" on cue is not the ambition he wants.

If he were writing the questions today, he would not hang the picture on a single operationalization of AGI. He would rather have an index: thirty forecasts that rhyme, the way a basket of tickers makes a composite, instead of one brittle criterion. The new "five years after AGI" series interests him more, because the answers reveal what people even count as a critical possibility. A running joke from his AI Objectives Institute days: if, in a post-AGI world, you do not think Jerusalem will be a united and peaceful city, maybe your AGI is not ambitious enough. A system worthy of the name should be able to resolve clashes inside the existing human sphere, not just simulate a convincing interlocutor.

Labenz asks the density problem. A million new questions would need a billion predictions. Turan agrees that human attention is the real currency. Ten thousand new questions a day would drown the system. That is why he wants indexes, and why AI forecasting is not a spectator sport: bots should be a scaffold so a human can reach a rigorous forecast faster. The win condition is not "AIs forecast and humans watch." Incoming questions already get heavy editing so they stay resolvable and meaningful; as Metaculus open-sources, he wants domain experts writing questions and hosting their own instances, maybe with different scoring rules. Add new lights, he says. Do not go back and rewrite what the old questions already accomplished.

Labenz read the three papers Turan had shared and came away "overall pretty impressed." The first, *Approaching Human-Level Forecasting with Language Models*, is from Jacob Steinhardt's Berkeley group. It is the intuitive stack: a retrieval system, fresh news, processing, a forecast, plus fine-tuning and scaffolding. It came close to the Metaculus community prediction and still fell short. Labenz wanted a percentile: if that system were a single human, where would it sit? Turan does not have the number. What he does believe, without data he expects to have soon, is that a bot in the hands of a human forecasting team "will just kill it." Isolation tests are still worth running. The systems' obvious strengths — reading huge volumes of text, taking a first pass at a world model — are what he wants to build on.

The second paper, from Tetlock and coauthors, gave human forecasters an AI assistant. One group got the best assistant the researchers could build; another got a deliberately biased one. Both groups improved, the biased assistant less so. The paper also makes a methodological claim Labenz likes: studying what language models can forecast is a way to ask how far they generalize, because the future is, by definition, out of distribution. That they do it reasonably well is at least a point for the side that says these systems are not stuck inside their training set.

The third paper, also from the Tetlock orbit and pre-registered in the usual careful way, is *Wisdom of the Silicon Crowd*. Ensembling language models beat the individuals. GPT-4 was far ahead of the others in the data Labenz saw; his suggested tweak is to drop the worst models and run extra copies of the best. The paper also logged familiar pathologies: a bias toward round numbers, a tilt toward positive resolution, and the logical inconsistencies Turan has been seeing in the live tournament. Labenz's other note is simpler. The models will take a question about a foreign leader he has never heard of and just start. Breadth is the unglamorous strength.

The live tournament had already resolved about fifteen questions in its first weeks, and some of them were traps. Ask a bot the probability that measles cases will be under 200 and it says 75%. Ask for more than 300 and it says 70%. That leaves a 5% chance for the window between 200 and 300 — and the same bot will then assign that window 65%. A trained pro forecaster would not do this. An ensemble can still fall apart, Turan says, because nothing is maintaining a mathematically rigorous world model. Untrained humans make the same class of error. The interesting gap is between intuitive reasoning and enforced consistency.

His proposed fix is model-based reasoning on top of the language model: as the system absorbs news, it should also be building a world model you can inspect, with constraints you can enforce — scope sensitivity, negation sensitivity, budget adding up. He names tools in that spirit, including a simple Squiggle integration and Open Agency-style architectures. A Stanford paper from about a year earlier, *From Word Models to World Models*, is the citation he wants more people to build on. The technical version is auto-formalization: take a natural-language decision, emit variables and constraints in an optimization language, and let a library such as CVXPY do the arithmetic a language model will not reliably parse.

That stack, he argues, is not only a path to better forecasts. It is a path to more auditable AI. He does not expect to get all the way there by staring at one model's weights. A composite system that separates world-model building, option-space construction, and preference elicitation is something you can point at and say which part is wrong. A forecaster could hand over thirty related estimates and ask where they are logically inconsistent, instead of keeping an Excel sheet of intuitive probabilities. If that works, the same architecture is a safer model, not just a sharper one.

The competition is the AI Forecasting Benchmark series, a year of quarterly cups. This quarter the bots will see about 250 to 400 questions; roughly $120,000 in prizes is allocated across the whole year, spread along the curve rather than reserved for the top three. There is no human in the loop. You may update the bot when you have a better strategy; you may not tune it for a specific question. Bots have to leave comments that show their reasoning. Those comments are not scored, but they are how Metaculus will later interview the winners. OpenAI and Anthropic have donated a generous pool of credits. Turan says they have already given some teams up to $5,000, and they will add more after a short description of the strategy. They do not host the bots. Entrants run their own infrastructure and submit through the API. Each question is open for 24 hours.

This quarter is binary only; Q4 will add non-binary questions. Continuous variables from the main quarterly cup — a geomagnetic K-index band, for example — have been carved into overlapping binaries so the organizers can test logical consistency against human forecasters. Early resolved items include French coalition politics, whether *Deadpool & Wolverine* would outgross an earlier *Deadpool* at the domestic box office (bots beat humans on that one), and a cluster of Democratic-party questions around Joe Biden. Scoring is a proper log-loss rule against a coin-flip baseline, then a peer score that asks how you did relative to everyone else. Overconfidence is expensive. If the crowd is at 60% and that is also your honest view, you should say 60%.

They chose an honor system. The volume — on the order of 600 questions in the blast — is itself a deterrent to quiet human intervention, and winning writeups will be scrutinized. Labenz notes that, unlike a puzzle contest where a hidden human would be an obvious edge, an individual is not clearly able to beat a decent bot, and might make it worse. Turan has already seen questions where the bots beat the aggregate of pro forecasters, even as they fail scope and negation tests. Newcomer bots have jumped the leaderboard fast. He tells late arrivals they are not too late, points them at a template that can spin up in about thirty minutes, and offers a short list of unromantic improvements: look up similar questions on other platforms, pull recent news, ensemble models, and notice that different labs are good at different chores. His anecdotal split is that Anthropic models have been stronger on negation and logical consistency, while OpenAI models have been more helpful at figuring out what in the question to attend to.

Labenz's meta-forecast is that ensembles of ensembles will, over the next year, become competitive with — maybe even surpass — the community prediction, especially as tokens get cheap. Turan will not celebrate that as a finish line. If a bot aggregate consistently beats the crowd, does that by itself bring a better world? He does not think so. What excites him is the cost of the explanation: it is already a couple of cents for a model to write a research-length account of its reasoning. That is much cheaper to probe than a human writeup. Companies from Elicit to FutureSearch are already in that neighborhood. The failure mode he fears is the opposite of transparency — an uncanny "seems legit" that is too expensive to scrutinize, the way legal language already forces laypeople to defer. If bots start to converse in a register that looks like higher epistemic status than a human can access, that is a failure of epistemic security, not a win.

Accurate forecasts already exist, he keeps repeating, and we have not seen every corporation and government suddenly run on them. The missing pieces are action-oriented: inflection points, short-term proxies, budget-allocation questions, and hypotheticals that may never resolve. He would like to ask what happens if San Francisco allocates $70 million to public transit under plan X versus plan Y, even if the city never does it, and to pay pro forecasters for that strategic information rather than for a leaderboard point. The same logic applies to Taiwanese sovereignty and to AI. Mini-Metaculus instances — he describes them as subreddit-like, goal-scoped copies that will appear as the platform is open-sourced — are one experiment. Indexes are another. He is also looking at the Foresight Institute's tech trees: if $5 million versus $15 million is the choice, which inflection points actually move.

Labenz asks the CrowdStrike question. In a world where a few models supply most of the forecasts, what does a correlated failure look like? Turan's answer is decentralization. He wants Metaculus to be accurate, and he wants many other people trying many other scoring rules in many other domains, because a political-outcome question and a synthetic-biology spending question are not the same sport. Superforecasters can beat domain experts; that is not an argument for firing the experts. It is an argument for pairing them. If someone else's AI ensemble became more reliable than the Metaculus prediction, he would treat that as a success to integrate, not a defeat.

Labenz asks whether there is a Metaculus–Polis mashup coming. Turan says Polis is probably the most influential platform in the evolution of his thinking. At AI Objectives Institute the long-running project was Talk to the City, an LLM-assisted aggregation and deliberation tool for unbounded qualitative input — text, even video — rather than Polis's tweet-sized recommender loop. They worked with Taiwan's Ministry of Digital Affairs on municipal questions and on same-sex-marriage data that had arrived through Polis. Another case he loves: a veterans' health labor union with a twenty-day negotiation window, too fast for real internal deliberation. The new president, Mark Smith, sent an open-ended survey; within 24 hours they had on the order of 200 responses, in more than one language, clustered into four viewpoints, and sent back out for another round. That cycle, Turan says, is now cheap enough to change governance. The missing pillar, once you can aggregate desirability, is whether the action will actually get people where they want to go. That is the conversation that brought him to the previous Metaculus CEO, and then into the job.

He is wary of the phrase liquid democracy. He has a name for the failure mode: the consensus illusion. Optimize for agreement and you get lukewarm policy that does not address the issue. He cites a 2022 DeepMind paper on fine-tuning language models to find agreement across diverse preferences — techniques that even made their way into Talk to the City — and then the bike-lane fable. Everyone wants more bike lanes, until you name a street. The model says no, we did not agree to narrow *that* street; we agreed to "more bike lanes." Consensus, in an electoral system, is power, and a drive toward consensus trades off against fidelity to a viewpoint. He does not want the most agreeable sentence. He wants three streams talking to each other: stakeholder desirabilities, policymakers' action possibilities, and domain experts' outcome likelihoods.

The healthier move is a double crux: I think A, you think not-A, and we both agree that some underlying fact would change our minds. Then you can bring research and forecasting. The unhealthy move is a risky alliance — we both endorse policy A, I because I think it enables B, you because you do not think B will happen, and we stay together for one step — which scales into a polarized society where unrelated attitudes travel as a package. One study he recalls asked not for an immigration slogan but for a statistical tolerance: how many mistaken admissions will you accept to admit one person who truly deserved to come. Self-described liberals split into people who would accept five mistakes and people who would accept 300. They were not operating on the same world model. Better policy, in his telling, is higher-visibility disagreement plus a shared picture of how the world works.

He does not expect a top-down conversion of national governments. He expects municipal and membership-organization experiments: Talk to the City with Japanese municipalities, interest from Taiwan and Singapore, Detroit groups focused on the wellbeing of African Americans and previously incarcerated people. The first step is visibility into what actually worked. Existential risk, as he inherited the thought from Peter Eckersley and the AI Objectives Institute, is a failure to coordinate in the face of a risk you can already see. The ozone layer recovering after the CFC ban, and the end of leaded gasoline, are the existence proofs he wants remembered. Shared world models should contain the disagreements, instead of collapsing them into the most agreeable action. Organizations working on climate, AI, nuclear consequences, or just better resource allocation are invited to try the tools on problems that pay off in the short term.