Nine of the top ten research ideas were Claude's

Nathan Labenz introduces Chenglei Si as a Stanford PhD student working on ways to automate research, and as lead author of a paper with a blunt title: *Can LLMs generate novel research ideas?* The study asked more than a hundred AI researchers for new ideas, paid cash prizes for the best ones, asked Claude to generate ideas of its own, then had experts score the anonymized texts. The AI side scored higher on novelty and excitement. Online, Labenz says, the paper became a Rorschach test. Believers called it a milestone. Skeptics attacked the methods, and more persuasively noted that every topic was a prompting technique, which may not travel to harder sciences.

Si came up through NLP as an undergraduate and arrived at Stanford the previous year, where he met advisors he names Steve and Tatsu. Language models were getting good at reasoning, math, and code, and he wanted a PhD project that was not just another task. Automating research felt larger than beating a benchmark. The advisors were excited. The project started about a year before the episode and took roughly a year to finish. Building the idea-generation agent took two or three months. Finding people and running the human study took about eight. The AI work was the fast part.

Two inspirations sit behind the agenda. One is self-improving AI, including early Stanford work he associates with a researcher named Eric: if models become smarter than the humans who train them, the next ideas may have to come from the models. The other is a long HCI tradition of research-assistant and creative-support tools, including work around Semantic Scholar. Si's group is more aggressive than the assistant frame. They want to know whether the entire research process can be automated, not only supported.

Labenz's own model of why the question matters is an intelligence-explosion story that is no longer original and that he does not claim as his. At some point, systems may make meaningful contributions to machine-learning research. If that research becomes ten, a hundred, or a thousand times more productive, the quality of the resulting models becomes very hard to bound. Clear the hump and the future looks unpredictable. Fail and things stay somewhat normal. Si has read Leopold Aschenbrenner's *Situational Awareness*. Tatsu warned him that "intelligence explosion" sounds too aggressive in an academic room. On this podcast he will say it. Aschenbrenner's figure that stuck with him is an automated Alec Radford around 2028. Radford, lead author of the GPT paper, had spent the previous day being nominated on Twitter as the most influential scientist who is still not widely known. That bar is the wrong first comparison. There is one Alec. Si wanted to know how the current generation compares with average PhD students, and to give the field a concrete benchmark.

Labenz walks through the recent attempts that, in his telling, have always been a little short. Shortly after GPT-4, Gabe Gomez's group at Carnegie Mellon released Co-Scientist. A user could say "please synthesize aspirin," and the system would look up a route, translate it into instructions, and drive a real automated lab at Emerald Cloud Lab through an API, until purified powder came out the other end. Labenz had Gomes on an earlier episode and still thinks the result was major. What it was not, in his review at the time, was idea generation. He called it text-to-protocol: the human supplied the scientific thought, and the model broke the thought into steps.

Sakana AI's AI Scientist took a step he cares about more. It generated many ideas, queried the Semantic Scholar API, and tried to ask whether each idea already lived in the literature. Papers written by that stack did not look like NeurIPS material. Sakana's own write-up compared the work to an early graduate student who still needs advising. Even so, Labenz saw a mechanism pointing at the destination: generate, check novelty against the corpus, prioritize what looks new. NVIDIA's Eureka paper, from Jim Fan's group, was the first time he was willing to retire his old GPT-4 headline of "no eureka moments." GPT-4 wrote reward functions for robot-hand reinforcement learning and beat the human authors. Writing a reward function is not a street skill, and the outcome was objectively measurable.

Si's definition of a eureka moment for automated research is stricter. One day an agent generates an idea, executes it into a full project, submits to a top conference, and wins best paper. That is the bar, because it would mean the work is better than the average paper those venues already accept. Sakana's prototype is useful. Its own AI reviewer, he notes, does not reach a NeurIPS bar, and automatic reviewers are not ready in any case. In their analysis, agreement between a language-model reviewer and a human reviewer is low. After a sponsor break, Labenz pulls Table 11. The setup is a simplified accept-or-not decision on the lower half of papers. Random is 50 percent. NeurIPS human pairs reach 66 percent. ICLR reaches 71.9 percent. Their model implementation reaches 56 percent. Humans still agree with humans more than the model agrees with humans. Even if that gap closed, Si would hesitate to hand a subjective, high-stakes reviewing task to a model. Expert agreement at ICLR and NeurIPS is already low. That is the evaluation problem the paper is really about.

Two design principles governed the study. First, numbers without a human baseline do not mean much, and PhD students are the experts at this task. Second, experts do all the judging, because language-model reviewers are not ready. One branch of ideas comes from AI. Another comes from human researchers. Expert reviewers score every idea without knowing which is which. A third condition exists because the generation agent already ranks thousands of ideas per topic and keeps the ones it likes. That AI ranking is imperfect. Some excellent ideas may sit in the pile and never surface. To estimate an upper bound, the authors also had a human look at the generated set and pick the best. Those texts are still AI-written. Only the re-ranking is human. Reviewers then score novelty, excitement, feasibility, and effectiveness.

Humans were paid $300 to take part. There were additional prizes on the order of a thousand dollars for ideas judged best among the people, which Labenz notes is not nothing on a graduate stipend. Participants were mostly PhD students in machine learning, with a focus on natural language. The AI workhorse was Claude 3.5 Sonnet. The team had started on GPT-4. Colleagues in the lab were saying Claude 3.5 was stronger on science, reasoning, and coding. Si compared the two idea piles by eye, decided Claude had the edge, and stayed there. OpenAI's o1 had just been released and was on his list; they had not benchmarked it yet.

The generation prompt is not "give me some NLP ideas." Related papers are retrieved for the topic and dropped in as inspiration. Six hand-built demo examples, drawn from existing papers, fix the shape of an idea: motivation, problem setup, proposed method, a step-by-step experiment plan, and test examples. Ideas are generated in batches of five, because a single completion cannot hold the volume they want. Previous titles are appended with an instruction not to repeat. The model repeats anyway. After semantic-similarity filtering, about 95 percent of the pile is gone. Roughly two hundred ideas per topic survive as non-duplicates. A curve in the paper shows early batches mostly new and later batches mostly echoes. Among the first thousand generations the unique count saturates near two hundred.

Labenz asks whether the uniqueness curve is a logarithm you can keep buying, or a flatline. Two stories fit the same plot. The topic may simply contain a finite set of ideas. Or the space may be large and the model may have a low ceiling on diversity. They did not squeeze hard. The anti-repetition trick was appending titles. Fine-tuning for diversity and smarter decoding were left on the table. Only after those are exhausted would he treat a ceiling as a fact about the base model. Turning the diversity knobs up still leaves a quality question. Unlike code, you cannot unit-test an idea. A proxy ranker with even a modest correlation between bad and good may be enough for inference scaling: generate more, hope the ranker surfaces something better. The cleaner endgame is an execution agent that implements each idea, reports baseline and proposed numbers, and turns ideation into something as checkable as a coding pass rate. Sakana's papers cost tens of dollars each. If one in a thousand is good, the economics already work, and prices are falling. Someone at a frontier lab told Labenz that people still have no intuition for inference compute. They try once, twice, three times. A thousand or a million is a different game.

Ranking is where a practical lesson sits. Asking the model for a one-to-ten score worked poorly. Pairwise comparisons worked much better. Labenz had imagined a March Madness bracket, then a double-elimination bracket, and both have the same failure: the two best ideas meet in round one and one of them dies with a zero. The team instead pairs each idea with another whose accumulated score is closest, Swiss-style, and never eliminates. A strong idea that loses still has a record. Chenglei then re-ranked the pile himself. He did not look at the AI order. A few ideas he immediately treated as top-tier had not been marked that way by the model. The discrepancy is the point of the third condition. The re-ranker, Labenz confirms, is Si, working alone. It takes forever.

Every idea in the study lives inside prompting: how to get more out of language models with different prompting techniques. That scope is the background for later criticism. Once the human pile, the AI-ranked pile, and Si's re-ranked pile exist, a rewriting agent puts them all through the same academic style. The prompt allows wording changes and forbids content edits. Human originals leak. Some sentences are ungrammatical. Some writers are conversational and eager in a way that does not sound like a model. Generic AI detectors are unreliable, but people in the field have an ear for *delve* and its cousins, and a preference for human warmth would dirty the scores. Both sides go through the same transfer so that reviewers are not grading voice.

Each idea is a fleshed-out sketch of about a thousand words, not a slogan. Reviewers score five dimensions from one to ten: novelty, excitement, feasibility, effectiveness, and a separate overall. Feasibility means "can this actually be run" — not train GPT-5, not label a gigantic set by hand, not fine-tune a closed model with no fine-tune access. Effectiveness is different: if you ran it, would it beat existing baselines? Si jokes that their own paper would fail the budget test. Humans cost about $30,000. API credits cost a little more than that. Generating four thousand ideas is a few thousand dollars, times seven topics, plus prototypes. The expensive parts are over-generation (they pushed some topics to eight or ten thousand ideas to test the limit), a prompt that grows as titles accumulate, and a novelty filter in the appendix that compares each idea to retrieved papers one by one. A valid, non-duplicate idea costs a couple of dollars.

The headline is the one Labenz had been holding. After the rewrite, expert reviewers scored AI ideas higher than human ideas on novelty and excitement, and the difference was statistically significant. Feasibility leaned slightly toward humans and did not clearly clear significance. Effectiveness was close. Overall favored AI, and the gap widened when Si's re-ranking selected the AI set. He was surprised by the novelty result. Numbers can be massaged. He also read every AI idea and every human submission, and the texts, to his eye, support the table. The AI pile has a vibe. The ideas are a little weird. They do not look like the typical paper of the last few years. They are less grounded in existing work and more willing to step outside the box. That is what he means by novelty.

Labenz's usual classroom story is that models vary wildly *across* tasks — superhuman at translation, clumsy at stacking objects — and stay fairly consistent *inside* a task. The standard deviations here were essentially the same for AI and humans, and a bit wider for AI on excitement. Peak scores were higher for the model: novelty eight versus ten, excitement eight versus nine. Tyler Cowen reads people for their peaks. One lifetime idea may be the whole contribution. Rank every idea by overall score and look at the top ten: nine are AI, one is human.

Watching reviews come in, Si saw the usual mess. Reviewers disagree. Some systematically love prompting and hand out sevens and eights. Some hate it and live in twos and threes. Some favor a topic and lift every idea they see on it. He and his advisor agreed in advance to run every statistical test they could think of, and to withhold the claim if any one of those tests failed. They treated each review as a point, then each idea's average as a point, then each reviewer's own AI-versus-human gap as a point, so a harsh grader who happened to get more AI ideas could not manufacture the result. Novelty survived all three. Each idea had two to four human reads, three on average.

Si is more confident in the novelty claim than in the idea that novelty is enough. An idea can look unlike anything in the recent literature and still be a weird idea that does not work. Reviewers already try to guess effectiveness, which even experts do poorly. Feasibility was a bit worse for AI, effectiveness roughly similar. The honest next measurement is to implement the ideas. The group is recruiting people to turn both the human and the AI proposals into full projects. Either outcome is interesting. If the strange ideas also work, the story hardens. If the ranking reverses, the failure is not only about AI. It would call into question whether human evaluation at the idea stage is reliable at all, especially after nine of the ten highest-rated ideas came from the model.

One online critique that Labenz found worth digging into is sycophancy. Models trained with RLHF have learned that pleasing a rater is not the same as being right. Asked for a novel idea, a system might sprinkle novelty-sounding words over something thin or incoherent. The shared paraphrase prompt was supposed to flatten that. Si does not claim it did so perfectly. Some AI drafts may have kept novel-sounding language that the rewriter did not strip. A planned follow-up will look at word choice and style. The only way to close the file, he says, is the execution study.

The ideas he personally likes cluster on uncertainty and confidence calibration, one of the seven topics. An example that made the paper upgrades self-consistency. Generate many diverse answers, then instead of majority vote, measure how each answer supports or refutes the others, build a graph, and use centrality or other graph metrics as a confidence score. Another idea, weirder and not in the paper, first asks the model for uncertainty exemplars: an answer that would deserve a ten, an answer that would deserve a nine, and so on. A new response is then matched to the nearest exemplar. It looks like no uncertainty estimator he knows. It has not been tested. If that kind of strange idea works, he will be delighted.

Why execute rather than rerun the study in another field? Both are possible. They could drop the prompting-only constraint and point the same generator at synthetic data, decoding, or alignment objectives. Critics are right that the human ideas are not the best humans can do. The writers themselves said the submissions were about average PhD quality, not top-PhD quality. A stronger baseline, Si hopes, is papers actually accepted at a top venue such as ACL: compare the generated ideas with work that already cleared peer review. Execution is the other track, because it replaces a prediction about whether an idea will work with a measurement.

A still larger step is an agent that does the running. Preliminary attempts already write code that executes and emit baseline and method numbers. Look at the code and the cracks show. On a text-classification task the agent used a keyword method as the baseline, which is not a serious baseline in this era. Sometimes it skipped steps of the proposed method. Those errors do not appear in the final number. They require checking intermediate implementation, not only the reported score. He wants that evaluation posted as a community challenge. If idea generation and automatic execution can be chained and scaled, the best-paper eureka moment becomes a search problem instead of a wish.

Labenz has been coding with o1 and Cursor. The model will read a whole repository and a feature plan, then return a critique and a better alternative. He thinks Claude 3.5 Sonnet and the GPT-4 class, given heavy system design and millions of tokens, can sometimes produce a meaningful new idea, but not yet in a way that makes frontier research widely accessible. o1 looks to him like the generation that might change that, especially if it can rerank near human level and shorten the cycle. Si has not measured it. He wants to know whether it is a better ranker and whether it is more diverse. He also has a pile of conference papers and is curious whether a smaller model, fine-tuned on that data, could run the same pipeline more cheaply. Listeners who want to implement an idea can be paid a few thousand dollars. That study is open.

The milestone he will defend in public is an end-to-end system that invents, executes, and wins a paper award at a top conference. A milder milestone is simply getting an automated paper accepted. His hot take is that the milder one is not far. Some of the best ideas they already have look, to him, like work that appears at ACL. Nail execution and the shape is there. Mass-producing average papers is not the agenda. The hope is work beyond average PhD quality, demonstrated at least once. He thinks that takes a couple of years. The personal deadline is graduation, three or four years out. He knows the estimate is aggressive. It is a goal, and it sits on the same calendar as Aschenbrenner's timelines. Labenz closes on the paper's question: it is one of the most important questions in the world today.