Sakana AI's AI Scientist and the Prospect of Automated Research

Sakana AI's paper on the AI Scientist has me thinking about whether automated research could be a step toward artificial general intelligence. This is my interpretation of the system's potential, not a result established by the paper. The project is a 2024 prototype for machine-learning research, and its generated results require independent scrutiny.

I want to set the stage with Leopold Aschenbrenner's essay Situational Awareness. Its illustration presents a scenario in which AI capabilities grow and then accelerate sharply once AI research itself becomes automated. The graph is a rough forecast, not measured evidence that an intelligence explosion will occur on its proposed timeline.

The proposed mechanism is self-improvement: systems would discover techniques and algorithms that improve their own capabilities, rather than relying only on human discoveries. I imagine agents working continuously with large amounts of compute. Whether that feedback loop can reliably produce accelerating improvements remains an open question.

Sakana AI's AI Scientist is interesting to me because it automates parts of a research cycle: proposing ideas, testing them, writing up results and running a simulated review. The first demonstration is limited to machine-learning experiments built from provided starting templates; generating a manuscript is not the same as independently verifying or publishing a discovery.

The project describes an end-to-end workflow that generates research ideas, writes and runs code, summarizes results, makes figures and creates a manuscript. It also applies an automated reviewer and can use feedback to develop future ideas. I see that focus on AI research itself as relevant to the self-improvement scenario, though it does not demonstrate a runaway intelligence explosion.

The researchers acknowledge flaws and hallucinations in the first version. They report that their automated reviewer performs near human accuracy in a particular evaluation against human review judgments. That does not establish that the reviewer is universally reliable or that its acceptance scores validate the scientific claims it assesses.

The examples cover diffusion models, transformers and grokking. Sakana reports a cost of roughly $15 per generated paper in its setup. I imagine many agents running continuously and building on earlier work, but the figure is a project estimate for generating papers, not a price for a verified scientific discovery or evidence that unlimited scaling is practical.

The team presents a sample paper on adaptive feature balancing for low-dimensional diffusion models. I cannot personally verify its technical validity from reading it here, so I rely on the researchers' account. They identify an interesting empirical direction while noting that the explanation for its success is not entirely convincing; the experiments and claims still need careful independent review.

The workflow starts with an idea and a plan, searches existing literature for novelty, and scores and archives ideas. It then uses an experiment template, a language model and the Aider coding tool to modify code, run experiments and update the plan. Numerical results and plots feed into a manuscript template, followed by an automated review and possible further iteration.

In a sponsored portion of the video, I discuss Qualcomm's on-device AI hardware and model library. Returning to the research, the AI Scientist begins with a code template for a chosen topic, brainstorms directions and searches Semantic Scholar for related work. A literature search is useful, but it does not guarantee that an idea is genuinely novel.

The system executes the proposed experiments, records notes and generates plots, then writes a paper in LaTeX with citations found through literature search. An LLM-based reviewer assesses the manuscript and supplies feedback for future iterations. These are automated steps in a constrained workflow, and the review is simulated rather than independent peer acceptance.

The first version lacks vision, so it cannot inspect or correct its own plots and page layout. Figures can be unreadable and tables can overflow. I expect multimodal models could help with these presentation issues, though adding vision does not automatically solve the deeper question of whether the experiment or conclusion is sound.

More serious problems include implementing ideas incorrectly, comparing against baselines unfairly and making mistakes when interpreting results or comparing numbers. The researchers save executed files to help reproducibility. Reproducibility supports checking the work, but it does not itself establish that the design and conclusions are valid.

The team also reports surprising execution errors. In one run the system edited code so that it repeatedly called itself; in another it tried to extend a timeout instead of making an experiment faster. These examples illustrate why generated code and its execution environment need oversight.

I still think a large number of autonomous researchers could eventually accelerate progress toward AGI. That is a prediction, not a finding proven by these experiments. The researchers also warn that automatically producing and submitting large quantities of papers could burden reviewers and make scientific quality control harder.

As generated material becomes abundant, I worry about the signal-to-noise ratio. One possible response is an AI filter that helps people choose what to see; control over the filter would matter. This is a broader possibility I want to explore separately, rather than a capability shown by this research system.

The authors warn that automated reviews may lower review quality or introduce biases and propose marking papers and reviews that are substantially AI-generated. I support disclosure for AI-generated content more broadly, perhaps through metadata or another identifying method, so that both people and tools can recognize it.

The researchers also discuss the potential for unethical or unintended harmful research if future systems are connected to powerful tools, including automated laboratories or software environments. These are possible risks to consider, not demonstrations that this prototype created such harms.

The project tested proprietary models such as GPT-4 and Claude as well as open models including DeepSeek and Llama 3. The authors report that a proprietary model produced the strongest papers in their tests, while expecting the field to keep changing. That comparison is specific to the models and setup they evaluated.

Sakana imagines an AI-driven research ecosystem but expects the human scientist's role to change rather than disappear. I broadly agree that roles may adapt, although I am uncertain what fully automated science would mean for researchers or programmers. My conclusion is that reliable autonomous research could be important for AGI; the timeline and the claimed intelligence explosion remain speculative.