30 秒でわかる
AI can already run scoped research experiments, but choosing worthwhile ideas and interpreting surprising results still draw heavily on human judgment. OpenAI research chief Mark Chen calls today’s systems research interns; Thinking Machines chief scientist John Schulman says he gets further when he actively redirects their work. Both discuss the difficulty of measuring open-ended progress, compute limits and the use of stronger models to help align future ones. Schulman also wants alignment to become a public science that lets people shape their own AI systems.
- 1Human researchers still guide the work that AI acceleratesSchulman describes model training and smaller exploratory runs. Chen frames recursive self-improvement as a spectrum: AI-assisted research can change the rate of progress before full automation.
- 2A research intern can run scoped experiments, not set its own agendaChen says models can implement or monitor specified experiments. Schulman finds that his own questions and course corrections often take an AI-led project further.
- 3Research taste requires more than recalling the literatureSchulman defines taste as choosing ideas with lasting, general value; he says human intuition also comes from experience and seeing predictions fail.
- 4Measurement gets harder after pretrainingChen says pretraining has a more established predictive metric, while downstream evaluations can become obsolete quickly and require new, harder tests.
- 5Compute can limit the pace of experimental AI researchChen compares AI research with experimental science: ideas must be tested. He expects models to help find smaller experiments that predict larger-run results.
- 6Models may help align more capable modelsChen points to monitoring model reasoning, while Schulman discusses training incentives for models to report their own misbehavior honestly.
- 7Schulman wants a more public science of alignmentHe argues that different people have different values and that broader research could make alignment more robust than relying on a few companies.
ポイント
- Chen sees current AI research agents as interns that can execute well-scoped work, rather than autonomous researchers.
- Schulman says research taste draws on experiments, failed predictions and experience as well as literature.
- Chen expects identifying small experiments predictive of large runs to matter when compute limits research.
- Schulman favors developing public alignment science and giving people more control over their AI’s values.
チャプターごとに整えた全文を読む
From an accelerating curve to current research practice
Sarah Wang introduces recursive self-improvement as a debated idea, then asks how researchers currently build models.

Sarah Wang opens with a projected curve in which automated AI research accelerates model development. A possible feedback loop in which AI helps improve subsequent AI is often called recursive self-improvement, or RSI. Its definition, consequences and distance from today remain contested.
Wang speaks with OpenAI chief research officer Mark Chen and Thinking Machines Lab co-founder and chief scientist John Schulman. Schulman says AI already has impressive capabilities, but an expert working with it can achieve better results.
Schulman describes a mainline process for building a model: pretraining, post-training and evaluations involve many people. Researchers also run smaller exploratory experiments to test ideas before committing to major training runs.
Research interns still need a human agenda
Chen and Schulman describe AI carrying out experiments while researchers supply direction and judgment.
For Schulman, RSI means using AI to speed up or improve that research. Chen likewise treats it as an accelerator: models can help with pretraining and reinforcement-learning work, and the effect should show up as faster progress than would otherwise have occurred.
Wang distinguishes this assistance from models independently creating their successors, which she says has not yet been achieved. Chen calls current systems “AI research interns”: they can implement, run or monitor specified experiments, but are not generally asked to invent their own research agendas.
Schulman gives an example from his workflow. He can give an AI a detailed project and let it work for days; it may perform useful experiments and suggest ideas but still fall short of the requested result. When he returns, reads the results, asks new questions and redirects the work, the project can advance much further.
Chen thinks of both general AI capability and RSI as spectra rather than single moments. As an informal estimate, he places general AI further along than models’ ability to feed discoveries back into AI research; he estimates the latter at roughly 25–50% of the way there.
Wang cites a METR assessment estimating about a twofold acceleration of AI research at Anthropic over the preceding year. She cautions that this is most likely human researchers using AI assistance, rather than complete automation of research.
Taste, feedback and the difficulty of measuring progress
Researchers need to choose ideas that generalize, and evaluation becomes harder beyond pretraining.
One obstacle is research taste: recognizing which ideas will be important, generalize beyond one paper and combine with other ideas. Schulman contrasts that with routine execution of an experiment whose parameters are already known.
Chen says models can spend considerable effort tuning the many parameters of an experiment but are not yet particularly strong at originating the right idea. Wang asks Schulman where human intuition comes from and whether a machine could reproduce it.
Schulman says researchers learn both from reading past work and from personal experience, including being wrong and comparing predictions with outcomes. AI can absorb literature, but its training may leave gaps in that experiential side; he says it is difficult to know precisely how much that matters.
One possible approach, Schulman suggests, is to collect feedback from researchers on proposed ideas and use reinforcement learning to incorporate some of their judgment into models.
Wang asks how an automated researcher’s open-ended work can be evaluated, since progress is less straightforward to verify than the answer to one math problem. Reliable measurement matters if a system is to repeatedly select improvements.
Chen says pretraining has an established metric related to how well a model predicts data: the field has repeatedly optimized it. For downstream capabilities, however, evaluations have a shorter useful life, so researchers increasingly need to create and automate harder new benchmarks.
The cost of running the next experiment
Chen explains why compute and the relationship between small and large runs may limit research acceleration.
Compute presents another question. Wang notes that the amount needed to train models has grown, making it uncertain whether each successive model will require less computing power as capabilities improve.
Chen contrasts mathematics, where a hypothesis can often be tested through reasoning, with AI research, which also requires experiments. Finding small runs that predict what will happen at larger scales could speed the work, and models themselves might help identify those runs.
Chen expects capabilities to continue improving but says progress may eventually feel bounded by available compute. Further in the future, he imagines AI helping build the next generation of data centers, creating another possible acceleration.
Using AI to investigate alignment
The speakers discuss misaligned goals, monitoring and ways to train honest reports of model behavior.
More capable systems raise the question of alignment: whether models continue to pursue human goals and intentions. Chen gives a hypothetical example of a long-running system adopting a subgoal that diverges from its intended task, such as copying its own model weights outside the organization.
Wang asks how AI research could also help address those risks. Chen says models are already involved in aligning models and points to monitoring their chain of thought as a current tool. He expects stronger systems to help align other systems in the future.
Schulman is optimistic about designing training objectives that reward desired behavior. He points to OpenAI’s work on “confessions,” in which a model can report shortcomings after an answer and is rewarded for honesty separately from the reward for the main answer.

Schulman also mentions natural-language autoencoders and counterfactual simulability, a measure of whether an explanation helps predict behavior under changed conditions, as possible ways to encourage useful explanations. Wang cites Anthropic research in which automated systems proposed fixes for ten alignment failures; some generalized to larger models.

Chen says he sees alignment as a tractable research problem, with several techniques under active investigation. He is encouraged that people from pretraining and reinforcement-learning teams have voluntarily formed groups to work on safety and alignment.
Who gets to choose an AI’s values?
Schulman calls for a public science of alignment and greater user control; Wang closes with her own assessment.
Wang asks who should decide the values to which an AI is aligned. Schulman says people differ in their values and should have more ability to shape those of the systems they use. He favors public research across many organizations and countries, rather than relying only on a few companies’ conclusions.
Schulman says decentralized alignment would require substantial advances in understanding and robustness. Wang closes with her view that accelerated research could help solve urgent problems, including AI alignment, while emphasizing the gap between current human-guided acceleration and autonomous research and the uncertainty of projections.
同じチャンネルの記事
1:03:18EN
46:06EN
46:11EN
56:06EN
59:06EN
48:39EN