Can AI Help Humans Align More Powerful AI?

RESUMEN

En 30 segundos

In July 2023, Matthew Berman reviews OpenAI’s newly announced Superalignment team and its proposal to use AI to help oversee and research the alignment of more capable systems. He explains reinforcement learning from human feedback, scalable oversight, adversarial testing and DeepMind’s work on automated red teaming. Berman welcomes the investment but questions whether automated overseers can themselves be trusted and who gets to define human values. The video’s timelines and extreme-risk scenarios are predictions and concerns, not established outcomes.

1The 2023 Superalignment planOpenAI announced a team co-led by Ilya Sutskever and Jan Leike and proposed dedicating 20% of the compute secured to date over four years.
2Limits of human feedbackBerman explains how human preferences can train models and why the same approach may be difficult to scale to systems beyond human expertise.
3Who evaluates the evaluator?AI-assisted oversight may amplify biases or miss its own failures, and selecting the values to align with is a social as well as technical question.
4A proposed research pipelineOpenAI described scalable training, validation and adversarial testing, including deliberately training misaligned models to test detection.
5Automated red teamingDeepMind described using language models to generate inputs that expose harmful behavior in a target model.
6Transparency and uncertaintyBerman welcomes safety research but questions how outsiders can assess techniques when powerful models and development details are closed.

Conclusiones

OpenAI’s 2023 plan proposed AI-assisted evaluation and automated alignment research to supplement human oversight.
Human feedback can guide current models, but evaluating systems much more capable than their supervisors poses a challenge.
The proposed pipeline included scalable oversight, model validation and adversarial testing, not a demonstrated solution.
Berman asks how to detect failures in AI evaluators and whose values alignment should represent.
OpenAI’s four-year goal and expectation that superintelligence could arrive in the 2020s were forecasts, not guarantees.

TemasAI Alignment AI Safety Validación con humanos en el bucle Red Teaming AGI

TEXTO COMPLETO

Lee el texto editado por capítulos

Ver transcripción

Solo la transcripción

The Superalignment announcement

Berman introduces OpenAI’s July 2023 proposal and explains alignment and human-feedback training. The arrival of superintelligence is presented as a forecast.

In July 2023 OpenAI suggested that superintelligence could arrive in the 2020s and announced a Superalignment team to study how humans might steer systems more capable than themselves. This is a forecast, not an established timetable. Alignment here means trying to make AI systems follow human intent and values.

The new team was to be co-led by Ilya Sutskever and Jan Leike. OpenAI said it would dedicate 20% of the compute it had secured to date over the next four years and recruit researchers and engineers. The company warned that powerful future systems could benefit humanity but might also pose severe risks, including loss of human control.

One existing technique is reinforcement learning from human feedback, or RLHF. For example, human evaluators can compare two responses and indicate which they prefer, providing information used to train a model. But OpenAI argues that humans may struggle to reliably supervise systems much smarter than themselves.

Present models can still fail to follow instructions, give inaccurate or biased responses, or fail to refuse harmful requests. Berman expects the challenge of oversight to grow as systems become more capable; that is a concern about possible future systems, not proof that a particular failure will occur.

The proposed response is to develop automated systems that help conduct alignment research. OpenAI’s 2022 approach had three pillars: training with human feedback, training AI to assist human evaluation, and training AI to help do alignment research. Human researchers would increasingly review AI-generated work rather than produce all of it themselves.

Using AI to help with alignment

He reviews OpenAI’s earlier three-part research approach. He asks whether AI evaluators could mislead humans or amplify bias and who chooses the values involved.

DIAPOSITIVA 1 / 3OpenAI’s 2022 alignment research pillarsThe earlier approach combines human feedback, AI-assisted evaluation and AI-assisted alignment research.

That raises a difficult question for me: if AI monitors AI, how can we tell whether the evaluator is itself misleading us or missing a problem? Another question is whose values should guide the system. OpenAI’s earlier paper acknowledges this sociotechnical issue without claiming to resolve it.

AI assistance could amplify subtle biases, inconsistencies or vulnerabilities. Future alignment problems may differ from those observed in today’s systems, and systems capable enough to help with research might themselves be risky if they are not properly aligned. The question remains: who aligns the aligners?

Oversight, testing, and self-critique

The new plan proposes scalable oversight, validation and adversarial testing. Berman discusses self-critique and the uncertainty of the four-year research goal.

DIAPOSITIVA 2 / 3The proposed Superalignment pipelineOpenAI outlines scalable oversight, model validation and adversarial testing of the pipeline.

The 2023 plan proposed a scalable training signal for tasks people find difficult to evaluate, using AI to help assess other AI and studying how oversight generalizes. It also proposed validating systems by searching for problematic behavior and internals, then stress-testing the pipeline with deliberately misaligned models. These were research directions, not a completed solution.

A related idea is self-critique. A model might write code to print numbers from one to 100, then receive a second prompt asking it to look for bugs or improvements. Such feedback can be useful on some programming tasks, but that example does not demonstrate that self-critique solves alignment.

OpenAI described an ambitious four-year goal and explicitly acknowledged that success was not guaranteed. Berman illustrates the scale of the computing commitment with a hypothetical based on Microsoft’s investment, but the 20% commitment is a share of secured compute, not a verified $2 billion budget.

Red teaming and transparency

Berman explains DeepMind’s automated red-teaming example and discusses OpenAI’s commitment to sharing alignment research. He closes with concern about uncertain future capabilities.

DeepMind had also published work on automated red teaming: one language model generates prompts intended to elicit harmful outputs from a target model. Examples include offensive language, leakage of private information and uneven treatment of groups. Such tests are one way to find problems, not a complete safety guarantee.

OpenAI said it wanted to share alignment research when safe and be transparent about how well its techniques work. I welcome the focus and resources, but I question how outsiders can evaluate the tools and methods if the systems themselves remain closed. Sharing research and releasing model weights are distinct choices.

I also worry about a hypothetical future superintelligence that humans might not recognize or understand, like fish unaware of the world beyond their tank. That is a speculative risk, not evidence that such a system exists. The uncertainty makes this period both exciting and unsettling.