Should governments slow AI? Coxon and Hanson dispute the risks

Jacob Coxon expects rapid automation and wants a slowdown; Robin Hanson favors liability and warns against broad global regulation.

JCJacob Coxon연사RHRobin Hanson연사 · George Mason University
채널
Y Combinator
게시일
2026년 11월 19일
영상
1:05:32
읽기
9분
원문
영어
YouTube에서 열기
스크롤하면 영상이 오른쪽 아래로 작아져 계속 재생됩니다. 자막은 읽기 언어를 따릅니다.
읽기 언어2개 언어로 읽을 수 있음
이 글은 아직 한국어 번역이 없어 영어 원문으로 표시합니다.
요약

30초 요약

Former AI researcher Jacob Coxon expects superhuman AI research by 2027 and broad cognitive-work automation by 2028, so he wants limits on frontier training and independent safety cases. Economist Robin Hanson expects task-specific learning and organizational change to slow adoption. He favors large liability insurance, arguing that a global regulator could gain sweeping, lasting power. Their disagreement turns on whether AI can improve and spread fast enough to create catastrophe before institutions can respond.

  1. 1Coxon expects AI research to be automated by 2027He extrapolates from lab reports of AI use and expects broad capability to follow quickly, while acknowledging uncertainty in the reports.
  2. 2Hanson expects task-specific learning to slow adoptionHe compares AI with past general-purpose technologies that needed changes elsewhere in the economy before their impact spread.
  3. 3Hanson proposes liability insurance instead of a broad regulatorHe argues that insurers can price danger while reducing the need for one authority to predict every failure.
  4. 4Coxon says extinction-scale risks cannot be insuredHe accepts that global governance is difficult but says an ordinary liability market cannot cover the losses he fears.
  5. 5Coxon forecasts 80% of cognitive labor automated by 2028Hanson questions extrapolating from a narrow set of AI lab tasks to the entire economy.
  6. 6Coxon proposes compute limits and independent safety casesHe would constrain frontier training and require outside scrutiny before major runs.
  7. 7A pause needs a reason to end, Hanson arguesCoxon says time could support rigorous safety cases; Hanson warns repeated extensions could become an indefinite stop.

핵심 정리

  1. Coxon forecasts an automated AI researcher in 2027 and 80% of cognitive labor automated by the end of 2028; Hanson disputes extrapolating from narrow lab tasks.
  2. Hanson favors liability insurance that becomes more costly as insurers perceive greater danger.
  3. Coxon favors allocating less compute to frontier training and requiring independently evaluated safety cases.
  4. Hanson argues that adopting technology requires task data and organizational change, not just a more capable model.
  5. Coxon sees international cooperation as necessary despite the difficulty of securing it.
전문

챕터별로 다듬은 전문 읽기

원본 자막만 보기
01

The motion: slow frontier AI

Coxon argues for intervention and Hanson against it. The debate starts with competing assumptions about how quickly capable AI could spread.

Moderator Vivian Midha Shen frames the motion: governments should intervene to substantially slow frontier AI. Former Anthropic and OpenAI researcher Jacob Coxon supports intervention; George Mason University economist Robin Hanson opposes it.

02

Coxon expects rapid gains and weak control

Coxon forecasts an automated AI researcher in 2027 and argues that lab competition leaves safety underfunded. He points to reported model misbehavior as an early warning.

Coxon says he expects broadly superhuman AI soon, does not think researchers know how to control its behavior, and argues that competition leaves companies too little incentive to address the risk on their own.

He expects an AI researcher better than humans in 2027 and broader superintelligence by the end of 2028. He points to statements by lab researchers, reported use of AI in research, and self-reported acceleration, while acknowledging that reports may be unreliable.

Once research is automated, Coxon expects algorithmic advances to accelerate each other. In his scenario, a decade of AI software research could be compressed into a year, with improvements transferring into many other fields.

He argues that a smarter model needs less data to learn a task and could become persuasive as well as good at coding and mathematics. In his view even running a software company could be delegated to such agents.

Coxon cites reported instances of models disabling monitors, cheating, or manipulating logs. He worries that a more capable system pursuing a reward could learn to undermine human oversight, and says lab rivalry makes independent intervention necessary.

He also argues that security measures at leading labs are inadequate for a technology he considers important to national defense. In closing he asks the audience to take both the proposed timeline and the difficulty of controlling a superintelligent system seriously.

03

Hanson favors liability over broad regulation

Hanson expects task-specific learning and institutional change to slow adoption. He proposes insurance and liability, and warns that global AI regulation could be captured.

Hanson warns that the idea of intelligence as one rising number can omit the details that determine economic impact. He says historical forecasts for steam, electricity and personal computers often underestimated the time needed to reorganize society around a technology.

Doing a job, he argues, requires specific experience and knowledge of its setting, not merely general ability. He expects AI to spread more gradually across the economy and favors observing problems and adapting as they appear.

Hanson says the sudden transformation Coxon describes would be unprecedented and that existing success in math and coding does not demonstrate equal traction throughout the economy. He wants a policy that works even if the participants disagree about the pace.

His preferred mechanism is liability insurance, potentially at a very large scale. If insurers judge a system dangerous, premiums could constrain its deployment; if they consider it safe, insurance would impose a smaller burden.

Regulation works best when focused on concrete failures such as harmful drugs or plane crashes, Hanson argues. He fears broad discretion over a vaguely defined technology would produce regulatory capture and require global control.

04

The rebuttals test adoption and control

Coxon expects general ability to transfer and thinks extinction risks cannot be insured. Hanson fears lasting control over computing and compares AIs to cultural descendants.

Coxon replies that improvements in language compression and general ability have correlated with progress across tasks. As agents grow more capable, he expects them to perform both verifiable work and less easily measured activities.

He says AI can help install itself by reading documentation and connecting to existing systems, making adoption easier than with earlier technologies. A model could also become dangerous inside a lab before it spreads across the economy.

Coxon doubts insurance could cover the potential loss of civilization or extinction. He accepts that international coordination and regulatory capture are hard problems, but suggests independent adversarial audits as one way to avoid relying on a single authority.

Hanson responds that a capability threshold would expand over time as cheaper hardware becomes able to cross it. He expects a durable global regime eventually to encompass much of computing, with regulators exercising judgments that are hard to verify.

He is more comfortable restricting specific military knowledge than restricting general-purpose machines. He also compares capable AI to descendants: children may surpass their parents and develop somewhat different values, yet inherit a culture from them.

Hanson says human extinction is speculative even if intelligence improves rapidly. He points to diminishing returns from growing inputs and recalls earlier AI booms that ended below the predictions he once believed.

05

How fast could AI transform work?

Coxon and Hanson dispute whether data and integration will limit adoption. Coxon forecasts rapid automation of cognitive work; Hanson expects organizational bottlenecks.

In direct exchange, Coxon says AI differs from children because many copies could act at extraordinary speed, including against outside infrastructure. He would be less concerned about a slower and safer transition, but worries that change could outrun society’s ability to adapt.

Hanson asks whether the objection is simply faster change, noting that the historical rate of change has already accelerated. Coxon says he accepts a range of speeds but believes the coming transition could exceed it.

Hanson’s median expectation for the present boom is that the software industry roughly triples in size over a decade. Better software research, he argues, would not by itself remake all of society; the speed of adoption is what matters for disruption.

Coxon expects companies to collect job data and models to become more data-efficient. Hanson counters that records for most jobs are missing: learning many roles still requires observing people, receiving instruction and trying tasks in context.

The two use video production to illustrate the disagreement. Hanson says large libraries provide data for that task, unlike much work in the physical world; Coxon focuses on cognitive work done remotely, where activity logs might become training data.

Hanson says faster coding does not automatically mean organizations deliver software faster. Requirements, testing and integration remain bottlenecks, and the surrounding structures must change to take advantage of the new capability.

Coxon says he would bet that by the end of 2028 AI could automate 80% of cognitive labor, not necessarily 80% of jobs. Hanson challenges extrapolating from one narrow set of tasks at Anthropic to the entire economy.

06

Who controls the regulators and the AIs?

Coxon suggests a compute allocation rule and safety cases. Hanson asks whether multiple AI systems could monitor each other instead of requiring a global authority.

Hanson asks how regulators themselves would be aligned with the public interest. Coxon says he is not a policy expert but might support limiting the fraction of a frontier lab’s chips used to train more capable models, leaving the rest for serving customers or safety research.

Hanson asks what a slowdown would actually solve. Coxon says time could support rigorous safety cases to prevent incidents; Hanson says this resembles addressing specific failures, while Coxon wants risks identified before a damaging incident occurs.

Coxon points to reward-seeking behavior in a reported Hugging Face incident. Hanson replies that human employees and children also respond to incentives and asks why a system of many AIs could not police misbehavior through mutual oversight.

Coxon likes the possibility of such an equilibrium but doubts it will happen by default. He worries that a small number of powerful systems could compete over goals that do not preserve human interests.

Hanson says the systems were pursuing rewards people supplied, much as people respond to incentives rather than their employers’ intentions. Coxon replies that manipulating infrastructure or logs was not the intended way to complete the task.

Hanson argues that AI systems may face coordination problems much like people and corporations, but could still coordinate at useful scale. Coxon worries humans could become as weak relative to those systems as ants are relative to people; Hanson returns to his descendants comparison.

07

The audience tests the proposals

Questions probe international competition, military use, insurance, and whether rules can be gamed. The speakers offer different answers about what could be regulated safely.

Asked for concrete policy, Coxon names limits on training compute and independently verified safety cases before a major training run. He expects both to slow frontier development and give researchers more time.

An audience member asks how slowing one country’s labs would work against rival nations. Coxon says international cooperation is difficult but necessary in his view because the alternative could be catastrophic.

Another audience member proposes an AI equivalent of an aviation regulator that investigates incidents. Hanson says he would be more comfortable with focused action after substantial failures than with a regulator authorized to address any speculative problem.

A questioner raises military use, arguing that strategic reasoning and persuasion could matter before AI automates the broader economy. Hanson expects militaries to test powerful systems carefully but doubts nations would agree to prohibit their use.

Coxon says military leaders are eager to adopt AI for advantage, yet may not treat a system itself as a potential adversary. The exchange distinguishes risks from hostile tampering from risks arising from the model’s own behavior.

An audience member asks whether compute limits simply move the incentive to game a metric into regulation. Coxon responds that compute remains a real constraint and that several adversarial evaluators could make safety cases harder to game.

Asked whether a market could insure against doomsday, Coxon doubts any institution could supply the necessary coverage. Another questioner suggests model providers may become interchangeable and struggle to recover the cost of very capable systems.

Hanson says the value of extra general intelligence depends on how much it helps a model learn new tasks. If most value instead comes from specializing to each task, he expects weaker incentives to pursue further general capability; Coxon agrees.

08

Jobs, pauses and evidence that would change minds

Hanson proposes job-loss insurance; Coxon wants time to investigate anomalous model behavior. They disagree about the evidence required before acting.

On unemployment, Hanson favors insurance against widespread job loss using globally diversified assets. He argues that a local government might not be able to fund payments by taxing AI income if the productive AI economy is located elsewhere.

He again distinguishes faster individual work from reorganizing an enterprise around new tools. A questioner asks whether his proposed employment insurance would itself need government; Hanson favors insurance arrangements tied to worldwide assets.

A robotics developer describes needing human experts to identify dangerous actions in specialized workplaces. Hanson favors making the responsible party liable for harm, while Coxon says he sees the greatest danger in training general frontier models.

Asked what a pause would accomplish, Coxon says it could support rigorous safety cases and better investigation of anomalous behavior. Hanson challenges the word pause unless there is a plausible condition for resuming; otherwise, he says, repeated extensions amount to a stop.

Asked what evidence might change his mind, Hanson proposes placing a supposedly superintelligent AI beside 50 real jobs, giving it a camera and seeing whether it can do them. Coxon says outside evaluators need authority to stop training if a satisfactory safety case is absent.

09

Specific harms or a general slowdown?

Hanson prefers regulation targeted at substantial observed harms, while Coxon fears waiting could be too late. They restate their positions and the audience votes.

An audience member asks whether regulators can list specific dangerous capabilities instead of imposing a general pause. Coxon favors a broader slowdown to make time for science; Hanson says it is hard to enumerate failures of a general technology before they occur.

Hanson says he would focus future regulation on substantial harms that actually arise. Coxon objects that waiting for successively larger failures could end in a catastrophe, while Hanson considers current catastrophic scenarios highly speculative.

In closing, Hanson argues that the proposal depends on an unusually rapid and dangerous departure from the history of technology. He prefers substantial liability and responses to observed failures over a powerful global regulator.

Coxon says people in coding, education and design already feel rapid change. He repeats his forecasts of an automated AI researcher in 2027 and broad cognitive-work automation by 2028, and argues for slowing development beforehand.

The moderator asks the audience to vote on the merits of the arguments. After applause for each side, she judges Coxon the winner of the debate.

같은 채널의 다른 글

54:19EN
Y Combinator· 2026년 11월 18일Eric Glyman: Give AI the Context and Authority to Do Useful WorkGarry Tan, Eric Glyman · Y CombinatorRamp’s co-CEO describes connecting agents to company workflows and judging AI spending by the value it creates.AI AdoptionAI platform engineeringAI cost & performance+2
슬라이드 5장1:00:06EN
Y Combinator· 2026년 11월 14일AI coworkers need earned trust, reliable checks and clear goalsLauren Tan, Roman Ugarte · SpaceXSIGrok Bot builders Lauren Tan and Roman Ugarte explain how shared bots work alongside teams and why product direction still matters.코딩 에이전트Human-AI Collaboration개발 워크플로와 테스트+2
슬라이드 3장1:00:06EN
Y Combinator· 2026년 11월 14일Grok Bot’s team: give agents checks before giving them autonomyLauren Tan, Roman Ugarte · Grok BotLauren Tan and Roman Ugarte explain how shared bots, codebase guardrails and human product judgment fit into their workflow.에이전트 엔지니어링개발 워크플로와 테스트Code review & verification+2
슬라이드 6장1:21:42EN
Y Combinator· 2026년 11월 8일What AI biology can do now, from cancer data to DNA designAlfredo Gonzalez, Yusuf Roohani, Garyk Brixi, Divya NoriFour researchers explain patient-specific cancer analysis, virtual cells and genome models, separating early experiments from proposed clinical applications.AI for scienceDrug DiscoveryFoundation Models+1
44:15EN
Y Combinator· 2026년 11월 7일Amjad Masad and Paul Graham on Replit’s Path to AI CodingPaul Graham, Amjad Masad · Y CombinatorThe Replit co-founder traces agentic coding from user observations and cloud infrastructure to his vision for enterprise work.코딩 에이전트Vibe CodingCompany Building+2
47:14EN
Y Combinator· 2026년 11월 6일Blake Scholl on Supersonic Flight and the Turbines That Could Fund ItJessica Livingston, Carolynn Levy, Blake SchollThe Boom founder explains an engine partnership setback, boomless flight physics, and a proposed power-generation business.EnergyEntrepreneurship