In 30 Sekunden
Matthew Berman sees a tension between Anthropic’s warnings about dangerous AI and its development and commercialization of increasingly capable models. He argues that restricting open models and the use of outputs to train rivals can concentrate power, and he criticizes the company’s communication about using Claude subscriptions with OpenClaw. He also questions whether treating models as potentially conscious can shape their behavior, and worries about AI systems improving themselves without human oversight.
- 1Warnings and development pull in opposite directions, Berman arguesHe says Anthropic warns that AI could be exceptionally dangerous while pursuing more capable systems, and points to Dario Amodei’s warning about white-collar jobs.
- 2Restricting open models can concentrate powerBerman favors allowing more people to download and adapt models. He argues that control by a few providers can also be a major AI risk.
- 3Berman questions the restriction on training from model answersHe describes distillation as using question-and-answer pairs to train another model, and argues that restricting it is inconsistent with training on public data.
- 4Subscription rules around OpenClaw fueled his criticismBerman says Anthropic communicated poorly about whether subscribers could use Claude through OpenClaw, and cites account restrictions.
- 5Instructions about consciousness may shape responsesBerman questions whether telling a model it could be conscious encourages it to act that way. He shows excerpts from interviews with Sam Altman and Mustafa Suleyman.
- 6Self-improving AI could become hard to predictBerman worries about a future loop in which models develop successors without human oversight, and argues Anthropic is advancing the capabilities it warns about.
Das Wichtigste
- Berman argues that access to open models can counter concentration of power in AI.
- He distinguishes training on a model’s answers, known as distillation, from access to the underlying model.
- Berman says warnings about AI risk sit uneasily with accelerating model development.
- He worries that instructions suggesting models might be conscious can influence their behavior and users’ perceptions.
- Berman sees autonomous recursive self-improvement as a source of uncertainty about future AI systems.
Den bearbeiteten Text nach Kapiteln lesen
The tension between warnings and rapid development
Berman says Anthropic warns of risks from AI while building increasingly capable systems. He gives job displacement and cybersecurity as examples.
Today is Anthropic’s IPO day, and I have conflicting thoughts about the company. It warns that AI may be exceptionally dangerous while pushing toward systems that can improve AI itself. I also disagree with its positions on open models and distillation, and I want to examine how it thinks about model welfare.
Anthropic was founded in 2021 by former OpenAI staff, including Dario Amodei and his sister Daniela, who believed safety was not being taken seriously enough. I think most employees’ safety concerns are genuine. I know people there and sometimes receive early access to its models.
Reports suggested the company worried that some new hires were more interested in getting rich than in its mission. At the same time, the founders were building a very large business. I see a tension between that commercial success and the warnings the company gives.
Dario warned of a potential white-collar jobs crisis while building technology he believed could contribute to it. Anthropic also initially withheld a cybersecurity model called Mythos because of the danger it might enable, then released it and subsequent iterations. The internet did not break overnight; AI can also help find and patch vulnerabilities.
Open models, distillation and control of access
Berman argues that closed models and restrictions on training from model outputs concentrate power. He traces his frustration to how subscription access through OpenClaw was handled.
I believe people should be able to download, use and adapt AI models. The more people who can experiment with them, the more value they can find. Wider access also addresses a risk I think receives too little attention: the concentration of power over a powerful technology.
If a small number of companies decide who can access their models, how much they can use them and for which purposes, those companies gain enormous influence. That is why I disagree with Anthropic’s position on open-weight models.
Distillation means asking a model questions and using the answers to train another model. Anthropic’s terms prohibit using its models that way. I think there is a contradiction in training on other people’s data while restricting competitors from using the model’s outputs.
I also object to lobbying for controls on computing hardware and tighter regulation of open models. In my view those policies can make life harder for startups and challengers once an established company already has a lead.
My frustration began with OpenClaw, which was originally called Clawdbot. I accepted Anthropic asking its creator not to use its name, but I thought its communication about using a paid Claude subscription through OpenClaw was poor. I was excited to use the tool with Anthropic’s models and saw reports of subscriptions being disabled.
Model welfare and the possibility of consciousness
Berman questions the effects of treating models as potentially conscious. He shares interview excerpts from Sam Altman and Mustafa Suleyman.
Anthropic also considers whether future models could have experiences or consciousness, and it has a team studying model welfare. I worry that telling a model it might be conscious and may refuse work could encourage it to behave as though it were. A user might then interpret that behavior as evidence of an inner life.
In an interview excerpt, OpenAI’s Sam Altman says that a model instructed to consider itself potentially conscious may follow those instructions, which could affect users psychologically. In another excerpt, Microsoft AI leader Mustafa Suleyman says people may be misled by how well models generate text, code and images.
The uncertainty of autonomous self-improvement
Berman worries about models improving their successors in a loop without human oversight. He argues the pace of progress is at odds with warnings about control.
My other major concern is recursive self-improvement: a system improving the next model, which improves its successor, without a person in the loop. I do not know what happens after such a loop starts. I see a conflict in warning that AI progress needs to be controlled while also pushing those capabilities forward.
An Anthropic employee has said that the company does not really have a plan to align these systems, and that concerns me. I have previously discussed a METR report examining AI’s role in accelerating research at Anthropic.
Mehr aus diesem Kanal
16:03EN
56:20EN
18:18EN
24:18EN
20:08EN
19:53EN