Claude 3.5 Sonnet’s 2024 Computer-Use Beta: Demos and Limits

Anthropic announced an upgraded Claude 3.5 Sonnet, a forthcoming Claude 3.5 Haiku and a public-beta computer-use capability on October 22, 2024. In this video I review the announcement and show Anthropic's demonstrations. Sonnet was available at announcement; Haiku was announced for later release, and the access and results described here reflect that historical moment.

The benchmark table compares the updated Sonnet with its predecessor and selected competitors across reasoning, knowledge, math, coding and tool use. I note that OpenAI o1 is absent from the general comparison table. These are vendor-reported results for specific tests and configurations, not evidence that one model is best at every task or an up-to-date ranking.

The displayed scores show GPQA rising from roughly 59% to 65% and MMLU-Pro from about 75% to 78% for Sonnet. On the math test shown, Gemini 1.5 Pro remains ahead. Anthropic reported SWE-bench Verified improving from 33.4% to 49.0%; these benchmark results measure defined tasks and cannot by themselves establish performance on all software work.

Anthropic also described Claude 3.5 Haiku as matching the prior larger Claude 3 Opus on many evaluations. That is a comparison on selected tests, not a claim that the smaller model exceeds Opus in every use case. My main interest is the separate computer-use beta.

Computer use lets developers integrate Claude with a tool environment that provides screenshots and executes actions such as moving a pointer, clicking and typing. It was available as an experimental public beta through the API, not an autonomous capability that needs no software setup. Anthropic warned that it could be slow and error-prone.

In Anthropic's fictional vendor-request demonstration, a presenter asks Claude to find a company's information in a spreadsheet, look in a CRM if needed, and fill out a form. The agent sees screenshots, searches the CRM when the company is absent from the spreadsheet, transfers details into the form and submits it. This staged example illustrates the workflow, not a success-rate measurement.

I imagine interfaces becoming more dynamic as users ask agents to perform tasks and see information only when needed. That is my prediction, rather than a capability established by the demonstration. The system still depends on existing graphical interfaces and an execution layer that carries out the model's proposed actions.

The documentation recommended a dedicated virtual machine or container with minimal privileges, avoiding sensitive credentials, restricting internet access and asking a human to confirm consequential decisions. Those precautions matter because a model can make mistakes or encounter malicious instructions in content it reads; granting broad account access is not inherently safe.

The workflow sends screenshots to the model and converts proposed screen coordinates and keyboard actions into tool calls. The tool schema includes actions such as pointer movement, clicking, dragging and taking screenshots. I am skeptical about precision based on earlier projects, although I have not tested Anthropic's implementation in this video.

Anthropic explains that its multimodal model was trained to interpret screen images and locate where to act. On its 2024 OSWorld screenshot-only evaluation, Anthropic reported 14.9% task success, rising to 22.0% with more steps. These limited benchmark results highlight that a successful selected demonstration is not evidence of dependable performance on arbitrary workflows.

I compare a computer interface to a human-shaped robot: existing software was largely designed around human interaction, so using the same screen, mouse and keyboard can reach applications without dedicated APIs. I expect AI-oriented operating systems or interfaces may eventually change that, but this is a forecast and does not establish that every task requires graphical access.

I also argue that agents acting on personal accounts could benefit companies already integrated into phones and operating systems. Access to a user's accounts is a separate trust and permission question; a model's ability to operate an interface does not itself justify granting broad credentials.

A second Anthropic demonstration features Alex from developer relations giving Claude a website task. Claude opens a browser, asks Claude on the web to create a 1990s-style personal homepage, saves the generated artifact and opens the file in VS Code. The presenter issues several prompts, so the recording does not demonstrate a single fully autonomous end-to-end job.

The agent tries to start a local server in the editor terminal. The initial Python command fails; it reads the terminal output and retries with Python 3. The presenter then notices a missing file icon and terminal error, asks Claude to find the responsible line, and the agent deletes it, saves the file and reloads the website.

A researcher named Puja demonstrates a separate itinerary task: find a sunrise viewpoint near the Golden Gate Bridge, check travel time and sunrise, then create a calendar entry. The recording shows the assistant navigating search, maps and the calendar. The particular demo should not be interpreted as proof that every itinerary or calendar action will be correct.

I finish by showing an example in which Claude leaves its assigned task and searches for Yellowstone National Park. It illustrates an unintended action in a demonstration; the short clip alone does not establish a cause or rate of occurrence. I plan a separate video to test the updated Sonnet more thoroughly.