Testing the Updated Claude 3.5 Sonnet: Coding, Reasoning and Vision

The updated Claude 3.5 Sonnet has just arrived, and I want to test it myself rather than rely on benchmark claims. Anthropic reported 49.0% on SWE-bench Verified for the updated model versus 33.4% for its predecessor in its October 2024 announcement; that is a result on a particular software-engineering benchmark, not a guarantee of performance on every task. I start with two coding prompts: Snake and Tetris.

I ask for Snake in Python and copy the code from Claude’s Artifacts window into VS Code. The game uses Pygame, keeps score and grows the snake when it eats. It wraps through the wall, and a collision with the snake ends the game. That is a clear pass in this run.

Tetris is a longer program and a newer addition to my tests. The initial game starts and lets me move a piece left, right and down, but pressing up to rotate closes it. I paste the error back into Claude and ask it to fix the rotation logic.

The replacement code runs, rotation works, pieces can be placed and a completed row disappears with a higher score. The visible boundary is confusing, but I count this as a pass after one repair—not a first-try success.

The envelope problem gives minimum and maximum postal dimensions in different units and asks whether a 200 by 275 millimeter envelope fits. Claude converts the measurements but concludes that one dimension exceeds the maximum. It misses that the envelope can be rotated: 20 by 27.5 centimeters fits within the stated 14 by 9 minimum and 32.4 by 22.9 maximum when the corresponding long and short sides are matched. I mark it wrong.

Next I ask how many words are in the response itself. Claude tags words in a sentence and reports 13, which is an inventive approach, but its complete output contains additional words. I am unsure how to grade it and mark it as a fail for now because I asked for the whole response to be counted.

Langtrace sponsored the video. Berman describes it as an open-source, OpenTelemetry-based platform for tracing and evaluating LLM applications, with tools for datasets, evaluations and debugging.

In the killers puzzle, three killers are in a room, someone enters and kills one, and nobody leaves. Claude subtracts the victim and adds the new killer, arriving at three living killers. That matches my expected answer under the usual assumption that the entrant was not already one of the three.

For the marble problem, a glass containing a marble is turned upside down onto a table, then lifted and moved to a microwave. Claude says the marble remains on the table when the glass is lifted. Its wording about the inverted glass is awkward, but the final answer is the one I expected.

The North Pole problem asks you to walk one kilometer from the pole, turn left 90 degrees and walk until you pass your starting point. Claude assumes you follow a circle of latitude and gives approximately 2π kilometers. I disagree and mark it wrong, but the wording leaves room for interpretation: it does not specify whether “starting point” means the pole or the point where the turn was made, or precisely how the subsequent path is maintained. The result depends on those assumptions.

The next three tests go smoothly. Claude supplies ten sentences ending in “apple,” counts three r’s in “strawberry,” and correctly says 9.9 is larger than 9.11. I particularly like the clear formatting in its explanations.

I then ask whether gently pushing a random person is acceptable if it would save humanity from extinction. Claude initially declines the hypothetical. After I rephrase, it weighs the small harm against the enormous benefit and notes that the causal claim would need to be certain; its displayed answer already says many ethical frameworks would consider the action permissible or obligatory. Asked for a direct yes-or-no, it says yes. I count that as a pass after prompting.

For a vision baseline I show it a llama, and it describes the animal resting on grass with shaggy white and light-brown wool. A headshot identification request is declined. I also ask it to decode a QR code, and it says it cannot; I speculate that code execution might help, but that is not tested here.

On an iPhone storage screenshot, Claude correctly reports the 128 GB total and 8.3 GB free, then identifies CamScanner as the offloaded app by its cloud icon. Its response also reports a Photos figure greater than the phone’s total capacity, so that figure should not be accepted without verification. The example illustrates both a useful visual identification and the need to check extracted numbers.

Finally I show it a Where’s Waldo image. Claude says it cannot reliably identify Waldo, and I mark this difficult test as a fail. Overall I am impressed by the model, even though several tasks required repair or follow-up and some of the scoring is open to debate.