Testing Llama 3.2 Vision: Refusals and Image Tasks

ZUSAMMENFASSUNG

In 30 Sekunden

Matthew Berman tests image description, refusals, sketch-to-code, table extraction and visual localization. The small, hosted test is a snapshot rather than a general benchmark.

1Basic descriptionThe model describes a simple scene.
2RefusalsRefusals do not alone establish underlying visual capability.
3Sketch to codeA sketch-to-code request receives an apparently mismatched refusal.
4MemeThe meme explanation satisfies the presenter.
5Table and screenshotResults vary across QR, table and phone screenshot tasks.
6LocalizationThe presenter reports an incorrect location in a difficult image.

Das Wichtigste

The model describes a simple scene.
Refusals do not alone establish underlying visual capability.
A sketch-to-code request receives an apparently mismatched refusal.
The meme explanation satisfies the presenter.
Results vary across QR, table and phone screenshot tasks.
The presenter reports an incorrect location in a difficult image.

ThemenVision & Video Benchmarks Open-weight models

VOLLTEXT

Den bearbeiteten Text nach Kapiteln lesen

Transkript zeigen

Nur das Transkript

A small vision test

Descriptions, refusals and visual accuracy.

In this September 2024 video, Matthew Berman tries the 90B Llama 3.2 Vision model through a hosted interface. His small informal test reflects that model, provider and settings at the time, not all versions or current performance. The video includes a sponsor.

The model describes a simple image of a llama, which Berman counts as a successful basic description.

The model refuses requests to identify a person and solve a CAPTCHA. Berman treats these as failures for his test; refusals can reflect service or safety policies and do not by themselves establish whether the model can recognize the underlying visual content.

When asked to turn a rough interface sketch into code, the model refuses and gives an apparently unrelated safety explanation. Berman retries with a more specific prompt and still receives a refusal.

The model explains a workplace meme. Berman considers the description appropriate for the example, while acknowledging other tests in the same session went poorly.

A QR-code request does not produce the requested URL. A table-to-CSV conversion is judged successful by Berman, while a phone-storage screenshot yields a mixture of correct and incorrect answers.

In a difficult Where’s Waldo image, the model gives a location that Berman says does not match the answer he found. He concludes that this small set of tests was disappointing, while leaving open the possibility that configuration affected the result.