Pixtral 12B: Testing a 2024 Vision-Language Model

In this September 2024 video I try Mistral AI’s Pixtral 12B, a vision-language model, using a cloud GPU hosted by Vultr, the video’s sponsor. I review the release and run a set of informal text and image prompts; the sponsorship is relevant context for the hosting demonstration.

Mistral describes Pixtral as a 12-billion-parameter multimodal model released under Apache 2.0, supporting images of varying sizes and multiple images in a long context. I review company-provided benchmark comparisons with contemporary models; those figures are limited to the reported tests and settings.

I run the model on a 48 GB Nvidia L40 through an API-compatible backend with Open WebUI. This is a cloud-hosted test rather than a measurement of performance on an ordinary local device.

A text-only request for a Python Tetris game produces code that fails with an attribute error. The model also miscounts the letters in “strawberry.” Two isolated failures are not a complete assessment of its coding or reasoning ability.

For image description, the model describes a llama resting in a field. It also identifies a public figure in another picture, according to my test; such a result is a single example, not a guarantee that visual identification is reliable.

The video includes a CAPTCHA-reading test and I report an apparent success. That individual example does not show that CAPTCHAs as a category are defeated or that a model will perform similarly across challenges.

Using a phone-storage screenshot, I ask about capacity and used space, and the model reads the displayed totals. I then ask which app is not downloaded; it answers incorrectly, failing to interpret the cloud icon in the way I expected.

The model lists applications and storage amounts from the screenshot, then describes a meme contrasting workers at startups and large companies. I find its explanation of the visual joke plausible; this remains a small, subjective sample.

I speculate that people may increasingly route work to specialized models: vision to one, reasoning or coding to another, depending on capability, cost and latency. This is my forecast, not a result demonstrated by a single model test.

When asked what URL a QR code resolves to, the model says it cannot determine the destination from the image. In another test, it converts a screenshot of a table to CSV, which I judge largely correct.

I ask Pixtral to generate HTML from a hand-drawn interface. The resulting simple page includes a dropdown and button and appears to work in my brief test, although this does not establish production-ready code.

A more complicated website screenshot leads to a weaker reproduction: the generated page captures some text but lacks images and arranges items vertically rather than in the original layout. I regard that attempt as only partially successful.

For a crowded Where’s Waldo image, the first answer explains the game rather than locating the character. After follow-up prompts, the model proposes approximate coordinates; I believe they point to Waldo, though a single low-resolution example is not a robust localization benchmark.

I conclude that Pixtral’s visual capabilities are promising while noting the text and interface-generation failures. The results are anecdotal and specific to the 2024 model version, prompts and deployment shown in the video.