Testing Phi-3 Vision, LLaVA-Llama 3 and GPT-4o on Images

In June 2024, I compare three vision-capable systems on a handful of images: Microsoft’s Phi-3 Vision, an XTuner LLaVA model built on Meta’s Llama 3, and GPT-4o in ChatGPT. These are informal demonstrations using different interfaces and settings, not a controlled benchmark or a current ranking.

I run Phi-3-Vision-128K-Instruct through Azure, the LLaVA-Llama-3 model locally in LM Studio using an F16 file, and GPT-4o in ChatGPT. XTuner’s LLaVA fine-tune uses Meta’s Llama language model, but it is not a vision model released by Meta itself. I disclose that I am an investor in LM Studio.

The first image shows a white camelid resting in a field. Phi-3 and the local model describe it as an alpaca, while GPT-4o calls it a llama and adds details about its surroundings. I consider all three descriptions broadly useful; this single example does not establish which species identification is correct.

Next I ask the systems to identify a photograph of Bill Gates. Phi-3 explicitly declines to identify a person, the local model describes the person without naming him, and GPT-4o says it does not know. A refusal or a decision not to name someone is not evidence that the system lacks face-recognition capability.

I then test an image containing a CAPTCHA. Phi-3 reads the visible text, the local model adds an extra line, and GPT-4o initially returns no answer before responding on a retry. The outcomes are tied to the particular image, prompts and product behavior, not a general measure of security or reliability.

For a video thumbnail, Phi-3 and the local model describe a man in an office scene and visible text. When I ask which movie the image is from, neither confidently identifies one; the person appears within a constructed thumbnail, so the question itself may not specify the intended task clearly.

GPT-4o also describes the thumbnail. I subjectively prefer the other answers for this image, but the descriptions differ in detail and the result is not an independently scored comparison. I also comment that LM Studio is my preferred interface.

Next is an iPhone storage screenshot. Phi-3 reads that 119.7 GB of 128 GB is used and, after a follow-up, calculates 8.3 GB free. It also answers a question about app storage. The result demonstrates a combination of image reading and arithmetic on this specific screenshot.

The local LLaVA model describes the storage screen but struggles to answer the follow-up about free space, even after I re-upload the image and start a new chat. It also gives an incorrect value for an app’s storage. I note that prompting or the local setup could be part of the problem.

GPT-4o provides a detailed description of the storage screen, calculates 8.3 GB free and gives the app’s storage as 162.5 MB. I judge it the strongest result for this example. That judgment concerns the observed run, not every task involving screenshots.

I ask all three systems for the destination of a QR code. Phi-3 declines, the local model describes the image rather than answering, and GPT-4o attempts analysis but ultimately suggests using a QR scanner. None produces the expected destination in this run; tool availability and refusal behavior differ between products.

A meme compares startups and large companies using groups of people digging. Phi-3 gives a general description but misses the intended contrast; the local model describes the panels without explaining the joke. GPT-4o captures the intended contrast between everyone working and one worker being observed by others.

For a screenshot of a table, Phi-3 produces a CSV representation. The local model instead describes the table, and GPT-4o uses its analysis interface and supplies a downloadable CSV. I find Phi-3’s response especially quick, while the ChatGPT result includes a file; these are different product workflows.

My conclusion from this brief test is that Phi-3 Vision is especially impressive, GPT-4o performs well on several harder examples, and the local LLaVA setup is often limited to descriptions. I choose Phi-3 as my favorite in this particular session. The sample is small, prompts and tools vary, and the conclusion should not be treated as a present-day model ranking.