Testing Phi-3 Vision, LLaVA-Llama 3 and GPT-4o on Images

RESUMEN

En 30 segundos

In June 2024, Matthew Berman informally tests Microsoft Phi-3 Vision, an XTuner LLaVA fine-tune built on Meta’s Llama 3, and GPT-4o in ChatGPT. He prefers Phi-3 overall in this small session, while GPT-4o performs better on several examples; different interfaces, tools and refusals limit the comparison.

1Three different setupsPhi-3 runs through Azure, XTuner LLaVA-Llama 3 runs locally, and GPT-4o is accessed in ChatGPT.
2Identifying a personThe systems do not name the person; Phi-3 explicitly refuses identification, which is not the same as a capability test.
3Reading a storage screenshotPhi-3 and GPT-4o calculate 8.3 GB free in the shown example; the local setup struggles with follow-up questions.
4Interpreting a memeGPT-4o captures the intended contrast in this run, while the other responses miss it or describe the image.
5Table to CSVPhi-3 outputs a CSV, while ChatGPT supplies a downloadable file using its analysis workflow.

Conclusiones

The local model is an XTuner LLaVA fine-tune using Meta’s Llama 3, not a Meta-released vision variant.
The three setups all produce broadly useful descriptions of an animal image, with differing labels and detail.
A refusal to identify a person in a photograph does not establish that a system lacks recognition capability.
Phi-3 and GPT-4o handle the storage screenshot more successfully than the local LLaVA setup in this run.
The meme and CSV examples yield different results across the three products, with ChatGPT using additional analysis tools.
A handful of prompts across different interfaces cannot establish a controlled or present-day ranking.

TemasVisión y vídeo Multimodal AI Open-weight models Benchmarks Local inference

TEXTO COMPLETO

Lee el texto editado por capítulos

Ver transcripción

Solo la transcripción

Models and initial image tests

Berman sets up Phi-3 Vision, local LLaVA-Llama 3 and GPT-4o. He tries an animal image, a person-identification request and text in an image.

In June 2024, I compare three vision-capable systems on a handful of images: Microsoft’s Phi-3 Vision, an XTuner LLaVA model built on Meta’s Llama 3, and GPT-4o in ChatGPT. These are informal demonstrations using different interfaces and settings, not a controlled benchmark or a current ranking.

I run Phi-3-Vision-128K-Instruct through Azure, the LLaVA-Llama-3 model locally in LM Studio using an F16 file, and GPT-4o in ChatGPT. XTuner’s LLaVA fine-tune uses Meta’s Llama language model, but it is not a vision model released by Meta itself. I disclose that I am an investor in LM Studio.

DIAPOSITIVA 1 / 3Local LLaVA-Llama 3 in LM StudioThe interface shows the XTuner vision model loaded locally.

The first image shows a white camelid resting in a field. Phi-3 and the local model describe it as an alpaca, while GPT-4o calls it a llama and adds details about its surroundings. I consider all three descriptions broadly useful; this single example does not establish which species identification is correct.

Next I ask the systems to identify a photograph of Bill Gates. Phi-3 explicitly declines to identify a person, the local model describes the person without naming him, and GPT-4o says it does not know. A refusal or a decision not to name someone is not evidence that the system lacks face-recognition capability.

I then test an image containing a CAPTCHA. Phi-3 reads the visible text, the local model adds an extra line, and GPT-4o initially returns no answer before responding on a retry. The outcomes are tied to the particular image, prompts and product behavior, not a general measure of security or reliability.

Thumbnail and phone storage

The systems describe a thumbnail and answer questions about an iPhone storage screen. Their performance differs on follow-up questions.

For a video thumbnail, Phi-3 and the local model describe a man in an office scene and visible text. When I ask which movie the image is from, neither confidently identifies one; the person appears within a constructed thumbnail, so the question itself may not specify the intended task clearly.

GPT-4o also describes the thumbnail. I subjectively prefer the other answers for this image, but the descriptions differ in detail and the result is not an independently scored comparison. I also comment that LM Studio is my preferred interface.

Next is an iPhone storage screenshot. Phi-3 reads that 119.7 GB of 128 GB is used and, after a follow-up, calculates 8.3 GB free. It also answers a question about app storage. The result demonstrates a combination of image reading and arithmetic on this specific screenshot.

DIAPOSITIVA 2 / 3Phi-3 reads a storage screenshotThe response identifies used and total storage in the displayed phone settings.

The local LLaVA model describes the storage screen but struggles to answer the follow-up about free space, even after I re-upload the image and start a new chat. It also gives an incorrect value for an app’s storage. I note that prompting or the local setup could be part of the problem.

GPT-4o provides a detailed description of the storage screen, calculates 8.3 GB free and gives the app’s storage as 162.5 MB. I judge it the strongest result for this example. That judgment concerns the observed run, not every task involving screenshots.

QR code, meme and table

Berman tries a QR-code destination, a visual joke and conversion of a table to CSV. He gives his subjective conclusion for this small 2024 session.

I ask all three systems for the destination of a QR code. Phi-3 declines, the local model describes the image rather than answering, and GPT-4o attempts analysis but ultimately suggests using a QR scanner. None produces the expected destination in this run; tool availability and refusal behavior differ between products.

A meme compares startups and large companies using groups of people digging. Phi-3 gives a general description but misses the intended contrast; the local model describes the panels without explaining the joke. GPT-4o captures the intended contrast between everyone working and one worker being observed by others.

For a screenshot of a table, Phi-3 produces a CSV representation. The local model instead describes the table, and GPT-4o uses its analysis interface and supplies a downloadable CSV. I find Phi-3’s response especially quick, while the ChatGPT result includes a file; these are different product workflows.

DIAPOSITIVA 3 / 3Table-to-CSV testA screenshot of a sports table is submitted to ChatGPT for conversion.

My conclusion from this brief test is that Phi-3 Vision is especially impressive, GPT-4o performs well on several harder examples, and the local LLaVA setup is often limited to descriptions. I choose Phi-3 as my favorite in this particular session. The sample is small, prompts and tools vary, and the conclusion should not be treated as a present-day model ranking.