With One-Twelfth the Data, an Open-Source Vision Model Beat GPT-4o

Will Hardman, founder of the AI advisory firm Veratai, tells Nathan Labenz that the field would have been easier to survey if it had sat still for five minutes. They are recording on December 20, 2024, still inside OpenAI’s “12 Days of Christmas,” so a new model could land tomorrow. Hardman wants to cover the architectures builders actually meet, the datasets and benchmarks behind them, and, at the end, what he calls true multimodality: not only reading images and text, but generating images as well.

The practical case is already large. Google’s Gemini team had recently told Labenz that plenty of people are building with language models and relatively few with vision-language models, and they expect that gap to be a growth area. Hardman lists medical assistants that combine imaging with a patient history, content filters that have to know what is in both a picture and its caption, archival and catalog search, and insurance workflows that check whether a photo of a damaged car matches the written claim.

He offers two longer reasons to care. The first is engineering: a VLM is two modalities that start separate and somehow get reasoned over together. Get the recipe right, and the same ideas extend to audio, touch, lidar, and the rest. A robot that can cook has to handle many sensors at once; vision plus language is the first rehearsal. The second reason is philosophical. It is not obvious that multimodal understanding sits on the path to AGI, but some researchers argue that integrating modalities could produce another leap in how a system understands the world.

The case against is that frontier language models already show abstraction, world models, and reasoning, with no obvious ceiling, so a grounded multimodal view of the world may not be required. Labenz doubts we will ever settle the thought experiment. Imagine building AGI while unable to see: it would be a major hurdle, and multimodal systems are clearly going to work. Whatever the philosophers conclude, he says, this is the path of least resistance, and these systems will walk among us sooner rather than later.

Before any full VLM, Hardman wants two building blocks on the table, because both keep returning. The vision transformer, or ViT, comes from Google’s 2020 paper *An Image is Worth 16x16 Words*. Until then, vision was mostly stacked convolutional filters. The Google team asked whether the transformer recipe could understand images: cut the picture into non-overlapping patches, linearly embed each patch as a token, and run full attention so every patch can see every other patch. A classification token is prepended, and the training target is image classification. Made large enough, the transformer beat the convolutional networks of the day.

A few design facts matter later. The original ViT took a fixed 224×224 image; 16×16 patches yield 196 visual tokens. Later VLM practice usually takes the entire sequence of hidden states, not just the class vector, so those 196 tokens all survive. Naming is shorthand: ViT-H is “huge,” about 600 million parameters, and ViT-G is “giant,” about 1.8 billion. Unlike text tokens, which map back through a codebook, visual tokens live on a continuum. There is no vocabulary to look up, and nothing in the transformer requires quantization before attention can run.

CLIP, from OpenAI in 2021, is the other fixture. A vision encoder and a text encoder are trained on web image–caption pairs so a matched pair has high cosine similarity and a mismatched pair does not. In a batch of N pairs there are N true matches and N²−N negatives; the contrastive loss pulls the true pairs together and pushes the rest apart. You can then lift out the trained vision transformer and use it downstream. Because it has already been pulled toward a language space, Hardman says, later VLM teams start from a CLIP checkpoint and are about two-thirds of the way there.

Labenz used CLIP at Waymark to pick stock images for small-business videos. It was the first time the field had left ImageNet’s thousand classes for web-scale captions, and the captions were noisy: jokes, poems, and literal text in the picture. Search “pizza” for a Detroit restaurant and images with the word pizza printed on them floated to the top. Aesthetics were almost absent, because few captions say “a beautiful picture of X,” and small businesses needed quality as much as content. Hardman says the story of later VLMs is an increasing obsession with filtering that noise.

Hardman has heard Flamingo, DeepMind’s 2022 model, called the GPT-3 moment for vision-language models. The pattern that follows is already here: encode text with a tokenizer, encode images with a vision transformer, pick a language-model backbone to do the reasoning, and then decide how to connect the two. Flamingo froze a Chinchilla language model and inserted cross-attention layers between every fourth transformer block, so the language model could look up what the vision encoder had seen. Two problems show up immediately. The number of visual tokens varies with how many images you feed and how large they are, while cross-attention wants a fixed width; and hundreds of visual tokens make attention expensive.

DeepMind’s answer was the Perceiver Resampler, a small extra model with 64 learnable query vectors. Instead of building a sequence-length-squared attention matrix, the resampler computes sequence length times 64, and the output is always 64 tokens of a known hidden size. Cross-attention can then be defined once and for all. A Hugging Face comparison later put numbers on the cost: in a 7-billion-parameter language model, adding cross-attention layers means about 25% of parameters need training, versus about 10% for a simple projection. You can still freeze both the vision transformer and the language model and train only the new connector.

Hardman still finds it slightly magical. It feels like “tipping the image into a blender,” pressing go, and discovering after training that the important features were retained instead of mashed into pulp. Intuition, he says, does not help much with mechanisms this sophisticated. Training itself is ordinary next-token prediction: image positions in the text become placeholders that tell the model to look up the resampler. The Flamingo team’s other major finding was interleaved data — HTML documents in which images and text keep their original order. Without that mix, downstream scores dropped.

Evaluated on visual question answering, OCR, and captioning, Flamingo was competitive in a few-shot setting with much more specialized models, and cheap to train because the language model stayed frozen. The basic cross-attention pattern has been reused since, but it is not the only way to wire a VLM. Labenz does not remember ever getting to use Flamingo; Hardman does not think DeepMind released it.

Instruction tuning was the unlock for language models, and Flamingo had no comparable step, in part because nobody had a vision instruction set. That is the contribution of LLaVA — Large Language and Vision Assistant — a 2023 model from a mixed Microsoft and academic team. Existing generative VLMs could caption and answer simple questions, but nothing like the range of a tuned language model. Hardman blames the data: interleaved web pages and caption sets were plentiful, task-oriented labels were not. Architecturally LLaVA also switched sides. Following Salesforce’s BLIP work, it takes CLIP visual tokens, trains a simple projection matrix, and injects them into the decoder stream with the text. No new cross-attention layers.

The autoregressive design is simpler and has costs. Every visual token has to be unrolled in the decoder’s attention. A linear projection has far fewer parameters than a stack of cross-attention blocks, so once that thin adapter is maxed out you have to unfreeze the language backbone and risk catastrophic forgetting. Labenz’s early episode with the BLIP-2 authors had already shocked him: a language model trained on a roughly 100,000-token vocabulary can accept image vectors that never appear in that vocabulary, land in regions of embedding space that text itself cannot reach, and keep working. Hardman calls a projection matrix that does not wreck the transformer an amazing finding.

The data recipe is the real cleverness. They started from Microsoft’s 2014 COCO set: about 200,000 images with captions and bounding boxes. Then they asked text-only GPT-4, with carefully written few-shot templates, to invent a conversation between a user and a vision assistant that could supposedly see the photo. A caption about people around a car with luggage on the ground, plus boxes for luggage, a bicycle, and a person, is enough for GPT-4 to ask what kind of vehicle it is, what sits to the left of the car, or what challenge the people face as they load bags. The model never saw the pixels; the answers were already in the boxes. Pre-training used 600,000 caption pairs; instruction tuning used 150,000 of these synthetic dialogues.

At release, LLaVA beat other VLMs on complex image reasoning and was slightly better as a conversational assistant. A year and a half later, ByteDance’s LLaVA-OneVision still sits well on the MMMU board, so the recipe has not gone stale. Labenz describes bootstrapping a 2022 video-OCR job the same way: label ten examples himself, let a model label a hundred, correct the mistakes, fine-tune, and grow a capability in days that human annotators hated. Hardman says synthetic OCR and synthetic LaTeX documents now show up all over VLM training. That is one reason frontier labs sound relaxed about a data wall.

Hardman’s default yardstick is MMMU, Massive Multi-discipline Multimodal Understanding, the multimodal cousin of MMLU. It is built to test three skills: perception (what is in the image), knowledge (what the model knows about that content), and reasoning (what it can infer). University students pulled questions from textbooks, lectures, and the web until they had about 11,000 items across roughly 30 subjects, including history, medicine, electronics, market research, and music. Many items need mathematical reasoning. One example shows harmonic intervals on a score and asks which of four options is constructed incorrectly. Every question is four-way multiple choice, so random guessing is 25%.

When the benchmark launched in November 2023, GPT-4V led at about 55%. The best open-source result, a LLaVA successor, was 34%. A little over a year later, o1 preview sits at 78%, a full eight points clear of the runner-up. Hardman flags the recording date again: Gemini 2 and the newest Claude may not be on the public board yet, and o1 preview could move. He later finds scores for several of those models, but the snapshot is still a point in time.

The more telling baseline is what the MMMU authors did with text-only GPT-4. They OCR’d text from the images, asked a LLaVA model for captions, and fed the prose to GPT-4. That pipeline scored 34% — well above chance. Hardman reads it as evidence that reasoning carries more of the score than people assume. Everything above 34% is what you can credit to actually interpreting visual tokens and reasoning over them. From here on he scores every model he discusses on MMMU.

For how people actually pre-train now, Hardman uses Alibaba’s Qwen-VL series — Qwen-VL in 2023 and Qwen2-VL in 2024. They are decoder-only, autoregressive models with a Qwen language backbone and a vision transformer. The connector is a single standalone cross-attention layer with learnable queries, in the spirit of a Perceiver Resampler, used to compress a variable number of visual tokens before they are injected into the decoder. In an autoregressive setup those tokens otherwise have to be fully unrolled, so shrinking them is worth a module.

The training split is the innovation that stuck. Stage one freezes the language model and trains only the vision encoder and connector on large caption and interleaved sets, with every image resized to 224×224. Stage two, multitask pre-training, unfreezes the whole language model, lifts images to 448×448 (four tiles), and mixes in synthetic OCR, visual-grounding data with bounding boxes, existing VQA and document-QA sets, and text-only data so the model does not forget how to write. Stage three is supervised fine-tuning on a still smaller, more carefully augmented mix. Hardman’s summary of the gradient: data quality goes up, data volume goes down, and more parameters come unfrozen.

Qwen2’s 72-billion-parameter version sits just behind the leading models from OpenAI, Anthropic, and Google on MMMU, and above almost everyone else. Smaller checkpoints are open; the 72B is behind an API, with a paper promise to release it later. Labenz notes that Western observers may be overconfident about their lead. Plenty of Chinese names already appear on Western papers, and the Chinese labs tend to publish more openly. Hardman parks a full handicapping for the next model, InternVL, which is about to make the open-source case even sharper.

InternVL comes from OpenGVLab in Shanghai, and Hardman calls it the leading open-source VLM. Top open models cluster around 70 to 80 billion parameters; the closed ones are presumably larger. InternVL 2.5 was trained on about 120 billion mixed tokens, against roughly 1.5 trillion — later 1.4 trillion — for Qwen2-VL. Shortly afterward the same lab released a dataset on the order of 1.5 trillion tokens, which Hardman reads as a hint they intend to scale further. Public multimodal data has jumped in months: LAION’s 5.8 billion filtered image–caption pairs were the old ceiling; Salesforce-led MINT-1T added a trillion interleaved tokens from HTML, PDFs, and papers; OpenGVLab’s OmniCorpus claims 2.2 billion documents, 8 billion images, and 1.6 trillion text tokens.

Each InternVL generation attacks a specific bottleneck. The first model scaled the vision transformer itself to 6 billion parameters — several times ViT-G’s 1.8 billion — and ran CLIP-style contrastive pre-training against a frozen Llama 7B decoder. The resulting encoder was aligned well enough that they could throw away that Llama and bolt the vision tower onto a different language model. Version 1.5 tackled resolution with dynamic high resolution: an image of any aspect ratio is tiled into 448×448 patches chosen to match the native shape, then a thumbnail of the whole image is concatenated so the model sees both detail and layout. Too many tokens are squeezed with pixel shuffle, moving information from height and width into the depth dimension so count falls without throwing information away.

Labenz later checked OpenAI’s image pricing and thinks GPT-4V is doing something very similar. Low resolution is a flat 75 tokens, which looks like the thumbnail; high resolution adds 512×512 tiles as the image grows, and the 75-token base never goes away. Version 2.5’s contribution is progressive scaling. Align the vision transformer to a small language model until that run converges, swap in a mid-size backbone and continue, then swap in the 78-billion-parameter model. They report 120 billion pre-training tokens to reach the complexity loss Qwen2-VL hit at about 1.4 trillion. On MMMU, InternVL 2.5 sits just behind o1 and ahead of May’s GPT-4o, the original Claude 3.5 Sonnet, and Gemini 1.5 Pro. It also leads on VQA and OCRBench.

Why would a smaller model learn faster? Hardman’s intuition is capacity versus search. A large model can represent more, but it has more free parameters, so gradient descent takes longer to find a good basin. A small model hits a useful alignment sooner; by then the vision encoder has learned most of what it needs, and the larger backbone starts from a decent place instead of searching widely. Labenz is reminded of Microsoft’s TinyStories work, where tens-of-millions-parameter models learned parts of speech and structure first and only later showed “micro-reasoning.” The savings compound: fewer parameters and far fewer tokens.

Everything since Flamingo in this survey has been autoregressive — until Llama 3.2 Vision. Meta used a ViT-H/14, about 600 million parameters, modified the vision transformer with extra cross-attention, and inserted new cross-attention layers into a 90-billion-parameter Llama 3. The rest of Llama 3 stayed frozen through pre-training and supervised fine-tuning; only the new layers and the vision encoder were trained. They also ran DPO at the end, the first time Hardman had seen that preference step in a VLM. The motive is easy to guess: a 90B language model is too valuable to risk degrading. The new blocks are roughly 25% of parameters, trained on about a trillion tokens after heavy cleaning, deduplication, safety filtering, and synthetic OCR, tables, and LaTeX.

The 90B Llama 3.2 Vision is the second-placed open-source model on MMMU, just behind InternVL 2.5. Hardman’s point is blunt: you can build a frontier open VLM with either recipe. The differences that do show up are practical. Several teams find cross-attention weaker on OCR and other fine-grained tasks, perhaps because a Perceiver-style resampler shuffles visual tokens at a spatial scale OCR cares about. Some authors also speculate that multimodal reasoning is easier when visual and text tokens share one decoder sequence, though nobody has a clean causal story. Those two notes, he thinks, are enough to steer most people toward decoder-only self-attention, with a caveat still to come.

Labenz offers an organizational reading from a Latent Space conversation with a researcher who had been at Google, then Adept, then Google again: some architecture choices are fossils of separate vision and language teams. At a sprawling lab, the language group ships the next backbone and the vision group is left a connector that does not force them to touch it. Llama 3 Vision already feels like an odd one out next to Meta’s own early-fusion work. Labenz expects the bitter lesson to unify the designs eventually. Hardman agrees it may simply have been a way to keep Llama 3’s language quality intact.

He pauses to recap what the last two years actually taught: interleaved data at scale, staged pre-training that climbs in quality as it shrinks in size, tiling plus a thumbnail for high-resolution images, progressive scaling of the language backbone, and LLaVA-style task diversity in instruction data. Two families remain. In the autoregressive family, visual tokens join the decoder stream. In the cross-attention family, new layers look up encoded vision. Recent groups have tried to hold data and backbones fixed and ask which connector wins.

Hugging Face’s Idefics papers keep the decoder and vision transformer fixed and swap only the connector. If the language model is frozen and only new parameters train, cross-attention wins, which is unsurprising given the extra capacity. When they update the backbone, the autoregressive setup pulls ahead — after they hit training instabilities on a full attention update and switched to low-rank adapters. Scaling either tower helps; for a fixed parameter budget, a larger language model buys more than a larger ViT. A Perceiver-style compressor speeds training in both families but does not necessarily help final quality. Dropping interleaved documents, in an ablation, hurts a lot, especially few-shot learning.

NVIDIA’s NVLM does a similar bake-off at larger scale, pairing a Qwen language backbone with InternVL’s 6-billion-parameter vision encoder. Decoder-only (D) beats cross-attention (X) on multimodal understanding, image reasoning, and OCR; X is cheaper to train because it never unrolls every high-resolution token in the decoder. They then build a hybrid: high-resolution tiles arrive through cross-attention, while a thumbnail is injected into the decoder stream so self-attention still has an image to reason over. D still wins OCR and slightly wins chart understanding, but the hybrid beats both parents by about a point on MMMU validation. On the public MMMU board, NVLM’s hybrid sits just under InternVL 2.5 and Llama 3.2 Vision. Hardman says the jury is still out; Labenz asks whether a lab trying only for the best model, damn the efficiency, would bother. Too early, Hardman says, but it is a data point that decoder-only is not hands-down the last word.

NVLM’s other result reaches back to the AGI question. Open VLMs, including a LLaVA and an InternVL, dropped on text-only suites such as MMLU, MATH, and HumanEval after vision training — except Llama 3 Vision, whose language backbone stayed frozen. NVIDIA invested in a large text-only supervised set and then saw text-only scores *rise* versus the original decoder, especially on math. They credit geometry-style questions in the multimodal mix: pictures of math improved numerical reasoning even on text-only tests. Labenz analogizes SAT diagrams; Hardman calls it a small data point that multimodal training helps, without proving it is necessary for AGI.

That is the moment Hardman names early versus late fusion. Everything so far encodes the two modalities separately and joins them mid-network, which he puts on the late-fusion side of a continuum. Early fusion would be one encoder, one token space, from the start. He is not a neuroscientist, but he bets brains do both, at multiple integration points, and NVLM already fuses twice: a thumbnail in the decoder and a cross-attention lookup for tiles. Joint pre-training is a different axis from where in the stack the merge happens. A truly early example is still coming.

Two older benchmarks still shape fine-tunes. VQA, built largely on COCO, pairs images with open-ended questions from Mechanical Turk workers that need vision, language, and common sense — on the order of 50,000 images and a million questions. Hardman’s example is a woman with bananas hanging from her upper lip: “What color are her eyes?” should not collapse to the yellow in the middle of the frame, and “What is the mustache made of?” needs to know where a mustache sits. DocVQA is the document cousin: about 50,000 questions over 12,000 workplace scans, charts, invoices, infographics, and handwriting, asking for precise text spans. If your product is document understanding, that is the number to watch. Hugging Face bundled some 50 such sets as The Cauldron; Hardman’s cheap trick is to find a neighboring task, copy its augmentations and prompt shape, and apply them to your own images.

BLINK, from a mixed academic and Allen AI team in mid-2024, is the perceptual counterweight. It has just under 4,000 multiple-choice items across 14 tasks that humans solve in a blink, at mid-90s accuracy, with random guessing a bit over a third. The authors argue that many MMMU items almost reduce to dense captioning: replace the image with a rich description and a language model can still score well, as the original MMMU text-only baseline showed. The jump from GPT-4o to o1 on MMMU looks, in that light, like a reasoning jump. BLINK tries to ask what the visual system is actually doing.

Some categories are already in decent shape. On art style — match a sketchy prompt image to one of two paintings — GPT-4o scored about 83% against a 95% human ceiling and 50% chance. Visual similarity, pairing photographs of the same waterfall from the same viewpoint, had GPT-4 Turbo at 80% and humans near 97%. The badly solved tasks are more interesting. An IQ-test completion item had humans at about four-fifths, chance at 25%, and GPT-4 Turbo at 32.67%. Hardman is reminded of François Chollet’s ARC-AGI: humans bring perceptual priors that shrink the search, while a VLM may have to brute-force the possibilities. Labenz adds that even describing ARC grids — counting squares, reading the dimensions — was something the best models failed at, and that o3’s public ARC results were apparently run on numeric arrays, not pixels.

Relative reflectance is another miss. A hotel photo puts a cream headboard against a white wall; window light makes the headboard pixels lighter, but a person still knows the headboard is darker paint. Humans are around 95%; the best VLM in the paper, a LLaVA, was a shade under 40%, with chance at 33%. GPT-4o improved on some BLINK tasks and *regressed* on others: counting fell from about 60% with GPT-4V to 49%. LLaVA 1.6 34B led counting at 66% against a 98% human score. A DeepMind scan of caption sets found number words too scarce to bind counts to objects. Labenz’s product note is that even “solved” aesthetic judgments still need prompt work: models hedge on “is this attractive,” and do better if you ask which photo a small business would be proud to put in its marketing, or which selfie to send to a spouse.

If ViTs dropped CNN inductive biases, and if most of them are then trained with CLIP-style contrastive loss, maybe both choices are leaving perception on the table. Apple’s AIMv2 work asks whether a vision encoder can be pre-trained generatively instead. A ViT and a decoder are trained from scratch on image–caption pairs. Image-patch tokens come first, text tokens follow; prefix attention randomly masks some visual tokens; the decoder must reconstruct the rest of the visual sequence (soft tokens, mean-squared error) and then the text (cross-entropy). They throw away that decoder, attach the new ViT to Llama 3 with a simple MLP, and fine-tune on the LLaVA mix. Against a same-size contrastive ViT, they see gains on every VLM benchmark they tried, especially captioning and VQA. Hardman would like to know what that encoder does to BLINK. Labenz asks whether the setup is early or late fusion; Hardman says a bit of both: the towers are still separate, but they were aligned much earlier.

Understanding-only VLMs can already ingest video as a sequence of frames. What they have not done, in this survey, is generate pixels. The simple product path is still “write a prompt, hand it to a diffusion model,” which is how Gemini and GPT-4 used Imagen 3 and DALL·E. GPT-4o was announced as a true multimodal model that can emit images, with the “o” standing for omni, but OpenAI had not released that path, citing safety and infrastructure. Meta’s line of truly multimodal work runs through CM3, CM3leon, and Chameleon, which quantized image tokens into a codebook so one decoder could emit either modality. Hardman skips most of that in favor of Transfusion, Meta’s later recipe, which he also sees in DeepSeek’s work.

Transfusion pre-trains one transformer on an even mix of images and text, with a different loss per region. Image tokens stay continuous: a VAE encoder, then an MLP or U-Net down-block, no vector-quantized codebook. Text uses ordinary next-token cross-entropy; image regions use a diffusion loss after a U-Net up-path and VAE decoder. Attention is causal over text and bidirectional within an image, so patches in the same picture can see each other but not the future. Versus the quantized Chameleon series, Transfusion reports similar image quality at about a third the FLOPs, more stable training, and matching prior text-to-text losses at half the FLOPs. A 7-billion-parameter run outperformed DALL·E 2 and Stable Diffusion XL on image metrics and reached Llama 1 level on text-only tasks. Labenz is more excited by the instruction-style local edits — graffiti on a truck turned into in-situ calligraphy — than by unconditional generation, because Waymark needs precise, realistic changes, not a new scene.

Hardman closes with a leaderboard assembled from scattered sources. On MMMU, o1 preview stands at 78%; Claude 3.5 Sonnet (new) and InternVL 2.5 are in the low 70s, Gemini 1.5 Pro a few points below, Grok 2 beta at 66%. InternVL 2.5 is 0.3 points under the new Sonnet and was released five months earlier, which Labenz reads as a reminder that the China gap is roughly zero and cannot be dismissed as distillation of Claude. On DocVQA, Qwen2-VL leads the numbers Hardman has at 96.5%, with every listed model above 90%. On BLINK, InternVL 2.5 edges GPT-4o, 63.8% to 63.2%; Gemini is about 61% and the new Sonnet 56.5%. Among small models, Gemini 2.0 Flash leads MMMU at 70.7%; InternVL 2.5 8B matches its 78B sibling at 95.1% DocVQA; Microsoft’s Phi-3.5 Vision, a 4-billion-parameter model trained on half a trillion mixed tokens and 33 billion SFT tokens plus DPO, is a surprise BLINK leader at 58.3%. Hardman expects more Transfusion-style generators, larger open VLMs via progressive scaling, new augmented fine-tunes, and much more work on VLM alignment. Open-source models are already good enough to start.