A Multimodal Paper Review: BLIP, CoCa, and BEiT-3

This installment continues the multimodal paper review, focusing on Transformer encoders and decoders. It begins with BLIP's two themes: improving data through bootstrapping and combining image–text understanding and generation in a flexible framework.

BLIP comes from the same Salesforce Research team as ALBEF. The paper frames problems in both models and data: existing architectures have complementary strengths but do not directly cover every task, while web image–text pairs are noisy.

Encoder models such as CLIP and ALBEF work well for understanding and retrieval, but text generation is less direct without a decoder. Encoder–decoder models can generate text but do not necessarily support retrieval directly. BLIP aims to bring these capabilities together.

On data, larger web collections can improve performance without eliminating mismatched image–text pairs. BLIP uses a captioner to generate descriptions and a filter to remove mismatches, aiming to improve the training set.

A webpage accompanying a cake image might describe the bakery's location rather than the cake. Such text can be useful for search and commerce without being a good visual description; a generated caption offers an alternative candidate.

In ALBEF, vision and text encoders first extract separate features and align them with image–text contrastive learning. Cross-attention then fuses the modalities, and an image–text matching objective trains the multimodal representation.

ALBEF divides text-side layers between unimodal encoding and fusion to control computation. VLMo instead shares self-attention and selects modality-specific feed-forward experts, allowing a flexible reuse of the model.

BLIP combines these ideas: a Vision Transformer handles images, while text-only encoding, image-grounded text encoding, and image-grounded text decoding provide three modes. Tasks select different paths while sharing many parameters.

Text and vision features feed the image–text contrastive loss. An image-grounded text encoder fuses visual features through cross-attention for image–text matching. The presenter compares these paths with ALBEF and explains the role of parameter sharing.

For generation, BLIP adds a text-decoding mode. It uses causal self-attention so the model predicts subsequent text from what is already available, rather than seeing future tokens.

The presenter distinguishes bidirectional encoding from causal decoding and explains how cross-attention and feed-forward components can still be shared. Language modeling predicts subsequent words, unlike masked language modeling, which reconstructs hidden words within a sentence.

BLIP's text modes support image–text contrastive, image–text matching, and language-modeling objectives. Parameter sharing provides flexibility, but multiple text-side forward passes still impose training cost.

The paper also draws on ALBEF techniques, including momentum encoders for distillation and contrastive similarity for hard-negative selection in image–text matching. The combined encoder–decoder arrangement is called MED.

The second contribution is CapFilt data bootstrapping. Web pairs can mismatch, whereas human-labeled data is generally cleaner. MED is first pretrained, then adapted on cleaner data to produce a filter that scores image–text compatibility.

The filter retains better-matched web captions, while the captioner generates new candidates for the same images. Synthetic captions vary in quality and also need filtering. The resulting richer, cleaned dataset is used to train BLIP again.

Paper examples show that a generated caption can describe a sunset, lake, and birds more precisely, while an original caption can be more specific than a generic generated one. The filter selects among candidates rather than assuming synthetic text is always superior.

Ablations compare model size, data scale, and the captioner and filter. The presenter says each module helps individually, the combination performs best, and generated captions add diversity.

Bootstrapping proceeds in stages: train a model, adapt a captioner and filter to generate or clean data, and pretrain again on the resulting dataset. The model used to produce the data can be larger than the eventual consumer model.

Because data generation is a separate step, the presenter suggests that BLIP-generated pseudo-labels can also serve other multimodal models. Its value lies not only in architecture but also in improving training data.

One example uses Pokémon images to fine-tune Stable Diffusion: the images were available, but captions were missing. Researchers used BLIP to generate descriptions and trained on those pairs; the presenter gives a green Pokémon caption as an example.

A second example is LAION-COCO: starting from an English-language LAION subset, BLIP generates candidate captions and CLIP models rank them, yielding about 600 million image–text pairs. The workflow combines generation and filtering.

The described process samples 40 captions per image, selects five with a vision Transformer model, reranks them with another model, and uses a language model to repair grammar and punctuation. Multiple candidates give the ranking stages choices.

A BLIP captioner might also help where descriptions are missing in video, remote sensing, or medical datasets, though the presenter says performance in those domains needs verification. Better image–text data may enable experiments in language-guided detection and segmentation.

Next is Google Research's CoCa, or Contrastive Captioning. It combines contrastive and captioning losses and reports strong multimodal, image-classification, and video-recognition results using large models and datasets.

CoCa places an image encoder on one side and a text decoder on the other. The text branch first extracts unimodal features, then incorporates image features; the former supports contrastive loss and the fused features support captioning.

Visual features pass through attention pooling and enter the multimodal decoder, where cross-attention fuses them with text. Learned pooling can extract features for different uses, while the text side uses causal attention from the start.

Training efficiency is a central concern. Multiple objectives can require repeated forward passes and increase training time; CoCa uses features from a single pass for contrastive and generation objectives, helping it scale.

The largest model has roughly 2.1 billion parameters and uses billions of training examples, including a large private image dataset. The presenter notes that the method is comparatively simple but its data and compute scale make full replication difficult.

The paper compares captioning, vision–language understanding, ImageNet, video, and retrieval with a polygon chart. Each vertex represents a task, illustrating one model's breadth; the comparisons should be read in their publication-era context.

The presenter concludes that ALBEF-style branches and BLIP or VLMo-style sharing are both viable approaches. As models and datasets grow, training data remains a major factor, so architecture comparisons must account for data conditions.

BEiT-3 aims to unify architecture, training objective, and scaling further. It treats images as language-like inputs, compares image–text pairs to parallel sentences, and learns representations through masked modeling.

The model uses a Multiway Transformer with shared self-attention and feed-forward experts selected for vision, language, or vision–language input. The presenter relates it to VLMo's multimodal expert design.

BEiT-3 reports classification, detection, segmentation, and multimodal results. The presenter contrasts its use of public training data with CoCa's additional private data, emphasizing data access when assessing reproducibility.

Two qualified lessons follow: more objectives are not automatically better, since objectives may or may not complement one another; more data does not guarantee better results either, because quality, model size, and training settings matter.

The introduction calls the trend Big Convergence: pretrain on large datasets, learn transferable representations, and aim for foundation models serving multiple tasks and modalities. The presenter treats this as a major multimodal direction.

One reason the presenter favors Transformers is their adaptability to language, images, and multimodal inputs. Cross-modal applicability matters for a unified framework beyond comparisons on a single vision benchmark.

Dual encoders suit fast retrieval, encoder–decoders suit generation, and fusion encoders suit understanding. Different downstream tasks often require input, output, or structural changes, so these systems remain short of a fully general model.

BEiT-3 chooses masked data modeling as a common objective. Fewer objectives reduce loss-weight tuning and optimization complexity and can help scaling; image embeddings form token sequences that can be processed similarly to text.

After pretraining, different paths can support image classification, detection, segmentation, language tasks, or image–text understanding; separate visual and language paths can support retrieval. The presenter likens this flexibility to building with blocks.

The closing review starts with Vision Transformer and traces ViLT, CLIP, and ALBEF: simpler visual feature extraction, efficient retrieval, and multimodal fusion all informed later work.

On that foundation, CoCa combines contrastive learning and captioning; VLMo explores shared parameters and experts; BLIP adds flexible encoder–decoder modes and data bootstrapping. The presenter describes these as related design choices.

Another line of work uses masking: BEiT and MAE explore recovering visual tokens or pixels, while BEiT-3 combines earlier vision, vision–language, and expert-model ideas. Reconstruction targets and input choices affect computation and representations.

MAE reduces computation by sending only unmasked image patches into the visual encoder. FLIP applies a related idea to CLIP: remove masked visual tokens during training, shortening the sequence and accelerating training.

The presenter then discusses language interfaces: systems such as MetaLM and PaLI use text prompts to specify a task and generate textual answers. Image classification and visual question answering can therefore share more similar input and output forms.

The final outlook covers generalist models such as Unified-IO and the Uni-Perceiver family, aiming to reuse one model for downstream tasks with fewer task-specific heads or structural changes. The presenter sees promise while noting that results were still developing at the time.