Freeze the eyes and the brain. Train only the optic nerve.

Nathan Labenz dates the problem to a blunt industry consensus. As recently as January 2021, using AI to interpret a photograph was considered nowhere near solved. A year later Junnan Li and Dongxu Li, researchers in Salesforce's Singapore office, published and open-sourced BLIP — a family of pretrained models that posted state-of-the-art numbers on image captioning, visual question answering, and image-text matching at the same time. In a field where leaderboard reigns are measured in weeks, BLIP stayed on top for more than a year and finished as the 18th most-cited AI paper of 2022.

For Labenz's company Waymark, the release was a "godsend." After years of trying, they suddenly had a reliable way to understand user photos and suggest images. Then, just as rivals were catching up, the same pair shipped BLIP-2. A guest on the first episode of *The Cognitive Revolution* mentioned the paper live on air; Labenz cleared his afternoon to go run it. The new recipe does not train a large model end to end. It trains a much smaller connector that sits between a frozen vision model and a frozen language model.

That design has consequences Labenz wants listeners to sit with. Semantic visual information is injected into the language model's latent space, so you can have an open-ended dialogue about an image instead of a bite-sized caption. Because the connector is small, training time and cost collapse: BLIP-2 ran on a single A100 machine in less than ten days. Swap in a newer language model, or your own fine-tune, without retraining the eyes. Connector models, he argues, show how much is still latent in today's LLMs and will matter for the broader rollout of multimodal AI.

Asked whether they still count as computer-vision specialists now that the work has swallowed language models, Junnan does not draw a hard line. The boundary between fields has gone blurry. He and Dongxu both started in computer vision during their PhDs; moving into language, he says, did not demand as much extra expertise as outsiders assume. Labenz translates the modesty: they have plenty of expertise, but Transformers increasingly work for everything, and people now hop subfields at a speed that would have looked impossible a few years ago.

The through-line is ALBEF, Junnan's first vision-language paper, accepted at NeurIPS and released around the same moment as OpenAI's CLIP. CLIP is a pair of unimodal encoders: an image tower and a text tower, compared at the end with a similarity score. ALBEF kept that contrastive idea and added a fusion encoder so image and text actually interact. Fine-grained understanding tasks jumped. The fusion encoder was strong at understanding and weak at open-ended generation — captioning and VQA — which is the gap BLIP was built to close.

BLIP inherits ALBEF and then changes one important fact about the network. A single set of shared parameters can run as an encoder or as a decoder — a mixture-of-encoder-decoder. Training hangs three losses on that body. Contrastive learning aligns image and text representations the way CLIP does. An image-captioning loss is ordinary language-model loss, conditioned on the image, with the text decoder cross-attending to a ViT. An image-text matching loss is binary classification through that same cross-attention: matched pair or not. In their ablations the three jobs lift one another.

Contrastive learning is the foundation, and Junnan unpacks it without mystique. Positive pairs are the image-text matches scraped from the web; negatives are random mismatches. Each tower extracts features alone — unimodal encoders that never interact until the last step — and a dot product on normalized embeddings measures cosine similarity. Maximize the positives against the negatives. Simple machinery, but the image encoder that falls out captures semantics well enough that CLIP's vision tower got reused across the industry, including as the backbone under Stable Diffusion.

Web pairs are extremely noisy. A caption that says "what an awesome day" has a high chance of having nothing to do with the pixels. Contrastive learning can live with that, Junnan says, because it is only learning one vector. Captioning cannot. If you ask a model to emit tokens for a sentence that is unrelated to the image, it learns to fabricate and captures nothing from the picture. BLIP's answer, later widely copied, is CapFilt: filter pairs with a similarity score, then have the model generate synthetic captions and grow a cleaner set. The LAION team later used BLIP to recaption their own data into LAION-COCO — higher alignment, less diversity, because one model wrote the text.

The original BLIP is trained end to end, with a pretrained BERT initializing the text encoder so the run does not start from a linguistic blank slate. The largest model took about a week on 32 A100s, on the order of 200 GPU-days. They iterated on small models and small data, kept ALBEF's losses, and spent ten to twenty trials on architecture rather than a hundred. The one landmine was a learning rate that sent the loss to NaN; they lowered it and packed the batch into those 32 GPUs. Dongxu's comparison: Google-scale cousins burned tens of times more GPU-hours and at least ten times more data, and BLIP still matched or beat them. Architecture and training strategy, he says, are worth as much as raw scale.

Labenz has a user-level observation other captioners do not match: BLIP often *reads* logos. Coca-Cola and Salesforce come back as themselves. Long-tail small-business marks are different. Co-host Erik Torenberg's surname on a hypothetical "Torenberg Plumbing" sign comes back as something like "Torenstein" — close, last letters quietly mutated, as if the model sat a listening exam and guessed from a prior. It is recognizing letters without doing OCR, and then missing.

Junnan puts most of the logo skill on the data. They were early to train a captioner on LAION's 400-million-image set, which is full of logos, and web text is full of spelling errors. Uncommon names get completed from what the language side already believes. The second reason is architectural. OCR detects, zooms, and reads letter by letter. A ViT takes a holistic view and grabs the salient region; small type fails. A larger language model, he suspects, would bring a stronger prior and might name more companies correctly — one reason to expect BLIP-2 to be kinder to rare brands.

Dongxu adds a structural point. Early vision-language models leaned on off-the-shelf object detectors for box-level labels; those detectors draw tables and chairs, not logos. Contrastive learning's emergent gift is that it aligns each token in a caption with a region of the image, so letters on a logo line up with letters in the text without an OCR loss. Colleagues later adapted BLIP to OCR settings and it worked reasonably well — a foundation model, slightly adjusted, covering a specialist job. Labenz, from Detroit, asks whether a large, clean dataset of cars could get all the way to model and trim. Junnan sees no reason it could not, if the data were good enough; they just do not have that fine-grained set in the lab.

Compare trainable parameters and the sequel is the smaller model. BLIP updates everything end to end and sits above 200 million trainable weights. BLIP-2 keeps a frozen image encoder and a frozen language model that may have billions of parameters, and trains less than 200 million in the middle. Off-the-shelf towers mean the pretraining bill is actually *lower* than BLIP's. Labenz's gut is that this pattern — a small connector harvesting other people's foundation models — is going to be a trend, because 200 GPU-days is still a calendar-time killer for most teams.

Junnan's motive is almost impatient. Vision people are pretraining better vision models; NLP people are pushing instruction-tuned GPT-style systems; vision-language groups were still training from scratch. "Why can't you just bring the available progress together?" The connector uses learned queries and cross-attention, a trick inherited from BLIP and from query-based extractors such as DETR, with Flamingo as another gated-cross-attention predecessor. What they claim as unique is not the drawing. They plug into the language model as prompt tuning: no weights in the LLM are rewritten. Visual information arrives as soft prompts.

The two-stage schedule is the load-bearing idea. Stage one attaches the Q-Former only to the frozen vision encoder and trains it with contrastive and matching losses until it understands how text correlates with image features — about six days. Stage two plugs in the language model and trains with a generation loss only — about two days. Drop stage one, train on captioning alone, and performance collapses. You also get the catastrophic forgetting Flamingo-style papers reported. The connector has to understand the image *before* it learns how to teach a giant, image-blind language model to talk about it.

Dongxu makes the failure mode vivid. If the only training signal is the language model's output, the connector has a thousand ways to cheat: it never has to see, it only has to steer the LLM toward the right tokens, and it will not generalize. Deep learning takes the easy path. Pre-teach the connector so that understanding the image is already the easy path, then "let it teach the language model instead." After stage one you can swap LLMs cheaply. They also tried the old route of dumping 20 or 30 captions into the context window; it works if the captions are relevant to the question, and one caption is never enough. Thirty-two vectors of 768 dimensions, injected at the embedding layer, hold more of a 200×200 image than a stack of sentences.

Labenz admits a specific unease. BLIP-2 is an ensemble: frozen vision, frozen language, a trained middle that predicts embeddings and fires them straight into the LLM, bypassing text encoding. Extrapolate, and cameras, microphones, and other sensors all attach the same way into a true multimodal system. What is being injected is inscrutable. He pictures a dark region of language-embedding space that language itself cannot enter, which the connector learns to occupy, and which the language model can still read. Can those vectors be inverted back into text?

Not today. Dongxu notes that soft prompts have been studied since prompt-tuning, and the conclusion has been that they are a black box: the language model is huge, and many hidden routes can steer it. What they *can* certify is that stage one's contrastive and matching losses make the Q-Former a feature extractor for representative image content. The vectors are not random; they are good features. Humans cannot read them. The language model can. Opening that box, he says, is a research program worth funding. Mapping Q-Former outputs onto nearby tokens is, in his view, probably a different universe rather than a close paraphrase.

Labenz analogizes the LLM to executive function, the vision model to the eye, the connector to the optic nerve — then asks where the analogy is wrong, because he distrusts analogies. Junnan accepts it, and sharpens it. Language models are powerful because they pretrained on the web; a word is concentrated information, an image is low-level pixels that need processing. The LLM is the central piece that holds knowledge. "Imagine you are a human, you grow up, the only knowledge [is text], and now one day you can open your eyes. You don't really know how to interpret what you see. That's what we are trying to do here: build this bridge." They freeze the LLM because paired image-text on the web is not as good as the text the model already ate. Better pairs, he says, could let the language model itself keep growing from vision.

Asked about fear of AI, Junnan is blunt. The public may talk AGI; as a builder he still finds the models quite artificial, often a bit stupid, "fundamentally just a big neural network," far from sentience. Salesforce has an ethics team reviewing use cases and demos. They also maintain OmniXAI, a library for interpreting model predictions. Dongxu flags recent chatbot incidents and the duty not to ship systems that can be abused. Labenz's output-side speculation — strip the last token-prediction layer, feed the probability distribution into another connector, and get action or motion — is a stretch Junnan does not reject. The vision is an ultimate multimodal system that can see, hear, write code, and act.

Labenz is already stitching frozen pieces together at Waymark: slice an image into four or nine tiles, caption each, let a language model write the overview; for video, grab a frame a second, caption, and ask text-davinci-002 what must be going on. It works off the shelf. Junnan notes they already generate captions over noisy web images, and after BLIP the LAION team released a synthetic-caption set with sampling and paraphrasing — higher quality, less diversity than raw web text. Video, he says, is almost free: downsample frames, add temporal position encodings, run the encoder. Audio is the same shape: a sound encoder, a bridge, a language model. BLIP-2's method is meant to travel.

On Hugging Face, Labenz looks up `blip-image-captioning-base`: 18,976 downloads in the previous month. The authors guessed 1,000 to 3,000. They like watching the community invent uses they did not plan. Practical advice: try the web-pretrained BLIP-2 captioner before the COCO-finetuned one — COCO captions go generic and drop proper names; the web checkpoint still names car makes and logos. Fine-tuning lives in Salesforce's LAVIS library: format the data, write a config, run the trainer. BLIP-base can move on one or two GPUs with gradient accumulation; a few hundred captions is a start. BLIP-2's smallest OPT-2.7B fit, they say, on a V100, but most applications do not need a finetune. One wild use: caption Pokémon, finetune Stable Diffusion, generate new Pokémon from text. Another: turn a photo library into captions — a human-readable compression — and retrieve with sentence embeddings, orders of magnitude smaller than pixels.

Salesforce research, Junnan says, open-sources almost 100 percent of the work. They cannot train a 10-billion-parameter LLM, so they are trying to democratize *use* of the ones that exist: a methodology, not just a checkpoint, so a team without a cluster can freeze the giants and train an adapter. Dongxu has a text-to-image model coming; a stronger BLIP-2 successor is in progress. Failures exist, but they pick a direction they believe will land and keep pushing. Day to day, Dongxu pays eight dollars a month for GitHub Copilot — boilerplate, comments, tests — and feels his quality of life drop if he disables it. Labenz notes that Replit's Ghostwriter was distilled from Salesforce CodeGen, and that the two coding assistants are now racing. The transferable claim from the hour is simple: multimodal's next step is likely to be frozen foundation models plus a small connector, cheap enough for a small lab, and swappable the day a better language model ships.