Zhang Xiangyu on Multimodal Research, Visual Reasoning and Learning

Zhang Xiaojun introduces co-host Li Guangmi and guest Zhang Xiangyu. Recorded in 2025, the conversation reviews multimodal research and possible directions in visual reasoning, reinforcement learning and autonomous learning.

Zhang Xiangyu describes his academic and research background and his path from computer vision toward combining vision and language.

He compares developments in vision and language research, discussing how tasks, data and supervision affect the representations models learn.

He says self-supervised language tasks can use large text corpora, with BERT and autoregressive language models learning in different ways that can support downstream tasks.

For computer vision, he revisits training approaches that introduce manually designed invariances and argues that they limit scalable representations. This is his interpretation of that research path.

He explains why he became cautious about a language-model-like breakthrough from static images alone: pixel distributions, image generation and human understanding are not simply equivalent.

In a thought experiment, he argues that modeling an image distribution well does not by itself guarantee human-relevant understanding. He considers language and image–text data possible sources of alignment.

He began looking for ways to align vision and language, using interleaved image–text data in the short term while acknowledging limits in the quality and coverage of web data.

He recounts attempts to train image understanding and generation together. In his experiments, both branches improved, but combining them did not naturally produce the expected synergy.

He says removing one branch had little effect on the other, leading him to question whether architectural combination meant genuine integration. The observation concerns his experiments, not every multimodal system.

The team tried having understanding and generation models provide training signals to each other. He says control and mutual benefit remained disappointing, and visual outputs could violate geometric or physical constraints.

After months of exploration, he turned to a model-training observation: in settings he saw, some general capabilities improved with scale while mathematical reasoning did not always improve in parallel.

He offers a clue: larger models sometimes tended to skip calculation steps and answer directly. This does not establish that all larger models behave that way; results depend on training and evaluation.

Using a thought experiment about compression and addition, he explains his hypothesis that better matching a training-data distribution does not necessarily maximize accuracy on a particular task.

He compares possible step-by-step behavior by a smaller model with direct answers by a larger one and invokes mode collapse as an analogy. The numbers and mechanism are illustrative hypotheses, not universal laws.

Zhang argues that reinforcement learning can orient training more directly toward task outcomes, and discusses the value of checking, backtracking and choosing during reasoning.

He describes reflection, verification and strategy selection within a chain of thought as reusable reasoning patterns, comparing them with search and reward-assignment challenges in game reinforcement learning.

He emphasizes that key language-model decisions may occur at a small number of steps. Repeated sampling can sometimes find a correct answer, but reliably selecting a better path remains challenging.

They use branching choices to discuss why a single prediction may be insufficient and how reflection and checking can let a model reconsider earlier steps.

Returning to vision, he says complex generation must satisfy many constraints and direct final output can be hard to control. He proposes intermediate reasoning steps in visual space.

He describes experiments with annotating, editing and reasoning step by step on images. The team found that manually constructed patterns did not necessarily generalize easily to other visual problems.

He distinguishes generalization to new domains from generalization of reasoning patterns, suggesting that checking and backtracking can help across tasks while noting limitations on game-like problems.

He hypothesizes that reflection patterns already scattered through pretraining data may be elicited and strengthened later. The interview alone cannot confirm other teams’ internal methods.

Discussing multimodal data, he notes possible tradeoffs among quality, data mixture and text capability, arguing that individual model effects need experimental evaluation.

He proposes two lines of research: expanding pretraining data that contains process information and giving models a richer set of visual operations, such as editing, drafting and regeneration.

Teaching videos may contain pointing, construction lines and step-by-step explanations, but he also notes difficulties in cleaning long videos, selecting quality data and aligning it with language.

Asked when a multimodal breakthrough might arrive, the guest gives an optimistic timeline while acknowledging unknown data and training obstacles. This was a forecast made in 2025.

The conversation turns to long context and memory. Drawing an analogy with different levels of human memory, he discusses the cost and limits of putting all history directly into one context.

He discusses retrieval, cooperation among models and architectural tradeoffs, arguing that systems should be designed around real tasks rather than optimizing only one long-context test.

Asked whether models might develop different roles, he suggests training could lead components to specialize in memory, processing or other functions. This is a research hypothesis.

He sees online or autonomous learning as a potential next breakthrough while identifying challenges in scaling rule-based rewards, environments and natural-language feedback.

He considers learning how to improve from environmental and human feedback a long-term problem, particularly difficult when clear evaluation criteria are absent.

The hosts summarize multimodal reasoning and autonomous learning as possible major advances. Their timelines are predictions, not verified milestones.

Finally, he discusses world models, robotics and autonomous driving. In his view, constrained settings can use partial approaches sooner, while more general visual understanding and control require substantial research.