Reconstructing Viewed Images From fMRI With Stable Diffusion

Matthew Berman introduces a study by Yu Takagi and Shinji Nishimoto that uses Stable Diffusion with functional MRI data to reconstruct images people viewed. He calls it mind reading, but the tested task is narrower: participants looked at photographs during controlled scans, and the system generated approximations from the resulting measurements. The work does not show that it can read arbitrary thoughts, memories or dreams.

Berman shows the paper’s examples, comparing original pictures with generated images such as a teddy bear. Some outputs preserve a recognizable subject or composition, while details and colors can differ substantially. These are generated reconstructions, not direct photographs taken from the brain; the diffusion model contributes prior knowledge learned during its own training.

The [paper](https://openaccess.thecvf.com/content/CVPR2023/papers/Takagi_High-Resolution_Image_Reconstruction_With_Latent_Diffusion_Models_From_Human_Brain_CVPR_2023_paper.pdf) uses a pretrained Stable Diffusion latent diffusion model. The authors emphasize that they did not train or fine-tune a large generative network on the fMRI data. They did, however, train separate linear regression models to map each participant’s brain signals into model representations; the method was not training-free.

Berman notes that visual representations in the brain and internal representations in deep models are difficult to interpret. Earlier image-reconstruction approaches often involved training or fine-tuning complex generators on limited neural datasets. The authors’ approach instead reuses a pretrained model and learns simpler mappings from scans, while still relying on extensive prior image-model training and a large, carefully collected brain dataset.

The paper also compares model features with brain activity to investigate possible relationships between the representations. Berman describes the brain and artificial networks as working similarly, but the study’s measured statistical correspondences do not establish that their mechanisms are the same. Such associations can provide hypotheses about visual processing without fully explaining either system.

The researchers used the Natural Scenes Dataset, collected with a 7-tesla fMRI scanner over 30–40 sessions while participants repeatedly viewed thousands of natural images. The paper analyzed four of the eight participants who completed all sessions. For each participant it used 24,980 training trials and a shared test set of 982 images across 2,770 trials, averaging repeated test responses. These individualized data requirements limit conclusions about use with a new person.

The authors fitted regularized linear models separately for each participant. One maps early visual-cortex signals to an image-related latent representation, z; another maps higher visual-cortex signals to a text-related semantic representation, c, learned using image annotations during training. The decoded representations guide the pretrained generator to produce a 512-by-512 image. No original test caption is directly supplied as the reconstruction prompt in this described decoding pipeline.

The researchers evaluated the 982 test images with perceptual-similarity metrics and six human raters. Their two-way identification tests ask whether a reconstruction matches the correct original more closely than a randomly selected alternative. Such a score is specific to that identification setup, not the percentage of all pixels recovered or the probability of reading an arbitrary thought.

Berman compares three columns of examples: z alone tends to preserve coarse visual layout but can miss the object’s identity; c alone often suggests the right subject while differing in appearance; the combined z-and-c result can capture more of both. A train, teddy bear, clock tower and skier illustrate the trade-off, but the generated details are not exact copies.

The paper also shows reconstructions for multiple participants, with quality varying between people and images. For the highlighted figure comparing components, the researchers generated five candidates per test image and selected those with the highest perceptual similarity, so those examples should not be interpreted as randomly selected or typical single-run outputs. Berman views the work as a step toward interpreting fMRI signals; the evidence supports reconstruction in this controlled setting, not unrestricted access to a person’s mind.