From forty hours in a scanner to one: reconstructing what you see from a shared brain space

Nathan Labenz opens with a fact he thinks is getting lost in the rest of the AI noise: machines can already reconstruct what someone is seeing from fMRI. Last year he had Tanishq Abraham, founder of the open research group MedArc, on the show to talk about MindEye, which needed 30 to 40 hours of scans to build a model for a single observer. The new guest is Paul Scotti, lead author of the follow-up, MindEye2. Scotti was a postdoc in Ken Norman's lab at Princeton and now leads neuro-AI at MedArc while drawing a salary and compute from Stability AI.

The project started in 2022, when Scotti joined the LAION Discord to learn how an open community was publishing strong machine-learning work. He found a channel on reconstructing images from brain activity, introduced himself to Tanishq, and realized the two tracks met: Norman's group wanted to decode memory, and reconstructing vision is the prerequisite. Timing helped. A new Natural Scenes Dataset had just appeared, and CLIP and Stable Diffusion had been released as open models. The pieces were there for a state-of-the-art pipeline, so they made MindEye an open-science project.

MindEye1 was well received, and contemporaneous papers such as Brain-Diffuser used similar ideas. The constraint was that every model was trained on one person. A new subject meant another 30 to 40 hours in the machine at about a thousand dollars an hour, which Scotti calls unrealistic for memory work, mental imagery, dream reconstruction, biomarker diagnosis, or anything a clinician would actually run. MindEye2 is the step toward that world: collect one hour from a new person and get reconstructions that are not quite 40-hour quality, but close — "near the same level of quality with 2.5% the amount of data as before."

MindEye1 and MindEye2 use the same Natural Scenes Dataset: eight people, each scanned for 30 to 40 hours in a heroic multi-lab collection. The loop is four seconds long — three seconds of an image, one second of blank. Subject one made 40 separate visits and saw 750 images per hour-long session. Every image appeared three times across the study, and the task was to press a button on a repeat, which kept people looking. A person saw on the order of 30,000 to 40,000 images in total. The team never used the clicker data; other papers that predict brain activity from behavior find it useful, but Scotti's group could not get it to help.

The scanner is not a continuous recording. It takes a full three-dimensional snapshot of the brain about once a second, the machine's repetition time, and a general linear model folds that time series into a single measurement per image. What it actually measures is blood oxygenation. Tissue that is working harder needs more blood, and fMRI reads the oxygenation change in cubes 1.8 millimeters on a side. Each of those voxels, Scotti says, contains hundreds of thousands of individual neurons compressed into one number — several steps away from which cells are firing, and crude enough that it is surprising the pictures come back at all.

The model sees only visual-cortex voxels, roughly 12,000 to 17,000 per person. Counts differ because brains differ in size and shape and because the functional map of vision is laid out differently from one cortex to the next, so the cutoff of "visual cortex" is not the same polygon twice. Large-scale parcellations can be drawn with automated software; finer maps use retinotopic localizers, flashing light in a corner of the field and watching where cortex lights up. The dataset contains the whole brain on every snapshot. Feeding all of it worked worse. There is also no voxel-to-voxel correspondence across people, which is why a shared-subject model is not a trivial concatenation.

The design choice that unlocks the rest of the pipeline is directional. Mapping tens of thousands of voxels onto hundreds of thousands of pixels would explode the parameter count and throw away the thing you actually want: what the image *is*. Scotti says you want a compressed, semantically rich space, which is why they map into CLIP. Better CLIP models even retain low-level structure, so you can reconstruct the original picture from the latent. Earlier brain-decoding work leaned on Gabor filters and convolutional nets. The modern move, in his telling, is to do whatever it takes to land in a frozen pretrained space.

He treats CLIP as another modality bolted onto a frozen model, and he expects medical applications to improve in lockstep with foundation models. MindEye1 decoded with Versatile Diffusion; MindEye2 uses Stable Diffusion XL. Better generators raise the ceiling on what a brain latent can become, and better representation spaces raise the quality of the latent itself. Labenz's analogy is the bottom of a U-Net: convolve away from pixels into fewer, deeper channels, then unpack the other side. Diffusion models do a similar unpacking from high meaning and low precision back into pixels.

Two advances share credit for the jump in the paper's figures. An ablation that isolates shared-subject training, even when every hour of data is used, already reaches state-of-the-art reconstruction and retrieval. New CLIP and image models help on their own if you hold the shared-subject piece fixed. A third, less advertised piece is an unCLIP model the team trained themselves: they fine-tuned SDXL so that a ground-truth CLIP image embedding returns both the high-level meaning and the low-level layout. Versatile Diffusion, given a CLIP embedding of people cooking in a kitchen, would give you a kitchen and a chef in the wrong places, in the wrong clothes, under the wrong light. That was the previous ceiling. They had to raise it.

In the paper's pipeline diagram, brain-side modules are marked as trained and image-side modules as frozen. Ideally those frozen pieces would already exist and MindEye2 would only train an MLP and a diffusion prior. The snag was unCLIP. Typical unCLIP models take CLIP's pooled classification token — the contrastive output that aligns images with captions — and turn it into pixels. That path is semantic on purpose. It is the "image variations" trick: feed a tiger, get many tigers. Scotti's group needed the opposite. They wanted to cut off the front of the network, feed only the latent, and get the original picture back.

Instead of the pooled token they used the last hidden layer of the vision transformer, every token's output, a much higher-dimensional object. From that unpooled CLIP latent you can recover essentially the original image. The contrastive objective never asked the model to preserve layout, yet CLIP somehow learned a representation of the whole picture. MindEye1 could not do this, so it ran a second, low-level pipeline: predict a blurry reconstruction with no semantics — a dark blob where a person stands, a blue sky — and start the diffusion process from that hint instead of from noise. The blurry branch used things like Stable Diffusion's VAE, not CLIP.

MindEye2 folded that low-level path into the main pipeline and then discovered it was barely needed. Ablations show that starting from the blurry reconstruction actually looks worse, even if it pads low-level statistics in a results table. A captioning module that predicts a text description from the brain and feeds it into refinement is the same story: cool as a demo, almost irrelevant to final quality. "The vast majority of the information here is contained in that CLIP latent," Scotti says. Throw out captions and the blurry start, and the images still look like what you expect.

The practical reading is awkward for anyone who lives in leaderboards. Strip the extra hints and a human looking at the reconstructions still says the system is working. Put them back and pixel-wise correlations and intermediate-layer similarities go up. Scotti has reconstructions that look obviously better to him and score worse on the metrics every paper in the field reports. The group ran online alternative-forced-choice tests: show two images, pick the one that better matches the original. Preference and the standard tables came apart. Forced choice has its own bias toward naturalistic pictures over retained low-level structure, so it is not a perfect referee either.

The blurry module is the exhibit. A crude averaging process scores extremely well on the official low-level numbers and, subjectively, looks worse. For a conference paper you still have to win the table. Labenz sees the same ceiling across AI: automatic scores are saturating, and human scores are noisy. He cites the AVA aesthetics dataset, which collected about a hundred ratings per image and still reports a mean and a standard deviation rather than a single score, because people disagree too much to crush taste into one number. He suspects even a single person would drift over time.

A separate puzzle is where the semantics come from. A naive picture of visual cortex as a convolutional stack of edge detectors does not obviously contain "tiger," let alone a caption. Scotti says the brain already holds both. They include higher visual areas, not only V1 and V2. As you move forward you reach inferior temporal cortex, which does specialized semantic work; the fusiform face area is the textbook example. Restrict the voxels to early visual cortex and mapping into CLIP gets worse. If the goal were a one-to-one copy of the stimulus, he says, tap the retina or put a camera in front of the eye. The reason to look at the brain is the difference between your representation and someone else's — past history, biases, the functional topography that made your map — which is what you need to compare clinical populations.

Among neuroimaging readers, the retrieval module often matters more than the pretty figures. A contrastive loss trains a brain latent as a third CLIP modality, so you can take nearest neighbors or slide the latent along any CLIP axis you choose. The cheap criticism is that the model merely knows there is an elephant. Scotti's reply is grain: the retrieval path can pick the exact zebra out of about twenty zebra images, with high accuracy. That is not animal-versus-not. It is a latent you can compare to any other CLIP vector, which is why the neuroscience community cares.

On the brain side of the diagram, each person gets an adapter into a shared latent, and everything after that is common. The natural test is hold-one-out: train on seven of the original subjects, then see whether a little data from the eighth is enough to land in that space. Labenz asks the complementary data question. Instead of eight people for forty hours, what if you had hundreds of people for an hour — data that may already be sitting in hospital archives? Scotti frames it as the precision-neuroscience tradeoff. Short sessions on many people teach you how brains differ; long sessions on a few people give the high-quality, subject-specific signal that NSD was built for. He wants both: a foundation model on the flood, then high-quality sets stacked on top.

That foundation model is already in progress. The plan is to train on essentially every public fMRI dataset they can reach, compress it into a generalizable latent, and treat that latent as the new way to pull signal out of a scan. He praises the field's habit of releasing data in the BIDS standard. The Human Connectome Project is a large public set; the UK Biobank is another, though it is paid. The stock is "millions of hours of people in machines," then he hedges: they have not finished the inventory, so it might be less than a million, but it is still orders of magnitude beyond the hundreds of hours in current reconstruction papers. Downstream use need not be pictures. Anything that wants a brain embedding could sit on top.

MindEye2 itself is about two billion parameters, larger than it needed to be because a conference deadline is an invitation to spend the GPUs you have. More than half of that count is a single linear layer: a 4,096-dimensional shared space mapped onto a flattened CLIP grid of 256 by 1,664, roughly half a million dimensions. An MLP that fat is still cheaper to run than a diffusion model. Pretraining took less than a day on one H100 node; each person's fine-tune took less than a day; the SDXL unCLIP fine-tune ran for less than one epoch. Labenz had Claude 3 guess the incremental compute at the high end of one-thousandth of training SDXL. Scotti does not quarrel with the scale.

In the longer picture he wants to map cheaper, worse scanners into a richer latent, and to merge small high-quality or even invasive datasets with the public flood. fMRI is still the best noninvasive tool, and it is a miserable one: expensive, motionless to the millimeter, and closed to anyone with claustrophobia. Implanted electrodes, the kind seizure patients already receive, are much higher quality and spatially limited; Neuralink sits in motor cortex, not over the whole sheet. EEG is cheap and cannot localize. Ultrasound and near-infrared proxies for the fMRI signal look to him like the direction of travel — close-to-fMRI quality without the tube, or a multimodal stack that spends high-quality data on bad inputs.

The shared-subject trick is simpler than the diagrams suggest, and that is the scientific result. The usual recipe is to show everyone the same pictures, then split what is common from what is individual. MindEye2 could not do that. People looked at mostly different images; the overlapping set was the test set and could not be used in training. So they took the obvious baseline: ridge regression, a linear map per person into one 4,096-dimensional space, after which the rest of the model is shared. It was the first thing they tried. It worked.

Scotti's reaction was to wonder why the literature was full of manifolds and complicated networks shared across people. A linear alignment of "my brain and your brain" was enough, even though the two people had seen different pictures. That, he thinks, is evidence of shared representations you can extract in a simple way; fine-tuning can add the individual remainder. Failures are less philosophical. A single giraffe in a simple setting comes back looking like a giraffe in the right place. A crowded marketplace does not. The images came from Microsoft COCO and were mostly simple scenes, so the farther a stimulus sits from that distribution, the less the model can account for it.

Labenz's larger claim is that if the incremental compute is a rounding error, the strongest foundation model wins a long tail of downstream jobs nobody budgeted for. When the team needed unCLIP to actually invert CLIP, Tanishq wrote to Robin Rombach, then head of the Stable Diffusion effort. Rombach's reply was to use OpenCLIP's unpooled embeddings; you would get the original image back almost perfectly, and he showed examples. They asked why that capability was not released. In Scotti's telling, Rombach did not see the potential. People who trained CLIP, he adds, did not imagine anyone would bolt a brain onto it.

The institutional setup is unusual enough that Labenz presses on it. Tanishq runs MedArc and supports several open medical projects; Scotti leads the neuro-AI team. Neither is paid through MedArc. Stability employs them and, more strikingly, lets external academic groups and a Discord community use the cluster. Scotti does not know another lab that looks like this. It is also cheaper than it sounds. Across more projects than this interview covers, they have gotten by on a few nodes — nothing like the Stable Diffusion team's budget. The foundation-model scale-up has not started, so the next ask for compute is still a petition, not a line item.

Labenz asks how to think about open-sourcing a model this strange: people can add a thousandth of the training compute and unlock something the authors did not expect. Scotti's default is to share. The point of a foundation model, in his view, is that most researchers will never have the compute to train one and can still map into it, fine-tune it, and run inference. The scans they use are already public, released with consent, so he does not treat the weights as a privacy leak. If the model worked orders of magnitude better than anyone expects, he would want more diligence about what is released. For fMRI, he is confident the limitations are large enough that sharing is still the right default.

The caution turns real, he says, as recordings get more invasive and higher quality. Then you have to be careful about who has access and whether companies take advantage of people's brain data. Labenz sketches the dual-use picture anyway: a future headset that scores whether you are telling the truth, volunteered in a high-trust society or imposed by the state, plus the face-recognition precedent of police at the wrong door on a false match. Scotti cuts off the fMRI version of that story. Lie detection and law-enforcement use are not relevant to this modality. You cannot access what the person is not directly thinking about. Move more than a millimeter and the data distorts. The signal-to-noise ratio is not in the same universe as a portable polygraph.

What fMRI *can* get good enough for, if the person concentrates and stays still, is comparing clinical groups, hunting biomarkers, and reconstructing mental imagery — the University of Minnesota collaboration already underway. Scotti does not want the field to err on the side of too much caution at the current data quality, because the point of the next few years is helping people with disorders and understanding how the brain works. A world where you read a brain that is not cooperating would take a large hardware jump. Labenz analogizes the moment to GPT-1 or GPT-2: just entering the zone where the work is useful, still far from easy consumer deployment, with at least a couple of generations before it is broadly simple to use.

Scotti closes by naming the rest of the team. Tanishq is senior author on both MindEye papers; Ken Norman and about a dozen others are on the work; they are still collecting new data with Princeton. Anyone who wants to help can go to medarc.ai, join the Discord, sit in on weekly project Zooms, and work against public GitHubs. Labenz's minimum ask is simpler: look at the paper's figures, the seen image beside the reconstruction, and then decide who you would let see your brain data when the hardware catches up.