Fifteen Thousand Brain Numbers Were Enough to Reconstruct What Someone Saw

Nathan Labenz introduces Tanishq Mathew Abraham as a guest who barely needs the usual résumé dump, then gives it anyway. Abraham started college at seven, gave a TED Talk shortly afterward, and finished a biomedical-engineering PhD as he turned 20. Labenz’s on-air verdict is blunt: from what he can tell, the output already sits at “ten-year professor” level. Abraham is also CEO of MedARC, a research group he founded to push AI progress into real medical work rather than leave it on arXiv.

Two papers dropped in the same week. This episode covers *Reconstructing the Mind’s Eye*: fMRI-to-image with contrastive learning and diffusion priors. The sequel episode will take up the other paper, which predicts how stained tissue should look from raw slides so a lab technician can read a diagnosis. Labenz frames both as the opposite of pie-in-the-sky research — applied, hands-on, and already producing images you can judge with your own eyes.

He tells listeners to pause and open the paper’s first figure. Similar, he says, is too weak a word. Side by side, the picture shown to the subject and the picture rebuilt from the brain scan look like the same photograph. Abraham credits a high-resolution public fMRI dataset plus the latest frozen generative models; neither ingredient existed in this combination a few years ago. The rest of the conversation is about how little raw brain data that result actually required.

Abraham did not build a scanner. Neuroscience, he notes, already has an open-data habit: EEG and fMRI collections sit in public databases, and several teams have asked subjects to look at pictures while their brains were measured. The set MindEye used came out around 2020 or 2021. Hospital magnets are typically 1.5 or 3 Tesla; this one is 7 Tesla, which yields a stronger, higher-resolution BOLD signal. Another group ran the sessions — about eight subjects, hours at a time, a few seconds of looking, a few seconds of measurement, a short break, then the next image — and released the files.

A voxel here is a two-millimeter cube. The original dataset covers the whole brain; MindEye uses only the visual-perception subset the dataset authors had already prepared, about 15,000 voxels. Each voxel is a single scalar tied to blood-oxygenation level: more local activity burns more oxygen, and fMRI records that slow, indirect change. There is almost no useful time dimension. The subject stares until the hemodynamic response settles, the scanner averages a few seconds, and the model receives one steady-state number per cube.

Labenz does the arithmetic out loud. A cubic centimeter holds 125 of those 2 mm cubes; 15,000 voxels is roughly 100 cubic centimeters of tissue, a slab on the order of 10 by 10 by 1 centimeters at the back of the head. A phone photo is millions of pixels. The brain signal that is supposed to say “what you saw” is 15,000 numbers. Abraham agrees it is striking, and that you would expect to need neuron-level electrical recordings. You do not, he says, because visual cortex is organized: different regions answer to different image features, so which cubes light up already sketches what is on the screen. That is also why the reconstruction is a neuroscience instrument, not only a party trick.

The shared first step is almost embarrassingly plain: a multilayer perceptron maps the 15,000 voxels toward CLIP space. No attention in the model Abraham’s team trained; Transformers live only inside the frozen CLIP and diffusion backbones. CLIP’s image embedding is not a single 768-dimensional vector. The image is cut into 256 patches, each a 768-vector, plus one global vector, for a 257×768 tensor. The team refuses average pooling. Averaging throws away spatial layout, and later low-level metrics depend on keeping it.

Those predicted vectors are similar to real CLIP image embeddings in cosine space, which is enough for retrieval and not enough for generation. Contrastive training makes neighbors and non-neighbors line up; it does not force the coordinates to match the values a frozen image generator expects. OpenAI hit the same snag in DALL-E: CLIP text and CLIP image embeddings of “a dog” are close, but you cannot feed the text vector into a model that wants the image vector. DALL-E added a diffusion prior to translate one into the other. MindEye does the same from fMRI embedding to CLIP image embedding, then hands the aligned tensor to a pretrained generator.

Retrieval skips that alignment. Cosine search against the study images returns the picture the subject actually saw; search against LAION-5B often returns a near-duplicate, which Abraham treats as generation without a generator. The zebra example is the fine-grained punchline. The dataset is full of zebras, yet the model retrieves the specific zebra on the screen, not “a zebra.” The signal is carrying identity, not just a category label. Labenz points listeners to an earlier Cognitive Revolution episode with Chroma’s Anton Troynikov, who used this same fMRI set to stress-test vector search at scale.

The third pipe is low-level. A simple model predicts Stable Diffusion’s VAE latent from the same voxels and decodes a blurry image that still has object position and rough color. That sketch becomes the image-to-image start for diffusion, the way a drawing becomes a finished AI painting while keeping composition. Abraham is careful: if you only care about semantics, start from noise and skip the sketch; objects will wander and hues will drift. With the sketch, low-level metrics beat prior methods that got the zebra right and the pose wrong. The generator they shipped is Versatile Diffusion, chosen because it consumes the full 257×768 tensor. Stable Diffusion Image Variations and LaFite, which take only the global 768-vector, threw spatial information away and underperformed.

Labenz asks the obvious engineering question: why two steps instead of one end-to-end map from voxels to a generator-ready embedding? Abraham’s answer is empirical, not theological. Joint training cannot hold the best retrieval score and the best reconstruction score at once. Split pipelines let each task peak; ablations in the paper back that up. He does not pretend to have a complete theory. Contrastive learning only constrains who is near whom. In a space this high-dimensional, that is a different demand from matching the exact coordinates a frozen diffusion model was trained on.

A UMAP in the appendix makes the geometry visible. Without the prior, fMRI embeddings and CLIP image embeddings sit in two clusters. After the prior, they overlap enough to feed the generator. Labenz tries on the idea that the retrieval subspace is simply a different place gradient descent likes, and that nobody can interpret it yet. Abraham shrugs toward CLIP itself: image and text towers were never in the same numeric neighborhood either, which is why DALL-E needed a prior. MindEye inherited a known contrastive-learning headache; it did not invent a new one.

The models are per subject. fMRI signatures differ across people, so subject one’s data is split into train and test, a model is fit, and the same recipe is copied to the others. All architecture search ran on subject one; whatever won was applied to the rest. Training images and test images do not overlap. Labenz had “just under a thousand” in mind because of a 982 figure in the paper; Abraham clarifies that 982 is the retrieval test set. Training uses several thousand image–voxel pairs per person — still tiny next to LAION’s five billion, and still enough to retrieve a particular zebra.

Why does a few thousand natural scenes generalize at all? Abraham’s account is that the MLP is not learning vision from scratch. It is learning to land in, or next to, CLIP’s embedding space, where images are already laid out with a great deal of structure. Even before the diffusion prior, the predicted fMRI embeddings inherit some of that layout. The catch is honest: they have not tested images outside natural scenes, and the generator is only fluent in the kinds of pictures it was trained on. Out-of-distribution generality is an open question, not a claim.

Labenz keeps returning to an older Cognitive Revolution episode with the BLIP authors at Salesforce Research in Singapore. His working approximation since then is that every high-dimensional space is bridgeable to every other if you place the pylons carefully. Abraham agrees that mapping one latent space to another is “a very powerful idea,” especially when you are data-poor, and that the new twist is multimodal spaces rather than transfer learning itself. He mentions a recent Meta paper in the “map anything to anything” spirit. He also refuses the too-tidy version: different CLIP and BLIP checkpoints were trained with different objectives, their spaces are not interchangeable, and which representation fits which task is still mostly unexplored.

CLIP, Labenz notes, is biased toward what a caption would say. Beauty and other uncaptioned qualities barely live there. That is why the VAE branch matters: it carries composition — which colors sit where — as a blurry “Coke bottle in front of the photo” sketch, while CLIP supplies the semantic steering. Abraham confirms you can drop the sketch and start from noise if semantics are enough. Labenz restates the whole system as a pincer: one arm projects voxels into the VAE so diffusion has a qualitative start; the other projects them into CLIP so the blur sharpens into the right objects. Abraham says that playback is about right. Swap in a stronger CLIP-conditioned generator later, and the reconstructions should rise with it. That modularity is the point of standing on frozen models.

Abraham did not run the training jobs. First author Paul Scotti and co-author Atmadeep Banerjee did. His recollection is hours, not days. Labenz reads the paper back: the shipped model is 940 million parameters, roughly GPT-2 scale, trained on a single A100 for 240 epochs at batch size 32. Larger MLPs did better in ablations, which is why they did not shrink it for sport. Inference is sub-second. The headline, for Labenz, is not the parameter count. It is that a result this loud fit on one GPU because the expensive world models were already paid for.

Cross-subject transfer is still a blank page. Labenz wants the obvious experiment: pour subject A’s voxels into subject B’s model and see whether the output is noise or a degraded picture. Abraham does not think they ran it. People differ in visual perception, which is why they still train one model per head. The hope is a shared fMRI latent space — a foundation model trained on many datasets — so a new person would only calibrate. Labenz compresses that hope to another thousand-fold cut: from a few thousand images down to maybe five. Abraham says that is the target Paul Scotti is driving toward. Even then, which five images, and whether they should differ by person, are open.

Labenz asks which future is more real: better brain science, or a well-person wearable. Abraham pours cold water on the second. fMRI means lying in a large magnet for a long time with clinicians in the loop. Carrying that field strength around is a hardware problem he will not timetable. EEG has better temporal resolution and worse spatial resolution than fMRI, which is already 2 mm cubes; he thinks a decent reconstruction from EEG is unlikely. Other optical methods exist, each with their own trade-offs. Until someone records high-quality signal outside a scanner, and until that signal is validated through the same pipeline, he does not see a consumer product in the next couple of years.

Labenz wonders whether they could coarsen today’s voxels to 4 mm — eight cubes averaged into one, 15,000 numbers down to under 2,000 — and find the resolution where reconstruction dies. Abraham has not fuzzed the data that way. A saner next step, he says, is to rerun the method on older datasets from weaker magnets, which is what the field used until this 7-Tesla set arrived. Even those weaker fMRI machines are still far finer than EEG. Signal-to-noise is the open measurement; a consumer headset is not a software patch away.

Clinical research is the honest near term. Reconstructions of patients with neurological disease could be tracked as illness progresses. Depression might show up as duller pictures; that is a hypothesis, not a result. Older imagery studies asked people to “imagine an animal” and scored a coarse category. Fine-grained reconstructions would let researchers see what someone is looking at or imagining. Locked-in or comatose patients are the longest-horizon medical case, and they still wait on better sensing. Labenz’s larger takeaway is how many recently opened doors — CLIP, diffusion, one A100, a public 7-Tesla set — had to exist before this paper could. Abraham adds the organizational door: MedARC ran the work open on GitHub, with weekly Discord meetings and compute from Stability AI. Scotti and Banerjee led; contributors joined from around the world, some from a laptop. That collaboration, he says, would not have been possible a few years ago either.