A Robot-Arm Shot That Took Months Now Lives in a Prompt

Stephen Parker, creative director at Waymark and a co-host for this episode of The Cognitive Revolution, has logged what Nathan Labenz calls a proper 10,000 hours with image and video generators, back to the original DALL-E. He treats Luma’s Ray 2 as one tool in a mix: different models have different strengths, and Luma sits at the cinematic end of the ones he actually uses. His opening hypothesis is that more image-to-video work now starts from AI stills, because stills are cheaper and faster to iterate. Jain agrees. You can generate a hundred images, pick the one that matches the picture in your head, and only then spend the cost of a video.

Parker then pushes on the weird ones: avocado chairs, pickle people, characters that never existed. There is, Jain notes, no training data for a pickle standing up and then sitting down on an avocado chair. The only way through is a strong base model that understands chairs, sitting, and anthropomorphism as separate ideas and can recombine them. These systems are not memorizing behaviors. They are distilling core capabilities — what it means to sit, to be a chair, to be anthropomorphic — and the better that foundation, the more stable the model gets when the request is fully out of distribution.

Parker has also noticed that image-to-video now carries the look of the source frame further into longer generations. Jain’s answer is that a lot of that is still a *system*, not a naked model. Extra machinery tries to understand what is in the image. Those extras are crutches. The first Dream Machine — retroactively renamed Ray 1 — needed many of them to parse motion and vocabulary. Ray 2, he says, does not need even 10% of that stack. Users then push the new model into new jobs, so the crutches come back. Character consistency in Ray 2 is still largely external; as they design Ray 3, it is meant to live inside the model.

The pattern is industry-wide. Application-layer companies wrap current models to solve a job; the next base model eats the job. Jain’s rule of thumb: anything that can be done in the model is going to be better than what is done external to it. He points at Jasper, the AI copywriting company: clever product, but once language models internalized the same skills, the outer system evaporated. Between the user’s intent and the latent space sits what he calls a blood-brain barrier. When people ask for controllability, they are asking to punch through that barrier and tell the latent space exactly what they want.

Chain-of-thought, in this telling, is another version of the same move. Citing a recent Anthropic paper, Jain says the words a model prints may be a way of seeding the latent space rather than a faithful transcript of its thought. Language labs already learned not to coerce JSON from the outside if they can make the model good at structured output — that is how agents appeared. Multimodality will replay the pattern: reason in audio, video, language, and image together inside one model, and the output gets richer than any bolted-on workflow.

Jiaming Song agrees. Collaborating with a model should feel like collaborating with a person: you do not program a colleague’s brain. Today’s video models still have to be told that this is an image-to-video job, that is a keyframe job. A more intelligent model would take the request and not even treat those as different tasks. Camera motion is the concrete example. Ray 1.6 had the feature, but it was hard to control and lower quality. Ray 2’s Bolt Cam works without the user specifying camera extrinsics. The next step is interactive editability in between “give me a Bolt Cam” and “here are the exact coordinates” — faster, slower, more angle, a different subject.

The Bolt Cam that now ships as a Ray 2 concept is the same move that opened *Severance* season 2 on Apple TV+ — a robotic-arm shot that took the production months and became a viral wow moment. Parker notes it is already in the concept library for anyone generating video. Visual models, Jain says, still lack the in-context learning that makes language models copy a style from a few shots. Creators still need to teach a motion, a color grade, or a pose that does not exist on the internet and cannot be mass-synthesized for a LoRA. Fine-tuning and LoRAs also tend to damage what the model already knew.

Luma’s answer is Concepts, taught in a tool they call Concept School: take the model to school, with the user as teacher. The model learns from one to a few examples, existing capabilities do not degrade the way they do in fine-tuning, and concepts compose. Bolt Cam, dolly, and reverse dolly were taught this way, with very few examples. New batches were slated to ship weekly. The goal is to teach the model everything about filmmaking this way, then open the teaching ability to users. It is still finicky, Jain says, so it is not open access yet.

Concept selection is a two-sided bet. On one side sit bread-and-butter storytelling tools used thousands of times a day — tracking a shot, moving left, moving right. On the other sit shots ordinary people cannot pull off. A smooth Bolt Cam, done well, actually needs a robot. MKBHD has one; James Cameron surely has a few; most people do not. Put it in the model. The customer is people who tell stories, or who want to become professionals who do.

Luma’s mission, Jain says, is multimodal general intelligence. Language-model intelligence looks like human abstract thought. Multimodal intelligence looks like a world in front of you: physical properties, physical phenomena, and intelligent beings that raise entropy by interacting. People say world simulator; he thinks simulation is too weak a word. It is a world model — a physical manifestation of the outside universe, a weak facsimile, an approximation. Storytelling is not a side quest on that path. Forcing intelligent systems to dream, play, and deviate from rules is how you get intelligence rather than a procedure.

How well someone tells a story — event A, then B, then C, each a consequence of the last — is, in Jain’s phrase, a very good bearer of IQ. Video models sit on that path. Combine video, audio, and language and they become partners in imagining. It is not an accident that AI helped creative work first, against the old prediction that mechanical labor would automate first. Creative work is where intelligence is required. Art will matter for AGI the way it has always mattered for general thinking.

Labenz asks the obvious AGI objection: if you want a robot to make coffee and vacuum toys, why did the model also learn dragons? Song splits the answer. Short term, a more realistic, less hallucinatory model is better for physical chores. But consider picking up clothes with a dragon on them and putting them in the washing machine. If the model has no concept of a dragon, it cannot. A dragon is a lizard body, reptile scales, bat and bird wings — humans doing out-of-distribution generation from real-world parts. Fiction is still useful for physical tasks.

Longer term, the model should know which world it is acting in. Agents already live in virtual worlds that are not physics. A future robot should handle the dragon T-shirt and then go to the computer to answer an email — switching, as humans do, between physical and imaginary or virtual worlds. That should not require a specialized model for each realm. Asked whether interpretability work would find separate features for realistic physics, magical realism, and Minecraft, Song says he is not an expert on the topic. It is very hard to find that neuron. It is easier to prompt the model to do the task. State-of-the-art depth estimation already often starts from these pretrained generators. The API between human neurons and model neurons, he says, is not that well welded yet.

Parker is unconvinced that video models understand 3D. Maybe they are just getting better at 2D pixel kinematics; he does not think 3D scaffolding is in the training recipe. Song analogizes to the Turing test: the test does not ask whether the model uses human grammar internally. If talking to it feels like talking to a person, it passes. Likewise, predicting the next few seconds of a physically consistent scene may not mean the internals match a physicist’s knowledge. For utility, that can still count as passing a visual-generation Turing test.

Jain goes further: humans may not have true 3D perception either. Stereo disparity, he says, dies out at about 20 centimeters. People drive with an eye patch; close-up grasping is harder, but the brain also has hands and proprioception. It is hard to argue that the brain maintains an explicit 3D representation. Why, then, demand a mesh inside a generative model, as long as the output looks consistent?

Then the analogies. Planes fly, but they do not flap their wings. Birds and planes produce lift; the mechanisms differ. Dishwashers have no hands; the dishes still come out clean. Humanoids already move unlike us. The human brain is a 20-watt piece of organic tissue; these models run on gigawatt-scale clusters. There is no prerogative for machines to think as we do. People focused on making machines think and be interpretable exactly as humans are, Jain argues, both cap the machines’ capability and waste researcher time that should go to scaling attention and designing more efficient learning.

Labenz grants that AIs need not think like us, but pushes on Golden Gate Claude: isolating a concept direction and turning it up is more than a toy. He had wondered whether Luma’s Concepts were the same trick — inject anime or Toy Story in latent space. Jain draws a line between treating a model as a black box you can run empirical experiments on and expecting a coherent, human-readable account of how it works. Quantum mechanics is entirely uninterpretable, yet measurements check out to six, eight, even 23 sigma. Models at this scale, he says, are just not rockable by the human brain.

Golden Gate-style work is archaeology: piecing together what already formed. In the physical world, archaeology is all we can do. Here, the team was present for the formation of the planet and all the eons it went through. That is called training. Better to understand data and training dynamics. Internally they do heavy information-theoretic analysis: where high-frequency versus low-frequency detail goes, what is learned early versus late, when to change the curriculum. One finding: information-theoretic pathways are set in the first 20,000 to 30,000 iterations. Early training is a uniform cornfield; walkers stomp paths and crop circles. Data mix and learning-rate regime at that stage decide a lot of what comes later. Later training mostly emphasizes or undoes those paths.

Labenz asks if they hand-build batches. Jain says that overreads it. They come from the other direction: massive curation and filtering. The way to train a great model is not to throw billions of garbage samples at it and hope it figures things out. Show it good examples of what you want, at scale. Song adds a theoretical snag: training assumes i.i.d. draws from a world distribution, but it is unclear that such a distribution is even well-defined. How much English belongs in a corpus cannot be derived from speaker population. The honest method is empirical and, he says, very messy.

Labenz asks Song for a short master class in the intellectual history of diffusion models. The 2015 ICML paper by Sohl-Dickstein wrote down the forward and reverse process, ran MNIST, and was ignored. GANs generated in one step and looked better. The 2020 DDPM paper by Jonathan Ho and colleagues made diffusion competitive with GANs and — the practitioner’s prize — stable to train. You can launch a job and sleep. GANs collapse without warning. Slow sampling, about a thousand network evaluations, was tolerated because stability was that valuable. Song, who had worked on GANs, called it the first method that was both stable and high quality.

His own DDIM paper then sped sampling 20 to 50 times. Stanford labmate Yang Song continuous-ized the time steps, connecting diffusion to 1980s Anderson stochastic differential equations and denoising score matching. OpenAI trained on ImageNet with classifier guidance. Classifier-free guidance, also from Jonathan Ho’s line of work, applied Bayes’ rule: an unconditional branch and a conditional branch of the same model replace a separately trained classifier. Early 2022 brought OpenAI’s GLIDE, then Google’s Imagen, then Stable Diffusion and the open-source explosion. After that, the field’s attention split among quality, video, and efficiency.

Song’s three criteria for a generative algorithm are high sample quality, stable training, and fast sampling. GANs fail at stability; diffusion fails at speed. Progressive distillation, again from Ho, halves steps repeatedly but is hairy to implement. Early-2023 consistency models aimed at one-stage, one-step generation; in practice consistency distillation from a pretrained diffusion model won, because one-stage consistency training was unstable. GAN-based distillation can look better at very few steps and is less stable — the old tradeoff, back again.

Inductive Moment Matching, Song’s latest work, tries to hit all three. It generalizes consistency models from pointwise matching to distribution-level matching: you do not need to reproduce a single trajectory, only align distributions, which loosens the constraint. The math is maximum mean discrepancy, at least fifteen years old in statistics. The discriminator is a closed-form function in a reproducing kernel Hilbert space, so there is no inner loop of training a neural discriminator. Ablations are blunt: one sample, the degenerate consistency case, is unstable; two is better but still collapses; four or more and training holds. That is their one-stage recipe for quality plus speed plus stability.

Asked where multimodality is going, Jain says the topic is the foundation of the company and one episode cannot hold it. His short version is sharp. What people currently think of as multimodality is not really it — language models fine-tuned to work with image, audio, and video. Helpful for whatever the language model was already doing. On object recognition, segmentation, and following long threads of causality, they still lose to dedicated computer vision models. If language models had been built and still lost to RNNs at understanding language, that would have been a sign the approach was wrong.

The bag-of-tokens approach is extremely helpful, he says, but the field has to look beyond retrofitting everything onto a language backbone. Luma’s bet is a unified singular latent space that treats every modality as one signal. Nature does not distinguish video, image, and audio; they are signals in the same environment, different facets of the same action. Most of the industry, he says, is shortsighted on this — for good reason, because there is still so much to do in language, and people should keep doing that work. Solving multimodality still needs a new approach, and that is the one they are taking.