Why AI Video Is Still a Four-Second Corgi

Devi Parikh grew up in India, moved to the United States after high school, and did her undergraduate work at Rowan University in southern New Jersey. That is where she first ran into what people still called pattern recognition — the phrase “machine learning” had not yet taken over — and a professor who thought she might contribute to research pulled her in. She enjoyed it enough to go to Carnegie Mellon, but she was not sure she wanted a PhD. She applied aiming for a thesis master’s. The year she applied, CMU’s ECE department dropped that track: coursework, or a doctorate. They slotted her onto the PhD path. Her advisor was confident she would fall in love with research, and he was right.

The turn into computer vision was almost embarrassingly practical. Early projects were not very visual. At CMU her advisor’s lab worked on image processing, and she liked that everyone could look at an algorithm’s output and see whether it was working. Abstract tasks left you staring at a metric. Over a long career that began before diffusion models, GANs, and large Transformers existed, she says she never ran a grand strategy. Looking back, one theme keeps showing up: she wanted humans to interact with machines in more meaningful ways.

That is how she moved from non-visual work to vision, then from vision to attributes and adjectives — furry, shiny — as a shared vocabulary so people could teach machines new concepts and machines could explain their decisions. Adjectives became natural language: visual question answering, image captioning. A few years ago she was still dabbling in AI-for-creativity side projects while her bread and butter stayed multimodal vision-and-language. Then she made generative modeling the agenda: Transformers and diffusion for images, video, and 3D video.

She has been at Meta for about seven years. The original plan was a one-year stint at FAIR while she moved from Virginia Tech to Georgia Tech, among colleagues she already knew from Microsoft Research internships and faculty summers. Both sides liked the arrangement, so for five years she split time: fall semester teaching in Atlanta, the rest of the year in Menlo Park. A few months before this conversation, Meta stood up a new generative AI organization covering large language models, images, video, 3D, audio, and music. Her pitch for why it matters is simple. People already consume content on every surface; the interesting question is whether more of them can create what they have in their heads instead of searching for a close-enough match.

Make-A-Video started before DALL-E 2, after DALL-E 1 had blown the team’s minds — images that, she notes, now look almost uninteresting if you go back and check. Image generation was moving fast. Video generation was still an open question almost nobody was working on, which is why they took it on. The technical bet was to reuse everything image models had already learned rather than start from scratch.

The intuition: use images and their associated text to learn what the world looks like and how people talk about it, and use video separately to learn how things move. Split appearance-and-language from motion. That split has three advantages. The model has less to learn because it inherits the image model’s knowledge. Visual diversity — dragons, unicorns, and other fantastical concepts that may be rare in video — comes in through images. And, maybe the biggest win, you do not need paired video-and-text data. You need image-text pairs plus unlabeled video for motion.

Concretely, the network is initialized from an image generator. Before any video training, it can already emit a few frames that are not temporally coherent: independent pictures of corgis playing with balls, not a clip. Training’s job is to make those frames cohere in time. That is where the video data comes in. Asked how the model could ever produce a flying corgi it has never seen, Parikh offers an intuition rather than a proof: images and captions teach what a corgi looks like and what “flying” looks like; videos of planes and birds teach the motion. Interpretability, she admits, is not a strength of these architectures. Nobody knows for sure what they actually learned.

Type “a corgi playing with a ball” and Make-A-Video returns a short clip: the corgi moves, the ball rolls. That was first-of-its-kind at the time. It is also, Parikh says, a four-second video that is essentially an animated image. Same scene, same objects, moving in reasonable ways. Objects do not appear, disappear, or reappear. There are no scene transitions. Compared with the videos people actually watch, it is far from enough. Making clips longer and more complex, giving the model memory so a reappearing object looks like the same object, and telling more complicated stories are all still open.

Elad Gil asks the dangerous question: in a year or two, how long and how complex will generated video be? She will not forecast a number. Make-A-Video was about nine months old at the time of the taping, approaching a year, and neither Meta nor other institutions had shown amazingly longer videos, significantly higher resolution, or much more complexity. Resolution and quality ticked up a little. The category is still “videos equal animated images.” That is unlike the step-changes people got used to in images.

The usual curve — wait six more months and there will be a different regime — may not apply. Video, she suspects, is harder than the language-and-image playbook implied, and progress will be slower than people guessed from those other modalities. Her blunter line: she wonders whether the field is “fundamentally missing” something in how it approaches video generation.

Asked to list why video is slow, she gives four. First, infrastructure: videos are larger, more expensive to store, process, and generate, so the iteration cycle is inherently slower than other modalities. Elad notes this is an old pattern. YouTube sold to Google relatively early in part because streaming infrastructure was so costly. Second, representation: consecutive frames are highly redundant, but models still treat them fairly independently, generating one after another or in parallel and then refining. Whoever finds an efficient video representation may unlock a breakthrough.

Third, architecture. Longer video is an extremely high-dimensional signal — far more pixels than anything else the field generates. Scene transitions and consistency that looks like memory both need hierarchical structure nobody has figured out. Fourth, data. The community has built muscle for massaging language and image data; it has not done that for video. Short clips with little happening produce animated-image outputs. Minutes-long videos with messy transitions are what you actually want, but dumping that complexity straight into a network may not teach well. Some kind of curriculum — picture books before *War and Peace* — might help. Nobody has nailed the recipe.

The same lag showed up in understanding, not just generation. Image understanding raced ahead for years while video understanding trailed in both absolute level and rate of progress. She thinks generative models are replaying that history. Gil asks about robotics. There, she says, video *understanding* matters more than generation: embodied agents consume visual streams, not stills. The twist is that a robot is not watching YouTube. The next frame is a consequence of the action it just took. It knows what it did and sees how that changed the pixels. That feedback loop is a different problem from disembodied computer vision, where the stream just happens and you do not control what you see.

Human interaction was the original research drive, so Parikh is impatient with the current loop: type a prompt, get an image or a video, take it or leave it. You can tweak the prompt, run prompt-engineering tricks, and hope you get lucky. Text is better than a random sample. It is not really control. If generative models are going to be tools for creative expression, they have to bring someone’s voice to life, and that requires better ways to steer them.

She splits “more control” into two directions. One is multimodal inputs. For images, a sketch that the model must respect as composition. For video, an image that specifies the scene, a short audio clip for the soundtrack, even a short video clip for style, with the model expected to combine them reasonably. The other is predictable iterative editing. Even with all those inputs, the model may go off and do its own thing. The ideal is an intuitive interface where you can say what to change and how, and iterate toward the intended result without spending hours learning a new tool on YouTube.

Her forecast is cold: control lags core capability. Images had to look good before people started arguing about whether the picture was the one they wanted. Video is not there yet. Sequence: get randomly good first, then text, then other prompts. That is for generating from scratch. Editing an existing video — stylizing it, for example — is a parallel track, already showing up in products such as Runway, and she expects more of it sooner. She does not claim strong product intuition. Gil lists near-term uses: personal GIFs, mid-stream style changes, short animations, marketing. He is more interested in the unexpected Uber-of-mobile moment. Her own hunch is that situations where people currently *search* for an image or a clip to express themselves will slowly become situations where they *generate*, because the generated thing is closer to what they meant.

She does not track text-to-speech as closely as text-to-audio: prompt “the sound of a car driving down the street” and get a sound effect. State of the art is a few seconds to tens of seconds, working reasonably well roughly one time in five — the closest thing to a metric she can offer in a field without hard numbers. Adding audio to visual content makes it more expressive and more delightful, and she thinks both audio and music are underinvested. Gil points at the old sound-effects-library business: huge, well-labeled catalogs licensed as IP. That industry, they agree, is likely to go away.

Audio has the same length and compositionality problems as video, plus a trick video does not: superposition. A car and a baby crying can happen at once. Describe a complex sequence, or two sounds at the same time, and current models cannot do it well. On applications she is modest. Besides search-turning-into-generation, she expects AI agents on many surfaces, which raises a separate question: what does media creation look like when the user is an agent? Gil adds the usual near-term list, then the observation that cynics keep saying not many people want to create imagery. Recent social media says otherwise. Instagram democratized photography. TikTok democratized short video by taking controls away. Midjourney’s usage suggests a lot of people, professional or not, want to generate high-quality images.

Parikh sees a spectrum. At one end, artists who already had a voice. At the other, people without the training who still want to express themselves more creatively. Both ends matter. Some artists — she names Sofia Crespo and Scott Eaton — built practices around AI years before Midjourney, including with GANs. She hesitates to call herself an artist (that, she says, is for other people to decide), though macrame hangs on the wall behind her and she works in digital, AI, and analog. Making things is why she tries every new model, and why she is obsessed with control: text prompts are too tight. Multimodal inputs are not only steering; they are more knobs to play with.

Artists relate to models on a spectrum too. Some have a specific vision and want the tool to get them there. Others treat unpredictability as part of the process and the finished piece — the model as collaborator, not instrument. Gil mentions Los Angeles artist Alex Israel, not a technical person, who had an unusually specific vision and implementation plan for an AI video project: the tool-with-a-destination end of that spectrum.

In rapid-fire, she flags two under-explored research problems: the control agenda they already discussed, and true multimodality. The field has models that generate text, images, or video as separate systems. There is no deep reason they must stay independent. She wants systems that ingest, understand, and generate across many modalities at once, and she is only starting to see that work. At CVPR she spoke on vision, language, and creativity, and the session that stuck with her was not a paper. It was a workshop called Scholars and Big Models, on how academic labs without huge compute should live in the large-model era. She likes venues that talk about the human and meta side of the work, not only the technical sessions.

The one social-media prediction she will make with confidence: these tools will show up in front of millions and billions of people. Creative expression and communication get cheaper, and entirely new ways of interacting appear. Even the social entities on the graph may change once AI agents sit inside it. Gil argues the impact is already under-discussed: Lensa reached tens of millions of people, Character.AI’s engagement is obvious, Midjourney is how a lot of art gets shared with friends. Bot-based interaction and new social-expression modalities are already here; people just have not labeled them as such.

She writes publicly about time management. The thesis of a post that resonated more than she expected: everything you want to do belongs on a calendar, not a to-do list. A list never forces you to estimate duration, so the work always looks finishable. A calendar makes the hours honest. Outside that post, the advice that has stuck with her is simpler. Don’t self-select. Want the job, apply. Want the fellowship, apply. Want the internship, apply. Do not sit there asking whether you are good enough. If you are not a fit, the world will tell you. Saying no is the world’s job.