Don't Ship Autopilot: Gavin Miller on AI That Leaves Creatives in Charge

Gavin Miller reaches for a court painter when the conversation turns to beautified selves and deepfakes. Holbein went to paint Anne of Cleves for Henry VIII; the king was not pleased, and the marriage was over almost as soon as it began. Miller's point is not the gossip. People have been commissioning a nicer version of their own face for centuries, long before a slider existed in Photoshop.

Miller is the head of Adobe Research, the group trying to push Photoshop, Premiere, Illustrator, and the rest of the suite out of tedious pixel work and into what Lex Fridman calls idea space. He is also a poet, a playwright, and a roboticist. In the early 1990s at Apple he wrote a poem line by line into a friend's new voice synthesizer, then built a talking apartment around 1993 or 1994: a voice that nagged him about moldy laundry, photo albums that sensed which page was open and played matching sound. The literary idea of an agent, he says, became a personality he wanted to live with.

The same instinct shows up in hardware. After a 1988 paper, *The Motion Dynamics of Snakes and Worms*, he spent years in the garage on snake robots. The fifth, S5, now sits in the International Spy Museum in Washington. He put a video of it on YouTube and had to keep answering commenters who swore it was fake, on the theory that a man from Adobe would never film a real machine. The background did not move, he told them, because the rig was on a tripod.

Miller's early career was 3D computer graphics, the Renaissance-style project of modeling light, color, and shape before movies leaned on digital effects. He now describes a second wave that is more impressionistic: an idea, even a sentence, can generate an image. Adobe, he says, still wants both ends of that spectrum, not a single bet on autopilot art.

At the analog end, the lab simulates oil and watercolor so every bristle behaves, water keeps flowing after the stroke, and a painter who cares about the trail of the hand keeps full control. At the other end sits what he calls content velocity. A finished poster still has to become a phone vertical, a web banner, a tablet square; switch the copy to German and the words get longer, so the layout has to reflow and still look designed. That work eats hours and is not where the story lives.

AI's job on that second end is to read intent from the first example: what is the hero, what is the empty space for, what can move. Then it fills the other formats well enough that nobody has to start over. Time saved, Miller argues, does not vanish. It returns to look, style, message, and story. Some people will stay hands-on artisans. Others will shift toward art director or conceptual artist, with the computer as the partner that polishes the idea.

Fridman, a heavy Photoshop and Premiere user, wants fewer clicks on the low-level pixel path. Miller's first answer is almost modest: many existing tools already have too many sliders, so an auto button that sets defaults from the image, not from the tool's factory numbers, is already a form of common sense. He calls them smart defaults. You start from a decent guess, then you tweak.

Selection is the longer story. For most of Miller's two decades at Adobe, Quick Select watched color boundaries and flood-filled regions that graph theory said were connected. It was a leap over clicking every pixel, and it still had no idea what a cat or a dog looks like. Neural nets change the bet. A single click can grab the subject. For familiar categories, people or animals, the tool can skip the click and take the person in the middle of the frame as the dominant object.

He is careful about how good is good enough. A joke postcard that pretends you are in Hawaii does not need a perfect edge. A professional matte does. Adobe already combines a cheap one-click pass with algorithms that pull a clean matte from a crude selection. At Adobe Max the lab showed a rough polygon around a dancer in front of a brick wall, then pulled a mask from the moving target. Workflows that used to take hours now take seconds, with a little cleanup at the end. The toolchain is layered. Users go as deep as the job requires.

Fridman pushes on the word robust: a Max demo that works in most cases is not yet a tool you trust on every job. Miller draws a line between academic research and industrial research. Academia rewards the person who first had the promising idea and showed it on a dataset. Adobe still publishes that way, with a record he says matches some universities. It also has to ship, then sit through customer review and product critics.

Shipping, he says, is not perfection every time. It is good enough most of the time, plus a way to intervene when the model is wrong. Adobe's customers are skilled. The company does not want them doing the tedious job on every file, but taking a result from 99 to 100 with a mouse is acceptable at the professional end. That escape hatch is what lets unfinished research reach the product. "If it had to be 100% automatic all the time, then that would delay the time at which we could get to market."

He adds a second test that he finds more useful than raw accuracy. The system should estimate its own competence on this image and hold back, or ask for help, when it is unsure. Further out he describes an ensemble of experts: one model that knows houses, one that knows trees, and a dispatcher that assigns the job. That is a lot of engineering. The practical path is to nail frequent workflows first, then branch out.

Fridman admits he has never touched many of Photoshop's best features. Miller splits the problem in two: help with the task in front of you, and help with the longer journey of learning the tool. He likes the metaphor of a city. You do not need every street. You need the ones that get you where you work. Stay long enough and you become an expert in your neighborhood, not in the whole map. He jokes about Adobe University: use the product long enough and the expertise arrives.

The lab has fed thousands of hours of online Photoshop tutorials into models that try to understand what those videos teach. A CHI paper asked a narrower question: given your last three or four actions, what did other people in tutorials do next? Suggestions can appear in context, so you do not leave the app, invent keywords, and hunt. Longer term, the same understanding of designer behavior is meant to power an assistant that proposes a next step or simply shows what would happen if you tried it.

The destination he wants is "an artist over one shoulder and a teacher over the other." Each release, he says, has to be useful on its own and also lay a foundation for the next level of expectation. He prefers putting new skill in an assistant rather than another control in the GUI. A new menu item adds visual complexity for every new user. A new skill on an assistant is additive if you know to ask for it. Power users still want the dense interface and the keyboard shortcuts; in their hands Photoshop becomes, as he puts it, a musical instrument.

Asked for a favorite among Adobe's vision demos, Miller picks Sky Replace because it is a compound action, not a single instruction. The system selects the sky, searches stock or disk for geometry that matches horizon, lighting, and perspective, and offers several moods on purpose. It mats the new sky behind the foreground, then recolors that foreground from the incoming image. Swap midday for evening and the objects pick up an orange glow.

In college he had Magritte's *The Empire of Light* on the wall, the painting where the house is night and the sky is day. Adobe is trying not to do that by default. Surrealism can be a choice. A default that looks fake just creates post-production. The packed workflow runs in about a second instead of a minute or two. Because it is cheap, you can try ten skies. You may not know what you want until you see it, then go back and polish the mask. That explore-then-refine loop used to belong only to projects with time and budget.

Concept Canvas attacks a different search failure. Ordinary image search takes tags: you say dog and you get a photo that contains a dog. Designers often need a person left of center, a dog at lower right, and empty sky for a headline. The tool lets you assign keywords to regions. The library is already indexed for where important concepts sit, so matching becomes a layout problem. Even though it is still search, Miller says people "feel oddly responsible for the image that comes back as if you invented it." Give enough control, and ownership follows.

Content-Aware Fill, the older patch method, is still excellent on gravel, grass, and brick. It copies similar texture into the hole and smooths lighting so the patch does not flash. Those materials have little global structure, so local plausibility is enough. The method collapses when it must invent hidden structure. A person standing in front of a house hides a quarter of the facade. The algorithm has no idea what a house is, so window lines fail to meet and the wall looks crumpled.

GANs and other neural nets bring common sense learned from the training set: houses have windows, windows repeat, eaves run level. Texture is only a fraction of what they know. The catch is that the common sense is only as wide as the data. Train on houses, show an octopus, and the model will still try to complete a house. Generative fills, Miller adds, were still stable only at low resolution. You either upsample with another method or wait for research to catch up. He would not name a date.

That is why he keeps returning to guardrails. A detector can estimate how well the algorithm will do on this picture. Later, specialists can vote on confidence or a dispatcher can send houses to the house expert and trees to the tree expert. It is a lot of models. The honest rollout is to cover the workflows you see most, then expand.

To get that far you need data, and Adobe sits on the workflows of tens of millions of people. Miller will not take it as a birthright. The modern rule, he says, is that you have to show users how they benefit from sharing. In the short term that might mean better intent and better recommendations. In the long term, people who earn a living in the tool may want it to get smarter faster. The lab is also looking at learning on the fly without keeping a permanent record of what someone did.

Fridman offers a comparison Miller likes. "I'm less comfortable sharing data with social networks than I am with Adobe," the host says, because the Photoshop side has an obvious payoff in the next workflow. Miller agrees there are rooms the company must not enter. Someone preparing evidence for a legal case does not want any step phoning home. A future design might take coarse, high-level behavior from most people and fine-grained traces only from those who opt into a more permissive relationship, perhaps on hobby work.

Today Adobe still learns a lot from in-person customer studies. Training an algorithm would need something more systematic, and Miller says that has to be conscious. Trust is one of the things people treasure about Adobe. Aggressive collection would spend it. The company has a chief privacy officer, and he wants privacy "front and center of thinking about AI rather than an afterthought."

Miller brings up Project Puppetron because outsiders still picture Adobe as a 2D company. The input is a still or a video of someone talking, plus a painting of someone else. The output wears that painting's style on the speaker's face, with mouth, expression, and head motion attached. It is not a screen-door filter that slides over the frame and tears as soon as the person moves. The system infers 3D structure, binds features to a model, and lets the picture follow the head.

Pure 2D stylization does not know which patch is a nose. Brushwork smears. A 3D prior knows what should stay rigid and what should deform. Faces are also the least forgiving subject; a small error drops into the uncanny valley. Miller thinks 3D vision will mature first on domains rich in prior knowledge, faces with a parameterized template you can fit, then move toward general objects. Users may never be told a reconstruction is happening. They will only notice that edits hold together.

He splits immersive media the same way. VR lifts you into another world, closer to a very wide television than to a tool; he worked on latency and nausea in the 1990s, solved enough of it, and found the rigs too expensive and too early. The new wave is all-in-one, splitting into consumer toys and professional uses such as architecture, where a building is the product and scale is the point. AR does the opposite: it takes digital assets off the screen and sets them on your table. Adaptation is the hard part. He compares it to a theater troupe doing *Hamlet* in an English country house, restaging each scene for the rooms they found. A recording of Nureyev might need every nuance kept, maybe only scaled. A frog telling a princess story can wander. Game-industry tricks for adaptive characters, he thinks, will spill into ordinary commercial AR.

Adobe's installed base is large enough that every change meets the sentence "I am used to this." Miller's rule is simple. Trade control for convenience and existing users will be offended. Add convenience while keeping control and they may accept it. Familiar metaphors should not be casually broken. Photoshop has never been static, and it has never been thrown out. A one-click object select can sit beside the old strokes as a mode or a starting point. It does not ask anyone to relearn the craft.

Radical simplicity, he says, is easier in a new container. On a phone the affordances are already different. Or you ship a second tool aimed at speed rather than precision. Premiere Rush is the example: half an hour at home for vacation footage, not the four hours a four-part television series needs in full Premiere. The same person can hold both mindsets. Smart defaults are also teaching. When the sliders interact in ways you cannot see, a "give me a start" button is a skilled person tuning the desk once. He would rather offer a few good defaults than one lucky setting.

A short poem of his interrupts the product talk: "Today I left my phone at home and went down to the sea." The sand was soft, the water like glass, "but I was still just me." Fridman asks how to live with AI versions of ourselves that look better than the day. Miller stays mild. Eighteenth-century aristocrats hired famous painters and almost certainly looked nicer in oil than at breakfast. Intent is the line. A flattering avatar is mostly harmless; people will meet you. The harm is holding yourself to a standard you cannot reach. He also worries that perfect previsualization, the opposite of the grainy moon landing he saw as a child, might dull the wish to go. Pluto, unseen until the photographs arrived, still surprised everyone. Stronger tools for imagining are, on balance, good. Abuses exist and should be discouraged. The public, he adds, needs to know what is possible, so a single image stops being enough evidence.

Miller has written that interns are the secret of a thriving research lab, and he means it in three ways. People bond with a few ideas at a certain stage of a career and keep returning to them. If the field solves your problem in five years, you are out of cards. A new class each year brings minds that are not yet stuck. Industrial research also spends heavily to make a feature robust, which makes staff conservative: they pick what product teams already want. An intern can spend three months on something fanciful. A paper is a win. A miss does not break the lab. The stack of those bets becomes the portfolio from which one or two ideas are hardened over the next year.

Interns are also a hiring path and a bridge to universities. Many of Adobe's strongest researchers started that way, and the rotation keeps professors and the company in the same conversation, often with academic gifts attached. The long-term legacy of a lab, Miller says, includes the people who leave and carry the industrial-research model elsewhere. That makes recruiting harder. He still reads it as a rising tide.

Who chooses the work? Adobe pushes freedom toward researchers and interns, then rewards impact: papers and product. The model he does not want is a director who thinks he is a genius, assigns every task, takes credit, and blames the room. Company priorities still exist. They trickle down as "X would be particularly appreciated this year," and people reinterpret X through what they already want to do. A broad product line helps. A good idea usually finds a curious team, so the lab does not have to over-qualify projects up front. Now and then a product group pays for an extra intern on a problem it really needs. That, he says, is icing, not the cake. Every summer starts with high hopes. Some projects become features, some become papers, some are not novel but would make a fine button, and some reveal how little anyone knows until the same question is tried again years later.

Looking past 2019, Miller names three bets. The hottest is a creative or analytics assistant that keeps guessing what you are trying to do and offers help. He will not predict when it is ready, which is why he wants to watch it. The second is generative adversarial networks becoming practical at high resolution for mainstream work. He likes even their failures: they err in a way that can look like dreaming. The third is infrastructure. Sensei is pulling neural nets into one platform with one interface, a Henry Ford version of what used to be a custom deal with a single product team. A good idea in one app can be called from the others, and the time from prototype to product shrinks. Combine that with real-time ray tracing, live shadows and reflections, and AI that adjusts the frame from an inferred intent, and he says you are basically "dancing with light."

The snake robots started as a graphics problem. In the 1980s he was simulating cloth and soft bodies that bounced and then died. He wondered what would happen if you animated the rest lengths of springs, like muscles. The simplest creature for that was an earthworm, which led to the 1988 paper and, after a move from Alias to Apple, a short film called *Her Majesty's Secret Serpent*. Building in metal made him honest about friction. The first machine only slithered sideways. Later ones gained wheels, then coordinated gaits. S3 could sidewind and go forward; it carried the rings down the aisle at his wedding and made the local paper. Early brains were discrete logic, then eight-bit chips with 256 bytes of RAM, radio-controlled rather than autonomous. A cheap clockwork toy taught him about backlash and humbled the computer scientist. S5, his favorite, is tapered like a real snake, every segment unique, which is why most roboticists refuse to do it. S9 runs on a Raspberry Pi and can take a vision accelerator, so onboard nets and object recognition are finally possible. Children find the machines charming. Adults find them creepy. They still cost more than they return, which is why they remain a hobby.

"Being in research is a license to be curious," he says, and the hobby is the same license. It sent him into physiology and into the tracks sidewinders leave in sand, to see whether his machines left the same marks. He is not chasing human-level intelligence. The Turing test, he thinks, is such a high bar that it is self-defeating. He wants a bounded companion you understand, closer to a dog than to a stranger who can talk about anything. A reasoning system that can explain why it knows something would both feel more sentient and be easier to debug. The robot is his muse for what AI should invent next. Adobe in the physical world is still "a twinkle in my eye." He thinks virtual objects will arrive first through AR, and that when intelligent robots are common they will not surprise anyone. People will already have lived with those personalities on screens. The reaction, he jokes, will be Siri with legs, or Alexa on hooves.

Fridman closes with the last lines of a Miller poem about mortality and a kind of immortality: ideas living on in other people's work. "Do not weep or mourn. It was enough." Across selection tools, sky replacement, talking houses, and garage snakes, the method is the same. Decide how a thing should behave, make it move, then make the motion look alive. The stack of techniques changes. That sequence does not.