Image Generation Is Still a Loot Box. Suhail Doshi Wants to Scale Pixels

This is Doshi's third company. Mixpanel made product analytics. Mighty was a browser. Playground is an image generator and editor that trains its own foundation diffusion models and recently open-sourced Playground V2.5. Sarah Guo asks the obvious founder question: why this, why now?

The date he keeps is April 2022. GPT-3.5 had landed, then DALL-E 2, and he was still trying to put AI into a browser address bar. The images were a strange, eye-opening moment. People had not expected weird, interesting art this soon. Stable Diffusion followed in June or July. He got a couple of weeks of early access to SD 1.4, and it blew his mind what people could already do. It also seemed odd that all of it lived in a Google Colab notebook. There ought to be a UI that made the same power easy.

Friends had told him to do music. He has produced it as a hobby for about six years. He could not see a useful product he would actually ship for other people, even though the tools have since gotten good. Language was the opposite problem: crowded. He guesses more than twenty companies in the category, with five or eight of them sitting on a billion dollars of funding. He did not want to work on something if extremely passionate people he respected were already grinding on it.

Images sat in the middle. At the time the field was Midjourney, some DALL-E work at OpenAI, and Stable Diffusion. It was unclear who was shipping a fun demo and who would spend years making the models better. OpenAI's commitment looked uncertain. Stability AI seemed focused on seven different things. "There's just not enough people that want to do this one thing and do it really, really great," he says. Images also came with built-in distribution: people want to share them. Creativity, tools, and sharing lined up.

Doshi wants to rename the category. What people call text-to-image is, right now, text-to-art. You write a prompt that is basically a caption. The model diffuses it into pixels. Most of those pixels are used for art. The unused potential is the rest of the job: insert an object into a photo you already have, with lighting that matches; restyle an existing picture; blend real and synthetic imagery into one frame you could actually use for something besides a post.

Playground trains from scratch instead of only fine-tuning Stable Diffusion, which is why he thinks they are closing gaps across use cases quickly. He puts them around number two at text-to-art for now. The product, though, is starting to diverge. Users have phones full of pictures they love. They want to tweak a likeness, keep a dog or a face consistent, drop a logo into a scene. Generation without editing is a loot box: you pull, you cannot really change it, and getting the thing you wanted takes too much effort. Text synthesis is another target. The destination is higher utility, not another Instagram dump.

Strong engineers, Doshi says, start from a naive recipe: pick an architecture, find data, buy compute, stir, and out comes DALL-E 3. It is more complicated than that, and more complicated than he expected even after he expected it to be complicated. Playground 2.5 was a test. Before the next architecture landed, how far could this research team push a known recipe? They took Stable Diffusion XL's UNet, CLIP, and the same VAE Robin Rombach had trained, and set one goal: beat SDXL and ship the best open-source model they could. They did not know by how much.

Comparing outputs, they noticed SDXL's average brightness. Color and contrast were off. Doshi had stared at the old look so long that the new images looked like an eval bug. He kept saying they could not be right. The team told him he had just gotten used to the new model. The fix was an EDM formulation, from work associated with NVIDIA researcher Tero Karras: a small change in how noise is sampled. Blacks came back with color in them. The muddy average brightness went away. One clever math trick, a large aesthetic jump.

The field is young enough that a new trick shows up every month. Karras's Power EMA, still underused, can make training converge much faster. Offset noise helps color and contrast. DPO, borrowed from language models, works on images too. Any one of these can be a 10 percent, 20 percent, or even 2x move. The number one trick is not really a trick. It is the last supervised-fine-tune stage on carefully curated data: meticulous work, deep care, and, more than in language, taste. Users are in the product to make images, not to label them, so Playground hides a complicated ranking pipeline behind a light preference interaction. He calls the details secret sauce.

Evals keep disappointing him. Each round they try to beat the last eval, then later he finds gaps the score never saw. A recent one: photorealism, faces that looked like they had not slept in three days. Most industry benchmarks, he argues, are built for marketing and poorly correlated with what users care about. Language models look like good students because the tests look like homework: LSAT items, biology, math. Coverage is the real problem. Playground's method is brute force. Look at thousands of images across grids and checkpoints, get exacting about what is off, and pick release candidates by eye. World knowledge, celebrity faces, paintings, 3D, illustrations, logos: all of it is still hard to cover.

If other labs are scaling text, Playground is trying to scale pixels. Images came first for two practical reasons. 3D tools, he says, tend not to make much money; the money is in making the content, Pixar-style. Video is extraordinarily expensive to train and to run, and video models pretrain on something like a billion images anyway to get a semantic grip on pixels. Images are computationally cheaper, and their practical utility is still low, which means the upside is large.

The long-term object does not even have a settled name. People have LLMs. They do not have a common word for a multitask vision model. Doshi wants one that can create, edit, and understand. Understanding is GPT-4V, or open models like CogVLM. Create and edit are what Playground ships today. Further out, the same stack might help a robot traverse a path or a maze, or sit on a surveillance feed and say what is happening. For now the company is focused on graphics.

His controversial architectural take is that Transformers are the right direction and DiT is not enough. Sora is widely assumed to be a diffusion transformer; Stable Diffusion 3's MM-DiT is a variant. A model trained only from captions to images, he argues, does not hold interpretable knowledge the way a language model does. Those two things have to be married, and he expects the image-model architecture to change within a year or two. Language is a low-dimensional compression that two people can speak. Pixels carry lighting, physics, space, size, and shape. Shatter a glass, describe it, and two people will draw two different pictures. Vision-language models finally made images annotatable, which is why the marriage is speeding up. Elad Gil adds the data twist: internet text may not be enough, and may even mode-collapse, while a robot walking down the street can take infinite pictures. The hard part is filtering, especially once synthetic images flood the web.

Audio is the other job he would take if Playground did not exist. He skipped music as a company partly because the whole industry is about $26 billion, which made it hard to see how large a music product could get. Audio in general, he thinks, will be enormous. ElevenLabs is the example he names. Using the tools as a user is how he tracks where the category is going.

Instrumentals are the easy part. Taylor Swift tracks, rap beats: the beds are not scarce. Vocals and lyrics are. Finding a singer, getting words, getting a performance: that has always been the bottleneck. Suno was the first time he heard AI rap with real flow, emotional lyrics, and breath that did not sound like it had been dragged through Autotune. His workflow is now stacked: generate a song in Suno, run another model to split the stems, keep the vocal, throw away the instrumental, and rebuild the track himself. He has posted the results on Twitter. Quality jumps. Weird errors remain. It is, he says, AI tools all the way down.