Image AI Is Still at the GPT-2 Stage: Suhail Doshi on Building a Unified Vision Model

Returning as the show's first guest, Suhail Doshi, founder of Playground AI, says the past year in image generation has been exciting on the surface but disappointing underneath. New foundation models such as DALL-E 3, Midjourney v6 and Stable Diffusion XL arrived, yet they are used mostly for art. That has some value, he says, but it is nothing like the broad, high-utility experience people get from language models.

Doshi compares today's vision models to language models a few years ago, when they were seen as good for summarization, sentiment analysis or little else. Scaling laws then turned them into tools that write code and answer questions. Nothing comparable has happened for vision, where the practical menu is still making art, removing an object or removing a background.

Asked whether we are at a GPT-2 moment, Doshi says that is exactly how he describes it when recruiting, and he is not sure vision has even reached GPT-3 level. He sees that as the exciting part: a huge open field of unsolved problems, with plenty of pixels left to use and far more visual data than text.

Host Nathan Labenz notes that, from the outside, progress looks dramatic compared with the swirling abstractions of two years ago. Doshi replies that much of the gap is a lack of imagination. Few people predicted how capable GPT-4 would become, and he says the same is true of images, where logos, spatial reasoning, inpainting and text rendering remain weak.

Playground thinks about graphics as three pieces: creating pixels, editing pixels and understanding them. GPT-4V is a rudimentary example of the understanding piece, good at captioning but short of deeper visual reasoning, and no model yet reasons well about something like three minutes of video. Doshi stresses that almost no effort has gone into manipulating real photographs rather than synthetic images.

His example is a Christmas card photo where the dog and the child will not both smile. He would like to highlight his son's face and simply ask the model to make him smile, or merge two shots. He calls that a tiny application and "the very tip of the iceberg" of what manipulating real pixels could allow.

Labenz shares how well GPT-4V works for him, including reading a Salem, Massachusetts parking sign and inferring the Halloween context, and filtering user-uploaded business images at his company. Doshi adds that GPT-4V is weak at image segmentation and bounding boxes, where models like Grounding DINO plus Segment Anything do better, and that it still hallucinates in descriptions. His favorite use is screenshotting code errors, and he wishes a model could sit alongside him and pair as he works.

Doshi argues that reading takes effort and patience, while a diagram, graph or video can explain something far more directly. Many people are waiting for image models that can reliably produce flowcharts and diagrams. Labenz has worked around the gap by asking GPT-4 to emit Mermaid syntax, which renders deterministically, but agrees it leaves a lot to the imagination compared with a real designer.

He adds that humans are exceptionally good at spotting faces that are slightly off. A cartoon version of you may look right to ten friends and still feel offensive to you, which shows how many needs are unmet. He expects this to change significantly within the next year.

He also points to iteration on visual artifacts. You could ask a model to rearrange a bedroom with the bed on the other wall, then ask for a different dresser material or a lower nightstand. The same applies to logos: swapping circles for rounded rectangles or removing a swirl. Today that kind of back-and-forth is nearly impossible, and Doshi thinks the field is underappreciated only because it is trapped in the art world.

Labenz asks how a product builder balances what users need now against what better models will make unnecessary. Doshi praises the community's fine-tunes for hands, eyes and upscaling but calls them all patchwork. They are not a true fix at the foundation-model level, and he thinks the real question is what comes after text-to-image.

Playground plans to focus this year on multitask editing: taking existing images and manipulating them at high fidelity. Local edits include aging a face, changing a hairstyle, swapping clothes or interior design. Global edits are bigger, such as turning a winter scene into spring or summer, which changes every object in the frame.

The team will also work on a model that understands everything in an image, because that understanding helps train the other two. The hope is to unify creation, editing and understanding in a single model by the end of the year, enabling more surprising use cases, even for still images alone.

Labenz lists the growing set of controls in Playground: outpainting, masking, line sketches, depth maps and guidance strength. He asks whether these disappear in favor of a conversational interface. Doshi says some will go away, since a model should understand depth, edges, color and lighting on its own, so that erasing part of an image accounts for shadows and foreground. Tools like ControlNet were a great innovation, but a vast majority of users find such power tools hard.

Still, he does not think the future is text alone. Text compresses a high-dimensional idea into something simple, but concepts like shattered glass or a brand style carry so much entropy that two people imagine them differently. He expects users to combine language with reference images, finger masking on a phone and "I like this" pointers, so vision models will need to be extremely multimodal.

Labenz wonders whether a future user might skip words and let a system read their thoughts. Doshi has not studied brain decoding and is unsure about accuracy, but Labenz points to recent work that maps brain activity into the latent space of a diffusion model using small per-user samples. One uses fMRI, which is far from consumer-friendly, while a Meta effort uses magnetoencephalography with lower accuracy.

Doshi muses that the ultimate multimodal model might not be about tokens at all. Vision transformers chop images into patches and language models use subwords, but a truly unified system might work at a lower level, even bytes, making the distinction between language and vision matter less. He admits he does not know whether transformers or tokens are the right answer and says it may be too early to tell.

He notes that most people he speaks to, including researchers, have not thought much about a unified vision model. They are either stuck on text-to-image or focused on narrow cases like e-commerce product placement, clothes swapping or Snapchat-style filters, which he describes as pre-generative-AI thinking. The promise of foundation models is generalization that surprises us, and that moment has not yet come for vision.

Labenz asks whether vision lacks training data, since web captions are noisy and before-and-after transformations are rare online. Doshi agrees such edits usually require advanced Photoshop skills and are seldom posted. But images are easy to generate and annotate in bulk, so he expects synthetic data to work well. He cites InstructPix2Pix, largely built on synthetic data by a single Berkeley researcher, as a glimpse of what a team that really cares could do.

He also expects progress from multimodal models such as GPT-4V and open-source work like LLaVA. The LAION dataset is widely known to be poorly annotated, and the DALL-E 3 paper showed that recaptioning the dataset improved prompt alignment. Better captions make puzzles like four bottles numbered in reverse order possible and help text rendering, so advances across AI help vision too.

Doshi sums up the tradeoff. Images are enormously rich in information but poorly labeled, while language is richly structured but lossy because we have too few words. Text models feeding multimodal models should help fix our understanding of images, and he guesses vision is about a year or two behind language.

Asked how to balance revenue, funding and milestones against a long-term vision, Doshi says Playground's main quest is a unified vision model spanning images, video and 3D. This year it is working on just two things, starting with a great graphics model for static 2D images. He thinks most people are not Photoshop experts and either settle for a worse picture or hire someone, so giving them these abilities directly would open many doors.

Labenz notes this departs from the usual advice to launch early and iterate, echoing what Sam Altman has said about OpenAI. Doshi says Playground shipped around a hundred things last year and has the same plan since founding. But complaints and requests such as consistent characters have no quick fix.

He describes the worst possible product as one with a fix-hands button, a fix-eyes button and a character-consistency button. Users would need training, a language model would have to pick which model to run, inference costs would rise, and each fix might break something else. Playground has bought many GPUs and hired strong researchers, and he thinks starting with a few broadly useful capabilities, like Midjourney's early versions, can grow into something larger, perhaps "a PlayStation 1" that is still fun to use.

Labenz asks what the rules of the road should be for training data, style imitation and characters like Mario and Luigi. Doshi tries to put himself in artists' shoes. He read Greg Rutkowski's story and says Playground links back to the artist's page with an extra credit line whenever a prompt references his name, in case the AI wave makes him harder to find. He thinks such small steps do not fix the issue, but may help.

He also argues that users who think of themselves as artists do not want to copy other artists, and that you could always make Mario in Photoshop. Fan art is generally fine, and brands often tolerate it, but commercial use such as selling Nintendo-branded T-shirts is where the line falls. He doubts models can be kept from representing famous characters even when scrubbed from the data, so it may be better to talk about use than strictly about training data.

Doshi thinks speed is the key variable. If change is gradual, society will probably assimilate it, but a step-function change must be dealt with, since you cannot ask people to retrain from a craft they have practiced for a decade or two within a couple of years. He is skeptical of revenue sharing for training data, comparing it to the "penny per stream" resentment in music, and thinks promoting real artists may help more because audiences build relationships with artists, not just images.

He has no strong view on solutions, floating ideas like a UBI-style tax and funded upskilling, and notes that some artists simply enjoy their craft and do not want to retool. What he does insist on is that the tech industry should not throw caution to the wind, since the last decade showed that moving fast disenfranchises people and creates bad outcomes for the industry.

Asked what minimum standards AI application developers should uphold even before laws exist, Doshi says there is already reasonable consensus around safety, though the word has been overloaded with alignment. Anyone operating these tools has seen extremely bad actors. Playground trained its own state-of-the-art safety filter that goes further than existing ones, and he says it blocks nude content and other classes of images that are more evil or illegal.

He worries about deepfakes, revenge uses and election-year manipulation, and about explaining to relatives that something they saw online was not real. The tools are powerful and valuable but easily used for nefarious purposes. He therefore sees combating disinformation and misuse as the most basic thing developers can do.

His wish is that safety work were far more open. OpenAI offers a free moderation model that Playground uses, but he would like broader collaboration, since every company has different data and a shared state-of-the-art model would beat the cat-and-mouse game with bad actors. He argues this would not take from anyone's core IP and that such a model's performance could serve as a definitive answer to what the minimum moral standard is.