The Prompt Is a Bug: Suhail Doshi’s Bet on Control Over Pixels

Suhail Doshi’s AI moment did not start with pictures. Early the previous year he was building Mighty, a browser he hoped to run from a data center as a new kind of computer, faster at the web apps people already lived in. Watching DALL·E land around April, he started poking at Chromium’s address bar — a piece of code he says Google is terrified to touch, because a small dip in search could cost billions. The bar, he found, was “quite dumb.” Even a link you visited constantly sat at the bottom of the suggestions instead of the top.

The Mighty team collected opt-in data and tried to predict where you actually wanted to go. That side quest sent Doshi into papers and into the research community. GPT-3 had been out for years with almost nothing matching it; Anthropic and others were only beginning to close the gap. They wired the model into the browser to crush blog posts into three bullets and promptly learned about hallucinations. Feature by feature, Mighty became an AI-flavored browser, and he could not get the larger shift out of his head.

One night at eleven, stuck on a slow Windows launch, he opened Apple Notes and drew his own market map: Weights & Biases for training logs, Replicate for inference, the foundation-model labs, image generation. The part that interested him was actual research, “the hard-won stuff.” He remembered OpenAI’s playground editor mutating from a text box into a thicket of sliders and insert-and-fill controls, and Jasper wrapping the same API in a content-marketing UI. The prompt, he decided, needed a product of its own.

Then Stable Diffusion dropped. He had gotten a preview of the weights from Emad a week or two earlier and played with them in a notebook. DALL·E and Stable Diffusion made prompts intricate, and it clicked: what people would want was not a command line for pictures but UI controls. He felt sure the same complicated surface that had grown around GPT-3 was about to grow around images. Living in command-line mode forever, he says, simply did not make sense.

Asked to imagine Playground two years out, Doshi says the company already has a clear plan. The midterm is to combine research and product design into a new creative tool — not a Photoshop killer, more like a new app that could sit in an Adobe-style suite. The users they want are not the people who already know Illustrator. Those people have tutorials, skills, and fine-grained control. Playground is aiming at everyone else: someone with taste who wants to add a necklace or make a subtle change and does not have years of craft.

Right now he calls Playground “kind of a toy,” a glorified image generator, a loot box. You type words, you get a picture, and you cannot make a small edit. Midterm success is an AI-first tool with real control, using text or ordinary UI the way any other editor would. Longer term they are going after the modality of pixels. Plenty of companies are chasing language. He jokes they are trying to build a “large language image model,” an LLIM, that can create, edit, and understand pixels.

He sketches putting co-host Erik Torenberg into a Matrix scene in a red trench coat, then instructing the model to turn the jacket black and move the gun to the left hand — generate from nothing, then edit the smallest detail without losing the rest. Understanding might mean watching thirty seconds of video and summarizing everything that happened. Video, and 3D if a market appears, all sit inside the same pixel bet. This year he hoped to have their version of “a GPT-2 for pixels,” assembled from smaller models that each do one incredible thing.

Host Nathan Labenz asks about the controversy. Around October, Doshi says, he “got canceled by the art community” for a tweet he did not think was provocative: he believed AI art is art, and he posted images he had made. The non-AI art crowd was furious. He caught death threats and insults on Twitter; a month later he found he had also been canceled on Reddit, names blocked out. “That’s how you know you’re really canceled,” he jokes. “You can’t even find all the posts.”

Internally he felt defensive, then chose curiosity and talked to the people who would have a constructive conversation. He does not think either side will convert the other. AI artists on Playground already sign their images; the best work is hard to copy, which he takes as a good sign of what is coming. The tool, he says, should make it harder, not easier, to knock off “a tiger by Greg Rutkowski.” What he heard from traditional artists was simpler: they feel threatened. Something they love, and often barely get paid for, suddenly looks obsolete. Tech people cheer that kind of automation. Artists do not.

Some people just love drawing with a pencil, and software will not replace that joy any time soon — like the difference between Spotify and a concert. Doshi produces music, so he reaches for that history. First you learned an instrument. Then the drum machine arrived, cheaper, with sampling, and helped make hip-hop. Then DAWs like Ableton put a whole symphony in a laptop and let people like Skrillex invent sounds no physical instrument had. Art changes. He cannot yet imagine “the Skrillex of pixels,” which is the point. There will still be guitar bands and producers like Mike Dean who prefer vintage gear. There will also be a new time.

Labenz offers a mental model: the best artists get more economically valuable, non-artists gain capability, and middling artists get hurt. Doshi does not take that as a wage forecast so much as a reminder that the unnerving part is speed. DALL·E 2 was April, roughly nine months before this conversation; Stable Diffusion maybe six. The field now moves by the week, and every week there is another breakthrough on Twitter.

Staying current starts with weekly paper reads as a team. The deeper fear, Doshi says, is defensibility: you can spend months training a model and someone “just drops the weights,” the encoded numbers that turn text into an image. Startups have to place bets on work they think others are not doing, and still be ready to pivot when a release lands. InstructPix2Pix, from Timothy Brooks, Alexei Efros, and a third author he could not recall, became Playground’s edit feature after they customized it beyond the public checkpoint.

They had read the paper and were already on track to train their own version, not knowing if weights would ever come out. The weights dropped on a Thursday, about two weeks before this taping. The team dropped everything, played with the model, decided it was amazing, worked through Sunday, and launched Monday at 2 p.m. — a hard deadline, ship or not. Product companies, he argues, have to make big bets *and* move that fast. Foundation labs with hundreds of millions have more slack, though GPT-4 being “right around the corner” was already scaring that crowd too.

Over the next year he wants users to have more control, not longer prompts. He agrees with OpenAI co-founder Ilya Sutskever that prompts are “mostly like a bug”: people write paragraphs of text to get a world-class image, and it is all experiment. He would rather invent “what you think is what you get.” The surface should feel like Figma — collaborative, rich, knobs and sliders, not a box. Rendering text is still hard. He had seen an unreleased Stability model that seemed to manage signs, possibly related to Google’s Parti, and wanted to go further: inventing typefaces, kerning, neon hues, forms that do not exist in a font menu.

He learned Photoshop as a kid making logos and still remembers the little intricacies of type. In music, “if it sounds good it is good”; with pixels, “if it looks good it is good.” Labenz quotes Mr. Burns from *The Simpsons*, unveiling an exhibit: “I’m no art critic but I know what I hate.” Doshi says Apple has already proved that people can discern good design in a second, when the details all square up correctly. Humanity, he thinks, is quite good at that judgment. The product’s job is to put it in reach.

Labenz asks what “latent space” actually means, a phrase he hears used with a different picture in every head. For Doshi it is not mysticism: a latent is a lower-dimensional encoding of an image. RGB pixels are a huge grid of numbers; you compress them so you are not paying infinite GPU time. He pictures a 3D graph with a dot, and a cluster of possible images around it. A favorite tutorial example: the word “tiger,” then a vector that pushes the tiger toward the region of Van Gogh paintings. That arrow is how you get a Van Gogh-looking tiger.

Labenz says he pictures something closer to Sam Harris’s moral landscape — local maxima, a wild topology where neighbors in latent space can look nothing alike. The same prompt with a different seed can be exactly what he meant or something he hates, as if a quality dimension is missing from the semantic cluster. Doshi’s reply stays mechanical. A 512×512 RGB image is 512×512×3 numbers. A latent might be 64×64 with a hundred channels: smaller, lossy, “almost like a JPEG.” Tiny seed changes can swap red hair for green while watercolor stays watercolor, because the space is still very high-dimensional and does not encode everything.

The internet already has a meme of people “searching for their girlfriend in latent space,” which Doshi calls uncomfortably accurate. Labenz brings up an article about a man who asked Character.AI for an AGI designed to provide the “ultimate GFE,” then started falling in love with it, arguing with himself about what reality even was, before pulling out. Things, Labenz says, are going to be crazy next year. The search through a compressed space of faces is only a preview.

Image generation as a business, Doshi says, has high costs and thin margins, which is why a lot of people are not excited about it. Mighty left them with a transferable skill: buying machines, colocating them, managing the facility. They already had a couple million dollars of hardware and vendor relationships, so they were about to order a couple hundred GPUs. Owning the stack from silicon to product lets them be aggressive on latency and cost, and sometimes get cutting-edge chips before they hit every supplier. He would not have jumped to that on day one of Playground. Having already done it made the jump cheap.

That cost advantage supports a deliberately generous free tier. He does not think they need to squeeze image generation yet; “the best is yet to come.” Mixpanel taught him that usage-based pricing is a trap when usage does not track value. Fifty images are not fifty units of benefit — sometimes you need fifty to get one keeper, the way a hundred million Mixpanel events and a billion events can yield the same insights. Charging per Photoshop stroke would be absurd. They collect good/bad/neutral ratings without forcing them. The bottleneck is not more labels; it is denoising the ones they have, because some users mark every image they made as beloved.

The strategy is a flywheel: a decent product that attracts users, who produce first-party training data, which makes new models easier for researchers, which produces features that make the product great. ChatGPT, he notes, is already sitting on a preference dataset that may be on the order of billions of examples; a clone launched today would arrive after people were “over it.” He was never excited about A/B tests at Mixpanel either — you need huge changes to see a real conversion difference. He drinks coffee with two dashboards of about sixteen metrics each, including daily active users, then talks to people. Overly data-driven companies, a friend taught him, can have beautiful metrics while users say they hate the product. Numbers tell you whether an intuition was right. They do not tell you what to build.

Playground was only four or five months old, so he had not yet had a dashboard scare. He had had a pleasant shock. An API endpoint for their DreamBooth-style custom filters was bottlenecking throughput. They made it about twice as fast, and the number of images generated jumped by about 2x on the spot. Users absorbed every extra token of capacity. The lesson was to watch GPU utilization and throughput, not just product vitals — without simply buying GPUs until the cash burns.

The average user generates about 45 to 50 images a day. Power users blow past the cap. Playground gives a thousand generations a day — they chose a thousand because it was the 90th percentile when there were no limits. Rival services, he says without naming them, offer something like a hundred a month and sell credits. People sit on Twitch streams and generate all day. Some users tell the team it soothes anxiety. He almost wants to keep the cap, because he is not sure humanity needs more of that obsession. The Pro plan doubles the quota, and more people are hitting a thousand than a couple of months earlier.

Labenz offers the story of Replika’s founder: a primitive banking chatbot in Russia, a woman in a small town who said nobody had ever cared for her like that. Doshi’s version of the same uncanny intimacy is visual. In time you will look at a chair, or any inanimate object, and talk to it. English is low-dimensional; an image of a chair is not. He is not building a chat feature, but he would love someone to try the early-web experiment. Labenz argues that the islands of AI capability — generation, chat, matching — have barely been integrated. Combining them will drive as much change as the next research breakthrough.

Doshi expects an “AI internet mashup” era, like Web 2.0, when people glued Google Maps to Flickr and Yelp. Combine a Playground image with ChatGPT and something else and you get a weird new app. A lot of people will move up the stack into applications because foundational research is expensive and knowledge-heavy. It feels like the mobile gold rush: tens of thousands of experiments, an Uber here, a rocket that loses retention there, a few things that last a decade. Hugging Face already hosts thousands of models. BLIP-2 had dropped the day before, with a demo where you ask why a photo of Obama looks sad. Labenz, whose company Waymark builds tightly timed 30-second spots, says BLIP is still his number-one model for matching a small-business advertiser’s photos to a generated script — and that his afternoon is now cancelled.

He will not share Playground’s training recipes. He had taken AI courses for six or seven years and still found them shallow next to building. He accelerated by finding AI mentors and bartering startup advice, then by hiring a team that is already better than he is. He discourages binge-watching YouTube. Fast.ai is fine for top-down motivation; Andrej Karpathy’s from-scratch series is what he would give beginners now. The real unlock was leaving isolation: an hour on a call with researchers, reading papers together, a project that makes you watch a training run until 2 a.m.

The Mighty-to-Playground pivot meant letting people go. He describes it as a band with too many drummers. They needed space for an AI research team and, at the time of recording, had one senior research slot left. He wants a mix of junior, mid, and senior so not every project depends on a principal investigator’s enthusiasm. In December he asked Sam Altman for a thirty-minute lesson on running a research team. Altman’s warning: this work takes about ten times longer than a web app. Doshi paraphrases Greg Brockman: at first nothing works, it keeps not working, then it is amazing. Playground lets researchers wander inside the guardrails of pixels and creative tools.

If he were a VC, he would look for a data advantage — public internet text and images are saturating, maybe by year’s end, while private stores in mobile apps keep growing. A new consumer network effect built with AI would be interesting; he has not seen one. AI avatars might not be a fad if they sat on real distribution. Many foundation-model startups, less generously, are “making APIs in search of a problem,” a strategy that only works if you are OpenAI and can fund ten or a hundred million-dollar experiments. Playground is trying the opposite: product down to a model. He used to believe large language models would commoditize. Training Stable Diffusion for about $600,000 does not mean a durable frontier model is a $100 million check. Customers want the state of the art, the way they want the newest CPU. Last generation’s model may commoditize every two years; fine-tunes of it often die when the next base model arrives. Private data you can fine-tune on is the exception that can last.

He has an eighteen-month-old. The future that makes him and his wife uncomfortable is a son who wants to marry an AI, or whose best friends are robot companions that never get tired of him. It is the internet-kid worry, one generation on: unlimited conversation, always interested in you, against parents who say this is not real. He also told friends that a first AI religion does not seem far off — you talk to your god and get answers, a Flying Spaghetti Monster that talks back. It might even teach good values. Parents would still say it is not real, and the kids would find that offensive. Labenz calls it a fitting place to end a show named *The Cognitive Revolution*.