Editing Video Like a Document: How Descript Puts AI in the Copilot Seat

Nathan Labenz opens by introducing Andrew Mason, CEO of Descript, the AI-powered editor that treats video more like a document than a timeline. Cognitive Revolution has been a longtime user; Labenz says the tool is now integral to how the show gets made, and he can attest to how unusually easy it is compared with traditional editors. The conversation is framed as both a product deep dive and a look at where AI-assisted creation is headed.

The production load is not small. Labenz credits Adi and Sai of AI Podcasting, a two-person team he has worked with for six months, with putting out eight or more episodes a month and building AI workflows around them. Short clips posted to Twitter, for example, are the output of a pipeline that generates six to eight candidates per episode; Labenz then picks one or two to publish. He has no stake in their shop, but he recommends them enough to mention they are taking on more clients.

Descript bundles its AI features under the name Underlord, branding Labenz calls inspired because it telegraphs the opposite of an overlord. The implication, he says, is that Mason’s team wants humans to stay at the helm of content creation even as AI takes over more of the grind. That tension — a tools company that also ships AI which could start to look like a service — is the question Labenz wants to press throughout the hour.

Mason traces Descript to an observation from about eight years ago: making audio and video was starting to feel as fundamental as writing, the distribution platforms kept getting better, and the act of making the stuff was getting no easier. Transcription quality was finally good enough to reimagine the editor. Instead of a timeline — which not only has a steep learning curve but is “irreducibly slow” — the company built something that felt like the one tool everybody already understands: a doc.

The hope, Mason says, was to put video and audio into every communicator’s toolkit and finish a longer journey of creation: from creative professionals, to full-time creators, to every knowledge worker. AI enabled that from the start. In a generative-AI world he now sees the models as a companion that can take even more tedium out of the process. Opening your mouth and speaking is the easiest thing in the world; looking and sounding good is the hard part, and that is where he thinks AI can actually help.

Labenz nominates pause removal and filler-word removal as first-ballot hall-of-fame features, and he has lately noticed retake removal as well. Mason adds Studio Sound, an effect trained to take audio that sounds like it was recorded on a computer mic — including in an echoey room — and make it sound like a studio with a professional microphone. It is not just noise reduction. Podcast guests often lack a trained producer’s ear: they know it does not sound like radio, but they could not begin to say why or how to fix it.

Mason’s own favorite is more personal. For scripted how-to videos, reading off-camera a sentence at a time is an editing nightmare, and a teleprompter is its own skill. Eye contact lets you read from a Google Doc on screen; turn the feature on and it looks as if you are talking to the camera. He calls it “creepy but it’s so cool.” Combined with retake removal, Underlord can tell you said the same thing several times and keep only the last take. Labenz admits the episode’s opening video uses the same trick.

Another workflow Mason wishes were more obvious in the app: write the script in Descript, lay out visuals as you go, generate a scratch track with AI speech, then record your real camera on top. The product transcribes what you just said and lines the layout changes up automatically. Traditional walls between pre-production, production, and post-production fall away. If you are only doing a one-shot intro with no B-roll, he concedes, a separate doc is almost as good.

That is close to how Labenz already writes intros. He feeds Claude 3.5 Sonnet the episode transcript plus about 30 intros he wrote himself; the model matches his cadence — “welcome back to the Cognitive Revolution,” a couple of notes, then “here’s my conversation” — and the both-sides tone he uses on thornier AI questions. Without those examples it can summarize, but it does not sound like him. He typically records a couple of full takes. Mason’s suggested Descript loop is simpler still: generate the intro at the top of the transcript, select it, and record over it.

Labenz has been impressed by ElevenLabs’ audio-cleanup model and will, when needed, run audio through it and bring it back into Descript. That raises a product-strategy question: will Descript become a hub that plugs in other people’s purpose-built models, or keep building its own? Mason’s answer is blunt. “We don’t have any religion about building our own stuff. Frankly we would rather not.” When Descript does build something, it generally does not release an API for it. The goal is the best creation tool, with integrations that let people use whatever is state of the art.

The exception is Overdub, also branded Regenerator: select a word you already spoke, change it, and Descript generates new speech blended into the same prosody and sound quality as the audio on either side. Mason says nobody else has built that, which is why it stays in-house. For long-form text-to-speech of complete sentences, outside voices are now every bit as good, and Descript plans to add more third-party voices and generative media libraries soon. If something beats Studio Sound, that is “great for us.”

The production team’s wish list is an API: they fine-tune models per client, then want to hit something like an “ignore” endpoint so cuts show up visually in Descript, where a last tweak is easy. They have tried pushing edits through FFmpeg-style tooling; the pain is at the end, when you do not love the result and cannot easily go back. Mason is not against an API — “it’s just a priorities question” — and floats in-product extensions and saved presets as a nearer alternative. He also offers to brainstorm with Adi and Sai, then laughs: “This is the weirdest podcast. I just feel like we’re doing a customer call.”

Descript took a significant investment from OpenAI. Mason’s read of Sam Altman’s logic is that generative AI was getting exciting across audio, visual, and text at once, and Descript sits at the intersection. Altman also believed that even as the models got better, a tool with “human-shaped controls” — a place for people to practice their craft alongside “the alien” — was not going away anytime soon. In practice the relationship has looked like a standard early investor plus conversations with researchers and some early access. Asked about custom models, Mason gives a no-comment “from a place of ignorance”: he is not even sure what the research teams are working on with them.

Labenz cites OpenAI’s work with the legal-tech firm Harvey, where continued pre-training on legal data reportedly produced a 97 percent preference rate over base GPT-4. He wonders whether Descript’s mountain of intensively edited content could yield a similar jump. Mason does not pick up the Harvey comparison, but he does confirm the direction of travel: the company has seen the value of fine-tuning “and we’re moving to largely fine-tuned models for most of our actions at this point.”

Labenz’s favorite new question is what a ten-times-more-expensive version of the product would look like if inference cost were no object. Mason’s obvious answer is generative video, still cost-prohibitive for the interesting use cases. His framework: most videos are “mostly window dressing,” with brief glimpses of actual content on a few frames. This very episode is an example — two talking heads that hold attention but add almost nothing to the substance. Speech is now cheap; creating the visuals is the hard part. Synthetic avatars are the most prominent experiment in generative window dressing, and he thinks they are only the most obvious option. Animation and other forms look more interesting.

Why not integrate Runway? Mason puts it at something like ten dollars per minute of video, which really limits what Descript’s customers can do. Tools like Midjourney work because they are not zero-shot; they invite back-and-forth experimentation. Multimodality is a surer near-term path. Today Underlord can only read the script and guess what is happening on screen or in the audio. When it can look at frames and listen to intonation, the assistant gets more accurate and can take on richer styles. Fine-tuning to the individual user is “definitely on our radar,” though he cannot say much yet. Labenz’s Claude intro prompt, with the full transcript and 30 examples, costs about ten cents — still well under a 10x ceiling even at eight episodes a month.

Who actually creates in Descript? Mason describes a mix: podcasters, video creators, and a lot of people making educational explainers and how-tos. The product has a built-in screen recorder that can fire off a Loom-style video message or overlay narration on a screencast. Inside companies, marketing teams and people making internal training materials show up. Beyond the original episode, those users also repurpose shows into clips and promo content.

The pain points Mason hears most from creators are monetization and finding an audience — “at least for now, kind of outside of our sweet spot.” Descript has its hands full making content easier to produce. Once creators get past the first list of ideas, keeping things fresh is the next struggle. He sorts the work into three buckets: ideation, making more content faster, and audience plus monetization. Labenz, whose sponsorships are largely handled by the Turpentine network, agrees that selling and producing high-quality sponsor reads is a real load for independent creators.

On AI avatars from companies like HeyGen and Synthesia, Mason does something founders rarely do on a podcast: he admits he missed it. “There are some companies that have gotten quite large building AI avatars… and I totally missed it. When I saw that I was like, this is dystopian.” Even out of the uncanny valley, he argues, the value as currently executed is all on the creator side — making output scalable — with “no value… for the audience.” He expects the interesting phase to be not recreating the old form but inventing something new that is “not quite as creepy.”

Labenz has just spent an episode with HeyGen co-founder Joshua, generating his own intro with a HeyGen avatar and then, after a corrupted guest video file, recreating the whole conversation with the guest’s avatar. His verdict is late uncanny valley: serviceable for a couple of minutes at the top, not something he wants for long-form yet. What has let him down in AI-generated podcasts is less the voice or the face than the lack of alpha. “The quality of ideas” matters more than the form factor. Both of them do see a near-term killer use: translation. A slightly weird Spanish version of a real episode still opens a new market.

Labenz cannot resist a Groupon question: what are small businesses sleeping on with AI? Mason, too busy building “this little video app” to stay in touch with the massage places Groupon used to feature, still thinks the obvious gap is an 80th-percentile online presence — videos, website, the works — whose cost AI has crushed. Labenz’s mind goes to appointment-taking phone agents. For the slightly larger SMBs and enterprises Descript does talk to, the pattern is different: they have a mandate to find AI tools, and they often cannot find something that adds meaningful value once it has to live inside the organization. Magical demos that do one specific thing tend to break down in real workflows. That is not a reason to call AI a fad; it is a reason to keep building tools that actually get used.

Hiring AI engineers is a different discipline depending on the work. Wrangling LLMs is one skill; building models like Studio Sound or Overdub is another. The model-building path upends a product commonplace. Normally you define requirements, design something, and build around known engineering constraints. With these models the constraints shift as the research unfolds. The user experience for a speech model that trains instantly is very different from one that trains in 15 minutes, which is very different from one that trains in a day. Descript has lived all three cycles.

Late in development, a much higher-quality model sometimes appears that takes three hours instead of 15 minutes, and the trade-off can completely shift the nature of the product. The other lesson is that AI research is a stack of bets. Some pan out; the ones that do not may hit an absolute dead end, forcing the team to start from scratch on a different idea. That has to be factored into how you build and run the team. Labenz hears his own advice reflected back: in AI engineering you do not know what will work in advance, so de-risk by doing the hard part first. Mason’s closing brand value is to stay clear-eyed and “really careful not to believe our own… anything. Life is too short.”