Cameras Optional: HeyGen's Joshua Xu on Avatars, Orchestration and Building Ahead of the Models

The conversation opens with the moment HeyGen went viral, when people began dubbing a speech by Argentina's president at the World Economic Forum into other languages. Xu calls it exciting to see people recognize the product's value, but the team had no advance warning. In his telling, the company was in "shipping mode" and kept releasing things it found cool.

His internal lesson is that if a team keeps shipping, keeps improving the experience and keeps listening to customers, a breakout moment will eventually find the market. He likes to tell his team that their role is closer to that of a magician, because AI advances keep opening chances to create a magic moment. Video translation, where someone on camera speaks another language with a natural voice and matching expression, was one of those moments.

Xu's own first magic moment was creating his first avatar and watching himself speak on screen. A later one came from daily use: he had a script ready, had a chatbot polish it, and generated an internal product-update video through his avatar rather than filming himself.

Xu describes traditional video production as two steps: film footage with a camera, then edit and post-produce it. HeyGen's original mission was to replace the camera, because many businesses lack access to expensive equipment and many individuals, including introverts, are uncomfortable speaking on camera. Generating the footage with AI, he says, makes the process roughly ten times faster and ten times cheaper.

He says the company began before "generative AI" was a common term, driven by a belief that a scene of video can be encoded as zeros and ones, and that a machine can therefore learn to generate it. That conviction extends to editing. Timeline editors exist, he argues, because cameras were expensive and people shot many takes to pick from; once footage is generated, that structure is no longer needed.

Xu says the team is still exploring what replaces the timeline, probably a mix of writing a script, editing it like a document, and other interfaces. On prioritization, he says everything comes back to AI quality, which is broader than a better loss function: lighting, how the person speaks, and body motion that matches the script all count. Evaluation is subjective, so he pushes everyone, himself included, to build an avatar they would be happy to use in their own work.

HeyGen has more than 40,000 customers, according to Xu, who groups usage into three jobs: create, localize and personalize. Creating means picking a stock avatar or your own and typing text to get a video. Localizing means taking an existing video into more than 175 languages and dialects while keeping voice tone and facial expression and applying lip-sync.

The third use, personalization, lets one video become more than 100,000 variations depending on the recipient's industry or the problems they face, much like personalizing an email. Asked whether the tool targets cutting-edge filmmakers or everyday marketers, Xu is clear it is not for professional editors who already have cameras and skills. It is for content people and marketers who write scripts but cannot produce video.

Teaching users a new way of working is a continuing effort. In the early days HeyGen concentrated on showing that the magic was possible. Because the platform is horizontal, spanning marketing, sales, support, customer success, internal training and individual creators, it now invests in showing use cases by vertical.

For Xu, the central question for any avatar is whether the footage is engaging, because in business the purpose of a video is to deliver a message and viewers who leave after three seconds defeat that. People do not speak only with their mouths, he says, but together with head movement, eyebrows, body motion and gesture. Coordinating all of those pieces is the hardest part of the problem.

HeyGen builds the video layer in-house with a dedicated research team. The recently released Avatar 3.0 renders the entire body rather than only the face. Xu compares the training to a person who watches talking videos every day and then learns to reproduce them, adding that data is one of the biggest pieces to solve alongside model architecture.

A photo avatar cannot capture how you actually talk, so HeyGen offers video avatars: you upload roughly 30 seconds to 2 minutes of footage, and the model learns your talking style. The company is working on capturing different modes, such as presentation or interview, and on recommending an avatar mode based on the script. Full body rendering is the next frontier, and Xu says gestures are not yet included.

On real-time use, an interactive avatar is already in beta and can attend Zoom meetings. The hurdle is inference speed as models grow, but Xu is optimistic that within 12 months much of this will run in real time and some of it on device. He imagines ads that differ for each viewer based on preferences and watch history, instead of everyone seeing the same video.

Asked how pure text-to-video models like Sora and Pika fit in, Xu says HeyGen focuses on business videos and sees two paths. One generates the whole video end to end, pixel by pixel and frame by frame. The other builds an orchestration engine that combines script, voice, music, avatar footage and other components.

HeyGen believes in the second approach because brands and enterprises want control, consistency and quality. He adds that for control, a mouse on a 2D canvas is a better interface than a prompt. Text-to-video partners are therefore treated as an input to the system, with HeyGen's engine on top as the layer customers use directly.

The next big piece he points to is brand personalization. A model like ChatGPT can learn a company's context and write in its tone, but video cannot yet absorb a brand's colors, style and opening clips. Xu pictures giving HeyGen a URL or past videos, having the system learn the brand, and baking that into the final assembly, with user input stored as memory and fed in as embeddings.

On competing with incumbents such as Snap, where he previously worked, and TikTok, Xu says HeyGen is not really fighting for the old market. Those platforms unlocked creators who could easily make video with a phone camera. HeyGen wants to make the camera obsolete and serve content people who do not have one, which he sees as a new market where startups and incumbents can both compete.

He sees a structural dilemma for platforms built around creators. Suppose 10% of content is AI-generated; how should it be ranked against the other 90%? If the ratio moves to 50-50, he says, existing creators would get far fewer views, so a new platform designed for AI-generated content could eventually emerge. HeyGen itself does not plan to build a consumption platform and wants to stay focused on creative tools.

HeyGen's recent enterprise push taught Xu two things. Quality and brand-consistency requirements are much higher, which fits the company's product strategy. And the technology must slot into daily workflows, so HeyGen builds integrations; for marketing that means CRMs and go-to-market tools, and he mentions an announced partnership and app integration with HubSpot.

On trust and safety, Xu calls it critical because HeyGen serves some of the largest enterprise customers. Every avatar creation requires a video consent, and AI checks that the consenting person matches the footage. A dynamically generated passphrase that expires every 10 to 15 seconds makes it, in his words, almost impossible to create someone's avatar without their consent.

Content moderation combines AI review with a human moderation team. The platform policy bans hate speech, misinformation and political campaigns. HeyGen has partnerships with some actors who have built avatars on the platform, but Xu says it has not explored IP much; he finds it interesting that newly generated persons with consistent images could become future IP, though no AI influencers are on the platform yet.

Asked about fundraising, Xu notes that the biggest costs in AI are GPUs and talent, and that the economics differ from classic software. A software company's marginal cost of serving another customer is close to zero, but AI consumes GPU compute with every use. Offsetting that, AI-native teams are more efficient, and he says he gets far more done himself with ChatGPT.

He points to the speed at which earlier AI companies grew, including ChatGPT reaching 100 million users, and expects go-to-market to accelerate because the industry is excited about AI. His conclusion is that, surprisingly, it takes less capital to build a great AI company. On a more generous free tier, he says HeyGen already has one but has not explored pushing it further.

Asked whether anything is held back by inference cost, Xu says no: the team does not wait. HeyGen designs the product for the model capability and cost it expects 12 months out, and what other pieces of the puzzle will have evolved by then.

In the quickfire round, Xu says the speed of AI delivering big business value is overhyped, while its final impact is underhyped. He also describes changing his mind on technology: in 2021 HeyGen invested early in 3D as a guiding layer under generated pixels, but when Stable Diffusion appeared he realized the pixel approach would move much faster because it can train on large-scale data.

Xu takes one or two customer calls a day. He is surprised by how closely customers scrutinize their own avatars. One European customer reported that in videos longer than about five or six minutes, gestures began to drift and no longer matched the script toward the end, which is hard to spot unless a video is long. Outside video, he says the AI product he is most excited about is Google's NotebookLM, which he used to turn his weekly product update into a two-voice audio summary that colleagues loved.

For the long-term vision, Xu imagines having a video agency sitting next to you: you share an idea, they film, edit, show you the result and revise it on your feedback. In five years, he says, everyone will have that agency in their pocket, with HeyGen as the interface. Creative tools open use cases nobody can foresee, he adds, just as nobody in 2012 could picture Instagram, Snapchat and TikTok from the mobile camera, and he says his greatest joy is seeing people create with the tools he builds.