Shipping Finn: Intercom's Product Playbook for the ChatGPT Era

Des Traynor, co-founder and chief strategy officer of Intercom, recalls the day ChatGPT launched in Dublin: a Slack message from Intercom's VP of AI, an evening spent playing with the product, and a late-night follow-up suggesting they talk the next day. The team was impressed by the model's ability to write a song in the style of Rage Against the Machine about installing a Windows driver. Traynor and the CEO then spoke on Sunday for a couple of hours about whether to rip up the entire AI/ML roadmap.

They decided to go all in even though they had mainly seen impressive song impersonations and Taylor Swift-style outputs. Intercom shipped something before Christmas, had a release in January, launched Finn in March, and then launched it broadly in July. Traynor explains that customer support is in the 'kill zone' of LLMs because the models are conversational and can look up facts, read documents, understand them, and summarize them.

He argues that if Intercom did not lead in support AI, someone else would, and there might not be much support left for the rest. That made the shift an all-hands moment rather than a normal roadmap item. The early joke was that Intercom could finally make support responses sound like Rage Against the Machine songs.

The first tracer bullet was a set of zero-downside AI features inside the Intercom inbox: summarize this conversation, translate this message, expand on this text, or collapse it. These features had an upside if useful and no downside if ignored, because users could simply not click the button. The team knew summarization was real support work, so they built it within weeks.

The 'just don't click the summary button' design also avoided a cost trap. Intercom handles about 500 million conversations a month, so automatically summarizing all of them would have been prohibitively expensive. Traynor jokes that OpenAI's valuation would be higher and Intercom would no longer exist. That forced the team to box clever about what made sense to automate.

GPT-4 beta was the next step because it helped contain hallucinations enough to launch a user-facing bot. Finn answered questions above a confidence threshold. Intercom had to make the bot trustworthy, reliable, and on-topic, and to stop it from giving political opinions or recommending competitors.

Inbox AI came afterwards as a plus-plus version of the original inbox features, including matching Intercom's tone of voice. Traynor says there is more to come in that area, including a significant new look at what Finn can do in the inbox.

Traynor says the core ingredient is a good torture test of misbehaviors and good behaviors, plus an internal sense of the acceptable trade-off. These bars shrink in both directions: the more constrained the system, the less creative it is and the more likely it is to miss answers it would otherwise have got right. The test suite loops back, refers to earlier context, merges contacts, and tries to trick the system into worst-case behavior.

One example is a pizza company being asked for the best way to cook pizza: the model may give a useful answer, but it may also recommend a stone-baked method the company does not support. Another example is a model knowing that sunbeds are legal in most of the world but being used in a country like Australia where they are illegal. Intercom's prompts must tell the model that the supplied context supersedes prior knowledge, and that conflicts between documents should be resolved with a specified weighting.

Intercom has run the same scenarios across GPT-3.5 Turbo, GPT-4, later GPT-4 builds, Anthropic's Claude, Llama, and other models. Trust is not the only dimension: cost, reliability, stability, uptime, malleability, and speed all matter. Traynor says speed is probably the one thing he would fix with a magic wand.

The model choice may eventually be passed to customers: Intercom's default would work one way, but customers could choose another path. For now Intercom picks a model based on the full set of trade-offs. Traynor is not certain the company will keep that decision internal forever.

Traynor revises the bankruptcy joke: bankrupting customers is not a great business. Early ChatGPT was significantly more expensive, and he estimates costs have since fallen by roughly 10x. At the time, 500 million summaries per month were simply unaffordable, which was the first time a good default feature could not be afforded.

Intercom was not in cost optimization; it was still in deep exploration. There were obvious savings like moving calls from GPT-4 to GPT-3.5 or to an open-source model like Llama, but Traynor did not think that was the best use of the AI team's time. AI opportunities were everywhere: reporting, augmenting human agents, and deciding which messages Intercom should send.

Even when cost optimization comes, the levers are model choice, the number of API calls, when calls happen, and latency. Magic-wand features still felt a little like modem-internet days, and latency improvement was a major hope. Traynor also expects Apple and Google shipping local LLMs to make instant AI feel normal.

Intercom's approach is to build the best possible product first and make the economics work later. The company believes technology generally gets cheaper and faster, so Finn improves even if Intercom takes its hands off the wheel. Cost optimization will matter when model progress plateaus, not while OpenAI is still accelerating up the S-curve; Traynor points to a future where GPT-7 is only incrementally better than GPT-6.

Traynor says it is hard to know what is missing because Intercom builds a lot itself. Prompt management is an obvious gap: subtle prompt changes, rerunning tests, versioning prompts against different models, and managing those changes over time. He is not sure it is a billion-dollar opportunity, but he thinks it could be a rich SaaS category.

Other missing pieces are more robust infrastructure. Getting Finn to work in the EU was messier than expected because servers were in America, which led to a relationship with Microsoft. Intercom also did not use Pinecone partly because it already had a significant AI product, Resolution Bot, with vector search capabilities, giving it a running start. Traynor warns that OpenAI could 'Sherlock' tooling startups because it is simultaneously an AGI lab, the AWS of AI, and a consumer company.

Intercom uses a central AI/ML team of about 17 to 20 people, up from about nine when the journey started, surrounded by roughly 150 product engineers. The AI team exposes endpoints, such as answer a question or suggest a next step, and product teams decide how to build them into the inbox or messenger. Traynor distinguishes AI-first companies, companies that use AI to build new products, and companies that only sprinkle AI around an existing product.

Traynor cautions product managers that AI/ML projects add a second wave of uncertainty: after design exploration, the team must ask whether the thing is even possible, and there may never be a clean 'no'. He gives sentence completion in email clients as an example that looked feasible but ran into PII and abstraction problems. That is why he recommends treating AI work as a portfolio of bets, with some 99% bets and some 20-40% bets that may take years to resolve.

Customers are moving from AI-curious to all-in, and Intercom tries to make adoption less of a trust fall and more of a dip-your-toe. Because Intercom had full customer data, it could suggest testing Finn only on free users, only on users who had been free for more than a year, only on weekends, or only when a conversation contained certain words. Finn has handed out over two million answers, and Traynor says it can answer seven-part questions with seven-part answers.

Traynor expects Apple and Google shipping LLM-enriched assistants to normalize talking to software. He compares it to the iPhone making design a competitive battleground: once people see how good AI-enriched software can be, they will become less AI-skeptical. Until then, Intercom goes support team by support team explaining the world they could be living in.

Finn uses retrieval-augmented generation rather than per-customer fine-tuning; tone of voice is handled through prompting, and Finn already picks up tone from the docs and support conversations it reads. Intercom's AI lab has 30-40% of the AI team exploring Llama, building its own model, or training on support data. The product roadmap mostly follows customers, with Finn by email as a current priority.

Future automation depends on the vertical. E-commerce support has seven or eight query types and an extreme Pareto distribution, so a support bot could nearly be hand-rolled. A product like Google Docs has thousands of query types, so it may never reach 100% automation, maybe 80-90%. Not all support is text, either: some queries require actions like issuing a Stripe refund. Traynor imagines a spectrum where some customers want full automation, while others want Finn to propose actions to a human manager, like a self-checkout ID check.

Traynor does not think actions are waiting only on model progress; it is a hard problem with lots of code. There are two paths: hand-code conditions and authentication for a refund, or ask the AI to read Stripe's developer docs and work it out. Either way, authentication, monitoring, and data logging require a lot of software.

Traynor believes entire product stacks will be thrown out because of AI. His friend's ad-optimization startup is an example: users log in, pick winning ads, create new versions, measure LTV, and pick a winner. In the future, that product may not exist as a login; a fixed-fee service could generate, run, measure, and close ads automatically. Not every category dies, though: email marketing still requires sending hundreds of millions of emails, managing reputation, DMARC, spam compliance, and trust. A startup may spend a year reaching that starting line, giving incumbents like Mailchimp, Klaviyo, or Braze time to copy the cute AI design features.

Startups should pick areas where the incumbent's tech stack is irrelevant, so irrelevant that the incumbent would rebuild it entirely differently if starting again. For incumbents, Traynor recommends first finding asymmetric upside and simple features that run water through the pipes. Then break the product into workflows and ask whether AI can reliably do each one above the customer's threshold; if AI can do it, delete the old workflow, and if it cannot remove it but can reduce it to a simple decision, optimize it.

After removal and optimization, a bit of AI sprinkling may be worth doing just to look like a proper AI-complete offering, but it is less important. Then the company must learn to sell the value to customers. Traynor summarizes the algorithm as remove what can be removed, optimize what remains, and sprinkle after that.

Traynor calls productivity tools that write emails or sales pitches overhyped. He expects people and filters to smell AI content, and he thinks people have forgotten what good writing is. He calls creativity underhyped: just as Instagram filters made everyone feel like a photographer, tools for artwork, music, and video may unlock a new type of creativity.

On incumbents, Traynor credits Adobe, Figma, and Miro with finding useful AI use cases, while Apple and Amazon have been surprisingly slow. He contrasts ChatGPT telling his daughter a 10-minute dragon story with Alexa failing a simple song request. He hopes 2024 levels out the gap between primitive assistants and advanced AI.

Jacob Effron and Jordan Segall reflect that Intercom moved unusually fast in a space where many companies only announced roadmaps. They highlight the all-hands decision to rip up the roadmap and go all in, which gave Intercom first-mover advantage. They also like the zero-downside starting point, comparing it to Notion's early AI features: if users ignore the AI, the core product still works.

The hosts discuss prompt management as a gap. They have seen many seed pitches, but every company they talk to is still doing prompt management in a spreadsheet. They mention tools like Braintrust, Gentrace, and Humanloop taking end-to-end approaches, but they are unsure whether OpenAI will build these tools itself.

On cost optimization, the hosts note Intercom is still exploring capabilities and has not shifted to optimization. They wonder when companies will move from 'we know the use case' to 'how do we do this most cost-effectively,' perhaps with a summarization model that is 100 times cheaper. They also like the portfolio-of-bets framing and the idea of rolling out to low-risk users on weekends.

They predict customer support will see early widespread adoption and want a follow-up episode in a year to check whether enterprises have rolled Finn out broadly and whether Intercom has moved to optimization. They then discuss Meta's announcement of a huge compute commitment and Llama 3 as evidence that open-source models will remain competitive.