Fin Climbed From 35% to 65% Resolution. Intercom Says the Models Only Did a Few Points

Fergal Reid runs a 50-plus-person AI group inside Intercom that includes working scientists and researchers. When a new model drops, someone on the team usually has a hacked version of Fin running against it within hours. Backtests come back the next day, sometimes in five hours. Tracking the outside world is treated as a core competence of the whole org, not a specialist's job description.

That speed does not make Reid an eval maximalist. Six months before this taping, a meme was circulating that every product team needed to specialize in evals as the thing that would solve all their problems. He is skeptical of silver bullets. Intercom has a mature backtesting setup for Fin: hallucination rate, how often the model answers, whether a RAG-style stack finds the right content when good answers exist. The tests are useful. They are not the referee.

The battle-tested rule is to always test in production, with an A/B test at scale. Reid says the team has watched exciting techniques crush backtests and then underperform once they hit the messiness of real human communication. You cannot build a perfect eval for that mess. You can build something good enough to tell you there is a signal. A tenth of a percentage point in resolution rate is something Intercom sweats, and that kind of sensitivity only shows up in massively overpowered live experiments.

They consider that experimental scale an edge. Fin is deployed across more than 400,000 businesses, spanning many large and diverse apps. Smaller competitors simply cannot run the same tests. Reid's advice is as plain as it gets: do not fall in love with your offline numbers.

Nathan Labenz asks where Intercom tries to sit on the spectrum from bleeding-edge to late-but-best. McCabe's view is that lock-in buys time, and Intercom does not have Apple's lock-in. Salesforce has multi-year contracts, a broad platform, and customers who become a "Salesforce shop." Once a disruptor breaks out with early adopters, though, it is unusual for an incumbent to show up later and take the sheen away. He is relatively bearish on older companies catching up in AI categories where momentum has already formed, because building AI, he argues, requires a different culture and talent mix than deciding to ship ABC instead of XYZ.

That thesis changes what they choose to build. In earlier years Intercom had a strong product instinct to ship the sexy thing first. Now McCabe wants the company to remain the one customers trust for new innovation in service agents, without panicking if an unknown startup ships a shiny feature first. Fin already talks on the phone, in chat, and over email, and he finds that surprisingly effective. An obvious next spectacle would be a 3D avatar, a little digital person in the app instead of a chat bubble.

As a builder, he would love to be first. As a CEO, he does not think it moves the needle today. Customers are still absorbing the current product. Someone else will probably build the talking head first. He does not think that will hurt the business. The art is knowing what customers actually need right now, and he is betting that reliability and resolution rate matter more than the demo.

In February 2024 Klarna said it was going all-in on AI and employing 700 fewer full-time people, with millions of conversations and better metrics attached to the announcement. Labenz asks whether that story matches what Intercom sees across a broad customer base. McCabe is careful not to trash another company, but he never believed the narrative. He likes a brave marketing moment when he sees one, and this looked more like a show than a reality. His simplest reason: he did not see anyone else do it. He also thinks Klarna later backtracked.

The pattern he sees matches other disruptive technologies. Before the new tool replaces humans, it increases the supply of work people could never afford to deliver. Companies put Fin in front of free users who never got support. They answer faster, so customers ask more. They add email or chat they never offered. Net service goes up. Intercom itself is a sample: it has dramatically slowed hiring of service agents and has not substantially grown that team since launching Fin more than two years ago. The first human disruption, McCabe says, is eating future headcount, not emptying the current floor.

Reid has not met a support team that is not underwater by about 30% versus the capacity it wishes it had. On day one Fin may resolve 30 to 50% of queries, enough to pull a team from underwater back to roughly even. People then move up the value chain. The major exception is BPOs: customers who outsourced tier-one support often deploy Fin and cut the vendor. Internally, Intercom's support team had been projected to grow 25% in Fin's first year. That expansion never happened. The headcount stayed flat while Fin's resolution rate kept climbing.

Even the 1% monthly climb has a catch. Resolution rate is in the low-to-mid 60s, years from the high 80s or 90s, and companies rarely deploy Fin to every channel where requests arrive. Each extra point is harder work; the easy volume was automated long ago. Ten more resolution points is not ten points of remaining labor. McCabe's longer view is that even at high-90s resolution, 20, 30, or 40% of the work may still sit with humans. Reid adds a warning: CFOs have not really cracked the whip yet. In a downturn, the conversation about post-AI operational efficiency would get much harder.

Labenz notes that Intercom's advertised resolution rate has moved from 56% to 65% in the months he has been reading ads for the product. McCabe says Intercom's own deployment sits in the high 60s, impressive given a sprawling product with frankly too many features. Some customers with a narrower question set are in the high 80s and 90s. Fin now handles a really large chunk of Intercom's own customer service.

Reid describes the climb as a scaling-law, or Moore's-law, phenomenon: about one percentage point, month on month, from roughly 35% at launch to about 65% now. Every six weeks the team lists three things to try. He is usually not that confident. Two work and one does not, or one works and two do not, and the net is that stubborn monthly point. A couple of recent months hit 2%. Only a very small amount of the gain, a couple of percentage points over two years, came from core LLM performance.

The rest is retrieval, rerankers, prompts, and a constantly growing machine of tests. A cohorted view backs the claim that the engine, not customer homework, is doing most of the lifting: recent customer cohorts get a similarly high resolution rate to long-time customers. Some accounts have mature help centers and use every optimization feature, and variance inside cohorts is real. Overall, most of the improvement is in Fin's core.

That is why Intercom has started training its own models. The team built a custom retrieval model and a custom reranker; the reranker beat one of Cohere's top models that Fin had been using. Token prices are falling, but Fin does more work than at launch: harder questions, more context in RAG, a more complex chunking strategy, plus supporting systems such as insights and content checks. The north star is the share of inbound volume Fin resolves, not cost optimization.

Labenz's own company, Waymark, tells the more common story: a lot of quality gains came because the underlying models got dramatically better. Reid says Fin is different, in part because Intercom built the first version on GPT-4 with months of advanced access. That model took them across the accuracy threshold they wanted, but it was a beast that ran on dedicated hardware and forced a careful architecture. GPT-4 Turbo, which many people have forgotten, was much smaller and more efficient at similar power. Moving to Anthropic's Sonnet added a couple of resolution points, which they care about a lot, against a 35-to-65 backdrop where the models are a relatively small part.

He is willing to say it directly: intelligence is not the bottleneck for a large part of customer service. The task is less cognitively demanding than the math olympiad, so a given level of model intelligence saturates it earlier. GPT-4-level intelligence was already enough for most of the work. That is what lets them replace some stages with small models they train themselves.

One of those stages is summarization. End users stuff random extra information into queries, and you do not want that junk hitting the search pipeline and embeddings, so Fin canonicalizes first. The step started on GPT-3.5 Turbo, later GPT-4.1, and for a while Claude Haiku, which they did not love. It is now a proprietary encoder-decoder plus a fine-tuned Qwen3, cheaper, lower-latency, more predictable, more reliable, and higher quality than the third-party LLMs they had used. Smaller third-party models never quite hit the tradeoffs they wanted.

Other parts of Fin are still frontier. Procedures that touch external systems, refunds, address changes, order lookups, have a very high reliability bar, and those still run on Anthropic's Sonnet. The hardest answering prompts also stay on third-party frontier models. Fin was always 10 or 15 prompts; email has a different architecture; Fin Voice is a whole different story. Reid calls the product a cloud of AI services. McCabe's product counterpart is the 3D head: technically possible, not the thing that wins.

If intelligence is mostly off the table and Intercom still has about 30 points of resolution to go, what closes the gap? Reid splits it. Given this article, answer the question as well as a human rep: the models are already human-competitive. Given a pile of external systems and a vague policy about when to refund, do it reliably with common sense, not 99 times out of 100 but 100 out of 100: that is still a frontier task. The way to attack it is the same way they attacked informational queries, using the model as a building block and tuning the envelope around it.

Then there is deployment. Convincing the security team, proving the system is secure enough, and penetrating the organization are slow, messy human factors. Breakthrough technology still has to roll out. McCabe adds that the more systems a product has to touch, the slower it moves through a company. Isolated informational queries were easy to switch on. A drop-in knowledge worker that collaborates across teams, respects permissions, and talks to external stakeholders is a much harder adoption problem. If someone cloned the sophistication of Fin's stack for other use cases, he does not think the base technology would be the thing holding them back. The raw work of pointing the system at a job would.

Reid also thinks the models themselves are missing capabilities. They typically run stateless; bolted-on memory kind of works and kind of does not. They cannot grind on a problem and learn it the way a person does. A new intern will do badly at a task the first time, in a way Sonnet will not, but the intern gets better in a task-specific way. Models do not. There are research prototypes that use reinforcement learning to update weights mid-problem after a long chain of thought. None of that is deployed, and it is risky. Even a country of geniuses in a data center would still need training. For customer support, it is not obvious you want a crazy genius rather than someone with ten years of great service experience.

Labenz pushes on the 100-out-of-100 standard. Waymo is already much safer than a human taxi, with something like a 90% reduction in accidents, and deployment is still cautious. Do customers fail to notice how inaccurate their humans already are? Reid gives conflicting answers because the team spends a lot of time on this. Some customers know their reps issue incorrect refunds at a measured rate and treat that as the bar. If he were a PM at Waymo, though, he would not try to match the human accident rate. He would try to beat it by two orders of magnitude, because new products are held to a higher bar whether that is fair or not.

Regulated customers make the split explicit. If a human makes a mistake, they can talk to the regulator and be understood. If the system makes a mistake, they worry the regulator will not. Other customers run the trial, decide Fin is superhuman, and go. McCabe expects the microscope to stay harsh at first. Once people feel how often the agent is vastly better, they will grant more permission to err. He has watched Waymo riders tell the stuck-behind-a-parked-car story as an endearing quirk. Society, he thinks, will warm to AI's eccentricities.

Customer service is a special case because people already hate reaching a human. They expect to wait hours or days, hit someone who does not want the job, and get a half answer. A snappy, expert concierge that answers in seconds makes them ask more questions. Labenz has the receipts from Waymark: the team responds in two minutes, strong for a small company, and that is just long enough for the customer to tab away. The thread then takes 30-plus minutes to resolve. Instant presence is an advantage humans will not catch.

McCabe has a reversal. If Fin answers too fast, people assume it is a pre-baked macro and do not trust it. If it takes 20 seconds, still far faster than a human, they get annoyed because it is obviously AI. Give it a couple of minutes instead of forcing a reply in seconds, which he has pressured the team to do, and it actually does a better job. Newer models can spend more runtime thinking. He is pessimistic about how fast base quality will jump, but he does expect models to get much faster and cheaper, which should collapse that awkward window.

Labenz asks how they know when to change the paradigm: from a methodically decomposed workflow to the "choose your own adventure" style Anthropic and others are pushing, where you give the model tools and let it pick. McCabe teases Intercom's Pioneer event in New York, later pinned to October 9 and livestreamed. Part of what they will share is a broader shift beyond the Q&A bot, which he still finds limited. These systems will need to become more agentic. He does not think their A/B tests tend to be radical; they live in the middle of the spectrum.

Reid confirms Labenz's synopsis. Fin was deliberately a building-block architecture: isolate the models, test them, optimize them. That has worked, and some blocks are durable. In a RAG system you still need a retrieval engine and a reranker. Switching today to a looser tool-using agent, he says with high confidence, would reduce Fin's quality on question answering. They have built prototypes. If a next-generation architecture looks competitive, they will A/B it in production, the same way they have migrated core Fin through about four generations, some of those shifts comparable in size to the one Labenz is describing.

Fin's 99-cents-per-resolution price is, as far as Labenz remembers, the first pay-per-outcome scheme in his corner of the market, and it is still in place. Under the hood the forces cut both ways: cost per token is down, but Fin spends a larger token budget on harder questions, fatter RAG context, thinking tokens, and supporting products. Reid's answer is that the north star is inbound volume resolved, not a cost-minimization exercise.

McCabe wants to separate Intercom from the negative-gross-margin narrative around AI agents. At launch he was told each resolution would cost $1.21. They priced at 99 cents because the number was attractive, took a hit on every resolution, then turned the dynamic positive and reached software-level gross margins, even while Fin does more work. Intercom's own all-in cost for a human resolution is $26, including salaries, offices, and benefits. Some companies claim $5 or less; he does not think they are doing the analysis correctly.

The design goal was simple pricing that maps to value. Complex, multi-metric packaging had been a nightmare in Intercom's past. Per-conversation or per-query billing would make Intercom richer when Fin is worse. Per resolution, every monthly point Reid's team adds makes customers happier and the CFO happier. Two other companies, Zendesk and Sierra, later announced they were first to outcome-based pricing. McCabe says Intercom was the crazy first. Education was a real cost. When they surveyed customers about switching back to per-conversation pricing, the answer was no.

He cites an old book, *Pricing on Purpose*: value-based pricing yielded higher profit than cost-plus, because cost-plus cannot price-discriminate. Some classes of resolution that save a rep an hour or more will eventually cost more, still cheap, maybe $1.99, $2.99, $3.99, or $4.99. He does not know the number. He does know they will keep it a no-brainer. The insights product, which he calls the best in the market, is still free, because it helps customers use Fin better and raises resolution rate. Science lost; simplicity and the art of pricing won.

Labenz had been surprised by how much Intercom's insights product was already doing in the background. Reid says they did not just dump conversations into a deep-research-style context window. High-volume customers blow past the fidelity of huge windows, so Intercom uses LLMs to turn conversations into topics and subtopics, then a BERT-style medium model to deliver the product at scale. An industrial-strength insights tool, in his view, still has to be built that way. Marketing that depth is hard. McCabe admits that only 26% of customers use a feature Fergal's team killed themselves to build, and that discovery is a chronic vendor problem, not something an onboarding popup will fix.

The product expansion in front of them is bigger than a unused dashboard. McCabe thinks it is plainly obvious that companies will not want multiple agents talking to their customers. A sales agent, a service agent, and an onboarding agent with different vendors, styles, interfaces, and goals will compete with each other. You cannot orchestrate that. Fin is already the leading service agent by customer count and revenue in Intercom's head-to-heads, so the next job is to be the leading customer agent across the whole lifecycle: a high-touch concierge that was never actually possible with humans.

Inside the company, the same tension shows up as a productivity mandate. CTO Dara Curran set a goal some quarters ago to 2x engineering output. McCabe is of two minds on broad AI adoption: firms that refuse will become dinosaurs, and a 1,200-person company told to "use more AI" will also buy a bunch of crap. Reid had mixed feelings about putting 2x in the performance system, because people optimize the metric the wrong way, but busy people will not adopt new tools without a push. He found Claude Code right after public launch, hacked on it late into the night, and demoed it at Friday all-hands. Some teammates said they already used Cursor. His reply: Cursor is great, but this is different.

He does not know if 2x is achievable. He does know designers and engineers now hack together prototypes they would not have attempted. The posture he wants is optimism plus skepticism: try the new thing, then ask ruthlessly whether it made you faster or slower, and double down only when it is real. That is the note Labenz closes on, with Pioneer still ahead in New York and online.