Most Ideas Fail: Ronny Kohavi on Running A/B Tests You Can Trust

Kohavi's favourite surprising result is an experiment on how ads were displayed in Bing. Someone proposed promoting the second line of the ad to the first line so the title became larger. It looked like a mediocre idea and sat in the backlog for months, until an engineer who kept seeing it spent a couple of days implementing it and launched the experiment.

A revenue alarm fired, as it had in the past for logging bugs, but this time there was no bug. The change increased revenue by about 12%, which Kohavi says was worth roughly $100 million when Bing was much smaller. Crucially, it did not significantly hurt the guardrail user metrics, which is the usual price of raising revenue by showing more ads. It was the biggest revenue impact in Bing's history.

Lenny recalls a similar Airbnb win from opening search results in a new tab, and Kohavi notes that Microsoft ran this experiment around 2008, despite designer pushback, and published it. When he joined Airbnb he found that the lesson had been half-forgotten, which for him shows why organisations must remember their winners.

Asked how often such gold nuggets turn up, Kohavi says they are very rare and most gains come in small steps. He cites a graph in his book of Bing ads revenue per thousand searches improving by small amounts each month, and a Bing relevance team of hundreds of people whose goal is to raise a metric by about 2% every year.

At Airbnb, about 250 search relevance experiments during his tenure added up to roughly a 6% revenue improvement, even though 92% of experiments failed to improve the target metric. His published failure rates are about two thirds at Microsoft, around 85% at Bing and 92% at Airbnb, with other companies such as Booking and Google Ads reporting 80 to 90%.

He clarifies that these are failure rates of experiments, not ideas: about 10% of experiments are aborted on the first day because of bugs, and teams may iterate through several pivots before a successful launch. Every group starting to experiment assumes its success rate will be higher, he says, and "they're all humbled".

Kohavi points to two resources for common patterns: a Microsoft paper called Rules of Thumb, drawn from thousands of experiments, and goodui.org, a site where people submit results that are grouped into roughly 140 patterns with how often each helped. He adds that many companies run lots of experiments and never go back to summarise what they learned.

He defines a surprising experiment as one where the predicted and actual results differ by a lot, in either direction. His example is a Windows indexer that was better in offline tests but drained laptop batteries, a lesson that shaped the next iteration. He recommends a quarterly meeting on the most interesting experiments, a deck of successes and failures, and a keyword-searchable history of all experiments.

The scale he describes is large: when he left Microsoft in 2019 the company ran roughly 20,000 to 25,000 experiments a year, or around 100 new treatments every working day. At that volume, being able to ask whether anyone has already tested an idea is part of the platform.

On the worry that experimentation breeds micro-optimisation, Kohavi says he is a fan of testing everything, since even small bug fixes can have unexpected effects. He advises a portfolio like stocks: many incremental experiments that reliably help, plus some high-risk, high-reward ideas that are likely to fail but could be a home run.

His example of a big bet is Bing's integration with Twitter and Facebook feeds, on which about 100 person-years were spent. It ran for around a year and a half, and experiments were negative to flat. He says it took too long to decide to stop, and that he had mailed Qi Lu statistics suggesting it was time to abort. Lenny notes that Netflix and Airbnb also saw social features fail.

Not every domain is amenable to A/B testing, he adds. You cannot test an acquisition, and you need enough users for the statistics to work. In software, though, once a platform exists the marginal cost of an experiment should approach zero, as it did at Microsoft, whereas Airbnb's less mature platform needed many more analysts.

In his talk on practical defaults, Kohavi says that below tens of thousands of users the statistics do not work out for most metrics. He gives an example of a retail site trying to detect changes of at least 5%, which he says startups should target rather than 1%, and says this needs about 200,000 users.

His rule of thumb is therefore to start experimenting in the tens of thousands of users, when only large effects can be detected, and to get "magical" at around 200,000. Below that, he suggests building the culture, the platform and the integrations so the value appears as you scale.

Kohavi says many people dabbling in A/B testing miss the overall evaluation criterion, or OEC: what are you really optimizing for? Revenue alone is the wrong answer because you can add ads and make more money in the short term while hurting the user experience. At Bing his team ran experiments mapping the number of ads to churn and to time to a successful result, and built an OEC with a trade-off. He also framed it as a constraint: a fixed budget of ad pixels per query.

The key word, he says, is lifetime value: define the OEC so that it is causally predictive of it, using metrics such as retention or time to complete a task. For a lodging marketplace, that could include the rating a guest will give once they stay, which requires predicting something you only learn months later. He suggests long-term experiments for learning and models built from historical data.

His Amazon example is the recommendation email team, which was initially credited for any purchase after an email. That led to spam. His team modelled the cost of an unsubscribe at a few dollars, and more than half of the campaigns then turned out to be negative. It also led to a new feature: by default, the unsubscribe link removes the user from that type of campaign, such as author emails, instead of all emails.

Lenny says his full product redesigns have never produced a positive result, and Kohavi agrees, noting that he has posted examples of big launches that failed. His advice is to redesign in steps, learning at each one: this is one-factor-at-a-time, and of 17 ideas, maybe four are good and would launch. He warns against the sunk cost fallacy of launching a redesign that is bad for users.

He is not opposed to big redesigns, and agrees that breaking out of a local maximum is valid, as long as teams expect to fail about 80% of the time. He tells a story from Microsoft, where he arrived from Amazon and found people denying that half their ideas might fail, saying they had better PMs. Bing's results, and support from Qi Lu and Satya Nadella, eventually brought Office to run experiments.

On shipping, he says a statistically non-significant result is a no-ship, because new code carries a maintenance cost, unless legal requires the change. Even then, he suggests trying three different ways of meeting the requirement and shipping the one that hurts least.

Asked about Airbnb's more top-down, founder-led approach, Kohavi says he is limited in what he can discuss. He notes that his own search relevance team tested everything, while other teams did so less. He believes Airbnb would be in a better state today with more experimentation, though he stresses that the counterfactual is unknowable.

On Covid, he argues that experiments matter even more when conditions change, because you want to know whether a change helps in the current environment and whether it will generalise later. If teams are wrong two thirds to 80% of the time in peacetime, he asks, why would they suddenly be right in wartime? He cites online experiences as a big investment whose early data was not promising and which he sees as a footnote today.

Kohavi says his book, Trustworthy Online Controlled Experiments, sold more than 20,000 copies in English and was translated into several languages, with all proceeds donated to charity. He sees the platform as a safety net and an oracle, and says trust builds slowly and is easily lost. His example is early Optimizely, which let users stop when a real-time P value reached significance, inflating the false positive rate to perhaps 30% instead of 5%, until the problem was fixed.

The most common sign of an invalid test, he says, is a sample ratio mismatch: a design split of 50/50 that comes out at, say, 50.2% on a million users can be astronomically unlikely by chance. At Microsoft about 8% of experiments suffered from it, often because of bots or data pipeline issues, and third-party tools later reported 6 to 10%. His team first added a warning banner, then blanked the scorecard, and finally highlighted every number in red.

He ties this to Twyman's law: any figure that looks interesting or different is usually wrong. If a typical movement is under 1% and you suddenly see 10%, hold the celebration and investigate. Around nine times out of ten, his teams found a flaw, though the Bing ad result was replicated several times and held up.

Kohavi cautions that the common reading of a P value, one minus P as the probability that the treatment is better, is wrong. A P value assumes the null hypothesis is true. To get the probability people want, you need Bayes' rule and a prior, which can be estimated from historical success rates.

The result is the false positive risk, which he says is much higher than 5%. In Airbnb search, with a success rate of only 8%, a result with a P value below 0.05 has about a 26% chance of being a false positive. His team's practice was that results between 0.01 and 0.05 should be replicated and combined with Fisher's or Stouffer's method.

Lenny suggests that launching positive experiments on balance is probably fine, and Kohavi agrees but says the 26% matters because organisations want to share real successes as institutional knowledge. Using stricter P values and replication lowers the false positive rate considerably.

For teams starting out, Kohavi says the key decision is build or buy, and that it is usually both. Today's third-party platforms are good and trustworthy, unlike when he built platforms at Amazon and Microsoft. To shift a culture, he recommends choosing a team that launches often, such as weekly or daily, rather than every few months, and making sure the OEC is clear.

He recalls the microsoft.com team that chose time on site as its OEC. When asked whether more time on a support site was good or bad, half the room said better and half said worse, which he says shows an OEC is bad if people cannot agree on its direction. Success at Bing, and people moving from Bing to other groups, spread the practice across Microsoft.

He wants the marginal cost of experiments near zero through self-service setup, templated metric sets and automated analysis, noting that at Airbnb weak analysis meant many data scientists compensated. To go faster, he suggests a scorecard within about a day and variance reduction techniques such as capping skewed metrics and CUPED, which uses pre-experiment data to reduce variance and thus the users needed.

In the lightning round, he recommends the hierarchy of evidence, from anecdote up to repeated controlled experiments, as a way to judge claims in the news, and asks listeners to use controlled experiments to make data-driven decisions.