Supriya Subudhi on Five Lessons From A/B Testing AI at Bumble

I'm Supriya Subudhi, a product manager at Bumble. A/B testing compares two versions of a feature, email or experience to see which performs better. When generative AI and large language models enter consumer products, however, a positive result is the beginning of the investigation rather than the end.

Bumble has more than 20 million monthly active users. As a product manager, I represent users and their problems while remaining accountable for the company's finances: we don't have unlimited resources to solve every problem.

Our example is AI photo feedback. Dating-app users add pictures to a profile, but often don't know what a good profile looks like. We drew on Bumble's knowledge of what works to generate suggestions for improving the pictures people add.

The first lesson is that people can ask for a feature and then ignore it. In research, people repeatedly told us that they didn't know whether their dating profile was good enough and wanted support in understanding how to improve it.

We showed people possible forms of AI support that would help their dating journey without doing the dating for them. About 78% of the people we spoke to said they would love feedback. But in an A/B test, with half receiving the feature and half not, only 10–15% actually used it.

Openness to AI doesn't predict usage. We brought the feature into journeys people were already taking: when someone added a picture, we offered feedback; when someone struggled to find matches, we suggested checking profile feedback. Adoption rose. Measure actual use, not just the survey response, so that you can improve it.

The second lesson is that an average win can hide the tail. The top-line results looked good: more people added pictures and made profile changes than in the group without feedback. But some AI tips told people to smile more when they were already smiling.

The acceptable scope of error depends on the application. A few poor selections from an AI music DJ are different from a banking app recommending a $2,000 expense someone cannot afford. This is why we established guardrails for what our generated feedback should and should not do.

For example, our feedback should not tell people to change things outside their control, such as skin color or height. We also read real outputs. Evaluation tools let us inspect what users were seeing and whether our prompts were doing a good job.

The success criterion changed from a simple green or red overall result to the distribution of experiences. Are women and men responding similarly? Are outcomes consistent across countries and languages? A French response, for instance, might differ from an English one.

The third lesson is the green dashboard and the red bill. We initially chose the best LLM to get the highest-quality, most accurate feedback. It cost roughly ten times as much as a simpler approach, before adding the cost of evaluating the system.

Bumble already uses AI in its matching algorithm, but the cost pattern differs. A traditional machine-learning algorithm can require considerable upfront compute and training, with a relatively small incremental cost per user after it is working. Generative AI bills grow as more people use the product more often.

Before picking a model, understand why the product uses AI. Some teams receive a mandate to add AI, but the budget should reflect whether the intended benefit is immediate or long-term. A short-term return may justify a quick calculation; a long-term strategic benefit may justify additional spending.

Consider Amazon's shopping assistant as an example. Better recommendations might increase near-term purchases, but even if they do not immediately do so, a trusted assistant could eventually make customers choose the platform more often. The objective determines what investment might make sense; this is an illustration, not a claim about Amazon's internal decision.

The fourth lesson is that the win may come from the model. Our initial model was too expensive to scale, so we switched to a cheaper one. Costs fell, but accuracy and the impact of the results fell as well. The A/B win depended on the model being tested, not simply on the concept of giving feedback.

We learned to test extensively and make decisions from data. Evals helped us compare models on different data sets, account for cost, and rapidly retest versions of prompts and feedback. In the end, we arrived at the cheapest option that still produced a good result.

The fifth lesson is that a global rollout may not be possible. AI regulation changes and differs between markets, and noncompliance brings legal and financial risk. We launched the feature in the United States but could not ship the same version unchanged to other countries.

Product managers and engineers alike need to consider legal and compliance requirements early, not after the product is built. One approach is to identify restrictions everywhere and design within them from the start. Another is to optimize for the market with the greatest impact, even if that means launching in only one country.

A team can also pay the extra cost to make the application work differently in different countries. It is a spectrum of product decisions: building the best feature has limited value if users cannot legally get it.

Before calling an AI test a win, ask five questions: Are people using the feature or merely saying they will? Do you read real outputs or just averages? Does the result justify all costs? Can you afford the model responsible for the win? And are you legally clear on the markets where you can ship?