The User Research Reckoning: Why Most Research Is the Wrong Kind

Judd Antin, who helped build the user research practice at Facebook and later led research and design teams at Airbnb, wrote an essay arguing that the user research discipline of the last 15 years is dying. He says the post drew mostly positive reactions, though he now thinks the word "reckoning" invited more drama than he wanted. The most surprising critique came from people who felt that asking researchers to care about profit was not research's job at all.

Antin says the trigger was the wave of layoffs that hit UX and UX research particularly hard. In his view that is a sign the system is more broken than people assume, and that research is not driving the value or impact it could. Some of the blame lies with what researchers do, but a lot lies with how research is integrated and positioned inside companies.

He ties the problem partly to the zero-interest-rate era, when it was easy to hire researchers without knowing what they were for. They were set up as a service function, and when that era ended the weakness showed. Antin is clear that he does not think research itself is dying: good researchers are more impactful than ever, but in a new space.

Antin describes a framework of macro, middle-range and micro research. Macro work is strategic and forward-looking: competitor and market analysis, TAM studies, long-horizon "concept car" projects. Micro work is tightly focused, such as usability testing and digging into A/B test results to ship a polished product.

The middle range is the blurry zone of general user understanding, for example how Airbnb hosts feel about their payment options. These questions are interesting, but they are not framed in the language of the funnel, the OKRs or the business strategy, so the findings are hard to operationalize. They also invite the response "that was obvious, we knew that already" and let teams feel customer-obsessed without changing anything.

Antin adds that researchers often do middle-range work simply because it is what they are asked for most. Lenny, speaking as a former PM, admits that this is exactly the kind of question PMs bring. The result, Lenny notes, is research that delays decisions or becomes a way for product teams to avoid making a call on their own.

Why do PMs keep asking for middle-range studies? Antin says it is because they have not worked deeply enough with a researcher to get the most from them. His answer is simple but not easy: restructure product development so research is part of the process from beginning to end. A researcher in the room can help frame exactly the right question, run something quickly and report back.

He describes a vicious cycle. Companies hire researchers into a reactive service role and include them only at the end. Without input on the questions, the researchers produce reactive work that matters less, executives conclude research is not impactful, and the team is sidelined or laid off. Antin is confident that a great researcher inserted consistently into a product process will drive product improvements, metrics and growth, but he says that is the exception today.

His own success metric follows from this. He never wanted research teams to have separate OKRs; PMs, engineers, designers and researchers should share the same measures. The real test is whether the team refuses to have a decision meeting without the researcher present.

Asked how to tell a great researcher from a weak one, Antin says the best are multi-method. He lists five tools: formative or generative research that looks ahead and is open-ended, evaluative research such as usability testing, rigorous survey design, applied statistics, and a technical skill that used to be SQL. In an age of dashboards and generative AI, he suggests the fifth tool may now be something like prompt engineering.

To hire for this, he gives candidates a juicy, open-ended research question and watches how they handle it. A strong answer is multi-method and realistic about time: here is what we could do in a day, a week or a month. Not every researcher is equally deep in all five tools, so Antin built teams where individuals are T-shaped and the team collectively fills the gaps, at Meta and at Airbnb.

He also says communication is one of the things he weighs most when hiring research leaders. Rigorous work is not enough if nobody hears it, and the presentation needs to fit the audience, whether that is a PM or Airbnb's CEO Brian Chesky.

Antin argues that UX practice is steeped in empathy and user-centeredness, which is good, but researchers need to be explicit about the overlap between what users need and what makes the business money. When researchers ask how to become more profit-focused, he asks whether they have read the last quarterly report or listened to the shareholder call. Usually they have not, because it is full of language they do not understand, and that language is exactly what they need to learn.

He recommends digging through internal folders for this quarter's and next half's strategy documents, learning the OKRs, and knowing the conversion funnel forwards and backwards. With that knowledge a researcher can say precisely where in the funnel work is needed, or which competitor is winning with a given group of users. Antin notes the same challenge runs the other way: many PMs sit outside that overlap too and only perform customer centricity.

Antin coined the term user-centered performance for customer obsession that is symbolic rather than aimed at learning. It is work teams do to signal to each other how customer-obsessed they are, not because they want to change a decision. The clearest example is a PM asking for a quick study late in the process "to validate our assumptions": it is too late to matter and the real goal is to check a box.

Another example is the executive listening session or focus group, which Antin says is well-intentioned but roughly "97% performance". The implicit forms come from confirmation bias and ego. His mantra is that research should falsify, not validate: a PM who really wants to learn says "I want you to show me where we were off base" so the team can fix it quickly.

Lenny reads a line from Antin's essay noting that PMs, designers and executives each ask for middle-range research for their own reasons, and in the end decide on their own opinions anyway. Antin says intuition has an important place in product work, but it is also where blind spots live. He adds that designers are just as guilty, since user-centered design and design thinking made the performance a ritual.

Antin, who taught a leadership class at UC Berkeley, describes the gut as the pattern-matching part of experience: heuristics that let leaders judge well without being able to say why. The same place is where cognitive biases live. His advice is not glamorous: engage System 2, the slow, analytic thinking from *Thinking, Fast and Slow*, to check the gut, though slow here means a short deliberate pause, not months of analysis.

The other tool is the wisdom of the crowd, which works only when people bring different information and judgment. To check a gut, he says, gather several different guts and have an open, direct conversation where disagreement is welcome. Antin says researchers are especially good at running that conversation, and when they are next to the team all the time they expose blind spots before they turn into product mistakes.

Antin collected tropes that researchers hear from PMs. The first is that research is too slow; he calls this wrong, since a good team can deliver in a day, a week or a month depending on the depth needed, and getting it wrong and fixing it is slower. The second is "I can do my own research": talking to users is fine, but anyone can talk to a user, and one user can be idiosyncratic. A researcher helps turn that input into something reliable and avoids the bias of just asking your cousin's family.

The third trope is "A/B test everything". Antin says A/B tests are the closest we get to causal claims, but they rarely tell you why a result happened, and teams end up guessing and running an endless flywheel of experiments. He says you need both: data scientists for causality and researchers for the hows and whys, ideally working together from the start. The fourth is the Henry Ford "faster horses" quote, which he says is probably apocryphal and in any case describes something a good researcher would never do, which is ask users what they want.

The last trope is "we already knew that, it was obvious". Antin points to Duncan Watts's book *Everything Is Obvious Once You Know the Answer*, which describes hindsight bias and the narrative fallacy: we selectively remember and construct neat stories that make us feel we already knew. Responding to a Patrick Collison tweet, he adds that research should build an evidence-based mental model and a framework for experimenting, not hand down exactly what to build.

To show how fast micro research can be, Antin tells the story Airbnb called the multimillion dollar button. Research revealed that people were not moving down the purchase funnel because the button's call to action made them afraid it would start a purchase when it only took them to the next step. With help from the content design team, Airbnb changed seven characters, tested other calls to action, and added about 1% to conversion, which Antin calls very hard to do.

His Facebook story is about a small sample. Around 2012 or 2013, engineers wanted to treat the posts hidden most often as the signal for bad posts. Antin found that hiding follows a power law, so he invited a "super hider" for a traditional think-aloud interview, with engineering directors and product people watching behind the glass.

The participant hid every story after reading it, going for an inbox-zero model even though the feed scrolled infinitely. The people behind the glass had their minds blown, and Antin says an N of one was enough to make them stop thinking naively about hides as a signal and find a better solution. Lenny adds that his own team's focus groups in Paris showed hosts there were very reluctant to connect Facebook, which made the same point about the value of seeing real people.

For researchers, Antin's advice is to develop diverse skills, learn the language of the business, use insights like a scalpel, and build two-way relationships, while saying no to focus on fewer things done better. He knows many researchers who are the only one on a team with seven PMs and twenty designers. For companies, he says to stop siloing insights functions: at Airbnb he set up an integrated insights group that combined UX research, market and competitor work, data science and customer service feedback, including the NPS program.

For PMs, the answer is to create a process that includes researchers and insights from beginning to end, and to help prioritize. Antin says a full plate for a researcher is about three things, two big projects and a small one, and a project can take as little as 48 hours. Great PMs also go into the field with researchers.

On how many researchers a company needs, Antin says there is no clean ratio. His principle is relationships: you have enough when everyone who needs a research partner has one, and it is better to create pain by protecting a researcher's time than to spread them thin. He says partners who feel the loss will argue for more headcount on his behalf. He also notes that only one researcher appeared on Lenny's list of the first ten employees at about 20 B2B companies, which he calls messed up, because a researcher with a broad toolkit helps startups avoid guessing.

Asked about NPS, Antin calls it the best example of the marketing industry marketing itself. He says survey scientists broadly agree it makes the classic mistakes: an 11-point scale, usually unlabeled, with more options than precision supports and many below the fold on mobile. The likelihood-to-recommend question is also flawed, since most people do not recommend an operating system to friends.

His alternative is a simple customer satisfaction question such as "Overall, how satisfied are you with your experience with Airbnb?". Working with Mike Murakami, who led survey science at Airbnb, he redid the analysis and found CSAT had better data properties, more precision and a stronger link to business outcomes. He also says NPS benchmarks are not apples to apples because of inconsistency in how it is asked.

Antin ends with a contrarian note on dogfooding. Every PM should use their own product and walkthroughs are a good way to list issues, but PMs are nothing like users, and some problems need the user's context to recognize. Prioritizing which problems matter and for whom is where you should be extremely wary of your own intuition.