Three Fairness Rules That Cannot All Be True

In 2019, University of Pennsylvania professor Michael Kearns and co-author Aaron Roth published *The Ethical Algorithm*. The book is built around a result that is more immediate than whether robots will take over the world: three fairness requirements that each look defensible cannot hold at the same time. That collision sits in front of anyone trying to put a fairness constraint on a lending model, a sentencing model, or a hiring model.

Kearns’s own path is part of the argument. He tells Lex Fridman that his favorite novel is David Foster Wallace’s *Infinite Jest*, and that he started college as an English major after a high-school stretch of gonzo journalism, then discovered that an English major teaches you to read, not to write. He switched to math and computer science, did his doctorate at Harvard with Les Valiant, and has since worked across learning theory, game theory, algorithmic trading, and computational social science. He once jointly sponsored a workshop with the Federal Reserve Bank of Philadelphia. He also has an uncle who is a moral philosopher. That last fact keeps coming back, because it marks the boundary Kearns draws around what an algorithm is allowed to settle.

Early in the technical work, Kearns emailed that uncle a definition of algorithmic fairness that he and Roth thought of as a variant of Rawlsian fairness, and he thought he was asking a yes-or-no question. He got back what he calls the classical philosopher’s reply: it depends; look at it this way and you conclude one thing, change an assumption and you conclude another. That was the moment he saw a real rift. Philosophers have been arguing about justice from a humanitarian angle for more than two thousand years, from Plato’s question of what justice is. Computer scientists have to write a constraint that runs.

He does not claim that algorithms are solving the biggest problems. The book’s metaphor is looking where the light is. If police are racist in who they stop and frisk, that judgment goes into the data, and Kearns says there is “no undoing that” with clever downstream methods. What algorithms can fix are the obvious disasters. A model that never considers fairness is often wildly unfair; a system that never considers privacy often violates it thoroughly. Patch those large failures and the gain is real. Push into the gray, and the argument becomes fair to whom, at whose expense — which is back in the territory of philosophers and legislators.

The contrast with privacy is the part that stayed with him. In privacy, he says, the research community has largely settled on a definition: differential privacy. Fairness is already visibly messier. The literature already contains theorems showing that three entirely reasonable, desirable notions of fairness cannot be had at once. Algorithmic fairness, compared with algorithmic privacy, is going to be the harder problem, and it will have to revisit arguments that many generations of scholars had before computer science arrived. It is very early days.

Fridman asks a plain question: computer scientists will say a sorting algorithm is O(n log n), or that a classifier is 97% accurate on cats versus dogs. Will there be a day when someone can say, with the same kind of force, that an algorithm is 97% ethical? Kearns’s answer is yes — under a specific premise. The book opens with sorting a stack of numbered cards by sweeping through for the smallest, then again for the second smallest. That is an algorithm, and a bad one: quadratic, while the optimum is n log n. Even on a problem specified that tightly, many algorithms exist, with different trade-offs in time, memory, and distributed implementation. Computer science is already used to comparing those quantities. *The Ethical Algorithm* is about the slice of ethics that can be put on that same quantitative footing.

Once a definition is fixed, the math can take over. For a lending model, Kearns says, you can report that the false-rejection rate on Black and white applicants is within 3 percent, and “we might call that a 97% ethical algorithm”; 100% would mean the gap is zero. The hard step is the one the machine cannot do. The user has to decide who to protect and what counts as harm. In lending, a false negative — rejecting someone who would have repaid — may look like the real injury, while a false positive looks lucky. It is not obvious that handing a loan to someone who cannot repay is harmless. Those are not technical answers.

Group fairness has a stranger gap still. If you are the person wrongly denied a loan, every mainstream group definition, Kearns says, essentially tells you that your compensation is the knowledge that the system is also wrongly denying other groups at the same rate. What the person feels is their own loss, not statistical parity. There is a live technical tension between group notions and anything that might feel like fairness to an individual. The field, he adds, is not even at the point of doing worst-case analysis on the single most harmed person. Almost all of the definitions in the literature are what he calls received wisdom: a researcher sits down and invents a technical fairness that people ought to want. His group had only just started human-subject experiments, showing pairs of people in a recidivism setting and asking whether, as a matter of fairness, they should be treated the same. To his knowledge there is not a large literature that elicits ordinary people’s subjective fairness. The same hole exists for interpretability. Everyone agrees deep learning is hard to understand, and that sparse integer linear models are easier, but almost nobody puts a model in front of people and asks whether they understand what it is doing.

A person is a combination of attributes: a racial group, a gender, an age, and a pile of non-biological facts that still strongly shape outcomes. If you require a model to be fair by race, by gender, and by age, each taken on its own, you should not expect fairness for a disabled Hispanic woman over 55 earning less than $50,000 a year. Kearns puts group fairness and individual fairness on one axis. Asking for fairness at the individual level is asking for group fairness on every possible combination at once. Each of us can be treated as a group of size one, because some attribute eventually separates any two people. Researchers know how to give guarantees at the coarse end of that axis. They do not yet know how to give sensible, tractable ones at the individual end. The research program is to creep toward the middle by protecting finer subgroups.

He gave the failure mode a name: fairness gerrymandering. Like political districting, you can keep an aggregate promise by favoring some combinations and discriminating against others. Could a human put weights on the groups — gender 0.6, race 0.4 — and drop them into the objective? Technically yes. Fairness is special, though, because the moment you discuss it you are inside debates that never close: why this group, as retribution for history, as economic opportunity, or something else. Privacy does not split that way. People want more of it; breaches cause damage and bad press; more data privacy is widely treated as better than less. Fairness immediately becomes fair to whom, at what expense to who else.

Criminal justice makes the missing party obvious. Conversations about sentencing, failure-to-appear prediction, and parole are almost always about fairness for the accused. Fairness for victims is not in the constraint. Saying the false-incarceration rate for Black and white people should be roughly the same does not mention potential victims at all. Kearns and Roth learned quickly that they could not do technical work, go talk about it, and expect to be taken seriously if they refused the extra-algorithmic argument about what society should protect in the first place.

Pick a fairness definition — worry about gender discrimination — and a harm, such as false rejection for a loan, and the first thing you can do is audit. From the data, Kearns says, it is easy to report that the false-rejection rate on women is this much higher than on men. Put fairness into the objective and you can trace what he would call the Pareto curve. Error on one axis, unfairness on the other; the algorithm has a knob for how strongly to enforce the constraint. The ideal is the origin, zero error and zero unfairness. Anyone who has trained a model knows you will not get to zero error even with no fairness constraint at all. What you get is a convex curve that says, concretely, what increase in unfairness you take to go from 17% error to 16%. Points off the curve can be strictly improved in one or both dimensions. They are not serious options.

He thinks that curve should be the near-term interface between researchers and stakeholders. Imagine a criminal jurisdiction that cares about racial fairness and also about accuracy: do not release people who will recommit a violent crime, do not keep incarcerated people who would not have. Those goals fight. You can put the curve in front of them and show what overall accuracy in release decisions costs if the false-positive gap between racial groups must be held to a given level. That choice, Kearns says, is not something computer scientists should be deciding for society. Policymakers, regulators, and people in the domain should pick the point. Ignoring the trade-off is still picking a point. You just do not know it, and you will not admit it.

On a parole model, numerical error is the frequency of releasing someone who then recommits a violent crime, or of keeping incarcerated someone who would not have. Unfairness might be the difference between racial groups in those false-positive incarcerations. Why that gap is unfair bleeds quickly into older arguments: equality of opportunity versus equality of outcome, whether a higher reconviction rate for one group is itself the residue of earlier deprivation, whether the algorithm is supposed to correct an injustice that happened years before. Some theories of fairness want a corrective; others want only to level the field right now. The curve does not settle that. It makes the price of each stance visible.

Face recognition is a clean picture of how bias enters without anyone programming malice. If the training set is mostly white men and the objective is overall accuracy, the cheapest way to cut error is to get the majority right, even if other groups are worse. A common reply is that the trade-off is fake: collect more minority images, or train a separate model per subgroup. Kearns’s answer is that those moves have costs too. Rare-group data costs money. Multiple models cost money to build, deploy, and maintain. Even if you dissolve the tension between error and fairness inside the loss, the cost reappears as budget and calendar. Discussions skip that, he suggests, because talking about money is less comfortable than talking about principle.

He also wants to kill a reading of the title. People see *The Ethical Algorithm* and fear the claim is that algorithms should decide social norms and grow their own notions of fairness and privacy. Kearns and Roth are definitely not suggesting that, especially in a moment when people already worry about robots as overlords. The stance in the book is narrower: if society, or the stakeholders in a domain, can agree on those things, there is often a way to encode the agreement. Reaching the agreement is not the algorithm’s job. Computer scientists, in his view, are not in a position to decide for society what the right social norms are. They can show that agreement, when it exists, is implementable.

Fridman points at the heads of companies who already injected values by shipping products. Kearns does not want to be an apologist, but he thinks it is too strong to say explicit decisions were made about polarization. Facebook and its predecessor Friendster started as curiosities; people wondered why anyone would spend time there, just as they wondered what the early web was for when it was mostly hobbyist pages. Engineers and executives made design choices. He does not think anyone ten years earlier anticipated that the platforms would acquire this much influence on political discourse or elections. The scrutiny now is appropriate. Looking back and saying they should have seen it is, he thinks, a little too harsh. These systems are not isolated algorithms operating on unaware users. They are social systems whose technical evolution, because machine learning is involved, is driven in no small part by how people adopt and use them.

A few lines of code will not fix that. He still thinks there are ways to make things better. The platforms already place people and content in a metric space. The same model that estimates your affinity for a post also estimates what you are likely to violently disagree with or be offended by. You could tune a knob toward showing some of that farther-away material. You could even give users a slider: how much content do you want that you might disagree with, or that sits further from your interests — almost an exploration button. Optimizing engagement, he says, is how the industry got here. Doing something different is definitely possible. It would likely cost revenue and profitability, at least in the short term, and if he worked at those companies, optimizing engagement would have seemed the most natural thing in the world. Engagement is measurable. Reducing divisiveness, or maximizing a person’s intellectual growth over a lifetime, is not. He is not claiming a different objective would immediately look like a social good. He still thinks it is worth trying.

If Mark Zuckerberg called and asked him to run Facebook for six months, Kearns’s first response would be no thanks. He does not think he would need to be CEO to try the limited change he has in mind: let people who opt in see a newsfeed not chosen solely to match their interests and beliefs. Conceptually he does not find that alien. The companies already live in a space with distances; the model that makes you as happy as possible also tells them how to make you as unhappy as possible. We are, he says, in something that feels like a bad equilibrium. The systems help each of us optimize myopically and selfishly. From any one person’s chair, why sit in a feed full of things you find irrelevant or offensive? Maybe because everyone let the platforms do that, the collective outcome — political discourse, tolerance of opposing views — is one society is unhappy with. He adds a word for the companies: nobody foresaw election influence or polarized argument when Friendster looked pointless. Those results were learned together, including by the firms.

Asked whether symbolic AI or other traditions might help with fairness, Kearns has not thought about that question in this context, though he agrees in the large that recent machine-learning successes are real and also narrow. He does not think the field is notably closer to general artificial intelligence than when he started. On the more loaded idea that the algorithm itself might explore what should count as fair, he is leery. The more you know about a field, he says, the more you see its limits, and there is already too much we do not know when humans are the ones picking definitions. Algorithmic auditing for unfair subgroups — the gerrymandering case, searching combinations of features for where discrimination is worst — is a narrower, useful statement. Rolling out the idea that the algorithm should decide what is fair fails a cost-benefit test for the field right now, emotionally loaded as the topic is.

Fridman argues that computer science is becoming a basic literacy: philosophers should learn to program so they can scrape Twitter and argue about the meaning of life, and in twenty or thirty years computer scientists will be back in a Donald Knuth-style theoretical core while everyone else does the applied work. Kearns does not reject the trend of other disciplines adopting code. He turns the responsibility around. The influence of computer science on society has magnified exponentially in ten or twenty years. When researchers were tinkering among themselves, they did not need to be citizens of the world. Those days, he says, need to be over very fast. The right move is not to assume everybody else will become a computer scientist. It is for computer scientists to wade out more. One reason they wrote the book in 2019 is that ten years earlier, public awareness of algorithms was too low; you would have had to write a primer first. Now plenty of people with no technical training know enough to enter the nuances.

He floats a bolder guess: in fifty years there may not be computer science departments, because the field will be ambient in every discipline, and a CS department will look like an electricity department. A Knuth-style theoretical core would remain. A discipline marginalized by its own success is not an implausible path. Computer scientists, he notes, have an imperialist streak, like physicists in an earlier generation: no scientific topic is off limits. They usually embarrass themselves for a while. Reinforcement learning is his example; control theorists could fairly say the early work reinvented the 1940s. Eventually computer scientists made real contributions. If they start doing psychology and human-subject research, they should expect to embarrass themselves for a good ten years and hope it turns out as well as the other fields they wandered into.

Algorithmic privacy, Kearns says, is the study of encoding privacy definitions inside algorithms, including the everyday practice of data anonymization. That practice is still the most widely adopted technical solution, and he thinks it is deeply, fundamentally flawed. Anonymization means operations like deleting the name column, reporting age in decades, keeping the first three digits of a zip code, and calling the result free of personally identifiable information. The conceptual flaw is that these definitions pretend the dataset in question is the only one in the world, or that ever will exist.

The Netflix Prize was an early lesson. Netflix released ratings with names removed but kept which movies you scored and when. Researchers joined that file to public IMDb ratings and re-identified people. There was a lawsuit. Kearns’s summary of the last decade is that knowing a few apparently irrelevant things about you can often act as a fingerprint: the ratings you gave ten movies, plus the dates, almost uniquely pick you out of millions of Netflix users. A Facebook study he likes showed that likes alone — no demographics, no friend graph — were enough, in aggregate, to predict sexual orientation, drug and alcohol use, and whether your parents divorced. He also mentions a then-recent paper, in *Science* or *Nature*, on the order of eighteen attributes uniquely identifying a large share of the population. The exact integer matters less than the magnitude: far fewer features than intuition suggests.

Differential privacy, which Kearns credits his co-author Aaron Roth with helping to found, is a much stronger promise. Imagine a database of medical records, one of them yours, used to train a disease model. Compare two worlds: the analysis on the full database, and the same analysis with only your record removed. Differential privacy says that harms from the analysis that included you should be nearly identical to harms from the analysis that did not. It does not promise that data analysis will not hurt you. It says those hurts were going to happen anyway.

The smoking-and-lung-cancer studies of the 1950s are the worked example. If your record was in the analysis, and smoking carried no stigma yet, the published finding could raise an insurer’s posterior that you might have lung cancer and raise your premiums. That is real economic damage. The same analysis on everyone else’s records would have found the same link, because the link is a fact about the world, discoverable from any large enough medical database. Your row was not idiosyncratic or crucial.

The mechanism is noise. Every useful differentially private algorithm is probabilistic: the same input, run twice, yields different outputs. The usual pattern is to compute the exact statistic, then add zero-mean Gaussian or exponential noise so the released number is close to the mean but cannot be reverse-engineered back to any one input. Kearns’s first reaction to the definition was that it was clever and possibly too strict to be usable. The field’s success was to show the opposite. Backpropagation and neural nets, decision trees, support vector machines, boosting, classical hypothesis tests — none of them is differentially private in original form, and each has a noisy variant that is. The scientific stack that depends on statistical modeling does not have to be torn down to give individuals a robust privacy guarantee.

Fridman half-jokes that a campaign claiming Kearns loves cauliflower might hide that he likes spaghetti. People have tried the idea. Helen Nissenbaum’s group at NYU built a browser plugin that, when you queried Google, also sent a torrent of irrelevant searches, so that of a thousand queries from your browser, one was yours. Kearns calls that privacy by obfuscation. It is interesting. It does not give differential privacy. More important, it cannot cover every setting. When he types mountain biking into Google, he wants mountain-biking results, and he wants Google to know that is what he typed. He does not want noise on that query. Medical care, personalization, and maps are the same class: the value of the data is that it is precisely yours. Designing a privacy notion that lets a service use your data exactly, while still leaving you a guarantee, is an open technical problem.

On the larger landing place, he is optimistic: a better compromise than today, with stronger privacy and more individual control. Algorithmic proposals will not get there alone. There will have to be a regulatory and legal process that prods companies to adopt them. He mentions fledgling research on compensating people for selective use of their data. If you are training a model for a rare disease, negatives are easy and positives are scarce, so patients’ records are more valuable for that application and perhaps should be paid more. Today’s internet economics are a two-sided market: publishers and advertisers, with Google and Facebook in the middle, and users not at the table. A three-sided market, with proxies representing users, is not a crazy technical idea. It would have extreme economic consequences. The cliché is still the point: users are the product, not the customer. The customer is the advertiser. If individual control arrived at sufficient scale, it would upend the economic model that has supported the internet, and something else would have to replace it.

Game theory, Kearns says, giving credit to the economists, is a mathematical framework for reasoning about collective outcomes among interacting individuals. You need at least two people to start. The prisoner’s dilemma is the famous case: everyone looking out for themselves gets a worse collective result than cooperation would, and cooperation is not an equilibrium. Algorithmic game theory, now about twenty years old, looks at settings with huge numbers of actors and complicated incentives, and still wants algorithmic ways to predict or influence what happens. The result he is a fan of is Nash’s: under very general conditions a competitive equilibrium exists. Without equilibria it is hard to reason about what might happen, because there is no stability to talk about. Existence does not mean that iterative real-world behavior will get there.

The most important technical contribution of algorithmic game theory, in his view, is the tight connection to no-regret learning. Players each doing something in their self-interest can reach equilibrium in a rather short number of steps. Stability by itself is neither good nor bad. The clean everyday case is navigation. You open Google Maps or Waze, enter origin and destination, and the app computes, given what everyone else is doing, a selfish best response for your driving time. Kearns thinks it is fair to say these apps are nudging everyone toward the Nash equilibrium of that game. Theory, and some limited studies of actual traffic, say that being in that competitive equilibrium can make collective driving time higher — maybe significantly higher — than under other solutions. Social platforms do the same kind of optimization on your behalf. One of game theory’s important lessons is that being at equilibrium does not mean there is not a solution in which some people, or even everyone, would be better off.

Fridman asks about mutually assured destruction as a supposedly stable system. Kearns is a fan of game theory’s history and still finds it surprising that people at RAND in the 1960s drew two-by-two tables with the United States as the row player and Russia as the column player and took them seriously for nuclear questions. It seems to have worked, he says, which is why it now looks naive; we are still here. There were only two nuclear powers then, and you did not have to deter new entrants. The game has more players now. He is not generally someone who advocates simple mathematical models when the stakes are that high and the complexities are political and social. Then he adds: we are still here.

Kearns has watched algorithms take over finance in the same order they take over work elsewhere: first the jobs computers are obviously better at. Electronic exchanges made algorithmic trading possible, because an API order is as valid as a keystroke. The first inroads were optimized execution. A brokerage buying a very large Apple stake for an institution has to split the order across time and across venues or it will push the price against itself. That is optimization and control, and machines, fed historical and real-time data, beat a person at the schedule. High-frequency trading sits next to that layer: spotting fleeting mispricings between exchanges or inside an asset, or reading granular order-book flow. Machines are good at the mechanics of trading.

The current sweet spot for quant funds, and it has been for some time, is statistical arbitrage: a directional view on price or returns that is meant to be valid from a few seconds to a few days. Warren Buffett’s timescale is another species. Kearns’s cartoon is Buffett sitting and asking what Apple’s long-term value should be, not looking at today’s tape, deciding the gap is large, and sitting on the position for ten or twenty years. At that horizon you hold through recessions, cycles, and wars. An HFT model only needs the next 500 milliseconds to stay quiet; news of a CEO scandal is about the only outside shock it must watch. A long-horizon investor needs views on politics, cycles, and social mood, and data on wildly different clocks. A workshop Kearns jointly sponsored with the Philadelphia Fed on machine learning for macroeconomic prediction left him more confident that the field has a long way to go. Asked whose Wall Street job is safe from the bots, he points at people on that longer timescale, with an appetite for the risks of long-term investing. He is not seeing a robo-Buffett anytime soon.

Fridman closes by asking for a day that made him truly happy or profoundly changed him. Kearns answers a slightly different question: a watershed. He went straight from undergrad to a Harvard doctorate with Les Valiant, from an academic family — father a professor, uncle a professor, both grandfathers professors, fields all over the map, including philosophy. The first year he was unhappy. Berkeley had been huge; he had started in English, taken history and art history, and had friends who did everything. Harvard’s computer science department, especially then, was small. He had dropped from a big world into a specialized one, every class computer science, every classmate math or CS. He thought often about stopping with a master’s and going back to the Bay for a well-paid job at one of the big tech companies of that era.

He remembers a late-spring day sitting in Boston Common, chewing over what to do. He realized he was not having a great time, and it felt narrowing, but he had come to do research, to make something original, to carve out a career with independence — and he had not really tried that yet. He decided to stick through the summer. A year later it was clear he would finish. Research had started going well, and it was interesting. He describes the shift every doctoral student makes: from studying what has been done to doing your own thing, and learning your strengths and weaknesses as a researcher. Fridman adds that it is also accepting the pain of the journey. Kearns agrees. That afternoon he told himself he was going to stick around for a while.