Statistics as Decision-Making: Michael I. Jordan on Bayesians, Frequentists and the Ideas Between Them

Asked, in his own words, an "absurd question" about what statistics is, Michael I. Jordan places it somewhere between mathematics, science and technology. It is a collection of principles that let you make inferences with some reason to be believed, and make decisions where you have grounds to expect you will not err. All of that, he notes, depends on assumptions about what counts as an error and what the probabilities mean. Once those are fixed, you can guarantee that the chance of error is small and the chance that you found something real is high.

Jordan sketches a short history of the field, which he dates to roughly 250 years ago as a formal discipline. It was first called inverse probability, because probability itself had been developed to explain gambling: given a roulette wheel with a certain mechanism, what outcomes should you expect over a long run? Turning that around, people asked whether observed outcomes could reveal the underlying mechanism. That inverse problem was the original subject of statistics.

The name itself, Jordan believes, goes back to Laplace, who worked in Napoleon's government and needed a census of France. Laplace gathered data and analyzed it to set policy, and the word for the study of data for the state, tied to the word "state" in French, stuck. By the time the field was formalized in the 1930s, the people shaping it did not think of themselves as belonging to computer science, statistics, control or economics; they were effectively all of the above.

That shared origin explains why statistics is about more than analyzing data. Jordan points to von Neumann developing game theory alongside decision theory, and to Wald, an econometrician, turning decision theory into statistics. The framing is a loss function describing what you care about, a question, a probability model, and the risk you face for each decision. To this day, he says, advanced statistical curricula teach decision theory first and only then branch into Bayesian and frequentist approaches.

When asked for the most surprising or beautiful idea in statistics, Jordan first mentions James-Stein estimation, a result he calls too technical for the conversation and one that defeats the mind's attempts to understand it. He cites a colleague at the University of Chicago, Stephen Stigler, who wrote a lovely paper treating it as a paradox. Jordan then turns to a bigger puzzle in the field: like quantum physics with its wave and particle interpretations, statistics has two ways of thinking, Bayesian and frequentist, that you have to get used to.

He defines them through decision theory. A loss function takes two arguments, the data and the parameter, and neither is known in advance: the data is random and the parameter is unknown. To say you want the loss to be small, you must turn that uncertainty into something certain by averaging or maximizing over one argument. Averaging over the data, with the parameter held fixed, gives the frequentist risk; averaging over the parameter, given the data you actually observed, is the Bayesian approach.

Jordan argues each view fits different circumstances. The frequentist guarantee, a stamp saying the procedure does the right thing 95 percent of the time across many datasets, is exactly what you want when you build software and ship it to people who run it on data you never considered. The Bayesian says the other datasets you could have gotten are irrelevant, and Jordan agrees with that when you are working with a scientist, learning about a domain and analyzing the one dataset you collected. He does not accept that frequentist guarantees are unnecessary, so the two camps have to fight a little and also blend.

The meeting point, Jordan says, is empirical Bayes. You start with the Bayesian framework, which he calls arguably more philosophically reasonable, and write down the mathematics that follows. Then you notice there are quantities you do not know because the real world is messy, and you ask whether there is a sensible way to plug in an estimate for them. In many cases there is a natural observable to substitute, and a bit more mathematics can reassure you the result is really good.

He values that the Bayesian framework lets you bring in human expertise while the math guides you and, at the end, lets you attach a stamp of approval under stated assumptions. For a more accessible example of a surprising idea he offers false discovery rate. When you make a whole bag of hypothesis tests rather than one, some subset of them become announced discoveries, and you want the fraction of false ones within that subset to be small. That is a different criterion from accuracy, precision, recall, sensitivity or specificity.

Those latter measures mostly have a frequentist flavor, because they go forward from the state of nature, assuming the null or the alternative is true, to the data. False discovery rate goes the other direction: given that you made a discovery, what is the probability that the hypothesis is true? Classical frequentist methods cannot answer that without priors, but the empirical Bayesian plows ahead and finds that some of the needed pieces can be estimated reasonably. Jordan credits the line of work to Robbins around 1960, to Brad Efron's writing, and to the Bayesian interpretation of the false discovery rate by Yoav Benjamini and Daniel Yekutieli, and he calls it a healthy way to think about statistics.