Daryl J. Bem's experiments on psi caught the world's attention, as I posted last month, because he used standard psychology-lab methods to gather and analyze his data. Imagine what astronomers might feel if NASA announced that the Hubble space telescope had found evidence for astrology: How do you scoff when you depend on the same instrument yourself? Some, though, had the opposite reaction. Like the foursome who recently threw this bucket of cold water on Bem's claim. If psychology's telescope yields evidence that people can sense future events, say the authors, then that thing needs a tune-up.

"The field of psychology," write Eric–Jan Wagenmakers, Ruud Wetzels, Denny Borsboom, and Han van der Maas of the University of Amsterdam, "currently uses methodological and statistical strategies that are too weak, too malleable, and offer far too many opportunities for researchers to befuddle themselves and their peers."

At the heart of their case against Bem is a fundamental tenet, which, I think, can be paraphrased as: Parapsychology? Dude, seriously?

If psi is for reals, they ask, why aren't the world's casinos bankrupt? In one of his experiments, Bem found that people could correctly predict, 53.1 percent of the time, which of two locations on a screen would show an erotic picture. That success rate, if applied to the black/red choice in roulette, would provide a very good living for gamblers and eventually bankrupt casinos, note Wagenmakers et al. Yet casinos are still in business. So either "psi effects are not operative in casinos, but they are operative in psychological experiments on erotic pictures," or psi effects are trivial to non-existent. Bem's "eight experiments are not enough to convince a skeptic that the known laws of nature have been bent."

The point here is not that they'd be more convinced by 800 experiments. What's needed to provide "extraordinary evidence," the authors say, is the testing of many alternative explanations, not many experiments. Instead, they say, Bem's procedures tested only two possible hypotheses: That psi is real and that the results mean nothing (the null hypothesis).

When only one hypothesis is tested against null, it's easy to make the logical error known as "the prosecutor's fallacy." That comes up, in real life, when a forensics lab assesses a crime-scene DNA sample and estimates that the odds of finding its particular combination of alleles is, say, one in three million. Suppose, unfortunately for you, the DNA is a match for yours. The prosecutor then argues that this means the odds are one in three million that anyone else but you committed the crime. 

That's wrong: His hypothesis (you're guilty!) requires a DNA match. But that doesn't mean a DNA match requires his hypothesis. Maybe someone else with those markers did the deed (perhaps a lot of people with that genetic signature are concentrated in your neighborhood). Maybe that is your DNA, but you didn't commit the crime. Point is: You aren't convicted because the DNA matches; you're convicted because other possible explanations were tested, and only the one where you're guilty held up to scrutiny. A prosecutor who thinks people cheat on the lottery could throw every winner in jail, the authors point out. After all, the odds against winning are so great, anyone who does it must be guilty! That doesn't happen because winning-by-cheating is even more improbable than winning by chance. If you never compare the won-by-cheating theory to the won-by-chance theory, you won't see this.

In attacking the Bem paper, Wagenmakers et al. seem to be saying that psychology is rife with the prosecutor's fallacy, known also as the fallacy of the transposed conditional. Experimenters often claim they've proven a hypothesis because they've found some data, without exploring how else that data could have been generated. Bem doesn't say what other hypotheses he tested, they write, so they don't know how sound his reasoning is.