The Truth Wears Off
http://www.sott.net/articles/show/221621-The-Truth-Wears-Off
In recent years, publication bias has mostly been seen as a problem for clinical trials, since pharmaceutical companies are less interested in publishing results that aren't favorable. But it's becoming increasingly clear that publication bias also produces major distortions in fields without large corporate incentives, such as psychology and ecology. [...]
the problem seems to be one of subtle omissions and unconscious misperceptions, as researchers struggle to make sense of their results. [...]
scientists find ways to confirm their preferred hypothesis, disregarding what they don't want to see. Our beliefs are a form of blindness. [...]
Nevertheless, the data Ioannidis found were disturbing: of the thirty-four claims that had been subject to replication, forty-one per cent had either been directly contradicted or had their effect sizes significantly downgraded. [...]
hey quickly discovered that the vast majority had serious flaws. But the most troubling fact emerged when he looked at the test of replication: out of four hundred and thirty-two claims, only a single one was consistently replicable. "This doesn't mean that none of these claims will turn out to be true," he says. "But, given that most of them were done badly, I wouldn't hold my breath." [...]
According to Ioannidis, the main problem is that too many researchers engage in what he calls "significance chasing," or finding ways to interpret the data so that it passes the statistical test of significance - the ninety-five-per-cent boundary invented by Ronald Fisher. "The scientists are so eager to pass this magical test that they start playing around with the numbers, trying to find anything that seems worthy," Ioannidis says. In recent years, Ioannidis has become increasingly blunt about the pervasiveness of the problem. One of his most cited papers has a deliberately provocative title: "Why Most Published Research Findings Are False."
The problem of selective reporting is rooted in a fundamental cognitive flaw, which is that we like proving ourselves right and hate being wrong. "It feels good to validate a hypothesis," Ioannidis said. "It feels even better when you've got a financial interest in the idea or your career depends upon it. And that's why, even after a claim has been systematically disproven" - he cites, for instance, the early work on hormone replacement therapy, or claims involving various vitamins - "you still see some stubborn researchers citing the first few studies that show a strong effect. They really want to believe that it's true."
That's why Schooler argues that scientists need to become more rigorous about data collection before they publish. "We're wasting too much time chasing after bad studies and underpowered experiments," he says. The current "obsession" with replicability distracts from the real problem, which is faulty design. He notes that nobody even tries to replicate most science papers - there are simply too many. (According to Nature, a third of all studies never even get cited, let alone repeated.) [...]
The disturbing implication of the Crabbe study is that a lot of extraordinary scientific data are nothing but noise. ... The problem, of course, is that such dramatic findings are also the most likely to get published in prestigious journals, since the data are both statistically significant and entirely unexpected. Grants get written, follow-up studies are conducted. The end result is a scientific accident that can take years to unravel.
This suggests that the decline effect is actually a decline of illusion. ... Many scientific theories continue to be considered true even after failing numerous experimental tests. [...]
Although many scientific ideas generate conflicting results and suffer from falling effect sizes, they continue to get cited in the textbooks and drive standard medical practice. Why? Because these ideas seem true. Because they make sense. Because we can't bear to let them go. And this is why the decline effect is so troubling. Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions. (Such shortcomings aren't surprising, at least for scientists.) And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that's often not the case. Just because an idea is true doesn't mean it can be proved. And just because an idea can be proved doesn't mean it's true. When the experiments are done, we still have to choose what to believe.
