Most science research findings are false. Here’s how we can change that by Akshat Rathi

Follow on Twitter as @AkshatRathi

Akshat Rathi, reporter for Quartz in London reports, "When done right, science remains the most reliable way of finding universal truths. Sadly, as the business of science has grown, the incentives to do it right have not kept up—by one estimate, nearly two-thirds of all published research cannot be confirmed by subsequent studies. In 2005, one groundbreaking analysis even tried to explain why that is, and how to change it."

At least the experiment is reproducible. 
Photo: Flickr/Julien Belli under CC-BY

In the year of its 10th anniversary, the author of the analysis, John Ioannidis, professor of medicine and health research and policy at Stanford University, and his colleague Bob Kaplan, emeritus professor at the University of California at Los Angeles, took questions from the public on Reddit on what can still be done to fix scientific research.

Quartz has condensed and edited the highlights of the discussion.

---

Nate Silver says that two-thirds of published research is unable to be replicated. Is that really true?

For most scientific fields, replication is not done frequently. In fields where empirical replication efforts have been done in a systematic scale, the failure-to-replicate rates are as high as you mention or higher.

For example, in psychology, two-thirds of 100 high-profile papers could not be replicated. In preclinical studies (such as oncology drug targets), failure-to-replicate rates have varied from 75% to 89%. Candidate gene studies failed-to-replicate about 99% of the time when we moved to the genome-wide era with more rigorous methods. But there is large diversity across scientific fields, so I am sure that there are fields where almost everything would replicate and others where almost everything would fail to replicate, if we had attempted replications.

Based on theoretical considerations, I would expect that over time the number of “correct” scientific results increases because we have more scientists, better methods, and more analyses done; but it could be that the proportion of “correct” results is actually declining, because increasingly we are working in more difficult areas of complex research where the yield of true discoveries may be relatively low and the noise is more prominent.

Even though scientists consider reproducibility an important aspect of how science is done, why do they not work towards making science more reproducible?

Sometimes I fear that we have met the enemy—and discovered it is us. To a large extent, decisions about promotion and tenure are governed by peers. And as a scientific community we have undervalued team science, replication, and evidence integration (as opposed to first or last authored original papers).

The tenure and promotion system can be a key determinant of what type of science we get done. If we reward people for the right reasons, this can be powerful. Tenure could be based on a combination of criteria which I call PQRST: productivity, quality, reproducibility, sharing, and translation.
Read more...

Source: Quartz