The Moral Dimensions of the Research Reproducibility Crisis
The labor of scientists has benefited society tremendously. Advancements in medicine and technology have improved both the length and the quality of human lives. Scientific studies have been and continue to be a crucial part of that process. Science, when done well, is indispensable to a healthy, happy, curious human race. Unfortunately, science isn’t always done well. When done poorly, studies can have disastrous effects. People tend to trust claims made by scientists, and that trust turns out to be unwarranted if something has gone wrong with the research.
Consider, for example, Andrew Wakefield’s 1998 research paper that purported to show that the vaccine for measles, mumps, and rubella (MMR) caused autism. In a press conference, Wakefield called for a temporary halt in the administration of the vaccine in light of the results of his study. Though Wakefield’s claim has been disproven and there was evidence that he engaged in deliberate fraud, there was no turning back the dial on public opinion. A 2015 Gallup poll found that 6 percent of the American population believes that vaccines cause autism. Fifty-six percent of the population was unsure about the causal relationship between vaccines and autism. On the basis of these concerns, plenty of parents since the publication of Wakefield’s study have opted not to take the critical public health precaution of vaccinating their children.
The scientific method includes a bulwark against bad science — reproducibility. We have much better reason to believe that the results of a study are true if the study can be repeated under the same initial conditions to generate the same conclusions. Wakefield’s study, for instance, turned out not to be reproducible. Behind the scenes, however, there is good reason to be concerned that the claims made by scientists aren’t actually attended with the defense replication is supposed to provide. A 2017 survey published in Nature revealed that more than 70 percent of researchers tried and failed to reproduce the results arrived at by other scientists. What’s more, in many cases, replication of published findings isn’t even attempted. This revelation constitutes what many have called “The Replication Crisis.”
A crisis for replication in science is a moral crisis for society at large. Misleading or downright false scientific claims can have disastrous consequences. Science builds on itself. As philosopher Rene Descartes pointed out in the seventeenth century, when a system of beliefs is built on a shaky epistemic foundation, the whole system is suspect. A shaky foundation in the sciences can have an impact on life and death problems. Attempts to reproduce the results in important cancer studies, for example, resulted in the conclusion that fewer than 50 percent of important studies in oncology could be reproduced.
The situation seems dire in the case of psychology as well. The Center for Open Science engaged in the project of reproducing 100 psychological studies. They were only able to reproduce 39. This has serious implications for the ways in which patients are treated, and on the decisions that researchers and laypeople who have read the studies make.
The incentive structure involved in research and publication may provide much of the explanation for the lack of frequency with which replication studies are conducted. It is in the interest of a journal to produce studies that are groundbreaking. Tenure and promotion are based on publication in prominent journals. It is, therefore, in the interest of scientists to spend their time engaging in the most potentially groundbreaking research. Producing replication studies doesn’t fit the bill. The result is that there is no real incentive to produce replication studies. One potential response to the replication crisis is to create a system with more incentive to attempt to reproduce the work of other scientists.
A further related concern is that it is often difficult to find funding for studies of this type. There may be little incentive for researchers to fund replication studies, but there may be even less incentive for institutions to fund them. Potential funders are also looking for the notoriety that comes with publication in a top journal. Again, structural changes may be morally necessary — changes that involve a reward system of some type for funding organizations willing to fund replication studies.
There are other practical and philosophical barriers to replication. Many of these involve arriving at a coherent notion of what, exactly, reproducibility is. Science will have a better chance at improving its track record with respect to reproducibility if we have a better understanding of exactly what the ideals of reproduction look like. Ideally, reproducibility is achieved at more than one stage of the research. First, methods must be reproducible. In order for this to be achieved, all of the key features of the method must be described by the scientists involved in the initial experiment. Though the research method is always described, often it is not done in sufficient detail. The relevant necessary details vary from one research discipline to the next. There is no widespread agreement when it comes to the necessary level of detail. This can be particularly difficult in the social sciences, since key features of the initial conditions involved in the original experiment may not be clear to those conducting the experiment in the first place.
There are also challenges when it comes to achieving reproducibility with respect to results. For example, often, results are represented in terms of statistical significance. It is unclear how margins of error should impact our assessment of whether a study should count as replicated or not.
Finally, there are problems when it comes to what experts call “inferential reproducibility.” Reproducibility of this type involves the conclusions that experts draw in response to data results. Different researchers may be comfortable drawing very different results from the very same data.
Some of the moral burden falls on the shoulders of the layperson consumers of popular science. People love flashy science that they can share on social media. The more controversial a headline, the more likely it is to be widely shared in various forums on the Internet. As a result, poorly designed studies are often passed around and, in virtue of their popularity, are accepted by large numbers of people as scientific fact. Of course, there is a moral obligation to ensure that reproduction happens at an early stage of the process, but consumers of scientific studies also have a responsibility to hone their critical thinking skills. Our communities may have a corresponding responsibility to teach critical thinking skills to children at earlier ages. In light of our unprecedented ability to gain access to research of various kinds and on a sliding scale of quality, information literacy is more important than ever.