The accuracy of “scientific” research has recently reappeared in the press. A previous blog (February 25, 2015) addressed the implication of erroneous published medical research results. An earlier blog (July 17, 2014) discussed falsifying peer review processes. The concern is that in today’s instant communication world, information can be widely spread without any checking of the truth. Once something is repeated a few hundred thousand times or millions of times, the concept becomes “fact”.
There is a recent report [Ref. 1] from the National Association of Scholars that considers the issue of the use and abuse of statistics in the sciences. While the paper is lengthy, 72 pages, there is significant information about various improper considerations in reporting research results. The writing is directed at a general audience and does not incorporate statistical equations of detailed mathematics.
In reality, the misleading information can be caused by more than abuse of statistics. Granted that abusing statistics is an important part of creating a narrative that the researchers are promoting. The announcement of significant results without providing information on the size of the experimental data set provides an easy means of misdirecting the readers. The one major medical “mis-Study” that used three related people to demonstrate a desired outcome comes to mind quickly. [Reference intentionally not listed to encourage investigation by the reader.]
Henry Bauer [Ref. 2] points out in his comments on the above-mentioned publication that the real concern is the role of science in society as the source of dissemination reliable information. This information is employed to make decisions that impact many regulations that guide society. I think his comment: “A further complication is that fundamental changes have taken place in science during the era of modern science and society as a whole is ignorant of that.”
There have been various reports in the scientific news that some efforts at reproducing research results have not been very successful. One report indicated that of roughly 100 experiments that were explained in sufficient detail to be reproduced, had a less than 20% success in repeating the results. Some of these efforts and failures were conducted by the original researchers!
In a synopsis of the previously mentioned article [Ref. 3], it is pointed out that a 2012 study of fifty-seven “landmark” reports in hematology and oncology, the findings were not able to be verified in forty-seven of them. And, medical decisions and policy are made on the basis of the studies.
Unfortunately, there seems to be a reluctance to share data so research can be reproduced and the results evaluated. Currently, there is a strong push-back by governmental agencies to keep data confidential in research that is being employed to set regulations.
The question that arises is “How can we trust results being employed to make governmental decisions and regulations if no one can examine the data?” It will be both interesting to observe the results and critical to the belief in the scientific work to see if this credibility issue is addressed.
- Randall, David and Christopher Welser. “THE IRREPRODUCIBILITY CRISIS OF MODERN SCIENCE”, April 2018, National Association of Scholars, https://www.nas.org/images/documents/NAS_irreproducibilityReport.pdf