Painting a more accurate picture of scientific findings
- November 13, 2015
- Cognitive scientist Vandekerckhove receives NSF grant to develop tool that will make the research process more accurate
The criteria used to decide which academic studies get published and thus reach the
masses have become a concern in the field of scientific psychology and more broadly
in academic research, says Joachim Vandekerckhove, UCI cognitive sciences assistant
“There’s an alarming overrepresentation of ‘positive findings’ in the literature,” he says. “Academic papers are far more likely to be published if the results are statistically significant – a concept in statistics that’s often misunderstood.”
Vandekerckhove, who has a courtesy appointment in the Department of Statistics at UCI, explains that the process statistical significance testing is quite controversial and produces a “devastating threshold for experimental results to see the light of publication.”
Called “publication bias,” the issue means that only studies which show a surprisingly large effect make the cut for publication and thus become the accepted literature and findings on the topic.
“This is very problematic because large effects can easily happen by chance. There could be, say ten studies that found no effect, but one that did, and that one is more likely to be published,” he says. “The result is that published studies are not representative of the real world that scientists study every day, and that skews the research and data on the topic.”
After a recent attempt to reproduce the findings from several major journals in psychology, a large international team concluded that less than 40% of those results were robust. Another team tried to replicate a phenomenon that had been reported in more than 40 experiments in 15 papers, but was unable to in spite of testing 1,600 participants. Vandekerckhove attributes this lack of reproducibility partly to the embattled null hypothesis significance testing procedure itself, which sets a low bar for scientific evidence, but mostly to publication bias causing flukish results to be published. “It's nearly impossible to know how many studies did not obtain a large enough effect to be published, and were relegated to academic file drawers.”
A mathematical psychologist who specializes in Bayesian statistics, Vandekerckhove has received a $260,000 grant from the National Science Foundation to develop a meta-analytic tool for more accurately measuring and undoing such bias in the process. The tool will be based on a Bayesian model averaging technique he calls “Bayesian bias correction.” Its impact will be most apparent in psychology while also branching to the social, health and medical sciences where his approach can be used to identify the effect of publication bias, confirm or falsify null hypotheses, and more accurately estimate effect size. He plans to make his tool available through publications, conferences, workshops, conferences, teaching, tutorials, and most importantly – software available freely online.
“By making the method accessible, this tool will enable all researchers to add statistical mitigation of publication bias to their meta-analytic toolkit, thereby making their work more accurate and reflective of current findings,” he says.
Funding for this work began in September and will run through August 2018.