‘Publish or perish’ blamed for extreme results

That is the conclusion of research by Daniele Fanelli, a Leverhulme early-career fellow in science, technology and innovation studies and John Ioannidis, C. F. Rehnborg professor in disease prevention at Stanford University.

The researchers looked at nearly 1,200 studies in health-related fields which were combined into 82 meta-analyses published between 2009 and 2012.

Their findings confirm earlier suspicions that studies based on behavioural, as opposed to physiological, data are more inclined to report “extreme effects”.

This is assessed according to the extent to which each paper’s reported results deviated significantly from the size of effect reported in the meta-analysis, which the researchers assume “should approximate the true effect that primary studies were trying to measure”.

The paper, US studies may overestimate effect sizes in softer research, published in Proceedings of the National Academy of Sciences of the United States of America, does not report the same effect in non-behavioural research.

The authors suggest the exaggeration of results is particularly rife in behavioural studies due to the field’s complexity and its lack of consensus over the correct methodological approaches.

“When choices are less rigidly determined by theory, they are more likely to be influenced, consciously or unconsciously, by scientists’ own beliefs, expectations, and wishes, and the most basic scientific desire is that of producing an important research finding,” the paper says.

The authors find that the tendency to exaggeration was particularly prevalent in US-based studies, which were also more likely to deviate in the direction predicted by the paper’s hypothesis. 

They suggest this is explained by “the fact that researchers in the US have been exposed for a longer time than those in other countries to an unfortunate combination of pressures to publish and winner-takes-all system of rewards.

“This condition is believed to push researchers into either producing many results and then only publishing the most impressive ones, or to make the best of what they got by making them seem as important as possible, through post hoc analyses, rehypothesising, and other more or less questionable practices.”

paul.jump@tsleducation.com