Trouble replicating cancer studies a ‘wake-up call’ for science

A landmark attempt to replicate the results of dozens of the most influential cancer research studies should be a “wake-up call” to science, a professor has said, as most experiments could not be repeated because methodological data was missing.

Later this year, the Virginia-based Center for Open Science will publish the final two papers of an initiative launched in 2013 with the aim of replicating experiments from 50 high-impact cancer biology papers.

Many replication efforts were dropped, however, as the organisers realised that they needed more information from the original authors because vital methodological instructions had not been included in published papers. Frequently such details – as well as required materials – proved difficult to obtain, often owing to a lack of cooperation from the laboratories.

Subsequent delays and cost increases meant that just 18 papers covering 52 separate experiments, out of an initial 192 experiments targeted, were eventually replicated. Papers outlining the attempted replication of 10 articles have already been published.

The forthcoming papers in the replication study – one detailing the results of all 18 replicated papers, one on why so few were completed – were likely to be a “wake-up call” to science “given that so much information is missing” from published papers, according to Brian Nosek, the centre’s director and co-founder, who is also professor of psychology at the University of Virginia.

Speaking to Times Higher Education, Professor Nosek said that the “biggest trouble [in conducting the replications] was what was available on how the research was done. We had problems every step of the way getting these studies done,” he said.

“Some of the papers we reviewed would have a short methods section for, say, 12 different experiments,” Professor Nosek explained. This meant extensive consultation with the original authors was required; none of the experiments could be replicated based solely on the information contained in the original studies.

“We would generate protocols and how we proposed to run the experiment, then seek clarification from the original authors before submitting [the experiments] as registered reports,” he explained of the lengthy process required to begin repeating studies.

Professor Nosek has previously examined the reproducibility of psychology research, and in a 2015 study found that only 40 per cent of the results in 100 papers he and his colleagues analysed were replicable. This set in train concerns about the reliability of research findings more broadly.

Asked whether the cancer research overview was likely to have a similar impact, Professor Nosek said that the results were “shocking but perhaps not comparable” to the 2015 study.

Professor Nosek added that he hoped any non-replicable work highlighted by the forthcoming study would not be “weaponised” to attack cancer research, or any other research area, as unreliable or error-strewn.

“We have to embrace the idea that science is error-producing but self-correcting – that it is OK to be wrong,” said Professor Nosek, who said errors would occur when researchers were “tackling hard problems [on the] outskirts of knowledge”, he said.

Even if errors were spotted, Professor Nosek argued that replication studies would improve trust in science, rather than harm it. “The reason to trust science, however, is that science does not trust itself,” he said.

“This challenge creates friction [within the scientific community], but I feel we could be making more progress and faster if we improved our research culture and transparency.”

jack.grove@timeshighereducation.com