Science Fictions: Exposing Fraud, Bias, Negligence and Hype in Science, by Stuart Ritchie

This book is a good primer about many of the challenges that the contemporary scientific enterprise faces. Although it seldom breaks new ground and is much better at describing problems than offering original solutions, I would certainly recommend it to new graduate students.

The chapters are well structured. Illustrative anecdotes, mostly of misdoings, are effectively integrated with analysis that draws on the author’s experience as a psychologist and researcher. They are certainly timely. Paolo Macchiarini, the surgeon whose deadly tracheal transplants are featured in the “Fraud” chapter, has recently been charged with aggravated assault in Sweden. Ritchie mentions the failures of the Karolinska Institute and The Lancet in addressing the misdeeds of Macchiarini and the role of the media in forcing those institutions finally to confront their roles in propagating the misconduct. The focus, however, is on the miscreant researchers. Yet it would appear that the major lesson of the episode is that many academic and publishing institutions have lost sight of their founding goal, the creation and diffusion of knowledge, and have become largely devoted to self-preservation.

Science Fictions excellently performs its task of explaining data analysis and many statistical terms. I especially liked the distinction it makes between “replicability”, the repeatability of results obtained in an independent study of the same phenomenon, and “reproducibility”, the capacity of others to obtain the same conclusions from the original data. Although these definitions are not universally accepted, they are useful.

Ritchie’s explanatory abilities shine out in the chapter called “Bias”. The topic is one that most people are likely to have considered at only the most superficial level but here is discussed in estimable detail. Again, its topicality is underlined by the Netherlands Board on Research Integrity’s recent affirmation that p-hacking and other statistical manipulations by a Leiden University researcher constituted violations of standards of academic integrity. I would just note that Ritchie may himself have fallen victim to the bias he decries. Selective pressures mean that articles claiming that “the data were insufficient for us to draw any conclusions about whether the literature is reliable” are less likely to get published. This imbalance means that Ritchie is probably overestimating the extent of fraud and bias on the basis of the published literature.

The chapter on “Hype” opens with a discussion of the Mono Lake bacteria, which an article in Science claimed could grow by incorporating arsenic instead of phosphorus into their biomolecules. This is a topic with which I am quite familiar, because it was my entrée into a more intense focus on the integrity of the scientific literature. There was failure at the institutions involved (in particular Nasa, Arizona State University and the United States Geological Survey), and failure by Science itself, peer reviewers, the media and TED (TED talks are a frequent target of Ritchie’s).

Yet the hype about the Science article was not the only problem. The data in the supplementary material showed clearly that the arsenate was contaminated with phosphate, as one might have expected. (Ritchie incorrectly refers to “arsenic” and “phosphorus” instead.) This contamination was first recognised, as far as I recall, by Antoine Danchin, Steven Benner and myself. Their own evidence, therefore, indicated that the authors knew their conclusions were unfounded. Yet the article is approaching its 10th anniversary of publication online and, shockingly, has not been retracted.

James Heathers has set up a Twitter account where he takes news headlines about murine health research that doesn’t mention the fact that the studies were conducted on rodents rather than humans and adds the words “IN MICE”. Often it seems that Ritchie’s broad statements about science should be similarly qualified “IN PSYCHOLOGY”. He certainly focuses on psychology when he states that work on the microbiome is the “most hyped” field. The discovery of the microbiome and its role in human health is an incredibly important development in bioscience. The fact that psychologists and neurobiologists have exaggerated the evidence for its role in cognition and behaviour hardly represents a legitimate indictment of all science.

Ritchie discusses the principle formulated by the anthropologist Marilyn Strathern that “when a measure becomes a target, it ceases to be a good measure”. An example is the pseudometric known as the h-index, now much used as a target for evaluating researchers. Given the chapter title “Perverse incentives”, we might even imagine spam email messages offering “h-index extension”.

The documentation provided in Ritchie’s “Notes” is extensive and mostly accurate, and contains important information.

Although I recommend this book as a primer for graduate students, I nonetheless feel compelled to offer a few corrective comments. First, it is unavoidable that there are references to blogs or Twitter accounts that could disappear from the internet tomorrow. If such unstable sources of information are the only ones available, so be it. However, citing Wikipedia as a source of information that can be obtained from a primary source, such as variation in human height, is not an acceptable practice. Wikipedia is certainly a great place to find sources, but it is not itself a citable source. Second, it is a bit troubling that an author of genome-wide association studies provides an inaccurate description of Southern blotting. Finally, in a passage that draws a strange parallel with a symphony by the Polish composer Henryk Górecki, Ritchie seems to blur the distinction between “moonshot research” and “blue-sky research”. The latter refers to an investigation of a phenomenon that has few immediate applications and is at the heart of scientific endeavours.

As I suggested above, the solutions supplied in the “Fixing science” chapter, though probably helpful, are neither original nor likely to be transformative. Two examples. First, Ritchie’s faith in collaborative science is misplaced. In many large multidisciplinary collaborations, it is nearly impossible for any individual researcher to have the expertise to evaluate all the experimental results, let alone the time to do so. There is therefore an inevitable diffusion of responsibility; if there are problems, no one is held accountable.

In addition, as an answer to the problems he has described, it almost seems as if Ritchie is advocating for a science that is nothing more than cataloguing rather than a field of human creativity that is involved in concept development.

Second, the notion that the system of preprints will improve the quality of the scientific literature seems unjustified. Ritchie is aware of some of the limitations of the preprint system but ultimately embraces it. I myself, although not opposed to preprints, am less sanguine. Many of the Covid-19 preprints were truly awful but have nevertheless entered public and media discourse as if they held some value. We also need to consider retraction protocols for preprints, and whether fabrication, falsification or plagiarism in a preprint should be treated as seriously as in an article published in a journal.

Finally, Ritchie misses an opportunity to make a clear, universal statement because of his enchantment with the idea of study preregistration (undoubtedly a positive reform). In my view, investigators who are going to rely on the services of statisticians for data analysis should almost always discuss the experimental design with them before conducting the research; such consultation should not be limited to preregistered studies.

Despite these reservations, however, Science Fictions is a worthwhile effort from which many younger scientists could benefit.

David A. Sanders is an associate professor of biological sciences at Purdue University.

Science Fictions: Exposing Fraud, Bias, Negligence and Hype in Science
By Stuart Ritchie
Bodley Head, 368pp, £18.99
ISBN 9781847925657
Published 16 July 2020

The author

Stuart Ritchie, a lecturer in the Institute of Psychiatry, Psychology and Neuroscience at King’s College London, was born in Edinburgh and grew up in the nearby town of Penicuik. “One of the weirdest coincidences in my career,” he reports, “was that the first person to cite one of my scientific papers was a Canadian academic whose surname was Pennycook – the archaic spelling of my hometown’s name.”

After attending the University of Edinburgh as an undergraduate, Ritchie says he became “one of those boring people who [also] did his MSc, PhD and postdoc at the same place…I managed to work with colleagues from a variety of different fields: psychology, brain imaging, genetics, medicine. This was great fun, and I learned a huge amount – but also started to notice that some of the same problems that afflict psychology research are far more widespread.”

One striking early example Ritchie recalls was when his “very understanding [PhD] supervisors let me try to replicate a notorious psychology study that claimed psychic powers existed”. When their replication effort failed, they “were immediately rejected by the same journal that had published the original. We were told that replication studies were of no interest.”

So what has the Covid crisis meant for the central claims of Ritchie’s book?

“In some ways, the arguments have been very strongly vindicated,” he replies. “The massive clamour to find treatments has led to a great deal of rushed, overhyped, low-quality studies appearing in the literature (and in the news).” Nonetheless, he also sees “two sources for optimism. First, the high-profile problems with science during the pandemic might strengthen the case that we need to change the way we do research. And second, although hardly immune to problems (no pun intended), the development of a vaccine has been far less prone to these kinds of issues.”

Matthew Reisz