The grey zone: How questionable research practices are blurring the boundary between science and misconduct

Earlier this year, Paolo Macchiarini – former star surgeon and professor at Stockholm’s Karolinska Institute – was dismissed from his post following a high-profile investigation prompted by a documentary broadcast on Swedish national television. Macchiarini was found guilty of failing to secure ethical approval for experimental transplant techniques and misrepresenting data in journal publications. The scandal continues to resonate throughout the scientific community and has so far led to the resignation of the vice-chancellor of Sweden’s most prestigious medical university and the secretary general of the Nobel Prize committee, among others.

Such egregious breaches of scientific protocol are serious, but mercifully rare. Far more prevalent – and therefore even more damaging – are research practices that fall into an ethical “grey zone” between overt misconduct and scholarly best practice. Academic misconduct refers to forms of fabrication, falsification and plagiarism (FFP) – in other words, the terrain of fraudsters, con artists and cheats. Questionable research practices (QRPs), however, are more difficult to pin down but typically involve misrepresentation, inaccuracy or bias. Recent research suggests that academics are becoming more adept at employing QRPs that skirt around the edges of misconduct, like athletes who optimise their performance with artificial enhancements without technically breaking the rules. To put it into perspective, one study found that only 2 per cent of scientists admit to FFP, while almost a third admit to engaging in QRPs.

One prominent example of a QRP is “HARKing”, standing for “hypothesising after the results are known”. Normally, researchers follow the standard scientific practice of developing a hypothesis and then testing it against the facts. But HARKing involves constructing or changing a hypothesis after the data have been collected and analysed. If this is concealed from journal editors, the integrity of the scientific process is compromised. Yet, strictly speaking, HARKing is not considered academic misconduct, even if it is frowned upon by many researchers.

In a new study, we highlight the problem of questionable research practices in the business school. Management scholars tend to publish articles that use the hypothetico-deductive method to show the effects of, say, leadership styles on organisational performance. Here, the study of management is seen as a scientific discipline that adheres to rigorous methodological standards. By the same token, journal editors will retract published articles if they fall short of these standards.

However, we found evidence that QRPs are widespread in this field. For example, it is common for researchers to play with numbers to get the best (read: most publishable) outcome. This could involve removing outliers to confirm their hypothesis or fishing within the data to find unanticipated results. Such practices fall into the grey zone if they are hidden from editors and reviewers during the peer-review process. Unlike FFP, QRPs are more difficult to detect. As one respondent acknowledged, “I can just delete like 100 data points [and] you would never know it. How would you know? How would anybody find out?”

Consequently, highly ranked journals may be flooded with papers with results that are simply too good to be true. This was aptly shown by a recent study that identified a “chrysalis effect” in business research – that is, how subpar results in doctoral dissertations miraculously transform into beautiful peer-reviewed publications. The implication is that researchers manipulate their hypotheses or misrepresent their data to meet the exacting standards found in academic journals.

The most common explanations for the prevalence of QRPs include inadequate methodological training and the pressure to publish. However, the chrysalis effect points to another explanation: the demands and expectations of highly ranked journals encourage researchers to engage in questionable research practices. This is highlighted by one of our respondents, who reported that journals sometimes insist that “you come up with a hypothesis that you didn’t have before, and then test it and then report that as a confirmatory analysis in the paper – which is actually not allowed”. There is, of course, a paradox here. To live up to the unrealistic ideal of science promoted by top-tier journals, scholars may find themselves transgressing this ideal.

The question is what should be done to discourage QRPs. The standard answer is for journals to improve the ethical guidelines for authors and strengthen the peer-review process. Others have called for more openness in science, such as publicly registering hypotheses or establishing central data repositories. But this does nothing to address the role that journals play in fostering QRPs.

One alternative is to develop a “transparency index”, as proposed by the founders of the Retraction Watch website. This would provide a numerical metric of the journal’s transparency in a number of areas, such as peer review, retraction record, mechanisms for detecting misconduct and procedures for dealing with questionable research practices. The transparency index could also determine whether journals compel authors to acknowledge changes to their method or hypothesis during the research process. Establishing such an index would refocus attention on how journals often serve to reproduce – or even foster – bad academic habits.

Of course, most journals would be unlikely to adopt this index if it clashed with their much-prized impact factor. So we may continue to rely on post-publication peer-review websites such as PubPeer to hold researchers and journals to account.

Nick Butler is assistant professor in the Stockholm Business School at Stockholm University, Sweden. Helen Delaney is senior lecturer in management and international business at the University of Auckland, New Zealand. Sverre Spoelstra is senior lecturer in the department of business administration at Lund University, Sweden.

Write for our blog platform

If you are interested in blogging for us, please email chris.parr@tesglobal.com