Does research need a replication stamp?

Research scandals have come thick and fast in recent years, with everything from p-hacking controversies to alleged data invention making headlines. These stories have consequences; as a social psychology researcher in a business school, those in industry with whom I engage on a daily basis are increasingly asking about the meaningful contribution research can make to management.

At the best of times, it can be difficult to explain what researchers do, how they do it, and at what point what results should be considered reliable enough to share with the world. These are important questions about how researchers mark discovery milestones or communicate them with the research world. All too often a modest step in understanding is portrayed by the media as a momentous step for humanity, rather than small increments that will hopefully provide some insight in the future.

Let’s consider an illustration. In 2013, psychologists Andrew Przybylski and Netta Weinstein published two studies concluding that the mere presence of mobile phones inhibits the formation of a good relationship between two people. Imagine the consequences of such a finding: by just putting your cellphone on the desk during meetings, you could harm the quality of the social interaction you’re having.

Unsurprisingly, these findings spread like wildfire and, for a few years, mainstream media “specialists” pounced on them to back arguments about the nefarious side effects of the cellphone.

In 2018, we sought to replicate these findings, if only to understand why they may occur. Replication is an essential part of research: it is only with recurrent evidence of a phenomenon’s existence that researchers begin to believe in it. In two experiments involving 356 participants, we recreated Przybylski and Weinstein’s original work. We also tracked how creative groups of people who had just met, either with or without a phone being present, performed in a subsequent creative task.

Our results suggest that, in fact, a smartphone’s presence may have absolutely no impact on the quality of relationships or group creativity. At the very least, if there is an impact, it is not as blatant as was assumed so far.

The question is how many minds our research will change. Not only did our findings go against Przybylski and Weinstein’s, they also conflicted with a strongly ingrained point of view in the media at that point. And, as psychologist Dan Gilbert observed, unbelieving something is harder than believing it. Studies show that findings that fail to be reproduced are cited more than findings that get successfully reproduced, while only 12 per cent of post-replication citations of non-replicable findings acknowledge the failure to reproduce results. As a result, research domains – from natural sciences to economics and psychology — end up entertaining a discussion of sensational but non-replicable findings for longer than they should, instead of pushing findings that may be relatively marginal but are more reliable.

It is also problematic when the media do report the fact that research has been invalidated because the subtext is often that the original research therefore had no value. Yet this conclusion is overly hasty. Not replicating a finding can indicate that findings are either unreliable in their original form or culturally moderated. When Przybylski and Weinstein gathered their data more than eight years ago, we lived in a different world, where the presence of mobile devices might have had a deeper psychological impact than in 2018 when we conducted our investigation.

Early research also deserves credit for opening a scientific debate around an important question, even if the original studies don’t stand up to replication. We should also applaud the willingness of authors who pioneer new research questions to take risks.

It takes a village of researchers to tackle any consequential question. A growing number of studies are now casting doubt on the idea that the impacts of smartphones on our behaviour are all negative, including recent work by Przybylski and Weinstein themselves, which suggests that early research may have overestimated the negative impact of digital technology on adolescents’ wellbeing. Today, it is easy to claim that we knew all along that the phone presence effect was not real, but this exhibits a well-researched (and replicated) tendency known as hindsight bias.

What to do? It is certainly important to adopt a more agnostic lens when confronted with novel findings. Both the research community and the media may benefit from using some kind of stamp to signal whether a research finding they mention has been independently replicated – and, if so, how many times. Indeed, the media should reflect on whether it should discuss research findings at all until researchers have independently replicated them a minimum number of times.

As technology accelerates, so does public demand for clarity and simple, sensational answers. But the common belief that new knowledge is born quickly and perfectly formed is a potentially dangerous myth. If we really want practice to be evidence-based, we need to make people comprehend that genuinely game-changing insights about the complex social and business worlds emerge over the long term – and rarely in a linear pattern.

Anne-Laure Sellier is professor of marketing at HEC Paris, one of Europe’s leading business schools.