Yes, peer review sucks. But attention-economy hellscapes would be worse

Peer review sucks. That is the conclusion of a recent viral blog post by the American psychologist Adam Mastroianni. He’s not the first person to say this, of course. Other academics have been beating this drum for years. But Mastroianni has struck a chord with his compellingly unabashed argument that peer review should be abandoned.

It helps that he’s right – peer review really does suck. It does a terrible job of weeding out bad science, but a surprisingly great job of slowing down and tripping up good science. But I want to focus on what comes next. If we scrap peer-reviewed journals, what on earth do we replace them with? Is it simply the case that peer review is the worst system of publication – except for all the others?

The key issue that any alternative system has to grapple with is discoverability. I’ll use my first-ever academic paper as an example. This paper followed the traditional publishing model. I submitted to a peer-reviewed journal, and, after more than a year and two rounds of revisions, it was published. It didn’t set the world on fire, but a steady trickle of citations over the years suggests that at least some people working in my field are reading it.

What would I have done with this paper in a world without peer-reviewed journals? I could have followed Mastroianni’s example and just uploaded a PDF to my website. Except I didn’t have a website. And even if I did, no one would have visited. I could have used social media to promote it, but I don’t use social media because, to adapt an old Stewart Lee joke, “the internet is a flood of sewage that comes unbidden into your home. Social media is like you constructed a sluice to let it in”.

This is a point made by several critics of Mastroianni’s piece. In a world without journals, a paper’s visibility will be determined largely by its authors’ ability and willingness to generate attention. A paper by a second-year PhD student with zero social media game would almost certainly sink without trace.

So without peer review, how will we avoid being swamped by an ocean of dreck? How will we prevent the devolution of scientific publishing into a YouTube-style attention-economy hellscape?

A good place to start has got to be the existing system of preprint publishing. Preprint repositories, such as the physics arXiv and its disciplinary imitators, are in essence minimally filtered databases of research papers in various states of completion. We could simply abolish journals and ask researchers to upload their papers to these repositories instead; however, the result would be a discoverability nightmare for the reasons we’ve already covered. Instead, I believe that a truly viable system would need at least three additional features.

First, there must be a way to assess and communicate research quality. The obvious way to do this would be to allow readers to publicly comment on and rate research papers. This is a form of post-publication peer review, which allows readers to easily see how a paper has been received by other scientists (unlike traditional pre-publication reviews, which typically disappear after a paper is published). This is not a new idea, but it is likely to play a much larger role in a world without journals.

Incidentally, post-publication review also limits the power of incompetent or malicious reviewers. In the current journal system, these reviewers can block a paper from being published at all. But under the post-publication model, they can only leave a negative public review (the merits of which other readers may judge for themselves).

Second, we will need to fall back to a much older conception of the academic journal – not as a venue for finished research products, but as a forum for scientists to talk to each other. These forums could be implemented as separate community-run “channels” on central repositories (different from “overlay journals”, which involve editorial oversight). Each would ideally be quite niche – formed by a community of scientists as a venue for discussing a single topic, or even a single hypothesis. This would help keep the flood of new papers manageable.

Finally, we need a way to break the link between the visibility of research and the ability to grab attention. Quality metrics derived from post-publication review would help: positively reviewed papers would float to the top of their respective forums (and those with rave reviews could be escalated to a more generalist channel – replicating the function of journals such as Science and Nature).

But authors would still have to hustle to get any reviews in the first place (a situation familiar to any Amazon seller or YouTube creator). To solve this problem, every new paper should be sent to random forum members for review. To retain posting privileges, forum members would have to review a small number of submissions, say every few months. These “reviews” could be as simple as a thumbs up, to signal to other community members that a paper is worth their time. These mandatory reviews would provide crucial visibility to those least able or willing to play the attention game.

I am not claiming that this is a perfect system – there will inevitably be problems I’ve not thought of. But the question we should ask of any new publishing model is not “does it have flaws?” but rather “are the consequences of those flaws worse than those of the system we already have?”. As Mastroianni so persuasively showed, this is a much lower bar than many people realise.

Robert de Vries is senior lecturer in quantitative sociology at the University of Kent.