There is a long-standing debate about whether the UK’s Research Excellence Framework is a waste of time and money given its insistence on re-assessing tens of thousands of papers that have already been reviewed by journals. Why not just base REF scores on journal rankings instead?
One answer is that, as Robert de Vries put it in a recent article for Times Higher Education, journal-administered peer review “sucks”. De Vries is conscious, though, that the obvious alternative to journals, post-publication review on subject repositories, might quickly descend into a social-media-style “attention-economy hellscape”, which would be even worse.
His solution is to oblige everyone who publishes on such platforms to undertake post-publication review to ensure that visibility is a function of merit. But I believe that a specific REF repository would be a better solution, eliminating reviewing redundancy while upholding high standards.
UK academics would be able to upload their articles to this hub at any point during the REF cycle. These would be directly reviewed by a REF reviewer and given a score of between one and four stars, as in the current grading system. Of course, assigning these grades would involve more reviewing work than the current REF does, not least because not all papers are currently entered for the REF. But the reviewing would be spread out across the entire seven-year cycle, and many more reviewers could be involved than the overburdened few who, under the existing rules, have to review a large number of papers in a very short space of time.
If an author was happy with their score, their paper would be published immediately on the repository. Alternatively, they could revise and resubmit. Or, if they thought the review was unfair, they could resubmit the article unrevised for review by a different reviewer. This novel option in publishing would prevent reviewers with ideological axes to grind from blocking publication or under-scoring.
If the second review gave a different score, the article would be sent to an arbitration panel, led by a senior REF reviewer. The first two rounds of review would be blind, but the arbitration panel would be able to see the names of the reviewers. If they saw nothing obviously untoward in either review, their final decision might be an aggregate, fractional score. But if they considered any of the reviews to be clearly inaccurate, training would be provided to that reviewer.
One advantage of this system is that it would provide universities with real-time data on their likely REF scores. Even cross-panel standardisation could occur dynamically. For example, a selection of outputs could be randomly sampled prior to release of the output score – much like how an external examiner picks a sample of assessments to review prior to a final award board. Alternatively, a selection of – or even all – outputs could be reviewed by two reviewers: a specialist from the corresponding Unit of Assessment and a reviewer from another panel.
But would the repository really limit redundancy? Wouldn’t UK academics still feel the need to publish in journals in order to preserve their international visibility? Perhaps initially. But if the repository were fully open access and were promoted internationally, it could become the go-to place to find high-quality UK research. As its renown grew, UK academics would feel less of a need to publish in journals.
An alternative arrangement would be to make the repository open access only for people with UK IP addresses, charging those outside the UK for access and thereby generating income to partially cover administration costs. To maintain international prominence in this case, journals would be encouraged to select articles from the repository and sell them around the world in special themed editions (with the authors’ permission, of course). There would be no need for the journals to re-review the articles, freeing up academics who previously worked as journal reviewers to offer their expertise to the REF repository instead. If journals wanted additional expert opinion as part of their publication process, they would have to pay for it – creating a new income stream for academics.
One exciting aspect of the REF repository is that it would also make post-publication review extremely easy to incorporate. As de Vries suggests, users could simply give a thumbs-up to articles they considered to be of good quality, or they might rate them out of four stars, in a way comparable to TripAdvisor reviews. All readers with “reviewer rights” would be registered academics with ORCID IDs, ensuring that the review process remained in the hands of professionals. And to avoid the bad-tempered hellscape of which de Vries warns us, comments would not be anonymised.
Articles receiving more attention would be highlighted to repository users based on algorithms that identified their own research interests. In this way, our articles would reach the people who were most interested in them and, if they were positively received, they would reach even more people because academic journals would pick them up and publish them outside the UK.
I believe that this arrangement would offer a peer-review process that is more efficient, transparent, informative and, above all, fair. This would incentivise research that is more rigorous and creative – to the benefit of academia and society as a whole.
Martin Lang is course leader for MA fine art and senior lecturer at the University of Lincoln.
