Journal to rank peer reviewers to improve response times

A behavioural economics journal is trialling a novel and controversial way of making sure peer reviewers submit their reviews on time – releasing a ranking of how quickly they respond.  

Advocates of the pilot scheme by the Journal of Behavioral and Experimental Economics argue that it will reward speedy reviewers, but critics say that it will only make academics more “fed up” of offering “free labour” to journals.

Journals, which generally do not pay reviewers or make their reviews public, normally have no carrots or sticks to incentivise a quick turnaround, and rely on academics’ goodwill.

However, with the peer review system creaking at the seams because of the ever-increasing number of articles requiring review, journals are experimenting with new ways to speed the process along, such as offering payment.  

The journal will release a ranking of reviewer speeds on its website, calculated by the time between an invitation to review and submission. However, the slowest 20 per cent will not be revealed, and the actual number of days taken will also be kept private.

The aim is to “encourage efficient and timely reviewing and to recognize the appreciation for the important work of reviewers”, according to an email sent out to reviewers at the end of last month.

But the idea has attracted criticism, with some academics saying the idea will only antagonise reviewers. Lenny Teytelman, a co-founder of Protocols, a repository for life sciences experimental methods, told Times Higher Education that academics were “so fed up” of offering “free labour” that “saying we’re going to rank you, it just exacerbates the problem”.

The fact that the journal is owned by the publishing giant Elsevier, which has been criticised for what some see as excessive profit margins, has stoked criticism further. “In this case, for a corporation which has really good profit margins, the optics aren’t good,” he added. However, he said that he thought the technique would be effective, even if unfair.

Jim Woodgett, director of research at the Lunenfeld-Tanenbaum Research Institute in Toronto, said that he found the idea “ridiculous”.

“While authors clearly would love to have rapid assessment of their papers, they also want their work evaluated by the most appropriate experts and would, I am sure, prioritise that over whether the review shaved off a few days,” he said.

He also asked: “How is this an incentive? Publishers and editors seem to think that scientists only review for them.”

A spokesman for Elsevier said that the idea was “only a pilot with one journal, alongside many pilots we have to innovate peer review. It’s too early to tell if the journal will continue with this approach in the long term, but reviewer feedback will be monitored over the coming months.”

He pointed out that there had been other experiments with making reviewer speeds “more transparent”. In 2014, the Journal of Public Economics, also owned by Elsevier, released the results of a study that looked at different approaches to speeding up peer review.

Telling reviewers that their turnaround times would be posted publicly was “less effective” than offering $100 (£75) to hit a deadline, but was “still surprisingly successful in reducing review times, particularly with tenured professors, who were less sensitive to cash and deadlines”, it found.

Many journals have a “big problem” in finding peer reviewers, said Dr Teytelman, as “there’s been such an explosion in the number of articles published”. But he said that the best solution was to give reviewers public credit for their work, not rank their speed.

He pointed to one such initiative, Publons, a tool that allows academics to verify how much reviewing they have done so that they can use their record when applying for promotions or funding.

david.matthews@tesglobal.com