Peer review should be a two-stage, science-first process

One day in the late 1980s, I walked into the biochemistry laboratory at the University of California, Berkeley, where I was a PhD student, and declared that since Time magazine had a “man of the year” (it was only renamed “person of the year” later), Science magazine should have a “molecule of the year”. It happened that I was well positioned to make the suggestion: my supervisor, Daniel E. Koshland Jr, was the editor of Science.

Why not a “scientist of the year”, he might well have asked. My view was that evaluation should be focused on the science, rather than the individuals conducting it. I disliked how the Nobel prizes provoke so many complaints about who was left out despite doing the foundational research, or the legwork on the specific project – perhaps because they had not lobbied hard enough for recognition. Honouring a molecule would allow everyone in the research area to take pride in the designation.

The response to my proposal was the expected ridicule from my lab colleagues. Nevertheless, some time after I left the lab, Dan called me and said: “Guess what is going to be in the end-of-the-year issue of Science?” I guessed that it was something related to the lab’s research. “No,” he replied gleefully: “Molecule of the year!”

I have a similarly motivated critique of our current review system for scientific grants, and I hope that my suggested solution meets a similarly favourable reception.

Far too much emphasis is placed on who is proposing to do the research and the institutions with which they are associated, rather than on the actual science. Therefore, I recommend that review be conducted in two stages. Reviewers should initially receive only descriptions of the proposed research, written in the third person, with no preliminary results section or indication of the authors’ identities or affiliations. This would require that the proposal be evaluated and scored solely on the detail of its merits.

Of course, some reviewers might still know or guess who the applicants are, but this would be made known to the other reviewers by the requirement to sign one of two statements: Either “I do not know the author(s) of this proposal nor the institutions with which they are associated and have not made any attempt to discover their identities”, or “The authors of this proposal and/or the institutions with which they are associated have been made known to me.” The reviews would be weighed accordingly in the subsequent process.

The second step would see the reviewers provided with the usual biosketches, preliminary results, facility statements and so on, to help them evaluate the capacity of the researchers to conduct the proposed research. There can be no denial that the investigator’s track record is a relevant factor regarding a project’s potential for success. It is, however, inequitable if their identity or affiliation is the dominant factor in assessment. 

Moreover, the intense pressure to manufacture the appearance of productivity in the pursuit of funding can be a potent corrupter of the scientific endeavour. It is possible that if grants were distributed according to the merit of the project rather than the persona of the scientists then the incentives for manipulating the publication and presentation enterprise would be greatly reduced.

The purpose of this two-part review is to reduce the biases, unconscious or conscious, that currently affect the evaluation of proposals. There will certainly be the possibility that bias will still influence the second part of the process, but that bias will be both more evident and easier to quantify if reviewers are forced to justify their assessment of the researchers in a separate process; it might even prompt them to reconsider their assumptions about the nature of the metrics used to measure scientific success.

Accountability could be further improved by attributing each review to its author. Some might object that confidentiality allows reviewers to be more honest, fearing retaliation less. In fact, confidentiality allows reviewers more scope to favour friends, retaliate against foes and exploit their privileged access to the information in the proposal to advance their own research programmes. The National Institutes of Health reports having detected examples of these forms of misconduct.

It is clear that my reforms are not necessarily applicable to every grant proposal process, and they will not solve all of the problems associated with financing scientific research. Nevertheless, they would have demonstrable benefits.

A well-known example is instructive here. It has been stated that in 1970, fewer than 5 per cent of the musicians in the top US symphony orchestras were female. Then the policy of blind auditions was instituted, whereby the applicant plays behind a screen. Now the proportion of female instrumentalists is more than 30 per cent.

Peer review is a powerful method for evaluating research funding proposals. However, improving the fairness and accountability of peer review is a necessity if the modern scientific enterprise is to achieve the more equitable and wider distribution of resources necessary to fund innovative exploration.

David A. Sanders is associate professor in the department of biological sciences at Purdue University.