Peer review: democracy, sausages and lotteries

ROYAL SOCIETY

`Like democracy, peer review is the least corruptible approach that we have'

The main job of peer review is to select the best research, the best papers and the best research workers on offer. It is used for making decisions about who to appoint, who to fund, what to publish. It involves large numbers of scientists making judgements about colleagues. Central to peer review is that reviewers are subject to the same disciplines they mete out - that they are scientists active in applying for grants and submitting papers for publication.

Like democracy, this is preferable to a system where a few people (editors, chairmen of grants committees) make all the decisions. It is the least corruptible approach we have. Like democracy, it also has its problems. They are not fatal but they are persistent and give rise to endless debate.

Peer review is expensive, especially in volunteer time. When resources are under sustained pressure, much effort is consumed in sifting many more ideas than can possibly be supported. The system is overloaded. Reviewers have to judge between good and good, which is disheartening and more difficult than choosing between good and mediocre. Decisions about the relative merit of competing proposals could become more capricious as pressures increase.

In a report on peer review published in December 1995, the Royal Society looked at measures being taken to reduce the cost of it (this falls largely on the peer reviewers themselves and their employers rather than on the funding agencies seeking advice). These included approaches to moderating demand so that it did not greatly outstrip the funding on offer; and ways of streamlining procedures, which were generally less controversial provided that they did not involve non-peers making scientific judgements and that panels were used to rank groups of proposals.

This streamlining can be enhanced by electronic communication. Postal delays are deleted, it is often the reviewer's preferred route and it provides a ready interface between all parties.

The second problem is that peer review may have some bias towards safe proposals. Really radical, high-risk ideas are not always well served by a system designed to weed out error. Peer review has the potential to eliminate peaks as well as troughs. Multiple sources of funding, using separate peer review procedures, are therefore important to preserve the vitality of the research enterprise - this increases the chances that the creative maverick will find support from somewhere. There must be scope for funding agencies to take risks.

Third is the problem of misuse of the information made available to peer reviewers, to advance their own careers or thwart someone else's. There is no evidence that such misuse occurs on a significant scale in the UK, but we must be alert to the possibility of it happening at all. If the integrity of peer review is seriously challenged, then we are all in trouble.

Finally, there is the corrosive influence of bibliometrics. There are several useful things that can be done with bibliometrics, but one thing that cannot properly be done is to convert bibliometric data into decisions about the quality of individual people, individual projects or individual publications. There is no substitute for the exercise of personal responsibility in forming judgements about quality. The judgements should be informed by relevant objective data so far as possible, and will be moderated by interaction with other peer reviewers, but personal responsibility remains at the heart of peer review.

Confidence that the public investment in the national science base is well managed can be sustained only if an effective system of peer review holds. This implies a constant effort to improve efficiency and adapt the system to meet scientific needs within the context of evolving policy objectives.

Peter Lachmann is biological secretary and Peter Collins is head of science advice at the Royal Society.