The worthy on the unworthy

This book promises to show whether the peer review process is fair, reliable and unbiased and whether it prevents fraud in science or hinders innovative research. It attempts far less. It examines the fate of 449 brief "communications'' submitted to Angewandte Chemie, a high-status German chemistry journal. The manuscripts were sent, with the author's name on, to two external referees and 72 per cent were accepted for publication. Hans Dieter Daniel analyses the data, asking much the same questions as previous writers in the field.

The figures show that the two referees examining each manuscript agreed only slightly more often than they could be expected to by chance. Since there was a higher level of agreement for rejection - they often agreed that the paper was very bad indeed - there was an even lower level of agreement on the quality of most manuscripts.

Hans Dieter Daniel shows that, not surprisingly, a higher correlation is produced if you count some disagreements as agreements. For example he counts a flat rejection as being the same as acceptance with major rewriting even though in this journal a paper may well be judged to be excellent technically, but presented in language too technical for the general reader and so requiring rewriting. My own experience in economics journals is that I am only likely to be asked to make major revisions when a referee is really enthusiastic about what I am saying and wants bits added or expanded. Papers written by professors were more likely to be accepted, but Daniel does not discuss whether this might be because good research in chemistry is done in teams.

None of the papers rejected by this journal was published in journals with a higher impact factor (more citations per paper on average). Daniel argues from this that the editors' decision was "highly valid''. To me all it means is that authors submit to high status journals first, and work their way down. It may be mentioned too that Angewandte Chemie gets many of its citations from review articles rather than from the communications analysed in this book.

He also makes much of the fact that communications rejected by this journal and published elsewhere are cited less often on average than ones accepted. However, he admits that this effect could be purely because they are published in journals with less impact.

Throughout the book there are brief references to conceptual problems raised by other people, but, infuriatingly, they are not taken up, discussed and incorporated into the analysis.

The book asks familiar questions about one journal and comes up with much the same answers as studies of other journals over the past 30 years. The fact that there are no startlingly new answers suggests that the wrong questions have been asked, because it is a startlingly different journal to nearly all those discussed in the literature. Angewandte Chemie does not publish papers but "communications'' of up to six manuscript pages. These are written to establish the researchers' priority and to give publicity to new ideas, and papers are printed within perhaps six months of the manuscript being submitted. The communications are written to be accessible to a general audience. Some 72 per cent of manuscripts are accepted, compared with as few as 10 per cent in other journals discussed in the literature.

Since it has a completely different function to most journals, one would expect that there would be completely different refereeing and editing criteria and outcomes. The fact that referees do not agree here means something quite different to the fact that they do not agree at the American Economic Review, for instance - but this is not discussed.

Many other questions could have been asked. What functions does Angewandte Chemie actually perform? Who reads it: is it academic teachers, academic researchers, industrial researchers or scientists for instance? Each of these would read it for different purposes, which should influence refereeing criteria. They would also have different citation patterns, which limits the validity of the citation analysis used.

Do they scan every issue as it is printed, or do they do a literature search then find the journals mentioned in the stacks? This determines the relative importance of staying power in relation to newsworthiness and style. Are they looking for new ideas hot from the press, for guides to operating new processes, or for solid science?

If one knows how a journal is actually used, in practice, one can start asking whether the editors and referees think that this is what it does and are consciously trying to make it perform this function better. Certainly referees will disagree if they have different ideas of what readers want.

Another line of enquiry might have been to ask referees how they referee. Do they see it as their job to reject what is bad? To reject what is good but inappropriate for this journal? To accept anything that is obviously competent, even it is not very interesting? To screen out people who are not quite sound (this is the justification given for telling the referee who wrote it)? To support their pals? If the referees do not agree on objectives or criteria, they will certainly not agree on judgements.

This is a very brief case study padded out to 118 pages, which would make a useful departmental report at £5 for a small group of specialists. Instead a £25 hardback has been marketed to a general academic market with a misleading blurb. Who referees books, I wonder?

Peter Bowbrick is the author of The Economics of Quality, Grades and Brands.