Citation pitfalls ahead

Charles Oppenheim (Letters, 21 February) says previous studies "have shown excellent correlations between research assessment exercise scores and citation studies and therefore indicate citation analysis would not greatly affect overall order of RAE score". But such correlations are consistent with individual outliers. A demonstrable correlation is no guarantee that citation metrics will reliably replicate rankings based on quality judgments for individual department. Since the point of the exercise is to allocate funding to individual departments, such discrepancies should not be glossed over.

There is a deeper problem. Quality of research is not the only factor influencing frequency of citation. In my discipline, some topics receive far more attention than others: for example, publications on Homer are more than 30 times more numerous than those on Heliodorus. So if you want to maximise citations, give up on Heliodorus - especially if you are engaged in research with a long-term payoff. Citations accumulate slowly in Classics, so you will need to focus efforts on provocative interventions in current debates to stand any chance of gaining citations that will influence the next funding round.

A funding regime that selectively rewards quick-return contributions on a narrow range of intensively discussed topics will drive high-quality, low-citation research to extinction. That would do wonders for the correlation between quality judgments and citation metrics, but the consequences for the health of the discipline would be horrendous.

Touting correlations without understanding the causal relations that underpin them is a perfect recipe for creating perverse incentives.

Malcolm Heath, Professor of Greek, University of Leeds.