An instrument too blunt to judge sharp minds

Metrics may appeal to the RAE-weary but, warns Claire Donovan, Australians fear the numbers do not add up.

I am quite partial to pointing out the injustices of using science-friendly metrics to evaluate the humanities, arts and social sciences. But even I was unprepared for the outburst of a senior scientist at a recent meeting.

Since the early 1990s, Australia's block research funding has been based on metrics - publication numbers, research students, grant income and the like - a system now much maligned for favouring science subjects and not stimulating research quality. So, just as the UK looks to ditch its research assessment exercise in favour of metrics, Australia is on the verge of switching to peer review.

Like the angry senior scientist, I was in Canberra as a member of a cross-disciplinary group convened to consider the transition to a system suspiciously like the RAE. The discussion turned to whether citation metrics could lighten the load of panels expected to judge all submitted research output. It was being suggested that they could be accepted as a reasonable substitute for peer review in the "basic" science disciplines when the scientist spoke out. "Hold on a minute," he said. "That would mean the humanities, arts and social sciences would get Rolls-Royce treatment in comparison with the sciences!"

This scientist had faith in citation metrics when applied to all subjects.

But he saw that science would be disadvantaged if it alone was judged in these terms while more time and resources were diverted to a superior treatment of other disciplines.

In a grab for cash, the UK's science, technology, engineering and medicine-focused institutions initially rushed to swap the Rolls-Royce treatment of the RAE for a metrics-based assessment of research quality.

Not only was this a gross act of treachery against the humanities, arts and social sciences, it was startlingly premature when the shape of the proposed system, its weightings or how they would be linked to funding were not known. Now the sector has seen some financial modelling, there is almost universal condemnation of a metrics-only approach.

For those of us who care about the future of all UK research, it is deeply worrying that the Government does not realise its proposed replacement for the RAE will not actually measure research quality. Who knows what perverse behaviours metrics will drive?

A system that rewards the number of publications has been explicitly dumped in Australia. Over two decades, this incentive has caused an increase in the number of academic papers. But these have been clustered in less prestigious journals.

The inherent quality of research is not measured by citation metrics either. Any field in which journals are not the main way of presenting research will be disadvantaged - a fact that could severely affect the future of UK research and diminish its international standing. Take the social sciences as a telling example of what is to come. In journals, it's the number-crunching papers that have the widest potential for citation.

They will become viewed as the "best" and will be rewarded as such. This will steer university block funding towards the pursuit of quantification and away from qualitative or interpretative research, which will be branded "inferior".

Research income does not guarantee research quality. We know that the capacity to raise funds varies between disciplines and subfields. But this is conveniently forgotten when, as proposed, this metric is applied at an institutional level and science-oriented universities clearly reap the rewards. This has been recognised by the British Government, and it is promised that "other options" will be explored for humanities, arts and mathematics. But will this mean retaining the Rolls-Royce system for such disciplines along with dedicated cash? If so, when they realise their mistake, will the UK's science-led institutions clamour for similar treatment just as our scientist in Canberra did?

And what will be the fate of the social sciences, acknowledged only by a footnote in the Government's consultation document?

Perhaps the moral of this story is that we should be careful what we wish for because it may come true. While many have, with good reason, longed for the demise of the RAE, a metrics-only system would leave us pining for the halcyon days of peer review.

When isolated from peer knowledge, metrics are blunt instruments that tell us little about quality. Yet peer review is not without its flaws. To enhance the UK's research standing, the way ahead is for all disciplines to stand together to lobby to retain an RAE with new field-specific metrics to aid panel deliberations. These metrics should include intellectual, social, cultural and economic returns. It will take time to develop, but in the meantime, we should all insist on a Rolls-Royce treatment.

Claire Donovan is a research fellow in the Research Evaluation and Policy Project at the Australian National University.