Category and other errors

The headline "'Serious defects' apparent in 'crude' European rankings project" and its associated news story suggest that your staff had not read the paper at the heart of the story (News, 24 June).

The paper, University Rankings: Diversity, Excellence and the European Initiative, argues that all ranking systems suffer from poor proxies for the things they are trying to measure, and that monotonic ranking systems suffer from the additional disadvantage of failing to recognise diversity: comparing higher education institutions with different functions as if they were the same.

In this latter regard, U-Map and U-Multirank are distinct improvements on Times Higher Education's own ranking system, and have the potential to provide real value rather than mere brand identity.

Many ranking schemes do what we teach our students not to do. They fail to define precisely what is being ranked, other than as an accumulation of proxy attributes. The proxy measures are unbounded: there is no estimation of their error in measuring the feature for which they stand, such that we are unable to assess confidence in the results.

They commit category errors by including institutions with different purposes. There is no rational method for combining proxies for, say, research, education and outreach to produce a summation: an error that U-Multirank does not commit. And there is no test of the hypothesis that a particular ranking represents.

Moreover, many rankings give credence to the view of universities as supermarkets offering a series of highly specific and measurable products that are currently in vogue. But they must be greater than the sum of their parts. Universities should have a greater sense of their potential to evolve beyond the ranking box to stimulate the creative talents that are their principal contribution to their societies.

Geoffrey Boulton, University of Edinburgh

Measured, and found wanting more