Concern over ‘basic’ indicators used to judge universities

Many European quality assurance agencies are using “quite basic” metrics to pass judgement on universities, according to the European University Association (EUA), following an analysis of how institutions are assessed, ranked and funded.

The EUA has mapped which indicators matter most for universities across Europe, determining their budgets, prestige and accreditation. These include their staff-to-student ratio, the number of degrees they hand out and even floor space.

On the whole, universities are judged on the basis of easy-to-quantify “input” indicators, such as their student and staff numbers, rather than trickier “output” numbers that try to gauge what students have actually learned.

Despite a growing focus in recent years on “learning outcomes” − including a project run by the Organisation for Economic Cooperation and Development to test student capabilities that never got beyond a pilot – the EUA report concludes that the metrics on which universities are measured “have exhibited relative stability”.

“The biggest surprise for us is that we don’t see real development of [the] indicators being used,” said Tia Loukkola, director of the EUA’s institutional development unit, and main author of the report, Exploring Higher Education Indicators, released on 25 May.

Funding bodies, quality assurance agencies and ranking agencies “seem to be using the same kind of indicators they did 10 years ago. There’s no real development in terms of trying to capture the educational mission,” she said. The report looks at indicators related to a university’s educational mission – it does not go into how research funding is allocated, for example.

When it comes to deciding what state support universities get, the most commonly used indicator used by funding bodies is how many students are enrolled. The number of degrees handed out is also a common measure.

About a third of funding bodies across Europe looked at diversity indicators, be it gender or socio-economic. The same proportion take graduate employment rates into account.

As for quality assurance, Ms Loukkola expressed concern that some agencies seemed to be using similar indicators to judge universities – such as dropout rates, for example – as funding agencies. “It’s the same thing being recycled from one thing to another,” she said.

Staff numbers, dropout rates, student numbers and the staff-to-student ratio were the most common metrics used by quality assurance agencies, the report found.

The staff-to-student ratio is a “quite basic” indicator for such an agency to scrutinise when making a judgement of a university, she said, although agencies do normally conduct campus visits as well.

The report also reveals that in two European University systems, a university’s international ranking table position is fed into the funding formula. “It’s not widespread,” she said. “But I would say it’s not necessarily the best indicator.”

“The ranking fluctuations from one year to another can be quite considerable depending on changes in the methodology and the inclusion of new universities,” she cautioned.

david.matthews@timeshighereducation.com