Careers at risk after case studies ‘game playing’, REF study suggests

Click here to view full graph

A significant number of academics’ careers could be tarnished by exclusion from the research excellence framework merely on the basis that their university could not find enough good impact case studies, an analysis indicates.

The number of case studies universities were required to submit to the 2014 REF was determined by the number of full-time equivalent (FTE) staff they submitted. Two case studies were required for submissions containing up to 14.99 staff, three from those between 15 and 24.99 staff, and so on.

Graeme Rosenberg, REF manager for the UK funding bodies, confirmed to Times Higher Education that his intention had been that decisions about the number of staff submitted would determine the number of case studies to submit, rather than vice versa.

However, an analysis of the REF results by Tim Horne, head of the Research Excellence Unit at Coventry University, reveals that the number of submissions containing staff numbers just below the threshold for an extra case study was far higher than would be expected statistically.

Dr Horne rounded down to whole numbers the number of staff in each of the 1,911 submissions to the REF, and plotted how frequently each number appeared. He found a series of sharp spikes in numbers ending with a four (view full graph).

Dr Horne said that his analysis confirmed what “everybody knew would happen anyway: that you would get institutions determining the number of people going into a submission by the number of case studies they had that they thought were good”.

He said the same pattern was evident for all mission groups and main panels. Examination of individual institutions’ submission patterns risked not being statistically significant. But he noted that of Durham University’s 23 submissions, 18 were within one FTE of a case study boundary. The institution was 19th by grade point average on impact, and 20th overall, submitting 79 per cent of its eligible staff.

Universities generally scored highly for impact: the sector’s overall grade point average was 3.24, compared with 2.90 for outputs. But Patrick Dunleavy, professor of political science and public policy at the London School of Economics, said it was “very disturbing” that universities appeared to have been “gaming the system” by not submitting individuals whose outputs otherwise merited submission. Universities typically insist that non-submission to the REF will not, in itself, have negative career consequences. But many academics remain sceptical of such claims: “Individual careers may well have been sacrificed for institutional convenience,” Professor Dunleavy said.

Claire Warwick, pro vice-chancellor for research at Durham, said that her institution had aimed to “make the best submission possible” within the rules.

“Decisions about whether [the] rules might be changed in future are a matter for the funding bodies to consider. We would suggest that if they wish to measure research intensiveness in a future REF, they might require universities to enter all academic staff,” she said.

Professor Dunleavy also advocated universal submission but preferred a metrics-based approach to avoid “raising the already huge workload” and to end the “current farce of assessment by eyeballing”.

Dr Rosenberg said that he was “aware that the case study requirements had an effect on submission size. We will need to look at the implications and consider this issue for the future.”

paul.jump@tesglobal.com