Metrics would ruin the REF

At the end of every research excellence framework (REF) cycle, the question is asked whether a more metrics-driven system could simplify and improve what is undeniably a complex, time-intensive exercise. This time around is no exception; since submissions closed at the end of March, there has been a flurry of articles addressing this question, and it is certainly a discussion worth having. However, it seems to me that the answer remains the same: no.

The REF aims, among other things, to identify the relative world-class strengths of UK academic departments, scoring them according to output produced, impact generated and their research environments. It seems likely that high output scores reflect the strongest research environments, in terms of protected research time, availability of research funding and frequency of research leave. So one simplification might be to do away with outputs and score the REF entirely on environment (which is much less labour-intensive to assess than outputs) and impact.

However, the incentives that doing so would introduce might damage quality. Departments might divert scarce resources to supportive measures but without maintaining sufficient regard of what they helped produce. Moreover, the correlation between quality of input and quality of output is hard to definitely prove, and if our aim is to assess the quality of the research produced, anything less than a close look at the outputs themselves would seem to be second best.

But could the burden of assessing outputs be lightened by switching from peer review to some form of metrics? I think not. Every metrics-driven model I have seen for assessing research quality tends to focus on what can be counted – and all are very imperfect proxies for what needs to be counted: quality.

Take journal rankings. Different academic fields disagree (including internally) about whether there is or could be a definitive ranked list of the best journals. In most, if not all, fields in the arts, humanities and social sciences, it is not uncommon for different journals to approach the same referees to assesses the same piece of work; specialist expertise is scarce, after all. Sometimes those approached recuse themselves if they have already reviewed the manuscript for another journal. Sometimes they don’t. Either way, the supposed “best” journals certainly don’t have a stranglehold on the best reviewers or the highest standards. Rather, in my experience, the “best” journals are generally considered to be those most commonly available within universities – and those tend to be the oldest titles.

What about other possible metrics? Grant income (and the need for it) varies enormously by field and scholarly approach. Citations, meanwhile, are a particularly unreliable indicator of quality in the humanities. In my own work, for instance, I cite work that I find mistaken as often, if not more so, than work supporting my points, since I aim to offer something new. Does that matter at an aggregate level? After all, universities are assigned funding based on the entirety of their REF submission, so you might ask how much granularity we really need. However, the minutiae of the rankings do matter enormously for those involved. Cutting corners might save time, but if it short-changes certain departments and institutions, it isn’t worth it, in my opinion.

Perhaps a metrics-based approach might be more relevant to the sciences, and there is nothing to say, a priori, that all disciplines must be judged in the same way; I am aware that, in Australia, for example, a hybrid approach is used depending on the discipline. But if we do want a universal approach, the worst option apart from all the others would appear to be peer review.

While it might be difficult to improve on the REF as a mode of national research assessment, I do worry that it is so utterly divorced from the assessment of teaching quality. This seems particularly misguided for those research-intensive institutions that champion their research-led teaching.

Some might worry that bringing teaching and research together would create more administrative workload. This is an important concern, and I agree with those who argue that we need to focus more on doing research and teaching rather than filling out reports about it. Yet it is also important that we think clearly about what we do and why we do it.

In this case, I don’t think the administrative burden need increase at all. We in the UK already spend a lot of time on teaching quality through assurance exercises, periodic departmental reviews and annual reviews of teaching – not to mention the teaching excellence framework (TEF) itself. How difficult would it be to reorient these exercises so that strategic planning about teaching focuses on research-led teaching excellence? After all, the staff and students who create and benefit from research and research-led teaching are whole human beings, so why should our institutional research and educational strategies be run by separate teams, each with their own future plans?

Of course, any joined-up assessment of research and its contribution to world-class education would probably lend itself even less to a metrics-driven approach than the REF does. But what I propose would, I believe, produce a far more holistic, far more useful measure of departmental and institutional quality.

We can and we should do better that the current approach – but not by grasping for low-cost, low-quality metrics.

Thom Brooks is the dean of Durham Law School and the president of the Society of Legal Scholars. He comments in a personal capacity.