Don’t let the REF tail wag the academic dog

There’s a great deal of interest in how we can improve the research process, both in terms of how we conduct research and the incentives we create that shape our behaviour. Greater transparency in research is increasingly being promoted to allow the assumptions and choices made during a research process to be scrutinised. At the same time, institutions are developing responsible approaches to research evaluation that eschew reductive metrics such as journal impact factor. This, in turn, is part of a wider focus on research culture, reflected in the UK government’s R&D people and culture strategy, as well as UKRI’s recently released strategy and activity over recent years by charitable foundation Wellcome.

An interesting time, then, for the research excellence framework (REF) results to be released.

Institutions and even academics themselves are perhaps their own worst enemies when it comes to the REF. Much of the time and effort associated with the REF arises from the way institutions prepare for it – which is understandable given the financial incentives involved and the complexity of the rules. While the REF analysis is at the level of the unit of assessment (UoA), inevitably individual academics feel pressure to demonstrate their value to their institution by producing what they hope will be 3* and 4* outputs, although they will never, in fact, know. Then, once the results are released, institutions are keen to trumpet their perceived successes and focus on metrics that allow them to say they are “top five” in something.

While REF preparations and submissions are now behind us, is there a way to approach the REF results that takes into account efforts to improve research culture?

One approach that indicates how we can do this is that taken by the University of Glasgow. They pre-registered their approach to communicating REF results some time ago. Why does this matter? All institutions will have prepared their approach internally, but this transparency means we can trust that the interpretation and communication of results will not be shaped by the results themselves, but instead by core guiding principles that have been considered in advance. By making these principles transparent, Glasgow has ensured that no one can accuse them of having tweaked these after the fact. This is exactly the reason why pre-registration of study designs and analysis plans before data have been collected has been promoted as one possible means by which research quality could be improved.

What are these guiding principles to REF communications that are laid out in Glasgow’s pre-registered approach?

1. They will be mission focused, in line with the university’s research strategy, research environment and research strength.

2. They will be contextualised within the larger picture of Glasgow’s research successes and future ambitions.

3. They will focus on collegiality over superiority, avoiding unnecessary comparisons and presenting the results as evidence of Glasgow’s significant contribution to the collaborative research endeavour.

These are taken directly from the Glasgow’s statement, but I have added emphasis to what I think are the key elements of these principles.

These are guiding principles that we collectively – as a sector – would do well to consider. The competitiveness that the REF fosters is not a necessary feature of the exercise, but it is perhaps unsurprising given the broader pressures on the sector that have led to excessive competition at both an individual and an institutional level – hence the need to focus on collegiality as an antidote to those pressures. And we should not need the REF to tell us what we’re good at, not least because the REF cannot capture everything that counts as success for an institution. Hence, there is a need for the results to be contextualised, and also to be mission focused. The REF is an important mechanism for accountability but should not be the be all and end all of academic endeavour within the UK.

Once the REF results are out, the pressure will be on institutions to emphasise the positives and downplay the negatives. If we’re not careful, this will lead to another round of institutions all claiming to be world leading on one dimension or another. This risks undermining all the positive work that has been done around research culture, and making academics feel that they simply exist to generate REF outputs that promote their institution in ways that feel decoupled from the academic endeavour. While few institutions have a publicly pre-stated approach to how they will communicate their results – I am only aware of Glasgow’s – we can perhaps all bear this in mind when the results are announced.

Marcus Munafò is associate pro vice-chancellor for research culture and a professor of biological psychology and a Medical Research Council (MRC) investigator at the University of Bristol.

If you found this interesting and want advice and insight from academics and university staff delivered direct to your inbox each week, sign up for the THE Campus newsletter.