Is it safe to trust that the next REF will reward a wider range of outputs?

It would be easy for academics and administrators to think that the Research Excellence Framework 2029 will basically be a rerun of the 2021 exercise. But look closely and you will see that REF 2029 is actually a very different animal.

It’s true that the REF’s structural mainframe is staying the same, with four main panels and 34 units of assessment, plus a continuing commitment to peer review and an arm’s-length use of metrics. The three main elements of the process are also still broadly based around outputs, impact and environment. But things have changed; there has been a major shift on what “excellence in research” means and how quality can be demonstrated.

In the past few rounds of the REF and its predecessor, the Research Assessment Exercise, a rather narrow, formal and short-term approach to understanding research excellence held sway. This limited conception did not go unnoticed: critics have long complained about the exclusion of new and emerging intellectual currents, an aversion to riskier outputs beyond the traditional journal article and substantial internal university bureaucracy to keep researchers in check. Instead of incentivising intellectual innovation and the mobility of knowledge across traditional disciplines, even the most recent exercise, REF 2021 (whose results were actually delayed until 2022 because of the pandemic) was widely seen as having the opposite effect – despite its conscious efforts to reward a wider range of outputs.

It is, therefore, not surprising that the latest REF iteration is seeking to expand how excellence in research is conceived, delivered and assessed. A new “enhanced assessment of excellence” will define quality much more broadly. This is likely to emerge as REF 2029’s defining feature, reflected in changes to the definition and weighting of the three main elements.

The “outputs” element of the REF has been renamed “contribution to knowledge and understanding” (CKU). This new emphasis is embodied in a new “explanatory statement” setting out how a unit of assessment's research has advanced knowledge, worth at least 10 per cent of quality scores. Also gone is the narrow talk of “impact”; instead, “engagement and impact” will be assessed. And the “environment” section has been replaced by “people, culture and environment” (PCE), accounting for 25 per cent of total scores – up from 15 per cent in REF 2021. That extra weighting comes at the expense of CKU, which will account for only 50 per cent of total scores in 2029.

Most strikingly, this “enhanced” way of assessing research excellence recognises a distinction between the business of doing research and what is described as the “wider infrastructure of research”. The former generally relates to the achievements of individuals or specific units of assessment, usually in the form of publications and patents. “Research infrastructure”, by contrast, focuses more on how research is actually done – how it is conceptualised, how it is made available, how the contributions of everyone in a team are recognised and how different forms of knowledge are embraced. It might also include how researchers are trained or developed professionally and whether their research crosses traditional disciplinary or sectoral boundaries – perhaps by working closely with experts in industry.

Source: 

Istock (edited)

The exercise seeks also to reward collective contributions to the national science base through research reciprocity. In other words, research units must put something back into the training, development and support of other researchers beyond their institutions.

These innovations are aimed at promoting the “crucible effect”: a concept of structured serendipity coined by Nesta and then developed by the Royal Society of Edinburgh whereby different elements are brought together to forge something far stronger than the base elements. In a research context, these elements are researchers, professionals and students from across the research “ecosystem” who would not normally get the chance to meet and learn about each other’s skills. Breakthroughs at world-class research environments do not just happen – it is by cultivating boundary-spanning relationships and innovating in relation to opportunities that new perspectives and insights emerge.

This principle of facilitating a much larger landscape, after all, is why UK Research and Innovation was formed six years ago, furthering connectivity across the research, development and innovation ecosystem, as envisaged by Sir Paul Nurse in his 2015 report. Now it is the turn of the REF, arguably the biggest driver of how research is done in the UK, to turbocharge cooperation and connectivity across the sector.

The document on “high-level decisions” made for REF 2029, published last June, outlines how this strategic shift in quality assessment will play out in practice. Review articles, meta-analyses, replication studies, translations and edited collections have not, for example, generally been seen as “REF-able”, but they are now explicitly identified as critical components of an expanded approach to research excellence. The sector should, the document suggests, look beyond the usual research outlets and channels, such as journal articles, which accounted for more than 80 per cent of the outputs submitted to REF 2021. In theory, this should open up the intellectual space within REF 2029 to acknowledge the value of a range of research contributions.

There have been some concerns about whether blunt metrics can capture the research culture (PCE) part of REF 2029; the year-long delay to REF 2029, whose results were originally supposed to be announced in December 2028, will largely examine the implementation of the PCE element. Nevertheless, the broader definition of research excellence has been largely welcomed by the sector. When Dame Jessica Corner, executive chair of Research England, announced that this was a “once-in-a-generation moment for change as we shift national research assessment away from a focus on individuals to how institutions and disciplines contribute to healthy, dynamic and inclusive research environments”, few demurred. 

But are academics, universities and even review panel experts ready to play a potentially very different REF game? Are they likely to respond to its invitation to embrace a more fluid, perhaps less clear idea of quality?

One potential pinch point is the internal management and selection processes that have been hardwired into university systems over recent decades. Internal vetting processes around “who’s in” and “who’s out” of an institution’s REF submission are governed by certain assumptions about what excellence looks like. With the national research assessment process suddenly wanting to “recognise and reward” a far more diverse and vibrant interpretation of excellence, how will risk-averse university managers and vice-chancellors adapt? What if they refuse to respond to the signals and, instead, continue to base their submissions on a very narrow interpretation of research excellence?

I’ve had conversations with many early-career researchers who like the sound of the “mood music” around REF 2029. They wish to undertake research that challenges conventions and ranges across disciplinary boundaries, and they want to disseminate it beyond the traditional publishing confines of journal articles and monographs. But the same early-career researchers often fear falling into a gap between rhetoric and reality – a gap that risks leaving them trapped with a body of research and scholarship that, as the cold, hard reality of REF 2029 approaches, is suddenly viewed as a little too risky, too unconventional for their university’s submission strategy.

Addressing this peril will be crucial as the ever-thorny topic of REF panel appointments arises, as it soon will. The traditional conservatism of review and assessment boards is well recognised within academia, so panels will need to be recruited, supported and empowered with an emphasis on promoting and rewarding a broader definition of research excellence. 

With such uncertainty hanging over the process, grand statements about valuing diversity of excellence will not be enough. Guidance to academics and institutions about what a wider definition of research excellence means and how it is to be measured will be vital to boosting trust that innovative research or projects really will be rewarded. 

But what exactly should this wider definition be? Widespread discussion of that question must start in earnest, yet it is unclear who should lead it – and ensure that different opinions and outlooks are channelled back into Research England, the lead body delivering REF 2029 on behalf of all four UK higher education funding bodies. The REF 2029 team itself is relatively small and lacks the resources to directly oversee an engagement programme. It could also be argued that it is not its job to oversee disciplinary discussions in a process that is, after all, undertaken by academics and based on peer review. More generally, it is not for funders to prescribe any definition of research excellence.

Better, then, for the discussions to be led by the subpanels themselves. But this could get very messy if significant disagreements arise between disciplinary areas, or if specific panel members feel uncomfortable engaging in discussions and debates that further muddy the waters on quality.

This all raises the simple question: “Who speaks for REF 2029?” The high-level decision to adopt a broader approach to research excellence has been welcomed, but it now needs research leaders – academic ambassadors – who are willing to put their heads above the parapet and lead the debate.

Some would argue that learned societies, the national academies or professional associations could take on this ambassadorial role. But entrusting this task to senior establishment figures surely risks promoting exactly the homogeneous type of excellence that REF 2029 has sought to push back against.

Within these discussions, therefore, the research leaders of the future – today’s early- and mid-career researchers – have a central role to play. If we are to avoid a rhetoric-reality gap opening up around REF 2029, those who most enthusiastically buy into the rhetoric must be closely involved in shaping the reality.

Matthew Flinders is professor of politics at the University of Sheffield, and vice-chair of the Political Studies Association of the United Kingdom.