Should REF environment statements get the chop?

Pointless or profound? A bureaucratic monster that entrenches research inequalities or a rich, narrative-led document that offers an insight into every university department in the UK?

Of all the thousands of documents entered into the Research Excellence Framework (REF), environment statements – of which about 2,000 unit- and institutional-level versions were submitted for the 2021 exercise – tend to polarise opinion more than anything else. For some, these detailed accounts of how research is supported within different disciplines – backed with information on research infrastructure expenditure, PhD completions and funding success – are the future of the REF; in their eyes, they represent a bold break from the harmful fixation on “star professors” and individual outputs and the chance to move to a team-science-led culture, in which all research colleagues are valued.

For its critics, however, such statements are little more than a glorified creative-writing exercise, sometimes relying on “boiler-plate” text about inclusion (as one REF sub-panel suggested). Moreover, they involve huge amounts of administrative effort, given that many universities will submit one for each of the 34 units of assessment, each of which is read and discussed by between three and five panellists.

With much of the content already covered by other schemes – notably Advance HE’s Athena Swan and Race Equality Charter schemes, and various sector-level concordats on research integrity and researcher development, plus institutional commitments on responsible metrics use – the statements also repeat existing work on research culture, others add.

Colette Fagan, vice-president for research at the University of Manchester, who leads the Russell Group’s group of research pro vice-chancellors, cited them as one of the biggest burdens of the REF, suggesting the unit-level statements should be abolished in favour of institutional-level documents.

“Environment statements can be very time-consuming for the proportion of weighting that they get,” Professor Fagan told a Times Higher Education webinar, referring to the 15 per cent of overall REF scores they account for.

However, it may be Russell Group universities – whose metrics for doctoral degrees, research grant funding and infrastructure spend naturally give them an edge over smaller institutions – who benefit most from such statements. Given that both research impact and environment scores track fairly closely to output scores – on which larger institutions generally fare better – awarding funding on the basis of environment essentially rewards these institutions three times for their excellent outputs, with this “Matthew effect” compounding inequality across the sector, some argue.

Mehmet Pinar, professor of economics at Edge Hill University, said the correlation between outputs, impact and environment meant the latter two may be “redundant” for the purpose of resource allocation, at least.

He also questioned whether environment really assessed the strength of departments. “Even though the environment evaluates the team effort, it is still an individual-level assessment in many aspects,” explained Professor Pinar, pointing to the data collected on external income generation and PhD completions. “Most of these metrics are not satisfied by all team members – not everyone generates a large sum of income or supervises the same number of postgraduate students.”

Submission size also plays a major role in the environment template outcomes as “even though smaller institutes tend to perform relatively well in outputs, their relative performance tends to be lower in environment scores”, he said, which means that “removing the environment template would relatively benefit smaller submissions”.

Others take a different view, however. John Senior, pro vice-chancellor (research and enterprise) at the University of Hertfordshire, said he would not favour a move to institutional-level statements only, stating the unit-level approach helped to “empower each unit of assessment to develop a bespoke research delivery plan, which aligned with the university’s overall research strategy”.

That led to important ambitions being achieved in this year’s REF, with Hertfordshire improving its research impact standing more than any other university and ranking the highest of any post-92 university, he added.

Rachel Norman, dean for research engagement and performance at the University of Stirling, agreed that the loss of unit-level statements on environment would be a mistake.

“The vast majority of research interactions come at a local level – even having statements at faculty or school level may be too broad to ​truly reflect how people experience research culture,” said Professor Norman.

If the statements are to rely more heavily on metrics, more “creative” ones should be used, she suggested. “We should have something focused on career pathways and precarious contracts, so maybe something about how many people have been on short-term contracts for several years – that might encourage more longer-term contracts or those that allow people to move between projects but provide some stability,” said Professor Norman.

Though the environment section may have its doubters, it could also be an opportunity to fix some of UK academia’s biggest problems, she added. “Universities have to change fundamentally at the moment – the traditional model in place now is not properly inclusive and only some people can succeed,” reflected Professor Norman.

If the international review panel examining the future of the REF is keen to make changes, scrapping these statements is not the way to go, she advised. “If we are to have a REF, let’s make it a force for good.”

jack.grove@timeshighereducation.com