The “metrics-lite” approach to assessing research culture that will be used in a pilot to inform the next Research Excellence Framework (REF) has been welcomed by sector figures.
A year-and-a-half after organisers of the REF announced that the people, culture and environment (PCE) section would be assessed using a “more tightly defined, questionnaire-style template” focused on “demonstrable outcomes”, replacing the lengthy environment statements submitted in REF 2021, the likely metrics and indicators on which universities and departments will be assessed in 2029 have been announced.
Under the pilot framework published on 16 January, universities will be asked to submit data regarding staff diversity, pay gaps and the proportion of fixed-term staff. Metrics on research staff well-being will also be sought for the pilot, involving 40 institutions which will submit data across a range of units of assessment, while information on the number of openly accessible research outputs and datasets will also be considered. Institutional data on carbon emissions will also be collected.
Institutions will also be allowed to submit qualitative evidence and contextual data about their achievements, though they will be limited to just 1,000 words in each of the five assessment sections.
However, the areas on which qualitative data can be submitted vastly outnumber the subjects on which metrics are sought, noted Cat Davies, dean for research culture at the University of Leeds.
“Colleagues might be relieved to see the balance between qualitative and quantitative data, which is slightly heavier on the former than perhaps what was expected given all the work around metric selection,” said Davies, noting the investigation of potential metrics had “cemented the limitations” of this kind of approach.
Advocating for “metrics-lite and rich narratives” for PCE, Davies said she “welcome[d] the mix of qualitative and quantitative data to evidence research culture change”.
“Although tentative, the framework will help research organisations start to map their data and practices against the proposed indicators. The five enablers align with the way that many institutions are structuring their research culture work,” she said.
It was important, Davies added, that the “final framework later this year [is] one which balances rigour, inclusivity and burden at a time when universities are having to make difficult financial decisions”.
Within the proposed framework universities will be able to refer to “external accreditation” schemes such as Athena Swan, the Race Equality Charter and various concordats. However, complying with these schemes is “not necessarily an indication of excellence, but examples of such external standards may be used to explain and justify the approaches taken”, the guidance warns.
That ambivalence towards specific charter marks and commitments was disappointing in the case of the Technician Commitment, suggested Paul Gilbert, chair of Technical Managers in Universities, which represents technical staff.
“I would perhaps like to see more explicit reference to technicians and technical initiatives in the framework, particularly with regard to the Technician Commitment, which has had an enormous impact on advancing good research culture by bringing to the fore the critical role that technicians play,” said Gilbert.
That said, the framework’s “reference to the development of career pathway frameworks for all research contributors is welcome, as this is an area that has been sharply in focus for technical staff and where a lot of positive progress is being made across the sector, not least at my own institution, the University of Liverpool”, he added.
“For many years the contribution of technical staff to research excellence has been overlooked and undervalued, so I welcome the broader inclusion of all who contribute to the research environment in our universities,” added Gilbert.
