It's evolution, not revolution for REF

The impact of academics' work on the wider world will determine one quarter of universities' overall score in the forthcoming research excellence framework, with the number of assessment sub-panels to be cut by more than half.

These major changes to the way research is assessed were set out by the Higher Education Funding Council for England this week.

The REF, which will replace the research assessment exercise, will put particular stress on the economic and societal benefits of research, and will also see the greater use of metrics to inform peer review and more consistency among panels.

Hefce's 56-page consultation, the academy's final formal chance to comment on the REF, comes 18 months after it was first proposed that a mix of metrics based on citations, research income and postgraduate student numbers would determine the allocation of nearly £2 billion in annual quality-related research funding from 2014.

However, the plan that has emerged, which follows numerous studies, a bibliometrics trial and intense discussions with the sector, looks very different from the original proposals.

It resembles an evolution of the peer-review-based RAE more than a revolution based on metrics. Hefce suggests that significantly reducing the burden of peer assessment through the use of research citations while retaining robust judgments on quality are all but incompatible aims.

"While we remain concerned to reduce the burden of the assessment, we believe we have exhausted the main options for any radically different alternative approach (to the RAE)," the document says.

Output, impact, environment

The consultation proposes an assessment regime based on three elements: "output quality" (weighted at 60 per cent); "impact" (25 per cent); and "environment" (15 per cent).

The impact measure is introduced as a distinct element of the research assessment for the first time, in line with the Government's so-called "economic-impact" agenda.

As in the RAE, "output quality" will continue to be based on peer review, referred to as "expert review" in the new model.

In a change from the previous system, however, citation data will be available to "supplement and inform" the assessment of various science subjects (see "REF at a glance", right). Other subject sub-panels will decide whether they want to use such data.

"Impact" will be judged by the sub-panels only for high-quality research. Impact must be evident during the REF assessment period, although the research could have been carried out ten to 15 years before. It will be assessed using departmental case studies - one for every five to ten staff submitted - and an impact statement from departments.

The "environment" measure will consider factors including research income, the number of postgraduate research students and their completion rates. Panels will decide whether bigger departments with "critical mass" should receive special recognition.

Hefce's draft plan is to apply the same 60:25:15 weighting across all units of assessment (UoAs). This will lead to "greater simplicity and comparability across the exercise" and reduce the scope for "tactical decision-making about which UoA to submit work to", it says.

In a controversial move, the number of UoAs and sub-panels is to be cut from 67 to 30, and the number of main assessment panels from 15 to four. "We believe that the proposed breadth and scale of coverage ... is manageable," Hefce says. "We are committed to substantially reducing the number of UoAs and do not wish to consider arguments for retaining comparatively small discrete UoAs."

A single sub-panel will assess outputs, environment and impact, with "specialist advisers" helping on outputs and "research users" doing so on impact. Here the role of academics will be downplayed.

Roughly the same number of experts will be involved in the assessment as in RAE 2008, David Sweeney, Hefce's director of research, said. The consultation, published on 23 September, asks for feedback on whether the number of outputs submitted per person should be reduced from four to three, and whether certain "substantive outputs", such as monographs, should be "double-weighted".

As in the RAE, the selection of staff and their outputs will be left to institutions, and it is proposed that the "profile-based" assessment pioneered in RAE 2008 will continue.

The consultation also proposes honing the descriptions of "world-leading" (4*) and "internationally excellent" (3*) research used in RAE 2008 to achieve "greater discrimination" at the top end of the scale.

It also seeks to resolve the issue of whether the proportion of eligible staff submitted should be made public, with Hefce altering procedures to ensure it will be.

"We recognise that some institutions feel strongly that they should be able to demonstrate their research intensity," the document says.

We're all ears

Mr Sweeney said Hefce was seeking a "genuine consultation" and was in "listening mode". He added that its intention remains to "identify excellence wherever it is found".

Bahram Bekhradnia, director of the Higher Education Policy Institute, said it was a "great relief" that Hefce was proposing a peer-review process informed by metrics.

"Although the proposals have a new name, they are a development of the RAE," he said. However, he suggested it was "surprising and perhaps unwise" that as much as 25 per cent of overall scores would be attributed to impact.

"It would have been wiser not to give so much weight to what is effectively an experiment," he said.

He described the reduction in the number of sub-panels as "understandable and brave", but said it would be "damaging to the credibility of the exercise if large numbers thought that they were being judged by people without the competence to do so".

The proposals were also welcomed by Research Councils UK.

"We support Hefce's position that there is a role for quantitative evidence, including citation information ... but believe that peer review should still form the backbone of any assessment of research excellence," an RCUK statement says.

The consultation closes on 16 December.

zoe.corbyn@tsleducation.com

REF AT A GLANCE

• Assessment to be based on three elements: research output (60 per cent); impact (25 per cent); and environment (15 per cent).
• Citation data to be used to inform expert review in units of assessment covering the medical, health, biological, physical and computer sciences, along with psychology and engineering.
• A two-tier structure is planned, with 30 sub-panels (one for each unit of assessment) working under the guidance of four main panels.