REF impact will be lighter but also more widespread

The impact weighting in the research excellence framework looks set to be reduced, but the broad methodology drawn up by funding chiefs should be adopted universally.

These are the likely outcomes of the pilot impact assessment exercises carried out in English, physics, medicine, social work and social policy, and earth systems and environmental sciences. The conclusions drawn from the pilots were unveiled on 11 November.

The submitting departments, drawn from 29 universities, were asked for one case study for every 10 academics. These detailed any research they had conducted during the previous 15 years that had made an impact in the past six years.

The panel chairs sum up their recommendations in the report Research Excellence Framework Impact Pilot Exercise: Findings of the Expert Panels. They conclude that the case study approach designed by funders is appropriate, and that a common approach and weighting should be possible in all disciplines.

The implementation of the REF was delayed by a year after David Willetts, the universities and science minister, cast doubt on the rigour of the criteria for assessing impact and their acceptance by the academy.

The original proposals, which included a 25 per cent weighting for impact, provoked particularly strong opposition from scholars in the arts, humanities and social sciences.

But Judy Simons, chair of the English panel, said her colleagues' initial scepticism about demonstrating impact in such fields had faded once they began to think in terms of the "benefit" of their work.

"We were won round by the strength of what emerged from the case studies," she said.

The REF is to be conducted by the Higher Education Funding Council for England.

David Sweeney, its director for research, innovation and skills, hailed the pilot as a success.

But he acknowledged that there were a number of issues to resolve before final decisions on the impact part of the REF are taken in January, following a further period of informal consultation with the sector and the UK's four governments.

One of these is the weighting that impact should be given. The report says that this should be "sufficient to ensure it is taken seriously" but not so high that the "developmental" nature of the assessment criteria undermines confidence in the REF.

It says the funding councils should consider lowering the proposed weighting of 25 per cent for the first REF.

It adds that the guidance for institutions will need to be improved to elicit better evidenced and articulated submissions.

The best case studies, it says, consist of a "coherent narrative, explaining what research was undertaken, what the claimed benefits or impacts are and how the research is linked to the benefits".

Some of these will be posted on Hefce's website next week.

The panel chairs also want more flexibility on the time limit delineating when research can no longer be considered.

Mr Sweeney said he was willing to consider the physics panel's request for a 25-year limit.

The panels were made up of roughly equal numbers of academics and "users" of research, but the report says there were no "general differences" in the two groups' judgements.

The panels graded case studies from four stars to unclassified and gave each department a "profile" depending on the proportion of case studies in each category.

Anonymised profiles show a wide range in performance, from a social work/social policy department with 70 per cent of its case studies awarded a four-star rating, to English and physics departments with 100 per cent one-star ratings.

Mr Sweeney said the value of gathering evidence on the impact of research had been demonstrated by the government's decision to ring-fence the research budget in last month's Comprehensive Spending Review.

But he added that to strengthen the case for public benefit in the future, it was necessary to "move beyond the wonderful, glossy publications universities produce talking about the benefits of their research to a more evidence-based set of materials that we can use to assess for quality".

paul.jump@tsleducation.com