Research Intelligence - REF guidance sets up the starting blocks

The research assessment exercise may have changed its name but the submissions guidance for the 2014 research excellence framework underlines that there will be few major changes to the way the exercise is conducted.

The one major new element of the REF is, of course, the evaluation of each unit of assessment's "impact" on the basis of one case study for every 10 academics.

As previously announced, impact will count for 20 per cent of the final scores across all units of assessment, with research environment counting for 15 per cent. "Esteem" will no longer be assessed.

The REF, which will inform how quality-related research funding will be distributed from the 2015-16 academic year, was originally intended to embrace bibliometric data. But the requirement was relaxed by the Higher Education Funding Council for England, which administers the REF on behalf of all four UK funding councils, after a pilot exercise indicated that citation information was "not sufficiently robust to be used formulaically or as a primary indicator of quality".

The funding councils' Assessment Framework and Guidance on Submissions for the 2014 REF, published last week, confirms that panels will be permitted to use figures on the number of citations garnered by research outputs only to "inform" their judgement and must not use them as a substitute for reading outputs, or penalise outputs for which no citation data are available.

Panels will also be prohibited from using rankings of journals in their assessment of research outputs.

Such rankings were used last year for the inaugural run of Australia's version of the REF, Excellence in Research for Australia. They have been jettisoned from the second iteration in 2012 amid concerns that they were encouraging universities to introduce targets for publishing in top-ranked journals.

Stories also abound of UK universities unofficially making hiring and promotion dependent on publishing "REF-able" numbers of papers in journals with high impact factors. But the submissions guidance says the funding councils "do not sanction or recommend" the use of citation information to inform selection of staff or outputs for REF submission.

Instead, universities should use their judgement to select submissions "in relation to the full range of assessment criteria (originality, significance and rigour), and in accordance with their codes of practice for the selection of staff".

These codes will have to be submitted to Hefce for approval. Particularly close scrutiny will be applied to equality considerations in the light of the relatively low numbers of eligible women and black academics submitted to the 2008 RAE.

The submissions guidance also makes clear that early career researchers or academics whose circumstances "have significantly constrained their ability to...work productively throughout the assessment period" will not be penalised for submitting fewer than the standard four outputs.

The assessment period will run from 1 January 2008 to 31 December 2013 for research outputs and to 31 July 2013 for the impact and environment elements.

The research assessed for its impact must have been carried out since 1 January 1993, but subpanels can make a special case for that date to be put back by another five years. The deadline for submissions will be 29 November 2013.

Details of how submissions will be assessed are to be put out for consultation at the end of the month, including details of which panels will be using citation data and the sorts of evidence they will require for the impact and environment elements of the exercise.

The criteria are being drawn up at main panel level to ensure consistency, but exceptions will be made for specific subpanels "where this is justified by differences in the nature of research in the disciplines concerned".

More assessors for subpanels

The guidance also confirms that more assessors will be added to sub-panels for the assessment phase of the exercise in 2013. Their selection will be informed by a survey of the kinds of outputs institutions intend to submit, to be carried out at the end of 2012.

REF manager Graeme Rosenberg emphasised that many of those already appointed to subpanels have interdisciplinary expertise. But he said subpanels would also be permitted to ask other subpanels for help with cross-disciplinary submissions.

If an output involves collaborators from more than one university, all institutions may submit it and subpanels will assess it on an "equal footing" to other outputs. Departments will also be required to state, in their environment submission, how they support interdisciplinary and collaborative research.

As in the RAE, results given at the end of 2014 will be in the form of "profiles", indicating what proportion - this time to the nearest percentage point - of a submission was deemed to fall within each of Hefce's four star categories.

The submissions guidance says that, after careful consideration, Hefce has concluded that "the absence of an appeals process does not make the assessment process any less robust".

David Sweeney, director for research, innovation and skills at Hefce, said he could not see how a fair appeals process could be devised.

He added that there was a "general acceptance" among universities that, as in student exams, there could be no appeal against what were purely academic judgements, and that it was best to "take the rough with the smooth" regarding unexpected results.

Full details of submissions, as well as "sub-profiles" for outputs, impact and environment, will be posted online in 2015 unless departments request redactions for specific reasons such as security or commercial sensitivity.

Each panel will also publish a report summarising its observations about the state of research in the areas falling within its remit. Feedback on the reasons for each profile will be sent to the vice-chancellor of each institution.

paul.jump@tsleducation.com