REF's effort to make knowledge visible may have cloudy results

Credit: James Fryer

Mark Twain once commented that educational institutions “have two great functions: to confer, and to conceal, knowledge”. If UK academics have been guilty of the latter historically, efforts to reward research “impact” can be understood as an attempt to ensure that this is no longer the case. In this respect, the intention is laudable, but the “impact” agenda has also met with fierce criticism, particularly from those who view it as part of an agenda to further marketise higher education.

The final criteria for assessing impact in the research excellence framework will reassure some of those who have raised concerns. The period in which impact can be demonstrated is relatively long (from 1993 for most subjects, even longer for architectural research); research “users” will be involved in assessments; case studies will be able to draw on changes to policies at non-governmental organisations as well as in the public and commercial spheres. There is also a quality “threshold”, which means that research outputs must be deemed to be of at least “two-star” standard to qualify for inclusion. Finally, the case-study approach means that not all academics will need to demonstrate impact.

Yet there are good reasons to remain cautious, even for policy-orientated academics. First, beyond the quality threshold, impact case-study assessments will explicitly ignore differences in the quality of research outputs. This may please those who dislike the REF’s approach to assessing quality, which prioritises international relevance over local. However, it also means that lower-quality research for which there is apparently concrete evidence of “impact” (policy citations, say) is liable to score more highly than higher-quality outputs for which the “impact” is more nebulous.

Second, there is little, if any, consideration of ethics in the impact assessment criteria. In theory, this means that researchers can be rewarded for evidence that their work - which may itself have been conducted ethically - is being used for ethically questionable purposes, such as research employed by companies profiting from the sale or use of harmful products like alcohol, tobacco or arms.

Another issue is that, despite a vast empirical literature demonstrating that the use of research within policy is often “symbolic” (helping to legitimise decisions that have already been taken rather than informing decisions), there appears to be no suggestion that assessors should try to consider different kinds of “use”. Rather, there is a perhaps naive assumption that where research is cited, this alone is indicative of influence. In practice, it may be hard to determine where research has been employed symbolically because decision-making is often opaque, and it may be in the interests of both researchers and research users to present influence in instrumental terms, even if this is an inaccurate portrayal.

In addition, rewarding evidence of impact may, over time, encourage academics to focus on producing research that is relevant to short-term policy agendas and thus narrow opportunities for research focusing on longer-term - yet potentially still policy-relevant - issues, such as food and water security.

It is also possible that the impact agenda could inadvertently reduce the clarity of research messages. If academics believe that their work is less likely to be used if it is perceived to be too radical, then challenging findings may be reframed in less critical terms.

Finally, by encouraging researchers to promote their work beyond academia at a time when resources and capacity are being squeezed, it is possible that busy decision-makers will be overloaded with information. This returns us to the issue of quality. For if the REF impact assessment fails to consider the varying quality of studies (beyond the threshold), there is little guarantee that it will promote the better use of evidence or the use of better evidence, only that it will encourage academics to “tout their wares” outside academia. This may well be extremely unhelpful for the potential research users who are targeted.

Assessing the impact of research will always be difficult and somewhat subjective, and the six potential “impacts of impact” outlined do not lead us to conclude that the REF approach is necessarily flawed. Indeed, the multitude of criticisms directed at the impact agenda speak to the robust character of UK academia. Yet it is also true that by measuring and rewarding certain behaviours, the REF will inevitably shape behaviours, and worthy intentions do not guarantee desirable results. Good-quality research requires an attention to research ethics, alertness to potential problems, including the impact of researchers and methodologies on what is being observed, critical thought, reflexivity and a willingness to learn. Approaches to assessing research impact require no less.