It’s high time we rethought policy impact

Two decades on from the first discussions about policy “impact” in higher education, UK universities have now fully swung behind this agenda. Indeed, the Research Excellence Framework’s inclusion of impact case studies, and UK Research and Innovation (UKRI) pathways to impact, have galvanised far greater engagement between researchers and policymakers.

But these frameworks for rewarding policy impact have also produced a number of unanticipated effects. REF definitions are based on simple, linear models of policy impact, which fail to capture the often indirect, incremental and serendipitous ways in which research can shape policy. Such frameworks overlook more subtle processes of learning and co-production, and the role of public engagement in enriching and diversifying public policy debate. And they tend to encourage researchers to gain credit for their individual impact, rather than working in collaboration or building on wider bodies of knowledge. Not least, impact frameworks fail to address forms of bias built into research-policy relations, with certain types of researchers more likely to gain traction in policy circles.

• How much should you publish?
• A guide to writing grant proposals
• How I plan to get through REF results day

So how can we promote forms of policy impact that avoid these pitfalls? In a new report, we suggest that funders – notably the REF and UKRI – need to understand their role in a much broader way. Importantly, the frameworks they devise are not just tools for evaluating performance or selecting projects to fund. Rather, these tools are generative: they have profound effects on the behaviour of universities and researchers, shaping what research they carry out, how they engage with policymakers and how they marshal their research to influence policy. Funders need to take these generative effects seriously, taking responsibility for the wider ramifications of their impact frameworks on the sector.

The other report authors and I propose six core principles that need to guide policy impact frameworks:

• Collaboration: Researchers should be encouraged to work collaboratively across disciplines and institutions, rather than homing in on their individual team or institution’s research.
• Bodies of knowledge: Similarly, researchers should be encouraged to build and communicate wider corpora of insights and evidence, rather than only focusing on their own findings.
• Equity and Diversity: Under-represented groups should be given targeted support to create a level playing field and diversify the research informing policy.
• Quality of policy engagement: Policy impact frameworks should reward “productive engagement” and co-creation as ends in themselves – rather than simply as a means to impact.
• Public and community engagement: Such frameworks should encourage researchers to engage with publics and communities, to enrich, inform and broaden the parameters of policy debate.
• Disruptive research: Support for policy impact should not crowd out or devalue innovative and potentially disruptive blue-skies research.

Alongside these principles, reform of policy impact frameworks should also take into account the more practical issue of resource constraints: changes to impact frameworks should, where possible, limit the burden on funders and HEIs. Moreover, given the political need to demonstrate the benefits of research, impact frameworks should be designed in a way that communicates the societal and economic impact of research in a compelling way.

Building on these principles, the report sets out a number of recommendations targeted at the REF, other funders and HEIs. The proposals work as a package, designed to have mutually reinforcing effects across these organisations.

One of the recommendations for the REF is retaining the focus on case studies. This format allows universities to showcase a wide range of research and impact while focusing on a selection of activity that does not require all researchers to engage in impact. Case studies are also a compelling way of communicating the benefits of research to wider audiences. However, case studies should be assessed not just based on impact but on the quality of engagement put in place to achieve that impact. And the REF should loosen its requirement for underpinning research to be of 2* quality, because this discounts valuable forms of activity – including the crucial role of syntheses and reviews in informing policy.

We also suggest ways in which REF guidance can be oriented to encourage collaborative submissions, moving away from the focus on “heroic”, individual impact. Crucially, the report offers a number of recommendations for promoting equality and diversity. It proposes that the REF asks universities to explain how they address barriers to impact and engagement faced by under-represented groups. We also suggest an early career flag for impact case studies to avoid case studies being authored predominantly by senior colleagues.

The recommendations for UKRI and other funders focus on joining up activity across often discrete projects. Hubs should be created for supporting impact and engagement across related projects. The advantages of this grouped approach would be twofold: researchers would benefit from more specialised support for their engagement activities, while policymakers would appreciate more synthesised and efficiently brokered inputs from a wide body of research. Funders should also do more to bolster public and community engagement, bringing in diverse voices and fostering more inclusive public policy debate. And they should continue to provide a space for blue-skies research that retains a (sometimes critical) distance from government.

Finally, we turn our attention to the role of higher education institutions. It has often been pointed out that universities are overly cautious in how they interpret REF and funder guidance, for example opting for “safer” impact case studies. The tendency to gold-plate funder guidance is not an easy one to counter, given the huge financial and reputational stakes. Crucial here will be more effective communication between the REF, funders and HEIs, creating a level of trust that enables universities to take more risks – for example, by supporting more collaborative approaches through synthesising research and pooling impact support across institutions – as well as by favouring diversity over the “usual” (often senior, white, male) impact case study authors.

Taken as a package, these measures could go a long way to achieving more equitable and effective forms of policy impact.

Christina Boswell is professor of politics and vice-principal for research and enterprise at the University of Edinburgh. She is also vice-president for public policy at the British Academy.

This article is based on the Royal Society of Edinburgh report Rethinking Policy Impact: Promoting Ethical and Effective Policy Engagement in the Higher Education Sector by Christina Boswell, Katherine Smith and Cleo Davies. The project was funded by the Economics and Social Research Council.

If you found this interesting and want advice and insight from academics and university staff delivered direct to your inbox each week, sign up for the THE Campus newsletter.