Many more UK research projects influence policy than official figures based on self-reporting suggest, with social and environmental studies having particularly strong and widespread impact, a new data analysis indicates.
The analysis compared citations in a large global database of policy documents called Overton with outcomes recorded by UK Research and Innovation’s Gateway to Research (GTR) portal for information on research grants.
It found that between 2014 and 2020, hundreds more research projects led to references in policy documents produced by governments, charities and thinktanks than the UKRI figures would suggest.
For instance, about 700 projects funded by the Natural Environment Research Council produced papers that received at least one policy citation, according to the Overton data, but UKRI figures show that only about 100 projects had been recorded as having policy influence.
Martin Szomszor, founder of research analytics firm Electric Data Solutions, which carried out the analysis, said he also compared UKRI and Overton data on projects that had widespread impact by influencing policy across the world or in many documents.
Although this showed a smaller gap, there was still a noticeable difference. For example, for NERC, almost 200 projects had at least one paper in the top 10 per cent of Overton citations, while fewer than 50 were recorded as having global influence in the UKRI data.
Another research council whose projects appeared to have widespread global impact on policy was the Economic and Social Research Council, where about 180 projects led to papers in the top 10th of cited articles for that year and subject.
Policy and other outcomes recorded by UKRI in its GTR data portal are typically self-reported by project leaders or their universities.
Dr Szomszor, former director of Clarivate’s Institute for Scientific Information, said the results of the analysis were not a criticism of the current system because it was virtually impossible for grant holders to be aware of every single occasion when documents such as clinical guidelines or a thinktank paper referenced their work.
But the analysis did suggest that research was having a large and wide policy impact, and being able to demonstrate this could help disciplinary areas such as the social sciences better advocate for funding.
Analysing databases such as Overton might have other potential, Dr Szomszor added, such as looking at whether different countries were using the same research to inform policy.
“Do multiple countries, say within Europe, base their policy on the same research, or are people looking at different evidence to inform this policy? That strikes me as quite an important question,” he said.
He added that such a citation database could also benefit areas such as social sciences and the arts, which “are often difficult to analyse with [traditional] bibliometrics because of the low volume of citations”.
However, Dr Szomszor, who has written a preprint exploring the Overton data alongside the platform’s founder Euan Adie, said that the community had to be careful that it did not create another set of metrics used to push incentives on to academics.
“Inevitably, that is a double-edged sword [using metrics to assess the arts and humanities’ impact] because as soon as you open up that opportunity, you’re inviting misuse of those metrics, and that’s something that would have to be considered,” he said.
