Universities are moving to a more holistic assessment of their research culture after an over-reliance on quantitative data led to narrow thinking and misrepresentation
In 2013, a group of journal editors at a conference in San Francisco agreed that there was a pressing need to improve how research outputs are evaluated. As a result, they issued a set of recommendations: the San Francisco Declaration on Research Assessment (DORA). Nevertheless, almost a decade on, challenges remain.
At a virtual round table with Times Higher Education and Clarivate, a global provider of trusted insights and analytics, education leaders from across the globe met to discuss how data can be used responsibly to better assess research performance at higher education institutions.
Max Lu, president and vice-chancellor at the University of Surrey, explained that the misuse of metrics has started to erode public confidence in research and researchers. “When you look at this topic from the point of view of public confidence in science and technology, we need to maintain trust and confidence – as exemplified by the Covid-19 pandemic. Fortunately, I’m pleased to say the tide is turning to focus on the more appropriate and responsible use of data and metrics to inform, but not drive, assessment.”
Echoing Lu’s input, Vivek Goel, president designate at the University of Waterloo and vice-president of research and innovation at the University of Toronto, argued that universities had become lazy in conducting their research assessments, relying too heavily on quantitative metrics. “You can't compare institutions based on metrics alone without having some adjustment for qualitative differences at the programme or institutional level.”
The panel members agreed that individual metrics are hugely reductive but were optimistic regarding recent trends towards more nuanced, holistic assessments of scholarship, which look more at the contribution researchers make to a field or wider society.
Martin Szomszor, director of the Institute for Scientific Information at Clarivate, explained his role in developing products that enable users to engage more actively with the data underpinning research metrics, which hopefully leads to a process where individuals “question their interpretation of the data more deeply”. However, he also stated that “building trustworthy data sources is becoming increasingly difficult as more and more people are seeking to undermine the system to gain an advantage, acting in direct opposition to the values outlined by DORA”.
Putting a different slant on the discussion, Lydia Snover, director of institutional research at the Massachusetts Institute of Technology (MIT), noted that instead of looking purely at research citations, MIT’s outputs are increasingly measured in terms of the impact on the economy and how its graduates are changing society. “We use a lot of natural language processing to measure our outputs, including things like the intellectual property that our graduates generate, the patents they publish, the companies they start and the wider impact they have.”
The discussion also addressed the challenges presented by huge, collaborative research papers – so-called “kilo-authored papers” – and the problems presented by university ranking tables. Phil Baty, chief knowledge officer at Times Higher Education, acknowledged the latter challenge and stressed that ranking organisations is “encouraging the disaggregation of data to produce more complex, nuanced university profiles”.
Finishing on a positive note, Lu argued that we need to move away from relying on data alone. “We must develop a culture that is inclusive, supportive and positive,” he said. “Funders, institutions, ranking agencies and the wider academic community should come together to focus on building a culture that will ultimately support the world’s brightest minds to tackle the challenges we face.”
Watch the webinar on demand above or on the THE Connect YouTube channel.
Find out more about Clarivate.