An Adelaide-based “statistics wonk” has devised a new method for comparing researchers without allowing their gender, discipline or length of service to disadvantage them.
Flinders University ecologist Corey Bradshaw says he has produced an alternative to the ubiquitous but flawed h-index, which implicitly favours “the old farts – me being one of them”.
His “epsilon index” is calculated from publicly available citation data extracted from researchers’ Google Scholar profiles. “[It] corrects for most biases and allows assessors to compare researchers at any stage of their career and from any discipline,” Professor Bradshaw’s team reports in the journal Plos One.
“It ranks women relative to women, and men relative to men, normalises them and then puts them in the same rank,” he said. “Because women are being compared to women, it’s a fairer assessment.”
He said that it also ironed out differences between disciplines with low citation rates, such as archaeology, and those “where they tend to cite the hell out of themselves. You rank relative to their peers in the same discipline, and then standardise and put them together.”
The ε-index is derived from an algorithm that crunches four indicators: the researcher’s most highly cited paper, h-index, years of publishing and tally of papers with at least 10 citations. A publicly available “R code” and online app are available to perform the calculation.
The combination of the four indicators protects the index from being skewed by “one-hit wonder” papers, Professor Bradshaw said. The metric also corrects for career breaks by deducting time spent doing other things from researchers’ years of publishing.
Professor Bradshaw said that he already uses the index to shortlist applicants for jobs, promotions and Australian Research Council grants. He had developed it out of frustration with the h-index, a default metric despite being “full of holes”, and the 50-odd alternative measures that were rarely used because they relied on data that was too hard to collect.
He said that the h-index always favoured long-standing researchers, with their scores continuing to accumulate even after they died. “Early career researchers will inevitably be ranked lower based on citation metrics, and that really bothered me because it wasn’t a good indicator of the potential of these people.”
The team tested the index on a sample of 480 researchers equally divided across two genders, three career stages and eight disciplines. The new metric achieved “broad consistency” in comparing their research strengths “despite the considerable variation” in citation patterns, the paper says.
While “no single index should be used exclusively to rank researcher capability”, the paper recommends the new metric “whenever assessors must compare research performance among researchers of different backgrounds”.
Professor Bradshaw said that while he did not anticipate a stampede of academics using his index, they should at least “try it out”. It required them to copy a handful of numbers per applicant into a text file, which “doesn’t take that long”.
“If you couldn’t be bothered, then you’re just going to go on your gut, and that’s part of the problem. It comes down to how fair you want to be to the people you’re assessing.”
He said that his new approach was not perfect and could benefit from more refinements. “I’m always open to improvements,” he said. “There’s a lot of very clever people out there who could probably give me some tips.”
