Reviewers change their mind after hearing from ‘peers’

Any academic who has submitted a research proposal or a journal paper will have felt the sting of an unwelcome comment from at least one peer reviewer – inevitably leading to grumbles among friends that said reviewer simply doesn’t know what they’re talking about.

Now a study by Harvard University researchers offers evidence to suggest that almost half of academic reviewers actually share that lack of confidence in their own expertise.

For the experiment, outlined in a working paper published by Harvard Business School, researchers recruited 277 academics from seven different US medical schools to assess funding applications. After evaluating a proposal, the scholars were shown a range of ratings and told that they had been given to the same application by experts in their field, although they had in fact been randomly assigned to be above or below a reviewer’s initial mark.

Having seen this, nearly half of reviewers – 47.1 per cent – chose to amend their rating.

The exercise was anonymised, and a control experiment in which reviewers were not shown others’ scores but were given the option to amend their own resulted in no changes, demonstrating that external influences were the key prompt for scoring revisions.

Significantly, women taking part in the experiment were 13.9 per cent more likely than men to update their review scores. Women working in male-dominated subfields were particularly susceptible to the influence of others, the study found.

Meanwhile, very highly cited “superstar” reviewers – who were more likely to be male – changed their scores 24 per cent less often than others.

Misha Teplitskiy, a postdoctoral fellow at Harvard’s Laboratory for Innovation Science and co-author of the study, said that willingness to change one’s views was not a bad thing in itself. “In fact, it’s generally good to take other opinions into account, all the more so when those opinions come from serious experts,” he told Times Higher Education. What was concerning, however, was that certain groups revised their scores more than others, “even when they have similar expertise”.

In all but one case, reviewers revised their scores towards those given by others. Reviewers who did change their own scores were most likely to do so by one point (86.6 per cent), but researchers cautioned that these “seemingly small updates can have dramatic implications for funding outcomes”.

The difference in self-confidence levels also has wider implications, the report concludes, because “individuals who underrate their competence may be less likely to apply for grants, ask for resources, or seek recognition for their achievements”.

rachael.pells@timeshighereducation.com