Still crying out for some criteria

Only complete transparency in review procedures will lead to better research, says Eileen Gambrill.

In an article published by the Journal of Social Work in 2002, I described my experience as an external research assessment exercise adviser in the area of social policy, administration and social work. I noted that no clear criteria describing exactly how different kinds of publications were evaluated by assessment panels were provided despite repeated requests. I raised concerns about how fair the review procedure could be in their absence.

The response to my article heightened my concerns. Rather than describing the criteria used, we were urged to trust the process "arriving at best judgements in the round". It seems that the new method for research assessment published last week and emerging from the Roberts review will continue to deprive researchers of a clear description of the criteria used to review different kinds of contributions.

Roberts emphasises the continued use of peer review and the need for transparency, but there is a striking lack of attention to well-known flaws in peer review or concrete suggestions of how to address them. The review would have been a golden opportunity to require a clear description of the criteria panels use and a description of their reliability and of the sources of disagreements among panel members.

We need more than assertions concerning transparency and consistency. This is an opportunity to make transparency a reality rather than a hope, a claim or a slogan.

The literature describing flaws in peer review highlights the importance of openness, not only for fairness, but also for discovering how to improve the process itself. Such openness also allows each institution and individual rated to see whether the criteria were applied fairly and consistently over different institutions and individuals.

Although there are benefits to keeping the system secret, there are many disadvantages, including the use of evaluation criteria that are inappropriate to a particular kind of scholarly work.

Here is what I had hoped to read in last week's report: "In view of the need for transparency and consistency in peer review:

• Each review panel will identify, in writing, the criteria used to assess each different kind of scholarly contribution it reviews
• Each panel will describe disagreements that occurred about how to rate a given kind of contribution. This will allow a review of the reasons for and against the use of certain kinds of criteria so that all can see and learn from them and so that the process of peer review can be improved and made fairer and more accurate
• Each review panel will also calculate the reliability with which criteria are used. Information will be collated over all panels so that an overview of the process can be obtained. All information will be made available on our website
• Any researcher who has had a contribution assessed will, at their request, be sent a description of the criteria used to rate their contribution and their rating. This will be accompanied by a reliability rating and a description of any disagreements that occurred when assessing their work."

Although peer review may be the best way of assessing academic work, it is far from ideal. Only complete transparency of the process will enable us to seek closer approximations to a fair, just process that also contributes to a continued improvement in the quality of research.

Eileen Gambrill is a professor in the School of Social Welfare, University of California, Berkeley. She was an external reviewer for RAE assessment in social work, social work administration and social policy in 2000-01.