When alpha is a non-starter

Between 1983 and 1985 I held an Economic and Social Research Council project grant - the first that I had ever applied for. Since that time my research has been funded by a variety of European Community and business sources.

Until 1995 I did not apply for further ESRC funding for my research. However, given the status of peer-reviewed ESRC funding for the research assessment exercise exercise, I decided to apply. Both my applications were graded alpha but were unfunded. Both of the rejections included copies of the referees' comments but no other comments.

The rejection letter from the ESRC on my first research proposal said: "The board's decision whether or not to recommend an award is based on the advice of referees, coupled with the judgement of board members. Thus the reasons for a particular outcome can often be complex and related to assessments made by a number of people. Because of this complexity, and because of the very considerable administrative costs involved, it is council's policy not to provide reasons for declining to support individual applications."

I read the referees' reports on my first proposal with interest. They seemed supportive but argued that the research findings would be best tested in an industrial context. Other comments seemed to me easy to counter. None of the comments provided a critique of my proposed methodology or of my theoretical conceptualisation.

Accordingly, I revised the proposal taking into account the referees' opinions. I revised and resubmitted the proposal but received another letter from the ESRC which said that "I have to tell you that in its present form, the application cannot be accepted for consideration . . . the proposal is a resubmission of an earlier application with the same title and the council has regulations concerning resubmissions which are strictly applied . . . uninvited resubmissions may be exceptionally considered provided you can demonstrate that the proposal has been substantially revised. We also require a covering letter to accompany the application which summarises the way in which this has been done."

So, I composed the missing covering letter and sent it to the ESRC. Its response was, to my mind, revealing. It said: "I have carefully considered the points you raise concerning the changes made . . . but have to inform you that the decision not to accept this as a resubmission has to stand. As approximately 80 per cent of the applications submitted to the board do not obtain funding, I am sure you will appreciate that the criteria for substantial revision has to be strictly applied . . . we would expect . . . considerable rewriting . . . For your future reference could I note that we cannot guarantee use of the same referees in resubmission."

So, unless a proposal is substantially revised it will, on revision, not pass the gatekeepers at the ESRC who control access to the referees. My contention is that mildly and sensibly revised proposals (in which the methodological and conceptual underpinnings are technically sound) that are re-orientated to match (or counter) the particular concerns of individual referees would be likely to elicit improved ratings from the original referees. The policy of gatekeeping practised by the ESRC blocks this sequence of events.

Compare the ESRC's evaluation of research with that of a refereed journal. My observations here stem from my editorship of the Wiley journal Journal of Behavioural Decision Making. An editor receives unsolicited manuscripts for consideration. Copies are sent out to referees for evaluation. Referees' reports are returned to the editor.

It is unusual for all referees to recommend straightforward acceptance. Most recommend changes. Often the changes recommended are contradictory. It is the editor's job to balance and weigh the referees' comments and recommend either rejection or give guidelines for a successful revision. If the referees are, to a reasonable extent, in favour of publication, then the editor will recommend what revisions are, in his view, necessary to secure acceptance.

The author is then invited to resubmit a revision with a covering letter which details how the referees' concerns have been addressed - or why they have not been addressed. On receipt of the revised manuscript, the editor then decides whether to accept the manuscript or send it back to the same set of referees (together with the covering letter).

The referees then submit a second report to the editor. Since the referees who compose the second report are identical to those who produced the first reports, the goalposts that the author has been asked to aim for are unlikely to be moved.

On consideration of the second set of referees' reports, the editor then usually makes a decision to ask for further (usually minor) amendments or rejects the paper. In my experience, the majority of manuscripts that proceed beyond the first stage of the reviewing process (ie secure a reasonably positive set of reviews) are eventually accepted. Citation counts of accepted papers that have been accepted by Journal Behavioural Decision Making do not show a positive correlation between favourableness of initial reviews and subsequent citations by other authors writing in the source journals of the Social Science Citation Index.

The impact of the ESRC's gatekeeping policy is to discourage resubmission of alpha-rated (but unfunded) proposals. Mild criticism by referees cannot result in mild revisions, since all revisions must be "substantial". No attempt is made by the ESRC to communicate their "complex reasons" as to why an alpha-rated proposal has been rejected for funding. So the proposal's author is left wondering if the area of the proposed investigation is, in itself, thought of worthy of funding. Should he submit another proposal in the same research field (in my case, the role of judgement in forecasting)? Is it worth the effort? How can one find out?

One other impact of the ESRC's gatekeeping policy is to reduce the number of proposals at each research round. There are three major research rounds a year where the ESRC makes decisions on research proposals. Last November, 284 applications were considered and 179 applications were alpha graded. Of these, 79 were funded and only two of the alpha-graded applicants were invited to revise and resubmit at a later date.

In my view, the other 98 should have also been invited to resubmit. A procedure analogous to the one I described for a journal's operation should then be implemented. That would mean, of course, that there might be near 400 applications (300 new and 100 revisions) for the next research round. Most of the revisions would receive improved reviewers' ratings.

Many of these would then be difficult to reject for funding. Since research funding is a limited resource, it follows that a lower proportion than previously of the 300 new applications would be funded. In time, the majority of funding would go to revised proposals rather than new proposals . . . This situation would equate to a journal's publication process, in which all (to my knowledge) manuscripts are revised before publication. In my view, a change in the ESRC's procedures for the consideration of research proposals to match that of a refereed journal's procedures would provide the research proposal's author with a stronger sense of fair process and could not degrade the quality of the research that is eventually funded.

Currently, the ESRC's processes seem unfair to unsuccessful, alpha-graded, applicants. The only benefit is that fewer applications need to be processed by the ESRC. This administrative benefit is, to my mind, far outweighed by the negative perception that the ESRC will gain in academic eyes, if the current gatekeeping process continues.

Most academics produce good ideas for research projects only infrequently. Their (personally prized) ideas deserve evidence of an even-handed and thorough evaluation.

Authors who receive, what appears to be, an unjustified rejection from a journal re-evaluate the journal's worth and try elsewhere. Social science researchers have little choice but to submit their research proposals to the ESRC.

George Wright is professor of business administration and deputy director of Strathclyde Graduate Business School.