Assessment reassessed

As Pounds 3,405 million of research funding is distributed according to the 1996 RAE ratings, THES readers examine the flawed basis of its judgements

In the year since I wrote in The THES about the double whammy which the teaching quality assessment and the research assessment exercise represent for university teachers, it has become increasingly obvious to everyone except the Government that no one can hope to achieve a high score in TQA and do well in RAE at the same time; or at least that anyone who gets 23 in TQA and 5 in RAE is either superhuman or just lucky in having only small groups to teach.

For the moment, though, TQA has no impact on funding. RAE does: just how much we are finding out now.

I would not mind so much if some RAE scores did not defy common sense. But, wishing to be paranoid and see conspiracy or prejudice at work, I can find only one rational explanation: those getting 3 or less are doing the "wrong" kind of research, or at least not enough of the "right" kind.

When the chips are down only monographs from academic publishers and full-length articles in refereed journals appear to count. All other forms of publication - translation, creative writing, bibliography, review articles, entries (however substantial) in reference books (however respectable), and so on - are not taken seriously by assessment panels.

So we must stop doing the things that earn us few Brownie points and instead concentrate exclusively on writing books for academic publishers and papers for refereed journals. I am now refusing all other approaches, although that is often hard to do, especially when the requests come from colleagues abroad who, quite frankly, think our research culture has gone mad.

They assume that someone who, like me, made his reputation when he was a junior academic, will, once he becomes a senior member of the profession, concentrate on fostering the research of those in the position he was himself in 30 years ago. They expect me to be supervising theses and dissertations, giving expert advice to monograph publishers, contributing review articles to research journals, writing authoritative entries in reference books, compiling bibliographies, and the like: activities that carry little weight with our assessors.

There is in most disciplines a natural cycle: you write the "big" books or papers when you are young, and as you approach retirement you guide the efforts and promote the work of those who will lead the profession long after you have gone. RAE subverts that natural cycle and (not to speak of the wholly disproportionate cost in time and effort) imposes on university teachers the degrading necessity of filling in at oppressively frequent intervals a sort of academic tax return.

But RAE is not about to go away, so we must focus on producing the publications which experience shows are alone acceptable to the assessors. Never was "publish or perish", once sneered at as an American aberration, a truer reflection of the shape of things to come. How we do it and teach the loads currently expected of us, God only knows.

JOHN FLETCHER

Professor of European literature at the University of East Anglia.

What is meant by research? This is a question that haunted departments in the run-up to the RAE. The only explicit criterion specified was that published work should be original - nothing was said about whether books were preferable to articles, or if research should always be documented by lengthy footnotes. Yet it has now become clear that the panels were looking for research of the same character as a PhD thesis: research was understood as sustained, carefully argued, richly documented, original study.

In retrospect, it is not surprising this was the understanding. The PhD still carries huge academic prestige. But as everyone knows, theses are hard to get published; as comprehensive investigations of very narrow subjects they are commercially unviable. Publishing houses, even university presses, must be aware of the marketplace. They refuse to produce unsaleable books, no matter how worthy.

This means that there is a discontinuity between the aspirations of academics and those of publishers. In order to gain promotion (and to satisfy the demands of RAE panels), dons seek to publish scholarly monographs. Publishers, on the other hand, are looking for more general works which will sell. With few exceptions, whenever a proposal is submitted to a publisher, the first question asked is whether there is a market for the book. The basis for acceptance tends to be its predicted commercial success, rather than its scholarly excellence.

This tension between academics and publishers is also evident in the reviewing process. Publishing houses send out review copies in the hope that their books will be widely noticed. Even bad reviews constitute free publicity. Academics, however, regard reviews as important critical comments. Good reviews are used as the basis for gaining promotion. Thus, while publishers are not particularly interested in the content of reviews, academics are agonised by these evaluations. A further paradox is that reviewing is regarded as the lowest form of scholarly activity. The most distinguished scholars are often too busy to keep to their reviewing schedule. Thus, reviews which cause so much heartache are often dashed off by less than expert colleagues anxious only to make the deadline.

The existence of the RAE has widened the gap between academe and the marketplace. Since the RAE will only accept published work, academics are desperate to find a press for their scholarly output. Yet publishers have to make a living, and everyone knows that scholarly monographs do not sell. Is there any solution to the problem? The answer is obvious. The role of universities is not only to extend the frontiers of human knowledge, but also to educate the general public. Thus, general studies and textbooks should also be given weight in the RAE remit. The proper place for the publication of pioneering research should be the learned journal with a limited circulation, since the commercial publisher can only be expected to produce books which have a wide appeal. Yet both products are an essential part of the work of a late 20th-century university, and both should be recognised in any future research exercise.

Rabbi DanCohn-SherbokUniversity of Kent, Canterbury 9With this week's funding announcement speculation about the future and re-evaluation of the research assessment exercise process abounds. The debates seem interminable but are, perhaps, a welcome displacement activity from the more arduous one of writing research papers and books. This leads me to ponder the relative merits of the procedures and how essential they are to research itself.

As a sociologist intimately involved in various ways, I feel professionally bound to reflect. In any event, as much of my reading for the sociology panel confirmed for me, a key feature of late modern society is what is now called "reflexivity", the inbuilt evaluations by individuals of their activities and subjectivities. I have been a wearer of many hats from subject-based to sectorwide groups. I was a member of the sociology panel, the person with special responsibility for my university's multidisciplinary submission to the social policy panel, a member of the committee of chairs of research committees in the new universities (CCRCNU) and a member of various professional groups.

In reflecting upon this multitude of roles and relative power bases I am sadly led to conclude plus ca change; that, despite that explosion of special interest groups and what appear to be new procedures, the changes to the system are relatively marginal. There may have been some modest shifts across the system from the sciences towards the social sciences. The rate of improvement of new universities as compared with the old has been greater, which is not fully reflected in the funding finally allocated. On the whole the old "system" continues.

What is remarkable is how wider social and cultural changes have had virtually no impact on this "male-centred" system of higher education. Women are spectacularly absent from the senior echelons of these various research communities, including the fact that there were only three female chairs of 70 panels, that women's studies as a subject in its own right managed to achieve only sub-panel status to the sociology panel, that I am the sole female member of CCRCNU, and that the successor to Dev R for the new universities has been called "son of DevR".

For the future from the point of view of women and research, the prospects remain bleak, with no woman having been asked to serve on the research subcommittee of the Dearing review. Changes in the research evaluation process are likely not to be so seismic - merely more tinkering with a male-dominated system.

MIRIAM DAVID

Director of the socialsciences research centre, South Bank University.

Will the funds awarded as a result of the research assessment exercise really be fair or will the fairness be distorted as a result of the way in which the ground rules were interpreted by the 102 universities which submitted their documents last May?

A student attending a university inter-acts with the whole university. Therefore it is more valid to recalculate the output of the assessment exercise on the basis that all staff in the university have been considered in the analysis. In this way a league table can be established which takes into account the average scholarly performance of all the academic staff within the university - not just the minority of staff selected by the university to be put up for consideration.

I have recalculated all universities' performance on this basis and come up with an alternative league table. This defines the average scholarly performance of all the academic staff for universities and eliminates the anomaly where some universities have been particularly effective in presenting their most advantageous image by removing a large number of non-scholarly staff from their research submission.

According to my analysis, Cranfield University would drop 41 places (from 17th to 58th) in the table if all academic staff of the university had been included in the assessment exercise. Napier would drop 25 places (from 69th to 94th) and the University of Wales College of Medicine would drop 23 places (33rd to 56th). These universities could be considered to have submitted highly efficient documents which greatly enhanced their position in the league tables while possibly reducing marginally the income that the exercise would yield to the universities themselves.

Universities which could have substantially improved their relative positions with more selective documentation include the University of Huddersfield (up 21 places from 88th to 67th), Aberdeen University (up 18 places from 55th to 37th) and De Montfort University (up 20 places from 83rd to 63rd.)

If future exercises are to be held, it is important for all universities to present their best situation. It would be preferable, though, since it is realistic for all academic staff to be seen and compared in the assessment exercise (as all should be involved in research activities), if the selectivity figures were produced to reflect the relative position of universities in relation to all the teaching staff of the establishment.

KEN STOUT

School of engineering, University of Birmingham. He can supply a full copy of hisre-calculated league tableon request

One of the most intriguing features of the RAE is the way in which the panels went about the business of assessment. For although it is well known that each department was assessed on a scale of 1 to 5*, it appears also to be the case that in some panels each individual was graded personally.

The HEFCE has acknowledged that these panels "formed a view about the research excellence of the output of an individual, before taking account of all the contextual evidence in deciding a final rating".

Suspecting that a process of individual assessment had been undertaken, I wrote to the chairman of the law panel, requesting details of my personal rating. In a response in the course of a telephone call, I was extremely surprised to learn that I could not be supplied with this. I was even more surprised to learn from HEFCE several weeks later that they too are "unable to meet" my request. It appears that the panels "did not record these individual scorings", it being "both unnecessary and invidious" to do so.

At one level it is perhaps reassuring that sensitive material of this kind is not recorded. But on the other hand, if it is the case that in some panels individuals have been scored personally, this means that judgements have been made, and that information has been generated about individual scholars on the basis of their work, and that the information so generated has been used, albeit only for the purpose of assessing departments. That alone is enough to give rise to questions about the transparency of the procedures and the accountability of the assessors.

If it is the case that a view has been formed about the work of individuals, it is at least arguable that they are entitled to expect the personal information to be made available to them confidentially on request: no one is suggesting that it should be published. It is also arguable that they are entitled to be told: (i) by whom the view was formed (was submitted work read by all the members of the panels?); (ii) the criteria which were developed by the panels in order to form that view; and (iii) the reasons for any view formed about them.

Transparency and accountability are not only self-evidently desirable on grounds of principle. Unnecessary secrecy is to be discouraged in academic life, of all places. But transparency and accountability are desirable as a matter of practice, for it is important that people should have an opportunity to assess the assessment of the assessors, and possibly also to challenge any judgement if it appears unfair or unreasonable. This can only be done if they we armed with information about who, how and why. It is clearly also fundamental that people know why their work did not reach the requisite standard of excellence, if they are to do better next time.

This is not to suggest that effective accountability is easy to achieve, nor that the arrangements adopted in the current exercise have operated unfairly in any particular case. But it is to suggest that if the allocation of research funds is to continue to be based to any extent on the scoring of individuals, steps should be taken to ensure that these individuals are given the fullest information about the way in which the process is conducted. The academics who take part in such assessments, and who accept the responsibility to judge others on behalf of a public body, are surely prepared fully to account for what they have done and should not be denied the opportunity of doing so.

KEITH EWING

Professor of public law King's College London

In theory, the idea of assessing research in order to provide an objective basis for funding is fair and reasonable; but in philosophy, at least, that is not how the RAE has worked in practice. The process is described as a "peer review", but what guarantee is there that the panel adequately represents the diversity of contemporary British philosophy? The short answer is: none.

Little attempt is made to present members of the panel as representative of the field. It is only a few years since a number of prominent philosophers opposed the award of a Cambridge honorary degree to Jacques Derrida. Are such philosophers suitable to conduct a "peer review" of the work of the followers of Derrida? Questions like this must be answered before the title of "peer review" can have any credibility.

The panel, it is claimed, assessed the quality of the work submitted objectively and impartially; but how "quality" was judged is shrouded in obscurity. "International" and "national" excellence are the key criteria. According to one panel member, however, these were not treated as "geographical" but rather as "value concepts". The panel does not explain or justify its decisions; nor is there any appeals procedure. At the end of the day, we are expected to defer to the wisdom of authority. This opaque and undemocratic system has been accepted with scarcely a murmur.

The exercise is designed to inject the "oxygen of competition" by rewarding "excellence" wherever it is found. It is unlikely to have this effect. "Oxford and Cambridge will get 5s," one member of the panel said, even before the submissions were in. Of course they did.

Warwick and Essex (the main centres for "Continental" philosophy) will not get 5s, he could well have added. They did not. As regards "international" excellence, a colleague was assured, "only the United States and Australia count". In short, what is regarded as philosophy in Oxford, Harvard and Sydney is the standard. If you want a high rating you would be well advised to follow that model.

This will have a deeply conservative impact. The system should be opened up and made publicly accountable. The panel should be selected by an open process with the aim of representing the diversity of approaches in the subject. The rating criteria should be specified more precisely, and the reasons for the panel's decisions explained and open to appeal. These are matters of elementary justice which echo Nolan.

Given the divisions within philosophy, they could be implemented only by taking some account of more objective standards. Sheer quantity of output is too crude, but citation rates and even market success could help make the system more responsive to diversity. These are crude indicators too, and should not be used mechanically. But rating philosophical work on a seven-point scale is a crude business; and at least they provide some objective indication of academic standing, relatively free of the subjective opinions of a few individuals.

When I was a student in the 1960s, British philosophy was entirely dominated by the narrow conception of the subject which then prevailed in Oxford and Cambridge. Mercifully, it has been transformed since then, even if old attitudes linger on.

Change came about because new approaches were cited and gained a market. These measures proved far more responsive to change and diversity than the judgements of senior academics of the sort who make up RAE panels. The sooner this diversity is duly recognised, the better for philosophy in Britain.

SEAN SAYERS

Lecturer in philosophy at the University of Kent. A longer version of this article will appear in Radical Philosophy in May