Scrutineers perpetuate reign of error

The assessment of research at Hong Kong's universities has confused quantity with quality, argues Orlan Lee

Hong Kong's academic community must be one of the most closely scrutinised in the world.

The University Grants Committee, the government-appointed committee that administers funding to the nine higher education institutions, conducted research assessment exercises in 1994 and 1996, and the teaching and learning process review in 1997.

In July another research assessment exercise - the third in five years - was announced for 1999.

In the 1996 exercise, the research "output" of academics in Hong Kong's seven traditional institutions was assessed to determine the university research budget for the next three years. In less than nine months, 110 experts, many chosen worldwide, reviewed 14,000 publications by 3,330 academics.

But only 57 per cent of all assessed faculty members made the "threshold" cut-off. But, the committee seemed caught off-guard when a number of staff at Hong Kong University were concerned enough about the potential effect of such quantitative measurement of their careers to request access to their scores under Hong Kong's new Personal Data Ordinance. The UGC consulted legal counsel, but claimed that the records had been destroyed upon evaluation.

The exercise, which was undertaken with the advice of many eminent scholars and university administrators from around the world stemmed from a widespread concern about the accountability of a publicly funded mass higher education system.

When the secretary general of the UGC, wrote to vice-chancellors in March 1996, he indicated a desire to undertake an administrative "assessment" of academic research in the UGC-funded institutions, and proposed two months of consultation on the nature and scope of the 1996 exercise with the 1994 subject panel chairmen and members. The assessments would start in July and provisional results were to be compiled in January 1997, and the exercise was to be completed by April 1997.

The purpose was to be solely "qualitative", ie, to determine whether the UGC was getting its money's worth from the government's institutional research budget. The problems from an academic perspective, were, however, that the assessment would be "quantitative" - how much "output" was published, and in which publications of what "prestige" ranking (although this was denied).

The measure of academic "productivity" in Hong Kong universities, like that at other "world-class" universities, is, by tradition, assessment by the profession. When one applies for appointment, for renewal of contract, or for tenure, one's work is submitted to authorities throughout the world.Evaluation of applications for awarding of research grants by the Hong Kong UGC follows the same pattern. This information is not destroyed upon completion of appointment decisions, at least. Therefore, presumably, it could have been re-evaluated by the funding authority. On the other hand, an entirely new "assessment" in traditional fashion, on the scale of what was now contemplated by the UGC, would have presented a monumental task that could hardly have been accomplished in time.

The declared "principle and philosophy" of this exercise explained: "Each UGC-funded institution has its own roles and missions I Not all academic staff need to be active researchers".

It is difficult to escape a somewhat condescending distinction between research and teaching in this statement, when one would have hoped that concern for "quality" on both counts would have produced an expectation of unified "research-based teaching".

The draft guidance notes continued that all academic staff, holding appointments at the institution concerned for a continuous period of 12 months from December 31 1995, and not holding concurrent paid positions at other institutions, were "invited" to "submit up to five items of output" from the five-year period from January 1 1992, to December 31 1995. Output was defined as "any publication, artefact etc. that is publicly accessible,provided it was published or officially accepted (without need for further amendments of any kind in that periodI" Non-traditional output - concert performance, video tape, computer software programme, or any creative work that can be evaluated could also be submitted.

It was conceded that "works of great impact could take more than four years to complete" but the notes added "they tend to be infrequent". In addition, "individuals (might) submit ... a description (limited to one page) of such things as honours, community or other service, and professional activities if they deem(ed) it appropriate and necessary".

It was apparent, verbal concessions to works of great impact and public service aside, that the design of the 1996 exercise was to capture a picture of a particular staff member's publication productivity, measured by reference to specific output during the designated census period. This "picture" was also meant to relate to the individual staff member, not the work in question.

Quality was, therefore, given a definition akin to what, in legal parlance, we take to be the measure of the standard of professional conduct in a given area when we are forced to determine the converse, ie, "professional negligence". Moreover, to serve the purpose of comparability of data, it was apparent that the intention was that quality must be quantitatively scored for individuals, not as scholars, but as staff members of "cost centres" of the various institutions. It was conceded, of course, that such an assessment might produce negative as well as positive results.

The UGC declared that it did not intend to publish comparable data tables for individual institutions. Yet, obviously, such data would be employed to reach variable funding commitments for the institutions up to 2001.

Prospective funding decisions made in this "what have you done for us lately", manner were bound to affect careers. Since the teaching contract span in Hong Kong is generally three years, this could very well affect funding for the entire period or a major portion of appointment. Moreover, for academic staff not on terms of substantiation, contributions to their retirement funds are made only upon completion of contract. Therefore, a resultant decision to leave early could mean loss of such contribution.

A period of big budgets followed by an era of contraction opens the way for management theory planning. To that extent we cannot quarrel with the need for institutional economies. It is rather the mixing of industrial productivity theory with notions of academic excellence that makes us cringe.

There may be those who pass for "dead wood" among their colleagues - either because they have ceased to contribute to the advancement of knowledge or perhaps because they never had a part in it. It is another matter, however, to judge every scholar's career by a picture of what has occurred in any one particular calendar-defined five year capsule. An academic career is not an assembly line. There are also scholars who pursue a second course of study, or who work in public service. Does the UGC contemplate shutting these people out of academic life?

Can the UGC assess whether scholars who follow a single path are not just producing more and more of the same results in different "venues" year after year?

Scholarly achievement is either measured by the total picture of a person's career or it has little meaning.

Over the past 20 years, higher education has had lavish funding and unprecedented growth. Full-time equivalent enrolments increased from 42,000 in 1990-91 to 62,000 in 1995-96, prompting concern about how institutions will be able to maintain the quality of their teaching and research.

During the expansion, universities sought to compete with world-class universities elsewhere.

Today, however, they are in a phase of consolidation, restructuring and re-evaluation in which, despite continued expansion in some areas, economies are encouraged in others.

Orlan Lee is lecturer in law, Hong Kong University.of Science and Technology.