Research assessment in New Zealand could be marked down

The results of New Zealand’s six-yearly assessment of university research performance, the Performance-Based Research Fund (PBRF), were released on 30 April. Assessment panels studied evidence portfolios compiled by about 8,000 academics across New Zealand’s eight universities and rated the research performance of each.

As in the three previous PBRF rounds, the universities of Auckland and Otago, which house New Zealand’s two medical schools, were assessed as having the largest number of staff producing high-quality research. Between them, Auckland and Otago employ 53 per cent of the university staff who earned an A grade (meaning that they are deemed world class). As a result, they capture half the quality assessment funding distributed on the basis of the PBRF, amounting to NZ$315 million (£160 million) a year.

But that is only part of the story. In assessing research performance, scale matters. Auckland is by far New Zealand’s biggest university by any measure: revenue, staff or student numbers. But controlling for staff numbers, Wellington’s Victoria University ranks first, meaning that it has the highest concentration of research staff and the lowest proportion of staff who didn’t meet the standard for being rated. Otago rates second and Auckland fourth.

Controlling for student numbers, meanwhile, sees Lincoln University (New Zealand’s smallest university by every measure) rank first, with Otago second and Auckland third.

The release of the results from this fourth round captured less public attention than earlier rounds, with few stories making the non-education media. This is partly because, 16 years on, the PBRF has lost its novelty. It also reflects the fact that it is no longer possible to rank universities by subject, reducing the scope for them to boast.

This change from previous rounds is one consequence of a major review of the PBRF conducted in 2012-13 in response to longstanding complaints about the exercise’s high transaction costs and the opportunities it presented for universities to “game-play”. But despite the resulting simplification of the system – not to mention the tweaks following the evaluations that followed the first two PBRF iterations – the government has now commissioned yet another review.

Already under way, this is the result of a manifesto commitment made by the incoming Labour-led coalition government in 2017. Radical change – or complete abolition – of the PBRF has long been an article of faith for New Zealand’s Tertiary Education Union (TEU), and its officials worked hard in the run-up to the election to make sure that the Labour party knew its view.

In response to the latest results, TEU president Michael Gilchrist reiterated the union’s complaints about compliance costs, despite the system’s simplification. He acknowledged that the PBRF had increased the focus of academics on research but added that “after four gruelling rounds of individual assessments those benefits are well past”. He also complained that the refocusing had come at the expense of teaching and had damaged locally focused research.

The union, in addition, has always been concerned that the PBRF assesses each individual academic staff member’s research. Many staff have argued that the “unit of analysis” in any research assessment should be research groups or academic departments – or anything other than the individual staff member.

But reputation and prestige are powerful motivators and there is little doubt that making the individual staff member the unit of analysis changed the focus of most academics’ work – making research a priority for many. Ministry of Education analysis shows that, as a result, the introduction of the PBRF saw an increase in the quality and quantity of research. New Zealand universities boosted both their share of the world’s premium publications and their share of global citations. Those gains increased, moreover, as the PBRF bedded in. And, whatever the union might argue, there is no evidence of any fall-off in the quality of teaching.

The complaints and the lobbying have continued unabated, however. Will the new review lead to the complete overhaul sought by the TEU? And if so, will that come at the expense of rising research performance? Watch this space.

Roger Smyth is an independent consultant and adviser. He is the former head of tertiary education policy at New Zealand’s Ministry of Education.