Is ‘risk-based’ quality assurance too risky?

The biggest shake-up of the university inspection system for a generation was unveiled at the end of last month, but it was overshadowed almost immediately by a newer, shinier policy.

Universities and science minister Jo Johnson used his first major policy speech to outline his vision for a teaching excellence framework that would “root out bad teaching”, halt grade inflation and force “coasting” students to work harder, just two days after the Higher Education Funding Council for England published plans to make fundamental changes to the quality assurance landscape.

While the TEF potentially could have far-reaching consequences, some argue that Johnson’s nascent and unformed ideas for the framework distract from the real game-changer in higher education: the new QA system set out in Hefce’s consultation document, titled Future Approaches to Quality Assessment in England, Wales and Northern Ireland.

For universities, the most attention-grabbing proposal in the document is the plan to exempt established institutions from the institutional reviews carried out by the Quality Assurance Agency every six years. Instead, there will be annual checks conducted by Hefce itself, which will analyse trends in student outcome indicators such as “student recruitment, progression and achievement” to see if a higher education institution is performing better or worse than similar universities. This will be combined with measures designed to “strengthen” governance arrangements and external examining, and to place greater reliance on universities’ internal review processes.

Removing cyclical institutional reviews for those institutions that have passed numerous QAA checks over the years is a sensible move away from a “one size fits all” approach, Hefce says in the document. While new providers should be subject to “detailed external independent peer review”, endless retesting against baseline requirements should not be necessary for established providers, particularly those predisposed to wanting to improve teaching quality, it suggests.

Similarly, universities’ governing bodies, which would be asked to vouch for academic standards at their institution, should be trusted to ask the right questions in order to guarantee that quality is upheld, the consultation maintains. Governors might be asked to sign an annual document stating that they had “challenged the executive where necessary”.

But there are concerns about the crucial role of overseeing established providers being taken away from an independent, accountable peer review process run by the QAA to a centrally managed, statistics-based vetting procedure, conducted by what one university quality officer calls “faceless Hefce bean-counters”.

Funding bodies and quality assurance processes should always remain separate, believes Geoffrey Alderman, professor of history and politics at the University of Buckingham and the former chairman of the University of London’s academic council. “Ofsted does not fund schools – it simply inspects them,” he says, adding that this separation is vital to safeguard against possible conflicts of interest, as well as ensuring transparency and accountability.

There are questions about how such a move would be viewed outside the sector. In the consultation paper, Hefce states that it would “publish confirmation, for each provider, that academic standards are set and maintained appropriately and are reasonably comparable” and introduce a “quality kitemark” system as evidence of the “continued good standing” of an individual provider.

However, the lack of a publicly available report – as is currently provided by the QAA – would arguably do little to build trust in the institution among staff, students, potential applicants and the sector more widely. This is one of the criticisms that has been levelled at the terse reports produced by Australia’s Tertiary Education Quality Standards Agency, which since 2011 has accredited institutions via a light-touch, metrics-driven system of checks, while allowing institutions to keep private their financial and institutional data.

Ian Kimber, the former executive director (regulation and review) of Teqsa and now director of quality development at the QAA, admits that the lack of transparency in Australia’s Teqsa reports is problematic. “Students need to know about the outcomes of [students’] work and want to find out what enhancement and improvement work is happening,” he says.

The transparency of the quality reform process itself has also been called into question, with critics arguing that no compelling reason for radical change to university inspections has been given.

When Times Higher Education asked Hefce to disclose some of the responses it received to the first stage of its consultation, a spokeswoman said that their release would “prejudice ongoing discussion”, “inhibit the free and frank provision of advice” and potentially create “perceptions that the outcomes may have been influenced by [certain] pressures”. That refusal comes despite advice in its own January 2015 consultation document that “responses to this document are unlikely to be treated as confidential except in very particular circumstances”.

A summary of anonymised responses has been published, but many will wonder whether the proposals are being driven by the sector or whether they are part of a bid to cut the estimated £1 billion regulatory burden on universities. In a document published alongside the Hefce consultation, consultants at KPMG estimated that this was the sum it cost the UK’s 130 higher education institutions to carry out required checks on quality. But this figure, and the estimated £90 million in potential savings outlined in the report, have been disputed by many universities who say that much of their work to remain QAA-compliant would need to be completed in any case.

Source: 

Getty

Some speculate that a more self-interested motive is driving the quality assurance reforms – self-preservation on the part of Hefce. Writing in THE in June, Gill Evans, emeritus professor of medieval theology and intellectual history at the University of Cambridge, pointed out that Hefce’s role as a funding body has been eroded, as the bulk of funding for teaching now comes from student fees, and that it now appeared to be countering that loss of influence by “beefing up” its role as lead regulator (“Will many hands make light work of regulation?”, Opinion, 11 June).

According to Roger Brown, the former head of the QAA’s predecessor, the Higher Education Quality Council, many will see the quality assurance plans “as Hefce putting itself in pole position if there is just a single regulator”.

“Hefce will be firmly in charge and governing bodies their unwilling lieutenants,” Brown adds, arguing that governing bodies made up of lawyers, accountants and other lay members do not have the “desire, skills or experience” to carry out that quality role.

The more fundamental shift to a “risk-based” quality assurance system is also likely to elicit many responses in the consultation, which runs until 18 September. (Responses on quality assurance in Wales are required by 31 August.)

This outcomes-based process, in which problems may be detected and resolved only after several years of a downward trend, could mean that affected students do not receive the consumer protection guarantees found at more frequently scrutinised institutions, argues Roger King, co-chair of the Higher Education Commission’s 2013 inquiry into higher education regulation, who has studied risk-based assurance in higher education.

“It means some consumers will not get the security that others get,” says King, who has previously warned that risk-based regulation potentially carries “increased risk for regulators and government, and key dangers for some institutions and their students”.

Champions of the current system will argue that it is far better to weed out problems before they flare up by using robust, cyclical checks on institutional policies, than to wait for a problem to arise and then act.

Others will also point to the failure of risk-based regulation in other sectors, such as healthcare, where neither internal governance nor analysis of outcomes at the Mid Staffordshire NHS Foundation Trust picked up extensive failings until many Stafford Hospital patients had died. Many contend that scandal – and deaths – could have been prevented if the concerns of staff, patients and their families had been heeded, and they argue that risk-based regulation requires robust whistleblowing procedures if it is to be effective.

The QAA already has a Concerns scheme for this purpose, which allows individuals to raise concerns about academic standards, quality and information provided by an institution, but this should be strengthened if the new risk-based method is introduced, argues David Lewis, head of the Whistleblowing Research Unit at Middlesex University.

“A risk-based approach is based on the assumption that there is an open culture where people will speak out freely,” says Lewis. This is not always the case at universities, where collegiality is highly valued and those breaking the code can find themselves ostracised by peers or even forced out of their institutions.

“People want to know they will not suffer retaliation,” adds Lewis, who believes that whistleblowing does make a difference at universities, even if successful cases generally go unreported.

Under the consultation plans, concerns about standards or quality at an institution would be reported by external examiners, the Office of the Independent Adjudicator, student groups and others to Hefce. “A quality assessment system that adopts a risk-based approach to routine monitoring must have, and be seen to have, ‘teeth’ where serious problems are identified,” the document says.

Many publicly funded universities are likely to welcome Hefce’s proposals, particularly if they escape regular institutional inspections by the QAA, according to King.

“Once an institution has been through the QAA review a few times, the process begins to feel very ceremonial and procedural,” he says. “People feel as if they are going through the motions, rather than learning much from the process, so they will welcome a relaxation of external scrutiny.”

But Hefce’s proposed audit system may not be as light touch as promised, with increased requests for information potentially proving just as onerous as the current QAA inspection system, King warns. “Risk-based regulation is an attractive proposition as it seems to promise deregulation for a great deal of providers,” he explains. “But when it comes to operationalising it, the new data required often leads to more compliance measures, rather than less”.

The introduction of risk-based assurance in Australia “quickly ran into trouble”, leading to a major review in 2013 of what data should be assessed just two years after the regulator was set up, King adds.

“Risk-based regulators tend to gather more information than is really necessary and keep a close eye on everyone, despite what they say,” he argues. “If a scandal happens somewhere they considered low-risk, they will face a backlash, so they take a lot of trouble to protect themselves by collecting more and more data.”

Any regulatory system will have some element of risk, but will Hefce’s lighter-touch approach be enough to provide students and taxpayers with the protection and reassurance they seek in a new risk-laden phase of funding cuts, growing undergraduate numbers and increased competition between institutions?

Hefce plans to pilot the new quality assurance system in 2016-17, and roll out the new system across the sector in 2017-18.

Fiats, fiefdoms, skirmishes and rebellions: a brief history of the quality wars

The planned move to a risk-based inspection system is the latest chapter in the long-running debate about quality assurance in UK higher education.

The story can be traced back to the late 1980s when, faced with the risk that the government might impose an inspectorate on them, universities agreed to set up a body that would monitor and review their practices and procedures in respect of academic standards.

This was the Academic Audit Unit, established in 1990. It carried out site visits, looked for evidence to support institutions’ own assessments of their performance, and then offered not recommendations but “points for further consideration”.

The AAU was only a year old, however, when the government announced its intention to allow polytechnics to gain university status, and this was enshrined in law the following year with the Further and Higher Education Act 1992. This change was to be accompanied by the introduction of new quality assurance arrangements for universities that would also replace the Council for National Academic Awards, which until then had been responsible for the other side of the binary divide.

The solution, put in place in 1992, was another dual regime. A new Higher Education Quality Council took on the role of monitoring systems and structures that supported teaching, but also took on a responsibility for sharing best practice.

Meanwhile, each higher education funding council was given the role of assessing the quality of the teaching that was actually provided, through subject-by-subject institutional inspections.

This system proved to be deeply unpopular with universities, who complained that the administrative burden was too heavy. It was also thought that having one organisation overseeing quality assurance, rather than two bodies competing in the so-called quality wars, would prove more efficient and effective.

As a result, the Quality Assurance Agency for Higher Education was set up in 1997 and, by late 2000, John Randall, its chief executive, had drawn up plans for a new academic review regime that would succeed the twin-track approach. Whether such a system would adequately reduce the administrative burden, however, was hotly debated.

Some Russell Group universities openly began to consider seceding from the QAA system. The response of David Blunkett (pictured, above), who was then education secretary, was to announce in 2001 that the volume of inspections would be reduced by 40 per cent.

All departments that had achieved good scores in the latest round of subject reviews would be exempted from the next round. By agreement, a small proportion would be sampled to provide a benchmark of good practice, Blunkett said.

Randall resigned, warning that such a “light-touch” approach would short-change students and employers. Nevertheless, his departure cleared the way for the introduction of a slimmed-down system in which, after an initial three-year cycle, institutional audits would take place every six years.

Wales broadly followed England and Northern Ireland in introducing new arrangements of this kind. But since 2001, quality assurance in Scotland has been based around a four-year cycle of institutional reviews, with a subject review process applying only at newer universities or where specific concerns have been raised. The emphasis, however, is on quality enhancement, with the aim of identifying and sharing good practice.

In England, the new quality assurance arrangements continued with little change for several years. However, they became the subject of debate again in 2008, when Peter Williams, then the QAA’s chief executive, publicly questioned the comparability of degree standards between different institutions. The following year, MPs on the Innovation, Universities, Science and Skills Select Committee laid the blame on the QAA itself, which they said had “no teeth”, and the result was a number of changes to the institutional review process.

It did not take long for the pendulum to swing back in favour of universities concerned about the administrative burden of quality assurance. The 2011 higher education White Paper, Students at the Heart of the System, proposed a risk-based regime under which leading institutions would have faced review once a decade, or perhaps not at all.

“For new providers, with an inevitably shorter track record of quality, a more regular and in-depth review is appropriate than has previously been applied. Conversely, for those providers with a sustained, demonstrable track record of high-quality provision, we would expect to see significantly less use of full institutional reviews,” the White Paper said.

However, the Higher Education Funding Council for England decided in 2012 to stick with the existing arrangements, although risk-based variation in the interval between reviews was introduced in 2013-14. Hefce revived the idea of a fully risk-based approach last October.

Williams, who retired in 2009, says that he believes that the QAA and its predecessors have done a “remarkably valuable” job in raising basic standards while protecting institutional autonomy.

But the past two and a half decades have shown how key tensions have remain unresolved, with ministers at times being won over by universities’ complaints about what they see as outside interference, and at other times wanting watchdogs to crack down harder on perceived lax standards.

With a planned teaching excellence framework now raising the prospect of a return to a twin-track system, these tensions show little sign of fading away.

Chris Havergal

Source: 

Getty

Exchange rate mechanisms: degree classifications return to the spotlight

The new quality assurance regime may herald a “light-touch” approach but it also brings a new focus on the comparability of degree classifications.

“Demonstrating that standards are maintained and are reasonably comparable across the system, despite market and other pressures, should be the core, non-negotiable component of a quality assurance system,” according to the Higher Education Funding Council for England’s consultation on the future of quality assurance.

The report notes concerns in the press about “grade inflation” and says that the sector has done little to counter such narratives. It is important for degree classifications to be “reasonably comparable”, particularly at the borderlines between a 2.1 and 2.2 and a pass and fail, it says.

The document proposes the establishment of subject-based “communities of peers”, with external examiners from across the sector meeting regularly to compare students’ work and calibrate standards – an idea discussed in a Times Higher Education cover feature last year (“Unknown qualities”, Features, 13 November 2014).

Universities’ governors would need to confirm in an annual statement that standards are comparable, and a task force should be set up to look at the algorithms used to decide degree classifications at different institutions to “determine a sensible range”. The proposal follows research on external examining by the Higher Education Academy that found that almost half of institutions surveyed had made changes to their degree classification algorithms in the past five years “to ensure that their students were not disadvantaged compared to those in other institutions”.

Times Higher Education reporters

Time’s up? QAA’s future is uncertain

The consultation raises a question mark over the future of the Quality Assurance Agency, which is not mentioned in the text of the Higher Education Funding Council for England’s document.

In a surprise move in October last year, the funding council said that it would invite external bodies to bid for work undertaken by the higher education watchdog.

However, the ideas outlined in the report, Future Approaches to Quality Assessment in England, Wales and Northern Ireland, suggest that the QAA could make a bid to retain some of its review work, despite plans to abolish cyclical QAA reviews of established providers.

According to the document, a number of quality assurance roles will be up for grabs, including the operation of “detailed external independent peer review” of providers seeking to enter the system, and reviews designed to re-test established providers where problems have been identified.

The development of a register and training for external examiners will also need to be overseen, and all these functions “would be undertaken by one or more organisations external to the funding bodies”.

Roger Brown, former head of the QAA’s predecessor, the Higher Education Quality Council, says that if the plans go ahead “there will be plenty of work for QAA but it will not do what it was set up specifically to do: assure academic standards and quality across the sector”.

Times Higher Education reporters