Students are given more say in quality

Phil Baty looks at the new quality assurance framework and how it will work.

Ministers have insisted on a key role for students in judging university standards and setting the quality inspectors' agendas as part of the new quality assurance regime.

This week, higher education minister Margaret Hodge finally gave the go-ahead to the framework that will abolish routine assessment of teaching at departmental level in favour of a hands-off audit of internal quality assurance systems.

Most universities will be visited by Quality Assurance Agency inspectors once every six years, compared with the 30 to 40 visits every six years under the current system.

But ministers have demanded a number of measures to ensure public accountability that the QAA has privately acknowledged will be less palatable for university staff.

Under the new framework, agreed by the Higher Education Funding Council for England, vice-chancellors and the QAA, universities will be required to bring external reviewers into all their internal departmental reviews. These reviewers will be outsiders employed by the universities to cast an independent eye over standards.

Institutions will also have to publish a range of externally audited information on quality and standards.

They will publish summaries of the reports from their external examiners, who will be asked to "sign off" each degree course.

Announcing the framework, which applies only to England, higher education minister Margaret Hodge said that "students will be put centre stage in the drive to raise standards". They will have a forum to tip off auditors about shortcomings on their courses, before visits, so the audit teams can investigate. They will also be asked to complete a satisfaction survey.

Ms Hodge said: "The views of students will be taken into account in a new national survey of what they think of their courses, their teaching and their institutions, which will be regularly published.

"As higher education expands, we need a new approach that strikes a balance between reducing inappropriate burdens on institutions and ensuring high and continuously improving standards."

Peter Williams, chief executive of the QAA, said: "The framework puts responsibility for quality and standards where it belongs - in the institutions themselves. We believe it will meet the needs of all key players."

After almost a decade of hated teaching-quality assessments of every university department, the QAA has accepted that universities have demonstrated, albeit at tremendous cost in time, resources and even to staff health, that their quality is in general, "very high".

So the new system "can rely more than previously on internal HEI quality procedures".

The QAA has apparently won a number of concessions for universities. Institutions' concerns that "drilling down" activity during the new audits would merely be subject review by another name have apparently been assuaged, as audit teams will not include any discipline specialists for routine subject-level checks.

Subject specialist auditors will now be involved only if the auditors raise specific concerns about a university's systems, where full subject reviews will take place.

Roderick Floud, president of Universities UK, said: "This is almost certainly the best news on external quality-assurance arrangements that the sector has received since 1991. For the first time in more than a decade we have a coherent and soundly based system."

National Union of Students president Owain James said: "NUS will continue to urge universities to listen to their students and student satisfaction surveys are a step in the right direction. Students are paying for their education and have the right to independent advice."

Ministerial approval has tough conditions

Universities will have to make public an unprecedented amount of information about their quality and standards.

Sir Ron Cooke's report for the Higher Education Funding Council for England on the information requirements demanded under the framework is widely seen as the tough but necessary element that convinced ministers to endorse plans for the "light touch" audit regime.

The group said it recognises that its recommendations will mean some additional work and expenditure.

It said the following information should be published by each university and audited for accuracy by the QAA:

Qualitative data:

• Summaries of external examiners' reports for each programme. Full reports would remain confidential but universities would publish an examiner's checklist, certifying that assessments were fairly conducted, that standards are appropriate for the awards, by reference to national subject benchmarks, and that student performance compares with similar programmes elsewhere
• An institution-wide commentary on the findings of external examiners' reports
• Student views on the quality of courses, teachers and facilities, would be published for each university as part of a national survey on student satisfaction but it is unlikely the survey would go beyond institution-wide detail
• Institutions' own internal student feedback surveys should be conducted on a more consistent basis than at present and published
• Summaries of institutions' learning and teaching strategies
• Summary statements on the results of internal programme and departmental reviews, and the action taken in response to problems
• Summaries of links with relevant employers spelling out how employer needs and opinions are used to develop programmes. This could include employers' views on the graduates they have recruited.

Quantitative data:

• Student entry qualifications, including A levels, access courses, vocational qualifications and Scottish Highers
• Dropout rates, progression and successful completion for full-time first-degree students, as set out in Hefce's performance indicators
• Data on the class of students' first degree, by subject area
• Students' employment destinations, for full-time first-degree students, as set out in Hefce's performance indicators.

 
Audit trails and sign-offs after six years

Institutions will be audited every six years.

The audits will be conducted over a five-day week by between four and eight QAA-appointed auditors. The auditors will be nominated by universities and will not audit their own institutions.

Purpose

To show that the institution is providing degrees of "both an acceptable quality and an appropriate academic standard".

Academic standards will be judged against the "reference points" provided by the QAA's four-pronged "standards infrastructure", developed since 1997.

A qualifications framework describes the characteristics of qualifications to ensure the standards on a course are appropriate to its level; national subject benchmark statements describe the intellectual characteristics expected to be found in degree holders in each subject; programme specifications ensure each university spells out exactly what students are expected to learn; and a code of practice sets out guidelines on good practice in the management of quality and standards.

Areas covered :

• The effectiveness of internal quality-assurance systems
• The accuracy and reliability of the material universities publish on the quality of courses and standards of awards
• The examination of "a number of examples of the institution's internal quality-assurance processes at work at the level of the programme, or across the institution as a whole".

Institutions can expect between four and six of these discipline "audit trails". They will be determined on a number of factors. They may take place where there is cause for concern, for example where student representatives have signalled problems.

Full subject reviews involving subject specialists will be conducted where problems are identified by QAA auditors.

Timing

The first new audit will take place in spring term 2003.

A preliminary discussion will be held about nine months before each visit. In these discussions the QAA will "consider the basis for choosing discipline areas for inquiry during the audit".

Thirteen weeks before the audit, institutions will hand over documentation, including self-evaluation documents. Confidential submissions from students will also be considered.

At this stage auditors will decide the discipline areas that will be subject to audit trail, in consultation with the institution, which will know in advance which areas will be inspected.

Five weeks before the visit a three-day briefing will take place.

Judgement

After each audit, a judgement will be published on "the level of confidence that can reasonably placed in the soundness of the institution's management of the quality of its programmes and the academic standards of its awards, and, through direct scrutiny of primary evidence, whether the institution is securing acceptable academic standards and quality". There will be no "binary judgement", indicating simple "confidence" or "no confidence".

The QAA said it would be very difficult to give a good report on institutions that could not show a "strong and scrupulous use of fully independent external examiners" and a similar use of independent external participants in internal review at discipline or programme level".

The visit

During the audit the inspectors will expect to be given access to external examiners' reports and internal review documentation. They will interview staff and students and will see examples of students' assessed work.

Action

A report will be published after the institution has had a chance to add comments, representing the "signing off" of the audit. Institutions will provide a brief progress report one year after the audit, and another three years later.

Where problems are identified, a programme of follow-up action will be initiated. An action plan will be required, with progress reports at regular intervals. In extreme cases, failing institutions will be re-visited within a year. Failure could mean the withdrawal of Hefce funding.