NSS results unrelated to teaching quality, study claims

Student satisfaction scores should not be used to measure teaching quality because they have no discernible link with exam performance, a leading University of Oxford academic has claimed.

While results from the National Student Survey are likely to be used as a key indicator in the government’s proposed teaching excellence framework, a study by Tim Lancaster, director of clinical studies at Oxford, concludes that they have “little or no value as a quality metric” .

Dr Lancaster compared the NSS results of 28 UK medical schools with the average pass rates achieved by their students in exams sat by all trainee doctors two years after graduation.

There was no correlation between good results in the NSS and performance in the exams set by the General Medical Council, according to the study “Assessing the quality of UK medical schools: what is the validity of student satisfaction ratings as an outcome measure?”, which is due to be published shortly by Dr Lancaster.

When medical schools were ranked by success according to both the NSS results and their pass rates, based on average scores achieved between 2008 and 2014, 13 schools did better on examinations than they did in the NSS and 14 did worse, the study says.

Only one medical school performed strongly on both measures, scoring top in both the NSS and on pass rates, according to the study undertaken with Tom Fanshawe, a statistician at Oxford’s Nuffield Department of Primary Care Health Sciences.

“One medical school with one of the highest NSS scores had the worst exam results,” Dr Lancaster, a GP and fellow at St Anne’s College, told Times Higher Education.

“NSS scores can still be relevant because they provide information about whether your students are happy or not, but [they do] not appear to correlate with teaching quality,” he added.

His study did suggest, though, that there is a strong correlation between institutions’ average entry scores for students – in terms of exams such as A levels – and subsequent medical assessment pass rates.

The study had wider significance for academia because medicine was one of the few subjects where a standardised “common national exam” enabled comparisons between students at different universities, Dr Lancaster said.

With universities minister Jo Johnson currently drawing up plans for the TEF, which will allow some institutions to raise tuition fees from 2017-18, the study’s findings are likely to stoke debate over which metrics should be included.

One of the NSS’ architects, Paul Ramsden, former chief executive of the Higher Education Academy, has argued that the survey’s scores are a “proxy for learning gain” because students who report better experiences gain better degrees, even controlling for entry scores.

But critics say that the NSS has fuelled grade inflation because students return positive results to lecturers who hand out higher marks.

Dr Lancaster concludes that if “the overriding measure of the success of a medical school is its ability to graduate competent doctors, the NSS appears to have little or no value as a quality metric for the teaching excellence framework”.

jack.grove@tesglobal.com