Questions, questions

Have you raised the banners yet? Ordered the free hot dogs? Set up the laptops? Yes, it’s that time again. Up and down the land, exhortations will flash across plasma screens, encouraging messages will fill noticeboards, and bright-faced student ambassadors will lie in wait at strategic points ready to grab any passing third-year. For next month, the National Student Survey begins.

Since 2005, when the Ipsos Mori poll first began, this annual judgement day has assumed more and more importance. It is increasingly viewed by the media, student applicants and their parents, and even the government as the absolute barometer of university health.

It’s not as if we’re not always trying to improve what we do. But we’re now so aware of how influential the NSS has become that we put massive efforts into making sure our students know that we have responded to their concerns.

Naturally we can’t directly influence what they say. Bribery is out. (We all remember the hapless lecturer at Kingston University a couple of years ago who was rapped on the knuckles for telling her students that they wouldn’t get jobs if the university got a bad score.) Instead, we operate more subtly. We remind students that they must give a fair reflection of their whole three years of study and not just their immediate mood. We point out that the survey is not a vehicle for complaint, as there are plenty of other outlets for that. We try to avoid giving out any assessments during the weeks that the survey is active, in case a poor mark might influence their assessment of us. And we also delay telling them whether their fashion collection, film or design has been chosen for the final show.

Despite all these efforts, most of us have long viewed the whole exercise with a healthy dose of scepticism. Like atheist mourners chanting prayers about the afterlife, we put in all this effort while harbouring grave doubts about the validity of the exercise.

For a start, the language of the survey is ambiguous. Students may not realise what “feedback” is. In my school, we’ve responded by renaming criticism sessions “feedback workshops”, and I encourage everyone to use the word as often as possible - even during corridor conversations. Nor are they always clear about what “academic support” might be. So now, when people stick their tutorial timetables on their doors, they label them “academic support sessions”.

We also know results will be misleading if the uptake is low, so we’re constantly devising ever more ingenious tactics to persuade the recalcitrants to fill in the survey. They’ll get discounts in the bookshop, vouchers for printing, beer and snacks on survey days and even, if they’re very lucky, a branded T-shirt or memory stick.

And we also need to be mindful that a good score on “teaching” doesn’t always equate with good teaching. Students don’t necessarily respond well to difficult concepts or challenging assignments. Indeed, Arlene J. Nicholas has proved that they don’t. Her research, reported in a recent edition of Times Higher Education, reveals the hardly surprising news that students don’t like writing long essays. They prefer short ones, and lots of bite-sized presentations.

And the results don’t deal well with contradiction, either. Some students will mark a tutor down for leaving them to their own devices, while others will praise her for giving them freedom. Even the most highly rated module can be dismissed as rubbish, often for unfathomable reasons.

It’s not that the NSS is necessarily worse than any other survey. All are open to bias and misinterpretation - and good old lying. The Health Survey for England 2010 reveals, for example, that men have almost twice as many sexual partners during their lifetimes as women (9.3 compared with 4.7). And while 13 per cent of women have more than 10 partners, there are twice as many males in this category. As Barbara Ellen pointed out in The Observer, something doesn’t quite add up. “Usually men crack on they’ve had a lot more sexual partners than is true,” she concludes, “while women own up to far fewer.”

Students may not resort to lying when responding to surveys, but they can get confused. “I’ve had no feedback at all this semester,” moaned one of ours last year. She then added helpfully, “but I haven’t handed in any work yet”. Others complain that their teachers are too critical, too demanding, and even - gasp - expect them to be familiar with the books on the reading list.

You can’t really blame students for getting grumpy. They suffer from survey fatigue. They’re already expected to fill in questionnaires rating each module, while at the University of Westminster we carry out our own student-experience survey in addition to the NSS. Last year a student claimed mitigating circumstances to justify missing his project deadline. He was, he explained, completely tied up responding to surveys.

Sometimes, though, survey respondents know enough to be able to distort their replies so as to gain real advantages. According to a criminologist friend, surveys are routinely carried out on prisoners in order to discover common traits or to identify treatment regimes that might be effective. One seasoned old lag had it down to a fine art. “You put ‘yes’ to questions 9, 16, 23, 31, 39 and 44,” he explained. “That way, you get a job in the laundry.”