‘Tide turning’ against using student evaluations to rate staff

Experts have predicted a global “sea change” away from the use of student evaluations to measure lecturers’ suitability for promotion or tenure, after a union successfully challenged the practice.

An arbitration case between Ryerson University in Toronto and its faculty association that had stretched on for 15 years finally concluded with a ruling that course surveys can no longer be “used to measure teaching effectiveness for promotion or tenure”.

Arbitrator William Kaplan said that “insofar as assessing teaching effectiveness is concerned – especially in the context of tenure and promotion – SETs [student evaluations of teaching] are imperfect at best and downright biased and unreliable at worst”.

The Ontario Confederation of University Faculty Associations argued that the outcome could pave the way for other Canadian universities to negotiate similar changes. The case had “established an important precedent for faculty associations and lends support to others who have been arguing that student questionnaires are deeply problematic instruments for the purpose of evaluating faculty members’ teaching effectiveness”, the organisation said.

But Philip Stark, associate dean of the Division of Mathematical and Physical Sciences at the University of California, Berkeley, who was an expert witness in the Ryerson case, said that the impact could be much broader.

The result, alongside recent voluntary changes to the use of teaching evaluations at US universities, including the University of Southern California, shows that the “tide is starting to turn”, he said.

“Seeing this happen at the level of a union dispute in Ontario is terrific. I think this is the start of something much bigger. I really hope that there is going to be a sea change in how people administer and rely on student evaluations and the items that they use,” Professor Stark said.

Professor Stark added that he hoped that other unions representing academics in Canada, the US and elsewhere would “negotiate to reduce or eliminate reliance on student evaluations” and that universities of their own accord would “move towards more sensible means of evaluating teaching”.

“I think that the time is right for class-action lawsuits on behalf of women and under-represented minorities against universities that continue to rely on student evaluations as primary input for employment decisions [and that this] will induce universities to do the right thing,” he said.

Professor Stark said that studies have shown that “there is little if any connection between students’ ratings of instructor effectiveness and actual instructor effectiveness, as measured by student performance on anonymously created final exams”, and evidence suggests that any association is “generally negative not positive”.

There is also evidence that students’ ratings are strongly influenced by academics’ personal characteristics, such as gender, race, age and physical attractiveness, he said.

Bob Uttl, professor of psychology at Mount Royal University in Alberta, who has published research showing that student evaluations of teaching and student learning are unrelated, said that the ruling provides a “strong reminder” that “one ought not to make high-stakes personnel decisions based on unreliable and/or invalid assessments”.

He added that the ruling is “likely to have substantial impact and encourage faculty associations to pursue discontinuation of SETs as [a] measure of teaching effectiveness in high-stakes personnel decisions including promotion, tenure, merit pay and teaching awards decisions”.

ellie.bothwell@timeshighereducation.com