Questionable results

Preliminary results of the pilot student satisfaction survey ("Students rate staff highly", THES , January 30) should be treated with caution.

Constructing a league table that allows students to choose between institutions would be problematic. A detailed analysis of the biggest courses shows that how students rate them has little effect on an institution's rating. Most are not statistically significantly different from the average. Courses with fewer students show even less differentiation between institutions. This is because the student numbers involved are a small part of the total.

Neither do we know how much of the variation between institutions really represents year-on-year variation within institutions. One or even two years' data may be a poor guide. Students' prior qualifications and experiences might also affect ratings. The response rate of little more than 40 per cent, rather short of the 60 per cent target, would need to improve to conclude that the response rate was representative.

Harvey Goldstein
Bedford Group
Institute of Education