Measuring teaching in a time of Covid

There has been remarkable inconsistency in universities’ approach to teaching in 2020.

If we were to try to understand the impact of online teaching on students, the gold standard would be some form of double-blind trial, where neither the participants nor the experimenters know who is receiving a particular treatment.

Of course, that isn’t viable, but there is usually a very good stand-in: the natural experiment. In a natural experiment, you look for a dividing line where groups with otherwise similar characteristics are split in some arbitrary way.

For example, if you wanted to explore whether voting in an election increases the chances of voting in subsequent elections, you could explore the behaviour of people who were 18 just before and just after an election, but otherwise identical.

Unfortunately, the varying conditions of the pandemic across the world mean that natural experiments are hard to do. There is no doubt that countries such as Japan have weathered the storm in a much stronger way than much of Europe, and few countries have fared as badly as the US. As a result, the experience of students has also varied hugely. This makes measurement of any type very difficult.

Another big challenge is that there isn’t a consistent opinion of how great remote teaching should look (although for some excellent insights it’s worth checking out our new THE Campus initiative). If we don’t know what people should be doing, how can we measure it?

One option is to look for signals from students themselves. In our teaching rankings for the the US and Japan, we ask students about their experience at their college or university.  Specifically, we ask them:

• The extent to which they are taught critical thinking
• If they have interaction with faculty
• The opportunities they have to collaborate with their peers
• If they would recommend their college or university

This year’s US student survey, which would have been carried out in the spring, had to be cancelled because of the pandemic and subsequent difficulties for institutions in the US. But this year’s Japan student survey is in the field now, and with Japanese universities moving from a semester of online learning to one with some physical classes, we’re starting to see some interesting results.

These are only preliminary insights, so the trends may change, but the initial data show that where students are in institutions with above median amounts of online learning taking place, the average score for interaction with faculty is significantly down when compared with last year. Although teachers are working hard to make up for the lack of traditional teaching approaches, it seems that it is difficult to replicate the in-person effect.

The score for collaboration among peers is also down significantly at institutions with high levels of online learning. Again, this isn’t surprising, but it is perhaps an area where universities could be directing more effort.

Another more unexpected result is that perception of support for critical thinking has risen among all groups. Perhaps this is related to the change in course material and delivery that has taken place since 2019, although the rise is especially strong where there is less online learning.

What does all this mean in terms of the likelihood of students recommending their university? The positive news is that this score has also risen, on average, and it seems to have risen equally regardless of the type of learning experienced. This suggests that in Japan, at least, universities have lived up to the expectations of students in this very challenging year.

Of course, Japan is just one country and both the nation and its universities are working in different contexts to the rest of the world. But nonetheless the results reveal some interesting signs that might provide hope for improved learning across many sectors in the future.

Do you have ideas about how we can improve our rankings? Send suggestions and questions to us at profilerankings@timeshighereducation.com.

Duncan Ross is chief data officer at Times Higher Education.