Online Learning Rankings 2024: methodology

Browse the results of the Online Learning Rankings 2024

Online learning has been a growing trend in higher education for many years. The Covid-19 pandemic drove the demand for online learning exponentially. However, there is no global standard on how to measure the teaching quality of higher education delivered online. The Times Higher Education Online Learning Rankings (OLR) 2024 is the first attempt to measure online learning on a global basis.

Definition and scope

The OLR is a teaching ranking similar to other THE teaching rankings, such as the Japan University Rankings or the Sub-Saharan Africa University Rankings. However, there are two main differences:

• OLR is a global ranking;
• OLR does not evaluate the whole university. Instead, it evaluates the part of a university that delivers online leaning.

The parts of universities that deliver online learning are defined as university programmes that:

• are advertised as “online”; and
• have as least 40 per cent of the course content delivered online.

The focus of the OLR is the teaching excellence of universities’ online programmes, rather than their technical capability for online delivery.

It is important to note that while the scope of the evaluation includes both undergraduate and postgraduate programmes, online MBA programmes are excluded.

Unlike other THE rankings that use rank position and scores, the OLR 2024 groups institutions into three categories: gold, silver and bronze. Institutions are listed alphabetically within each category. This approach has been taken to ensure fair representation for all institutions in this inaugural, pilot year of the ranking.

Methodology overview

Similar to other THE teaching rankings, the OLR comprises four pillars, namely:

• Resources – the level of resources devoted to online learning;
• Engagement – the level of student engagement in online learning;
• Outcomes – the outcomes for students participating in online learning;
• Environment – the environment in which the online learning takes place.

These four pillars are further broken down into a total of 17 metrics. The weight of each metric is shown in the diagram below.

Metric definitions

Resources: 35%

• Finance per student: 12%
• Faculty per student: 12%
• Development hours per staff: 11%.

The first metric is defined as the amount of institution income dedicated to online academic activities per online student in all years and of all programmes that lead to a degree, certificate, university credit or other qualification. The second metric is the total full-time equivalent (FTE) number of staff employed in an online academic post divided by the FTE number of online students in all years and of all programmes that lead to a degree certificate, university credit or other qualification.

The third metric is the number of hours of professional development dedicated to online teaching staff online faculty (FTE).

All three metrics are subject-weighted.

Engagement: 25%

• Interaction with staff: 4%
• Collaboration with other students: 4%
• Convenience: 4%
• Ease of use: 4%
• Accessibility for disabled: 4%
• Number of available programmes: 5%.

The first five metrics in this pillar are based on responses to a student survey (see below). Students were asked to rate from 0 to 10 the extent to which they can interact with faculty members at their institution; the extent to which their institutions provide collaborative learning; the level of convenience and ease of use of their online learning platforms; and the accessibility of the online learning platforms to learners with disabilities.

The final metric is defined as the number of online programmes offered at an institution this year, adjusted by the number of students (FTE). This metric is subject-weighted.

Outcomes: 15%

• Student progression rate: 7%
• Student recommendation: 8%.

The first metric in this pillar is defined as the proportion of online students progressing to the next year, adjusted by the number of students (FTE). This metric is subject-weighted.

The second metric is based on a question in the student survey that asked respondents to rate from 0 to 10 their likelihood to recommend their institutions to friends or family members.

Environment: 25%

• Student inclusion: disability: 3%
• Age diversity: 3%
• Staff gender diversity: 3%
• Support staff per student: 6%
• Connectivity support: 5%
• Other offline resources: 5%.

The first four metrics in this pillar measure the proportion of online students with disabilities; the mix of online students in different age brackets; the mix of genders among online faculty members; and the number of non-teaching staff that support the delivery of online content for the institution per student.

The connectivity support metric seeks to explore whether the university is providing additional, dedicated support to ensure students can connect to the internet and access resources.

Although we are exploring online learning, it is often an advantage if access to more traditional, offline resources are also available to online students on an equal footing to in-person students. Universities are therefore asked whether online students have access to a library, faculty, study groups or other resources.

For these final two metrics, universities are asked to provide evidence. We give credit for the evidence and for the evidence being specific and publicly accessible.

Data sources

Student survey

Institutions that signed up for the OLR were asked to distribute an online survey to the students who were currently enrolled in their online programmes. The students were asked a range of questions about their learning experiences, as well as basic demographic questions. Responses from students were collected directly by THE.

Responses that did not meet the following criteria were discarded. The respondent must:

• be over the age of 18;
• be currently enrolled as a student in one of the institutions that signed up for the ranking;
• not be enrolled in an online MBA programme with the institution;
• have taken more than 90 seconds to complete the survey;
• not give artificially high answers (9s and 10s) to all questions.

Survey responses tend to have a strong geographic bias. Respondents in some countries are much more likely to give high scores than those in other countries. A Bayesian technique is used to normalise the responses and make them comparable across countries.

Institutional data

Institutions that signed up for the OLR were requested to provide their institutional data to THE via a web portal between April and June 2024. This data was submitted, approved and authorised by representatives of the institution and was validated by THE. Universities were asked to submit data from either the financial year, calendar year or academic year that ended in 2022.

The data collected can be classified into two groups: quantitative data and evidence data. For details on the data collected from the portal, please refer to the OLR Portal Guide.

Supplemental data

Institutional financial data is translated into US dollars at purchasing power parity (PPP) rates so that it can be measured and compared on a global basis. PPP rates are sourced from the World Bank.

In addition, for any institution that submitted their financial data in a currency that is different from their domestic currency, their data is first translated back to the domestic currency before the PPP rates are applied. This translation is done using the annual average exchange rates from the UK’s HM Revenue and Customs.

Data from the year 2022 were used in the ranking calculation.

Eligibility criteria

In order for an institution to be included in the OLR, it must:

• Sign up for the OLR;
• Distribute the student survey to its online students and have at least 50 valid responses;
• Submit OLR data to THE via an authorised data provider;
• Have at least 40 per cent of the course content delivered online;
• Not have more than two missing metrics;
• Not be in THE’s custom exclusion list – THE reserves the right to exclude any institution from its rankings even if it satisfies all other requirements.

Institutions that met the criteria, except for receiving at least 50 valid survey responses and/or missing more than two metrics, have been assigned “reporter” status.

We welcome your feedback: if you would like to share your insights and ideas on measuring excellence in online learning, or if you are interested in participating in future years, email onlinelearningrankings@timeshighereducation.com.