BRICS & Emerging Economies Rankings 2015 methodology

A slight variation on a tried-and-trusted method

The Times Higher Education World University Rankings methodology used for the BRICS & Emerging Economies Rankings 2015 has been adjusted for less mature research systems and differing development priorities

The THE World University Rankings are the only global university performance tables to judge research-led universities across all their core missions – teaching, research, ­knowledge transfer and international outlook. The BRICS & Emerging Economies Rankings apply the same 13 carefully calibrated performance indicators as the World University Rankings to provide the most comprehensive and balanced comparisons, which are trusted by students, academics, university leaders, industry and even governments.

Our 13 performance indicators are grouped into five areas:

• Teaching (the learning environment)
• Research (volume, income and reputation)
• Citations (research influence)
• International outlook (staff, students and research)
• Industry income (innovation)

Exclusions
Universities are excluded from the Times Higher Education BRICS & Emerging Economies Rankings if they do not teach undergraduates or if their research output amounted to fewer than 500 articles between 2008 and 2012 (100 a year).

Data collection
Institutions provide and sign off their institutional data for use in the rankings. On the rare occasions when a particular data point is not provided – which affects only low-weighted indicators such as industrial income – we enter a low estimate between the average value of the indicators and the lowest value reported: the 25th percentile of the other indicators. By doing this, we avoid penalising an institution too harshly with a “zero” value for data that it overlooks or does not provide, but we do not reward it for withholding them.

Scores
To calculate the overall rankings, “Z-scores” were created for all datasets except for the results of the academic reputation survey. The calculation of Z-scores standardises the different data types on a common scale and allows fair comparisons between different types of data – essential when combining diverse information into a single ranking. Each data point is given a score based on its distance from the mean average of the entire dataset, where the scale is the standard deviation of the dataset.

The Z-score is then turned into a “cumulative probability score” to arrive at the final totals. If University X has a cumulative probability score of 98, for example, then a random institution from the same data distribution will fall below University X 98 per cent of the time. For the results of the reputation survey, the data are highly skewed in favour of a small number of institutions at the top of the rankings, so in 2011-12 we added an exponential component to increase differentiation between institutions lower down the scale, a method we have retained.

Teaching (the learning environment): 30%

• Reputation survey: 15%
  Thomson Reuters carried out its latest reputation survey in spring 2014. It examined the perceived prestige of institutions in teaching. The responses were statistically representative of global higher education’s geographical and subject mix.
• Staff-to-student ratio: 4.5%
• Doctorate-to-bachelor’s ratio: 2.25%
• Doctorates awarded to academic staff ratio: 6%
  As well as giving a sense of how committed an institution is to nurturing the next generation of academics, a high proportion of postgraduate research students also suggests the provision of teaching at the highest level that is thus attractive to graduates and effective at developing them. This indicator is normalised to take account of a university’s unique subject mix, reflecting that the volume of doctoral awards varies by discipline.
• Institutional income: 2.25%
  This measure of income is scaled against staff numbers and normalised for purchasing-power parity. It indicates an institution’s general status and gives a broad sense of the infrastructure and facilities available to students and staff.

Research (volume, income and reputation): 30%

• Reputation survey: 18%
  The most prominent indicator in this category looks at a university’s reputation for research excellence among its peers, based on the responses to our annual academic reputation survey.
• Research income: 6%
  Research income is scaled against staff numbers and normalised for purchasing-power parity. This is a controversial indicator because it can be influenced by national policy and economic circumstances. But income is crucial to the development of world-class research, and because much of it is subject to competition and judged by peer review, our experts suggested that it was a valid measure. This indicator is fully normalised to take account of each university’s distinct subject profile, reflecting the fact that research grants in science subjects are often bigger than those awarded for the highest-quality social science, arts and humanities research.
• Research productivity: 6%
  We count the number of papers published in the academic journals indexed by Thomson Reuters per academic, scaled for a university’s total size and also normalised for subject. This gives an idea of an institution’s ability to get papers published in quality peer-reviewed journals.

Citations (research influence): 20%

Our research influence indicator is the flagship, the single most influential of the 13 indicators. It looks at the role of universities in spreading new knowledge and ideas.

We examine research influence by capturing the number of times a university’s published work is cited by scholars globally. This year, our data supplier Thomson Reuters examined more than 50 million citations to 6 million journal articles, published over five years. The data are drawn from the 12,000 academic journals indexed by Thomson Reuters’ Web of Science database and include all indexed journals published between 2008 and 2012. Citations to these papers made in the six years from 2008 to 2013 are also collected.

The citations help to show us how much each university is contributing to the sum of human knowledge: they tell us whose research has stood out, has been picked up and built on by other scholars and, most importantly, has been shared around the global scholarly community to expand the boundaries of our collective understanding, irrespective of discipline. The data are fully normalised to reflect variations in citation volume between different subject areas. This means that institutions with high levels of research activity in subjects with traditionally high citation counts do not gain an unfair advantage.

We exclude from the rankings any institution that publishes fewer than 100 papers a year to ensure that we have enough data to make statistically valid comparisons.

International outlook (staff, students and research): 10%

• International-to-domestic-student ratio: 3.3%
• International-to-domestic-staff ratio: 3.3%
  The ability of a university to attract undergraduates, postgraduates and faculty from all over the planet is key to its success on the world stage.
  
• Research: 3.3%
  In the third international indicator, we calculate the proportion of a university’s total research journal publications that have at least one international co-author and reward higher volumes. This indicator is normalised to account for a university’s subject mix and uses the same five-year window as the “Citations: research influence” category.
  

Industry income (innovation): 10%

A university’s ability to help industry with innovations, inventions and consultancy has become a core mission of the contemporary global academy. This category seeks to capture such knowledge transfer activity by looking at how much research income an institution earns from industry, scaled against the number of academic staff it employs.

The category suggests the extent to which businesses are willing to pay for research and a university’s ability to attract funding in the commercial marketplace – useful indicators of institutional quality.