Reputation voting behaviour in the Arab University Rankings

As part of our Arab University Rankings, we conduct an annual survey of academics in the region to gather their perspectives on the leading universities in the Arab world. We invite academics from our database, who have published research in the past five years and are based in the Arab region, to participate. They are asked to vote for up to 10 universities that excel in research and another 10 that are strong in teaching. This is strictly by invitation only, and universities are not invited to submit lists of names of potential respondents.

In 2024, 3,225 academics responded to the survey, casting a total of 36,043 votes. Upon processing the results, we observed significant increases in votes for certain universities compared with 2023, prompting further analysis.

Three data scientists independently conducted a deep analysis using different techniques to understand the cause of these vote increases. All three reached the same conclusion: 21 universities across seven countries were engaging in mutual voting, casting very few votes for institutions outside their group. Moreover, these universities received minimal votes from academics at other institutions. The statistical signal was clear and strong, indicating a voting syndicate.

After careful consideration of various actions, we decided the best course was to remove all votes exchanged between the 21 universities, effectively neutralising the impact of the syndicate. This also serves as a minor penalty; with more natural voting patterns, these universities would likely have earned some useable votes from each other. Although a review of the 2023 survey provided a much weaker statistical signal of a smaller syndicate, we have decided to use only the 2024 survey data for the upcoming edition of the Arab University Rankings, rather than combining two years’ worth of data as we have done in the past.

We considered alternative actions, such as eliminating all votes received by these universities or removing the institutions from the ranking entirely. However, we chose not to pursue these options this time, as university leaders may have been unaware of the irregular voting behaviour. If similar behaviour is observed in the future, more severe action will be taken.

We have taken action in the past when issues arose. For instance, to address increasing self-voting in academic surveys, we introduced a cap on self-votes in the Arab University Rankings, and also in the World University Rankings. We believe some self-voting is acceptable, as academics can reasonably consider their own institution among the best. Additionally, we have examined the distribution of votes received by universities and introduced a limit on the ratio of votes to distinct universities (votes per respondent institution, or VPRI), which helped curb some unusual voting patterns.

Now, to the statistics. Within the 21-university syndicate, over 81 per cent of their votes went to another member of the group, whereas only 6 per cent of votes from universities outside the syndicate were directed toward syndicate members. The following table illustrates the vote distribution between the two groups (the percentages in brackets are the percentages of votes cast):

The level of statistical significance is immense, with a p-value of 0.0000 and a z-score of 190.1.

We are also expanding our analysis of this syndicate beyond the Arab survey votes:

• We are working with our partners at Elsevier to determine whether this group is also engaging in mutual citation of academic papers. If patterns emerge in recent research papers, further action may be taken, including potentially suspending journals involved.
• As we begin collecting data from universities for next year’s rankings, we will be scrutinising submissions from the universities in the syndicate. Where evidence of manipulation exists, we will adjust or exclude their data from the rankings.
• We have already analysed voting patterns in our global academic survey and found some weak signals related to this syndicate. These have been mitigated by the implementation of our VPRI cap, and as such have had no material effect on the reputation metrics or overall rankings in the World University Rankings.

We will continue to research and develop new tools to detect manipulative behaviour as soon as it arises. The integrity and transparency of our rankings is of paramount importance, and we reserve the right to take any necessary action to protect them, including the ultimate sanction of removing universities from all our rankings.

If you are aware of or suspect malpractice at your university, I encourage you to contact me directly in strict confidence.

David Watkins is managing director of data at Times Higher Education. He can be contacted at david.watkins@timeshighereducation.com