Revamped US research PhD rankings cause bewilderment

The National Research Council’s Database Assessment of Research Doctoral Programmes – the first to be published since 1995 – includes data on more than 5,000 programmes at 212 universities across the US.

It was scheduled to be published in 2007, and its late arrival means that the data used are now almost five years old.

Because of the intervening turnover of faculty and changes in funding, some are questioning how relevant the findings are, especially given the impact of the recession, which the data pre-date.

An extra question mark has been raised over the new methodology, which gives doctoral programmes a ranking within a range of positions rather than a specific numerical value.

For example, the English language and literature programme at Arizona State University could be placed anywhere between fifth and 29th position out of 122.

More broadly, at the University of Michigan, the percentage of PhD programmes in the top 25 per cent nationally could be as high as 95 per cent but could also be as low 20 per cent, according to the assessment.

The 2010 exercise also sees programmes being put in a range in two categories, known as the “S-rankings” and the “R-rankings”.

The S-rankings are based on a series of statistical analyses of characteristics of programme quality. The council surveyed academics to determine which characteristics were important to them and then used a statistical technique called “random halves” to determine the ranking range. This involved half of the responses being chosen at random and statistically compared to the characteristics of individual programmes to come up with a ranking. This process was then repeated 500 times, and the scores at the top and bottom of the spectrum were used to create a final ranking range.

The second group of rankings, the R-rankings, were based indirectly on a reputation survey undertaken by staff. Using the same random halves method, survey responses were compared to characteristics of programmes in a certain discipline.

If, for instance, faculty rated programmes in a certain field at Harvard University highly, other universities’ programmes that had characteristics similar to Harvard’s were also rated highly.

The confusing approach, which has left universities unable to compare their positions in these rankings to those of the 1995 assessment, has proved controversial.

The study committee responsible for compiling the rankings has not endorsed it, and at the press briefing some committee members even went as far as to suggest that there may be better ways to rank than this exercise, which cost the sector $4 million (£2.5 million).

sarah.cunnane@tsleducation.com