University rankings, dissected

1. University rankings, dissected Răzvan V. Florian Ad Astra association of Romanian scientists Center for Cognitive and Neural Studies, Cluj, Romania

2. How are indicators used in global university rankings chosen? • Best indicator for efficiency of education (and economic efficiency): the salaries of graduates in the first years after graduation (normalized to national averages); obtaining such data is difficult and costly! • Many global university rankings use bibliometric indicators; this is because research performance is relevant to universities’ mission, but also because such data is relatively easy to obtain from existing databases.

3. Indicators are evolving • The methodology of global university rankings is constantly evolving (THE-QS methodology constantly changed between 2004-2007); • New rankings appear; they may dominate in the future “traditional” rankings because of better methodology; • Better indicators appear (the eigenfactor as a replacement of the impact factor).

4. Web 2.0 as a source of information for rankings • Lots of people are online (e.g., graduates on Facebook, employers on job sites, Linkedin), so it will be easier to perform surveys; • From the impact factor, back to peer review: it will be possible to evaluate individual papers by aggregating the feedback of all scientists that read these papers; indexing by databases will not matter anymore (also the problem of evaluating humanities scientific literature, conferences, books, etc. will be solved); • Network theory will improve scientometry (Google & the eigenfactor);

5. What should universities do? • Do not try to game current indicators; if effects are obtained in ~5 years, the methodologies of rankings will change significantly by then; • Just do well your job: • Have graduates that employers will hire on good salaries; • Produce internationally relevant, excellent research; research is likely to still dominate education in rankings in the near future (as data will remain easier to obtain);

6. An example: the methodology of the Ad Astra ranking • 2005-2007: the number of ISI indexed papers per faculty was a good proxy for research performance (also inspired by the Shanghai ranking); • 2009: 57 Romanian ISI-indexed journals, with no or very low impact factors (except one); ISI-indexed Romanian papers are dominated by papers in Romanian journals, mostly of low quality; hence, the simple indexing by ISI is not anymore a good proxy for research performance; • The future Ad Astra rankings will likely change methodology, by counting the domain-normalized impact factor and not just counting papers; a threshold for the domain-normalized impact factor will likely be imposed; CNCSIS also already changed the criteria for the awards they give for ISI papers; • Conclusion: universities that invest in internationally-relevant research win, universities that focus on just ISI-indexing their journals loose;

7. What should Romanian universities do to climb the rankings? • Allocate resources exclusively on internationally-recognized performance (not seniority, rank / power in hierarchy, etc.); • Support excellence, considering the exponentially-increasing efforts to reach higher levels; • Specialize (emergence of subject-oriented rankings); • Restructure personnel (diaspora as a source of qualified scientists); the proportion of faculty that obtained a degree from a top 500 university as an indicator of potential; • Reduce bureaucracy (informatisation, use of electronic signature, etc.); • The government should also do its part (funding, bureaucracy, evaluation criteria, etc.)

8. Questions? • See also: http://coneural.org/florian/papers/2009_recomandari_rankinguri.pdf(in Romanian) • My email: florian@ad-astra.ro