U-Multirank On the feasibility of a new approach

1. U-Multirank On the feasibility of a new approach Frans Kaiser, CHEPS Mini-seminar organised by the Transparency Tools Working Group for the BFUG Cracow 12 October 2011

2. Rankings in highereducation • instruments to compare higher education institutions • and to judge their relative position, usually in league tables • based on the actual performance of these institutions

3. The rise of global rankings • Academic Ranking of World Class Universities (ARWU) Shanghai Jiaotong University, since 2003 • Times Higher Education Supplement World Rankings (THE) Times Higher Education, since 2004 • Higher Education Evaluation and Accreditation Council of Taiwan Ranking (HEEACT), since 2007 • The Leiden Ranking (LR) Leiden University, since 2008

4. Critique of existing rankings • focus on ‘whole institutions’ (ignoring internal variance) • concentrate on ‘traditional’ research productivity and impact • focus on ‘comprehensive research universities’ • aggregate performance into composite overall indicators • use constructed ‘league tables’ • imply cultural and language biases • imply bias against humanities and social sciences

5. Designing an alternative: the EC call for Tender • development of concept and feasibility study of a global ranking (not only European) • multi-dimensional, not only research • multi-level (institutional and ‘field’) • all types of higher education and research institutions • multiple stakeholders

6. Newapproach to ranking Subset of institutional activity profiles

7. Newapproach to ranking Subset of institutional activity profiles

8. Institutional performance profiles multidimensional perspective of ‘institutional profiles’ no overall ‘league tables’ no composite institutional indicators two-level analysis (institutional and ‘field’) stakeholders driven approach & specific ‘authoritative rankings’ allowing ‘performance profiles’ in various dimensions base for benchmarking, for inter-institutional cooperation, for effective communication and profiling base for informed governmental differentiation policies at system level

9. Dimensionsand indicators

10. Pilot • Two levels: • Institution (FIR) • Field (FBR) • Global sample of higher education and research institutions: 159 (target: 150), 2/3 Europe, 109 completed institutional questionnaires • Two fields: • Business studies • Engineering (electrical and mechanical)

11. Pilot • Self-reported institutional data by means of four online questionnaires: • U-Map (Institutional) • U-Multirank (Institutional level) • U-Multirank (department level) • Student survey • International databases • Bibliography • Patent data

12. Pilot: conclusion Thereareissuestoberesolved (like regional engagement, employability) but itmaywork Whatcoulditlooklike?

13. Choose the type of ranking you are interested in. Ranking of Business studies programs Ranking of institutions Ranking of Electrical engineering programs

14. Selection of matching institutions (1) Doctorate production Other research products Academic publications Distance education Total enrolment % Expenditure on research Cultural activities Start-up firms Orientation of degrees % Expenditure on teaching Income knowledge transfer Subjects covered Mature students Teaching & learning Student profile Research involvement Knowledge transfer Degree level focus Part time students Patent applications Professional publications

15. Selection of matching institutions (2) Exchange stud; incoming Non-national academic staff Foreign degree seeking stud Graduates in the region Income from regional sources Income from internat sources International orientation Regional engagement Exchange stud; sent out New entrants from region Continue to ranking

16. Performance profiles (institutional level) Student internships in region Graduationratebac % incomefrom the region Graduationrate master Time todegreebac Research contracts withregionalenterprise Regional joint research publications Time todegree master % exp on teaching Graduatesworking in the region Graduateunemployment % interdisciplinary programs Highlycited research publications Patents awarded Field normalisedcitationrate Post docs per academicstaff Start up firms Art related research output Size of TTO % of research incomefromcompetitive sources Co-patenting % incomethird party funding Createpersonalised ranking table % expenditure on research Incentives forknowledge transfer Interdisciplinary research Univ-industry joint publ. CPD courses offered International doctorategraduationrate. International joint research publications Research publication output. % programs in foreignlanguage. % internationalstaff % students in joint degree programs 293 148 152 4 222 111 98 196

17. Personalise your ranking Teaching & learning Research Knowledge transfer International orientation Regional engagement Show personalisedtable

18. Personalised institutional ranking sort 293 148 152 4 222 111 98 196

19. Personalised institutional ranking Sorted by indicator #1 sort 293 148 152 4 222 111 98 196

20. Personalised institutional ranking Sorted by indicator #3 sort 293 148 152 4 222 111 98 196

21. Background information Name of institution: 293 Address URL U-Multirank profile Mission statement U-Map profile

22. Choose the type of ranking you are interested in. Ranking of Business studies programs Ranking of institutions Ranking of Electrical engineering programs

23. Selection of matching institutions (1) Doctorate production Other research products Academic publications Distance education Total enrolment % Expenditure on research Cultural activities Start-up firms Orientation of degrees % Expenditure on teaching Income knowledge transfer Subjects covered Mature students Teaching & learning Student profile Research involvement Knowledge transfer Degree level focus Part time students Patent applications Professional publications

24. Selection of matching institutions (2) Exchange stud; incoming Non-national academic staff Foreign degree seeking stud Graduates in the region Income from regional sources Income from internat sources International orientation Regional engagement Exchange stud; sent out New entrants from region Continue to ranking

25. Default business studies ranking Default field based ranking 293 148 152 4 222 111 98 196

26. Default business studies ranking Default field based ranking Sorted by indicator #1 293 148 152 4 222 111 98 196

27. Default business studies ranking Default field based ranking Sorted by indicator #12 personalise 293 148 152 4 222 111 98 196

28. Personalise your ranking Teaching & learning Teaching & learning; student satisfaction Research International orientation Knowledge transfer Regional engagement Show personalisedtable

29. 4 Personalised ranking business studies 98 115 138 139 144

30. Personalised ranking business studies Sorted by indicator #1 251 115 292 546 4

31. Personalised ranking business studies Sorted by indicator #3 613 4 546 144

32. Discussion No ‘U-Multirank league table’: lesspresscoverage, betteruse? Authorativerankings: rankings in a user-defined setting? U-Multirank; a new approach? ** Multi-dimensional * Multi-level *** Inclusive ** Comparability check (U-Map) **** User-driveness

33. Thankyouforyour attention www.u-multirank.eu