Procedures and experience in verification of data on higher education institutions in Romania

1. Procedures and experience in verification of data on higher education institutions in Romania Paul SerbanAgachi Member of the Executive Committee of IREG Observatory University Babes-Bolyai, Cluj-Napoca, Romania

2. Outline • Why necessary verify HE data? • Verification at the level of the university • Verification at the level of the system/Ministry • Procedures of verification • Results

3. Why necessary verify HE data? • Since the impact of the rankings on university and national policies is increasing • Since the complexity of rankings is increasing • Since the aims of using the rankings diversified it is a problem of responsibility of the ranking agencies to give the most reliable rankings

4. U-Map and U-Multirank Classification U-Map Activity profiles of institutions Teaching andlearning Profile A Profile B ... Research involvement Knowledgeexchange International orientation Multidimenisonalrankings Regional engagement Dimen-sion 1 Dimen-sion 2 Dimen-sion 3 Dimen-sion ... Student profile Multiple excellences Multi-dimensional global university ranking

5. Criticism of existing rankings (I) (Vincke, Dubrovnik, 2009, Bucharest 2012) • Selection of criteria and their relative importance • Research • Education • Costs • Services • Social aspects • National context, legislation • Financial resources • Selection of indicators • Validation of data

6. Criticism of existing rankings (II) (Vincke, Dubrovnik, 2009, Bucharest 2012) • Bibliometrics • Quality of data • Discrimination between scientific fields • Different traditions (journals, books, proceedings, number of authors, duration of research validation) • Supremacy of publishing in English • Which indicators? (IF, citation index, h index, …) • Experts • Are there ranking experts? • How are they selected? • How are the questionnaires structured? How are the answers handled?

7. Suppose 6 universities

8. Result of Ranking C D A F H E B

9. Change of one Data

10. Result B C D A F E

11. TWO CONCLUSIONS • Robust algorithms for calculating the positions of the universities in rankings • Reliable data similar problems with the allocation of funding for the universities in Romania

12. PROPOSAL OF A RANKING AGENCY romania

13. Verification at the level of the university • Database with the Human Resource of the university (teaching, research, technical, administrative staff) – HR compartment • International staff and their status – HR and International offices • International students and their status – Registrar and International office • National Student Enrollment Registry - Registrar • Individual verification of each registration (consistency, nature, values, intervals etc.) • Students with the situation incomplete at the end of the academic year • Coherence of he output and input data from different consecutive years • Unique position in the NSER

14. Verification at the level of the university • IT conditions – IT department • Databases with researchers and doctoral students – Research and Doctoral studies compartment • Research data- Research compartment • Scientometric compartment • Internal Database of research activity • Citations, IF • Comparison with the International Data Bases (Thomson, Scopus) • Consistency with the two databases

15. Verification at the level of the university • Database with alumni – Alumni office • Databases with alumni at the level of departments/faculty • Database with employers at the level of faculties and university • Employment situation – in UBB – Center for University Development one of the major problems is that all these data are not correlated and their consistency is not verified by one unique body

16. Verification at the level of the system • Ministry (National Student Enrollment Registry) • Individual data for each student • Data concerning the university • University structure • University teaching programs • Data concerning schooling capacity figures approved • Titles and diplomas

17. Verification at the level of the system • CNFIS • Allocation of funds based on the equivalent student number • Allocation of funds based on quality: • Teaching staff and its quality • Scientific performance: publications, invention patents etc. • Extra-budgetary funds and their allocation • Quality of social services • Internal management of the allocations

18. Verification at the level of the system • CNFIS • Analysis of the students’ number reported on October 1 of the current academic year relative to the report of the same number on January 1 of the previous academic year • Consistency of the statistical data based on the correlation between their value and their significance • number of students participants to the pedagogical seminar < number of BA students

19. Verification at the level of the system • CNFIS • Analysis of the students’ number reported on October 1 of the current academic year relative to the report of the same number on January 1 of the previous academic year • Consistency of the statistical data, comparing the enrollment figures on January 1 of the previous academic year with the schooling capacity: • variations larger than +/-10%; data referring to the continuing education for teaching staff in the pre-university system – not considered because of the annual fluctuations • Variations larger than +/-3%; data referring to period 1 Oct – 1 Jan of the same academic year • Identify situations of discordance between situations from Jan 1 of the two consecutive academic years • “surplus” of the declared number of students and schooling capacity

20. Verification at the level of the system • CNFIS • Analysis of the academic staff number reported on October 1 of the current academic year • Primary analysis of the consistency • Number of academic staff should be < or = to the total number of professors of all grades (full, associate, lecturers etc.) • Number of young staff should be < or = to the total number of lecturers, assistants • Secondary analysis of the consistency • Comparison with the situation of a reference (2008) for the difference larger that +/- 10% • Academic tenure staff change; difference in the number of professors of all grades; teaching staff under 35; teaching staff with PhD degree; teaching positions number; vacancies; auxiliary staff change

21. Verification at the level of the system • CNFIS • Analysis of the number of students enrolled and schooling capacity • Comparison of the number of students enrolled at October 1 of the academic year and the allocated figure from the Ministry (OMECTS) in the year verified • Schooling capacity compared with first year students reported at October 1 of the academic year • Comparison between the number of students enrolled in the university, reported in the previous years with the schooling capacity in the same year (e.g. II-nd year students)

22. Verification at the level of the system CNCS • Analysis of the overall scientific contribution (articles, books, proceedings, patents, prizes, conferences etc.) • Analysis and comparison of the data reported by the universities with international databases: ISI Thomson, SCOPUS • Result: report for Ministry, ANCS and CNFIS

23. Verification at the level of the system • ARACIS • Analysis of the number of students enrolled and schooling capacity for each program • Analysis of the number of academic staff for each program • Analysis of the state of the properties of the universities • Analysis of the teaching/research space

24. CONCLUSION • HE data are verified at three levels • University • Ministry • Councils Bad experience with the validation of the data at the last classification New classification/ ranking

25. Thank you for your attention