Quality assurance practices and University evaluation systems

1. Qualityassurancepractices and Universityevaluationsystems Stefano Fantoni President, ANVUR Euro-MediterraneanUniversity Network TETHIS meeting University of Palermo 10°-11° of April 2014

2. The Italian National Agency for the Evaluation of Universities and Research Institutes • ANVUR is the Italian National Agency for the Evaluation of Universities and Research Institutes. It started operating in May 2011. • ANVUR is in charge of • Evaluating the quality of processes, results, and products of all the activities concerning research, teaching and technological transfer • Defining methods and criteria to be used for the periodical accreditation and evaluation of Universities (undergraduate, postgraduate and Ph.D. courses) • Providing general advices to internal evaluation bodies • Evaluating public funding programs aimed at research and innovation activities • Every two years, ANVUR will edit a general Report on the italian Higher Education status

3. The Italian National Agency for the Evaluation of Universities and ResearchInstitutes • ANVUR, in its evaluation process, refers to the Quality Assurance system as described in European Standard Guidelines ESG/ENQA 2005/2009. • ANVUR has performed the research evaluation exercise for the period 2004-2010 (VQR 2004 - 2010, Valutazione della Qualità della Ricerca), whose results have been presented in July 2013 • ANVUR is currently performing the accreditation and assessment program for courses and institution (AVA, Accreditamento, Valutazione e Autovalutazione) • ANVUR is currently developing the accreditation and the evaluation models for the PhD courses.

4. The contest of the evaluation EvaluationofResearch EvaluationofUniversities Courses evaluation Management evaluation public funding Incomingresearchersevaluation Placement & transitionto work

5. The Evaluation of Research VQR 2004-2010 Each university professor or researcher has been asked to submit three research products (six instead of three to the researchers of the Research Institutes) published in the 2004-2010 period. Each research product (for a total of about 180 thousands) has been evaluated by bibliometric or/and peer review to evaluate NOT THE INDIVIDUALS but RESEARCH AREAS, DEPARTMENTS, UNIVERSITIES International character of the italian production: 62,1% of productsisin English; mainly in scientificand technicalareas(maximum value 88,6% in informatics and communicationengineering). Humanities, law and in general social areashavethe majority of workswritten in italian ( 5,7% of notitalian in Law; 13.2% for Humanities and historicalsciences).

6. The Evaluation of Research2004–2010 (VQR 2004-2010) • Products are classified in 14 scientific areas, as established by the National University Council (CUN). • For eight areas (mathematics, physics, chemistry, earth Sciences, biology, medicine, agricultural and Veterinarian Sciences, part of Architecture and Civic Engineering, Industrial and communication Engineering) the research quality has been assessed mainly on the basis of bibliometric indicators ( impact factor and number of citations of the article in a mixed way). • For six areas (Humanities and Social Sciences) the products have been assessed through peer review. • Bibliometric indicators were used also in part of Architecture and Civic Engineering, Psychology Sciences, Economics and Statistics. • Some areas were “importing” or “exporting” products, giving an image of the processes of crossed fertilization or of boundaries processes in Italian research

7. Some Numbers • 95 Universities, 67 public and 28 private, 11 of which are at distance institutions • 12 Research Institutes under the control of the Ministry of Education and University • 26 Research Institutes which have voluntary submitted an application to be evaluated by ANVUR • 450 national and international Scientific experts from all the 14 Areas of Research • 14.770 reviewers, 10.150 Italian and 4.620 working in non Italian institutions • 18 months the full time-length of the evaluation • 184,878 research products have been submitted for evaluation; 5.1% of “missing products” from -not active, or partially active- researchers. • The typology of products varies in different scientific areas: book and book chapters (average value 19.9%) are more common in humanities and social sciences, articles (average value 73.5%) in scientific and technical areas.

8. Expected vs. submittedproducts

9. Submittedproductsbyarea of the product

10. The Evaluation Results • A research product has been classified as follows: • Excellent, when it falls in the top 20% in terms of quality according to the international scientific community in the field • Good, when it falls in the 60-80% • Adequate, when it falls in the top 50-60% • Limited, when it falls in the bottom 50% • Penalized (not pertinent or copied): 2076 products (1,1%)

11. NOTE on the Evaluation Results • Comparisons only within the same area, NOT between different areas • The products evaluation results have been used to elaborate the final assessment of the structures. Its weight was 50% • The others indicators for the final assessment were • ability to attract funds in a number of competitive calls (10%) • mobility index, calculating the results of the new coming researches in the examined period (10%) • Internationalization index (incoming and outcoming researchers, products with non Italian co-authors), 10% • High qualification index (ph.d. students, post graduation grants…), 10% • Funds for research coming from the institution, 5% • Improvement index, as a difference from the former evaluation process (5%)

12. AVA and the Qualityassessment Courses & institutionsassessment Accreditation: criteriaforQualityAssessment Outcomesassessment International comparison “on-site” visits Qualityassessmentgroup

13. SUA: the AnnualSummarizing Chart • Each dean or responsible of the course of study is required to fill • the names of three professors and their teaching subjects yearly in charge of the course • the aims and methods of the course • the professional profile for the graduates • The labor market needs • the curriculum (characterizing compulsory or optional subjects, laboratories, stages, international programs…). • SUA chart provides information for the students enrolling the following year

14. The annualreview report • Each University has a committee for the audit of the Quality Assessment (Nucleo di Valutazione). In November, the Nucleo • Analize • the SUAs of the former year • the results of the questionnaires filled by students and graduates • evaluates Quality Assurance system • writes a rewiev report • Suggestes the changes to improve the quality

15. The ExternalEvaluation • ANVUR will also perform the evaluation of the system as a whole • Universities have an external evaluation every five years • Courses of study have an external evaluation every three years • The external assessment is performed by a group of experts appointed by ANVUR (CEV, Experts of Evaluation Commission). • The CEVs work starting from the review of the Nucleo and from other documents produced by the course/university • Then they do an on site visit of three to five days, depending from the dimension of the university. Their report is submitted to the University governance for comments . • The final report gives an A, B or C classification.

16. AVA process • Teaching • service • Self-assessment • Studentquestionnaire • Project toimprovequality • Review report • Finalassessment report • On site visit • Provisional report discussion • Stakeholders and useers • CEV commissionofexperts

17. THANK YOU FOR YOUR ATTENTION!