J.A. Rafael and M. Assunção , 10-11 July 2008

1. Experience of self evaluation as part of a long-term change process : the UA as a case study from Portugal Conference on self-evaluation of HE institutions, Belgrade, Serbia J.A. Rafael and M. Assunção , 10-11 July 2008

2. Contents • About the University of Aveiro • profile • organisation • QA internal implementation • Setting the scene at European and National level • Legal changes • Opportunity to re-think QA in UA

3. Portuguese Public Universities

4. Ol. DE AZEMÉIS AVEIRO AGUEDA Ol. DE AZEMÉIS AVEIRO AGUEDA The University of Aveiro as a Regional Network for Education and Training • The U. Aveiro incorporates University Departments and Polytechnic Schools (since 1997). Four years ago a programme aimed at post-secondary education was started and is now run by the Polytechnic Schools. • 17 University departments (Aveiro) • 35 graduation degrees • 4 Polytechnic Schools (3 cities) • 15 graduation degrees • 16 CETs (11 cities)

5. University Departments • engineering • › Electronics and Telecommunications • › Environment and Planning • › Ceramics and Glass Engineering • › Mechanical Engineering • › Civil Engineering • education • › Sciences of Education • › Didactics and Educational Technology • science • › Chemistry • › Biology • › Physics • › Mathematics • › Earth Sciences • arts and social sciences • › Languages and Cultures • › Media and Artistic Studies • › Social and Political Sciences • › Economics, Management and Industrial Engineering health Sciences

6. RegisteredStudents source: GAGI, December 2007

7. Research Centres 13 Research Units 4 Associated Labs source: Research Institute, May 2007

8. Rector Senate Scientific Council Pedagogic Council Research Institute Undergraduate University Education Institute Postgraduate University Education Institute Polytechnic Education Institute … Polytechnic School 1 Polytechnic School 4 Departments (17) Functional Organisation & Management

9. … … Study Programme 1 Study Programme 2 Study Programme n [Degree Director] [Degree Director] [Degree Director] Dept. 1 ● ● ● Dept. 2 ● … … Dept. m ● ● Resources Study Programme Matrix Organisation

10. … … Research Unit1 Research Unit 2 Research Unit n [Unit Coordinator] [Unit Coordinator] [Unit Coordinator] Dept. 1 ● Dept. 2 ● ● ● … … Dept. m ● ● Resources Research Units Matrix Organisation

11. Promotion of a Quality Culture: the beginning (1) • 1992, Portuguese Rectors Conference hosts a seminar on Quality Systems (GB, F, NL) for HE • 1995 CRE institutional quality audit • 1997, Vice-Rector for Quality • 1999 Follow-up of CRE audit • Initiatives to create awareness of the quality issue • Atenção UA! • Hosting a CRE/IMHE seminar - UA was presented as a case study • Repensar os Curricula (“Bologna before Bologna”)

12. Promotion of a Quality Culture: the beginning (2) • Decision to develop a single (integrated) information system • Creation of specific bodies responsible for quality development • GAGI (Information Management Office) • GAQAP (Quality, Evaluation and Procedures Office) • Redefinition of the university structure to include institutes for coordination of function

13. Internal Processes (1) • Transversal instruments • LEGUA (Legislation, Regulation and Procedures Knowledge Base) • Quality Protocol • Single Information System • Supporting and monitoring all the university activities in an integrated way

14. Single Information System

15. Internal Processes (2) • Teaching activities • Managed through the Education Institutes • Centrally • Locally through the figure of the Study Programme Director • Transparency of the global information system enabling that teaching and learning activities have their results available to all relevant actors • Indicators referring access numbers, drop-out rates, time to graduation and student performance are produced on a semester basis; employment data and feedback from employers - initially on a five year cycle, now on an annual basis • Since 1994, on a five-year cycle basis, the UA study programmes have been evaluated in accordance with a national framework

16. Study Programme Evaluation (at national level) • Promoted by FUP (Portuguese University Foundation) and, afterwards, CNAVES • Comprises • Self-evaluation (SE) • SE Committee appointed by the Head of the Department responsible for the Study Programme • Based on standard evaluation guidelines (include questionnaires to teachers and students) • External evaluation • Evaluation team appointed by FUP, constituted by experts of other Portuguese universities • Site visit • To analyse and discuss the SE report with SE Committee, Teachers and Students • Evaluation report with recommendations • Recommendations of the evaluation report were discussed and totally or partially implemented

17. Internal Processes (3) • Research activities • Research Institute coordinates the activities of research units and encourages quality practices through support for high quality research • At Research Unit/Associated Labs level there exist internal mechanisms which regulate production of research, promoting quality through internal funding rules • Research activities are subject to an external quality processes in the sense that they are only funded if they meet the requirements of the funding institution (e.g. FCT, EU, …) • The Research Units are also subject of an external evaluation programme by FCT, carried out by international panels, on a three-year basis • Academic career itself foments quality research by valuing some types of publications over others and making promotion more dependent on research rather than teaching or other activities

18. Research Evaluation (at national level) • Promoted by FCT (Science and Technology Foundation) • Comprises • Activity report • Written by the Coordinator of the Research Unit • Based on standard evaluation form issued by FCT • External evaluation • Evaluation team designated by FCT, constituted by international experts • Site visit • To analyse and discuss the Activity Report with the members of the Research Unit • Evaluation report and a classification of the Research Unit (from Poor to Excellent) with direct impact on Unit funding

19. Internal Processes (4) • Administrative processes • Definition and management of these quality processes are the responsibility of the appropriate Head of Service • Mechanisms which monitor and regulate administrative processes are enforced by an internal network of quality-pivots coordinated centrally (by GAQAP) • They are disseminated by internet through LEGUA, and/or incorporated into informatics applications that support actual activities • Annually GAQAP gives a general questionnaire to all staff and a specific one to Heads of Units for identifying staff training needs • Implementation of a system for evaluation of administrative and technical staff within the scope of the national SIADAP (Integrated System for Evaluation in Public Administration)

20. Setting the scene at European level (1) • The initiative to introduce the European Higher Education Area was launched in 1998, in Paris, and confirmed, one year later, in Bologna where ministers from 29 countries signed a declaration to the purpose of creating a borderless EHEA by 2010. • The central idea of the Bologna process is to promote student and staff cross-border mobility through: • the adoption of a system of comparable two cycle degrees (Bachelor and Master) and, • the introduction of a harmonized credit system ECTS (European Credits Transfer System).

21. Setting the scene at European level (2) • Quality has always played a role in Higher Education but its importance increased enormously during the last decade. • Among the new challenges facing Higher Education Institutions are of paramount importance the combined requirements of promoting the Bologna process and participating in the European Higher Education Area and of creating the European knowledge society. • In both cases quality is seen as a key factor to achieve these objectives.

22. Setting the scene at European level (3) • Why Quality? • To guarantee the necessary trust between institutions, which is essential to make sure they will accept students who have acquired a certain number of credits in another institution, particularly in another country, the quality of institutions and its audit, as well as the recognition of degrees, have gained a central role in the whole process

23. Setting the scene at European level (4) • However • The emergence of quality as a key factor to the successful implementation of the EHEA is quite recent. • In 2003 it became the first policy objective of the Berlin Communiqué (September 2003). There, it is stated that quality of higher education is at the heart of the setting up of a European Higher Education Area and consequently it is essential the development of quality assurance at institutional, national and European level

24. Setting the scene at European level (5) • Henceforth the need to develop mutually shared criteria and methodologies on quality assurance. • It is also stressed that the primary responsibility for quality assurance in higher education lies with each institution itself. • It also invited the QA community, together with the Higher Education Institutions, to develop an agreed set of standards, procedures and guidelines on quality assurance and to explore the possibility of a peer-review of QA agencies

25. Setting the scene at European level (6) • In 2005 the E4 group (ENQA, EUA, ESIB and EURASHE) proposed to create an European Register for QA agencies. • In April 2007, at the 4th EUA Convention, a recommendation was issued emphasizing two points: • quality should be all embracing and quality assessment must also recognize diversity and encourage, rather than constrain, innovation and creativity in teaching and learning. • the European Register of QA agencies, expected to be approved at the 2007 Ministers meeting, has been developed by an innovative partnership between stakeholders and is seen as an important block in this area, and an essential component of the global attractiveness of European Higher Education.

26. Setting the scene at National level (1) • The Portuguese Higher Education System has profoundly changed in the last thirty five years: the Public University Sector was enlarged and a private higher education sector emerged and expanded very rapidly. An important feature is the existence of a binary divide between university-level and polytechnic-level programmes. • The system now comprises: • University Public Sector • 14 universities and one non-integrated university institute • 4 Military University Institutions • Polytechnic Public Sector • 15 Institutes • 21 non-integrated colleges which include nursing, education, military and police

27. Setting the scene at National level (2) • The private sector is much larger in terms of number of institutions, with more than 100, but accounting for less than 30% of the students, revealing the fragmentation of the system. • It is also noteworthy that all the public universities are comprehensive in terms of fields of study and they are, by large, the main actors of research in Portugal, carried primarily by the academic staff.

28. Setting the scene at National level (3) • In the current context of stabilization in the number of enrolments and severe shortage of public funds for higher education, the system and the institutions face considerable constraints. • This setting presents great challenges for the regulation and steering of the system, requiring clear policies on the overall mission and goals to attain, supported by transparent and stable funding mechanisms and by a regular and consistent evaluation framework. • Failing to do this will not only hinder institutions’ efforts to adopt a strategic approach, but will also fail to provide the adequate information for students, parents and other stakeholders, an issue of paramount importance for the enrolment decision process and for the relationships between institutions and the society.

29. Evaluation at National level: the institutions • Most public universities have sought international expertise to carry out a institutional evaluation. This was the case, in particular, of the Institutional Evaluation Programme run by the European University Association (EUA), formerly the Council of European Rectors (CRE). • During the nineties most public universities, Aveiro, Porto, Minho, Technical University of Lisbon, University of Lisbon, Universidade Nova de Lisboa, University of Algarve and Universidade Católica, have invited EUA to carry out a evaluation of the institution. Some of these institutions have also done one follow-up. • These exercise was self-contained within the institution evaluated, so the benefits of the exercise were not extended to the different type of institutions and a systemic perspective could not be obtained.

30. Evaluation at National level: the system • The Higher Education sector in Portugal is supporting the Glasgow declaration: Strong Universities for a strong Europe. Europe needs strong universities as motors in the knowledge triangle of education, research and innovation. • At national level it is accepted that Portugal needs universities/Higher Education Institutions that are able to move out of the shadows of nation state and are able to do both: go for excellence in research as well as in teaching and provide broad, equitable access to basic higher education in Europe. • In 2005 the government launched their programme which included as a priority the integration the Higher Education System in the European Area. In this context it was emphasised the need to carry out an all embracing evaluation of the system, according to the best practices in Europe and elsewhere.

31. New developments and new legal framework (1) • Background: Portuguese government promoted discussion on three levels: • HE system level (OECD); • setting-up of a national evaluation system (ENQA); • Institutional level (EUA) • Ministry (of CTES) supports programme for institutional evaluation

32. New developments and new legal framework (2) • During 2006 ENQA reviewed the accreditation and quality assurance practices in the Portuguese higher education against the best European practices and following the Guidelines for Quality Assurance in the European Higher Education Area and issued recommendations that are now being implemented at national level. • These recommendations emphasised the European trend of a distinction between accreditation at programme level and academic audit and evaluation at institutional level

33. New developments and new legal framework (3) • The Portuguese Quality Assurance Agency will adopt the following definition of accreditation: • The characteristic of accreditation is that the process is based on established external criteria and that the result of the process is a decision as to whether these criteria have in fact been met by the programme under accreditation.

34. New developments and new legal framework (4) • The government decided to encourage ALL the Portuguese Higher Education Institutions, universities and polytechnic public and private to apply for an Institutional Evaluation according to the EUA International Evaluation Programme and set up the following focus for such an evaluation: • The main characteristics, positive and negative of the Portuguese Higher Education institutions, their stage of development in the European context and their efficiency. • The level of institutional autonomy and the practices related to governance and management. The report should make suggestions for improvement.

35. New developments and new legal framework (5) • The institutional capacity for change and the way institutions respond to the increasing challenges facing higher education. • The access and the way students are recruited. Special attention should be paid to LLL and the opening of the institution to the new publics and professional continuous education. • The relevance of the diplomas awarded. • The diversification of funds and the way these are managed. • Recommendations on how to promote the rationalization of he offer and diversification of institutions.

36. New developments and new legal framework (6) • 2007 • RJIES • Law that establishes new governance models for HEI. Institutions should have new statutes until June 2008 • RJAES • Law that establishes the legal environment for the evaluation of HE (focus on evaluation of study programmes)

37. New developments and new legal framework (7) • The UA decided (2006) to apply for EUA’s institutional evaluation programme • The UA is finalising its statutes • Changes in university governance bodies, both at central and Department/School levels, may/will lead to a new approach and organisation for the quality assurance mechanisms

38. Administrative support Administrative support Research Research Education Education Local Management Re-Structuring the QA Organisation Management Organic Unit 1 ….. Organic Unit n Central Management Operational role

39. Key areas of the Action Plan • From self-evaluation (SWOT) and contribution of external evaluation team • Performance indicators • Bologna Process • Partnerships • Funding • Organisation and governance

40. Thanks for your Attention