Training of E xternal R eviewers

1. Training of External Reviewers Twinning project “Support to strengthening the higher education system in Azerbaijan” (No AZ/14/ENI/OT/01/17 (AZ/49)) 11 September 2019 This project is funded by the European Union

2. Outline • Review principles and site-visit • Session 3: Interview ethics and techniques • Session 4: Practical exercise on interviewing techniques: simulation of interviews

3. Principles • The evaluation is an evidence-based process carried out by an independent review following the approved methodology; • Reviewers should neither assume the roles of inspectors nor consultants of HEIs, but act as critical, but friendly external experts; • The information provided by the Higher Education Institutions (HEIs) is assumed to be factually correct unless other evidence point to the contrary; • The evaluation is a process of verification of the information provided in the self-evaluation report (SER) and other documentation and the exploration of any matters which are omitted from that documentation and interviews with representatives of HEI under evaluation.

4. Review process Self-evaluation report Preparation for the visit Visit Final report

5. A site-visit by the Experts to the HEI • To verify and clarify information provided in the SER; • To gather evidence to support the findings; • To explore the programme compliance with the set criteria and discuss it during meetings and interviews with key individuals at the HEI; • To formulate the review team’s findings regarding compliance with the set criteria, strengths and areas of improvement and communicate these to the HEI; • To make sure that the team has all necessary information and evidence for producing a clear, detailed and evidence-based final evaluation report. • The site visit evaluation should be rigorous, fair, transparent and consistent.

6. A site-visit by the Experts to the HEI During the visit the expert team will have meetings/interviews with key personnel of the study programme, students and stakeholders. Also, the experts will visit facilities and material resources dedicated to the evaluated study programme. There are three major sources of information available during the site visit: • Interviews with teachers, administration, students, graduates and representatives of employers; • Documentary information the team asks for or the HEI wishes to make available on site; • Visiting of teaching and learning facilities.

7. Interview ethics and techniques Session 3

8. Ethical principles Main values which review team should pursue are Unbiased and fact-based findings,Objectivity, Respect for diversity, Confidentiality and Collaboration. Also, review team members should follow principles such as respect for the participants of the evaluation, impartiality, transparency, accuracy, professionalism, lawfulness etc.

9. The principle of unbiased and fact-based findings The experts shall act in an honest and unbiased manner in higher education institution efforts to reach the aim of the evaluation. When expressing higher education institution opinion, formulating conclusions or taking decisions, the expert shall rely on the facts, observations, and his/her personal competence.

10. Objectivity principle An expert shall be fair in his/her efforts to achieve the aims of the evaluation and to evaluate the study programme objectively. While expressing his/her opinion, formulating conclusions or taking decisions, an expert shall draw on precise facts and information and his/her own competence. The expert shall act independently and shall not represent the interest of the higher education institution or other party.

11. Respect for diversity Within the assessment process, the expert shall act in good faith as a professional. The expert shall not exceed his/her powers as specified in his/her tasks. The expert shall treat the parties involved in the assessment process as persons capable of taking responsibility for higher education institution actions, therefore he/she shall rely on facts and observations when referring to the strengths and weaknesses of the study programme.

12. Confidentiality principle All the information related to the evaluation (opinions of the interviewees, the self-evaluation report, and additional information provided by the higher education institution) shall be used exclusively for the evaluation process.

13. Collaboration principle Each expert, as a member of the experts group, shall be open to collaboration with other members of the experts group. The collaboration of the experts shall be coordinated by the chair of the experts group. The experts group shall develop mutual understanding with the representatives of the higher education institution and make efforts to assist the higher education institution to enhance quality culture.

14. Declaration of Interests Any expert performing the task must complete the Confirmation of Absence of Conflict of Interest and indicate a potential conflict of interests if there is a situation that could hinder the expert to be impartial and objective while fulfilling the task.

15. Declaration of Interests • the expert or someone among the expert’s close relatives (spouse, cohabiting partner, the partner where partnership is registered in the manner established by laws as well as higher education institution parents (adoptive parents), children (adopted children), brothers (step brothers), sisters (step sisters), grandparents, grandchildren and higher education institution spouses, cohabitants or partners - (hereinafter referred to as the close persons) in the last 5 years worked in the institution which programmes are being evaluated; • the expert or someone among the expert’s close relatives during the last 2 years participated in the evaluative activities of the institution (acted as a member of the Senate, the Academic Council, the Committee of the Final Thesis Defence, the Committee of Doctoral Studies, a member of the management bodies, shareholder or similar);

16. Declaration of Interests • the expert or someone among the expert’s close relatives during the last 3 years studied in the institution which programmes are being evaluated; • the expert or someone among the expert’s close relatives participates in any ongoing project or other joint activity with the institution he works for, with the institution which programmes are being evaluated; • other circumstances which are unspecified herein and are related to the expert or his close relative and which may prevent the expert from being objective (i.e., he has publicly expressed a negative attitude towards the institution which programmes are being evaluated; has terminated work relations after the conflict with the institution which programmes are being evaluated).

17. Expert roles Team leader – tells what to do • Team leader • Team members (academics) • Representative of employers • Student • Coordinator Coordinator – tells how to do Coordinator is the mediator between the expert group and the HEI No direct contact with Higher Education Institution!

18. Responsibilities (Team leader) Before the site visit • Takes overall responsibility for the functioning of the evaluation; • Distributes tasks to other team members; • Together with other members reviews the documentation, including the SER and any other information available prior to the site visit to ensure it is complete and adequate for the needs of the team; • Agrees the issues to be clarified during the site visit – undertaken with contributions from the rest of the team. During the site visit • Chairs all the meetings and discussions or delegates it to a responsible/other expert; • Ensures that all lines of inquiry identified are satisfactorily answered. After the site visit • Makes sure the team prepares drafts of the final evaluation reports on time – based on the documentation provided and the notes taken during the site visit; • Assures that all amendments are included (if any and if accepted by the expert team, if not – includes arguments why the remarks were not taken into account) in the draft evaluation report; • Makes sure the evaluation is done in accordance with procedures and final evaluation reports are well-founded and comprehensive.

19. Responsibilities (Expert team members) Before the site visit • Review the documentation, including the SER and any other information available prior to the site visit; • Contribute to the development of the lines of inquiry to be pursued during the site visit; • Prepare the preliminary evaluation reports (or their parts) on assigned programmes; • Prepare accurately for the site visit. During the site visit • Participate in all of the meetings and discussions and contribute to pursuing agreed lines of inquiry; • Take notes at the meetings attended and during private discussions of the panel. After the site visit • Produce an initial draft of the evaluation report and circulate it to all expert team members; • Consider the comments on the draft evaluation report (if any and if accepted by the expert team, if not – include arguments why the remarks were not taken into account) and amend the draft evaluation report; • Comment on the initial draft evaluation reports produced by other experts.

20. Student member is a full member of the review team. His/her role is particularly important in ensuring that the student experience is given appropriate consideration. He/she is responsible for ensuring that the student voice is heard and reflected appropriately within action planning, contributing to discussions from the perspective of students and exploring any themes he/she wishes that impact on the student learning experience. The student representative is usually responsible for describing the study process and student assessment.

21. Representative of employers is expected to ensure that the professional requirements, public needs and the needs of the labor market are given appropriate consideration. The social partner is usually responsible for input to all reports on programme learning outcomes compliance to the needs of state/ private/ organisational and individual interests. Also, he/she is expected to provide insights if acquired competences and professional activities of the majority of graduates meet social partners' expectations. The social partner is expected to review all evaluation areas pertaining to employment in all sectors (state, private, nongovernmental) or to an individual practice of the profession.

22. Evaluation coordinator • Coordinates agendas with the expert team and HEI, arranges logistics, takes care of payments after the submission of final reports; • Consults the expert team on evaluation process, legal requirements, other specific issues related to evaluation task; • Communicates between the expert team and the HEI; • Takes care of circulation of the reports; • Is present during the expert work in Azerbaijan. The coordinator does not act as a member of the expert group.

23. Visit timetable AGENDA OF THE VISIT Be on time and observe the time

24. Introduction • Welcome everyone • Introduce the Team • Remind the aims of the meeting, confidentiality • Follow the agenda • Mind the language issues • Encourage to speak • Write down the remarks Please, turn off the phones and remind others to do the same

25. Interview techniques • Tone – objective and neutral, professional, polite, respectful • Adopted to different target groups • Different organizational cultures must be in mind • Prior perceptions/ prejudices should not appear • Intercultural communication is important

26. Interview techniques • Attention to data in the self-evaluation report • Looking for facts (who? when? how?); • Exploration (“please, tell more...”; “please, explain...”; “we have read, that...”); • Generalization (“did you have in mind, that...”, “ can I summarize...”).

27. Questions and search for the evidence • Approx. 6 overarching questions for one meeting • Simple, short questions • Avoid biased, tendentious questions (“Don’t you think that…”) • Avoid too difficult questions • Avoid giving your institution as an example (“We would always…”) • Ask for examples • Adjust questions to different groups. Repeat them to hear different opinions.

28. Questions and search for the evidence • Explain the context if the question remains unanswered • Ask, if there are any questions, but be careful on going into discussions or arguing • Don’t discuss answers among yourselves during the interview • No comments! (this is bad or this is good situation) • Don’t show if the answer is not good enough • No conclusions in advance! Don’t forget to thank participants

29. Example Ask qualitative questions which are open-ended and allow “informants to tell their story”: Questions for students and for student union: • Do students‘ representatives or student union members participate in the meetings of the administration (closed question) – Factual information. • Is student opinion heard at the decision making level? • Could you present any examples when the students’ opinion was taken into account? Possible change: • How students are integrated into the decision making process at your HEI? Please present us with any example. (open question)

30. Place for the ideas • Good examples • One best thing about this study programme • One thing that needs to be improved • Wish list

31. Practical exercise on interviewing techniques: simulation of interviews Session 4

32. Participants divided into groups (interviewers, target group, observers) • Different roles explained • Extract from self evaluation report (reading in each group) • Positives and negatives of the area • Questions to the target groups • Interview simulation • Discussions • If possible – round two