QAC Seminar Introduction to QAC Audit

1. QAC Seminar Introduction to QAC Audit HKBU 12 Mar 2008 Trevor Webb Assistant Secretary-General QAC Secretariat Quality Assurance Council

2. Seminar Objectives To provide participants with: • Background to the QAC; and an • Overview of the QAC audit process; • Outline of the responsibilities of audit panel members; • Introduction to the audit methodology and to the auditing of teaching and learning; • Explanation of methods of gathering information, and audit and interviewing techniques • Opportunityto identify issues and raise questions Quality Assurance Council

3. Overview of QAC • QAC Structure • The Council • The Secretariat • Auditors and Audit Panels • Primary roles of the QAC • Conduct Quality Audits of Teaching & Learning • Promote QA & QE / spread good practice • Rationale for establishing the QAC Quality Assurance Council

4. Progress To Date • Audit Manual (http://www.ugc.edu.hk/eng/qac/index.htm) • Auditor Briefing Notes • Initial Audit Cycle • Register of Auditors • Auditor Induction & Professional Development • CUHK and HKBU Audits Quality Assurance Council

5. QAC Quality Audits #1 QAC Quality Audits: • will NOT question institutional autonomy • willNOT validate or re-accredit institutions (self-accrediting status is recognised) • willNOT be excessively intrusive • willNOT be confrontational or trying to find fault • willNOT be about surprises • will NOT compare institutions • will NOT determine funding of institutions Quality Assurance Council

6. QAC Quality Audits #2 QAC Quality Audits are: • about self-evaluation • about peer review • about working with institutions • about partnership • about dialogue • for institutions and students • a constructive but robust process • about helping institutions to improve Quality Assurance Council

7. Roles & Responsibilities of Auditors Auditors are expected: • to be familiar with the QAC’s approach/methodology • to read thoroughly the Institutional Submission (IS) and the Supplementary Material • to provide written comments on the IS before the Initial Meeting of the Audit Panel • to participate fully in meetings, contributing to the discussion • to avoid applying pre-conceived templates to their consideration of the audited institution • to contribute to, and agree, the Panel’s Report • to work as a Team (QAC Audit Manual Annex B.3) Quality Assurance Council

8. QAC Approach to Audit QAC Audits are about: Fitness for Purpose in teaching & learning, where the Quality of Student Learning is the Centre-piece (focal point), with an emphasis on Quality Enhancement, i.e. continuous improvement [QAC Audit Manual Section 3.1] Quality Assurance Council

9. QAC Audit Methodology & Focus Areas#1 Audit Panels will use the ADRI methodology: Approach (What is the institution’s purpose?) Deployment (How does the institution achieve its purpose?) Results (What evidence does the institution have that its purpose is being achieved?) Improvement (What processes are in place for improvement) in relation to 11 core Focus Areas (tobe revisited later in the Seminar) Quality Assurance Council

10. QAC Audit Methodology & Focus Areas#2 Audit Panels examine : • how an institution’s practices in each focus area contribute to student learning quality • whether an institution’s objectives, procedures and outcomes are informed by appropriate external reference points • a sample of programmes in depth, as “test cases” (QAC Audit Manual Sections 3.3, 3.4 and 3.5) Quality Assurance Council

11. The QAC Approach: “‘Fitness for Purpose’ where institutions have different purposes which reflect their missions and role statements they have agreed with the UGC…not a review against pre-defined standards.” Quality Assurance Council

12. QAC Audit Scope & Focus “Student learning quality is the focal point of the audit system” “The QAC has selected a set of…activities common to all institutions, as the ‘focus areas’ of audit….taken together, the focus areas effectively define thescope of QAC audit.” (QAC Audit Manual, Section 3.4) Quality Assurance Council

13. Scope of QAC Audit Activities • All HK first degree programmes and above, however funded, that are provided by UGC- funded institutions; • Includes provision by continuing and community education arms; • Includes self-financing and joint programmes leading to a HK award; • Includes teaching and learning aspects of research degree programmes. Quality Assurance Council

14. QAC Audit Focus Areas • Articulation of appropriate objectives • Management planning & accountability • Programme development and approval processes • Programme monitoring and review • Curriculum design • Programme delivery, resources, mode, learning env. • Experiential / out of classroom learning, i.e. “external engagement” (e.g. OS exchange, work-integrated-learning, Professional and Industry linkages, etc.) • Assessment • Teaching quality and staff development • Student participation • Activities specific to research degrees Quality Assurance Council

15. The QAC Audit Process Preparation Panel Selection (QAC) Self-Review & Institutional Submission (Auditee) Preliminary Phase The Initial Meeting Preliminary & Offshore Visits(Auditors) The Audit Visit (The ‘main event’) The Audit Report(QAC) & Follow-Up (Auditee & QAC)

16. Preparation • Initiation of the process • Nomination of additional focus areas • Panel selection • Self-review • Institutional Submission Quality Assurance Council

17. Self-review & Institutional Submission • Self- review is “an opportunity for the institution to analyse and reflect on its objectives for student learning, on its procedures, standards and benchmarks, and…measures” (Audit Manual, Section 5.4) • Self-review also forms the basis of the Institutional Submission to the audit • QAC does not prescribe approach, but there are some key issues that should be addressed. Quality Assurance Council

18. Key Self-review Questions • What student learning outcomes are we trying to achieve? (graduate attributes, standards etc.) • How are we trying to achieve our desired student learning outcomes? (policies & procedures etc.) • How do we know we are achieving them? (what measures and benchmarks, etc.) • How do we respond to what the evidence tells us? (what feedback mechanisms, impact on programme standards etc.) Quality Assurance Council

19. Institutional Submission • QAC does not prescribe a detailed format, but…. • Auditee is expected to submit a draft Institutional Submission to QAC six months prior to Audit Visit and the final three months prior; Two components • Main Submission • Supplementary Materials – key documents referred to in Main Submission. Quality Assurance Council

20. Main Submission • 12,000 words • Section 1 – Introduction/Structures • Section 2 – Overview of Teaching & Learning QA System • Section 3-N Focus Areas • Section N+1 Conclusion • References *** Should be “Stand-alone” *** Quality Assurance Council

21. Preliminary Phase • Initial Meeting of the Panel (about 9 wks before AV) - Refresh on QAC audit procedures - Review of Institutional Submission to identify major issues & questions - Selection of programmes for sample review - Identification of further information/documentation required - Planning of Audit Visit - Decisions on whether to visit provision outside HK - Decisions on who to meet during the Audit Visit • Preliminary Visit to Auditee to plan the Audit Visit (about six weeks beforehand) • Other visits, such as to operations outside HK, as required. Quality Assurance Council

22. The Audit Visit (s) • Main Visit length 3-5 days • Panel typically operates as a group, but may split for some parallel sessions • An Audit Visit Programme and Worksheets are prepared to provide a common reference for the panel and to ensure that the necessary issues are covered. Quality Assurance Council

23. Audit Visit – Usual Sessions • Opening meeting with President/Vice-Chancellor • Staff (various groups, senior through to junior) • Students (including those taking sample programmes) • Key boards and committees • Range of stakeholders (Council Members; Employers) • Panel deliberation sessions • “Sessions on request” • “Call-back sessions” • Exit meeting (QAC Audit Manual, section 7 and Annex A.2) Quality Assurance Council

24. The Audit Report Format: • Executive Summary • Findings for each Focus Area: - Commendations - Affirmations - Recommendations • Programmes sampled (but not named) • Institutional response (Appendix) Quality Assurance Council

25. The Audit Report (cont.) • Concern by QAC over “fairness, accuracy and balance”; • Produced by the Audit Co-ordinator - exposure draft sent to auditee - feedback considered by AC and Chair • Brief institutional response • Consideration by QAC • Publication • Consideration by the UGC Quality Assurance Council

26. Follow-Up • 18 Month Progress Report: - Action (s) taken - Person responsible - Date by which action is expected to be completed - Means by which the institution will assess effectiveness • Consideration by QAC • Consideration by UGC Quality Assurance Council

27. Any Questions? Discussion Quality Assurance Council

28. Audit Method Components • Approach to audit (Fitness-for-purpose √) • Audit Scope & Focus (Areas of Focus√) • Triangulation of data & information sources (Institutional Submission, supporting documents, sampling of programmes, meetings • Document Analysis and Interpretation of Information - ADRI (Identification of key quality issues and lines of enquiry) • Identification and Reporting of audit findings (to be covered in depth in the pre-audit Intensive Workshop) • Auditing, interviewing, questioning, and meeting organisation tools & techniques required for effective performance of the above. Quality Assurance Council

29. A The Critical Questions: D I R • What are the institution’s objectives and on what basis does it set these objectives? (A) • How does the institution go about achieving its objectives? (D) • How does the institution measure achievement of its objectives and what evidence is there that the objectives are being achieved? (R) • What processes are in place for improvement (and howdo these processes feed back to the setting and reporting of achievement of the objectives)? (I) Quality Assurance Council

30. Managing Meetings • How might participants be feeling on entering the meeting room? • How can the panel help participants feel more at ease? • How can the panel ensure that all participants are involved in the discussion? Quality Assurance Council

31. Questioning Techniques #1 In general, auditors should avoid: • asking complex or multiple questions up-front (a more straight-forward question, with a follow-up question/(s) is generally more effective); • telling anecdotes or making speeches; • detailing the situation in their own organisation; and, • offering gratuitous advice. Quality Assurance Council

32. Questioning Techniques #2 • Creating an atmosphere for genuine dialogue during the Audit Visit is extremely important • Panel members should try to act as colleagues and peers of the participants, rather than acting as inspectors • To this end, the questioning and discussion must be fair and polite. It must also be rigorous and incisive, as the audit report must reflect both the institution’s achievements and its areas for improvement (Commendations, Affirmations, and Recommendations) • Panel members should ensure that all participants are engaged in the discussion, for the encouragement of an organisational culture of continuous quality improvement. Quality Assurance Council

33. Any Questions? Opportunity for final discussion of any Seminar topics. Quality Assurance Council

34. Thank you for your participation Quality Assurance Council