Hep quality audit manual draft v1 l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 93

HEP Quality Audit Manual (draft v1) PowerPoint PPT Presentation


HEP Quality Audit Manual (draft v1). © 2007 Martin Carroll, Salim Razvi, Tess Goodliffe Training Module 21 v1 Oman Accreditation Council. Module Objectives. Participants will: Have a detailed knowledge of the HEP Quality Audit Manual draft v1;

Download Presentation

HEP Quality Audit Manual (draft v1)

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Hep quality audit manual draft v1 l.jpg

HEP Quality Audit Manual (draft v1)

© 2007 Martin Carroll, Salim Razvi, Tess Goodliffe

Training Module 21 v1

Oman Accreditation Council


Module objectives l.jpg

Module Objectives

Participants will:

  • Have a detailed knowledge of the HEP Quality Audit Manual draft v1;

  • Be able to prepare for the Quality Audit with confidence;

  • Develop a deeper appreciation of the Quality Audit process;

  • Develop a deeper appreciation of the nature of evidence in Quality Audit;

  • Have an opportunity to provide comments on the draft manual.


The manual s table of contents l.jpg

The Manual’s Table of Contents

Part A:Quality Audit Overview

9.00am – 9.30am Presentation, Q&A

Part B:The Self Study

9.30am – 9.40am Presentation

Part C:The External Review

9.40am – 10.00am Presentation

10.00am – 10.20am Short Break

10.20am – 10.40am Working Groups

10.40am – 11.00am Group Feedback

Part D:Methods Of Analysis

11.00am – 11.20am Presentation

11.20am – 11.40am Working Groups

11.40am – 12.00pm Group Feedback

Part E:Appendices


Part a quality audit overview l.jpg

PART A:QUALITY AUDIT OVERVIEW


Slide5 l.jpg

Provider QA Processes

First cycle

commencing

HEP Accreditation

2008

HEP Licensure

Stage

1

:

Quality

Audit

4 years

≤4 years

Appeal

HEP Accreditation

HEP Accreditation

Pass

Certificate

Stage

2

:

Standards

Assessment

Fail

KEY

1-2 years on

Probation

Start/End

Pass

Process

HEP

Accreditation

Terminated

HEP Standards

Fail

Reassessment

Document


Slide6 l.jpg

Provider QA Processes

HEP Accreditation

Stage

1

:

Quality

Audit

4 years

≤4 years

HEP Accreditation

HEP Accreditation

Pass

Certificate

Stage

2

:

Standards

Assessment

STAGE 1: QUALITY AUDIT

Self Study

External Review

External

Quality Audit

Quality Audit

Self Study

Review

Portfolio

Report

Activities

Activities


Slide7 l.jpg

Provider QA Processes

HEP Accreditation

Stage

1

:

Quality

Audit

4 years

≤4 years

HEP Accreditation

HEP Accreditation

Pass

Certificate

Stage

2

:

Standards

Assessment

STAGE 2: STANDARDS ASSESSMENT

Self Assessment

External Assessment

Self

Assessment

Activities

Standards

Assessment

Report

External

Assessment

Activities

Assessment

Application


Slide8 l.jpg

Program QA Processes


What is quality audit l.jpg

What is Quality Audit?


What is a quality audit l.jpg

What is a Quality Audit?

In the Sultanate of Oman, a Quality Audit is a public and systematic determination of whether:

  • A HEP’s goals and objectives are based on appropriate regulations, standards and benchmarks;

  • Its planned arrangements are suitable to achieve those goals (i.e. check the overall approach);

  • Its actual practice conforms to the planned arrangements (i.e. check the deployment);

  • The arrangements achieve the desired results;

  • The organisation is learning from a self-evaluation of its approach, deployment and results, and can demonstrate improvements.


What is a quality audit11 l.jpg

What is a Quality Audit?

In the Sultanate of Oman, a Quality Audit is a public and systematic determination of whether:

  • A HEP’s goals and objectives are based on appropriate regulations, standards and benchmarks;

  • Its planned arrangements are suitable to achieve those goals (i.e. check the overall approach);

  • Its actual practice conforms to the planned arrangements (i.e. check the deployment);

  • The arrangements achieve the desired results;

  • The organisation is learning from a self-evaluation of its approach, deployment and results, and can demonstrate improvements.


Slide12 l.jpg

Reviewing Graduate Destinations

ADRI Cycle of Quality Assurance

Quality Audit ‘Double Loop’ Process

IMPROVEMENT

APPROACH

RESULTS

DEPLOYMENT

Internal Review of the System (results in a Portfolio section with Strengths & OFI)

External Review of the System (results in Audit Report section with Coms, Affs, Recs)

  • Evidence examples:

  • Internal program review reports

  • Input obtained from industry & professional bodies

  • Resulting action plans

  • Reports on progress against actionplans

  • Evidence examples:

  • HEP graduate destination targets

  • Industry needs analysis

  • Careers Service Plans

  • Curriculum demonstrating links to industry needs

Topic:

Graduate

Destinations

  • Evidence examples:

  • Graduate destinationstatistics

  • Evidence of employersatisfaction (e.g. surveyresults)

  • Evidence examples:

  • Interviews with academic staff,careers service staff, recent graduatesand employers

ADRI is a model of quality assurance used by agencies in many countries around the world (e.g. AUQA in Australia)

It can be applied to any topic – in this case, the analysis of a HEP’s graduate destinations


What a quality audit is not l.jpg

What a Quality Audit is Not

  • Secret (although some of the information and deliberations will be confidential).

  • An ‘accreditation’ in itself (it is part of the overall accreditation system).

  • A measurement of performance against each OAC HEP standard (like Standards Assessment), but audit does use those standards to help define the scope (i.e. chapter headings in the Portfolio).

  • A summative assessment (it does not result in a pass/fail or a grade or score).

  • A strategic review (it is focused on how well a HEP is doing, not what future direction it should head in).


Slide14 l.jpg

Evaluations / Audits Accreditations

National Agencies for Accreditation & Quality Assurance in Europe, 2005

Rolf Heusser, Chairman of European Consortium for Accreditation (2006) Mutual Recognition of Accreditation Decisions, INQAAE Workshop, Den Haag, Nederlands.


Scope of audit section 3 4 in the quality audit manual l.jpg

Scope of Audit(section 3.4 in the Quality Audit Manual)


Comparison with old rosqa l.jpg

What does Quality Audit Consider?

Comparison with old ROSQA

NEW SCOPE

PREVIOUS SCOPE

  • Governance & Management

  • Student Coursework

  • Student Research

  • Staff Research & Consultancy

  • Community Engagement

  • Academic Support Services

  • Students & Student Support Services

  • Staff & Staff Support Services

  • General Support Services & Facilities

  • Mission, Goals & Objectives

  • Governance & Administration

  • Learning & Teaching

  • Student Services

  • Learning Resources

  • Facilities & Equipment

  • Financial Planning & Management

  • Staffing & Employment Procedures

  • Research

  • Community Relationships


Part b the self study l.jpg

PART B:THE SELF STUDY


Self study principles 1 of 2 l.jpg

Self Study Principles (1 of 2)

  • A Self Study is a rigorous and comprehensive evaluation of the HEP and all its significant areas of activity

  • The Self Study should be championed by the very highest levels of the organisation.

  • The Self Study should involve many people.

  • The Self Study will take time. There is a large amount of information to be collected, analysed, interpreted and reported.


Self study principles 2 of 2 l.jpg

Self Study Principles (2 of 2)

  • A Self Study is evaluative, not just descriptive. The idea is to find out not just what is happening, but how well it is happening. If we think we are doing something well, can we prove it?

  • A Self Study will require valid and reliable quantitative and qualitative information.

  • If done well, the Self Study will also have a value to the HEP independent from its purpose as the submission document for Quality Audit.


What is a quality audit portfolio l.jpg

What is a Quality Audit Portfolio?

  • Covers the entire institution.

  • Explains the HEP, its mission and vision, its special characteristics and its context.

  • Provides a concise summary of the HEP’s quality assurance systems for each function.

  • Provides a robust analysis of the effectiveness of those systems

  • Identifies strengths.

  • Identifies where improvements are needed.


What does a portfolio look like l.jpg

What does a Portfolio Look Like?

  • Maximum 30,000 words; 100 pages (including Appendices).

  • In English (and Arabic, if necessary/requested).

  • Hard copy and PDF form.

  • List of Supporting Documents and optional supporting Materials.

  • Final version should be professionally edited, type set, printed and published.


Typical portfolio table of contents l.jpg

Typical Portfolio Table of Contents

Chairperson’s Introduction

Overview of [name of HEP]

The Self Study Method

1. Governance and Management

2. Student Coursework

3. Student Research

4. Staff Research and Consultancy

5. Community Engagement

6. Academic Support Services

7. Students and Student Support Services

8. Staff and Staff Support Services

9. General Support Services and Facilities

Appendices

Supporting Material Index

Available Supporting Material Index


Slide23 l.jpg

Each Chapter of the Portfolio should conclude with ‘Areas of Strength’ and ‘Opportunities for Improvement’ (OFI)

(Although not every issue will necessarily conclude with a strength or OFI)


Areas of strength l.jpg

Areas of Strength

  • ‘Areas of Strength’ refer to areas in which the HEP can be justifiably proud

  • An Area of Strength is where the HEP can show effectiveness in achieving intended results

  • An Area of Strength highlights an example of potential good practice for the Audit Panel


Areas of strength examples l.jpg

Areas of Strength Examples

Area of Strength #1

Hogwarts College has an effective system for annual course review conducted by the Academic Board that incorporates student evaluations, teacher reflections, industry input and a review of scholarship in the field and that leads to course improvements.

Area of Strength #2

Hogwarts College provides excellent pathways for graduates into foreign universities through the establishment of eight active student exchange and articulation agreements.


Opportunities for improvement l.jpg

Opportunities for Improvement

  • During the Self Study, the HEP identifies issues in need of particular attention

  • Identifying an OFI demonstrates the HEP’s understanding of the issue and its commitment to taking appropriate action

  • Therefore, if verified, may result in an ‘Affirmation’ rather than a ‘Recommendation’ in the Audit Report

  • The main advantages of identifying OFIs is to aid the HEP’s ongoing quality improvement activities


Ofi examples l.jpg

OFI Examples

Opportunity for Improvement #1

Hogwarts College needs to implement a systematic approach to analysing and acting upon the feedback it receives from its student surveys.

Opportunity for Improvement #2

Hogwarts College needs to redesign its research funding scheme in order to achieve desired results.


Part c the external review l.jpg

PART C:THE EXTERNAL REVIEW


Quality audit protocols section 9 in the quality audit manual l.jpg

Quality Audit Protocols(section 9 in the Quality Audit Manual)


Quality audit protocols l.jpg

Quality Audit Protocols

  • Conflicts of Interest

  • Undue Influence

  • The Chatham House Rule

  • Transparency vs Protectionism

  • Sensitive Information

  • Complaints about the HEP


External reviewers section 11 in the quality audit manual l.jpg

External Reviewers(section 11 in the Quality Audit Manual)


Register of external reviewers l.jpg

Register of External Reviewers

  • The OAC is assembling a Register of External Reviewers. The OAC has sought nominations. All decisions are made by the OAC Board.

  • Includes local and international leaders in academia, higher education management, professional bodies and industry.

  • Membership lasts for 3yrs, and may be renewed.

  • The External Reviewers will receive thorough training and an Auditor Manual.

  • Only External Reviewers in the approved Register may be used for Quality Audits (and other types of external panels).


Quality audit panel membership l.jpg

Quality Audit Panel Membership

  • Membership will be assembled approximately as follows (subject to HEP consultation):

    • Two academicians (1 local; 1 international)

    • Two people with HEP management experience (1 local; 1 international)

    • One person from a relevant industry or profession

  • Each panel is supported by an Executive Officer from the OAC.

  • Each Panel may be accompanied by an Observer (who has no role in the process).


The audit process see table 1 in the quality audit manual l.jpg

The Audit Process(see Table 1 in the Quality Audit Manual)


Key roles in the audit process l.jpg

Key Roles in the Audit Process


Slide36 l.jpg

KEY

Process

Decision

Document

Terminator / Link


The audit visit section 16 in the quality audit manual l.jpg

The Audit Visit(section 16 in the Quality Audit Manual)


About the audit visit l.jpg

About the Audit Visit

  • Typically gets all the attention, but it is ONLY ONE PART of the overall quality audit process.

  • A chance for the Audit Panel to test the accuracy and completeness of the Portfolio.

  • Audit panel members can interview people and review resources and facilities.

  • May be 2+1 or 3+1 days, depending on the size and complexity of the HEP.

  • Highly professional, but also conducted in a positive and friendly manner.


Slide39 l.jpg

Dean

Heads of Department

Academic Staff

Students (UG)

External Stakeholders

Laboratories

Council Members

Administrative Heads

Academic Staff

Students (PG)

Student Services

Library & IT

TIME

DAY 1

DAY 2

DAY 3

0900-0945

Interview 1

Interview 7

Callback interviews(if required)

1000-1045

Interview 2

Interview 8

Panel deliberationsand report drafting

1100-1130

Panel Review

Panel Review

1130-1215

Interview 3

Interview 9

1230-1345

Interview 4 (lunch)

Interview 10 (lunch)

1400-1500

Panel Review

Panel Review

1500-1545

Interview 5

Interview 11

1600-1715

Interview 6 (on site)

Interview 12 (on site)

Verbal Feedbackto HEP

1730-1830

Panel Review

Panel Review

Example of a typical 2+1 Audit Visit program. Precise details will vary for each Audit.


Interview session logistics l.jpg

Interview Session Logistics

  • Unless otherwise arranged:

    • Maximum 8 interviewees per session.

    • All people will be selected by the Panel.

    • People should meet with the Panel once only.

    • Managers & staff interviewed separately.

  • The sessions are confidential, in that the Panel may use the information received, but may not reveal the identity of the provider (Chatham).

  • HEPs are also expected to respect the confidentiality of the process and may not coerce interviewees to say certain things or to divulge what was said.


Random interviews l.jpg

Random Interviews

  • Support collection of independent data.

  • Carried out by Panel members and take place during the audit visit.

  • Panel members will only speak with staff and students.

  • Potential interviewees are not obliged to participate

  • Interviews will usually be conducted confidentially with individual people.

  • The type of questions that will be asked during the random interviews are in Appendices M and N in the Quality Audit Manual.

(see section 16.2.4 of the Quality Audit Manual)


Public submissions l.jpg

Public Submissions

  • Important aspect of accountability – supports the independence of the Quality Audit.

  • The public are invited (through the HEP, the local press and the OAC website) to submit their views.

  • Anonymous submissions will not be accepted.

  • Personal grievances and vague allegations will not be considered by the Panel.

  • Submissions will be considered in the context of the overall Quality Audit.

(see section 15 of the Quality Audit Manual)


Preliminary feedback l.jpg

Preliminary Feedback

  • Takes place at the end of the Audit Visit.

  • Audit Panel meets with HEP CEO and up to seven other HEP representatives.

  • Panel gives verbal feedback to HEP based on the preliminary indication of the Panel’s findings.

  • The feedback presented to the HEP during this session is confidential and not open for discussion.

  • This is preliminary feedback only and notbinding on the OAC.

(see section 16.2.8 of the Quality Audit Manual)


The audit report section 17 in the quality audit manual l.jpg

The Audit Report(section 17 in the Quality Audit Manual)


About the audit report l.jpg

About the Audit Report

  • A text report of about 40 pages.

  • Based on an analysis of all the evidence.

  • Public (posted on OAC website).

  • Does not include a pass/fail or a grade.

  • Does include Recommendations, Affirmations and Commendations (but not every topic considered will result in a Rec, Aff or Com).

  • Will not include every issue that was touched on during the audit, but will attempt to provide a balanced view of the HEP.


Audit report table of contents l.jpg

Audit Report Table of Contents

  • Overview of Audit Process

  • Executive Summary of Findings

  • Summary List of Commendations

  • Summary List of Affirmations

  • Summary List of Recommendations

  • Chapters (mirroring the HEP’s Portfolio, but OAC could modify if required)

  • Appendix A: Audit Panel Members

  • Appendix B: Acronyms and Terms


Audit report drafts l.jpg

Audit Report Drafts


Commendations l.jpg

Commendations

  • The Panel is looking for evidence of good practice.

  • Where the Panel finds an example of an effective process resulting in positive results, this may lead to a Commendation.

  • Especially if it is unusual, recent, or something that the rest of the sector could benefit from knowing about.


Commendation examples l.jpg

Commendation Examples

Commendation #1

The Oman Accreditation Council commends Hogwarts College for developing a framework for performance indicators that is demonstrably aiding the management and planning processes.

Commendation #2

The Oman Accreditation Council commends Hogwarts College for successfully implementing a peer mentoring for academic staff system that has resulted in improved student satisfaction.


Affirmations l.jpg

Affirmations

  • Sometimes the Panel will find an OFI that the HEP has already identified and fully understood through its self review process AND which the HEP has made a clear commitment to addressing.

  • While this is still an OFI, it is also evidence of a self review system that works, and therefore should not be treated like other Recommendations.


Affirmation examples l.jpg

Affirmation Examples

Affirmation #1

The Oman Accreditation Council affirms that Hogwarts College has accurately identified, and is responding to, the need for a comprehensive risk management system.

Affirmation #2

The Oman Accreditation Council affirms that Hogwarts College has accurately identified, and is responding to, the need to redesign its research funding scheme in order to achieve desired results.


Recommendations l.jpg

Recommendations

  • The Panel found an opportunity for improvement (OFI) that the HEP either did not accurately identify, or to which it is not responding satisfactorily.

  • The Panel identifies the OFI, NOT the solution. Therefore, the Recommendation will state what needs to be done, but not how (at least, not in prescriptive detail).

  • Recommendations will not be prioritised but the Panel may add emphasis words like “strongly recommends” or urgency words like “immediately”.


Recommendation examples l.jpg

Recommendation Examples

Recommendation #1

The Oman Accreditation Council recommends that Hogwarts College implement a systematic approach to analysing and acting upon the feedback it receives from its student surveys.

Recommendation #2

The Oman Accreditation Council recommends that the Hogwarts College Council develop strategies to ensure it is able to inform and balance its fiduciary governance responsibilities with its academic governance responsibilities.


Hep feedback on draft v5 l.jpg

HEP Feedback on Draft v5

  • The HEP’s response to the draft Report is a vital part of the Quality Audit Process.

  • Opportunity for the HEP to give feedback on the Report before it is finalized.

  • The HEP can comment on:

    - factual inaccuracies;

    - unbalanced emphases or perspectives;

    - significant omissions; and

    - deviations from the agreed audit process.

  • The HEP must provide clear evidence in support of their claims (see Appendix R).


Approximate quality audit time line l.jpg

Approximate Quality Audit Time Line

A more detailed time line is provided in the Quality Audit Manual section 3.3


Slide56 l.jpg

Workshop Activity #1


Slide57 l.jpg

Workshop Activity

  • Form into groups.

  • Respond to the questions on the template, referring as appropriate to the Quality Audit Manual – but also drawing on your own thoughts.

  • One person will record the group’s responses.

  • One person will report back to the full workshop.

  • There are 20 minutes for this activity and 20 minutes for the feedback session


Slide58 l.jpg

Workshop Activity


Part d methods of analysis l.jpg

PART D:METHODS OF ANALYSIS


Self study vs external review l.jpg

Self Study vs External Review

Similar methods of analysis, with the following differences:


Methods of analysis issues l.jpg

Methods of Analysis Issues

  • Obtaining a General Overview of the HEP

  • Sampling

  • Types of Evidence & Data Analysis

  • Gaining a Comprehensive Picture

  • Gaining Confidence in the Evidence

  • Interview Methods

  • Reaching Conclusions

    To be followed by Group Workshops


General overview of hep l.jpg

General Overview of HEP

  • First section of the Portfolio.

  • Provides a shared view of the HEP to help contextualise the subsequent analyses.

  • Necessary for Self Study and External Review.

  • Includes:

    • a brief history

    • campus location(s)

    • general description of the HEP and its context (e.g. academic and general structure; list of the programs offered)

    • statistical information


Sampling 1 2 l.jpg

Sampling (1/2)

  • A HEP reviews all its major activities

  • An External Audit Panel uses sampling due to limited:

    - number of reviewers

    - amount of time

    - access to HEP

  • Two things are sampled: issues and evidence

  • The Audit Panel makes the sampling decisions.


Sampling 2 2 l.jpg

Sampling (2/2)

  • Sampled Issues:

    • All main headings (the 9 Scope topics) will be covered, but not every topic/issue under those headings

    • The Panel will seek to achieve an overall balanced account of the HEP

  • Sampled Evidence:

    • People to interview

    • Administrative & academic departments to investigate

    • Programs and courses to consider

    • Documents to consider


Types of evidence l.jpg

Types of Evidence

  • Conclusions in an Audit Report are based on consideration of evidence.

  • There are different types of evidence, each with its own methods of presentation & collection.

  • A notable distinction is between Quantitative and Qualitative evidence.

  • All types have something useful to offer in reaching a comprehensive conclusion.

  • Therefore, Quality Audit is a mixed method exercise.


Qualitative evidence l.jpg

Qualitative Evidence

  • Examples include:

    • Case Studies

    • Folk Lore (shared legends)

    • Lived Experiences

  • Methods of Collection include:

    • Interviews

    • Surveys (e.g. using open-ended questions)

    • Observations


Quantitative evidence l.jpg

Quantitative Evidence

  • Examples include:

    • Enrolment statistics

    • Financial results

    • Group opinions

    • Student results

  • Methods of Collection include:

    • Surveys (e.g. using Likert-type scales)

    • Documented reports

    • Records


Gaining a comprehensive picture l.jpg

Gaining a Comprehensive Picture

  • Saturation – a method used to explore an issue until no new information about it comes to light.

  • Triangulation – a method for strengthening the analysis using combination of:

    • Multiple original source of data (eg. students, staff, other stakeholders)

    • Multiple methods of data collection (e.g. surveys, interviews, literature)

    • Different types of data

  • Process Mapping – a method for depicting the steps in a process and their relationships.


Gaining confidence in the evidence l.jpg

Gaining Confidence in the Evidence

  • In reaching a conclusion, Panel Members must have confidence that the evidence is not only comprehensive, but also valid, reliable and honest!

  • Examples of methods for gaining confidence:

    • Chatham House Rule

    • Discourage ‘rehearsing’ for interviewees

    • Random Interviews

    • Non-attributable surveys (e.g. student evaluations of teaching)

    • Independence (of survey analysis, program reviews etc.)


Wet paint syndrome l.jpg

“Wet Paint” Syndrome

  • HEPs will usually seek to addressa range of problems before theAudit Visit occurs.

  • HEPs should acknowledge thenewness of any such improvements(rather than presenting the issue as being of long standing).

  • HEPs should not be embarrassed that an improvement is new.

  • OAC supports new improvement efforts – it is evidence of a quality management system in practice.


But there are limits l.jpg

But there are limits!!!


Interview session techniques l.jpg

Interview Session Techniques

Interviewers will use a range of questioning techniques in order to:

  • Explore how the documented evidence compares to the ‘lived experience’ of staff and students.

  • Seek the authoritative voice on a topic.

  • Seek validation from non-authoritative sources.

  • Determine how widely known an issue/policy etc. may be.

  • Saturate a topic.

  • Triangulate a topic.


Reaching conclusions l.jpg

Reaching Conclusions

  • Conclusions should be based on a comprehensive analysis considering different types of evidence.

  • They represent the judgment of the team involved – not individual members.

  • Each chapter in a Portfolio should conclude with Areas of Strength and Opportunities for Improvement.

  • Conclusions in the Audit Report are presented in the form of Commendations, Affirmations and Recommendations.


Slide74 l.jpg

Workshop Activity #2


Slide75 l.jpg

Workshop Activity

  • The following scenarios are adapted from HEP self-study reports (in Oman and abroad).

  • Form into groups.

  • Each group analyses one of the scenarios and answers the questions on the following slide.

  • One person will record the group’s responses.

  • One person will report back to the full workshop.

  • There are 20 minutes for this activity and 20 minutes for the feedback session


Slide76 l.jpg

Workshop Activity

  • What type of evidence is provided?

  • Is this (a) appropriate and (b) comprehensivein relation to the objective?

  • What are the (a) advantages & (b) limitations of this type of evidence in relation to the objective?

  • How would you check on the accuracy / validity / reliability of this evidence? (How does this differ between internal and external reviews?)

  • What further kinds of evidence would be required? (Triangulation)


Slide77 l.jpg

Workshop Activity

  • Teaching & Learning

  • Student Services

  • The Library

  • Information Technology

  • Graduate Employability

  • Community Services

  • Staff Development & Communication

    End


Slide78 l.jpg

Workshop #1 ScenarioTeaching and Learning

“The University aims to provide outstanding quality teaching and learning. Teaching and course evaluation questionnaires (TEVALs – see SM018) are administered for all subjects in all semesters, and the results are favourable. In each of the nine semesters in 2001 to 2003, teacher effectiveness was rated at 6 or 7 on a 7-point Likert-type scale by 60% of students (see 2006 Semester 2 Report: SM019). TEVALs are administered early for visiting staff, so they get feedback from the students while there is time to adjust their course, if necessary.”


Slide79 l.jpg

Workshop #1 ScenarioTeaching and Learning – Model Answers 1&2

  • What type of evidence is provided?

    • Quantitative (psychometric) statistics

  • (a) Is this appropriate in relation to the objective?

    • Yes, it is appropriate for teaching

      (b) Is this comprehensive in relation to the objective?

    • Not comprehensive in respect of teaching

    • Nothing mentioned about student learning


Slide80 l.jpg

Workshop #1 ScenarioTeaching and Learning – Model Answers 3a

  • (a) What are the advantages of this type of evidence in relation to the objective?

    • Internationally accepted method (generally – although the details of student evaluation systems can vary a lot in ways which may strengthen or undermine the integrity of the system)

    • Obtains feedback directly from the primary stakeholder (the students)

    • Benchmarks are obtainable.


Slide81 l.jpg

Workshop #1 ScenarioTeaching and Learning – Model Answers 3b

  • (b) What are the limitations of this type of evidence in relation to the objective?

    • Feedback only from students, who are not equipped to comment expertly on all aspects of teaching

    • Likert-type scale data reveals issues but not causes (need qualitative responses for that)

    • Aggregates of Likert-type data can be misleading and misinterpreted (e.g. the international literature shows significant differences between certain disciplines and extreme class sizes).


Slide82 l.jpg

Workshop #1 ScenarioTeaching and Learning – Model Answers 4

  • How would you check on the accuracy / validity / reliability of this evidence?

    • Quality controls in the statistical program

    • Process samples as a quality control

    • The HEP could commission an independent, expert review of the TEVAL policy and system

    • The Panel could review the TEVAL policy and system directly

    • The Panel could interview TEVAL staff to explore data management issues

    • The HEP/Panel could interview staff to gauge the acceptance of the system by the staff.


Slide83 l.jpg

Workshop #1 ScenarioTeaching and Learning – Model Answers 5

  • What further kinds of evidence would be required? (Triangulation)

    • Qualitative responses from students

    • Peer review

    • Teacher self reflection

    • Investment in professional development

    • Benchmarked student learning outcomes


Slide84 l.jpg

Workshop #2 ScenarioStudent Services

“The University aims to provide the same or comparable services to its distance education and its on-campus students. For example, telephone counselling is available to external students, and secure video conferencing for counselling is available from the Tartan campus. Another example is that learning assistance staff offer comment on assignments and assistance by phone, e-mail or post. Recent student satisfaction survey data indicates that over 50% of students rated the quality of student services as ‘high’ or ‘very high’, 4 or 5 on a 5-point scale.”


Slide85 l.jpg

Workshop #3 ScenarioThe Library

“The Library aims to provide a standard of excellence in meeting the information needs of clients. The Library has an informative Website that provides a gateway into its resources and services www.xxx.edu.om/library/. A library guide is mailed to all external students to inform them of the services provided. The Library has several methods for seeking feedback from students and staff. In order to maximize the relevance of the library collection, the Library works in close consultation with academic staff. Feedback from clients, informal and formal, indicates a high level of satisfaction with the Library’s service provision. The national benchmarking survey identified the Library’s highest performing areas and noted a major challenge in the size of the Library collection and resources.”


Slide86 l.jpg

Workshop #4 ScenarioInformation Technology

“To provide contemporary IT facilities that are stable and supportive of teaching and learning activities, and to provide an effective IT working environment and support for staff in their day-to-day activities, IT has implemented a strong policy framework. This includes a number of key policies developed to support service provision and covering areas such as conditions of use, network security, email, web hosting and publishing and associated guidelines and standards. There is an IT and telecommunications services catalogue, which describes the services and facilities provided. The University’s Helpdesk system escalates, alerts and tracks all requests, and is also used for quality monitoring and improvement. The external Hunter Review found widespread support for the services provided, and a high level of regard by clients generally. A University-wide survey of students was conducted in 2002 and demonstrated that on average, 60% of students rated IT services at 4 or 5 on a 5-point scale.”


Slide87 l.jpg

Workshop #5 ScenarioGraduate Employability

“The college aims to qualify graduates to be highly employable in their chosen professional areas. The academic departments across the college are engaged in innovative efforts aimed at increasing rigor and enriching the curriculum to ensure a comprehensive undergraduate experience and effective transition into either a post graduate program or into the workplace. To that end, the college has included excellent training and graduation projects in the study plans in all programs. In addition to enhancing the quality of students’ academic endeavors, this process promotes confidence and set up habits of reflection that will permit increasingly sophisticated career exploration over time.”


Slide88 l.jpg

Workshop #6 ScenarioCommunity Services

“The college aims to ensure that its community service is of high quality, reflects and enhances the distinctive role of the college in the higher education. Most of the faculty members contribute to society service through their participation in training seminars in multiple fields. The college shows a great interest in the academic research carried out by the staff members and published in specialized specific magazines. In addition, staff members are engaged in supervising and assessing academic researches for parties outside the college. The college also provides the local society with scientific books authored by the staff members. Furthermore, the college publishes a great number of educational articles in the cultural local or international magazines.”


Slide89 l.jpg

Workshop #7 ScenarioStaff Development & Communication

“The university aims to develop a whole-of-enterprise approach to staff development and communication. As one example, TechNet (see TechNet ToR; SM035) is an initiative of the technical staff of the university run by a volunteer committee. The network provides a means of communication for the technical community, facilitates resource sharing and seeks to improve the profile and development of technical staff within the university. The TechNet website is a focal point for all technical staff in the university. Chemicals, equipment and resources no longer needed are redistributed via the Technet email forum, saving $34k in 2006 on additional purchases (see TechNet Financial Benefits Report: SM036).”


Part e appendices l.jpg

PART E:APPENDICES


What next l.jpg

WHAT NEXT?


Next steps l.jpg

Next Steps

  • The Manual will be revised in light of the feedback received (by end of October) and approved by the end of the year.

  • Two pilot Quality Audits are currently underway.

  • The Manual may be amended after the pilot audits have been completed (mid 2008).

  • External Reviewers are currently in the process of being selected. A two day training program will be held for External Reviewers in November.

  • The National Quality Audit Schedule is being arranged now and will be published by the end of the year.


Hep quality audit manual draft v193 l.jpg

HEP Quality Audit Manual (draft v1)

Thank you and congratulationson completing this training module.


  • Login