Evaluator orientation
Download
1 / 72

EVALUATOR ORIENTATION - PowerPoint PPT Presentation


  • 129 Views
  • Uploaded on

EVALUATOR ORIENTATION. Serving on Off-Site and On-Site Committees OVERVIEW. TYPES OF COMMITTEES. Reaffirmation Committee (Off-Site Review Committee or On-Site Review Committee) Substantive Change Committee Special Committee Candidacy Committee Accreditation Committee.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'EVALUATOR ORIENTATION' - lev


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Evaluator orientation

EVALUATOR ORIENTATION

Serving on Off-Site and On-Site Committees

OVERVIEW


Types of committees
TYPES OF COMMITTEES

  • Reaffirmation Committee (Off-Site Review Committee or On-Site Review Committee)

  • Substantive Change Committee

  • Special Committee

  • Candidacy Committee

  • Accreditation Committee


Invitations to serve as an evaluator
INVITATIONS TO SERVE AS AN EVALUATOR

  • Invitations Come From Commission Staff Member

  • Usually Invitations Are By E-Mail

  • Reply Promptly And Check For Conflict of Interest Because Declines Must Be Replaced As Quickly As Possible

  • Evaluators Are Selected from Evaluator Registry


Evaluator registry
EVALUATOR REGISTRY

  • Arranged by level, type institution, category

  • Contains records of visits and ratings by chair and staff

  • Contains relevant comments concerning performance of committee members


Helpful information
Helpful Information

  • Handbook for Peer Evaluators

  • Modules which can be accessed at www.sacscoctraining.org

  • COC Staff and Committee Chairs


Logistics of visits
Logistics of Visits

  • Staff Member Sends Invitations

  • Staff Sends Packets with Forms, Handbooks, Principles, Travel Information, Information Outline Form

  • Committee Member Arranges Travel and Submits Expense Voucher After the Visit for Expenses


Logistics
Logistics……..

  • Institution (for On-Site Review Committees and other types of committees) or Commission Staff (for Off-Site Review Committees meeting in Atlanta) Arranges for Accommodations and Provides Information Concerning Transportation To and From Airport

  • Institution (for On-Site and others) Or Staff (for Off-Site) Seeks Information on Dietary Needs and Computer Needs (for On-Site)

  • Staff, Chair, and Institution Communicate with Evaluators


Qualities of an effective committee member
QUALITIES OF AN EFFECTIVE COMMITTEE MEMBER

  • Responds Promptly to Invitations

  • Within Reason, for On-Site Review Is Considerate of Institution in Making Travel Plans and Providing Information

  • Reviews Materials Thoroughly Before the Visit


Qualities
Qualities…….

  • On Time for Meetings, Transportation

  • Has an Understanding of Best Practices in Higher Education in His/Her Area of Expertise

  • Uses Professional Judgment in Applying Best Practices


Qualities1
Qualities…..

  • Is Fair and Objective In Evaluation

  • Is Willing To Make Difficult Decisions

  • Works with Other Committee Members in Reaching Consensus about Issues

  • Submits Expense Voucher Immediately after Visit with Appropriate Receipts


Commonly used materials
COMMONLY USED MATERIALS

  • Principles of Accreditation

  • Handbook for Peer Evaluators

  • Training Modules

  • Review and Writing Assignments

  • Tentative Agenda

  • Materials from Institution(s)

  • Materials from Commission Staff


The review process for reaffirmation of accreditation overview
The Review Process for Reaffirmation of AccreditationOVERVIEW

  • Review of Compliance by Off-Site Review Committee

  • Report of Off-Site Review Committee Shared with Institution


The review process
The Review Process…..

  • Review by On-Site Committeeof Quality Enhancement Plan and Compliance Issues (as identified by Off-Site Review Committeeor which arise during On-Sitevisit)

  • Report Completed by On-Site Review CommitteeSent to Institution and Commission as Basis on which Action Is Taken


Evaluator orientation1

EVALUATOR ORIENTATION

The Off-Site Review



Off site review committee 7 9 members

Chair

Finance Evaluator

Institutional Effectiveness Evaluator

Organization and Administration Evaluator

Student Support Evaluator

Learning Support Evaluator

Educational Program Evaluators (Number Varies Depending on Complexity of Cluster Institutions)

OFF-SITE REVIEW COMMITTEE7-9 MEMBERS


Off site review committee tasks phase i
OFF-SITE REVIEW COMMITTEETASKS Phase I

  • Phase I:

    Each Committee Member Receives Materials and Reviews Compliance in Assigned Area(s) for Each of the Three or Four Institutions in the Cluster(approximately six to eight weeks before meeting in Atlanta)

    Each Committee Member Completes a Draft Report for His/Her Review Assignments for Each Institution in the Cluster and Sends It to the Chair


Phase ii
Phase II

  • Phase II:

    Committee Meets in Atlanta for Parts of Three Days to Review, Discuss and Determine Compliance for Each Institution in the Cluster

    Committee Completes a Report for Each Institution in the Cluster


What are clusters
WHAT ARE CLUSTERS?

  • A Group of Three or Four Institutions Sharing Similar Characteristics

  • Assigned to an Off-Site ReviewCommittee for Review (Called a Cluster Committee)


The off site review purpose
THE OFF-SITE REVIEWPurpose

  • Provide Assessment of Compliance

    • Core Requirements 1-11

    • Comprehensive Standards and Federal Requirements

  • Provide Advice To Staff as Appropriate Concerning Make-Up of On-Site Committee


The off site review process key players
THE OFF-SITE REVIEWPROCESS“Key Players”

  • Institution

  • Commission Staff

  • Off-Site Review Committee


Role of institution
Role of Institution

  • Conducts Compliance Review

  • Completes Compliance Certification

  • Submits Compliance Certification and Materials Demonstrating Compliance To Off-Site Review Committee Members (Hard Copy or Transmitted Electronically)


Materials sent by institution
MATERIALS SENT BY INSTITUTION

HARD COPY

Compliance Certification

Summary Form (with identification of technical support person if some of materials have been sent electronically)

Catalog

Financial Audit and Management Letter and Financial Aid Audit


Materials sent by institution1
Materials Sent by Institution…

HARD COPY OR ELECTRONICALLY TRANSMITTED

Institutional Effectiveness Materials

By-Laws, Minutes, Make-Up of Board

Faculty Roster Denoting Qualifications

Policy Manuals

Other Materials Demonstrating Compliance


Role of commission staff
Role of Commission Staff

  • Select and Invite Chair and Members of Off-Site Review Committee for Each Cluster

  • Send Roster and Pertinent Information to Chair and Committee Members

  • Send Roster of Off-Site Review Committee to Institutions in Cluster


Role of staff
Role of Staff……..

  • Make Arrangements for Meeting in Atlanta

  • Communicate with Chair and Committee Members and Institutions in the Cluster

  • Provide Support during Meeting in Atlanta


Materials sent to chair and committee members by staff
Materials Sent to Chair and Committee Members by Staff

  • Roster of Off-Site Review Committee

  • Handbook for Peer Evaluators

  • Principles of Accreditation

  • Blank Report of Report of the Reaffirmation Committee


Materials sent by staff
Materials Sent by Staff…..

  • Review Assignments for Each Committee Member

  • Review Schedule for Institutions in Cluster

  • Information Concerning Travel and Housing for the Meeting in Atlanta

  • Expense Vouchers


The off site review prior to the meeting in atlanta phase i
THE OFF-SITE REVIEW PRIOR TO THE MEETING IN ATLANTA (PHASE I)

  • Review the Principles of Accreditation

  • Review the Handbook for Review Committees

  • Review List of Institutions in the Cluster

  • Contact Staff IMMEDIATELY if you discover a conflict of interest with any of the institutions in the Cluster


Check the List of Materials Sent by the Institution and by Staff

Call the Staff Member if an Item Is Missing

Call the Technical Support Person at the Institution If You Cannot Access Materials Sent Electronically

Do Not Otherwise Contact the Institution

Familiarize Yourself With the Institution


Tasks
TASKS Staff

  • Scan the Compliance Certification Document

  • Review Your Assignment (For Institutional Effectiveness Evaluators, Primarily Core Requirements 4 and 5 and Section 3).

  • Review the Institution’s Compliance in the Area(s) of Your Assignment

  • Communicate with Other Committee Members and the Chair as Appropriate


Tasks1
Tasks Staff

  • Complete a Draft Report Reflecting Preliminary Assessment and Send It to the Chair, Other Committee Members, and Staff

  • Chair Will Compile Drafts Into One Draft Report and Will Send It To Committee Members and Staff


Assessing compliance institutional effectiveness
Assessing Compliance StaffInstitutional Effectiveness

  • Familiarize yourself with mission of the institution

  • Familiarize yourself with size of institutions and programs offered

  • Review institutional effectiveness documents provided by institution


Assessing compliance
Assessing Compliance…….. Staff

  • Read the narrative provided by the institution describing its process

  • Determine who is responsible for the process at the institution

  • Determine whether there is a timeline provided by which process functions


Assessing compliance1
Assessing compliance…… Staff

  • Is the process ongoing?

  • Is there evidence that it functions? What evidence might signal a functioning process?

  • Is the process integrated? Is there evidence that all areas of the institution are engaged in planning and evaluation?


Assessing compliance2
Assessing compliance…… Staff

  • Is there evidence that the process is research-based?

  • Does it appear that meaningful goals and objectives are set by various units?

  • Are there various types of assessment means used?

  • Is there evidence that results of assessment are used for improvement?


Assessing compliance3
Assessing compliance…… Staff

  • Are there representative samples from various units indicating that student learning outcomes have been identified and that they are assessed on a regular basis?

  • Is there evidence of improvement in the various units?


Communicate with other committee members
COMMUNICATE WITH OTHER COMMITTEE MEMBERS Staff

  • Throughout the process of off-site evaluation, it is crucial that Committee members communicate with each other.

  • Communication is especially important for the institutional effectiveness evaluator.

  • The roster of Committee members will contain e-mail addresses for assistance in communication.


Communicate
Communicate… Staff

  • Committee members should monitor e-mail frequently during the time of review and respond as quickly as possible.

  • Staff and Chair should be copied on all e-mail so that they can render assistance and be aware of concerns.


Write a draft report
WRITE A DRAFT REPORT Staff

  • For each institution in the Cluster, mark Compliance, Non-Compliance, or Did Not Review for each of your assigned Core Requirement(s), Comprehensive Standard(s), and/or Federal Requirements.

  • Write narrative explaining specifically why you made the determination of compliance or non-compliance


Draft
Draft……… Staff

  • Make suggestions to staff as appropriate concerning the On-Site Review Committee

  • Send your draft electronically to the Chair and other Committee members and Staff


Completing the off site review in atlanta phase ii
COMPLETING THE OFF-SITE REVIEW IN ATLANTA (Phase II) Staff

  • Orientation Meeting of Committee

  • The Committee, Led by the Chair and With Discussion by All Committee Members, Determines Compliance of Each Institution in the Cluster

  • The Committee Produces a Report for Each Institution in the Cluster


Completing the review in atlanta
Completing the Review in Atlanta… Staff

  • The Committee Agrees on All Areas Marked Compliance or Non-Compliance and Agrees on Narrative Supporting Each (Both Compliance and Non-Compliance)


The final report of the off site committee
THE FINAL REPORT OF THE OFF-SITE COMMITTEE Staff

  • Compliance: The Committee has determined that the institution has presented acceptable narrative and documentation demonstrating compliance.

  • Non-Compliance: The Committee has determined, after review of the narrative and documentation presented by the institution, that the institution is not in compliance.


Completion of work
Completion of Work Staff

  • The Committee’s Work Is Finished After the Report Is Completed

  • The Committee Has No Contact with Institutions Following Off-Site Review


A special note confidentiality
A Special Note StaffCONFIDENTIALITY

  • Maintain Confidentiality At All Times

  • Discuss the Institutions in the Cluster ONLY with Other Committee Members, Chair of the Committee, and Commission Staff

  • Do Not Contact the Institution (except tech support) Before, During, or After the Review Activities


Challenges of the off site review
CHALLENGES OF THE OFF-SITE REVIEW Staff

  • Difficulty in assessing compliance only on paper

  • Sometimes problems in accessing materials (especially electronically)

  • Amount of time needed to review three or four institutions


Evaluator orientation2

EVALUATOR ORIENTATION Staff

The On-Site Visit


The on site review
THE ON-SITE REVIEW Staff

  • The Make-Up of the On-Site Committee

  • The Tasks of the On-Site Committee

  • Preparing for the Visit

  • Conducting the Visit

    Reviewing Compliance

    Reviewing the QEP

    Writing the Report


Make up of the on site committee
Make-Up of the On-Site Committee Staff

  • Chair

  • CEO

  • Institutional Effectiveness

  • Educational Programs and Faculty

  • Learning or Student Support Services

  • Quality Enhancement Plan Evaluator

  • Others as Necessary


Tasks of on site committee
TASKS OF ON-SITE COMMITTEE Staff

  • Review of Quality Enhancement Plan (Core Requirement 12)

  • Review Areas of Non-Compliance Cited by Off-Site Committee

  • Consider Any Other Areas of Non-Compliance which Come to its Attention

  • Refine or Revise Narrative Provided by Off-Site Committee for Compliance

  • Write a Report To Be Sent to Committees on Compliance and Reports

  • Conduct an Exit Conference


Preparing for visit
PREPARING FOR VISIT Staff

  • Make Arrangements (Communicate with Staff, Chair, Institution as Appropriate)

    • Travel—Arrive by first meeting time

    • See Handbook concerning preparations

  • Review Materials Thoroughly

    Off-Site Committee Report

    Institution’s Focused Report

    Quality Enhancement Plan

    Review/Writing Assignments


Preparing for visit1
Preparing for Visit…… Staff

  • Determine Interviews Desired and Communicate to Chair

  • Make Notes and Be Prepared To Discuss Compliance Issues and QEP Observations during a Conference Call (if the chair arranges one) and at the Orientation Meeting of the Committee


Conducting the visit
CONDUCTING THE VISIT Staff

  • Visit Takes Place Over Parts of Three Days

  • Review Compliance Issues (Arrive at Consensus Regarding Recommendations)

  • Review Quality Enhancement Plan (Determine Acceptability Consistent With Core Requirement 12)


Conducting the visit1
Conducting the Visit………. Staff

  • Complete Report (Indicate Compliance or Non-Compliance and Write Narrative as Appropriate)

  • QEP recommendations are placed under Core Requirement 12.

  • QEP supporting narrative is under Part III of the report

  • Conduct an Exit Conference

    Discussion of Findings (Staff, Chair, On-Site Committee, Institution’s Leadership Team)


Compliance issues
COMPLIANCE ISSUES Staff

  • Consider Best Practices and Apply Professional Judgment in Determining Compliance

  • Draft Recommendations for Areas Where the Institution Appears To Be Out of Compliance

  • Draft Narrative Explaining Specifically Why There Is Non-Compliance

  • Reach Consensus with Entire Committee Concerning Compliance or Non-Compliance, the Wording of Recommendations, and the Narrative for All Requirements and Standards


Compliance issues1
Compliance Issues…. Staff

  • Compliance Issues Not Cited by the Off-Site Committee May Come To the Attention of the On-Site Committee


Evaluating the acceptability of the qep
EVALUATING THE ACCEPTABILITY OF THE QEP Staff

  • Part of the Planning and Evaluation Process?

  • Focused (significant and manageable)

  • Institutional Capacity

  • Institutional Assessment Provisions

  • Broad-Based

  • Linked to Student Learning Outcomes

  • Provide Input to Two Quality Enhancement Evaluators


Summary tasks of on site evaluator
SUMMARY StaffTASKS OF ON-SITE EVALUATOR

  • Prior to visit reviews Off-Site Report, Focused Report of Institution, and QEP

  • During visit reviews any compliance issues

    Reviews documents

    Interviews as necessary

    During visit assists in reviewing QEP

    Writes for assigned areas (recommendations and/or narrative)


A special note confidentiality1
A Special Note StaffCONFIDENTIALITY

  • Maintain Confidentiality At All Times

  • Discuss the Institution ONLY with Other Committee Members, Chair of the Committee, and Commission Staff

  • Do Not Contact the Institution After the Review Activities


Evaluator orientation3

EVALUATOR ORIENTATION Staff

Using Professional Judgment and Applying Best Practices


Evaluating institutions
Evaluating Institutions Staff

  • An Evaluator

    • must consider what are best practices in any given area

    • must carefully consider the institution’s “case” that it complies with a requirement or standard

    • must apply professional judgment in determining compliance


Exercising professional judgment in determining compliance
EXERCISING PROFESSIONAL JUDGMENT IN DETERMINING COMPLIANCE Staff

Exercised by peers during evaluation

Based on your study in higher education

Based on your experiences

Based on your expertise in dealing with particular issues in higher education


Professional judgment
Professional Judgment……. Staff

Based on your understanding and application of Principles of Accreditation

Based on your evaluation of the soundness of the institution’s analysis relating to compliance

Based on your evaluation of the cogency of the evidence presented by the institution




Evaluating the case for compliance
Evaluating the “Case” for Compliance Staff

Reviewers ascertain

accuracy and adequacy of summaries and interpretations of the data to which they refer and whether the narratives and the data combined support the institution’s claim of compliance.

As a reviewer on-site, you may wish to examine documents and identify a group of persons whom you wish to interview to gain additional insight.


Conducting the review
Conducting the Review Staff

  • For Committees reviewing a cluster of institutions during an Off-Site Review, each institution should be reviewed separately, taking into consideration the mission of the institution.

  • Avoid “comparative compliance,” i.e., determining compliance based on whether an institution is “at least more in compliance than another one.” Avoid comparing the institution to your institution.


Conducting the review1
Conducting the Review…. Staff

  • On-Site evaluators should likewise avoid “comparative compliance.”

  • Remember that it is the responsibility of the institution to establish compliance. It is the Committee’s responsibility to determine whether the institution has established compliance or not.


Conducting the review2
Conducting the Review….. Staff

  • Look first at the “case” the institution makes for compliance.

  • Then assess whether you think the institution’s “case” and the documents which it provides in support of its “case” are sufficient.


Conducting the review3
Conducting the Review…. Staff

  • Review components of your assignment which can be quickly determined

  • Then review components of your assignment which may require discussion with other Committee members and more in-depth analysis


Peer evaluation
PEER EVALUATION Staff

  • Integrity of Accreditation Depends on the Willingness of Peer Evaluators to Make Fair, Well-Thought-Out, Objective, and Sometimes Difficult Decisions

  • Decisions about Compliance Are Made To Assist Institutions in Improving the Quality of Their Operations and Services


ad