Improving Accountability by Using Internal Quality Assurance and Peer Review

1. Improving Accountability by Using Internal Quality Assurance and Peer Review Presentation to the Canadian Evaluation Society/Société Canadienne D’ Évaluation Vancouver, B.C., June 4, 2003 Session 302 Presentation by Phyllis M.Thomas Mid-continent Research for Education and Learning (McREL)

2. Phyllis M. Thomas McREL 2550 South Parker Road, Suite 500 Aurora, CO, USA, 80014 (303) 632-5564 pthomas@mcrel.org

3. What is the Strategy? Internal Quality Assurance and External Peer Review

4. Primary Purpose and Use of these Strategies (in general): • Formative review processes for assuring the quality of a product before it is finalized. • QA holds the org. and staff accountable for quality. • Applies guidelines & commonly accepted professional standards to a draft product. • Identifies strengths & weaknesses for improvement. • Enhances integrity of an organization & public image. • Adds credibility with use of external/peer reviewers.

5. How Was This Strategy Used in REL Performance Indicators Work?  A combination of formative review strategies within and across organizations: • Internal QA review using common guidelines & forms. • External subgroup/peer review.  Overall responsibility/accountability for data and QA resides within each lab/with each evaluator … and with peers for aggregation of data.

6. How Was This Strategy Used in REL Performance Indicators Work? (cont’d)  Internal formative QA review conducted to assure data quality and document problems.  Each lab collected & reviewed its own regional data.  Applied guidelines & common standards to data/ processes.  Work plan & “Algebra problems” (example).  QA form based on govt. standards for quality (example).  Identified problems and areas for improvement.  Submitted regional data and QA form to cross-lab peer subgroup for review & aggregation across thesystem.

7. How Was This Strategy Used in REL Performance Indicators Work? (cont’d) • External/cross-lab peer subgroup review to assure data quality and document problems before aggregation.  Identified problems and areas for improvement (example).  Provided feedback/asked questions (example).  Opportunity for evaluator to revise data/resubmit (example).  Subgroup resolved issues/negotiated changes/made aggregation decisions (example). (Full indicators group reviewed aggregated data.)

8. Benefits, Limitations, and Lessons Learned • Benefits  Increased scrutiny of data & processes.  Review at multiple levels.  Shared responsibility & accountability.  Enhanced quality, accuracy, & consistency.  Enhanced credibility with external peers.  Helped create a culture of accountability.  Good preparation for the external audit.

9. Benefits, Limitations, and Lessons Learned (Cont’d) • Limitations • Additional resources required/increases costs. • Additional time required/accelerates the schedule. • Potential for omission & error. • External peer reviewers may miss the boat. • Potential for conflict with peers. • Benefits of review are limited by the inherent quality of the data. • Credibility still somewhat limited without external audit/review.

10. Benefits, Limitations, and Lessons Learned (Cont’d) • Lessons Learned • Costs vs. benefits of QA & external peer review. • Clear role definition & good working relationships. • Balancing need for accountability & collaboration.