Benchmarks of Quality Training

1. Benchmarks of Quality Training Kentucky Center for Instructional Discipline www.kycid.org

2. Training Objectives • Introduction to the Benchmarks of Quality (BoQ) • Understand how to complete, submit, and use the BoQ

3. Why is Evaluation of PBIS Implementation Important? • To examine the fidelity of implementation • “Are we really doing what we think we are doing?” • To document effectiveness of implementation • “Is what we’re doing working?” • To identify and examine strengths and weaknesses of implementation • Celebrate successes • Identify areas to improve

4. What We’ve Been Doing • Team Implementation Checklist (quarterly) • Effective Behavior Support survey (annually) • School-wide Evaluation Tool (annually)

5. What We’re Now Doing • Team Implementation Checklist (three times a year) • Effective Behavior Support survey (annually) • School-wide Evaluation Tool (annually) • Benchmarks of Quality (annually)

6. Why the BoQ and not the SET? • The BoQ provides deeper analysis • Self-assessment heightens awareness • Takes less time • SET scores may sometimes be inflated • SET may not reflect current activities

7. Benchmarks of Quality • Developed by Florida PBIS at the University of South Florida • Assesses development and implementation of school-wide PBIS • Lists 53 benchmarks of quality in school-wide PBIS programs within 10 critical elements • Completed by school teams annually to identify areas of strength and weakness • Used by state and districts to guide TA and training and evaluate outcomes related to level of implementation

8. Benchmarks of Quality • 53 items aligned with PBIS training process addressing areas of: • Faculty commitment • Effective procedures for dealing with discipline • Data entry and analysis plan • Expectations and lesson plans developed • School-wide recognition system established • Implementation plan • Crisis plan • Evaluation

9. BoQ Validation Process • Expert review • Pilot studies in Florida and Maryland • Reliability: Test-retest, inter-rater both >.01 • Concurrent Validity – SET/ODR’s • For more details see JPBI, Fall 2007

10. Completing the Benchmarks

11. Timeline • January – training for coaches on the BoQ • February – complete the BoQ assessment • By March 30 – submit results to your Area Coordinator

12. Benchmarks of QualityRating and Scoring Components • Team Member Rating Form • Completed by team members independently • Returned to PBS Coach • Scoring Form, Tally Sheet, and Max Scores per Element • Completed by Coach using Scoring Guide • Used for reporting back to team • Scoring Guide • Describes administration process • Rubric for scoring each item

13. Method of Completion • Coach • Uses Scoring Guide to ascertain the appropriate score for each item • Collects Team Member Rating forms • Resolves any discrepancies by reaching consensus with the team • Reports final findings to the team

14. Completion of BoQStep 1: Coach’s Scoring • The Coach uses his or her best judgment based on personal experience with the school and the descriptions and exemplars in the Benchmarks of Quality Scoring Guide toscore each of the 53 items on the Benchmarks of Quality Scoring Form. • Do not leave any items blank.

15. Step 2: Team Member Rating • The Coach will give the Benchmarks of Quality Team Member Rating Form to each PBIS Leadership Team member to be completed independently and returned to the Coach upon completion. • Team members should be instructed to rate each of the 53 items according to whether the component is In Place, Needs Improvement, or Not in Place. Team members place a check in the box to indicate their rating.

16. Step 2: Team Member Rating Note: Some of the items relate to product and process development, others to action items; in order to be rated as In Place, the item must be developed and implemented (where applicable).

17. Step 3: Scoring the BoQ • Coaches collect the Team Member Rating Forms and tally the team’s most frequent response using the Tally Sheet. • Record the most frequent response for each item on the Benchmarks of Quality Scoring Form in the Step 2 Column, using ++ for In Place, + for Needs Improvement, and – for Not In Place. • In the Step 3 Column, place a check in the box where any discrepancies in scoring occur between the team and coach ratings

18. Step 3: Scoring Guidelines • For items with a range of 0 to 1: • 1 = In Place • 0 = Needs Improvement or Not in Place • For items with a range of 0 to 2: • 2 = In Place • 1 = Needs Improvement • 0 = Not in Place • For items with a range of 0 to 3: • 3 = In Place • 2 or 1 = Needs Improvement • 0 = Not in Place

19. Step 4: Team Report • The Coach completes the Team Summary on p. 3 of the Benchmarks of Quality Scoring Form, recording areas of discrepancy, strength, and weakness.

20. Step 4: Team Report • Discrepancies - If there were any items for which the team’s most frequent rating varied from the Coach’s rating based upon the Scoring Guide, the descriptions and exemplars from the guide should be shared with the team. • If upon sharing areas of discrepancy, the Coach realizes that there is new information that according to the Scoring Guide would result in a different score, the item and the adjusted final score should be recorded on the Scoring Form.

21. Step 5: Reporting Back to Team • The coach reports back to the team using the Team Summary page of the Benchmarks of Quality: Scoring Form. • If needed, address items of discrepancy and adjust the score.

22. Step 5: Reporting Back to Team • The coach then leads the team through a discussion of the identified areas of strength (high ratings) and weakness (low ratings). • The team identifies celebrations to share with staff. • The team identifies area of focus and documents on the PBIS Tier 1/Universal Action Plan.

23. Step 6: Submitting Your BoQ • The Coach submits the Benchmarks of Quality: Scoring Form (both pages) and the Team Summary to his or her KYCID Area Coordinator. • BoQ results should be submitted by April 30.

24. Final Notes • District level PBIS team members or KYCID staff may validate BoQ findings using the PBIS Walkthrough or the SET. • In addition, your team can use the PBIS Walkthrough anytime throughout the year for a quick self-assessment

25. Questions or Assistance? • For additional information and support on completing and/or using the data from the BoQ, contact your Area Coordinator: -Judy Boggs, Central -Lea Brown, West -Kelly Davis, West -Janeah Gullett, East -Steve Hutton, North

26. For more information about the BoQ: