Data Overview Iowa Behavioral Alliance PBS 2007 Conference October, 2007

1. Iowa Department of Education Data OverviewIowa Behavioral AlliancePBS 2007 Conference � October, 2007 Robin Galloway, PhD, Research Institute for Studies in Education Sonmez Pamuk, PhD, Research Institute for Studies in Education

2. Research Institute for Studies in Education Presentation Organization Evaluation of PBS state-wide efforts How were data analyzed? Are schools in Iowa interested in adopting PBS? Are participating schools implementing PBS with fidelity? Does PBS impact social and behavioral adjustment? Does PBS impact academic performance? Does PBS impact school culture and climate?

3. Research Institute for Studies in Education: Demographics Source. PBS Demographic Database, FFY 2006.

4. Research Institute for Studies in Education Demographics Figure 1. Percent of Population Free and Reduced Lunch: PBS Sites and the State

5. Research Institute for Studies in Education Demographics Figure 2. Iowa PBS Site Overall Percent Minority Population Compared with State Minority Population.

6. Research Institute for Studies in Education How were data analyzed? By Cohort Year, Implementation Year, and across graduation, dropout, achievement, Iowa Youth Survey, ODRs, suspension and expulsion, and instructional time saved�..

7. Research Institute for Studies in Education How were data analyzed? PBS and Matched sites Enrollment Location Level of �at-riskness� Free-reduced lunch Dropout Graduation Ethnicity Pregnancy Overall Poverty

8. Research Institute for Studies in Education Are schools in Iowa interested in Adopting PBS? Figure 3. Number of Participants Attending Iowa Behavioral Alliance Professional Development.

9. Research Institute for Studies in Education So, are schools in Iowa interested in adopting PBS? Participation has increased more than 10-fold 8 original demonstration sites to the current 90 school sites. AEA 11 � 40 Sites in 2006-2007 Des Moines � 11 Sites in 2006-2007 Total of 48,049 students impacted in 2006-2007 school year By and large, schools who adopt PBS have demographics that are significantly different from the state average demographics 25% more students qualify for free and reduced lunch than the state 50% more students who represent a minority population. Demographics may influence impact on key outcomes compared to other sites

10. Research Institute for Studies in Education Are Participating Schools Implementing PBS with Fidelity? Figure 4. Percent Fidelity as Measured by SET Mean Scores Across Implementation Years.

11. Are Participating Schools Implementing PBS with Fidelity? Research Institute for Studies in Education

12. Are Participating Schools Implementing PBS with Fidelity? Research Institute for Studies in Education

13. Research Institute for Studies in Education So, are participating schools implementing PBS with fidelity? Data indicate that schools are implementing PBS with fidelity with treatment fidelity scores well above 80% in years 2 and 3. First-year treatment fidelity scores are somewhat lower, 73% overall. Development of a PBS team, Discipline policy, School-wide Information Systems (SWIS) training, and Behavior expectation curriculum. First-year implementation scores below 80% are also reflected in the literature on Positive Behavioral Supports. Correction Systems category � slow to implement

14. Research Institute for Studies in Education Does PBS impact social and behavioral adjustment? To determine impact, the following measures were analyzed by Cohort, Implementation, SET scores, and achievement between PBS and matched sites: Office Discipline Referrals, Graduation Rates, Dropout Rates, and Suspension and Expulsion Rates Office Discipline Referrals Major ODRs per school per 100 students per day National averages Elementary Schools: .43 ODR/100/Day Middle Schools: .95 ODR/100/Day High Schools: .99 ODR/100/Day

15. Research Institute for Studies in Education OFFICE DISCIPLINE REFERRALS Figure 5. ODRs/100 Students/Day for PBS schools by Implementation Year. Through yr 2005-2006Through yr 2005-2006

16. OFFICE DISCIPLINE REFERRALS Research Institute for Studies in Education Cohort 1 (2) and 2 (6) are together in implementation year 4 Through yr 2006-2007Cohort 1 (2) and 2 (6) are together in implementation year 4 Through yr 2006-2007

17. Research Institute for Studies in Education GRADUATION RATE Figure 6. Average Graduation Rate for PBS Sites, Matched Sites and the State.

18. Research Institute for Studies in Education DROPOUT Figure 7. Dropout Rate for PBS Sites, Matched Sites and the State.

19. Research Institute for Studies in Education DROPOUT Figure 8. Dropout Rate for PBS and Matched Sites by Implementation Year for Students with IEPs.

20. Research Institute for Studies in Education SUSPENSIONS/EXPULSIONS Figure 9. Suspension and Expulsion Rate for PBS Sites, Matched Sites and the State for All Students.

21. Research Institute for Studies in Education SUSPENSIONS/EXPULSIONS Figure 10. Suspension and Expulsion Rate for PBS Sites, Matched Sites and the State for Students with IEPs

22. Research Institute for Studies in Education So, does PBS impact social and behavioral adjustment? Data indicate PBS sites experienced: Reduction in ODRs and dropout rates over time A greater reduction in suspension/expulsion rates for all students and for students with IEPs when compare to matched sites and the state. Increased graduation rates for all students as compared to matched sites and the state No statistically significant change over time in ODRs, graduation, dropout, or suspension/expulsion rates

23. Research Institute for Studies in Education Does PBS impact academic performance? To determine impact, the following measures were analyzed by Cohort, Implementation, SET scores, and social/behavioral measures: Percent proficiency in reading across grades 4, 8 and 11; Percent proficiency in mathematics across grades 4, 8 and 11; and SET fidelity mean scores impact on percent proficiency in reading and mathematics

24. Research Institute for Studies in Education Academic performance: 4TH GRADE Figure 11. Percent Proficiency in 4th Grade Mathematics and Reading for PBS Sites and the State.

25. Research Institute for Studies in Education Academic performance: 8TH GRADE Figure 12. Percent Proficiency in 8h Grade Mathematics and Reading for PBS Sites and the State.

26. Research Institute for Studies in Education Academic performance: 11TH GRADE Figure 13. Percent Proficiency in 11h Grade Mathematics and Reading for PBS Sites and the State.

27. Research Institute for Studies in Education So, does PBS impact academic performance? Data indicate PBS sites experienced: An increase in percent proficiency in mathematics in grades 4 and 8; An increase in reading proficiency in grades 4, 8 and 11; No statistically significant change over time in percent proficiency in reading or mathematics

28. Research Institute for Studies in Education Does PBS impact school culture and climate? To determine impact, the following measures were analyzed: Increase in instructional time; 10 sites cohorts 1 - 4 Focus group data; and Specific factors in the Iowa Youth Survey Instructional time Administrator and Instruction Time Gain

29. Research Institute for Studies in Education Does PBS impact school culture and climate? Student Focus Group Data � 2006-2007 school year Strengths: were aware of the PBS program and how it worked saw their learning environment as positive reported rewards were used to acknowledge positive behavior Needs Improvement: reported a problem with teacher inconsistency with implementation reported seeing negative teacher behavior � examples shared included favoritism, surliness and sarcasm suggested they saw no implementation of PBS in the community reported not being involved in decision-making activities reported that they did not think the rewards were improving behavior reported a problem with bullying

30. Does PBS impact school culture and climate? Adult Focus Group Data � 2006-2007 school year decrease in adult comments around behavior in the school with increases in overall comments by participants regarding drop out and mental health adult participants from 6 of the 6 groups made references to the PBS program as supportive of dealing with behavior issues at school perception of school support for dealing with behavior issues has remained fairly consistent over 4 years adults reported an increase in community support for students at-risk of dropping out of school adults reported concern over the existence of poor student behavior in the community Research Institute for Studies in Education

31. Research Institute for Studies in Education Does PBS impact school culture and climate? Iowa Youth Survey There are clear rules about what students can and cannot do The school principal and teachers consistently enforce school rules Many teachers care about me My teachers notice when I am doing a good job and let me know about it Students in my school treat each other with respect

32. Does PBS impact school culture and climate? Research Institute for Studies in Education

33. Does PBS impact school culture and climate? Research Institute for Studies in Education Cohorts 1, 3 and 4Cohorts 1, 3 and 4

34. Research Institute for Studies in Education So, does PBS impact school culture and climate? Data indicate that PBS sites experienced: A decrease in ODRs with corresponding increase in administrator and instructional time Decrease of 2076 ODRs/100/Day 692 hours (115 days) of administrator time gained 1,557 hours (260 days) of instructional time gained An increase in student strongly agree and agree response on five relevant Iowa Youth Survey questions over all grades and cohorts

35. Future Evaluation Activities IYS Questions PBS sites with Matched sites Teacher Focus Groups 6 PBS sites representing cohorts 1, 3 and 4 3 sites implementing with fidelity and decrease in ODRs 3 sites not implementing with fidelity and increase in ODRs Case Studies 1 PBS site implementing with fidelity and decrease in ODRs Research Institute for Studies in Education