Out of School Time in America

1. Out of School Time in America International Conference Network on Extracurricular and Out-of-School Time Educational Research Denise Huang CRESST/UCLA November 23rd to 25th, 2010 Giessen, Germany

2. The Evolutions of Afterschool Programming for At-Risk Youths • Safe Haven • Build Resiliency • Opportunities for Enrichment • Closing the Gap • Academic Achievement

3. Theoretical Framework

4. ANewDayForLearning “A comprehensive , seamless approach to learning that values the distinct experiences that families, schools, after-school programs, and communities provide for children.” A Report from the Time, Learning, and Afterschool Task Force 2007

5. Community/Neighborhood School Afterschool Program Family Child The Ecological Model Afterschool programs are social organizationsembedded with cultural values coming from families, school, and neighborhood climate and norms.

6. Provisional Features of Positive Developmental Settings (Larson et. al) • Physical and Psychological Safety • Clear and consistent structure • Supportive relationships • Opportunities to belong • Positive social norms • Support for efficacy and mattering • Opportunities for skill building • Integration of family, school, and community efforts

7. Core Positive Youth Development Constructs Context Person Developmental Success Work Place Peers View of the Child Reduction in High-Risk Behaviors Family Congregation Community School Developmental Strengths Promotion of Health Well-being Thriving Programs Neighborhood Search Institute Insights & Evidence, November 2006

8. Leveraging Social Capitals in Afterschool Programs Internal Networks External Networks Innovation and Staff Development Organizational Capital Competence Relationships Attitudes Social Capital Trust Intellectual Capital Expectations & Social Norms Problem-solving & Agility Staff & Student Relationships Student Engagement

9. Theoretical Logic Model for the Afterschool Partnership Study Program Support Management Staff /Resources Periodical Evaluation for Continuous Improvement Align activities to goals Motivation Engagement Periodic Assessment of Student Performance After School Program Quality Content Practice Quality Opportunities to Practice (Attendance) Set Goals Linkage to School Day Content Aligned to Standards Research Based Practice & Strategies Content Process Structure

10. The CDE Indicator Model Student Engagement Evaluation System Satisfaction Monitoring Goals ProgramOrientation Academic Management CASHEE External Connections STAR School Attendance Program Environment ProgramClimate Positive Youth Development Relationships Setting Features Safety Instructional Features Expectation Resources Aspirations Alignment Staff Efficacy Professional Development

11. Evaluation

12. The Need of Evidences to Support Expansions • Do afterschool programs contribute to positive academic development? • Do afterschool programs contribute to positive youth development? • What aspects of the program functioning contribute to these positive outcomes?

13. Prevalence of Different Evaluation Research Process Evaluations: Formative studies • Evaluations that assesses the conduct of the program during the initial design and testing stages with the intent to improve the program Program monitoring • Systematic examination of program coverage and delivery- (target population, fidelity, efficiency) • Identifying successful implementation strategies for program diffusion

14. Outcome Evaluation Summative Evaluation • Summative evaluation provides information on the product's efficacy ( it's ability to do what it was designed to do) • By looking at the intervention group, the evaluator can examine the learning materials and learning process together with the outcomes-- hence the name Summative Evaluation. Impact Evaluation • Impact evaluation involves constructing a counterfactual • Random selection and isolation from interventions are seldom practicable and sometimes ethically difficult to defend. • Quasi-experimental method is often used.

15. Indicators for Program Effectiveness • Student attendance (in regular school and afterschool programs) • Performance measures (achievement tests, homework completion, classroom grades, language re-designation, school retention, future aspirations, etc.) • Non-cognitive measures (safety, attitudes towards school, relationships with adults, social competence, conflict resolution skills, self-esteem, and self-efficacy etc.) • Parent involvement • Professional development • Long-term effects (drop out rate, life satisfaction, etc)

16. Challenges in After School Studies • Diversity of program characteristics • Self-selectiveness -Consent forms • Comparison groups -Ethical issues • Transience • Availability of archived data sources • Meaningful outcome measures • Importance of dosage

17. Cognition & Achievement

18. Common Outcome Measures for Cognition and Achievement • Academic achievement scores • Attitudes towards school/learning • Development of study skills (time management, organization, memory, etc. ) • Development of academic enabler (self-efficacy, motivation, effort, etc.) • School dropout • Future aspirations

19. High Scope Afterschool Quality and Day School Outcomes 2007 Safe Environment Reading Achievement *** Program Attendance Positive Interaction Student Engagement Autonomy & Opportunities to reflect and practice School Suspensions

20. Relationship between Program Attendance and Youth Outcomes

21. ASES Program Attendance

22. ASSETs Program Attendance

23. Improved School Attendance and Engagement in Learning Elementary school students attending LA’s BEST afterschool program improved their regular school day attendance and reported higher aspirations regarding finishing school and going to college. Additionally, LA’s BEST participants are 20 percent less likely to drop out of school compared to matched nonparticipants. (UCLA National Center for Research on Evaluation, Standards and Student Testing, June 2000, December 2005 and September 2007)  A New Hampshire statewide study of students participating in academically focused afterschool programs, including those funded by the federal 21st Century Community Learning Centers Program (21st CCLC), found that more than half of regular attendees improved both behaviorally and academically. (RMC Research, 2005) Ninety-two percent of high exposure of Citizen Schools participants were promoted on time to the tenth grade compared to 81 percent of matched nonparticipants. This is critical, since earning promotion to tenth grade on time is a key predictor of high school graduation (i.e. preventing drop out). (Policy Studies Associates, Inc., December 2006)

24. Improved School Attendance and Engagement in Learning-High School Ninth grade students who formerly participated in The After-School Corporation (TASC) in middle schools had higher daily school attendance and credit accumulation than matched nonparticipants. (Policy Studies Associates, Inc., October 2007) Participants in the Breakthrough Collaborative program enroll in college-preparatory mathematics courses at double the national average, and are accepted to college-preparatory high schools by more than 80 percent. (Breakthrough Collaborative, 2006) High school students participating in Chicago's After School Matters program—which offers paid internships in the arts, technology, sports, and communications to teenagers in some of the city's most underserved schools—have higher class attendance, lower course failures and higher graduation rates than similar students who do not participate in the program. (University of Chicago, Chapin Hall Center for Children, 2007)

25. Improved Test Scores and Grades Annual performance report data from 21st CCLC grantees across the country demonstrate that students attending 21st CCLC programs improve their reading (43%) and math grades (42%). (Learning Point Associates, November 2007)  The Promising Afterschool Programs Study, a study of about 3,000 low-income, ethnically-diverse elementary and middle school students, found that those who regularly attended high-quality programs over two years demonstrated gains of up to 20 percentiles and 12 percentiles in standardized math test scores respectively, compared to their peers who were routinely unsupervised during the afterschool hours. (Policy Studies Associates, Inc., 2007)  Participants in North Carolina’s Young Scholars Program with at least 280 hours in the program averaged double-digit increases annually for proficiency in both math and reading. Promotion rates rose by 38 percent. Furthermore, the number of Young Scholars receiving A’s and B’s increased an average of 38 percent, while the number receiving F’s decreased an average of 50 percent. (Z Smith Reynolds Foundation, 2006)  Active participants in programs offered by The After-School Corporation (TASC) were more likely to take and pass the Regents Math Sequential 1 exam by ninth grade than were nonparticipants. Thirty-two percent of active ninth grade participants took and passed the exam, compared to one percent of ninth grade nonparticipants. Fifty-two percent of active participants took and passed the Math Sequential 2 and 3 exams, compared to 15 percent of nonparticipants in the same grades. (Policy Studies Associates, Inc., 2004)  Participants of St. Paul Minnesota’s 21st CCLC Pathways to Progress program received better grades in English and math than nonparticipants. (University of Minnesota, Center for Applied Research and Educational Improvement, March 2004)

26. Keeping Kids Safe and On Track for Success A meta-analysis of 73 afterschool evaluations concluded that quality programs were consistently successful in producing multiple benefits for youth including improvements in children's personal, social and academic skills, as well as their self-esteem. (University of Illinois at Chicago, Collaborative for Academic, Social, and Emotional Learning, 2007)  Children attending LA’s BEST Afterschool program are 30 percent less likely to participate in criminal activities than their peerswho do not attend the program. Researchers estimate that every dollar invested in the LA’s BEST program saves the city $2.50 in crime-related costs. (UCLA National Center for Research on Evaluation, Standards and Student Testing, September 2007)  Youth attending 23-40 or more days of Maryland’s After School Opportunity Fund Program showed a more positive gain on commitment to education and academic performance, and a reduction in delinquency and contact with the police.(University of Maryland, Department of Criminology and Criminal Justice, June 2004)  Teens who do nothree times more likely to skip classes , use marijuana or other drugs, drink alcohol, smoke cigarettes and engage in sexual activity.t participate in afterschool programs are nearly (YMCA of the USA, March 2001)

27. Helping Families Parents who are concerned about their children’s after-school care miss an average of eight days of work per year. Decreased worker productivity related to parental concerns about after-school care costs businesses up to $300 billion per year. (Brandeis University, Community, Families and Work Program, 2004 and Catalyst & Brandeis University, December 2006) In an evaluation of LA’s BEST, three quarters of the parents surveyed indicated that since enrolling their children in the program, they worried significantly less about their children’s safety and had more energy in the evening. (UCLA Center for the Study of Evaluation, June 2000 and December 2005) Parents in the TASC study said that the program helped them balance work and family life: 94 percent said the program was convenient, 60 percent said they missed less work than before because of the program, 59 percent said it supported them in keeping their job, and 54 percent said it allowed them to work more hours. (Policy Studies Associates, Inc., 2004) After School Alliance: http://www.afterschoolalliance.org/after_out.cfm

28. Parents’ Perspectives

29. Parents’ Satisfaction

30. Obstacles for Parent Involvement Conflicts with job Care for other children Language barrier Lack of transportation ASES • 60% visited the program • One in three attended any event • One in five volunteered or given feedback ASSETs • One in five visited the program • One in nine attended any event • One in twenty volunteered or given feedback

31. Program Quality

32. Afterschool Accomplishments • Created a knowledge field that is in general in consensus of program characteristics for high quality indicators • Created a database now possible for investigating longer term effects • Established a “nitch” and demonstrated the significance/importance of ASP in educational research

33. Common Practices of High Performing Afterschool Programs • Offering a broad array of enrichment activities • Provide a wide range of experiences that promote skill-building and mastery • Intentional relationship-building • Employ strong managers, differentiated Staffing, and • Partner Organizations Provided Support to Project Leaders and Participants Tasc, 2007

34. Successful Program Features Best practices evidenced in the literature on out-of-school time suggest that several critical components such as: goal-oriented programs program structure and program process These components contribute to the effectiveness and success of programs.

35. Indicators for High Quality Programs • Program Structure Clearly defined goals Set up program structures to meet these goals Set up program mission and vision to motivate staff • Process Strong leadership High quality staff Clear communications and support Positive relationships • Content Research-based curriculum and strategies Build in assessment and continuous improvement loop

36. Theory of Change Making Programs Accountable and Making Sense of Program Accountability • Outcomes-based • Causal model • Articulate underlying assumptions

37. Evidence-based Teaching Approach-CAESL • Sequenced - a sequenced set of activities to achieve skill objectives • Active - the use of active forms of learning • Focused – program component to be focused on specific skills • Explicit – the targeting of specific skills

38. Use Internal Evaluation for Program Improvement • monitoring student progress • measuring program growth • researching program needs • defining areas for professional development • gauging program impact

39. Model of Data-based Decision Making & Continuous Improvement Process Data Based Decision-making Process Analysis of Results Plan of Action Assessment of Student Outcomes Specification of monitoring Continuous Adjustment

40. Pyramid of Program Quality – High Scope Engaged Learning Opportunity for Interaction Encouragement Reframing conflict Skill building Session flow Active engagement Welcoming atmosphere Supportive Environment Psychological safety Physically safe environment Emergency procedures Program space and equipment Healthy food and drinks Safety