Office for Evaluation and Needs Assessment Social Science Research Institute University of Hawaii

1. Evaluation of Hawaii’s Supplemental Educational Services ProgramDRAFT Evaluation PlanPresentation to Service Providers and Complex Areas April 2006 Office for Evaluation and Needs Assessment Social Science Research Institute University of Hawaii

2. Evaluation Team • Dr. Judith Inazu, P.I., Associate Director, Social Science Research Institute • Dr. Shuqiang Zhang, Statistician, College of Education • Dr. Daniel Anderson, Consultant, Planning and Evaluation, Inc. • Dr. Aiko Oda, Consultant, Planning and Evaluation, Inc. • Ms. Nancy Marker, Educational Specialist, Social Science Research Institute • Dr. Patty Reiss, Lecturer, College of Education • Ms. Julie Holmes, Graduate Assistant, Social Science Research Institute

3. Goal of SES To increase the academic achievement of economically disadvantaged students in low-performing schools by providing additional academic instruction outside of the regular school day.

4. Purpose of Evaluation To determine the effectiveness of service providers in increasing students’ academic achievement. Providers must: ► Increase students’ achievement for 2 consecutive years, and ► Provide services consistent with applicable federal, state, and local health, safety, and civil rights requirements.

5. Guiding Questions 1. Effectiveness Did the provider increase student achievement in reading and/or mathematics? • Customer Satisfaction Are parents, schools, and complex areas involved in SES satisfied with the service provider? • Service Delivery Did the provider comply with applicable federal and state laws and regulations, and contractual procedures and requirements associated with the delivery of SES?

6. Interim and Final Evaluation Plans Interim Evaluation Plan Year 1 (Fall 2005 - Summer 2006) Year 2 (Fall 2006 - Summer 2007) Year 3 (Fall 2007 - Summer 2008) Final Evaluation Plan Year 4 (Fall 2008 - Summer 2009)

7. Measuring Academic Achievement In Year 1 of the Evaluation, academic achievement will be the sole criterion for evaluating service providers

8. Measuring Academic Achievement • Analysis to be conducted separately for each grade by subject matter, and for each provider. • Example: For 4th graders who received tutoring in reading by College Connections.

9. Minimum Data Set for AnalysesData on all SES-eligible students. Arrayed by grade level, subject matter, and service provider.

10. Analyses Analyses of Covariance (ANCOVA) with prior HSA score as the covariate, SES status (yes/no) as the binary independent variable, and 2006 HSA score as the dependent variable. The ANCOVA asks the question, “If you hold the pre-test scores constant (since all students begin at different levels of performance), is there a significant difference in post-test scores between students who received tutoring and students who did not?”

11. Analyses for Students Without Pre-test Scores Conduct t-test analyses (difference between mean scores) • Compare 2006 HSA test scores between students who received tutoring and those who did not. • Again, analyses conducted separately by grade level, subject matter, and for each service provider.

12. Measuring Customer Satisfaction Pilot tested with selected samples in Years 1-3: • Parents • Teachers • Complex Area Administrators • Principals • Students (To-be-determined)

13. Measuring Customer Satisfaction Pilot testing to begin in May 2006 • Paper and pencil questionnaires with a select sample of parents, teachers, complex area administrators, and principals. • Translation requirements for parents. • Internet survey for teachers, school administrators, and complex areas.

14. Sample Questions: Parental Satisfaction 1. The school staff was qualified and supportive in helping me get tutoring for my child. • Yes • No 2. I would recommend this tutor to other parents. • Yes • No 3. My child's school work improved because of the tutoring. • Yes • No 4. My child found tutoring was a positive experience. • Yes • No 5. The tutors were good at reporting to me about my child’s progress. • Yes • No 6. Overall how would you rate this tutor? • Strong • Average • Weak

15. Measuring Service Delivery Pilot testing to begin in May 2006 • Site visits and observations of tutoring sessions • Interview(s) with contact person(s) at the site • Document review during site visit • Self-administered compliance checklist

16. Instructional Observation • Clear academic expectations are set and articulated • Instruction is on task without interruption • Criterion material is covered with instructor’s direction • Instructional time is adequate for material yet flexible • Learning modalities are active, variable and appropriate • Instructional pacing is appropriate for student’s interest and ability • Student progress is frequently assessed • Evidence of mutual respect, positive interaction and feedback • Evidence of the instructor’s enthusiasm and interest

17. Instructional Delivery • Student-teacher ratio • Computer, lectures, etc.

18. Compliance Checklist (Examples) SES Provider Assurances • Use research-based strategies designed to improve academic achievement • Offer instruction consistent with state academic content and achievement standards • Meet federal, state and local health, safety and civil rights law • Provide services that are secular, neutral, and non-ideological • Provide information on student’s progress, as agreed upon with the LEA • Remain financially viable • Abide by Family Educational Rights and Privacy Act of 1975

19. What We’ll Need From You Complex Areas • Written permission to be on campus to conduct site observations • Names and addresses of parents • Completion of satisfaction survey

20. What We’ll Need From You Service Providers • Assistance in scheduling site visits • Set aside time for an interview • Documents available for review (e.g., attendance logs, student records, pre/post test reports) • Completion of compliance checklist

21. No Vendor Left Behind(NVLB)