1 / 28

Theory and Methods in OST Research

Theory and Methods in OST Research. Denise Huang. American Educational Research Association Annual Meeting AERA San Diego, CA - April 14, 2006. What do all these mean?. The gold standard for research design and analysis involve experimental randomization.

doris-foreman
Download Presentation

Theory and Methods in OST Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Theory and Methods in OST Research Denise Huang American Educational Research AssociationAnnual Meeting AERA San Diego, CA - April 14, 2006

  2. What do all these mean? • The gold standard for research design and analysis involve experimental randomization. • Satisfactory research findings required generalization and replication. • The afterschool population has distinct characteristics Ethical issues Self-selective group Difficulty in obtaining comparison group (consent forms) Accuracy of data and availability of longitudinal records High transience • Qualitative strategies are vital tools in finding out the intricate rich data about a program and help to explain, elaborate, and triangulate quantitative findings… • Matching study questions with design and appropriate methodology

  3. Most afterschool studies are evaluation research The systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social intervention programs (Strauss & Corbin, 1990). • Focus on finding and explaining educational /social effects or outcomes from the intervention, and • Devising instructional strategies that will improve the outcomes. • Formative and summative evaluations • Process and impact evaluations

  4. Different types of evaluation research Process Evaluations: Formative studies • Evaluations that assesses the conduct of the program during the initial design and testing stages with the intent to improve the program Program monitoring • Systematic examination of program coverage and delivery- (target population, fidelity, efficiency) • Identifying successful implementation strategies for program diffusion

  5. Impact Evaluations Assesses the changes in the well-being of individuals that can be attributed to a particular intervention, such as a project, program or policy (Strauss & Corbin, 1990). Summative evaluation • Summative evaluation provides information on the product's efficacy ( it's ability to do what it was designed to do) • By looking at the intervention group, the evaluator can examinethe learning materials and learning process together with the outcomes-- hence the name Summative Evaluation. Impact Evaluation • Impact evaluation involves constructing a counterfactual • Random selection and isolation from interventions are seldom practicable and sometimes ethically difficult to defend. • Quasi-experimental method is often used.

  6. Which theoretical perspective informed your work, and why did you select this theory? • Goal setting theory • Social cognitive theory • The contextual understanding of the social ecological model

  7. The role that theory play in research Theory is an invention aimed at organizing and explaining specific aspects of the environment (Pedhazur & Schmelkin, 1991) • Theory provides the researcher with a “selective point of view-an orientation • Helps to determine what variable is relevant and which are not relevant • Being a way of seeing, a theory is also a way of not seeing.

  8. How did this theory shape your research methods? • Mixed methods • Process/outcome • SEM • HLM

  9. Examples: Formative, Process, & Summative The ASL/KidzLit Formative Study ASL is a reading and writing project that was based on the theoretical foundation of the literature-based instructions (social cognitive theory) • How do the LA’s BEST teachers evaluate the ASL project in terms of training, materials, and support? • To what extent have staff capabilities been enhanced after the ASL training? • If the staff members teach during the regular school day and in the after-school program, to what extent have they carried the ASL strategies into their regular school day teaching? • What changes in student attitudes might be associated with ASL? Are there any preliminary indicators that participation in ASL is affecting students?

  10. Teacher survey I-formative evaluation For the effectiveness of the ASL training, the evaluation based on ASL’s four core components: • Care • Talk • Read • Connect To evaluate the ASL project as a whole, teachers’ perspectives were examined: • ASL materials • ASL training • Support for the ASL project For an affective/motivational outcome the following indicators were examined: • Students’ enjoyment • Teacher attitude

  11. The student survey Based on the social cognitive theory that learning is a social phenomenon and social collaboration is essential in the classroom climate, we investigate on students’: • Enjoyment of their reading class; • Perception of the social support in the classroom; • Perception of their relation with their teachers. Based on the motivational theory, we look at the students’: • Attitudes towards reading and writing; and • Reading and writing self-efficacy. And students’ perception of the classroom climate: • The autonomy orientation of their classroom climate; and • The emphasis on effort allocation.

  12. Qualitative study Innovation and Staff Development Internal Networks External Networks Organizational Capital Organizational Capital Competence Intellectual Capital Social Capital Relationships Intellectual Capital Social Capital Attitudes Trust Problem-solving & Agility Expectations & Social Norms Staff & Student Relationships Student Engagement Exploring the intellectual, social, and organizational capitals at LA’s BEST

  13. Quantitative outcome evaluations Keeping kids in school study • 4 cohorts of students (all LA’s BEST participants and a stratified random sample of non-participants) • 3 participation level ( 1,2,3+ years) • 6-9th grade in 1998-1999 followed through to 2002-2003 • Chi-square analysis • Cox survival analysis

  14. Comparison of dropout rates for LA’s BEST vs. LAUSD non-participants (3+ Years).

  15. The Afterschool Hours-SEM and HLM Examining the Relationship between Afterschool Staff-Based Social Capital and Student Engagement in LA’s BEST • What are LA’s BEST staff perceptions of collective staff efficacy, teamwork, and communication, and the quality of their relationships with students? • What are student perceptions of their relationships with LA’s BEST staff? To what extent do they value education and have high aspirations for their futures? What are their reported levels of student engagement in LA’s BEST and the day school?   • How are staff-student relationships, teamwork and communication, and collective staff efficacy at the site level, related to student perceptions of their relationships with staff?   • What is the association between student perceptions of their relationship with staff, their value of education, future aspirations, and engagement in the afterschool program and day school?

  16. Hierarchical Linear Modeling • Examine the relationship between the social capital predictors measured by the staff surveys (i.e., staff-student relationships, collective staff efficacy, and communication and teamwork) and student perceptions of social capital (i.e. staff-student relationship) as measured by the student surveys. • The demographic variables of gender, grade level, and languages spoken were also included in each model to control for individual student differences. • A total of 2,270 students and 395 staff from 50 school sites were included in the HLM analysis.

  17. Structural Equation Modeling

  18. DOJ study-using propensity matching & cost-benefit analysis • Quasi-experimental design • Longitudinal sampling of academic and juvenile crime data • Advanced multilevel propensity scores methods to establish study samples • Hierarchical growth modeling and survival analysis (multilevel discrete-time hazard) • Students were followed from 1994-1995-2002-2003 school years • Cost-benefit analysis

  19. How did your method(s) contribute to our knowledge of OST, and what were the limitations of the method? • Better comparison sample—propensity matching • The importance of “dosage” • Importance of goal setting • Importance of employing a continuous improvement model

  20. Why results are inconsistent? • Linking results to program content __Design___________________________________________ • Linking results to implementation procedures(program fidelity) __Treatment________________________________________ • Linking results to program attendance __Dosage___________________________________________

  21. What does that theory mean for what you would do with children and youth during the out-of-school time and inform program design? Importance of having • A theory of change • A logic model • A continuous improvement strategy

  22. Why create a logic model? • They are powerful tools for designing, planning, implementing, and evaluating OST programs • They set up a plan of how a program is expected to work • Provide a map of how to achieve goals (cause and effect-program process and outcomes) • Stimulate clear thinking, preparation, and organization

  23. Example: Theoretical-based logic model designed for the Afterschool Partnership study Program Support Management Staff /Resources Periodical Evaluation for Continuous Improvement After School Center Quality Align activities to goals Motivation Engagement Periodic Assessment of Student Performance Content Practice Quality Opportunities to Practice Set Goals Linkage to School Day Research Based Practice & Strategies Content Aligned to Standards Process Content Process Structure

  24. Example: Outcome based logic model Content Academic Design Linkage with Standard Attitudes & Skills Research-based Engagement Internal Program Structure Structure Program Structure External Communication & Support Process Afterschool Program Implementation Student/Adult Interaction Parent Satisfaction Indicators Design & Process Immediate Outcome Expected Outcome Long-term Outcome Linkage with School Achievement Professional Development

  25. Using data for continuous improvement Data Based decision Making Process Theory and Research-based Programming Student Engagement Student Retention Student Improvement Analysis of Results Plan of Action Periodic Assessment Specification of Monitoring Continuous Adjustment

  26. Looking Forward Gap in knowledge as we are working to develop the 21st century skills: • Technology divides • The contextual environmental • Social capital and human networking system

  27. dhuang@cse.ucla.edu

More Related