Social Welfare 580 Advanced Research Methods & Design

1. Social Welfare 580Advanced Research Methods & Design ExperimentsQuasi-Experiments and Quantitative Program Evaluation

2. Types of correlational analysis methods • Bivariate correlation • Regression and prediction • Multiple regression • Factor analysis • Cluster analysis • Covariance structure modeling • Confirmatory Factor Analysis (CFA) • Structural Equation Modeling (SEM) • Latent Growth Curve Modeling (LGM) • Survival analysis • Mixture modeling • Hierarchical linear modeling (HLM)

3. Experiments, Quasi-Experiments &Quantitative Program Evaluation • Features of Experimental Designs • Things to look for in critiquing the strengths of an experiment • Factorial designs • Quasi-experimental designs • Program evaluation

4. Experimental and Quasi-experimental Designs Two primary purposes • To verify causes in an etiological framework • Intervention evaluation • Efficacytrials: does a proposed program work? • Effectiveness trials: does a tested program work when brought to scale? • Program evaluation: is the currently applied program working?

5. Quantitative Program Evaluation Efficacy Effectiveness

6. Experiments Random assignment to conditions Observations at Baseline Treatment/Intervention Post-Intervention Assessment Defined: A study in which at least one variable is manipulated and units are randomly assigned to different levels or categories of the manipulated variable(s). Most common experiment: Two group designs with a pretest & controlR O1 X O2R O1 O2

7. Features of Experiments • random assignment– reduces the plausibility of alternative explanations of observed effects. This is the main strength of experiments. • laboratory vs. field experiments • manipulation strength ≠ strength of the actual effect • treatment integrity • manipulation check: intended & unintended effects

8. Distinction between random sampling and random assignment random sampling: external validity sample random assignment to conditions: internal validity R O1 X Intervention Target OutcomeR O1 Intervention Target Outcome Population

9. Experiment Randomization Check: see that all relevant variables are equivalent across groups at baseline….was group equivalence achieved through randomization? If not, include vars which are different as covariates. R O1 X Intervention Target OutcomeR O1 Intervention Target Outcome

10. Experiment Treatment Integrity: what procedures were implemented to ensure that the treatment was implemented fully, as designed? R O1 X Intervention Target OutcomeR O1 Intervention Target Outcome

11. Experiment Manipulation Check: Did the intervention change the independent variable as intended? R O1 X Intervention Target OutcomeR O1 Intervention Target Outcome

12. Experiment Outcome Analysis: Did the intervention affect the target outcome? R O1 X Intervention Target OutcomeR O1 Intervention Target Outcome

13. Example: Intervention to change teaching methods to enhance bonding to school in children randomly assigned to 2 conditions Randomization Check: gender? ethnicity? child initial levels of school bonding? child grades? teacher characteristics? Were the groups equivalent at baseline? 40 teachers (30 kids per classroom) R O1 X Teacher Behavior BondingR O1 Teacher Behavior Bonding 20 20

14. Example: Intervention to change teaching methods to enhance bonding to school in children Treatment Integrity: program materials standardized/manualized? teacher attendance at workshops? workshop leaders well trained? teacher attrition? child attrition? R O1 X Teacher Behavior BondingR O1 Teacher Behavior Bonding

15. Example: Intervention to change teaching methods to enhance bonding to school in children Manipulation Check: Did the teachers in the intervention condition actually implement in the classroom the things covered in the workshops? R O1 X Teacher Behavior BondingR O1 Teacher Behavior Bonding

16. Example: Intervention to change teaching methods to enhance bonding to school in children Outcome Analysis: Were the kids whose teachers got the intervention significantly more bonded to school than non-intervention kids? R O1 X Teacher Behavior BondingR O1 Teacher Behavior Bonding

17. Study Questions: Sobell, et al. 2002 “General Mail Intervention Can Reduce Alcohol Use Among Problem Drinkers” A generalized mail intervention is as successful in reducing alcohol abuse as is providing personalized feedback, according to a study of problem drinkers in Toronto, Ontario (Canada). Problem drinkers who had never sought treatment were randomly assigned to one of two types of interventions, which consisted of receiving through the mail either generalized pamphlets with information about the effects of alcohol, or personalized advice and feedback based on their individual drinking and related behaviors.

18. “General Mail Intervention Can Reduce Alcohol Use Among Problem Drinkers” The study found that either form of intervention resulted in significant reductions in the percentage of days drinking from one year before to one year after the intervention.

19. “General Mail Intervention Can Reduce Alcohol Use Among Problem Drinkers” Both types of interventions also resulted in a decline in binge drinking rates and the number of alcohol consequences experienced, as well as an increase in the percentage who had received alcohol treatment. According to the authors, “These results, coupled with the low cost to deliver the intervention, suggest that public health campaigns could have a substantial effect on reducing alcohol problems and associated cost as well as getting some individuals into treatment”

20. Study Questions Does a generalized mail intervention result in a decline in binge drinking rates among problem drinkers? Does a motivational enhancement/personalized feedback intervention result in a decline in binge drinking rates among problem drinkers?

21. Study Questions • methodological strengths and weaknesses: • conceptualization • operationalization • sampling, power • Potential problems of: • threats to construct validity • threats to internal validity • threats to external validity.

23. Experiments with a Factorial Design R O1 X1 O2 treatment 1 R O1 X2 O2 treatment 2 R O1 X12 O2 treatment 1 and 2 combined R O1 O2 no treatment control group

24. Example, Parent and Teacher Training to Enhance School Bonding teacher training yes no 1 only control no parent training 2 only yes 1 and 2

25. Graphically bonding no parent training 5 parent training 1 no teacher teachertraining training

26. Graphically bonding no parent training 5 parent training 1 no teacher teachertraining training

27. BREAK?

28. Quasi-Experiments • Defined: An experiment in which units are not randomly assigned to conditions. • Used to compare groups, but random assignment not possible, • For example, groups already exist (male vs. female, depressed vs. non-depressed), or politically, ethically unfeasible, etc.; • All of the other issues of experiments still apply: lab vs. field; manipulation strength; treatment integrity; manipulation checks and unintended effects. • In addition, one must tend to additional threats to internal validity (e.g., group equivalency at baseline.) • Cause of group differences not always clear.

29. Weak quasi-experimental designs:one group, no comparison One group designs with no control group:One-group pretest posttestO1 X O2 This design is strengthened by adding pretest observations, and/or by adding or removing treatments: O1 O2 X O3or O1 X O2 X O3 X O4 But still a pretty weak design.

30. Stronger quasi-experimental designs:two group designs Two group designs but no pretestNR X ONR O Most common quasi-experiment: Two group designs with a pretest & controlNR O1 X O2NR O1 O2

31. Time Series / Interrupted Time Series / Multiple Baseline Designs • Time series designs involve many observations over time. • Interrupted time series typically involve three or more observations of a group before and after the intervention. O1 02 03 04 X 05 06 07 08 • Multiple Baseline Time Series design involves successive implementation of the same intervention in different groups over a time-series. Grp1 O1 02 03 X 04 05 06 07 08 Grp2 O1 02 03 04 X 05 06 07 08 Grp3 O1 02 03 04 05 X 06 07 08

32. Time Series / Interrupted Time Series / Multiple Baseline Designs • These designs emphasize point-specific causal hypotheses. If the cause-effect link is quick acting, then effective treatment should lead to change in level, slope or variance of the time series at the point where treatment occurred. • The test of intervention effect is whether the obtained data show a change in the series at the pre-specified point. • Analyses typically involve multi-level modeling to account for the fact that observations over time of the same group are correlated with themselves.

33. Example Interrupted Time Series Proportion of tobacco outlets in Prineville community that were willing to sell to those under 18 before and after the implementation of a reward and reminder program. (Biglan, et al. 2000).

34. Example Multiple Baseline Design Proportion of tobacco outlets in four communities that were willing to sell to those under 18 before and after the implementation of a reward and reminder program. (Biglan, et al. 2000).

35. Example Multiple Baseline Design Proportion of tobacco outlets in four communities that were willing to sell to those under 18 before and after the implementation of a reward and reminder program. (Biglan, et al. 2000).

36. Shadish’s Ten Lessons About Field Experimentation • The logistics of selection and assignment can make or break your experiment. • It is essential to study program implementation. • Use what has been learned about preventing and analyzing attrition. • We know how to do better analyses of quasi-experiments. • The unit-of-analysis problem is dead: long live multi-level modeling. • We also know how to do better analyses of longitudinal data. • Pay more attention to statistical power and effect size. • Place brackets around your estimates. • Both internal and external validity are important in field experimentation. • Learn about Rubin's causal model.

37. Quantitative Program Evaluation Efficacy Effectiveness

38. Quantitative Methods to Evaluate Interventions • Randomized Controlled Trials • Quasi-Experiments • Outcome evaluations increasingly incorporate • program theory or logic model • extended data collection to include • program implementation • participant characteristics • patterns of change (multiple measures of the outcome).

39. Evaluation Research: A broader view • Needs Assessment – Is the program needed? • Not an easy assessment. • Get multiple indicators from multiple sources: - measures of prevalence of the problem - measures of level and distribution of risk - measures of stakeholder perceptions of need

40. Evaluation Research: A broader view • Process Evaluation – How does the program operate? • Assessment of organizational structure and operation. • Assessment of factors that contribute to successful (or unsuccessful) implementation. • Formative Evaluation – Process evaluation used to shape and refine program operations.

41. Evaluation Research: A broader view • Outcome Evaluation – What is the program’s impact? Does it work? • AKA, Impact Evaluation or Summative Evaluation • The methods we have been discussing today (experimental and quasi-experimental analysis) used to conduct outcome evaluations.

42. Evaluation Research: A broader view • Efficiency Analysis – How efficient is the program? • Cost-Benefit Analysis • Identify the specific costs and benefits to be studied. • Program clients, program staff, taxpayers all will have a different perspective on what can be considered a benefit or a cost. • Typically, benefits and costs are monetarized. • Cost-Effectiveness Analysis • Program costs are compared across programs while the benefits are listed but not assigned a cash value.

43. Example: WSIPP Cost-Benefit Analysis Washington State Institute for Public Policyhttp://www.wsipp.wa.gov/ Does prevention pay? Can an ounce of prevention avoid (at least) an ounce of cure? More specifically for public policy purposes, is there credible scientific evidence that for each dollar a legislature spends on “research-based” prevention or early intervention programs for youth, more than a dollar’s worth of benefits will be generated? If so, what are the policy options that offer taxpayers the best return on their dollar?

44. Example: WSIPP Cost-Benefit Analysis Step 1: Outcome analysis. What works? What doesn’t? Step 2: What long-run information is known about populations to which a program could be applied? Step 3: What is the value to taxpayers and crime victims of reducing crime by one unit?The cost of crime to taxpayers is estimated by modeling the marginal operating and capital costs of Washington’s state and local government criminal justice system, and the way in which juvenile and adult criminal cases are processed in Washington. The costs incurred by crime victims are obtained from national sources. Step 4: What do different approaches cost? Step 5: What are the comparative costs and benefits of programs?

45. Example Cost-Effectiveness Analysis http://www.wsipp.wa.gov/pub.asp?docid=04-07-3901

46. In reviewing the economic results, several findings emerge: • Investments in effective programs for juvenile offenders have the highest net benefit. Such programs yield from $1,900 to $31,200 per youth. • Some forms of home visiting programs that target high-risk and/or low-income mothers and children are also effective, returning from $6,000 to $17,200 per youth. • Early childhood education for low income 3- and 4-year-olds and some youth development programs provide very attractive returns on investment. • While their net benefits are relatively low, many substance use prevention programs for youth are cost effective, because the programs are relatively inexpensive. • Few programs are effective at reducing teenage pregnancy. • Each program area we examined has interventions that are not cost effective. Some prevention and early intervention programs are very expensive and produce few benefits.

47. Example: Cost-Effectiveness Analysis Gibbard, Deborah; Coglan, Louisa; MacDonald, John. Cost-effectiveness analysis of current practice and parent intervention for children under 3 years presenting with expressive language delay. International Journal of Language & Communication Disorders. Vol 39(2) Apr-Jun 2004, 229-244. Two treatment strategies for children diagnosed with speech and language delay: • Parent-based intervention: parents were trained in 11 sessions every other week over 22 weeks to use daily routines and naturally occurring situations to advance language objectives • General Care: parents and child seen individually with speech and language therapists every 2 weeks.

48. Example: Cost-Effectiveness Analysis Costs in£:

49. Example: Cost-Effectiveness Analysis Costs in£:

50. Example: Cost-Effectiveness Analysis