Evaluation: Using Multiple Methods to Demonstrate Impacts

1. 1 Evaluation: Using Multiple Methods to Demonstrate Impacts Youth and Families with Promise Utah State University

2. 2 Overview of Youth and Families with Promise An early intervention, two-level mentoring program designed to reduce delinquency, enhance social and academic skills, and improve family relations in at-risk youth ages 10 to 14 and their families.

3. 3 What Makes Youth and Families with Promise Unique? Two level mentoring approach Young adult and grand mentors Mentoring activities are asset based with curriculum for both youth and families Youth and family involvement Family Night Out Integrates youth into 4-H youth programs Service Activities

4. 4 Multiple Methods of Data Collection Why is it useful? Increases reliability. Presents a more accurate picture of outcomes than a single method.

5. 5 Methodologies Utilized Quantitative Pre-Post Testing Post-then-Pre Testing Intensity of Mentoring (Dosage) Qualitative Focus Group Interviews Individual Interviews Refer to instruments available on the websiteRefer to instruments available on the website

6. 6 Standard Pre-Post Testing Respondents evaluated the youth�s behavior in several areas prior to involvement in the program. After participating, they evaluated the youth again on the same behaviors.

7. 7 Advantages of Pre-Post Testing Widely accepted as the standard methodology for program evaluation. Respondents are generally familiar with this type of survey instrument.

8. 8 Disadvantages of Pre-Post Testing Incomplete �sets� must be excluded If the pre-test data was not collected for a respondent, their post-test data cannot be used. Pre-Post testing does not allow for a paradigm shift by the individual.

9. 9 Response Shift Bias A paradigm shift which occurs while the individual is involved in the program. Examples: Vitamin C Communication

10. 10 Post-then-Pre Testing Respondents evaluated the youth�s behavior after their involvement in the program, and at the same time retrospectively evaluated the behavior prior to participation.

11. 11 Advantages of Post-then-Pre Testing Overcomes the problems associated with response shift bias. Avoids incomplete data sets. All needed data was collected at one time. Reduced expenses. Data collection only occurred once. Less paperwork for respondents and administrators.

12. 12 Disadvantages of Post-then-Pre Testing Not widely utilized or accepted as standard. Social desirability bias People like the program and want to make it look good by indicating improvement.

13. 13 Focus Group Interviews Small groups of respondents were interviewed together about their experiences. Interviews were transcribed verbatim and analyzed qualitatively.

14. 14 Advantages of Focus Group Interviews More in-depth responses and emotion than quantitative data. Human interest stories bring the research to life. Allows participants to focus on and discuss the things they feel are most important (which may not be asked quantitatively).

15. 15 Disadvantages of Focus Group Interviews More time and resource intensive. Training moderators, logistics of coordinating interview Self selection bias Those most invested in the program are more likely to participate.

16. 16 Individual Youth Interviews Youth were interviewed individually by telephone about their experiences in the program. Researchers never interacted directly with youth maintaining anonymity.

17. 17 Advantages of Individual Youth Interviews More in-depth responses and emotion. The outcomes are reported directly by those involved, not through a third party. Provides insight about why youth participate. Can�t force change, if youth don�t want to participate or don�t enjoy the program, they won�t benefit from it.Can�t force change, if youth don�t want to participate or don�t enjoy the program, they won�t benefit from it.

18. 18 Disadvantages ofIndividual Youth Interviews Logistically difficult to select sample and conduct interviews while maintaining confidentiality. Raises questions about how well adolescents can reflect on personal growth.

19. 19 What can you do when there is not a control group?

20. 20 Dosage Data Information about the level of participation or amount of the program or study received. Examples: Units completed Attendance Amount of substance given

21. 21 Intensity of Mentoring:Dosage Data Data were collected about how frequently youth had contact with their mentors and the amount of time the mentors and youth spent together. Data were also collected about family involvement and participation in program activities.

22. 22 Advantages of Dosage Data Sets up a control group without denying participation to youth who could potentially benefit. Dosage data can be analyzed separately as a sub-sample of those in the program so youth without this data are not excluded from analysis.

23. 23 Disadvantages of Dosage Data High likelihood of missing cases because accurate data is difficult to collect. More paperwork at every level of the program.

24. 24 Example of using multiple methods from YFP

25. 25 Summary of 2001-2002 Outcomes Each reported variable was statistically significant at p<0.001. Most youth showed significant improvement in at least one area.

26. 26 Percent of Youth Showing Improvement Socially