Task and Process: Assessment and Evaluation of Outreach Efforts

1. Task and Process: Assessment and Evaluation of Outreach Efforts Gary D Glass, Ph.DDuke University Forrest Seymour, MSW, LICSW Keene State College Teresa Michaelson-Chmelir, Ph.D, LMHC University of Central Florida Eva G. Schoen, Ph.D University of Iowa Chris Meno, Ph.D Indiana University

2. Task and Process: Assessment and Evaluation of Outreach Efforts Why evaluate? What are the challenges?(Dealing with the “UGH!”) What are steps forward?

3. Why are we here? • AUCCCO Research subcommittee charge:Explore options in developing measures to assess the quality & value of outreach, particularly workshops & presentations. • Ongoing process & dialogue:We are looking for your input and feedback. Why are you here?

4. Our Frame and Quest… Empires of Outreach

5. Our Frame and Quest… 1. Identifying/Clarifying Goals of Outreach • Introducing therapeutic elements and messages into the campus environment • Marketing Counseling Center • Providing skills and knowledge • Increasing Awareness of mental health and student development issues • Responding to Acute Needs in Community (crises/tragedy/trauma)

6. 2. ReviewingModalities of Outreach Our Frame and Quest… • Delivering Presentations • Providing Training to professional colleagues and student para-professionals • Facilitating Discussions • Organizing Experiential/Interactive Events • Producing/Distributing Content (print, electronic, multimedia, etc) • Building/Maintaining Liaison Relationships w/ constituencies • Screenings/Early Detection Activities

7. Our Frame and Quest… 3. Reviewing Methods/Tools of Conducting Assessment • Printed Evaluation Forms • Online Evaluation Forms • Tablet/On-site computer tools • Accessing institutional data (including, obviously, counseling center

8. UGH!

9. Our Players and Problem • The Players: • UCF: 60,000+ CAPS: Clinical Staff: 20 Outreach Committee: 7 members (Psychologists, Mental Health, SW, and a doctoral intern); meeting bi-weekly • The Problem: • We needed to better evaluate the effectiveness of outreach • But there is just so much variation in what we define as “outreach”(psychoeducational/developmental programs, tabling's, signature events, etc) • The OTHER variables, WHAT do we want to measure? • Does programming increase utilization of services? • Did we teach new skills/awareness?? • Did our outreach program improve retention???

10. Shifting from “UGH, there is too much!!” …to….”ok, we got this.” • As a committee, we decided to focus on one area of our outreach ….psychoeducational / developmental programming. • We began to think of aaaaalllllllll the program titles we have done, and coded them under topics headings

11. Initial action steps…… As you can see….we struggled with duplication. Some topics just seemed to fit under several areas….

12. A second look…. • We took another look at the themes and decided that we could narrow them down a bit further AND thought more about how WE want to define our programs…. make it fit with our strategic plans, our mission, our whole existence!!!

13. We can define ourselves • Our overarching presentation themes became: • Academic Success • Social Connections • Wellness and Life Skills • Mental Health Awareness • Diversity Awareness UCF CAPS Outreach Website

14. Now what? • In order to evaluate our efforts, we had to identify what we wanted to know about our services…. • Overall effectiveness of workshop in achieving success at UCF and in life • Topic specific value • Likelihood of using the skill/information in the future

15. Evaluation Questions: • Learning Objectives: • Overarching: • To learn in this workshop that will help me be more successful at UCF and in my life. • Topic Specific : • To learn to improve my (Academic Success, Social Connectedness, Wellness and Life Skills, Mental Health Awareness, Diversity Awareness) Skills Information Skills and Information

17. Task and Process: Identifying Variables Of Effectiveness What would actually go on the forms that we use to evaluate our programs?

18. Variables of Effectiveness: Our Process: From A Single Standardized Form to a Series of Variables to Choose From • Outcome Measures Related to Change In Attendees As a Result of the Program/Presentation • Outcome Measures Related to Attendees’ Evaluation of the Program/ Presentation • Macro-Level Outcome Measures (Related to Prevalence and Severity of issues directly or indirectly addressed by Counseling Center)

19. Outcome Measures Related to Change In Attendees • Student reports Intention/Plan to Implement (specified) Change After Program • Student Gained Knowledge/Information (re: topic of program) • Student Gained/Improved Skills from Program • Student Increased/Gained Awareness or Insight (regarding Self and Self-in-the Context of his/her environment) • Student reports achieving SPECIFIED Desired Change in Experience or Behavior

20. Outcome Measures Related toChange In Attendees • Student reports achieving SPECIFIED Desired Change in Experience or Behavior • Student reports decrease in specified problem indicators (i.e., stress, symptoms, behaviors, etc) • Student reports increase in specified resilience or satisfaction indicators (i.e., positive self-concept, improved sleep, increased focus, etc). • Student Reports Experiencing Connection With/Among Other Attending Students • Student Reports Improved Efficacy in Being Change/Support Agent to peers and community

21. Outcome Measures Related to Change In Attendees • Students indicates change in attitude • Regarding topic/behavior (specified in particular program) • Regarding counseling center • Regarding help-seeking in general

22. Outcome Measures Related to Attendees’ Evaluation of the Program/ Presentation • Overall Rating of Presenter • Overall Quality/Value of Program • Open Ended Questions on Program(+ and –) • Produced Desired Knowledge and Attitudes of Counseling Center • Assessment of whether program was Cross-Culturally Competent/Attended to Diversity

23. Macro-Level Outcome Measures • Overall Reduction in Symptoms/Incidence of dysfunction or impairment in local clinical data (LCD) • Overall Reduction in Prevalence of Problematic Behaviors in campus data • Overall Increase in presence of individual resilience or satisfaction Factors in institutional data • Increased utilization of counseling center resources for high risk or populations that under-utilize services • Decrease utilization of counseling center resources for low-severity issues that are identified as linked to problematic campus environment norms. • Overall Increase in Campus Activity Toward Promoting/Creating Supportive Campus • Decrease in instances of bystander-apathy/increase in instances of bystander-intervention (whenever someone calls to consult concerned about a student they are worried about, how many were students calling about other students, etc.) • Outreach Messages Showing up in student circles

24. Where To Go From Here? • Examining and Respecting Process so Concrete Examples emerge • Maintaining Flexibility and Creativity as Organization to accommodate different staff and program/presentation styles with need for organizing structure (some “required” measures?) • Reminder not to allow structure to stifle creativity and expansion of what constitutes outreach

25. What do YOU think? • Questions and Comments? • Continue the Conversation • Interested in participating in the subcommittee? • Provide input on a google.doc for follow up comments and questions to this workshop? • (sign email list and we’ll send you the link!) https://docs.google.com/document/d/1lVyAtAQs0BVYitDzYNx6lailDn5VaOpSRM6RK288QTc/edit?usp=sharing