1 / 27

Evaluation Communities of Practice: Building capacity from the inside out

Evaluation Communities of Practice: Building capacity from the inside out. Ben Silliman, Department of Youth, Family, & Community Sciences NC State University. Communities of Practice. Informal networks (online) Focused on improvement, growth Characterized by…

garrettelkins
Download Presentation

Evaluation Communities of Practice: Building capacity from the inside out

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation Communities of Practice:Building capacity from the inside out Ben Silliman, Department of Youth, Family, & Community Sciences NC State University

  2. Communities of Practice • Informal networks (online) • Focused on improvement, growth • Characterized by… • Structured and spontaneous dialogue • Strong facilitation • Flexible levels of participation • Public and private spaces • Balance of routine and novelty • Flexible, interactive technology

  3. Sample Communities of Practice • NC Extension/Youth Development • Learning Circle 1 (2008-11) • Learning Circle 2 (2015- + E-Basics online training • Multi-state Extension System • eXtension/webinar (2009-15) • Evaluation Champions (2013-15) • Youth Development Evaluation (2015-

  4. NC 4-H Evaluation Learning Circles (E-LC) • Short-term projects (2006-07) • Evaluation research and publication on Conferences, Public Speaking, Camps • Long-term projects (2009-2012) • Evaluation of clubs, camp, science fair • New Learning Circle (2015- • Monthly training webinars • Small group projects and teaching tools

  5. NC 4-H E-LC1 • Learning of E-LC Members (2009-2012) (based on Evaluation Skills Self Assessment) • Pre-test: most E-LC Unfamiliar/Minimal Familiarity (Self-ratings 1.15-1.44 for phases) • Post-test: > 50% Implement with Assistance on most items; 25-49% on some items

  6. Results:Perceived Skill Improvement Cohort 1, Y1-2: ^p < .05 Cohort 1, Y1-3: +p < .05, * p < .01 (N = 7)

  7. Results:Perceived Skill Improvement Cohorts 1 & 2, Y1-2: +p < .05, *p < .01 (N = 15)

  8. Evaluation Champions StudyeXtension Evaluation CoP • Extension ECB Framework • Professional Development (training, TA, collaborative projects, mentoring/coaching, CoPs) • Resources and Supports (expertise, materials, champions, assets, financing, technology, time) • Organizational Environment (leadership, demand, incentives, structures, policies/procedures) Taylor-Powell, E., & Boyd, H.H. (2008). Evaluation capacity-building in complex organizations. In M.T. Braverman, M.Engle, M.E. Arnold, & R.A. Rennekamp (Eds.), Program evaluation in a complex organizational system: Lessons from Cooperative Extension. New Directions for Evaluation, 120, 55-69.

  9. Evaluation Champions StudyeXtension Evaluation CoP • Extension ECB Framework • Professional Development (training, TA, collaborative projects, mentoring/coaching, CoPs) • Resources and Supports (expertise, materials, champions, assets, financing, technology, time) • Organizational Environment (leadership, demand, incentives, structures, policies/procedures) Taylor-Powell, E., & Boyd, H.H. (2008). Evaluation capacity-building in complex organizations. In M.T. Braverman, M.Engle, M.E. Arnold, & R.A. Rennekamp (Eds.), Program evaluation in a complex organizational system: Lessons from Cooperative Extension. New Directions for Evaluation, 120, 55-69.

  10. Evaluation Champions StudyeXtension Evaluation CoP • Evaluation Champions (2013-15) • 4 states, purposive sample • 10/state; all disciplines, levels, settings • 30-40 min. phone interview Part 1 • What do you do? • What motivates you? (then and now) Part 2 • What do you need? (esp. technology) • How can your organization help?

  11. Evaluation Champions Part 1eXtension Evaluation CoP The reason I work with Extension is that I want to have an impact on my community and improve it, so measuring it and knowing that I have an impact is important to me. —county agent

  12. Evaluation Champions Part 1eXtension Evaluation CoP • Champions’ Roles • Advocate: embedded, strategic: talking to peers, administrators • Practitioner: exemplar, innovator engaging clients, managing resourcefully, measuring precisely but pragmatically, using data creatively…esp. to learn • Mentor/Trainer: investor, expert capturing ‘teachable moments’ in project work, professional development, personal contacts

  13. Evaluation Champions Part 1eXtension Evaluation CoP • Advocate I meet with other supervisors and remind them of the importance of evaluation and advocate for evaluation when we choose staff training priorities. —county director

  14. Evaluation Champions Part 1eXtension Evaluation CoP • Practitioner: Thinking I tend to be a science person and think about how and why things work, so evaluation is a natural part of science and Extension.—county agent

  15. Evaluation Champions Part 1eXtension Evaluation CoP • Practitioner: Managing I have partnered with faculty and their students in applied research fields at the university on several projects. —community development agent

  16. Evaluation Champions Part 1eXtension Evaluation CoP • Practitioner: Measuring In 4-H livestock with novice learners, I evaluated what they knew before, during, and after…then tracked 3-10 years and measured. —county agent

  17. Evaluation Champions Part 1eXtension Evaluation CoP • Practitioner: Using That [program] value is recognized in the individual stories, testimonies, quotes, and capturing themes from focus groups. —county agent

  18. Evaluation Champions Part 1eXtension Evaluation CoP • Mentor I supervise and mentor staff and require them to do projects. We start with logic models and identify appropriate points in a program to conduct evaluation. --county director

  19. Evaluation Champions Part 1eXtension Evaluation CoP • How Champions Roll • Personal commitment to field, people, CES • Few with formal, extensive training • More with personal, practical experience Also… • Early, intensive, relevant training, program/ evaluation success provides intrinsic rewards • Organization rules, positive feedback from stakeholders offers extrinsic support VS… • Inconsistent process, unused results

  20. Evaluation Champions Part 1eXtension Evaluation CoP Looking at evaluation from a broader perspective I could see that it was not just ‘bean counting,’ but could help me, my supervisor, and the system to understand stakeholder needs and Extension impacts better and decide what programs should be continued or modified.—county agent

  21. Evaluation Champions Part 2eXtension Evaluation CoP • What Champions Want (and see as needs for everyone) • Support and Resources • Basic  Advanced training • Team support, innovation • Tools for all phases of the evaluation cycle • Rewards for advancement, innovation • Policies and Practices • Value and reward

  22. Evaluation Champions Part 2eXtension Evaluation CoP • What Champions Want …maybe a video that discusses a specific topic like how to analyze data in Excel and then how to effectively present the results… We need to do a better job in thinking about how to include evaluation before we begin a program. If there was a tool that helped us do that, it would be a great help.—county agent

  23. Evaluation Champions Part 2eXtension Evaluation CoP • What Champions Want (and see as needs for everyone) • Technology • Tools and equipment for doing evaluation • Efficient/effective data and reporting systems • Tools for learning evaluation, resource materialsand people

  24. Evaluation Champions Part 2eXtension Evaluation CoP • What Champions Want Last year I hired a videographer to capture the message in a way that stakeholders get excited about.—county agent

  25. Evaluation Champions Part 2eXtension Evaluation CoP • What Organizations can do • Talk the Talk • Clear, consistent emphasis • Walk the Walk • Consistent training, implement, rewards • Get out of the way • Remove barriers, inconsistencies • Support advocacy, practice, training/mentoring (informal, formal)

  26. Evaluation Champions Part 2eXtension Evaluation CoP • What Champions Can Do Maybe it’s a matter of changing perceptions that it’s everybody’s job and everybody can do it. —county agent

  27. Literature Review:Extension Capacity-building • Extension ECB Framework • Professional Development (training, TA, collaborative projects, mentoring/coaching, CoPs) • Resources and Supports (expertise, materials, champions, assets, financing, technology, time) • Organizational Environment (leadership, demand, incentives, structures, policies/procedures) Taylor-Powell, E., & Boyd, H.H. (2008). Evaluation capacity-building in complex organizations. In M.T. Braverman, M.Engle, M.E. Arnold, & R.A. Rennekamp (Eds.), Program evaluation in a complex organizational system: Lessons from Cooperative Extension. New Directions for Evaluation, 120, 55-69.

More Related