Aggregating Outcomes for Effort and Effect: What NTAC Learned from its Site Review

1. Aggregating Outcomes for Effort and Effect:What NTAC Learned from its Site Review Ella L. Taylor, Ph.D. NTAC Teaching Research Institute Western Oregon University

2. Agenda • Site review response to evaluation • Revisions to evaluation plan • Collecting effort and effect data • Aggregating data • Questions

3. Site Review Concerns about Evaluation • Very complex • Less likely to succeed than a simpler plan • Needs to be simplified • Needs to be made more realistic and appropriate

4. Constraints of Evaluation • Everyone wants evaluation to be seamless and transparent • Reticence to see the direct benefit for improving the project • Evaluation is seen as cumbersome and confusing

5. Primary purpose of evaluation • Did we do what we said we were going to do? (effort) • What was the impact of what we did? (effect)

6. Effort vs. Effect • Effort – actions carried out by the project • Satisfaction data • Numbers of participants, events, etc. • Effect – impact of the actions • What outcome resulted from the activity? • Change of awareness • Change of knowledge • Change in skill/implementation (service provider, family, systems) • Change in child (child change data)

7. Alignment

8. Did we do what we said we were going to do? (EFFORT) • Grant objectives • Met = explain how we met them • Not met = explain why not • Data • Number of events • Number of participants • Satisfaction with effort

9. What was the impact of what we did? (EFFECT) • What to measure? • How to measure? • How to report succinctly? • How to aggregate?

10. What was the impact of what we did? (EFFECT) • Outcome and Performance Indicators (NTAC’s OPIs) • Outcome • A statement of a measurable condition or an expected result or change. (i.e. increase, improvement, progress toward). • Performance Indicator • A statement that helps quantify the outcome and indicates whether the outcome has been achieved. Often, multiple indicators may provide better evidence of the achievement of an outcome.

11. Kudos to… • John Killoran • Kathy McNulty • Paddi Davies • Many, many hours of development and refinement of NTAC’s OPIs

12. OPIs • Comprehensive outcomes for children (15), families (9), service providers (19) and systems (6) • Embed in all aspects of planning, delivery and evaluation • Aligns needs assessment, project activities and measurement of impact • On web (http://www.tr.wou.edu/ntac/evalforms/)

13. How we are using OPIs • Planning • Identify the stakeholder group (service provider, child, family and/or systems) • Identify the outcomes you will target • Identify the performance indicators that will help you determine attainment of targeted outcome(s) • Delivery of service • Implement TA that is targeted to the outcomes selected • Evaluation • Tailor assessment/evaluation measures to targeted outcomes and performance indicators

14. An Example

15. Webinar Example: Planning • Needs Assessment • Comments at Project Directors’ Meeting during self-evaluation breakout sessions • Conversations with state project directors, coordinators and staff • NTAC Advisory committee meeting • Outcome Goal: The use of formative and summative evaluation of the systems change and/or capacity building has increased (S4) • Performance Indicator: Uses outcomes measures (S4g)

16. Webinar Example: Delivery of Service • Activity: Webinar • Align the intensity of our evaluation with the intensity of the activity • One time activity results in less intense evaluation than sustained professional development. • Align needs with activity with evaluation

17. Webinar Example Evaluation: Change of awareness

18. A slightly more complicated example…

19. Example 2 • Planning/Needs: Ongoing training and support • Outcome: Use of formative and summative evaluation (Systems 4) • Performance Indicators: • Uses satisfaction measures (S4d) • Uses awareness, knowledge or skills measures (S4e) • Uses outcomes measures (S4g) • Activity: Series of webinars for one region • Evaluation: Change of knowledge & skill

20. Evaluation:Change of knowledge & skill

21. Another more complicated example…

22. Example 3 • Planning/Needs: Development, implementation, analysis & support in the use of formative and summative evaluation • Outcome: Use of formative and summative evaluation (Systems 4) • Performance indicators: • Uses participant demographic data (S4c) • Uses satisfaction measures (S4d) • Uses change in awareness, knowledge or skills measures (S4e) • Uses outcomes measures (S4g) • Uses formative and summative evaluation information for ongoing feedback and continuous improvement (S4j) • Disseminates evaluation results of the systems change or capacity building activities (S4k)

23. Sustained Professional Development • Activity: Multiple visits by TAS and evaluation specialist to 2 states • Evaluation: • Change of knowledge & skill • Follow-up

24. Follow-up

25. Aggregating the Data

26. What data did we collect? Number of events/activities (effort) Satisfaction (effort) Change of awareness (effect) Change of knowledge & skill (effect) Follow-up evaluation (effect) On what, did we collect data? Outcome: Use of formative & summative evaluation (Systems 4) Performance indicators: demographic data (S4c) satisfaction data (S4d) awareness, etc. (S4e) outcomes measures (S4g) ongoing feedback (S4j) dissemination (S4k) Data Collection

27. Data

28. Aggregate the data • Each event carries equal weight • Assign different weight to each event because events have different levels of importance • Follow-up evaluation should carry more weight because the intensity of the effort was greater

29. Conversion to 4 point scale • Convert 5 point scale to 4 point • 5 (strongly agree) = 4 (achieved) • 4 (agree) = 3 (nearly) • 3 (neither) = 2 (emerging) • 2 & 1 (disagree) = 1 (non-existent) • Convert 3 point scale to 4 point • 3 (substantial) = 4 (achieved) • 2 (some) = 3 (nearly) • 1 (no) = 1 (non-existent)

31. What does this mean? • Effort • NTAC conducted one national webinar, a series of webinars for one region, and several onsite consultations with two states to increase the states’ capacity to use formative and summative evaluation systems. Across the trainings, participants indicated 90% satisfaction with the skill of the consultants and the content of the activities.

32. What does this mean? • Effect • Across the trainings and consultations, participants report that they are very near achieving the ability to develop, implement, and analyze formative and summative evaluation measures to increase capacity and systems change (m = 3.03/4.0 scale). • Could elaborate by listing performance indicators if needed.

33. Response to NTAC Site Review: 2004 – 2005 Field test 1) Embed Outcomes and Performance Indicators (OPIs) in planning and delivery of service. 2) Embed OPIs in all evaluation measures. 3) Share our evaluation systems and data with our state/multi-state partners

34. Addressing the constraints • Aligning needs, delivery and evaluation through the OPIs yields a more seamless system (Constraint 1) • Sharing data facilitates the use of data (Constraint 2) • Consistency helps diminish confusion (Constraint 3)

35. Questions?

36. Region 1 Shawn Barnard Paddi Davies Region 2 Jon Harding Barb Purvis Region 3 Nancy Donta Amy Parker Region 4 Kathy McNulty Therese Madden Rose Contact

37. Additional Examples • The following information will not be shared during the discussion, but is being provided as additional material.

38. Weighting the events differently • Using the previous examples, let’s say that we believe the follow-up data should carry more weight since it indicates more long-term implementation and attainment of the outcome. • We want the follow-up evaluation to carry 40% of the weight.

40. Embed in basic evaluation (Service Provider 1a) • Satisfaction data • “I was satisfied with my opportunity to learn about the impact of deaf-blindness on an individual’s overall development (i.e. social, emotional, cognitive).” • Change of awareness • “I have increased my awareness about the impact of deaf-blindness on an individual’s overall development (i.e. social, emotional, cognitive).” • Change of knowledge/skill • “As a result of the training, I can use my knowledge about the impact of deaf-blindness on an individual’s overall development (i.e. social, emotional, cognitive) to plan instruction.”

41. Embed in Follow-up • Service Providers • “Based on the recent training provided on understanding how a combined vision and hearing loss impacts learning and social/emotional development, please indicate your progress in performing the following tasks…” • Child change • “Three months ago, you received technical assistance on understanding how a combined vision and hearing loss impacts learning and social/emotional development. As a result of that training, please indicate any progress the student has made in the following skills…”