Building Bridges: Embedding outcome evaluation in national and state TA delivery Ella Taylor Diane Haynes John Killor

1. Building Bridges: Embedding outcome evaluation in national and state TA deliveryElla TaylorDiane HaynesJohn KilloranSarah BeairdAugust 1, 2006

2. Session goals Situate classroom based technical assistance within the broader framework of teacher education Provide a rationale for the development of outcomes and performance indicators for children who are deaf-blind and their service providers Share how classroom based TA can use the OPIs, in conjunction with the IEP, to maximize student outcomes Demonstrate a relational database that facilitates the management and utility of the process Address participant questions ET will go over the sessions goals and introduce everyone.ET will go over the sessions goals and introduce everyone.

3. It�s all about the outcomes! Similarities TA providers & education professionals are being required to demonstrate the impact of our efforts TA providers & education professionals must do this within a standards based environment that demands learning for all students Differences Education professionals usually operate within only one venue (i.e., one state, one school district) TA providers operate across multiple venues Issue for TA providers is the use of a common language across the multiple venues I want to situate what we�re doing within the broader framework of educationI want to situate what we�re doing within the broader framework of education

4. As TA providers our goal is to help teachers and service providers with each aspect of their classroom in order to maximize student learning. We must also figure out how we demonstrate our impact across multiple classrooms, multiple students, and multiple service providers. Within the packet will be a more detailed version of this slide that explains each of these components. I�ll make the connections between what we�ll be talking about and how that fits in with classrooms designed for student learning.As TA providers our goal is to help teachers and service providers with each aspect of their classroom in order to maximize student learning. We must also figure out how we demonstrate our impact across multiple classrooms, multiple students, and multiple service providers. Within the packet will be a more detailed version of this slide that explains each of these components. I�ll make the connections between what we�ll be talking about and how that fits in with classrooms designed for student learning.

5. Measuring impact across and within states Outcome evaluation system comprised of Targeted outcome statements for each group of stakeholders (i.e., children, families, service providers, and systems). Performance indicators associated with each outcome that help quantify how the expected result or change will be measured. A TA process that embeds the targeted outcomes and performance indicators at the beginning of the TA cycle. A computerized relational database system that ties the planning, delivery and evaluation of TA together. I�ll explain that John will explain the development of the outcome and performance indicators. Sarah will share how all these components come together in a relational database system.I�ll explain that John will explain the development of the outcome and performance indicators. Sarah will share how all these components come together in a relational database system.

6. Measuring impact within the classroom Extend outcome evaluation system to include IFSP outcomes IEP benchmarks State academic standards Alternate assessment components Transition targets Clearly align and connect these components within the outcome evaluation system to facilitate their utility for the education professional I�ll explain that Diane will explain this in a few momentsI�ll explain that Diane will explain this in a few moments

7. NTAC�s OPI System Provides a consistent coding system used in the development and implementation of technical assistance for ongoing collection, analysis & reporting of data. The system was developed across four domains: Service Providers Families Systems Children John begins his sectionJohn begins his section

8. How to demonstrate impact across trainings, individuals, students? Outcome A statement of a measurable condition or an expected result or change. (i.e. increase, improvement, progress toward). Performance Indicator A statement that helps quantify the outcome and indicates whether the outcome has been achieved. Often, multiple indicators may provide better evidence of the achievement of an outcome.

9. A Little History of the NTAC OPI System�

10. Federal emphasis on collecting and reporting outcome results has been challenging for many TA & D projects � including NTAC! The need for a systematic process of collecting and reporting outcome information (effect data) was identified. NTAC responded through a comprehensive four year process of identifying measurable outcomes for service providers, children/youth, families, and systems; developing instruments for collecting the data; field-testing and redesigning the instruments multiple times; and, analyzing the outcome data. Multiple stakeholders were involved in this endeavor including; NTAC staff, NTAC advisory board members, State deaf-blind project personnel, Families and consumers, and Higher education faculty and researchers in the field.

11. NTAC�s OPIs Comprehensive outcomes for� children (16) families (12) service providers (22) systems (8) Multiple performance indicators for each outcome Embedded in all aspects of planning, delivery and evaluation Aligns needs assessment, delivery of TA and measurement of impact On web (http://www.tr.wou.edu/ntac) FileMaker Pro templates

12. Putting it all together and beginning a system� The NTAC Outcomes & Performance Indicator (OPI) System� A process created to assist with� planning collecting aggregating, and reporting effort, effect and outcome data

13. How NTAC uses the OPIs� Needs assessment and planning Identify the stakeholder group service provider child family, and/or systems Identify the outcome(s) to be targeted Identify the performance indicators that will help determine attainment of targeted outcome(s)

14. How NTAC uses the OPIs� Delivery of service Implement technical assistance that is targeted to the outcomes selected through� needs assessment, and planning process

15. Evaluation Tailor assessment/evaluation measures to� the intensity of the TA targeted outcomes, and performance indicators How NTAC uses the OPIs�

16. More on evaluation� Align the intensity of the evaluation with the intensity of the activity One time activity results in less intense evaluation than sustained professional development. Align needs (OPIs) of the activity with evaluation

17. Incorporating outcomes into project activities�

18. Examples of incorporating OPIs into evaluation Change of knowledge �As a result of the training, I have increased my knowledge about the receptive and expressive array of language and communication skills for children with deaf-blindness.� (SP.07.a) Follow-up (Service Providers) �Based on the recent training provided on use of strategies to improve communication and/or language, please indicate your progress in performing the following tasks�� responding to the child�s expressive communication attempts (SP.07.d) implementing expressive communication strategies across environments (SP.07.e) Either John can continue or I can pick up from here to talk a little bit more about how we do evaluationEither John can continue or I can pick up from here to talk a little bit more about how we do evaluation

19. Examples of incorporating OPIs into evaluation Follow-up (Child change) �Three months ago, you received technical assistance on understanding how a combined vision and hearing loss impacts learning and social/emotional development. As a result of that training, please indicate any progress the student has made in the following skills�� Rate of expressive communication (C.04.a) Use of multiple communication intents (C.04.b)

20. Embedding the OPIs into initial Change of Knowledge evaluation

21. Embedding the OPIs into Follow-up evaluation

22. Embedding child change OPIs into the follow-up evaluation

23. Reporting Example Effect Language Percentage of service providers reporting an increase in knowledge about strategies to improve communication and language for students who are deaf-blind (8 out of 10 = 80%) Percentage of service providers reporting an increase in skills to implement strategies to improve communication and language for students who are deaf-blind (5 out of 6 = 83%)

24. APR Example Effect Language Mean rating of students demonstrating a change in receptive communication (4.0 on a 5.0 scale) Mean rating of students demonstrating a change in expressive communication (3.5 on a 5.0 scale)

25. Using the OPIs to develop the �big picture� within a state Diane�s section beginsDiane�s section begins

26. Building the �big picture� Needs assessment Uses outcome statements from the child OPIs to focus on the student first Action plan Uses performance indicators to target specific impact on child Incorporates IFSP/IEP, state standards, alternate assessment to clearly align with state mandates Action plan for service provider Uses service provider outcomes and performance indicators to link impact on student learning to effort of service provider Evaluation Clearly articulates how student learning/progress will be measured This is where Diane would walk the participants through each aspect of these forms. They will have a handout that replicates the KY templates.This is where Diane would walk the participants through each aspect of these forms. They will have a handout that replicates the KY templates.

27. Linkages Needs assessment & action plan provide clear direction for service provider and/or family Needs assessment helps TA providers determine common needs across the state Action plan identifies common strategies and measures of impact on student learning (outcomes) Evaluation can be used by both the service provider and the TA provider Diane would talk about connections between what is good for service provider is also good TA (and good for TA providers).Diane would talk about connections between what is good for service provider is also good TA (and good for TA providers).

28. Keeping track FileMaker Pro database Sarah begins FMP demonstrationSarah begins FMP demonstration

29. Questions?