aggregating outcomes for effort and effect what ntac learned from its site review n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Aggregating Outcomes for Effort and Effect: What NTAC Learned from its Site Review PowerPoint Presentation
Download Presentation
Aggregating Outcomes for Effort and Effect: What NTAC Learned from its Site Review

Loading in 2 Seconds...

play fullscreen
1 / 41

Aggregating Outcomes for Effort and Effect: What NTAC Learned from its Site Review - PowerPoint PPT Presentation


  • 142 Views
  • Uploaded on

Aggregating Outcomes for Effort and Effect: What NTAC Learned from its Site Review. Ella L. Taylor, Ph.D. NTAC Teaching Research Institute Western Oregon University. Agenda. Site review response to evaluation Revisions to evaluation plan Collecting effort and effect data Aggregating data

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Aggregating Outcomes for Effort and Effect: What NTAC Learned from its Site Review' - bunny


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
aggregating outcomes for effort and effect what ntac learned from its site review

Aggregating Outcomes for Effort and Effect:What NTAC Learned from its Site Review

Ella L. Taylor, Ph.D.

NTAC

Teaching Research Institute

Western Oregon University

agenda
Agenda
  • Site review response to evaluation
  • Revisions to evaluation plan
  • Collecting effort and effect data
  • Aggregating data
  • Questions
site review concerns about evaluation
Site Review Concerns about Evaluation
  • Very complex
  • Less likely to succeed than a simpler plan
  • Needs to be simplified
  • Needs to be made more realistic and appropriate
constraints of evaluation
Constraints of Evaluation
  • Everyone wants evaluation to be seamless and transparent
  • Reticence to see the direct benefit for improving the project
  • Evaluation is seen as cumbersome and confusing
primary purpose of evaluation
Primary purpose of evaluation
  • Did we do what we said we were going to do? (effort)
  • What was the impact of what we did? (effect)
effort vs effect
Effort vs. Effect
  • Effort – actions carried out by the project
    • Satisfaction data
    • Numbers of participants, events, etc.
  • Effect – impact of the actions
    • What outcome resulted from the activity?
      • Change of awareness
      • Change of knowledge
      • Change in skill/implementation (service provider, family, systems)
      • Change in child (child change data)
did we do what we said we were going to do effort
Did we do what we said we were going to do? (EFFORT)
  • Grant objectives
    • Met = explain how we met them
    • Not met = explain why not
  • Data
    • Number of events
    • Number of participants
    • Satisfaction with effort
what was the impact of what we did effect
What was the impact of what we did? (EFFECT)
  • What to measure?
  • How to measure?
  • How to report succinctly?
  • How to aggregate?
what was the impact of what we did effect1
What was the impact of what we did? (EFFECT)
  • Outcome and Performance Indicators (NTAC’s OPIs)
  • Outcome
    • A statement of a measurable condition or an expected result or change. (i.e. increase, improvement, progress toward).
  • Performance Indicator
    • A statement that helps quantify the outcome and indicates whether the outcome has been achieved. Often, multiple indicators may provide better evidence of the achievement of an outcome.
kudos to
Kudos to…
  • John Killoran
  • Kathy McNulty
  • Paddi Davies
  • Many, many hours of development and refinement of NTAC’s OPIs
slide12
OPIs
  • Comprehensive outcomes for children (15), families (9), service providers (19) and systems (6)
  • Embed in all aspects of planning, delivery and evaluation
  • Aligns needs assessment, project activities and measurement of impact
  • On web (http://www.tr.wou.edu/ntac/evalforms/)
how we are using opis
How we are using OPIs
  • Planning
    • Identify the stakeholder group (service provider, child, family and/or systems)
    • Identify the outcomes you will target
    • Identify the performance indicators that will help you determine attainment of targeted outcome(s)
  • Delivery of service
    • Implement TA that is targeted to the outcomes selected
  • Evaluation
    • Tailor assessment/evaluation measures to targeted outcomes and performance indicators
webinar example planning
Webinar Example: Planning
  • Needs Assessment
    • Comments at Project Directors’ Meeting during self-evaluation breakout sessions
    • Conversations with state project directors, coordinators and staff
    • NTAC Advisory committee meeting
  • Outcome Goal: The use of formative and summative evaluation of the systems change and/or capacity building has increased (S4)
    • Performance Indicator: Uses outcomes measures (S4g)
webinar example delivery of service
Webinar Example: Delivery of Service
  • Activity: Webinar
  • Align the intensity of our evaluation with the intensity of the activity
    • One time activity results in less intense evaluation than sustained professional development.
  • Align needs with activity with evaluation
example 2
Example 2
  • Planning/Needs: Ongoing training and support
    • Outcome: Use of formative and summative evaluation (Systems 4)
    • Performance Indicators:
      • Uses satisfaction measures (S4d)
      • Uses awareness, knowledge or skills measures (S4e)
      • Uses outcomes measures (S4g)
  • Activity: Series of webinars for one region
  • Evaluation: Change of knowledge & skill
example 3
Example 3
  • Planning/Needs: Development, implementation, analysis & support in the use of formative and summative evaluation
  • Outcome: Use of formative and summative evaluation (Systems 4)
  • Performance indicators:
    • Uses participant demographic data (S4c)
    • Uses satisfaction measures (S4d)
    • Uses change in awareness, knowledge or skills measures (S4e)
    • Uses outcomes measures (S4g)
    • Uses formative and summative evaluation information for ongoing feedback and continuous improvement (S4j)
    • Disseminates evaluation results of the systems change or capacity building activities (S4k)
sustained professional development
Sustained Professional Development
  • Activity: Multiple visits by TAS and evaluation specialist to 2 states
  • Evaluation:
    • Change of knowledge & skill
    • Follow-up
data collection
What data did we collect?

Number of events/activities (effort)

Satisfaction (effort)

Change of awareness (effect)

Change of knowledge & skill (effect)

Follow-up evaluation (effect)

On what, did we collect data?

Outcome: Use of formative & summative evaluation (Systems 4)

Performance indicators:

demographic data (S4c)

satisfaction data (S4d)

awareness, etc. (S4e)

outcomes measures (S4g)

ongoing feedback (S4j)

dissemination (S4k)

Data Collection
aggregate the data
Aggregate the data
  • Each event carries equal weight
  • Assign different weight to each event because events have different levels of importance
    • Follow-up evaluation should carry more weight because the intensity of the effort was greater
conversion to 4 point scale
Conversion to 4 point scale
  • Convert 5 point scale to 4 point
    • 5 (strongly agree) = 4 (achieved)
    • 4 (agree) = 3 (nearly)
    • 3 (neither) = 2 (emerging)
    • 2 & 1 (disagree) = 1 (non-existent)
  • Convert 3 point scale to 4 point
    • 3 (substantial) = 4 (achieved)
    • 2 (some) = 3 (nearly)
    • 1 (no) = 1 (non-existent)
what does this mean
What does this mean?
  • Effort
    • NTAC conducted one national webinar, a series of webinars for one region, and several onsite consultations with two states to increase the states’ capacity to use formative and summative evaluation systems. Across the trainings, participants indicated 90% satisfaction with the skill of the consultants and the content of the activities.
what does this mean1
What does this mean?
  • Effect
    • Across the trainings and consultations, participants report that they are very near achieving the ability to develop, implement, and analyze formative and summative evaluation measures to increase capacity and systems change (m = 3.03/4.0 scale).
    • Could elaborate by listing performance indicators if needed.
response to ntac site review 2004 2005 field test
Response to NTAC Site Review: 2004 – 2005 Field test

1) Embed Outcomes and Performance Indicators (OPIs) in planning and delivery of service.

2) Embed OPIs in all evaluation measures.

3) Share our evaluation systems and data with our state/multi-state partners

addressing the constraints
Addressing the constraints
  • Aligning needs, delivery and evaluation through the OPIs yields a more seamless system (Constraint 1)
  • Sharing data facilitates the use of data (Constraint 2)
  • Consistency helps diminish confusion (Constraint 3)
contact
Region 1

Shawn Barnard

Paddi Davies

Region 2

Jon Harding

Barb Purvis

Region 3

Nancy Donta

Amy Parker

Region 4

Kathy McNulty

Therese Madden Rose

Contact
additional examples
Additional Examples
  • The following information will not be shared during the discussion, but is being provided as additional material.
weighting the events differently
Weighting the events differently
  • Using the previous examples, let’s say that we believe the follow-up data should carry more weight since it indicates more long-term implementation and attainment of the outcome.
  • We want the follow-up evaluation to carry 40% of the weight.
embed in basic evaluation service provider 1a
Embed in basic evaluation (Service Provider 1a)
  • Satisfaction data
    • “I was satisfied with my opportunity to learn about the impact of deaf-blindness on an individual’s overall development (i.e. social, emotional, cognitive).”
  • Change of awareness
    • “I have increased my awareness about the impact of deaf-blindness on an individual’s overall development (i.e. social, emotional, cognitive).”
  • Change of knowledge/skill
    • “As a result of the training, I can use my knowledge about the impact of deaf-blindness on an individual’s overall development (i.e. social, emotional, cognitive) to plan instruction.”
embed in follow up
Embed in Follow-up
  • Service Providers
    • “Based on the recent training provided on understanding how a combined vision and hearing loss impacts learning and social/emotional development, please indicate your progress in performing the following tasks…”
  • Child change
    • “Three months ago, you received technical assistance on understanding how a combined vision and hearing loss impacts learning and social/emotional development. As a result of that training, please indicate any progress the student has made in the following skills…”