march 10 2011 judith m ottoson ed d m p h l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
March 10, 2011 Judith M. Ottoson, Ed.D., M.P.H. PowerPoint Presentation
Download Presentation
March 10, 2011 Judith M. Ottoson, Ed.D., M.P.H.

Loading in 2 Seconds...

play fullscreen
1 / 34

March 10, 2011 Judith M. Ottoson, Ed.D., M.P.H. - PowerPoint PPT Presentation


  • 233 Views
  • Uploaded on

Evaluation Theory and Practice Framework : UCSF EPI246 Translating Evidence Into Practice: Individual-Centered Implementation Strategies . March 10, 2011 Judith M. Ottoson, Ed.D., M.P.H. Evaluation Questions. What is it?. Evaluation practice. Is it good or bad?.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'March 10, 2011 Judith M. Ottoson, Ed.D., M.P.H.' - yosefu


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
march 10 2011 judith m ottoson ed d m p h

Evaluation Theory and Practice Framework: UCSF EPI246Translating Evidence Into Practice: Individual-Centered Implementation Strategies

March 10, 2011

Judith M. Ottoson, Ed.D., M.P.H.

slide2

Evaluation Questions

What is it?

Evaluation

practice

Is it good or bad?

How do you know?

Of what use is

this information?

evaluation is
Evaluation is…...

… the systematic assessment of the operation and/or outcomes

of a program or policy, compared to a set of explicit or implicit

standards as a means of contributing to the improvement

of the program or policy

Weiss, p4

slide6

Steps

Engage Stakeholders

Ensure use and share lessons learned

Describe the program

Standards

Utility

Feasibility

Propriety

Accuracy

Focus the evaluation design

Justify conclusions

Gather credible evidence

Framework for Program Evaluation in Public Health, MMWR, 1999

the linchpin
The Linchpin

The fundamental purpose of evaluation theory is to specify feasible practices that evaluators can use to construct knowledge of the value of social programs that can be used to ameliorate the social problem to which programs are relevant.

Shaddish, Cook and Leviton, 1991, p36

slide8

Theories of

program evaluation

ProgramNeed/problem

- Structure

- Context

- Change process

  • Practice
  • Engage the stakeholders
  • Describe the program
  • Ask questions
  • Make values transparent
  • Focus the design
  • Gather credible evidence
  • Justify conclusions
  • Ensure use & lessons
  • -Manage evaluation

Feasibility

Accuracy

  • Valuing
  • - Criteria of success
  • Standards of success
  • Who decides?
  • Knowledge
  • -“Real” knowledge
  • Design: who? when?
  • Methods
  • - Analysis

Utility & accountability

Propriety

Use

-Kinds of use

-By when & who

-Reporting

-Dissemination

Adapted from:

Shadish, Cook, & Leviton,

Foundations of Program Evaluation, 1991

slide9

Theories of

program evaluation

ProgramNeed/problem

- Structure

- Context

- Change process

  • Practice
  • Engage the stakeholders
  • Describe the program
  • Ask questions
  • Make values transparent
  • Focus the design
  • Gather credible evidence
  • Justify conclusions
  • Ensure use & lessons
  • -Manage evaluation

Feasibility

Accuracy

  • Valuing
  • - Criteria of success
  • Standards of success
  • Who decides?
  • Knowledge
  • -“Real” knowledge
  • Design: who? when?
  • Methods
  • - Analysis

Utility & accountability

Propriety

Use

-Kinds of use

-By when & who

-Reporting

-Dissemination

Adapted from:

Shadish, Cook, & Leviton,

Foundations of Program Evaluation, 1991

program as evaluand
Program as Evaluand
  • What is the problem or need?
  • What is the “program?”
    • policy, program, project, component, element
    • before, during, after
  • Internal process and structure
  • External context
  • Change process – levers of change
types of program failure
Types of Program Failure

Desired

effect

Successful

Program

“causal

Process”

Program

Which

led to

Set in

motion

“causal

Process”

Desired

effect

Program Theory

failure

Which did

not lead to

Program

Set in

motion

“causal

process”

Desired

effect

Implementation

failure

Program

Did not

set in

motion

Which would

have led to

Source: Weiss, 1972

basic logic model
Basic Logic Model

Input

Resources &/or

barriers that enable

or limit program

effectiveness, e.g.

funds, people,

supplies, equipment

Activities/process

Activities, techniques,

tools, events, actions,

technology, e.g.

products, services,

infrastructure

Output

Size & scope of

services; dose

e.g. # materials,

rate participation,

# hours

Outcome

Short-term

Changes in attitudes,

behavior, knowledge,

skills, status, level

of functioning

Outcome

Long-term

Changes in organization,

community, &/or

system, e.g. improved

condition, capacity,

policy changes

Adapted from: W.F. Kellogg Foundation, Logic Model Development Guide: http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf

development strategies for logic models
Development Strategies for Logic Models
  • “But how?” questions: reverse logic
    • Start with distal effects
    • Ask: how to generate that effect
    • Create downstream of proximal effects & activities connected to needs
  • “But why?” questions: forward logic
    • Start with needs
    • Ask: so what happens next to create distal effects
    • Create upstream of activities and proximal effects connected to distal effects
  • “But how?” & “But why?” questions: middle-road logic
    • Start with activities or outputs
    • Ask: what will create downstream activities that connect to needs
    • Ask: but how will these activities or outputs create upstream distal effects
slide14

Referral from GP/Specialists

Patient Needs:

Common medical problem w/ great impact resulting in ↓ Quality of Life (QOL) and loss of work

Provincial Health Authority (PHA) Needs:

Save $$, ↓ length of stay (LOS), and ↓waiting list

Rapid Access Disc Herniation Program

(RADH)

Patient

PHA

Teaching

 Resources: CD, pamphlet, helpline, contact no.

 Small group discussion

 Paramedical interview and teaching: nurses, PT

Surgeons

 Knowledge  Training  Expertise with Minimally Invasive Surgery (MIS)

Surgery

Administration

 Support/ attitude

 Resources ($$,equipment)

 Facility

Practice

 Empowerment

Opportunity with involvement

 Planning care and activities after surgery

Output

# patients participated

# surveys

# follow-ups

# discharges

Output

# packages distributed

# procedures performed

# interviews with provider

Resources

 OR suited for MIS

 Day Surgery Unit

 Help line

Outcome

↑QOL

-pain reduction

-improved activities

Early RTW

Outcome

 ↓ LOS

 ↓waiting list

 ↓ $$

Impact

 Reallocation of hospital resources

Center of Excellence

 Raise standard of Disc treatment to Nat’l level

slide15

Theories of

program evaluation

ProgramNeed/problem

- Structure

- Context

- Change process

  • Practice
  • Engage the stakeholders
  • Describe the program
  • Ask questions
  • Make values transparent
  • Focus the design
  • Gather credible evidence
  • Justify conclusions
  • Ensure use & lessons
  • -Manage evaluation

Feasibility

Accuracy

  • Valuing
  • - Criteria of success
  • Standards of success
  • Who decides?
  • Knowledge
  • -“Real” knowledge
  • Design: who? when?
  • Methods
  • - Analysis

Utility & accountability

Propriety

Use

-Kinds of use

-By when & who

-Reporting

-Dissemination

Adapted from:

Shadish, Cook, & Leviton,

Foundations of Program Evaluation, 1991

focusing the evaluation by asking questions
Focusing the Evaluation by Asking Questions
  • Ask guiding (key) questions about the evaluand
    • Based on the logic model
    • Program level, not individual level
  • Questions are something in which multiple stakeholders can engage vs. developing “measures,” writing objectives, or study design
  • Questions become the guide to measures, analysis, & reporting
  • One question may cover multiple measures. Group evaluation results by the forest (questions), not the trees (measures)
determining the value of a program or policy
Determining the Valueof a program or policy
  • What is valued?
  • Who decides?
  • Prescribe or describe values
  • Making values transparent
  • Valuing logic
    • dimensions (criteria) of worth/merit
    • standards of worth/merit
    • performance
the logic of valuing

criteria = bar

Standard =how high

The logic of valuing
  • Determine criteria of “success”
    • the dimensions of the evaluand on which stakeholders…
      • …have questions
      • …identify as key, core, or essential
      • …are willing to hang evaluand success
    • Dimensions include: input, process, output, outcomes
    • Process ex: diversity, enrollees, materials, activities
    • Outcome ex: knowledge, behavior, jobs, scores, health status
  • Set standards of “success”
    • how well must performance be on the criteria
    • Ex: #, %, increase or decrease, spread,
  • Measure performance
slide19

Theories of

program evaluation

ProgramNeed/problem

- Structure

- Context

- Change process

  • Practice
  • Engage the stakeholders
  • Describe the program
  • Ask questions
  • Make values transparent
  • Focus the design
  • Gather credible evidence
  • Justify conclusions
  • Ensure use & lessons
  • -Manage evaluation

Feasibility

Accuracy

  • Valuing
  • - Criteria of success
  • Standards of success
  • Who decides?
  • Knowledge
  • -“Real” knowledge
  • Design: who? when?
  • Methods
  • - Analysis

Utility & accountability

Propriety

Use

-Kinds of use

-By when & who

-Reporting

-Dissemination

Adapted from:

Shadish, Cook, & Leviton,

Foundations of Program Evaluation, 1991

knowledge construction
Knowledge construction
  • Is evaluation knowledge special?
  • What counts as “real” evaluation knowledge to you? To others?
  • What are feasible, ethical, useful, and accurate ways to construct knowledge?
evaluation design basics
Evaluation Design Basics
  • Who? (sample)
  • When? (timing)
  • What? (answer the questions)
  • How? (data collection)
slide22

Theories of

program evaluation

ProgramNeed/problem

- Structure

- Context

- Change process

  • Practice
  • Engage the stakeholders
  • Describe the program
  • Ask questions
  • Make values transparent
  • Focus the design
  • Gather credible evidence
  • Justify conclusions
  • Ensure use & lessons
  • -Manage evaluation

Feasibility

Accuracy

  • Valuing
  • - Criteria of success
  • Standards of success
  • Who decides?
  • Knowledge
  • -“Real” knowledge
  • Design: who? when?
  • Methods
  • - Analysis

Utility & accountability

Propriety

Use

-Kinds of use

-By when & who

-Reporting

-Dissemination

Adapted from:

Shadish, Cook, & Leviton,

Foundations of Program Evaluation, 1991

evaluation use
Evaluation Use
  • Kinds of use
    • instrumental
    • conceptual
  • Who uses & when
  • Facilitators and obstacles to use
the program evaluation standards
The Program Evaluation Standards
  • key features
    • Standards identify and define evaluation quality
    • Guide evaluators and evaluation users in pursuit of evaluation quality
    • “laws” vs voluntary, consensus
  • Revised 2011
    • Clarifications
    • Now fifth standard of evaluation accountability
  • Trade-offs among standards
standards of program evaluation
Standards of Program Evaluation
  • Utility-- The utility standards support high quality evaluation use through attention to all aspects of an evaluation (8)
  • Feasibility -- The feasibility standards encourage evaluation to be effective and efficient. (4)
  • Propriety -- The propriety standards are intended to ensure that an evaluation will be proper, fair, legal, right, acceptable, and just. (7)
  • Accuracy -- Accuracy is the truthfulness of evaluation representations, propositions, and findings, which is achieved through sound theory, methods, designs, and reasoning. (8)
  • Evaluation Accountability-- Documenting and improving evaluation accountability requires similar efforts to those required for program accountability, i.e., an evaluation of the evaluation (metaevaluation) (3)
evaluation standards describing the program
Evaluation StandardsDescribing the program
  • U2: attention to stakeholders
  • F2: Practical procedures
  • F3: Contextual viability
  • P1: Responsive & inclusive orientation
  • P5: Transparency & disclosure
  • P6: Conflicts of interest
  • A2: Valid information
  • A3: Reliable information
  • A4: Explicit program & context descriptions
  • A7: Explicit evaluation reasoning
  • A8: Communication & reporting
  • E1: Evaluation documentation

The program evaluation

standards, 2011

evaluation practice debates
Evaluation Practice Debates
  • Whether to evaluate? Who wants the evaluation and why?
  • Do evaluators need to be content experts and/or evaluation experts?
  • What are the advantages and disadvantages of being an internal vs external evaluator?
  • Who counts as a stakeholder for you? Do you hold bias towards any stakeholders?
  • Evaluation design, data collection, analysis, and reporting.
  • Are evaluation and research the same?
program debates
Program Debates
  • What counts as the evaluand, e.g., process, structure, context, people, planning?
  • How do programs relate to policy (landscape) and components (portrait)?
  • What assumptions guide the program? How is it supposed to work?
  • What is the relationship between type of evaluation & stage of program development/ stability?
  • What factors influence incremental vs radical program change? What are the levers of change?
valuing debates
Valuing Debates
  • What kinds of values determine program value, e.g., instrumental, terminal?
  • Where are the values embedded, e.g. needs, goals, problems?
  • How might the use of prescriptive or descriptive values influence the evaluation?
  • Who decides on evaluation criteria and standards, e.g. evaluator? stakeholders?
  • Should evaluators synthesize or describe findings?
knowledge construction debates
Knowledge Construction Debates
  • What counts as “real” knowledge to you? To other stakeholders?
  • What is acceptable evidence of program success? To whom?
  • Is the quantitative / qualitative debate over for everybody?
  • Is it possible to combine these different understandings of knowledge construction?
evaluation use debates
Evaluation Use Debates
  • What counts as use of an evaluation: conceptual vs instrumental use?
  • How much? By when? With what fidelity should evaluation findings be used?
  • Who uses evaluation results?
  • What are the evaluator’s responsibilities towards use?
key evaluation resources
Key Evaluation Resources
  • American Evaluation Association
    • www. eval.org
    • November 2-5, 2011, Anaheim
  • San Francisco Bay Area Evaluators
    • www. sfbae.org/
  • The Evaluators Institute: http://tei.gwu.edu
    • April 4-16, 2011
references
References
  • Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR 1999;48 (no.RR-11) http://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm
  • Ottoson, J.M. & Martinez, D. (2010). An ecological understanding of evaluation use: A case study of the Active for Life evaluation. The Robert Wood Johnson Foundation. www.rwjf.org/pr/product.jsp?id=71148
  • Ottoson, J.M. & Hawe, P. (eds). (Winter 2009). Knowledge utilization, diffusion, implementation, transfer, and translation: Implications for evaluation. New Directions for Evaluation, 124. Jossey-Bass: San Francisco.
  • Shadish, W.R., Cook, T.D.,& Leviton, L.C. (1991). Good theory for social program evaluation. In The foundations of program evaluation. Newbury Park: Sage.
  • The Joint Committee on Standards for Educational Evaluation. (2011). The program evaluation standards (3rd ed.). Thousand Oaks: Sage.
  • Weiss, C.H. (1998) Evaluation. (2nd Ed.). New Jersey: Prentice-Hall.
  • W.K. Kellogg Foundation, Logic Model Development Guide. (2004). http://www.wkkf.org/knowledge-center/resources/2010/Logic-Model-Development-Guide.aspx