Evaluating program outcomes
1 / 36

- PowerPoint PPT Presentation

  • Uploaded on

Evaluating Program Outcomes. Farrokh Alemi, Ph.D. July 04, 2004. Your Experience. Do you have experience collecting satisfaction surveys? Using the data from satisfaction surveys? What do these data tell you? Are they useful?. Objectives.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about '' - albert

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Evaluating program outcomes

Evaluating Program Outcomes

Farrokh Alemi, Ph.D.

July 04, 2004

Your experience
Your Experience

  • Do you have experience collecting satisfaction surveys?

  • Using the data from satisfaction surveys?

  • What do these data tell you? Are they useful?

Farrokh Alemi, Ph.D.


  • Use statistical process control to evaluate effectiveness of programs?

  • Use satisfaction surveys and health status measures to examine impact of care

  • Understand how program evaluations can go wrong

Farrokh Alemi, Ph.D.

Evaluating changing programs
Evaluating Changing Programs

  • Ideal program evaluation

    • Program is constant

    • Patients are assigned randomly

    • Takes months to complete

  • Continuous Program Evaluation

    • On-going data and timely results

    • Allows continuous change in the program

    • Few evaluation requirements

Farrokh Alemi, Ph.D.

Study design
Study Design

Farrokh Alemi, Ph.D.

Evaluation measures
Evaluation Measures

Farrokh Alemi, Ph.D.

Why measure satisfaction
Why measure satisfaction?

  • Give customers a voice

  • Evaluate based on customer’s own values

  • Measure facet of care not easily examined

    • compassionate bedside skills

    • efficient attendance to needs

    • participation in decision-making

    • adequate communication and information

  • Change market share

Farrokh Alemi, Ph.D.


  • Not too rare as death or other adverse events

  • Available on all disease for all institutions

  • Not likely to be affected by case mix, though affected by expectations

  • Positive outcome

Farrokh Alemi, Ph.D.

Definition of patient satisfaction
Definition of Patient Satisfaction

  • Pascoe envisages patient satisfaction as healthcare recipients‘ reactions to their care.

  • A reaction that is composed of both a cognitive evaluation and an emotional response.

  • No right or wrong, all reactions are valid.

Farrokh Alemi, Ph.D.

Process of making satisfaction judgments
Process of Making satisfaction Judgments

  • Each patient begins with a comparison standard against which care is judged.

  • Standard can be an ideal care, a minimal expectation, an average of past experiences, or a sense of what one deserves.

  • The patient can assimilate discrepancies between this expected and actual care.

  • What is not assimilated affects patient ratings of satisfaction

We do not knowhow to measure


Farrokh Alemi, Ph.D.

Examples of satisfaction surveys
Examples of Satisfaction Surveys

  • Patient Satisfaction Questionnaire (PSQ)

  • Patient Judgments of Hospital Quality Questionnaire (PJHQ)

  • Medline database includes surveys specific to different clinical areas

Farrokh Alemi, Ph.D.

Psq contains eight dimensions

Technical care

Interpersonal behavior



Continuity of care



PSQ Contains Eight Dimensions

Farrokh Alemi, Ph.D.

Patient judgments of hospital quality questionnaire

Nursing care

Medical care

Hospital environment


Admission procedures

Discharge procedures


Patient Judgments of Hospital Quality Questionnaire

Farrokh Alemi, Ph.D.

Constructing a satisfaction instrument
Constructing a Satisfaction Instrument

  • Let a focus group to generate the questions.

  • Group the questions into general dimensions

  • Survey a large sample of patients

  • Drop questions with skewed distributions or with a high rate of missing responses

  • Create a shorter version by dropping highly correlated items

  • Construct validity of the questions by comparing responses to objective measures

Farrokh Alemi, Ph.D.

Problems with satisfaction measures

Problems with Satisfaction Measures

Are patients satisfied with their overall care or a specific agent of care?

Problems with satisfaction measures1

Problems with Satisfaction Measures

Are reports of satisfaction biased by patients' respect, trust, confidence, and gratitude to their doctors, nurses, and healthcare?

Problems with satisfaction measures2

Problems with Satisfaction Measures

Statistical analyses of satisfaction ratings suggest that technical and interpersonal dimensions are not always evaluated independently by patients

Are we measuring quality of care or quality of the cure?

Problems with satisfaction measures3

Problems with Satisfaction Measures

The earlier satisfaction is measured, the higher the satisfaction rating

Problems with satisfaction measures4

Problems with Satisfaction Measures

Most people are satisfied and to discover impact large databases are needed to detect small changes

Focus on time to dissatisfied customer

Patients health status
Patients’ Health Status

  • Rely on client’s self report

  • Measure ability to function not preferences for life styles

  • Available on numerous diseases across many institutions

Farrokh Alemi, Ph.D.

Patients health status1
Patients’ Health Status

  • Many disease-specific measures are available. See Medline for details

  • A widely used example is the general SF-36 and its shortened version SF-12

Farrokh Alemi, Ph.D.

Components of sf 36
Components of SF-36

Farrokh Alemi, Ph.D.

Reliability validity
Reliability & Validity

  • More than 4000 studies

  • With few exceptions, overall reliability exceeding 70%

  • Numerous studies showing the validity of the instrument in differentiating among sick and well patients

  • Translated and used widely

Farrokh Alemi, Ph.D.

Reliability of components
Reliability of Components

Farrokh Alemi, Ph.D.

Problems with health status measures

Problems with Health Status Measures

Patients’ perception of adequacy of their health status should depend on their life style choices. SF-36 does not take this into account.

Assumptions in use of statistical process control tools
Assumptions in Use of Statistical Process Control Tools

  • Data needs to be collected over multiple time periods and not just before and after the intervention

    • Patients are likely to be recruited over multiple time periods any way

  • Data need to be collected from both the experimental and control groups

Farrokh Alemi, Ph.D.

Use of statistical process controls
Use of Statistical Process Controls

  • Process limits are set based on our expectations

    • Calculated from patterns among outcomes of the control group

  • Observed rates are compared to expected limits

Farrokh Alemi, Ph.D.

Control cases are weighted based on their similarity to experimental group
Control Cases are Weighted Based on Their Similarity to Experimental Group

  • O = (∑ j=1, …, M Wj Oj)/ ∑ j=1, …, M Wj

  • S = [∑ j=1, …, M Wj (Oj - O)2/(-1+∑ j=1, …, M Wj)]0.5

  • Upper limit = O + 3 S

  • Lower limit = O - 3 S

Farrokh Alemi, Ph.D.

Limitation poorly defined populations
Limitation: Experimental Group poorly defined populations

  • Threat of poorly defined populations. It is not reasonable to compare patients and organization with such differences to each other

  • How this threat is addressed?  The proposed method weighs cases similar to the program site more heavily. It formalizes what is implicitly always done: It allows the investigators to specify the characteristic on which cases must be matched. 

Farrokh Alemi, Ph.D.

Limitation treatment contamination
Limitation: Experimental Group treatment contamination

  • Potential threat of treatment contamination.  Typically one expects that well-defined interventions be used to address construct validity and to protect against diffusion, contamination and imitation of treatments by the comparison groups.  This ensures that any "significant" difference can indeed be ascribed to a specific intervention. 

  • How this threat is addressed?  In the proposed evaluation, if the program has led to improved outcomes, then improvements will be detected when data are compared to historical trends.  Furthermore, since cycles of improvement are introduced at specific points in time, we will be able to attribute the improvement to specific modification of the program.   

Farrokh Alemi, Ph.D.

Limitation poorly defined control group
Limitation: Experimental Group poorly defined control group

  • Potential threat of poorly defined control group. Changes in "control" may introduce biases that limit our understanding maturation and regression towards the mean in the control group.  Moreover, it might generate variability. 

  • How the threat is addressed?  If the observed improvement is due to aging or maturation of the clients, such differences are also occurring in the control cases.  Similarly regression towards the mean occurs for both the program and the control cases.  So the proposed design protects against both maturation and regression towards the mean.

Farrokh Alemi, Ph.D.

Limitation baseline differences
Limitation: Experimental Group baseline differences

  • Potential threat of baseline differences.  Clients differ in their baseline values.  Some are at high risk and others are at low risk for incidence of adverse outcomes.  Where you end up, in part depends where you started.  To ignore these differences would brand effective programs as useless.

  • How this threat is addressed? We gather data on baseline.  Patients are asked to provide data prior to intervention and after the intervention.  The analysis compares outcomes to historical trends.

Farrokh Alemi, Ph.D.

Limitation not having a replication
Limitation: Experimental Group not having a replication

  • Potential threat of not having a replication.  In the Western scientific method, there is an absolute requirement for replication of scientific findings.  Without replication, it is impossible to refute the claim that any "result" was simply due to chance variation among study subjects.

  • How this is addressed? Though the intervention is change, there is a replication of a different sort going on in the proposed approach. If outcomes have not changed, then we consider the intervention to be a replication of the earlier one.

Farrokh Alemi, Ph.D.

Take home lessons

Take Home Lessons Experimental Group

You can evaluate outcomes of changing programs (e.g. patients satisfaction and patients’ health status) using statistical process controls

Limitation: Experimental Group

Farrokh Alemi, Ph.D.