Using consumer surveys for quality management a modified pes as one option
1 / 24

Using Consumer Surveys for Quality Management: A Modified PES as One Option - PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Using Consumer Surveys for Quality Management: A Modified PES as One Option. Presentation to Cash & Counseling States. Sara Galantowicz July 11, 2006. Agenda. Consumer Surveys and Quality Management Modified Participant Experience Survey as one Survey Option PES Overview ME CD-PAS version

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Using Consumer Surveys for Quality Management: A Modified PES as One Option

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Using Consumer Surveys for Quality Management: A Modified PES as One Option

Presentation to Cash & Counseling States

Sara Galantowicz

July 11, 2006


  • Consumer Surveys and Quality Management

  • Modified Participant Experience Survey as one Survey Option

    • PES Overview

    • ME CD-PAS version

    • WV hybrid

  • Additional Issues for Consideration

Consumer Surveys for Self-Direction

Consumer Surveys

  • Consumer surveys are one of many sources of quality data

    • Consumer experience

    • Knowledge, attitudes, and practices

    • Satisfaction

  • Where you want to end up is where you want to start

    • Research questions

    • Data gaps

    • Suspected problem areas

    • Mandates

    • Changes/policies requiring evidentiary support

    • CMS and other reporting

    • Who needs to know what

Consumer Surveys for Self-Direction

Consumer Surveys Options

  • Designing your own instrument

    • Flexible

    • Specific

    • Reflects stakeholder input

  • Using/modifying existing instruments

    • Stealing can be good!

      • Efficient

      • Testing

      • Comparability

  • Either approach requires involving the right players and technical expertise

Consumer Surveys for Self-Direction

Looking at Existing Tools

  • Multiple tools available

  • Vary on several dimensions

    • Target population

    • Domains or areas of emphasis

    • Method of administration

    • Role of proxy respondents

    • Length of survey

    • Resulting measures

  • Evaluation Criteria

    • Scope

    • Testing

      • Reliability and validity

    • Cost

    • Availability of benchmark data and support

Consumer Surveys for Self-Direction

Participant Experience Survey

PES is a technical assistance tool for States

  • Solicits participant feedback on HCBS waiver services/supports

  • Generates indicators for QA/QI

  • Developed by The Medstat Group, Inc. for CMS/DHHS

  • Experience survey for participants only

    • Not a satisfaction survey

    • No proxy version

    • Designed to be administered in-person

    • Optional

  • Extensively tested before release

    • Cognitive testing of individual items

    • Field testing to evaluate administration and tool properties

  • Consumer Surveys for Self-Direction

    PES Development

    • Combination of quantitative and qualitative methods

      • Work Group provided technical assistance and oversight, and specified priority areas

      • Review of related instruments

      • Draft items to operationalize domains

      • Cognitive testing (Phase I)

      • Field testing (Phase II)

    Consumer Surveys for Self-Direction

    PES: Current Versions

    PES E/D

    • Frail elderly and adults with physical disabilities

    • 33 indicators

    • Free automated survey software available

      • Responses entered directly into laptops

      • Pull-down menus for data analysis


    • Adults with mental retardation or developmental disabilities

    • 51 indicators

    • Includes 8 “core” questions for participants with severe cognitive impairments

    • Automated version

  • PES BI

    • Adults with acquired brain injuries

    • Up to 58 indicators (two optional modules)

  • Consumer Surveys for Self-Direction

    PES: Domains

    • Access to Care

      • Are participants’ needs for personal assistance, adaptive equipment, and case manager access being met?

    • Choice and Control

      • Do program participants have input into the types of services they receive and who provides them?

    • Respect and Dignity

      • Are program participants treated with respect by providers?

    • Community Integration

      • Do program participants engage in activities and events of their choice outside their homes, when they want to?

    Consumer Surveys for Self-Direction

    Sample Items

    • Access to Care

      • Do you ever go without eating when you need to?

      • Is this because there is no one there to help you?

      • Have you ever talked with your case manager or support coordinator about any special equipment, or changes to your home, that might make your life easier?

    • Choice and Control

      • Do you help pick the people who are paid to help you?

      • Would you like to help pick the people who are paid to help you?

      • When you are at home, can you eat when you want to? (MR/DD)

      • Can you talk to your case manager or support coordinator when you need to?

    Consumer Surveys for Self-Direction

    Sample Items

    • Respect and Dignity

      • Do the people paid to help you treat you respectfully in your home? (E/D)

      • Do support staff in other places, such as at work, or at a day program, say “please” and “thank you” when they ask for something? (MR/DD)

      • Does anyone ever do mean things to you, such as yell at you? (MR/DD)

    • Community Integration

      • Do you like to go out to eat? (MR/DD)

      • Do you go out to eat? (MR/DD)

      • Do you get to help pick where you eat out? (MR/DD)

      • Is there anything you want to do outside your home that you don’t do now?

    Consumer Surveys for Self-Direction

    Maine’s CD-PAS Survey

    • State seeking feedback from CD-PAS participants

      • Non-elderly adults with physical disabilities

    • Reviewed PES E/D and identified gaps

    • Crafted new survey items

    • Stakeholder input

      • Operating agency

      • Medicaid agency

      • Provider (Alpha One)

      • Consumers

    • Testing

      • Cognitive

      • Field

    Consumer Surveys for Self-Direction

    Maine’s CD-PAS Survey

    • Domains

      • Program Referral and Support

      • Access to Services

      • Respect and Dignity

      • Technology and Community Integration

      • Interviewer comments and observations

    • Second year of use

      • Interviews conducted by staff from the Muskie School’s survey unit

      • Early results reflected political uncertainty

      • Modifications made based on findings

    Consumer Surveys for Self-Direction

    Modifying the PES in Maine

    • Added items specific to self-direction

      • How PAs located and hired

      • PAs’ relationship to waiver participant

      • PA retention and back-up

      • Support from Alpha One Independent Living Specialists

      • Experience with self-direction

    • Retained relevant items from PES

      • Respect and dignity

      • Unmet need for personal assistance

      • Adaptive equipment

      • Physical and verbal abuse, theft

    Consumer Surveys for Self-Direction

    Crafting the new items in Maine

    • General design principles and traps to avoid

      • Consistent response categories

      • Appropriate response categories

      • Adequate response categories

      • “Problem” words

      • Sound-alike words

      • Compound or double-barreled questions

      • Jargon and unfamiliar words

      • Complicated concepts

      • Relevance to consumers (screening questions)

      • Sensitive, appropriate language

      • Hypothetical questions

      • Scales

      • Specificity of times, dates, frequency, and amounts

    Consumer Surveys for Self-Direction

    Testing the new items in Maine

    • How people answer a survey question

      • Comprehension

      • Retrieval

      • Estimation

      • Response

    • Each area is a potential source of bias

    • Cognitive testing: determining if the question you think you are asking is the one waiver participants are hearing and answering

    • Iterative process of asking draft questions with probes

      • Small n, longer interviews

      • Discuss and act on results

      • Resulted in many changes

    Consumer Surveys for Self-Direction

    Field Testingin Maine

    • Larger n, shorter interviews

    • Evaluate survey logistics

      • Reaching consumers

      • Scheduling interviews

      • Interview length

      • Interview acceptability

    • Obtain baseline data

    • Evaluate appropriateness of tool

    • Examine utility of findings

    Consumer Surveys for Self-Direction

    West Virginia’s Combined PES Tool

    • State seeking feedback from both traditional E/D waiver participants and those enrolling in new Personal Options waiver

    • Developed hybrid tool with separate skip patterns for traditional and self-directing waiver participants

    • Multiple players

      • Medicaid agency

      • Operating agency

      • UCED staff

      • Stakeholder advisory group

    Consumer Surveys for Self-Direction

    WV’s Combined PES tool (con.)

    • Many common items

      • Access to care/unmet need

      • Respect and Dignity

      • Abuse/theft

    • Choice and control items for traditional waiver

      • Primarily same questions from PES E/D

      • Some new items

    • New Program Referral and Support items for Personal Options participants

      • Modified the Maine questions

      • Additional items on resource consultants and financial management services

    Consumer Surveys for Self-Direction

    Testing the WV Survey

    • Iterative development process

      • Reviewed PES E/D and Maine tool

      • Developed new questions

      • Lots of review and feedback

        • Consumer review

        • Expert review

      • Limited cognitive testing

    • Field testing to begin this summer

    Consumer Surveys for Self-Direction

    Issues for Consideration

    • What are your goals for the survey effort?

      • Whom do you want to survey and why?

      • Where will these data fit into your QM strategy?

      • What additional information would you like to capture?

      • With whom do you plan to share the data?

      • One-time vs. longitudinal use

    • What is your target population?

      • Sampling frame

        • Criteria for inclusion or exclusion

      • Sampling plan

        • Sample size

        • Stratification

        • Proxies?

    Consumer Surveys for Self-Direction

    Additional Issues for Consideration

    • Interviewer Selection and Training

      • Who will be conducting interviews?

      • Who will attend training? Who will conduct it?

      • Level of effort

        • Ongoing data collection vs. limited period

    • Contacting participants

      • How will sample respondents be contacted?

        • State staff, case managers, other

      • Scheduling and briefing respondents

      • Data collection on non-respondents

    Consumer Surveys for Self-Direction

    Additional Issues for Consideration

    • Pre-survey information

      • What kind of information can you access about participants before the interview?

      • How can you use this information to tailor the interview and provide appropriate supports

      • Assess ability to complete survey

        • Special issue for consideration!

    • Timeframes

      • When will interviews start

      • How long will the process last

      • When would you like to be able to have and share results

    Consumer Surveys for Self-Direction

    Other Issues for Consideration

    • Electronic data entry

    • Psychometrics

      • Inter-rater reliability studies

      • Validity concerns/cognitive screening

    • Data Analysis

      • Who will develop the analysis plan?

      • Who will conduct the analysis

    • Acting on and sharing your results

      • Who should see the findings and how best to share them?

      • Determining appropriate QI projects

        • Involving stakeholders

        • Need to consider this up front

    Consumer Surveys for Self-Direction

  • Login