Presented by Peg Capo, Executive Director, DDRB Kathy Williams, Project Manager, DDRB

1. 5 Habits of Highly Effective Consumer Satisfaction Research A case study of capturing meaningful consumer feedback for the DDRB of St. Charles county MACDDS Annual Conference October 2009 Presented by Peg Capo, Executive Director, DDRB Kathy Williams, Project Manager, DDRB Lisa Richter, richterIntel Cynthia Cluff, richterIntel

2. Introduction • DDRB is a public taxing entity that enters into funding contracts with agencies that serve individuals with developmental disabilities in St. Charles County. This public tax was established in 1977 when voters of St. Charles County approved to tax themselves up to sixteen cents per $100.00 of assessed property value. • DDRB funds: • 29 agencies plus case management for DMH and DDRB • 11 service areas or groupings of programs • 53 programs, plus 2 case management programs • Serving approximately 1,500 consumers in St. Charles County • A little less than half of the consumers receive 1 service, a little more than one-third of the consumers receive 2-3 services, and a little less than 20% receive 3 or more services • We needed feedback directly from the families and individuals that we serve in a way that is consistent across agencies and meaningful to us. • Two key uses of Consumer Satisfaction: • Planning tool for continuous improvement • Accountability that we are good stewards of the money we are handling for our constituents

3. What is Consumer Satisfaction Market Research? • It is a systematic and scientific approach to collecting feedback from the consumer’s viewpoint. • Shifts planning and decision making from intuitive information gathering to orderly and objective investigating • Emphasizes understanding consumer’s needs and measuring success as decided by the consumer • Characteristics of valuable customer satisfaction research are: • Timely – research is conducted on a schedule where it can be used to influence planning and decision making, the data is current • Complete – right amount of information • Relevant – collects measures that are meaningful to customer satisfaction, anticipates the kinds of information that will be useful • Quality – Data represents the true situation

4. What Does Consumer Satisfaction Research Allow You to Do? • Describe: • Paint a picture of the situation from the customer’s viewpoint – who, what, when … • Diagnose: • Explain the data – reasons behind something from the customer’s view point, dig deeper • Predict: • Using statistics predict the results of something

5. Habit #1: First Things First – Develop a Blueprint • First step in consumer satisfaction research is a well laid out plan. • A well laid out plan includes determining: • Your OBJECTIVE for conducting the research • Who are you wanting to talk to – SAMPLE PLAN • How are you going to collect the data – METHODOLOGY • Clear and precise OBJECTIVE of what is to be achieved, what information is being sought. • DDRB county-wide Consumer and Family Satisfaction Survey objective: • By listening directly to the voice of the consumer and their families or guardians measure the quality and effectiveness of DD services in St. Charles County • Identify areas for quality improvement • Measure satisfaction in a consistent format across agencies • Look at survey results in the following ways: • System-wide • By Consumers and Parents/Guardians • By Service Area (11) • Taking each of 29 agencies plus 2 case management agencies and look at results for each their funded programs

6. Habit #1: First Things First – Develop a Blueprint • A well defined SAMPLE PLAN - how many and who • DDRB was going to sample the entire population. • DDRB definition of population included: • All consumers 16 years of age or older • The parent/guardian of every consumer • A survey for each service consumer received – one to consumer if 16+ and one to the parent, could be multiple surveys to one consumer and one parent • Served in the last fiscal year – a definite time frame, go back no more than 12 months

7. Habit #1: First Things First – Develop a Blueprint • A database was developed, capturing all consumers, every service they received, their parent/guardian. • Sent Excel spreadsheet to each agency requesting a list of their population served in fiscal year, with defined columns of data needed, exact format specified • Assumed no email addresses available. • One row of data per consumer with 1 service listed, many consumers were listed multiple times for each service received. • This step of database development took a lot of time! • End result was 5,000+ records merged together from all 31 agencies.

8. Habit #1: First Things First – Develop a Blueprint • Identify most appropriate METHOD for collecting data considering the situation: • Characteristics of population • Database information you have on population • Budget • DDRB study done by mail. Targeted postcard reminders. Targeted telephone follow-up. • Things to keep in mind when doing a mail survey. • Lower cost compared to other methods (i.e., telephone) • Takes longer time to collect the data • Have to be very clear – wording and instructions, no interviewer to clarify • Have to be aware of the length of questionnaire – can’t be too long • Hard copy only – risk of order bias • Lose control of sample – it’s self selection • Many won’t respond • Can’t control who actually answers the survey • Those not responding may impact the representativeness of the sample obtained

9. Habit #2: Begin With the End in Mind – Develop a Questionnaire • Purpose of the questionnaire: takes the objectives and translates them into specific questions that are meaningful to the client and understandable to the respondent. • Develop a valid and reliable QUESTIONNAIRE. • Valid - Measuring what it is supposed to measure – do the scope of questions cover the topic area? • Reliable - Provide consistent results • Validity – What is important to consumers and families to be satisfied? • Use expertise, knowledge • Stakeholder panel – agencies, board members … • Past surveys • Other organization’s surveys • Reliability – Will it provide consistent results over time and under different circumstances? • Clear wording, simple, understandable, not ambiguous • One question addresses one thing • Can person answer, is it at their level, can they recall • Reliable measurement scale • At least 5 point scale, to get intensity of feelings, label scale either every pt or end pts • Balanced – equal satisfied and not satisfied points on scale, some exceptions • Non-forced choice – give them a don’t know or doesn’t apply option, some exceptions

10. Habit #2: Begin With the End in Mind – Develop a Questionnaire • Give clear directions. Break down barriers to participation • Give them comfort in the privacy of their feedback – Tell them it is anonymous and their name won’t be tied to them or identified to the sponsor of the study in any way • Give them a contact to reach if they have any questions, issues, or need clarifications • DDRB cover letter with survey did this • Identify the consumer, the agency, and the service – they are to think of when they answer this questionnaire • Pre-test!!! • Confirms if questionnaire is valid and reliable. • Have a small number of consumers and families complete the survey. Be there in person. See how they answer things. Ask them if anything is unclear, or left out.

11. Habit #2: Begin With the End in Mind • Benchmark2005 DDRB Consumer Satisfaction Q2. eliminate Q6. eliminate

12. Habit #2: Begin With the End in Mind Guardian Satisfaction Survey Thinking about the specific AGENCY and the specific PROGRAM listed in the box below, tell us what you think by checking a response for each item. If you do not know about an item or it does not apply to you, please check ‘don’t know / doesn’t apply’ for that item. NEW-directions Your opinions about this AGENCY: «Agency1»«Agency1_ID» Your opinions about this PROGRAM: «SERVICE1» «SERVICE1_CODE» Consumer for whom you are Guardian: merge in consumer first and last name NEW pre-coding NEW- ID consumer to think about Poor Not Very Average Very Excellent Don’t Know or Thinking of this service the consumer Good Good Doesn’t Apply receives from this agency, give us your opinion on… 1 2 3 4 5 0 NEW Scale and DK 1. Staff is respectful and helpful    reworded 2. Staff follows through in a timely manner (with appointments, meetings, requests, care plans, etc.)       NEW 3. Staff returns phone calls promptly       NEW 4. Staff meets your needs       NEW 5. Staff keeps you informed      NEW 6. Staff is well trained and knowledgeable       NEW 7. Program you participate in helps you       8. Program you participate in is well organized       NEW 9. Program service location is safe and healthy       10. Program service location is convenient       NEW 11. Overall rating for quality of staff and program       reworded 12. What do you like best about the staff and this program? (Please be as specific as possible) NEW – directed open ended questions 13. If you rated any of the above areas a ‘Poor’ or ‘Not Very Good,’ how can we improve this area? (Please be as specific as possible) NEW Not At Not Very Neutral Somewhat Very Don’t Know or All Likely Likely Likely Likely Doesn’t Apply 1 2 3 4 5 0 14. Likely to recommend this program to a friend  15. Is there anything else that can be done to better meet your needs? (Please be as specific as possible) NEW «G_ID» NEW Scale and DK

13. Habit #2: Begin With the End in Mind • Benchmark2005 DDRB Case Management Satisfaction Survey • Eliminated most of the questions: • Q1, Q2, Q3, Q4, Q4a, Q5a

14. Habit #2: Begin With the End in Mind Guardian Satisfaction Survey Thinking about the specific AGENCY and the specific PROGRAM listed in the box below, tell us what you think by checking a response for each item. If you do not know about an item or it does not apply to you, please check ‘don’t know / doesn’t apply’ for that item. NEW pre-coding Your opinions about this AGENCY: «Agency1»«Agency1_ID» Your opinions about this PROGRAM: «SERVICE1» «SERVICE1_CODE» Consumer for whom you are Guardian: merge in consumer first and last name A definition of Case Management was stated NEW- ID consumer Poor Not Very Average Very Excellent Don’t Know or Thinking only of case management you Good Good Doesn’t Apply receive from this agency, give us your opinion on… 1 2 3 4 5 0 NEW Consistent scale Consistent q’s 1. Staff is respectful and helpful    NEW (with appointments, meetings, requests, care plans, etc.)       NEW 3. Staff returns phone calls promptly       NEW 4. Staff meets your needs       NEW 5. Staff keeps you informed      NEW 6. Staff is well trained and knowledgeable       NEW 7. Staff makes you aware of services       NEW 8. Staff is an advocate for your needs       NEW 9. Staff’s ability to solve problems       NEW 10. Have consistent case management staff       NEW 11. Overall rating for quality of case mgmt staff       reworded Q7-Q10 unique to CM 12. What do you like best about the case management staff and service you receive? NEW – directed open ended questions 13. If you rated any of the above areas a ‘Poor’ or ‘Not Very Good,’ how can we improve this area? (Please be as specific as possible) NEW 14. Do you have an Annual Written Plan? Yes No Don’t Know definition of plan provided 15. Have you seen or do you have a copy of your Annual Written Plan? Yes No Don’t Know NEW 16. Is there anything else that can be done to better meet your needs? (Please be as specific as possible) NEW «G_ID»

15. Habit #3: Two Ears One Mouth – Listen to the Consumer • COLLECT THE DATA. • This takes time! DDRB took about 7 weeks to collect the data.

16. Habit #3: Two Ears One Mouth – Listen • Follow up to achieve return rate. • Post card reminders • DDRB email reminders to agencies and several public administrators • Telephone follow up to targeted smaller populations, in certain agencies, service areas, or programs. • Things to keep in mind with telephone surveying: • Can obtain people that may not complete mail self administer – more representative sample (all public opinion polls are done by telephone because it is currently accepted as the method that gives the most accurate representation of the population) • Capturing open ended responses, maybe more likely to answer, interviewer can probe • Telephone is more costly compared to mail or web-based

17. Habit #4: Synergize – Tell a Story • Weave the data and findings into a story that paints a picture of what consumers and parents/guardians say. • Reveal strengths and opportunities. • Include not only key findings but drill down to tie in underlying reasons (i.e., using open-ended responses). • Provide insightful and actionable recommendations that go beyond identifying a broad service area for improvement. • Tools to help weave a story and organize information. • Organize performance attributes – For DDRB listed attributes most important to least important. • We conducted some statistics and determined the strongest correlating/highest impact attributes on Overall Performance. • Decide on a reporting system – For DDRB we reported top two box score (rating Excellent/Very Good) • Handle open-ended questions – Group open ended responses and quantify

18. Habit #4: Synergize – Tell a Story • 2005 Story • Questions in order of questionnaire • 6 numbers reported for each question – 42 %’s in total on table for Guardians – • Guardians on one slide, consumers on another, no total combining

19. HIGHER IMPACT MODERATEIMPACT LOWERIMPACT High Score 88%+ Low Score < 80% Moderate Score 80%-87% Habit #4: Synergize – Tell a Story • DDRB funded agencies receive a high score of 90% for Overall quality of staff and program. Excellent work by agencies! • Key strength is Staff is respectful and helpful (92%). • Several attributes receive moderate scores of between 80% - 87%. These are areas for improvement. • Attributes ordered by impact on Overall • High/low score targets set • Top 2 box score for each attribute-11 %’s for each grouping of data • Show total, consumers, & guardians on one slide

20. Habit #4: Synergize – Tell a Story • 2005 Story – open ended “Comments” listed in total and were reported by agency in the format as seen here. • The lady I got is super nice, he/she helps me 100 percent. He/She is very responsive and he/she was there for me even right after his/her surgery. I cannot read or write and the burden was lifted of my chest after I got this service. Thank you very much!! • The staff/ teachers are absolutely awesome! My (child) has made huge changes in his/her speech and behavior in the last year. These guys are wonderful we would have never been ready for kindergarten next fall if it wasn’t for them! • He/She is happy and well cared for. The staff are all caring and provide a loving family atmosphere for him/her. I can not say enough good things about (Agency). I feel confident that all his/her needs are being met. • The staff are Saints! The staff are all angels! Our association with this (Service) gives (Name) an indescribable excitement and anticipation for the activities he/she has done. The work, sweat, planning, staffing of this activity is worth all their hard work for the participants! They talk to each other about the classes & can’t wait to begin each year! Thanks to you for the facilities and the volunteers. The kids LOVE them. • The agency has been great for our child. It has allowed him/her to participate in group sporting activities & to succeed with the help of the staff. The peer & staff support has allowed our (child) to feel confident in his/her "abilities." • It took them 9 months before they would return my call. Once they did come out, they said they were going to send me information and never did; there was no follow up. • Should respond quicker and let me know why you can’t fill requests. • They promise things but do not follow through • Need more employees • Inconsistent discipline • My child complains that the staff people there always yell and he/she thinks that is not necessary • Staff not always as responsive as I think they should be • I am not satisfied with the cleanliness of the carpet in the group home. Dirty and unsafe condition. • When there was trouble or a problem on the job, always sided with employer • Need more staff

21. Habit #4: Synergize – Tell a Story • Organize open ended questions by creating mutually exclusive and exhaustive answer categories. Assign the participant’s answers to the categories. • Tell a story of the open ended comments by frequency of response. What do you like best about the staff and this program? • With coding of open ends can diagnose systematically • Can sort by agency, program, service area • No new areas of importance are mentioned that were not included on the survey Base for total is all 1768 respondents, 41% of the total gave no answer (not shown in table above). Multiple responses accepted.

22. Habit #4: Synergize – Tell a Story If you rated any of the above areas a ‘Poor’ or ‘Not Very Good,’ how can we improve this area? • Can obtain insight in consumer’s or guardian’s own words as to areas to improve – in a systematic way Base for total is 146 respondents (146 out of 1768) who gave at least one ratings question a ‘Poor’ or ‘Not Very Good.’ Of this 146, 29% of the total gave no answer (not shown in above table). Multiple responses accepted.

23. Habit #4: Synergize – Tell a Story • Told the story of Consumer and Family Satisfaction by organizing and telling the findings: • With a System Wide Report & Report for the Board With an Agency Report Card

24. Habit #4: Synergize – Tell a Story • Conclusion from 2005 • It would appear that there are at least four areas, where quality improvement initiatives can be focused. These areas would be person centered planning, case management, Individualized Support Services, and Employment Services. An examination of the qualitative comments provided by respondents can also be helpful in combination with survey data to generate areas of focus for quality improvement initiatives.

25. HIGHER IMPACT MODERATEIMPACT LOWERIMPACT Low Score < 80% High Score 88%+ Moderate Score 80%-87% Habit #4: Synergize – Tell a Story • 2008 Employment Service Area – Staff keeps you informed looks like an opportunity area. Employment

26. HIGHER IMPACT MODERATEIMPACT LOWERIMPACT Low Score < 80% High Score 88%+ Moderate Score 80%-87% Habit #4: Synergize – Tell a Story

27. Habit #5: One Mouth Two Ears - Talk, Plan, Action, Reward • Take the data and recommendations and connect with stakeholders, develop action plans for improvement areas, reinforce positive areas. • System-wide recommendations to the DDRB of St. Charles. • Communicate to agency’s staff appreciation for care being given • Communicate that programs are making a difference • Strengthen communications and responsiveness given to families • Strengthen organizational or structural aspects • Provide additional training on disabilities, job knowledge, and appropriate conduct • Consider demand or need for enhancements to programs, additional programs or activities • Put forth effort to have less staff turnover • DDRB examples of using the data for quality improvement.

28. What is Consumer Satisfaction Market Research • It is a ___________ and ____________ approach to collecting feedback from the ____________ viewpoint • Characteristics of valuable customer satisfaction research are: • ________ – research is conducted on a schedule where it can be used to influence planning and decision making, the data is current • ________ – right amount of information • ________ – collects measures that are meaningful to customer satisfaction, anticipates the kinds of information that will be useful • ________ – Data represents the true situation

29. What is Consumer Satisfaction Market Research The questionnaire take the objectives and translates them into specific questions that are meaningful and understandable. • A questionnaire should be: • __________________ Scope of questions cover the topic area • __________________ Provide consistent results • What is something you can do with the questionnaire before the data is actually collected to test if the questionnaire is of good quality. • ___________________

30. Conclusion Content for these slides prepared by Cynthia Cluff