LiveWell Colorado Evaluation

1. LiveWell Colorado Evaluation LiveWell Colorado Taskforce Meeting July 16, 2009

2. LWCO Evaluation Team: Mission To guide and support LiveWell Colorado communities in evaluation and to report the overall outcomes and impact of LiveWell initiatives.

3. KPCO Community & Local Gov’t Relations Corina Lindley, MPH Monica Buhlig, MPH Principal Investigator Diane King, PhD Sr. Evaluation Manager Bonnie Leeman-Castillo, PhD Data Management & Reports Data Collection Community Reports & TA Data Analysis Sr. Research Specialist Erica Morse, MA Sr. Research Asst. Lisa Harner, MA TristanSanders, BS Statistical Analyst Sophia Newcomer, MPH Sr. Research Asst. Deanna McQuillan, MA LWCO Evaluation Team: Who We Are

4. What We Do • Collect • Enter • Analyze • Report …………………………………………………Data

5. Primary Sources of Data Demographic & Process Data • Community Action Plan (year end) • Continuation Progress Report (year end) • Annual Site Visits (summer) • Key informant interviews with key community coalition members (fall) • TA Reports (monthly) Outcomes Data • Continuation Progress Report (year end) • IVR Surveys (varies by community)

6. Other Sources of Data • Media and Website Hits • Community Highlights • Local Evaluators’ Reports • School surveys • CDPHE surveys e.g., BRFSS and Child Health Survey

7. Today’s Plan • LWC IVR Survey: Analytic Update • Progress Milestones • Current indicators of future success • Community Reports • Small group discussion

8. LiveWell Colorado IVR Survey:Analytic UpdateSophia Raff Newcomer, MPHBiostatisticianInstitute for Health Research, Kaiser Permanente Colorado

9. Agenda • What’s new with the IVR survey? • IVR survey updates • Timing of community surveys • Analytic update • Discussion of raw vs. weighted data • Examples

10. What’s new with the IVR survey? • Timing of survey • Survey will be conducted: • M&P Year • Year 1 (first year of implementation) • Year 5 • Year 10 • Questions • Consistent “core” questions • Limiting number of questions on survey

11. Unweighted data • In the LiveWell Colorado IVR survey, a sample (n) of community members participate in a phone survey. • The intent of the survey is to use the responses from that sample of community members (n) to better understand key health and behavior indicators of the entire community (N). Community (N) Sample (n)

12. Unweighted data (cont.) • The data collected is raw data • Raw data: the actual responses from survey participants • Sometimes refered to as “unweighted” • If we use the raw data from the sample (n) to draw conclusions about the community (N), then we assume that: • Non coverage and non response bias is the same throughout all demographic groups in the community • The sample (n) is representative of the community • These assumptions are violated in the LWCO IVR

13. Example: Park Hill 2006Question: # of fruits and vegetables/dayN=19,315 (2000 Census) n= 321

14. Weighted survey analysis (Park Hill data) • Weighted survey analysis adjusts the raw, collected data from the sample (n) to try to “look” like the population (N) • Weighting on age and sex, the individual case weights (ICW) are: Small cell sizes; collapse with next age group

15. Definitions…and then some data! • Survey sample size: number of respondents that started the IVR survey (provided a valid age and pressed a key for the first question) • Item sample size: number of respondents who provided a valid response for that question • 95% confidence interval: range of values in which the “true” weighted percentage is expected to occur, with 95% probability • If we drew 100 samples from that same community, we would expect the weighted percentage from 95 of those samples to fall within the reported 95% confidence interval

16. Have you heard of Park Hill Thriving Communities?

17. Have you heard of Park Hill Thriving Communities? 39.3% 33.8% 44.8% Weighted percentage for response=“Yes” 2008 33.4% 28.0% 38.7% 2007 17.0% 12.4% 21.6% 2006 10% 20% 30% 40%

18. Meeting guidelines of 30 minutes of moderate physical activity at least 5 days per week?

19. Meeting guidelines of 5 or more servings of fruits and vegetables per day?

20. Have you heard of LiveWell Commerce City?

21. Meeting guidelines of 30 minutes of moderate physical activity at least 5 days per week?

22. Meeting guidelines of 30 minutes of moderate physical activity at least 5 days per week? Weighted percentage of respondents meeting moderate PA guidelines COMMERCE CITY 35.5% 27.9% 43.1% 2008 27.3% 20.4% 34.2% 2007 26.7% 18.9% 34.6% 2006 20% 30% 40%

23. Meeting guidelines of 5 or more servings of fruits and vegetables per day?

24. Data weighting summary • Advantages: • Allows the collected IVR data to “look” more like the population from which it is sampled • Limitations • 2000 Census data • Weighting on other sociodemographic factors, such as race, is impractical • Small sample sizes and “extreme” ICWs

25. Any questions?

26. Progress Milestones

27. How do we answer that question????? • Collaborative Partnerships & Community Capacity • 4 Intermediate Outcomes that Predict Impact • Volume of changes: number of new or modified community & systems changes. • Intensity of changes: modifying access & removing barriers more intense than providing information about services. • Permanence (durability) of changes: ongoing versus once • Penetration of changes: was contact made (reach) through multiple sectors & settings (adopters) or geographic areas.

28. Collaborative Partnerships & Capacity Building • Are Partners Broadly Representative? • e.g., 4 core sectors, parents, policy makers, police, target populations, non-profit organizations • Are Partners (including residents) Engaged? • e.g., shared leadership & work teams • Is Community Capacity Increasing? • e.g., amount of resources leveraged

29. Volume & Intensity • How many organizational systems changes adopted? • How many environmental changes made? • How many sectors are you working in?

30. Permanence (a.k.a “maintenance”) • Are strategies durable? • e.g., policy change > curriculum change > events • Has HEAL mission been adopted by partner organizations? • Has HEAL language been incorporated into planning & policy documents?

31. Broad Penetration (a.k.a. “reach”) • What proportion of target population(s) have been reached? • Are residents being reached at multiple levels of the ecological model? • e.g., cooking class (individual / program); corner store/farmer’s markets (family / environment); school rules involving snacks (organization / policy)

32. Broad Penetration (a.k.a. “adoption”) • Are changes holistic (i.e., inter-linked)? • e.g., park improvements + safe routes to park >park improvements alone • Are single goals addressed in multiple sectors? • e.g., breastfeeding addressed through: healthcare (training providers to counsel, changing formula bags for lactation kits), social marketing (norms & attitudes), workplaces (space, time, supervisor awareness)

33. Community Reports

34. Community Reports: Activity Questions for Discussion: • Who would use this information: partners, stakeholder, and/or residents? • How might it be used? • Is it a good way to display this information? • Does it need more/less text? • What would you add or subtract on this topic/subsection of the report?