1. How Much Pollution Is My Program Preventing? Betsy Elzufon Larry Walker Associates June 13, 2002

2. Tools for Measuring Source Control Program Effectiveness Water Environment Research Foundation Project #98-WSM-2 www.werf.org 703-684-2470

3. Project Goals • Identify and develop evaluation tools for residential and commercial pollution prevention programs • Address misconceptions regarding effectiveness measurement • Develop a framework for incorporating effectiveness measurement into pollution prevention programs • Assess framework and evaluation tools through demonstration projects

4. Project conducted in 2 Phases • Phase 1 - Develop framework and identify measurement tools • Tools to Measure Source Control Program Effectiveness (2000) • Phase 2 - Assess Tools through Demonstration Projects • Controlling Pollution at Its Source: Wastewater and Stormwater Demonstration Projects (2001)

5. WERF Demonstration Projects • SFWPPP Mercury Reduction Project • Santa Monica New Development Program Assessment • Woodland Oil and Grease Reduction • LACSD Lindane Reduction • LA County Used Oil Campaign • Davis Healthy Gardens Program Evaluation

6. Getting Started • What are the specific issues to be addressed? • What do I already know about these issues? • What are my goals, what are we trying to accomplish? • How will I know if I have achieved my goals? • How will I know if my program is having an impact? • Can influent concentration changes be attributed solely to our P2 program? • What other things can I measure besides concentration changes? • Are there intermediate steps for which change can be measured?

7. Stages to Environmental Improvement • Design and implement program • Increase awareness of pollution issues • Change behavior • Reduce inputs of targeted pollutants • Improve water quality of discharges • Improve environmental condition

8. Framework for Developing an Effective Program • Identify the issue • Identify and assess sources • Determine available control strategies • Evaluate and prioritize control strategies • Establish a goal • Implement the program • Evaluate effectiveness • Modify program

9. Program Stages • Project planning • Audience characterization • Baseline information • Existing program review • Implementation • Tracking progress • Achieving milestones • Meeting expectations • Completion • Goals achieved • Most successful strategies • Future needs/ next steps

10. Effectiveness Measurement Tools Tested • Planning Tools • Estimated Load Reduction • Review existing information • Surveys • Measuring increased awareness • Phone surveys • Mailed surveys • Intercept surveys • Measuring behavior change • Sales Tracking • Participation Tracking • Surveys • Site visits • Measuring pollutant load reductions (monitoring)

11. Planning Tools Estimated Load Reductions Existing Information Review Surveys

12. Guidelines for Developing and Evaluating a Program • Identify and prioritize issues/pollutants • Prioritize sources • Baseline information • Develop a strategy for each identified source • Assign a goal to each strategy • Identify an effectiveness measure for each source

13. SF Estimated Hg Load Reduction

14. SFWPPP Mercury Reduction Project • Project planning • Estimated load reduction • Previous dental outreach evaluation (mailed survey) • Dental outreach program site visits • Thermometer exchange program • participation tracking • phone survey • intercept questions

15. Santa Monica New Development Program • Co-permittee in LA County Stormwater permit • New Development requirement that projects submit an Urban Runoff Mitigation Plan • Goal was to assess program and determine future directions • Assessment of process, submitted plans, and finished project

16. Santa Monica Urban Runoff Mitigation Plans

17. New Development Program Assessment Results • Agency staff and building community could be better educated on BMP design and implementation and URMP requirements • Building community may benefit from information on wider variety of BMPs • Workshop developed to address these issues

18. Woodland Oil and Grease Reduction • Existing restaurant control program with high compliance rate • Oil and grease collection system upsets • Existing program review to identify other sources

19. Woodland Oil and Grease Assessment

20. Residential Outreach Program on Proper Oil and Grease Handling

21. Mailed Surveys • One page - front and back • Pre-addressed, stamped envelope • One word answers, check boxes • Colorful layout

23. Dental Survey Results • 27% of 843 Dental surveys received • Best source of environmental information is CDA, SFDS (80%) NOT government agencies (31%) • 60% recalled 1997 brochure • 66% recycle scrap amalgam. ~50% recycle traps, ~27% recycle vacuum filter waste • Over half of respondents recommend non-amalgam fillings

24. Dental Outreach Program Development • Work with CDA and SFDS, they co-signed all outreach letters • Focus on increasing implementation of BMPs regarding recycling of trap and filter waste and waste log • Increase awareness of amalgam separators • Used targeted outreach/ site visits as outreach strategy

25. Measuring Increased Awareness Quantitative Surveys (phone, mail) Non-quantitative (Intercept)

26. Davis Healthy Gardens Program • Purpose to educate residents regarding impact of pesticide use on local waterways and on general stormwater issues • Outreach included signs, theatre slides, workshops, newspaper articles, fan brochure • Awareness Surveys • 1996 survey on stormwater, wastewater, hazardous waste disposal awareness • 1999 phone survey and mail survey

27. 1999 Phone Survey • Phone survey repeated questions from 1996 survey to measure changes in awareness • Phone survey added questions on Healthy Gardens Program awareness • Recall of program materials • Recall of program message • Connection between pesticides and local water pollution • 140 surveys completed (goal was 250)

28. 1999 Mailed Survey • Survey repeated Healthy Gardens questions from phone survey • Mailed to 500 randomly selected residents w/ a pre-addressed stamped envelope • Received 198 responses in 2 weeks • Combined survey response was 340 responses • Mailed survey responses and phone survey responses were statistically the same so responses could be combined

30. Healthy Gardens Program Assessment • 47 - 50% of respondents recalled program • 18% remember the programs message • 4 - 17% of survey respondents reported using less pesticides • Items recalled most often - Logo, North Pond map, theatre slide, demonstration signs • Locations recalled most often - North Pond, Farmer’s Market, movies, stores

31. Pollution Prevention Program Assessment - Changes since 1996 • Decrease in respondents reporting using City’s disposal programs (47% to 38% for used oil) • Increase from 2% to 26% of respondents reporting they don’t use pesticides • Increase in awareness of wastewater treatment plant (37% to 82%) • Increased awareness that stormwater goes to Davis ponds (3% to 14%) • Significant portion still believe stormwater receives treatment (39% to 34%)

32. What is an Intercept Survey? • Short interview conducted at a fixed location where target audience is likely to gather • Non-quantitative • Important design factors • Appropriate venue • Day and time of day • Oral or written survey • Person who conducts survey • Incentive

33. Grocery Store Intercept Survey • Events conducted outside local grocery stores in problem areas • Provided outreach materials (potholders, can lids, brochures) • Conducted survey on grease handling in Spanish and English • Distributed materials at local apartment complexes • Conducted follow-up intercepts to measure change in awareness after 3 months

34. Grease Survey Results • 19% - 24% pour used cooking oil down the drain • Apartment dwellers more likely to dispose of oil improperly than in single family homes • Low recall of outreach materials (4%)

35. LA County Used Oil Campaign • Annual campaign to promote used oil recycling • Targets Latino & African American men 16-35 • Bill boards, theatre slides, radio, bus stop signs • Intercept survey conducted at auto parts stores to determine campaign recall and used oil disposal practices • Conducted in English and Spanish • Saturday morning • Conducted by males fitting target group • Participants received two lottery tickets • 188 of 328 approached completed survey

36. Intercept Survey Results • 90% disposing of used oil and oil filters properly • 65-70% dispose of other automotive fluids properly • 40% recalled advertising regarding used oil recycling • 20% recalled advertising as radio or billboards • 60% recalled incorrectly TV or auto parts store advertising

37. Measuring Behavior Changes Site Visits Particpation Tracking Surveys Sales Tracking

38. Behavior Change Measurement may be very project specific • Commercial/ business audience - Site Visits • Residential - tracking participation in collection campaign, changes in sales patterns • Workshop, training - quiz • Surveys still useful

39. Dental Outreach Materials

40. Dental Site Visits • 50% of general dental practices visited (34) • 74% recycle scrap amalgam - higher than mail survey results (67%) • 15% used no amalgam - lower than mail survey results (23%) • 16% recycle traps properly • 38% dispose of traps as medical waste • 40% report placing >90% non-amalgam fillings

41. San Francisco Thermometer Collection Campaign • Purpose to increase awareness regarding mercury as a water pollutant and reduce use of mercury thermometers • Outreach included newspaper articles, PSAs, media event, direct mail, water bill insert • Evaluation through participation tracking and phone survey

42. Thermometer Turn-in - Participation Tracking • 2 questions regarding where participants lived and where they heard about program • Results • 3300 people turned in 4700 thermometers • Most effective outreach - Newspaper (42%), Radio (26%), Word-of-mouth (18%) • Best collection locations -40% collected at 1 of 9 fire stations

43. New Development Workshop Goals • Increase understanding of ordinance and SUSMP requirements • Improve knowledge regarding effective post construction BMPS

44. Santa Monica Workshop Evaluation • Participants knowledgeable regarding general Stormwater issues before workshop • Increased understanding of certain regulations • >100% increase in number of correctly identified post-construction BMPs • Increase in types of BMPs mentioned

45. Workshop Follow-Up Survey Mailed to Participants

46. LACSD Lindane Reduction • Goal to reduce lindane levels in wastewater • EPA grant to conduct pilot program • Conducted outreach targeting health care professionals • Evaluated using surveys, water quality monitoring and sales tracking

47. Survey Results - Awareness Change After After Before Percent Before Highly Toxic to Humans Pollutes Environment

48. Survey Results - Physician Behavior Change Before After Percent

49. Pharmacy Study Results - Sales Ratios Ratio Lindane to Non-Lindane Remedies Control Outreach

50. Bonus Result! • AB 2318 Lowenthal • Ban medical uses of lindane in California • Signed by Governor Davis in Sept. 2000 • In effect as of January 1,2002