How Much Pollution Is My Program Preventing?

1. How Much Pollution Is My Program Preventing? Betsy Elzufon Larry Walker Associates June 13, 2002

2. Tools for Measuring Source Control Program Effectiveness Water Environment Research Foundation Project #98-WSM-2 www.werf.org 703-684-2470

3. Project Goals • Identify and develop evaluation tools for residential and commercial pollution prevention programs • Address misconceptions regarding effectiveness measurement • Develop a framework for incorporating effectiveness measurement into pollution prevention programs • Assess framework and evaluation tools through demonstration projects

4. Project conducted in 2 Phases • Phase 1 - Develop framework and identify measurement tools • Tools to Measure Source Control Program Effectiveness (2000) • Phase 2 - Assess Tools through Demonstration Projects • Controlling Pollution at Its Source: Wastewater and Stormwater Demonstration Projects (2001)

5. WERF Demonstration Projects • SFWPPP Mercury Reduction Project • Santa Monica New Development Program Assessment • Woodland Oil and Grease Reduction • LACSD Lindane Reduction • LA County Used Oil Campaign • Davis Healthy Gardens Program Evaluation

6. Getting Started • What are the specific issues to be addressed? • What do I already know about these issues? • What are my goals, what are we trying to accomplish? • How will I know if I have achieved my goals? • How will I know if my program is having an impact? • Can influent concentration changes be attributed solely to our P2 program? • What other things can I measure besides concentration changes? • Are there intermediate steps for which change can be measured?

7. Stages to Environmental Improvement • Design and implement program • Increase awareness of pollution issues • Change behavior • Reduce inputs of targeted pollutants • Improve water quality of discharges • Improve environmental condition

8. Framework for Developing an Effective Program • Identify the issue • Identify and assess sources • Determine available control strategies • Evaluate and prioritize control strategies • Establish a goal • Implement the program • Evaluate effectiveness • Modify program

9. Program Stages • Project planning • Audience characterization • Baseline information • Existing program review • Implementation • Tracking progress • Achieving milestones • Meeting expectations • Completion • Goals achieved • Most successful strategies • Future needs/ next steps

10. Effectiveness Measurement Tools Tested • Planning Tools • Estimated Load Reduction • Review existing information • Surveys • Measuring increased awareness • Phone surveys • Mailed surveys • Intercept surveys • Measuring behavior change • Sales Tracking • Participation Tracking • Surveys • Site visits • Measuring pollutant load reductions (monitoring)

11. Planning Tools Estimated Load Reductions Existing Information Review Surveys

12. Guidelines for Developing and Evaluating a Program • Identify and prioritize issues/pollutants • Prioritize sources • Baseline information • Develop a strategy for each identified source • Assign a goal to each strategy • Identify an effectiveness measure for each source

13. SF Estimated Hg Load Reduction

14. SFWPPP Mercury Reduction Project • Project planning • Estimated load reduction • Previous dental outreach evaluation (mailed survey) • Dental outreach program site visits • Thermometer exchange program • participation tracking • phone survey • intercept questions

15. Santa Monica New Development Program • Co-permittee in LA County Stormwater permit • New Development requirement that projects submit an Urban Runoff Mitigation Plan • Goal was to assess program and determine future directions • Assessment of process, submitted plans, and finished project

16. Santa Monica Urban Runoff Mitigation Plans

17. New Development Program Assessment Results • Agency staff and building community could be better educated on BMP design and implementation and URMP requirements • Building community may benefit from information on wider variety of BMPs • Workshop developed to address these issues

18. Woodland Oil and Grease Reduction • Existing restaurant control program with high compliance rate • Oil and grease collection system upsets • Existing program review to identify other sources

19. Woodland Oil and Grease Assessment

20. Residential Outreach Program on Proper Oil and Grease Handling

21. Mailed Surveys • One page - front and back • Pre-addressed, stamped envelope • One word answers, check boxes • Colorful layout

23. Dental Survey Results • 27% of 843 Dental surveys received • Best source of environmental information is CDA, SFDS (80%) NOT government agencies (31%) • 60% recalled 1997 brochure • 66% recycle scrap amalgam. ~50% recycle traps, ~27% recycle vacuum filter waste • Over half of respondents recommend non-amalgam fillings

24. Dental Outreach Program Development • Work with CDA and SFDS, they co-signed all outreach letters • Focus on increasing implementation of BMPs regarding recycling of trap and filter waste and waste log • Increase awareness of amalgam separators • Used targeted outreach/ site visits as outreach strategy

25. Measuring Increased Awareness Quantitative Surveys (phone, mail) Non-quantitative (Intercept)

26. Davis Healthy Gardens Program • Purpose to educate residents regarding impact of pesticide use on local waterways and on general stormwater issues • Outreach included signs, theatre slides, workshops, newspaper articles, fan brochure • Awareness Surveys • 1996 survey on stormwater, wastewater, hazardous waste disposal awareness • 1999 phone survey and mail survey

27. 1999 Phone Survey • Phone survey repeated questions from 1996 survey to measure changes in awareness • Phone survey added questions on Healthy Gardens Program awareness • Recall of program materials • Recall of program message • Connection between pesticides and local water pollution • 140 surveys completed (goal was 250)

28. 1999 Mailed Survey • Survey repeated Healthy Gardens questions from phone survey • Mailed to 500 randomly selected residents w/ a pre-addressed stamped envelope • Received 198 responses in 2 weeks • Combined survey response was 340 responses • Mailed survey responses and phone survey responses were statistically the same so responses could be combined

30. Healthy Gardens Program Assessment • 47 - 50% of respondents recalled program • 18% remember the programs message • 4 - 17% of survey respondents reported using less pesticides • Items recalled most often - Logo, North Pond map, theatre slide, demonstration signs • Locations recalled most often - North Pond, Farmer’s Market, movies, stores

31. Pollution Prevention Program Assessment - Changes since 1996 • Decrease in respondents reporting using City’s disposal programs (47% to 38% for used oil) • Increase from 2% to 26% of respondents reporting they don’t use pesticides • Increase in awareness of wastewater treatment plant (37% to 82%) • Increased awareness that stormwater goes to Davis ponds (3% to 14%) • Significant portion still believe stormwater receives treatment (39% to 34%)

32. What is an Intercept Survey? • Short interview conducted at a fixed location where target audience is likely to gather • Non-quantitative • Important design factors • Appropriate venue • Day and time of day • Oral or written survey • Person who conducts survey • Incentive

33. Grocery Store Intercept Survey • Events conducted outside local grocery stores in problem areas • Provided outreach materials (potholders, can lids, brochures) • Conducted survey on grease handling in Spanish and English • Distributed materials at local apartment complexes • Conducted follow-up intercepts to measure change in awareness after 3 months

34. Grease Survey Results • 19% - 24% pour used cooking oil down the drain • Apartment dwellers more likely to dispose of oil improperly than in single family homes • Low recall of outreach materials (4%)

35. LA County Used Oil Campaign • Annual campaign to promote used oil recycling • Targets Latino & African American men 16-35 • Bill boards, theatre slides, radio, bus stop signs • Intercept survey conducted at auto parts stores to determine campaign recall and used oil disposal practices • Conducted in English and Spanish • Saturday morning • Conducted by males fitting target group • Participants received two lottery tickets • 188 of 328 approached completed survey

36. Intercept Survey Results • 90% disposing of used oil and oil filters properly • 65-70% dispose of other automotive fluids properly • 40% recalled advertising regarding used oil recycling • 20% recalled advertising as radio or billboards • 60% recalled incorrectly TV or auto parts store advertising

37. Measuring Behavior Changes Site Visits Particpation Tracking Surveys Sales Tracking

38. Behavior Change Measurement may be very project specific • Commercial/ business audience - Site Visits • Residential - tracking participation in collection campaign, changes in sales patterns • Workshop, training - quiz • Surveys still useful

39. Dental Outreach Materials

40. Dental Site Visits • 50% of general dental practices visited (34) • 74% recycle scrap amalgam - higher than mail survey results (67%) • 15% used no amalgam - lower than mail survey results (23%) • 16% recycle traps properly • 38% dispose of traps as medical waste • 40% report placing >90% non-amalgam fillings

41. San Francisco Thermometer Collection Campaign • Purpose to increase awareness regarding mercury as a water pollutant and reduce use of mercury thermometers • Outreach included newspaper articles, PSAs, media event, direct mail, water bill insert • Evaluation through participation tracking and phone survey

42. Thermometer Turn-in - Participation Tracking • 2 questions regarding where participants lived and where they heard about program • Results • 3300 people turned in 4700 thermometers • Most effective outreach - Newspaper (42%), Radio (26%), Word-of-mouth (18%) • Best collection locations -40% collected at 1 of 9 fire stations

43. New Development Workshop Goals • Increase understanding of ordinance and SUSMP requirements • Improve knowledge regarding effective post construction BMPS

44. Santa Monica Workshop Evaluation • Participants knowledgeable regarding general Stormwater issues before workshop • Increased understanding of certain regulations • >100% increase in number of correctly identified post-construction BMPs • Increase in types of BMPs mentioned

45. Workshop Follow-Up Survey Mailed to Participants

46. LACSD Lindane Reduction • Goal to reduce lindane levels in wastewater • EPA grant to conduct pilot program • Conducted outreach targeting health care professionals • Evaluated using surveys, water quality monitoring and sales tracking

47. Survey Results - Awareness Change After After Before Percent Before Highly Toxic to Humans Pollutes Environment

48. Survey Results - Physician Behavior Change Before After Percent

49. Pharmacy Study Results - Sales Ratios Ratio Lindane to Non-Lindane Remedies Control Outreach

50. Bonus Result! • AB 2318 Lowenthal • Ban medical uses of lindane in California • Signed by Governor Davis in Sept. 2000 • In effect as of January 1,2002