Low-Income Program Evaluation: Informing Program Management and Cost-Effectiveness

1. Low Income Program Evaluation Insights Informing Program Management and Cost-Effectiveness Stu Jerue, PGW Jackie Berger, APPRISE April 4, 2019

2. Presentation Overview

3. Background

4. PA Universal Service Programs

5. Intersection of CAP and LIURP 2017 Natural Gas CAP Programs Source: 2017 Report on Universal Service Programs & Collections Performance. PA Public Utility Commission, Bureau of Consumer Services.

6. LIURP Requirements

7. LIURP Reporting Requirements

8. Pgw approach

9. Home Comfort • Program goals and workflow: • 3 Contractors schedule audits, develop work scopes, and return to perform efficiency measures • Achieve deep cost-effective savings, based on TRC • Seek whatever savings possible, even when work scope is limited • Over 20,000 homes treated in program to date • Many receive only “core measures” at audit • Customers assigned in program; no application

10. Budget and Jobs

11. Contractors & Scorecard • Diversity of contractor types & subcontracting models • Competition-driven, with funds allocated based on scorecard • Total Savings • Cost-effectiveness • QA • Data entry, issue resolution, other metrics

12. Project Example • Reduced usage from 196 MMBtu by 85 MMBtu (43%) at a cost of $8,831 • 25% leakage reduction and insulation • New heater and chimney liner • Low-flow devices • Electrical rewiring Source: Philly Mag

13. challenges

14. Savings Decline

15. PGW Pre-Treatment Usage Decline 23% decline in pre-treatment usage from 2011 to 2016.

16. Pre-Treatment Usage Decline (Electric Heat) 18% decline in pre-treatment usage from 2011 to 2017.

17. Level of Treatment

18. Health and Safety Issues

19. Health and Safety Issues There were 687 health & safety issues reported in 2016.

20. Change in Jobs Fully Treated

21. Change in Jobs With at Least 1 Major Measure

22. Energy Savings Decline

23. Energy Savings Decline

24. Energy Savings Decline

25. Energy Savings Decline – Electric Heating 34% decline in savings from 2011 to 2016.

26. Challenges Servicing Customers • Customer refusals / no-shows • Landlord authorization • Program name and awareness • Other challenges serving low income and senior populations

27. evaluation

28. 2016 Evaluation – Energy Savings

29. Energy Savings by Contractor

30. 2016 Evaluation – Comprehensive Jobs

31. 2016 Evaluation – Comprehensive & Limited Jobs

32. 2016 Evaluation – Comprehensive Jobs Major measures include roof insulation, heating system replacement, infiltration work, and programmable thermostats.

33. 2016 Evaluation – Limited Jobs Major measures include roof insulation, heating system replacement, infiltration work, and programmable thermostats.

34. 2016 Evaluation – Measure Savings ***Statistically significant at the 99 percent level; **95% level. *90% level.

35. 2016 Evaluation – Measure Savings by Contractor ***Statistically significant at the 99 percent level; **95% level. *90% level.

36. 2013 Evaluation – Measure Savings ***Statistically significant at the 99 percent level; **95% level. *90% level.

37. 2016 Evaluation – Realization Rates

38. 2016 Evaluation – Benefit-Cost Ratio

39. Findings & Recommendations • Contractor performance inputs • Percent of jobs that are comprehensive • Major and minor measures installed • Level of investment • Contractor performance measurement • Level of savings • Realization rates • Benefit-cost ratio

40. Findings & Recommendations • Measures with statistically significant savings • Heating system replacement • Roof insulation • Water heater replacement • Air sealing • Programmable thermostats • Variation in measure-specific savings by contractor

41. Program improvements

42. Putting into Context • Develop hypotheses • Not defining thermal envelope or using blower door properly • Unclear expectations and communication • Crews not encouraged to seek deep savings • “Quality” not defined • Realistic understanding of program limitations and ability to change contractors • Steering a barge, not a speedboat

43. Reviewing Evaluation Report • Body of evidence • Impact evaluation report • Other annual reporting • Inspection reports • Aggregated measure and case data • Customer complaints • Contractor field visits

44. Improvement Plan and Schedule

45. Inspection Updates

46. Contractor-Centric Training & Mentoring • On-site shadowing for all auditors and crews • Classroom trainings for all program staff • Web conferences & meetings at contractor offices • Availability for technical questions & QA report review

47. Program Manual • Define quality • Reduce communication issues • Provide inspector tool • Set clear expectations • Provide to field staff, management and administrators

48. Health and Safety Documentation Tree • Fewer blower door tests = low realized savings • Perform work safely, and tell us why you chose that step • Recognizing contractor’s sense of risk

49. Blower Door Estimation Protocol

50. Zonal Pressure Requirements • Zonal pressure pre and post tests required for zones receiving air sealing and insulation. • Goal to achieve 45pa @ 50CFM for zones outside the envelope