PTCS Service Provider Review

1. PTCS Service Provider Review Background • RTF assumed responsibility for maintaining PTCS specifications in March 2003 • Developed PTCS Service Provider Standards to create means for approving new Providers (initially Climate Crafters acted as the single Provider within the region with NEEA funding) • Standards require Providers to submit certain data to RTF upon request • In 2010, the RTF hired Northwest Energy Works (through PECI) to request data from PTCS providers and review for compliance with the Standards and, to the extent possible, technical specifications

2. PTCS Service Provider Review • The Review • Developed methodology by pulling substantive requirements from the Standards • Determined what data would be requested and how it would be reviewed to determine compliance • Reviewed with - and affirmed by - RTF staff and the PTCS Subcommittee • Identified 10 parties, sent nine RTF-recognized Providers a request for information in December 2010, followed up with personal contact and reminders tothose who did not respond • Sought information on all trainers, all technicians and PTCS measures installed from July 2009 through June 2010

3. PTCS Service Provider Review • Six Entities Responded with Data • Conservation Services Group (CSG) • Ecos Consulting (Ecos) • Efficiency Services Group (ESG) • National Center for Appropriate Technology (NCAT) • Proctor Engineering, CheckMe! program (Proctor) • Washington State University Energy Extension (WSU)

4. PTCS Service Provider Review • Of those Six: • Three conduct training as sole PTCS service (ESG, NCAT, WSU) • One, Ecos, delivers a full suite of PTCS services • One, CSG, maintains a systems registry, does QC of data, performs QA field audits on a sample of measures • One, Proctor, delivers a heat pump commissioning program with real time date collection and QC that the RTF has determined to be equivalent to PTCS without QA field audits being required

5. PTCS Service Provider Review

6. PTCS Service Provider Review • Recommendations – five primary areas for action • Changes to Providers’ processes • Provisions within the PTCS Service Provider Standards • Administration of the PTCS Service Provider Standards and measure technical specifications • Role of utilities and other entities contracting for or providing PTCS services • Future Reviews of PTCS Service Providers (Program Impact Evaluation across PTCS programs implemented throughout the region.)

7. PTCS Service Provider Review • Changes to Providers’ Processes -- Recommendations • Improved Data Systems – both Ecos and CSG track technician follow-up outside of the databases available for reporting. Needs improvement • Communication Between Providers – with Ecos housing the single technician registry (Proctor aside), there must be regular communication to track technician follow-up and non-compliances, inform trainers of needed changes to training to correct common technician errors, etc. Need formal communication protocols. • QA Sampling – random sampling is required unless a Provider has had an alternative approach approved by the RTF. The CSG and Ecos data did not contain any evidence of random sampling being performed. Providers need to demonstrate their sampling protocols. • Proctor PTCS Equivalency – current emphasis on outdoor temp lockout is something for which statistical analysis of technician data is not feasible. Need field data to determine how well Proctor systems compare to PTCS for compliance.

8. PTCS Service Provider Review • Provisions in PTCS Service Provider Standards • Simplify Standards • Option One. Require Providers to assume responsibility for all PTCS elements. Also recommend that the Standards be reduced to brief description of the core PTCS elements that must be included in a Provider’s program: technician performance management, measure tracking and quality control, and independent QA field review of required sample of measures. (This would not preclude organizations from sub-contracting with others to perform some duties.) • Option Two. Consolidate the present five PTCS service categories listed in the Standards into three: Technician Registry – combines training, certification and technician registry functions; PTCS Service Provider – combines system registry and QC functions; and QA Provider.

9. PTCS Service Provider Review • RTF Administration of PTCS Standards and Specs. • Recommend RTF delegate authority to Subcommittee to maintain technical specs., bringing only items with energy savings, measure life or significant program delivery implications to RTF for final decisions • Committee Charter with stated roles, voting, membership, etc. • RTF-appointed subcommittee chair with no PTCS Provider affiliation, reports committee activity to RTF, brings items to RTF for decisions, identifies and manages any potential conflicts of interest • Restrict subcommittee to recommendations on matters involving Provider Standards

10. PTCS Service Provider Review • Role of Entities Contracting for or Providing PTCS Services • Parties contracting for, or providing, PTCS programs must ensure that their scopes of work allow for all aspects of PTCS Standards and Specifications to be implemented.

11. PTCS Service Provider Review • Future Reviews • Recommend that reviews be performed as site audits of PTCS Providers. • Use the actual data systems to view registry information, confirm tracking and QC procedures • Allow Provider’s staff to demonstrate how all aspects of PTCS administration are handled • Evolve the first generation “information request” into a proper survey instrument to better structure responses from Providers • Follow-up to this review • Require written response from each Provider that addresses how deficiencies will be addressed and a timeline for bringing program into full PTCS compliance • Program Impact Evaluation • This review did not assess compliance of PTCS measures in the field. Both CSG and Ecos showed a significant number of measures that failed to “pass” QA inspection. We do not know that the work in the field is being performed comparably to that done for the initial model calibration when PTCS was being developed. A field review and analysis of measure savings and measure life is needed.

12. PTCS Service Provider Review • Recommended RTF Actions on Report. Move for decision at the next meeting to: • Terminate recognition of Providers that did not supply information for review • Require written responses from Providers to address non-compliances and offer timeline for bringing programs into compliance • Require full PTCS field Quality Assurance for CheckMe! measures (with effective date set to minimize program delivery issues, if beneficial) • Charge RTF staff with drafting subcommittee charter (to the extent permitted under RTF’s bylaws) • Charge RTF staff with creating draft of simplified PTCS Service Provider Standards in keeping with “Option One”