Making the Case for Measurement: A Nationwide Logic Model and Measures for SBOs/SBEAPs

1. Making the Case for Measurement: A Nationwide Logic Model and Measures for SBOs/SBEAPs May 11, 2006 National Environmental Partnership Summit Yvonne M. Watson and Kenya Stump

2. Road Map • Background • Approach to Developing Measures • SBEAP Measures • Benefits and Challenges of Common Measures

3. Background • ~ 507 Small Business Ombudsman (SBOs) and Small Business Environmental Assistance Programs (SBEAPs) nationwide as of 2005 • Provide key services to help small businesses comply with environmental regulations • Technical assistance • Regulatory review • Training and Compliance Workshops • Compliance assistance tools • Emissions inventory development

4. Why Measure? • The level of influence of any organization is dependent on its ability to tell its story and demonstrate results. • Increasing demands from EPA, State sponsors and others to demonstrate and communicate results.

5. Why Measure? • Desire to answer questions from internal and external stakeholders and sponsors regarding performance. • Do you have enough resources? • Are you operating efficiently? • What are the benefits to the consumer? • Did you accomplish your goals? • What is your impact to the community?

6. Small Business by the Numbers SBA Office of Advocacy. “The Impact of Regulatory Costs on Small Firms”. September 2005. http://www.sba.gov/advo/research/rs264tot.pdf HOW DOES COMPLIANCE ASSISTANCE AFFECT THESE NUMBERS?

7. Kentucky Example • 3:1 benefit to cost ratio • ~$37,500 in net benefits per business assisted • ~$1,875 in net benefits per employee RESULTS ARE NOT INDICATIVE NATIONWIDE PROGRAM PERFORMANCE NOR ARE THEY TRANSFERABLE TO OTHER PROGRAMS

8. SBEAP National Measurement Subcommittee • 14 Members • (SBEAP staff from VT, MN, IA, ID, IN, CO, SC, WI, VT, KS, KY and EPA Regions and staff) • Mission • The overall mission/purpose of the measurement subcommittee is to work cooperatively to support the SBO/SBEAP National Steering Committee, improve SBEAP measurement information, training, and resources, and respond to SBEAP measurement needs in a timely manner.  • Purpose •   "Did we (SBEAPs) accomplish what we set out to do as mandated under the Clean Air Act?"

9. SBEAP National Measurement Subcommittee • Goals • Develop a mechanism to translate SBEAP activities into value added outcomes • Work with the promotional and multi-media subcommittee to incorporate measurement issues • Provide a mechanism to assess small business compliance assistance needs • Provide measurement and evaluation training, tools and resources • Develop common measures for the SBEAP programs to use nationwide and individually

10. Approach to Developing Measures To produce these outputs… So that these customers can change their ways… We use these resources… For these activities… Which leads to these outcomes… Leading to these results! A logic model is a diagram and text that describes/ illustrates the logical (causal) relationships among program elements and the problem to be solved, thus defining measurements of success.

11. Approach to Developing Measures • Conducted two online Logic modeling training sessions • Used GoToMeeting (Sponsored by KSBDC) • Developed national SBEAP Logic models for three program components • Technical Assistance Programs • Small Business Ombudsman • Compliance Advisory Panel • National level models were based on Logic models developed by 6 states

12. Compliance and Technical Assistance Program Outcomes Resources Activities Outputs Customers Short-term Intermediate Long-term Ultimate Goal • Staff • Institutional Knowledge • Budget ($) • Partners • Program Host • National Steering Committee • Small Businesses (SB) • Compliance Advisory Panel (CAP) • EPA Office of the Small Business Ombudsman • Small Business Ombudsman (SBO) • COMPLIANCE ASSITANCE ACTIVITIES • Answer phone calls and e-mail request for assistance • Develop compliance assistance tools • Comment on new regulations/participate in rulemaking • Prepare mailing on new regulations • Conduct Site Visits • Make compliance recommendations • Prepare emission spreadsheets and permit applications • Research rules and regulations • Develop training seminars/workshops (permitting, P2, compliance) • Assist with emissions inventory • Make referrals to partners Small Businesses (SB) • # of assistance requests answered • # of compliance tools developed • # of rule comments • # of new rule mailings • # of site visits conducted • # of compliance recommendations • Tons of emissions calculated and # of applications prepared • # of rules and regulations researched • # of training seminars developed & conducted • Emissions inventory completed • # of referrals to partners • Environment is improved • Community is safer • Increased understanding of environmental resources available (SBEAP, EPA, Partners) • Increased understanding/ awareness of air emissions • Increased understanding of regulatory requirements • Increased understanding of P2 opportunities • SB submit permit applications/registrations • SB make changes to comply • SB decrease violations • SB decrease the number of accidents and releases • SB educate workforce • SB improve management of environmental responsibilities • SB implement P2 and reduce emissions • SB become environmental steward • SB productivity increase • SB become more economically viable • SB work environment improves • SB apply for awards • SB promote SBEAP services • SB advocate for comprehensive environmental assistance to new businesses • # of committees serving on • # of promotional materials developed • # of program web pages • # of partners projects • # of exhibit and speaking events attended • # of newsletters developed and distributed • # of award activities conducted • Increased awareness of SBEAP and Partners • SB are recognized by the State and Community for accomplishments • PROGRAM PROMOTION ACTIVITIES • Serve on local, state, and federal committees • Develop promotional materials • Webpage development • Meet and coordinate projects with partners • Exhibit and speak at partner functions/events • Develop and distribute program newsletter • Coordinate and promote awards program • SBEP becomes known as steward and valued resource to the community Partners; Legislators; Host; Other Decision-makers • Increased awareness of SBEAP and Small Business concerns • SBEAP services are expanded into other environmental media • Increased resources and support available to SBEAP • PROFESSIONAL DEVELOPMENT ACTIVITIES • Attend meetings and conferences • Serve on local, state, and federal committees • Networking • Acquire and maintain relevant certifications • CTAP improves consulting services and network • # of conferences and meetings attended • # of committees serving on • # of networking events attended • # and type of certifications Increase in professional knowledge and network, resources available • SBEAP expands tools and services offered • New partnerships developed CTAP Staff • Economic Development Agencies advocate for comprehensive environmental assistance to new businesses Economic Development Organizations; Partners Increased understanding of how regulatory requirements impact economic development • Economy Grows • BUSINESS DEVELOPMENT ACTIVITES • Assist Economic development projects when there are environmental concerns • Assist partners with environmental issues relating to constituents • # of economic development projects assisted with • # of partner environmental assistance requests Small Businesses • Increased awareness of SBEAP, Partners and State Incentives • SB recommends host state as a place to grow/move • SB locate to state

13. Small Business Ombudsman Outcomes Resources Activities Outputs Customers Short-term Intermediate Long-term Ultimate Goal • Staff • Institutional Knowledge • Budget ($) • Partners • Small Business Ombudsman (SBO) Host Agency • National Steering Committee • Small Businesses (SB) • Compliance Advisory Panel (CAP) • EPA Office of the Small Business Ombudsman • Compliance and Technical Assistance Program (CTAP) • ADVOCATE • Advocates for SB experiencing “roadblocks” in permitting • Advocates for SB experiencing enforcement or compliance issues • Write articles for publications • Attends permit discussion meetings on behalf of SB • Attend pre-enforcement meetings • Serve on local, state, and federal • # of compliance/ enforcement issues received • # of promotional pieces developed and articles submitted to publications • # of enforcement meetings attended • # of committees/ boards/ workgroups serving Small Businesses • Increased awareness of SBEAP and Ombudsman services • Increased understanding by permitting agency of “roadblocks” • Increased understanding between permit writers, SB, and inspectors • Communication increases between SB, permit reviewer, and inspectors • Changes in permitting strategies are considered • SB perception of permitting agency improves • Permitting burden is lessened • State Regulatory Environment improves for SB • State is seen as SB friendly • Environment improves • SBEAP becomes known as steward and valued resource to the community • ADVISORY • Review upcoming regulations for SB issues • Review permit applications • Field phone calls and e-mail request from SB and partners • Ombudsman makes referrals to partners • Make recommendations to CTAP on sector projects • Oversee and make recommendations on CTAP compliance tool development • # of regulations reviewed and comments submitted • # of applications reviewed • # of SB assisted • # of referrals to partners Increased awareness of SBEAP and Ombudsman Services SB contacts CTAP, partners for assistance SB compliance rates improve • CTAP awareness of service gaps improves • CTAP services are improved • # of sectors identified and # of compliance tools developed • New compliance projects are implemented CTAP • Support increases for SB needs and concerns in the regulatory process • Coordination increases among partners • More SB friendly regulations are written • SBEAP services are expanded • FACILITATION • Coordinate CAP activities • Coordinate conference and award ceremonies • # of CAP meetings/activities • # of conferences coordinated Increased understanding of regulatory burden on SBs CAP, CTAP, SB, Partners, Decision-makers • PROFESSIONAL DEVELOPMENT • Attend conferences and meetings Increased professional knowledge and network • Ombudsman is seen as value added to state • # of conferences/meetings attended SBO • Ombudsman improves services

14. Compliance Advisory Panel Outcomes Resources Activities Outputs Customers Short-term Intermediate Long-term Ultimate Goal • Members • Small Business (SB) knowledge as represented on the CAP • Small Business Ombudsman (SBO) • EPA Office of the Small Business Ombudsman • Compliance and Technical Assistance Program (CTAP) • National Compliance Advisory Panel (NCAP) • Agencies • Partners • ADVISORY • Advises CTAP, SBO, and Agency on SB compliance issues • Advises SBO and CTAP on ways to deal with SB • Identifies areas in which CTAP and SBO should be working # of recommendations for SBO and CTAP # of sectors identified for SBO and CTAP # of regulations reviewed Increased understanding of SB concerns / problems • New SB compliance projects are implemented • SB services improve • SB compliance rates improved • SB perceptions of Agency changes • Environment improves • Economy grows • SBEAP becomes known as steward and valued resource to the community Compliance and Technical Assistance Programs (CTAP) and Small Business Ombudsman (SBO) • REVIEW • Review applicants for awards programs • Review CTAP and SBO compliance materials • Review regulations for SB impact when applicable # of award applicants reviewed # of materials reviewed # of regulations reviewed Increased awareness of type of compliance tools necessary SBEAP tools are improved SB compliance rates improved • Support increases for SB needs and concerns in the regulatory process • Regulatory changes are made • Legislative support increases for SBEAP • State is an SB friendly environment • SBs locate to host state • SBEAP services are expanded • Increased understanding of regulatory burden on small businesses • Increased awareness of SBEAP importance Legislators and Other Decision Makers • ADVOCATE • Lobbies on behalf of SBEAP # of Lobbying activities • ALL ENCOMPASSING • Attend quarterly meetings and conferences # of meetings held and conferences attended SBO; CTAP Agencies; Partners; Decision Makers Increased professional knowledge and network Increased communication and coordination between all customers State SB Services are improved

15. Key Steps in the Measurement Development Process • Conducted two online Performance measurement training sessions • Used GoToMeeting (Sponsored by KSBDC) • Identified measures in existing documents • Reviewed SBEAP Report to Congress measures • Reviewed the SBEAP logic model and identified/selected priorities • Focused on Technical Assistance Program Model • Developed performance measures for logic model elements

16. SBEAP Measures SBEAP PERFORMANCE MEASURES

20. SBEAP Measures WORK QUALITY MEASURES

23. Benefits and Challenges of Common Measures • Benefits • Creates uniformity in measures reported • Provides information for comparison of performance • Challenges • Diversity among the SBEAP State programs • Burden for obtaining/collecting measurement information • Limited or lack of resources to collect measurement information

24. Next Steps • Take draft measures out to comment for the entire SBEAP group- Online Survey • Develop Guidelines and Tools for gathering data on these measures • Work with Promotional and Multimedia subcommittees • Help develop a “Value Added” Report on SBEAP activities and outcomes nationwide

25. Contacts Kenya Stump Kentucky Small Business Environmental Assistance Program Small Business Environmental Assistance Program Measurement Sub-committee chair (800) 562-2327 kenya.stump@uky.edu Yvonne M. Watson Evaluation Support Division National Center for Environmental Information U.S. Environmental Protection Agency (202) 566-2339 watson.yvonne@epa.gov