Framework for Integrating Program Design, Management, Accountability, and Evaluation in Federal Environmental Research P

1. Session 853Extending Organizational Capacity and Capability to Evaluate Federal Environmental Research ProgramsDeveloping a Framework that Integrates Program Design, Management, Accountability & EvaluationDale Pahl* and Emma NorlandU.S. EPA Office of Research and Development October 29, 20052005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation Association*Corresponding author contact: pahl.dale@epa.gov

2. Presentation Focus • Question: How can we develop a framework that • Integrates program design, management, accountability, and evaluation? • Responds to OMB guidance about the Research & Development Investment Criteria? • Communicates clearly—to evaluators, clients, and external stakeholders—about the program’s environmental research and outcomes?

3. Presentation Focus • Response: Articulating the program theory for EPA’s environmental research creates a logical framework that . . . • Integrates program design, management, accountability, & evaluation; • Engages research managers, clients, scientists, and stakeholders across the program’s scope and lifetime; and • Enables independent expert panels to evaluate evidence about program relevance, quality, performance, and leadership—with client input

4. Communicating Program Theory for Environmental ResearchHelps Integrate Design, Management, Accountability, & Evaluation Clients use research (short-term outcomes) … … e.g., to make environmental decisions Clients Research Program Outcomes and Environmental Results Clients Resources Research Topics & Activities Research Outputs Short-Term Outcomes Intermediate Outcomes Long-Term Outcomes Effective Transfer Specific Organizations & Individuals Client Decisions & Actions Strategic Goals & Objectives Mission Program Managers Have Direct Influence Program Managers Have Direct Control Agencies Have Indirect Impact

5. Communicating Program Theory for Environmental ResearchHelps Integrate Design, Management, Accountability, & Evaluation Programs are designed from RIGHT to LEFT analysis synthesis Short-term Outcomes Intermediate Outcomes (e.g., Improved environmental quality, reduced human exposure) Long-Term Outcomes (e.g., Improved human & ecosystem health) Research Topics & Activities Specific Clients Key Research Outputs Resources Intended Changes in Decisions or Actions by Specific Clients )

6. Environmental Outcomes, Risk Assessment, & AccountabilityAdapted from Presentations to EPA’s Board of Scientific Counselors by Hugh Tilson, Larry Cupitt, and John Vandenberg Source Emissions Health Impacts Fate and Transport Early Signs of Effects Ambient Conditions Exposure And Dose • Risk assessment helps identify & prioritize scientific questions & knowledge gaps across a program’s environmental outcomes • Risk assessment is essential to help: • Decide whether or not to take regulatory action . . . Is there an environmental hazard? • Decide what actions are most effective . . . What actions do we take to protect human health? • Understand how to evaluate the effectiveness of our decisions . . . Were we effective?

7. Environmental Outcomes, Risk Assessment, & AccountabilityAdapted from Risk Assessment in the Federal Government: Managing the Process (NRC, 1983); 1997 Update to ORD’s Strategic Plan (EPA, 1997); and OIG-ORD Presentation to EPA’s Deputy Administrator (Pahl & Norland, March 2002) RISK ASSESSMENT RISK MANAGEMENT EXPOSURE 2. Implementation DecisionsManagers make decisions about how to implement, comply with & enforce regulations or how to remedy environmental problems. Pollution Sources Biological Effect Adverse Health Effect Emissions Legal Considerations Ecosystem and Human Health Considerations Internal Dose Health Assessment Transport & Transformation Dose- Response Assessment Risk Management Options Exposure-Dose Relationships Risk Characterization Environmental Concentrations 1.Environmental Decisions and Regulations Hazard Identification Social, Economic, & Political Factors Exposures Exposure Assessment 3. AccountabilityDeveloping and measuring appropriate environmental indicators demonstrates whether environmental decisions result in improved human and environmental health. Dale Pahl Nov ‘02

8. Short-Term Outcomes:A Critical Link between Research & Impact Short-Term Outcomes (Research Contributions to Environmental Decisions) Limitations or Gaps in Knowledge about the Environmental Problem Transfer (e.g., dissemination thru publications) Research Topics & Activities Intermediate Outcomes (e.g., improved environmental quality, reduced human exposure) Long-Term Outcomes (e.g., improved human & ecosystem health) Research Outputs Specific Clients Intended Changes in Decisions or Actions by Specific Clients Limitations or Gaps in Attitudes Transfer (e.g., guideline manuals & training) Limitations or Gaps in Skills and Abilities Needed to Respond to the Environmental Problem

9. Short-Term Outcomes: A Critical Link between Research & Impact • Short-term outcomes are achieved when key research contributions are transferred to, and used by, intended clients • Research creates improved knowledge and applications for risk assessment, environmental decisions, regulatory decisions, and accountability: • Setting or revising standards—formal EPA rulemaking • Implementing standards—e.g., Regions or states develop plans to comply with standards, restore ecosystems, or manage environmental exposure & risk • Applying environmental indicators to “measure” progress to achieve regulations and environmental outcomes—e.g., assessing whether legislation & regulations have the intended environmental impact ____1 See, for example: Strategic Research Plan for Particulate Matter, Air Quality Subcommittee of the Committee on the Environment and Natural resources (CENR), December, 2002; Science to Support Rulemaking, EPA Office of Inspector General (OIG) Report 2003-P-00003, November 15, 2002; and Air Quality Management in the United States, National Research Council, 2004.

10. Communicating Program Theory for Environmental ResearchHelps Integrate Design, Management, Accountability, & Evaluation Programs are designed from RIGHT to LEFT analysis synthesis Short-term Outcomes Intermediate Outcomes (e.g., improved environmental quality, reduced human exposure) Long-Term Outcomes (e.g., improved human & ecosystem health) Research Topics & Activities Specific Clients Key Research Outputs Resources Intended Changes in Decisions or Actions by Specific Clients )

11. Organizing Research, Topics, & Activities • Both applied1 and use-inspired basic research2 contribute to regulatory decisions and improved understanding about environmental outcomes: • Applied1 environmental research is targeted at understanding and solving particular environmental problems. • Basic2 environmental research elucidates problems that involve complex environmental processes and nonlinear systems with multiple causes and effects. • EPA’s basic research is “use-inspired” basic research2—targeted to answering questions about complex problems that cut across EPA’s programs to ensure that decisions are based on a foundation of sound science. • Typically, the distinction between these types of research is not clear cut. __________ 1 EPA's applied and basic research programs implement recommendations from the National Research Council in Building a Foundation for Sound Environmental Decisions, (NRC, 1997). 2 Stokes, D.E. “Renewing the Compact between Science and Government,” in 1995 Forum Proceedings, Vannevar Bush II—Science for the 21st Century. Pages 15-32. Sigma Xi, 1995.

12. Organizing Research, Topics, & Activities Identify existing and emerging issues for a specific problem Use risk assessment to rank issues and pinpoint largest uncertainties FEEDBACK Narrow EPA focus based on client needs and recognition of what others are doing Identify research topics that improve understanding, reduce uncertainties, and develop client applications Adapted from National Research Council, Building a Foundation for Sound Environmental Decisions (NRC, 1997). Problem-Driven Research Core Research Elucidation of Complex Environmental Processes Development of Tools Collection of Data Re-evaluate priorities regularly Select projects based on broad applicability, relevance to EPA, and scientific merit.

13. Organizing Research, Topics, & Activities • Question: • How do we organize and “measure” research when there are no objective methods to: • Measure new knowledge as it develops? • Manage the pace at which research progresses? • Measure research quality and impact?

14. Organizing Research, Topics, & Activities • Response: • Organize the research with priority research topics • Assess research progress and priorities with periodic meetings • Convene independent expert panels to evaluate the improved knowledge and its applications with indicators that span a program’s scope & lifetime __________ 1 For example, see Averch, H.A., “The Systematic Use of Expert Judgment,” pages 294-295 in Handbook of Practical Program Evaluation. J.S. Wholey, H.P. Hatry, and K.E. Newcomer (eds). San Francisco, Jossey-Bass, 1994. 2 The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be Uneven. U.S. GAO/GGD (1997)

15. Developing Indicators Across a Program’s Scope & Lifetime Very useful Useful Limited usefulness

16. The Importance of Independent Expert Evaluation • The systematic use of independent expert judgment is important1,2 when: • It is difficult to measure program progress or outputs (e.g., advances in research knowledge) needed to achieve outcomes • Multidisciplinary expertise is needed to evaluate scientific progress that responds to research topics and scientific questions • It is difficult to determine when outcomes can be attributed to the program • Agency programs involve research, regulation, or external partners such as state agencies • These criteria illustrate why independent expert review is important for evaluating EPA’s research programs _____ 1 For example, see Averch, H.A., “The Systematic Use of Expert Judgment,” pages 294-295 in Handbook of Practical Program Evaluation. J.S. Wholey, H.P. Hatry, and K.E. Newcomer (eds). San Francisco, Jossey-Bass, 1994. 2 The Government Performance and Results Act: 1997 Governmentwide Implementation Will Be Uneven. U.S. GAO/GGD (1997)

17. Summary: • EPA follows a systematic approach1,2 to organize, integrate, synthesize, and evaluate research that informs environmental decisions by specific clients: • Develop an integrated risk assessment framework to synthesize available information about a specific environmental problem • Assess client needs, knowledge gaps, research questions, and uncertainties • Create research topics needed to develop knowledge & evaluate evidence related to the gaps, questions, and uncertainties • Monitor progress in implementing the research topics and in applying improved knowledge base • Evaluate the improved knowledge base and client applications • The next slide illustrates how program theory creates a framework that integrates these objectives . . . __________ 1 For an example of EPA’s systematic approach, see Strategic Research Plan for Particulate Matter, Air Quality Subcommittee of the Committee on the Environment and Natural resources (CENR), December, 2002. 2For additional background information ,see Risk Assessment in the Federal Government: Managing the Process. National Research Council, 1983; and Environmental Research and Development: Strengthening the Federal Infrastructure. Carnegie Commission on Science, Technology and Government, 1992.

18. Programs are designed from RIGHT TO LEFT HOW? WHAT? The right science WHY?The right questions Environmental Outcomes and Strategic Goals Resources & Inputs Specific Clients Short-Term Outcomes Long-Term Outcomes Research Outputs Research Topics & Activities