Technical Assistance Guide Prepared for Web Cast on AT Act Outcomes Wednesday, October 22, 2003

1. Understanding the Government-Wide Shift in Accountability: Using Performance Management to Plan for Outcomes & Demonstrate Results Technical Assistance Guide Prepared for Web Cast on AT Act Outcomes Wednesday, October 22, 2003 by Margaret L. Campbell, Ph.D. National Institute on Disability and Rehabilitation Research U.S. Department of Education

2. Toprovide NIDRR staff, grantees, applicants and reviewers with a comprehensive resource for understanding the government-wide shift in accountability and how the concepts and tools of performance management can be used to: Plan for outputs & outcomes Improve programs Monitor progress, and Document results and communicate successes Purpose of Technical Assistance Guide

3. Part 1: The “What and the Why” of the Accountability Shift

4. What is the Government-Wide Shift in Accountability? • Shift from primary focus on: • Dollars(how much is spent & on what) • Activities& processes (what you are doing to/with whom and how well you are doing it) • Productivity– (how much are you doing) • To an expanded focus on: • Performance -- (are you meeting stated objectives and standards of quality), and • Results – (what you are achieving and who’s benefiting?)

5. Shift in Accountability is Not “Either- Or” Situation • Rather amounts to an added dimension of accountability for “results.” • Still must meet accepted standards of practice for: • Continuous quality improvement • Program development and delivery of services, and • Consumer involvement • Systems change & advocacy

6. What Does “Accountability for Results” Mean? • In general, “accountability for results” refers toaccomplishments that are under the direct or indirect influence of project resources and activities & occur within the boundaries of a funding/budget cycle. • Within the NIDRR context, this translates into accountable for three types of results: • Outputs– planned and produced to date • Short-Term Outcomes– anticipated and actual, and in some cases • Intermediate Outcomes– anticipated or actual

7. Origins of the Accountability Shift in Government: GPRA • In 1993, Congress passed the Government Results and Reporting Act, which established uniform requirements for strategic planning, annual planning and reporting, and agency-grantee partnerships. • Key Provision of GPRA: Provides a new definition of program evaluation – “an assessment, through objective measurement and statistical analysis, of the manner and extent to which federal programs achieve intended objectives.”

8. Recent Policy Initiatives Reinforcingthe Accountability Shift • The President’s Management Agenda (PMA) FY 2002, with its emphasis on: • Performance management & measurement • Objective criteria to assess program results (i.e., the PART) • Performance-based budgeting • The Dept. of Education’s New Strategic Goals, 2002-2007, with an emphasis on: • Transforming Education into an Evidenced-based Field.

9. Part 2: The Language and Logic of Outcomes Planning and Managing for Results (Key Concepts and Definitions)

10. What is Meant by “Managing for Results?” • “Managing for results” requires developing a new mindset based on outcomes-oriented goals and performance measures,as opposed to activity-oriented goals and objectives . • The primary focus is on planning for what outputs and outcomes will be achieved not on what activities will be conducted. • For most of us, this involves re-expressing existing program goals and objectives into the language of planned outputs and anticipated outcomes.

11. Concepts and Tools You Need to Succeed • Performance management • Performance measurement • Performance goals & measures • The Performance Spectrum • Logic modeling – i.e., reverse mapping • Components of Performance measurement • Target systems and/or populations

12. What is Performance Management vs. Performance Measurement? • Performance management is an approach to planning and evaluation that utilizes the concepts and tools of performance measurement and logic modelingto identify performance goals and assess progress towards goals. • Performance measurement “involves the ongoing monitoring and reporting of program accomplishments, particularly progress towards pre-established goals” (see GPRA requirements).

13. What are Performance Goals? • According to OMB, performance goals are the desired levels of performance expressed as measurable objectives against which actual achievement can be compared. • To be complete, performance goals should incorporate performance measures and performance targets into timeframes.

14. Definitions of Performance Goals (Cont’d.) • Performance measures are the quantitative or qualitative metrics and indicators that are used to demonstrate progress towards goals and document results/ accomplishments. • Performance targets are the quantifiable or otherwise measurable characteristics that tell how well a program or project must accomplish a performance measure.

15. The Spectrum of Performance: Outcomes-Oriented Goals • Although the PART emphasizes outcome goals, that focus on changes and/or improvements in the target system, performance goals can also be stated in terms of “planned outputs” (and sometimes “quality of activities”). • The key: All performance goals should clearly distinguish between components of performance and there should be a logical connections among them with outputs supporting outcomes(i.e., outcome-oriented performance goals).

16. What is Logic Modeling? • A vehicle for dialogue, planning, program improvement and evaluation • A graphic representation or “blueprint” of the key elements of a program or project, and how it will work under certain conditions to “solve” identified problems. • A helpful tool for identifying outcomes and depicting the “chain of events” that link inputs and activities to outputs and outcomes.

17. What Does A Basic Logic Model Look Like? Prepared by the NIDRR PPB&E Division1 HOW WHY Span of Accountability for Results Inputs & Infrastructure Program Activities Outputs Short-Term Outcomes Intermediate Outcomes Longer Term Outcomes Customers Who Use Outputs PROGRAM DELIVERED RESULTS FROM PROGRAM EXTERNAL CONDITIONS INFLUENCING SUCCESS (+/-) ANTECEDENT/ MEDIATING MEASURES 1Source: Adapted from McLaughlin, J. A. & Jordan, G. B. (In Press/2003) Logic models: a tool for describing program theory and performance, in Wholey, et. al.: Handbook of Practical Evaluation. Jossey-Bass. The lead author may be contacted by email at:macgroupx@aol.com Note:The above logic model does not include the standards by which the quality, relevance, and productivity of performance will be measured and evaluated across the spectrum. For NIDRR grantees this is provided by the “centers of excellence” (CoE) criteria. Updated on June 27, 2003

18. What are the Components of Performance Measurement? • Inputs & Infrastructure: The human and financial resources and systems needed to conduct a high quality, outcomes-oriented program or project. • Examples: agency priorities and requirements, host institutional support and leveraging, staff competencies and expertise, management practices and evaluation plans, partnerships and collaborations, and previous accomplishments.

19. What are the Components of Performance Measurement? (Cont’d) • Program Activities: Are the action steps, tasks, procedures, and services performed in conjunction with implementing your planned program of public awareness, technical assistance and training, Outreach, and interagency coordination to carry out objectives and produce results.

20. What are the Components of Performance Measurement? (Cont’d) • Outputs: Are the direct results of program activities and consist of the findings or conclusions, products and services produced and reported to external audiences. • Outputs typically arereported using statistics or quantitative counts– e.g., the # of individuals trained, newsletters distributed, publications produced, & participants attended. • But they also can be described qualitatively in terms of the nature of the findings or discoveries.

21. What are the Components of Performance Measurement? (Cont’d) • More On Outputs: • Outputs are important in their own right as indicators of productivity; they also are the essential building blocks of outcomes. • However,not every output will have a corresponding outcome, nor should it.

22. What are the Components of Performance Measurement? (Cont’d) • Utility: Although not an official component of the performance spectrum as defined by GPRA, NIDRR has included utility because of its centrality to our Mission, and because demonstrating the “perceived usefulness” of outputs to sub-groups within the broader target systems constitutes an important bridge between outputs and short-term outcomes.

23. What are the Components of Performance Measurement (Cont’d) • Outcomes: Are the anticipated or actual effects of program activities and outputs and constitute changes or improvements in the target systems being served. Although the emphasis is on planning for outcomes, outcomes may also occur as unexpected changes. • Outcomes are the most complex component of PM and are difficult to define outside of the context of a specific program and target system.

24. What are the Components of Performance Measurement (Cont’d) • Factors to take into account in planning for and documenting outcomes: • The basic outcomes equation is: Outcomes = change/improvement in the target population or system, which is a function of: • Outputs + Documented External Use • Outcomes typicallyoccur sequentially over time and, as a result, may affect more than one target system. Because of this, we refer to “chain of outcomes,” starting with short-term outcomes and progressing to intermediate and longer-term outcomes.

25. Types of Outcomes • Short-term outcomes are expected or actual changes or improvements in the identified target system that are more under the direct influence of center activities. Short-term outcomes represent the first level of change that must occur in order to bring about intermediate outcomes. Typically, short-term outcomes are focused on changes/improvements in the learning & knowledge system at the individual or environmental level of analysis.

26. Types of Outcomes • Intermediate outcomes are the “gold-standard” in terms of accountability for results. They consist of expected or actual changes or improvements in the “action” system that occur in part as a result of the use or adoption of project outputs. Unlike short-term outcomes that occur under the direct influence of program activities, there may be other causal factors contributing to the achievement of intermediate.

27. Types of Outcomes (Cont’d) • Longer-term outcomes are the desired end-results of a program, and constitute changes or improvement in the overall condition of a population or system. Given their scope, longer-term outcomes usually take more than one funding cycle to achieve, and therefore are not required under the “accountability for results” umbrella. Their primary function is to serve as critical anchor points in the planning process.

28. Examples of Types of Outcomes (Changes or Improvements) at the Individual Level

29. Examples of Types of Outcomes (Changes or Improvements) at the Environmental Level

30. What is a Target System? • Formally defined, a “target system” refers to a unit of analysis or bounded set of interacting elements. • In the NIDRR context, a target system or population refers to the level(s) of the environment and/or to the group(s) of individuals that program activities and outputs are intended to affect. • Specifying the particular sub-groups or sample populations within the broader target system that your program intends to affect is key to identifying anticipated outcomes.

31. Examples of the Types of Target Systems Typically Involved in NIDRR-Funded Projects • Individual level systems: persons with disability and/or their family members, clinicians, service providers, other researchers, policy makers, and industry representatives. • Environmental-level systems: the knowledge base in a field, treatment/practice standards, product development standards, availability of new methods, tools, products, services, and technologies, policies and/or service delivery systems, manufacturers, and societal conditions.

32. Part 3: How to Apply Performance Management Tools to Plan for Outcomes & Manage for Results

33. Objective of Outcomes Planning To identify a limited number of high-priority, problem-focused, reasonably ambitious and measurable outcomes-oriented performance goals that: • Reflect NIDRR priorities or statutory requirements, and • Capture the anticipated effects and benefits of project activities & outputs on identified target populations and/or systems.

34. Two Ways to Get to Outcomes:Prospective vs. Retrospective Planning • Prospective outcomes planning is the preferred strategy, but it requires starting with a “managing for results” mindset and identifying performance goals and measures to guide activities and monitor progress (slides 35- 42 ). • Retrospective outcomes planning is less than ideal, but is how most of us beginning to get our feet wet. It involves identifying outcomes after-the-fact and collecting data to support them (slides .

35. Prospective Outcomes Planning Step 1: Preparation • Describe the overall problem you are trying to “solve” and specify what part or parts of the problem will be addressed by your program activities (i.e., theory of the problem clarifies scope of your project). • Identify the specific sub-groups or sample populations within the broader target systems that you anticipate will change/ improve as a result of your program. • Explain how your program activities & outputs will change or benefit the specific target systems or populations you have identified (i.e., theory of the program)

36. Prospective Outcomes Planning Step 2: Constructing a Logic Model • Translate requirements and objectives of program into anticipated outcomes, starting with longer-term, end-results and working backwards to identify intermediate and short-term outcomes (see Worksheet below); • Identify the specific target populations and systems to be affected by each type of outcome • Plan the outputs that will be produced to bring about anticipated changes in target systems; • Design the activities that must be conducted to produce planned outputs; • Identify the resources needed to implement activities; and • Document the contextual factors that could influence the success of your program.

37. Worksheet for Constructing a Logic Model of Performance for Your Program Prepared by the NIDRR PPB&E Division1 Important Tip: When planning, begin with the end in mind and work from right to left. When implementing, begin with inputs and infrastructure and move from left to right Span of Accountability for Results Short-Term Outcomes Intermediate Outcomes Inputs and Infrastructure Program Activities Longer-term Outcomes Outputs Utility What resources and management practices are you using, or developing, to conduct planned activities and produce desired results consistent with the priority and program requirements? What evidence do you have of the potential merit and/or usefulness of your outputs? What changes or improvements will occur in awareness, understanding or knowledge and for which target populations as a result of your program activities and outputs? And, what measurable indicators will you use to track progress towards achieving these short-term outcomes/ performance goals? What activities are you currently conducting, or plan to conduct, across program requirements to fulfill objectives, produce intended outputs, and assess their relevance and utility for target audiences? How well are you implementing these activities; and what is your progress to date, including problems encountered and actions taken? What products, services, and information are you providing, or do you plan to provide to target audiences or customers to support anticipated outcomes? And what is your productivity to date? What changes or improvements do you anticipate will occur in behavior, practice and/or policy, and for which target populations as a partial result of the use or adoption of new knowledge generated by your program? And, what measurable indicators will you use to track progress towards achieving these intermediate outcomes/ performance goals? What are the long-term changes or improvements in societal conditions that you hope to contribute to overtime through your program activities, outputs and short-term and intermediate outcomes? Internal and External Factors That Can Influence Success 1Source: Adapted from McLaughlin, J. A. & Jordan, G. B. (In Press/2003) Logic models: a tool for describing program theory and performance, in Wholey, et. al.: Handbook of Practical Evaluation. Jossey-Bass. The lead author may be contacted by email at:macgroupx@aol.com Note:The above logic model does not include the standards by which the quality, relevance, and productivity of performance will be measured and evaluated across the spectrum. For NIDRR centers and model systems this is provided by the “centers of excellence” (CoE) criteria. Revised: June 27, 2003

38. Prospective Outcomes Planning Step 3: Formulate Performance Goals • Convert outcome statements into performance goals by: • Identifying or developing the performance measures that will be used to document progress and demonstrate accountability for results (i.e., outputs, short-term and intermediate outcomes). • Establishing a baseline and identifying performance targets that must be met to achieve outcomes. • Establishing the timeframe within which progress & results will occur.

39. Prospective Outcomes PlanningStep 4: Evaluation of Performance • Develop an evaluation plan that identifies the types of evidence you need to collect to determine how well your program is working and if it is achieving the anticipated results. • Specify the sources and methods of data collection you will use to measure performance and demonstrate progress towards outcomes-oriented goals (see Table on next slide forexamples of data sources & methods).

40. Source of Information Method of Collecting Information • Participants (consumers, persons with disabilities) • Usability Assessments, pre/post tests • Family Members • Observation • End-users & Other Stakeholders • Interviews or surveys • Key Informants • Focus Group • Practitioners • Record reviews • Documents • Citation analysis • Data sets/ public use data files • Literature Reviews • Secondary Data Analysis Examples of Sources and Methods of Data Collection for Demonstrating Outcomes

41. Prospective Outcomes PlanningStep 5: Evaluation of Performance • Integrate your performance management plan into the ongoing operation of your program activities to measure performance, collect outcomes data and monitor progress.

42. Prospective Outcomes PlanningStep 6: Analysis & Documentation Analyze the qualitative and quantitative performance data to documentprogress towards anticipated outcomes. Interpret the data to determine if an outcome has occurred or is likely to occur within the specified timeframe. Making this determination relies on qualitative judgments of quantitative and/or qualitative data. Document results and communicateaccomplishments to external audiences is an evidence-based claims-making process, in which grantees present evidence to support the claim that their program activities & outputs have resulted in changes/ improvements in the identified target system.

43. Retrospective Outcome Identification: Problems with Approach • Missing pre-established performance goals to guide activities and outputs • Missing specification of target systems where outcomes are expected to occur • Missing a logic model that clarifies the interconnections among activities, outputs and outcomes and between short-term and intermediate outcomes • Missing performance measures and systematic outcomes data to monitor progress and demonstrate results

44. Retrospective Outcome Identification: Suggested Implementation Strategies • Form an AT Act Work Group to translate statutory requirements into a “limited number of high-priority” (i.e., 2-3) and measurable outcomes-oriented performance goals for each core activity: public awareness, TA & training, outreach, and interagency coordination. (suggested # of outcomes is • Review and categorize accomplishments produced to date into outputs and types of outcomes (short-term, intermediate and long-term).

45. Retrospective Outcome Identification: Implementation Strategies (Cont’d.) • Link accomplishments to newly formulated performance goals and identify the types of data available and the data elements needed to demonstrate progress towards outcomes and document the contribution of AT Act project activities and outputs. • Revise web-based annual performance reporting form for FY 2004 to collect necessary data and rethink how existing data can be analyzed to provide evidence of progress or achievement of outcomes.

46. Common Weaknesses in Prospective & Retrospective Outcomes Identification 1. Lack of specificity in “what” and “where” 2. Too many outcomes and expressed too narrowly or too broadly 3. Outcomes expressed as speculations vs. as expectations 4. Confusion between levels of the performance spectrum (i.e., activities vs. outputs and outcomes and short-term vs. intermediate outcomes) 5. Missing supporting links between outputs and short-term outcomes 7. Missing evidence of outcomes

47. Exercise: What’s Missing in the Following AT Act Project Outcomes? • Outreach: Within the last grant cycle, State X provided technical assistance to 4, 127 individual and demonstration of loan of equipment to 1,415 individuals. • Training: State X provided 102 training sessions attended by 3,8288 which resulted in increased knowledge levels that have prompted systems to address AT needs and responsibilities within policies and procedures.

48. Exercise: Illustration of Reformulated Outcome Statement • Original Statement Systems Change Intermediate Outcome: State AT program worked with the state election officials on the HAVA state plan which resulted in adoption of the FEC 2002 access standards for voting machines. • Suggested Reformulation: Conduct legislative initiatives that result in the development (short-term outcome) and adoption by states (intermediate outcome) of new standards for voting machines to increase access for adults with moderate to severe disabilities.

49. Part 4: Lessons Learned: The Utility and Challenges of Implementing Performance Measurement

50. Benefits of Utilizing Performance Management • Provides a common language and integrated framework to facilitate: • Strategic planning & program design • Identification of performance goals • Program integration – coordinates the contributions of multiple projects and tasks • Objective measurement of progress towards goals • Program monitoring & improvement • Program accountability -- documentation of results, and • Communication of success stories