Distinguishing Inputs, Outputs, Outcomes and Impacts
Download
1 / 89

Distinguishing Inputs, Outputs, Outcomes and Impacts - PowerPoint PPT Presentation


Distinguishing Inputs, Outputs, Outcomes and Impacts. www.summerinstitute.eu. Important points to start with. Decision making. Policy analysis. Limited rationality. Incentives. Process and learning. Contents. Outputs & inputs. Outcomes. Indicators. Logic model. Budgeting & PI.

Related searches for Distinguishing Inputs, Outputs, Outcomes and Impacts

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha

Download Presentation

Distinguishing Inputs, Outputs, Outcomes and Impacts

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Distinguishing Inputs, Outputs, Outcomes and Impacts

www.summerinstitute.eu


Important points to start with

  • Decision making

  • Policy analysis

  • Limited rationality

  • Incentives

  • Process and learning


Contents

  • Outputs & inputs

  • Outcomes

  • Indicators

  • Logic model

  • Budgeting & PI

  • Future


Contents

  • Outputs & inputs

  • Outcomes

  • Indicators

  • Logic model

  • Budgeting & PI

  • Future


Outputs - definition

  • Services or goods which are provided by ministries and other state organizations to external stakeholders / beneficiaries (directly or through other organizations)

  • External stakeholders include: general public, citizens, businesses, NGOs, media and other state bodies including the minister, other ministries and state organizations, and the parliament


Examples

  • Provision of prosthetic devices to disabled

  • Provision of schooling

  • Health centre services

  • Analysis and policy advice

  • Laws

  • Inspections

  • Training


Outputs should not be

  • Activities

  • Processes

  • Inter-mediate output

  • Capacity building initiatives


Example: output – schooling

  • Number of graduates from school

  • Activity - teaching

  • Process - recruitment of teachers

  • IMO – Number of enrolled chidlren

  • Capacity building – increasing professional qualification of teachers


Consumption or provision?

  • Economic theory view: the delivery of service takes place between producer and consumer, and therefore transaction is complete, when service is used (output consumed)

  • Theatre plays

  • Patients checked

  • Public administration view: for many outputs it does not mater whether they are consumed or not – the service should be there in case it is needed

  • Emergency services

  • Prison services


Output characteristics

  • Volume or quantity of provision

  • Timeliness in provision

  • Quality of provision

  • Satisfaction - surveys and evaluation

  • Comparison – to others or standard

  • Coverage, meting of demand and accessibility

  • Equity


Example: output – provision of prosthetic devices

  • Volume – number of disabled people served

  • Timeliness – service provided in specific time – for example, one week after request

  • Quality – satisfaction of users of devices

  • Coverage – % of people in need of devices served

  • Equity – people with access difficulties served at home or transported to the service point


Selectivity

  • Not all output characteristics can/should be used for all outputs – be selective

  • Focus on key outputs for external users

  • For all other outputs – have at least the volume


Aspects to consider

  • Control of provision

  • Other government interventions

  • Measurability

  • Attribution

  • Users

  • Incentives & gaming


Control of provision

  • Provision of a single output is controlled (directly or indirectly) by government organization

  • But not its effectiveness (achieving desired policy objective). Many government outputs make sense only when used together with other outputs

  • Road safety and reduction of fatal accidents depends ona group of outputs – quality of road, speed control, technical state of cars, use of seat belts.


Control of provision (contd.)

Many organizations

Policy

Budgeting

Delivery

One or more organizations

One organization


Other government interventions

  • To implement policies governments provide not only goods and services, but also:

  • regulation

  • capital investments

  • social and various other benefits & compensations

  • subsidies

  • If these are not outputs, how do they relate to them?


Other government interventions(contd.)

  • It can be helpful to think of three broad categories

  • goods and services = outputs

  • administered items

  • capital investment

  • mandatory benefits & compensations

  • subsidies and other payments

  • general regulation

  • But remember – most of administred items and general regulation require outputs


Example – road safety

  • In budgets, some countries separate outputs / investment / financing in separate programs while others do not (but separate using economic classification)


Measurability

  • Some outputs are harder to measure than others due to the nature of organizations providing them

Measurement difficulty

Embassies

Army

Residential care

Ambiguity

(output)

Road police

Job Counselling

Training institutions

Unemployment agencies

Policy making ministries

Routine

(process)


Attribution

  • Outputs mean something if they can be attributed to some objective (outcome) – they contribute to achieving of objective, e.g. Change in outcome

  • Generalized attribution: re-training of unemployed contributing to reduction of unemployment

  • Specific: re-training of unemployed contributing to employability of the trained person during the next year after training


Difficulties in establishing attribution

Clear attribution:

  • Fixed arm

  • Medical service

  • Broken arm

Less clear attribution:

  • Survival

  • Services

  • Cancer

  • Earlier diagnosis?

  • Better treatment?

  • Healthier life style?

  • Beneficial effects of affluence?


Users

  • Different PI users and decision making stages might have different information needs (and therefore require different formats of presentation of PI)

Parliament

Government

Ministry of Finance

Minister

Permanent Secretary

Unit Manager

Employee

Output focus

  • Using of PI means analysis and learning rather than money and / or promotion


Principal – Agent Problem

  • Principals are “owners” and “bosses”, but they lack time and expertise to do everything

  • Therefore principals hire agents, who have time and expertise, but are self interested

Modern

Medieval

Business

Principal

King

Shareholder

People

Agent

Lords

CEO

Bureaucrats

Delegated authority

Tax collection

Profit making

Service delivery


Principal – Agent Problem(contd.)

  • Agents have one strong advantage over principals - information

  • This creates problem called “gaming”, which is based on two factors:

  • There is information asymmetry between the principal and agent; and

  • There are wrong incentives for disclosing or hiding information


Gaming & incentives

  • Bevan and Hood: "reactive subversion such as 'hitting the target and missing the point' and / or reducing performance where targets do not apply"

  • Gaming = manipulation with information that might affect service delivery

  • Incentives = factors (financial or non-financial) that provide motives for a particular course of action, or count as a reason for preferring one choice to the alternatives

  • Targets are the most common causes for perverse incentives


Gaming & incentives

(contd.)

High

Gaming potential

Information asymmetry

Low

Loose

Tight

Relationship with decision-making


Impact of gaming

Strategy A

Strategy B

Strategy C

Strategy D

Arrival on time at final destination

“Saboteur” manipulation of service and data

“Smart Liar” or manipulation of data

“False Champion” or service manipulation

“Honest Fool” or no manipulation

TARGET


Impact of gaming (contd.)

  • Gaming can result in:

  • Cherry picking – handling of easy cases, patients etc. at the expense of difficult ones

  • Ratchet effect – slowing down today to be on time tomorrow

  • Distortions – improving areas where measurement is dome while neglecting other areas

  • Threshold effect – focusing on 2nd best and forgetting about the best and worst, as best will be achieved anyway while worst require too much of an effort


Dealing with gaming

  • Learning and data improvements

  • Avoiding strong negative incentives

  • Clarify use of PI in advance

  • “Loose” rather than “tight” connection to decisio making

  • Provide feedback on data relevance and quality

  • Reward those with good PI

  • Do not overload with data – use the key measures


Dealing with gaming(contd.)

  • Dealing with principal – agent problem at all levels

  • Use of “auditors”

  • National Audit Office

  • Internal audit

  • Encourage ownership at production level


Summarizing about outputs

  • Goods and services

  • Basic element in performance information system

  • Can be measured in different ways, but there are difficulties

  • Thee are three most common difficulties:

  • Measurability

  • Attribution to objectives (outcomes)

  • Scope for manipulation

  • Loose link between output data and decision making generally is a better incentive to avoid gaming


Inputs - definition

  • Inputs are resources used in the delivery of goods and services

  • Labour and other means of service provision (facilities, computers, teachers, books etc.)

  • Money is not an input. It s a cost of input


Inputs – output combination

More outputs

V

?

Less inputs

More inputs

?

X

Less outputs


Improving performance

More outputs

Production possibility frontier

Less inputs

More inputs

Less outputs


Contents

  • Outputs & inputs

  • Outcomes

  • Indicators

  • Logic model

  • Budgeting & PI

  • Future


Outcomes - definition

  • The effects which government is trying to achieve for the public. They are often expressed as improvement in the living conditions of people or favourable changes that contribute to these changes

  • Outcomes are changes in the economic, physical, social and cultural environments which the state agency(ies) is trying to influence through provision of goods and services, general regulation and financing

  • Outcomes are those events, occurrences, or conditions that are the intended or unintended results of government actions (OECD)


Examples

  • reduction in maternal / child mortality

  • increase in export

  • decrease of unemployment

  • reduced crime levels

  • improved literacy rate of the population

  • reduction in fatalities from road accidents

  • land productivity improvement


Examples (contd.)


Examples (contd.)


Examples (contd.)


State Measures and Change

LATVIA

2004

2005

change %

Number of road accidents

48912

47353

-3.2

Number of injured

6416

5600

-12.7

Number of fatal injuries

442

516

-14.3

Number of registered cars

966242

-7.6

898145


State Measures and Change(contd.)

  • the same data can be called state measure or performance measure

  • it depends whether you attribute policy (outputs, regulation and finances) to it or not

  • if you do attribute – you are interested in change influenced by your policy (the question – whether different policy generates different result, e.g. marginal changes in result depending on inputs)

  • + - 0


State Measures and Change(contd.)

LATVIA

2004

2005

change %

Number of road accidents

48912

47353

-3.2

Number of injured

6416

5600

-12.7

Number of fatal injuries

442

516

-14.3

Number of registered cars

966242

7.6

898145

  • outputs

attribution of policy to change

  • financing

  • regulation


Different outcomes – results & impacts

Impact or longer term effects (FO)

Global objectives

Results or direct & immediate effects (SO)

Specific objectives

Operational objectives

Outputs

Activities

Inputs


Results & impacts defined

  • Results relate to the direct and immediate effect brought about by a policy or program. They provide information on changes to, for example, the behaviour, capacity or performance of direct beneficiaries. Such indicators can be of a physical (reduction in journey times, number of successful trainees, number of roads accidents, etc.) or financial (leverage of private sector resources, decrease in transportation cost) nature

  • Impactsrefer to the consequences of theprogram beyond the immediateeffects on its direct beneficiaries. Two concepts of impact can be defined. Specificimpacts are those effects occurring after a certain lapse of time but which are, nonetheless, directly linked to the action taken. Global impacts are longer-term effectsaffecting a wider population


Examples

Tax collection

Economy

Justice

Inputs

Inputs

Inputs

inputs

Tax audits

Regulatory burden studies

Prisoner rehabilitation

outputs

Reduced tax evasion

Better regulation

Re-integration into community

results or SO

Reduced shadow economy

Enhanced competitiveness

Reduced repeat crimes

impacts or O


Results & impacts defined(contd.)

  • Results = short to medium term outcomes = specific outcomes

  • Impacts = medium to long term or final outcomes = overall outcomes

  • Sometimes immediate, intermediate and final outcomes


Single outcomes and indexes

  • Sometimes indexes are used to describe and compare performance

  • Outcomes from several areas and sometimes even sub-areas are merged into one index

  • These are of particular use for policy planners

  • For example: the Enabling Trade Index


Example: The Enabling Trade Index 2007/8

MARKET ACCESS

T&C INFRASTR.

BUSINESS ENVIRONM.

BOARDER ADMIN.

  • Regulatory environment

  • Availability & quality of transport infrastructure

  • Efficiency of customs administration

  • Tariff & non-tariff barriers

  • Proclivity or openness to trade

  • Physical security

  • Availability & use of ICTs

  • Efficiency of import – export procedures

  • Transparency of boarder administration


Example: The Enabling Trade Index 2007/8 (contd.)


Example: Tourism Competitiveness Index 2008

REGULATORY FRAMEWORK

HUMAN, CULTURAL & NATURAL RESOURCES

BUSINESS ENVIRON. & INFRASTR.

  • Policy & regulations

  • Air transport infrastructure

  • Human resources

  • National tourism perception

  • Environmental regulations

  • Ground transport infrastructure

  • Natural & cultural resources

  • Safety & security

  • Tourism infrastructure

  • Health and hygiene

  • Price competitiveness

  • Prioritization of T&T


Example: The Tourism Competitiveness Index 2008(contd.)


Problems with outcome measurement

  • Timing

  • Data availability for measurement and transaction costs

  • Outcome (state) measures remain important even in the absence of policies and programs directed at them

  • Impact of other influences

  • Attribution


Attribution to outputs

Other influences

Influence of other factors

Degree of outcome attribution to outputs

Attribution

Immediate – intermediate - final


Attribution modelling

Outcomes

IMMEDIATE

INTER

MEDIATE

FINAL

Knowledge & understanding

Changes in environment

Behavioural changes


Attribution modelling(cont.)

Outputs

Inputs

Outcomes

Target group

Outputs

Medium term

Short term

Long term

Improved knowledge about impact of farming on environment

Educational events

Less polluting farming

Systematic pollution monitoring in farms

Staff

Finances

Research

Materials

etc.

Improved understanding about the use of pollutants

Control system

Polluting farmers

Changes to farming practices

Improved understanding about the ways to control pollution

Site visits / inspections

Improved environment


Summarizing about outcomes

  • Outcomes are changes in the economic, physical, social and cultural environments which the state agency(ies) is trying to influence

  • Focus on change (+/-/0) in outcomes is what makes attribution between outputs and outcomes)

  • Outcomes can be of immediate, medium term and long term nature. More immediate they are – stronger the attribution to outputs

  • Outcomes can range from single to more compplex ones based on some kind of a model

  • Outcomes are particularly useful for policy planning and evaluation


Contents

  • Outputs & inputs

  • Outcomes

  • Indicators

  • Logic model

  • Budgeting & PI

  • Future


Indicators - definition

  • An indicator can be defined as the measurement of an objective to be met, a resourcemobilized, an effect obtained, a gauge of quality or a context variable

  • Indicators demonstrate how to measure elements of program that you are interested in or program in general


Different levels of indicators


Examples


Important aspects of use of indicators

  • Output indicators

  • State indicators and baselines

  • Targets

  • Single and ratio indicators


Output indicators

  • Volume or quantity of provision is the most often used and most relevant to budgeting. However, other output dimensions can have indicators, too:

  • Timeliness in provision

  • Quality of provision: satisfaction and/or comparison

  • Coverage, meting of demand and accessibility

  • Equity


Example: technical certification of cars service

  • Volume: X number of cars per years

  • Coverage & demand: 100% every day

  • Timeliness in provision: less than ½ hour waiting, service within 1 hour

  • Quality: satisfaction of car users; quality in weakest areas improved; online registration etc.

  • Equity: special service for disabled


State indicators and baselines

  • State indicators describe the state of affairs in a given area of policy concern – unemployment, GDP, mortality, lack of labour mobility, demand for a service exceeding supply etc.

  • Baseline indicators refer to the initial value against which inputs, outputs or outcomes are subsequently measured.

  • State indicator = photograph of situation; Baseline indicator = starting point of measuring impact of policy


Targets

  • Show desired level of performance

  • Make sense only if time bound and have baseline

  • Customer satisfaction from a service increased from 25 to 75% by June 2009

  • Every input, outputs and outcome can have indicators, but should it?

  • UK PSAs: from 300+ targets in 1998 to 30 targets in 2007

  • Often, if you want to have loose connectionto accountability and control, simple expected performance projections can work better than targets


Static and dynamic targets

STATIC CONCEPT

DYNAMIC CONCEPT

Value

Needs

Needs

Value

Target

Target

Baseline

Baseline

Time


Single and ratio indicators


Single and ratio indicators (cont.)

Needs

Objective

Input

Effectiveness

Productivity

Costs

Cost-effectiveness

Efficiency

Output

Outcomes


Summarizing about indicators

  • Indicators are measures of change

  • Baseline and target data is important for good use of indicators

  • But be careful – not too many targets. When targets are set, make sure gaming potential is minimized


Contents

  • Outputs & inputs

  • Outcomes

  • Indicators

  • Logic model

  • Budgeting & PI

  • Future


Model for structuring performance information

Other influences

Problems and needs

Policy level

Policy

input

Structure, institutional and managerial arrangements

Program level

Activities and processes

Outputs

Short to medium term outcomes

Final outcomes

Objectives

input

Context


Challenges

Value to the nation

Measurement challenges

Measurement costs

Time delays

Relative magnitude

Linkages to our actions

OUTPUTS

Outcomes

IMMEDIATE

INTER

MEDIATE

FINAL


Bottom-up vs. top-down approach in modelling and planning

  • Outputs to outcomes (bottom-up) approach is most common to budgeting

OUTPUTS

Outcomes

IMMEDIATE

INTER

MEDIATE

FINAL

  • While outcomes to outputs (top-down) – in policy

Policy objective

Options

Instruments

  • However, several OECD countries use the later in budgeting, focusing on outcomes and leaving connection to outputs loose


Contents

  • Outputs & inputs

  • Outcomes

  • Indicators

  • Logic model

  • Budgeting & PI

  • Future


Budgeting and performance information

  • Performance budgeting has been THE THEME for several decades now

  • It has not succeeded and no one does it right

  • There are two most discussed topics in this theme:

  • Outputs or outcomes focus?

  • Can everyone do it?


Challenges

  • Informational challenges, e.g. inserting output and outcome information into budget documentation

  • Decision making challenges, e.g. how much difference PI makes


Informational challenges

  • Information overload

  • Difficulties to delegate

  • Cost of data

  • Information and knowledge gaps

  • Principal agent problems


Decisional challenges

  • Better information = better decisions – only assumption

  • Hard to attribute action to outcome change. Requires high quality policy analysis

  • Fixed budgets are not performance budgets

  • But non-fixed budgets can lead to gaming

  • Information asymmetry and principal agent problems

  • Time delays


Outputs or outcomes?

inputs

outputs

outcomes

Program budgeting

Strategic planning

Evaluation and spending reviews

Attribution

Need for strong policy analysis


Outputs or outcomes? (cont.)

  • Performance budgeting is about using the budget to promote performance rather than a specific technique such as program or output budgeting. What all attempts at performance budgeting have in common is the attempt to increase the influence of policy analysis, strategic decision support, performance information in the budget system (Graham Scott)

  • Appropriate use of performance information by key actors at each stage of the budget cycle to inform their decisions concerning resource allocation and to improve efficiency of resources (World Bank)


Can everyone do it?

  • Get the basic parts of the budget to perform first

  • Willingness and ability to execute voted money, to control expenditure

  • Sound financial accountability, respect for financial management procedures, and effective internal controls

  • Functioning government

  • Implanting of performance information in the budget process requires conductive environment – values, minimal degree of informality, less corruption, leadership etc.


Where to start?

How much money you can spend or should save for the whole of Government?

Macroeconomic policy as in Budget Memorandum

1. Aggregate Fiscal Discipline

2. Allocative Efficiency

In what areas we want / need to spend the money on – health, education, transport?

Strategic policy documents of the Government

2.1 Inter-sectoral

In each area, what are the needs and priorities we want to spend money on – preventive health care, treatment etc.?

Sector level policy documents

2.2 Intra-sectoral

How well we spend the money we got? Could we spend it better?

Monitoring and analysis

3. Operational Efficiency


Where to start? (cont.)

Aggregate fiscal discipline

However, any failure to successfully achieve macro fiscal goals will seriously undermine allocative and operational efficiency goals

Allocative efficiency

Operational efficiency

Importance of PI


What preconditions?(cont.)

  • Baseline projections

  • Trend analysis

  • Attribution to change


What preconditions?

  • Get the basics right (execution, control, accounting, reporting)

  • Sort out budget process (actors, information flows, incentives and capacity) – little sense to attempt performance budgeting within environment where short term political priorities and deal making prevail over rational analysis of spending options

  • Focus on performance, and not performance information

  • Focus on process, not only targets. Process is as important as targets

  • Role and capacity of Ministries of Finance and centres of Government

  • Holistic rather than patchy approach


Contents

  • Outputs & inputs

  • Outcomes

  • Indicators

  • Logic model

  • Budgeting & PI

  • Future


Future - open questions

  • How does your PI dimension affect your budget process in terms of actors, information flows between them, capacity and incentives?

  • How to improve usage of PI in budgetary decision making?

  • How to improve measurement?

  • How to improve quality / user friendliness / usability of information?

  • How to get politicians to use PI in decision making?


www.cpmconsulting.eu


ad
  • Login