slide1
Download
Skip this Video
Download Presentation
Distinguishing Inputs, Outputs, Outcomes and Impacts

Loading in 2 Seconds...

play fullscreen
1 / 89

Distinguishing Inputs, Outputs, Outcomes and Impacts - PowerPoint PPT Presentation


  • 204 Views
  • Uploaded on

Distinguishing Inputs, Outputs, Outcomes and Impacts. www.summerinstitute.eu. Important points to start with. Decision making. Policy analysis. Limited rationality. Incentives. Process and learning. Contents. Outputs & inputs. Outcomes. Indicators. Logic model. Budgeting & PI.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Distinguishing Inputs, Outputs, Outcomes and Impacts' - Michelle


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
important points to start with
Important points to start with
  • Decision making
  • Policy analysis
  • Limited rationality
  • Incentives
  • Process and learning
contents
Contents
  • Outputs & inputs
  • Outcomes
  • Indicators
  • Logic model
  • Budgeting & PI
  • Future
contents4
Contents
  • Outputs & inputs
  • Outcomes
  • Indicators
  • Logic model
  • Budgeting & PI
  • Future
outputs definition
Outputs - definition
  • Services or goods which are provided by ministries and other state organizations to external stakeholders / beneficiaries (directly or through other organizations)
  • External stakeholders include: general public, citizens, businesses, NGOs, media and other state bodies including the minister, other ministries and state organizations, and the parliament
examples
Examples
  • Provision of prosthetic devices to disabled
  • Provision of schooling
  • Health centre services
  • Analysis and policy advice
  • Laws
  • Inspections
  • Training
outputs should not be
Outputs should not be
  • Activities
  • Processes
  • Inter-mediate output
  • Capacity building initiatives
example output schooling
Example: output – schooling
  • Number of graduates from school
  • Activity - teaching
  • Process - recruitment of teachers
  • IMO – Number of enrolled chidlren
  • Capacity building – increasing professional qualification of teachers
slide9
Consumption or provision?
  • Economic theory view: the delivery of service takes place between producer and consumer, and therefore transaction is complete, when service is used (output consumed)
  • Theatre plays
  • Patients checked
  • Public administration view: for many outputs it does not mater whether they are consumed or not – the service should be there in case it is needed
  • Emergency services
  • Prison services
output characteristics
Output characteristics
  • Volume or quantity of provision
  • Timeliness in provision
  • Quality of provision
  • Satisfaction - surveys and evaluation
  • Comparison – to others or standard
  • Coverage, meting of demand and accessibility
  • Equity
example output provision of prosthetic devices
Example: output – provision of prosthetic devices
  • Volume – number of disabled people served
  • Timeliness – service provided in specific time – for example, one week after request
  • Quality – satisfaction of users of devices
  • Coverage – % of people in need of devices served
  • Equity – people with access difficulties served at home or transported to the service point
selectivity
Selectivity
  • Not all output characteristics can/should be used for all outputs – be selective
  • Focus on key outputs for external users
  • For all other outputs – have at least the volume
aspects to consider
Aspects to consider
  • Control of provision
  • Other government interventions
  • Measurability
  • Attribution
  • Users
  • Incentives & gaming
control of provision
Control of provision
  • Provision of a single output is controlled (directly or indirectly) by government organization
  • But not its effectiveness (achieving desired policy objective). Many government outputs make sense only when used together with other outputs
  • Road safety and reduction of fatal accidents depends ona group of outputs – quality of road, speed control, technical state of cars, use of seat belts.
control of provision contd
Control of provision (contd.)

Many organizations

Policy

Budgeting

Delivery

One or more organizations

One organization

other government interventions
Other government interventions
  • To implement policies governments provide not only goods and services, but also:
  • regulation
  • capital investments
  • social and various other benefits & compensations
  • subsidies
  • If these are not outputs, how do they relate to them?
other government interventions contd
Other government interventions(contd.)
  • It can be helpful to think of three broad categories
  • goods and services = outputs
  • administered items
  • capital investment
  • mandatory benefits & compensations
  • subsidies and other payments
  • general regulation
  • But remember – most of administred items and general regulation require outputs
slide18
Example – road safety
  • In budgets, some countries separate outputs / investment / financing in separate programs while others do not (but separate using economic classification)
measurability
Measurability
  • Some outputs are harder to measure than others due to the nature of organizations providing them

Measurement difficulty

Embassies

Army

Residential care

Ambiguity

(output)

Road police

Job Counselling

Training institutions

Unemployment agencies

Policy making ministries

Routine

(process)

attribution
Attribution
  • Outputs mean something if they can be attributed to some objective (outcome) – they contribute to achieving of objective, e.g. Change in outcome
  • Generalized attribution: re-training of unemployed contributing to reduction of unemployment
  • Specific: re-training of unemployed contributing to employability of the trained person during the next year after training
slide21
Difficulties in establishing attribution

Clear attribution:

  • Fixed arm
  • Medical service
  • Broken arm

Less clear attribution:

  • Survival
  • Services
  • Cancer
  • Earlier diagnosis?
  • Better treatment?
  • Healthier life style?
  • Beneficial effects of affluence?
users
Users
  • Different PI users and decision making stages might have different information needs (and therefore require different formats of presentation of PI)

Parliament

Government

Ministry of Finance

Minister

Permanent Secretary

Unit Manager

Employee

Output focus

  • Using of PI means analysis and learning rather than money and / or promotion
principal agent problem
Principal – Agent Problem
  • Principals are “owners” and “bosses”, but they lack time and expertise to do everything
  • Therefore principals hire agents, who have time and expertise, but are self interested

Modern

Medieval

Business

Principal

King

Shareholder

People

Agent

Lords

CEO

Bureaucrats

Delegated authority

Tax collection

Profit making

Service delivery

principal agent problem contd
Principal – Agent Problem(contd.)
  • Agents have one strong advantage over principals - information
  • This creates problem called “gaming”, which is based on two factors:
  • There is information asymmetry between the principal and agent; and
  • There are wrong incentives for disclosing or hiding information
gaming incentives
Gaming & incentives
  • Bevan and Hood: "reactive subversion such as 'hitting the target and missing the point' and / or reducing performance where targets do not apply"
  • Gaming = manipulation with information that might affect service delivery
  • Incentives = factors (financial or non-financial) that provide motives for a particular course of action, or count as a reason for preferring one choice to the alternatives
  • Targets are the most common causes for perverse incentives
slide26
Gaming & incentives

(contd.)

High

Gaming potential

Information asymmetry

Low

Loose

Tight

Relationship with decision-making

impact of gaming
Impact of gaming

Strategy A

Strategy B

Strategy C

Strategy D

Arrival on time at final destination

“Saboteur” manipulation of service and data

“Smart Liar” or manipulation of data

“False Champion” or service manipulation

“Honest Fool” or no manipulation

TARGET

impact of gaming contd
Impact of gaming (contd.)
  • Gaming can result in:
  • Cherry picking – handling of easy cases, patients etc. at the expense of difficult ones
  • Ratchet effect – slowing down today to be on time tomorrow
  • Distortions – improving areas where measurement is dome while neglecting other areas
  • Threshold effect – focusing on 2nd best and forgetting about the best and worst, as best will be achieved anyway while worst require too much of an effort
dealing with gaming
Dealing with gaming
  • Learning and data improvements
  • Avoiding strong negative incentives
  • Clarify use of PI in advance
  • “Loose” rather than “tight” connection to decisio making
  • Provide feedback on data relevance and quality
  • Reward those with good PI
  • Do not overload with data – use the key measures
dealing with gaming contd
Dealing with gaming(contd.)
  • Dealing with principal – agent problem at all levels
  • Use of “auditors”
  • National Audit Office
  • Internal audit
  • Encourage ownership at production level
summarizing about outputs
Summarizing about outputs
  • Goods and services
  • Basic element in performance information system
  • Can be measured in different ways, but there are difficulties
  • Thee are three most common difficulties:
  • Measurability
  • Attribution to objectives (outcomes)
  • Scope for manipulation
  • Loose link between output data and decision making generally is a better incentive to avoid gaming
inputs definition
Inputs - definition
  • Inputs are resources used in the delivery of goods and services
  • Labour and other means of service provision (facilities, computers, teachers, books etc.)
  • Money is not an input. It s a cost of input
inputs output combination
Inputs – output combination

More outputs

V

?

Less inputs

More inputs

?

X

Less outputs

improving performance
Improving performance

More outputs

Production possibility frontier

Less inputs

More inputs

Less outputs

contents35
Contents
  • Outputs & inputs
  • Outcomes
  • Indicators
  • Logic model
  • Budgeting & PI
  • Future
outcomes definition
Outcomes - definition
  • The effects which government is trying to achieve for the public. They are often expressed as improvement in the living conditions of people or favourable changes that contribute to these changes
  • Outcomes are changes in the economic, physical, social and cultural environments which the state agency(ies) is trying to influence through provision of goods and services, general regulation and financing
  • Outcomes are those events, occurrences, or conditions that are the intended or unintended results of government actions (OECD)
examples37
Examples
  • reduction in maternal / child mortality
  • increase in export
  • decrease of unemployment
  • reduced crime levels
  • improved literacy rate of the population
  • reduction in fatalities from road accidents
  • land productivity improvement
state measures and change
State Measures and Change

LATVIA

2004

2005

change %

Number of road accidents

48912

47353

-3.2

Number of injured

6416

5600

-12.7

Number of fatal injuries

442

516

-14.3

Number of registered cars

966242

-7.6

898145

state measures and change contd
State Measures and Change(contd.)
  • the same data can be called state measure or performance measure
  • it depends whether you attribute policy (outputs, regulation and finances) to it or not
  • if you do attribute – you are interested in change influenced by your policy (the question – whether different policy generates different result, e.g. marginal changes in result depending on inputs)
  • + - 0
slide43
State Measures and Change(contd.)

LATVIA

2004

2005

change %

Number of road accidents

48912

47353

-3.2

Number of injured

6416

5600

-12.7

Number of fatal injuries

442

516

-14.3

Number of registered cars

966242

7.6

898145

  • outputs

attribution of policy to change

  • financing
  • regulation
different outcomes results impacts
Different outcomes – results & impacts

Impact or longer term effects (FO)

Global objectives

Results or direct & immediate effects (SO)

Specific objectives

Operational objectives

Outputs

Activities

Inputs

results impacts defined
Results & impacts defined
  • Results relate to the direct and immediate effect brought about by a policy or program. They provide information on changes to, for example, the behaviour, capacity or performance of direct beneficiaries. Such indicators can be of a physical (reduction in journey times, number of successful trainees, number of roads accidents, etc.) or financial (leverage of private sector resources, decrease in transportation cost) nature
  • Impactsrefer to the consequences of theprogram beyond the immediateeffects on its direct beneficiaries. Two concepts of impact can be defined. Specificimpacts are those effects occurring after a certain lapse of time but which are, nonetheless, directly linked to the action taken. Global impacts are longer-term effectsaffecting a wider population
slide46
Examples

Tax collection

Economy

Justice

Inputs

Inputs

Inputs

inputs

Tax audits

Regulatory burden studies

Prisoner rehabilitation

outputs

Reduced tax evasion

Better regulation

Re-integration into community

results or SO

Reduced shadow economy

Enhanced competitiveness

Reduced repeat crimes

impacts or O

results impacts defined contd
Results & impacts defined(contd.)
  • Results = short to medium term outcomes = specific outcomes
  • Impacts = medium to long term or final outcomes = overall outcomes
  • Sometimes immediate, intermediate and final outcomes
single outcomes and indexes
Single outcomes and indexes
  • Sometimes indexes are used to describe and compare performance
  • Outcomes from several areas and sometimes even sub-areas are merged into one index
  • These are of particular use for policy planners
  • For example: the Enabling Trade Index
example the enabling trade index 2007 8
Example: The Enabling Trade Index 2007/8

MARKET ACCESS

T&C INFRASTR.

BUSINESS ENVIRONM.

BOARDER ADMIN.

  • Regulatory environment
  • Availability & quality of transport infrastructure
  • Efficiency of customs administration
  • Tariff & non-tariff barriers
  • Proclivity or openness to trade
  • Physical security
  • Availability & use of ICTs
  • Efficiency of import – export procedures
  • Transparency of boarder administration
example tourism competitiveness index 2008
Example: Tourism Competitiveness Index 2008

REGULATORY FRAMEWORK

HUMAN, CULTURAL & NATURAL RESOURCES

BUSINESS ENVIRON. & INFRASTR.

  • Policy & regulations
  • Air transport infrastructure
  • Human resources
  • National tourism perception
  • Environmental regulations
  • Ground transport infrastructure
  • Natural & cultural resources
  • Safety & security
  • Tourism infrastructure
  • Health and hygiene
  • Price competitiveness
  • Prioritization of T&T
problems with outcome measurement
Problems with outcome measurement
  • Timing
  • Data availability for measurement and transaction costs
  • Outcome (state) measures remain important even in the absence of policies and programs directed at them
  • Impact of other influences
  • Attribution
attribution to outputs
Attribution to outputs

Other influences

Influence of other factors

Degree of outcome attribution to outputs

Attribution

Immediate – intermediate - final

attribution modelling
Attribution modelling

Outcomes

IMMEDIATE

INTER

MEDIATE

FINAL

Knowledge & understanding

Changes in environment

Behavioural changes

attribution modelling cont
Attribution modelling(cont.)

Outputs

Inputs

Outcomes

Target group

Outputs

Medium term

Short term

Long term

Improved knowledge about impact of farming on environment

Educational events

Less polluting farming

Systematic pollution monitoring in farms

Staff

Finances

Research

Materials

etc.

Improved understanding about the use of pollutants

Control system

Polluting farmers

Changes to farming practices

Improved understanding about the ways to control pollution

Site visits / inspections

Improved environment

summarizing about outcomes
Summarizing about outcomes
  • Outcomes are changes in the economic, physical, social and cultural environments which the state agency(ies) is trying to influence
  • Focus on change (+/-/0) in outcomes is what makes attribution between outputs and outcomes)
  • Outcomes can be of immediate, medium term and long term nature. More immediate they are – stronger the attribution to outputs
  • Outcomes can range from single to more compplex ones based on some kind of a model
  • Outcomes are particularly useful for policy planning and evaluation
contents58
Contents
  • Outputs & inputs
  • Outcomes
  • Indicators
  • Logic model
  • Budgeting & PI
  • Future
indicators definition
Indicators - definition
  • An indicator can be defined as the measurement of an objective to be met, a resourcemobilized, an effect obtained, a gauge of quality or a context variable
  • Indicators demonstrate how to measure elements of program that you are interested in or program in general
important aspects of use of indicators
Important aspects of use of indicators
  • Output indicators
  • State indicators and baselines
  • Targets
  • Single and ratio indicators
output indicators
Output indicators
  • Volume or quantity of provision is the most often used and most relevant to budgeting. However, other output dimensions can have indicators, too:
  • Timeliness in provision
  • Quality of provision: satisfaction and/or comparison
  • Coverage, meting of demand and accessibility
  • Equity
example technical certification of cars service
Example: technical certification of cars service
  • Volume: X number of cars per years
  • Coverage & demand: 100% every day
  • Timeliness in provision: less than ½ hour waiting, service within 1 hour
  • Quality: satisfaction of car users; quality in weakest areas improved; online registration etc.
  • Equity: special service for disabled
state indicators and baselines
State indicators and baselines
  • State indicators describe the state of affairs in a given area of policy concern – unemployment, GDP, mortality, lack of labour mobility, demand for a service exceeding supply etc.
  • Baseline indicators refer to the initial value against which inputs, outputs or outcomes are subsequently measured.
  • State indicator = photograph of situation; Baseline indicator = starting point of measuring impact of policy
targets
Targets
  • Show desired level of performance
  • Make sense only if time bound and have baseline
  • Customer satisfaction from a service increased from 25 to 75% by June 2009
  • Every input, outputs and outcome can have indicators, but should it?
  • UK PSAs: from 300+ targets in 1998 to 30 targets in 2007
  • Often, if you want to have loose connectionto accountability and control, simple expected performance projections can work better than targets
static and dynamic targets
Static and dynamic targets

STATIC CONCEPT

DYNAMIC CONCEPT

Value

Needs

Needs

Value

Target

Target

Baseline

Baseline

Time

single and ratio indicators cont
Single and ratio indicators (cont.)

Needs

Objective

Input

Effectiveness

Productivity

Costs

Cost-effectiveness

Efficiency

Output

Outcomes

summarizing about indicators
Summarizing about indicators
  • Indicators are measures of change
  • Baseline and target data is important for good use of indicators
  • But be careful – not too many targets. When targets are set, make sure gaming potential is minimized
contents71
Contents
  • Outputs & inputs
  • Outcomes
  • Indicators
  • Logic model
  • Budgeting & PI
  • Future
model for structuring performance information
Model for structuring performance information

Other influences

Problems and needs

Policy level

Policy

input

Structure, institutional and managerial arrangements

Program level

Activities and processes

Outputs

Short to medium term outcomes

Final outcomes

Objectives

input

Context

challenges
Challenges

Value to the nation

Measurement challenges

Measurement costs

Time delays

Relative magnitude

Linkages to our actions

OUTPUTS

Outcomes

IMMEDIATE

INTER

MEDIATE

FINAL

slide74
Bottom-up vs. top-down approach in modelling and planning
  • Outputs to outcomes (bottom-up) approach is most common to budgeting

OUTPUTS

Outcomes

IMMEDIATE

INTER

MEDIATE

FINAL

  • While outcomes to outputs (top-down) – in policy

Policy objective

Options

Instruments

  • However, several OECD countries use the later in budgeting, focusing on outcomes and leaving connection to outputs loose
contents75
Contents
  • Outputs & inputs
  • Outcomes
  • Indicators
  • Logic model
  • Budgeting & PI
  • Future
slide76
Budgeting and performance information
  • Performance budgeting has been THE THEME for several decades now
  • It has not succeeded and no one does it right
  • There are two most discussed topics in this theme:
  • Outputs or outcomes focus?
  • Can everyone do it?
slide77
Challenges
  • Informational challenges, e.g. inserting output and outcome information into budget documentation
  • Decision making challenges, e.g. how much difference PI makes
slide78
Informational challenges
  • Information overload
  • Difficulties to delegate
  • Cost of data
  • Information and knowledge gaps
  • Principal agent problems
slide79
Decisional challenges
  • Better information = better decisions – only assumption
  • Hard to attribute action to outcome change. Requires high quality policy analysis
  • Fixed budgets are not performance budgets
  • But non-fixed budgets can lead to gaming
  • Information asymmetry and principal agent problems
  • Time delays
slide80
Outputs or outcomes?

inputs

outputs

outcomes

Program budgeting

Strategic planning

Evaluation and spending reviews

Attribution

Need for strong policy analysis

slide81
Outputs or outcomes? (cont.)
  • Performance budgeting is about using the budget to promote performance rather than a specific technique such as program or output budgeting. What all attempts at performance budgeting have in common is the attempt to increase the influence of policy analysis, strategic decision support, performance information in the budget system (Graham Scott)
  • Appropriate use of performance information by key actors at each stage of the budget cycle to inform their decisions concerning resource allocation and to improve efficiency of resources (World Bank)
slide82
Can everyone do it?
  • Get the basic parts of the budget to perform first
  • Willingness and ability to execute voted money, to control expenditure
  • Sound financial accountability, respect for financial management procedures, and effective internal controls
  • Functioning government
  • Implanting of performance information in the budget process requires conductive environment – values, minimal degree of informality, less corruption, leadership etc.
slide83
Where to start?

How much money you can spend or should save for the whole of Government?

Macroeconomic policy as in Budget Memorandum

1. Aggregate Fiscal Discipline

2. Allocative Efficiency

In what areas we want / need to spend the money on – health, education, transport?

Strategic policy documents of the Government

2.1 Inter-sectoral

In each area, what are the needs and priorities we want to spend money on – preventive health care, treatment etc.?

Sector level policy documents

2.2 Intra-sectoral

How well we spend the money we got? Could we spend it better?

Monitoring and analysis

3. Operational Efficiency

slide84
Where to start? (cont.)

Aggregate fiscal discipline

However, any failure to successfully achieve macro fiscal goals will seriously undermine allocative and operational efficiency goals

Allocative efficiency

Operational efficiency

Importance of PI

slide85
What preconditions?(cont.)
  • Baseline projections
  • Trend analysis
  • Attribution to change
slide86
What preconditions?
  • Get the basics right (execution, control, accounting, reporting)
  • Sort out budget process (actors, information flows, incentives and capacity) – little sense to attempt performance budgeting within environment where short term political priorities and deal making prevail over rational analysis of spending options
  • Focus on performance, and not performance information
  • Focus on process, not only targets. Process is as important as targets
  • Role and capacity of Ministries of Finance and centres of Government
  • Holistic rather than patchy approach
contents87
Contents
  • Outputs & inputs
  • Outcomes
  • Indicators
  • Logic model
  • Budgeting & PI
  • Future
slide88
Future - open questions
  • How does your PI dimension affect your budget process in terms of actors, information flows between them, capacity and incentives?
  • How to improve usage of PI in budgetary decision making?
  • How to improve measurement?
  • How to improve quality / user friendliness / usability of information?
  • How to get politicians to use PI in decision making?
ad