Agency level measurement can it work and does it matter
This presentation is the property of its rightful owner.
Sponsored Links
1 / 29

AGENCY-LEVEL MEASUREMENT: Can it work, and does it matter? PowerPoint PPT Presentation


  • 80 Views
  • Uploaded on
  • Presentation posted in: General

AGENCY-LEVEL MEASUREMENT: Can it work, and does it matter?. ACDI/VOCA Vision . Vision A world in which people are empowered to succeed in the global economy. Mission

Download Presentation

AGENCY-LEVEL MEASUREMENT: Can it work, and does it matter?

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Agency level measurement can it work and does it matter

AGENCY-LEVEL MEASUREMENT: Can it work, and does it matter?


Acdi voca vision

ACDI/VOCA Vision

  • Vision

    A world in which people are empowered to succeed in the global economy.

  • Mission

    To promote economic opportunities for cooperatives, enterprises and communities through the innovative application of sound business practice.

  • Our Work

    • Financial Access

    • Enterprise Development

    • Community Development

    • Food Security

    • Agribusiness


Results based oriented

Results Based Oriented

  • M&E systems at ACDI/VOCA have two primary purposes: accountability to stakeholders and learning to facilitate results-oriented program design and knowledge-based management.

    • Accountability refers to measuring project efficiency, effectiveness, relevance, and sustainability of projects, as well as transparently (disclosing) sharing findings from project evaluation to stakeholders, and using the findings to inform resource allocation and other decisions.

    • Learning refers to the systematic generation of knowledge that can be used to refine designs and introduce improvements into future efforts with the goal of achieving meaningful and sustainable results. M&E systems should ultimately demonstrate that ACDI/VOCA’s activities are contributing to its vision and producing sustainable and equitable development results for the communities in which it works.


Results based guiding principles

Results Based Guiding Principles

  • Integrated into program management and technical approaches;

  • Systematic so as to produce reliable, accurate and useful information, while also

  • Flexible to the complex environments in which ACDI/VOCA operates;

  • Participatory of stakeholders and beneficiaries;

  • Ethical in the collection and reporting of information about and to beneficiaries, partners and stakeholders (Confidentiality and anonymity-Do no harm);

  • Supportive of building local capacity of staff, partners and beneficiaries in M&E;

  • Foster inclusivity and equity of program interventions for both men and women and traditionally marginalized groups; and.

  • Results-oriented: in addition to focusing on implementation processes; ACDI/VOCA M&E systems will emphasize results by asking the “so what” questions.


Capturing performance indicators on a statistical canvas capisc

Capturing Performance Indicators on a Statistical Canvas (CaPISC)

  • CaPISC is an online database that captures the performance monitoring plans for all of ACDI/VOCA’s 70 active projects. To help ACDI/VOCA measure progress towards the goals at the project and practice area level to support our attention to accountability and learning. The system will provide ACDI/VOCA with agency-wide performance measures for:

  • Reporting to stakeholders and donors

  • Communicating with the development community

  • Managing regional and country programs through comparative performance measures

  • Measuring beneficiaries reached by AV intervention

  • Measuring integration of activities

  • Reporting on Companywide common indicators

  • Managing Practice Areas/ individual Project

  • Developing New Business (Marketing)


Why global performance indicators

Why Global Performance Indicators?

  • To assess the performance of its projects, A/V would like to use a comprehensive performance measurement framework.

  • ACDI/VOCA will track this progression of its performance through results reporting against indicators from the project level (predominantly outputs), to the broader program level (predominantly outcomes).

  • The indicators will be standardized to the greatest degree possible, thus allowing ACDI/VOCA and its partners to compare performance across the portfolio of projects.


A starting point

A Starting Point

  • Global Level Indicators

    • Per capita expenditures (as a proxy for income) of USG targeted beneficiaries

    • Number of jobs attributed to A/V implementation

    • Number of hectares of natural resources showing improved biophysical conditions as a result of USG assistance

    • Percent of beneficiaries adopting xx number of improved practices/technologies


Goals of capisc

Goals of CaPISC

CaPISC allows for current project reporting thus more responsive management and program adaptation.

Reporting with CaPISC will allow for enhanced program and institutional learning.


Uses of capisc as a results based management tool

Uses of CaPISC – as a Results Based Management Tool

Visually depict progress on project goals and indicators

Determine progress of key project indicators COP Dashboards

Provides tools to help you analyze your data

Create reports for donors


Use capisc to help in new business development

Use CaPISC to Help in New Business Development

  • Illustrate results over time and across projects

  • Show past experience and results

  • Demonstrate capability using indicator data

  • Model PMPs after similar ones in the system

    • Uniform terminology

    • Global indicators

      • Indicator library


Challenges lessons learned

Challenges/Lessons Learned

  • It has been a long path to persuade the management to invest in this database.

  • It is taking some time to train project level M&E specialist on how to use and enter the data

  • Data quality (especially data that comes from the COs).

  • Recourses (human and financial)


Interaction forum 2012

InterAction Forum 2012

Measuring Agency-Level Results

CRS

Beneficiary and Service Delivery Data (BSDI)

May 1, 2012


Beneficiary and service delivery indicators bsdi

Beneficiary and Service Delivery Indicators (BSDI)

Output Level Indicators

that

Measure Agency-wide Performance

on

CRS Fundamental Program Outputs

Services Delivered

Beneficiaries Served

Beneficiary and Service Delivery Indicators (BSDI) Presentation


Illustrative program area and services catalogue water and sanitation

Illustrative Program Area and Services Catalogue: Water and Sanitation

Beneficiary and Service Delivery Indicators (BSDI) Presentation


Bsdi data flow map

BSDI Data Flow Map

Point of Service Delivery

Typically data are recorded on paper by individual beneficiary

Beneficiary and service delivery data are registered

Data

Typically data are transcribed to Excel files

Regional/Partner Office

Beneficiary and service delivery data are recorded and records are updated

Data

Project Management Office

Typically data are stored in a project database

Beneficiary and services delivery data are stored, maintained, analyzed and reported

Data

Reports

On-line Agency

Database (PIMS)

Donor

Beneficiary and Service Delivery Indicators (BSDI) Presentation


Bsdi performance products

BSDI Performance Products

BSDI allows accurate, annual agency-wide tracking of CRS performance

BSDI allows integrated technical and financial performance analysis

BSDI will track “double counted” beneficiaries in a multi-sector projects

Beneficiary and Service Delivery Indicators (BSDI) Presentation


Agency level measurement can it work and does it matter

Mission Metrics


Agency level measurement can it work and does it matter

Incoming Data

Analyze

Reflect/ Discuss

Cycle of Learning

Review Results/Learn

Make Decisions

Take Action


Measuring agency level results interaction forum 2012 may 01 2012

Measuring Agency Level ResultsInterAction –Forum 2012May 01, 2012


Purpose key drivers and elements of agency level measurement

Purpose, key drivers and elements of Agency Level Measurement

Purpose

Measure progress towards multi-year strategic plan—specifically to support an overarching objective of “accountability for results”

Communicate w/staff and board and on performance against our intended results for children within the context of our “Theory of Change”

Key Drivers

Accountability to ourselves and to others (specifically children)

Need to have a tool for management purposes from the senior leadership of the organization (including the board).

  • Key Elements

  • Operational Metrics (eg, budget performance, revenue generation, brand recognition, media rating etc)

  • Programmatic and Policy Metrics (eg. Reach, Global Indicators around our Theory of Change)

  • Audience and Use

  • A dashboard for Senior Management Team/Board (Accountability)

  • Staff at all levels (Global, Regional, CO staff) – Learning, new business development

  • Public/Donors – public report (Accountability)


Theory of change how we work to create impact for children

Theory of ChangeHow we work to create impact for children


Understanding results

Understanding Results

The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention.

RESULTS

Impact

Outcome

Input

Output

Input

Output

Outcome

Impact

The financial, human, and material resource used for the development intervention.

The products, capital goods and services which result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of outcomes.

The likely or achieved short-term and medium-term effects of one or more interventions’ outputs.

Positive and negative, primary and secondary long-term effect produced by a development intervention, directly or indirectly, intended or unintended

Definition

E.g. $ spent, staff time

e.g. # health workers trained, teachers trained,

e.g. children accessing life saving interventions;

Policy change implemented

e.g. children’s lives saved; under 5 mortality rate over district, region or country

Examples

REACH

GLOBAL INDICATORS

EVALUATIONS


Components of our program policy results monitoring system towards our strategic goals

Reach

Measures the number of children and adults who have received program input and/pr services directly and indirectly by Save the Children and its partners in the fiscal year disaggregated by theme and sub-theme area.

Global Indicators

A set of outcome indicators linked to Save the Children’s global outcome statements which measure achievement of outcomes.

Advocacy Measurement

A tool to measure progress against an advocacy objective aimed at policy change and/or implementation

Components of our program/policy results monitoring system (towards our strategic goals)


Example of global indicators

Example of Global Indicators

  • Intended Impact: Children learn and develop with age appropriate care and education

  • Outcome Statement: In support of MDG2, by 2015, Save the Children will have significantly contributed to getting 2 million of the hardest to reach children into school, raising the Quality of Learning Environments (QLE), and improving the learning outcomes of more children

  • Indicators:

    • Quality of LE: % of Basic Education and Early Childhood Care and Development schools / sites supported by SC that achieve 4 guiding principles


Lessons learned and key challenges

Buy in and support from the senior leadership at all levels (global and country) is critical for the success of such a system.

Agency dashboard is used to discuss results quarterly—informing decision making on a regular basis.

Data quality and management can be overwhelming, capacity (at all levels) and resource constraints to institute a strong system at all levels.

Shared global objectives has helped to establish shared metrics across the Save the Children Global Movement – however determining attribution is a challenge.

Keeping the balance between a concise dashboard (with few metrics) while providing a comprehensive picture of our results for children and the complex nature of our work.

Reach figures do not tell us about our program impact. But measuring outcome and impact level indicators are resource intensive.

Lessons learned and key challenges


Contact

Contact


  • Login