monitoring evaluation l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
MONITORING & EVALUATION PowerPoint Presentation
Download Presentation
MONITORING & EVALUATION

Loading in 2 Seconds...

play fullscreen
1 / 143

MONITORING & EVALUATION - PowerPoint PPT Presentation


  • 107 Views
  • Uploaded on

MONITORING & EVALUATION. Capacity Building for Program Improvement. Global AIDS Program Centers for Disease Control and Prevention  ORC Macro. Training Focus. Program Improvement. Share Data with Partners. Reporting/ Accountability. Training Objectives.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'MONITORING & EVALUATION' - maeve


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
monitoring evaluation

MONITORING & EVALUATION

Capacity Building for Program Improvement

Global AIDS Program

Centers for Disease Control and Prevention

ORC Macro

training focus
Training Focus

Program

Improvement

Share

Data with

Partners

Reporting/

Accountability

training objectives
Training Objectives
  • Describe the use of M&E data for program planning, management, and improvement
  • Understand and initiate program monitoring and other evaluation activities as standard program components within technical strategies
  • Work towards developing strategy-specific M&E plans and an integrated M&E strategy
  • Identify M&E TA needs

By the end of the training, participants will be able to:

ideal relationship among caps budget m e reporting
Ideal Relationship among CAPs/Budget, M&E & Reporting

Develop Program

Plan & Budget Plan

Due Oct. 1

Country

Assistance

Plan

Objectives

Due Oct. 1

Logic Model

Annual

Report

Due Dec. 1

Design M&E/

M&E Plan

Implement M&E Activities

relationships among caps budget m e reporting
Relationships Among CAPs/Budget, M&E & Reporting

Develop Program

Plan & Budget Plan

Due Oct. 1

Country

Assistance

Plan

Objectives

Due Oct. 1

Logic Model

Annual

Report

Due Dec. 1

Design M&E/

M&E Plan

Implement M&E Activities

cdc gap program model
CDC/GAP Program Model

Care and Treatment

Surveillance

Primary Prevention

Capacity and Infrastructure Strengthening

  • Voluntary counseling and testing (VCT)
  • Blood safety
  • STI prevention and care
  • Youth interventions
  • Public-private partnerships
  • Behavior change communication
  • Preventing HIV transmission in drug-using populations
  • HIV, STI, and TB disease surveillance and behavioral surveillance
  • Preventing mother-to-child transmission (PMTCT)
  • Prevention and treatment of opportunistic infections
  • TB prevention and care
  • Palliative care
  • Appropriate use of antiretroviral drugs
  • Monitoring and evaluation
  • Training
  • Laboratory support
  • Information systems
cdc hiv strategic plan
CDC HIV Strategic Plan

International Objectives:

  • Decrease sexually transmitted HIV infections
  • Develop capacity in host countries for HIV prevention & care
  • Strengthen HIV/STD/TB surveillance
  • Improve scientific knowledge of HIV and the safety and efficacy of new biomedical interventions
  • Decrease HIV infections transmitted from mother to child
  • Increase access to HIV care and support, including prevention & treatment of opportunistic infections
  • Decrease parentally transmitted HIV infections
slide10

Outputs

Short-Term Outcomes

Intermediate Outcomes

NAP

Program

Program

Program

Program

Program

Program

Program

Program

Global Fund

inputs

World Bank

inputs

Long-Term

Impacts

USG

inputs

Other

inputs

Multi-agency M&E Logic Model

Adapted from Milstein & Kreuter. A Summary Outline of Logic Models: What are They and What Can They Do for Planning and Evaluation? CDC 2000

slide11

Program

Program

Program

Program

Program

Program

Program

Program

Multi-agency M&E Logic Model

Short-Term Outcomes

Intermediate Outcomes

Outputs

NAP

CDC

inputs

CDC

USAID

inputs

Long-Term

Impacts

USG Partnership

Agency X

inputs

Global Partnership

Agency Y

inputs

slide12

Global M&E Indicator Pyramid:Levels of Indicators

Global

Level

Indicators

(UNGASS)

Country Level Indicators

(CDC, NAP/UNAIDS, USAID Missions)

Project Level Indicators

(MTCT, STI, VCT, TB, Care and Treatment, etc.)

cdc gap m e mission statement
Strengthen M&E systems that inform HIV/AIDS policy and program decisions at the local, national and global levelsCDC/GAP M&E Mission Statement
cdc global aids program monitoring and evaluation goals
CDC/Global AIDS ProgramMonitoring and Evaluation Goals

The CDC/GAP M&E system has two goals:

  • Determine the progress and effectiveness of CDC/GAP programs and assistance activities
  • Strengthen the capacity of National AIDS Programs to conduct monitoring and evaluation
global aids program m e framework and illustrative data types
Global AIDS Program M&E Framework and Illustrative Data Types

Impact(Long-termEffects)

Input

(Resources)

Assessment & Planning

Output

(Immediate

Effects)

Outcomes(Intermediate Effects)

Activities

( Interventions,

Services)

Situation Analysis

Response Analysis

Stakeholder Needs

Resource Analysis

Collaboration plans

Staff

Funds

Materials

Facilities

Supplies

Trainings

Services

Education

Treatments

Interventions

# Staff Trained

# Condoms Distributed

# Test Kits Distributed

# Clients Served

# Tests Conducted

Provider Behavior

Risk Behavior

Service Use

BehaviorClinical Outcomes

Quality of Life

HIV Incid/Prev

Social Norms

STI Incid/Prev

AIDS Morb/Mort

Economic Impact

Program Development Data

Population-based Biological, Behavioral & Social Data

Program-based Data

In addition to monitoring these illustrative data types, select programs conduct enhanced process and outcome evaluations.

9 critical elements of gap m e strategy
9Critical Elements of GAP M&E Strategy

Phase 3

Start-up/Phase 1

Phase 2

Systematic Review

Case Studies, Operation & Intervention Research, Economic Evaluations

Partnerships

CAPs & Logic Models

National-Level Impact Monitoring

M&E Plans, Annual Reports, & Program Reviews

National-Level Outcome Monitoring

M&E Needs Assessments & Trainings

Program Monitoring & Process Evaluation

a public health questions approach to unifying aids m e
A Public Health Questions Approach to Unifying AIDS M&E

Are collective efforts being implemented on a large enough scale to impact the epidemic (coverage; impact)?Surveys & Surveillance

Determining Collective Effectiveness

OUTCOMES

& IMPACTS

MONITORING

Are interventions working/making a difference?

Outcome Evaluation Studies

Monitoring & Evaluating National Programs

OUTCOMES

Are we implementing the program as planned?

Outputs Monitoring

OUTPUTS

What are we doing? Are we doing it right?

Process Monitoring & Evaluation, Quality Assessments

ACTIVITIES

What interventions and resources are needed?

Needs, Resource, Response Analysis & Input Monitoring

INPUTS

Understanding Potential Responses

What interventions can work (efficacy & effectiveness)? Are we doing the right things?

Special studies, Operations res., Formative res. & Research synthesis

What are the contributing factors? Determinants Research

Problem Identification

What is the problem? Situation Analysis and Surveillance

strategic planning for m e setting realistic expectations
Strategic Planning for M&E: Setting Realistic Expectations

Monitoring and Evaluation Pipeline

All

Most

Some

Few *

#

of Projects

Impact Monitoring/Evaluation

Outcome Monitoring/ Evaluation

Process Evaluation

Input/Output Monitoring

* Supplemented with impact indicators from surveillance data.

Levels of Monitoring & Evaluation Effort

Adaptation of Rehle/Rugg M&E Pipeline Model, FHI 2001

strategic planning for m e setting realistic expectations19
Strategic Planning for M&E: Setting Realistic Expectations

All

Most

Some

Few *

Phase 1: Typical Methods

  • Situational analysis
  • Response analysis, stakeholder needs & resource analysis
  • Inputs/outputs monitoring (e.g., # staff trained, #condoms distributed, # clients served)
strategic planning for m e setting realistic expectations20
Strategic Planning for M&E: Setting Realistic Expectations

All

Most

Some

Few *

Phase 2: Typical Methods

  • Process evaluation (e.g., quality of training, client satisfaction or perceptions)
  • Quality assessments
  • Operations research & formative evaluation
  • Case study
  • Cost analysis
strategic planning for m e setting realistic expectations21
Strategic Planning for M&E: Setting Realistic Expectations

All

Most

Some

Few *

Phase 3: Typical Methods

  • Monitoring outcome indicators (e.g., increase in condom use, increase in knowledge about HIV transmission)
  • Outcome evaluation (e.g., was the program responsible for behavior change)
strategic planning for m e setting realistic expectations22
Strategic Planning for M&E: Setting Realistic Expectations

All

Most

Some

Few *

Phase 4: Typical Methods

  • Impact monitoring (e.g., disease surveillance)
  • Impact evaluation (e.g., rise or fall of disease incidence/prevalence as a function of AIDS programs)
international m e standards
International M&E Standards
  • Monitoring and Evaluation Reference Group (MERG)
    • UNAIDS, since 1998…
    • International gold standard for indicators (now sharing with WHO)
  • UNGASS- United Nations General Assembly Special Session
    • Global commitment embodied in accepted national indicators to be tracked and reported at global level
    • CRIS (Country Response Information System)
  • USG support
    • USAID-DHHS financial and technical assistance (leading funder)
    • National M&E trainings and workshops
  • These efforts support the crucial link between standardized indicators and standardized data collection methods/ mechanisms at the country level
international m e standards harmonized tools and methods guidance
International M&E Standards: Harmonized Tools and Methods Guidance
  • UNAIDS, WHO, UNICEF, USAID, HRSA, GFTAM, World Bank are major CDC/GAP partners in development of Indicators and M&E guidance to national governments
  • New indicator and M&E guidance on:
    • VCT
    • PMTCT
    • Care and Support
    • ART
    • Orphans
unaids and ungass indicators
UNAIDS and UNGASS Indicators
  • National indicators; subset are globally tracked
  • Core national program indicators with focus on:
    • Coverage (% population “at risk” getting intervention or responding to intervention) and
    • Impact: HIV infection, morbidity, mortality
what is monitoring evaluation
What Is Monitoring & Evaluation?
  • Monitoring:
    • Tracks priority information relevant to national program planning and intended outputs, outcomes and impacts.
    • Tracks costs & program functioning.
    • Provides basis for program evaluation when linked to a specific program.
  • Evaluation:
    • Is a rigorous, scientifically based collection of information about program activities, characteristics, and outcomes to determine the merit or worth of a specific program.
    • Is used to improve programs and inform decisions about future resource allocations.
m e terminology
M&E Terminology
  • Assessment & Planning:
    • Collection of information and data needed to plan programs and initiatives. These data may describe the needs of the population and the factors that put people at risk, as well as the context, program response, and resources available (financial and human).
  • Answers questions such as:
      • What are the needs of the population to be reached by the program/initiative?
      • How should the program/initiative be designed or modified to address population needs?
      • What would be the best way to deliver this program/initiative?
m e terminology continued
M&E Terminology (continued)
  • Input/Output Monitoring:
    • Collects data describing the individuals served, the services provided, and the resources used to deliver those services.
    • Answers questions such as:
      • What services were delivered? What population was served and what numbers were served? What staffing/resources were used?
  • Process Evaluation:
    • Collects more detailed data about how the intervention was delivered, differences between the intended population and the population served, and access to the intervention.
    • Answers questions such as:
      • Was the intervention implemented as intended? Did the intervention reach the intended audience? What barriers did clients experience in accessing the intervention?
m e terminology continued30
M&E Terminology (continued)
  • Outcome Monitoring:
    • Basic tracking of measures related to desired program outcomes. With National AIDS programs, outcome monitoring is typically conducted through population-based surveys to track whether or not desired outcomes have been reached. May also track information directly related to program clients, such as change in knowledge, attitudes, behavior.
    • Answers the question:
      • Did the expected outcomes occur, e.g., increase in condom use; increase in knowledge or change in behavior; increase in client use of services?
m e terminology continued31
M&E Terminology (continued)
  • Outcome Evaluation:
    • Collects data about outcomes before and after the intervention for clients as well as with a similar group that did not participate in the intervention being evaluated.
    • Answers the question:
      • Did the intervention cause the expected outcomes?
m e terminology continued32
M&E Terminology (continued)
  • Impact Monitoring and Evaluation:
    • Collects data about HIV infection at the jurisdictional, regional, and national levels.
    • Answers the question:
      • What long-term effects do interventions have on HIV infection?

Distinction between Impact Monitoring and Evaluation

    • Impact monitoring (e.g., disease surveillance).
    • Impact evaluation (e.g., rise or fall of disease incidence/prevalence as a function of AIDS programs).
slide33

M&E Compared to Other Concepts

  • Academic Research: Primarily, hypothesis testing in a controlled environment.
  • Disease Surveillance: Ongoing systematic collection, analysis, and interpretation of data that describe diseases and their transmission in populations.
  • Operations Research/Evaluation: Applies systematic research techniques to improve service delivery and influence related program policies.
  • Policy Evaluation: Assessments of application and effectiveness of policies.
  • Economic Evaluation: Assessments to identify, measure, value, and compare the costs and outcomes of alternative interventions.
other notes on m e language
Other Notes on M&E Language

Case Studies: A methodological approach that typically incorporates a number of data-gathering activities (e.g., interviews, observations, and questionnaires) at select sites or programs. In the GAP context, case studies are done at the country level to determine CDC’s overall “value added”. The findings are then used to report to stakeholders, make recommendations for program improvement, and for sharing lessons with other countries.

m e related to planning implementation and outcomes
M&E Related to Planning, Implementation and Outcomes

Outcomes

Planning

Implementation

-Assessment & Planning

-Input/Output Monitoring

-Process Evaluation

-Outcome Monitoring

-Outcome Evaluation

-Impact Monitoring

-Impact Evaluation

implications of not knowing how an intervention was implemented
Implications of Not Knowing How an Intervention Was Implemented

Planning

???????

Outcomes

cdc gap field office m e activities gap m e racetrack
CDC/GAP Field Office M&E Activities “GAP M&E Racetrack”
  • In field/regional-office groups discuss M&E activities that are planned, initiated, completed, institutionalized or ongoing.
  • Complete the racetrack by placing dots in appropriate boxes.
  • Choose a field/regional-office spokesperson to present a 10 minute overview of your M&E activities.
goals and objectives
Goals and Objectives
  • Goal:

Statement of a desired, broad, long-term outcome of

    • the program; expresses general program intentions
    • and helps guides the program’s development.
  • Objective:
  • Statement of desired, specific, reasonable, and measurable
  • program results.
slide41

M&E Training Goal and Objectives

  • Goal of training:
    • To equip GAP staff with an understanding of M&E and the knowledge and skills needed to incorporate M&E activities into everyday work with programs.
  • Training objectives:
    • By the end of this training, participants will be able to:
      • Describe the use of M&E data for program planning, management, and improvement.
      • Understand and initiate program monitoring and other evaluation activities as standard program components within technical strategies.
a public health questions approach to unifying aids m e42
A Public Health Questions Approach to Unifying AIDS M&E

Are collective efforts being implemented on a large enough scale to impact the epidemic (coverage; impact)?

Determining Collective Effectiveness

OUTCOMES

& IMPACTS

MONITORING

Are interventions working/making a difference?

Monitoring & Evaluating National Programs

OUTCOMES

Are we implementing the program as planned?

OUTPUTS

What are we doing? Are we doing it right

ACTIVITIES

What interventions and resources are needed?

INPUTS

Understanding Potential Responses

What interventions can work (efficacy & effectiveness)? Are we doing the right things?

What are the contributing factors?

Problem Identification

What is the problem?

slide43

Process and Outcome Objectives

Output: # Clients tested for HIV receive test results.

Objective: By the end of the first program year, 98% of clients tested for HIV will receive test results.

Outcome: Clients (HIV+ and HIV-) develop and adhere to personalized HIV risk-reduction and treatment strategy.

Objective: By the beginning of the second program year, 65% of clients receiving HIV test results will have formed personalized risk-reduction/treatment strategies.

slide44

Sample Objectives

  • Process Objective:
    • Provide VCT clients with HIV test results.
  • Outcome Objective:
    • Assist VCT clients with developing personalized risk-reduction and treatment strategies.
slide45

SMART Method

  • Specific:Identifies concrete events or actions that will take place.
          • Does the objective clearly specify what will be accomplished and by how much?
  • Measurable:Quantifies the amount of resources, activity, or change.
          • Is the objective quantifiable?
  • Appropriate:Logically relates to the overall problem statement and

desired effects of the program.

          • Does the objective make sense in terms of what the program is trying to accomplish?
  • Realistic:Provides realistic dimension that can be achieved with available resources and plans for implementation.
          • Is the objective achievable given available resources and experience?
  • Time-based:Specifies a time within which the objective will be

achieved.

          • Does the objective specify when it will be achieved?
slide46

Examples of SMART Objectives

  • Process Objective:

By the end of the first program year, 98% of clients tested for HIV will receive test results.

  • Outcome Objective:

By the beginning of the second program year, 65% of clients receiving HIV test results will have developed and adhered to personalized risk-reduction/treatment strategies.

  • Specific: Does the objective clearly specify what will be accomplished and by how much?
  • Measurable: Is the objective measurable?
  • Appropriate: Does objective make sense in terms of what the program is trying to accomplish?
  • Realistic: Is the objective achievable given available resources and experience?
  • Time-based: Does the objective specify when it will be achieved?
examples of objectives
Examples of Objectives
  • Improve use of TB register data (as measured by decrease in the transfer-out rate, interruption rate & missing data, as well as improved validity of data) in 8 of 9 provinces by March 31, 2004. ______________________________________________________
    • Pilot and evaluate a Window-based Electronic TB Register (ETR) in 1 province by 3/31/03.
    • Decrease the transfer-out rate to < 7% in 8 provinces within 12 months of implementation of ETR (baseline 2000, 11.1%).
    • Decrease the interruption rate to <10% in districts within 12 months of implementation of ETR (baseline 2000, 15.1%).
    • Decrease missing data to less than 10% within 12 months of implementing ETR.
    • Conduct TA visits in 8/9 provinces and support implementation of the ETR particularly regarding validation and use of data.
small group work on writing objectives instructions
Small Group Work on Writing Objectives “Instructions”
  • Develop and/or revise CAP objectives for a minimum of two technical strategies (and complete as many technical strategies as possible).
  • Link specific activities to CAP objectives for as many technical strategies as possible.
    • Note: It may be easier to work through one technical strategy at a time (e.g., develop CAP objectives for VCT and link specific activities to these objectives before moving on to another technical strategy).

* Field/regional offices will have an opportunity to report back on their experience writing CAP objectives during the country/region debrief, Wednesday 2:45-3:45.

logic model definition
A logic model describes the main elements of a program and how they work together to prevent HIV in a specific population.

This model is often displayed in a flow chart, map, or table to portray the sequence of steps leading to program outcomes.

Logic Model Definition
headache logic model
Headache Logic Model

Problem Statement:

Stress and tension

have produced a headache

Intervention:

Take two aspirin and

rest for 30 minutes

Outcome:

Headache pain will

be reduced

logic model for hiv prevention
Logic Model for HIV Prevention

Problem Statement

Implementation

Inputs Activities Outputs

Outcomes

Impacts

definitions of logic model components
Problem Statement: Factors that put a population at risk, such as knowledge, attitudes, beliefs, behaviors, skills, access, policies, and environmental conditions.

Problem Statement:HIV infection rates continue to rise, underscoring the importance for people to know their serostatus, develop personalized risk-reduction strategies, and access care and treatment services.

Definitions of Logic Model Components

Problem Statement

Implementation

Inputs Activities Outputs

Outcomes

Impacts

definitions of logic model components54
Inputs: Resources used in an program, such as money, staff, curricula, and materials.

GAP, government, & other donor fundp

C&T personnel

VCT protocols and guidance

Training materials

HIV test kits

Activities: Services that the program provides to accomplish its objectives, such as outreach, materials distribution, counseling sessions, workshops, and training.

Train C&T personnel and site managers

Provide pre-test counseling, HIV tests, post-test counseling

Outputs: Direct products or deliverables of the program, such as intervention sessions completed, people reached, and materials distributed.

# personnel certified

# clients receiving pre-test counseling, HIV tests, post-test counseling

Definitions of Logic Model Components

Problem Statement

Implementation

Inputs Activities Outputs

Outcomes

Impacts

definitions of logic model components55
Outcomes: Program results that occur both immediately and some time after the activities are completed, such as changes in knowledge, attitudes, beliefs, skills, behaviors, access, policies, and environmental conditions.

Quality of VCT improved

Access to VCT increased

Clients develop & adhere to personalized risk-reduction and treatment strategy

Definitions of Logic Model Components

Problem Statement

Implementation

Inputs Activities Outputs

Outcomes

Impacts

definitions of logic model components56
Impacts: Long-term results of one or more programs over time, such as changes in HIV infection, morbidity, and mortality

HIV transmission rates decrease

HIV incidence decreases

HIV morbidity and mortality decrease

Definitions of Logic Model Components

Problem Statement

Implementation

Inputs Activities Outputs

Outcomes

Impacts

vct program implementation logic model
VCT Program Implementation Logic Model

Problem Statement:HIV infection rates continue to rise, underscoring the importance for people to know their serostatus, develop personalized risk-reduction strategies, and access care and treatment services.

OUTPUTS

INPUTS

ACTIVITIES

OUTCOMES

IMPACTS

Funding from gov’t, GAP, & other donors

Train counseling & testing personnel and site managers

# personnel certified in VCT

Quality of VCT increased

HIV transmission rates decreased

# clients received pre-test counseling,

# clients received HIV tests,

# clients received results & post-test counseling***

Provide pre-test counseling, HIV testing, and post-test counseling to all clients tested

Counseling and Testing Personnel*

Access to VCT increased

HIV incidence decreased

Risk behaviors decreased

VCT protocols,

guidelines, and training documents**

Clients (HIV+ and -) develop & adhere to personalized HIV risk-reduction & treatment strategy

Provide pregnant HIV+ women with PMTCT services

# pregnant HIV+ women receive PMTCT services

HIV morbidity & mortality decreased

Health outcomes of HIV + improve

HIV test kits

Screen HIV+ clients for (or refer to screening for) OIs and TB specifically

# HIV+ clients referred to or provided OI/TB screening

*Personnel include counselors, lab techs & VCT site managers.

**At the beginning of programs, inputs such as VCT guidelines, protocols, management information systems (MIS), and referral systems will need to be developed and would be considered “activities” and “outputs”. When these products and systems are in place, they become “inputs”.

***With rapid testing, pre-test counseling, testing, and post-test counseling occur within a short time-frame. Each step is identified because it is possible that there might be a short time lag between steps that may present the possibility of some client attrition.

Increase in care, prevention, and treatment services for HIV+, HIV-, and discordant couples

VCT MIS**

Referral system for

prevention & Tx services**

# HIV+ clients referred to Tx, care & support including ARVs

Refer HIV+ clients to Tx services

vct program implementation logic model58
VCT Program Implementation Logic Model

Problem Statement:HIV infection rates continue to rise, underscoring the importance for people to know their serostatus, develop personalized risk-reduction strategies, and access care and treatment services.

OUTPUTS

INPUTS

ACTIVITIES

OUTCOMES

IMPACTS

Funding from gov’t, GAP, & other donors

Counseling and Testing Personnel*

VCT protocols,

guidelines, and training documents**

HIV test kits

VCT MIS**

Referral system for

prevention & Tx services**

vct program implementation logic model59
VCT Program Implementation Logic Model

Problem Statement:HIV infection rates continue to rise, underscoring the importance for people to know their serostatus, develop personalized risk-reduction strategies, and access care and treatment services.

OUTPUTS

INPUTS

ACTIVITIES

OUTCOMES

IMPACTS

Funding from gov’t, GAP, & other donors

Train counseling & testing personnel and site managers

Provide pre-test counseling, HIV testing, and post-test counseling to all clients tested

Counseling and Testing Personnel*

VCT protocols,

guidelines, and training documents**

Provide pregnant HIV+ women with PMTCT services

HIV test kits

Screen HIV+ clients for (or refer to screening for) OIs and TB specifically

VCT MIS**

Referral system for

prevention & Tx services**

Refer HIV+ clients to Tx services

vct program implementation logic model60
VCT Program Implementation Logic Model

Problem Statement:HIV infection rates continue to rise, underscoring the importance for people to know their serostatus, develop personalized risk-reduction strategies, and access care and treatment services.

OUTPUTS

INPUTS

ACTIVITIES

OUTCOMES

IMPACTS

Funding from gov’t, GAP, & other donors

Train counseling & testing personnel and site managers

# personnel certified in VCT

# clients received pre-test counseling,

# clients received HIV tests,

# clients received results & post-test counseling***

Provide pre-test counseling, HIV testing, and post-test counseling to all clients tested

Counseling and Testing Personnel*

VCT protocols,

guidelines, and training documents**

Provide pregnant HIV+ women with PMTCT services

# pregnant HIV+ women receive PMTCT services

HIV test kits

Screen HIV+ clients for (or refer to screening for) OIs and TB specifically

# HIV+ clients referred to or provided OI/TB screening

VCT MIS**

Referral system for

prevention & Tx services**

# HIV+ clients referred to Tx, care & support including ARVs

Refer HIV+ clients to Tx services

vct program implementation logic model61
VCT Program Implementation Logic Model

Problem Statement:HIV infection rates continue to rise, underscoring the importance for people to know their serostatus, develop personalized risk-reduction strategies, and access care and treatment services.

OUTPUTS

INPUTS

ACTIVITIES

OUTCOMES

IMPACTS

Funding from gov’t, GAP, & other donors

Train counseling & testing personnel and site managers

# personnel certified in VCT

Quality of VCT increased

# clients received pre-test counseling,

# clients received HIV tests,

# clients received results & post-test counseling***

Provide pre-test counseling, HIV testing, and post-test counseling to all clients tested

Counseling and Testing Personnel*

Access to VCT increased

Risk behaviors decreased

VCT protocols,

guidelines, and training documents**

Clients (HIV+ and -) develop & adhere to personalized HIV risk-reduction & treatment strategy

Provide pregnant HIV+ women with PMTCT services

# pregnant HIV+ women receive PMTCT services

Health outcomes of HIV + improve

HIV test kits

Screen HIV+ clients for (or refer to screening for) OIs and TB specifically

# HIV+ clients referred to or provided OI/TB screening

Increase in care, prevention, and treatment services for HIV+, HIV-, and discordant couples

VCT MIS**

Referral system for

prevention & Tx services**

# HIV+ clients referred to Tx, care & support including ARVs

Refer HIV+ clients to Tx services

vct program implementation logic model62
VCT Program Implementation Logic Model

Problem Statement:HIV infection rates continue to rise, underscoring the importance for people to know their serostatus, develop personalized risk-reduction strategies, and access care and treatment services.

OUTPUTS

INPUTS

ACTIVITIES

OUTCOMES

IMPACTS

Funding from gov’t, GAP, & other donors

Train counseling & testing personnel and site managers

# personnel certified in VCT

Quality of VCT increased

HIV transmission rates decreased

# clients received pre-test counseling,

# clients received HIV tests,

# clients received results & post-test counseling***

Provide pre-test counseling, HIV testing, and post-test counseling to all clients tested

Counseling and Testing Personnel*

Access to VCT increased

HIV incidence decreased

Risk behaviors decreased

VCT protocols,

guidelines, and training documents**

Clients (HIV+ and -) develop & adhere to personalized HIV risk-reduction & treatment strategy

Provide pregnant HIV+ women with PMTCT services

# pregnant HIV+ women receive PMTCT services

HIV morbidity & mortality decreased

Health outcomes of HIV + improve

HIV test kits

Screen HIV+ clients for (or refer to screening for) OIs and TB specifically

# HIV+ clients referred to or provided OI/TB screening

Increase in care, prevention, and treatment services for HIV+, HIV-, and discordant couples

VCT MIS**

Referral system for

prevention & Tx services**

# HIV+ clients referred to Tx, care & support including ARVs

Refer HIV+ clients to Tx services

slide63

Relationship Between Logic Model Components and Objectives

INPUTS

ACTIVITIES

OUTPUTS

OUTCOMES

IMPACTS

# clients received pre-test counseling, # clients received HIV tests, # clients received results and post-test counseling

HIV incidence decreased

Clients (HIV+ and -) form /adhere to personalized HIV risk-reduction & treatment strategy

Risk behaviors decreased

Counseling and Testing Personnel

Provide pre-test counseling, HIV testing, post-test counseling

LOGIC MODEL

By the end of the 1st program year, 98% of clients tested will receive their HIV test results.

By the beginning of the 2nd program year, 65% of clients receiving HIV test results will have developed personalized risk-reduction/treatment strategies.

OBJECTIVES

QUESTIONS

MEASURES

what outcomes and impacts do we want to achieve to remedy the problem
What outcomes and impacts do we want to achieve to remedy the problem?

PROBLEM STATEMENT

OUTPUTS

INPUTS

ACTIVITIES

OUTCOMES

IMPACTS

Quality of VCT increased

HIV transmission rates decreased

Access to VCT increased

HIV incidence decreased

Risk behaviors decreased

Clients (HIV+ & -) develop & adhere to personalized HIV risk-reduction & treatment strategy

HIV morbidity & mortality decreased

Health outcomes of HIV + improve

Increase in care, prevention, and treatment services for HIV+, HIV-, and discordant couples

what activities might we do to achieve these outcomes
What activities might we do to achievethese outcomes?

PROBLEM STATEMENT

OUTPUTS

INPUTS

ACTIVITIES

OUTCOMES

IMPACTS

Train counseling & testing personnel and site managers

Quality of VCT increased

HIV transmission rates decreased

Provide pre-test counseling, HIV testing, and post-test counseling to all clients tested

Access to VCT increased

HIV incidence decreased

Risk behaviors decreased

Clients (HIV+ & -) develop & adhere to personalized HIV risk-reduction & treatment strategy

Provide pregnant HIV+ women with PMTCT services

HIV morbidity & mortality decreased

Health outcomes of HIV + improve

Screen HIV+ clients for (or refer for screening for?) OIs and TB specifically

Increase in care, prevention, and treatment services for HIV+, HIV-, and discordant couples

Refer HIV+ clients to Tx services

what outputs are needed to achieve each desired outcome
What outputs are needed to achieve each desired outcome?

PROBLEM STATEMENT

OUTPUTS

INPUTS

ACTIVITIES

OUTCOMES

IMPACTS

Train counseling & testing personnel and site managers

# personnel certified in VCT

Quality of VCT increased

HIV transmission rates decreased

# clients received pre-test couns.,

# clients received HIV tests,

# clients received results & post-test counseling***

Provide pre-test counseling, HIV testing, and post-test counseling to all clients tested

Access to VCT increased

HIV incidence decreased

Risk behaviors decreased

Clients (HIV+ & -) develop & adhere to personalized HIV risk-reduction & treatment strategy

Provide pregnant HIV+ women with PMTCT services

# pregnant HIV+ women receive PMTCT services

HIV morbidity & mortality decreased

Health outcomes of HIV + improve

Screen HIV+ clients for (or refer for screening for?) OIs and TB specifically

# HIV+ clients referred to or provided OI/TB screening

Increase in care, prevention, and treatment services for HIV+, HIV-, and discordant couples

# HIV+ clients referred to Tx, care & support including ARVs

Refer HIV+ clients to Tx services

what resources inputs are needed to achieve each desired output
What resources (inputs) are needed to achieve each desired output?

PROBLEM STATEMENT

OUTPUTS

INPUTS

ACTIVITIES

OUTCOMES

IMPACTS

Funding from gov’t, GAP, & other donors

Train counseling & testing personnel and site managers

# personnel certified in VCT

Quality of VCT increased

HIV transmission rates decreased

# clients received pre-test couns.,

# clients received HIV tests,

# clients received results & post-test counseling***

Provide pre-test counseling, HIV testing, and post-test counseling to all clients tested

Counseling and Testing Personnel*

Access to VCT increased

HIV incidence decreased

Risk behaviors decreased

VCT protocols,

guidelines, and training documents**

Clients (HIV+ & -) develop & adhere to personalized HIV risk-reduction & treatment strategy

Provide pregnant HIV+ women with PMTCT services

# pregnant HIV+ women receive PMTCT services

HIV morbidity & mortality decreased

Health outcomes of HIV + improve

HIV test kits

Screen HIV+ clients for (or refer for screening for?) OIs and TB specifically

# HIV+ clients referred to or provided OI/TB screening

Increase in care, prevention, and treatment services for HIV+, HIV-, and discordant couples

VCT MIS**

Referral system for

prevention & Tx services**

# HIV+ clients referred to Tx, care & support including ARVs

Refer HIV+ clients to Tx services

slide68

VCT Technical Assistance Logic Model

Problem Statement: VCT is a critical entry point to HIV/AIDS prevention, care, and treatment services and needs to be provided consistently in a quality manner.

OUTPUTS

INPUTS

ACTIVITIES

OUTCOMES

IMPACTS

# of effective VCT sites increased

Knowledge & skills to plan and implement VCT increased

Conduct VCT Training of Trainers (TOT)

Participants complete TOT

Funding

HIV transmission rates decreased*

MOH & NGO staff

# receiving VCT services increased*

Knowledge of necessary referral linkages increased

Provide TA to trainers to train district level staff

Trainers train district level clinic staff

VCT training curriculum

Knowledge of HIV status increased*

HIV incidence decreased *

Knowledge about prevention , care, support, and Tx resources incrsd.*

VCT MIS

Provide TA to clinics to recruit & train VCT staff

Clinic staff recruit & train VCT staff

Quality of VCT increased

VCT Trainers and TA providers

Condom use increased*

HIV morbidity & mortality decreased *

Risk behaviors decreased*

* Outcomes & impacts accomplished indirectly through implementation of VCT by trained staff.

cyclical logic model for hiv prevention
Cyclical Logic Model for HIV Prevention

Process Monitoring

and Evaluation

Inputs

Problem

Statement

Activities

Outputs

Impacts

Outcomes

Outcome Monitoring

and Evaluation

planned versus actual logic models
Planned Versus Actual Logic Models
  • Planned implementation and outcomes
    • During program planning, a logic model can
      • describe intended implementation
      • show expected outcomes
      • use theory, experience, and scientific evidence
  • Actual implementation and outcomes
    • Once the program is implemented, a logic model can
      • describe how program actually occurs
      • demonstrate achieved outcomes
      • contribute to theory, experience, and scientific evidence
logic model for hiv prevention71
Logic Model for HIV Prevention

Problem Statement

Implementation

Inputs Activities Outputs

Outcomes

Impacts

a good logic model
A Good Logic Model
  • Includes a problem statement, inputs, activities, outputs, outcomes, and impacts
  • Reflects agreement among major stakeholders about intended implementation and outcomes (planned logic model)
  • Illustrates clear, sequential, and logical linkages between each part of the logic model
  • Contains a problem statement that identifies underlying causes
a good logic model continued
A Good Logic Model (continued)
  • Includes outcomes responsive to the issues identified in the problem statement
  • States outcomes as changes in knowledge, attitudes, beliefs, intentions, skills, behaviors, access, policies, or environmental conditions
  • Includes outcomes that are realistic for the stated activities
  • States outcomes that are within the scope of the program’s influence
logic model benefits
Logic Model Benefits
  • Communicate the fundamental purpose of the program by explicitly outlining the intended outputs and outcomes of the program
  • Provide a reference point for everyone involved in the program
  • Illustrate the internal logical consistency of the program helping planners identify gaps or unrealistic results
  • Reveal assumptions about how the program leads to outputs and outcomes
  • Involve stakeholders and promote communication about the program among contractors, funders, community members, and other stakeholders, and evaluation
logic model benefits continued
Logic Model Benefits (continued)
  • Identify potential obstacles to program operation so that staff can address problems as soon as possible
  • Help monitor progress by providing a clear plan for tracking changes to the program so that successes can be replicated and mistakes avoided
  • Focus evaluation of the program by revealing appropriate evaluation questions and relevant data needs
  • Improve program staff’s expertise in planning, implementation, and evaluation
creating logic models instructions
Creating Logic Models“Instructions”
  • Select a program (technical area) you are currently funding, implementing, planning or for which you provide TA.
  • Complete the worksheet describing planned implementation and outcomes.

Note: Components of the logic model can be completed in any order.

  • Once you are comfortable with the logic model elements, write them onto post-it notes and begin to create a logic model with functional relationships on the flip chart paper.

* Field/regional offices should be prepared to report back on their experience developing CAP objectives and logic models at the country debrief to follow (10 minutes per country).

data use and users case study instructions
Data Use and Users Case Study“Instructions”
  • Using your own example of a technical strategy or program, answer the questions in the left-hand column with reference to two types of stakeholders (e.g., but not limited to, donors and service providers).
  • In the “stakeholder” boxes, fill in country specific examples of these types of stakeholders to personalize the exercise.
  • Be sure to describe concrete uses of the data.
strategies to promote utilization
Strategies to Promote Utilization
  • Develop buy-in among evaluation stakeholders
  • Clearly identify the intended users of the evaluation data
  • Identify evaluation questions meaningful to the intended users
  • Decide how the data will be used before the evaluation is conducted
  • Present data in a user-friendly format
evaluation logic
Evaluation Logic

Don’t attempt to measure impact

until you’ve demonstrated outcomes.

Don’t attempt to measure outcomes until you’ve demonstrated implementation (or process).

implications of not knowing how an intervention was implemented81
Implications of Not Knowing How an Intervention Was Implemented

Impact

AIDS Educ. Effect

On Incidence

Outcome

????

Planning

????

Implementation

????

slide82

Relationship Between Logic Model Components and Objectives

INPUTS

ACTIVITIES

OUTPUTS

OUTCOMES

IMPACTS

# clients received pre-test counseling, # clients received HIV tests, # clients received results and post-test counseling

HIV incidence decreased

Clients (HIV+ and -) form /adhere to personalized HIV risk-reduction & treatment strategy

Risk behaviors decreased

Counseling and Testing Personnel

Provide pre-test counseling, HIV testing, post-test counseling

LOGIC MODEL

By the end of the 1st program year, 98% of clients tested will receive their HIV test results.

By the beginning of the 2nd program year, 65% of clients receiving HIV test results will have developed personalized risk-reduction/treatment strategies.

OBJECTIVES

QUESTIONS

MEASURES

vct program implementation logic model83
VCT Program Implementation Logic Model

Problem Statement:HIV infection rates continue to rise, underscoring the importance for people to know their serostatus, develop personalized risk-reduction strategies, and access care and treatment services.

OUTPUTS

INPUTS

ACTIVITIES

OUTCOMES

IMPACTS

Funding from gov’t, GAP, & other donors

Train counseling & testing personnel and site managers

# personnel certified in VCT

Quality of VCT increased

HIV transmission rates decreased

# clients received pre-test counseling,

# clients received HIV tests,

# clients received results & post-test counseling***

Provide pre-test counseling, HIV testing, and post-test counseling to all clients tested

Counseling and Testing Personnel*

Access to VCT increased

HIV incidence decreased

Risk behaviors decreased

VCT protocols,

guidelines, and training documents**

Clients (HIV+ and -) develop & adhere to personalized HIV risk-reduction & treatment strategy

Provide pregnant HIV+ women with PMTCT services

# pregnant HIV+ women receive PMTCT services

HIV morbidity & mortality decreased

Health outcomes of HIV + improve

HIV test kits

Screen HIV+ clients for (or refer to screening for) OIs and TB specifically

# HIV+ clients referred to or provided OI/TB screening

Increase in care, prevention, and treatment services for HIV+, HIV-, and discordant couples

VCT MIS**

Referral system for

prevention & Tx services**

# HIV+ clients referred to Tx, care & support including ARVs

Refer HIV+ clients to Tx services

slide84

Relationship Between Logic Model Components and Objectives – VCT

INPUTS

ACTIVITIES

OUTPUTS

OUTCOMES

IMPACTS

# clients received pre-test counseling, # clients received HIV tests, # clients received results and post-test counseling

HIV incidence decreased

Clients (HIV+ and -) form personalized HIV risk-reduction & treatment strategy

Risk behaviors decreased

Counseling and Testing Personnel

Provide pre-test counseling, HIV testing, post-test counseling

LOGIC MODEL

By the end of the 1st program year, 98% of clients tested will receive their HIV test results.

By the beginning of the 2nd program year, 65% of clients receiving HIV test results will have developed personalized risk-reduction/treatment strategies.

OBJECTIVES

QUESTIONS

Were 98% of clients who were counseled, actually tested by the end of year 1?

Did 65% of clients who received HIV test results developed personalized risk-reduction/treatment strategies by year 2?

  • # clients tested
  • # clients received results
  • # individuals (by sex) received test results in VCT sites supported by GAP*

MEASURES

  • # clients receiving counseling for HIV testing
  • # clients tested
  • # of clients gaining knowledge of HIV status*
  • # clients developed personalized risk-reduction/treatment strategies

*GAP Indicator

slide85

VCT Technical Assistance Logic Model

Problem Statement: VCT is a critical entry point to HIV/AIDS prevention, care, and treatment services and needs to be provided consistently in a quality manner.

OUTPUTS

INPUTS

ACTIVITIES

OUTCOMES

IMPACTS

# of effective VCT sites increased

Knowledge & skills to plan and implement VCT increased

Conduct VCT TOT

Participants complete TOT

Funding

HIV transmission rates decreased*

MOH & NGO staff

# receiving VCT services increased*

Knowledge of necessary referral linkages increased

Provide TA to trainers to train district level staff

Trainers train district level clinic staff

VCT training curriculum

Knowledge of HIV status increased*

HIV incidence decreased *

Knowledge about prevention , care, support, and Tx resources incrsd.*

VCT MIS

Provide TA to clinics to recruit & train VCT staff

Clinic staff recruit & train VCT staff

Quality of VCT increased

VCT Trainers and TA providers

Condom use increased*

HIV morbidity & mortality decreased *

Risk behaviors decreased*

* Outcomes & impacts accomplished indirectly through implementation of VCT by trained staff.

slide86

Relationship Between Logic Model Components and Objectives – VCT TA

INPUTS

ACTIVITIES

OUTPUTS

OUTCOMES

IMPACTS

LOGIC MODEL

VCT trainers & TA providers

Provide TA to clinics to recruit & train VCT staff

Clinic staff recruit & train VCT staff

Knowledge of necessary referral linkages increased

# of effective VCT sites increased

HIV transmission rates decreased*

Risk behaviors decreased*

By the end of the 1st program year, 15 VCT staff will be recruited and trained

VCT providers make referrals to relevant referral sites at least 80% of the time

OBJECTIVES

QUESTIONS

Were 15 VCT staff trained by the end of year 1?

Were at least 80% of referrals made to relevant referral sites?

  • # of trainers trained
  • # district staff trained
  • # VCT clinic staff trained
  • # country nationals trained in provision of VCT services*
  • # VCT sites supported by CDC/GAP*
  • Client chart notations indicating referrals

MEASURES

*GAP Indicator

process evaluation
Process Evaluation
  • Examples of Methods:
  • Questionnaires (self-administered or
  • interviewer-facilitated)
  • Focus groups
  • Observation of program implementation
  • Site visits
  • Case studies
outcome evaluation
Outcome Evaluation
  • Experimental Design: Experimental v. control group
  • Quasi-Experimental Design: Comparison group through
  • natural or non-coerced process of assignment
  • Non-Experimental Design: May compare client variables
  • (e.g., attitudes, knowledge, behavior) before and
  • after or during participation; no comparison or
  • control group. Could complement outcome monitoring
economic evaluation
Economic Evaluation

Applied analytic methods to identify, measure, value, and compare the costs and outcomes of alternative interventions.

why care about economic evaluation methods
Why Care About Economic Evaluation Methods?
  • Resources are scarce, but wants are unlimited
    • Every choice has an associated opportunity cost
    • Trade-offs must be made
  • Objective of economic evaluation is to facilitate the use of scarce resources to maximize health outcomes
  • Complements effectiveness measures gathered through monitoring, evaluation, and research
common economic evaluation approaches in public health
Common Economic Evaluation Approaches in Public Health
  • Cost Studies
    • Program Cost Analysis
    • Cost-of-Disease Analysis
  • Cost-Outcome Studies
    • Cost-Effectiveness Analysis
    • Cost-Utility Analysis
    • Cost-Benefit Analysis
  • Resource Allocation and Other Studies
cost analyses
Cost Analyses
  • Program Cost Analysis
    • Estimates total costs of running a program
      • Labor: personnel, training
      • Supplies
      • Overhead
  • Cost-of-Disease Analysis
    • Estimates total costs incurred because of a disease or condition
      • Direct medical costs
      • Productivity losses
    • Can indicate potential benefits of health interventions
cost outcome analyses
Cost-Outcome Analyses

Compares net cost of an intervention to net outcomes achieved

  • Cost-Effectiveness
    • Effectiveness measured in health outcomes achieved
      • e.g., per HIV infection averted; per appropriate provision of antenatal services
    • Allows comparison of alternative interventions to achieve same health outcomes
    • Note: “Cost-Effective” ≠ “Cost-Saving”
cost outcome analyses cont
Cost-Outcome Analyses (cont.)
  • Cost-Utility
    • Utility is expressed as the number of life years saved adjusted to account for loss of quality or for disability (QALYs or DALYs)
      • e.g., per QALY gained
    • Allows comparison of different health interventions
      • Provision of ARVs vs. PMTCT
      • Provision of ARVs vs. polio vaccination
cost outcome analyses cont96
Cost-Outcome Analyses (cont.)
  • Cost-Benefit
    • All benefits, such as health outcomes and lives saved, are expressed in monetary terms
    • Allows comparison of disparate programs with a wide range of health and non-health outcomes
      • VCT programs vs. secondary education
      • Business-case for provision of ARVs to workers
building economic evaluations
Building Economic Evaluations
  • Sources for Model Inputs
    • Ongoing monitoring and evaluation activities
    • Special studies
    • Published literature
  • Dealing with Uncertainty
    • Sensitivity analysis
    • Monte Carlo simulations
conclusion
Conclusion

Economic Evaluation can inform policy and programming by

  • Estimating cost per health outcome achieved
  • Identifying which of competing interventions maximizes health gains
  • Determining efficient budgetary allocations given resource constraints
  • Providing evidence to key policy-makers demonstrating value of particular interventions
m e plan template
M&E Plan Template

* Field/regional offices will have an opportunity to report back on their experience developing an M&E plan during the country/region debrief to follow (10 minutes per country).

gap planning budgeting and reporting system pbrs

GAP Planning, Budgeting and Reporting System (PBRS)

  • Country Assistance Plan
  • Budget Plan
  • Annual Report & Indicators
  • Development of Data Entry & Analysis Tool
gap planning budgeting reporting system annual requirements
GAP Planning, Budgeting & Reporting SystemAnnual Requirements

1. Program Planningdue Oct 1

Country Assistance Plan (CAP)

2. Budget Planningdue Oct 1

by object class, projects, technical strategies

list of contracts, cooperative agreements

3. Progress Reportingdue Dec 1

Annual Report including indicators

ideal relationship cap budget m e and reporting
Ideal Relationship CAP, Budget, M&E and Reporting

Develop Program

Plan & Budget Plan

Due Oct. 1

Country

Assistance

Plan

Objectives

Due Oct. 1

Logic Model

Annual

Report

Due Dec. 1

Design M&E/

M&E Plan

Implement/Monitor M&E

annual planning reporting status
Annual Planning & Reporting: Status

Country Assistance Plans

  • FY2001: none
  • FY2002: 24 country programs; 1 regional office
  • FY2003 (expected): 25 country programs; 4 regional offices

Annual Reports

  • FY2001: 18 country programs
  • FY2002: 20 country programs; 1 regional office
  • FY2003 (expected): 25 country programs; 4 regional offices
importance of gap annual reports
Importance of GAP Annual Reports
  • Compile overall GAP report (see FY01 report)
  • Respond to repeated requests from HHS, OMB, the White House and Congress for information about what GAP has accomplished
    • Over the last 12 months …more than 12 requests:

Steiger report; HHS PMTCT report; several OMB requests;

Tobias briefing

technical program areas most frequently implemented 2001 2002
HIV/AIDS, TB & STI Surveillance Systems (n=15)

Laboratory capacity (n=14)

VCT (n=11)

Care and treatment (n=11)

Information systems (n=11)

TB prevention and care (n=9)

Laboratory capacity (n=21)

HIV/AIDS, TB & STI Surveillance Systems (n=20)

VCT (n=16)

Care and treatment (n=16)

Information systems (n=15)

PMTCT (n=10)

Technical Program Areas Most Frequently Implemented: 2001-2002

N = 18; 2001 CDC/GAP Annual Reports

N = 21; 2002 CDC/GAP Annual Reports

gap cooperative agreements
GAP Cooperative Agreements

* Includes 22 ‘new coags’ undergoing clearance

gap support for national policies guidelines
GAP Support for National Policies & Guidelines

Start-up:

  • Country Needs Assessment: All Countries
  • 2001 Strategic Meetings Supported: 358

GAP Countries with support for policies/guidelines/operational plans for national services

cumulative country nationals trained fy 01 02
Cumulative Country Nationals Trained: FY 01-02

4,030

FY 02 Total Nationals Trained: 12,065

FY 01 Total Nationals Trained: 4,620

2,644

2,499

1,882

1,548

1,546

1,161

716

647

326

248

69

20

prevention
Prevention

n = number of countries reporting on this indicator

care treatment
Care & Treatment

n = number of countries reporting on this indicator

surveillance infrastructure
Surveillance & Infrastructure

n = number of countries reporting on this indicator

monitoring and evaluation
Monitoring and Evaluation

n = number of countries reporting on this indicator

annual planning reporting status119
Annual Planning & Reporting: Status

Strengths

  • Understanding of importance / value of information
  • Interesting and informative

Weaknesses

Field offices:

  • Late submissions
  • Incomplete information
  • Quality: e.g, objectives not measurable, indicator data missing/inconsistent
annual planning reporting status120
Annual Planning & Reporting: Status

Weaknesses

Headquarters:

  • Repeated requests for additional information
  • Late notification of changes
  • Inefficient ‘handling’ of information
gap planning budgeting reporting system purpose
GAP Planning, Budgeting & Reporting System Purpose

To develop a unified Planning, Budgeting & Reporting System

  • to assist with reporting requirements
  • to determine program progress
  • to provide data for decision-making
  • to share information in a timely and efficient manner
country assistance plan cap
Country Assistance Plan (CAP)

1. Demographics and Epidemiology of HIV/AIDS, STIs, TB

2. National HIV/AIDS Response

3. GAP Country Profile:

  • Start of program
  • Budget
  • Technical Strategies supported: description & rationale
  • Institutional arrangements & collaboration
country assistance plan cap123
Country Assistance Plan (CAP)

4. Program Goals, Annual Objectives, Activities & Required Indicators

For *all relevant Technical Strategies:

  • Goals, Objectives, Activities, Required Indicators for FY04, FY05

5. *M&E Activities

For all relevant Technical Strategies:

  • M & E Activities (planned, ongoing)

Appendix: *CDC/GAP overall M&E Plan

*NEW information

annual report
Annual Report

1. Executive Summary:

  • Accomplishments
  • Challenges
  • Recommendations for Future Direction

2. FYO3 Achievements & *Budget spent

3. *Key Findings from Evaluation Studies

4. *FY03 Indicators

!!! Use “Indicator Guide for Annual Reporting”

Appendices: *Evaluation Study Reports

*NEW information

m e data use
M&E Data Use

Program

Improvement

Share

Data with

Partners

Reporting/

Accountability

which type of indicators are reported to gap hq
Which type of indicators are reported to GAP HQ ?

GAP support for

  • Development of Policies / Guidelines / Operational Plans
  • Training
  • Infrastructure building
  • Program / Service Implementation
  • Quality of Program / Service
global aids program evaluation framework and illustrative data types
Global AIDS Program Evaluation Framework and Illustrative Data Types

Impact(Long-termEffects)

Input

(Resources)

Assessment & Planning

Output

(Immediate

Effects)

Outcomes(Intermediate Effects)

Activities

( Interventions,

Services)

Situation Analysis

Response Analysis

Stakeholder Needs

Resource Analysis

Collaboration plans

Staff

Funds

Materials

Facilities

Supplies

Trainings

Services

Education

Treatments

Interventions

# Staff Trained

# Condoms Distributed

# Test Kits Distributed

# Clients Served

# Tests Conducted

Provider Behavior

Risk Behavior

Service Use

BehaviorClinical Outcomes

Quality of Life

HIV Incid/Prev

Social Norms

STI Incid/Prev

AIDS Morb/Mort

Economic Impact

Program Development Data

Population-based Biological, Behavioral & Social Data

Program-based Data

In addition to monitoring these illustrative data types, select programs conduct enhanced process and outcome evaluations.

gap support
GAP Support

Any one or any combination of the following:

(1) Direct funding

(2) Personnel support

(3) Materials / commodities

(4) Technical assistance

example m e indicators for vct
Example: M&E Indicators for VCT

1.1 CDC/GAP supported the development of an operational plan for expansion of VCT services: yes/no

1.2 # country nationals trained in provision of VCT services

1.3 # VCT sites supported by CDC/GAP

1.4 # individuals (by sex) tested in VCT sites supported by CDC/GAP

1.5 # individuals (by sex) testing HIV-positive in VCT sites supported by CDC/GAP

1.6 # individuals (by sex) receiving HIV-test result in VCT sites supported by CDC/GAP

example m e indicators for vct130
Example: M&E Indicators for VCT

1.7 # of HIV-infected individuals identified in CDC/GAP-supported VCT sites that have been screened for TB

1.8 # of HIV-infected individuals screened for TB in CDC/GAP-supported VCT sites that have initiated TB treatment

1.9 # of HIV-infected individuals screened for TB in CDC/GAP-supported VCT sites that have initiated TB preventive therapy

1.10 # and % observed HIV counseling sessions meeting international or national standards for quality counseling in VCT sites supported by CDC/GAP

gap planning budgeting reporting system development of data entry analysis tool
GAP Planning, Budgeting & Reporting System Development of Data Entry & Analysis Tool

Step 1: CAP, AR - HQ function

To assist with

  • collating / analyzing CAP & AR information for overall GAP annual report
  • sharing of information

Step 2: CAP, AR - Field function

To provide a data entry tool for CAP and AR

gap planning budgeting reporting system development of data entry analysis tool132
GAP Planning, Budgeting & Reporting System Development of Data Entry & Analysis Tool

Step 3: Budget information

To assist with linking budget information to CAP, AR

Next Steps to be explored

For example:

To track TA requests, response, provider, schedule, FU

planning for next steps instructions
Planning for Next Steps“Instructions”
  • Review the Action Planning Worksheet for completeness (note: this worksheet is for your use only).
  • Complete the Technical Assistance Needs Worksheet (note: please hand in to Kanchan Reed before leaving).
m e capacity
The ability to conduct effective monitoring and evaluation and use M&E findings for program management and improvement.M&E Capacity
general readiness to evaluate
General Readiness to Evaluate

Not Ready

  • Is there a willingness to evaluate?
  • Have the intended users of the evaluation been identified?
  • Is there a logic model describing planned implementation and outcomes?
  • Have evaluation questions been identified?
  • Is there a desire to use the evaluation findings?
  • Have data needs been determined?
  • Are the data needed available or feasible to collect?
  • Have evaluation resources been secured?
  • Have data collection, management, and analysis procedures been developed?
  • Is there a strategy to disseminate and use the evaluation findings?

Ready to

Evaluate!

readiness to evaluate outcomes
Readiness to Evaluate Outcomes

Not Ready

  • Is the program sustainable?
  • Is the program implemented with fidelity to its plan?
  • Is the program stable over time?
  • Does the program reach a sufficient number of people?
  • Is the program delivered with sufficient dosage?

Ready to

Evaluate!

framework for m e capacity
Framework for M&E Capacity
  • Motivational Forces
    • The reasons why M&E happens
      • Internal incentives
      • External policies
  • Organizational Environment
    • The characteristics of an agency in which M&E occurs
      • Leadership
      • Organizational norms
      • Job responsibilities
  • Workplace and Professional Development
    • The knowledge, skills, and abilities of the people who conduct M&E
      • Technical skills
      • Training opportunities
framework for m e capacity141
Framework for M&E Capacity
  • Resources and Support
    • The practical assets needed for M&E
      • M&E funds
      • Management information systems
      • Logic models and SMART objectives
  • Learning from Experience
    • The changes that occur in people, agencies, and interventions during and after M&E
      • Planning to use M&E data
      • Actually using M&E data
evaluation technical assistance resources
Evaluation Technical Assistance Resources
  • GAP M&E Team, CDC
  • UTAP Providers
  • National and local consultants
  • University faculty
  • Graduate students
  • Volunteers (e.g., evaluation association members)
  • Websites
working with an evaluation consultant
Working with an Evaluation Consultant
  • Select a TA provider who knows the topic
  • Select a TA provider who is culturally competent and can communicate clearly with different stakeholders (e.g., program managers, front line staff, community members)
  • Clarify the roles and responsibilities of the TA provider and stakeholders
  • Establish a workplan and timeline
  • Meet regularly with the TA provider to monitor progress
  • Use the TA provider to build internal capacity