MONITORING AND EVALUATION IN TB/HIV PROGRAMS

MONITORING AND EVALUATION IN TB/HIV PROGRAMS Presentation for the ProTEST Lessons Learnt workshop, Durban, RSA By Wilfred Nkhoma Medical Officer, WHO/AFRO/TB UNIT

FOCI OF MONITORING AND EVALUATION IN TB/HIV PROGRAMS • Assessment of the design of interventions • Monitoring of program implementation • Assessment of program utility (effectiveness and efficiency):

Monitoring program conceptualisation • Has a problem been appropriately conceptualized ? • is the program designed to meet intended objectives ?)

MONITORING PROGRAMME IMPLEMENTATION: • To determine the extent to which a program is reaching the appropriate target population [Program coverage] • To determine whether or not delivery of program elements (services and treatments) is consistent with program design specifications[Program delivery] • To determine what resources are being expended in the conduct of the program [Program costs]

Monitoring of Program Implementation questions There is no point in being concerned with the impact or outcome of a program unless it has indeed taken place and served the appropriate participants in the way intended (as designed). [One not uncommon reason impact evaluations reveal little or no impact is not that a program is ineffective, but rather that its implementation is faulty or incomplete] • Is the program reaching the specified target population (program coverage)? • Are the intervention efforts being conducted as specified in the program design (program delivery)

Monitoring programme coverage • Coverage refers to the extend to which the participation by the target population reaches the levels specified in the program design [conclusions include undercoverage, bias and overcoverage].Bias is the degree to which sub-groups of the target population participate differentially Estimating of efficiency of programme coverage: Coverage efficiency = 100 *{number Number not } {in need served - in need served } {Total number Total number } {in need served }

Monitoring delivery of services Measures the extent to which program specifications are met in the delivery of the interventions. Common causes of delivery failures (Rossi, 1978): • No treatment or not enough is delivered: “Nonprograms”. Little evidence that a program existed ! • Wrong treatments delivered • Treatment is unstandardised, uncontrolled or varies across target populations [e.g too much discretion in the hands of service providers!] resulting in non-comparability between sites or communities

ASSESSMENT OF PROGRAM UTILITY Assessing the utility of intervention programmes involves systematic examination of : i) Programme impacts [extent to which programme produces desired outcomes] ii) Programme efficiency [the benefits of a programme in relation to its costs] Thus utility assessments help to distinguish useful from ineffective and inefficient intervention programs/ interventions

IMPACT ASSESSMENTS Undertaken to establish whether or not interventions produce their intended effects (the extent to which a program causes change in the desired direction) • Basic aim is to produce an estimate of the “ net effects” of an intervention free of the influence of other processes and events that may produce the same results [establishing causality]. • Comparison of outcomes in targets who have experienced an intervention and those who have not is the underlying common frame of reference for all impact evaluations Basic requirements: • Well articulated objectives • Specification of measures of goal achievement • Sufficiently well implemented interventions that there is no question that its critical elements have been delivered to appropriate targets. • Data collection plan that allows measures of goal achievement to be derived [commonly quantitative or quantifiable qualitative data]

EFFICIENCY ASSESSMENTS • Provide a frame of reference for relating program costs to program results (i.e guages the extent to which a program is producing sufficient benefits to program participants for the costs incurred) • Efficiency assessments employ cost-benefit and cost-effectiveness analysis methodologies • These can be undertaken before or after a program has been in existence but often only possible after program has been in existence

Cost-benefit analysis • Program inputs and program outcomes measured in monetary terms • Requires specification, measurement and valuation of all program costs and benefits (costs include both direct and indirect costs of undertaking the program).

Cost-effectiveness analysis • Requires program costs in monetary terms and benefits in outcome units • Generally less controversial than cost-benefit analysis as regards the need to express outcomes in monetary terms • Final measure is expressed as the cost of achieving a magnitude of substantive intervention outcome • Amenable to rank ordering of programs in terms of their costs for reaching given goals or the various inputs required for different degrees of goal achievement

PROGRAM UTILITY QUESTIONS • Is the programme effective in achieving its intended goals ? • Can the results of the program be explained by some alternative process that does not include the programme ? • What are the costs to deliver services and benefits to program participants ? • Is the programme an efficient use of resources, compared with alternative uses of the resources ?

CONCLUSIONS • Monitoring and evaluation are essential elements of the process of implementation of TB/HIV programs (as for other public health and social programs) • Program implementation and utility with particular focus on coverage, delivery, impact and cost-effectiveness of interventions constitute the cardinal areas in monitoring and evaluating TB/HIV activities (and should be planned for at the design stage of intervention programs). • Meaningful interpretation of monitoring and evaluation efforts entails specification of services: defining each kind of service in terms of the activities that should take place or in terms of participation by target populations and providers [time, costs, procedures, or a product, i.e. Specific simple program elements that can be identified, and reliably meaningfully counted !]

MONITORING AND EVALUATION IN TB/HIV PROGRAMS