slide1
Download
Skip this Video
Download Presentation
Monitoring and Evaluation of Implementing National ICT Policies for the ICB4PAC Project

Loading in 2 Seconds...

play fullscreen
1 / 61

Monitoring and Evaluation of Implementing National ICT Policies for the ICB4PAC Project - PowerPoint PPT Presentation


  • 129 Views
  • Uploaded on

Monitoring and Evaluation of Implementing National ICT Policies for the ICB4PAC Project . Honolulu, Hawaii 18-20 July, 2011. Day 1 Session 2-1: Substance of M&E. Norman H. Okamura, Ph.D., Faculty Specialist Christina Higa, Associate Director

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Monitoring and Evaluation of Implementing National ICT Policies for the ICB4PAC Project ' - bayard


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Monitoring and Evaluation of Implementing National ICT Policies

for the ICB4PAC Project

Honolulu, Hawaii 18-20 July, 2011

Day 1 Session 2-1: Substance of M&E

Norman H. Okamura, Ph.D., Faculty Specialist

Christina Higa, Associate Director

Telecommunications and Information Policy Group

Social Science Research Institute

University of Hawaii at Manoa

slide2
Welcome!

and

Aloha!

national ict plans step back review
National ICT Plans – Step Back Review
  • Pacific Island Countries have developed National Information and Communication Technology (ICT) Plans
  • Basic Definition of a Plan
    • A plan is a statement of a desired future state and the activities that will be undertaken and the resources that will be committed to achieve that future state.
    • Planning is a process by which the plan is developed.
  • The Development of the Plans were a trilateral scientific activity.
definition of a plan and planning
Definition of a Plan and Planning
  • Pacific Island National ICT Plans have established goals and objectives.
  • The goals and objectives were based on a trilateral scientific assessment based on empirical, critical, and prescriptive theory.

Desired State

Resources (Inputs)

Actual State

Actions and Activities

(Processes)

theory and implementation
Theory and Implementation
  • Pacific Island National ICT Plans have also identified actions and resources that will ensure that a country moves forward to achieve the goals.
  • The idea behind the plans are that by allocating resources and undertaking specific tasks, an outcome will result that moves the actual state closer to the desired state.

Desired State

Resources (Inputs)

Actual State

Actions and Activities

(Processes)

the science in planning
The Science in Planning
  • The development of a plan contains a world view which is based on a theory(ies) that certain actions and allocation of resources will lead to a desired state.
  • Inherent in this world view are an assessment of the actual state, a critical evaluation of the actual state, and a theory of how things will be better if resources were committed and actions undertaken.
  • Three “scientific” activities were undertaken.

Desired State

Resources (Inputs)

Actual State

Actions and Activities

(Processes)

epistemology of plans
Epistemology of Plans

Plans begin with a view and observations of the world.

Plans usually begin with a description of a country situation.

Data

“Real World”

From Johan Galtung, 1972, Empiricism, Criticism, Constructivism: Three Activities of Science

From Johan Galtung, 1972, Empiricism, Criticism, Constructivism: Three Activities of Science

representation of world space
Representation of World Space

Cartesian Representation of X and Y Variables

(World Space)

The “world” is classified and defined.

Y

s

y

_

y

_

t

X

x

x

Crudely and Finely Meshed

slide9
Data
  • We use Data sentences to represent of “reality” or the World Space.
  • Data statements usually contain an implicit or explicit “measurement model.”
    • Classification … helps to organize the measurement
  • The representations of the world are through data measurement model is based on some sort of scaling.
    • # of phone, minutes, people
    • Ratios – Internet capacity/person
  • The level, type, and construction of the scale depends on the how finely meshed the analyst would like to be.
epistemology of plans role of theory
Epistemology of Plans – Role of Theory

We understand the world through theories.

Theory

From Johan Galtung, 1972, Empiricism, Criticism, Constructivism: Three Activities of Science

theory sentences
Theory Sentences
  • Theory is a statement that describes, explains, and/or predicts a phenomena.
  • The theories are represented as “sentences”
    • ICT leads to economic development
    • Competition leads to lower cost, improved services, and economic development
  • The “science” in theory attempts to test theory by comparing theory statements to data using empirical methodologies.
    • We can test theories by comparing the theory sentences with the data sentences
    • e.g. World Bank data on ICT competition and economic development
theory sentences1
Theory Sentences

A theory as defined by Webster’s Dictionary is:

  • the analysis of a set of facts in their relation to one another
  • abstract thought: speculation
  • the general or abstract principles of a body of fact, a science, or an art
  • a belief, policy, or procedure proposed or followed as the basis of action
  • an ideal or hypothetical set of facts, principles, or circumstances
  • a plausible or scientifically acceptable general principle or body of principles offered to explain phenomena
epistemology of plans1
Epistemology of Plans

1: the analysis of a set of facts in their relation to one another

What are some theories on ICT and development?

Theory

Empiricism

Are there any operating theories that underlie the FSM Plan? ….

What are some examples of empiricism?

Data

What does the World Bank mean by “evidence based?”

From Johan Galtung, 1972, Empiricism, Criticism, Constructivism: Three Activities of Science

epistemology of plans2
Epistemology of Plans

Value is a concept that describes the beliefs of an individual or culture. A set of values may be placed into the notion of a value system.

—Synonyms

1. utility. Value, worth imply intrinsic excellence or desirability. Value is that quality of anything which renders it desirable or useful: See appreciate.

Values

the world can be valued
The World can be Valued
  • Value sentences can also dichotomize the world space
  • Data sentences from world space can be categorized by the “preferred or rejected,” “right-wrong,” “desired-undesired,” “beautiful-ugly”

Cartesian Representation of X and Y Variables

(World Space)

s

Y

Good Right Desired

Good Right

Bad Wrong

Bad Wrong Undesired

_

X

t

x

x

Crudely and Finely Meshed

epistemology of plans3
Epistemology of Plans

When you compare values (value sentences) with data (data sentences) you end up with an a different science - evaluation … or criticism.

Data

Values

Criticism

Evaluation

epistemology of plans4
Epistemology of Plans

How do the National ICT Plans evaluate the current situation (data)?

Are we OK with things as is?

In criticism … we hold in abeyance … the theory … or why

Data

Values

Criticism

Evaluation

moving on epistemology of plans
Moving On … Epistemology of Plans

We have two types of sciences … defined … but what happens when we compare theory and values alone?

Theory

Empiricism

Data

Values

Criticism

epistemology of plans5
Epistemology of Plans

Constructive and Prescriptive theories are not bound by “what is” or the “why” of empiricism.

Imagine!

Theory

Constructivism

Prescription

Empiricism

Data

Values

Criticism

epistemology of plans6
Epistemology of Plans

The National ICT Plans are all based on empirical, normative and constructive or prescriptive statements.

Implicit or Explicit

Validated or Unvalidated

Theory

Constructivism

Empiricism

Data

Values

Criticism

implications a concept of trilateral science
Implications - A Concept of Trilateral Science
  • “[F]irst, there is a rejection of empiricism alone, as a carrier of the definition of scientific activity.
  • “Second, there is a rejection of the idea that trilateral science is ‘applied science’ and all might be called ‘action research.’”
  • Third, “There is a rejection of the scientist/politician division of labor” (Town – Gown)
  • The planner does all three scientific activities …
  • The plan represents all …
  • But …. The Plan Does More! The Plan Recommends certain Actions … To implement the Plan … Requires …
epistemology of plans7
Epistemology of Plans

Praxis

Theory

Constructivism

Empiricism

Data

Values

Criticism

a moment on praxis
A Moment on Praxis
  • Praxis is the process by which a theory, lesson, or skill is enacted, practiced, embodied, or realized.
  • "Praxis" may also refer to the act of engaging, applying, exercising, realizing, or practicing ideas.
    • This has been a recurrent topic in the field of philosophy, discussed in the writings of Plato, Aristotle, St. Augustine, Immanuel Kant, Søren Kierkegaard, Karl Marx, Martin Heidegger, Hannah Arendt, Paulo Freire, and many others. It has meaning in political, educational, and spiritual realms.
a moment on praxis1
A Moment on Praxis
  • “Philosophers have only interpreted the world in various ways; the point is to change it."
  • “Praxis” the highest and most important level of the active life.
  • Georg Lukács held that the task of political organization is to establish professional discipline over everyday political praxis, consciously designing the form of mediation best suited to clear interactions between theory and practice.
  • Praxis is reflection and action upon the world in order to transform it.“
  • In social work theory, praxis is the reflexive relationship between theories and action.
a moment on praxis2
A Moment on Praxis
  • Georg Lukács held that the task of political organization is to establish professional discipline over everyday political praxis, consciously designing the form of mediation best suited to clear interactions between theory and practice.
  • In Ancient Greek the word praxis (πρᾶξις) referred to activity engaged in by free men. Aristotle held that there were three basic activities of man: theoria, poiesis and praxis. There corresponded to these kinds of activity three types of knowledge: theoretical, to which the end goal was truth; poietical, to which the end goal was production; and practical, to which the end goal was action.
a concept of trilateral science
A Concept of Trilateral Science
  • “Fourth, the scientist himself has to get out of this schizophrenia and start integrating his own activities, not submitting to a distinction of what belongs to office hours and what belongs to extracurricular activities.
  • That distinction makes little sense to the social scientist as to an architect – who is possibly the best model from which to learn integrated science.”
  • “Instead, there should be symmetrical dialogues with either party presenting his vision of the three worlds.”
galtung the reality
Galtung – the reality
  • “Of course, in reality it is not so clear-cut because any scientific activity is a mixture of all three, which is one reason why one might just as well be honest and bring them all in explicitly.”
  • “Thus, the scientific process does not end with a written product, where verbal consonance – ‘paper-solutions” – are obtained. It is concluded only when reality is changed and empirical consonance obtained.”
  • “A good theory is not one that accounts for empirical reality, but one that leads to the realization of a preferred potential reality.”
final note
Final Note
  • . . . [I]t is also another story whether one should stop with trilateral science or could image a fourth or fifth type of sentence defining worlds, and tetralateral or pentalateral science. We stop at this point, seeing unilateral science (descriptive, theoretical, axiological) as basic; bilateral science as more advanced because consonance enters, defining empiricism, criticism, and constructivism; and finally trilateral science, which with the assumption of value-priority becomes a practice-oriented science, making scientists not so very different from other people.”
final note1
Final Note

In fact, that may be the basic point in the whole story: people in general observe, foresee, and prefer, and have to do so in order to live. Why should scientists not do the same, to live rather than to make a living by restricting themselves to only two of them? Let the scientist be one among all the rest of mankind, with goals built into his daily life and work, making himself accountable to others by telling openly what his preferred world is – not pretending to be like a famous recently deceased statement: neither to the left or to the right, but above.

national ict plans step back review ii
National ICT Plans – Step Back Review II
  • Basic Definition of a Plan
    • A plan is a statement of a desired future state and the activities that will be undertaken and the resources that will be committed to achieve that future state.
    • Planning is a process by which the plan is developed.
  • Good Plans will be able to specify a clear logic model which enables a clear articulation of Goals/Objectives, Inputs (Resources), Activities, Outputs, and Outcomes.
  • Good plans have examined options
general evaluations of plans
General Evaluations of Plans

There are several approaches to evaluating a plan

  • Plan meet Need (Integrity … Does the Plan Achieve the Goals and Objectives?)
    • Program Logic Model (Is the theory of intervention (actions + resources) clear?)
  • Workability
    • Does the seem workable and doable?
      • Are there sufficient resources allocated to undertake the activities?
      • Will the activities lead to success
  • Effectiveness/Cost
    • What are the options? What is the relative E/C?
  • Return on Investment(ROI) – Net Present Value
slide33
Plan Evaluation

Policy, Program, and Project

Logic Models

program or project logic model
Program or Project Logic Model

There are several approaches to evaluating a plan

  • Program or Project Logic Model is an analytic tool which helps the project planner to think through the relationships between actions and outcomes.
  • Building a logic model
    • Antecedents
    • Goals
    • Inputs
    • Work Plan Activities
    • Outputs
    • Outcomes
slide36
Plan Evaluation

LOGFRAME

or

LOGicalFRAMEwork

general evaluations of plans wb plan
General Evaluations of Plans (WB Plan)

There are several approaches to evaluating a plan.

  • The logical framework or LOGFRAME is an analytical tool used to plan, monitor, and evaluate projects. It derives its name from the logical linkages set out by the planner(s) to connect a project’s means with its ends. The LogFrameis only one monitoring and evaluation tool and its use does not pre-empt the use of other evaluation tools such as priority-setting or rate-of returnanalysis.
  • The framework or structure behind the LogFrameis a 4x4 matrix. That is, four rows and four columns.
implementation monitoring evaluation
Implementation – Monitoring - Evaluation
  • The plan is intended to direct activities and resources that are intended to close the gap between the actual state and the desired state.
  • Success is generally measured when the desired state and actual state converge.
    • But …. as resources are allocated and activities conducted ….. will it automatically bring success?
    • How do we know we are making progress (closing the gap between the actual and desired states?
    • We need to monitor and evaluate our activities since it will impact the “actual state… and could even create unintended consequences.”
national ict plans step back review ii1
National ICT Plans – Step Back Review II
  • Implementation is the carrying out, execution, or practice of a plan, a method, or any design for doing something. As such, implementation is the action that must follow any preliminary thinking in order for something to actually happen.
  • In an context of the National ICT Plans, implementation encompasses all the processes, activities, and resources applied to doing things so that we actualize the future state.
  • So … Let’s assume that policymakers have blessed the plan and we are moving ahead!
theoria and praxis
Theoria and Praxis

The Town-Gown Philosophies

Town: “Mine is not to reason why, Mine is but to do or die!”

Gown: “Mine is not to do or die, Mine is only to reason why!”

Integrated Scientist … Planner … Community Participant ….

“XXXXXXXXXXXXXXXXX”

World Bank – Building a Results-Based Monitoring and Evaluation System

the m e framework
The M&E Framework

World Bank – Building a Results-Based Monitoring and Evaluation System

monitoring and evaluation m e
Monitoring and Evaluation (M&E)
  • Major Questions … How do we know…
    • Whether we are successful in moving the actual state closer to the desired state?
    • How well we are doing?
    • Whether we could do something a different way?
    • Whether our activities and resources are having intended or other unintended consequences?
    • Is success, or failure, a result of the theory of the plan or the implementation of a plan?
  • At all levels …. We answer these questions through Monitoring and Evaluation (M&E)
  • In policy, program, and project planning, M&E are fundamental processes to effective management and often to the success or failure of plans.
slide45
Monitoring and Evaluation

Definitions and Distinctions

what do we mean by monitoring
What do we mean by “Monitoring”?
  • Monitoring - The monitoring process looks at what is being done and how it is being done.
    • It could involve continuous tracking of activities, review of the flow of services and activities provided by the program, compliance with laws, regulations, guidelines, etc. Much of the information desirable to monitor a program is also valuable for the implement patient and evaluation component.
  • Monitoring - The routine tracking of key elements of program/ project performance, usually inputs and outputs, through record-keeping, regular reporting, and surveillance systems such as client surveys.
what do we mean by evaluation
What do we mean by “Evaluation”?
  • Evaluation

Evaluation is the episodic assessment of the change in the targeted results (goals and objectives) that can be attributed to the program/ project intervention. Evaluation attempts to link a particular output or outcome directly to an intervention after a period of time has passed.

Evaluations could tell us whether success or failure was a function of theory or a function of the way the policy/program/project was implemented.

formative v summative evaluation
Formative v. Summative Evaluation
  • Evaluation - Evaluation looks at performance against goals. This can and should take place while the program or research project is underway, and is concerned with evaluating how the intervention is meeting its performance goals. The early aspect is called formative evaluation; it overlaps with monitoring to the extent that it uses the data gathered during the monitoring that is specifically performance related. Near the end of the project, there should be an evaluation of how effective the program has been or rather the research project has met its original objectives. This is called a summative evaluation.
formative v summative evaluation1
Formative v. Summative Evaluation
  • Formative Evaluation - Formative evaluation is a type of evaluation which has the purpose of improving programs.
  • Formative evaluation is a method of judging the worth of a program while the program activities are forming or happening. Formative evaluation focuses on the process (Bhola 1990).
why monitoring and evaluation
Why Monitoring and Evaluation?
  • Help Make Resource Allocation Decisions
  • Help Rethink the Causes of the Problem
  • Identify Emerging Problems
  • Support Decision-making in Analysis of Competing or Best Alternatives
  • Support Public Sector Reform and Innovation
  • Build Consensus on the Causes of a Problem and How to Respond

World Bank – Building a Results-Based Monitoring and Evaluation System

for managers
For Managers
  • Strategic Level: Are the right things being done?

- Rationale or justification

- Clear theory of change

  • Operations Level: are things being done right?

- Effectiveness in achieving expected outcomes

- Efficiency and optimizing resources

- Client satisfaction

  • Learning Level: Are there better ways to do things?

- Alternatives

- Best practices

- Lessons Learned

World Bank – Building a Results-Based Monitoring and Evaluation System

who is the audience for the evaluation
Who is the Audience for the Evaluation?
  • Who will receive what information?
  • In what format?
  • When?
  • Who will prepare the information?
  • Who will deliver the information?

World Bank – Building a Results-Based Monitoring and Evaluation System

broad implementation strategies
Broad Implementation Strategies
  • Performance logic chain assessment
  • Pre-Implementation assessment
  • Process Implementation evaluation
  • Rapid appraisal
  • Case study
  • Impact evaluation
  • Meta-evaluation

World Bank – Building a Results-Based Monitoring and Evaluation System

scientific evidence and data availability
Scientific Evidence and Data Availability
  • Systems, including financial and human resources, available for surveillance, measurement of targets and monitoring.
  • National expert recommendations.
  • Sources of information, e.g. data sets, and evidence in the country.
  • Relationship between research and policy, and the means by which information is transferred to policy-makers and vice-versa.
slide58
Steps in the Monitoring

and Evaluation Process

implementation
Implementation
  • So … let’s assume that we have approved plans and begin to implement the plan by allocating resources and undertaking some actions.
implementation1
Implementation
  • So … let’s assume that we have approved plans and begin to implement the plan by allocating resources and undertaking some actions.

Christina will tell us how!

ad