Monitoring and Evaluation of Implementing National ICT Policies for the ICB4PAC Project

1. Monitoring and Evaluation of Implementing National ICT Policies for the ICB4PAC Project Honolulu, Hawaii 18-20 July, 2011 Day 1 Session 2-1: Substance of M&E Norman H. Okamura, Ph.D., Faculty Specialist Christina Higa, Associate Director Telecommunications and Information Policy Group Social Science Research Institute University of Hawaii at Manoa

2. Welcome! and Aloha!

3. National ICT Plans – Step Back Review • Pacific Island Countries have developed National Information and Communication Technology (ICT) Plans • Basic Definition of a Plan • A plan is a statement of a desired future state and the activities that will be undertaken and the resources that will be committed to achieve that future state. • Planning is a process by which the plan is developed. • The Development of the Plans were a trilateral scientific activity.

4. Definition of a Plan and Planning • Pacific Island National ICT Plans have established goals and objectives. • The goals and objectives were based on a trilateral scientific assessment based on empirical, critical, and prescriptive theory. Desired State Resources (Inputs) Actual State Actions and Activities (Processes)

5. Theory and Implementation • Pacific Island National ICT Plans have also identified actions and resources that will ensure that a country moves forward to achieve the goals. • The idea behind the plans are that by allocating resources and undertaking specific tasks, an outcome will result that moves the actual state closer to the desired state. Desired State Resources (Inputs) Actual State Actions and Activities (Processes)

6. The Science in Planning • The development of a plan contains a world view which is based on a theory(ies) that certain actions and allocation of resources will lead to a desired state. • Inherent in this world view are an assessment of the actual state, a critical evaluation of the actual state, and a theory of how things will be better if resources were committed and actions undertaken. • Three “scientific” activities were undertaken. Desired State Resources (Inputs) Actual State Actions and Activities (Processes)

7. Epistemology of Plans Plans begin with a view and observations of the world. Plans usually begin with a description of a country situation. Data “Real World” From Johan Galtung, 1972, Empiricism, Criticism, Constructivism: Three Activities of Science From Johan Galtung, 1972, Empiricism, Criticism, Constructivism: Three Activities of Science

8. Representation of World Space Cartesian Representation of X and Y Variables (World Space) The “world” is classified and defined. Y s y _ y _ t X x x Crudely and Finely Meshed

9. Data • We use Data sentences to represent of “reality” or the World Space. • Data statements usually contain an implicit or explicit “measurement model.” • Classification … helps to organize the measurement • The representations of the world are through data measurement model is based on some sort of scaling. • # of phone, minutes, people • Ratios – Internet capacity/person • The level, type, and construction of the scale depends on the how finely meshed the analyst would like to be.

10. Epistemology of Plans – Role of Theory We understand the world through theories. Theory From Johan Galtung, 1972, Empiricism, Criticism, Constructivism: Three Activities of Science

11. Theory Sentences • Theory is a statement that describes, explains, and/or predicts a phenomena. • The theories are represented as “sentences” • ICT leads to economic development • Competition leads to lower cost, improved services, and economic development • The “science” in theory attempts to test theory by comparing theory statements to data using empirical methodologies. • We can test theories by comparing the theory sentences with the data sentences • e.g. World Bank data on ICT competition and economic development

12. Theory Sentences A theory as defined by Webster’s Dictionary is: • the analysis of a set of facts in their relation to one another • abstract thought: speculation • the general or abstract principles of a body of fact, a science, or an art • a belief, policy, or procedure proposed or followed as the basis of action • an ideal or hypothetical set of facts, principles, or circumstances • a plausible or scientifically acceptable general principle or body of principles offered to explain phenomena

13. Epistemology of Plans 1: the analysis of a set of facts in their relation to one another What are some theories on ICT and development? Theory Empiricism Are there any operating theories that underlie the FSM Plan? …. What are some examples of empiricism? Data What does the World Bank mean by “evidence based?” From Johan Galtung, 1972, Empiricism, Criticism, Constructivism: Three Activities of Science

14. Epistemology of Plans Value is a concept that describes the beliefs of an individual or culture. A set of values may be placed into the notion of a value system. —Synonyms 1. utility. Value, worth imply intrinsic excellence or desirability. Value is that quality of anything which renders it desirable or useful: See appreciate. Values

15. The World can be Valued • Value sentences can also dichotomize the world space • Data sentences from world space can be categorized by the “preferred or rejected,” “right-wrong,” “desired-undesired,” “beautiful-ugly” Cartesian Representation of X and Y Variables (World Space) s Y Good Right Desired Good Right Bad Wrong Bad Wrong Undesired _ X t x x Crudely and Finely Meshed

16. Epistemology of Plans When you compare values (value sentences) with data (data sentences) you end up with an a different science - evaluation … or criticism. Data Values Criticism Evaluation

17. Epistemology of Plans How do the National ICT Plans evaluate the current situation (data)? Are we OK with things as is? In criticism … we hold in abeyance … the theory … or why Data Values Criticism Evaluation

18. Moving On … Epistemology of Plans We have two types of sciences … defined … but what happens when we compare theory and values alone? Theory Empiricism Data Values Criticism

19. Epistemology of Plans Constructive and Prescriptive theories are not bound by “what is” or the “why” of empiricism. Imagine! Theory Constructivism Prescription Empiricism Data Values Criticism

20. Epistemology of Plans The National ICT Plans are all based on empirical, normative and constructive or prescriptive statements. Implicit or Explicit Validated or Unvalidated Theory Constructivism Empiricism Data Values Criticism

21. Implications - A Concept of Trilateral Science • “[F]irst, there is a rejection of empiricism alone, as a carrier of the definition of scientific activity. • “Second, there is a rejection of the idea that trilateral science is ‘applied science’ and all might be called ‘action research.’” • Third, “There is a rejection of the scientist/politician division of labor” (Town – Gown) • The planner does all three scientific activities … • The plan represents all … • But …. The Plan Does More! The Plan Recommends certain Actions … To implement the Plan … Requires …

22. Epistemology of Plans Praxis Theory Constructivism Empiricism Data Values Criticism

23. A Moment on Praxis • Praxis is the process by which a theory, lesson, or skill is enacted, practiced, embodied, or realized. • "Praxis" may also refer to the act of engaging, applying, exercising, realizing, or practicing ideas. • This has been a recurrent topic in the field of philosophy, discussed in the writings of Plato, Aristotle, St. Augustine, Immanuel Kant, Søren Kierkegaard, Karl Marx, Martin Heidegger, Hannah Arendt, Paulo Freire, and many others. It has meaning in political, educational, and spiritual realms.

24. A Moment on Praxis • “Philosophers have only interpreted the world in various ways; the point is to change it." • “Praxis” the highest and most important level of the active life. • Georg Lukács held that the task of political organization is to establish professional discipline over everyday political praxis, consciously designing the form of mediation best suited to clear interactions between theory and practice. • Praxis is reflection and action upon the world in order to transform it.“ • In social work theory, praxis is the reflexive relationship between theories and action.

25. A Moment on Praxis • Georg Lukács held that the task of political organization is to establish professional discipline over everyday political praxis, consciously designing the form of mediation best suited to clear interactions between theory and practice. • In Ancient Greek the word praxis (πρᾶξις) referred to activity engaged in by free men. Aristotle held that there were three basic activities of man: theoria, poiesis and praxis. There corresponded to these kinds of activity three types of knowledge: theoretical, to which the end goal was truth; poietical, to which the end goal was production; and practical, to which the end goal was action.

26. A Concept of Trilateral Science • “Fourth, the scientist himself has to get out of this schizophrenia and start integrating his own activities, not submitting to a distinction of what belongs to office hours and what belongs to extracurricular activities. • That distinction makes little sense to the social scientist as to an architect – who is possibly the best model from which to learn integrated science.” • “Instead, there should be symmetrical dialogues with either party presenting his vision of the three worlds.”

27. Galtung – the reality • “Of course, in reality it is not so clear-cut because any scientific activity is a mixture of all three, which is one reason why one might just as well be honest and bring them all in explicitly.” • “Thus, the scientific process does not end with a written product, where verbal consonance – ‘paper-solutions” – are obtained. It is concluded only when reality is changed and empirical consonance obtained.” • “A good theory is not one that accounts for empirical reality, but one that leads to the realization of a preferred potential reality.”

28. Final Note • . . . [I]t is also another story whether one should stop with trilateral science or could image a fourth or fifth type of sentence defining worlds, and tetralateral or pentalateral science. We stop at this point, seeing unilateral science (descriptive, theoretical, axiological) as basic; bilateral science as more advanced because consonance enters, defining empiricism, criticism, and constructivism; and finally trilateral science, which with the assumption of value-priority becomes a practice-oriented science, making scientists not so very different from other people.”

29. Final Note In fact, that may be the basic point in the whole story: people in general observe, foresee, and prefer, and have to do so in order to live. Why should scientists not do the same, to live rather than to make a living by restricting themselves to only two of them? Let the scientist be one among all the rest of mankind, with goals built into his daily life and work, making himself accountable to others by telling openly what his preferred world is – not pretending to be like a famous recently deceased statement: neither to the left or to the right, but above.

30. Evaluating Plans

31. National ICT Plans – Step Back Review II • Basic Definition of a Plan • A plan is a statement of a desired future state and the activities that will be undertaken and the resources that will be committed to achieve that future state. • Planning is a process by which the plan is developed. • Good Plans will be able to specify a clear logic model which enables a clear articulation of Goals/Objectives, Inputs (Resources), Activities, Outputs, and Outcomes. • Good plans have examined options

32. General Evaluations of Plans There are several approaches to evaluating a plan • Plan meet Need (Integrity … Does the Plan Achieve the Goals and Objectives?) • Program Logic Model (Is the theory of intervention (actions + resources) clear?) • Workability • Does the seem workable and doable? • Are there sufficient resources allocated to undertake the activities? • Will the activities lead to success • Effectiveness/Cost • What are the options? What is the relative E/C? • Return on Investment(ROI) – Net Present Value

33. Plan Evaluation Policy, Program, and Project Logic Models

34. Program or Project Logic Model There are several approaches to evaluating a plan • Program or Project Logic Model is an analytic tool which helps the project planner to think through the relationships between actions and outcomes. • Building a logic model • Antecedents • Goals • Inputs • Work Plan Activities • Outputs • Outcomes

36. Plan Evaluation LOGFRAME or LOGicalFRAMEwork

37. General Evaluations of Plans (WB Plan) There are several approaches to evaluating a plan. • The logical framework or LOGFRAME is an analytical tool used to plan, monitor, and evaluate projects. It derives its name from the logical linkages set out by the planner(s) to connect a project’s means with its ends. The LogFrameis only one monitoring and evaluation tool and its use does not pre-empt the use of other evaluation tools such as priority-setting or rate-of returnanalysis. • The framework or structure behind the LogFrameis a 4x4 matrix. That is, four rows and four columns.

39. Implementation – Monitoring - Evaluation • The plan is intended to direct activities and resources that are intended to close the gap between the actual state and the desired state. • Success is generally measured when the desired state and actual state converge. • But …. as resources are allocated and activities conducted ….. will it automatically bring success? • How do we know we are making progress (closing the gap between the actual and desired states? • We need to monitor and evaluate our activities since it will impact the “actual state… and could even create unintended consequences.”

40. Monitoring and Evaluation

41. National ICT Plans – Step Back Review II • Implementation is the carrying out, execution, or practice of a plan, a method, or any design for doing something. As such, implementation is the action that must follow any preliminary thinking in order for something to actually happen. • In an context of the National ICT Plans, implementation encompasses all the processes, activities, and resources applied to doing things so that we actualize the future state. • So … Let’s assume that policymakers have blessed the plan and we are moving ahead!

42. Theoria and Praxis The Town-Gown Philosophies Town: “Mine is not to reason why, Mine is but to do or die!” Gown: “Mine is not to do or die, Mine is only to reason why!” Integrated Scientist … Planner … Community Participant …. “XXXXXXXXXXXXXXXXX” World Bank – Building a Results-Based Monitoring and Evaluation System

43. The M&E Framework World Bank – Building a Results-Based Monitoring and Evaluation System

44. Monitoring and Evaluation (M&E) • Major Questions … How do we know… • Whether we are successful in moving the actual state closer to the desired state? • How well we are doing? • Whether we could do something a different way? • Whether our activities and resources are having intended or other unintended consequences? • Is success, or failure, a result of the theory of the plan or the implementation of a plan? • At all levels …. We answer these questions through Monitoring and Evaluation (M&E) • In policy, program, and project planning, M&E are fundamental processes to effective management and often to the success or failure of plans.

45. Monitoring and Evaluation Definitions and Distinctions

46. What do we mean by “Monitoring”? • Monitoring - The monitoring process looks at what is being done and how it is being done. • It could involve continuous tracking of activities, review of the flow of services and activities provided by the program, compliance with laws, regulations, guidelines, etc. Much of the information desirable to monitor a program is also valuable for the implement patient and evaluation component. • Monitoring - The routine tracking of key elements of program/ project performance, usually inputs and outputs, through record-keeping, regular reporting, and surveillance systems such as client surveys.

47. What do we mean by “Evaluation”? • Evaluation Evaluation is the episodic assessment of the change in the targeted results (goals and objectives) that can be attributed to the program/ project intervention. Evaluation attempts to link a particular output or outcome directly to an intervention after a period of time has passed. Evaluations could tell us whether success or failure was a function of theory or a function of the way the policy/program/project was implemented.

48. Formative v. Summative Evaluation • Evaluation - Evaluation looks at performance against goals. This can and should take place while the program or research project is underway, and is concerned with evaluating how the intervention is meeting its performance goals. The early aspect is called formative evaluation; it overlaps with monitoring to the extent that it uses the data gathered during the monitoring that is specifically performance related. Near the end of the project, there should be an evaluation of how effective the program has been or rather the research project has met its original objectives. This is called a summative evaluation.

49. Formative v. Summative Evaluation • Formative Evaluation - Formative evaluation is a type of evaluation which has the purpose of improving programs. • Formative evaluation is a method of judging the worth of a program while the program activities are forming or happening. Formative evaluation focuses on the process (Bhola 1990).

50. Monitoring and Evaluation World Bank –