ON THE NEED TO LEARN IN MONITORING AND EVALUATION

1. ON THE NEED TO LEARN IN MONITORING AND EVALUATION Paper for SAMEA 29 March 2007 Prof. Gert J van der Westhuizen University of Johannesburg gertvdw@uj.ac.za

2. THIS PRESENTATION: 1. Why consider learning in Monitoring and Evaluation? 2. Some relevant learning theory perspectives 3. How evaluation approaches open up spaces for learning 4. What is the potential for and assumptions about learning in one M+E report 5. Recommendations for enhancing learning in M+E My interest in learning: From an Educational Psychology perspective, drawing on recent articulations of learning theories, and how they inform practices that promote learning equity in education and evaluation

3. 1. WHY CONSIDER LEARNING IN M+E? LEARNING is the central purpose of M+E work This Learning purpose is mostly underestimated: • Little reference is made to WHO should learn, and WHAT should be learned, and WHEN, WHERE, HOW • M+E reports treat the need for learning as incidental, and rarely PLAN for learning • It is not clear how evaluation stakeholders understand learning and how learning may best be facilitated over time, across situations • Explicit emphases on learning may enhance the relevance, responsiveness and utility value of an evaluation • Critical to consider what the learning is in Higher Education M and E, especially given the CHE M+E Framework of 2004

4. 2. SOME RELEVANT LEARNING THEORY PERSPECTIVES • Learning is a social process of meaning making, drawing on technical [pen, paper, computer, etc] and psychological tools [concepts, symbols, social norms, etc.][Vygotsky] • Learning involves conceptual change under specific conditions, based on specific beliefs [Duit] • Learning is strongly shaped by activity [Engestrom], in situations [Rogoff, Brown et al.] • Learning is distributed, continuous, across situations and times, and a kind of apprenticeship [Brown et al.] • Semiotic approaches describe the use of socio-cultural tools in meaning making [Lemke] • Learning is facilitated in communities of practice and activity [Lave and Wegner] • Concepts of emancipatory and transformational learning advances social change

5. 3. HOW RECENT APPROACHES TO EVALUATION INVITE LEARNING IN EXPLICIT WAYS • Responsive/deliberate democratic evaluation [Stake 2004, House 2004, others] – importance of experience, practical participation, dialogue, deliberation • Theory driven evaluation [Weiss 2000, Chen 2005] – clarifying the theory of transformation • Constructivist models [Lincoln and Guba 2004] – emphasise meaning making, meaning deriving activities in evaluation • Utilisation-focused evaluation [Patton 1997] – primary users as learners • Practice-based evaluation [Schwandt 2004] – learning from practice; evaluation as discursive practice [Fischer 2003]

6. 4. CONTEXT: THE SOUTH AFRICAN HIGHER EDUCATION M+E FRAMEWORK [2004] • The M+E Framework sets the scene – clarifying roles, responsibilities in higher education M+E, with the Council for Higher Education [CHE] the main monitoring and evaluation agency • Main purpose of M+E in higher education is to assess progress towards transformation goals [including equity and redress, development, quality, efficiency, democratic practice, etc.] • Monitoring of the HE system is a key Council for Higher Education responsibility – to produce data that should have utilisation at system and institutional levels; to contribute knowledge both for application in policy making and for the advancement of a theoretical understanding of change [p. 20]

7. 5. THE CHE FRAMEWORK FOR M+E: THE PROMISE OF LEARNING • Some clarity about the WHAT of learning – system performance, progress towards transformation goals • No explicit emphasis on the WHO, HOW of learning • No reference to processes, dynamics, conditions of learning

8. 6. EXEMPLARY ANALYSIS – WHAT IS THE LEARNING IN ONE HIGHER EDUCATION M+E REPORT? [1] • Report by the CHE M+E Directorate: Higher Education responses to students with disabilities [Howell 2005] • Framed as research/baseline report • Design: questionnaire and interviews • Main learning: history still influences access for disabled students [DS]; resource disparities exist for DS across higher education institutions; there is a need for flexibility in teaching to DS; clear need for mainstreaming support to DS

9. 6. EXEMPLARY ANALYSIS – WHAT IS THE LEARNING IN ONE HE M+E REPORT? [2] Some findings relevant to the need to enhance learning: • The concept of working with Disabled Students in higher education “in the social model of restructuring society” seems limited to the evaluators? [limited levels of understanding, meaning making] • Indicators of successful integration of DS into higher education have not been developed [no accounting for interpretations, construction of knowledge]. • Not clear what it would take to change peoples minds about student disabilities as deficit [underestimated need for conceptual change]. • Practice experiences of lecturers working with DS in specific situations not well documented [lack of understanding how learning is situated, experiential]. • Diversity of views about mainstreaming of DS not well accounted for [unclear which are the semiotic tools people use when they talk about DS?]. • No “communities of university practitioners working with DS” established [missing out on knowledge construction in communities].

10. 7. RECOMMENDATIONS FOR THE ENHANCEMENT OF LEARNING IN THE M+E OF THE INTEGRATION OF STUDENTS WITH DISABILITIES INTO SA HIGHER EDUCATION INSTITUTIONS [1] • Ensure/enhance/increase participation in M+E [in indicator development, institutional monitoring systems, data gathering - to develop shared understandings, build communities of practice… • Focus on what may be learned from practice, from activity • Develop learning capacity – to ask questions and use learning tools • Building communities of learning

11. 7. RECOMMENDATIONS [2] • Successful M+E work may be measured by the extent to which learning is enhanced in meaningful and emancipatory ways • To problematise learning in M+E one may ask questions about the learning purpose, the audiences for learning, and the WHAT, WHY, HOW, WHEN, WHERE of learning • Emancipatory learning involves using M+E for social change