A MERICAN I NSTITUTES FOR R ESEARCH

1. AMERICAN INSTITUTES FOR RESEARCH An Analysis of the Research and Impact of ICT in Education in Developing Country Contexts Nitika Tolani-Brown, PhD Meredith McCormac, MA Roy Zimmermann, PhD

2. Background • Within the education sector, the importance of ICT lies in its ability to provide increased access to knowledge and services that such tools provide marginalized or traditionally underserved populations. • There is a sense of positive impact among practitioners but no rigorous research to support “feeling” • In the absence of rigorous experimental literature, it is easy to persist in the notion that ICT is “a magic bullet that will provide the answer to long standing educational challenges” • This review is a first step to establishing recommendations and developing a research agenda for academicians, practitioners and policymakers to estimate impacts of ICT on educational outcomes in the developing world

3. Scope • Developing country contexts • Experimental, quasi-experimental and qualitative/case studies • Educational outcomes for learners, excluding impacts on other beneficiaries • Clear consensus has not emerged on the direction or the magnitude of ICTs effects on students • Policymakers need to know more about the combinations of factors that enhance or mitigate the effects of ICTs on students in educational settings if they are to make decisions based on evidence rather than intuition.

4. Methodology • Literature Review • Qualitative Interviews with Experts • Contacted 40 experts, including researchers, academicians, policymakers and practitioners in the US, Europe and developing countries (e.g. Zambia, Nepal and Afghanistan) • 25 interviews conducted

5. Qualitative Literature • Small pool of purely qualitative evaluations within our criteria • Our report highlights three evaluations that represent a diverse set of approaches and methodological rigor within the qualitative arena • Methodologies included in-depth structured interviews or focus group discussions with program beneficiaries at the local and policy levels, classroom and school observations, and literature reviews • Case studies alone do not provide a rigorous enough base from which to draw concrete programmatic and policy recommendations or impact estimates • Need for replication – a study’s findings may not be relevant after a few years

6. Experimental Evaluations • Very few experimental evaluations have been conducted on the impact of ICT on student outcomes • Experimental Evaluations are important: • Provide internal and external validity and generalizability • RCTs can help identify differential effects across subgroups that might otherwise remain uncovered • Boys versus girls, special needs students, socio-economic status, etc • Report highlights three key RCTs conducted in India on computer assisted learning programs • Results suggest the importance of individualizing ICT solutions and investigating how ICT solutions interact with the learning environment and subsequently impact student outcomes

7. Challenges to Evaluation in Developing Countries • Absence of standard framework for evaluation of ICT programs in educational settings • Limited local capacity for evaluation • Limited funding/scarce resources • Lack of demand and ownership of evaluation

8. Lessons Learned & Recommendations • Impacts of ICT are varied in literature • Positive, negative and insignificant impacts on learning outcomes are reported • Research suggests type of impact often contingent upon other variables, including teacher pedagogy, infrastructure, learning environment, etc • Additionally, design of many ICT interventions is based upon perceptions and anecdotal evidence

9. Lessons Learned & Recommendations (cont’d) • Design longitudinal studies • Look for differential rates of impact across subgroups • Evaluations conducted utilizing mixed methods could be highly useful • Develop international standards for evaluation of ICT’s impact on learning • Explore other research areas, such as effects of learning styles, teacher training, and others