Evaluating the Impact of Educational Technology

1. Evaluating the Impact of Educational Technology Erno Lehtinen University of Turku Finland European Association for Research on Learning and Instruction (EARLI)

2. New challenges Managing complex, ill-defined problems Managing rapid change Continuous surpassing of existing expertise Learning socially shared and distributed expertise Effective use of cultural artifacts Traditional schooling Dealing with well-defined pieces of knowledge Conveying established traditions of knowledge Fulfilling external standards Cultivating individual minds Emphasis of pure mental operations New challenges of learning - are the schools prepared?

3. Self-evident benefits of information and communication technology • the power of multimedia • interactivity • possibilities for multiple representations • possibilities to simulate real phenomena • world wide access to information • tools for synchronous and asynchronous communication • How real are these (technology-driven) benefits?

4. Desires for using ICT to improve learning: Theory-driven arguments • Technology as inspiration for advancement of the theories and models of learning • applications of constructivist epistemology • new notions of collaborative learning • communities for progressive inquiry and knowledge building • authentic and anchored learning • from knowledge acquisition to participation metaphor of learning

5. Promises of ICT: results of the experimental research • Reviews and meta-analyses of more than 1500 experiments show that: • ICT students learn more • ICT supports social interaction • ICT improves student motivation but also causes motivational problems • the quality of learning depends on the type of ICT application

6. Promises of ICT: Experiences in the innovative projects • Higher order learning in traditional school tasks • complex problem solving • meta-cognitive skills • Results going beyond the aims of traditional education: • communities of distributed expertise • new media skills • active participation

7. ICT and the quality of learning in ordinary classrooms • A few very good spontaneous examples • Very much superficial quasi-activities • In many technologically well equipped schools ICT has only occasionally been applied in teaching and learning • Due to negative side effects many teachers do not use Internet

8. How to explain this contradiction between different results?

9. Critical reflection of the experimental evidence • Publication bias: only articles with positive effects are submitted and accepted • “Unfair” experimental designs • Anatomy of the experimental evidence • the size of experimental groups • the duration of experiments

10. Analysing the impact of ICT: methodological challenges • Why it is so difficult to measure the impact of ICT in education • Moving from ”traditional” to ICT environments: difficulties to define the ”independent variable” (systemic change) • In innovative use of ICT also the learning aims change

11. New methods are needed: Example CSCL • Collaboration around and through technology • Analysis of document structures • Content analysis of CSCL documents • Analysis of participation activity • Analysis of participation structures

12. Critical reflection of innovative projects • Marketing of scientific ideas and applications • Theoretical concepts have a tendency to turn into ideological slogans • Individual differences have been neglected • New ICT based environments change the motivational and social interpretations • virtual open learning environments are obviously increasing achievement differences between learners • Unrealistic investments in terms of preparation and support of the teaching

13. A scaling up problem: why ”best practices” do not transfer • Experimental evidence is partly misleading • selected teachers, major investments etc. • Shortage of realistic models • Shortage of ”killer applications”; ICT based practices able to solve the hard nuts of teaching and learning • Shortage of models for long term everyday use • Direct transfer of practices is not possible – a transformation an localisation is needed

14. Preparing students for information society and the responsibility of the educational systems • New literacy for all students – avoiding new inequality • Innovative use of new technologies to improve access to education and quality of learning – not fashionable application of virtual environments at the expense of quality

15. Stronger teacher involvement in the development of the ICT tools an practices: • Teachers must have ownership (eg. learning objects approach) • Innovative use of ICT is not a private business of isolated teachers but it emphasises the community nature of the schools • The use of ICT in education should not be fully commercialised but a coordinated public-private partnership is needed

16. The duration of the experiment • Effect sizes in experiments of different duration

19. Next

20. Back Back