Potential impact of PISA

1. Organisation for Economic Cooperation and Development (OECD) Potential impact of PISA Michael Davidson OECD/Directorate for Education

2. What is PISA? A three-yearly assessment that… … examines the performance of 15-year-olds in key subject areas as well as a wider range of educational outcomes • Including students attitudes to learning, their beliefs about themselves, and their learning strategies … collects contextual data from students schools, parents and systems to identify policy levers • Contextual data explain 71% of the performance variation among schools internationally Coverage • Representative samples of between 3,500 and 50,000 15-year-old students drawn in each country • Most federal countries also draw regional samples

3. Results from PISA 2006 • Scientific knowledge and use of that knowledge in… … identifying scientific issues, … explaining scientific phenomena, … using scientific evidence

4. PISA provides key benchmarks for the quality of education systems 1. Overall performance of education systems 2. Equity in the distribution of learning opportunities • Measured by the impact students’ and schools’ socio-economic background has on performance… … not merely by the distribution of learning outcomes 3. Consistency of performance standards across schools 4. Gender differences

5. Mean science scores The centre line is the mean The boundaries indicate the 95% confidence limits There is only a significant difference if there is no overlap of scores OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Table 2.1c

6. Is it all innate ability?Variation in student performance Performance variationacross schools 20 OECD (2007), Learning for tomorrow’s world: First results from PISA 2006, Table 4.1a

7. Variation in student performance PISA 2006: Science Competencies for Tomorrow’s World, Figure 4.1a.

8. Variation in student performance Variation of performance within schools Variation of performance between schools PISA 2006: Science Competencies for Tomorrow’s World, Figure 4.1a.

9. High science performance Average performanceof 15-year-olds in science – extrapolate and apply Equity … 18 countries perform below this line Low science performance

11. Impact of selected student and school factors on school performance (after accounting for all other factors in the model) Schools with greater autonomy (resources) School activities to promote science learning One additional hour of self-study or homework One additional hour of science learning at school School results posted publicly Academically selective schools but no system-wide effect Schools practising ability grouping for all subjects 20 Effect after accounting for the socio-economic background of students, schools and countries OECD (2007), PISA 2006 – Science Competencies from Tomorrow’s World, Table 6.1a

12. Evaluating the policy impact of PISA

13. External evaluation of the policy impact of PISA • Objective • To what degree does PISA impact policies, meet expectations and priorities of the participants? • Evaluation criteria • Relevance • Effectiveness • Sustainability • Unexpected / Unplanned impacts

14. External evaluation of the policy impact of PISA • Team of three external evaluators • David Hopkins, Dianne Pennock, Jo Ritzen • Two-step procedure • Questionnaire to stakeholders in the country • In-depth evaluation of 5 case studies • Stakeholder groups • Policy makers at the national/federal and local level as well as members of the PISA Governing Board • School principals, representatives of teacher and parent organisations • Academics and researchers • Representatives of the business community and the media • Evaluation report • Conclusions and recommendations based on the empirical results obtained

15. Strategy for an evaluation of the policy impact of PISA • Effectiveness – Specific questions • To what extent are the participating countries and other stakeholders aware of the output results of PISA? • To what extent are these output results being used as input into the policy-making processes of clients and other stakeholders? • How significant is the influence of the PISA output results used in the policy-making processes? • What is the role of PISA in the context of national assessment strategies, i.e. to what extent have countries used PISA to add to, or complement, their national assessments or data? • To what extent has PISA provided added value to the national assessment strategies? • What impact has PISA had on policy coherence in participating countries?

16. Outcomes – Relevance of PISA • Policy makers are considered the most significant stakeholder group in relation to PISA and its results. • Policy makers are also considered as the major group responsible for implementing policies in light of PISA. • The various stakeholder groups assume relatively low levels of responsibility for the PISA results in their countries.

17. Outcomes – Relevance of PISA • Overall, PISA is seen as a useful tool for identifying and addressing nationally relevant themes and problems • In many countries PISA is used to monitor and evaluate the quality and equity of the education system • Impact of PISA varies from country to country: • In countries with unsatisfactory outcomes often direct policy impact • In high-performing systems: PISA as an evaluation mechanism

18. Important aspects of PISA

19. Outcomes – Effectiveness and sustainability of PISA • In all countries, PISA is seen as an important instrument for policy making with a high level of credibility • Impact of PISA on policy formation increasing from PISA 2000 to PISA 2006 • Influence of PISA greater at the national / federal level than at the local level or in school practices and classroom instruction • Policy makers, academics and researchers and the media most aware of PISA

20. Addressing policy needs

21. Contributing to change

23. Outcomes – Effectiveness and sustainability of PISA • According to respondents, the influence of PISA could be increased by: • a better coordinated and strategic approach for the dissemination of PISA results; • further support for various stakeholder groups in interpreting PISA results and in designing strategies for improvement in light of PISA; • In some countries, clearer linkage of PISA results to national or federal assessment strategies; and, • greater utilisation of the PISA results by participating countries.

24. Unexpected / unplanned outcomes • Great public interest and debate • PISA as an instrument to justify reform • “Culture of blame” • Increased interest in empirical educational research

25. Next step

26. Thank you very much for your attention! http://www.pisa.oecd.org Michael.davidson@oecd.org