Programme in International Student Assessment (PISA)

1. Programme in International Student Assessment (PISA) Jenny Bradshaw

2. PISA in brief • Over half a million students… • representing 28 million 15-year-olds in 65 countries/economies … took an internationally agreed 2-hour test… • Goes beyond testing whether students can reproduce what they were taught… … to assess students’ capacity to extrapolate from what they know and creatively apply their knowledge in novel situations • Mathematics, reading, science, problem-solving, financial literacy • Total of 390 minutes of assessment material … and responded to questions on… • their personal background, their schools and their engagement with learning and school • Parents, principals and system leaders provided data on… • school policies, practices, resources and institutional factors that help explain performance differences .

3. PISA in brief • A shared learning tool for all involved • ‘Crowd sourcing’ and collaboration • PISA draws together leading expertise and institutions from participating countries to develop instruments and methodologies… … guided by governments on the basis of shared policy interests • Cross-national relevance and transferability of policy experiences • Emphasis on validity across cultures, languages and systems • Frameworks built on well-structured conceptual understandingof academic disciplines and contextual factors • Triangulation across different stakeholder perspectives • Systematic integration of insights from students, parents, school principals and system-leaders • Advanced methods with different grain sizes • A range of methods to adequately measure constructs with different grain sizes to serve different decision-making needs • Productive feedback, at appropriate levels of detail, to fuel improvement at every level of the system .

4. The structure of the PISA assessment

5. Questionnaires • PISA2012: • Student and school questionnaires • Options: • ICT questionnaire • Educational career questionnaire • Parent questionnaire

6. PISA 2012 Sample Question 1 Climbing Mount Fuji Mount Fuji is a famous dormant volcano in Japan. Mount Fuji is only open to the public for climbing from 1 July to 27 August each year. About 200 000 people climb Mount Fuji during this time. • On average, about how many people climb Mount Fuji each day? • 340 • 710 • 3400 • 7100 • 7400

7. PISA 2012 Sample Question 1 Percent of 15-year-olds who scored Level 2 or Above

8. PISA 2012 Sample Question 4 Revolving Door A revolving door includes three wings which rotate within a circular-shaped space. The inside diameter of this space is 2 metres(200 centimetres). The three door wings divide the space into three equal sectors. The plan below shows the door wings in three different positions viewed from the top. The two door openings (the dotted arcs in the diagram) are the same size. If these openings are too wide the revolving wings cannot provide a sealed space and air could then flow freely between the entrance and the exit, causing unwanted heat loss or gain. This is shown in the diagram opposite. What is the maximum arc length in centimetres (cm) that each door opening can have, so that air never flows freely between the entrance and the exit? Maximum arc length: ____________ cm

9. PISA 2012 Sample Question 4 Percent of 15-year-olds who scored Level 6 or Above

10. High mathematics performance … Shanghai-China performs above this line (613) Average performanceof 15-year-olds in Mathematics Fig I.2.13 … 12 countries perform below this line Low mathematics performance US

11. Change in performance between PISA 2003 and 2012 Fig I.2.18 PISA 2003 performance below the OECD average PISA 2003 performance above the OECD average Improving performance Deteriorating performance

12. Of the 65 countries 45 improved at least in one subject

13. What accounts for variation?

14. Variation in mathematics performance between systems, schools and students  Fig IV.1.2 Variation in mathematics performance attributable to differences: Between systems Between schools Between students OECD countries All participating countries and economies

15. In most countries and economies, the disciplinary climate in schools improved between 2003 and 2012 Fig IV.5.13 Disciplinary climate improved Disciplinary climate declined

16. Schools with more autonomy perform better than schools with less autonomy in systems with more accountability arrangements Fig IV.1.16 School autonomy for curriculum and assessment x system's level of posting achievement data publicly Score points

17. Performance and equity: a tradeoff ? 17 Fig I.2.13 High mathematics performance High impact of socio-economic condition on student outcomes Low impact of socio-economic condition on outcomes Low mean performance

18. Percentage of resilient students Fig II.2.4 A resilient student is situatedin the bottom quarter of the PISA index of economic, social and cultural status (ESCS) in the country of assessment and performs in the top quarter of students among all countries, after accounting for socio-economic status. Socio-economically disadvantaged students not only score lower in mathematics, they also report lower levels of engagement, drive, motivation and self-beliefs. Resilient students break this link and share many characteristics of advantaged high-achievers. More than 40% resilient Between 20%-40% of resilient students Less than 20%

19. Gender differences in mathematics performance Fig I.2.25 Boys perform better than girls Girls perform better than boys

20. Resources make a difference… …but only up to a point

21. Spending per student from the age of 6 to 15 and mathematics performance in PISA 2012 Fig IV.1.8 Cumulative expenditure per student less than USD 50 000 Cumulative expenditure per student USD 50 000 or more

22. Educational resources are more problematic in disadvantaged schools, also in public schools in most countries Fig IV.3.8 Disadvantaged and public schools reported better educational resources Advantaged and private schools reported better educational resources

23. Among high-income countries high-performers pay teachers more Fig IV.1.10 Per capita GDP less than USD 20 000 Per capita GDP over USD 20 000 In 33 countries schools where a higher share of principals reported that teacher shortages hinder learning tend to show lower performance Among low-income countries a host of other resources are the principal barriers

24. Countries with better performance in mathematics tend to allocate educational resources more equitably Fig IV.1.11 Adjusted by per capita GDP 30% of the variation in math performance across OECD countries is explained by the degree of similarity of educational resources between advantaged and disadvantaged schools OECD countries tend to allocate at least an equal, if not a larger, number of teachers per student to disadvantaged schools; but disadvantaged schools tend to have great difficulty in attracting qualified teachers. Greater equity Less equity

25. Which systems have improved… … and how have they done it?

26. Country case study- Brazil Brazil’s average scores in PISA assessments • PISA 2000 – lowest ranked country; over 50% of students scored below Level 1 in reading. • Since then Brazil has improved in all PISA subjects

27. Some actions taken by Brazil • Significant increase in funding • raising spending on educational institutions from 4% of GDP in 2000 to 5.2% of GDP by 2009 • allocating more of those resources to raising teachers’ salaries • spending more equitably and more efficiently • Federal funds are now directed towards the poorest of the country’s 26 states, providing schools in those states with resources comparable to those available to schools in wealthier states. • Accountabilitysystems development • PISA-benchmarked Basic Education Development Index (IDEB), created in 2005

28. Video series on Strong Performers and Successful Reformers in Education http://www.pearsonfoundation.org/oecd

29. What’s next? PISA 2015

30. PISA 2015 • Main subject: Science • Number of participants : 72 • Field trials in 2014 • Main survey 2015 • Results released in December 2016

31. PISA 2015 in Kosovo • An opportunity to engage with educational policies and educational colleagues worldwide • For maximum benefit, transparency and public dialogue is essential • Preparation before the results are released, and early planning of dissemination, are important

32. Engagement of all is important: • Policy-makers • Teachers and Schools • Students and Parents • Media • Research community

33. Thank you ! Find out more about PISA at www.pisa.oecd.org National and international publications The complete micro-level database Email: Jenny.Bradshaw@oecd.org