PISA for teachers Interpreting and using information from an international reading assessment in the classroom

1. PISA for teachers Interpreting and using information from an international reading assessment in the classroom ACER Research Conference Perth, 16-18 August, 2009 Juliette Mendelovits Dara Searle Tom Lumley

2. Outline • Why do countries participate in international studies of educational achievement? • Australia’s performance in PISA reading, 2000 to 2006 • What do our results tell us about the way Australian 15-year-old read? • What can teachers do to improve reading performance? • New directions in PISA reading 2009: Electronic reading: • How has PISA conceptualised electronic reading? • How can this conceptualisation be used at the classroom level?

3. Why do countries participate in international studies of educational achievement? • To collect national-level data when there is no other mechanism for doing so • To compare themselves with other countries • To be involved in – and help to shape - new developments in education

4. Australia’s performance in PISA reading, 2000 to 2006

5. In PISA 2000 Australia Ranked 4th out of the 31 participating countries

6. Australia’s result for PISA Reading 2000 • Mean= 528 • OECD average =500 • Standard deviation =102 • OECD average = 100

7. Australia’s results for PISA Reading 2000 Subgroup performance • Gender • Difference between girls and boys (34 points) was about the same as the OECD average (32 points). • Indigenous students • Performed about 80 PISA scale score points below non-indigenous students – the equivalent of about two years of schooling • High performing students • The top performing 5% of students in Australia performed as well as any other countries’ top 5%

8. PISAsurvey cycle

9. PISA Reading 2000 – 2006 • Between PISA 2000 and PISA 2006: • 7 countries showed an improvement in their results • 15 countries showed no change in their reading literacy performance • 14 countries showed a decline in their results … • including Australia

10. Australia’s result for PISA Reading 2006 • Mean= 513 • OECD average =492 • Standard deviation =94 • OECD average = 99

11. PISA Reading in Australia2000 – 2006 • Australia’s mean score in reading declined by 15 scale score points – the equivalent of about 4 months of schooling • Results declined for all states and territories • Maybe it was the sample? • But Australia showed no significant change in performance in Mathematical literacy between PISA 2003 (when Mathematics was the major domain) and PISA 2006

12. PISA Reading in Australia 2000 – 2006Subgroup performance in reading • Gender • Both girls and boys declined in performance (the gap remained about the same) • Indigenous students • Indigenous performance declined from 2000 to 2006, but not significantly • http://www.acer.edu.au/ozpisa/reports.html • High performing students • There was a significant decline in performance of the top 25% of Australian students • already observable in 2003, worsened in 2006 • top 5% declined the most

13. What happened and what can we do about it? • The decline is NOT attributable to: • the sample (compare maths) • differences between states and territories • a gender difference problem • a decline in indigenous performance • Maybe it can be attributed to: • disproportionate focus on improvement of the least proficient (high performers showed the biggest decline) • the way reading is taught in Australia …

14. PISA’s definition of reading Reading literacy is understanding, using, reflecting on and engaging with written texts, in order to achieve one’s goals, to develop one’s knowledge and potential, and to participate in society.

15. PISA uses five task characteristics in the construction of all reading literacy tasks • Context • Medium • Environments • Text format – text type • Aspect

16. Text format Continuous Non-continuous Aspects Access and retrieve Integrate and interpret Reflect and evaluate Reported Subscales for PISA Reading 2000

17. Text format • Continuous • Information presented in sentences and paragraphs • Argument • Description • Exposition • Instruction • Narration • Non-continuous • Information presented in formats other than prose, for example • tables, • graphs, • diagrams • maps • forms

18. In PISA 2000, Australia performed better on non-continuous texts than continuous texts Our weakest performance, in terms of text type, was on narrative and argumentative texts.

19. Aspects • Access and retrieve • Integrate and interpret • Reflect and evaluate

20. In PISA 2000, Australia performed best on access and retrieve tasks, and worst on reflect and evaluate tasks

21. Comparing Australia’s results with those of other English speaking countries in PISA 2000

22. Aspect means for English-speaking countries

23. Sample reflect and evaluate item, narrative textThe Motorcycle Have you ever woken up feeling that something was wrong? It was a day like that for me. I sat up in bed. A little later I opened the curtains. It was terrible weather – the rain was pouring down. Then I looked down into the yard. Yes! There it was – the motorcycle. It was just as wrecked as last night. And my leg was starting to hurt. Why does the writer begin the story with a question? Because the writer wants to know the answer. To involve the reader in the story. Because the question is hard to answer. To remind the reader that this kind of experience is rare.

24. Sample reflect and evaluate item, narrative text Macondo Dazzled by so many and such marvellous inventions, the people of Macondo did not know where their amazement began. They stayed up all night looking at the pale electric bulbs fed by the plant that Aureliano Triste had brought back when the train made its second trip, and it took time and effort for them to grow accustomed to its obsessive toom-toom. They became indignant over the living images that the prosperous merchant Don Bruno Crespi projected in the theatre with the lion-head ticket windows, for a character who had died and was buried in one film, and for whose misfortune tears of affliction had been shed, would reappear alive and transformed into an Arab in the next one. The audience, who paid two centavos apiece to share the difficulties of the actors, would not tolerate that outlandish fraud and they broke up the seats. The mayor, at the urging of Don Bruno Crespi, explained by means of a proclamation that the cinema was a machine of illusions that did not merit the emotional outburst of the audience. With that discouraging explanation many felt that they had been the victims of some new and showy gypsy business and they decided not to return to the movies, considering that they already had too many troubles of their own to weep over the acted-out misfortunes of imaginary beings. Do you agree with the final judgement of the people of Macondo about the value of the movies? Explain your answer by comparing your attitude to the movies with theirs.

25. Example answers for Macondo

26. Sample task from an argument text Democracy in Athens One purpose of the speech in part B was to honour soldiers who fell in the first year of the Peloponnesian War. What was ANOTHER purpose of this speech? PART A Thucydides was a historian and military man who lived in the fifth century BC, during the Classical Greek period. He was born in Athens. During the Peloponnesian War (431 BC to 404 BC) between Athens and Sparta he was in command of a fleet whose mission was to protect the city of Amphipolis in Thrace. … PART B Thucydides attributes to Pericles (fifth century BC), the Athenian ruler, the following speech in honour of the soldiers who fell in the first year of the Peloponnesian War. Our system of government does not copy the laws of neighbouring states; we are rather a pattern to others than imitators ourselves. Our system is called democracy, since its administration depends on the many instead of the few. Our laws afford equal rights to all in their private affairs, whereas the prestige in public life depends on merit rather than on social class. …

27. Add democracy in Athens?

28. Based on information from PISA, what can teachers do to improve reading performance of Australian students? • Reconsider approaches to reflective and evaluative reading; • Change the emphasis of what is done with continuous texts, in particular narratives and argument texts; and • Make particular efforts to challenge the most able students.

29. Why do countries participate in international studies of educational achievement? • To collect national-level data when there is no other mechanism for doing so • To compare themselves with other countries • To be involved in – and help to shape - new developments in education

30. New directions in PISA reading: Electronic reading:

31. Why introduce an Electronic Reading Assessment (ERA)? • As of 2007, almost one and a half billion people were reading on line • Between 2000 and 2007, Internet use doubled in OECD countries, and more than trebled worldwide

32. Electronic reading is not ICT literacy Reading literacy is understanding, using, reflecting on and engaging with written texts, in order to achieve one’s goals, to develop one’s knowledge and potential, and to participate in society. (PISA definition of reading literacy) Electronic reading is not just reading print text on a computer screen

33. What distinguishes electronic reading from print reading? • Different kinds of texts (more multiple texts, more bits and pieces, more opportunity to interact with the text) • More onus on the reader to evaluate the text • More onus on the reader to select and construct the text because • There’s usually no pre-determined order in which a text should be read • It is a more abstract and unknown navigation space

34. Analogies … Page of a go

35. In PISA, electronic reading is conceptualised in terms of two components • Text processing • Navigation

36. Relationship between navigation and text processing for ERA tasks

37. ERA Sample Items http://erasq.acer.edu.au

38. Relationship between navigation and text processing for ERA tasks E022Q09: Write a reply to Mischa. In your reply, answer her question… E022Q08: Which suggestion does Doctor Nauckunaite make? E022Q01: Who wrote the first reply… E022Q04: Which writer would disagree…

39. Relationship between navigation and text processing for ERA tasks E022Q09: Write a reply to Mischa. In your reply, answer her question… E022Q08: Which suggestion does Doctor Nauckunaite make? E022Q01: Who wrote the first reply… E022Q04: Which writer would disagree…

40. Based on information from PISA, what can teachers do to improve electronic reading performance? • 1) Different kinds of texts • Teachers could check that students are exposed to a range of text forms from the digital medium • 2) More onus on the reader to evaluate the text • Teachers could refer to the framework descriptions and the items that reflect them as models of critical reading in the electronic medium. • 3) More onus on the reader to select and construct the text • Teachers could inspect this range of tasks to help construct a sequence of lessons on classifying and mastering different navigation techniques ALONG WITH the thinking (reflection and evaluation) that’s necessary to find information efficiently

41. An assessment system like PISA can help teachers to think about • The kinds of texts that students are taught • The kinds of approaches to reading (PISA’s ‘aspects’) that can be covered • Ways of attending to the full distribution of student achievement

42. The PISA Proficiency Scale Reading Literacy Level 5 Evaluating information and building hypotheses Drawing on specialised knowledge Accommodating concepts contrary to expectations 10% 22% Australian Average 29% OECD Average 22% Reading Literacy Level 1Recognise main theme on a familiar topic, make simple connections 12% 6% Below Level 1These students have not acquired the skills to use reading for effective participation in society

43. The PISA Proficiency Scale Reading Literacy Level 5 Evaluating information and building hypotheses Drawing on specialised knowledge Accommodating concepts contrary to expectations 10% 22% Australian Average 29% OECD Average 22% Reading Literacy Level 1Recognise main theme on a familiar topic, make simple connections 12% 6% Below Level 1These students have not acquired the skills to use reading for effective participation in society

44. Thank you For more information about PISA go to www.pisa.oecd.org

45. Continuous Non-continuous Mixed Multiple Narration Exposition Description Argumentation Instruction Text formats … further categorised by text types