PISA 2009 – New Approaches to Assessing Reading Literacy

1. PISA 2009 – New Approaches to Assessing Reading Literacy Gerry Shiel Rachel Perkins Educational Research Centre

2. Overview • Status of reading literacy in PISA 2009 • Overview of the framework • Engagement and metacognition • Demonstration of electronic reading items • Scaling reading literacy in PISA 2009 and reporting on trends

3. Reading Literacy as a Major Assessment Domain in PISA 2009

4. Definition of Reading Literacy in 2009 • No change since 2000 • Reading literacy is understanding, using and reflecting on written texts, in order to achieve one’s goals, to develop one’s knowledge and potential, and to participate in society. • Written texts is understood to mean a range of texts – handwritten, printed and electronic

5. Dimensions of Reading Literacy • Situation • Text (medium, format, rhetorical structure) • Aspect (strategy or cognitive approach)

6. Reading Situations • Refers to uses for which the author constructed a text

7. Distribution of Reading Tasks by Situation

8. Texts • Medium – printed vs. electronic (hypertext) • Text format – continuous, non-continuous, mixed and multiple • Rhetorical structure – description, narration, exposition, argumentation and instruction

9. Text - Medium • Electronic texts – hypertext – a text or texts with access structures that make possible and encourage non-linear reading. • Electronic texts exist in a number of environments, including web environment, desk top and e-mail. • PISA 2009 excludes electronic texts outside computer environments (e.g., text messages)

10. Continuous and Non-continuous Texts – Print Medium

11. Continuous and Non-continuous Texts – Electronic Medium

12. Distribution of Continuous and Non-continuous Texts (Print)

13. Reading Processes (Aspects)

14. Three Broad Aspect Categories • retrieving information; • interpreting texts; • reflection and evaluation.

15. Relationship between task, text and aspects in the print medium

16. Relationship between task, text and aspects in the electronic medium

17. PISA 2009 – Distributions of Items in Printed Text, by Aspect

18. Print Texts Multiple choice Constructed Response Electronic Texts Multiple choice Constructed response Closed response (e.g., Drag and drop; copy and paste; select by clicking) Open response (e.g., generating and entering a search term) PISA 2009 Reading Response Formats

19. Reading Engagement • In PISA 2000, there was a stronger association between engagement in reading and reading performance than between SES and reading in all participating countries • It was argued that reading engagement could reduce the gap in achievement between subgroups • New definition of engagement, covering print and electronic texts, proposed for 2009.

20. Components of Reading Engagement • interest in reading– disposition to read literature and information text for enjoyment and the satisfaction of curiosity; • perceived autonomy– perceived control and self-direction of one’s reading activities, choices, and behaviours; • efficacy– belief in one’s capacity to read well in the present and future; • social interaction– social goals for reading and interactive competence; • mastery goal– aim of reading to comprehend, understand, and explain text; • reading practices– behavioural engagement referring to the amount and types of reading activities.

21. Measuring Engagement in Reading • Concept of unified engagement • Equal emphasis on reading motivation (interest, autonomy, efficacy, social interaction and mastery goal) and reading practices • Contextualisation of engagement across a range of types of reading material (fiction, information and social communication).

22. Educational Context of Engagement • Relevance: My teacher tells us how the reading relates to our experience. • Autonomy support: This teacher lets me choose what to read. • Self-efficacy support: My teacher shows me how to read the material for this course. • Social interaction in literacy: In class, I talk with other students about what the text means. • Mastery goal structure: The teacher explains how the topics fit together. • Opportunity to read: In this class the teacher helps us with a lot of challenging reading.

23. Metacognition and Reading • Metacognition in reading refers to the awareness of and ability to use a variety of appropriate strategies when processing texts in a goal oriented manner. • It also involves an understanding of the differential processing demands associated with various kinds of tasks and how to apply this understanding. • Metacognition has both a significant correlation with reading proficiency, and is amenable to teaching and learning

24. Measuring Metacognition

25. Assessing Metacognition

26. Pilot Assessment of Electronic Reading in the PISA 2009 Field Trial (2008) • 22 countries participated in international option for reading of electronic texts • Australia, Austria, Belgium, Canada, Chile, Denmark, France, Germany, Hong-Kong, Hungary, Iceland, Ireland, Japan, Korea, Macao-China, New Zealand, Norway, Poland, Russian Federation, Spain, Sweden and United Kingdom. • Decision on participation in electronic reading component of main study to be made in October 2008.

27. Relationship between navigation and text processing for ERA tasks

28. Reading Literacy Scales in PISA 2009 • Reading trend scale (based on 28 ‘common’ items) • PISA 2009 reading literacy scale (print only) • PISA 2009 electronic reading literacy scale • PISA 2009 reading literacy scale (print and electronic combined)

29. Reading Literacy Sub-scales in PISA 2009 (Print) • Access and retrieve • Integrate and interpret • Reflect and evaluate • Continuous • Non-continuous • (Items based on mixed and multiple texts not to be scaled – too few items)

30. Reading Literacy Sub-scales in PISA 2009 (Electronic) • Access and retrieve • Integrate and interpret • Reflect and evaluate • Complex (processes)