pisa mathematics assessment apec tokyo feb 2010 l.
Skip this Video
Loading SlideShow in 5 Seconds..
PISA Mathematics Assessment APEC Tokyo Feb 2010 PowerPoint Presentation
Download Presentation
PISA Mathematics Assessment APEC Tokyo Feb 2010

Loading in 2 Seconds...

play fullscreen
1 / 59

PISA Mathematics Assessment APEC Tokyo Feb 2010 - PowerPoint PPT Presentation

  • Uploaded on

PISA Mathematics Assessment APEC Tokyo Feb 2010. Kaye Stacey University of Melbourne, Australia. Data and images in this presentation are from OECD websites and official publications and from ACER publications on PISA in Australia.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'PISA Mathematics Assessment APEC Tokyo Feb 2010' - zenia

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
pisa mathematics assessment apec tokyo feb 2010

PISA Mathematics AssessmentAPEC Tokyo Feb 2010

Kaye Stacey

University of Melbourne, Australia

Data and images in this presentation are from OECD websites and official publications and from ACER publications on PISA in Australia.

The views expressed here are those of the author and do not represent the OECD or associates.

  • What is PISA and what does it test?
  • What is mathematical literacy?
  • A small sample of results
    • Country comparisons
    • Levels of proficiency
    • Performance of subgroups
    • Social gradient
  • Possibilities for Computer-based Assessment of Mathematics
pisa programme for international student assessment
PISA: Programme for International Student Assessment
  • Test years 2000, 2003, 2006, 2009, 2012, ..
  • 15 year olds
  • assesses “the knowledge and skills that students have acquired at school and their ability to use them in everyday tasks and challenges”
    • reading literacy
    • scientific literacy
    • mathematical literacy
  • statistically rigorous, to ensure that the results are as meaningful as possible, measuring
    • student performance
    • data on the student, family, school and system factors
key features of pisa from oecd
Key features of PISA (from OECD)
  • policy orientation
    • major aim is informing educational policy and practice
    • aim to significantly improve understanding of the outcomes of education
  • concept of “literacy” (discussed later)
  • relevance to lifelong learning
    • motivation to learn,
    • attitudes towards learning
    • learning strategies;
  • surveys to explore features associated with educational success
    • characteristics of students and schools
    • trends monitored every 3 years;
  • breadth
    • by 2006, around 90% of the world economy
    • nearly 400 000 students
  • Recent studies tracking young people in the years after age 15 show PISA measures knowledge and skills relevant to a life success
participating countries
Participating countries
  • PISA 2000: 43
  • PISA 2003: 41
  • PISA 2006: 57
    • nearly 400 000 students
  • PISA 2009: 66
    • PISA Plus : +9
  • PISA 2012: about 90?
asia pacific 2009
Asia-Pacific 2009
  • China
    • Hong Kong
    • Macao
    • Shanghai
  • Indonesia
  • New Zealand
  • Thailand
  • Japan
  • Korea
  • Australia
  • Singapore
  • Chinese Taipei
  • Chile
  • Peru
  • Panama
  • Argentina
  • Mexico
  • USA
  • Canada
  • +9 PISA Plus
    • Malaysia, …
survey methods
Survey methods
  • Schools randomly selected by PISA (usually 150+)
  • Random sample of 35+ students per school
    • between age 15 yrs 3 mths & 16 yrs 2 mths
  • Strict sampling criteria to be included in reports
    • e.g. Netherlands in 2000 below required so not in trend data
  • Some countries oversample for their own purposes
  • Each student does 2 hour test and 30 min questionnaire
  • Items are in rotating booklets (about 13)
    • results of individual students not available/meaningful
timss trends in international mathematics and science study
TIMSS: Trends in International Mathematics and Science Study
  • Independent body, not OECD
  • Tests every 4 years, since 1994/5
  • Grade based sample (years 4, 8, 12)
  • Tests randomly sampled intact classes
    • hence teacher survey makes sense
  • Aims to test achievement of curriculum goals
    • Careful and extensive curriculum comparisons
  • More Asian countries have participated in TIMSS
    • Singapore’s high results very famous
schedule of performance measures



Schedule of performance measures
  • Additional cognitive assessments:
    • 2003: problem solving
    • 2006: computer based assessment of science
    • 2009: electronic reading
    • 2012: problem solving
    • 2012: computer based assessment of mathematics
questionnaire components




Questionnaire components
  • School context and attitudes
    • themselves and their homes
    • attitudes to learning
  • School questionnaire, optional teacher questionnaires
  • TRENDS since 2000 (nearly)



statistically better than australia maths
Statistically better than Australia - Maths
  • PISA 2003
    • Hong-Kong-China, Finland, Korea, Netherlands, Liechtenstein, Japan, Canada
  • PISA 2006
    • Chinese Taipei, Finland, Hong-Kong-China, Korea, Netherlands, Switzerland, Canada, Macao-China
  • Movements: 5 stay above Australia, 2 drop to Australia’s group, 1 rises from Australia’s group, 2 new entrants
what percent of students in lowest performance bands
What percent of students in lowest performance bands?

OECD: PISA Proficiency Level 2 “a baseline level of proficiency at which students begin to demonstrate skills that enable them to actively use mathematics”

what is mathematical literacy16
What is mathematical literacy?
  • PISA assesses “the knowledge and skills that students have acquired at school and their ability to use them in everyday tasks and challenges”
  • Reflect recognition that globalisation and computerisation are changing labour markets and societies, and that a different set of skills is needed
  • US evidence:
    • greatest decline in jobs over the past decade has not been in manual labour, but in routine cognitive tasks – those that can easily be done at less cost by computer (Levy & Murnane, 2006).
mathematical literacy
Mathematical literacy
  • 2003/2006 "an individual’s capacity to identify and understand the role that mathematics plays in the world, to make well-founded judgments, and to engage in mathematics in ways that meet the needs of that individual’s current and future life as a constructive, concerned and reflective citizen." 
  • Strong links to other concepts
    • Mathematical modelling (in PISA framework)
    • Numeracy (but certainly not just “basic skills”)
    • Quantitative literacy
sample domain items
Sample domain items
  • PISA: “Take the test”
    • Reading – page 13
    • Science – page 187
    • Mathematics – page 97
  • Questionnaires
    • Also download from “MyPISA”


% teachers unaware

  • Written text followed by questions
  • This one: answer as graph
a test across countries needs
A test across countries needs…
  • cultural breadth and balance in tests
    • bullying, Ministry of Education, ….
    • not a question of intersection of school curricula around the world (TIMSS)
    • school curricula (e.g. reading graphs) influence success rate and hence usefulness of items for constructing a measure
  • high quality in translations
    • two “masters” for each item (English and French)
    • translations from both masters compared
    • back translation etc
    • some informal language will not translate
      • corner vs vertex

English: corner or vertex

French: vertex

growing up
Growing Up
  • 6.1 A height of female in 1980 (given increase since then)
  • 6.2 Explain how graph shows growth rate of girls slows down after 12 yrs of age
  • 6.3 When are females taller than males of same age?
growing up 6 2 growth rate girls
Growing Up (6.2 – growth rate girls)
  • Classification
    • Scientific; Change and Relationships; Connections
    • Difficulty 574 PISA score points.
  • The question requires students to:
    • Analyse different growth curves
    • Evaluate characteristics of data set, represented by graph.
    • Note and interpret different slopes along graphs.
    • Reason and communicate the results of this process, within explicit models of growth.
  • OECD average 45%
  • Most successful countries: Netherlands (77%), Finland (68%), Belgium (64%), Canada (64%)
  • Large omission rates: Austria (44%) and Greece (43%).
scoring for 6 2 growth rate for girls
Scoring for 6.2 (growth rate for girls)
  • Score (Code) 1 : Response refers to “change” of gradient of female graph, explicitly or implicitly.
  • Code 11:Refers to reduced steepness, using daily-life language.
    • It does no longer go up, it straightens out; The curve levels off; It is more flat after 12; The line of the girls’ starts to even out and the boys’ line just gets bigger; It straightens out and the boys’ graph keeps rising.
  • Code 12: Refers to reduced steepness ,using mathematical language.
    • You can see the gradient is less; The rate of change of the graph decreases from 12 years on; (uses words like “gradient”, “slope”, or “rate of change”)
  • Code 13: Compares actual growth (comparison can be implicit).
    • From 10 to 12 the growth is about 15 cm, but from 12 to 20 the growth is only about 17 cm; The average growth rate from 10 to 12 is about 7.5 cm per year, but about 2 cm per year from 12 to 20 years.
  • Score (Code) 0
  • Code 01: Student indicates that female height drops below male height, but does NOT mention the steepness of the female graph or a comparison of the female growth rate before and after 12 years.
    • The female line drops below the male line.
  • Code 02: Other incorrect responses. For example, the response does not refer to the characteristics of the graph, as the question clearly asks about how the GRAPH shows….
    • Girls mature early; Girls don’t grow much after 12.
growing up 6 2 growth rate girls29
Growing Up (6.2 – growth rate girls)
  • Answer type:
    • Daily life language: over 70% of correct answers in 24 countries
    • Mathematical language : 56% of correct answers in Korea
    • Comparing actual growth: common in Austria (34%); Mexico (26%), Greece (23%), France and Turkey (19%).
  • Common errors
    • Most common error: not referring to graph e.g. “girls don’t grow much after 12”.
    • Around 40% of incorrect answers in France, Korea and Poland refer to graph, only to show the female height drops below the male height.(concept of gradient??)
heatbeat m537 graph
Heatbeat (M537) - graph

Heartrate = 200 – age

Heartrate = 208 – 0.7*age

  • Newspaper statement in text alerts student to phenomenon
  • Question 46.1: from which age does the recommendation increase?
  • Question 46.2: write formula for most effective training heartrate (80% of max)

Possible solution to Q46.1

  • Classification
    • Quantity; Occupational; Connections
    • Difficulty rating: 499 PISA score points (Mean is set to 500)
  • The question requires students to:
    • Develop a strategy to connect two pieces of information for each component: how many available, how many needed per set
    • Use logical reasoning to link that analysis across the components to produce the required solution.
    • Communicate the mathematical answer as a real-world solution (not 5.5 bookshelves)
  • Most successful:
    • Finland and Hong Kong-China (74%),
    • Korea, the Czech Republic, Belgium and Denmark (72%).
  • OECD average: 61% correct, 29% of students attempted & incorrect and 10% did not attempt.
2012 cbam computer based test of mathematics
2012 CBAM: computer based test of mathematics
  • New opportunities for presentation of items to measure same material better
    • More attractive presentation
    • Better presentation (e.g. animation)
    • Better response formats (e.g. move an animation?)
  • Able to test some aspects of doing mathematics by computer and so extend notion of literacy to better match world
    • What are these aspects?

Mathematical Skills in the Workplace

Final Report to the Science, Technology and Mathematics Council, UK, 2002

C. Hoyles, A. Wolf, S. Molyneux-Hodgson & P. Kent

  • Recommendation 1 . Raising Visibility and Awareness of the Importance of Mathematical Literacy in the Workplace
  • The focus should be:
    • The nature of mathematical literacy: that it is anchored in real data, in the context of a particular workplace.
    • That maths used in the workplace has economic benefits in the market-place.
    • That mathematics may be present quite implicitly in jobs and tasks, which are not obviously mathematical.
    • Many employees, regardless of their level of employment, are required to use mathematical literacy.
    • That IT and mathematical skills are interdependent.
what it maths skills in ml41
What IT-maths skills in ML?

Use 3-D views

See GoogleSketchUp demo

what it maths skills in ml43
What IT-maths skills in ML?

Write formula?

What tools?

(graphics) calculator?

computer – what software?

mobile phone?

Enter data?

how well can you use a spreadsheet to plot a graph

MyPISA query

How well can you…..use a spreadsheet to plot a graph?
  • “Do well by myself”
    • OECD total: 42.09%
    • Macao: 28.49%
    • Thailand 25.75%
    • Australia 58.42%
studying subgroups of students
Studying subgroups of students
  • Gender
  • Ethnic and home background
    • Migration
    • Language background
    • Indigenous students
  • Social gradient
mathematical literacy by socio economic background australia
Mathematical literacy by socio-economic background (Australia)
  • Graphic shows:
    • mean and confidence interval (white)
    • 5th, 10th, 25th, 75th, 90th, 95th percentiles
social gradient science 2006 sci literacy score against social index
Social gradient (Science 2006)(Sci-literacy score against social index)

From: Thomson & de Bortoli Exploring Scientific Literacy: How Australia Measures Up

position of line strength of relationship gradient of line curvature length of line
Position of line; strength of relationship; gradient of line; curvature; length of line

From: Thomson & de Bortoli Exploring Scientific Literacy: How Australia Measures Up

interpreting the social gradient
Interpreting the social gradient
  • Strength of association (variance explained)
    • Hong Kong 6.9% < Australia 11.3% < OECD 14.5%
    • Asian countries tend to be low
  • Gradient
    • Australia 43 > OECD 40 > Hong Kong 26
  • Length
    • Australia has less variation in social index than OECD
    • US has higher top level than OECD; same bottom
  • Position
    • Australia does better than the OECD average
  • Curvature
    • low and high groups do not differ on relationship (cf NZ, Canada)
  • Australia – maths slightly less affected by social index than reading and science – relatively more affected by school
  • NOTE: For almost all countries, the effect of school average ESCS outweighs effect of student’s own ESCS.
equally prepared for life oecd
“Equally prepared for life?” OECD
  • Reading PISA2000
    • females significantly outscored males in all countries
  • Mathematics PISA2003
    • males often outscored females
  • Science PISA2006
    • no significant difference between males and females overall, but some difference in patterns of strengths
    • no significant differences in attitudes to school science
    • marked differences in expectations of science career
australia s indigenous girls and boys
Australia’s indigenous girls and boys

Graphic shows:

mean and confidence interval (white)

5th, 10th, 25th, 75th, 90th, 95th percentiles

information about pisa
Information about PISA

http:// www.oecd.org


PISA: “Take-the-test”

MyPISA: public database and analysis

thoughts and discussion points
Thoughts and discussion points
  • Immense amount of information and excellent reports
    • Available to you through public databases (except secure ‘trend items’)
    • But no study answers all questions
      • in-depth understanding of thinking e.g. of algebra
      • What has caused the results (e.g. Finland’s success; Asia’s success)
  • An international study operates under severe constraints
    • Create items that work internationally to measure target construct validly
    • Anomalies can reveal differences
      • e.g. Korea-Australia average, 103 vs 104
  • Concepts of mathematical and scientific literacy
    • Major contribution to educational aspiration in many countries
    • Still developing
      • How does computer-based mathematics affect definition of math literacy?
      • Has mathematical literacy changed from 2003 to 2012?
  • Aim is to find the school systems, schools, teaching and societies that best prepare all future citizens for living productive and satisfying lives
thank you

Thank you


http:// www.oecd.org


Stacey, K. & Stephens, M. (2008). Performance of Australian School Students in International Studies in Mathematics. Schooling Issues Digest 2008/1. Canberra http://www.dest.gov.au/sectors/school_education/publications_resources/schooling_issues_digest/.