PIAAC = Programme for the International Assessment of Adult Competencies

2. PIAAC = Programme for the International Assessment of Adult Competencies

3. Through the assessments and background questionnaire, PIAAC situated respondents within a continuum of life-long learning

4. From Whence We Came .

5. PISA • The OECD also coordinates the Programmefor International Student Assessment (PISA) • Every 3 years, starting 2000 • 15 year old students from randomly selected schools worldwide • Assessments in reading, mathematics and science • Students and principals complete background questionnaires looking at family background, approaches to learning and school administration • In some cases, parents also complete a questionnaire

6. What PIAAC Assessed • Literacy • Single scale measuring reading of prose, document, digital and mixed format texts • IALSS reported literacy as two separate domains (prose and document comprehension on two separate scales)

7. What PIAAC Assessed • Numeracy • Essentially the same concept of numeracy as used in IALSS • PIAAC, however, gathered significantly more data for the construction of its numeracy scale

8. What PIAAC Assessed • Problem Solving in a Technology Rich Environment (PS-TRE)

9. PS-TRE • Newly introduced domain • PIAAC broadens the concept of literacy by including digital technology, communication tools and networks to acquire and evaluate information, communicate with others, and perform practical tasks • At a basic level, it represents the intersection between computer literacy skills and the cognitive skills required to solve problems • PIAAC assessed adults capacity to use computer tools (keyboards and mice, file management tools, applications, and graphic interfaces) to assess the capacity of adults to access, process, evaluate and analyze information effectively

10. PIAAC employed adaptive testing – a feature of computer-based testing that adjusts the difficulty of questions as participants take the test • Provides a much more nuanced distinction in scores than were provided by paper-based predecessors

11. If respondents could not (or would not) use the computer based instrument, they were provided with a paper-based instrument that measured literacy and numeracy • Respondents could score at levels 4/5 in literacy and numeracy using the paper-based instrument

12. Main Elements of PIAAC Direct assessment (skills measured by the survey) • Literacy • Numeracy • Problem solving in technology-rich environments Background questionnaire (information about respondents) • Demographic characteristics • Educational attainment and training • Employment status and income • Social and linguistic background Module on skills use (information on how skills are used every day) • cognitive skills (e.g., engagement in reading) • non-cognitive skills (e.g., capacity to work in a team) and communications, organization, and influencing skills • workplace skills (e.g., autonomy over key aspects of work)

13. The Background • 157,000 adults 16-65 in 24 countries • Second round with nine additional countries • Third round proposed for 2014 • Language of assessment was the official language(s) of the countries participating • Sample sizes were determined nationally • Canada chose a larger sample size to ensure PIAAC data could be compared to IALSS

14. PIAAC Sample By Country

15. PIAAC Sample in Canada

16. PIAAC Sample in Canada • Aboriginal status was self-declared, as determined through other Stats Canada reporting • Hours and hours of pan-Canadian conference calls and meetings were required to achieve this level of partipation • Conference calls and meetings continue as we start to work with the data • Confidentiality assured. The Ministry of Education will have access to the full data set and this was communicated to respondents

17. The Results …

18. Computer Based Assessment

19. Computer Based Assessment • At 86%, Saskatchewan had the third highest proportion of the population (in the world) that engaged in computer based survey – after Sweden and the Netherlands. • The next highest proportion in Canada was Nova Scotia at 84%

20. Computer Based Assessment • Japan had a relatively low rate of completion via computer, BUT scored highly in literacy and numeracy • PS-TRE could not be measured by the PIAAC paper-based assessment instrument

21. Saskatchewan is … • At the OECD average in literacy • At the OECD average in PS-TRE • Below the OECD average in numeracy

22. Saskatchewan … • Saskatchewan’s overall performance mirrors Canada’s performance • Matches the OECD average for people at the highest proficiency levels • Has a higher proportion of the population at the lowest proficiency levels in numeracy and PS-TRE compared to the OECD average

23. Saskatchewan … • Had 10% of respondents whose first language was not English and French • Matches the OECD average, but is 13% below the Canadian average • Youth (16-24) is performing below the OECD average in all three domains • In PS-TRE, Saskatchewan youth at levels 2/3 is 38% vs. the OECD average of 51%

24. Nationally, educational achievement for both Aboriginal and non-Aboriginal people affects their overall literacy attainment in similar ways • Saskatchewan’s results do not show the same correlation, and we are starting to work with Manitoba and other pan-Canadian partners to try and understand why the data shows this • The gap between Aboriginal and non-Aboriginal populations in Saskatchewan (at a 26 and 35 score point difference in literacy and numeracy respectively), is larger than in Canada overall at 14 and 22 points.

25. Proficiency Levels - Literacy • Below Level 1 • 3% OECD, 4% Canada, 3% Saskatchewan • Level 1 • 12%, 13%, 14% • Level 2 • 34%, 32%, 33% • Level 3 • 39%, 38%, 39% • Level 4/5 • 12%, 14%, 11%

26. Proficiency Levels - Numeracy • Below Level 1 • 5% OECD, 6% Canada, 6% Saskatchewan • Level 1 • 14%, 17%, 18% • Level 2 • 33%, 32%, 33% • Level 3 • 35%, 33%, 33% • Level 4/5 • 13%, 12%, 11%

27. Proficiency Levels - PS-TRE • Non-respondents • 24% OECD, 19% Canada, 15% Saskatchewan • Below Level 1 • 12%, 15%, 18% • Level 1 • 29%, 30%, 35% • Level 2 • 28%, 29%, 28% • Level 3 • 6%, 7%, 5%

28. So Brett, how does our performance compare to IALSS? • (and, wink, wink, I know we’re doing worse)

29. PIAAC and IALSS • It is not possible to make a direct comparison of the results from IALSS and PIAAC; the scales used are not the same • The data from the prose, document and numeracy measures of IALSS have been rescaled at the national level to fit with the literacy measure used in PIAAC – these can be compared, but are still do not represent a 1 for 1 correlation • Level 3 in PIAAC is not equivalent to Level 3 in IALSS. In IALSS the OECD identified Level 3 as a minimum threshold of the skills needed to manage the demands of work and life. IN PIAAC, Level 3 was not defined the same way.

30. PIAAC and IALSS • The OECD designed the PIAAC survey to help jurisdictions determine the degree to which their skills address their needs. As a result, the OECD proposes that results for each jurisdiction should be assessed in terms of the association between skill levels and other characteristics, such as job, level of education, and income specific to the populations in each jurisdiction. • Does that mean all of our existing benchmark work is now thrown out? No – at this stage, the results from PIAAC have limited bearing on existing benchmarks. Much further analysis and research will be needed before the results can be applied to literacy programs and the measures we all use.

31. Overall, the difference between the performance of people from different backgrounds within jurisdictions is much larger than the difference between jurisdictions themselves • e.g., The gap between Canadians (and Saskatchewanians) at the 95th and 5th percentiles is larger than the gap between OECD countries on average

32. Skills in the labour force Literacy ― Proficiency of population aged 16 to 65 by employment status, OECD average and countries, 2012

33. Skills in the Labour Force • Contrary to what might be expected, Canada and the United States have an almost equal proportion of working people at levels 4/5 and level 1 and below • Italy’s and Korea’s performance better fits expectations

34. Immigrants Literacy — Average scores and score point differences of population aged 16 to 65 by immigrant status, OECD average and countries, 2012

35. Immigrants • Canada and Australia have very diverse populations with a range of educational backgrounds, but immigration status does not factor as highly in determining overall literacy attainment in our two countries as in others

36. The skills of young adults

37. The skills of young adults PS-TRE ― Proficiency of population aged 16 to 65 at Level 2 or 3 by age groups, OECD average and Canada, 2012

38. Youth • Do we need to look at new ways of knowledge transfer between older and younger workers? • How do employment opportunities (primarily p/t vs. primarily f/t affect attainment)? • Other questions will be forthcoming …

39. Where Does Canada Go Next? • A series of thematic reports • Education • Official Language Minority Communities • Labour Market • Social Outcomes • Aboriginal Populations • New Canadians • PIAAC Online

40. Where do WE Go Next?

41. Together • WE can look at building richer programs and services for individuals, families, employees and communities • WE can support individuals in a life long learning continuum that is reflective of their changing circumstances • WE can honour the holistic teachings of the communities we have all worked with

42. www.piaac.ca http://www.oecd.org/site/piaac/ http://piaacdataexplorer.oecd.org/ide/idepiaac/

43. Questions? Brett Waytuck Provincial Librarian / Executive Director Provincial Library & Literacy Office brett.waytuck@gov.sk.ca