PISA for Development
This presentation is the property of its rightful owner.
Sponsored Links
1 / 30

PISA for Development Technical Strand 1: Contextual questionnaires Overview of PISA instruments (constructs, indices and variables) EDU/DCD PowerPoint PPT Presentation


  • 116 Views
  • Uploaded on
  • Presentation posted in: General

PISA for Development Technical Strand 1: Contextual questionnaires Overview of PISA instruments (constructs, indices and variables) EDU/DCD. PISA for Development Contextual questionnaires. Purpose of PISA contextual questionnaires How are they used? How they are developed

Download Presentation

PISA for Development Technical Strand 1: Contextual questionnaires Overview of PISA instruments (constructs, indices and variables) EDU/DCD

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Pisa for development technical strand 1 contextual questionnaires overview of pisa instruments constructs indices and variables edu dcd

PISA for Development

Technical Strand 1:

Contextual questionnaires

Overview of PISA instruments

(constructs, indices and variables)

EDU/DCD


Pisa for development contextual questionnaires

PISA for DevelopmentContextual questionnaires

  • Purpose of PISA contextual questionnaires

  • How are they used?

  • How they are developed

  • What will PISA for Development seek to do?

  • Questions for discussion


Pisa for development purpose of pisa contextual questionnaires

PISA for DevelopmentPurpose of PISA contextual questionnaires

  • Student and school questionnaires are part of the core assessment design of PISA

  • Countries may also opt for additional questionnaires:

    • Parents, Teachers, Health, Time Use, ICT familiarity, Career…

  • Studentquestionnaires: ~ 30 minute* stand-alone document (students respond after test questions)

  • School questionnaires: ~ 30 minute stand-alone document (school authority responds before or on day of testing)

  • * Rotated questionnaires in 2012 to cover additional material


  • Pisa for development purpose of pisa contextual questionnaires1

    PISA for DevelopmentPurpose of PISA contextual questionnaires

    • Contextualise the performance results in reading, mathematics and science*

    • Allow analyses of equity issues across sub-groups and populations

    • Provide indications of possible policy interventions

    • Some information collected is used to scale cognitive assessment data

    • Country-specific questions can be included for additional analysis

    • * Additional domains such as financial literacy and problem-solving


    Pisa for development purpose of pisa contextual questionnaires2

    PISA for DevelopmentPurpose of PISA contextual questionnaires

    • Have been used in 74 countries/economies

    • In more than 40 languages

    • Publicly available

      • PISA 2009 Student Questionnaire

      • PISA 2012 Student Questionnaire

      • PISA 2012 School Questionnaire

      • PISA 2009 School Questionnaire


    Pisa contextual questionnaires how they are used

    PISA Contextual QuestionnairesHow they are used

    This is the PISA Analytic Framework that guides data collection and analysis

    Outputs and Outcomesimpact of learning

    Policy Leversshape educational outcomes

    Antecedentscontextualise or constrained policy

    Individual learner

    Quality and distribution of knowledge & skills

    Individual attitudes, engagement and behaviour

    Socio-economic background of learners

    Quality of instructional delivery

    Student learning, teacher working conditions

    Instructional settings

    Teaching, learning practices and classroom climate

    Levels of an education system

    The learning environment at school

    Community and school characteristics

    Output and performance of institutions

    Schools (other institutions)

    Social & economic outcomes of education

    National educ., social and economic context

    Country or system

    Structures, resource allocationand policies


    Pisa contextual questionnaires how they are used1

    PISA Contextual QuestionnairesHow they are used

    This is how the PISA Analytic Framework is operationalised for data collection and analysis

    • Main data collection

    • Student Questionnaire, School Questionnaires and System Level*

    Variables included in instruments:

    *Options include questionnaires geared for Teachers, Parents, Health, Time Use, ICT familiarity, Career


    Pisa contextual questionnaires system level data collection

    PISA Contextual QuestionnairesSystem-level data collection

    • Standards and procedures for data collection in PISA

      • Questionnaires for non-OECD economies

      • Panel of experts to complete some data collection

    These are examples of the types of information collected at the system level:

    • Ratio: students / teaching staff

    • Spending /student

    • Instruction time (by age)

    • Teachers’ salaries (by education, experience)

    • Evaluation & accountability

    • Educational stratification

    • Decision-making at different levels

    • School choice

    • Changes in education policy

    • Others (…)

    Sample of relevant areas to address in developing contexts

    • Within-country disparities

    • Quality, supply and distribution of educational materials and resources

    • (Extra-)tuition

    • Parental support

    • Contractual arrangements of school teaching staff


    Pisa contextual questionnaires school questionnaires

    PISA Contextual QuestionnairesSchool questionnaires

    • School resources, policies and practices

    This is the type of information collected through the school questionnaires, that can then be cross-referenced with other sources

    These are the constructs that will be reviewed as part of the technical work of PISA for Development

    • Grades served

    • Public-Private

    • Funding sources

    • Community type

    • Enrollment (boys, girls)

    • Average class size

    • Non-native speakers

    • Grouping/tracking

    • Teachers, by qualifications

    • Teacher shortages

    • Physical building limitations

    • Instructional materials

    • Teacher attitudes, expectations, relations

    • Computer/network/web access

    • Absenteeism, disruptions

    • Teacher morale

    • Student morale

    • Teaching environment

    • Time for special programmes

    • Non-native speaker offerings

    • Admissions requirements

    • Assessment types

    • Uses of assessment

    • Assessment of teachers

    • Decision making authority

    • External decision makers


    Pisa contextual questionnaires student questionnaires

    PISA Contextual QuestionnairesStudent questionnaires

    Student attitudes, behaviours and approaches to learning

    These are examples of the types of

    non-cognitive outcomes

    on which PISA collects information

    These are the constructs that will be reviewed as part of the technical work of PISA for Development


    Pisa contextual questionnaires examples of analysis

    PISA Contextual QuestionnairesExamples of analysis

    … How are questionnaire data used for relevant analysis and comparative policy insights?

    (a few examples from PISA results)


    Pisa for development technical strand 1 contextual questionnaires overview of pisa instruments constructs indices and variables edu dcd

    • Examples of analysis

    • Range of performance among non-OECD countries

    PISA Results for different years


    Pisa for development technical strand 1 contextual questionnaires overview of pisa instruments constructs indices and variables edu dcd

    • Mexico

    Student performance

    PISA Index of socio-economic background

    Disadvantage

    Advantage

    • Examples of analysisSchool performance and socio-economic background

    • PISA 2009 Results

    School performance and students’ socio-economic background within schools

    Private school

    Public school in rural area

    Public school in urban area

    Student performance and schools’ socio-economic background

    Schoolswith a similar socio-economicprofile


    Pisa for development technical strand 1 contextual questionnaires overview of pisa instruments constructs indices and variables edu dcd

    • Kyrgyzstan

    Student performance

    PISA Index of socio-economic background

    Disadvantage

    Advantage

    • Examples of analysisSchool performance and socio-economic background

    • PISA 2009 Results

    School performance and students’ socio-economic background within schools

    Private school

    Public school in rural area

    Public school in urban area

    Student performance and schools’ socio-economic background

    OECD average


    Pisa for development technical strand 1 contextual questionnaires overview of pisa instruments constructs indices and variables edu dcd

    • Examples of analysisPolicy findings from contextual information

    Policy briefs on these issues can be found in the PISA in Focus series


    Pisa for development technical strand 1 contextual questionnaires overview of pisa instruments constructs indices and variables edu dcd

    Based on the analysis of cognitive outcomes and contextual information

    • Examples of analysisPolicy insights using contextual information

    Lessons from PISA from successful education systems (equitable and improvers)


    Pisa contextual questionnaires how they are developed

    PISA Contextual QuestionnairesHow they are developed

    • ‘Crowd sourcing’ and collaboration

      • PISA draws together leading expertise and institutions from participating countries to develop instruments and methodologies…

        …guided by governments on the basis of shared policy interests

    • Cross-national relevance and transferability of policy experiences

      • Emphasis on validity across cultures, languages and systems

      • Frameworks built on well-structured conceptual understandingof assessment areas and contextual factors

    • Continuous review and updating of assessment frameworks (cycles)

      • Guided by participating countries and various expert groups formed by leading international experts in different areas related PISA

      • Technical Advisory Group, Reading Expert Group, Mathematics EG, Science EG, Questionnaire EG, and international contractors


    Pisa contextual questionnaires how they are developed1

    PISA Contextual QuestionnairesHow they are developed

    • PISA 2012 Questionnaires

    • Linkages between cognitive and non-cognitive outcomes

    Student questionnaire: Rotated design (total of 65 questions – 25 core and 3 blocks of 27 questions)

    School questionnaire: 33 questions (5 additional questions on financial education at school)

    Parental questionnaire: 25 questions

    ICT questionnaire: 12 questions

    Education careers: 14 questions


    Pisa contextual questionnaires how they are developed2

    PISA Contextual QuestionnairesHow they are developed

    • Purpose of the PISA Field Trial (for Main Study)

    Selection of cognitive test items to be included in the PISA Main Study instruments (e.g. countryXcountryDIF, gender DIF)

    Validation of psychometric equivalence of translated instruments (and adaptations from source versions)  issues addressed

    Identify constructs from contextual questionnaires (indices and variables) that are associated with performance for inclusion in the Main Study

    Determine cross-national validity of the questions in the questionnaire

    Technical standards ensure that data from participating countries are internationally comparable


    Pisa contextual questionnaires what will pisa for development seek to do

    PISA Contextual QuestionnairesWhat will PISA for Development seek to do?

    • Adapt and enhance contextual instruments to “better fit” diverse contexts found in developing countries while maintaining the comparability with the main PISA scales and international results

      • Adapt existing constructs (and indices and variables)

      • Identify and introduce new constructs (indices and variables)

      • Confirm and validate enhancements through field trials and main data collection in participating countries


    Pisa contextual questionnaires what will pisa for development seek to do1

    PISA Contextual QuestionnairesWhat will PISA for Development seek to do?

    • Examples of variables and indices

    Student Questionnaire

    Section 1: ABOUT YOU

    • Section 2: YOUR FAMILY AND YOUR HOME

    • Section 3: YOUR READING ACTIVITIES

    • Section 4: LEARNING TIME

    • Section 5: YOUR SCHOOL

    • Section 6: YOUR SCHOOL LESSONS AND SUBJECTS

    • Section 7: YOUR STRATEGIES IN READING AND UNDERSTANDING TEXTS

    • Section 8: YOUR VIEWS ON <BROAD SCIENCE>

    • Section 9: CAREERS AND <BROAD SCIENCE>

    • Section 10: YOUR MATHEMATICS EXPERIENCES


    Pisa contextual questionnaires student questionnaires1

    PISA Contextual QuestionnairesStudent questionnaires

    • Examples of variables and indices

    • Student Questionnaire

      SECTION 2: YOUR FAMILY AND YOUR HOME

      2009-ST08: Family structure

      2009-ST09: Mother’s main job (ESCS, HISEI, BMMJ)

      2009-ST10: Mother’s education (ESCS, HISCED, PARED, MISCED)

      2009-ST11: Mother’s qualifications (ESCS, HISCED, PARED, MISCED)

      2009-ST12: Mother’s employment status (ESCS, HISEI, BMMJ)

      2009-ST13: Father’s main job (ESCS, HISEI, BMFJ)

      2009-ST14: Father’s education (ESCS, HISCED, PARED, FISCED)

      2009-ST15: Father’s qualifications (ESCS, HISCED, PARED, FISCED)

      2009-ST16: Father’s employment status (ESCS, HISEI, BFMJ)

      2009-ST17: Country of birth for student and parents (IMMIG)

      2009-ST18: Age at arrival in country of test

      2009-ST19: Language spoken at home

      2009-ST20: Home resources (ESCS, HOMEPOS, WEALTH, HEDRES, CULTPOS)

      2009-ST21: Family wealth (ESCS, HOMEPOS, WEALTH)

      2009-ST22: Books in home (ESCS, HOMEPOS)


    Pisa contextual questionnaires student questionnaires examples of indices

    PISA Contextual QuestionnairesStudent questionnaires – examples of indices

    PISA index of Economic, Social and Cultural Status

    (ESCS)

    • Comparability across countries (not a country-specific income indicator)

    WEALTH

    CULTPOSS

    Highest educational level of parents

    PARED

    HEDRES

    # of books in home

    Home possessions

    HOMEPOS

    Highest occupational status of parents

    HISEI


    Pisa contextual questionnaires student questionnaires examples of indices1

    PISA Contextual QuestionnairesStudent questionnaires – examples of indices

    PISA index of Economic, Social and Cultural Status

    (ESCS)

    Index of family wealth

    a room of their own, a link to the Internet, a dishwasher (treated as a country-specific item), a DVD player, and three other country-specific items (some items in ST20), and their responses on the number of cellular phones, televisions, computers, cars and the rooms with

    a bath or shower (ST21).

    Index of cultural possessions

    classic literature, books of poetry and works of art (ST20)

    WEALTH

    Index of home educational resources

    For example, availability of a desk, a quiet place to study, a computer that students can use for schoolwork, educational software, books to help students, technical reference books and dictionary (some items in ST20)

    CULTPOSS

    HEDRES

    # of books in home


    Pisa contextual questionnaires student questionnaires examples of indices2

    PISA Contextual QuestionnairesStudent questionnaires – examples of indices

    Self-efficacy in science and mathematics

    • 2006-ST17: Self-efficacy in science (SCIEEFF)

    • 2003-ST31: Self-efficacy in mathematics (MATHEFF)

     Stays the same in PISA 2012

    Relationship between constructs, indices and variables (questions)


    Pisa contextual questionnaires student questionnaires examples of indices3

    PISA Contextual QuestionnairesStudent questionnaires – examples of indices

    Self-efficacy in science

    • 2006-ST17: Self-efficacy in science (SCIEEFF)


    Pisa contextual questionnaires student questionnaires examples of indices4

    PISA Contextual QuestionnairesStudent questionnaires – examples of indices

    Instrumental motivation to learn science

    • 2006-ST35: Instrumental motivation to learn science (INSTSCIE)


    Pisa for development how will we do this

    PISA for DevelopmentHow will we do this?

    • Identify constructs in need of adaptation and important constructs that are missing

    • Obtain input and advice from experts in the field (grounded in developing-country contexts)

    • Obtain input and guidance from participating countries

    • Draw on existing work and empirical evidence

    • Test the constructs (indices and variables) to ensure comparability with PISA scales


    Pisa for development technical strand 1 contextual questionnaires overview of pisa instruments constructs indices and variables edu dcd

    PISA for Development

    Technical Strand 1:

    Contextual questionnaires


    Pisa for development questions for discussion

    PISA for DevelopmentQuestions for discussion

    • Are the intentions and rationale of this strand of work clear and understood?

    • Which components of the PISA questionnaires appear to demonstrate the greatest need of adaptation?

    • Where do you see the largest hurdle to overcome?

    • What existing studies and evidence should contribute to this strand of work?


  • Login