The implementation of national educational standards in austria
Download
1 / 45

The Implementation of National Educational Standards in Austria - PowerPoint PPT Presentation


  • 81 Views
  • Uploaded on

The Implementation of National Educational Standards in Austria. Michael Bruneforth BIFIE - Austrian Federal Institute for Education Research, Innovation and Development of the Austrian School System EURASIAN EDUCATIONAL DIALOGUE 17 .-19. April, Yaroslavl.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' The Implementation of National Educational Standards in Austria' - hans


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
The implementation of national educational standards in austria

The Implementation of National Educational Standards in Austria

Michael Bruneforth

BIFIE - Austrian Federal Institute for Education Research, Innovation and Development of the Austrian School System

EURASIAN EDUCATIONAL DIALOGUE

17.-19. April, Yaroslavl


  • BIFIE - Austrian Federal Institute for Education Research, Innovation and Development of the Austrian School System

  • Areas of work

    • International Studies (PISA, TIMSS, PIRLS, TALIS)

    • Implementation of national education standards

    • Formative and summative Evaluation of standards

    • Central examination for upper secondary education preparing for tertiary education

    • National Education Reports


Spotlight on austrian education
Spotlight on Austrian education Innovation and Development of the Austrian School System

  • Very rich country (4th highest GDP per Capita in EU)

  • High expenditure per student (140% of EU Average) in primary and secondary education

  • Low student teacher ratio and class size

  • But PISA shock in 2003 and 2009:

    • Very low PISA results in reading (470, similar to Russia 459), no EU country scores less

    • Strong relation between education outcomes and family background


Changing in paradigm in educational governance
Changing Innovation and Development of the Austrian School System in paradigm in educationalgovernance

  • Since more than a decade there is a change in governance of schools and education in German speaking countries from input oriented control to evidence based output oriented governance and extension of school autonomy

  • In Austria at the centre is shift towards competence oriented teaching based on the definition of nationwide compulsory education standards and a mechanism of formative and summative evaluation


Legal basis and conditions
Legal Basis and Conditions Innovation and Development of the Austrian School System

  • From 2004: pilotphasewithselectedpilotschools

  • 2008: alterationofthelawofteaching in schoolstoallowforintroductionofeducationalstandards

  • 2009: educationalstandardsbecomecompulsoryfor German, Mathand English

  • 2009 (grade 8) and 2010 (grade 4): baseline-tests in samples of about 10.000 students each

  • From 2012 onwards yearly assessment of national standards (one subject per year)


Annex of the legal regulation
Annex of the legal regulation Innovation and Development of the Austrian School System

  • Competency Models anddescriptorsareformulatedasCan-Do-Statements

  • Competency Models anddescriptorsarelegallyestablished

  • Educational Standards canbeinterpretedasmandatoryandagreed upon socialobjectivesofinstruction

  • Educational Standards arederivedfrom national curriculaandclearlydefinelearningoutcome.


Areas covered
Areas Innovation and Development of the Austrian School Systemcovered

  • German (language of instruction) – grades 4 & 8

  • Listening & Speaking

  • Reading

  • Writing texts

  • Spelling/Grammar

Math – grades 4 & 8

4 content dimensions

4 process dimensions

  • English, grade 8

  • Listening

  • Reading

  • Speaking

  • Writing

English Assessment:

Common European Framework of Reference for Languages (CEF)


Competencies are
Competencies are … Innovation and Development of the Austrian School System

  • „fashionable termwith a vaguemeaning“ (Weinert 2001)

  • „cognitiveprerequisitestocopingwith a specificrangeofsituations“ (Klieme, Hartig & Rauch, 2008)

  • „cognitiveabilitiesandskillsavailableto/acquiredbyindividualsthatenablethemtosolvespecificproblems, aswellastherelated motivational, volitional andsocialreadinessandskillstoutilizethesolutionssuccessfullyandresponsibly in variable situations“ (Weinert, 2002)

  •  BIFIE use: subject (domain) specificcompetencymodels, focusing on cognitiveskillsandproficiencies


Competency m odels
Competency Innovation and Development of the Austrian School SystemModels

  • Models ofcompetencestructures

    • Dimensionsofcompetence: E.g. TIMSS, grade 8: 4 contentdomains (number, algebra, geometry, dataandchance); 3 cognitivedomains (knowing, applying, reasoning)

  • Models ofcompetencelevels

    • Describespecificskillsandproficienciesusuallyshownbyindividualslocatedatspecificpointsoftheabilitycontinuum

    • Usedtoprovide a criterion-referencedinterpretationofmeasurementresults


Competency m odels1
Competency Innovation and Development of the Austrian School SystemModels

  • Models ofcompetencestructures

  • Models ofcompetencelevels

  • Development models

    • Models describingtheacquisitionofskillsandproficiencies

    • May belinkedtoneurological/psychologicaltheoriesofdevelopment

    • Only a fewexist

    • Wouldbegreattohave a theoreticalfoundationfortests in different grades! (IKM – BIST)


Example mathematics grade 8
Example Innovation and Development of the Austrian School Systemmathematics, grade 8


M8 competency model
M8 – Innovation and Development of the Austrian School SystemCompetency Model


M8 competency m odel
M8 – Innovation and Development of the Austrian School SystemCompetencyModel

Complexity

Competency (H3, I2, K2)

Content

Process

The 4x4 gridleadsto 16 nodes, usingcomplexityas a 3rd dimension = 48 nodes

Not veryhandyandthetheoreticalconceptofcomplexitydoes not lenditselftoassessment


Example competency model m8
Example Innovation and Development of the Austrian School System:Competency Model M8

  • P: „demonstrating, modelling “ C: „numbersandmeasures“

  • Formulatedascan-do-statements: The studentsareableto

    • transformgivenarithmeticaldatainto (another) mathematicalrepresentation. In doing so, directuseofbasicskillsisnecessary

    • transformgivenarithmeticaldatainto (another) mathematicalrepresentation. In doing so, connectionstoothermathematicalcontents (terms, definitions, representations) oractivitieshavetobeestablished

    • makeandevaluatestatementsabouttheappropriatenessas well asweaknessesandstrengthsof different mathematicalrepresentations (models) ofarithmeticaldata

P1/C1/K1

P1/C1/K2

P1/C1/K3


Educational standards the austrian version
Educational Standards – Innovation and Development of the Austrian School SystemThe Austrian Version

  • Three areas of activities

    • Implementation of standards at school and capacity building for teacher: Support of teachers and school heads in implementation

    • Formative evaluation to support teachers diagnostic: On-line evaluation system to be used by teachers

    • Census to monitor and summative evaluation at system, sub-system, school and classroom level


Educational standards the austrian version1
Educational Standards – Innovation and Development of the Austrian School SystemThe Austrian Version

  • School-specificdevelopmentofinstruction

    • Mandatoryeducationalstandards

    • Census-survey ofall studentsandschools

    • Feedback to all students, teachers, schools

    • Feedback moderation („Rückmeldemoderation“)

    • Stimulation ofqualitydevelopmentandsupportbyMoE

    • Strong linkageofmonitoringandpracticalschoolandinstructionaldevelopment

    • Strengtheningofschools, schoolmanagementand regional qualitydevelopment


Main functions of standards according to the legal regulation
Main functions of standards according to the legal regulation

  • Output orientation: Sustainable output orientation in planning and implementation of instruction

  • Promotion („Förderfunktion“): Diagnosis and specific individual remediation and challenge

  • Evaluation: reliable information about the output of instruction, schools and education system


Implementation of standards
Implementation of Standards regulation

  • Supporting materials for instruction

    • Praxishandbücher

    • Themenhefte

    • Aufgabenbeispiele, Interaktive Beispiele

    • Best-Practice-Beispiele

  • Integration with teacher training institutions

    • In-service teacher training (partially mandatory)

    • Pre-Service teacher trainin

    • Research and development work by members of teacher training institutions


Diagnosis tool for informal monitoring of competencies
Diagnosis tool for Informal Monitoring of Competencies regulation

  • Teacher administered on-line system

  • Grade 3 and 7, one year before Assessment

  • All Areas of standards

  • Only teachers themselves have access to results


Assessment of educational standards
Assessment regulationof Educational Standards

  • Legal assessment cycle: three (school) years

  • In each cycle, the whole target population has to be assessed in all prescribed competency subdomains

    • Grade 4: German, Mathematics

    • Grade 8: German, Mathematics, English

  • Students, teachers, schools and regional education authorities must get feedback about the results (to be used for quality improvement)


Domain orientated test design
Domain- regulationorientated Test Design


  • Pilot testing: about 10% of population, up to 9000 Students regulation

  • Main test: Census up to 83000 students

    • Speaking domain only with sample for system monitoring

  • 10% of census with external administration and items for next census to ensure linking over time

  • Example: Test session Mathematics Grade 4


  • This picture regulationshows about 1/10 of the test materials


Item development
Item Development regulation

  • Items are developed to measure sub-competencies or domains, eg. Content = numbers and units and process = demonstrating

  • Dimension „complexity“ is not used as such, since it can‘t be measured empirically

  • We have 16 nodes, upon which an item should fall. We strive for unidimensional items, i.e. items that can be classified as belonging to one specific node only


Item development1
Item Development regulation

  • Items aredevelopedtomeasure sub-competenciesordomains, eg. Content = numbersandunitsandprocess = demonstrating

  • Dimension „complexity“ is not usedas such, sinceitcan‘tbemeasuredempirically

  • Wehave 16 nodes, upon which an item should fall. Westrivefor unidimensional items, i.e. itemsthatcanbeclassifiedasbelongingtoonespecificnodeonly


Item development2
Item Development regulation


Item development3
Item Development regulation

  • Test blueprintspecifiedtheitemstobedeveloped:

    • Mathematicalnode

    • Item format: Multiple Choice, Constructedresponse, …

    • Estimateddifficultyof item

  • Further requirements:

    • adequatforagegroup,

    • nobiasedtowardsgender, race, religion, also takingintoaccount rural/urban differences.

    • does not requireotherknowledgethanmathscontents.

    • Items shouldbe uni-dimensional


Validation
Validation regulation

  • In ordertosecurevalidityofthetestitems, theywere …

    • writtenbyschoolanduniversityteachers

    • reviewedbyteachers (peerreviewandoutsiderreview)

    • trialled on a sample of grade 8 students – each item was answeredbyat least 200 students

    • item propertieswerereviewedby a teamofpsychometriciansandschoolteachers

    • itemswithlessthandesirableorsurprisingpropertieswereretiredand not used in themaintest


Test design
Test Design regulation

  • Multiple matrixsampling

  • Two-stepapproachforthetest design

    • Allocationof „empty“ blockstoformsthatsatifyseveral global prerequisitesforthe design:

      • positionalandcontextualbalance, item security (S90 vs. S10), longitudinal link (BL and BIST-UE 2015)

    • Allocationofitemstoblocksthatsatisfyseverallocalprerequisites:

      • Balanceddifficulty, format, representativeforthecompetenceconstruct, dependency (enemyitems), testmotivation

      • Test information, especiallyalongcutoff-scores


Scaling
Scaling regulation

  • UsingRasch model

  • Allowsto express difficultiesofitemsandstudentsability on onecommonscale

  • Allowstomatchtestcyclesover time

  • Allowsforcomplextest design


Standard setting
Standard Setting regulation

  • Standard settinggoal: meaningfullydescribedlevels

  • criterion-referenced

  • Phases:

4 levels in Austria in mostdomains

0: standards not reached

1: partiallyreachedstandards

2: reachedstandards

3: exceededstandards

Easy tounderstandfor „laymen“


P roficiency l evel descriptions m8
P regulationroficiencyLevel Descriptions M8

3 – standardsexceeded

Studentspossess fundamental knowledgeandskills in all partsofthemathematicscurriculumandadvancedknowledgestructures, whichexceedtherequirementsoflevel 2, specificallymorepronouncedabilitiesofabstractionandhigherproficiency in combiningpartsofknowledge, methodsorrules. Theyareabletoapplytheseindependently in novelsituations in a flexible way.

2 – standards reached

Students posessfundamental knowledge and skills in all parts of the mathematics curriculum and can use these in a flexible way. They are able to find and apply problem solving strategies, to describe and reason an approach. They are able to handle verbal, graphical and formal representations of mathematical facts in a flexible way and can apply these appropriately. Theay are able to extract relevant information from a differently represented facts (e.g. texts, data material, graphics) and can interpret them in the respective context. They are able to relate their mathematical knowledge and can check, evaluate and/or reason mathematical statements.

1 – standardspartiallyreached

Studentspossess fundamental knowledgeandskills in all partsofthemathematicscurriculumandcanmasterreproductivetasksandcarry out routineprocedures.


Standard setting1
Standard Setting regulation

  • „ A standard setting method is a mechanism for helping panelists translate their intentions to the score scale.“ (Reckase, 2006)

  • Goal: Match predetermined descriptors of competency levels to an empiric scale defined by test items

    • Determinecutscoresbetweenlevelsofcompetence

    • Bifie invited a groupofstakeholderstosetthecutscores: teachers(loweranduppersecondary), teachersofdidacticsofmath; representativesofparentsassociations; representativesoftheMoE; representativesofchamberofcommerce; bifie staff

    • Methodused: Item descriptormatchingmethod


( regulationModified)

Item descriptormatchingmethod

Item-descriptormatches

Difficult Item

Level 3

Level 3

Level 2

Level 2

Level 1

Level 1

Difficulty-ordereditems

Easy Item

Ordered Item Booklet

Cut Score 2

Cut Score 1

Freunberger & Yanagida, in press.


Standard setting2
Standard Setting regulation

  • Standard settingismoreaboutfinding a societalconsensusthanfinding a constant

  • The goalshouldbeclearbeforetestdevelopmentetcstarts


Feedback regulationand ReportsInformation aboutresultsof Educational Standards

SCHOOL LEVEL

School report(forschoolmanagement) per school, online

4600 schools

about 3200 primaryand1400 secondary I

School andinstructiondevelopment

10.000 classesabout 6000 primaryand4000 secondary I

Feedback toteachersper teacher, per class/group; online

Feedback tostudents

per participant, online

170.000 students

about 85.000 in grade 4 and 8 each


Feedback regulationand ReportsInformation aboutresultsof Educational Standards

SYSTEM LEVEL

National report (Minister for Education) aggregated

MoE

System development

9 provinces

Provincereports (pres& vice-) foreachprocinceand grade

Supervisoryreports (federalsuperv. authoritiesfor PS & LS/AS) aggregated on fed. stateandsupervised-school type

9 provincesPrimary/4

Lower sec/8, academic sec/8

School development

99 schooldistricts

15 Cities – 84 districts

Supervisoryreports (districtsuperv. authoritiesfor PS and LS) per districtand grade


School report overview of content
School regulationreport: overviewofcontent

  • Information aboutschoolresults:

  • compared toAUT

  • compared tothedefinedgoals(in theregulation)

  • compared tootherschoolswithsimilarbasicconditions („fair comparison“)

  • Areas ofconflict:

  • Educational Standards applyto all students – andthereforeto all schools

  • Conditionsofschoolsvarywidely


Feedback

The regulationschooltakescentrestage

Feedback


Feedback example school report competency levels
Feedback – regulationexample (schoolreport)„Competencylevels“


Feedback example teacher feedback
Feedback – regulationexample (teacherfeedback)


Feedback example school report
Feedback – regulationexample (schoolreport)


Support system for schools
Support regulationsystemforschools


Outlook
Outlook regulation

  • Bringing IKM (grade(s) before BIST) and BIST assessments closer together. Using items in both assessments could lead towards a better understanding of learning

  • Backwash from doing the assessments towards research in the areas of item development, standard setting, feedback … through evaluation of results

  • Outcomes of the assessments should not only allow us to give feedback to schools, teachers and students but also to improve the competence models we use for the assessment

  • Prepare for computer based testing

  • Open databases for external researchers and improve system level monitoring


Thank you for your attention

Thank regulationyouforyourattention!

MichaeIBruneforth

[email protected]

Bundesinstitut BIFIE

Salzburg | Zentrum für Bildungsmonitoring & Bildungsstandards

www.bifie.at


ad