Current Trends in Educational Assessment in Hungary: Research and Development

1. BenőCsapóhttp://www.staff.u-szeged.hu/~csapo/ Current Trends in Educational Assessment in Hungary:Research and Development Prague, 26 November 2014 UNIVERSITY OF SZEGED INSTITUTE OF EDUCATION

2. Outline • Current trends in educational assessment • Educational assessment in Hungary • Developing an online diagnostic assessments system • a large-scale project at the University of Szeged • Further plans to develop the educational assessment in Hungary

3. Function of assessment: providing feedback “Measure what is measurable, and make measurable what is not so.”Galileo Galilei “If you can not measure it, you can not improve it.”Kelvin

4. Current trends in educational assessment

5. What is new? (1) New constructs (2) New contexts (3) New instruments

6. (1) Assessment of new constructs “21st Century Skills”

7. (2) Assessments in new contextAssessment for learning • Formative and diagnostic assessment • Embedded in learning and teaching • Based on a better understanding of learning and development • Detailed, frequent • Provides immediate feedback • Helps to determine the directions of intervention

8. (3) New InstrumentsTechnology-Based Assessment • Innovative item formats, improved validity • Improved objectivity and reliability • Improved precision • Automated scoring • Immediate feedback

9. TBA in PISA • 2006, major domain: science • Technology-Based Assessment of Science • international option • 2009, major domain: reading • Electronic Reading Assessment • international option • 2012, major domain: mathematics • Computer-Based Problem Solving • innovative domain • 2015, major domain: science • Computer-based assessment at all domain • Collaborative problem solving as innovative domain

10. USA: Assessment in the “Race to the Top” • “Adopting standards and assessments that prepare students to succeed in college and the workplace and to compete in the global economy” • “Building data systems that measure student growth and success, and inform teachers and principals about how they can improve instruction”

11. Characteristics of the assessment projects in “Race to the Top” • Two grant categories • Comprehensive Assessment Systems • High School Course Assessment • Coalition of states • Two large coalitions received approximately $330 million • Partnership for Assessment of Readiness for College and Careers (PARCC) $170 million • SMARTER Balanced Assessment Consortium (SBAC) $160 million

12. Educational assessment in Hungary

13. History and context: Two research and development communities Assessment centers at Budapest National Educational Institute (-1990) Hungary participated in the first IEA project in the 1970s and 1980s After 1990: Several reorganizations Today: Educational Authority International and national assessment programs Research and development centers at the University of Szeged Test development projects from the 1970s Several research groups established in the 1980s and 1990s Today Research Group on the Development of Competencies (1996-) Hungarian Educational Longitudinal Program Center for Research on Learning and Instruction Developing Diagnostic Assessments Training of educational assessment experts (1991-)

14. Educational Assessments in Hungary PIRLS TIMSS PISA Grade Diagnostic Assessment System Diagnostic Assessments Assessment of Basic Competencies Matura (Abitur)

15. Characteristics of the Assessment of Basic Competences Reading and mathematics are the assessment domains Every student is measured at grade 6, 8 and 10 The data are longitudinally connected Sophisticated value-added models are used School-level results are published (‘soft accountability’)

16. Developing Diagnostic AssessmentsA project implemented by the Center for Research on Learning and Instruction University of SzegedTwo phases: 2009-2011, 2012-2014 edia.hu

17. http://edia.hu English

18. http://edia.hu/?q=en Publications / Books Demo test

19. Online Diagnostic Assessment System • A complex system for supporting learning and teaching • Based on the analysis of learning and developmental processes, and the outcomes of learningSee -> framework development • An online accessible assessment platform See-> eDia • An item bank containing thousands of items in innovative formatsSee -> demo test for item formats • Teacher trainingpreparing the teachers for the utilization of the system • Developing intervention materials (e.g. computer games) for enhancing students’ skills if needed

20. Devising assessment frameworks for reading, mathematics and science for the first six grades

21. How an assessment framework looks like? PISA TIMSS Diagnostic - application - content - content (literacy) - reasoning - reasoning - application - application

22. A three-dimensional model for framework development GENERAL ABILITIES Internal, psychological There may be unexploited potentials here LITERACY External, social expectations Disciplinary content CONTENT KNOWLEDGE – EXPERTISE PISA, three main literacy domains Curriculum-based assessment, 1st and 2nd IEA Science and Mathematics Study

23. Devising assessment frameworks:working methods and products • Broad national and international cooperation • Results published in 2x3 volumes (in English and in Hungarian) • Opening theoretical chapters (international co-authors) • Detailed standards by Hungarian expert groups

27. Result of the Framework Development Three volumes, published both in Hungarian and in English May be downloaded in .pdf

28. The eDia platform • The system runs at the servers of the University of Szeged • Item builder module • Item bank unit • Test delivery module • Data bank • Data analysis module • Feedback module

29. The item development process • Training the item writers (350+) • assessment theory and item writing • content of the assessment (frameworks) • ICT and the eDia platform • Item writing • Reviewingby the group leader • Supervising by the „dimension chief” • Piloting • Determining the item parameters

30. Innovative item formats • Text and voice for the grades 1-3 • Stimulus • text, voice • picture, animation • video • simulation • Response capture • everything what is possible on paper, keyboard, mouse • manipulating and moving objects on the screen, drag and drop • touchscreen solutions

34. Characteristics of the Online Diagnostic Assessment System • Serving ca. 600 000 students (grades 1-6) • Communicating with ca. 20 000 teachers • Making the service availablein ca. 3 000 schools • Managing ca. 10 000 test items • Maintaining and analyzing students’ data • Providing both students and teachers with sophisticated feedback in user friendly format

35. Assessments of the major domains • Reading, mathematics and science are the major domains • Each major domain is measured in 3 dimensions in different scales • Large item banks are used to measure the major domains

36. Assessment of minor domains • There are 16 minor domains in the system • school readiness • writing skills, musical abilities, English as a second language • health literacy, financial literacy, visual skills, civic competencies • combinatorial reasoning, inductive reasoning • problem solving, learning to learn, ICT literacy, creativity • social skills, motivation • Test batteries are prepared for the minor domains

37. Plans for the next phase: 2015- • Online developmental exercises for helping slow students to catch up • Gamification • Visualization • Integration with the regular instructional processes • Extending to the kindergarten population

38. The economists’ view The significance of early education(The Heckman curve) The developmental psychologists’ view:Solid foundation for later learning Basic skillsInterestAttitudesMotivation

39. Can online assessments be used in kindergarten?

40. YES!

41. Yes!

42. Further plans to develop educational assessment in Hungary

43. Digitizing the Assessments of Basic Competencies • Present problems • only reading and mathematics is assessed • expensive • slow feedback • difficult to interpret • Plans for TBA • adding science • developing item banks • immediate feedback • visualizing the data to help the interpretation

44. Technology-based reform of the ‘matura’ • Present problems • too many elective subjects • low objectivity • no comparable scales • etc. • Plans for the reform • technology based assessments only • 5 compulsory subjects: Hungarian language and literature, history, mathematics, science, English (no electives)

45. Conclusions Technology-based assessments work well in kindergarten and primary schools Technology can be extended to other domains and age groups, the precision, objectivity and reliability can be improved Further work is needed to integrate the assessments into the regular classroom activities and educational processes

46. www.staff.u-szeged.hu/~csapo Thanks for your attention!