1 / 79

Computer-Based Assessments

Computer-Based Assessments. CBA Pilot for NILA & NINA Autumn 2013 Information Seminar for Principals September 2013 Anne Cromie and Dorothee Wagner. Learning Intentions. At the end of the session you will be aware of: the context for computer-based assessment (CBA ) in autumn 2013;

xenon
Download Presentation

Computer-Based Assessments

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Computer-Based Assessments CBA Pilot for NILA & NINA Autumn 2013 Information Seminar for Principals September 2013 Anne Cromie and Dorothee Wagner

  2. Learning Intentions At the end of the session you will be aware of: • the context for computer-based assessment (CBA) in autumn 2013; • developments to NILA and NINA; • next steps for the pilot schools; and • guidance and support available from CCEA and C2k.  Will have an opportunity to: • contribute to discussion about using the NILA and NINA outcomes.

  3. Overview of the Seminar 9.30 Registration and tea/coffee 10.00 Welcome Context of Computer-Based Assessment in NI Developments - NILA and NINA 12.30 Lunch 13.30 Reporting Developments Next Steps for CBA Pre-Pilot Guidance and Support available from CCEA 15.00 Questions, Evaluation & Close

  4. Part 1 Setting the Context for the CBAs

  5. Review of Computer-Based Assessment • A review by the Council for the Curriculum, Examinations and Assessment on the operation of the new assessments; 2. A review by the Education and Training Inspectorate which will determine how effectively schools make use of assessment information; • An independent review of the technical issues to establish why they occurred and what lessons can be learnt for the future; and • A review by the Department of the policy of statutory Computer Based Assessment.

  6. Legislative Context In his letter to schools in May 2013 the Minister for Education outlined that: • CBA assessment will not be specified as mandatory in 2013/14; • Use of CBA assessments (NILA and NINA) in 2013/14 will be optional for schools. “Although CBA will not be specified in 2013/14, the Department expects schools to carry out diagnostic assessment using an assessment tool of its own choice and that the results of this assessment should be shared with the pupil’s parent/guardian at a mutually convenient time.” “I would like to encourage as many schools as possible to participate in this pilot.”

  7. Scale of the CBA Pilot Total number of schools participating in CBA Pilot: approx. 230 • Initial Pilot Testing Schools: 2 (1 NT, 1 T) • Pre-Pilot: 10 (9 NT, 1 T) • Pilot, Phase 1 (NT Schools): 61 • Pilot, Phase 2 (NT Schools): 62 • Pilot, Phase 3 (NT Schools): 64 • Phase 4 (T Schools): 33

  8. Moving Forward with the CBA Pilot

  9. Computer-Based Assessments NILA - Northern Ireland Literacy Assessment NINA - Northern Ireland Numeracy Assessment • Assessments are available in English & Irish. • Assessments are adaptive and not time-bound. • Assessments are web-based and pupil responses to questions are saved automatically online. • Assessments may be completed by Year 4 to Year 7 pupils from the pilot schools during autumn 2013. • Assessments are processed immediately following completion and schools canaccess results after 48 hours. The purpose of the computer-based assessments is to help teachers plan the learning and teaching during the school year to meet the needs of pupils in their classrooms.

  10. Computer-Based Assessments The question items in the Literacy / Numeracy Assessments have been developed using the following criteria: • Definition of Literacy / Numeracy as set out by CCEA; • The expectations of the NI Language & Literacy / Mathematics & Numeracy Curricula; • CCEA’s revised Lines of Development; • Advice from ELBs regarding expectations at the end of each academic year; and • Levels of Progression.

  11. Focus on Raising Standards “The use of data from computer-based assessments remains a key aspect of the Department’s school improvement policy” Paragraph 3 – DE Letter to Principals Nov 2011

  12. Computer-Based Assessments

  13. NILA Overview

  14. The Literacy Assessment • Assesses some crucial aspects of Listening, Reading and Writing. • Draws on topics from across the Northern Ireland Curriculum. • Is varied in form, ranging from everyday texts, such as advertisements and signs, to high quality literary passages. • Uses 8 different answer techniques (cloze, drag and match, drag into order, drag punctuation, drag to hot spot, pick multiple, pick one and select word). • Uses the potential of the on-screen presentation to provide interactive and engaging experiences in different ways.

  15. The Literacy Assessment • There are two age related bands in the assessment, one for Years 4 and 5 and one for Years 6 and 7. • The questions cover a range of difficulty within each band, designed to allow pupils show what they can do whatever their ability. • The assessment adapts to easier or more challenging questions based on pupils’ responses. • The assessment should take the majority of pupils around 35 minutes to complete. • Provides teachers with diagnostic information about the skills and understanding of their pupils.

  16. Focus on Skills

  17. Adaptive Routing in NILA

  18. 2012 Outcomes

  19. NILA 2012 Outcomes English Medium Assessment Blue = Years 4 & 5 Red = Years 6 & 7 This shows that across all year groups the proportion of pupils in the ‘below expected’ and ‘above expected’ categories remains relatively even.

  20. NILA – Progression across Year Groups These charts show that there is progression across year groups, where over time, a greater proportion of pupils are attaining higher competence levels.

  21. NILA – moving through question sets The proportion of pupils staying (red) is greatest between sets 2 & 3. A larger proportion of pupils in Y5 & Y7 moving up (green) in sets 3 & 4. On average, the majority of pupils advance again in sets 4 & 5.

  22. NILA Assessment Duration This chart shows little difference in the proportions where year groups are compared. However, a greater proportion of Year 4 pupils taking longer than 70 minutes. Blue = Y4 Red = Y5 Green = Y6 Purple = Y7

  23. NILA Question Answer Techniques This chart shows the proportion of correct answers by the type of the question answer technique. ‘Drag punctuation’, ‘pick multiple’ and ‘select word’ were observed as potentially more difficult. Blue = Years 4 & 5 Red = Years 6 & 7

  24. NINA Overview

  25. The Numeracy Assessment • Samples the key concepts of Numeracy through two assessments / strands: • Strand 1 Understanding & Using Number • Strand 2: Shape & Space, Measures; Handling Data • Each strand will be assessed using an adaptive approach to assessment. • Assessment for each strand will take approximately 20 minutes. • Only mental arithmetic questions are timed. • English Medium database is multi-instance.

  26. Focus on Skills

  27. NINA Assessment Process

  28. Adaptive Assessment • All pupils start in their target year and are presented with questions which are one year below their current school designated year e.g. Year 3 questions for a Year 4 pupil. • On the basis of their responses, pupils are promoted, demoted or stay in their curricular year. • A promotion may be reversed by a subsequent demotion and vice versa. • The overall result is then computed as a weighted average of the two strand scores using the number of threads as the weighting. • Outcomes from the assessment give both numerical and qualitative diagnostic information about individual pupils.These will become available 48 hours after completion of Strand 2. .

  29. Adaptive Assessment

  30. 2012 Outcomes

  31. Questions and Duration 74% of questions were delivered and responded to online in less than 60 seconds

  32. Assessment Performance Performance of pupils answering questions in their target year:

  33. Comparison of mean Chronological Age with mean Age Related Outcome (ARO)

  34. Promotions and Demotions by Year

  35. Year 5 Promotions & Demotions Curriculum Year Y1 Y2 Y3 Y4 Y5 Y6 Y7

  36. Question Item Analysis

  37. Question Item Analysis Key Factors • number of times the response type is deployed; • the objective to be assessed; • the level of difficulty required, as determined by curricular year and band of difficulty; • the selection of appropriate question response type/behaviour; • opportunities for pupils to practise the behaviours; and • the ability of the pupil.

  38. Part 2 Developments to NILA and NINA

  39. 2012 CCEA Review Evaluation of: • online CBA training (users); • CBA implementation during autumn 2012 (principal, teacher and pupil); • AROs and SSs (principal and teachers); and • parent perspective. Feedback from: • helpdesk calls; • ELB support teachers seminars; • Principals’ focus groups; and • written and verbal feedback from special schools

  40. Developments • NILA / NINA website improvements and alignments; • NILA and NINA administrator’s access/permissions aligned; • Revised Pupil Demos and associated resources; • NINA Practice Area; • Item Analysis and Question Banks; • SEN Improvements; • Monitoring Assessment Status; • Improved Reports (simpler, sortable, printable and saveable); • Charts and Tables showing progress over time; • Improved IM Language selection; • Modified Pupil Checklist; and • Streamlined Helpdesk.

  41. Pupil Demos Pupil Demos for autumn 2013 • Pace of the voice will be slowed • Optional sub-titles available • Scripts will be available. NILA Practice Assessment Demo / NINA Practice Area Demo: Introduces pupils to NILA / NINA and the techniques they need to be aware of when completing their assessment. NILA / NINA Real Assessment Demo: Reinforces this knowledge and prepares the pupils for logging in to complete their actual assessment.

  42. Revised Practice Area

  43. Revised Practice Area

  44. Item Analysis NILA • Additional questions added to NILA at the lower and upper end. • Drag punctuation questions: • Improved ‘grab’ of the box; and • More accurate placement of the punctuation mark. • Improved graphics and screen display. • Improvements for Hearing Impaired (HI) pupils. NINA • Reviewed small number of questions with high proportion of incorrect answers.

  45. General SEN improvements • Specific guidance included in the NILA / NINA Guidance Documents; • Improved font size, style and colour; • Language modification; • Setting an SEN entry point for individual pupils; • Setting an HI entry point (removes Listening questions from NILA); • Improved ‘drag punctuation’ questions in NILA; • Voice in NILA and NINA Demos slowed down; • Improved pupil demos (for the practice areas and the real assessment); • Streamlined NINA Practice area; and • Simplified pupil checklists.

  46. Introduction of Special Entry Points What qualifies a pupil to use the SEN entry point? • Teacher and SMT must: • reasonably expect that the pupil is performing 2 or more years below their chronological age; and • be in agreement that an SEN entry point is appropriate for the pupil. What qualifies a pupil to use the HI entry point? • Teacher and SMT must: • have evidence that the pupil is receiving special support for HI; and • be in agreement that an HI entry point is appropriate for a pupil. N.B: Only the administrator can select the SEN and HI entry point.

  47. NILA SEN and HI Entry Point SEN Entry Point Pupils will: • have access to the same number of questions as their peers; • allows pupils quicker access to easier and shorter questions; and • aims to deliver a more positive and potentially shorter assessment. Hearing Impaired Entry Point Pupils will: • only be assessed in Reading and Writing; and • use an appropriate checklist.

  48. Setting an SEN Entry Point for a Pupil (NILA)

  49. Setting an SEN Entry Point for a Pupil (NINA)

  50. NINA SEN Entry Point

More Related