1 / 30

Proposed Cognitive Test for the Main Study

Proposed Cognitive Test for the Main Study. National Research Coordinators Meeting Windsor, June 2008. Contents. First impressions Summary of recommendations Possible outcomes Item deletions, minor changes. ≅ 50min. Group discussion of specific recommendations. ≅ 60min.

fisk
Download Presentation

Proposed Cognitive Test for the Main Study

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Proposed Cognitive Test for the Main Study National Research Coordinators Meeting Windsor, June 2008

  2. Contents • First impressions • Summary of recommendations • Possible outcomes • Item deletions, minor changes. ≅ 50min • Group discussion of specific recommendations ≅ 60min • Feedback from group discussions, comments, recommendations ≅ 70 min

  3. First impressions • Items worked well, can comprise single scale (NRC 2, p10) • CIVED trend set can be scaled with and used for single (NRC 2 pp16,17) • Test time per item seemed appropriate according to ‘not reached’ – still to be investigated further • No pattern of gender DIF by item type or substance (NRC 2, p12) • Some patterns of ‘regional’ item-by-country interaction (NRC 2, p15)

  4. First impressions (cont.) • Open-ended response items showed higher ‘non-attempts’ and slightly more variability (item-by-country interaction and opportunity for inter-rater reliability issues) (NRC 2, p17) • Targeting good (NRC 2, p9)

  5. Open-ended response item scoring guides • Revision: • Structural changes informed by suggestions from countries* • Addition of “real” student responses (thank you to Julie Nelson at NFER) • Addition of relevant scoring codes on the advice of countries* • Finalisation of scoring guides at the marker training seminar in Hamburg * Feedback from NRCs through questions, voluntary reports and field-trial operations survey.

  6. Summary of Item Recommendations *T/F

  7. Possible Outcomes • Shorten test to 45 minutes • Increase number of clusters to 7 • Delete two open-ended response items and approximately 15 MCQ to maintain test-taking rate. • Some updates on meeting documents resulting from further analyses and PAC meeting. • True/False items (NRC 5, pp 25-26) • PAC recommendations/suggestions (tabled document)

  8. Latest News • Some updates on meeting documents resulting from further analyses and PAC meeting. • True/False items (NRC 5, pp 25-26) • PAC recommendations/suggestions (tabled document)

  9. Part 2Group discussions of items

  10. Context of Group Discussions • Assessment framework developed in consultation with NRCs and other experts before the field-trial. • All countries invited to submit items before field-trial • All items made available for online feedback by NRCs before the field-trial • All items revised on the basis of the online feedback for the field-trial and approved for inclusion at the NRC meeting in Rome.

  11. Context for Group Discussions • Field-trial empirical data collected from 31 countries and approx. 19, 000 students • Recommendations for changes to instrument are based on empirical (and some qualitative) data from the field trial.

  12. Context for Group Discussions • The only changes that can be made to the cognitive instrument now are: • Deletion of items • Minor changes to items only if we are very confident that a change will fix a problem (and not introduce a new problem) • Amendments to the scoring guides

  13. Context for Group Discussions • All items with “good” data are to be included in the MS instrument • Groups to consider each of the ISC recommendations and make one of three recommendations for each: • Accept definitive ISC recommendation • Reject definitive ISC recommendation (delete item) • Offer alternative “solution” to ISC recommendation

  14. Session 2 – Until lunch • 30 minutes – finish group discussion • 60 minutes – BRIEF presentation from each group NOTE: Please submit all your comments and feedback to us (me) and we will include them in our deliberations regarding the final instrument.

  15. Part 3 Feedback from group discussions, comments, recommendations

  16. Booklet rotation

  17. Not reached for Latin American Countries

  18. Recommended delete Possible delete

  19. Recommended delete Possible delete

  20. Recommended delete Possible delete

  21. True/False Items Yuck! 

  22. True/False Items Also yuck! 

  23. True/False Items Disc: 0.18

  24. True/False Items Disc: 0.16

  25. True/False Items Disc: 0.36

  26. True/False Items

  27. True/False Items Disc: 0.31

  28. True/False Items

  29. True/False Items Disc: 0.15

  30. True/False Items Disc: 0.18

More Related