What do the kids think ? A quantitative analysis of feedback questionnaires in standardised reading tests

1. What do thekidsthink?A quantitative analysis of feedback questionnaires in standardised reading tests Eva Konrad & Annabell Marinell

2. Outline • Research questions • Secondary school types in Austria • The research instrument • Results • Conclusions A. Marinell & E. Konrad IATEFL TEASIG 2011

3. Research questions • Is there a connection between • the test takers’ familiaritywith the topics • their interestin the texts used • how difficult the test takers felt the texts to be • their familiarity withthe test methods and how well they felt it tested their reading ability? • Is there a connection between the test takers’ total scores andhow well they felt the standardised reading test tested their reading ability? • Is there a difference in test takers’ responses from different school types? A. Marinell & E. Konrad IATEFL TEASIG 2011

4. Secondary school types in Austria AHS BHS Classical secondary academic school Secondary academic school emphasizing mathematics and science Secondary academic school emphasizing economics Upper-secondary academic school specializing in instrumental music Upper-secondary academic school specializing in fine arts HUM: comprise business studies, theory in the respective area as well as compulsory work placements HAK: secondary schools for occupations in the business sector HTL: secondary schools for engineering BAKIP: nursery schools BASOP: colleges for social education HLFS: colleges for agriculture and forestry A. Marinell & E. Konrad IATEFL TEASIG 2011

5. Research instrument: test booklet • 4 independent reading tasks • Test formats: • Multiple Choice (MCQ) • Multiple Matching (MM) • Sequencing (SEQ) • Note Form (NF) • True/False/Justification (TFJ) (dropped: True/False/Not given) • Answer sheet • Questionnaire A. Marinell & E. Konrad IATEFL TEASIG 2011

6. Research instrument: questionnaire Questionnaire excerpt A. Marinell & E. Konrad IATEFL TEASIG 2011

7. Selected questions for research • How familiar were you with the topic of the texts? • How interesting did you find the texts? • How difficult did you find the texts? • How familiar were you with the test methods? • How well could you show your reading competency in this test? A. Marinell & E. Konrad IATEFL TEASIG 2011

8. Face validity “The extent to which a test meets the expectations of those involved in its use, e.g. administrators, teachers, candidates and test score users; the acceptability of a test to its stakeholders.” (McNamara 2000) For our research:How well did the test takers feel the standardised reading test was able to measure their reading competency? A. Marinell & E. Konrad IATEFL TEASIG 2011

9. Test population A. Marinell & E. Konrad IATEFL TEASIG 2011

10. Method Programme • MSB statistics programme • Developed by Gerhard Marinell and Gabriela Steckenberger Method A. Marinell & E. Konrad IATEFL TEASIG 2011

11. MSB statistics interface

12. Data matrix Significance level as quoted in Henning (1987): α=0.05 A. Marinell & E. Konrad IATEFL TEASIG 2011

13. Question 1 Is there a connection between • the test takers’ familiarity with the topics • interest in the texts used • how difficultthey felt the texts to be and how well they felt their reading ability was tested? Is there a connection between these factors and the test takers’ total score? A. Marinell & E. Konrad IATEFL TEASIG 2011

14. Values in % A. Marinell & E. Konrad IATEFL TEASIG 2011

15. Values in % A. Marinell & E. Konrad IATEFL TEASIG 2011

16. Values in %

17. Question 2 Is there a connection between the test takers’ familiarity with the test methods and how well they felt their reading ability was being tested? A. Marinell & E. Konrad IATEFL TEASIG 2011

18. Values in % A. Marinell & E. Konrad IATEFL TEASIG 2011

19. Question 3 Is there a connection between the test takers‘ perception of how well their reading competence was tested and their total scores? A. Marinell & E. Konrad IATEFL TEASIG 2011

20. Values in %

21. Conclusions • We could observe certain tendencies: • Correlations in all of the factors were more frequent in 2011 than in 2008 • Correlations with face validity were more frequent than with total score • Further analysis of feedback questionnaires is needed • We will be able to say more about BHS after more trials • We still have several questions: • Are there differences between the different BHS school types? • What further information might we gain from the open questions? A. Marinell & E. Konrad IATEFL TEASIG 2011

22. Bibliography • Alderson, C. J., Clapham, C., & Wall, D. (2005). Language testconstructionandevaluation. Cambridge: CUP. • Bachman, L. F. (2004). Statistical analysesforlanguageassessment. Cambridge: CUP. • Henning, G. (1987). A guidetolanguagetesting: development, evaluation, research. Cambridge, Mass: Newbury House Publishers. • McNamara, T. (2000). Language testing. Oxford: OUP.

23. Thank you for your attention and enjoy the rest of the conference! A. Marinell & E. Konrad IATEFL TEASIG 2011