1 / 20

Important Exams and the Concept of Good Practice

Important Exams and the Concept of Good Practice. Dávid Gergely. Interpret focus of lecture. Hardly feasible task to cover all important exams M any websites, information booklets, etc. provide the information anyway, for interested fellow professionals New focus about

arabella-elvery
Download Presentation

Important Exams and the Concept of Good Practice

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Important Exams and the Concept of Good Practice Dávid Gergely

  2. Interpret focus of lecture • Hardly feasible task to cover all important exams • Many websites, information booklets, etc. provide the information anyway, for interested fellow professionals • New focus about • quality of examinations • what may be considered good professional practice. • „Tools for fishing”: a series of criterial questions

  3. What can you do? • Study test specifications and other printed info packs, mock exams and any published docs. • Study regulations • Carry out an observation task, collect information.

  4. Studying the specifications 1 • Is the target language sampledwidely enough? • The nature of foreign lang. Proficiency • Skills, communicatve language activities, abilities • Are all four skills + grammar/vocab tested? • Litmus test: Is listening comprehension tested?

  5. Studying the specifications 2 • Is the candidate required to produce a large sample of the target language? • Are the papers long enough? • Not enough to include 5-6 papers, but there must be enough items etc. in each to give candidates as many chances (fresh new starts, fairness) to show what they know. • In terms of our litmus listening test, how many items based on how many sound tapes (hangzóanyag) is good enough? • 20 items based on 3 sound tapes is minimum.

  6. Explanation • Not covering enough skills (subskills), activities, abilities and/or doing that through too few items is called the underrepresentation of the test construct – a typical problem with many tests. • The difficulty of inferring underlying proficiency from test proficiency. • The quest for information: • Student 1: 1,0,1,0,0,1,0,0,1,1 • Student 2: 1,2,1,0,2,0,1,1,0,2

  7. Studying the specifications 3 • Static specs of papers, tasks types, etc. Do not speak about what the test provider do to ensure quality • Is the testing process specified too? • Procedural validity

  8. Model testing process: test material • Drafting tasks • Moderation • Revision of drafts • Pretesting draft tasks • Revision 2 • Administration • Post-test quality control • Score calculation • Standard (level) setting

  9. Model testing process: examiner • Training • Benchmarking • Administration • Post-test quality control • Score calculation • Standard (level) setting

  10. Study regulations, observe practice • Is the model testing procedure followed? • If it is not followed in every respect, how do they „make up for it”?

  11. Applying the procedure to ensure validity • There must be a quality control mechanism that assumes, especially: • Candidate response data (all such data) are collected systematically • Response data are analysed statistically after the pretest, but most importantly the post-test, in order to decide how good the items were, how well the raters marked, whether intended level was maintained, etc. • Are response data collected systematically? • Are the data analysed statistically?

  12. Illustrations of data problems • How is the oral scored? • Two raters added up? Averaged? • Raters agree? Separately given scores recorded? • Is it possible to put together, candidate by candidate, scores from the written papers with scores from the oral?

  13. Data? Analysis? Why the hell? • The Holy Grail of language testing • Want to find out candidates’ lang. proficiency • Not directly measurable • Must use tools (items, raters, criteria, i.e. test method) to make that proficiency observable • Whatever the method, it will unavoidably influence test performance and results • Unknown proficiency+method effect = scored test performance

  14. 3232,1,63-74,1,1,1,1,1,1,1,1,0,1,1,1 3233,1,63-74,1,1,1,1,1,1,0,1,1,0,1,1 3234,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3235,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3236,1,63-74,0,0,0,1,0,1,1,1,0,0,0,0 3237,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3238,1,63-74,1,1,0,1,1,1,0,1,0,1,1,1 3239,1,63-74,1,1,1,1,1,1,0,1,1,1,1,1 3240,1,63-74,0,1,1,1,0,1,1,0,1,0,1,0 3241,1,63-74,0,1,1,1,0,0,1,1,0,0,0,0 3242,1,63-74,1,0,1,1,1,1,1,1,1,1,1,1 3243,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3244,1,63-74,0,1,1,1,1,1,0,1,0,1,1,1 3245,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3246,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3247,1,63-74,0,0,1,1,0,0,0,0,0,1,0,0 3248,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3249,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3250,1,63-74,1,0,1,0,1,1,1,1,1,1,1,0 3251,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3252,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3253,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3254,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3255,1,63-74,0,1,0,1,0,1,0,1,1,0,0,0 3256,1,63-74,1,1,1,1,1,1,0,1,1,1,0,0 3257,1,63-74,0,1,0,0,1,1,0,0,1,0,1,1 3258,1,63-74,0,0,0,1,1,0,0,1,0,1,0,0 3259,1,63-74,0,0,0,0,1,0,1,1,0,1,1,0 3260,1,63-74,0,1,1,1,0,1,1,1,1,0,1,0 3261,1,63-74,0,1,1,0,1,0,0,1,1,1,1,0 3262,1,63-74,0,0,0,1,1,0,0,0,1,0,0,0 3263,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3264,1,63-74,1,1,1,1,1,1,0,1,1,1,1,0 3265,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3266,1,63-74,1,1,0,1,1,1,0,1,1,1,1,1 3267,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3268,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3269,1,63-74,0,1,1,1,0,1,1,1,1,1,0,1 3270,1,63-74,1,0,0,1,1,1,1,1,0,0,0,0 3271,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3272,1,63-74,0,1,1,1,1,1,0,1,0,0,0,0 3273,1,63-74,0,0,0,1,0,1,0,1,1,1,1,0 3274,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3275,1,63-74,1,1,1,1,1,1,0,1,1,1,1,1 3276,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3277,1,63-74,0,1,0,1,1,1,0,1,1,1,0,0 3278,1,63-74,1,1,0,0,1,1,0,1,0,0,1,0 3279,1,63-74,1,1,0,1,0,1,1,0,1,0,1,0 3280,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3281,1,63-74,1,1,1,1,1,1,0,1,0,1,0,0 3282,1,63-74,1,1,1,1,1,1,0,1,1,1,1,1 3283,1,63-74,1,1,0,1,1,1,1,1,0,1,1,1 3284,1,63-74,0,0,0,1,0,0,0,1,0,1,0,0 3285,1,63-74,1,1,1,1,1,1,0,1,1,0,0,1 3286,1,63-74,0,1,1,1,0,0,1,0,1,1,1,0 3287,1,63-74,0,0,1,1,1,1,1,0,1,0,1,0 3288,1,63-74,1,0,1,1,1,1,0,1,1,1,1,0 3289,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3290,1,63-74,0,0,0,1,1,1,1,1,0,0,0,0 3291,1,63-74,0,1,0,1,1,0,1,1,1,1,0,0 3292,1,63-74,1,0,1,1,1,0,0,1,1,0,1,0 3293,1,63-74,1,1,1,1,1,0,1,1,0,1,1,0 3294,1,63-74,1,1,0,1,1,0,0,1,0,0,1,0 3295,1,63-74,0,1,1,1,1,1,0,1,1,1,1,0 3296,1,63-74,0,1,1,1,0,1,1,1,1,1,1,1 3297,1,63-74,0,0,0,1,0,1,1,1,0,0,1,0 3298,1,63-74,0,1,0,1,1,1,0,1,0,1,0,0 3299,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3300,1,63-74,1,1,1,1,1,1,1,1,1,1,0,1 3301,1,63-74,1,0,1,1,1,1,1,1,0,1,0,0 3302,1,63-74,1,1,1,0,1,1,1,1,1,1,1,1 3303,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3304,1,63-74,1,0,0,1,1,1,0,1,1,0,1,1 3305,1,63-74,1,0,1,1,1,0,1,1,1,0,0,0 3306,1,63-74,1,1,1,1,1,1,0,1,1,0,0,0 3307,1,63-74,0,0,1,1,0,1,0,1,0,1,0,0 3308,1,63-74,0,1,0,1,0,0,0,1,1,0,0,0 3309,1,63-74,1,1,1,1,1,1,1,1,1,1,0,1 3310,1,63-74,0,1,0,1,1,1,0,1,0,1,1,0 3311,1,63-74,0,0,0,1,0,1,0,1,0,1,0,1 3312,1,63-74,1,1,1,1,1,1,1,1,0,1,1,0 3313,1,63-74,1,1,1,1,1,1,0,1,0,0,1,0 3314,1,63-74,0,0,0,1,0,1,0,1,0,1,0,0 3315,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3316,1,63-74,0,0,0,1,0,0,0,1,0,1,0,0 3317,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3318,1,63-74,1,1,1,1,1,1,0,0,1,1,0,0 3319,1,63-74,0,0,0,1,0,0,0,1,0,0,1,1 3320,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3321,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3322,1,63-74,0,1,0,1,1,1,1,1,0,1,1,1 3323,1,63-74,1,1,1,1,0,1,0,1,1,1,1,1 3324,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3325,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3326,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3327,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3328,1,63-74,0,1,0,0,0,1,1,1,0,1,1,0 3329,1,63-74,1,1,1,1,1,1,0,0,1,0,1,0 3330,1,63-74,0,1,0,1,1,1,1,1,0,0,0,0 3331,1,63-74,0,1,0,0,1,1,0,1,0,1,0,1 3332,1,63-74,1,1,1,1,1,1,0,1,1,0,0,0 3333,1,63-74,0,1,1,1,0,1,1,1,1,1,1,1 3334,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3335,1,63-74,1,1,0,1,1,1,1,1,0,1,1,1 3336,1,63-74,1,1,0,1,1,1,1,1,0,1,1,1 3337,1,63-74,1,0,1,0,1,1,1,1,1,0,1,0 3338,1,63-74,1,1,1,1,1,1,1,1,1,1,0,0 3339,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3340,1,63-74,0,0,0,1,1,0,0,0,1,1,0,0 3341,1,63-74,1,0,1,0,1,1,1,1,1,1,1,1 3342,1,63-74,1,1,1,1,1,1,0,1,1,0,1,0 3343,1,63-74,0,1,1,1,0,1,0,1,0,1,0,1 3344,1,63-74,1,1,0,1,0,1,0,1,1,1,0,1 3345,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3346,1,63-74,1,1,0,1,0,1,1,1,0,1,1,0 3347,1,63-74,0,1,1,1,0,0,0,1,1,1,1,0 3348,1,63-74,0,0,0,0,0,1,0,0,0,1,0,0 3349,1,63-74,1,1,1,1,1,1,0,1,1,1,1,0 3350,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3351,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3352,1,63-74,1,1,1,1,1,1,0,1,0,1,1,1 3353,1,63-74,0,1,1,0,1,1,0,1,1,1,1,0 3354,1,63-74,1,1,1,1,1,1,0,1,0,0,0,0 3355,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3356,1,63-74,0,0,1,1,1,0,0,1,1,0,1,0 3357,1,63-74,1,1,1,0,1,1,0,1,1,1,1,1 3358,1,63-74,1,1,1,1,1,1,0,1,1,1,1,0 3359,1,63-74,1,1,1,1,1,1,0,1,0,1,1,0 3360,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3361,1,63-74,1,1,1,0,1,1,0,1,1,1,1,0 3362,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3363,1,63-74,0,1,0,0,1,1,1,1,1,1,1,1 3364,1,63-74,1,1,1,1,0,1,1,1,1,0,1,1 3365,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3366,1,63-74,1,1,1,1,1,1,0,1,0,0,0,0 3367,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3368,1,63-74,1,0,0,1,1,1,0,1,0,0,0,1 3369,1,63-74,1,1,1,1,1,0,0,0,0,1,0,0 3370,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3371,1,63-74,0,1,1,1,0,1,1,1,1,1,1,0 3372,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3373,1,63-74,0,0,1,1,1,1,0,1,1,0,1,0 3374,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3375,1,63-74,1,1,1,1,1,1,0,1,1,1,1,1 3376,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3377,1,63-74,1,1,1,1,1,1,1,0,1,1,1,0 3378,1,63-74,0,0,1,1,0,1,1,1,0,0,0,0 3379,1,63-74,1,1,1,1,1,1,0,1,1,1,1,0 3380,1,63-74,1,1,1,1,1,1,1,1,0,1,1,0 3381,1,63-74,1,0,1,0,1,1,0,1,1,0,1,0 3382,1,63-74,0,1,0,1,1,1,1,1,1,1,1,0 3383,1,63-74,1,1,0,1,1,1,0,1,1,1,1,1 3384,1,63-74,1,1,1,1,1,1,1,1,1,1,0,1 3385,1,63-74,1,1,1,1,1,1,1,1,0,1,0,1 3386,1,63-74,1,0,0,1,1,0,1,0,0,0,0,0 3387,1,63-74,1,0,1,1,0,1,0,1,0,0,0,1 3388,1,63-74,1,1,1,1,1,1,0,1,1,1,1,0 3389,1,63-74,0,1,1,1,0,1,1,1,1,1,1,0 3390,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3391,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3392,1,63-74,0,1,1,0,1,0,0,1,0,1,0,1 3393,1,63-74,0,1,0,1,1,1,1,1,0,1,0,0 3394,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3395,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3396,1,63-74,0,0,0,1,1,1,0,1,0,0,1,1 3397,1,63-74,0,0,1,1,1,1,0,1,0,0,0,0 3398,1,63-74,0,1,1,1,0,1,1,1,1,1,1,0 3399,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3400,1,63-74,1,1,1,1,1,1,0,1,1,1,1,1 3401,1,63-74,0,1,1,1,0,1,0,1,1,1,0,1 3402,1,63-74,0,0,1,1,1,0,0,1,0,0,0,0 3403,1,63-74,0,0,1,1,0,0,0,1,1,0,1,0 3404,1,63-74,0,1,1,1,0,0,0,0,0,0,1,0 3405,1,63-74,1,1,1,1,1,1,1,0,1,1,1,1 3406,1,63-74,0,0,0,1,0,0,0,1,0,0,0,0 3407,1,63-74,1,1,1,1,1,1,0,1,1,1,0,1 3408,1,63-74,0,0,0,0,0,1,0,0,0,0,1,0 3409,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3410,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3411,1,63-74,0,0,1,0,1,0,1,0,1,0,1,0 3412,1,63-74,0,1,0,1,1,1,0,1,1,1,1,0 3413,1,63-74,1,0,0,1,1,1,0,1,0,1,1,0 3414,1,63-74,1,1,1,0,1,1,1,1,0,1,1,0 3415,1,63-74,1,1,1,1,1,1,0,1,1,1,1,1 3416,1,63-74,0,1,1,1,1,1,0,1,1,1,0,0 3417,1,63-74,1,1,1,1,1,1,1,1,1,1,1,0 3418,1,63-74,1,1,1,1,1,1,1,1,0,1,1,1 3419,1,63-74,1,0,0,1,0,1,0,1,0,1,0,0 3420,1,63-74,0,1,1,1,0,1,1,1,0,0,1,1 3421,1,63-74,0,1,1,1,1,1,0,1,0,0,1,1 3422,1,63-74,1,1,1,1,0,1,0,1,1,1,0,0 3423,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3424,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3425,1,63-74,0,0,0,0,0,0,0,0,0,0,0,0 3426,1,63-74,1,1,1,1,1,1,0,1,0,1,1,1 3427,1,63-74,0,0,1,1,0,1,0,0,1,1,1,0 3428,1,63-74,1,1,1,1,1,1,0,0,0,1,0,0 3429,1,63-74,0,0,0,0,0,0,0,0,1,0,0,0 3430,1,63-74,1,1,1,1,1,1,0,1,0,1,1,1 3431,1,63-74,0,0,1,1,0,0,0,0,1,1,0,1 3432,1,63-74,0,0,0,0,1,0,0,0,0,0,1,0 3433,1,63-74,0,1,0,1,0,1,1,1,0,0,1,1 3434,1,63-74,1,1,1,1,1,1,1,1,1,1,1,1 3435,1,63-74,0,1,0,1,1,1,0,1,0,1,1,0 3436,1,63-74,0,0,0,1,0,0,0,0,0,0,1,0

  15. Study regulations, observe practice • Has the exam gone through accreditation procedure by the Ministry? • Many exams have, some have not (list www.nyak.hu) • The Hungarian accreditation system not perfect, but at least • a body of experts to review exams regularly and on demand • documentation must be submitted • Some achievements

  16. Study regulations, observe practice • How soon are results announced? • If announced very soon, it is unlikely to be a good test, as it is impossible to do the quality control work that needs to be done.

  17. Study regulations, observe practice • Are most complaints handled favourably for the candidates? • If quality control procedures are weak, such practice sends a wrong message: • „Kaparj kurta neked is jut”, making the candidate responsible for finding problematic items, raters, etc. • Implicit in law • Unfair. Quality control should not depend on candidates’ awareness of their rights.

  18. Study regulations, observe practice • Are pass rates very high? • Bear national educational context in mind: • falling numbers in the age-group where candidates might come from (14-22) • falling foreign language education standards • should predict falling pass rates too if standards (CEFR levels) are maintained . • The horrible case of a prestigious German exam 10 years ago.

  19. Summary • What is good testing practice? • Use this checklist of questions • Read up on language testing and add more questions • What you need • a keen eye • a sane mind.

  20. Required reading • McNamara, T. (2000). Language Testing. Oxford Introductions to Language Study. Series Editor: H.G Widdowson.Oxford: OUP. • The required chapters are • The Testing Cycle (Chapter 3) pp. 23-33; Validity: testing the test (Chapter 5) pp. 47-54; • Measurement (Chapter 6) pp.55-65.

More Related