1 / 29

Manual for Relating Examinations to the Common European Framework

Manual for Relating Examinations to the Common European Framework. Sauli Takala EALTA Confererence Slovenia, Granjska Gora, May 14-16, 2004. Overview. Some background Why is the Manual needed? What is the approach adopted? What is/is NOT the Manual for? Procedures proposed in the Manual

Olivia
Download Presentation

Manual for Relating Examinations to the Common European Framework

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Manual for Relating Examinations to the Common European Framework Sauli Takala EALTA Confererence Slovenia, Granjska Gora, May 14-16, 2004

  2. Overview • Some background • Why is the Manual needed? • What is the approach adopted? • What is/is NOT the Manual for? • Procedures proposed in the Manual • Current situation • Future prospects and challenges

  3. Some historical notes The Convention for Cultural Cooperation (CDCC), which provided the mandate for the Council of Europe´s action within the domain of cultural affairs was established 50 years ago Modern languages were included in the agenda about 45 years ago After a number of more ad hoc set of activities, action related to modern languages was organised into medium-term projects Work focused on exploring how learning objectives could be determined in a manner that reflected the social and learner needs The Threshold specification model of learning objectives is a good example of outcomes of this exploration and development work In 1991, at an intergovenmental symposium in Rüschlikon, Switzerland, it was agreed to launch a programme to develop a framework of reference for languages, which would provide a coherent conceptual tool for the various needs of language education. The Portfolio project was also launched. After some ten years of work, inlvolving broad consultation, the Common European Framework of Reference for Languages was adopted at an intergovernmental conference in Strasbourg In recent years both the Council of Europe and European Union have adopted the CEF as the basic tool for their work on language education

  4. Current CoE Projects • Common European Framwork of Reference of Languages (CEF) • European Language Portfolio • Manual for Relating Examinations to the CEF • Language Policy analyses and reviews • Other: CEF-sign language; Romani language curriculum; bank of descriptors; language-specific reference frameworks (German, French ...)

  5. What is the CEF – some points to • consider • CEF provides: • a ‘vertical dimension’ and outlines an ascending series of common reference levels for describing learner proficiency (6-point scales) • a ‘horizontal dimension’ - aset of descriptive categories map out made up of parameters of communicative activity and communicative language competence (not all are scaled)

  6. CEF construct of language proficiency: the key to understanding the CEF (p. 9, English version) • Language use, embracing language learning, comprises the actions performed bypersons who as individuals and as social agents • develop a range of competences, both general and in particular communicative language competences. • They draw on the competences at their disposal in various contexts under various conditionsand under various constraints • to engage in language activities involving languageprocesses • to produce and/or receive texts in relation to themes in specific domains, • activating those strategies which seem most appropriate for • carrying out the tasksto be accomplished. • The monitoring of these actions by the participants leads to thereinforcement or modification of their competences.

  7. Language learning potential progresses on two dimensions (Source: John de Jong,2004) Quantity Quality

  8. Scale hierarchy: quantitative dimension

  9. Frequently asked questions: How far do pupils/students progress in L2 in the school system (no PISA in L2; EU Barcelona indicator; European item bank) • What is the currency value of national school reports/certificates nationally and internationally? How could it be improved? • How can we relate our exams to the CEF? (need of guidance) • How do I know that my B1 is your B1? (need of guidance) • Need of guidance in relating exams to CEF in a credible manner led to the project of developing a Manual

  10. Some history related to the Manual: Invitational Expert Seminar; Helsinki, summer 2002 • Report: DG IV/EDU/LANG (2002) 15 • Recommended development of a Manual as a collaborative endeavour building on current work in member states. • Describe the examination coverage, administration and analysis procedures; • Align results reported from the examination to the CEF common reference levels; • Provide supporting evidence that reports on the procedures followed to do so.

  11. Who done it? • Authoring Group/Phase 1: (aka ”Manual workers”) : Brian North, Neus Figueras, Piet Van Avermaet, Sauli Takala, Norman Verhelst • Sounding Board of consultants: other Helsinki Seminar participants • Ad hoc Project Group (new): Charles Alderson, John de Jong, John Trim... Participants can vary according to topic of discussion

  12. Development Timeline • July 2002: Helsinki seminar • September 2002: Project approval • November 2002: Authoring group set up • Nov-Jan 2003: Consultation on Outline with Sounding board (consultants) • Jan-May 2003: Authoring Group: revision of draft text • May 19-20, 2003 Meeting of Authoring Group and consultants to discuss version 0.5 (Strasbourg) • Sep 2003: English & French drafts (DG IV/EDU/LANG (2003) 5 • Sep 2003: Piloting Feedback invited (guide provided) • April 15-16, 2004: Seminar on Piloting, Strasbourg; further action discussed

  13. Aims of Manual • contribute to competence building in the area of linking assessments to the CEF • encourage increased transparency on the part of examination providers • encourage the development of both formal and informal national and international networks of institutions and experts • Thus: Manual is not just a technical guidance tool but a rich resource for thoughtful action

  14. What the Manual NOT? • Not a guide how to write good tests; presupposes that exam results are reliable and their interpretation is valid • Does not demand any particular type of examination format nor exclude automatically some types of examination • Being CEF-based, it cannot provide resources that are not in the current CEF-version; however, new descriptive categories can be added in the Specification job (->CEF-revision) • Does not prescribe any one method of linking; other approaches are acceptable (cf. Reference Supplement) • Does not present a mechanism which would provide accreditation of examinations; this requires a specific rationale of its own • Is not a recipe; does nor provide rule-of-thumb shortcuts • Does not guarantee success in linking examinations to the CEF; the linking activities must be of high standard

  15. Overview of Linking Scheme • Chapter 1: Introduction gives a very good overview (“read me first”) • Familiarisation: thorough knowledge of the CEF “philosophy” and its resources (Chapters 2 & 3, pp. 13-28) • Specification: aims, CEF audit of coverage • Objectives, coverage, sections, weighting, text & item types, marking criteria, reporting (Chapter 4, pp. 29 – 63) • Standardisation: understanding CEF levels • Training with calibrated examples, benchmarking local examples for standardisation training; standard setting (Chapter 5, pp. 65-98) • Validation: Collection and analysis of data • Internal (test characteristics) External (relation to calibrated tests and descriptors • Actually ALL four activities are essential elements in building and substantiating an argument for a high quality work with linkage to the CEF • Chapter 6, pp. 99-122

  16. What do we claim we are testing? (What is our construct?) We now need to check what out examination really looks like. (check the claim; awareness raising) If we wish to relate our examination to the CEFR, we need to profile our examination drawing on its descriptive scheme and scales. Preliminary Pilot Manual proposes a procedure to profile examinations vis-a-vis the CEFR: SPECIFI-CATION (Ch. 4, 29-63) We need to use the descriptive resources the CEFR provides as much as is possible, and supplement the scheme if it is not sufficient for profiling the exam

  17. Specification • General description of exam (Form A1): purpose, target population, domains etc (”this is our product”) • Test development procedures, marking, reporting results • More specific description in terms of communicative activities: reception, interaction, production, mediation • More specific description in terms of aspects of language competences in: reception, production, mediation ...

  18. Specification sample

  19. Specification sample

  20. How do our exam results relate to CEFR levels? How do I know what level our exam results correspond to? How do I know my B2 corresponds to your B2? We obviously need to standardise our procedures (in Europe)– STANDARDISATION (Ch. 5) We need to set standards, including cut-off scores. We need rigorous rater training: check intrarater consistency, interrater consistency We need to choose scales: CEF scales; validation of local scales We need benchmarks: European, ”local” We will choose to apply one of the several possible methods of standard setting; Manual suggests one approach – Reference Supplement, Section B, contains more information

  21. External check of standard setting: verification • What it the evidential basis of our linking? • Internal validation • External validation • Chapter 6 in the Manual • Reference Supplement

  22. Link to CEF (Fellianka Kaftandjieva, Belgrade, 2003) Examination Through another scale Direct Standard Setting Through another test CEF scale

  23. Possible outcome: European chart which shows how validated examination results are located on the Framework scales. Not only ”My B1 is your B1” but also ”My own exam´s grade 6 (or ”Good”) corresponds to B2”, grade 7 (”Very good”) is C1.1 ....

  24. What the Manual is about? • A tool to promote competence to link examinations to CEF; a guide • A tool for raising awareness about the particular examination (a tool for reflection) • A tool with a certain approach outlined • Represents one particular approach; other approaches may be equally valid (need to present rationale and evidence) • Is mainly concerned with standard setting (not test development or construction) • Requires thoughtful application • Presupposes the provision of extensive and adequate empirical evidence to substantiate any claims

  25. Contributions welcome: • FEEDBACK on reading the Manual • PILOTING on the basis of using the Manual • CASE STUDY – writing up experience of using the Manual • PERFORMANCE SAMPLES – to be collected for speaking + writing; LC and RC items These will contribute to a revised and extended Manual “kit” www.coe.int/lang First feedback Seminar: Strasbourg, April 15-16,2004

  26. Final word • Many claim linkage to the CEF but few produce evidence, and so far – arguably - no evidence is completely adequate (DIALANG probably the best exception) • So: • Quis custodiet ipsos custodes? • Caveat emptor! • When there are good reports of linkage, what happens to them next?

More Related