slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
University of Michigan English Language Institute Testing and Certification Division PowerPoint Presentation
Download Presentation
University of Michigan English Language Institute Testing and Certification Division

Loading in 2 Seconds...

play fullscreen
1 / 16

University of Michigan English Language Institute Testing and Certification Division - PowerPoint PPT Presentation


  • 114 Views
  • Uploaded on

14 th International GALA conference, Thessaloniki, 14-16 December 2007. Behavioural scales of language proficiency: insights from the use of the Common European Framework of Reference Spiros Papageorgiou. University of Michigan English Language Institute

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'University of Michigan English Language Institute Testing and Certification Division' - wayne


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

14th International GALA conference, Thessaloniki, 14-16 December 2007

Behavioural scales of language proficiency: insights from the use of the Common European Framework of Reference Spiros Papageorgiou

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli

slide2

Outline

  • Background
  • Aims
  • Data collection
  • Data analysis
  • Results
  • Implications

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli

slide3

Background

  • Advent of the CEFR: increased interest in behavioural scales of language proficiency
  • Using the CEFR scales: Problems
  • Designing test specifications (Alderson et al., 2006)
  • Measuring progression in grammar (Keddle, 2004)
  • Describing the construct of vocabulary (Huhta & Figueras, 2004)
  • Designing proficiency scales (Generalitat de Catalunya, 2006)

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli

slide4

Background(2)

  • Using the CEFR scales: Criticism
  • Equivalence of tests constructed for different purposes (Fulcher, 2004b;Weir, 2005)
  • Danger of viewing a test as non valid because of not claiming relevance to the CEFR (Fulcher, 2004a)
  • Progression in language proficiency not based on SLA research but on judgements by teachers (cf. North 2000; North & Schneider 1998)

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli

slide5

Aims of the study

  • Investigation of three research questions:
  • Can users of the CEFR rank-order the scaled descriptors in the way the appear in the 2001 volume?
  • If differences in scaling exist between the users of the CEFR and the 2001 volume, why does this happen?
  • Can training contribute to more successful scaling?

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli

slide6

Data collection

  • 12 users of the scales acting as judges in relating two language examinations to the CEFR
  • Data collected during Familiarisation sessions described in the Manual for relating examinations to the CEFR
  • Part of a doctoral thesis at Lancaster University (Papageorgiou, 2007) and a research project at Trinity College London
  • Task: sort descriptors into the six levels

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli

slide7

Data collection (2)

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli

slide8

Data analysis

  • Analysis: FACETS Rasch computer program
  • 3 facets: descriptors-raters-occasions
  • Rank-ordering of elements of facets on a common scale
  • Fit statistics (Bond and Fox, 2001; McNamara, 1996)
  • Overfit: too predictable pattern
  • Misfit: more than expected variance
  • Acceptable range of fit statistics
  • Descriptors: .4-1.2 (Linacre & Wright, 1994)
  • Raters: .5-1.5 (Weigle, 1998)

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli

slide9

Results: Writing Levels A1-B1

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli

slide10

Results: Writing Levels B2-C2

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli

slide11

Results: Raters

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli

slide12

Results: Occassions

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli

slide13

Results: Correlations

Correlations of scaling between the judges and the CEFR volume

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli

slide14

Summary of results

  • Trained judges perceived language ability as intended in the CEFR
  • Almost identical scaling
  • Cut-offs between B2-C1 and C1-C2 unclear
  • Competences other than linguistic: misfitting descriptors
  • Unclear and inconsistent wording resulted in level misplacement by the judges
  • Mixed effect of training

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli

slide15

Implications of findings

  • Common understanding of the construct in the CEFR scales can be achieved, but
  • How valid is it to claim that a test is linked to B2 instead of C1 and C1 instead of C2?
  • How can sociolinguistic and strategic competences be tested in relation to the CEFR?
  • Can SLA research help better understand these issues?

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli

slide16

Contact details

Spiros Papageorgiou

University of Michigan English Language Institute

500 East Washington Street

Ann Arbor, MI

48104-2028

USA

spapag@umich.edu

University of Michigan

English Language Institute

Testing and Certification Division

www.lsa.umich.edu/eli