Educating the stakeholders l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 63

Educating the Stakeholders PowerPoint PPT Presentation


  • 134 Views
  • Uploaded on
  • Presentation posted in: General

Educating the Stakeholders. ECOLT 2008 November 7, 2008 ILR Testing Committee. Overview of Presentation. Intuitive Language Testing Testing Misconceptions Educating Stakeholders Continuing Challenges. Why educate stakeholders?.

Download Presentation

Educating the Stakeholders

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Educating the stakeholders l.jpg

Educating the Stakeholders

ECOLT 2008

November 7, 2008

ILR Testing Committee


Overview of presentation l.jpg

Overview of Presentation

  • Intuitive Language Testing

  • Testing Misconceptions

  • Educating Stakeholders

  • Continuing Challenges


Why educate stakeholders l.jpg

Why educate stakeholders?

  • Consequences of language tests reach far beyond the testing sessions.

  • Stakeholders

    • Examinees

    • Language teachers

    • Supervisors of linguists

    • USG agencies

    • Policy makers

    • Citizens


Eggen d 2006 october 11 fbi agents still lacking arabic skills washington post p a01 l.jpg

What is limited proficiency?

ILR Level 2

Proficiency in what?

Speaking

What type of Arabic?

Modern Standard Arabic

“Five years after Arab terrorists attacked the United States, only 33 FBI agents have even a limited proficiency in Arabic …”

Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01


Eggen d 2006 october 11 fbi agents still lacking arabic skills washington post p a015 l.jpg

Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01

Were all the agents tested?

Most agents do not have a language requirement.

What is meant by “any familiarity?”

Speaking Modern Standard Arabic

What does a score of zero mean?

“Counting agents who know only a handful of Arabic words -- including those who scored zero on the standard proficiency test -- just 1 percent of the FBI’s 12,000 agents have any familiarity with the language, the statistics show.”


Eggen d 2006 october 11 fbi agents still lacking arabic skills washington post p a016 l.jpg

Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01

What is ‘advanced professional proficiency?’

ILR Level 4

The tests were administered by the ILR?

The ILR does not administer tests.

“Youssef … is one of only six FBI agents who scored a 4 for ‘advanced professional proficiency’ in Arabic on standardized speaking tests administered by the Interagency Language Roundtable for federal agencies.”


Eggen d 2006 october 11 fbi agents still lacking arabic skills washington post p a017 l.jpg

Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01

Who is interpreting the language testing statistics?

An attorney for a former linguist, suing the FBI.

What language test was given?

Speaking, not Translation

Who do agents rely on?

Language Analysts and Intelligence Analysts

Tested in Reading, Listening, Translation, Speaking

“Youssef’s attorney, Stephen M. Kohn, said the statistics indicate that most FBI agents have no way to gauge the accuracy of translated materials and must rely on linguists or other third parties for their information.”


Intuitive test theory braun and mislevy l.jpg

Intuitive Test Theory Braun and Mislevy

  • General testing P-prims

    • A test measures what it says at the top of the page.

    • A test is a test is a test.

    • A score is a score is a score.

    • Any two tests that measure the same thing can be made interchangeable with a little “equating” magic.


P prims continued l.jpg

P-Prims, continued

  • You score a test by adding up scores for items.

  • 70% is passing – on any and every assessment.

  • You can tell if an item is good by looking at it.

  • Multiple-choice questions only measure recall.

  • Multiple-choice tests equal standardized tests equal high-stakes tests.


Ilta code of ethics l.jpg

ILTA Code of Ethics

Principle 7

  • Language testers in their societal roles shall strive to improve the quality of language testing, assessment and teaching services, promote the just allocation of those services and contribute to the education of society regarding language learning and language proficiency.


Ilta code of ethics11 l.jpg

ILTA Code of Ethics

Principle 8

Language testers shall be mindful of their obligations to the society within which they work, while recognising that those obligations may on occasion conflict with their responsibilities to their test takers and to other stakeholders.


Overview of presentation12 l.jpg

Overview of Presentation

Intuitive Language Testing

Testing Misconceptions

Educating Stakeholders

Continuing Challenges


Quote 1 l.jpg

Quote 1

Hassan (Foreign Service Specialist):

“I worked for an NGO in Mosul and helped several US-based companies set up their computer systems. I constantly talked to vendors, made arrangements to access the appropriate routers, negotiated bandwidth for their Internet hubs, the works! In my spare time, I talked to my Iraqi buddies about all sorts of things. They were great at tipping me off whenever good stuff came at the local grocery store. I thought I was a 5 in speaking Iraqi Arabic. You scored me a 2. Need I tell you what I think about your test?”


Testing misconception l.jpg

Testing Misconception

A 3 is a 3 is a 3.

  • Proficiency testing does not equal achievement or performance testing.

  • Scores in one skill do not transfer across other skills.

  • A 3 given by a non-USG testing organization may not equal a 3 given by a USG agency.


Educating stakeholders types of tests l.jpg

Educating Stakeholders: Types of Tests

  • Achievement = Memorized responses using the content of a specific textbook or curriculum.

  • Performance =Rehearsed ability to communicate in specific, familiar settings.

  • Proficiency = Unrehearsed general ability to accomplish communication tasks across a wide range of topics and settings.

    The following self assessment used by permission, Dr. Ray Clifford, ACTFL 2007


A self assessment quiz l.jpg

A Self Assessment Quiz

  • You will be asked about 6 different testing purposes.

  • For each of those test purposes, which type of test would you choose?a. Achievementb. Performancec. Proficiency


Which type of test would you choose achievement performance or proficiency l.jpg

Which type of test would you choose:Achievement, Performance, or Proficiency?

  • To assess students’ language learning after Chapter 3 of a beginning language course?


Which type of test would you choose achievement performance or proficiency18 l.jpg

Which type of test would you choose:Achievement, Performance, or Proficiency?

  • To assess students’ language learning after Chapter 3 of a beginning language course?

  • To place students into a university’s sequence of courses?


Which type of test would you choose achievement performance or proficiency19 l.jpg

Which type of test would you choose:Achievement, Performance, or Proficiency?

  • To assess students’ language learning after Chapter 3 of a beginning language course?

  • To place students into a university’s sequence of courses?

  • To test students completing a year-long, intensive language course?


Which type of test would you choose achievement performance or proficiency20 l.jpg

Which type of test would you choose:Achievement, Performance, or Proficiency?

  • To assess students’ language learning after Chapter 3 of a beginning language course?

  • To place students into a university’s sequence of courses?

  • To test students completing a year-long, intensive language course?

  • To screen job applicants for a specific job with well-defined, repetitive tasks?


Which test type would you choose achievement performance or proficiency l.jpg

Which test type would you choose:Achievement, Performance, or Proficiency?

  • To assess students’ language learning after Chapter 3 of a beginning language course?

  • To place students into a university’s sequence of courses?

  • To test students completing a year-long, intensive language course?

  • To screen job applicants for a specific job with well-defined, repetitive tasks?

  • To select someone to be the NATO spokesperson on a news show with a “hostile” moderator?


Which test type would you choose achievement performance or proficiency22 l.jpg

Which test type would you choose:Achievement, Performance, or Proficiency?

  • To assess students’ language learning after Chapter 3 of a beginning language course?

  • To place students into a university’s sequence of courses?

  • To test students completing a year-long, intensive language course?

  • To screen job applicants for a specific job with well-defined, repetitive tasks?

  • To select someone to be the NATO spokesperson on a news show with a “hostile” moderator?

  • To document a language ability rating for a person’s military or civilian personnel record?


Achievement performance or proficiency which test type did you choose l.jpg

Achievement, Performance, or ProficiencyWhich test type did you choose?

  • To assess students’ language learning after Chapter 3 of a beginning language course?

  • To place students into a university’s sequence of courses?

  • To test students completing a year-long, intensive language course?

  • To screen job applicants for a specific job with well-defined, repetitive tasks?

  • To select someone to be the NATO spokesperson on a “hostile” news show?

  • To document a language ability rating for a person’s military or civilian personnel record?


Educating stakeholders fsi examinees l.jpg

Educating Stakeholders:FSI Examinees

Information on the Skill Level Descriptions available online at www.govtilr.org

Self appraisals are suggested/required for some language positions

Formal presentations on language testing given to new entrants


Fsi stakeholder education l.jpg

FSI Stakeholder Education

Before Speaking Self-Assessment: 85% of applicants with self-assessed proficiency tested at 42% passing rate.

After Speaking Self-Assessment: 61% of applicants with self-assessed proficiency tested at 58% passing rate.


Quote 2 l.jpg

Quote 2

Nasra (Swahili language applicant):

“Oh yeah, that test won’t be a problem at all. It’s my mother tongue, so of course I’m a Level 5. I speak Swahili all the time. It’s not a big deal. You can test me right now, if you want. I’m ready.”


Testing misconception27 l.jpg

Testing Misconception

All native speakers are Level 5.

  • Native speaker as a social variable.

  • Native speaker as the ideal.


Typical scales l.jpg

Typical Scales

Satisfactory

Good

Excellent

Poor

Fair

C

B

A

F

D

0

50

100


Fbi opi data l.jpg

FBI OPI Data

  • Testing Native Speakers

    • 80% + of FBI Linguists are native or heritage speakers of the language.

  • Native Speaker Self Appraisal Scores

    • Native Language

      • Level 5: 70%

      • Level 4/4+: 10%

      • Level 3/3+: 20% (largely heritage speakers)

    • English

      • Level 5: 51%


Quote 3 l.jpg

Quote 3

James (Manager of a new national security initiative):

“We need at least 10 linguists dedicated to this assignment in Arabic alone. Since this matter is of highest importance, we need them all to be Level 5. It’s important that we have the best people on the assignment.”


Who would you hire as a foreign service officer l.jpg

Aleem:

heritage speaker of Saudi Arabic

went to Arabic classes at a private language school

Language school-Listening: 4

Language school-Reading: 4

Language school-Speaking-MSA: 3+

Richard:

native speaker of English

learned Arabic for 4 years while in college, (1 year in Qatar)

DLPT-Listening: 3

DLPT-Reading: 3

OPI-Speaking-MSA: 2

Who would you hire…As a Foreign Service Officer?


Who would you hire as a military interpreter l.jpg

Adrienne:

worked at an Iraqi refugee program for 1 year

DLPT-Listening: 2+

DLPT-Reading: 2

OPI-Speaking-MSA: 2+

Consecutive Interpretation Test-Iraqi/English: 2

Catherine:

worked as a translator for the United Nations for 2 years

DLPT-Listening: 2+

DLPT-Reading: 4

OPI-Speaking-MSA: 3

Translation Test: 3

Who would you hire…As a Military Interpreter?


Who would you hire as a foreign language monitor l.jpg

Jennifer:

native speaker of English

DLPT-Listening: 2

DLPT-Reading: 3

OPI-Speaking-MSA: 1+

Translation Test: 2

English Comp: 3+

Who would you hire…As a Foreign Language Monitor?

Mirna:

  • native speaker of Arabic

  • DLPT-Listening: 2+

  • DLPT-Reading: 2

  • OPI-Speaking-MSA: 2+

  • Translation Test: 2

  • English Comp: 2


Who would you hire as a msa oral proficiency interview tester l.jpg

Mike:

heritage speaker of Jordanian Arabic

OPI-Speaking-MSA: 3

OPI-Speaking-Jordanian: 3

OPI-Speaking-English: 3+

Imad:

native speaker of Jordanian Arabic

OPI-Speaking-MSA: 5

OPI-Speaking-Jordanian: 4

OPI-Speaking-English: 1

Who would you hire… As a MSA Oral Proficiency Interview Tester?


Who would you hire as a document translator l.jpg

Julie:

native speaker of English

DLPT-Listening: 2

DLPT-Reading: 3

OPI-Speaking-MSA: 1+

Translation Test: 2+

Hiam:

native speaker of Moroccan Arabic

DLPT-Listening: 3

DLPT-Reading: 3

OPI-Speaking-MSA: 4

English Composition: 3

Who would you hire… As a Document Translator?


Testing misconception36 l.jpg

Testing Misconception

Higher scores make better linguists.

  • Some job requirements do not necessitate high scores, or need high scores in only some skills.

  • Tests are usually not performance.

  • Other non-language skills may be required.


Audio sample l.jpg

Audio Sample

  • Native Speaker of English

  • College Graduate

  • Was in Peace Corps for 14 months in Uzbekistan


Educating management l.jpg

Educating Management

Stressing proficiency and performance

Understanding job requirements and working with HR to establish criteria


Quote 4 l.jpg

Quote 4

Debra, a newly hired Foreign Service Officer signed up for a test in Swedish, hoping to be assigned to Stockholm. The testing team reports that Debra understands Swedish quite well, but she speaks Norwegian.


Testing misconception40 l.jpg

Testing Misconception

I know what languages I can understand, speak, and write.

  • Mutually intelligible does not equal bilingual.

  • Languages are sometimes misidentified by examinees.

  • One skill ability does not equal all skills ability.


Educating stakeholders l.jpg

Educating Stakeholders

Ensure recruiters understand language requirements

Define language for testing purposes

Resources such as the NVTC site, Languages of the World web site


Quote 5 l.jpg

Quote 5

Hanke (Polish OPI Tester Applicant, has an ILR Speaking 5 on record, 10 years old):

“Why would I want to lower my score on record? Who and why decided that a Polish linguist working for the Bureau cannot score higher than a 4? And, if this applies only to Polish, why are the requirements lower for the Polish language?”


Testing misconception43 l.jpg

Testing Misconception

Test batteries and qualifying scores should be the same across languages for the same position.

All tests from a particular battery may not be available in a language.

Some tests may not have the same maximum possible score across languages.


Educating stakeholders44 l.jpg

Educating Stakeholders

Document testing standards

Back up decisions with research, where available

Ensure policies are documented and available to language professionals


Quote 6 l.jpg

Quote 6

Silke (German linguist who has to retest because of expired scores):

“I don’t see why I have to retest. I’m a 5. Even assuming that my command of the German language somewhat diminished over the years (which it has not), it could not have gone lower than a 4.”


Testing misconception46 l.jpg

Testing Misconception

Once a 4, always a 4.

Test scores are considered valid for varying timeframes, depending on the agency and the level scored

Regular retesting with new forms should be encouraged.

Score may reflect the highest score given from that exam, not the ‘true score’ of the examinee.


Educating stakeholders47 l.jpg

Educating Stakeholders

Research project underway at the Center for the Advanced Study of Languages to investigate testing periodicity.


Quote 7 l.jpg

Quote 7

Olga (Russian teacher):

“I’ve had a lot of experience in test development. I’ve been giving tests in my classroom for 20 years, and I give oral exams at the end of the class, so I could be an OPI tester. I don’t think that I even need to attend the training.”


Testing misconception49 l.jpg

Testing Misconception

The best teachers make the best testers.

Teachers sometimes have difficulty swapping hats from someone who fosters language learning to someone who is taking an impartial measurement.

True whether or not the examinee is the teacher’s student.


Educating stakeholders tester training l.jpg

Educating Stakeholders:Tester Training

Extensive testing training:

Up to 1 year (Tester or Examiner training)

Discuss FSI’s Testing Manual

Rate taped tests

Observe and discuss live tests

Conduct (unofficial) progress tests

Complete online program & mastery exercise

Conduct certification tests


Tester accountability l.jpg

Tester Accountability

Display sufficient understanding of testing principles!


Quote 8 l.jpg

Quote 8

Jennifer (Foreign Service applicant): “I took 2 semesters of Arabic, was at the top of my class, never missed an assignment, always contributed in class, got straight As and you guys tested me S0+.

Your test sucks!”


Testing misconception53 l.jpg

Testing Misconception

Consistent practice with material in the foreign language will result in a good score and regular improvement on ILR-based tests.

Improvement depends on consistent practice with increasingly complex language material.

ILR Level ranges are not equidistant.


Slide54 l.jpg

ILR Skill Level Descriptions

5

Level 5

4

Level 4

Level 3 Range

3

3

Level 3

2

Level 2

1

Level 1

0

Level 0


Quote 9 l.jpg

Quote 9

Anna

“I just applied to work at XYZ and they told me I had to pass their language test. How do I know if I’m at the right level to pass their test?”


Testing misconception56 l.jpg

Testing Misconception

USG language tests will tell me all that I need to know about the examinee.

  • Most ILR-based tests measure proficiency, not achievement.

  • Most USG tests are not diagnostic in nature.


Educating test takers l.jpg

Educating Test Takers

DoD Familiarization Guides

Online language training (LangNet, LingNet)

New Self Assessments

www.govtilr.org


Educating stakeholders what does level x mean l.jpg

Educating Stakeholders:What does Level X mean?

  • Through tester training, illustrate Levels

  • Provide professional development training throughout


The next big thing is us lev grossman time march 2006 l.jpg

The Next Big Thing is UsLev Grossman (Time March 2006)

It goes against everybody's inner cynic to read (or forthat matterto write) a sentence like the following: We are on the verge of the greatest age of creativity and innovation the world has ever known. It smacks of treacly dotcomism. It smacks of I Love the '90s. My inner cynic is a tiny bit queasy right now. But lately it's a conclusion I've had a hard time avoiding. Consider the following idea. Things, broadly speaking, used to be invented by a small, shadowy élite. This mysterious group might be called the People Who Happened to Be in the Room at the Time. These people might have been engineers, or sitcom writers, or chefs. They were probably very nice and might have even been very, very smart. But however smart they were, they're almost certainly no match for a less élite but much, much larger group: All the People Outside the Room.


Educating stakeholders the public l.jpg

Educating Stakeholders:The Public

  • ILR Website

    • www.govtilr.org

  • National Security Language Initiative

    “They are also evaluating initial results and outcomes to gain a more concrete understanding of program efficacy and best practices in foreign language instruction. Building on their program-specific work, the NSLI agencies are exploring other ways to advance and measure foreign language learning, such as enhanced assessments of language proficiency, and additional resources and materials for teachers” (Paper from August 2008)


Educating stakeholders additional online resources l.jpg

Educating Stakeholders:Additional Online Resources

Department of State Web site section on testing:

Overview

FSI Test

Notes to Examinee

DVC/Field tests

Skill Level Descriptions

FAQs

MLAT


Overview of presentation62 l.jpg

Overview of Presentation

Intuitive Language Testing

Testing Misconceptions

Educating Stakeholders

Continuing Challenges


Continuing challenges l.jpg

Continuing Challenges

  • Finding time an resources to ‘spread the word’

  • Continuing Interagency collaboration

  • Testing the LCTLs

  • Testing Listening

  • Testing Proficiency AND Performance

  • Testing across the ILR Scale


  • Login