Educating the Stakeholders. ECOLT 2008 November 7, 2008 ILR Testing Committee. Overview of Presentation. Intuitive Language Testing Testing Misconceptions Educating Stakeholders Continuing Challenges. Why educate stakeholders?.
November 7, 2008
ILR Testing Committee
ILR Level 2
Proficiency in what?
What type of Arabic?
Modern Standard Arabic
“Five years after Arab terrorists attacked the United States, only 33 FBI agents have even a limited proficiency in Arabic …”Eggen, D. (2006, October 11). FBI agents still lacking Arabic skills. Washington Post. p A01
Were all the agents tested?
Most agents do not have a language requirement.
What is meant by “any familiarity?”
Speaking Modern Standard Arabic
What does a score of zero mean?
“Counting agents who know only a handful of Arabic words -- including those who scored zero on the standard proficiency test -- just 1 percent of the FBI’s 12,000 agents have any familiarity with the language, the statistics show.”
What is ‘advanced professional proficiency?’
ILR Level 4
The tests were administered by the ILR?
The ILR does not administer tests.
“Youssef … is one of only six FBI agents who scored a 4 for ‘advanced professional proficiency’ in Arabic on standardized speaking tests administered by the Interagency Language Roundtable for federal agencies.”
Who is interpreting the language testing statistics?
An attorney for a former linguist, suing the FBI.
What language test was given?
Speaking, not Translation
Who do agents rely on?
Language Analysts and Intelligence Analysts
Tested in Reading, Listening, Translation, Speaking
“Youssef’s attorney, Stephen M. Kohn, said the statistics indicate that most FBI agents have no way to gauge the accuracy of translated materials and must rely on linguists or other third parties for their information.”
Language testers shall be mindful of their obligations to the society within which they work, while recognising that those obligations may on occasion conflict with their responsibilities to their test takers and to other stakeholders.
Intuitive Language Testing
Hassan (Foreign Service Specialist):
“I worked for an NGO in Mosul and helped several US-based companies set up their computer systems. I constantly talked to vendors, made arrangements to access the appropriate routers, negotiated bandwidth for their Internet hubs, the works! In my spare time, I talked to my Iraqi buddies about all sorts of things. They were great at tipping me off whenever good stuff came at the local grocery store. I thought I was a 5 in speaking Iraqi Arabic. You scored me a 2. Need I tell you what I think about your test?”
A 3 is a 3 is a 3.
The following self assessment used by permission, Dr. Ray Clifford, ACTFL 2007
Information on the Skill Level Descriptions available online at www.govtilr.org
Self appraisals are suggested/required for some language positions
Formal presentations on language testing given to new entrants
Before Speaking Self-Assessment: 85% of applicants with self-assessed proficiency tested at 42% passing rate.
After Speaking Self-Assessment: 61% of applicants with self-assessed proficiency tested at 58% passing rate.
Nasra (Swahili language applicant):
“Oh yeah, that test won’t be a problem at all. It’s my mother tongue, so of course I’m a Level 5. I speak Swahili all the time. It’s not a big deal. You can test me right now, if you want. I’m ready.”
All native speakers are Level 5.
James (Manager of a new national security initiative):
“We need at least 10 linguists dedicated to this assignment in Arabic alone. Since this matter is of highest importance, we need them all to be Level 5. It’s important that we have the best people on the assignment.”
Aleem: Arabic skills.
heritage speaker of Saudi Arabic
went to Arabic classes at a private language school
Language school-Listening: 4
Language school-Reading: 4
Language school-Speaking-MSA: 3+
native speaker of English
learned Arabic for 4 years while in college, (1 year in Qatar)
OPI-Speaking-MSA: 2Who would you hire…As a Foreign Service Officer?
Adrienne: Arabic skills.
worked at an Iraqi refugee program for 1 year
Consecutive Interpretation Test-Iraqi/English: 2
worked as a translator for the United Nations for 2 years
Translation Test: 3Who would you hire…As a Military Interpreter?
Jennifer: Arabic skills.
native speaker of English
Translation Test: 2
English Comp: 3+Who would you hire…As a Foreign Language Monitor?
Mike: Arabic skills.
heritage speaker of Jordanian Arabic
native speaker of Jordanian Arabic
OPI-Speaking-English: 1Who would you hire… As a MSA Oral Proficiency Interview Tester?
Julie: Arabic skills.
native speaker of English
Translation Test: 2+
native speaker of Moroccan Arabic
English Composition: 3Who would you hire… As a Document Translator?
Higher scores make better linguists.
Stressing proficiency and performance
Understanding job requirements and working with HR to establish criteria
Debra, a newly hired Foreign Service Officer signed up for a test in Swedish, hoping to be assigned to Stockholm. The testing team reports that Debra understands Swedish quite well, but she speaks Norwegian.
I know what languages I can understand, speak, and write.
Ensure recruiters understand language requirements
Define language for testing purposes
Resources such as the NVTC site, Languages of the World web site
Hanke (Polish OPI Tester Applicant, has an ILR Speaking 5 on record, 10 years old):
“Why would I want to lower my score on record? Who and why decided that a Polish linguist working for the Bureau cannot score higher than a 4? And, if this applies only to Polish, why are the requirements lower for the Polish language?”
Test batteries and qualifying scores should be the same across languages for the same position.
All tests from a particular battery may not be available in a language.
Some tests may not have the same maximum possible score across languages.
Document testing standards
Back up decisions with research, where available
Ensure policies are documented and available to language professionals
Silke (German linguist who has to retest because of expired scores):
“I don’t see why I have to retest. I’m a 5. Even assuming that my command of the German language somewhat diminished over the years (which it has not), it could not have gone lower than a 4.”
Once a 4, always a 4.
Test scores are considered valid for varying timeframes, depending on the agency and the level scored
Regular retesting with new forms should be encouraged.
Score may reflect the highest score given from that exam, not the ‘true score’ of the examinee.
Research project underway at the Center for the Advanced Study of Languages to investigate testing periodicity.
Olga (Russian teacher):
“I’ve had a lot of experience in test development. I’ve been giving tests in my classroom for 20 years, and I give oral exams at the end of the class, so I could be an OPI tester. I don’t think that I even need to attend the training.”
The best teachers make the best testers.
Teachers sometimes have difficulty swapping hats from someone who fosters language learning to someone who is taking an impartial measurement.
True whether or not the examinee is the teacher’s student.
Extensive testing training:
Up to 1 year (Tester or Examiner training)
Discuss FSI’s Testing Manual
Rate taped tests
Observe and discuss live tests
Conduct (unofficial) progress tests
Complete online program & mastery exercise
Conduct certification tests
Display sufficient understanding of testing principles!
Jennifer (Foreign Service applicant): “I took 2 semesters of Arabic, was at the top of my class, never missed an assignment, always contributed in class, got straight As and you guys tested me S0+.
Your test sucks!”
Consistent practice with material in the foreign language will result in a good score and regular improvement on ILR-based tests.
Improvement depends on consistent practice with increasingly complex language material.
ILR Level ranges are not equidistant.
ILR Skill Level Descriptions Arabic skills.
Level 3 Range
“I just applied to work at XYZ and they told me I had to pass their language test. How do I know if I’m at the right level to pass their test?”
USG language tests will tell me all that I need to know about the examinee.
DoD Familiarization Guides
Online language training (LangNet, LingNet)
New Self Assessments
It goes against everybody's inner cynic to read (or forthat matterto write) a sentence like the following: We are on the verge of the greatest age of creativity and innovation the world has ever known. It smacks of treacly dotcomism. It smacks of I Love the '90s. My inner cynic is a tiny bit queasy right now. But lately it's a conclusion I've had a hard time avoiding. Consider the following idea. Things, broadly speaking, used to be invented by a small, shadowy élite. This mysterious group might be called the People Who Happened to Be in the Room at the Time. These people might have been engineers, or sitcom writers, or chefs. They were probably very nice and might have even been very, very smart. But however smart they were, they're almost certainly no match for a less élite but much, much larger group: All the People Outside the Room.
“They are also evaluating initial results and outcomes to gain a more concrete understanding of program efficacy and best practices in foreign language instruction. Building on their program-specific work, the NSLI agencies are exploring other ways to advance and measure foreign language learning, such as enhanced assessments of language proficiency, and additional resources and materials for teachers” (Paper from August 2008)
Department of State Web site section on testing:
Notes to Examinee
Skill Level Descriptions
Intuitive Language Testing