How t o Write a Reliable Listening Comprehension Test ?. Zhenlin Qiao Defense Language Institute April 16, 2009. Characteristics of oral communication Aspects of listening comprehension Defining the construct Approaches to assess listening Selecting text Creating tasks
How to Write a Reliable Listening Comprehension Test?
Defense Language Institute
April 16, 2009
Characteristics of oral communication
Aspects of listening comprehension
Defining the construct
Approaches to assess listening
Properties of a good test
Encoded in the form of sound, acoustic signal
Linear and takes place in real time with no chance to review
Linguistically different from written language
Stress and intonation to indicate clausal boundaries
Redundancy, hesitation, accent
Various speech rate
Listening comprehension is an active construction of meaning, and that this is done by applying knowledge to the incoming sound.
Declarative knowledge（Anderson, 1976）
Procedural knowledge（Anderson, 1976）
Spreading activation（Cole and Jakimik, 1980)
Processing idea unit（Sachs, 1967)
Processing connected discourse/cohesion
(Halliday and Hasan, 1976)
- propositional inference （Hildyardand Olson, 1978)
- enabling inference （Hildyardand Olson, 1978)
- pragmatic inference （Hildyardand Olson, 1978)
- bridging inference（Haviland and Clark 1974)
- elaborative inference
Script theory (Schank and Abelson, 1977)
Schemata (Rumelhart and Ortony, 1977；Rumelhart, 1980)
Sociolinguistic appropriacy (Hymes, 1972)
Pragmatic interpretation (Leech, 1983)
Speech acts (Austin, 1962; Searle, 1969)
Grice’s Maxim (Grice, 1975):
- maxim of quantity
- maxim of quality
- maxim of relation
- maxim of manner
Principles of analogy and minimal change (Brown and Yule, 1983)
The cognitive environment (Sperber and Wilson, 1986
The flow chart approach (Nagle and Sanders, 1986)
1) echoic memory
2) working memory
3) long-term memory
Mental modes: there are 3 main ways in which texts can be represented mentally (Johnson-Laird, 1983;1985):
1) as proposition
2) as mental models
3) or as image
The thing we are trying to measure is called construct (Buck, 2002).
Construct validity is to make a test somehow measures that construct (Buck, 2002).
- construct-irrelevant variance
General language proficiency
Collaborative or non-collaborative listening
Differing interpretations of texts
A construct based on the interaction between competence and task
A default listening construct
The knowledge, skills and abilities we think test takers should have.
Knowledge here refers to procedural knowledge- the ability to apply the knowledge in efficient and automatic language processing.
- comprehension processes
- storing and memory processes
- using and retrieval processes
- assessing the situation
Providing suitable texts
- levels of difficulty
Ensuring good quality sound
- speech rate
- background noise
Setting (place)/time of day
- time allotment
Format (long/short texts)
Language of input
a. use tasks that depend on knowledge that everyone has;
b. use tasks that depend on knowledge that no one has,
c. use tasks that depend on knowledge that has been provided in the test
Self-evident comprehension tasks (true/false)
Body movement tasks
General passage comprehension
Information transfer tasks (e.g. following direction on a map; filling in information in a grid)
The main problem is that it is very difficult to say with certainty that any inference is completely wrong.
Usefulnessincorporates the test’s measurement properties, as well as the social consequences of test use and the practicality of the test for its particular purpose.
It is a function of six properties: reliability, construct validity, authenticity, interactiveness, impact and practicality.
(Bachman and Palmer, 1996)
It is concerned with how accurately the test measures.
We should always get the same result testing the same person in the same situation.
The more important the decision based on the test, the more reliable the test needs to be.
It is the extent to which the test measures the right construct.