Loading in 2 Seconds...
Loading in 2 Seconds...
An IPO Task Difficulty Matrix for Prototypical Tasks for Task-based Assessment. Sheila Shaoqian Luo School of Foreign Languages Beijing Normal university Sept 22 2007. The presentation structure …. introduction literature the rationale of the research
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Sheila Shaoqian Luo
School of Foreign Languages
Beijing Normal university
Sept 22 2007
Affect and attitudes
ability for use
(1) Organization: - Grammar; Text;
(2) Pragmatics: - illocution; sociolinguistics
(1) to inform task selection (to predict the relative difficulty of each task); (2) to ensure the full range of candidates’ ability will be tapped); (3) to assist test developers in structuring test tasks and the conditions under which these tasks are performed in appropriate ways; (4) to inform development of rating scale descriptors; (5) to facilitate interpretation of test scores (which may differ according to tasks)
Context of testing
Ability for Use
(Skehan, Dec. 2006)
If tasks are a relevant unit for testing, the research problem is to try to systematically “develop more refined measures of task difficulty” . (Skehan, 1998:80)
The Problem of Difficulty
(Skehan, Dec. 2006)
Weaknesses in previous findings on task difficulty: were of only moderate support for the proposed relationships between the combinations of cognitive factors with particular task types…
(Elder et al., 2002)
How can language ability in TBLT in mainland
Chinese middle schools best be assessed?
(1) competence-oriented underlying abilities;
(2) a structure made up of different interactive and inter-related components (Canale & Swain, 1980; Bachman, 1990);
(3) different performances drawing upon these underlying abilities (Bachman, 1990);
(4) sampling such underlying abilities in the comprehensive and systematic manner so to provide the basis for generalizing to non-testing situations.
matrix studies matrix
(1) A correlational analysis to explore the relationship between tasks and task difficulty components; and
Two research phases:
(1) Phase one: Study One~Study Four (March~May 2004)
Application of the Norris-Brown et al. task difficulty matrix
(2) Phase two: Study Five~Study Ten (Oct 2004~2005)
Establishing and evolution of the IPO task difficulty matrix
1 The IPO-CFS task difficulty scheme
2 CNEC-theme related tasks (Table3)
Personal information; Family, friends and people around; Personal environments; Daily routines; School life; Interests and hobbies; Emotions; Interpersonal relationships; Plans and intentions; Festivals, holidays and celebrations; Shopping; Food and drink; Health and fitness; Weather; Entertainment and sports; Travel and transport; Language learning; Nature; The world and the environment; Popular science and modern technology; Topical issues; History and geography; Society; Literature and art
(making oral/written expression more accurate and fluent)
(1) Encouraging correlations: all but one range from .52 to .83. The exceptional pair of .34leads to further data collection from both raters and students for the matrix reliability and validity.
(2) The matrix is improving, but needs input from actual raters; Inseparable Input, Processing, Output
(1) Study Eight correlation range: .69 to .92
(2) Study Nine correlation range: .62 to .91
(3) Study Ten correlation range: .75 to .87.
Dimensions: INPUT PROCESSING OUTPUT
II Language: Level of syntax; Level of vocabulary
III Performance Conditions:Modality; Time pressure
Primary research question; Similar purposes; similar design of matrix; an example of an assessment alternative; Sources
Test Objects; Task Themes; Task Focus; +(-)related to curriculum; Task Selection; Definitions/Labels; Characteristics; Layout; Rating System; Raters
(1) Estimating task difficulty: to use learner performances on sampled tasks to predict future performances on tasks that are constituted by related difficulty components. (Norris et al., 1998:58)
(2) Students with greater levels of underlying ability will be able to successfully complete tasks which come higher on such a scale of difficulty. (Skehan, 1998:184)
-- to promote the generalizability: more research needed in different regions in EFL contexts
This is a presentation based on my Ph.D. research under
the supervision of Professor Peter Skehan. My great gratitude
goes to my supervisor, Professor Skehan. I also thank my
committee members, Professor Jane Jackson and Professor
David Coniam at the Chinese University of Hong Kong, who
have contributed thoughtful and helpful suggestions to this
study. My thanks go to the the participants in the research.
Bachman, L. F. (2002). Some reflections on task-based language performance assessment. Language Testing, 19(4), 453-476.
Brown, J. D., Hudson, T., Norris, J. & Bonk, W. J. (2002). An investigation of second language task-based performance assessments. Second Language Teaching & Curriculum Center, University of Hawai’i at Manoa.
Coniam, D., & Falvey, P. (1999). Assessor training in a high-stakes test of speaking: the Hong Kong English language benchmarking initiative. Melbourne Papers in Language Testing, 8 (2), 1–19.
del Pilar Garcia Mayo, M. (Ed.). (2007). Investigating tasks in formal language learning. Clevedon: Multilingual Matters.
den Branden, K. V. (Ed.) (2006), Task-based language education: From theory to practice (pp. 1-16). Cambridge: Cambridge University Press.
Elder C., Iwashita N., & McNamara, T. (2002). Estimating the difficulty of oral proficiency tasks: What does the test-taker have to offer? Language Testing, 19,4, 343-368.
Elder, C., Knoch, U., Barkhuizen, G., & von Randow, J. (2005). Individual feedback to enhance rater training: Does it work? Language Assessment Quarterly, 2(3), 175-196.
Ellis, R. (2003). Task-based language learning and teaching. Oxford: Oxford University Press.
Ellis, R., & Barkhuizen, G. (2005). Analyzing learner language. Oxford: Oxford University Press.
Iwashita N., Elder C., & McNamara T. (2001). Can we predict task difficulty in an oral proficiency test? Exploring the potential of an information-processing approach to task design. Language Learning, 51(3), 401-436.
Knoch, U., Read, J., & Von Randow, J. (2006, June). Re-training writing raters online: How does it compare with face-to-face training?Paper presented at the 28th Annual Language Testing Research Colloquium of the International Language Testing Association, University of Melbourne, Australia (June 29, 2006).
Norris, J. M., Brown, J. D., Hudson, T. D., & Bonk, W. (2002). Examinee abilities and task difficulty in task-based second language performance assessment. Language Testing, 19(4), 395-418.
Nunan, D. (1993). Task-based syllabus design: Selecting, grading, and sequencing tasks. In G.. V. Crookes & S. M. Gass (Eds.), Tasks in a pedagogical context: Integrating theory and practice (pp. 55-68). Clevedon, Avon: Multilingual Matters.
Nunan, D. (2004). Task-Based Language Teaching. Cambridge: Cambridge University Press.
Robinson, P. (2001). Task complexity, task difficulty, and task production: Exploring interactions in a componential framework. Applied Linguistics, 22 (1), 27 – 57.
Skehan, P. (1996). A framework for the implementation of task-based instruction. Applied Linguistics, 17 (1), 38-62.
Skehan, P. (1998). A Cognitive approach to language learning. Oxford: Oxford University Press.
Skehan, P. (1999). The influence of task structure and processing conditions on narrative retellings. Language Learning, 49(1), 93–120.
Skehan, P. (2001). Tasks and language performance assessment. In M. Bygate, P. Skehan & M. Swain (Eds.), Researching pedagogic tasks: Second language learning teaching and testing (pp. 167-185). London: Longman.
Skehan, P. (2003). Task-based instruction. Language Teaching, 36(1), 1-14.
Skehan, P., & Foster, P. (1997). Task type and task processing conditions as influences on foreign language performance. Language Teaching Research, 1(3), 185–211.
Wolfe-Quintero, K., Inagaki, S., & Kim, H. (1998). Second language development in writing: Measures of fluency, accuracy & complexity. Second Language Teaching & Curriculum Center, Honolulu: University of Hawai‘i Press.