Download Policy: Content on the Website is provided to you AS IS for your information and personal use only and may not be sold or licensed nor shared on other sites. SlideServe reserves the right to change this policy at anytime. While downloading, If for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
1. Understanding Difficulty & complexity in writing tasks: Toward a useful analytic framework Anne Nebel
Lancaster University firstname.lastname@example.org
Georgetown University email@example.com
School of Foreign Service in Qatar
3. Writing assessment tasks My research investigates the difficulty and complexity of a set of 8 writing tasks
Not experimentally designed tasks
Fixed set of retired test tasks (ECCE)
Writing task option pair: letter or essay
4. Task choice? Challenge of task equivalence
Within each test and across versions
Potential for eliciting sufficient output equally rich in features under assessment
Raises validity & reliability issues
Fairness is a primary concern
What decisions and conclusions are reached from the test results? (Bachman & Palmer, forthcoming)
5. Research context (1) High-stakes nature of language proficiency assessment (e.g. Greece)
May impact performance (affective variables)
Over-trained and test-savvy
Test context as a facet in the complexity equation?
6. Research context (2) Low-stakes environments (e.g. Qatar)
Opportunity to look at test tasks outside the test context
Test-like conditions (timed, no resources, etc.)
None of the affective and pragmatic interference of a real test situation
7. My research: writing assessment tasks Perceptions of task difficulty
Student (N=20 pilot questionnaire and raking activity)
Teacher (N=5, 30-minute interviews & ranking activity)
Student (N=? interviews and raking activity)
Analysis of task complexity
8 task set of 4 pairs (letter & essay)
Various model adaptations toward the development of a framework
Analysis of performances on tasks
Measures of CAF (which, why)
Interactions between perceptions, task complexity and performance (what, how)
8. How to analyze difficulty & complexity? Expert judgments, VPAs, interviews, surveys and questionnaires.
(e.g. Elder, Iwashita and McNamara, 2002; Hamp-Lyons, 1994)
Existing models (Robinson, 2001, 2005; Skehan & Foster, 1999, 2001; and Ishikawa, 2008; Kuiken & Vedder, 2007)
Taxonomies of characteristics (Bachman & Palmer, 1990; Carr, 2006)
Coding & classification systems (Silver et al., 2009)
learner factors / perceptions COMPLEXITY
task structure / analysis
9. Student perceptions of difficulty (phase I) Task choice? letter (18), essay (2)
S1: I chose the letter because it gives me the way how to write and explain my opinion.
S4: Letter, because its easiest.
S14: Because I can explain a situation better in a letter than in the essay.
S17: Its easier to give an opinion on a letter (less formal) than writing an essay.
S18: In my view a letter is more personal and it is easy to write their own opinion.
S6*: Because I dont know how to write an essay.
10. View from the learner (Ortega, 2005) Whether through discourse analysis, psychological questionnaires, or post-task interviews, a process-product approach that considers the full landscape of variables contributed by task, leaner and linguistics outcomes, has the potential to illuminate our theories of the role of planning in task-based language learning (Ortega, 2005: 116).
of task complexity?
11. Approach to perceptions data (phase II) Phenomenography
Qualitative approach that makes use of in-depth interview data to explore how participants conceive of a phenomenon (Marton, 1981);
Reveals categories of understanding that can be ordered in an outcome space (Micari et al., 2007);
Characterizes ways of seeing something in terms of the critical aspects of the phenomenon (Pang, 2003: 145)
12. Theory and method PCP (Kelly, 1955) / Repertory Grid Technique
Personal construct theory is a theoretical framework that provides a vocabulary with which to interpret the constituents and processes by which a person construes his or her experience (Roberts, 1999:119).
Dyadic difference elicitation technique
Elicit constructs based on a comparison of elements.
There is nothing sacrosanct about the triad (Fransella, Bell and Bannister, 2004: 28).
13. Elements: 8 tasksConstruct elicited (sample): Specific / non-specific
Personal / general (private/public orientation Hamp-Lyons & Mathias,1994: 54-55)
Real life / artificial or hypothetical
Meaningful / meaningless
Concrete / general or vague
Specified content / need to invent content
Voc-ed domain / academic domain
1st person / 3rd person
Suggested structure / no suggestion of structure
14. Task ranking Teachers (N=5) ranked 8 tasks in order of difficulty.
In every case, task 6 was ranked as the most difficult or the second most difficult.
In every case (but one), task 1 was ranked as the easiest or the second easiest task of the set.
15. Task analysis (essay option, task pair 3)
Should public and private resources be used to support development in locations where natural disasters, such as floods, forest fires, and volcano eruptions, could occur? Explain your views. Give specific reasons and examples.
(letter option, task pair 1)
Write a letter to the Editor of the City Times. Name one person you think should go on the Millennium Exchange Program. Describe the person, and tell why you think this person should go. You may nominate yourself or someone else you know.
Task 6 (most difficult) Task 1 (easiest)
16. Task complexity analysis: constructs -specific
6 = more complex task
Task 6 (rebuild city with funds essay) Task 1 (Millennium nominee letter)
17. Task complexity analysis: (Silver et al., 2009) High cognitive demand
6=more complex task Low cognitive demand
Task 6 (rebuild city with funds essay)
Task 1 (Millennium nominee letter)
18. Task complexity analysis: LACM and TCF(compared in Kuiken & Vedder, 2007) Code complexity
Familiarity of topic
6=more complex task
Here and now
No reasoning demands
Task difficulty LACM (Skehan) TCF (Robinson)
19. Task complexity analysis: 1 adaptationfor speaking (Elder, Iwashita and McNamara, 2002) 2 performance conditions (+/-)
4 task dimensions:
Perspective: your own/anothers
Adequacy: complete/incomplete input
Planning time: [constant]
Which is more complex: 6? Equal? Inconclusive
20. Complex tasks, complex writing? Although they dont always take into account the same task dimensions, the models are capable of accounting for complexity in writing tasks; which dimensions are essential and how do we operationalize them?
Can the models predict the corresponding features of writing that will demonstrate complexity of thinking in writing? Which features and why?
In language testing, an equally important consideration is how well we can measure these features with a scoring rubric or scale.
21. References Bachman, L. and Palmer, A. (1990) Language Testing in Practice. Oxford: Oxford University Press.
Carr, N. (2006) The factor structure of test task characteristics and examinee performance. Language Testing 23, 269-289.
Chalhoub-Deville, M. (2001) Task based assessments: characteristics and validity evidence. In Bygate, M. Skehan, P. and Swain, M. (2001) Researching pedagogic tasks: second language learning, teaching and testing. Essex: Pearson Education Limited. Ch 10, pp. 210-228.
Elder, K, Iwashita, N. and McNamara, T. (2002) Estimating difficulty or oral proficiency tasks: what does the test-taker have to offer? Language Testing 19 (4); 347-368.
Fransella, F., Bell, R. and Bannister, D. (2004) A Manual for Repertory Grid Technique. West Sussex: John Wiley and Sons.
Hamp-Lyons, L. and Mathias, S. (1994) Examining expert judgments of task difficulty on essay tests. Journal of Second Language Writing 3 (1); 49-68.
Ishikawa, T. (2008) The effect of manipulating task complexity along the [+/- here and now] dimensions on L2 written narrative discourse. In M. Garcia May (ed) Investigating tasks in formal language learning. Clevedon, UK: Multilingual Matters.
Kelly, G. (1955) The psychology of personal constructs. New York: Norton.
Kuiken, F. and Vedder, I. (2007) Task complexity and measures of linguistics performance in L2 writing. IRAL 45, 261-284.
Micari, M., Light, G., Calkins, S. and Streetweiser (2007) Assessment beyond performance: phenomenography in educational evaluation. American Journal of Evaluation 28 (40); 458-476.
Marton, F. (1981) Phenomenography: describing conceptions of the world around us [online version]. Instructional Science 10, 177-200. Retrieved 2/12/08 from http://www.ped.gu.se/bjorn/phgraph/misc/constr/html)
22. References Ortega, L. (2005) What do learners plan? Leaner-driven attention to form during pre-task planning. In Ellis, R. (ed) Planning and task performance in a second language. Amsterdam/Philadelphia: John Benjamins. Pp. 77-110.
Pang, M. F. (2003) Two faces of variationon continuity in the phenomenographic movement. Scandinavian Journal of Educational Research 47 (2); 14-156.
Roberts, J. (1999) PCP as a framework for research into teacher learning and thinking. Language Teaching Research 3 (2); 117-144.
Robinson, P. (2001) Task complexity, task difficulty and task production: exploring interactions in a componential framework. Applied Linguistics 22, 27-57.
Robinson, P. (2005) Cognitive complexity and task sequencing: studies in a componential framework for second language task design. IRAL 43, 1-32.
Silver, E., Mesa, W., Morris, K., Star, J. and Benken, B. (2009) Teaching mathematics for understanding: an analysis of lessons submitted by teachers seeking NBPTS certification. American Educational Research Journal, 46 (2); 501-535.
Skehan and Foster (1999) The influence of task structure and processing conditions on narrative retellings. Language Learning 49, 93-100.
Skehan and Foster (2001) Cognition and tasks. In P. Robinson (ed) Cognition in second language instruction (pp. 185-205). Cambridge: Cambridge University Press.
Wigglesworth, g. (2001) Influences on performance in task-based oral assessments. In Bygate, M. Skehan, P. and Swain, M. (2001) Researching pedagogic tasks: second language learning, teaching and testing. Essex: Pearson Education Limited. Ch 9, pp. 186-209.
23. Anne Nebel
firstname.lastname@example.org Thank you.