understanding difficulty complexity in writing tasks toward a useful analytic framework
Skip this Video
Download Presentation
Understanding Difficulty & complexity in writing tasks: Toward a useful analytic framework

Loading in 2 Seconds...

play fullscreen
1 / 23

Understanding Difficulty complexity in writing tasks: Toward a useful analytic framework - PowerPoint PPT Presentation

  • Uploaded on

TBLT 2009 Lancaster. Understanding Difficulty & complexity in writing tasks: Toward a useful analytic framework. Anne Nebel Lancaster University [email protected] Georgetown University [email protected] School of Foreign Service in Qatar. TBLT research.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Understanding Difficulty complexity in writing tasks: Toward a useful analytic framework' - benjamin

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
understanding difficulty complexity in writing tasks toward a useful analytic framework

TBLT 2009 Lancaster

Understanding Difficulty & complexity in writing tasks: Toward a useful analytic framework

Anne Nebel

Lancaster University [email protected]

Georgetown University [email protected]

School of Foreign Service in Qatar

tblt research
TBLT research
  • Classroom context (sequencing tasks for learning)
  • Speaking tasks
  • Assessment context (Chalhoub-Deville, 2001; Elder, Iwashita & McNamara, 2002; Wigglesworth, 2001)
  • Writing tasks(Ishikawa, 2008; Kuiken & Vedder, 2007)
  • Predominant models developed for speaking (Robinson, 2001, 2005; Skehan & Foster, 1999, 2001)
writing assessment tasks
Writing assessment tasks
  • My research investigates the difficulty and complexity of a set of 8 writing tasks
    • Not experimentally designed tasks
    • Fixed set of retired test tasks (ECCE)
    • Writing task option pair: letter or essay
task choice
Task choice?
  • Challenge of task equivalence
    • Within each test and across versions
    • Potential for eliciting sufficient output equally rich in features under assessment
  • Raises validity & reliability issues
    • Fairness is a primary concern
    • What decisions and conclusions are reached from the test results? (Bachman & Palmer, forthcoming)
research context 1
Research context (1)
  • High-stakes nature of language proficiency assessment (e.g. Greece)
    • Employment opportunities
    • Career advancement
    • Pay scale
    • May impact performance (affective variables)
      • Over-trained and test-savvy
    • Test context as a facet in the complexity equation?
research context 2
Research context (2)
  • Low-stakes environments (e.g. Qatar)
    • Opportunity to look at test tasks outside the test context
    • Test-like conditions (timed, no resources, etc.)
    • None of the affective and pragmatic interference of a real test situation
      • Motivation?
my research writing assessment tasks
My research: writing assessment tasks
  • Perceptions of task difficulty
    • Student (N=20 pilot questionnaire and raking activity)
    • Teacher (N=5, 30-minute interviews & ranking activity)
    • Student (N=? interviews and raking activity)
  • Analysis of task complexity
    • 8 task set of 4 pairs (letter & essay)
    • Various model adaptations – toward the development of a framework
  • Analysis of performances on tasks
    • Measures of CAF (which, why)
    • Interactions between perceptions, task complexity and performance (what, how)
how to analyze difficulty complexity
How to analyze difficulty & complexity?


learner factors / perceptions


task structure / analysis

  • Expert judgments, VPAs, interviews, surveys and questionnaires.

(e.g. Elder, Iwashita and McNamara, 2002; Hamp-Lyons, 1994)

  • Existing models (Robinson, 2001, 2005; Skehan & Foster, 1999, 2001; and Ishikawa, 2008; Kuiken & Vedder, 2007)
  • Taxonomies of characteristics (Bachman & Palmer, 1990; Carr, 2006)
  • Coding & classification systems (Silver et al., 2009)
student perceptions of difficulty phase i
Student perceptions of difficulty (phase I)

Task choice? letter (18), essay (2)

S1: “I chose the letter because it gives me the way how to write and explain my opinion.”

S4: “Letter, because it’s easiest.”

S14: “Because I can explain a situation better in a letter than in the essay.”

S17: “It’s easier to give an opinion on a letter (less formal) than writing an essay.”

S18: “In my view a letter is more personal and it is easy to write their own opinion.”

S6*: “Because I don’t know how to write an essay.”

view from the learner ortega 2005
“View from the learner”(Ortega, 2005)
  • “Whether through discourse analysis, psychological questionnaires, or post-task interviews, a process-product approach that considers the full landscape of variables contributed by task, leaner and linguistics outcomes, has the potential to illuminate our theories of the role of planning in task-based language learning” (Ortega, 2005: 116).
  • …of task complexity?
approach to perceptions data phase ii
Approach to perceptions data (phase II)
  • Phenomenography

Qualitative approach that makes use of in-depth interview data to explore how participants conceive of a phenomenon (Marton, 1981);

  • Reveals “categories of understanding” that can be ordered in an outcome space (Micari et al., 2007);
  • Characterizes “ways of seeing something in terms of the critical aspects of the phenomenon” (Pang, 2003: 145)
theory and method
Theory and method
  • PCP (Kelly, 1955) / Repertory Grid Technique

Personal construct theory is a theoretical framework that “provides a vocabulary with which to interpret the constituents and processes by which a person construes his or her experience” (Roberts, 1999:119).

  • Dyadic difference elicitation technique

Elicit constructs based on a comparison of elements.

“There is nothing sacrosanct about the triad” (Fransella, Bell and Bannister, 2004: 28).

elements 8 tasks construct elicited sample
Elements: 8 tasksConstruct elicited (sample):
  • Specific / non-specific
  • Personal / general (“private/public orientation” Hamp-Lyons & Mathias,1994: 54-55)
  • Real life / artificial or hypothetical
  • Meaningful / meaningless
  • Concrete / general or vague
  • Specified content / need to invent content
  • Voc-ed domain / academic domain
  • 1st person / 3rd person
  • Suggested structure / no suggestion of structure
task ranking
Task ranking
  • Teachers (N=5) ranked 8 tasks in order of difficulty.
  • In every case, task 6 was ranked as the most difficult or the second most difficult.
  • In every case (but one), task 1 was ranked as the easiest or the second easiest task of the set.
task analysis
Task analysis

Task 6 (most difficult)

Task 1 (easiest)

(essay option, task pair 3)

Should public and private resources be used to support development in locations where natural disasters, such as floods, forest fires, and volcano eruptions, could occur? Explain your views. Give specific reasons and examples.


(letter option, task pair 1)

Write a letter to the Editor of the City Times. Name one person you think should go on the Millennium Exchange Program. Describe the person, and tell why you think this person should go. You may nominate yourself or someone else you know.


task complexity analysis constructs
Task complexity analysis: constructs

Task 6 (rebuild city with funds essay)

Task 1 (Millennium nominee letter)



-concrete (abstract)


6 = more complex task

+ specific

+ personal

+ concrete

+ meaningful

task complexity analysis silver et al 2009
Task complexity analysis: (Silver et al., 2009)

Task 6 (rebuild city with funds essay)

Task 1 (Millennium nominee letter)

  • High cognitive demand
    • analyze
    • evaluate
    • interpret
    • create
    • justify
    • make decisions
  • 6=more complex task
  • Low cognitive demand
    • recall
    • remember
    • describe
    • guided
    • routine
    • constrained
task complexity analysis lacm and tcf compared in kuiken vedder 2007
Task complexity analysis: LACM and TCF(compared in Kuiken & Vedder, 2007)

LACM (Skehan)

TCF (Robinson)

  • Code complexity
    • Vocab
  • Cognitive complexity
    • Familiarity of topic
    • Predictability
    • Information organization

6=more complex task

  • Resource directing
    • Few elements
    • Here and now
    • No reasoning demands
  • Resource dispersing
  • Single task
  • Prior knowledge
  • Task difficulty
task complexity analysis 1 adaptation for speaking elder iwashita and mcnamara 2002
Task complexity analysis: 1 adaptationfor speaking (Elder, Iwashita and McNamara, 2002)

2 performance conditions (+/-)

4 task dimensions:

  • Perspective: your own/another’s
  • Immediacy: there-then/here-now
  • Adequacy: complete/incompleteinput
  • Planning time: [constant]

Which is more complex: 6? Equal? Inconclusive

complex tasks complex writing
Complex tasks, complex writing?
  • Although they don’t always take into account the same task dimensions, the models are capable of accounting for complexity in writing tasks; which dimensions are essential and how do we operationalize them?
  • Can the models predict the corresponding features of writing that will demonstrate complexity of thinking in writing? Which features and why?
  • In language testing, an equally important consideration is how well we can measure these features with a scoring rubric or scale.

Bachman, L. and Palmer, A. (1990) Language Testing in Practice. Oxford: Oxford University Press.

Carr, N. (2006) The factor structure of test task characteristics and examinee performance. Language Testing 23, 269-289.

Chalhoub-Deville, M. (2001) Task –based assessments: characteristics and validity evidence. In Bygate, M. Skehan, P. and Swain, M. (2001) Researching pedagogic tasks: second language learning, teaching and testing. Essex: Pearson Education Limited. Ch 10, pp. 210-228.

Elder, K, Iwashita, N. and McNamara, T. (2002) Estimating difficulty or oral proficiency tasks: what does the test-taker have to offer? Language Testing 19 (4); 347-368.

Fransella, F., Bell, R. and Bannister, D. (2004) A Manual for Repertory Grid Technique. West Sussex: John Wiley and Sons.

Hamp-Lyons, L. and Mathias, S. (1994) Examining expert judgments of task difficulty on essay tests. Journal of Second Language Writing 3 (1); 49-68.

Ishikawa, T. (2008) The effect of manipulating task complexity along the [+/- here and now] dimensions on L2 written narrative discourse. In M. Garcia May (ed) Investigating tasks in formal language learning. Clevedon, UK: Multilingual Matters.

Kelly, G. (1955) The psychology of personal constructs. New York: Norton.

Kuiken, F. and Vedder, I. (2007) Task complexity and measures of linguistics performance in L2 writing. IRAL 45, 261-284.

Micari, M., Light, G., Calkins, S. and Streetweiser (2007) Assessment beyond performance: phenomenography in educational evaluation. American Journal of Evaluation 28 (40); 458-476.

Marton, F. (1981) Phenomenography: describing conceptions of the world around us [online version]. Instructional Science 10, 177-200. Retrieved 2/12/08 from http://www.ped.gu.se/bjorn/phgraph/misc/constr/html)


Ortega, L. (2005) What do learners plan? Leaner-driven attention to form during pre-task planning. In Ellis, R. (ed) Planning and task performance in a second language. Amsterdam/Philadelphia: John Benjamins. Pp. 77-110.

Pang, M. F. (2003) Two faces of variation—on continuity in the phenomenographic movement. Scandinavian Journal of Educational Research 47 (2); 14-156.

Roberts, J. (1999) PCP as a framework for research into teacher learning and thinking. Language Teaching Research 3 (2); 117-144.

Robinson, P. (2001) Task complexity, task difficulty and task production: exploring interactions in a componential framework. Applied Linguistics 22, 27-57.

Robinson, P. (2005) Cognitive complexity and task sequencing: studies in a componential framework for second language task design. IRAL 43, 1-32.

Silver, E., Mesa, W., Morris, K., Star, J. and Benken, B. (2009) Teaching mathematics for understanding: an analysis of lessons submitted by teachers seeking NBPTS certification. American Educational Research Journal, 46 (2); 501-535.

Skehan and Foster (1999) The influence of task structure and processing conditions on narrative retellings. Language Learning 49, 93-100.

Skehan and Foster (2001) Cognition and tasks. In P. Robinson (ed) Cognition in second language instruction (pp. 185-205). Cambridge: Cambridge University Press.

Wigglesworth, g. (2001) Influences on performance in task-based oral assessments. In Bygate, M. Skehan, P. and Swain, M. (2001) Researching pedagogic tasks: second language learning, teaching and testing. Essex: Pearson Education Limited. Ch 9, pp. 186-209.