Sophisticated tasks in e assessment l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 23


  • Uploaded on
  • Presentation posted in: General

SOPHISTICATED TASKS IN E-ASSESSMENT. WHAT ARE THEY? AND WHAT ARE THEIR BENEFITS?. Andrew Boyle Research & Statistics team QCA. Importance of e-assessment. Important in major government initiatives eLearning strategy White Paper on 14 – 19 education QCA’s e-assessment vision

Download Presentation


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Sophisticated tasks in e assessment l.jpg



Andrew Boyle

Research & Statistics team


Importance of e assessment l.jpg

Importance of e-assessment

  • Important in major government initiatives

    • eLearning strategy

    • White Paper on 14 – 19 education

    • QCA’s e-assessment vision

      • E-assessment commonplace by 2009

      • Currently developing KS3 ICT test

  • CAA well established in UK HE

Importance of sophisticated tasks l.jpg

Importance of sophisticated tasks

  • Substantially different task types in the near future

    • From pencil-and-paper tests and from early CAA

  • Tomlinson report/14 – 19 White Paper

    • E-assessment not limited to multiple-choice testing

    • Potential to test learners in both structured and unstructured environments

    • Both short and long answer questions

    • Ability to use techniques (sic) such as video clips

    • Assess a wider range of knowledge, skills and understanding than possible at present

Possible e assessment futures l.jpg

Possible e-assessment futures

  • ‘Intelligent measurement’

    • Sophisticated assessment methods (e.g. simulations) become more practicable

  • ‘Reinvented’ learning and assessment

    • Radically different assessment tasks

    • Integration of learning and assessment

    • Highly dynamic and adaptable content

Examples of sophisticated tasks l.jpg

Examples of sophisticated tasks

  • World Class tests

  • Key Stage 3 ICT test

  • TRIADs


  • Concept-mapping tasks

Taxonomies of e assessment tasks l.jpg

Taxonomies of e-assessment tasks

  • Parshall’s five-dimensional framework

    • Item format

    • Response action

    • Media inclusion

    • Level of interactivity

    • Scoring algorithm

Towards a definition l.jpg

Towards a definition?

  • Parshall

    • Five dimensions likely to be inter-related

    • Each dimension acts as a continuum

      • No simple-to-sophisticated cut points

  • Difficult to find a definition

    • Cf Mackenzie

  • Possible to suggest core features

Core features of sophisticated tasks l.jpg

Core features of sophisticated tasks

  • Contain media-rich stimulus material

    • Would rule out essays – even machine-marked

  • Test taker interacts with stimulus in various ways

    • Choosing responses (various ways)

    • Constructing responses (writing answers, plotting data points)

    • Directly demonstrating capability (simulations)

  • Test taker produces complex work

    • More complex than simple CAA, not necessarily more sophisticated than pencil-and-paper tests

Comments on core features l.jpg

Comments on core features

  • Must contain several features to be ‘sophisticated’

  • Notion of ‘sophistication’ always linked to other aspects – not just task design

    • Test model

    • Type of data produced

    • Marking

    • Reported information

    • Assessment purpose

  • Description of tasks as ‘sophisticated’ usually amounts to claim about merits of tasks/test

Possible benefits of sophisticated tasks l.jpg

Possible benefits of sophisticated tasks

  • Measuring different things

  • Addressing different assessment purposes

Measuring different things l.jpg

Measuring different things

  • Simple items based on a ‘dated form of psychology’

    • Learner as a ‘collector of facts and skills’; added to repertoire independently

    • At odds with recent understanding of learning and experts’ cognition

  • Conventional test models at odds with test takers’ cognitive strategies

    • Success in tests depends on at least two dimensions (accuracy and speed)

      • Test-taking strategies

    • Models should account for varied sources of difficulty

Measuring different things 2 l.jpg

Measuring different things (2)

  • Curriculum coverage

    • ‘Higher-order learning outcomes (HLOs)’

      • Valued in curriculums, but not easily tested by simple item formats

      • Simple-item tests might not represent the curriculum adequately

    • Particularly important if modern curriculums emphasise new types of abilities, etc.

      • ‘Shared rhetoric in education’

      • Emphasis on: Maths and Science; ICT; problem solving and communication

Definition of ict capability l.jpg

Definition of ICT capability

  • Construct measured in KS3 ICT test

    • Technical and cognitive proficiency to access, use and communicate information using technological tools

    • Purposefully applying technology to solve problems, analyse information, develop ideas, create models and exchange information

    • Discriminating in use of information and ICT tools

  • Contrasted with ICT skills

    • Technical competences to carry out tasks using common software applications

Challenge for sophisticated task users l.jpg

Challenge for sophisticated-task users

  • Sophisticated tasks depart from outdated psychological models

    • Need for new mental models

    • Significant challenge to researchers

    • Account for implications of sophisticated tasks

      • When specifying assessment model

      • During validation

Threats to validity from sophisticated tasks l.jpg

Threats to validityfrom sophisticated tasks

  • Construct-irrelevant variance

    • Sophistication and complexity in user interfaces - ‘nuisance variables’

    • Detract from validity

  • Differential impact of new task styles

    • Learners of differing backgrounds

    • Different cognitive styles/dispositions

Different assessment purposes l.jpg

Different assessment purposes

  • Substantial strand of recent research addresses assessment purposes

    • Formative assessment can improve learning

    • Formative assessment wide-reaching notion

      • Applies to formal test instruments and to day-to-day teaching techniques (e.g. questioning styles)

Formative e assessment l.jpg

Formative e-assessment

  • Potential of e-assessment to provide useful feedback on aspects of sophisticated tasks

  • Well-designed formative information more supportive of learning than simple right|wrong

  • Varied presentation styles

    • Providing feedback in range of visual and interactive modes

    • Might include a wider range of learning styles

Limits to the optimism 1 l.jpg

Limits to the optimism (1)

  • E-assessment is expensive to develop

    • Costs of e-assessment shifted towards front of cycle

    • Large budgets necessary for test development

    • Especially true of sophisticated tasks

  • Large organisations tend to have most resource to commit to big projects

    • Generally focused on summative assessment

    • Often conservative, risk averse

  • Makes use of sophisticated tasks for innovative formative assessment less likely

Limits to the optimism 2 l.jpg

Limits to the optimism (2)

  • Writing good test items is difficult.

    • Good quality items take a long time to produce; iterations of reviewing, editing, etc.

    • Teachers don’t necessarily have item writing skills

    • Claim that difficulty in writing good items is particularly pronounced for sophisticated tasks

  • Sophisticated tasks

    • Expensive

    • Slow to develop

    • Not easily written by non-specialist

    • May be difficult for teachers and learners to ‘own’

    • Lack key characteristic of formative assessment

      • Assessment closely integrated with classroom practice

Discussion l.jpg


  • No clear-cut definition of ‘sophisticated tasks’

    • Device to distinguish from alleged reactionary use of simple question styles in early CAA

  • Critique of benefits of sophisticated tasks

    • Implies a position of informed scepticism

    • Central assertion is accepted

      • Sophisticated tasks will soon be ubiquitous

      • They will bring real benefits

      • Benefits have been asserted rather than proven

      • Need to move to informed implementation

Discussion 2 the two benefits l.jpg

Discussion (2) – the two benefits

  • Assessment of HLOs

  • Formative assessment

    • Often confounded – they are distinct

  • Argument stronger in case of HLOs

    • Empirical evidence in near future

  • Sophisticated tasks’ ability to facilitate formative assessment – less secure

    • Difficulty in authoring tasks – continue to be created by large central organisations

    • Formative assessment ought be to low-tech – usable in everyday classrooms

    • Use of day-to-day software applications for formative assessment – e.g. track changes in word processors

Other work to do on sophisticated tasks l.jpg

Other work to do onsophisticated tasks

  • Deriving scores for each task

  • Measurement models

    • summarise scores

    • provide meaningful information about performance/ability, etc.

  • Validation of tests

Mackenzie definition l.jpg

Mackenzie definition

  • Advanced Computer-Based Assessment …

    • goes beyond the simple multiple-choice/response item types into the area of complex question types, adaptive branching, scenarios and simulations of real life situations or problems. In formative mode, feedback may be immediate, extensive, context-sensitive and include annotation of diagrams, links to web resources or full courseware/e-learning tutorials making it an extremely powerful learning tool.

  • Login