sophisticated tasks in e assessment
Skip this Video
Download Presentation

Loading in 2 Seconds...

play fullscreen
1 / 23


  • Uploaded on

SOPHISTICATED TASKS IN E-ASSESSMENT. WHAT ARE THEY? AND WHAT ARE THEIR BENEFITS?. Andrew Boyle Research & Statistics team QCA. Importance of e-assessment. Important in major government initiatives eLearning strategy White Paper on 14 – 19 education QCA’s e-assessment vision

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'SOPHISTICATED TASKS IN E-ASSESSMENT' - craig

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
sophisticated tasks in e assessment



Andrew Boyle

Research & Statistics team


importance of e assessment
Importance of e-assessment
  • Important in major government initiatives
    • eLearning strategy
    • White Paper on 14 – 19 education
    • QCA’s e-assessment vision
      • E-assessment commonplace by 2009
      • Currently developing KS3 ICT test
  • CAA well established in UK HE
importance of sophisticated tasks
Importance of sophisticated tasks
  • Substantially different task types in the near future
    • From pencil-and-paper tests and from early CAA
  • Tomlinson report/14 – 19 White Paper
    • E-assessment not limited to multiple-choice testing
    • Potential to test learners in both structured and unstructured environments
    • Both short and long answer questions
    • Ability to use techniques (sic) such as video clips
    • Assess a wider range of knowledge, skills and understanding than possible at present
possible e assessment futures
Possible e-assessment futures
  • ‘Intelligent measurement’
    • Sophisticated assessment methods (e.g. simulations) become more practicable
  • ‘Reinvented’ learning and assessment
    • Radically different assessment tasks
    • Integration of learning and assessment
    • Highly dynamic and adaptable content
examples of sophisticated tasks
Examples of sophisticated tasks
  • World Class tests
  • Key Stage 3 ICT test
  • TRIADs
  • Concept-mapping tasks
taxonomies of e assessment tasks
Taxonomies of e-assessment tasks
  • Parshall’s five-dimensional framework
    • Item format
    • Response action
    • Media inclusion
    • Level of interactivity
    • Scoring algorithm
towards a definition
Towards a definition?
  • Parshall
    • Five dimensions likely to be inter-related
    • Each dimension acts as a continuum
      • No simple-to-sophisticated cut points
  • Difficult to find a definition
    • Cf Mackenzie
  • Possible to suggest core features
core features of sophisticated tasks
Core features of sophisticated tasks
  • Contain media-rich stimulus material
    • Would rule out essays – even machine-marked
  • Test taker interacts with stimulus in various ways
    • Choosing responses (various ways)
    • Constructing responses (writing answers, plotting data points)
    • Directly demonstrating capability (simulations)
  • Test taker produces complex work
    • More complex than simple CAA, not necessarily more sophisticated than pencil-and-paper tests
comments on core features
Comments on core features
  • Must contain several features to be ‘sophisticated’
  • Notion of ‘sophistication’ always linked to other aspects – not just task design
    • Test model
    • Type of data produced
    • Marking
    • Reported information
    • Assessment purpose
  • Description of tasks as ‘sophisticated’ usually amounts to claim about merits of tasks/test
possible benefits of sophisticated tasks
Possible benefits of sophisticated tasks
  • Measuring different things
  • Addressing different assessment purposes
measuring different things
Measuring different things
  • Simple items based on a ‘dated form of psychology’
    • Learner as a ‘collector of facts and skills’; added to repertoire independently
    • At odds with recent understanding of learning and experts’ cognition
  • Conventional test models at odds with test takers’ cognitive strategies
    • Success in tests depends on at least two dimensions (accuracy and speed)
      • Test-taking strategies
    • Models should account for varied sources of difficulty
measuring different things 2
Measuring different things (2)
  • Curriculum coverage
    • ‘Higher-order learning outcomes (HLOs)’
      • Valued in curriculums, but not easily tested by simple item formats
      • Simple-item tests might not represent the curriculum adequately
    • Particularly important if modern curriculums emphasise new types of abilities, etc.
      • ‘Shared rhetoric in education’
      • Emphasis on: Maths and Science; ICT; problem solving and communication
definition of ict capability
Definition of ICT capability
  • Construct measured in KS3 ICT test
    • Technical and cognitive proficiency to access, use and communicate information using technological tools
    • Purposefully applying technology to solve problems, analyse information, develop ideas, create models and exchange information
    • Discriminating in use of information and ICT tools
  • Contrasted with ICT skills
    • Technical competences to carry out tasks using common software applications
challenge for sophisticated task users
Challenge for sophisticated-task users
  • Sophisticated tasks depart from outdated psychological models
    • Need for new mental models
    • Significant challenge to researchers
    • Account for implications of sophisticated tasks
      • When specifying assessment model
      • During validation
threats to validity from sophisticated tasks
Threats to validityfrom sophisticated tasks
  • Construct-irrelevant variance
    • Sophistication and complexity in user interfaces - ‘nuisance variables’
    • Detract from validity
  • Differential impact of new task styles
    • Learners of differing backgrounds
    • Different cognitive styles/dispositions
different assessment purposes
Different assessment purposes
  • Substantial strand of recent research addresses assessment purposes
    • Formative assessment can improve learning
    • Formative assessment wide-reaching notion
      • Applies to formal test instruments and to day-to-day teaching techniques (e.g. questioning styles)
formative e assessment
Formative e-assessment
  • Potential of e-assessment to provide useful feedback on aspects of sophisticated tasks
  • Well-designed formative information more supportive of learning than simple right|wrong
  • Varied presentation styles
    • Providing feedback in range of visual and interactive modes
    • Might include a wider range of learning styles
limits to the optimism 1
Limits to the optimism (1)
  • E-assessment is expensive to develop
    • Costs of e-assessment shifted towards front of cycle
    • Large budgets necessary for test development
    • Especially true of sophisticated tasks
  • Large organisations tend to have most resource to commit to big projects
    • Generally focused on summative assessment
    • Often conservative, risk averse
  • Makes use of sophisticated tasks for innovative formative assessment less likely
limits to the optimism 2
Limits to the optimism (2)
  • Writing good test items is difficult.
    • Good quality items take a long time to produce; iterations of reviewing, editing, etc.
    • Teachers don’t necessarily have item writing skills
    • Claim that difficulty in writing good items is particularly pronounced for sophisticated tasks
  • Sophisticated tasks
    • Expensive
    • Slow to develop
    • Not easily written by non-specialist
    • May be difficult for teachers and learners to ‘own’
    • Lack key characteristic of formative assessment
      • Assessment closely integrated with classroom practice
  • No clear-cut definition of ‘sophisticated tasks’
    • Device to distinguish from alleged reactionary use of simple question styles in early CAA
  • Critique of benefits of sophisticated tasks
    • Implies a position of informed scepticism
    • Central assertion is accepted
      • Sophisticated tasks will soon be ubiquitous
      • They will bring real benefits
      • Benefits have been asserted rather than proven
      • Need to move to informed implementation
discussion 2 the two benefits
Discussion (2) – the two benefits
  • Assessment of HLOs
  • Formative assessment
    • Often confounded – they are distinct
  • Argument stronger in case of HLOs
    • Empirical evidence in near future
  • Sophisticated tasks’ ability to facilitate formative assessment – less secure
    • Difficulty in authoring tasks – continue to be created by large central organisations
    • Formative assessment ought be to low-tech – usable in everyday classrooms
    • Use of day-to-day software applications for formative assessment – e.g. track changes in word processors
other work to do on sophisticated tasks
Other work to do onsophisticated tasks
  • Deriving scores for each task
  • Measurement models
    • summarise scores
    • provide meaningful information about performance/ability, etc.
  • Validation of tests
mackenzie definition
Mackenzie definition
  • Advanced Computer-Based Assessment …
    • goes beyond the simple multiple-choice/response item types into the area of complex question types, adaptive branching, scenarios and simulations of real life situations or problems. In formative mode, feedback may be immediate, extensive, context-sensitive and include annotation of diagrams, links to web resources or full courseware/e-learning tutorials making it an extremely powerful learning tool.