reflections on innovations in e assessment
Skip this Video
Download Presentation
Reflections on innovations in e-assessment

Loading in 2 Seconds...

play fullscreen
1 / 28

Reflections on innovations in e-assessment - PowerPoint PPT Presentation

  • Uploaded on

Reflections on innovations in e-assessment. John Winkley May 2014. About Alpha Plus. Specialist education and assessment consultancy Particular strength in qualifications, assessment and examinations Extensive work in technology-supported learning and assessment

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Reflections on innovations in e-assessment' - jaclyn

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
About AlphaPlus
  • Specialist education and assessment consultancy
    • Particular strength in qualifications, assessment and examinations
    • Extensive work in technology-supported learning and assessment
  • Based in the UK, with an international portfolio of work, covering
    • Primary, secondary and high school examinations for progression to college university
    • Vocational and technical qualifications
    • Professional certifications
About me!
  • Started out as an engineer/technologist – two organisations providing e-assessment and e-learning.
    • (millions of online exams each year)
    • (recently enrolled its 1.3 millionth learner)
  • Then moved in to e-assessment, and then assessment more generally.
  • It’s not really about the technology any more…
  • I work with educators, test writers, certification agencies and governments to help make qualifications and assessment work better
potential of e assessment for large scale testing
Potential of e-assessmentfor large-scale testing

Bennett (2001)

In the same way that it is already helping to revolutionize commerce, education, and even social interaction, the Internet will help revolutionize the business and substance of large-scale assessment.

but has potential gone unrealised
But has potential gone unrealised?

Isabel Nisbet, outgoing chief executive of Ofqual

February 2011

Isabel in text

‘The way that students are tested should be the same as the way that they learn, otherwise it’s not a very good test…

‘…the way that they use technology to learn that that’s just native to them… …sitting at desks writing down, that may not be testing what they’ve learnt.

‘…let’s get this off the too difficult pile and just think a little bit about the future, how are we going to make sure that tomorrow’s kids have the right kind of tests?’

ofqual in 2013

Ofqual in 2013

Jeremy Benson, Director of Policy, Ofqual at FAB national conference 2013

The starting point for my speech today is standards in vocational qualifications. There has [not] been … nearly enough [talk] about standards.

But in my view, standards are as important in vocational qualifications as in general qualifications. Standards covers not only the skills that have to be demonstrated to pass the qualification, but also the level of demand from the design and structure of assessments, and how well candidates have to perform.

How can we, as regulator, be confident that your assessment developers and your assessors have the right skills, so you can be confident in the standards and validity of your awards? Where will the assessors of the future come from? – particularly in an increasingly e-enabled world?

focus of this talk
Focus of this talk
  • Maybe high-stakes testing in schools isn’t where e-assessment can have the most (immediate) benefit.
  • Maybe we can get more ‘wins’ from non-testing, less ‘schools’ technologies and services.
  • Some “case studies”:
    • Changes in “traditional e-assessment”
    • On-screen essays
    • Elephants don’t forget!
    • Secure exam delivery by pdf
    • Low volume marking
    • Statistical services for AOs
  • I will illustrate the case studies and explain why they are successful (or at least interesting) e-assessment innovations.
case study 1 on screen tests in 2014

CASE STUDY 1: on screen tests in 2014

Actually, a few disparate experiences from various projects‘it’s really not about the technology any more’

Our own experience in deploying e-assessment for a major project for BIS

Our experience in running a training course recently in Saudi Arabia

Falling price of e-assessment “per-test”. Falling minimum volumes too.

Cloud-based delivery appears to have solved a lot of problems for centres

case study 2 on screen essays

CASE STUDY 2: on screen essays

The brief:

Specialist professional body offering international examinations

Small volumes, limited internal resources – previous investigations of “traditional” e-assessment not fruitful

But board level drive to undertake service modernisation to support growth

Examinations take the form of lengthy essay-based papers. Changing the nature of the examination was outside the scope of the project.

The administration involves moving a lot of paper around the world.

Candidates are typically mid 30s busy professionals working in large companies

case study 2 on screen essays1

CASE STUDY 2: on screen essays

What we helped the professional body to do:

Review the options in the market including some of the “less trodden paths”

Deploy a “Bring Your Own Device” e-assessment solution using the Exam4 “armored word processor”

Support centres and learners participating in the live pilot

Obtain detailed feedback on how they found the assessment

Deploy a secure exam paper delivery system

Support centres using the technology

case study 2 on screen essays2

CASE STUDY 2: on screen essays

What we learned:

Students really valued being able to use their own machine


Able to type-and-edit faster than they could write

Removed concerns about untidy and illegible scripts

Many students were willing and able to work through the setup in advance

Technology effective and inexpensive

“Start to finish” in about 6 months

case study 3 elephants don t forget

CASE STUDY 3: Elephants don’t forget

Employer ownership pilots are showing interest in unfamiliar approaches to assessment and “awarding”, particularly in areas where employers have existing assessment (usually internal compliance or safety-related) which is similar to external assessment.

  • Seems likely that BIS will want Ofqual to consider “approval” of such innovative approaches.
  • Despite the jokey website, this approach could potentially deliver very interesting effects in terms of assessment predictive validity (we think).
CASE STUDY 4: Secure exam delivery

Awarding organisations:

  • Upload Exam Packs to the system. An Exam Pack contains all the PDF files you want the centre to be able to download in advance of the examination.
  • Create Examination Windows. An Examination Window specifies when an Exam Pack becomes available for download, and when the download window closes.
  • Allocate the Examination Window to one or more Centres.
  • Mass email the Centres (via SecurePDF) to let them know about their upcoming Examination Windows.
CASE STUDY 4: Secure paper delivery

What we learned:

  • Greatly increased security for delivery of examination documents (versus postal delivery or website/portal download)
  • Awarding organisation has complete control over when centres download examination documents
  • SecurePDF maintains a record of downloads, identifying users and centres that downloaded each Exam Pack
  • Centres were happy to use the system and valued the efforts made to improve security
case study 5 low volume marking

CASE STUDY 5: Low volume marking

There are four key problems when it comes to marking digital files:

How do we view all the different file types without lots of software?

Files can be very large – e.g. PowerPoint

How do we actually mark/annotate the files without changing the work?

How do we track different users’ annotations clearly?

red pen how it works

Red pen – how it works

The Benefits:

Mark and annotate a variety

of file types completely online

No need for a multitude of

software - just a Flash plug-in

Does not alter the work

Links to mark schemes

Provision for student self,

peer, teacher and verifier


Case Study – Red Pen Tool for Moodle

Oldham Hulme Grammar School

  • Using Moodle to support work at KS3 & 4
  • Found that there was still too much printing, especially for marking work
  • They started using Red Pen Tool in February 2011
  • Primarily using the Red Pen Tool to support coursework for GCSE ICT (Module 2: controlled assessment - WJEC)
  • Proved to be a huge success and as a result they are planning to extend use this coming year.
case study 4 statistical analysis for aos

CASE STUDY 4: Statistical analysis for AOs

AOs who use on-screen tests will have large quantities of data.

It is possible to use such data to do a wide range of analysis.

Such analyses are very useful as a ‘reality check’.

Helps you understand tests’ performance empirically.

“Experts consider this item difficult, but when we presented it to 5,000 students, the facility value was 0.75, so it's pretty easy.”

Statistical analysis does not replace expert judgement, it supplements it.

It’s part of a general commitment to quality.

Helps AOs to spot problems before anyone else does!

There are ‘pushes’ as well as ‘pulls’

Ofqual increasingly looking into this area.

Likely to ask ‘do you know how reliable your tests are?’

Parallel forms an area of particular weakness, we believe



Lower cost, lower volume solutions ever more viable for organisations that would not consider themselves as experts in IT

The view of on-screen assessment as a commodity is becoming more prevalent (and more true) but is not the whole story

Diversity of options increasing too. Careful market searches identify tools that solve different problems to the mainstream products

Technology offers AOs proven competitive advantage, at a time of extraordinary market turbulence (for the Ofqual regulated sector)

reflections on innovations in e assessment thank you for listening questions

Reflections on innovations in e-assessmentThank you for listening.Questions?

John Winkley

May 2014