Reflections on innovations in e-assessment. John Winkley May 2014. About Alpha Plus. Specialist education and assessment consultancy Particular strength in qualifications, assessment and examinations Extensive work in technology-supported learning and assessment
In the same way that it is already helping to revolutionize commerce, education, and even social interaction, the Internet will help revolutionize the business and substance of large-scale assessment.
Isabel Nisbet, outgoing chief executive of Ofqual
‘The way that students are tested should be the same as the way that they learn, otherwise it’s not a very good test…
‘…the way that they use technology to learn that that’s just native to them… …sitting at desks writing down, that may not be testing what they’ve learnt.
‘…let’s get this off the too difficult pile and just think a little bit about the future, how are we going to make sure that tomorrow’s kids have the right kind of tests?’
Jeremy Benson, Director of Policy, Ofqual at FAB national conference 2013
The starting point for my speech today is standards in vocational qualifications. There has [not] been … nearly enough [talk] about standards.
But in my view, standards are as important in vocational qualifications as in general qualifications. Standards covers not only the skills that have to be demonstrated to pass the qualification, but also the level of demand from the design and structure of assessments, and how well candidates have to perform.
How can we, as regulator, be confident that your assessment developers and your assessors have the right skills, so you can be confident in the standards and validity of your awards? Where will the assessors of the future come from? – particularly in an increasingly e-enabled world?
Actually, a few disparate experiences from various projects‘it’s really not about the technology any more’
Our own experience in deploying e-assessment for a major project for BIS
Our experience in running a training course recently in Saudi Arabia
Falling price of e-assessment “per-test”. Falling minimum volumes too.
Cloud-based delivery appears to have solved a lot of problems for centres
Specialist professional body offering international examinations
Small volumes, limited internal resources – previous investigations of “traditional” e-assessment not fruitful
But board level drive to undertake service modernisation to support growth
Examinations take the form of lengthy essay-based papers. Changing the nature of the examination was outside the scope of the project.
The administration involves moving a lot of paper around the world.
Candidates are typically mid 30s busy professionals working in large companies
What we helped the professional body to do:
Review the options in the market including some of the “less trodden paths”
Deploy a “Bring Your Own Device” e-assessment solution using the Exam4 “armored word processor”
Support centres and learners participating in the live pilot
Obtain detailed feedback on how they found the assessment
Deploy a secure exam paper delivery system
Support centres using the technology
What we learned:
Students really valued being able to use their own machine
Able to type-and-edit faster than they could write
Removed concerns about untidy and illegible scripts
Many students were willing and able to work through the setup in advance
Technology effective and inexpensive
“Start to finish” in about 6 months
Employer ownership pilots are showing interest in unfamiliar approaches to assessment and “awarding”, particularly in areas where employers have existing assessment (usually internal compliance or safety-related) which is similar to external assessment.
CASE STUDY 4: Secure exam delivery
Monitor the status of downloads
CASE STUDY 4: Secure paper delivery
What we learned:
There are four key problems when it comes to marking digital files:
How do we view all the different file types without lots of software?
Files can be very large – e.g. PowerPoint
How do we actually mark/annotate the files without changing the work?
How do we track different users’ annotations clearly?
Mark and annotate a variety
of file types completely online
No need for a multitude of
software - just a Flash plug-in
Does not alter the work
Links to mark schemes
Provision for student self,
peer, teacher and verifier
Oldham Hulme Grammar School
AOs who use on-screen tests will have large quantities of data.
It is possible to use such data to do a wide range of analysis.
Such analyses are very useful as a ‘reality check’.
Helps you understand tests’ performance empirically.
“Experts consider this item difficult, but when we presented it to 5,000 students, the facility value was 0.75, so it's pretty easy.”
Statistical analysis does not replace expert judgement, it supplements it.
It’s part of a general commitment to quality.
Helps AOs to spot problems before anyone else does!
There are ‘pushes’ as well as ‘pulls’
Ofqual increasingly looking into this area.
Likely to ask ‘do you know how reliable your tests are?’
Parallel forms an area of particular weakness, we believe
Reflections on innovations in e-assessment
Lower cost, lower volume solutions ever more viable for organisations that would not consider themselves as experts in IT
The view of on-screen assessment as a commodity is becoming more prevalent (and more true) but is not the whole story
Diversity of options increasing too. Careful market searches identify tools that solve different problems to the mainstream products
Technology offers AOs proven competitive advantage, at a time of extraordinary market turbulence (for the Ofqual regulated sector)