e-Assessment: From Implementation to Practice

e-Assessment: From Implementation to Practice Myles Danson Farzana Khandia Loughborough University

Learning Outcomes • Identify the benefits of online assessment • Understand the pros and cons of using different question types • Construct questions based on the advice given • Translate statistical output of questions to improve the quality

Overview of CAA “CAA is a common term for the use of computers in the assessment of student learning. The term encompasses the use of computers to deliver, mark and analyse assignments or examinations.” (Bull et al, 2001, p.8)

Benefits – For Staff • Reduces lecturer administration (marking) “ ... online assessment is capable of greater flexibility, cost effectiveness and timesaving. It is these attributes that have made it appealing in an age of competitive higher education funding and resource constraints.” McLoughlin, 2002, p. 511. • Supports distance learning assessment • Potential to introduce a richer range of material (audio, visual) • Ability to monitor the quality of the questions • Question reusability

Benefits – For Students • They can revise and rehearse at their own pace • Flexible access • Instant Feedback “Action without feedback is completely unproductive for a learner.” (Laurillard, 1993, p. 61) • Alternative form of assessment

Types of CAA - Web • Authoring via PC • Delivery on PC / Web

Types of CAA - OMR • Authoring / delivery on paper • Marked using technology • Example

Scenarios of use - Diagnostic One-off test during start of academic year, or at intervals throughout the year to gauge: • Variety in student knowledge • Gaps in student knowledge • Common misconceptions Or help • Plan lecture content

Scenarios of use - Formative • Promote learning by providing feedback • may wish to use objective tests at regular intervals within a course to • determine which topics have been understood, or • to motivate students to keep pace with the teaching of the module

Scenarios of use - Summative can be used to • test the range of the student's understanding of course material • Norm or Criteria Referenced • High Risk

Demo • Lboro Tests • Bruce Wright Excel test • Maths practice

Drivers • Widening participation (Student diversity) • Increasing student retention • Enhanced quality of feedback • Flexibility for distance learning • Coping with large student numbers • Objectivity in marks / defensibility • QCA / DFES / HEFCE / JISC

QCA by 2009 • e-Assessment will be rolled out in post-16 education by 2009 • e-Assessment will make a significant contribution to reducing the assessment burden and improving the quality of assessment • e-Assessment field trials should be taking place in at least two subjects per awarding body during 2005 • 75% of key and basic skills tests will be delivered on screen by 2005 • All new qualifications will include an option for on-screen assessment • All Awarding Bodies should be set up to accept and assess e-portfolios

Barriers • Availability of resources • Lack confidence in e-assessment • Fear of technical failure high stakes assessments • Work pressures on academic staff (insufficient time to give to developing potential of e-assessment) • Fit for purpose / assessing appropriate levels of learning? • Authentication issues e.g. learner identity, plagiarism

Support and Training • Student support • Practice tests and fall back • Demonstration of the software • Special needs • Staff support • Introductory session • Follow-up session • Pre, during and post-examination procedures

Question Design It’s very easy to write objective questions but difficult to write good objective questions.

Parts of a MCQ A multiple choice question consists of four discrete elements: As children’s language skills increase in complexity, from the pre-linguistic phase to telegraphic speech, the progression is most noticeable in the area of • semantics • intonation • syntax • inference • clause combinations STEM DISTRACTERS KEY OPTIONS

Writing Stems • When possible, state the stem as a direct question rather than as an incomplete statement. e.g. Alloys are ordinarily produced by ... How are alloys ordinarily produced?

Avoid irrelevant clues such as grammatical structure, well known verbal associations or connections between stem and answer e.g. A chain of islands is called an: *a. archipelago. b. peninsula. c. continent. d. isthmus Grammatical clue!

The height to which a water dam is built depends on a. the length of the reservoir behind the dam. b. the volume of water behind the dam. *c. the height of water behind the dam. d. the strength of the reinforcing wall. Connection between stem and answer clue

Use negatively stated stems sparingly. When used, underline and/or capitalise the negative word. e.g. Which of the following is not cited as an accomplishment of the Kennedy administration? Which of the following is NOT cited as an accomplishment of the Kennedy administration?

Eliminate excessive verbiage or irrelevant information from the stem. e.g. While ironing her formal, Jane burned her hand accidentally on the hot iron. This was due to a transfer of heat be ... Which of the following ways of heat transfer explains why Jane's hand was burned after she touched a hot iron?

Include in the stem any word(s) that might otherwise be repeated in each alternative. e.g. In national elections in the United States the President is officially a. chosen by the people. b. chosen by members of Congress. c. chosen by the House of Representatives. *d. chosen by the Electoral College. In national elections in the United States the President is officially chosen by a. the people. b. members of Congress. c. the House of Representatives. *d. the Electoral college.

Present a definite, explicit and singular question or problem in the stem. e.g. Psychology ... The science of mind and behaviour is called ...

Writing Distracters • Use the alternatives "none of the above" and "all of the above" sparingly. When used, such alternatives should occasionally be used as the correct response. • Ensure there is only one unquestionably correct answer e.g. The two most desired characteristics in a classroom test are validity and a. precision. *b. reliability. c. objectivity. *d. consistency. The two most desired characteristics in a classroom test are validity and a. precision. *b. reliability. c. objectivity. d. standardisation.

Make the alternatives grammatically parallel with each other, and consistent with the stem. What would do most to advance the application of atomic discoveries to medicine? *a. Standardised techniques for treatment of patients. b. Train the average doctor to apply radioactive treatments. c. Remove the restriction on the use of radioactive substances. d. Establishing hospitals staffed by highly trained radioactive therapy specialists. What would do most to advance the application of atomic discoveries to medicine? *a. Development of standardised techniques for treatment of patients. b. Training of the average doctor in application of radioactive treatments. c. Removal of restriction on the use of radioactive substances. d. Addition of trained radioactive therapy specialists to hospital staffs.

Make alternatives approximately equal in length. e.g. The most general cause of low individual incomes in the United States is *a. lack of valuable productive services to sell. b. unwillingness to work. c. automation. d. inflation. What is the most general cause of low individual incomes in the United States? *a. A lack of valuable productive services to sell. b. The population's overall unwillingness to work. c. The nation's increased reliance on automation. d. An increasing national level of inflation.

Make all alternatives plausible and attractive to the less knowledgeable or skilful student. e.g. What process is most nearly the opposite of photosynthesis? a. Digestion b. Relaxation *c. Respiration d. Exertion a. Digestion b. Assimilation *c. Respiration d. Catabolism • Vary the position of the correct answer in a random way

Exercise Item Intimations

Question / Test Performance • Caution is needed when using statistics based on small samples. • External awarding bodies and test producers aim to trial items with at least 200 candidates to get reliable item statistics • If there are fewer than 50 candidates the statistics will only give an approximate indication of how the items are working

Statistics for whole test • Mean • Standard deviation • Reliability.

= Average mark Expected value: 65-75% for most tests (50-75% acceptable) Up to 90% in formative tests Mean should be similar to previous years. Mean

Candidates are more / less able than in past (or than expected) Candidates are better / less well taught Test is too difficult / too easy Time allowance was too short. Possible causes ifMean is not as expected:

= a measure of the spread of the marks Standard deviation (s.d or σ)

Expected value: 10-15% Will be much lower if mean is high S.D. should be similar to previous years. Standard deviation (s.d. or σ)

The candidates are genuinely of similar ability The test is not distinguishing satisfactorily between the better and weaker candidates The test covers two or more quite different topics (or abilities) . Possible causes ifStandard Deviation is low:

= a measure of internal consistency (0 – 1) Given as a decimal; the theoretical maximum is 1.0 Expected value: above 0.75 Reliability

Too many items with low Discrimination Test is too short . Possible causes ifReliability is low:

Number of candidates151 Max possible score 120 Range of scores63 / 112 (53-93%) Mean 96.2 (80.2%) Standard deviation 9.96 (8.3%) K-R 20 reliability not calculated Pass Rate 96.7% Example : Beauty Therapy exam

Facility Discrimination Number and % choosing each option Mean for outcome Statistics for each item:

Also called Difficulty or P-value = Proportion or percentage of candidates answering question correctly = Proportion choosing the key (multiple-choice) Expected value: 40-90% for multiple-choice May be lower in other item types. Facility

Key is not the best answer Some candidates didn’t have time to answer Topic is too difficult The item is unclear or contains error One or more distractors are unfair Item is too complex. Possible causes ifFacility below 40:

Topic is too easy Wording contains clues One or more distractors are weak. Possible causes ifFacility above 90:

Theoretically possible values from –1.0 to +1.0 Shows whether the candidates who answered this item correctly were the candidates who generally did better on the whole test Expected value: +0.2 or above . Discrimination / Facility

Topic tested is unlike rest of the test Item is very easy (> 90) or very difficult (<40) Item is misleading One or more distractors are unfair. A negative Discrimination is never acceptable. Possible causes ifDiscrimination is too low:

A B* C D Number 10 137 1 3 % 6.6 90.7 0.7 1.9 Mean for outcome 87.7 97.4 74 76 Facility 90.7 Discrimination 0.39 Display for each option

Also called Frequency and Percent or ‘Times answered’ Expected values: at least 5% for each distractor % for distractor should not be more than % for the key Number and % choosing each option

There is a clue in the wording or the distractor is silly Topic is very easy. Possible causes ifdistractor chosen is under 5%:

Item is misleading in some way. Possible causes if distractor attracts more candidates than the key:

Mean (average) mark on the whole test of the candidates who chose this option. Expected value: Mean for key (or correct outcome) should be higher than Mean for each of the distractors (or for incorrect outcome). Mean for outcome

e-Assessment: From Implementation to Practice