e assessment from implementation to practice n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
e-Assessment: From Implementation to Practice PowerPoint Presentation
Download Presentation
e-Assessment: From Implementation to Practice

Loading in 2 Seconds...

play fullscreen
1 / 55

e-Assessment: From Implementation to Practice - PowerPoint PPT Presentation


  • 74 Views
  • Uploaded on

e-Assessment: From Implementation to Practice. Myles Danson Farzana Khandia Loughborough University. Learning Outcomes. Identify the benefits of online assessment Understand the pros and cons of using different question types Construct questions based on the advice given

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'e-Assessment: From Implementation to Practice' - zubeda


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
e assessment from implementation to practice

e-Assessment: From Implementation to Practice

Myles Danson

Farzana Khandia

Loughborough University

learning outcomes
Learning Outcomes
  • Identify the benefits of online assessment
  • Understand the pros and cons of using different question types
  • Construct questions based on the advice given
  • Translate statistical output of questions to improve the quality
overview of caa
Overview of CAA

“CAA is a common term for the use of

computers in the assessment of student

learning. The term encompasses the use of

computers to deliver, mark and analyse

assignments or examinations.” (Bull et al,

2001, p.8)

benefits for staff
Benefits – For Staff
  • Reduces lecturer administration (marking)

“ ... online assessment is capable of greater flexibility, cost effectiveness and timesaving. It is these attributes that have made it appealing in an age of competitive higher education funding and resource constraints.” McLoughlin, 2002, p. 511.

  • Supports distance learning assessment
  • Potential to introduce a richer range of material (audio, visual)
  • Ability to monitor the quality of the questions
  • Question reusability
benefits for students
Benefits – For Students
  • They can revise and rehearse at their own pace
  • Flexible access
  • Instant Feedback

“Action without feedback is completely unproductive for a learner.” (Laurillard, 1993, p. 61)

  • Alternative form of assessment
types of caa web
Types of CAA - Web
  • Authoring via PC
  • Delivery on PC / Web
types of caa omr
Types of CAA - OMR
  • Authoring / delivery on paper
  • Marked using technology
  • Example
scenarios of use diagnostic
Scenarios of use - Diagnostic

One-off test during start of academic year,

or at intervals throughout the year to gauge:

  • Variety in student knowledge
  • Gaps in student knowledge
  • Common misconceptions

Or help

  • Plan lecture content
scenarios of use formative
Scenarios of use - Formative
  • Promote learning by providing feedback
  • may wish to use objective tests at regular intervals within a course to
  • determine which topics have been understood, or
  • to motivate students to keep pace with the teaching of the module
scenarios of use summative
Scenarios of use - Summative

can be used to

  • test the range of the student's understanding of course material
  • Norm or Criteria Referenced
  • High Risk
slide11
Demo
  • Lboro Tests
  • Bruce Wright Excel test
  • Maths practice
drivers
Drivers
  • Widening participation (Student diversity)
  • Increasing student retention
  • Enhanced quality of feedback
  • Flexibility for distance learning
  • Coping with large student numbers
  • Objectivity in marks / defensibility
  • QCA / DFES / HEFCE / JISC
slide13
QCA

by 2009

  • e-Assessment will be rolled out in post-16 education by 2009
  • e-Assessment will make a significant contribution to reducing the assessment burden and improving the quality of assessment
  • e-Assessment field trials should be taking place in at least two subjects per awarding body during 2005
  • 75% of key and basic skills tests will be delivered on screen by 2005
  • All new qualifications will include an option for on-screen assessment
  • All Awarding Bodies should be set up to accept and assess e-portfolios
barriers
Barriers
  • Availability of resources
  • Lack confidence in e-assessment
  • Fear of technical failure high stakes assessments
  • Work pressures on academic staff (insufficient time to give to developing potential of e-assessment)
  • Fit for purpose / assessing appropriate levels of learning?
  • Authentication issues e.g. learner identity, plagiarism
support and training
Support and Training
  • Student support
    • Practice tests and fall back
    • Demonstration of the software
    • Special needs
  • Staff support
    • Introductory session
    • Follow-up session
    • Pre, during and post-examination procedures
question design
Question Design

It’s very easy to write objective questions but

difficult to write good objective questions.

parts of a mcq
Parts of a MCQ

A multiple choice question consists of four discrete

elements:

As children’s language skills increase in complexity, from

the pre-linguistic phase to telegraphic speech, the

progression is most noticeable in the area of

  • semantics
  • intonation
  • syntax
  • inference
  • clause combinations

STEM

DISTRACTERS

KEY

OPTIONS

writing stems
Writing Stems
  • When possible, state the stem as a direct question rather than as an incomplete statement.

e.g.

Alloys are ordinarily produced by ...

How are alloys ordinarily produced?

slide19
Avoid irrelevant clues such as grammatical structure, well known verbal associations or connections between stem and answer

e.g.

A chain of islands is called an:

*a. archipelago.

b. peninsula.

c. continent.

d. isthmus

Grammatical clue!

slide20
The height to which a water dam is built depends on

a. the length of the reservoir behind the dam.

b. the volume of water behind the dam.

*c. the height of water behind the dam.

d. the strength of the reinforcing wall.

Connection between stem and answer clue

slide21
Use negatively stated stems sparingly. When used, underline and/or capitalise the negative word.

e.g.

Which of the following is not cited as an accomplishment of the Kennedy administration?

Which of the following is NOT cited as an accomplishment of the Kennedy administration?

slide22
Eliminate excessive verbiage or irrelevant information from the stem.

e.g.

While ironing her formal, Jane burned her hand accidentally on the hot iron. This was due to a transfer of heat be ...

Which of the following ways of heat transfer explains why Jane's hand was burned after she touched a hot iron?

slide23
Include in the stem any word(s) that might otherwise be repeated in each alternative.

e.g.

In national elections in the United States the President is officially

a. chosen by the people.

b. chosen by members of Congress.

c. chosen by the House of Representatives.

*d. chosen by the Electoral College.  

In national elections in the United States the President is officially chosen by a. the people. b. members of Congress. c. the House of Representatives. *d. the Electoral college.

slide24
Present a definite, explicit and singular question or problem in the stem.

e.g.

Psychology ...

The science of mind and behaviour is

called ...

writing distracters
Writing Distracters
  • Use the alternatives "none of the above" and "all of the above" sparingly. When used, such alternatives should occasionally be used as the correct response.
  • Ensure there is only one unquestionably correct answer

e.g.

The two most desired characteristics in a classroom test are validity and

a. precision.

*b. reliability.

c. objectivity.

*d. consistency.  

The two most desired characteristics in a classroom test are validity and

a. precision.

*b. reliability.

c. objectivity.

d. standardisation.

slide26
Make the alternatives grammatically parallel with each other, and consistent with the stem.

What would do most to advance the application of atomic discoveries to medicine?

*a. Standardised techniques for treatment of patients.

b. Train the average doctor to apply radioactive treatments.

c. Remove the restriction on the use of radioactive substances.

d. Establishing hospitals staffed by highly trained radioactive therapy specialists.  

What would do most to advance the application of atomic discoveries to medicine?

*a. Development of standardised techniques for treatment of patients.

b. Training of the average doctor in application of radioactive treatments.

c. Removal of restriction on the use of radioactive substances.

d. Addition of trained radioactive therapy specialists to hospital staffs.

slide27
Make alternatives approximately equal in length.

e.g.

The most general cause of low individual incomes in the United States is

*a. lack of valuable productive services to sell.

b. unwillingness to work.

c. automation.

d. inflation.  

What is the most general cause of low individual incomes in the United States?

*a. A lack of valuable productive services to sell.

b. The population's overall unwillingness to work.

c. The nation's increased reliance on automation.

d. An increasing national level of inflation.

slide28
Make all alternatives plausible and attractive to the less knowledgeable or skilful student.

e.g.

What process is most nearly the opposite of photosynthesis?

a. Digestion

b. Relaxation

*c. Respiration

d. Exertion

a. Digestion

b. Assimilation

*c. Respiration

d. Catabolism

  • Vary the position of the correct answer in a random way
exercise
Exercise

Item Intimations

question test performance
Question / Test Performance
  • Caution is needed when using statistics based on small samples.
  • External awarding bodies and test producers aim to trial items with at least 200 candidates to get reliable item statistics
  • If there are fewer than 50 candidates the statistics will only give an approximate indication of how the items are working
statistics for whole test
Statistics for whole test
  • Mean
  • Standard deviation
  • Reliability.
slide32
= Average mark

Expected value:

65-75% for most tests (50-75% acceptable)

Up to 90% in formative tests

Mean should be similar to previous years.

Mean
possible causes if mean is not as expected
Candidates are more / less able than in past (or than expected)

Candidates are better / less well taught

Test is too difficult / too easy

Time allowance was too short.

Possible causes ifMean is not as expected:
standard deviation s d or1
Expected value:

10-15%

Will be much lower if mean is high

S.D. should be similar to previous years.

Standard deviation (s.d. or σ)
possible causes if standard deviation is low
The candidates are genuinely of similar ability

The test is not distinguishing satisfactorily between the better and weaker candidates

The test covers two or more quite different topics (or abilities) .

Possible causes ifStandard Deviation is low:
reliability
= a measure of internal consistency (0 – 1)

Given as a decimal; the theoretical

maximum is 1.0

Expected value:

above 0.75

Reliability
example beauty therapy exam
Number of candidates151

Max possible score 120

Range of scores63 / 112 (53-93%)

Mean 96.2 (80.2%)

Standard deviation 9.96 (8.3%)

K-R 20 reliability not calculated

Pass Rate 96.7%

Example : Beauty Therapy exam
statistics for each item
Facility

Discrimination

Number and % choosing each option

Mean for outcome

Statistics for each item:
facility
Also called Difficulty or P-value

= Proportion or percentage of candidates answering question correctly

= Proportion choosing the key (multiple-choice)

Expected value:

40-90% for multiple-choice

May be lower in other item types.

Facility
possible causes if facility below 40
Key is not the best answer

Some candidates didn’t have time to answer

Topic is too difficult

The item is unclear or contains error

One or more distractors are unfair

Item is too complex.

Possible causes ifFacility below 40:
possible causes if facility above 90
Topic is too easy

Wording contains clues

One or more distractors are weak.

Possible causes ifFacility above 90:
discrimination facility
Theoretically possible values from –1.0 to +1.0

Shows whether the candidates who answered

this item correctly were the candidates who

generally did better on the whole test

Expected value: +0.2 or above .

Discrimination / Facility
possible causes if discrimination is too low
Topic tested is unlike rest of the test

Item is very easy (> 90) or very difficult (<40)

Item is misleading

One or more distractors are unfair.

A negative Discrimination is never acceptable.

Possible causes ifDiscrimination is too low:
display for each option
A B* C D

Number 10 137 1 3

% 6.6 90.7 0.7 1.9

Mean for outcome 87.7 97.4 74 76

Facility 90.7 Discrimination 0.39

Display for each option
number and choosing each option
Also called Frequency and Percent or ‘Times

answered’

Expected values:

at least 5% for each distractor

% for distractor should not be more than % for the key

Number and % choosing each option
mean for outcome
Mean (average) mark on the whole test of the

candidates who chose this option.

Expected value:

Mean for key (or correct outcome) should be higher than Mean for each of the distractors (or for incorrect outcome).

Mean for outcome
recent innovations
Recent Innovations
  • Confidence Assessment
  • Adaptive Testing
  • Free Textwww.sagrader.comwww.intelligentassessment.com
  • JISCe-Assessment Glossarye-Assessment Case Studiese-Assessment Road Map
  • www.caaconference.com/
references
References
  • Bull, J. and C. McKenna (2001). Blueprint for Computer-assissted Assessment, CAA Centre.
  • Laurillard, D. (1993). Rethinking university teaching : a framework for the effective use of educational technology. London ; New York, Routledge.
  • McLoughlin, C. (2002). "Editorial." British Journal of Educational Technology33(5): 511-513.
further reading
Further Reading
  • Introduction to e-assessment
    • Warburton (2005) Whither E-assessment, Proceedings of 9th CAA Conference (Loughborough University), 471-482.
  • CAA Benefits for Staff and Students
    • Brown, Race & Bull (1999), Computer Assisted Assessment in Higher Education (Kogan Page), 7-8.
    • Bull and McKenna (2000), Blueprint for CAA (CAA Centre), 8-10.
    • Ashton, H.S. et al (2003) Pilot summative web assessment in secondary education, Proceedings of 7th CAA Conference (Loughborough University), 33-44.
  • Student Acceptance
    • Sambell et al (1999) Students’ Perception of the learning benefits of computer-assisted assessment: a case study in electronic engineering, in: S.Brown, J. Bull & P. Race (Eds) Computer-assisted assessment in higher education (Birmingham, SEDA), 179-191.
  • Item Banks
    • Sclater, et al (2004) Item Banks Infrastructure Study (JISC).
  • Future Developments
    • Boyle A (2005) Sophisticated Tasks in E-Assessment: What are they and what are their benefits? Proceedings of 9th CAA Conference (Loughborough University), 51-66.