james christie n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
James Christie PowerPoint Presentation
Download Presentation
James Christie

Loading in 2 Seconds...

play fullscreen
1 / 20

James Christie - PowerPoint PPT Presentation


  • 64 Views
  • Uploaded on

James Christie. Automated Essay Marking Does it work?. Essay Definition 1 of 2.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'James Christie' - elom


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
james christie

James Christie

Automated Essay Marking

Does it work?

essay definition 1 of 2
Essay Definition 1 of 2

… requires a response composed by the examinee, usually in the form of one or more sentence, of a nature that no single response or pattern of responses can be listed as correct, and the accuracy and quality of which can be judged subjectively only by one skilled or informed in the subject, …

essay definition 2 of 2
Essay Definition 2 of 2

… but even an expert cannot usually classify a response as categorically right or wrong. Rather, there are different degrees of quality or merit which can be recognized. …

attributed to

Stalnaker, 1951

possible criteria for automated essay marking
Possible criteria for automated essay marking
  • Ease of creating a scoring schema
  • Ability to score on various mark regimes
  • Ease of identification on non-scoring elements
  • Ease of modification
      • should scoring error(s) occur
  • Consistent and reproducible scoring
  • Acceptability of results to
      • human markers, essayists, …
  • Defensibility
  • Accuracy and precision
  • Coachability avoidance
  • Cost
slide5

Sample of a schema for Robert Gordon, founder of the Robert Gordon University.

Essay is marked out of 25, while the sum for individual items is 35 ~ not a mistake!

Mark Item

0 Robert Gordon

3 born 1668 Castlegate Aberdeen

3 inherited 1680 £1,000

5 graduated 1689 Marischal College Aberdeen

3 retired 1720 Aberdeen

1 died 1731

[max:15]

1 Father

2 Arthur Gordon

2 Edinburgh advocate

[max:5]

1 Mother

2 Isabella Gordon

2 nee Isabella Menzies

[max:5]

1 Grandfather

2 Robert Gordon

1 Catographer

3 Blaeu's Atlas 1654

[max:7]

1 Trader

2 Danzig Baltic

[max:3]

[max:25]

model essay
Model Essay

The cat sat on the mat.

marking schema
Marking Schema

Mark Item

3 cat sat mat

[max:3]

essay set
Essay Set
  • ALPHA
    • The cat sat on the mat.
  • BRAVO
    • The cat sat on the floor.
  • CHARLIE
    • The dog lay on the floor.
  • MODEL
    • The cat sat on the mat.
schema report
Schema Report

Essay set ............. catmat

Schema Report using ... catmat

Entities 1 - 4 :

Entity ID : 1 2 3 4

Entity Type : a cat sat mat

Part ID's : z

Essay Name :

ALPHA.EXT: y y y y

BRAVO.EXT: _ y y _

CHARLIE.EXT: _ _ _ _

MODEL.EXT: y y y y

content report
Content Report

Essay set .............. catmat

Content Report using ... catmat

Essay Name : Words Sentences Usage[%] Coverage[%]Part: z[ 3] Mark[ 3] %[100]

ALPHA.EXT: 6 1 50.00 100.00 3 3 100.00

BRAVO.EXT: 6 1 33.33 50.00 0 0 0.00

CHARLIE.EXT: 6 1 0.00 0.00 0 0 0.00

MODEL.EXT: 6 1 50.00 100.00 3 3 100.00

Started on Thursday, February 06 2003 at 15:59:05

Finished on Thursday, February 06 2003 at 15:59:06

Marked 4 file(s): scanned 4 file(s)

usage coverage
Usage & Coverage
  • Usage
    • How much of the Essay matched with the Content Schema
  • Coverage
    • How much of the Content Schema matched with the Essay
marking performance
Marking Performance

Pearson / Spearman

Significance **= 0.01 *= 0.05

future work content
Future work [content]
  • maximise use of active and passive voices
  • cope with spelling & grammar errors
  • increased coverage of Bloom’s Taxonomy
  • include non-textual feature(s)
  • develop
    • better feedback to the essayist
    • better feedback to the examiner
    • plagiarism detection mechanism(s)
future work style
Future work [style]
  • obtain marked essays for style marking
    • plain ASCII essays
      • using a common set of metrics
    • word-processed essays
      • using a common set of metrics
      • augmented with word-processing based metrics
slide20

Automated Essay Marking

Does it work? Yes, but …

http://www.rgu.ac.uk/staff/jrc

j.christie@rgu.ac.uk

T: +44 [0]1224 262725

F: +44 [0]1224 262725