Kevin Waugh, Neil Smith, Pete Thomas Department of Computing The Open University

1. Kevin Waugh, Neil Smith, Pete Thomas Department of Computing The Open University

2. Toward the automated assessment of ERDs

3. The investigators Diagram Understanding Neil Smith Natural Language Processing Kevin Waugh Assessment, Teaching and Learning Pete Thomas

4. Diagrams

5. What is a diagram? A picture isn't

6. What is a diagram? Free and structured text aren't

7. What is a diagram?  We are using Segmentable Feature-based A common sense filter This isn't a diagram

8. What is a diagram?  These are ….

9. What is a diagram?  and these are ….

10. Traditional take on diagrams Often treated as formal "visual" languages So they're expected to be parsable grammatical correct complete

11. but real diagrams aren't formal they're not always grammatical they're often incomplete they're often incorrect they are not always parsable (especially when drawn by students!)

12. Interesting question: What if we treat diagrams in the same way that we treat text?

13. Text and diagram - a simple correspondence Characters/punctuation - segments Words – features Phrases - "minimal meaningful units“ Sentences – mmu aggregations

14. Natural language A grammar is an approximation to actual language use Pragmatic - rather than correct/complete Do we even need a grammar?

15. Sub-languages Specific grammars for specific domains Stylistic conventions textbooks novels poetry instruction manual Interpretation is domain specific No "universal" solution

16. Research question: What if we process diagrams the same way we process text? Bag of words - Syntactic - Semantic - Statistical analysis -

17. Our take on diagrams Diagrams: are intended to carry meaning, in a given domain domain limited user is domain aware they have an interpretation use domain specific diagram notations domain expert can assess correct use of notation both well-formed and ill-formed (imprecise) domain expert can assess "correctness", "understandability" of diagram domain expert can interpret and correct incomplete and incorrect diagrams (sometimes!)

18. What does this represent? A diagram without domain context non-expert cannot interpret

19. Diagram interpretation needs context and domain knowledge Diagram with domain non-expert cannot interpret (informed guessing)

20. Domain expert does not need correct, complete diagrams Diagram with domain Expert can interpret, criticize and correct.

21. Could supply a notation description but that doesn't equate to a regular user’s knowledge of the domain it helps decide if a diagram is properly drawn, not if it is meaningful.

22. Our larger investigation If we attempt to process diagrams in ways comparable to the ways we process formal, natural and sub-language texts……. can we do useful things with diagrams? Things such as automated assessment?

23. Automated assessment

24. Automated assessment Coursework and Examinations Self-assessment and revision support Grade + automated feedback Grading alone is not sufficient Directed, appropriate, focused feedback is a requirement (Multiple choice - not our concern)

25. Successful automated assessment: Textual assessment (essay and short text) bag-of-words bag-of-phrases sequences (ordered-bag-of-words/phrases) syntactic structure abstracting and comparison (semantic-syntactic) semantic analysis Diagram assessment restricted choice and "slot filling" multiple choice "Free" diagram assessment has not been successfully achieved

26. What if we assess diagrams the same way that we assess text? What are the diagram assessment equivalents to bag-of-words bag-of-phrases sequences abstracting and comparison syntactic structure semantic analysis Can we achieve automated assessment of diagrams comparable to that achieved by a human marker? Can we provide focused feedback comparable to a human tutor?

27. Our initial experiment with ERDs

28. Feasibility experiment: pipelines Approach: comparable to bag-of-words

29. Why entity relationship diagrams? Scope: right/wrong – interpretable Range: small – large Range: simple – complex Correctness: notation - meaning Question, Sample solution, Marking guide (familiarity) Aggregations –  m:n decomposition,  relationship signatures, ...

30. The question

31. Solution and marking scheme:

32. On the risks of using a drawing tool: Slot filling? Prompting? No segmentation or feature extraction? Drawing "correct" diagrams because tool enforces correctness?

33. First results 21 human marked answers Human: Mean 21.29 StdDev 3.757 Tool: Mean 22.24 StdDev 2.508 Spearman’s rho correlation coefficient: 0.957  (significant at the 0.01 level, two-tailed), N=21 Pearsons correlation coefficient: (significant at the 0.01 level, two-tailed), N=21

34. Simplistic? Yes - but .... First step in our assessment of diagrams as text - comparable to bag-of-phrases processing The pipeline experiment was bag-of-words Essentially uses same algorithm as the marking of short answer texts Gives us a base-line when considering adding aggregation etc. We are also aware of ... need to investigate how will we express complex marking schemes (if we need them) the above assessment is not dependent on aggregation nor interpretation

35. Where next Take what we have, add feedback and we have a revision support tool. More complex marking schemes inc. alternative solutions. Include aggregation and abstraction. ERD questions with scope for interpretation – scenario-based rather than translation based.

36. DEAP: Diagrammatic Electronic Assessment Project Thank you