Designing an Interactive Teaching Tool with ABML Knowledge Refinement Loop

1. Designing an Interactive Teaching Toolwith ABML Knowledge Refinement Loop enablingarguing to learn Matej Zapušek1, Martin Možina2, Ivan Bratko2, Jože Rugelj2, Matej Guid2 1Faculty of Education, University of Ljubljana, Slovenia 2Faculty of Computer and Information Science, University of Ljubljana, Slovenia 12thInternationalConference on IntelligentTutoringSystems ITS 2014: Honolulu, Hawaii 2014

2. Some Concepts are Difficult to Explain... How to distinguish edible from toxic mushrooms?

3. IntroducingDomainExperts Evenfordomainexperts it is hard to articulatetheirknowledge!

4. MachineLearning: How to Involve a DomainExpert? Theexpertcanstateconstraintsandthedomainknowledge in advance... … verify, evaluate, andcorrectresultsofmachinelearning… … ortheexpertandthecomputeriterativelyimprovethe model. ABML argument-basedmachinelearning

5. Argument-BasedMachineLearning given • set oflabeledlearning examplesei • described withattributevaluesDi • whereCiis classification of learning example ei goal • learnprediction model (hypoteshis) H IF ... THEN ... IF ... THEN ... ... H ei: Di Ci ai • example ei may have argument ai

6. It is MuchEasier to ExplainIndividualCases! Is thismushroomtoxic? Why?

7. ABML Knowledge RefinementLoop Step 1: Learn a hypothesis with ABML Step 2: Find the “most critical” example (if none found, stop) Step 3: Expert explains the example Return to step 1 learn data set Argument ABML critical example

8. ABML Knowledge Refinement Loop Step 1:Learn a hypothesis with ABML Step 2:Find the “most critical” example (if none found, stop) Step 3:Expert explains the example Return to step 1 Step 3a: Explaining a critical example (in a natural language) Step 3b:Adding arguments to the example Step 3c: Discovering counter examples Step 3d:Improving arguments Return to step 3c if counter example found

9. IllustrativeExample: Learning to DiagnoseFlu • Thecurrent model: • IF Temperature<veryhighTHEN Flu= no • cannotexplainwell Pacient 2. • Thequestion to theexpert: • „What is thereasonfor Pacient 2 havingtheflu?“

10. Expert‘sExplanation Expert‘sexplanation: „Pacient#2 hasthefluebecauseof a high temperature.“ • Expert‘s argument • is attached to learningexample #2. New model is built. Flu = yes BECAUSE Temperature> Normal

11. WhatiftheExpert‘s Argument is not goodenough? ML methodnowinduced a rule consistentwithargument: IF Temperature > Normal THEN Flu= yes The rule is inconsistentwith data! Expert is presentedwithcounterexample: „Comparepacients #2 and #4. Why Pacient #4 doesn‘thavetheflu?“

12. TheExpertmayImprovethe Argument ExpertfindsthecrucialdifferencebetweenPacients #2 and #4: „Pacient 2 didn‘tgetvaccinatedagainsttheflu.“

13. ImprovedRulesmayExplainUnseenExamples as Well ML methodinduces a new rule: IF Temperature > Normal AND Vaccination = no THEN Flue= yes • Thenew rule alsoexplainsdiagnosisfor Pacient #3: • „Hasflubecauseof a high temperature anddidn‘tgetvaccinatedagainst it.“

14. ABML Knowledge Refinement Loop: TheInnerLoop • Step 3a: Explaining a critical example (in a natural language) • „Pacient#2 hasthefluebecauseof a high temperature.“ Step 3b:Adding arguments to the example Temperature > Normal Step 3c: Discovering counter examples Step 3d:Improving arguments with counter examples IF Temperature > Normal AND Vaccination = no

15. ABML Refinement Loop & Knowledge Elicitation critical examples counter examples IF ... THEN ... IF ... THEN ... ... ABML argument-based machine learning arguments easier for experts to articulate knowledge explain single example expert provides only relevant knowledge “critical” examples detect deficiencies in explanations “counter” examples

16. ExpertcanIntroducenewConcepts (Attributes) • FluSymptoms • no • yes • yes • no • ... Possible rule withthenewattribute: IF Temperature > Normal AND FluSymptoms = yesTHEN Flu= yes

17. ExpertcanCorrectClassificationofLearningExample • no Thequestion to theexpert: „What is thereasonfor Pacient 37 havingtheflu?“ Expertcorrectstheclassificationof Pacient 37: „Pacient 37 doesn‘thavetheflu.“

18. Knowledge Elicitation with ABML critical examples counter examples IF ... THEN ... IF ... THEN ... ... ABML argument-based machine learning arguments experts introduce new attributes human-understandable models suitable for teaching inconsistencies in labels are detectedautomatically misclassificatedexamples are easilyrecognizedandcorrected

19. IntroducingStudents critical examples counter examples IF ... THEN ... IF ... THEN ... ... ABML argument-based machine learning arguments How to use ABML in educationalsetting?

20. TheOuterLoop Step 1:Learn a hypothesis with ABML Step 2:Find the “most critical” example (if none found, stop) Step 3:Studentexplains the example Return to step 1

21. TheOuterLoop & theInnerLoop Step 1:Learn a hypothesis with ABML Step 2:Find the “most critical” example (if none found, stop) Step 3:Studentexplains the example Return to step 1 USING TEACHER‘S ATTRIBUTES! Step 3a: Explaining a critical example (in a natural language) Step 3b:Adding arguments to the example Step 3c: Discovering counter examples Step 3d:Improving arguments with counter examples Return to step 3c if counter example found

22. Arguing to Learn Argumentationinvolves elaboration, reasoning, and reflection. These activities have been shown to contribute to deeper conceptual learning (Bransford, Brown, & Cocking, 1999) Participatingin argumentation helps students learn about argumentative structures (Kuhn, 2001)

23. A New Paradigm arguing to learn with argument-basedmachinelearning

24. Let‘ssee how thisworks in practice

25. BasicorAdvanced? „basic“ „advanced“

26. Learning Data Set 121 solutionsof 62 differentexercises teacherlabeledeachsolution as „basic“or„advanced“ test data: 30 examples learn data: 91 examples

27. KnowledgeElicitationfromtheTeacher TEACHER‘S GOALS: relevantdescriptionlanguage: newattributes consistentlylabeledexamples

28. ResultsofKnowledgeElicitationfromtheTeacher • 9iterations • 9newattributes • 9rules • only1 out of 5 initialattributesremained

29. A Student-Computer Interactive Learning Session • STUDENT‘S TASK • obtainrulesfordistinguishing„basic“and„advanced“solutions rules must consist of attributesin teacher‘sfinal model useteacher‘sdescriptivelanguage

30. A Student-Computer Interactive Learning Session • RECOMMENDATIONS TO THE STUDENT • usethe most important featuresforexplanations • use the smallest possible number of features in a single argument • try not to repeat the same arguments

31. The First „Critical“ Example • Thequestion to thestudent: • „Why is thissolutionadvanced?“ Student‘s argument: „Becausefunction zip is present and the number of rows is low.“ • Solution= advanced BECAUSE Zip=True AND cRows = low

32. CounterExample • IF Zip=True AND cRows = low THEN Solution= advanced The rule is inconsistentwith data! Student is presentedwithcounterexample: „Comparethesetwosolutions. Whyis thesecondsolution a basic one?“

33. Improving Argument Student‘sextended argument: „Becausefunction zip is present,the number of rows is low, anda list comprehensionoccurs. “ • IF Zip=True AND cRows = lowAND LiCom = TrueTHEN Solution= advanced nextiteration no more counterexamples

34. At theEndoftheInteractiveSession • 5iterations • half anhour • 90%accuracy on (previouslyunseen)testing data • severalsuggestionsofnewdescriptivefeatures

35. Assesment • Results: experimentwith 7 students • 7.1iterations • 87.1%classificationaccuracyofobtained rule model • 86.7%correctly „manually“ classified (previouslyunseen) examples • verypositivequalitativefeedbackfromthestudents

36. Conclusions New paradigm: arguingto learnwithargument-basedmachinelearning • Future work: • applications in severaldomains • assessingargument‘squalityforimprovedimmediatefeedback • goal-orientedextension (seeour ITS 2012 paper)

37. Questions & Discussion Thankyou! enablingarguing to learn slides: ailab.si/matej