Ontology-based Open-Corpus Personalization for e -Learning

1. Ontology-based Open-Corpus Personalization for e-Learning Sergey Sosnovsky Committee: Peter Brusilovsky DarinaDicheva Daqing He Chad Lane Michael Spring School of Information Sciences, University of Pittsburgh

2. Definition: Open-corpus vs. Closed-corpus • Definition 1 (Closed-corpus adaptive system):“…operates on a closed corpus of documents, where documents, relations between the documents, and relations between the documents and the knowledge models of the system are defined at design time” • Definition 2 (Open-corpus adaptive system):“…operates on a open corpus of documents, where documents, relations between the documents, and relations between the documents and the knowledge models of the system are not known at design time and, moreover, can constantly change and expand” School of Information Sciences, University of Pittsburgh

3. A Closed-Corpus Adaptive System School of Information Sciences, University of Pittsburgh

4. A Closed-Corpus Adaptive System in the Open-Corpus Settings Missing link School of Information Sciences, University of Pittsburgh

5. Open-corpus Personalization: Step by Step • discovery of open-corpus content • identification of individual learning objects in open-corpus content • modeling of open-corpus content • connection of open-corpus learning objects to each other and to the preexisting content • providing adaptive access to open-corpus content • tracing of users’ interaction with the open-corpus content • user modeling based on interactions with the open-corpus content School of Information Sciences, University of Pittsburgh

6. The Proposed Approach Compared to the Existing Solutions School of Information Sciences, University of Pittsburgh

7. Structured e-Learning Content on WWW Textbooks Encyclopedias • High-quality, domain-oriented • Their content has been chosen, structured and formatted for the purpose of explaining the domain knowledge by the experts in the domain • The structure of the domain knowledge have been used as a mean for organizing these resources • If extracted, this structure represents the model of the resource and the model of the domain as the author understands it Tutorials Dictionaries Digital Libraries Links between Pages Headers of Sections Table of Contents Content Formatting School of Information Sciences, University of Pittsburgh

8. Structured content on WWW School of Information Sciences, University of Pittsburgh

9. Structured content on WWW School of Information Sciences, University of Pittsburgh

10. Structured content on WWW School of Information Sciences, University of Pittsburgh

11. Structured content on WWW School of Information Sciences, University of Pittsburgh

12. Proposed Approach • Extraction of a topic-based model from an open-corpus resource • Mapping a topic-based model into the central reference ontology • Maintaining open-corpus personalization School of Information Sciences, University of Pittsburgh

13. Step1: Extraction of Open-Corpus Models School of Information Sciences, University of Pittsburgh

14. Step 2: Mapping Topic-based Models into a Reference Ontology School of Information Sciences, University of Pittsburgh

15. Step 3: Open-Corpus Personalization School of Information Sciences, University of Pittsburgh

16. Ontology-based Open-corpus Personalization Service • Recommends supplementary reading as a list of links to the topics’ content • Works as a value-added service foran exercise system (QuizJET) • Implements two adaptation technologies: • Context-based adaptive link recommendations/ordering • Knowledge-based adaptive link annotation • Domain: Java Programming School of Information Sciences, University of Pittsburgh

17. OOPS Interface: Recommendation Phase A QuizJET Exercise List of open-corpus topics recommended and annotated by OOPS School of Information Sciences, University of Pittsburgh

18. OOPS Interface: Reading Phase Feedback/exitbuttons Navigation links to the next and the previous topics content of the chosen topic School of Information Sciences, University of Pittsburgh

19. Personalization in OOPS School of Information Sciences, University of Pittsburgh

20. Evaluation: Basic Info • Subjects: • 40 undergraduate and graduate Pitt students with little experience in Java programming • Material: • Two sets of introductory Java topics: • Easy: objects, classes, conditionals • Difficult: loops, arrays, ArrayLists • Task: • “Solve QuizJet exercises, use supplementary reading, when necessary” School of Information Sciences, University of Pittsburgh

21. Evaluation: Schedule Easy topics Difficult topics (17 exercises) (16 exercises) School of Information Sciences, University of Pittsburgh

22. Evaluation: Three Versions of the System Adaptive recommendation of Open-corpus content The original textbook instead of recommendation Adaptive recommendation of Closed-corpus content--------------------------------------Based on manual indexing of the textbook topics with the concepts of the ontology School of Information Sciences, University of Pittsburgh

23. Evaluation: Experiment Design 40 students School of Information Sciences, University of Pittsburgh

24. Evaluation: Data Collection • Pre-test & Post-test => • Questionnaire • Transaction logs from QuizJET • Transaction logs from OOPS • A topic recommended • A recommendation accepted • Browsing occurred • A recommendation exited (by using which button) School of Information Sciences, University of Pittsburgh

25. General Learning Effect • H1: Scrorepre-test< Scorepost-test across all groups and conditions      p = 0.089 School of Information Sciences, University of Pittsburgh

26. Effect on Learning Difficult Material  • H2: KGopen-corpus = KGclosed-corpus for easy material • H3: KGopen-corpus = KGclosed-corpus for difficult material • H4: KGopen-corpus > KGtextbook for easy material • H5: KGopen-corpus >Kgtextbook for difficult material       p = 0.043 School of Information Sciences, University of Pittsburgh

27. Effect on Learning Conceptual Material ≈ • H6,7: KGopen-corpus > KGtextbook for conceptual learning material There are two values of type boolean: ______________ and ______________ • What is the output of the following code segment? • inta = 3 + 3; • int b = 2 + 2; • if (a != b) System.out.println(“ Not equal ”); • if (a == b) System.out.println(“ Equal ”); p = 0.023 p = 0.089 School of Information Sciences, University of Pittsburgh

28. Recommendation Acceptance Rate • H8.9: RecAccRateopen-corpus = RecAccRateclosed-corpus    School of Information Sciences, University of Pittsburgh

29. Perceived Value of Recommendation   • H12,13: PersRecValueopen-corpus = PersRecValueclosed-corpus   School of Information Sciences, University of Pittsburgh

30. Questionnaire: Open-Corpus vs. Closed-Corpus Recommendation • I benefited from the materials recommended by the OOPS service • The recommendations of the OOPS service did not help me to solve the questions • Most often, when I clicked on a recommended link, I had to browse to find a helpful page • Most often, I was not able to find a helpful page even by browsing • Most often, I did not have to browse, one of the recommended pages was helpful for the question • Most often, the helpful page was the first or second in the recommended list Open-corpus Closed-corpus School of Information Sciences, University of Pittsburgh

31. Questionnaire: Open-Corpus vs. Textbook • The usual textbook was more helpful for solving questions than the recommendations of the OOPS service • It was easier to find a helpful page among the recommendations of the OOPS service than in a textbook • OOPS recommendations were better only because they were accessible within the same system, which was easier to use than the book • OOPS recommendations were better because OOPS brought the helpful pages much closer, and I did not have to look for them in the entire book Open-corpus School of Information Sciences, University of Pittsburgh

32. Contributions of this dissertation • A fully-automated approach towards open-corpus personalization and semantic open-corpus content modeling for e-Learning • An adaptive e-Learning service implementing the approach • A new algorithm for mapping ontologies and topic models • Evaluation experiment demonstrating: • several significant learning effects of the developed service compared to a regular textbook • no significant difference between the open-corpus and the closed-corpus recommendation from point of learning • quality of open-corpus recommendation comparable to the closed-corpus one (based on both transactional logs and subjective evaluation) • subjects strongly prefer OOPS over a textbook for the “right” reason School of Information Sciences, University of Pittsburgh

33. Future Work Directions • Further evaluation of the proposed approach: • A larger-scale control study • Evaluation in a course settings • Evaluation with multiple collections of open-corpus content • Further Development of the service: • Implementation in other domains • Model extraction component • Mapping algorithm • Cross-mapping with multiple collections • Use the social feedback to refine the recommendation School of Information Sciences, University of Pittsburgh

34. Acknowledgements School of Information Sciences, University of Pittsburgh

35. Thank you Questions? School of Information Sciences, University of Pittsburgh

36. Characteristics of the Approach • it supports a full cycle of open-corpus personalization • it allows for creation of semantic models of open-corpus content • it requires little to no involvement of a human author • it is domain-independent • it is compatible with a wide range of adaptation technologies • it is potentially applicable in a wide range of application fields beyond e-Learning School of Information Sciences, University of Pittsburgh

37. OOPS: Architecture School of Information Sciences, University of Pittsburgh

38. Modeling open-corpus content School of Information Sciences, University of Pittsburgh

39. Knowledge-based Adaptive Navigation School of Information Sciences, University of Pittsburgh

40. Indexing of QuizJET questions School of Information Sciences, University of Pittsburgh

41. Recommendation statistics School of Information Sciences, University of Pittsburgh

42. Exercises statistics School of Information Sciences, University of Pittsburgh

43. Questionnaire: General Questions + Questions about Adaptive Annotation • The adaptive annotations of the recommended links helped me to choose appropriate recommendations • I did not use the adaptive annotations of the recommended when choosing the recommendations • I think the idea of recommending reading material to help solve the question is good, but implementation could be better Open-corpus • I like the interface of the OOPS service • I tried to use the help of the OOPS service School of Information Sciences, University of Pittsburgh

44. Algorithm for Topics to Ontology Mapping (ATOM) • Input: the central ontology and a topic-based model • Output: matrix of similarities between concepts and topics School of Information Sciences, University of Pittsburgh

45. Average Accepted Recommendation Rank  • H10,11: AvgAccRecRankopen-corpus = AvgAccRecRankclosed-corpus   School of Information Sciences, University of Pittsburgh