ADAPTIVE ASSESSMENT IN WEB-BASED LEARNING

1. ADAPTIVE ASSESSMENT IN WEB-BASED LEARNING By Dunwei Wen (1), Sabine Graf (2), Chung Hsien Lan (3), Terry Anderson (1), Kinshuk (1), Ken Dickson(1) (1) Athabasca University, Alberta, Canada (2) Women's Postgraduate College for Internet Technologies, Vienna University of Technology, Austria (3) Nanya Institute of Technology, Jhongli, Taiwan Presenter: Dunwei Wen IEEE ICME 2007 - International Conference on Multimedia & Expo, July 2-5, 2006, Beijing

2. OUTLINE • I. Am I Ready – An Adaptive Readiness Self-assessment System • 1. Introduction • 2. Knowledge Modeling • 2.1. User Model • 2.2. Counseling Model • 2.3. Process Model • 2.4. Assessment Model • 3. System Structure And Realization • 4. Conclusion of Am I Ready • II. Performance Self-Assessment • III. Peer Assessment IEEE International Conference on Multimedia & Expo, Beijing, 2007

3. AM I READY for Athabasca Unv.A DISTANCE EDUCATION READINESS SELF-ASSESSMENT SYSTEM • Features • Is an adaptive online readinessself-assessment system • Helps prospective students understand their • requirements • readiness • Provides information of distance learning at Athabasca University. • Based on integrated knowledge models • dynamically adjusts the contents of the self-assessment according to the interaction between • the knowledge models • the user’s responses. IEEE International Conference on Multimedia & Expo, Beijing, 2007

4. 1. Introduction • Online readiness self-assessments are widely used in universities that provide distance education services. • Most current readiness self-assessment tools are online questionnaires. • The shortcoming of those systems: • The same question set for different users • The same question sequence for every user • Less remedial information to help users IEEE International Conference on Multimedia & Expo, Beijing, 2007

5. BASIC IDEAS - features • User-oriented • Questions are filtered for different users • Dynamic and adaptive • Next questions rely on users’ previous answers • Remedial • Instant feedback and remedial assessment information IEEE International Conference on Multimedia & Expo, Beijing, 2007

6. BASIC IDEAS – methods • Technologies to model counselling knowledge • User Model • Counselling Model • Process Model • Assessment Model. • Operators adopted according to the above models of a user: • Enabling some questions • Disabling some questions • Sorting questions by priorities of questions • Checking contradictory answers • Showing real time information IEEE International Conference on Multimedia & Expo, Beijing, 2007

7. 2. Knowledge Modeling And Architecture IEEE International Conference on Multimedia & Expo, Beijing, 2007

8. 2.1 USER MODEL (1)STATIC MODEL Acting as the first assessment filter: • Highest level of education • Employment status • Financial capability • Computer skills • Disability • Full or partial assessment IEEE International Conference on Multimedia & Expo, Beijing, 2007

9. STATICUSERMODEL IEEE International Conference on Multimedia & Expo, Beijing, 2007

10. (2)DYNAMIC MODEL Records of a user’s response history are dynamically modified in real time as the assessment proceeds. • The answer to each question • The records of questions that are disabled, enabled, or changed to higher priority IEEE International Conference on Multimedia & Expo, Beijing, 2007

11. 2.2. COUNSELLING MODEL • Modeling questions • Dividing counselling information into fields and their sub-fields • Assigning questions to each sub-field • Setting up relations between questions • Authoring instant information • Instant information (including descriptions, links to Web pages with multimedia resources etc) can be assigned to each choice of a question. They will show whenever you make a choice. • Pre-defined answer types • Yes and No • Yes, No and Not Clear • Grades • Multi-Checks IEEE International Conference on Multimedia & Expo, Beijing, 2007

12. Questions &InstantInformation IEEE International Conference on Multimedia & Expo, Beijing, 2007

13. FOUR KINDS OF RELATIONS • Enable • Disable • Plus • Contradictory Examples: (Enable (answer(i) of question(j), ( question list ) (sub-field list) ) ) (Contradict (answer(i) of question(j), (list of answers of some questions))) IEEE International Conference on Multimedia & Expo, Beijing, 2007

14. 2.3 PROCESS MODEL Modeling the behavior that controls the counseling process • Actions: • Utilizing user model and counseling model • Changing question priority in real time • Sorting questions by priority • Filtering questions IEEE International Conference on Multimedia & Expo, Beijing, 2007

15. NEXT PAGE IEEE International Conference on Multimedia & Expo, Beijing, 2007

16. 2.4 ASSESSMENT Provides assessment information in response to user’s answers • Two kinds of assessment information: • Question related • Question-group related Example: (Assessment (i)  (answer(i1)of question(j1), (answer(i2)of question(j2), … ) ) IEEE International Conference on Multimedia & Expo, Beijing, 2007

17. 3. SYSTEM STRUCTURE IEEE International Conference on Multimedia & Expo, Beijing, 2007

18. ProcessInteraction IEEE International Conference on Multimedia & Expo, Beijing, 2007

19. REALIZATION • Putting the knowledge into PostgreSQL database (21 tables) • including rules, relations, facts and text information • Determining the question sequences • Reasoning by the support of SQL • Interacting with users • Java/JSP based HTML Webpage Dialog IEEE International Conference on Multimedia & Expo, Beijing, 2007

20. 4. CONCLUSION Am I Ready • User oriented & user specific self-assessment • Dynamic & adaptive self-assessment process • Real time guidance and remedial information to improve the readiness of students for distance education • More effective and more natural self-assessment process • Flexible design for more general use IEEE International Conference on Multimedia & Expo, Beijing, 2007

21. II. Performance Self-Assessment • How can performance self-assessment help to address the different learning styles of students in learning management systems? • For providing adaptive courses • For improving adaptivity by gathering additional information about the students IEEE International Conference on Multimedia & Expo, Beijing, 2007

22. Motivation • Learning Management Systems (LMS) are often and successfully used in e-education but provide little or in the most cases no adaptivity • Learners have different needs • Considering learning styles makes learning easier and increases the learning progress IEEE International Conference on Multimedia & Expo, Beijing, 2007

23. Felder-Silverman Learning Style Model • Richard M. Felder and Linda K. Silverman, 1988 • Each learner has a preference on each of the four dimensions: • Active – Reflective learning by doing – learning by thinking things through learning by discussing & group work – work alone • Sensing – Intuitive concrete material – abstract material more practical – more innovative and creative patient and careful – not patient and careful with details standard procedures – challenges • Visual – Verbal learning from pictures – learning from words • Sequential – Global learn in linear steps – learn in large leaps good in using partial knowledge – need “big picture” interested in details – interested in the overview IEEE International Conference on Multimedia & Expo, Beijing, 2007

24. Adaptive Features for Self-Assessments • Courses can be adapted with respect to the order and number of included learning objects • Regarding self-assessments we consider: • Theoretical tests • Practical exercises • Adaptation features for theoretical tests • Presentation at the beginning of each chapter • Presentation at the end of each chapter • Presentation at the end of the course • Adaptation features for practical exercises • Number of presented exercises • Presentation at the beginning of each chapter • Presentation at the end of each chapter IEEE International Conference on Multimedia & Expo, Beijing, 2007

25. Adaptation with respect to learning styles • Active learning style: • Theoretical tests and exercises at the beginning and end of each chapter (due to the preference for active learning) • High number of exercises • Reflective learning style: • First present learning material, then self-assessments  students have time to reflect about the material • Low number of exercises • Sensing learning style: • First present learning material, then self-assessments  students can use the learned material for performing the self-assessments • High number of exercises (due to the preference for problem solving) • Intuitive learning styles: • Present self-assessments first  provide learners with challenges • Low number of exercises IEEE International Conference on Multimedia & Expo, Beijing, 2007

26. Adaptation with respect to learning styles • Sequential learning style: • Avoid theoretical tests at the end of the course (since sequential learners prefer to test their knowledge in short intervals) • Theoretical tests at the end of each chapter are more suitable • Global learning style: • Avoid self-assessments at the beginning of the chapter (since global learners need the big picture for performing self-assessments) • Theoretical tests can be provided at the end of the course IEEE International Conference on Multimedia & Expo, Beijing, 2007

27. Self-Assessments for Improving Adaptivity • Providing adaptivity requires knowing the students’ learning styles • Self-assessments can be used to gather additional information from learners • This information can help to identify the learning styles of students more accurately and therefore enable the system to provide more suitable adaptivity • Examples for patterns: • Time students spend on self-assessments  sensing/intuitive • Number of performed exercises  active/reflective • Performance on questions about multimedia content  visual/verbal • Performance on questions about details and overview  sequential/global IEEE International Conference on Multimedia & Expo, Beijing, 2007

28. III. Peer Assessment Peer assessment is one form of group assessment The issue of fairness has to be concerned in group assessment The proposed methodology can aggregate students’ marks and consider individual learning styles to Reduce personal bias Enhance the accuracy of the assessment IEEE International Conference on Multimedia & Expo, Beijing, 2007

29. Adaptive Peer Assessment The process of adaptive peer assessment Detect reviewers’ learning styles Learning Styles: Active/reflective, sensing/intuitive, visual/verbal, and sequential/global Consider assessment criteria and learning styles Assessment criteria: Creativity, Completeness, Execution, and Security Aggregate all marks and sends the result to the original author Assessment feedback = w1x1+w2x2+…wixi Weight of learning styles Assessment criteria IEEE International Conference on Multimedia & Expo, Beijing, 2007

30. Relations between Assessment Criteria and Learning Styles Execution and Security Active students tend to be experimentalists Completeness Sensing students like solving problems by standard methods and are patient with detail Creativity Intuitive students like innovation IEEE International Conference on Multimedia & Expo, Beijing, 2007

31. FUTURE WORK • A knowledge editor to enhance the speed and accuracy of knowledge modeling, modification and upgrade of AM I READY. • Graphical statistical information representation of readiness in self-assessment report. • Data mining for analyzing the question-answer pairs and their relations, and expanding the knowledge base. • A formal knowledge description and a SQL-based reasoning method. • Distance education readiness self-assessment is only a rudimentary step to general dynamic and adaptive counseling systems. • More multimedia resources. IEEE International Conference on Multimedia & Expo, Beijing, 2007

32. THANK YOU! Questions? IEEE International Conference on Multimedia & Expo, Beijing, 2007