Creating highly engaging online discussions to support sustainable formative learning

1. Creating highly engaging online discussions to support sustainable formative learning Brant Knutzen Learning Designer Faculty of Education University of Hong Kong

2. Overview • Social Constructivism • Transactivity • Building engagement • Assessment methods • Participation rating • Measuring engagement

3. Computer-mediated Conferencing • CMC : Computer-mediated Conferencing • Synchronous : “chat”, or instant messaging • Asynchronous : “discussion forum” • Asynchronous • Participants take turns posting • Minutes, hours, or days apart • More time to digest, research, compose response

4. Learning Interactions using CMC

5. Social Constructivism • The effective construction of knowledge is a product of a functional collaborative group producing artifacts for public display and use • Efficacy has been found to be linked to the process that learners utilize in working on the task together (Fischer et al 2002) • Process: social negotiation of arguments and argument sequences (Leitão 2000; Voss & Dyke 2001)

6. Social Constructivism • John Biggs captured the educational value of discussion when he stated: "Good dialogue elicits those activities that shape, elaborate, and deepen understanding“ (Biggs 1999 p. 5)

7. Transactivity • Transactivity: the method by which students build on the contributions of their fellow learners (Berkowitz & Gibbs 1983)

8. Transactive online discussions • Transactive communication: • Participants respond to and build on each other’s contributions • Peer exchange of information and ideas • Social negotiation of knowledge • Each participant brings their own experiences to apply to a common educational goal

9. Transactivity • A key theoretical construct for measuring collaboration • How can we formulate the instructional design conditions which consistently result in more productive and transactive learning activities? • How can we describe it in easily grasped ways? • Quantitative • Qualitative

10. Online discussion – the potential • An online discussion is a great formative learning activity for building depth of student understanding, or supporting project work • Highest potential for social construction of knowledge and transactive collaboration • Unfortunately, this activity is also the most likely to FAIL! Why?

11. Study #1 – 2007 – Building Engagement • Case study of one course (Knutzen, 2007) • International school in Hong Kong – secondary level • 1-to-1 laptop blended learning environment • Introduction to Psychology course • Sample size = 24 • Investigation of instructional design conditions to achieve a highly productive online discussion • At start of study, average student production in optional online discussions = 1 post

12. Productive Communication • Four conditions to achieve productive online discussions: • Teacher facilitated social formation of small groups • Class time to initiate oral and online discussion interaction • Setting open-ended, challenging topic questions that encourage discussion and debate • Assessment system that reinforces production and peer interaction • At end of study, average studentproduction: over 10 posts per discussion!

13. Extensive use of discussion design • Over the past six years: • Extensive use of the online discussion design • Full-time instruction of secondary students • 1-to-1 laptop environment • IT classes • Psychology • Implemented at University of Hong Kong: 2009 • MSc IT in Education course on eLearning • Implemented in 12 courses at LingnanUniv : 2010 • Business, History, Social Studies, Philosophy, English • Design continues to result in good production

14. Designing discussion topic questions • One to three questions around one topic or area of content / concepts • Advantage of multiple questions: • Instructor can design a “gradient” of difficulty which can elicit a range of student answers • From basic knowledge -> higher-order thinking skills (HOTS) • Use a taxonomy of active verbs to specify the levels of understanding expected in answers (Blooms Revised, SOLO) • Examples: Multi-structural (list, describe, classify)Relational (compare/contrast, explain, analyze, relate)Extended Abstract (hypothesize, generate, reflect)

15. Designing discussion topic questions • Objective of multiple questions: • Make discussion accessible to all students • Challenge the advanced students • Other topic question gradients found to be useful: • Concrete facts -> abstract concepts • Textbook context -> personal context (unique answers!)

16. Designing discussion topic questions Example of a Topic Question gradient: Can you demonstrate what you have learned in your study of the Porter management models? • Can you list and describe the Porter models? (Basic understanding ). • How can you compare Porter's models? (Relational understanding)Can you relate these models to each other in several ways, or on several dimensions?. • Based on these models, can you create your own model? What factors do you theorize are important, and why? (Extended abstract understanding)

17. Discussion marking schemes • Traditional – teacher-assessed subjective marking • Review contributions by each student • Award mark based on: • Participation – any contribution to discussion • Interaction - responding and seeking feedback • Transaction – sharing / exchanging useful information and resources • Transformation – perspective change due to interaction with others • Best method for summative assessment • Should a discussion be a summative activity?

18. Using a Rubric for assessment • Criteria examples: • Locate, organize, analyze, evaluate, synthesize, and ethically use information from a variety of sources and media • Interact, collaborate, and publish with peers, experts, or others employing a variety of digital environments and media • Develop cultural understanding and global awareness by engaging with learners of other cultures • Contribute to project teams to produce original works or solve problems Source: ISTE NETS for Students

19. Problems with teacher assessment • A highly productive discussion can easily produce over 200 posts a week! • A teacher can become a victim of their own success • How much time can they devote to quantitative marking? • How much time remains for qualitativefeedback?

20. Peer-assessment • Desired graduate attributes: • Critical thinking skills • Excellent cooperative skills • Integrity • Personal responsibility • Subjective peer-assessment can directly address the development of these attributes • Requires student training • Requires review and evaluation by teacher

21. Peer-assessment - Subjective • Possible problems: • Revenge grading : 報復 • “you gave me a low grade, I will give you a low grade” • Back-scratching : 賄賂 • “If you give me a high grade, I will give you a high grade “

22. Sustainable forum assessment? • Teacher evaluation of forums: time intensive • Peer evaluation of forums: requires monitoring • Subjective evaluation of formative work tends to distort student learning • Optional forums do not get sufficient participation to create viable discussions

23. Leveraging the power of the LMS • Mysolution: automate the rating based on participation • No subjective judgment, just rating using a systematic method: • Moodle can automatically average these grades!

24. Rating based on Participation • Moodle averages the ratings • Grades produced by participation: • One post = 6 -> D- • Two posts = 8 -> B- • Three posts = 8.6 -> B • Four posts = 9 -> A- • Five posts = 9.2 -> A- • Six posts = 9.33 -> A • Seven posts = 9.42 -> A • Eight posts = 9.5 etc • More Q&A participation = higher grade

25. More participation -> higher grade

26. A foundation of participation • This assessment method for assessing a forum creates a foundation of participation: on average over 90% of students will engage • As always, the teacher’s role is to create the educational value of the activity: • Set up challenging topic questions to guide exploration • Maintain a “visible presence” in the forum activity • Successfully moderate the discussion

27. Discussion moderation techniques • Salmon’s 5-stagemodel ofeModeration • Teacher guidesstudents upthe levelstowardknowledgedevelopment Source: http://www.atimod.com/e-moderating/5stage.shtml

28. Participation-based forum grade • The Moodle LMS handles the awarding of quantitative marks for participation • Teacher can focus on quality: • Monitor progress • Guide the discussion • Challenge students • Time for thoughtful qualitativefeedback • Minimum: 30 mins to 1 hour each week per discussion

29. Motivating students to engage • Motivation on three levels: • Constructive alignment between formative discussions and summative final assignmentsExamples: • Reflective posts build towards a Learning Portfolio blog • Group collaboration on final project development • Social Constructivism: a sense of social obligation to support the group • Quantified Participation: each contribution is counted as participation towards a forum grade

30. The Participation Forum • Automates the awarding of participation points • Reduces technical requirements (modifying roles) • No need to train the students • No need to monitor rating accuracy • No need to motivate student ratings of posts • Finished development and testing in Dec 2011 • Currently in trials at HKU • Planned release as open-source project soon

31. Measures of discussion activity • Quantitative: • Production = Total number of discussion posts / n • Interactivity = Total number of feedback posts / n • Group Activity = Total number of discussion posts / # topics • Transactivity = Production × Interactivity • Qualitative: a new type of graphical representation – the “Participation Map” • Developed at Lingnan University in 2011 • Continued development as open source project

32. The Participation Map • New Moodle plug-in automates the production of the map of discussion activity: • Quantitative statistics • Qualitative graphic display: a “data portrait” • Useful for displaying discussion activity • Feedback to students in the class (who is working?) • Feedback to teacher (what approach is working?) • Anonymous mode for reporting research results

33. Example Participation Map Overall discussion statistics Red line shows rating cutoff Time scale from Day 1 to Day 14 Students can interact in multiple groups Student profile pictures Group comparison stats

34. Quantitative Data • Quantitative ratings of overall discussion

35. Participation Map is a teacher tool • After install on Moodle server, this plugin is only available to those with Teacher roles • New option on CourseAdministration block

36. Participation Map operation • Select theforum, andthen plot type: • Normal plotfor feedbackto teacher andstudents • Anonymousfor reportingresults outside

37. Questions and Discussion • Currently setting up a website to make these open source projects available to the Moodle educational community: Brant.Knutzen.se • Goal is to release both as free open source plugins for Moodle 2 by Sep 2012 • Q & A ?? • My email is brantknutzen@gmail.com