Carolyn Penstein Rosé Language Technologies Institute Human-Computer Interaction Institute

1. Souflé: A Three Dimensional Framework for Analysis of Social Positioning in Dyadic and Group Discussions Carolyn Penstein Rosé Language Technologies Institute Human-Computer Interaction Institute School of Computer Science With funding from the National Science Foundation and the Office of Naval Research

2. Developing technology capable of shaping conversation and supporting effective participation in conversation to achieve positive impact on… Human learning Health Wellbeing

3. Developing technology capable of shaping conversation and supporting effective participation in conversation to achieve positive impact on… Human learning Human learning Health Wellbeing

4. Outline • Introducing the Problem of Supporting Productive Discussion for Learning • Discussion of Souflé • Transactivity • Engagement • Authoritativeness • Application to Dynamic Support for Group Learning • Conclusion and Current Directions

5. Outline • Introducing the Problem of Supporting Productive Discussion for Learning • Discussion of Souflé • Transactivity • Engagement • Authoritativeness • Application to Dynamic Support for Group Learning • Conclusion and Current Directions

6. Reward structures encourage students to focus on performance over learning • Well crafted instruction provides opportunities for learning • Opportunities only help if students take them Purpose of Interaction in Learning Contexts Take Home Message: Introducing reflection points provides opportunities for students to take advantage of learning resources

7. Important Ingredients for Learning Reflection through Rich Interaction Carefully Structured Conceptual Knowledge

8. Second-Year Thermodynamics • End of Fall Semester: Students learn about Rankine Cycles • 1 Week of lectures • Homework assignment on analysis of Rankine Cycles • Tutorial on using CyclePadsoftware package (Developed at Northwestern University (Forbes et. al. 1999) • Allows students to construct and analyze a variety of Thermodynamic Cycles) • Instructed on Effects of Changing System Variables (Temperature, Pressure) on System Output (Power, Waste Heat)

9. Collaborative Task • Learning Goal: Encourage students to reflect on interactions between cycle parameters • Design Goal: Design a power plant based on the Rankine Cycle paradigm • Competing Student Goals: • Power: Design a power plant that achieves maximum power output • Motivated by economic concerns • Green: Design a power plant that has the minimum impact on the environment • Motivated by environmental concerns • Each pair turns in exactly one design Increasing heat increases power but also waste heat Increasing pressure increases efficiency Reduction in Steam Quality Power Waste Heat

10. Computer Supported Collaborative Learning

12. Outline • Introducing the Problem of Supporting Productive Discussion for Learning • Discussion of Souflé • Transactivity • Engagement • Authoritativeness • Application to Dynamic Support for Group Learning • Conclusion and Current Directions

13. 3 Dimensions: • Transactivity • Engagement • Authoritativeness Souflé Framework(Howley et al., in press) Souflé Framework(Howley et al., in press) Person Person

14. Sociolinguistics Language Use Computational Models Of Discourse Analysis Discourse Analysis Machine Learning Language And Identity Applied Statistics Multi- Level Modeling

15. Souflé Framework(Howley et al., in press) Transactive Knowledge Integration Person Person

16. i • Definition of Transactivity • building on an idea expressed earlier in a conversation • using a reasoning statement well, for power and efficiency, we want a high tmax, but environmentally, we want a lower one. We don't want tmax to be at 570 both for the material and [the Environment]

20. Findings • Moderating effect on learning (Joshi & Rosé, 2007; Russell, 2005; Kruger & Tomasello, 1986; Teasley, 1995) • Moderating effect on knowledge sharing in working groups (Gweon et al., 2011) • Computational Work • Can be automatically detected in: • Threaded group discussions (Kappa .69) (Rosé et al., 2008) • Transcribed classroom discussions (Kappa .69) (Ai et al., 2010) • Speech from dyadic discussions (R = .37) (Gweon et al., 2012) • Predictable from a measure of speech style accommodation computed by an unsupervised Dynamic Bayesian Network (Jain et al., 2012) Transactivity (Berkowitz & Gibbs, 1983)

21. Souflé Framework(Howley et al., in press) Transactive Knowledge Integration Person Person Engagement Engagement

22. System of Engagement • Showing openness to the existence of other perspectives • Less final / Invites more discussion • Example: • [M] Nuclear is a good choice • [HE] I consider nuclear to be a good choice • [HC] There’s no denying that nuclear is a superior choice • [NA] Is nuclear a good choice? Engagement(Martin & White, 2005, p117)

24. Findings • Correlational analysis: Strong correlation between displayed openness of group members and articulation of reasoning (R = .72) (Dyke et al., in press) • Intervention study: Causal effect on propensity to articulate ideas in group chats (effect size .6 standard deviations) (Kumar et al., 2011) • Mediating effect of idea contribution on learning in scientific inquiry (Wang et al., 2011) Engagement (Martin & White, 2005)

25. Souflé Framework(Howley et al., in press) Authority Transactive Knowledge Integration Person Person Engagement Engagement Authority

26. Analysis of Authoritativess Water pipe analogy: Water = Knowledge or Action Source = Authoritative speaker Sink = Non-authoritative Speaker Authoritativeness Ratio = Source Actions Actions

27. The Negotiation Framework(Martin & Rose, 2003) K2 requesting knowledge, information, opinions, or facts Type of Content? Knowledge Action Source or Sink? Primary Secondary A2 Instructing, suggesting, or requesting non-verbal action K1 giving knowledge, information, opinions, or facts A1 Narrating or performing your own non-verbal action Additionally… ch(direct challenge to previous utterance) o(all other moves, backchannels, etc.)

28. The Negotiation Framework(Martin & Rose, 2003) K2 requesting knowledge, information, opinions, or facts Type of Content? Knowledge Action Source or Sink? Primary Secondary A2 Instructing, suggesting, or requesting non-verbal action K1 giving knowledge, information, opinions, or facts A1 Narrating or performing your own non-verbal action

29. The Negotiation Framework(Martin & Rose, 2003) K2 requesting knowledge, information, opinions, or facts Type of Content? Knowledge Action Source or Sink? Primary Secondary A2 Instructing, suggesting, or requesting non-verbal action K1 giving knowledge, information, opinions, or facts A1 Narrating or performing your own non-verbal action

30. The Negotiation Framework(Martin & Rose, 2003) K2 requesting knowledge, information, opinions, or facts Type of Content? Knowledge Action Source or Sink? Primary Secondary A2 Instructing, suggesting, or requesting non-verbal action K1 giving knowledge, information, opinions, or facts A1 Narrating or performing your own non-verbal action

31. The Negotiation Framework(Martin & Rose, 2003) K2 requesting knowledge, information, opinions, or facts Type of Content? Knowledge Action Source or Sink? Primary Secondary A2 Instructing, suggesting, or requesting non-verbal action K1 giving knowledge, information, opinions, or facts A1 Narrating or performing your own non-verbal action Additionally… ch(direct challenge to previous utterance) o(all other moves, backchannels, etc.)

32. The Negotiation Framework(Martin & Rose, 2003) K2 requesting knowledge, information, opinions, or facts Type of Content? Knowledge Action Source or Sink? Primary Secondary A2 Instructing, suggesting, or requesting non-verbal action K1 giving knowledge, information, opinions, or facts A1 Narrating or performing your own non-verbal action Authoritativeness: K1 + A2 Additionally… ch(direct challenge to previous utterance) o(all other moves, backchannels, etc.) K1 + K2 + A1 + A2

33. K2?

34. Set up! K2 K1

36. Findings • Authoritativeness measures display how students respond to aggressive behavior in groups (Howley et al., in press) • Authoritativeness predicts learning (R = .64) and self-efficacy (R = .35) (Howley et al., 2011) • Authoritativeness predicts trust in doctor-patient interactions (R values between .25 and .35) (Mayfield et al., under review) • Computational Work • Detectable in collaborative learning chat logs (R = .86) • Detectable in transcribed dyadic discussions in a knowledge sharing task (R = .95) (Mayfield & Rosé, 2011) • Detectable in transcribed doctor-patient interactions (R = .96) (Mayfield et al., under review) Authoritativeness (Martin & Rose, 2003)

37. Outline • Introducing the Problem of Supporting Productive Discussion for Learning • Discussion of Souflé • Transactivity • Engagement • Authoritativeness • Application to Dynamic Support for Group Learning • Conclusion and Current Directions

38. Computer Supported Collaborative Learning

39. Automatic Analysis Of Conversation Positive Learning Outcomes Conversational Interventions

40. Foundational study: students work with a partner and dialogue agent for support Learn 1.24 s.d.more than individuals without support (Kumar et al., 2007a) • Results inform iterative design of agent behavior • Personalized agents increase supportiveness and help exchange between students (Kumar et al., 2007b) • Agents are more effective when students have control over timing of the interaction (Chaudhuri et al., 2008; Chaudhuri et al., 2009) • Agents that employ Balesian social strategies are more effective than those that do not (Kumar et al., 2010; Ai et al., 2010) • Students are sensitive to agent rhetorical strategies such as displayed bias (Ai et al., 2010), displayed openness to alternative perspectives (Kumar et al., 2011), and targeted elicitation (Howley et al., 2012) • Bazaar architecture enables efficient, principle based agent development (Kumar & Rosé, 2011; Adamson & Rosé, 2012) 6 Years of Positive Results

41. Outline • Introducing the Problem of Supporting Productive Discussion for Learning • Discussion of Souflé • Transactivity • Engagement • Authoritativeness • Application to Dynamic Support for Group Learning • Conclusion and Current Directions

42. Conclusions and Current Directions • Transactivity is a conversational behavior that is important for learning • Authoritativeness and Engagement are dimensions of conversation that play a supporting role • Positioning students to exchange Transactivecontributions • We have made progress at automating analysis of Transactivity and Authoritativeness • Automated analysis enables dynamic triggering of supportive interventions for online group learning

43. Conclusions and Current Directions • In the future, CSCL activities will be part of societies of online learners • Vision began with Virtual Math Teams/ The Math Forum • Now we’re already seeing the shift through companies like Coursera and Udacity • We are taking steps towards this future • Fully distance learning studies (UCSB, Drexel) • Sustainable CSCL (online office hours agent) • As we look to the future: we must understand the emergent effects of our local interventions in order to maximize positive benefit on a grand scale

44. Thank You!Questions?

45. Revoicing Agent

46. Experimental Paradigm • Day 1: Pretest on conceptual questions related to the unit (Diffusion or Punnett Squares) • Day 2: Online collaborative activity + Immediate Posttest isomorphic to pretest • Day 3: Whole class teacher led discussion + Delayed Posttest isomorphic to pretest • Participants: consenting 9th grade biology students, randomly assigned to groups of 3 • Experimental Design: Simple between subjects design • Groups randomly assigned to Revoicing condition or Control Condition Revoicing Agent Studies

47. Revoicing Agent

48. Study 1: Year 1, Diffusion Lab (50 students) • Students learn more on explanation questions in supported conditions (F(1,46) = 4.3, p < .05, effect size 1 standard deviation) • Students in supported conditions more active in whole group discussion (F(2,26) = 4.2, p < .05, effect size .75 standard deviations) • Study 2: Year 2, Diffusion Lab (78 students) • Students learn more on immediate post test in Revoicing Agent condition (F(1,74) = 4.3, p < .05, effect size .51 standard deviations) • Study 3: Year 2, Punnett Square Lab (78 students) • Students learned more on delayed post-test in Revoicing Agent condition Results of CSCL Studies