1 / 34

Thinking Hard Together: the Long and Short of Collaborative Idea Generation in Scientific Inquiry

Thinking Hard Together: the Long and Short of Collaborative Idea Generation in Scientific Inquiry. Hao-Chuan Wang, Carolyn P. Rosé, Yue Cui Carnegie Mellon University Chun-Yen Chang National Taiwan Normal University Chun-Chieh Huang, Tsai-Yen Li National Chengchi University.

lam
Download Presentation

Thinking Hard Together: the Long and Short of Collaborative Idea Generation in Scientific Inquiry

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Thinking Hard Together: the Long and Short of Collaborative Idea Generation in Scientific Inquiry Hao-Chuan Wang, Carolyn P. Rosé, Yue Cui Carnegie Mellon University Chun-Yen Chang National Taiwan Normal University Chun-Chieh Huang, Tsai-Yen Li National Chengchi University Funded through the National Science Foundation, The Pittsburgh Science of Learning Center, and the Office of Naval Research

  2. Why Collaborative Idea Generation? • Supported by common CSCL environments • Idea generation (problem finding) is recognized as preparation for problem solving (e.g., Hmelo-Silver, 2004) • Its value as a learning task is not well established Hmelo-Silver, C. (2004). Problem-Based Learning: What and How do Student Learn, Educational Psychology Review 16(3), pp235-266

  3. Problem: Process Losses in Group Idea Generation • Well known problem (Connelly, 1993; Diehl & Stroebe, 1987; Kraut, 2003) • Non-interacting groups may produce more and better ideas (Hill, 1982; Diehl & Stroebe, 1987) • How many people does it take to screw in a light bulb?

  4. Research Questions • Can we design support for collaborative idea generation that mitigates process losses? • Are there long term learning benefits of collaborative idea generation that outweigh the short term productivity losses?

  5. Outline • Vision: Using language technologies to offer context sensitive support for collaborative idea generation • Experimental Study (2X2 factorial design) • Pairs versus Individuals • With versus without support from an automatic brainstorming support agent • Main results and process analysis • Conclusions and current directions

  6. Outline • Vision: Using language technologies to offer context sensitive support for collaborative idea generation • Experimental Study (2X2 factorial design) • Pairs versus Individuals • With versus without support from an automatic brainstorming support agent • Main results and process analysis • Conclusions and current directions

  7. The Debris-FlowHazard Task (Chang & Tsai, 2005) • Task1: “What are the possible factors that might cause a debris-flow hazard to happen?” • Task 2: “How could we prevent it from happening?” Hmelo-Silver, C. (2004). Problem-Based Learning: What and How do Student Learn, Educational Psychology Review 16(3), pp235-266

  8. Domain Knowledge Ideas Bridging Inferences The Constructive Process of Idea Generation (Brown & Paulus, 2002) • Idea Generation Prompt • What are the possible factors that might cause a debris-flow hazard to happen? • Domain Knowledge • Debris flow refers to the mass movement of rocks and sedimentary materials in a fluid like manner • There are many typhoons (i.e. hurricanes) in Taiwan during the summer • Bridging Inferences • Heavy rain implies the presence of massive amounts of water • The presence of massive amounts of precipitation is likely to cause the fluid movement of rocks • Idea • Typhoons could be a factor causing a DFH to happen

  9. Labeled Texts Labeled Texts Idea Labels TagHelper Unlabeled Texts A Model that can Label More Texts Time Automatic Support for Collaborative Idea Generation • Student 1 People stole sand and stones to • use for construction. • Agent Yes, steeling sand and stones • may destroy the balance and • thus make mountain areas unstable. • Thinking about development of • mountain areas, can you think of a • kind of development that may • cause a problem? • Student 2 Development of mountain areas • often causes problems. • Student 1 It is okay to develop, but there • must be some constraints. • Trigger dialogue agents with an automatic analysis of a collaborative learning interaction http://www.cs.cmu.edu/~cprose/TagHelper.html

  10. Tutorial: • points the student in a direction for moving on with brainstorming • Designed based on “category labels” shown to be effective in previous studies of idea generation (Dugosh et al., 2000; Nijstad & Stroebe, 2006) Comment: acknowledges the idea the student just contributed 2 Part Feedback Generation Student 1 People stole sand and stones to use for construction. Agent Yes, steeling sand and stones may destroy the balance and thus make mountain areas unstable. Thinking about development of mountain areas, can you think of a kind of development that may cause a problem?

  11. Outline • Vision: Using language technologies to offer context sensitive support for collaborative idea generation • Experimental Study (2X2 factorial design) • Pairs versus Individuals • With versus without support from an automatic brainstorming support agent • Main results and process analysis • Conclusions and current directions

  12. Method • Experimental Design: 2X2 Factorial design • Working in Pairs vs Working Individually • Feedback from VIBRANT vs No Feedback • Participants: 42 10th grade students from a central Taiwan high school were randomly assigned to 4 conditions • Individual+NoFeedback (7 students) • Individual+Feedback (7 students) • Pair+NoFeedback (7 pairs) • Pair+Feedback (7 pairs)

  13. Collaborative Idea Generation Support Using Dialogue Agents Task Description Student 1’sContribution Student 2’sContribution Conferencing Mode: Student 1, Student 2 & Dialogue Agent Agent’sFeedback

  14. Experimental Procedure • Background reading (10 min) • Pretest (15 minutes) • Brainstorming 1 (30 minutes) • Experimental manipulation takes place during this phase • Brainstorming 2 (10 minutes) • Serves as a “transfer task” • Post test (15 minutes)

  15. What counts as a good idea? • The Debris-Flow Hazard task was designed by a panel of science educators as an assessment of creative problem solving ability • It has been used in a series of classroom studies in Taiwan • The student responses from these assessment studies have been evaluated by the panel • Based on this data, the panel decided on a set of “valuable ideas” for each task, which are used to measure idea generation success

  16. Analysis of Verbal Data: First Brainstorming Task • Conversation logs segmented into idea units • Typically at contribution boundaries • Contributions containing more than one idea broken up • Agreement of 2 coders on 10% of data was satisfactory (Kappa .7) • Idea units coded for one of 19 specific idea labels, other ideas related to correct solutions, and other ideas • Agreement of 2 coders on 10% of data was good (Kappa .82)

  17. Analysis of Verbal Data (Second Brainstorming Task) • Idea units from second brainstorming task already segmented • Coded with 15 pre-defined “valuable” ideas defined by experts or other • Agreement of 2 coders on 10% of the data was acceptable (Kappa .74)

  18. Outline • Vision: Using language technologies to offer context sensitive support for collaborative idea generation • Experimental Study (2X2 factorial design) • Pairs versus Individuals • With versus without support from an automatic brainstorming support agent • Main results and process analysis • Conclusions and current directions

  19. Expected Productivity loss in pairs NominalDyad • D.V.: Number of unique ideas • Results of ANOVA:- Nominal Dyad > Real Dyad(p<.001, d=1.51) • Productivity loss is evidenced 14 RealDyad 12 10 8 Num. Unique Idea (Brainstorming 1) 6 4 2 0 Nominal Real Conditions

  20. 10 8 6 Adjusted Posttest Score 4 2 0 Ind Pair Conditions Students in Pairs Learned Less • D.V.: Post test score • Covariate: Pre test score • Results of ANCOVA:Individuals > Pairs(p<.01, Cohen’s d=.1.68)* • Effect mediated by process loss • R2=.48, p< .005, N=42 • *Based on conversion formula by • Cohen(1988): • d = 2 * f for k=2 Individuals Pairs

  21. 10 8 6 Adjusted Postest Score 4 2 0 NoF F Conditions System Feedback as Learning Support during Brainstorming System Feedback • D.V.: Post test score • Covariate: Pre-test score • Evaluating the influence of system effect on domain learning (pre to post learning increases) • Result:System Feedback > No Feedback (p<.05, d=.70) No Feedback

  22. 7 6 5 4 Num. Unique Ideas (Brainstorming 2) 3 2 1 0 Ind Pair Conditions Effect of Working in Pairs on Task 2 Success • D.V.: Number of unique ideas in the subsequent idea generation task • Note: All students worked alone on this task • Results:Pairs > Individuals(p<.01, d=.92)* • No main effect of System Feedback Pairs Individuals

  23. 7 6 5 4 Num. Unique Ideas (Brainstorming 2) 3 2 1 0 Pair+F Ind+NoF Ind+F Pair+NoF Condition Effect of Working in Pairs on Task 2 Success • Sig. Interaction Effect • (p<.05, d=.78)* • Best: Pairs with Feedback • Worst: Individuals with Feedback Pairs Individuals

  24. Connection between Task 1 and Task 2 in Pairs Conditions • Main effect on Pair/Ind • F(1,38)=4.19, p<.05, d=.6 • Pair > Individual • No interaction • Evidence of mediating success on Task 2 • Significant correlation with success on task 2 • R2=.49, p < .0001, N=42

  25. “Off-task Ideas” 2.5 2.5 2.0 2.0 • Solutions were mentioned more frequently in the pairs conditions and in the feedback conditions • Note: Only marginal • Mention of solutions during the first task correlated with success during the second task • R2=.13, p< .05, N=42 1.5 1.5 Mean (valuable solutions) Mean (valuable solutions) 1.0 1.0 0.5 0.5 0.0 0.0 Nominal Real Type of Dyad

  26. Connection between Task 1 and Task 2 in Pairs Conditions Dialogue from Task1

  27. Greater Cohesiveness in Pairs with Feedback • t(38)=2.57, p<.05, d=.76 • Individuals in the Pair+F condition tended to stay in the same topic more • t(24)=1.994, p<.05, d= .77 • Pair+F > Pair+NoF • Peers in the Pair+F condition ‘talked’ more similarly

  28. Pairs versus Individuals • Pairs approached the problem from a broader perspective • Better preparation for problem solving • Feedback increased the intensity of Task 1 performance • Trend of effect on Task 2 consistent with the overall positive effect of working in pairs • Negative effect on Task 2 only in the Individual condition where focus was narrowly on Task 1 • But how can we balance success at learning/Task 1 with success at Task 2?

  29. Process Analysis • Conjecture: Process losses occurred at early stage of group idea generation (Diehl&Stroebe, 1991) • Never tested directly • Folk wisdom: brainstorm alone before group brainstorming • Note: Opposite has been proven effectively previously (Brown & Paulus, 2002) • Finding: about half of unique ideas contributed in first 5 minutes • Cognitive interference strongest during that time • Feedback will show an effect after 5 minutes

  30. Individuals+ Feedback Individuals+ NoFeedback Pairs+ Feedback Pairs+ NoFeedback Unique Ideas 12 Nom+N Nom+F Real+N 10 Real+F 8 #Unique Ideas 6 4 2 0 0 5 10 15 20 25 30 Time Stamp Process Analysis Individuals+Feedback Process loss Pairs vs Individuals: F(1,24)=12.22, p<.005, 1 sigma Individuals+NoFeedback Pairs+Feedback Pairs+NoFeedback Process loss Pairs vs Individuals: F(1,24)=4.61, p<.05, .61 sigma Negative effect of Feedback: F(1,24)= 7.23, p<.05, -1.03 sigma Positive effect of feedback: F(1,24)=16.43, p<.0005, 1.37 sigma

  31. Outline • Vision: Using language technologies to offer context sensitive support for collaborative idea generation • Experimental Study (2X2 factorial design) • Pairs versus Individuals • With versus without support from an automatic brainstorming support agent • Main results and process analysis • Conclusions and current directions

  32. What did we learn? • Good News • Interaction with dialogue agents during brainstorming increases learning • Feedback increases idea generation in pairs and individuals after 5 minutes • Bad News • Process losses in group brainstorming may hinder learning • Feedback decreases idea generation in the first five minutes

  33. Current Directions • Follow-up study • Students brainstorm alone for five minutes without feedback • Then work together for 25 minutes with feedback • Continuing to investigate automatic forms of collaborative learning support • Thermodynamics (Kumar et al., 2007) • Middle school math (Kumar et al, to appear) • Continued work on automatic collaborative learning process analysis (Rosé et al, Under Review)

  34. Any Questions? Htpp://www.cs.cmu.edu/~cprose/TagHelper.html

More Related