computer supported collaborative learning
Download
Skip this Video
Download Presentation
Computer Supported Collaborative Learning

Loading in 2 Seconds...

play fullscreen
1 / 44

Computer Supported Collaborative Learning - PowerPoint PPT Presentation


  • 124 Views
  • Uploaded on

Computer Supported Collaborative Learning. Language Technologies Institute Carnegie Mellon University. Wednesday, March 18, 2009: Speech and NLP for Educational Applications. Research Group. Rohit Kumar. Our work: Questions. Conversational Agents (Among other things)

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Computer Supported Collaborative Learning' - etoile


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
computer supported collaborative learning

Computer Supported Collaborative Learning

Language Technologies Institute

Carnegie Mellon University

Wednesday, March 18, 2009: Speech and NLP for Educational Applications

research group
Research Group

Rohit Kumar

our work questions
Our work: Questions
  • Conversational Agents (Among other things)
    • To support human users at various tasks
      • What kind of tasks?
    • Supporting Learning tasks
      • What kind of support? (Agents ofcourse, but…)
        • What role?
        • Manifestation
        • How do we make sure the support is getting through?
      • What kind of users?
        • Individuals / Pairs / Groups ?
        • Students ? Teachers/facilitators?
      • What environment?
      • How do we build this support?
      • How do we evaluate if the support helps?
our focus today
Our focus today:

Rohit Kumar, Carolyn P. Rosé, Yi-Chia Wang,

Mahesh Joshi, Allen Robinson

Tutorial Dialogue as Adaptive Collaborative Learning Support

Artificial Intelligence in Education 2007

Particularly interesting:

Borderlining the transition of the CycleTalk project from Tutorial Dialog to CSCL

cycletalk
CycleTalk
  • Cycle Pad

(Forbus et. al. 1999)

    • Thermodynamic cycles simulation environment
    • Designed to engage students in engineering design
    • Exploratory learning
  • Cycle Talk Goals:
    • Support engineering students learning the principles involved in designing thermodynamic cycles
    • Study the benefits of tutorial dialogue in an exploratory learning context
cycletalk history

Build

cycle

Explore relationships between parameters

Assume component parameters

Incorporate conceptual understanding

Generate plan to improve cycle

Investigate variable dependencies

Compare

multiple cycle improvements

Compare cycle

to alternatives

CycleTalk: History
  • Rosé et. al., CycleTalk: Towards a Dialogue Agent that Guides Design with an Articulate Simulator, 2004
    • Cognitive Task Analysis
    • Observation: Cycle Pad’s significant pedagogical potential tends to be underutilized when students do not receive tutorial guidance
cycletalk history1
CycleTalk: History
  • Rosé et. al., A First Evaluation of the Instructional Value of Negotiable Problem Solving Goals on the Exploratory Learning Continuum, 2005
      • 3 conditions
        • NPSG: Human Tutoring + written material (script)
        • PS: Example tracing tutors (Aleven et. al.) + script
        • S: Script only
      • Overall order: PS < S < NPSG
      • Significant difference between NPSG and PS
      • Marginal difference between PS and S
    • By-product: Conversational data collected from human tutoring
    • Difference between tutors
      • One of the tutors was very impatient and terse.

Most of his students learnt less than expected from pre-test!

>> Content/presentation of tutorial dialog is critical to success

cycletalk history3
CycleTalk: History
  • Since we had some conversational data now:
    • Analysis of Topics
      • Using Info-Magnets

Arguello et. al. 2006

    • 15 topics identified
      • Each human-tutoring log, covered between 4 to 11 of these topics
      • Correlation between coverage of these topics on learning from the human-tutoring data
      • So, lets get the machine to do the talking.
cycletalk history4
CycleTalk: History
  • Developing a Tutorial Dialog Systems for CycleTalk
    • 12 Knowledge Construction Dialogs authored based on this topics identified
      • 7 related to optimization relationships
      • 5 related to theoretical concepts
    • Authored for TuTalk
  • When do we bring these up?

Tutor: Let’s stop for a minute and reflect on what we have seen. Having seen how changing T-Max (i.e., the Maximum Temperature at the Heater Outlet) affects the efficiency of the cycle, do you think it is a good idea in general for the Maximum Temperature to be increased and kept high?

Student: yes

Tutor: Okay. Let’s think now about one important consequence of keeping T-Max high. What happens to the steam quality at the turbine exit if you in-crease the maximum temperature of the cycle?

Student: Increases

Tutor: Right. As increasing the temperature makes the efficiency and steam quality better, we should increase the maximum temperature of the cycle. But can we keep on increasing the maximum temperature at the heater outlet indefinitely, or are there limitations to keep in mind?

Student: there are limitations such as material constraints

Tutor: Great. we should try to keep T-Max (i.e., the maximum temperature at the Heater outlet of the cycle) as high as possible without risking safety concerns or exceeding the maximum temperature the material can withstand. Keeping T-Max high increases efficiency and keeps the steam quality high, which are both important. This is an important principle to keep in mind while optimizing the efficiency of your cycle.

cycletalk history5
CycleTalk: History
  • Developing a Tutorial Dialog Systems for CycleTalk
    • 12 Knowledge Construction Dialogs authored based on this topics identified
      • 7 related to optimization relationships
      • 5 related to theoretical concepts
    • Authored for TuTalk
  • Integrated with the example tracing tutor at relevant nodes
    • So, instead of hints, a window with the dialog agent would pops up
      • Not a clean integration
cycletalk history6
CycleTalk: History
  • Ran an experiment with our first CycleTalk with Tutorial Dialog

Kumar et. al., Evaluating the Effectiveness of Tutorial Dialogue Instruction in an Exploratory Learning Context, 2006

    • 3 conditions
      • S – Script only
      • PSHELP – PS + Tutorial Dialog triggered in place of some hints
      • PSSUCCESS – PSHELP + Tutorial Dialog triggered on successful completion of certain trace nodes
    • Effect size
      • CMU: 0.35σ comparing PSHELP to PSSUCCESS
      • USNA: 0.25σ comparing S to PSSUCCESS
    • Average KCD launches:
      • PSHELP - 1.8 , PSSUCCESS – 2.7
      • Human Tutoring – 4 to 11
cycletalk fall 2006
CycleTalk: Fall 2006
  • Collaborative Learning Setup: Implementations
    • Interaction Model: Keep the students more engaged:
      • Hinting prompts

“Try to think of an idea related to manipulating a property of the pump.”

      • Every one minute (too many!)
      • Targeted prompting (based on contribution rate)
    • Dynamic/Adaptive support for collaboration vs. Scripting
      • Topic Filter for triggering of dialogs
        • Trained models to classify turns into topics (Taghelper)
        • Training data from Human-Tutoring corpus

(Rosé et. al., 2005)

      • 2 step classification
        • Topic worthiness: SVM (Q: How do we know if worthy?)
        • Topic labeling: TDIDF Scoring
    • Chatting software
      • Agent as an observer/participant in chat
cycletalk procedure
CycleTalk: Procedure
  • 15min: CyclePad training(led by experimenter)
  • 70min: Work through material on domain content and using CyclePad
  • 15min: Pre-Test
    • 42 multiple choice, 8 open response questions
  • Contest announcement
  • 25min: Review and write notes about what students learnt into the chat window 
  • 10min: Planning/Synthesis of 2 design plans
  • 25min: Implementation of designs in CyclePad
  • 15min: Post-Test
  • Questionnaire (Pairs only)
cycletalk experimental design
CycleTalk: Experimental Design
  • Manipulation in step 4: Review and write notes about what students learnt into the chat window
  • 3x2 Full factorial design
    • Support
      • None (N): No support
      • Static (S): Written material (Script)
      • Dynamic (D): Adaptive dialog agent
    • Collaboration: Alone (I) / Pair (P)
  • CMU Sophomores: 87 students over 4 days
cycletalk outcome metrics
CycleTalk: Outcome Metrics
  • Pre/Post Tests
    • Objective type questions
    • Open response questions
  • Ability to design, implement an efficient Ranking Cycle
  • Questionnaire
cycletalk results collaboration
CycleTalk: Results: Collaboration
  • Effect size: 0.4σ(Q: Does this mean significant?)
  • Reflection in pairs is more effective than reflection alone
cycletalk results support
CycleTalk: Results: Support
  • Positive effect of Dynamic support
  • Dynamic Support > No support
  • Effect size: 0.7σ
cycletalk results combined
CycleTalk: Results: Combined
  • Marginal Interaction, p=.07
  • Pair+Dynamic > Individual+No Support, 1.2σ
  • Pair+Static > Individual+No Support, .9σ
  • Individual+Dynamic > Individual+No Support, 1.06σ
cycletalk more results
CycleTalk: More Results
  • Open response questions
    • Advantage for dialog based support, but not collaboration
    • Effect size: 0.5σ (Simpler ANCOVA model)
  • Practical Assessment:
    • No significant differences
    • 91% students built one fully defined cycle
    • 64% built two
  • Questionnaire:
    • Dynamic support student rate high on benefit, but low on engagement when in pairs
    • Collaboration & Dynamic support not working together??
    • Desirable Difficulty? (Ref: Robert Bjork)
cycletalk observations
CycleTalk: Observations
  • Individual interaction with agents
    • Highly tutor directed
    • Students rarely ignore tutor prompts
  • More complicated dynamics with pairs
    • Students talk around tutor agent (Q)
    • Tutor agent is an interruption
    • Students treat agent contributions like hints
    • Evidence that they read although they often don’t respond
    • Frustration
aied 2007 your questions q
AIED 2007: Your questions (Q)
  • Is this reflection?
    • Dialogs were about material already given to them which they review in step 2
    • Whether students were receiving instruction twice?
  • Multi-Party Turn-Taking
    • This was a problem back then
      • Recent attempts to improve this
      • And more needs to be done…
  • Why do students ignore the Tutors?
  • Pairwise Tukey Posthoc analysis
    • Difference of means/SE > qcritical
that was fall 2006 since then
That was Fall 2006: Since then…
  • Making the agent more “Social”
    • Kumar et. al., SLaTE 2007
      • Engaging students through small talk
      • Promoting Collaboration
    • Unpublished Manuscript (Kumar et. al.)
      • Attention Grabbing
    • Chaudhuri et. al., AIED 2009
      • Pointers
  • Basilica
    • Architecture for developing conversational agents
  • Virtual Environments
    • ConcertChat
    • SecondLife
  • More than 2 students?
    • Recent work in a freshmen Mechanical Engineering class
kumar et al slate 2007
Kumar et. al., SLaTE 2007
  • Mathematics Problem Solving
    • Fractions
      • Addition, Subtraction, Multiplication, Division

CTAT Problem Solving Interface with Cognitive Tutors based feedback [Aleven, Koedinger]

(shared by students)

Conversation Interface

(shared by students & conversational agents)

kumar et al slate 2007 social strategy
Kumar et. al., SLaTE 2007: Social Strategy

Tutor:Student1, If you had to choose between a long flight and a longcar ride, which seems less uncomfortable?

Student1: I’ll take the car ride

Tutor:Ok Student1.

Tutor:Student2, Which are more entertaining – books or movies?

Student2: definitely books!

Tutor:Ok Student2.

Tutor:Please work on the problem shown in the side panel.

Motivated by the intention of engaging students

    • By showing interest in their personal preferences
  • Goal of this social conversation is to make the students feel that they worked together to construct the problem statement
  • Comes up every time the students are about to start solving a new problem
  • Related: (Bickmore & Cassell)
    • SmallTalk by Embodied Conversational Agent REA

Example

Jan packed several books to amuse herself on a long car ride to visit her grandma. After 1/5 of the trip, she had already finished 6/8 of the books she brought. How many times more books should she have brought than what she packed?

kumar et al slate 2007 questionnaire
Kumar et. al., SLaTE 2007: Questionnaire

Significant (Effect Size = 1.15)

Students perceived higher help offering by their partners in the Experimental condition

Significant (Effect Size = 1.18)

Students perceived they offered more help to

their partners in the Experimental condition

kumar et al slate 2007 results
Kumar et. al., SLaTE 2007: Results
  • Observations from Conversation Analysis
    • Average number of Help Provisions not significantly different across conditions
    • More help related episodes per problem in the Experimental condition

Mean (Control) = 0.30 Mean (Experimental) = 0.69

F(1, 15) = 16.8 p < 0.001

    • More episodes of Deny Help in Control condition

Mean (Control) = 40.2 Mean (Experimental) = 24.7

F(1, 62) = 3.46 p = 0.001

    • Students displayed more negative attitude in Control conditions
      • Insults (“you stink”, “stupid”) occurred only in Control condition
that was fall 2006 since then1
That was Fall 2006: Since then…

Making the agent more “Social”

Kumar et. al., SLaTE 2007

Engaging students through small talk

Promoting Collaboration

Unpublished Manuscript (Kumar et. al.)

Attention Grabbing

Chaudhuri et. al., AIED 2009

Pointers

Basilica

Architecture for developing conversational agents

Virtual Environments

ConcertChat

SecondLife

More than 2 students?

Recent work in a freshmen Mechanical Engineering class

cycletalk fall 20071

Two Motivational Prompts

At 2 minute mark

At 30 minute mark

Tutor

Tutor

Tutor

St13BP

Tutor

Tutor

St13BP

Tutor

St13BP

Tutor

St13BP

Tutor

Tutor

Tutor

There will be more potential for

Cooling. If there is more potential

for cooling, is there more or less

potential for power generation?

Solving this problem can be a lot of

fun if you cooperate well together

Consider the effect of increasing Qin

which is heat input to a cycle. First,

let’s consider what happens to

power out when Qin is increased.

What happens to power out when

steam is cooled?

because that increases power

Winning isn’t everything.

Don’t worry. Be Happy.

Power is generated when steam is

cooled. If there is more heat input,

will there be more or less potential

for cooling?

Now might be a good time for

reflection.

Power is generated when steam is

cooled. If there is more heat input,

will there be more or less potential

for cooling?

well the idea is to increase the heat

in

because that increases power

Now might be a good time for

reflection.

There will be more potential for

Cooling. If there is more potential

for cooling, is there more or less

potential for power generation?

Consider the effect of increasing Qin

which is heat input to a cycle. First,

let’s consider what happens to

power out when Qin is increased.

What happens to power out when

steam is cooled?

well the idea is to increase the heat

in

Instructive Conversationwith Attention Grabbing

Instructive Conversationwith Attention Grabbing

CycleTalk Fall 2007
cycletalk fall 2007 conversation analysis
CycleTalk Fall 2007: Conversation Analysis

More Attention to the Tutor

More Relevant Responses

Numbers here are:Average ( Percentage of Instructional turns each student responded to)

i n t r o d u c t i o n : e x p e r i m e n t 1 : e x p e r i m e n t 2 : d i r e c t i o n s

cycletalk fall 2007 conversation analysis1
CycleTalk Fall 2007: Conversation Analysis

i n t r o d u c t i o n : e x p e r i m e n t 1 : e x p e r i m e n t 2 : d i r e c t i o n s

cycletalk fall 2008 chaudhuri et al 09
CycleTalk Fall 2008 (Chaudhuri et. al. ‘09)

B6: so i think the best thing to go with first is to increase the pressure entering the connector

Tutor: You might want to review what you saw earlier in the sensitivity analysis investigating the effect of changing P-max on steam quality and efficiency, from the top half of page 13 in your book.

Tutor: type HELP WITH CHANGEPMAX if you want to discuss it with me.

B1: help with changepmax

Tutor: Remember the sensitivity analysis you have completed where you manipulated P-Max and observed it\'s effect on the efficiency of the cycle. You can refer to the figure that I have pasted onto your whiteboard for your convenience. Having seen how changing P-Max, the Maximum Pressure at the Heater Inlet, effects efficiency and Steam, do you think as a general principle the Maximum pressure should be increased ?

B6: Yes

Tutor: So the Efficiency of a Cycle increases by increasing the maximum pressure of the cycle. In addition to the effect on efficiency, what happens to the steam quality at the turbine exit if you increase the maximum pressure of the cycle ?

B6: Decreases

Tutor: So let\'s think for a minute about these competing effects: Increasing P-Max makes the efficiency better but the steam quality degrades. Is the steam quality at the turbine exit acceptable at the maximum allowable pressure within safety limits?

cycletalk fall 2008 experiment
CycleTalk Fall 2008: Experiment
  • Manipulation (2x2):
    • Pointer Hints: Yes/No
    • Dialog Support: Yes/No
  • Results
    • Higher learning gains for Pointer+Dialog condition
      • Pointer+Dialog vs Dialog: 0.8σ
      • Pointer+Dialog vs None: 0.6σ
      • Pointer vs None: 0.35σ
    • Few dialogs in Pointer + Dialog condition compared to Dialog only condition
      • Too many dialogs distracting?
that was fall 2006 since then2
That was Fall 2006: Since then…

Making the agent more “Social”

Kumar et. al., SLaTE 2007

Engaging students through small talk

Promoting Collaboration

Unpublished Manuscript (Kumar et. al.)

Attention Grabbing

Chaudhuri et. al., AIED 2009

Pointers

Basilica

Architecture for developing conversational agents

Virtual Environments

ConcertChat

SecondLife

More than 2 students?

Recent work in a freshmen Mechanical Engineering class

basilica
Basilica
  • Multi-Expert model of building conversational agents
  • Terminology:
    • Components, Actors, Filters, Events, Connections
  • Actors: Generate user perceivable events
  • Filters: (Everything else, mostly): Interpret events generated by other components
  • Data is encapsulated as Events
basilica1

Student1

Student2

ConcertChat

Server

OutGoingMessage

SpoolingFilter

TextMessageEvent

Presence

Actor

Hinting

Actor

Prompting

Actor

TuTalk Server

CCText

Filter

Channel

Filter

TurnTaking

Filter

Hinting

Filter

Launch

Filter

TextMessageEvent

ProduceHintEvent

LaunchEvent

TextMessageEvent

x

x

TookTutorTurnEvent

TextMessageEvent

TextMessageEvent

LaunchEvent

TextMessageEvent

TextMessageEvent

x

TextMessageEvent

TextMessageEvent

TextMessageEvent

ProduceHintEvent

LaunchEvent

TakeTutorTurnEvent

Attention

Grabbing

Actor

Tutoring

Actor

GrabAttentionEvent

StartTutoringEvent

TakeTutorTurnEvent

TextMessageEvent

TextMessageEvent

TutoringStartedEvent

TookTutorTurnEvent

DoneTutoringEvent

Polling

Attention

Grabbing

Filter

Tutoring

Filter

TakeTutorTurnEvent

TookTutorTurnEvent

LaunchEvent

TutoringStartedEvent

DoneTutoringEvent

AttentionGrabbedEvent

GrabAttentionEvent

TextMessageEvent

GrabAttentionEvent

AttentionGrabbed-

Event

TakeTutorTurnEvent

TookTutorTurnEvent

StartTutoringEvent

GrabAttentionEvent

Basilica
basilica2
Basilica
  • Novel Architecture
  • Re-usable components
    • Rapid prototypes
  • Easy integration of the same agent with many environments
  • Incremental developments of components
  • Meta-architecture for bringing together other dialog management components
conversational agents in second life
Conversational Agents in Second Life

MIDDLEWARE

Session1

Session2

Session3

Session4

Session5

Object 1 Internal Representation

Object 2 Internal Representation

TRANSLATION

O

B

J

E

C

T

1

O

B

J

E

C

T

2

Message Receiver

Message Receiver

I

N

T

E

R

F

A

C

E

I

N

T

E

R

F

A

C

E

Message

Queue

Message

Queue

HTTP

HTTP

that was fall 2006 since then3
That was Fall 2006: Since then…

Making the agent more “Social”

Kumar et. al., SLaTE 2007

Engaging students through small talk

Promoting Collaboration

Unpublished Manuscript (Kumar et. al.)

Attention Grabbing

Chaudhuri et. al., AIED 2009

Pointers

Basilica

Architecture for developing conversational agents

Virtual Environments

ConcertChat

SecondLife

More than 2 students?

Recent work in a freshmen Mechanical Engineering class

supporting groups
Supporting Groups
  • Mechanical Engineering Freshmen class
    • Sort of repeat the process of CycleTalk
      • Initial data collection done with agents though
      • One advantage of Basilica here
    • Observations: (3 out of 6 sessions)
      • New vocabulary collection (for dynamic triggers)
      • Lots of Sarcasm, Teasing, Cursing, Discontent, Silliness
        • Some positiveness too
      • Abuses towards Tutor
      • Opportunities identified that next version of tutor can use
        • Current irresponsiveness
        • frequent questions/concepts
  • GRASP
    • Supporting teachers/facilitators for long-term group assessment
our work questions1
Our work: Questions
  • Supporting Learning tasks
    • What kind of tasks?
    • What kind of support? (Agents ofcourse, but…)
      • What role?
      • Manifestation
      • How do we make sure the support is getting through?
    • What kind of users?
      • Individuals / Pairs / Groups ?
      • Students ? Teachers/facilitators?
    • What environment?
    • How do we build this support?
    • How do we evaluate if the support helps?
slide44
Done:

Thanks for tuning in.

Most questions are good questions.

So, Please Ask.

i n t r o d u c t i o n : e x p e r i m e n t 1 : e x p e r i m e n t 2 : d i r e c t i o n s

ad