1 / 171

Why2 Tutors that teach mental models using natural language dialog

Why2 Tutors that teach mental models using natural language dialog. Kurt VanLehn & the Natural Language Tutoring Group at the University of Pittsburgh Art Graesser & the Tutoring Research Group at the University of Memphis. 1. Can natural language tutorial dialog add pedagogical value?.

odele
Download Presentation

Why2 Tutors that teach mental models using natural language dialog

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Why2Tutors that teach mental models using natural language dialog Kurt VanLehn & the Natural Language Tutoring Group at the University of Pittsburgh Art Graesser & the Tutoring Research Group at the University of Memphis

  2. 1. Can natural language tutorial dialog add pedagogical value? • Expert Human tutors • Computer mediated communication in print • Spoken computer mediated communication • Computer tutors • Why2/AutoTutor • Why2/Atlas • Reading text • Reading textbook • Focused lectures

  3. 2. Can deep linguistic and dialog techniques add pedagogical value?

  4. 3. How feasible is a deeplinguistic tutoring system? • Existing deep dialogue systems (e.g., Trains) are not applied • Existing applied dialogue systems (e.g., call centers) are not deep • Tutoring is not easy • Non-trivial knowledge (i.e., worth teaching) • Vague, incomplete, ill-formed language • Operate in real time

  5. Task domain: Writing qualitative physics explanations • Why explanations? • Generating explanations > studying explanations • But coaching and feedback may be necessary • Why conceptual physics? • Inert knowledge (e.g., F=m*a) may exist already but may be difficult to apply to problems • Correct principles can be identified objectively for problems, rather than being unconstrained • Subject matter is verbal and requires some analytical precision, but is not mathematics • Persistent misconceptions

  6. An example essay Question: Suppose you are running in a straight line at constant speed. You throw a pumpkin straight up. Where will it land? Explain why. Student’s explanation: Once the pumpkin leaves my hand, the horizontal force that I am exerting on it no longer exists, only a vertical force (caused by my throwing it). As it reaches it’s maximum height, gravity (exerted vertically downward) will cause the pumpkin to fall. Since no horizontal force acted on the pumpkin from the time it left my hand, it will fall at the same place where it left my hands.

  7. Evaluation framework • Pretest • Training:For each problem/question, do: • Student enters initial explanation • Tutor and student discuss it • Student enters final explanation • Tutor presents ideal explanation • Posttest Only this varies with the condition

  8. How might better discussion increase learning? • Better NLU => better dialogue goals • Better misconception detection • Doesn’t adopt remediation goal for correct statements • Better ways to achieve dialogue goals • Increase student explanation by better hints, prompts… • Managing complexity by postponing new goals • Maintain engagement

  9. Research plan • Develop tutoring systems with natural language dialog • Why2/AutoTutor • Why2/Atlas • Develop several control conditions • Human tutors • Reading text • Doing nothing • Compare computer tutors and controls • Hypothesis: learning is proportional to linguistic and dialog competence • Conduct research on computational models and mechanisms of tutorial dialog

  10. Broad Outline • Evaluation (60 min) • Why2-AutoTutor R&D (60 min) • Coffee break (30 min) • Why2-Atlas R&D (60 min)

  11. Next Evaluation Outline • Tutors • Human tutoring (Pam Jordan) • Why-Atlas (Pam) • Why-AutoTutor (Art) • Lecturer (Kurt) • Experiments • 1 (Kurt) • 2 (Art) • 3 (Tanner Jackson) • 4 (Pam) • 5 (Kurt) • Discussion (Kurt)

  12. Student’s screen for human tutoring Problem Dialogue history Student’s essay Student’s turn in the dialogue

  13. Example (slide 1 of 4) • Suppose you are running in a straight line at constant speed. You throw a pumpkin straight up. Where will it land? Explain. • Essay: The pumpkin will land behind me Problem Student’s explanation

  14. Example (slide 2 of 4) • Tutor: Please give reasons for your answer. • Student: The pumpkin will land behind me because the gravitational force will cause the pumpkin to land behind the position where I threw it Some “explanations” are incomprehensible even for us

  15. Example (slide 3 of 4) • Tutor: Let us argue systematically. What was the pumpkins velocity just bfore it was thrown? • Student: 0 m/s Ignores “explanation.” Starts long line of reasoning with a short-answer question wrong

  16. Example (slide 4 of 4) [Many lines omitted] • Tutor: Ok. If you have no questions I shall show you the expert essay? • Student: ok • Tutor: The man holding the pumpkin is running with constant horizontal velocity. In order to throw the pumpkin vertically up, the man will have to exert an upward vertical force on the pumpkin such that the net vertical force on it causes it to accelerate upward and …[20 more lines] • Student: ok Ideal essay

  17. Student’s screen for Why-Atlas Problem Dialogue history Student’s essay Student’s turn in the dialogue

  18. Why2-Atlas dialogue: outer loop • Identify flaws in the student’s essay • Misconceptions • Missing expectations (= points that a good essay is expected to cover) • Discussion: • Select a flaw • Select a KCD (knowledge construction dialogue) plan • Run it • Invite student to fix the essay • Repeat until essay is flawless

  19. Why2-Atlas dialogue inner loop: running a KCD plan • A KCD plan has a main line of reasoning • Each tutor turn ends in a question • Branch on student’s response: • Correct => positive feedback, may restate answer • Recognized incorrect or partial => specific remediation • Otherwise => generic remediation • Repeat until main line of reasoning is finished

  20. Example (slide 1 of 4) • Suppose a man is running in a straight line at constant speed. He throws a pumpkin straight up. Where will it land? Explain. • Essay: The pumpkin would land in front of me because of the horizontal speed that is exerting on the pumpkin.

  21. Example (slide 2 of 4) • Tutor: I see some opportunities for growth. Let's chat. So let's discuss a point that seems to be missing in your essay. Can you tell me what forces are acting on the pumpkin after it is released? • Student: Gravity force Picks a missing expectation & starts KCD Correct, so goes to next step of the KCD

  22. Example (slide 3 of 4) • Tutor: Good. Since we know that gravity is the only force acting on the pumpkin, let me ask you this. As the gravitational force pulls the pumpkin down, does it also pull the pumpkin horizontally a little bit? • Student: Yes. • Tutor: Let's look at this more closely. In which direction does the earth's gravitational force act? • Student: Vertical • Tutor: Let's consider this some more. Is that vertical upwards or vertical downwards? • Student: Vertical downwards Wrong, so subdialogue Incomplete, so subdialogue

  23. Example (slide 4 of 4) • [many lines omitted] • Tutor: Fine. I think this is an important point to include. See if you can add some sentences that convey this. Modify your essay please. • Essay: Since there is no gravity acting on the pumpkin vertically downward, then there is no force acting on the pumpkin when it is in the air. So the pumpkin does not land horizontally, but it lands in the same direction that i threw it up at. • Tutor: Terrific. Although we're making progress, I can see a few more things I'd like to discuss…

  24. Next Evaluation Outline • Tutors • Human tutoring (Pam Jordan) • Why-Atlas (Pam) • Why-AutoTutor (Art) • Lecturer (Kurt) • Experiments • 1 (Kurt) • 2 (Art) • 3 (Tanner Jackson) • 4 (Pam) • 5 (Kurt) • Discussion (Kurt)

  25. Talking head • Gestures • Synthesized speech • Presentation of the question/problem Student input (answers, comments, questions) • Dialog history with • tutor turns • student turns

  26. Distinctive features ofWhy2-AutoTutor • Animated agent • Attempt to have mixed initiative dialogue rather than a coached construction of an essay • Final request to generate essay in a single turn rather than ongoing modification of an essay • Accommodates spontaneous student questions (which they rarely ask) • Prompts for one student question (per problem) • Asks one diagnostic question about a misconception (per problem)

  27. Managing One AutoTutor Turn • Short feedback on the student’s previous turn • Advance the dialog by one or more dialog moves that are connected by discourse markers • End turn with a signal that transfers the floor to the student • Question • Prompting hand gesture • Head/gaze signal

  28. Dialog Moves • Positive immediate feedback: “Yeah” “Right!” • Neutral immediate feedback: “Okay” “Uh huh” • Negative immediate feedback: “No” “Not quite” • Pump for more information: “What else?” • Hint: “How does tossing the pumpkin affect horizontal velocity?” • Prompt for specific information: “Vertical acceleration does not affect horizontal _______.” • Assert: “Vertical acceleration does not affect horizontal velicity.” • Correct: “Air resistance is negligible” • Repeat: “So, once again, how does tossing the pumpkin affect horizontal velocity?” • Summarize: “So to recap, [succinct summary].”

  29. Cycle fleshes out one expectation at a time Exit cycle when: LSA-cosine(S, E ) > T S = student input E = expectation T = threshold Hint-Prompt-Assertion Cycles to Cover Good Expectations Hint Prompt Assertion Hint Prompt Assertion

  30. Next Evaluation Outline • Tutors • Human tutoring (Pam Jordan) • Why-Atlas (Pam) • Why-AutoTutor (Art) • Lecturer (Kurt) • Experiments • 1 (Kurt) • 2 (Art) • 3 (Tanner Jackson) • 4 (Pam) • 5 (Kurt) • Discussion (Kurt)

  31. The Lecturer • Intended as control condition • Monologue not dialogue • Everything else is the same • Content is the same

  32. What the student does • Read the problem • Write an initial explanation/essay • Read several minilessons (text only) • Revise the explanation/essay • Study the ideal essay

  33. Controlling for content • Each KCD converted to a minilesson • T: What direction is the gravitational force?S: Straight down • => The direction of the gravitational force is straight down • Ignoring the student’s essay, the Lecturer presents every relevant minilesson • Student may not read (or reread) it.

  34. Example (slide 1 of 4) • Suppose you are running in a straight line at constant speed. You throw a pumpkin straight up. Where will it land? Explain. • Essay: Once the pumpkin leaves my hand, the horizontal force that I am exerting on it no longer exists, only a vertical force (caused by my throwing it). As it reaches it's maximum height, gravity (exerted vertically downward) will cause the pumpkin to fall. Since no horizontal force acted on the pumpkin from the time it left my hand, it will fall at the same place where it left my hands.

  35. Example (slide 2 of 4) • Lecturer: Here is an important point that a good explanation should cover. The relationship between the velocity of the runner and the velocity of the pumpkin which the runner is holding, before the runner releases the pumpkin, is that the two velocities are the same. As an example let's consider in a different scenario the relationship between the velocities of two objects that move together. • Student Click

  36. Example (slide 3 of 4) • Lecturer: A woman is carrying a suitcase through an airport. The suitcase does not move relative to the woman's body. Therefore, over any interval of time, the displacement of the woman compared to the displacement of the suitcase is equal to the displacement of the suitcase. • Student Click • Lecturer: Velocity is defined as the displacement vector over a time interval divided by that time interval. Thus, the relationship between the velocity of the woman and the velocity of the suitcase is that the two velocities are the same.

  37. Example (slide 4 of 4) • [Many lines omitted] Lecturer: Please revise your essay now. • Essay: As I am running with the pumpkin, the velocity of the pumpkin is equal to my velocity. …. • Student Click • Lecturer: Now that you've finished your essay, here is an essay that covers everything needed to fully answer the problem. Student edits initial essay => final essay Ideal Essay goes here

  38. Similar format for all 4 conditions • Pretest • Training:For each problem/question, do: • Student enters initial explanation • Tutor and student discuss it • Student enters final explanation • Tutor presents ideal explanation • Posttest Only this varies with condition But same expectations & misconceptions per essay

  39. Next Evaluation Outline • Tutors • Human tutoring (Pam Jordan) • Why-Atlas (Pam) • Why-AutoTutor (Art) • Lecturer (Kurt) • Experiments • 1 (Kurt) • 2 (Art) • 3 (Tanner Jackson) • 4 (Pam) • 5 (Kurt) • Discussion (Kurt)

  40. Experiment 1 details • Participants had college physics • 10 Training problems • Tests (two equivalent forms) • 40 multiple choice questions • 4 essay questions • Componential scoring • Wholistic scoring

  41. All four groups learned Essays

  42. All groups learned the same amount • Measures • Multiple choice test: Raw, normalized, malleable • Essay test: Lenient, stringent, wholistic • Combined measure of principle/misconceptions • Statistical power was sufficient • Subpopulations • Test items that are similar to training vs. not • Subjects with low vs. high pretests

  43. Why did all 4 groups learnthe same amount? • Content alone controls learning, and all 4 groups covered the same content equally diligently? • Experiment 2 • Tests are not deep enough? • Experiment 3 • Not really “learning”, just “reminding”? • Experiment 4

  44. Next Evaluation Outline • Tutors • Human tutoring (Pam Jordan) • Why-Atlas (Pam) • Why-AutoTutor (Art) • Lecturer (Kurt) • Experiments • 1 (Kurt) • 2 (Art) • 3 (Tanner Jackson) • 4 (Pam) • 5 (Kurt) • Discussion (Kurt)

  45. Experiment 2: AutoTutor versus Reading a Textbook Three tutoring conditions • Why-AutoTutor • Read textbook control • Read nothing 63 participants knowledge of physics

  46. Phases of Experiment • Survey on participant information • Pretest • Multiple choice test • 4 essay questions • Training on 10 problems vs. control • Posttest • Multiple choice test • 4 problems – essay answers

  47. Multiple Choice Scores

  48. Content: Textbook  Others Transcript = one tutor, one student, all 10 training problems For each pair of transcripts, compute the LSA cosine, with “object” substituted by problem-specific objects

  49. Conclusions from Experiment 2 • Learning from Why2-AutoTutor is better than reading chapters from a textbook for nearly the same amount of time. • Reading from the textbook is no different from doing nothing at all. • The textbook may cover slightly different content than Why2-AutoTutor • Reading the textbook may not be particularly engaging.

  50. Next Evaluation Outline • Tutors • Human tutoring (Pam Jordan) • Why-Atlas (Pam) • Why-AutoTutor (Art) • Lecturer (Kurt) • Experiments • 1 (Kurt) • 2 (Art) • 3 (Tanner Jackson) • 4 (Pam) • 5 (Kurt) • Discussion (Kurt)

More Related