1 / 22

Uncovering the Problem-Solving Process:

Uncovering the Problem-Solving Process:. Cued Retrospective Reporting, Eye Tracking, and Expertise Differences. Tamara van Gog, Fred Paas, & Jeroen J. G. van Merriënboer I 3 CLEPS Workshop/Mini-conference, August 29, 2005. Overview. Experiment: Theory Design

salena
Download Presentation

Uncovering the Problem-Solving Process:

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Uncovering the Problem-Solving Process: Cued Retrospective Reporting, Eye Tracking, and Expertise Differences Tamara van Gog, Fred Paas, & Jeroen J. G. van MerriënboerI3CLEPS Workshop/Mini-conference, August 29, 2005

  2. Overview Experiment: • Theory • Design • Comparison of 3 verbal methods • The 3 methods & expertise differences • Uncovering expertise-related performance differences through eye movement data • Present limitations and future research • Discussion

  3. Theory Use of process-tracing techniques to uncover problem-solving processes in order to advance / inform: • Psychological theory • Expert systems • User-system interaction, But also • Instructional design e.g., design of process-oriented worked examples

  4. Theory From the literature (Kuusela & Paul, 2000; Taylor & Dionne, 2000): + of concurrent reporting (“think aloud”): more information on actions taken + of retrospective reporting: more information on rationale for actions taken and strategies that control the process Needed: A method that combines + & + : Cued retrospective reporting based on a record of eye movements & mouse/keyboard operations?

  5. Design Within-subjects, 26 participants, electrical circuits troubleshooting tasks: Seq. Condition + Tasks 1 CR 1+2 CRE 3+4 RR 5+6 CRR 7+8 2 CRE 3+4 CRR 7+8 CR 1+2 RR 5+6 3 RR 5+6 CR 1+2 CRR 7+8 CRE 3+4 4 CRR 7+8 RR 5+6 CRE 3+4 CR 1+2 CR = concurrent reporting; CRE = concurrent reporting with eye tracking; RR = retrospective reporting; CRR = cued retrospective reporting.

  6. Comparison of 3 Methods: Hypotheses 1. Concurrent reporting (CR): more ‘action’ info than RR 2. Retrospective reporting (RR): more ‘why’, ‘how’, & ‘metacognitive’ info than CR 3. Cued retrospective reporting (CRR):-> more ‘action’ than RR-> more ‘why’, ‘how’, & ‘metacognitive’ than CR

  7. Comparison of 3 Methods: Analyses Segmentation based on speech  sentences / utterances (preceded & followed by a pause) Coding scheme task-oriented main categories: ‘action’ ‘why’ ‘how’ ‘metacognitive’ 20% of protocols scored by 2 raters: kappa = .79 good; proceeded with 1 rater Analyses on nr. of codes on main categories, obtained by summing codes on subcategories

  8. Comparison of 3 Methods: Results Friedman Tests with Conover (1999) comparisons CR vs RR:as hypothesized: ‘action’  CR >RRhowever: ‘why’ and ‘how’  CR > RR, and‘metacognitive’ CR = RR CRR vs RR:as hypothesized: ‘action’  CRR >RR‘why’: CRR = RR‘how’ and ‘metacognitive’: CRR > RR

  9. Expertise Differences: Explorative 5 “highest” and 5 “lowest” expertise participants (from 26). Determined by performance efficiency: “highest”: higher performance, lower mental effort, lower time-on-task “lowest”: lower performance, higher mental effort, higher time-on-task • Differences in elicited information? • Differences in preferences/experiences?(open-ended debriefing questions)

  10. Expertise Differences: Elicited Information Differences in elicited information? (Mann-Whitney U Tests) CR: ‘how’ and ‘metacognitive’ info: “lowest” > “highest” RR: ‘why’ info: “highest”> “lowest” ‘how’ info: “lowest” > “highest” CRR: ‘action’ and ‘metacognitive’ info: “lowest” > “highest”

  11. Expertise Differences: Experience Differences in preferences/experiences? “lowest”: experience: CR  (4/5)preference: CRR > CR & RR (4/5) “highest”:no differential experiences/preferences Mediating factors mentioned re. experience / preference, by both “lowest” and “highest”: • Time-on-task • Cue

  12. Studying Expertise-Related Performance Differences: Eye Movement Data 1 Eye fixation data provide insight in the allocation of attention, and hence differ with expertise Research use: provide information about the problem-solving process at a finer grained level than verbal protocols? • (Ultimate) educational use: guiding novices’ attention? 1 Data from Van Gog, Paas, & Van Merriënboer (2005), Applied Cognitive Psychology

  13. Eye Movement Data: Participants & Procedure Same 5 “lowest” and 5 “highest” expertise participants Data collected in first 3 phases of the process: • Problem orientation (until pushing switch to observe circuit behavior) • Problem formulation and action decision • Action evaluation and next action decision % time spent on phase, mean fixation duration (MFD), and in 1st phase fix. related to faults

  14. Task Short-circuit Only 3 Volt

  15. Eye Movement Data: Results Phase 1: problem orientation (Mann-Whitney U Tests, 2-tailed, α = .10) % of time: “highest” > “lowest” MFD: “lowest” > “highest” % fixations on battery: “highest” > “lowest” Gaze switches short-circuit: “highest” > “lowest” (NB: only trend)

  16. Eye Movement Data: Results Phase 2: problem formulation & action decision (Mann-Whitney U Tests) % of time: “highest” = “lowest” MFD: “highest” = “lowest” MFD First ½: “highest” > “lowest” MFD Second ½: “highest” = “lowest”

  17. Eye Movement Data: Results Phase 3: action evaluation & next action decision (Mann-Whitney U Tests) % of time: “highest” > “lowest” MFD: “highest” = “lowest” MFD First ½: “highest” = “lowest” MFD Second ½: “highest” = “lowest”

  18. Eye Movement Data: Results MFD over phases (Friedman + Nemenyi post-hoc): n.s. for “lowest”; “highest” 1 < 2.1., 2.2., 3.2 & 2.1 >3.1

  19. Limitations • CRR and fabrication? • Cue: combination of eye movements AND mouse/keyboard operations • Only quantitative analyses of protocols • Eye movement data: distinction of phases • Performance efficiency measure:very relative distinction (lowest and highest within this group of participants) • Small nr of participants in analyses related to expertise differences

  20. Future Research • Qualitative differences between CRR and RR? • Cue: different effects with only eye movements OR mouse/keyboard operations? • Cue: technical optimization? • (RR/)CRR: effects of other prompts? • Further study of performance efficiency measure to distinguish expertise levels • Replications with larger N

  21. Thank you for your attention! tamara.vangog@ou.nl

More Related