Uncovering the problem solving process
1 / 22

Uncovering the Problem-Solving Process: - PowerPoint PPT Presentation

  • Uploaded on

Uncovering the Problem-Solving Process:. Cued Retrospective Reporting, Eye Tracking, and Expertise Differences. Tamara van Gog, Fred Paas, & Jeroen J. G. van Merriënboer I 3 CLEPS Workshop/Mini-conference, August 29, 2005. Overview. Experiment: Theory Design

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Uncovering the Problem-Solving Process:' - kenton

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Uncovering the problem solving process
Uncovering the Problem-Solving Process:

Cued Retrospective Reporting, Eye Tracking, and Expertise Differences

Tamara van Gog, Fred Paas, & Jeroen J. G. van MerriënboerI3CLEPS Workshop/Mini-conference,

August 29, 2005



  • Theory

  • Design

  • Comparison of 3 verbal methods

  • The 3 methods & expertise differences

  • Uncovering expertise-related performance differences through eye movement data

  • Present limitations and future research

  • Discussion


Use of process-tracing techniques to uncover problem-solving processes in order to advance / inform:

  • Psychological theory

  • Expert systems

  • User-system interaction,

    But also

  • Instructional design e.g., design of process-oriented worked examples


From the literature (Kuusela & Paul, 2000; Taylor & Dionne, 2000):

+ of concurrent reporting (“think aloud”): more information on actions taken

+ of retrospective reporting: more information on rationale for actions taken and strategies that control the process

Needed: A method that combines + & + :

Cued retrospective reporting based on a record of eye movements & mouse/keyboard operations?


Within-subjects, 26 participants, electrical circuits troubleshooting tasks:

Seq. Condition + Tasks

1 CR 1+2 CRE 3+4 RR 5+6 CRR 7+8

2 CRE 3+4 CRR 7+8 CR 1+2 RR 5+6

3 RR 5+6 CR 1+2 CRR 7+8 CRE 3+4

4 CRR 7+8 RR 5+6 CRE 3+4 CR 1+2

CR = concurrent reporting; CRE = concurrent reporting with eye tracking; RR = retrospective reporting; CRR = cued retrospective reporting.

Comparison of 3 methods hypotheses
Comparison of 3 Methods: Hypotheses

1. Concurrent reporting (CR): more ‘action’ info than RR

2. Retrospective reporting (RR): more ‘why’, ‘how’, & ‘metacognitive’ info than CR

3. Cued retrospective reporting (CRR):-> more ‘action’ than RR-> more ‘why’, ‘how’, & ‘metacognitive’ than CR

Comparison of 3 methods analyses
Comparison of 3 Methods: Analyses

Segmentation based on speech  sentences / utterances (preceded & followed by a pause)

Coding scheme task-oriented main categories:





20% of protocols scored by 2 raters: kappa = .79 good; proceeded with 1 rater

Analyses on nr. of codes on main categories, obtained by summing codes on subcategories

Comparison of 3 methods results
Comparison of 3 Methods: Results

Friedman Tests with Conover (1999) comparisons

CR vs RR:as hypothesized: ‘action’  CR >RRhowever: ‘why’ and ‘how’  CR > RR, and‘metacognitive’ CR = RR

CRR vs RR:as hypothesized: ‘action’  CRR >RR‘why’: CRR = RR‘how’ and ‘metacognitive’: CRR > RR

Expertise differences explorative
Expertise Differences: Explorative

5 “highest” and 5 “lowest” expertise participants (from 26). Determined by performance efficiency:

“highest”: higher performance, lower mental effort, lower time-on-task

“lowest”: lower performance, higher mental effort, higher time-on-task

  • Differences in elicited information?

  • Differences in preferences/experiences?(open-ended debriefing questions)

Expertise differences elicited information
Expertise Differences: Elicited Information

Differences in elicited information?

(Mann-Whitney U Tests)


‘how’ and ‘metacognitive’ info: “lowest” > “highest”


‘why’ info: “highest”> “lowest”

‘how’ info: “lowest” > “highest”


‘action’ and ‘metacognitive’ info: “lowest” > “highest”

Expertise differences experience
Expertise Differences: Experience

Differences in preferences/experiences?

“lowest”: experience: CR  (4/5)preference: CRR > CR & RR (4/5)

“highest”:no differential experiences/preferences

Mediating factors mentioned re. experience / preference, by both “lowest” and “highest”:

  • Time-on-task

  • Cue

Studying expertise related performance differences eye movement data 1
Studying Expertise-Related Performance Differences: Eye Movement Data 1

Eye fixation data provide insight in the allocation of attention, and hence differ with expertise

Research use: provide information about the problem-solving process at a finer grained level than verbal protocols?

  • (Ultimate) educational use: guiding novices’ attention?

1 Data from Van Gog, Paas, & Van Merriënboer (2005), Applied Cognitive Psychology

Eye movement data participants procedure
Eye Movement Data: Movement Data Participants & Procedure

Same 5 “lowest” and 5 “highest” expertise participants

Data collected in first 3 phases of the process:

  • Problem orientation (until pushing switch to observe circuit behavior)

  • Problem formulation and action decision

  • Action evaluation and next action decision

    % time spent on phase, mean fixation duration (MFD), and in 1st phase fix. related to faults

Task Movement Data


Only 3 Volt

Eye movement data results
Eye Movement Data: Movement Data Results

Phase 1: problem orientation

(Mann-Whitney U Tests, 2-tailed, α = .10)

% of time: “highest” > “lowest”

MFD: “lowest” > “highest”

% fixations on battery: “highest” > “lowest”

Gaze switches short-circuit: “highest” > “lowest” (NB: only trend)

Eye movement data results1
Eye Movement Data: Movement Data Results

Phase 2: problem formulation & action decision

(Mann-Whitney U Tests)

% of time: “highest” = “lowest”

MFD: “highest” = “lowest”

MFD First ½: “highest” > “lowest”

MFD Second ½: “highest” = “lowest”

Eye movement data results2
Eye Movement Data: Movement Data Results

Phase 3: action evaluation & next action decision

(Mann-Whitney U Tests)

% of time: “highest” > “lowest”

MFD: “highest” = “lowest”

MFD First ½: “highest” = “lowest”

MFD Second ½: “highest” = “lowest”

Eye movement data results3
Eye Movement Data: Movement Data Results

MFD over phases (Friedman + Nemenyi post-hoc):

n.s. for “lowest”; “highest” 1 < 2.1., 2.2., 3.2 & 2.1 >3.1

Limitations Movement Data

  • CRR and fabrication?

  • Cue: combination of eye movements AND mouse/keyboard operations

  • Only quantitative analyses of protocols

  • Eye movement data: distinction of phases

  • Performance efficiency measure:very relative distinction (lowest and highest within this group of participants)

  • Small nr of participants in analyses related to expertise differences

Future research
Future Research Movement Data

  • Qualitative differences between CRR and RR?

  • Cue: different effects with only eye movements OR mouse/keyboard operations?

  • Cue: technical optimization?

  • (RR/)CRR: effects of other prompts?

  • Further study of performance efficiency measure to distinguish expertise levels

  • Replications with larger N

Thank you for your attention
Thank you for your attention! Movement Data

[email protected]