The Development of a Delayed Retrospective Report Method to Increase the Reliability and Validity of...
1 / 23

The Development of a Delayed Retrospective Report Method to Increase the Reliability and Validity of Verbal Reports abou - PowerPoint PPT Presentation

  • Uploaded on

The Development of a Delayed Retrospective Report Method to Increase the Reliability and Validity of Verbal Reports about Past Events. David W. Eccles, Paul Ward, Kevin R. Harris, Lauren Tashman, K. Anders Ericsson, and Laura B. Hassler Expert Performance Research Team,

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'The Development of a Delayed Retrospective Report Method to Increase the Reliability and Validity of Verbal Reports abou' - bella

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Slide1 l.jpg

The Development of a Delayed Retrospective Report Method to Increase the Reliability and Validity of Verbal Reports about Past Events

David W. Eccles, Paul Ward, Kevin R. Harris, Lauren Tashman,

K. Anders Ericsson, and Laura B. Hassler

Expert Performance Research Team,

Human Performance Laboratory

Learning Systems Institute,

The Florida State University, USA

Using conventional interviews to provide insights into cognition during performance l.jpg
Using Conventional Interviews to Provide Insights into Cognition during Performance

  • Conventional interview - common method of obtaining insights into cognitive processes mediating differentially skilled performance

  • For example, an interview conducted after a military mission to investigate the nature of a decision making during that mission

Problems with interviews l.jpg
Problems with Interviews Cognition during Performance

  • Key problem with interviews and other self-report measures - tendency by respondents to interpret, explain, summarize thoughts (Nisbett & Wilson, 1977).

Think aloud technique ericsson and simon 1993 l.jpg
Think Aloud Technique - Ericsson and Simon (1993) Cognition during Performance

  • A task-concurrent cognitive process-tracing method – used in domains ranging from chess to medical practice (Ericsson & Lehmann, 1996).

  • Performers think out aloud their thoughts, as if alone, while performing a task

  • Perform TA exercise

Think aloud technique l.jpg
Think aloud technique Cognition during Performance

  • Example think aloud: Think aloud while identifying the 4th letter in the alphabet after the letter P.

    Reports might look like “There’s the P, then there’s err Q, then R, S, and then T, it’s T”.

  • Thought to elicit verbal reports that reflects contents of WM during task execution

  • Reduces interpretations, explanations, and summaries of thoughts

  • Not the performer’s interpreted account of their thoughts but a more direct reflection of their actual thoughts

Constraints on applicability of think aloud method l.jpg
Constraints on Applicability of Think Aloud Method Cognition during Performance

  • Physical constraints (e.g., from rapid respiration)

  • Environmental constraints (e.g., task is undertaken in water)

  • Event danger/importance (i.e., unwillingness to verbalize during critical events)

  • In some domains, neither feasible nor desirable to apply think aloud method

Solution immediate retrospective report method l.jpg
Solution: Immediate Retrospective Report Method Cognition during Performance

  • After completing a task, the participant reports immediately on the thoughts experienced during the task

  • Perform TA and IRR exercise

Immediate retrospective report l.jpg
Immediate retrospective report Cognition during Performance

  • Example alphabet task IRR:

    Experimenter: What was the first thought you remember having while coming up with the answer, and the next, and the next, and so on?

    Participant: “I saw a P, and then there was a Q, R, S, and finally a T, it was T”.

Limits on irr applicability l.jpg
Limits on IRR Applicability Cognition during Performance

  • Limited to those situations in which the researcher can apply the method immediately following task completion

  • To return to the example of examining mission decision making, access to military personnel might not be possible for some time after the mission

Solution delayed retrospective report l.jpg
Solution: Delayed Retrospective Report Cognition during Performance

  • Our research group has been developing a delayed retrospective report (DRR) method

  • DRR is identical in essence to the IRR but comprises many more directions for the participant that support the process of recalling thoughts experienced during the event

  • Incremental approach

  • participant directed to provide a brief overall description of the event

  • a timeline around a critical event established that includes clearly recalled “event markers”

  • participant directed to recall thoughts and actions during each event marker, staring with first, then second, third, and so on

  • Crucially, like the IRR method, the DRR method directs the participant to:

    - only report those thoughts and actions that can be recalled with certainty

    - avoid interpreting, explaining, or summarizing thoughts or actions

Early attempts to use drr with police recall of thoughts while performing under stress study l.jpg
Early Attempts to Use DRR with Police– Cognition during PerformanceRecall of Thoughts While Performing Under Stress Study

Step 1: Initial overall description of event obtained and establishment of sub-events and their timeline

  • I was dispatched to an attempted armed robbery call. The robbery was attempted in a man’s yard. Two suspects actually tried to rob an individual who…was armed. There was a stand off, no shots were fired, and the suspects fled.

  • I chose a route that I thought would most likely bring me into contact with the suspect vehicle. I…meet the suspect vehicle on the road, turned around on it, initiated my overhead lights. The suspect vehicle yielded. I exited my vehicle with my shotgun, conducted a felony [vehicle] stop [procedure], and took the driver into custody.

Step one the timeline l.jpg
Step One: The Timeline Cognition during Performance



Car seen

Car stopped

Radio call

Step 2 identifying most stressful part of event l.jpg
Step 2: Identifying Most Stressful Part of Event Cognition during Performance

  • The most stressful part of the whole encounter was from the period that I realized that he was going to yield [his vehicle] to me and the point that I actually exited my vehicle and had my shotgun trained on him

  • I was at my most vulnerable from the time that he stopped and I stopped behind him because the suspect had an advantage then in that he’s already made up his mind what he’s gonna do and if he decides to exit his vehicle quickly with a handgun, he will probably be able to get shots off at me before I can either respond or extract myself from the situation

Delayed retrospection of thoughts and actions using timeline l.jpg
Delayed Retrospection of Thoughts and Actions using Timeline Cognition during Performance

Section of transcript:

  • I: So what was the first thought you remember having as, as the vehicle started to yield?

  • PO: There’s only one guy in the car.

  • I: What was your next thought after that?....

  • PO: I do absolutely recall thinking “be careful, don’t let the [shotgun] sling get tangled up on say the gear shift or the radio”…and if it does get tangled up…I need to just drop it and immediately draw my handgun and engage him with that.

  • I: What was the next thought you remember?

  • PO: I’m, I’m thinking light him up [with the shotgun flashlight] and then go through a felony stop ritual.

Attempts to validate drr l.jpg
Attempts to Validate DRR Cognition during Performance

  • Current research in our laboratory with police officers involves studying the relative validity and reliability of the methods described here

Some context http www wnbc com news 4706691 detail html l.jpg
Some Context - Cognition during Performance

July 11 2005

  • LOS ANGELES – Police shot and killed a man Sunday night when he emerged from his home carrying a young girl, about 17 months old, following an hours-long standoff and opened fire, wounding one officer. The girl was also killed in the gunfire.

  • [LAPD Chief McDonnell] said. "The officers are taking it very hard," McDonnell said. "Anytime you have a baby killed, it takes its toll."

Police simulation in human performance laboratory l.jpg
Police Simulation in Human Performance Laboratory Cognition during Performance

Expert Police Officer “Truman”

Drr validation police study l.jpg
DRR Validation – Police Study Cognition during Performance

12 expert (SWAT, TAC) and 12 novice (cadet) police officers (but I’m not going to report here on expert-novice differences)

Initial training in think aloud and IRR techniques with established exercises

In a simulator, officers equipped with modified gun, exposed to 20 film clips depicting representative and stressful police scenarios, requiring shoot/no shoot decision

Video captures behavior of officer – handles gun, unholsters gun, points and aims gun relative to events occurring on film – all verbal and non-verbal commands

For each shot, computer captures initial trigger squeeze time point, full trigger depression time point, and gun shot placement relative to events occurring on screen

IRR captures thoughts experienced during scenario

Police scenario simulation human performance laboratory l.jpg
Police Scenario Simulation, Cognition during PerformanceHuman Performance Laboratory

  • Plan of equipment






Video of whole body behavior

Video record of trigger behavior

Procedure l.jpg
Procedure Cognition during Performance

DRR is then performed about thoughts and actions undertaken during two

(a priori determined) stressful scenarios, indicated with*

Procedure triangular comparisons l.jpg
Procedure – triangular comparisons Cognition during Performance

  • Compare reports from IRR obtained immediately after certain scenarios with events in the film and recorded response behaviors of officer during those scenarios

  • Compare IRR reports with those obtained from DRR undertaken some time after all scenarios are completed

  • Compare DRR reports with events in the film and recorded response behaviors of officer during certain scenarios

Towards the damn konfident l.jpg
Towards the “Damn Konfident” Cognition during Performance

  • In terms of the reliability of knowledge continuum proposed by Rainer Martens, the DRR method, when compared to a conventional interview method, should help us move away from feeling like we “Don’t Know” and towards feeling “Damn Konfident” about the findings we obtain from verbal report methods (Martens, 1987).

Don’t Know

Damn Konfident

Acknowledgements l.jpg
Acknowledgements Cognition during Performance

  • US Office of Naval Research, Arlington, VA

  • The Florida State University Center for Research and Creativity

  • Judd Butler, Stavroula Perdikogiannis, Jason Ramratten, Mike Tuffiash, Ning Zhuang

  • Pat Thomas Law Enforcement Academy, Quincy, FL