think aloud protocols n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Think-Aloud Protocols PowerPoint Presentation
Download Presentation
Think-Aloud Protocols

Loading in 2 Seconds...

play fullscreen
1 / 66

Think-Aloud Protocols - PowerPoint PPT Presentation


  • 328 Views
  • Uploaded on

Think-Aloud Protocols. February 5, 2010. Today’s Class. Probing Question Think-Aloud Protocols Assignments. Probing Question for Friday, February 5. Let’s say that the NSF has just given you a $100,000 grant to study student note-taking during classroom lecture

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Think-Aloud Protocols' - kayla


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
think aloud protocols

Think-Aloud Protocols

February 5, 2010

today s class
Today’s Class
  • Probing Question
  • Think-Aloud Protocols
  • Assignments
probing question for friday february 5
Probing Question for Friday, February 5
  • Let’s say that the NSF has just given you a $100,000 grant to study student note-taking during classroom lecture
    • i.e. you can assign one minion to the task full-time for a year
  • Which qualitative method would you prefer to use, and why?
    • What factors would you be especially attentive to, to ensure validity of your results?
today s class1
Today’s Class
  • Probing Question
  • Think-Aloud Protocols
  • Assignments
i need a volunteer
I need a volunteer
  • A fast typist
  • To try to record things people say while thinking aloud
  • Only record the person thinking aloud
i need two more volunteers1
I need two more volunteers
  • Volunteer #1: Will think-aloud
  • Volunteer #2: Will run the think-aloud
what should volunteer 2 remember while running the think aloud1
What should Volunteer #2 rememberwhile running the think-aloud?
  • Use appropriate instructions
  • Prompt the subject
  • Write down interesting events
  • Focus on what the subject does and says, not “why”
  • Don’t help the subject
long division
Long division
  • Volunteer #1 will think-aloud
  • Volunteer #2 will run the think-aloud
  • So, Volunteer #2, give Volunteer #1 some instructions…
long division1
Long Division

7140

16, 429, 782, 600

how did it go
How did it go?
  • Volunteer #1:
  • How hard was the task?
  • Did thinking aloud make it harder?
how did it go1
How did it go?
  • Volunteer #2:
  • How hard was it to get him/her to think-aloud?
  • Did you worry about disrupting him or her?
  • Audience:
  • Any (friendly and constructive!) criticism for Volunteer #2?
thoughts comments
Thoughts/comments
  • From anyone
  • On why this task might have been difficult/easy for the person thinking aloud, or the experimenter
  • (or anything else)
drawing a scatterplot
Drawing a scatterplot
  • Volunteer #1 will think-aloud
  • Volunteer #2 will run the think-aloud
how did it go2
How did it go?
  • Volunteer #1:
  • How hard was the task?
  • Did thinking aloud make it harder?
how did it go3
How did it go?
  • Volunteer #2:
  • How hard was it to get him/her to think-aloud?
  • Did you worry about disrupting him or her?
  • Audience:
  • Any (friendly and constructive!) criticism for Volunteer #2?
thoughts comments1
Thoughts/comments
  • From anyone
  • On why this task might have been difficult/easy for the person thinking aloud, or the experimenter
  • (or anything else)
the 9 dot problem kershaw ohlsson 20041
The 9-Dot Problem(Kershaw & Ohlsson, 2004)

Volunteer #2 will connect all 9 points with just 4 straight lines(you don’t already knowthe answer, right?)

how did it go4
How did it go?
  • Volunteer #2:
  • How hard was the task?
  • Did thinking aloud make it harder?
how did it go5
How did it go?
  • Volunteer #1:
  • How hard was it to get him/her to think-aloud?
  • Did you worry about disrupting him or her?
  • Audience:
  • Any (friendly and constructive!) criticism for Volunteer #1?
thoughts comments2
Thoughts/comments
  • From anyone
  • On why this task might have been difficult/easy for the person thinking aloud, or the experimenter
  • (or anything else)
insight problems
Insight Problems
  • The biggest problem with thinking-aloud as a technique.
  • When people are trying to figure something out, and it requires “insight”, thinking-aloud can be very disruptive.
  • Therefore, if your participant is really baffled,
  • Now might not be exactly the time to push them to “KEEP TALKING”
  • In general, it’s best to avoid using think-aloud in these situations
  • What else might you use to study process in these situations?
trogdor
Trogdor!
  • http://www.homestarrunner.com/trogdor.html
how did it go6
How did it go?
  • Volunteer #1:
  • How hard was the task?
  • Did thinking aloud make it harder?
how did it go7
How did it go?
  • Volunteer #2:
  • How hard was it to get him/her to think-aloud?
  • Did you worry about disrupting him or her?
  • Audience:
  • Any (friendly and constructive!) criticism for Volunteer #2?
thoughts comments3
Thoughts/comments
  • From anyone
  • On why this task might have been difficult/easy for the person thinking aloud, or the experimenter
  • (or anything else)
ok enough for now
OK enough for now
  • What have we learned about think-alouds from all of this?
  • Bonus points for expressing your thought as a Beyer & Holtzblatt-style aphorism
what could you do
What could you do?
  • If you want to do a think-aloud on some uninterruptable process?
example
Example
  • Take, for example, a helicopter pilot who needs to drop a ladder to someone while still flying
task splitting
Task-splitting
  • You might want to do a think-aloud of the pilot’s processes as (s)he does each of these tasks separately: ie, a think-aloud as (s)he lowers a ladder, and a think-aloud as (s)he flies the plane
  • This would show you the pilot’s major sub-goals, decision-points, and evaluation criteria for each task
retrospective think aloud
Retrospective Think-Aloud
  • Videotape the participant as they are doing the task
  • Take a 10 minute break
  • Then show the video to the participant and have them do colour-commentary.
  • Take notes on their colour-commentary
  • (Or even tape-record it)
retrospective think aloud is also useful
Retrospective Think-Aloud is Also Useful…
  • When you are studying someone who is so expert at an action, that their actions are completely automatized.
  • In this case, thinking-aloud can disrupt or change someone’s behavior.
volunteer 1
Volunteer #1
  • Volunteer #1: Please tell me the cash value of the change you have in your pocket. Don’t think aloud.
volunteer 2
Volunteer #2
  • Volunteer #2: Please THINK ALOUD while you determine the cash value of the change you have in your pocket.
volunteer 21
Volunteer #2
  • Volunteer #2: Please THINK ALOUD while you determine the cash value of the change you have in your pocket.
  • Volunteer #1, is this the procedure you used?
analysis1
Analysis
  • Let’s take the longest of the protocols that our noble live transcriber has produced
  • Can you email it to me?
    • I’ll put it up on the screen
segmentation
Segmentation
  • It’s better to do segmentation from an audio recording than a transcript in general, but for this time…
segmentation1
Segmentation
  • Your thoughts on where we should segment?
cognitive processes
Cognitive Processes
  • What are the cognitive processes found in these utterances?
cognitive processes1
Cognitive Processes
  • Do we have an “irrelevant to task” category?
    • Recommended by van Someren
coding
Coding
  • Let’s code the whole transcript with regard to the list of cognitive processes
  • For each segment, quickly write down your code, and then read it out loud
    • What level of agreement do we obtain?
    • Why do we disagree?
question
Question
  • Let’s think about the coding scheme we used
  • How did the coding scheme in Gobert (1999) differ from this?
question1
Question
  • Let’s think about the coding scheme we used
  • How did the coding scheme in Gobert (1999) differ from this?
  • Atheoretical .vs. theoretical
advantages disadvantages
Advantages/Disadvantages
  • What are the advantages and disadvantages of
    • atheoretical coding schemes
    • theoretical coding schemes
what kind of participants could you involve in a think aloud1
What kind of participants could you involve in a think-aloud?
  • Complete novice – to see learning
  • Expert – to see how “it should be done”
  • Recent learner – to see process of someone who has just acquired skill (may be more applicable for designing instruction)
for each think aloud we did what kind of participant did we have
For each think-aloud we did,what kind of participant did we have?
  • Complete novice
  • Expert
  • Recent learner
  • Other
where do think alouds fit on this diagram
Where do think-aloudsfit on this diagram?

HOLISTIC

EXISTENTIALIST

ESSENTIALIST

ENTITATIVE

janice
Janice
  • Do you have anything you’d like to add, as one of the world’s foremost experts on think-alouds?
today s class2
Today’s Class
  • Probing Question
  • Think-Aloud Protocols
  • Assignments
assignment 1
Assignment #1
  • Very high quality submissions overall!
  • Complete grading and commentary for assignment #1 will be a little bit delayed
assignment 2
Assignment #2
  • Any questions?
assignment 3
Assignment #3
  • Will be handed out Monday (by email)