Observational Methods - PowerPoint PPT Presentation

observational methods n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Observational Methods PowerPoint Presentation
Download Presentation
Observational Methods

play fullscreen
1 / 116
Observational Methods
164 Views
Download Presentation
eilis
Download Presentation

Observational Methods

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Observational Methods January 20, 2010

  2. Today’s Class • Survey Results • Probing Question from Wed, Jan. 20 • Observational Methods • Probing Question for Fri, Jan. 22 • Survey

  3. Survey Results • Only 3 responses  • I will be switching to a better technology for the next survey • Mostly positive  • I’ve read through your comments, and will try to take them into account • Quick comment – if you think a key term is being incompletely or unclearly defined, please let me know, and I will try to explain it better • The concepts covered in the first class will be re-visited as we go, and perhaps will become clearer as we get more examples

  4. Today’s Class • Survey Results • Probing Question from Wed, Jan. 20 • Observational Methods • Probing Question for Fri, Jan. 22 • Survey

  5. HOLISTIC EXISTENTIALIST ESSENTIALIST ENTITATIVE

  6. Today’s Class • Survey Results • Probing Question from Wed, Jan. 20 • Observational Methods • Probing Question for Fri, Jan. 22 • Survey

  7. Observational methods: top level • You don’t manipulate the research setting in any way • You just look at what is happening, or what has happened, and you note it • Later, you analyze the data

  8. Observational Methods • This class we will focus on quantitative observational methods • Where observations are noted in terms of a defined set of categories • Called “quantitative” (numerical) because of the analytic methods commonly used with this kind of data

  9. Observational Methods • This week we will focus on quantitative observational methods • Next week we will focus on qualitative methods • Where observations are noted in a more free-form way that does not attempt to rigorously apply a defined set of categories, but attempts to understand cases in their full particularity

  10. What is the research setting? • Where the learning (or other phenomenon you want to study) is taking place

  11. What is the research setting? • Usually a real learning setting • Classroom • University computer lab • Dorm room • Workplace • Any others? • Occasionally a lab setting • Needs to be justification for why the lab setting is valid

  12. Types of observation • Enhanced video coding • Video coding • Field observation • Screen replay coding • Text replay coding • (and more)

  13. High-level notes • High-level methodological similarity • Can be used for many of the same research goals • Advantages and disadvantages to each

  14. Types of observation(that we’ll focus on today) • Video coding • Field observation • Text replay coding

  15. Types of observationthat we’ll focus on today • Video coding • Field observation • Text replay coding

  16. Let’s watch a video • http://www.youtube.com/watch?v=lmoEWCejQIg&feature=related • From 0:00 to 1:15

  17. What student behaviors did you observe? • Please explain both what the behavior is, and how you could recognize it from the video • Does anyone disagree with how a behavior is defined? Why?

  18. List of behaviors • Let’s briefly write down a list of behaviors seen

  19. OK, let’s watch it again • Please write every behavior you see that did not fit into the original coding scheme

  20. What additional student behaviors did you observe? • Please explain both what the behavior is, and how you could recognize it from the video • Does anyone disagree with how a behavior is defined? Why?

  21. What is the total list of behaviors?

  22. Now let’s make a coding scheme • A list of “interesting” behaviors, how they are defined, and a code letter • Example: • Hint abuse (a type of gaming the system) • Clicking through hints at high speed until reaching the bottom-out hint, obtaining and entering the answer, and immediately moving on to the next problem step

  23. Now let’s make a coding scheme • Should not be infinitely long – eliminate • “Uninteresting” behaviors – be guided by • your intuition • prior theory and empirical results • “Extremely rare” behaviors – under approximately 1 in 200 • Behaviors occuring between 1-5% of the time are often very interesting behaviors! • Effects on learning are often disproportionate • Gaming the system occurs 3-5% of the time, and correlates to learning at 0.3-0.4

  24. Which behaviors in your list • Are “uninteresting”? • Are “extremely rare”?

  25. Now let’s make a coding scheme • And combine related behaviors • For example, different types of gaming the system (e.g. hint abuse, systematic guessing, intentional rapid mistakes) often get classified together • common • antecedents (poorly known skill) • characteristics (very fast responses) • results (poor learning gains) • differences are no longer considered theoretically interesting (some debate on this still) • tend to co-occur

  26. Which behaviors • Should be combined?

  27. Our Coding Scheme

  28. OK, let’s watch it for the 3rd time • Each person in our class will be assigned a student in the video (minimum of 2 of us for each 1 of them) • Every five seconds of video, I am going to stop the video, and you need to write down the best code for the previous 5 seconds • If 2 behaviors observed, pick the dominant one • If you are not sure, put down a “?” • Do NOT talk to your partner

  29. OK, now work with your partner • Find out what % of time you agreed • Find out where and how you disagreed • There are better ways to assess inter-rater reliability, we’ll discuss these next class

  30. Each group’s % error

  31. What categories did you disagree on?

  32. OK, now let’s watch • Same video, but times 1:06 to 2:10 • Observe the same student • Follow the same procedure as before

  33. Were the behaviors the same in this segment? • It is important to: • Develop your coding scheme based on sufficient initial qualitative observation (and/or theory and prior work) • Note behaviors that do not fit in your coding scheme

  34. OK, now work with your partner • Find out what % of time you agreed • Find out where and how you disagreed

  35. Across sessions • Did your agreement go up or go down? • By how much?

  36. A good strategy • Repeatedly code the same data separately and compare • Until your inter-rater reliability is “high enough” (more next time) • And then code different data separately

  37. Incidentally • There is nothing “magic” about the video coding method we used • It was largely selected for tractability in class

  38. Other things we could have done • Watch the same time segment multiple times • Advantages • Disadvantages

  39. Other things we could have done • Watch the same time segment multiple times • Advantages: Greater certainty in coding • Disadvantages: Takes a lot longer; allows second-guessing and hair-splitting

  40. Other things we could have done • Code when the behavior changes, not on a pre-selected time interval • Advantages • Disadvantages

  41. Other things we could have done • Code when the behavior changes, not on a pre-selected time interval • Advantages: Get better assessment of the duration of each behavior • Disadvantages: Enforces coding one student at a time; some code switches do not have “hard” edges (especially in case of coding affect)

  42. Other things we could have done • Code the first action in a time window, not the predominant one • Advantages • Disadvantages

  43. Other things we could have done • Code the first action in a time window, not the predominant one • Advantages: Less ambiguity and decision-making • Disadvantages: May be less representative; very brief behaviors may be missed

  44. Other things we could have done • What else?

  45. A limitation of one camera

  46. What is she doing?

  47. Other ways to use video • Multiple cameras • Addresses issue of the student off-screen • Synchronization becomes a challenge • Time to code increases with each camera you add

  48. Other ways to use video • Individual student video • Often from web-cams on top of machines

  49. Example

  50. Example • https://umdrive.memphis.edu/sdmello/memphis-affect-dialogue.wmv?ticket=t_dPrPbv8H