1 / 18

Interobserver agreement

Interobserver agreement. The degree to which observers agree on the ___________________________ of behavior Why is it important? Are your measures ___________? Are the effects and levels of behavior_________? You need ___ independent observers Evaluate in at least _____ of observations

sumi
Download Presentation

Interobserver agreement

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Interobserver agreement • The degree to which observers agree on the ___________________________ of behavior • Why is it important? • Are your measures ___________? • Are the effects and levels of behavior_________? • You need ___ independent observers • Evaluate in at least _____ of observations • Minimum IOA should average above ___% Lect 4

  2. ____ frequency X 100% ____ frequency Total Agreement • Used with _______________ recording • Observers tally _______ of responses • Easy to calculate • Limitations: • Unsure whether agreement in on the _________________ • Difficult to assess with _________ behaviors Lect 4

  3. Calculating Total Agreement OBSERVER 1 | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 50 OBSERVER 2 | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 45 IOA ? 75% 90% Lect 4

  4. Interval (point-by-point) Agreement Used when specific records can be compared on a point-by-point basis (_________ ________________________) • _____________ • _____________ (X) • _____________ (0) agreements disagreements Lect 4

  5. Calculating Interval Agreement _______________ Agreement: = Agreements X 100% (Agreements + Disagreements) Lect 4

  6. Obs. 1 Obs. 2 1 D X O A 2 X X A 3 X X A X X 4 A O O A 5 X X D 6 __ O X A IOA = x 100% O O ___ 7 A X X A 8 O O 9 10 Calculating Interval Agreement Overall Agreement (occurrence and nonocurrence) = ___%

  7. Obs. 1 Obs. 2 1 A O O D 2 X O A 3 O O A O O 4 A O O A 5 O O D 6 O X A O O 7 A O O A 8 O O 9 10 Overall agreement high (80%), but observers disagree on when behavior _______ Lect 4

  8. Obs. 1 Obs. 2 1 A X X D 2 X X A 3 O X A X X 4 A X X A 5 X X D 6 X X A X O 7 A X X A 8 X X 9 10 Overall agreement high (80%), but observers disagree on when behavior ____________ Lect 4

  9. Calculating Interval Agreement Because overall agreement doesn’t determine whether disagreements occurred in the occurrence or nonoccurrence of behavior, researchers also calculate occurrence and non-occurrence agreement _______________ Agreement: = Occurrence Agreements X 100% (Occurrence Agreements + Occurrence Disagreements) _______________ Agreement: = Nonoccurrence Agreements X 100% (Nonoccurrence Agreements + Nonoccurrence Disagreements)

  10. 1 D X O 2 A X X 3 A X X 4 A X X 5 A O O 6 A X X 7 D ___ O X 8 A IOA = x 100% O O 9 ___ A 10 X X A O O Calculating Interval Agreement Obs. 2 Obs. 1 Ignore intervals in which observers agreed on ________________ Occurrence = _____%

  11. 1 D X O 2 A X X 3 A X X 4 A X X 5 A O O 6 A X X 7 D __ O X 8 A IOA = x 100% O O 9 ___ A 10 X X A O O Calculating Interval Agreement Obs. 1 Obs. 2 Ignore intervals in which observers agreed on ________________ Non-occurrence = ___%

  12. 1 O O 2 X X 3 X X 4 X X 5 X O 6 X X 7 O O 8 O O 9 10 X X O O Obs. 1 Obs. 2 IOA? Overall: 80% ? 90% ?

  13. 1 O O 2 X X 3 X X 4 X X 5 X O 6 X X 7 O O 8 O O 9 10 X X O O IOA? Occurrence: 90%? 83% ? Nonoccurrence 44%? 80% ? Obs. 1 Obs. 2

  14. 1 X O 2 O X 3 X X 4 X X 5 X O 6 O X 7 O O 8 O X 9 10 X X O O Obs. 1 Obs. 2 IOA? Overall: 70% ? 50% ?

  15. 1 X O 2 O X 3 X X 4 X X 5 X O 6 O X 7 O O 8 O X 9 10 X X O O Obs. 1 Obs. 2 IOA? Occurrence: 38% ? Nonoccurrence: 39% ? 29% ?

  16. Factors influencing inter-observer agreement • Observer _______________ • Observers aware their observations will be checked and score behavior ___________ • Prevent by watching them from a unobtrusive location, without their awareness, or by watching them often. • Observer _________ • Observers might gradually ______ what they measure • Prevent by making sure they are well _______, periodic _______, good ___________ definitions, spot checks. Lect 4

  17. Factors influencing inter-observer agreement • _____________ of the measurement system • If observers are recording too many _______ or too many __________ they you may get low IOA • If data sheets too _____________ you may get low IOA • Keep data collection and data sheets as ______ as possible • Observer ________________ • May be biased and record what they ______should happen • To prevent observers should be _____ to purpose of study Lect 4

More Related