Data Collection: Designing an Observational System

1. Data Collection: Designing an Observational System

2. What is an observational system? • An observational system is the system you use to collect the data that you need to analyze in order to evaluate your program. It details the way you’ll look at the process, progress, and outcomes of your work, and how you’ll examine the behavior, conditions, or events that you’re concerned with.

3. Why design an observational system? • It can help you get reliable information. • It can help you find out exactly what you need to know, eliminating or reducing wasted effort. • It can ensure that observations are made. • It can make it easier to analyze your data. • It can help you avoid haphazard evaluation.

4. Why design an observational system? • It will make it easier to justify your findings. • It can help you gain credibility with funders and policy makers. • It can let you pass on your practices with confidence. • Most important, it can give you the best information possible about what’s working in your program, and what you need to adjust.

5. When should you design an observational system? • If you can, you should design the system before your program begins, so that you can watch it and its effects from the very beginning. • If that’s not possible, you should design your system before you start your evaluation, ideally at the start of a program cycle.

6. Who should design an observational system? • Observational systems are usually best designed by a participatory group that includes both researchers or evaluators and people who will do the actual observation.

7. How do you design an observational system? • Review your evaluation questions. • Decide what you need to observe to answer your questions. • Participants’ behavior. • Someone else’s behavior. • Conditions. • Observations of results of behavior. • Participants’ knowledge or attitudes. • Someone else’s knowledge or attitudes. • Goal attainment. • Interactions. • Program process or implementation (e.g., numberof participants).

8. How do you design an observational system? • Decide how the observations will be conducted. • Direct observation. • Participant observation. • Self-reports, including individual and group interviews, focus groups, journals, surveys, etc. • Second-hand reports, including interviews, journals, surveys, etc. • Electronic or mechanical observation. • Tests or other similar observation tools. • Public records and the like for community-level indicators.

9. How do you design an observational system? • Decide when you need to observe. • Pre- and post- observation. • At regular intervals during the evaluation period. • At irregular intervals during the evaluation period. • At specific times during the evaluation period. • Continuously.

10. How do you design an observational system? • Define and describe the behaviors, conditions, and/or events that observers should be concerned with. • Train observers in: • What it’s important to record, and why. • The definitions and descriptions of the behaviors, conditions, events, or situations to be observed. • Effects of observation. • Observer bias. • Observer drift.

11. How do you design an observational system? • Devise checks for reliability and accuracy. • Use an external standard. • Check for inter-rater reliability. • Use random third-party checks. • Adjust the system for the next evaluation.