Review of Evaluation Research

1. Review of Evaluation Research Research Process Step One – Conceptualization of Objectives Step Two – Measurement of Objectives Step Three – Determine Sampling Technique Step Four – Determine Data Collection Design Step Five – Collect and Analyze Data Step Six – Develop Graphs and Charts to Present Data. Step Seven – Write a Report – Reporting Results

2. Conceptualization Proposition Independent Variable Dependent Variable

3. Outcome Measurements • Identify Independent and Dependent Variables • Independent • Intervention Strategies • Dependent • Change Resulting from Intervention • Develop Reliable and Valid Indicators of these Measurements • Reliable – Indicator provides consistent measurement across time • Valid – Indicator provides accurate measurement

4. Example - Conceptualization Hypothesis Dependent Variable Independent Variable

5. Operationalization of Concepts:Identifying Indicator Concept Indicator

6. Operationalization of Concepts;Identifying Indicator Concept Indicator

7. State Relationships Between Indicators Indicator Indicator

8. Selecting a Sampling Technique • Non-Random Sample – for Descriptive Statistics • Convenience Sample • Snowball Sample • Theoretical Sample • Random Sample – For Inferential Statistics

9. Examples- Non-Random Sampling Techniques • Non-Random Sample • Convenience Sample – Select units that are convenient (i.e., the nearest fields of crops) • Snowball Sample – Have one farmer refer you to another who will refer you to another etc. • Theoretical Sample – Your theory states that this fertilizer only works for innovative farmers so you select only innovative farmers as your experimental group.

10. Random Sampling Technique • Random Sample • This type of sample should be used when • You want to publish in a peer-reviewed journal • You want to generalize to the population • Every unit (field or farmer) in your population has an equal probability of being selected for your study.

11. Example - Random Sampling Technique • Random Sample • Make a list of all the farmers in your county who plant this crop. • Assign a number to each farmer • Place numbers in a bin/hat and blindly draw out the number of farmers you need for your study.

12. Selecting a Data Collection Technique • Qualitative • Case Studies • Gather detailed information from one or a small group of individuals. • Intensive Interviews/Focus Groups • In-depth Understanding of Subjects • Disadvantage – Bias of interviewer can impact interpretation of results. • Participant Observation • Watch ongoing process • Disadvantage – Hawthorne Effect – People act differently when they know they are being observed • Content Analysis • Study materials objects (e.g., content of fields themselves)

13. Examples of Data Collection Technique s • Qualitative • Case Studies - Tell Farmer Brown’s story about his experience with fertilizer. • Intensive Interviews/Focus Groups • Sit down with individual farmers and asked open-ended questions, or sit down with a group of farmers and “focus” the open-ended questions on fertilizer and crop yield. • Participant Observation • Spend a summer as a farmer who uses fertilizer and live amongst farmers who use fertilizer. • Content Analysis • Study the content of fields that have been fertilized – measure amount of grain/hay grown per square inch etc.

14. Selecting a Data Collection Technique • Quantitative • Laboratory Experiment • Study experimental and control groups in a laboratory situation • Field Trials • Structure a experiment out in the field/community • Surveys • Construct questionnaires and mail/read to farmers. • Secondary • Use information collected by someone else

15. Examples of Data Collection Techniques • Quantitative • Laboratory Experiment • Plant small plots of land in a laboratory. Half of them would be fertilized and the other half would not. • Field Trials • Select farms that are fertilized and compare those to ones that are not. • Surveys • Use survey questions to ask farmers how satisfied they are with fertilizer and to report how much it has improved their crops. • Secondary • Find old records that contain information about fertilizers and crop yield

16. Experimental Designs • Field Trials • Data is collect literally “out in the field” or the community • Laboratory Experiment • Data is collected in a laboratory setting • Different Types of Experiments • One shot post-test • One group pre- and post-test • Classical Experimental design • Experimental and Control Groups – Pre- and Post-tests

17. One Group Pre and Post Tests Year 1 and 2 are crop yields before fertilizer; year 3 is crop yield after fertilizer

19. Surveys • Surveys • Self reported attitudes and behavior • Develop survey instrument (see power point on the development on survey instrument – have link here) • Mail out survey • Face to face survey • On-line survey • Telephone survey • Collect data • Analyze data

20. Data Analysis • Descriptive Statistics • Mean • Median • Mode • Standard Deviation • Inferential Statistics (Relationships) • Statistical Significance • Chi Square

21. Data Presentation • Should Include the Following: • Introduction • Literature Review • Methods Section • Results • Discussion – Summary and Conclusion • References/Appendix

22. Questions or Comments, Contact: • Dr. Carol Albrecht • Assessment Specialist USU Ext • carol.albrecht@usu.edu • (979) 777-2421