PRR 475 Review - Tips. Powerpoint Presentations – widgets too Topic Handouts Sample Questions Other Handouts e.g. Statistics Exercises from Lab Trochim chapters Other texts, websites. Research/Evaluation Concepts – evaluation.ppt Proposals & Research Design – propdesb.ppt
Research/Evaluation Concepts – evaluation.ppt
Proposals & Research Design –propdesb.ppt
Survey methods –survey99.ppt
Basic Statistics/Data analysis - handouts
Observ., qualitative, other methods –QLQN.ppt, econevaluation.ppt
Communication & Ethics researchwriting.ppt, ethics.ppt
Body of Knowledge
Method of Inquiry
1. Problem Statement - define program to be evaluated/problem to be studied, users & uses of results. Justify importance of the problem/study.
2. Objectives : Concise listing . In evaluation studies, the objectives usually focus on the key elements of program to be evaluated & the evaluation criteria. These are the study objectives NOT the program objectives.
3. Background/Literature Review - place for more extensive history/structure of program. Focus on aspects most relevant to proposed evaluation. Discuss previous studies or the relevant methods.
4. Methods - details on procedures for achieving objectives - data gathering and analysis, population, sampling, measures, etc. Who will do what to whom, when, where, how and why?
5. Attachments - budget, timeline, measurement instruments, etc.
NOTE: Most “programs” must be narrowed to specific components to be evaluated. Think of a “Program of studies” rather than a single evaluation study. The proposal should define this specific study & how it fits into a broader program of studies.
1. Estimate benefits and costs of program
2. Estimate economic impacts of program on local community (social, environmental, fiscal).
3. Determine effects of program on target population.
4. Describe users and non-users of program
5. Assess community recreation needs, preferences
6. Determine market/financial feasibility of program
7. Evaluate adequacy or performance of program
Definition & Measurement
“measurement is the beginning of science, … until you can measure something, your knowledge is meager and unsatisfactory” Lord Kelvin
Nominal/Conceptual Definition - define concept in terms of other concepts, links concepts without tying them to real world
Operational definition - equates definition with measurement, specify procedures/operations to generate the concept.
Validity vs Reliability
1. Preliminary Info
Who are subjects
Method of communication
2. Question Content
3. Question Wording
4. Response Format
5. Question Sequencing/Layout
Demographic, Socioeconomic, Physical
Cognitive - Knowledge & beliefs
Affective - attitudes, feelings, preferences
Behavioral - actions
Research Designs/Data Collection Approaches
1. Describing a population - surveys
2. Describing users/visitors - on-site survey
3. Describing non-users, potential users or general population - household survey
4. Describing observable characteristics of visitors - on-site observation
5. Measuring impacts, cause-effect relationships - experiments
6. Anytime suitable secondary data exists - secondary data
7. Short, simple household studies - phone
8. Captive audience or very interested population - self-administered survey
9. Testing new ideas - experimentation or focus groups
10. In-depth study - in-depth personal interviews, focus groups, case studies
Qualitative vs Quantitative
open ended measures
interpretation of data
Qualitative vs Quantitative Approaches
Secondary data analysis
Secondary data analysis
Survey vs Experiment
Survey - measure things as they are, snapshot of population at one point in time, generally refers to questionnaires
(telephone, self-administered, personal interview)
Experiment - manipulate at least one variable (treatment) to evaluate response, to study cause-effect relationships
(field and lab experiments)
STEPS IN A SURVEY
1. Sample equivalent experimental and control groups
2. Isolate and control the treatment
3. Measure the effect
R MB1 X MA1 Experimental group
R MB2 MA2 Control group
R denotes random assignment to groups
X denotes the treatment
Measure of effect = Expmt gp - Control gp
= (MA1-MB1) - (MA2-MB2)
= with vs without
Key Results & Recommendations
Justify from study