Goals of Sampling. Maximize External ValidityThe extent to which the results of a study generalize to the population of interestTo be confident about such a generalization, the sample must be representative of the population of interest.External Validity is NOT Everything.. In Defense of External
2. Goals of Sampling Maximize External Validity
The extent to which the results of a study generalize to the population of interest
To be confident about such a generalization, the sample must be representative of the population of interest.
External Validity is NOT Everything.
3. In Defense of External Invalidity (Mook, 1983) Sometimes researchers DO NOT necessarily care if their findings generalize to “real-life behavior in the real world”
Sometimes the point is to test theoretically-derived hypotheses. Proving it wrong in a non-random sample is enough.
The task is to decide what kinds of conclusions that you want to draw from a particular research project.
4. Goals of Many Experiments Minimize Threats to Internal Validity
The random assignment of individuals to treatment conditions means that confounding variables are equally distributed across conditions
As such, confounding variables are unlikely to be responsible for observed differences between treatment conditions.
5. Terminology Population
Set of all people, objects, or events of interest to the researcher
A variable that divides the population into mutually exclusive segments (e.g., gender, SES, political group)
A single member of the population
A subset of the population used in research
6. Definitions Redux A population refers to “the aggregate of all of the cases that conform to some designated set of specifications” (Chein, 1981, p. 419)
The aggregate is the target of generalization.
A sampling frame is the list that identifies the elements of the population.
7. Two Ways to Get a Sample Probability Sampling: (a) Every element of the population has a known nonzero probability of being selected; (b) Random selection is used at some point in the process
Nonprobability Sampling: Something else.
Bottom Line: With nonprobability sampling it is NOT possible to estimate sampling errors. Moreover, judgments about external validity are rarely on firm ground.
8. Probability Sampling Any method of sampling that ensures that the elements in a population have a KNOWN and CERTAIN probability of being chosen. It is not the case that all elements MUST have the SAME probability.
Simple Random Sampling: All elements have the SAME probability of being selected.
Sampling Frame: List or Specification of the Population
9. Simple Random Sampling All elements have an equal probability of being selected.
Population size is N. Sample size is n.
If N = 1000 and n = 100 then the chance that any one element would be selected is .10.
10. A Word About Sample Size The precision of our estimates increases with sample size.
It is sample size and NOT the size of the population that plays the vital role in determining precision.
The function between sample size and precision is non-linear.
11. The Standard Error Decreases as Sample Size Increases (Example SD =10)
12. Another Example of Probability Sampling Stratified Random Sampling
Divide the population into strata and then take a simple random sample from each subgroup.
Sometimes called proportional random sampling
Here we can over-sample a group if more statistical precision is desired for that group. This is called disproportionate stratified random sampling.
13. Disproportionate Stratified Random Sampling We can over-sample particular groups. This way we can obtain more precise estimates for strata that are small relative to the total population.
Say there are 100 green people in State X which has 10,000 people (Green people are 1% of the population). If we really want to know what green people are thinking then we should over-sample them!
We must still draw from the entire population of green people at random!
14. Nonprobability Sampling This method does not involve random selection.
Let’s be Blunt: None of these methods are terribly good for supporting inferences about external validity.
However, sometimes you cannot use probability sampling or do not need to use probability sampling.
15. Types of Nonprobability Samples Accidental, Haphazard or Convenience Sampling
The "person on the street" interviews
College student samples
Clinical practice samples
16. Purposive Sampling One or more specific predefined groups being sought. (They are the purpose!)
Example: People in a mall with a clipboard
Modal Instance Sampling: Sampling the most frequent case, the "typical" case, or the “average” person
OK: What is the typical or modal person (e.g., the average voter)?
17. Quota Sampling Specify the minimum number of sampled units that you want in each category.
Not always concerned with having numbers that match the proportions in the population. Sometimes, you simply want be able to talk about even small groups in the population.
18. Snowball Sampling Begin by identifying someone who meets the criteria for inclusion in your study
Then ask them to recommend other potential participants who meet inclusion criteria.
Useful when you are trying to reach populations that are inaccessible or hard to find (e.g., homeless)
19. Experience Sampling
20. Sampling of Events In addition to sampling people, it is possible to sample Events or Experiences
Examples: Age differences in emotional experiences; Job satisfaction in the workplace; Motives for drinking
Often we use technology to help record thoughts, feelings, or behavior in the moment.
21. Sampling Strategies Sampling at Random
Signaling device beeps at random periods throughout the day.
Participants are asked to record thoughts, feelings, or behaviors.
Answer a short questionnaire after an event of interest occurs.
Timing of response is determined by the participant
22. Triggers other than Events Signaling
Use an electronic device to signal/remind the participants to complete a questionnaire
Timing Controlled by Researchers
End of day reporting about experiences and reactions
Subject to more biases due to memory
Does not require electronic devices
23. General Issues Brevity – Keep it Short
Frame of reference (“at this moment”)
Duration of the Study?
Common for 7 Days
14 Days is probably better
Possible for much longer intervals
24. Programs for conducting ESP ESP: The Experience Sampling Program
A free software package for conducting experiments by experience sampling
Runs on PDAs (Palm Pilots)
Asks questions of the participant and records the answers and the participant's response time.
The data may later be uploaded to a computer for analysis.
25. Experience sampling How do you feel right now?
Please rate each feeling on the scale given. A rating of 0 means that you are not experiencing that feeling at all. A rating of 6 means that this feeling is a very important part of the experience. Not at all Very Much Happy . . . . . . . . . . . . . . . 0 1 2 3 4 5 6 Frustrated/annoyed . . . . 0 1 2 3 4 5 6
26. The research team:
Daniel Kahneman, Princeton University
Alan Krueger, Princeton University
David Schkade, University of Texas
Norbert Schwarz, University of Michigan
Arthur Stone, Stony Brook University
27. Day Reconstruction Method
28. Sample 909 women who worked the previous day
49% were European American
24% were African American
22% were Latinas/Hispanic
5% were Other
Average Age: 38 years
Average Income: $54,700
29. Step 1: Construct a Short Diary of the Previous Day “Think of your day as a continuous series of scenes or episodes in a film. Give each episode a brief name. Write down the approximate times at which each episode began and ended.”
Average Number = 14.1 (SD = 4.8)
Average Duration = 61 minutes (Range 15 minutes to 120 minutes)
30. Step 2: Answer Structured Questions about Each Episode
What were you doing? (check all that apply)
__ commuting __ working
__ shopping __ preparing food
__ doing housework __ taking care of your children
__ eating __ pray/worship/meditate
__ socializing __ watching TV
__ nap/resting __ computer/internet/email
__ relaxing __ on the phone
__ intimate relations __ exercising
__ other (please specify________________)
33. How did you feel during this episode? Please rate each feeling on the scale given. A rating of 0 means that you did not experience that feeling at all. A rating of 6 means that this feeling was a very important part of the experience. Please circle the number between 0 and 6 that best describes how you felt.
Not at all Very much
Impatient for it to end . . . . . . . . . 0 1 2 3 4 5 6
Happy . . . . . . . . . . . . . . . . . . . . . . 0 1 2 3 4 5 6
Frustrated/annoyed . . . . . . . . . . . 0 1 2 3 4 5 6
Depressed/blue . . . . . . . . . . . . . . 0 1 2 3 4 5 6
Competent/capable . . . . . . . . . . . 0 1 2 3 4 5 6
Hassled/pushed around . . . . . . . 0 1 2 3 4 5 6
Warm/friendly . . . . . . . . . . . . . . . 0 1 2 3 4 5 6
Angry/hostile . . . . . . . . . . . . . . . . 0 1 2 3 4 5 6
Worried/anxious . . . . . . . . . . . . . 0 1 2 3 4 5 6
Enjoying myself . . . . . . . . . . . . . . 0 1 2 3 4 5 6
Criticized/put down . . . . . . . . . . . 0 1 2 3 4 5 6
Tired . . . . . . . . . . . . . . . . . . . . . . . 0 1 2 3 4 5 6
35. Affect Calculation Positive: Average of Enjoyment, Warm, Happy
Negative: Average of Frustrated, Worried, Depressed, Angry, Hassled, Criticized
36. Some Results - Activities
37. Some Results – Interaction Partners