Drawing Samples in “Observational Studies”

2. Drawing Samples in “Observational Studies” • Sample vs. the Population • How to Draw a Random Sample • What Determines the “Margin of Error” of a Poll?

3. Sample vs. the Population - 1 • An observational study simply observes cases, WITHOUT attempting to impose a treatment and WITHOUT requiring any quasi- or natural experimental design. • Researchers can ask their cases questions in order to measure some variable. • Most of the time, researchers look closely at a small sample of the overall population.

4. Sample vs. the Population - 2 • A population is the ENTIRE group of cases about which you want information. • A sample is a SUBSET of the population which is used to gain information about the whole population. Population Sample

5. Sample vs. the Population - 3 • A parameter is a number describing a population. It is a usually a mystery. • A statistic is a number describing a sample. Statistics vary from sample to sample. • If our sample is representative of the population, sample statistics will closely approximate population parameters.

6. How to Draw a Random Sample - 1 • A simple random sample gives all members of the population an EQUAL chance to be drawn into the sample. • Draw names out of a hat, a really big hat • Label every case in the population with a number, then draw some random numbers • In a telephone poll, random digit dialing uses a random number generator to get even those with unlisted numbers.

7. How to Draw a Random Sample - 2 • WARNING! Random Sampling and Random Assignment are different! • Random sampling selects which cases in your population you will study. • Random assignment starts with a sample, then divides it into two or more groups. • Observational studies may use random sampling, but not random assignment, and this hurts their internal validity.

8. How to Draw a Random Sample - 3 • Samplingbias is consistent deviation of the sample statistic from the parameter • Sampling variability describes how far apart statistics are over many samples.

9. Nonrandom methods of drawing a sample (Note: These are Bad!) = 1 • A voluntary response sample includes the members of the population who voice their desire to be included in the sample. • 1936 “Literary Digest Poll” mailed 10 million ballots to magazine readers to volunteer participate in their Presidential election survey. 2 million surveys came back, predicting that FDR would lose 43%-57%. • FDR won, 61%-39%.

10. Nonrandom methods of drawing a sample (Note: These are Bad!) - 2 • A convenience sample studies the segment of the population that is easiest for the researcher to reach. • Polls only of people who have telephones. (Less of a problem than it used to be). • Studies by college students of their dormmates. • Internet polls at www.foxnews.com or www.sierraclub.org

11. Fundamentals of Significance Testing - 1 If you flip a “fair” (i.e., unbiased) coin 10 times there is LESS than a 25% chance you will obtain 5 heads and 5 tails. Put another way, there is approximately a 75% chance that you will get an UNEQUAL number of heads and tails (e.g., 7 and 3, 8 and 2, rather than 5 and 5).

12. Fundamentals of Significance Testing - 2 A “fair” (i.e., unbiased) coin will flip a DIFFERENT number of heads in repeated 10 flip trials. This is known as SAMPLING VARIATION (i.e., that repeated 10 flip samples of the same “fair” coin will NOT produce the same percentage of heads). Sampling variation raises this critical question:

13. Fundamentals of Significance Testing - 3 If we flip a coin 10 times and get an unequal number of head and tails did this occur because the coin was “biased” (not equally weighted – e.g., biased in favor of flipping heads) OR because of sampling variation (i.e., in the long run – say 10,000 flips - the coin would flip an equal number of heads and tails, it’s just in this 10 flip sample there was an unequal number of heads and tails)? This example is critical to the following discussion.

14. Fundamentals of Significance Testing - 4 In the coin flipping example, we will NEVER KNOW FOR SURE if the coin is UNBIASED. All we can do is ESTIMATE the PROBABILITY that it is UNBIANSED given the results that we have. Our “DECISION RULE” will be as follows: if the test results show that there is a 5%, or less chance, that the coin is unbiased, we will reject the notion that it is unbiased and conclude that it is biased.

15. Fundamentals of Significance Testing - 5 For example, if we find that the amount of time a person studies for this course (the independent variable), effects the grade they make in this course (the dependent variable), we need to ask the previous question applied to our circumstances: Is this because increased studying actually results in a higher grade, OR is the result due to sampling variation (i.e., on average, increased studying does not result in a higher grade, but did in our sample)?

16. Fundamentals of Significance Testing - 6 The remaining slides in this sequence are designed to help answer this critical question. Keep this discussion in mind as we proceed!

17. Confidence Intervals - 1 Do NOT worry about the “math” on the next group of slides. It is much more important that you understand the LOGIC of the text (i.e., the written words) than the mathematics. Immediately ahead we will construct “confidence intervals” for our statistical results. The following example should demonstrate the importance of a confidence interval.

18. Confidence Intervals - 2 Suppose we randomly survey 1,100 adult Americans and find that 52% think President Obama is doing a good job as president. The president’s job approval rating is important because it impact’s the president’s ability both to get his legislative program enacted and his party’s ability to retain both the presidency and congressional strength.

19. Confidence Intervals - 3 Thus, it is important to know how much confidence we can place in the 52% approval rating Obama had in the poll. For example, how likely would the result from our SAMPLE be that Obama had a 52% approval rating when his ACTUAL approval rating among all adult Americans was 46%? In a two candidate race, you can’t win with 46% of the vote! The following slides show us how to answer such a question.

20. Confidence Intervals - 4 What are the two principles of any test of statistical significance? See if you can apply these principles to the formulas on the following slide.

21. Confidence Intervals – 5 – DON’T WRITE THE FORMULA! • The 95% confidence interval around a sample proportion is: And the 99.7% confidence interval would be:

22. Confidence Intervals – 6 – DON’T WRITE THE FORMULA! Some years ago, in a poll of 505 likely voters, the Field Poll found 55% support for the recall of Gov. Davis

23. Confidence Intervals – 7 • The margin of error for this poll was plus or minus 4.4 percentage points. • This means that if we took many samples using the Field Poll’s methods, 95% of the samples would yield a statistic within plus or minus 4.4 percentage points of the true population parameter.