Sampling and External Validity

Sampling and External Validity KNR 164

Populations and Samples • Census: all obtainable persons • Population: The entire group about which information is desired • Sample: A proportion or part of the population • usually the proportion from which information is gathered

Populations and Samples • In research we can rarely obtain data on census; and, it is very hard to obtain data on populations…..so; • We try to find representative samples so we can infer our findings to all person in a population • Statistics are sometimes called inferential statistics

Target Population • The participants to whom the answer to the question pertains. • The target population definition has two aspects: • Conceptual – who we wish to infer to • Operational – the limits we are bound

Sampling • In its broadest sense, sampling is a procedure by which one or more members of a population are picked from the population • The objective is to make certain observations upon the members of the sample and then, on the basis of these results, to draw conclusions about the characteristics of the entire population

Sampling Model Generalize back Population Draw sample Sample

Selecting a Sample • Haphazard Sample: Haphazard samples are constructed by arbitrarily selecting individual sample members • Random Sample: There are several methods for constructing random samples—we consider only simple random samples. • Simple Random Sample: This process operates so that each member of the population has an equal chance of being selected into the sample

Looking at the Process • When we randomly select a sample from a population, we can use the mean for the sample as an estimate or guess as to the value for the mean of the population. • This should bring up the question as to how good is this sample mean or sample statistic as a guess for the value of the population mean or population parameter. • The essence of this question has to do with how well this processworks—the process of using a sample to make guesses about the population.

How Good is a Sample Mean • The essential question is “How good is a sample mean as an estimate of the population mean?” • One way to examine this question is to understand that we used a process that involved randomly selecting a sample from the population and then calculating the mean for the values of the observations in the sample • We can repeat this process as many times as we wish and examine what it produces

156 121 149 105 201 Sample with n = 5 46 198 217 189 149 172 162 42 121 198 201 309 111 220 100 201 261 156 … 133 Population of weights Sample of 5 weights

Ten Different Samples, n = 5

Validity • Validity: the best available approximation of the truth of a given proposition, inference, or conclusion • Measures, samples, and designs lead to valid conclusions

Internal Validity vs. External Validity • Internal validity refers both: • to how well a study was run (research design, operational definitions used, how variables were measured, what was/wasn't measured, etc.), • andhow confidently one can conclude that the observed effect(s) were produced solely by the independent variable and not extraneous ones • In experimental research, internal validity answers the question, "Was it really the treatment that caused the difference between the subjects in the control and experimental groups?"

Internal Validity vs. External Validity • External validity • represents the extent to which a study's results can be generalized or applied to other people or settings • the approximate truth of conclusions that involve generalizations • the degree to which the conclusions in a study would hold for other persons in other places and at other times

External Validity and Sampling • Sampling • process of selecting units (e.g., people, organizations, occasions) from a population of interest so that by studying the sample you can fairly or reasonably generalize your results to the population from which the units were chosen • primary approach is referred to as the sampling model

Sampling Model • goal is to claim representative sampling • potential problems… • knowing the population you want to generalize to in the first place • ability to draw a representative sample • ability to sample across all times • we can never generalize with absolute certainty, rather we have to decide how reasonable is it to assume similar results give different circumstances

Threats to External Validity • Interaction of selection and treatment • Maybe it is just these people • Interaction of setting and treatment • Maybe it is just these places • Interaction of history and treatment • Maybe it is just these times

Population Random Sampling Sample Use theory Our Study Places Times People Replicate, Replicate, Replicate settings Our Study times people places Improving External Validity

Sampling and External Validity