Download
1 / 27
270 likes | 351 Views
Research Design and the Logic of Control. What Do We Mean by “Control”.
E N D
What Do We Mean by “Control” During the 2010 congressional elections and even now, Republicans repeatedly asserted that the Bush/Obama fiscal stimulus did not work. Rather, it just built up trillions in federal debt that future generations would have to pay off. For example, our newly elected local congressman, Bill Flores, advertised then and recently said that joblessness increased by over seven million during the time the stimulus was occurring. What we need is to allow the private sector to work by getting government out of the economy. While these arguments undoubtedly paid political capital and may have sounded good to those inclined to vote for him, the science behind these arguments is problematic. Why? There was no control. In order to be able to conclude anything at all about the success of the stimulus, one would need to know what happened to a group which did NOT receive the stimulus. We need a both a treatment and a control in order to make causal explanations.
Perhaps the stimulus did actually work, but that other factors were functioning simultaneously to make the economy worse. • Thus, as scientists we need to ask: • What would the economy have been like without the stimulus? • For those who did not receive the stimulus (an imaginary group), what would have happened. • This is the issue of control for rival explanations. • The issue of control is fundamental to being able to make causal explanations. Politicians and people live in a much simpler world and are often willing to accept explanations offered by those they trust, especially if they coincide with one’s predispositions. • However, as scientists it is our obligation to flesh out the truth whenever it is possible to do so, regardless of its implications for our political views.
A Complicated World The political world is a very complicated place. For every explanation there are often competing explanations. Our job is to determine which of the competing explanations is true. In the case of the Bush/Obama economic stimulus it will probably not be possible to flesh out the truth. Why? There was no control group which did not receive the stimulus. It was applied uniformly to the entire nation. It would have been unethical to withhold the stimulus because of the economic implications for those who did not receive it. Similarly, there are often rival explanations for most things we want to explain.
Consider our earlier discussion on concealed handguns on college campuses. • We hypothesized that partisanship was probably a good explanation for supporting this initiative. • However, there are other possible rival explanations which require “controlling for” if we are to be able to make causal statements. • Gender might explain this phenomenon. Women are more likely to be Democrats and Men are more likely to be Republicans. Further, Women are generally stronger gun control advocates than men. • These rival explanations undermine our ability to reach causal conclusions about how partisanship affects support for guns on campus. • Is the independent variable, partisanship, causing support/non-support for guns on campus. Or, is some other variable at work such as gender, distorting our results and leading us to erroneous conclusions.
Research Design for Control Our ability to rule out competing explanations depends on the power of our research design. We can approach this problem by designing an experimental study. An experimental study typically contains a treatment group and a control group. The experimental subjects are similar in every way, except that the treatment group receives the stimulus and the control group does not. This isolation of only one group receiving the stimulus allows us to make causal explanations. We can also approach this problem by designing an observational study. Here the researcher makes controlled comparisons. That is, the researcher observes the effect of the independent variable of interest on the dependent variable, while holding constant all other plausible causes of the dependent variable.
Experimental Studies In all experiments, the investigator manipulates a treatment group and a control group in such a way that, in the beginning, the two groups are virtually identical in every way. Measurement is taken prior to the application of a stimulus. The two groups then receive different values of the independent variable of interest. Typically, the treatment group receives the stimulus, while the control group does not. Measurement is then taken of both groups after the application of the stimulus. Since the two groups are identical in every way, except in their receipt of the stimulus, any observed differences in the dependent variable cannot be attributed to rival explanations.
Types of Experiments • In a laboratory experiment, the treatment group and control group are studied in an environment wholly created by the investigator. For example, we have an experimental lab in this building where students participate in experiments while sitting in front of a computer screen. Participants are generally aware that an experiment is taking place, but are often unaware of the purpose. In laboratory experiments subjects are recruited to a common location, the experiment is largely conducted at that location, and the researcher controls almost all aspects in the environment in that location except for subject behavior. • In a field experiment, the treatment and control group are studied in their natural environment, living their lives normally, probably unaware that an experiment is taking place. In a field experiment the researcher’s intervention takes place in an environment where the researcher has only limited control beyond the intervention conducted. • Both types of experiments generally depend on random assignment for control. Random assignment occurs when each prospective participant has an equal probability of being in the treatment and control groups.
Example of a Laboratory Experiment • Suppose we want to study the effect of candidates’ racial attributes on people’s political behavior. For example, I have a colleague who studied the effect of racial attributes in Mexican election behavior. • In Mexico there is a diversity of racial features among the population. There are people of Spanish origin who appear caucasian; there are people of Mulatto origin who are very dark; there are people of Indian origin whose features are less dark but have distinctly Mongol features. • The treatment is showing participants pictures of three candidates. • One picture depicts Candidate A with obvious Spanish features. • A second picture depicts Candidate B who is very dark, probably mulatto (mix between caucasian and African). • A third picture shows Candidate C who has clear Indian features, probably descended from mixed caucasion/Mayan or Aztec populations.
The participants are read identical candidate profiles for the three. Each is well-educated, has a legal background, and is very successful. Participants are randomly assigned to three groups. Each of the three groups is shown a different picture at the same time they see the candidate profiles. Participants in each group are then asked which candidate they prefer in the upcoming Mexican election to represent their district. A significant difference is observed between groups in preferences for the three candidates. The candidate who appears Spanish is preferred over the candidate who appears Indian, who is in turn preferred over the candidate who appears African. What can we conclude? Why? What is the role of “control” in allowing us to make these conclusions? In order to make conclusions we need good random assignment. In other words, the three groups need to be identical in every way. Random assignment is the great equalizer of rival explanations.
Internal and External Validity • Internal validity means that the experiment successfully controls for rival explanations that might cause the treatment response. • External validity refers to whether or not the results from an experiment are “generalizable” to the population. • Laboratory experiments generally have high internal validity, but low external validity. For example, the preceding experiment involved participants who knew they were participating in an experiment. • How might this have affected the results? • It is common for political scientists to use “college students” or “internet respondents” when conducting laboratory experiments. How might this affect the external validity of experiments?
An Example of a Field Experiment • Field experiments are conducted in the real world and can sometimes overcome the limitations of laboratory experiments. • Gerber, Karlan, and Bergen (2007) conducted a field experiment on how exposure to various media outlets affects voting behavior. They randomly selected households in Prince William County Virginia to receive treatments prior to the 2005 Virginia gubernatorial election.. • Postcard to Subjects Congratulations! You have won a free Ten week subscription to The Washington Post/Washington Times! We have held a drawing to award free ten-week subscriptions of The Washington Post to households in Prince William County. Delivery begins this week. Delivery will automatically end after ten weeks, you do not need to call to cancel. However, if you want to cancel before the end of the ten weeks, please call 1-800-635-9224 and we will remove you from this promotion. Thank you for trying out the newspaper.
Group A was selected to receive a free newspaper subscription to the Washington Post (a liberal media outlet). Group B was selected to receive a free newspaper subscription to the Washington Times (a conservative media outlet). Group C was a control group which received no free newspaper subscription. A public opinion survey was administered to all subjects after the 2005 Virginia gubernatorial election. Those receiving the subscription to the Washington Post were eight percentage points more likely to have voted for the Democrat than those in the control group. A subscription to the Washington Times produced no change in voting behavior relative to the control group. Did this field experiment have internal and external validity? What, if any, are the limitations?
Observational Studies Experiments are the “gold standard” for research in political science. However, many research questions are not suited for conducting experiments. As political scientists we study concepts as we find them naturally in society. We generally cannot manipulate variables such as people’s party id, their relative liberalism, a state’s level of economic development, a nation’s institutional design, people’s gender, or education levels. The list is long of the things we would find difficult to manipulate in an experiment, but which are deemed important political variables. Hence, we must often rely on observational studies. An observational study is one in which we make controlled comparisons of data where we find it. In some cases we use surveys based on random selection of respondents from a population. Then, we compare groups in this randomly selected sample.
However, if randomization is not complete, there may be factors which “creep into” our observational studies which can affect the outcome. One potential problem is generally labeled “selection bias.” Selection bias occurs when subjects find their way into the treatment group based on some systematic factor relating to the dependent variable. Examples: It was widely predicted in 1948 that Thomas Dewey would win the 1948 presidential election over Harry Truman. These predictions were based on telephone surveys of respondents voting intentions prior to the election. Why were they wrong? Suppose we conduct an observational study of how a person’s income affects their propensity to vote. However, our sample contains only respondents from neighborhoods where people could be safely surveyed. As a result, we have too few low income respondents in our sample. What is the result of this procedure?
Survey Experiments Wood and Vedlitz (2007) conducted a survey experiment concerning the determinants of people’s views on global climate change. At the start of the survey, they asked people how concerned they were about various issues facing the nation. The issues included global terrorism, global climate change, the economy, discrimination, deteriorating moral values, etc. At this stage people had no idea that the researchers’ primary interest was their level of concern about global climate change. About 30 minutes into the survey after questioning people about various issues, people were randomly selected to receive three different scenarios. The three groups randomly received the scenarios: Suppose I told you that 70/50/30 percent of Americans believe that global climate change is a serious problem.
Respondents were then asked how concerned they are about the issue of global climate change. Comparing the scenario responses to the pre-measured level of concern, those receiving the high treatment (70) were more concerned about global climate change than those in the mid-treatment (50), who were more concerned than those in the low treatment (30). The authors claimed that this difference suggested the importance of social pressure to people’s level of concern.
Controlled Comparisons • A controlled comparison is accomplished by examining the relationship between an independent and dependent variable, while holding constant other variables, especially those suggested by rival explanations. • Returning to our example of “support for guns on college campuses.” Our theory says that this support is based on a person’s party identification. However, a rival explanation is gender determines support. How could we evaluate which is correct? • We could look at support for guns on campus only among women, and only among men. If it differs between Republican and Democrat women, and it differs between Republican and Democrat men, then we would know that partisanship is an important explanation. • More generally, we can also accomplish the same thing statistically, rather than by splitting the sample into women and men. You will learn how to do this in future weeks. For now, it is enough to note the concept.
Controlling for Z: Three Scenarios This brings us back to our earlier discussion of causality, spuriousness, mediating relationships, and interactions.
Here, Gender affects Partisan ID and Gender affects Gun control opinion. If the relation between X and Y disappears when Z enters the relationship, then we say the relation between X and Y is spurious.
One way of seeing a spurious relationship is simply to construct a graph of the two groups. Above we can see that Republican and Democrat women are about the same in their opinions about gun control. Similarly, men Republicans and Democats are also similar. Therefore, the relationship between party id and opinions on gun control is spurious, fully determined by gender.
However, suppose that Gender does not fully explain both partisan id and opinion on gun control. Then we can say that Gender both directly and indirectly affects opinion on gun control. Gender affects gun control opinion directly (Z to Y). It also affects gun control opinion indirectly (Z to X to Y). The two effects are called additive.
Here is the same line chart as before, but showing that partisanship affects gun control opinion. Note that the two lines are parallel. However, this need not be the case. The relationship can also be “interactive.”
Imagine a scenario in which support for gun control is such that women do not differ by partisanship, but that men do differ by partisanship. We call such a relationship an “interactive” relationship. In other words, gender affects support for gun control, but partisanship only affects it for men.
