Research Design and the Logic of Control. What Do We Mean by “Control”.
Research Design and the Logic of Control
During the 2010 congressional elections and even now, Republicans repeatedly asserted that the Bush/Obama fiscal stimulus did not work. Rather, it just built up trillions in federal debt that future generations would have to pay off.
For example, our newly elected local congressman, Bill Flores, advertised then and recently said that joblessness increased by over seven million during the time the stimulus was occurring. What we need is to allow the private sector to work by getting government out of the economy.
While these arguments undoubtedly paid political capital and may have sounded good to those inclined to vote for him, the science behind these arguments is problematic. Why?
There was no control. In order to be able to conclude anything at all about the success of the stimulus, one would need to know what happened to a group which did NOT receive the stimulus. We need a both a treatment and a control in order to make causal explanations.
The political world is a very complicated place. For every explanation there are often competing explanations. Our job is to determine which of the competing explanations is true.
In the case of the Bush/Obama economic stimulus it will probably not be possible to flesh out the truth. Why?
There was no control group which did not receive the stimulus. It was applied uniformly to the entire nation. It would have been unethical to withhold the stimulus because of the economic implications for those who did not receive it.
Similarly, there are often rival explanations for most things we want to explain.
Our ability to rule out competing explanations depends on the power of our research design.
We can approach this problem by designing an experimental study. An experimental study typically contains a treatment group and a control group. The experimental subjects are similar in every way, except that the treatment group receives the stimulus and the control group does not. This isolation of only one group receiving the stimulus allows us to make causal explanations.
We can also approach this problem by designing an observational study. Here the researcher makes controlled comparisons. That is, the researcher observes the effect of the independent variable of interest on the dependent variable, while holding constant all other plausible causes of the dependent variable.
In all experiments, the investigator manipulates a treatment group and a control group in such a way that, in the beginning, the two groups are virtually identical in every way.
Measurement is taken prior to the application of a stimulus.
The two groups then receive different values of the independent variable of interest. Typically, the treatment group receives the stimulus, while the control group does not.
Measurement is then taken of both groups after the application of the stimulus.
Since the two groups are identical in every way, except in their receipt of the stimulus, any observed differences in the dependent variable cannot be attributed to rival explanations.
The participants are read identical candidate profiles for the three. Each is well-educated, has a legal background, and is very successful.
Participants are randomly assigned to three groups. Each of the three groups is shown a different picture at the same time they see the candidate profiles.
Participants in each group are then asked which candidate they prefer in the upcoming Mexican election to represent their district.
A significant difference is observed between groups in preferences for the three candidates. The candidate who appears Spanish is preferred over the candidate who appears Indian, who is in turn preferred over the candidate who appears African.
What can we conclude? Why?
What is the role of “control” in allowing us to make these conclusions?
In order to make conclusions we need good random assignment. In other words, the three groups need to be identical in every way.
Random assignment is the great equalizer of rival explanations.
You have won a free
Ten week subscription to The Washington Post/Washington Times!
We have held a drawing to award free ten-week subscriptions of The Washington Post to households in Prince William County. Delivery begins this week. Delivery will automatically end after ten weeks, you do not need to call to cancel. However, if you want to cancel before the end of the ten weeks, please call 1-800-635-9224 and we will remove you from this promotion. Thank you for trying out the newspaper.
Group A was selected to receive a free newspaper subscription to the Washington Post (a liberal media outlet).
Group B was selected to receive a free newspaper subscription to the Washington Times (a conservative media outlet).
Group C was a control group which received no free newspaper subscription.
A public opinion survey was administered to all subjects after the 2005 Virginia gubernatorial election.
Those receiving the subscription to the Washington Post were eight percentage points more likely to have voted for the Democrat than those in the control group. A subscription to the Washington Times produced no change in voting behavior relative to the control group.
Did this field experiment have internal and external validity? What, if any, are the limitations?
Experiments are the “gold standard” for research in political science.
However, many research questions are not suited for conducting experiments. As political scientists we study concepts as we find them naturally in society. We generally cannot manipulate variables such as people’s party id, their relative liberalism, a state’s level of economic development, a nation’s institutional design, people’s gender, or education levels. The list is long of the things we would find difficult to manipulate in an experiment, but which are deemed important political variables.
Hence, we must often rely on observational studies. An observational study is one in which we make controlled comparisons of data where we find it.
In some cases we use surveys based on random selection of respondents from a population. Then, we compare groups in this randomly selected sample.
However, if randomization is not complete, there may be factors which “creep into” our observational studies which can affect the outcome.
One potential problem is generally labeled “selection bias.” Selection bias occurs when subjects find their way into the treatment group based on some systematic factor relating to the dependent variable.
It was widely predicted in 1948 that Thomas Dewey would win the 1948 presidential election over Harry Truman. These predictions were based on telephone surveys of respondents voting intentions prior to the election. Why were they wrong?
Suppose we conduct an observational study of how a person’s income affects their propensity to vote. However, our sample contains only respondents from neighborhoods where people could be safely surveyed. As a result, we have too few low income respondents in our sample. What is the result of this procedure?
Wood and Vedlitz (2007) conducted a survey experiment concerning the determinants of people’s views on global climate change.
At the start of the survey, they asked people how concerned they were about various issues facing the nation. The issues included global terrorism, global climate change, the economy, discrimination, deteriorating moral values, etc. At this stage people had no idea that the researchers’ primary interest was their level of concern about global climate change.
About 30 minutes into the survey after questioning people about various issues, people were randomly selected to receive three different scenarios.
The three groups randomly received the scenarios: Suppose I told you that 70/50/30 percent of Americans believe that global climate change is a serious problem.
Respondents were then asked how concerned they are about the issue of global climate change.
Comparing the scenario responses to the pre-measured level of concern, those receiving the high treatment (70) were more concerned about global climate change than those in the mid-treatment (50), who were more concerned than those in the low treatment (30).
The authors claimed that this difference suggested the importance of social pressure to people’s level of concern.
This brings us back to our earlier discussion of causality, spuriousness, mediating relationships, and interactions.
Here, Gender affects Partisan ID and Gender affects Gun control opinion. If the relation between X and Y disappears when Z enters the relationship, then we say the relation between X and Y is spurious.
One way of seeing a spurious relationship is simply to construct a graph of the two groups. Above we can see that Republican and Democrat women are about the same in their opinions about gun control. Similarly, men Republicans and Democats are also similar. Therefore, the relationship between party id and opinions on gun control is spurious, fully determined by gender.
However, suppose that Gender does not fully explain both partisan id and opinion on gun control. Then we can say that Gender both directly and indirectly affects opinion on gun control. Gender affects gun control opinion directly (Z to Y). It also affects gun control opinion indirectly (Z to X to Y). The two effects are called additive.
Here is the same line chart as before, but showing that partisanship affects gun control opinion. Note that the two lines are parallel. However, this need not be the case. The relationship can also be “interactive.”
Imagine a scenario in which support for gun control is such that women do not differ by partisanship, but that men do differ by partisanship. We call such a relationship an “interactive” relationship.
In other words, gender affects support for gun control, but partisanship only affects it for men.