The Social Scientific Research Process: The Mythbusters’ TM Myths?

1. The Social Scientific Research Process: The Mythbusters’TM Myths? PO 390W: Research Design for Political Science

2. MythbustersTM as a Pedagogical Tool • Why am I showing Mythbusters today? • Highlights the utility of the scientific method to explaining (even far-fetched or everyday) claims about empirical reality • Allows us to illustrate the important components and nuances of the scientific process • Will allow us to comment on the soundness of the Mythbusters’ methods • The show is freakin’ awesome

3. The Research Question: Is Yawning Contagious? • A question about real world phenomena that is: • Specific (not overly general) • Interesting (should not be underestimated) • Testable through scientific inquiry

4. The Hypothesis • Different types of research: • Exploratory/Descriptive: Basic observation • Hypothesis testing: Explanatory (the current question) • “If Subject A is exposed to a yawning stimulus by Subject B, Subject A will be more likely to subsequently yawn than if not exposed to a yawning stimulus”

5. Units of Analysis and Variables • Unit of Analysis – The discrete individual/event which is to be analyzed (the “stimulus” yawn and the “resultant” yawn) • Variables: • Explanatory: All variables part of some specified relationship • Dependent (Y): Variable that the researcher is interested in explaining/predicting (“resultant” yawn) • Independent (X): Variables that do the influencing and explaining/predicting (“stimulus” yawn) • Extraneous: Not part of the explanatory set, but perhaps quite important in determining outcomes and can complicate analysis • Antecedent: Occurs at time prior to Y or X (fatigue, oxygen deprivation) • Intervening: A variable that is influenced by X, and in turn influences Y (participants’ knowledge of the experiment’s purpose; missing psychological variable?) • *Control can be exercised to mitigate the influence of extraneous variables (more on this later)

6. Operationalization of Variables • Operationalization refers to the ways in which researchers specify or measure the value or category of a variable in each observation • Pretty straightforward for “yawning” (dropping of jaw, inhalation, exhalation), but: • How well do they operationalize extraneous variables? • Can be difficult in other settings (How can we operationalize war? Terrorism? Democracy? Technology? • While often imperfect, operational definitions must be logical, attainable, and acceptable to standards of the scientific community

7. Data Collection • As we will explore much more thoroughly later in the semester, data for hypothesis testing is collected and analyzed in several ways: • Experimentation (used here) • Field research • Survey research. • Also, existing data can be analyzed • Essentially, experimentation divides the subjects/observations into two groups: • The control group consists of observations in which subjects are not exposed to a stimulus (a certain value of the independent variable) • The experimental (or stimulus) group is exposed to the exact same conditions as the control group, with the sole exception of the introduction of the stimulus • In the yawning experiment, what are the control and experimental groups? How well is control accomplished?

8. Data Analysis • Once collected (or as collected), data are analyzed by observing the impact of actual or artificial variations in the independent variable • The ideal strategy of varying the independent variable and observing the impact of that variation on the dependent variable, while controlling for extraneous variables, is basically the same for all methods • To determine the validity of the hypothesis, a host of analytical methods can be used. Two of the most widely used are: • Quantitative Analysis: In general, mathematically analyzing the probability of Y given X (simple form used here) • Qualitative Analysis: Using outcomes of a few cases to determine wider applicability of hypothesis

9. Inference and Conclusions • The Mythbusters’ results – that, in the second experiment, 29% of the experimental group yawned, as compared to 25% of the control group – lead them to confirm the “myth” that yawning is contagious • In essence, their experiments lead them to accept the hypothesis laid out previously; or, technically, they lead them to reject the “null” hypothesis that stimulus yawns have no contagious effect (more on this later) • The act of generally applying the results of an experiment or other analysis of a sample to a broader population is known as scientific inference

10. How’d the Mythbusters Do? WHAT THEY DID WELL… • In general, though they didn’t fully lay it out, their concepts, hypothesis, and explanatory variables are well-defined. • Concepts are clear; explanatory variables are clearly operationalized and observable; hypothesis is relational, directional, and logical • They attempt to control for extraneous variables. • In fact, when they feel that control is insufficient because people suspect their intentions, they reformulate the experiment • They do a pretty good job of introducing (artificially varying) the independent variable in a systematic fashion.

11. How’d the Mythbusters Do? … WHAT THEY COULD HAVE DONE BETTER… • They have no theory of contagion. • This is extremely problematic because, absent any supportable propositions about the mechanism by which yawning is contagious, we are left to wonder if there’s any reason to believe that their hypothesis is valid • In other words, theoretical weakness can lead to the conclusion that their findings are spurious (that there is no causal relationship) or an artifact (that the association of the two variables is a by-product of another condition or association) • They really need to control for even more extraneous variables. • Was it the stimulus or the possibility that certain subjects were tired or out of breath that led them to yawn? How can this be controlled for in a systematic way?

12. How’d the Mythbusters Do? … WHAT THEY COULD HAVE DONE BETTER (cont’d)… • They have no set standardsfor accepting or rejecting the hypothesis WITHIN the context of the experiment. • When Adam says that the 29%-25% difference is significant, he has no leg to stand on; he is simply speculating that that difference is significant, either actually or statistically • Thus, because of a flaw in their research design, one cannot even say if their experiment enjoys internal validity (i.e., that their inferences for the sample they’re testing are actually valid) • There is great difficulty in accepting or rejecting the hypothesis outside the context of the experiment. • Their inference of the results to the general “myth” is problematic. They do try to get a representative sample; but is it truly representative? Moreover, is fifty people enough to infer to a broader population? • Thus, because of a flaw in their research design, one cannot say if their experiment enjoys external validity (i.e., that their inferences for the global population are actually valid)

13. How’d the Mythbusters Do? …AND A PROBLEM THEY MIGHT BE UNABLE TO OVERCOME. • It is very difficult to determine whether or not yawning in Person A ACTUALLY CAUSES yawning in Person B. • There is rarely – if ever – any way to determine whether causation truly exists

14. Conclusion • Since causation is almost never observable, perhaps the best we can do is develop reasonable theories about causal associations, and test them by providing as much evidence as possible regarding the direction of influence and doing what we can to eliminate spuriousness • The Mythbusters example shows that, since flaws in theory and research design can prevent us from accomplishing even basic goals, careful and systematic construction of theory and research design is essential to meaningful scientific inquiry