Chapter 11 Enhancing Rigor in Quantitative Research

1. Chapter 11Enhancing Rigor in Quantitative Research

2. Types of Validity • Statistical conclusion validity • Internal validity • Construct validity • External validity

3. Controlling Intrinsic Source of Extraneous Variability • Randomization • Crossover • Homogeneity • Blocking/Stratification • Matching (pair matching) • Statistical control

4. Benefits and Limitations of Control Methods: Randomization Benefits: • Controls all extraneous variables • Does not require advance knowledge of which variables to control Limitations: • Ethical and practical constraints on manipulation • Possible artificiality of conditions

5. Benefits and Limitations of Control Methods: Crossover (Repeated Measures) Benefits: • If done with randomization, strongest possible approach • Reduces sample size requirements Limitations: • Cannot be used if there are possible carryover effects from one condition to another

6. Benefits and Limitations of Control Methods: Homogeneity Benefits: • Easy to achieve • Enhances interpretability of relationships Limitations: • Limits generalizability • Requires knowledge of which variables to control

7. Benefits and Limitations of Control Methods: Blocking Benefits: • Enhances interpretability of relationships • Offers possibility of examining blocking variable as an independent variable Limitations: • Manageable only with a few blocking variables • Requires knowledge of which variables to control

8. Benefits and Limitations of Control Methods: Matching Benefits: • Enhances interpretability of relationships • Easy to do if there is a large pool of available comparison group subjects Limitations: • Manageable only with a few matching variables • Requires knowledge of which variables to control • May be hard to find comparison group matches

9. Benefits and Limitations of Control Methods: Statistical Studies Benefits: • Enhances interpretability of relationships • Easy and economical • Can be used with a large number of extraneous variables Limitations: • Requires knowledge of which variables to control • Requires statistical sophistication

10. Threats to Statistical Conclusion Validity • Low statistical power • Low precision • Factors that undermine a strong operationalization of the independent variables • Inadequate participation in treatment conditions

11. Threats to Internal Validity • Temporal ambiguity • Selection • History • Maturation • Mortality/Attrition • Testing and Instrumentation

12. Research Designs and Threats to Internal Validity

13. Construct Validity • Concerns inferences from the particular exemplars of a study to the higher order constructs that they are intended to represent

14. Threats to Construct Validity • Reactivity to the study situations • Researcher expectancies • Novelty effects • Compensatory effects • Treatment diffusion or contamination

15. External Validity • Concerns inferences about the extent to which relationships observed in a study hold true over variations in people, conditions, and setting, as well as over variations in treatments and outcomes

16. Threats to External Validity • Interaction between relationship and people • Interaction between causal effects and treatment variation

17. Accessible vs. Target Population • Accessible population: The population available for a particular study • Target population: The total group of people in whom a researcher is interested and to whom results could be generalized

18. Threats to External Population • Inadequate sampling • Expectancy effects • Novelty effects • Interaction of history and treatment effects • Experimenter effects • Measurement effects