Research Design in Clinical Psychology

1. Research Design in Clinical Psychology Lecture 2 Reliability, Validity, and Artifact/Bias (Chapters 2-4 in Kazdin)

2. Internal Validity • To what extent can the intervention, and not other factors, account for study results • History • Maturation • Testing (Practice) • Instrumentation • Statistical Regression • Selection Bias • Attrition • these can occur across all groups or only to select groups • Diffusion/Imitation of treatment • Special treatment/reaction of controls

3. External Validity • To what extent can the study results be generalized to other samples with different characteristics than the study sample

4. External Validity: Examples 1 • Sample characteristics • Differences b/n study sample and other samples • Include age, gender, culture, education • Stimulus characteristics and settings • Extension across characteristics of the study • Includes setting, experimenter, materials/apparatus • Ecological validity? • Reactivity of experimental arrangements • Awareness of being in a study may affect behavior • Are responses relevant to those who are not in a study • Multiple-treatment interference • Subjects receives multiple interventions • Relevant to those who did not receive other interventions?

5. External Validity: Examples 1 • Novelty • Is effect due to newness • Reactivity of assessment • Similar to experimental arrangements, but focuses on awareness of what the measures are tapping • Test sensitization • Does pre-testing or the test itself (in the case of post-test sensitization) alter subject experience and responses • Timing of Measurement • When assessments are given could alter results

6. Parsimony • Least complex explanations first • Frequently, a threat to internal validity is most parsimonious • When considering limitations in both internal and external validity, however, parsimony suggests that “findings are the best statement of a relationship, unless there are clear reasons to think otherwise.

7. Why internal validity precedes external validity • Can’t have ExtVal, without IntVal • It would be like asking, “gee can we generalize these results we have no confidence in to a wide variety of individuals?” • One can still have important findings that elucidate basic principles without much ExtVal. • A lack of generalization of a finding across samples may be very important and can span other research

8. Construct Validity • What is the causal agent and conceptual basis underlying an effect (what is the intervention and why did it lead to change?) • In clinical research methods, CV is different from in test construction where CV is the extent that a measure captures a construct of interest.

9. Threats to construct validity • Contact time • Placebo effects • Nonblind (single and double) designs • Single operations and narrow stimulus sampling • Is the effect due to the selected IV • Does other aspects of the intervention have an effect beyond the aspect identified by the experimenter • Expectancies • Cues and Demand characteristics • Others exist based on conceptual relevance

10. Statistical Conclusion ValidityAKA Data Evaluation Validity • Refers to the facets of the quantitative evaluation that influence the conclusions reached about experimental conditions and effects (to what extent are “real” effects demonstrated and interpreted) • Does one understand the stats used • Has one done stats correctly

11. Rejecting the Null Hypothesis I • Alpha = probability of rejecting the null when you shouldn’t (Type 1 error) • Saying groups are different when in reality are same • Beta = probability of accepting the null when you shouldn’t (Type 2 error) • Saying groups are same when in reality are different • Power = probability of correctly accepting or rejecting the null

12. Rejecting the Null Hypothesis II • Standard deviation = variability around mean • sqrt [(each observation – mean)2 / (N -1)] • SS/df • Effect size = Beyond significance, it is the magnitude of difference b/n groups • Expressed in terms of SD units • ES = (M1 – M2) / SD

13. Threats to statistical conclusion validity • Low power • Inability to detect real differences • Variability in the procedures • Subjects in same group get different treatment • Subject heterogeneity • Subjects in same group differ on potentially confounding variables • Unreliability of measures • Multiple comparisons/error rates • Experiment-wise error

14. Experimental Precision • Controlling vs holding constant • Tradeoffs

15. Sources of Artifacts and Bias

16. Rationale, scripts, & procedures • Imprecision in carrying out procedures • Experimenter may not have clear a protocol • Loose protocol effect • Experimenter may ignore protocol • Strategies for overcoming include • developing a clear protocol • actually testing the protocol out ahead of time • Get feedback from participants • Train experimenters together • Document all deviations

17. Experimenter Expectancy • Experimenters’ knowledge of study hypotheses may bias study implementation • Strategies for overcoming include • Keep experimenters naïve to purpose • Double blind procedures for group assignment • Success of these strategies can be empirically reviewed

18. Sample selection • Convenience samples and Volunteers • Do these sample differ in ways that affect the results or their generalizability • May select certain types of convenience or volunteer samples to answer particular questions • Attrition • Difference in # of dropouts across groups could have huge effect on results • Reminders and Commitment tactics (backloading $$$) • Select those unlikely to drop-out? • Plan ahead from past research

19. Other sources • Experimenter characteristics • Situational and context cues • Demand characteristics – clues on how to respond • Subject roles (pg 95 in book) • Data recording and analysis