Experimental Design:
This presentation is the property of its rightful owner.
Sponsored Links
1 / 35

Experimental Design: Threats to Validity PowerPoint PPT Presentation


  • 43 Views
  • Uploaded on
  • Presentation posted in: General

Experimental Design: Threats to Validity. EXPERIMENTS: The independent variable is manipulated to determine its effect on the dependent variable(s) whilst holding all other potential influences on the dependent variable(s) constant .

Download Presentation

Experimental Design: Threats to Validity

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Experimental design threats to validity

Experimental Design:

Threats toValidity


Experimental design threats to validity

EXPERIMENTS:

The independent variable is manipulated todetermine its effect on the dependent variable(s)whilst holding all other potential influences onthe dependent variable(s) constant


Experimental design threats to validity

Does the design allow us to answer thequestion, or might there be alternativeexplanations for the results obtained?

VALIDITY OF A STUDY

INTERNAL VALIDITY:

To what extent does the design of the studyallow us to attribute changes in the dependentvariable(s) to the effects of changes in theindependent variable?


Experimental design threats to validity

THREATS TO INTERNAL VALIDITY

Changes in the dependent variable may be dueto a variety of extraneous factors (confoundingor potential independent variables), rather thanto the manipulation of the independent variable


Experimental design threats to validity

How meaningful are the results whenapplied to the real world?

VALIDITY OF A STUDY

EXTERNAL (ecological) VALIDITY:

To what extent does the design of the studyallow us to generalise the results to populationsother than that from which the sample was drawn,or to similar populations in different settings orat different times?


Experimental design threats to validity

THREATS TO INTERNAL VALIDITY

Threats relating to…

The passage of time

Selection of participantsTesting and manipulations


Experimental design threats to validity

Example 1: Teaching styles

THREATS RELATING TO THE PASSAGE OF TIME

MATURATIONChanges within the participants due to thepassage of time (developmental, ageing,hunger, fatigue etc.)


Experimental design threats to validity

Example 2: Veggie study

THREATS RELATING TO THE PASSAGE OF TIME

HISTORYEvents occurring between pre- and post-test in addition to the experimental manipulation


Experimental design threats to validity

Example 3: Foul play study

THREATS RELATING TO THE PASSAGE OF TIME

INSTRUMENTATIONChanges in the way the dependent variableis measured (measurement errors,different tests, calibration problems, etc)


Experimental design threats to validity

54321

1 2 3 4 5

Unsupervised

Intentions

Supervised

1 2 3 4 5

Example 4: Exercise adherence

Pre-test

Post-test

1 2 3 4 5


Experimental design threats to validity

54321

1 2 3 4 5

Unsupervised

Intentions

Supervised

1 2 3 4 5

Example 4: Exercise adherence

Pre-test

Post-test

1 2 3 4 5

MORTALITY

Differentialdrop-out from groups betweenpre- and post-test


Experimental design threats to validity

RANDOMISED PRE-TEST POST-TESTCONTROL GROUP DESIGN

RO1XO2

RO3O4

Controls for threats to internal validity due tothe passage of time provided that:

Randomisation works (groups equated at pre-test)

There is no differential mortality between groups


Experimental design threats to validity

RANDOMISED PRE-TEST POST-TESTCONTROL GROUP DESIGN

RO1XO2

RO3O4

Threats relating to the passage of timeare controlled because they shouldmanifest themselves equally in each group


Experimental design threats to validity

Dependent Variable

Pre-test

Post-test

RO1XO2

RO3O4


Experimental design threats to validity

RO1XO2

RO3O4

Dependent Variable

Pre-test

Post-test


Experimental design threats to validity

THREATS RELATING TO SELECTIONOF PARTICIPANTS

SELECTIONBias resulting from differential selection

when assigning participants to groups


Experimental design threats to validity

Recovery time

Steroid

Shoulder

ACL

Pre-operative anxiety and recovery from anaesthesia.

A study was conducted to determine whether a brief pre-operative relaxationprocedure could reduce recovery time in surgical patients with sports injuries.Athletes about to undergo three different types of surgery were assigned toone of three conditions. Those about to receive a steroid injection for frozenshoulder were assigned to a relaxation condition in which they listened to a30 min. relaxation tape immediately prior to transfer to the operating theatre.Athletes about to undergo repair of ruptured anterior cruciate ligament wereassigned to a placebo condition in which they listened to a 30 min. tape-recordedshort story. Athletes about to undergo reduction of dislocated shoulder actedas no-treatment controls.


Experimental design threats to validity

Dissociation and endurance performance (Morgan et al., 1983).

A study was performed to evaluate whether a mental dissociation strategy couldfacilitate endurance performance. 27 navy personnel underwent a submaximalexercise test to determine VO2 max. Participants returned to the lab 48hours later and ran to exhaustionon a treadmill at an exercise intensity of80% VO2max. Time to exhaustion was recorded. Participants returned again48 hours later and were randomly assigned to a treatment group or a controlgroup. The treatment group were instructed in how to employ a dissociationstrategy whilst running. Time to exhaustion at 80% VO2max was assessed again.

Results showed that significantly more of the dissociation participantsincreased their endurance time between trials two and three compared tocontrol participants (78% of dissociation participants versus 31% of controlparticipants; p < .05). This suggests that the dissociation strategy facilitatesendurance performance.


Experimental design threats to validity

Dissociation and endurance performance (Morgan et al., 1983).

A study was performed to evaluate whether a mental dissociation strategy couldfacilitate endurance performance. 27 navy personnel underwent a submaximalexercise test to determine VO2 max. Participants returned to the lab 48hours later and ran to exhaustionon a treadmill at an exercise intensity of80% VO2max. Time to exhaustion was recorded. Participants returned again48 hours later and were randomly assigned to a treatment group or a controlgroup. The treatment group were instructed in how to employ a dissociationstrategy whilst running. Time to exhaustion at 80% VO2max was assessed again.

Results showed that significantly more of the dissociation participantsincreased their endurance time between trials two and three compared tocontrol participants (78% of dissociation participants versus 31% of controlparticipants; p < .05). This suggests that the dissociation strategy facilitatesendurance performance.


Experimental design threats to validity

THREATS RELATING TO SELECTIONOF PARTICIPANTS

SELECTION X MANIPULATION Effects of the manipulation only hold forthe particular population sampled

A threat to external validity


Experimental design threats to validity

Attitudes (mean)

Control

Treatment

Pre-test

Post-test

Example 5: Drugs intervention


Experimental design threats to validity

THREATS RELATING TO TESTINGAND MANIPULATIONS

EFFECTS OF TESTINGThe effects of taking the pre-test on scoreson the post-test

Post-test scores may be affected by practice,memory etc.

Pre-test sensitisation


Experimental design threats to validity

Example 5: Drugs intervention

Attitudes (mean)

Pre-test

Post-test


Experimental design threats to validity

THREATS RELATING TO TESTINGAND MANIPULATIONS

REACTIVE EFFECTS OF TESTINGTesting X manipulation interactionThreat to external validity

Pre-testing changes the participants’ responsesto the manipulation. The results obtained frompre-tested participants may be unrepresentativeof results that would be obtained from thosewho only received the manipulation


Experimental design threats to validity

REACTIVE ARRANGEMENTS

The experimental setting influences participantsto respond differently to how they would respondin a real-life setting

Research settings are unnatural situationsso we might expect participants to behaveunnaturally


Experimental design threats to validity

Dissociation and endurance performance(Morgan et al., 1983).

Long distance runners in Tibet have used thistechnique to run 300 miles non-stop in 30 hoursand many successful marathoners are known to use similar techniques… By dissociating in thisway you will not feel the same amount of fatigue,pain or discomfort you experienced during the lasttrial…


Experimental design threats to validity

REACTIVE ARRANGEMENTS

DEMAND CHARACTERISTICS (Orne, 1962)

“… the totality of cues which convey an experimental hypothesis to the subject.”


Experimental design threats to validity

Dissociation and endurance performance(Morgan et al., 1983).

Long distance runners in Tibet have used thistechnique to run 300 miles non-stop in 30 hoursand many successful marathoners are known to use similar techniques… By dissociating in thisway you will not feel the same amount of fatigue,pain or discomfort you experienced during the lasttrial… Please go as long as you can in order to makeour experiment a success.


Experimental design threats to validity

NON-SPECIFIC TREATMENT EFFECTS

Placebo/attention effectsHawthorne effects

Expectations of benefitCredibility of the treatment


Experimental design threats to validity

Rosenthal’s rats (1963)

EXPERIMENTER EFFECTS

The Pygmalion effect

Rosenthal & Jacobson (1968)


Experimental design threats to validity

RANDOMISED PRE-TEST POST-TESTCONTROL GROUP DESIGN

RO1XO2

RO3O4

Controls for threats relating to the passage of timeprovided that:

Randomisation works (groups equated at pre-test)

There is no differential mortality between groups

All potential independent variables includingnon-specific treatment effects are held constant


Experimental design threats to validity

RO1XO2

RO3O4

Threats relating to selection and testingare controlled because participants arerandomly assigned to groups

EXCEPT:

Because there is a pre-test, the reactiveeffects of testing are not controlled


Experimental design threats to validity

Alsocontrols for threats to internal validityprovided that:

Randomisation works (groups equated at pre-test)

There is no differential mortality between groups

All potential independent variablesincludingnon-specific treatment effectsare held constant

POST-TEST ONLY CONTROL GROUP DESIGN

RXO1

RO2


Experimental design threats to validity

RXO1

RO2

PLUS:This design also controls for reactive effectsof testing because there is no pre-test

BUT:We don’t know if the groups were equated at pre-test


Experimental design threats to validity

EXTERNAL VALIDITY

With the exception of the post-test only design’scontrol for reactive effects of testing, logically, wecannot control for threats to external validity

In an internally valid study we can onlydemonstrate that the effects (of X) hold underthe specific conditions of the study

e.g. The effects of X hold for(pre-tested)participants from theparticular populationsampled at thispoint in timein this particularplace … etc.


  • Login