Experimental Design:
Download
1 / 35

Experimental Design: Threats to Validity - PowerPoint PPT Presentation


  • 92 Views
  • Uploaded on

Experimental Design: Threats to Validity. EXPERIMENTS: The independent variable is manipulated to determine its effect on the dependent variable(s) whilst holding all other potential influences on the dependent variable(s) constant .

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Experimental Design: Threats to Validity' - minh


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Experimental Design:

Threats toValidity


EXPERIMENTS:

The independent variable is manipulated todetermine its effect on the dependent variable(s)whilst holding all other potential influences onthe dependent variable(s) constant


Does the design allow us to answer thequestion, or might there be alternativeexplanations for the results obtained?

VALIDITY OF A STUDY

INTERNAL VALIDITY:

To what extent does the design of the studyallow us to attribute changes in the dependentvariable(s) to the effects of changes in theindependent variable?


THREATS TO INTERNAL VALIDITY

Changes in the dependent variable may be dueto a variety of extraneous factors (confoundingor potential independent variables), rather thanto the manipulation of the independent variable


How meaningful are the results whenapplied to the real world?

VALIDITY OF A STUDY

EXTERNAL (ecological) VALIDITY:

To what extent does the design of the studyallow us to generalise the results to populationsother than that from which the sample was drawn,or to similar populations in different settings orat different times?


THREATS TO INTERNAL VALIDITY

Threats relating to…

The passage of time

Selection of participantsTesting and manipulations


Example 1: Teaching styles

THREATS RELATING TO THE PASSAGE OF TIME

MATURATIONChanges within the participants due to thepassage of time (developmental, ageing,hunger, fatigue etc.)


Example 2: Veggie study

THREATS RELATING TO THE PASSAGE OF TIME

HISTORYEvents occurring between pre- and post-test in addition to the experimental manipulation


Example 3: Foul play study

THREATS RELATING TO THE PASSAGE OF TIME

INSTRUMENTATIONChanges in the way the dependent variableis measured (measurement errors,different tests, calibration problems, etc)


54321

1 2 3 4 5

Unsupervised

Intentions

Supervised

1 2 3 4 5

Example 4: Exercise adherence

Pre-test

Post-test

1 2 3 4 5


54321

1 2 3 4 5

Unsupervised

Intentions

Supervised

1 2 3 4 5

Example 4: Exercise adherence

Pre-test

Post-test

1 2 3 4 5

MORTALITY

Differentialdrop-out from groups betweenpre- and post-test


RANDOMISED PRE-TEST POST-TESTCONTROL GROUP DESIGN

R O1 X O2

R O3 O4

Controls for threats to internal validity due tothe passage of time provided that:

Randomisation works (groups equated at pre-test)

There is no differential mortality between groups


RANDOMISED PRE-TEST POST-TESTCONTROL GROUP DESIGN

R O1 X O2

R O3 O4

Threats relating to the passage of timeare controlled because they shouldmanifest themselves equally in each group


Dependent Variable

Pre-test

Post-test

R O1 X O2

R O3 O4


R O1 X O2

R O3 O4

Dependent Variable

Pre-test

Post-test


THREATS RELATING TO SELECTIONOF PARTICIPANTS

SELECTIONBias resulting from differential selection

when assigning participants to groups


Recovery time

Steroid

Shoulder

ACL

Pre-operative anxiety and recovery from anaesthesia.

A study was conducted to determine whether a brief pre-operative relaxationprocedure could reduce recovery time in surgical patients with sports injuries.Athletes about to undergo three different types of surgery were assigned toone of three conditions. Those about to receive a steroid injection for frozenshoulder were assigned to a relaxation condition in which they listened to a30 min. relaxation tape immediately prior to transfer to the operating theatre.Athletes about to undergo repair of ruptured anterior cruciate ligament wereassigned to a placebo condition in which they listened to a 30 min. tape-recordedshort story. Athletes about to undergo reduction of dislocated shoulder actedas no-treatment controls.


Dissociation and endurance performance (Morgan et al., 1983).

A study was performed to evaluate whether a mental dissociation strategy couldfacilitate endurance performance. 27 navy personnel underwent a submaximalexercise test to determine VO2 max. Participants returned to the lab 48hours later and ran to exhaustion on a treadmill at an exercise intensity of80% VO2max. Time to exhaustion was recorded. Participants returned again48 hours later and were randomly assigned to a treatment group or a controlgroup. The treatment group were instructed in how to employ a dissociationstrategy whilst running. Time to exhaustion at 80% VO2max was assessed again.

Results showed that significantly more of the dissociation participantsincreased their endurance time between trials two and three compared tocontrol participants (78% of dissociation participants versus 31% of controlparticipants; p < .05). This suggests that the dissociation strategy facilitatesendurance performance.


Dissociation and endurance performance (Morgan et al., 1983).

A study was performed to evaluate whether a mental dissociation strategy couldfacilitate endurance performance. 27 navy personnel underwent a submaximalexercise test to determine VO2 max. Participants returned to the lab 48hours later and ran to exhaustion on a treadmill at an exercise intensity of80% VO2max. Time to exhaustion was recorded. Participants returned again48 hours later and were randomly assigned to a treatment group or a controlgroup. The treatment group were instructed in how to employ a dissociationstrategy whilst running. Time to exhaustion at 80% VO2max was assessed again.

Results showed that significantly more of the dissociation participantsincreased their endurance time between trials two and three compared tocontrol participants (78% of dissociation participants versus 31% of controlparticipants; p < .05). This suggests that the dissociation strategy facilitatesendurance performance.


THREATS RELATING TO SELECTIONOF PARTICIPANTS

SELECTION X MANIPULATION Effects of the manipulation only hold forthe particular population sampled

A threat to external validity


Attitudes (mean)

Control

Treatment

Pre-test

Post-test

Example 5: Drugs intervention


THREATS RELATING TO TESTINGAND MANIPULATIONS

EFFECTS OF TESTINGThe effects of taking the pre-test on scoreson the post-test

Post-test scores may be affected by practice,memory etc.

Pre-test sensitisation


Example 5: Drugs intervention

Attitudes (mean)

Pre-test

Post-test


THREATS RELATING TO TESTINGAND MANIPULATIONS

REACTIVE EFFECTS OF TESTINGTesting X manipulation interactionThreat to external validity

Pre-testing changes the participants’ responsesto the manipulation. The results obtained frompre-tested participants may be unrepresentativeof results that would be obtained from thosewho only received the manipulation


REACTIVE ARRANGEMENTS

The experimental setting influences participantsto respond differently to how they would respondin a real-life setting

Research settings are unnatural situationsso we might expect participants to behaveunnaturally


Dissociation and endurance performance(Morgan et al., 1983).

Long distance runners in Tibet have used thistechnique to run 300 miles non-stop in 30 hoursand many successful marathoners are known to use similar techniques… By dissociating in thisway you will not feel the same amount of fatigue,pain or discomfort you experienced during the lasttrial…


REACTIVE ARRANGEMENTS

DEMAND CHARACTERISTICS (Orne, 1962)

“… the totality of cues which convey an experimental hypothesis to the subject.”


Dissociation and endurance performance(Morgan et al., 1983).

Long distance runners in Tibet have used thistechnique to run 300 miles non-stop in 30 hoursand many successful marathoners are known to use similar techniques… By dissociating in thisway you will not feel the same amount of fatigue,pain or discomfort you experienced during the lasttrial… Please go as long as you can in order to makeour experiment a success.


NON-SPECIFIC TREATMENT EFFECTS

Placebo/attention effects Hawthorne effects

Expectations of benefit Credibility of the treatment


Rosenthal’s rats (1963)

EXPERIMENTER EFFECTS

The Pygmalion effect

Rosenthal & Jacobson (1968)


RANDOMISED PRE-TEST POST-TESTCONTROL GROUP DESIGN

R O1 X O2

R O3 O4

Controls for threats relating to the passage of timeprovided that:

Randomisation works (groups equated at pre-test)

There is no differential mortality between groups

All potential independent variables includingnon-specific treatment effects are held constant


R O1 X O2

R O3 O4

Threats relating to selection and testingare controlled because participants arerandomly assigned to groups

EXCEPT:

Because there is a pre-test, the reactiveeffects of testing are not controlled


Alsocontrols for threats to internal validityprovided that:

Randomisation works (groups equated at pre-test)

There is no differential mortality between groups

All potential independent variablesincludingnon-specific treatment effectsare held constant

POST-TEST ONLY CONTROL GROUP DESIGN

R X O1

R O2


R X O1

R O2

PLUS:This design also controls for reactive effectsof testing because there is no pre-test

BUT:We don’t know if the groups were equated at pre-test


EXTERNAL VALIDITY

With the exception of the post-test only design’scontrol for reactive effects of testing, logically, wecannot control for threats to external validity

In an internally valid study we can onlydemonstrate that the effects (of X) hold underthe specific conditions of the study

e.g. The effects of X hold for(pre-tested)participants from theparticular populationsampled at thispoint in timein this particularplace … etc.


ad