experiments
Download
Skip this Video
Download Presentation
EXPERIMENTS

Loading in 2 Seconds...

play fullscreen
1 / 85

EXPERIMENTS - PowerPoint PPT Presentation


  • 150 Views
  • Uploaded on

EXPERIMENTS. © Pine Forge Press, an imprint of Sage Publications, 2004. Different Types of Experimental Design. true experiments quasi-experiments evaluation research nonexperimental designs. © Pine Forge Press, an imprint of Sage Publications, 2004.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'EXPERIMENTS' - snow


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
experiments

EXPERIMENTS

© Pine Forge Press, an imprint of Sage Publications, 2004

different types of experimental design
Different Types ofExperimental Design
  • true experiments
  • quasi-experiments
  • evaluation research
  • nonexperimental designs

© Pine Forge Press, an imprint of Sage Publications, 2004

example of non experimental design before after pretest posttest

Example of non-experimental design [before-after/pretest-posttest]

The Imaginary Seattle Bike Patrol Study

slide4
Pretest Posttest

O X O

Look at change in crime rates

Subtract pretest score from posttest

slide5
O X O

Could have been;

Change in Economy

Change in weather

Change in media coverage

etc.

Instead of X, or combined with X

confounding
“Confounding”
  • Combining a second variable with the independent variable is Confounding
external events
External events
  • Such additional variables in the research situation which provide an alternative explanation to the one that X is changing Y, in this case are classified as “external events” because they are occurring outside the study at the same time as the independent variable is occurring.
history
HISTORY
  • Eternal events
  • HISTORY - ENVIRONMENTAL CHANGES …all events that occurred during the time of the study that might affect the individuals studied and provide a rival explanation for the change in the dependent variable…all events between the pretest [if given] and the posttest…
  • Changes in the economy
  • Changes in the weather
  • Changes in information provided by the media
  • War
  • Natural disaster
  • Etc.
exogenous events
Exogenous events
  • Note: “history” variables are also known as “exogenous” events. This means that the events are “outside” the study.
slide10
How can we ensure that the change was due to X and not one or more of the external events?... “History”
classical experimental design
Classical experimental design:

Pretest Bike patrol Posttest

  • RA
      • Pretest No bike patrol Posttest
slide12
We have randomly selected a bunch of cities for the E group and the C group…then…
  • We compare changes in each group.
simultaneous variation
Simultaneous variation

Pretest Bike patrol Posttest

Change in economy

Change in weather

Media coverage Etc.

  • RA

Pretest no bike patrol Posttest

change in Economy

Change in weather

Media coverage

Etc

simultaneous variation14
Simultaneous variation
  • Simultaneous variation allows for equal changes in all other variables in the experimental group and control group.
  • Since randomization would equalize [out] the effects in both groups, the only reason for different results in the experimental and control group should be due to X.
  • S.V. allows for changes to occur in each group yet it “CONTROLS” for effects of other variables.
slide15
N.B.
  • Be able to explain how simultaneous variation and random assignment do their work to “control.”
review
Review
  • History is controlled by random assignment and allowing the History Variables to make changes in Y equally in both the experimental group and the control group. Hence the only difference in changes in the two groups can be attributed to X.
  • If this doesn’t make immediate sense ask now, work on it, and get help!
true experiments
True Experiments

True experiments must have at least one experimental group (subjects who receive some treatment) and at least one comparison group (subjects to whom the experimental group can be compared).

01

01

© Pine Forge Press, an imprint of Sage Publications, 2004

true experiments18
True Experiments

True experiments must have at least three things:

  • Two comparison groups (in the simplest case, an experimental and a control group)
  • Variation in the independent variable before assessment of change in the dependent variable
  • Random assignment to the two (or more) comparison groups

© Pine Forge Press, an imprint of Sage Publications, 2004

true experiments19
True Experiments

True experiments must have at least one experimental group (subjects who receive some treatment) and at least one comparison group (subjects to whom the experimental group can be compared).

01

01

© Pine Forge Press, an imprint of Sage Publications, 2004

true experiments20
True Experiments

All true experiments have a posttest—that is, measurement of the outcome in both groups after the experimental group has received the treatment. Many true experiments also have pretests that measure the dependent variable prior to the experimental intervention. A pretest is exactly the same as a posttest, just administered at a different time.

01

01

© Pine Forge Press, an imprint of Sage Publications, 2004

internal validity
INTERNAL VALIDITY
  • The issue of X being responsible for the changes to Y is an issue of INTERNAL VALIDITY.
  • Since we ruled out all the outside factors, history factors, external events which could alternatively explain why the bike patrols worked, we may tentatively assume that the study indicates internal validity.
  • Why tentatively? RA & INTERACTION
tentativeness
Tentativeness
  • RA…random assignment doesn’t always truly equalize the E group and the C group.
  • Interaction allows for the possibility that bike patrols work sometimes and not others and we may not “uncover” a hidden relationship that work in different directions.
slide23
RA doesn’t always equalize E & C group in values of Y.
  • Or, Interaction may mask actual relationship:
  • Z1 O1 O2
  • Z2 O1 O2
  • X1 X2
internal validity24
Internal Validity
  • The 5 subtypes of internal validity examined here are:
  • 1. External [exogenous outside] events / History
  • 2. Endogenous inside events / Maturation, Instrumentation, Testing, Regression
  • 3. Selection bias / Selection Mortality
  • 4. Treatment misidentification / Reactivity, Experimenter Bias, Demoralization, Compensation, Placebo effect
  • 5. Contamination / Experimental Diffusion, Contamination
causal internal validity
Causal (Internal) Validity

There are four basic sources of noncomparability

(other than the treatment) between a comparison group and an experimental group. They produce four of the five sources of internal invalidity:

  • When characteristics of the experimental and comparison group subjects differ SELECTION BIAS
  • When the subjects develop or change during the experiment as part of an ongoing process independent of the experimental treatment ENDOGENOUS EVENTS.
  • When something occurs during the experiment, other than the treatment, which influences outcome scores HISTORY.
  • When either the experimental group or the comparison group is aware of the other group and is influenced in the posttest as a result (Mohr, 1992)CONTAMINATION.
causal internal validity26
Causal (Internal) Validity

The fifth source of internal invalidity can be termed TREATMENT MISIDENTIFICATION: Variation in the independent variable (the treatment) is associated with variation in the observed outcome, but the change occurs through a process that the researcher has not identified.

© Pine Forge Press, an imprint of Sage Publications, 2004

review27
Review
  • We have done history and found that it is controlled by
  • E
  • RA
  • C
  • SIMULTANEOUS VARIATION
endogenous change
Endogenous Change
  • The next set of variables which may interfere with internal validity are exogenous variables. “When the subjects develop or change during the experiment as part of an ongoing process independent of the experimental treatment.” These include: Maturation, Instrumentation, Testing, & Regression artifact.
maturation
Maturation
  • The bike patrol period was the beginning of a baby bustlet a small baby bust for folks growing into their teen age years. Fewer teenagers…probably fewer crimes. Folks grow out of lots of crimes when they get older, marry, get decent jobs etc.
maturation30
Maturation
  • MATURATION – GROWING OLDER, WISER, TIREDER, DEVELOPMENTAL CHANGES…biological, psychological, or social processes that produce changes in the individuals or units studied with the passage of time that are not produced by the independent variable(s) under study in the experiment.
  • Changes in the age distribution of the population
  • Changes in the interest in participating in the dependent variable [getting tired of it]
how is it controlled
How is it controlled?

E O X O

  • RA Bike patrol

Age changes

  • C O O
  • Age changes
  • SIMULTANEOUS VARIATION
instrumentation
INSTRUMENTATION
  • Police could have learned to be more vigilant…thus may have caught more crooks.
  • Alternatively, police may have grown tired and thus been less vigilant.
instrumentation33
INSTRUMENTATION
  • INSTRUMENTATION – INSTRUMENT “DECAY” OR IMPROVEMENT…changes in the measuring instrument between the pretest and posttest.
  • Note changing springs on a scale…
  • Note changing experience of interviewers…
  • Note changing ability of coders…
how to control
How to control?
  • O X O
  • Bike patrol
  • Police Reporting changes
  • RA
  • O O
  • Police Reporting changes
  • SIMULTANEOUS VARIATION
testing
TESTING
  • The study of crime may have made the potential victims more careful or may have made the criminals more careful and thus less likely to be caught.
testing36
TESTING
  • TESTING – PRACTICE OR AWARENESS CHANGES THE SUBJECT…the possible reactivity of measurement …testing itself may change the phenomenon being measured…through awareness & reflection, through experience & practice…
how to control37
How to control?
  • E X
  • & changes in testing
  • RA
  • C changes in testing
  • SIMULTANEOUS VARIATION
regression atrifact
REGRESSION ATRIFACT
  • Crime scores were unreliably measured at the top & the only way they could go was down as they varied on second test.
example using unreliable scale extreme scores can only go down
300 [297]

300 [400]

300 [295]

300 [298]

300 [296]

____

1500

1500/5=300

292

300

293

294

300

____

1479

1479/5=295.8

300-295.8=4.2

Example using unreliable scale“Extreme” scores can only go down.
regression atrifact40
REGRESSION ATRIFACT
  • REGRESSION ATRIFACT – UNRELIABLE MEASURES OF GROPUS WITH EXTREME SCORES; THERE’S ONLY ONE WAY TO GO…if only extreme cases are selected, and if the measurement instrument is unreliable, when remeasuring the only way unreliably low scores can go is up and …high scores…down
how to control for regression artifact
How to control for “regression artifact”
  • RA and simultaneous variation
  • Sam Ting w.r.t. changes due to unreliable measures with only one way to go will happen in the E group and the C group
review42
Review
  • All endogenous changes involve changes within the experiment other than X. They are controlled in the same way. SAM TING!
  • They are controlled by SIMULTANEOUS VARIATION – changing equally in the E group and the C group. RA should place the same kinds of cases in both groups so that change [aside from X] is “equalized/”
  • Hence SIMULTANEOUS VARIATION.
  • O X O
  • Other stuff
  • RA
  • O Other stuff O
slide44
e.g.1 Seattle may have already had a “thing” about crime such that the community was going to reduce it and the police on bikes was just a temporal coincidence.
  • Seattle was already hell bent on changing crime and so the selection of bikes coincided with an already established tendency.
  • So Seattle has “selected” to make the change apart from the bike patrol.
slide45
e.g.2 the film may not be the cause of change when a survey is taken on film effects. Filmgoers may have been more liberal before they saw the film.
confounding by selection
No X = not going to film

Non liberal folks avoid film

It isn’t film exposure .

Film exposure doesn’t change people.

X = film

Liberal folks go to such a film

It’s the initial liberalism of the audience

Confounding by selection
how to control for selection
How to control for selection
  • See above
  • Sam Ting
  • RA will place the same kinds of cases in the E group and the C group. Folks won’t select themselves.
slide48
INTERACTIONS WITH SELECTION – SUBJECT SELECTION - FOLKS CHOOSE WHICH GROUP THEY’RE IN SO GROUPS AREN’T REALLY EQUAL subject may be selected with a predilection to change or may appear to change because of how they have selected themselves…
experimental mortality
EXPERIMENTAL MORTALITY
  • Perhaps all the crooks were jailed …thus there was a smaller population of criminals committing crime at time 2.
slide50
EXPERIMENTAL MORTALITY - “DYING” / DIFERENTIAL ATTRITION / DROPPING OUT OF THE EXPERIMENTAL OR CONTROL GROUP…dropout problems that prevent the researcher from obtaining information on all cases.
how to control for mortality
How to control for mortality?
  • Same as above…assuming that the experimental treatment or the control treatment doesn\'t “kill off” subjects at a different rate.
treatment misidentification
Treatment misidentification
  • some process of which the researcher may not be aware is responsible for the apparent effect of treatment
treatment misidentification53
Treatment misidentification
  • N.B. Controlling for these sources of internal invalidity involves procedures different from employing a “true” experimental design.
reactivity
REACTIVITY
  • Realizing that the bike “experiment” was in place, criminals may have been on guard. Potential victims may have protected each other.
reactivity55
Reactivity
  • REACTIVITY – KNOWLEDGE ONE IS IN AN EXPERIMENT CHANGES HOW ONE ACTS [“REACTS”]…sometimes inappropriately known as the “Hawthorne effect” …subjects sometimes become aware that they are in an experiment and, because of this awareness, “react” differently…Subjects may “feel special” and work harder. Alternatively they may suffer from an “audience effect.”
how to control56
How to control?
  • Have both the E group and the C group think they’re the “experimental” group.
  • E.g. they’re “blind” to what group they’re in.
experimenter bias
Experimenter bias
  • Maybe there was not less crime but the cops liked the bikes so they cheated.
  • Maybe the cops misperceived.
slide58
EXPERIMENTER BIAS – INADVERTANTLY GUIDING SUBJECTS TOWARDS RESULTS TO “CONFIRM” THE HYPOTHESIS ; OUTRIGHT CHEATING
how to control59
How to control?
  • The experimenter must be “blind” to what condition the subject is in…during initial instructions…conducting the experiment and coding / analyzing the data.
demoralization
Demoralization
  • DEMORALIZATION – SUBJECTS IN THE CONTROL GROUP GIVING UP
  • Note: not relevant in the Seattle study, because no control group
how to control61
How to control?
  • Have subjects “blind” to which group they’re in
expectancies of the experimental staff compensation
EXPECTANCIES OF THE EXPERIMENTAL STAFF/ COMPENSATION
  • EXPECTANCIES OF THE EXPERIMENTAL STAFF/ COMPENSATION – STAFF PROVIDING “EXTRAS” TO THE CONTROL GROUP
how to control63
How to control?
  • Have staff “blind to which group subjects are in.
  • When not possible train and monitor the staff.
placebo effect
PLACEBO EFFECT
  • Anticipating cop’s bike patrols’ effectiveness, criminals did less crime.
placebo effect65
PLACEBO EFFECT
  • Placebo effect: Subjects change because of expectations of change, not because of treatment itself
how to control66
How to control
  • o bike patrol o
  • RA
  • o faux bike patrol? o
  • Sugar-pill analogy
  • S’s [and experimenters] must be BLINDw.r.t. nature of treatment. Both must see the subject as in the experimental group.
review treatment misidentification
Review: Treatment Misidentification
  • Random Assignment is insufficient here. For all these issues, either the subjects or the experimenters must be “blind.”
  • You may have heard of medical experiments wherein the procedures are “double blind.” This is to ensure that neither knowledge of the experimenter or the subject will alter the results.
contamination
Contamination
  • Think of stuff from the experimental group analogous to leeching through the soil to the control group or vice versa.
experimental diffusion
EXPERIMENTAL DIFFUSION
  • EXPERIMENTAL DIFFUSION – SUBJECTS ON ONE EXPERIMENTAL CONDITION INFLUENCING ANOTHER…by sharing information.
how to control70
How to control?
  • Isolate…sepArategroups.
contamination71
CONTAMINATION
  • CONTAMINATION – COMPARISON [CONTROL] GROUP IS AFFECTED BY OR AFFECTS THE EXPERIMENTAL GROUP
  • Compensatory rivalry or “John Henry effect.” …or demoralization…is different from # 13 in that in #13 information is passed on whereas in 14 only the knowledge of what group one is in is passed on such that group members behave differently. Wienir sees it as analogous to # 8 and doesn’t feel it is the same sort of contamination as in # 13.
how to control72
How to control?
  • Separate groups and keep members form communication about which group they’re in.
comparison of two contamination situations
Comparison of two contamination situations:
  • Diffusion: sharing content form experiment.
  • Contamination: One set of subjects letting the other know who the “experimental subjects” are.
review74
Type of invalidity:

History

Endogenous variables

Selection bias

Treatment misidentification

Contamination

How to control:

RA & simultaneous variation

RA & simultaneous variation

RA & simultaneous variation

Blind

Separate

Review
slide75
N.B.
  • Be able to explain fully how controlling works in the situations in the slide above!
true experiments76
True Experiments

An Example:

Richard Price, Michelle Van Ryn, and Amiram Vinokur (1992) hypothesized that a job-search program to help newly unemployed persons could reduce the risk of depression among this group.

The researchers tested this hypothesis with a sample of unem-ployed persons who volunteered for job-search help at Michigan Employment Security Commission offices.

© Pine Forge Press, an imprint of Sage Publications, 2004

true experiments77
True Experiments

The unemployed volunteers were randomly assigned either to participate in eight 3-hour group seminars over a 2-week period (the treatment) or to receive self-help information in the mail on how to conduct a job search (the comparison condition).

Those in the seminars were more likely to obtain jobs, which would naturally decrease their risk of depression.

© Pine Forge Press, an imprint of Sage Publications, 2004

quasi experiments
Quasi-experiments

A quasi-experimental design is one in which the comparison group is predetermined to be comparable to the treatment group in critical ways, such as being eligible for the same services or being in the same school cohort (Rossi & Freeman, 1989:313).

  • Nonequivalent control group designs have experimental and comparison groups that are designated before the treatment occurs and are not created by random assignment.
  • Before-and-after designs have a pretest and posttest but no comparison group. In other words, the subjects exposed to the treatment serve, at an earlier time, as their own controls.

© Pine Forge Press, an imprint of Sage Publications, 2004

quasi experiments79
Quasi-experiments

Ruth Wageman (1995) used a quasi-experimental design to investigate how the way tasks were designed and rewards allotted affected work team functioning.

Her research question was whether it was preferable to organize work tasks and work rewards in a way that stressed team interdependence or individual autonomy.

© Pine Forge Press, an imprint of Sage Publications, 2004

quasi experiments80
Quasi-experiments

David P. Phillips’s (1982) study of the effect of TV soap-opera suicides on the number of actual suicides in the United States illustrates a more powerful multiple group before-and-after design.

© Pine Forge Press, an imprint of Sage Publications, 2004

nonexperimental designs
Nonexperimental Designs

The ex post facto control group design has experimental and comparison groups that are not created by random assignment. Unlike the groups in nonequivalent control group designs, the groups in ex post facto (after the fact) designs are designated after the treatment has occurred.

© Pine Forge Press, an imprint of Sage Publications, 2004

other nonexperimental designs
Other Nonexperimental Designs

Cross-sectional designs, termed “one-shot case studies” in the experimental design literature, are easily able to establish whether an association exists between two variables, but we cannot be anywhere near as confident in their conclusions about appropriate time order or nonspuriousness as with true experiments or even quasi-experiments.

Longitudinal designs improve greatly our ability to test the time order of effects, but they are unable to rule out all extraneous influences.

© Pine Forge Press, an imprint of Sage Publications, 2004

external validity
External Validity

Researchers are often interested in determining whether treatment effects identified in an experiment hold true for subgroups of subjects and across different populations, times, or settings.

There is always an implicit tradeoff in experimental design between maximizing causal validity and generalizability. The more that assignment to treatments is randomized and all experimental conditions are controlled, the less likely it is that the research subjects and set-ting will be representative of the larger population.

© Pine Forge Press, an imprint of Sage Publications, 2004

slide84

The intersecting

complexity of societies, social relationships, and social beings—of people and the groups to

which they belong—is so great that it often defies reduction to the simplicity of a laboratory or

restriction to the requirements of experimental design.

Although it may be possible to test a hypothesis with an experiment, it may not always bedesirable to do so.

© Pine Forge Press, an imprint of Sage Publications, 2004

true experiments85
True Experiments

Randomization, or random assignment, is what makes the comparison group in a true experiment such a powerful tool for identifying the effects of the treatment. A randomized comparison group can provide a good estimate of the counterfactual—the outcome that would have occurred if the subjects who were exposed to the treatment actually had not been exposed but otherwise had had the same experiences

© Pine Forge Press, an imprint of Sage Publications, 2004

ad