Quasi-experimental Design

1. Quasi-experimental Design CRJS 4466EA

2. Introduction • Quasi-experiment • Describes non-randomly assigned participants and controls subject to impact assessment • Most common design involves constructed comparisons • Matching participants and comparisons • Statistical adjustment

3. Rationale for quasi-experimental usage • Random assignment not within the evaluator’s capability • Powerful stakeholders oppose randomization

4. Issues • Quasi-experiment often results from a randomized experiment • Program staff subvert the randomization process (assigning only those random subjects that will yield good results) • Attrition from treatment • Problems with data collection

5. Measuring impacts in quasi-experiments • Net effect = gross outcome for an intervention group – gross outcome for a constructed control group + or – uncontrolled difference between intervention and control groups + or – design effects and stochastic error

6. “when there is a possibility that one or more relevant differences exists between the members of the intervention and comparison groups, as there typically is in quasi-experiments, then it is also a possibility that these differences – not the intervention – cause all or part of the observed effects” (Rossi, Freeman and Lipsey, 1999)

7. “In evaluations in which selection bias is at work, net effects would tend to be over-estimated because a portion of the difference between the intervention group and its comparison would result from the stronger potential for positive (or, sometimes, negative) effects inherent in the persons selected for intervention” (Rossi, Freeman and Lipsey, 1999)

8. Ex ante quasi-experiments • Occurs before intervention to plan selection of the comparisons • Generally maximizes potential for equivalence • Considers motivations • Identifies characteristics • Locates potential comparisons • Allows review of prior, like evaluations

9. Ex post quasi-experiments • Decision to undertake evaluation occurs after program is underway • Targets are enrolled • Insufficient time to enroll a fresh group and follow them to termination • Issues can be managed to some extent through statistical controls

10. Constructing control groups by matching • Participants are sought first and comparisons are matched afterwards • Matching is based on prior knowledge and theoretical understanding of the social processes in question • Matching information is often sought in the published literature • Attend to variables that are potentially related to self-selection processes • Use only as many variables for matching as are necessary • Pertinent characteristics will tend to be intercorrelated and, therefore, somewhat redundant

11. Characteristics useful in devising constructed control groups – Exhibit 9A of text • Characteristics of individuals • Age, sex, educational attainment, socio-economic status, ethnicity, etc. • Characteristics of families (households) • Life-cycle stage, number of members, number of children, etc. • Characteristics of organized units (schools, classes, unions, etc.) • Size differentiation, levels of authority, growth rate, budget, etc. • Characteristics of communities (territorially organized units) • Population size, territorial size, industry mix, governmental organization, etc. • Note: not a substitute for priori knowledge directly relevant to the phenomena being studied

12. Matching procedures • Options are either individual or aggregate matching • Individual matching – draws a “partner” for each participant from the unexposed pool • Aggregate matching – overall distributions in the participant and control groups on each matching variable are made to correspond • Individual matching is usually preferable (the more characteristics especially) but is more expensive, time consuming and difficult to execute for a large number of matched variables (Rossi, Freeman and Lipsey, 1999)

13. Equating groups by statistical procedures • Statistical procedures, rather than matching, are now generally used in both ex ante and ex post quasi-experimental evaluations as the primary approach to dealing with selection bias and other unwanted differences between groups (Rossi, Freeman and Lipsey, 1999) • Multivariate statistical methods are commonly used to adjust for a number of contaminating variables simultaneously • Matched and statistical controls are equivalent ways of proceeding, with statistical controls possessing some superior qualities arising from the retention of observations that might have to be discarded under matching procedures (Rossi, Freeman and Lipsey, 1999)

14. Multivariate statistical models • Allows for creation of a statistical model to account for initial measurement differences between the intervention and comparison groups • The model adjusts the outcome difference between those groups to subtract the portion attributable entirely to those initial differences • Whatever difference on outcomes remains after this subtraction, if any, is interpreted as the net effect of the intervention (Rossi, Freeman and Lipsey, 1999)

15. Need also to control for variables dealing with selection of individuals into the intervention vs. the control group • Examples of these control variables include: proximity of individuals to the program site, motivation to enroll in the program, whether they had the characteristics program personnel used to select participants, etc. • Of course, the variables related to selection are only useful if they also relate to outcome (hamburger/hotdog example) (Rossi, Freeman and Lipsey, 1999)

16. Regression-discontinuity designs • Evaluator is given selection variables up-front • Limited applications, but most rigourous method • Also called cutting points designs • “regression-discontinuity designs approximate randomized experiments to the extent that the known selection process is modeled properly, which is generally straightforward because it is explicit and quantitative” (Rossi, Freeman and Lipsey, 1999)

17. Generic controls • Examples: age, sex, income, occupation and race; distributions of certain characteristics and processes (e.g. birth rates, sex ratios, proportions of persons in various labour force categories; and, derivatives of these measures • Best examples of appropriate use of generic controls are from epidemiological studies • E.g. detection of epidemics rests heavily on the epidemiologist’s knowledge of ordinary incidence rates for various diseases

18. Generic controls used successfully by epidemiologists because selection processes are either known or unimportant (i.e., use of morbidity rates to detect epidemics) • Issue of insufficient norms for using generic controls (i.e. achievement tests for inner-city children) • Generic controls are tempting because of low cost and limited time to collect data

19. A final note • Numerous comparisons between randomized designs and quasi-experimental designs relative to net effects measuring the same thing • “Lipsey and Wilson (1993) compared the mean effect size estimates reported for randomized versus non-randomized designs in 74 meta-analyses of psychological, educational, and behavioural interventions. In many of the sets of studies included in a meta-analysis, the effect estimates from nonrandomized comparisons were very similar to those from randomized ones. However, in an equal number of cases there were substantial differences in both directions” (Rossi, Freeman and Lipsey, 1999)