a bayesian meta cost benefit model n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
A Bayesian, Meta Cost-Benefit Model PowerPoint Presentation
Download Presentation
A Bayesian, Meta Cost-Benefit Model

Loading in 2 Seconds...

play fullscreen
1 / 65

A Bayesian, Meta Cost-Benefit Model - PowerPoint PPT Presentation


  • 125 Views
  • Uploaded on

A Bayesian, Meta Cost-Benefit Model. John Roman, Ph.D. P. Mitchell Downey District of Columbia Crime Policy Institute Partnership for Greater Washington Research The Urban Institute The Brookings Institution. The Brookings Institution June 30, 2010. Presentation Overview. Background

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

A Bayesian, Meta Cost-Benefit Model


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
a bayesian meta cost benefit model

A Bayesian, Meta Cost-Benefit Model

John Roman, Ph.D.

P. Mitchell Downey

District of Columbia Crime Policy Institute

Partnership for Greater Washington Research

The Urban Institute

The Brookings Institution

The Brookings Institution

June 30, 2010

presentation overview
Presentation Overview
  • Background
    • Existing Models
    • DCPI
    • Drug Courts
    • Meta-Analysis
  • Proposed Model
  • Analytic Strategy
  • Results
  • Next Steps
general approach
General Approach

Estimate:

  • Costs of drug court in DC;
  • Benefits of reduced crime (general);
  • Expected benefits for a DC drug court population;
  • Effect of drug court on criminal behavior;
  • Translate effects into benefits, difference the costs.
presentation overview1
Presentation Overview

Background

Existing Models

DCPI

Drug Courts

Meta-Analysis

Proposed Model

Analytic Strategy

Results

Next Steps

the origins of dcpi
The Origins of DCPI

Funded by the Justice Grants Administration (JGA) in the Washington, D.C. Mayor’s office using Recovery Act funds.

Key goal was to create an entity similar to the Washington State Institute of Public Policy (WSIPP).

WSIPP is a non-partisan, non-profit that provides evidence to support the Legislature;

Created in the late 1980s, WSIPP (under the leadership of Steve Aos and Roxanne Lieb) conducted research funded by the legislature to inform decision-making.

the origins of dcpi1
The Origins of DCPI

In 1998, WSIPP researchers create a meta, cost-benefit model (often called the Aos model).

Performs meta analysis across a range of policies and programs;

Links to Washington state specific costs of operations;

Monetizes outcomes;

Prioritizes policy choices and makes recommendations to the legislature.

In 2008, recommended funding several cost-effective initiatives. Savings from those programs allowed plans to build two new prisons to be shelved. Passed by the Legislature.

how can existing models be improved
How Can Existing Models be Improved?

The WSIPP model embodies a quantum leap is evidence-based policymaking. However, there are two (intractable) problems with the model:

First, general problem in meta analysis that if the underlying studies are not identified, no aggregate causal relationship can be established;

Second, the model does not account for uncertainty in several steps of the estimation process, nor does it account for uncertainty in combining multiple estimates;

identification issues in the existing model
Identification Issues in the Existing Model

Consider the question of whether DC should implement an Adult Drug Court in Washington.

In order to believe that there is a causal relationship between drug court participation and successful outcomes (Y) must believe:

YT > YC and, that assignment is exogenous, such that in the absence of drug court YT = YC

If assignment is not exogenous and YT ≠ YC then no causal attribution is possible. Problem gets worse with aggregation.

We have not solved this identification problem.

integrating uncertainty into the model
Integrating Uncertainty into the Model

If model is not identified, and causal attribution is not possible, must include all sources of uncertainty into the model. Two types of uncertainty:

Estimation. Each step in the process should yield both a point estimate and a distribution.

Aggregation. Final results should integrate uncertainty from each estimate.

WSIPP model produces a single point estimate with no confidence interval. Cannot determine probabilistically whether expected outcomes will occur.

Problem akin to comparison of fingerprints and DNA

integrating uncertainty into the model con t
Integrating Uncertainty into the Model, con’t

Why isn’t uncertainty included in the WSIPP model?

The problem was intractable using classical techniques at the time the model was conceived due to limitations in computing power.

Also, difficult to revisit estimates in a policy environment.

Solution: Bayesian inference allows distributions rather than ‘moments’ to be aggregated, maximizing how much uncertainty is included in the model.

donald rumsfeld on the importance of incorporating uncertainty
Donald Rumsfeld on the Importance of Incorporating Uncertainty

"Reports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns -- the ones we don't know we don't know.“

other benefits of a bayesian meta cost benefit analysis
Other Benefits of a Bayesian, Meta, Cost-Benefit Analysis

The model is designed to be replicable;

Does not requires abundant reputational capital with stakeholders, because;

It allows for stakeholder input;

Can be adjusted in real-time to test different assumptions;

All assumptions are transparent.

Reduces the number of stages of estimation.

presentation overview2
Presentation Overview

Background

Existing Models

DCPI

Drug Courts

Meta-Analysis

Proposed Model

Analytic Strategy

Results

Next Steps

what is the district of columbia crime policy institute dcpi
What is the District of Columbia Crime Policy Institute (DCPI)?

A nonpartisan, public policy research organization focused on crime and justice policy in Washington, DC.

DCPI’s mission is to support improvements in the administration of justice and public safety policies, through evidence-based research. 

Collaboration of UI and Brookings researchers to inform long-term strategic and short-term operations research on juvenile and criminal justice mattres.

Goal is timely, practitioner-oriented research to develop and implement evidence-based crime policy in DC.

what will dcpi do
What will DCPI Do?

Year 1 DCPI activities include:

Develop a cost-benefit model to identify cost-effective DC based interventions;

Develop a research library (DCCrimePolicy.org);

Conduct independent research:

An Evaluation of the Mayor’s Focused Improvement Area Initiative;

A Study of Promising Practices at the Metropolitan Police Department;

A Study to Understand the Impact of Pre-Trial Detention on Defendants and its Implications for Evidence-Based Practice

presentation overview3
Presentation Overview

Background

Existing Models

DCPI

Drug Courts

Meta-Analysis

Proposed Model

Analytic Strategy

Results

Next Steps

what are drug courts
What are Drug Courts?

Drug courts use court-based treatment as an alternative to incarceration.

Begun in Miami in 1989, drug courts are specialized dockets that process drug-involved offenders.

Client behavior closely monitored by a judge.

Key premises:

Drug involved offenders would commit fewer crimes if they desisted from drug use.

Relapse is part of recovery.

Treatment participation can be encouraged (coerced?).

do drug courts work
Do Drug Courts Work?

Yes? Can’t randomly assign courts to have a drug court or not. So next best studies find client outcomes are moderately better using;

RCTs in a single site;

Meta-analyses of many studies with varying designs;

Simulation studies using population data.

Generally, recidivism is reduced 10-15 percent. Potential for large net social benefits.

presentation overview4
Presentation Overview

Background

Existing Models

DCPI

Drug Courts

Meta-Analysis

Proposed Model

Analytic Strategy

Results

Next Steps

what is meta analysis
What is Meta Analysis?

Policymakers seem most convinced by meta-analysis as it uses evidence from many settings.

Steps in meta analysis are to:

Calculate an effect size (standardized mean effect size comparing T and C);

Sum across studies (weighting by the inverse of the variance of each effect size);

Account for heterogeneity;

Apply ad hoc weights.

what is meta cost analysis
What is Meta, Cost Analysis?

Translate effect size into net benefits to account for:

Savings to law enforcement - reducing new crime;

Savings to courts – not prosecuting new offenders

Savings to corrections - not locking those people up;

Savings to those who are not victimized;

Add in costs of new policies and programs.

presentation overview5
Presentation Overview

Background

Existing Models

DCPI

Drug Courts

Meta-Analysis

Proposed Model

Analytic Strategy

Results

Next Steps

what is meta cost analysis again
What is Meta, Cost Analysis (Again)?

What are the criticisms?

Garbage in, garbage out (identification);

No uncertainty in these estimates.

Can’t solve identification issues at this time;

Can introduce additional uncertainty.

How? Use Bayesian inference rather than classical (inferential, repeated sample) statistics.

Analogous to measuring using the metric system. Represents the same fact, but standardized Base 10 can be more efficient than Base Whatever.

why go bayesian statistically
Why go Bayesian (Statistically)

Accounts for uncertainty in the presence of common problems:

Multi-stage model.

Output of one stage is next stage’s input.

Bayesian inference takes a weighted average of all output (weighted by the probability) versus just using the mode (frequentist).

Non-symmetric distributions. Default is ‘mode’ in frequentist inference.

Presence of Missing Data. Again, uses distribution rather than MI or listwise deletion.

why should dcpi go bayesian
Why Should DCPI Go Bayesian?

Policy Buy-In.

Presents data more intuitively (e.g. weather forecaster does not predict an expected mean precipitation of 0.5 inches with a standard deviation of .25).

Allow Policymaker input. (e.g. can provide information about prior beliefs that can improve model performance (effect of policy changes on various problems, changes in crime patterns in DC, demographic changes, etc.).

Flexibility. Can adapt to all manner of common problems in inference (missing data, etc.);

Can’t combine multiple estimates with uncertainty without going Bayes. Replaces sensitivity analyses.

presentation overview6
Presentation Overview

Background

Existing Models

DCPI

Drug Courts

Meta-Analysis

Proposed Model

Analytic Strategy

Results

Next Steps

analytic strategy general
Analytic Strategy: General

Estimate:

  • Costs of drug court in DC;
  • Benefits of reduced crime (general);
  • Expected benefits for a DC drug court population;
  • Effect of drug court on criminal behavior;
  • Translate effects into benefits, difference the costs.
analytic strategy costs
Analytic Strategy: Costs
  • Taken from a prior study of a DC drug court (Harrell, et al. 1998)
  • Inflation adjusted to get present day per participant costs of drug court operations
  • Estimates $11,500 per participant (for entire process)
  • Limitations:
    • Based on FY 1995 costs – could have changed
    • Assumes same size drug court (returns to scale)
  • Advantages:
    • DC specific
analytic strategy valuing benefits of reduced crime
Analytic Strategy: Valuing Benefits of Reduced Crime

The cost of crime is the product of the price of crime (P) and the quantity of crime (Q), so:

Costi= Pi*Qi

Where i’s are the categories of crime costs.

  • Law enforcement costs of investigating crime;
  • Court costs of processing crime;
  • Corrections costs of supervision;
  • Victims costs of being victimized.

Main source of uncertainty: price of crime to victims. Extant estimates produce point estimates, Roman (2009) includes distribution of prices.

analytic strategy calculating benefits
Analytic Strategy: Calculating Benefits

The goal of this analysis is to create an estimate of the distribution of net harms averted.

Calculate reduced crime (impact)

  • From large scale, multi-site evaluation of drug courts find:
    • the rate of re-arrest among comparison group (not participating in drug court);
    • the distribution of crimes committed by those comparisons who were re-arrested;
    • the proportion of arrests that led to incarceration.
analytic strategy benefits cont
Analytic Strategy: Benefits (cont.)

In the next step, we will estimate how many crimes are prevented by drug court.

  • For each arrest prevented, estimate what the crime would have been by sampling from the distribution of crimes committed by controls;
  • Estimate the price of that crime by sampling from Roman (2009);
  • Find the probability that would have led to incarceration (calculate costs of incarceration);
  • These last two prices are the benefits of preventing one crime.
analytic strategy impact estimates
Analytic Strategy: Impact Estimates
  • Meta-analysis
  • Data:
    • From Shaffer (2006)
    • Coded typical meta-data (effect size, research design, length of follow-up, sample size, etc.)
    • Called drug courts which had been evaluated and interviewed about policies and practices
  • Our approach:
    • Bayesian linear regression of estimated effect size (correlation between re-arrest and drug court participation)
    • Condition effect size on court and study characteristics
analytic strategy impact estimates1
Analytic Strategy: Impact Estimates
  • Variable selection (court traits):
    • Preliminary regressions of effect size on theoretically meaningful characteristics (treatment type, program design, response to graduation/failure, eligibility, etc.)
    • Selected those which were significant: violent offenders eligible and motivation required for enrollment
  • Variable selection (study traits):
    • Length of follow-up;
    • Methodology (random trial, matching, etc.);
    • Similarity between treatment and control groups (race, age, gender, prior criminal history);
    • Composition of control group (dropouts, ineligible, etc.).
presentation overview7
Presentation Overview

Background

Existing Models

DCPI

Drug Courts

Meta-Analysis

Proposed Model

Analytic Strategy

Results

Next Steps

results effect size estimates
Results: Effect Size Estimates
  • The final results are many simulated possible effect sizes
  • From these simulations, we can estimate which effect sizes are more probable than others
  • The following slide presents the probability (on the y-axis) that the effect size will be less than (larger reduction) various values (on the x-axis)
  • It is calculated separately for courts that require motivation, don’t require motivation, and allow violent offenders
  • The vertical blue line indicates the probability that the effect size will be below zero (drug court reduces crime)
results effect sizes tests eligibility rules
Results: Effect Sizes, Tests Eligibility Rules

Does not Require Motivation

Requires Motivation

Allows Violent Offenders

Good Effects

Bad Effects

results effect size estimates1
Results: Effect Size Estimates
  • The results before clearly indicate that courts that do not accept violent offenders are nearly guaranteed to reduce offending
  • However, often, it is not enough to know that there will likely be some reduction, but the size of that reduction is critical
  • A key advantage of Bayesian methods is that they allow one to estimate the probability that the reduction will be greater than any particular value
  • If we arbitrarily define a large effect to be an effect size less than -0.08, then we can calculate the probabilities of a large effect under each of the three scenarios
results effect sizes probability of a large effect
Results: Effect Sizes, Probability of a Large Effect

Does not Require Motivation

Requires Motivation

Allows Violent Offenders

Good Effects

Bad Effects

results effect size estimates2
Results: Effect Size Estimates
  • The previous slide indicates that there is a large difference in the probability that a court will achieve a large reduction, depending on whether or not motivation is required
  • This result is important because both courts had near equal likelihood of leading to a reduction in reoffending
  • Motivation eligibility requirements only start to matter when there is a large target effect
the effect size over time
The Effect Size Over Time
  • The following plot traces the effect size over time, based on the relationship between measured effect and length of follow-up
  • The findings indicate that studies with longer follow-up periods tended to estimate larger effects, suggesting that the effect of drug court grows over time
  • For simplicity, only the mean effect size at each time point is displayed
results effect size over time
Results: Effect Size Over Time

Allows Violent Offenders

Does not Require Motivation

Requires Motivation

the problem with effect sizes
The Problem with Effect Sizes
  • The past three plots indicate that:
    • Drug courts which do not accept violent offenders are almost certain to have a true re-arrest reducing effect
    • The effects are larger in courts that require motivation for enrollment
    • The effects appear to grow over time
  • However, all of the above results use the effect size
  • Effect sizes are basically measures of the mean effects
  • There is no certainty that any jurisdiction’s experience will match the mean effect size
the problem with effect sizes1
The Problem with Effect Sizes
  • The problem with presenting effect sizes can best be thought of by considering the sample size problem ubiquitous in social science statistics
  • Some effect is measured, but with a small number of units, we cannot be sure whether the observed effect reflects the true effect (hence tests for statistical significance)
  • This problem is reversed here
  • We estimate the true effects of a drug court, but if only a small number of offenders are enrolled, we cannot be sure that the realized, observed effect will match the underlying true effect
the problem with effect sizes2
The Problem with Effect Sizes
  • In other words, when presenting effect sizes, we are presenting estimates of what the true effect is, not what effect could be expected from establishing a new drug court
  • This problem is greatest when considering programs with a relatively small number of participants, which typifies most criminal justice programs
  • The following plot displays the densities of expected number of arrests prevented, given 230 participants (the mean number from our sample)
densities of change in rearrest
Densities of Change in Rearrest

Reduction in Rearrest | Increase in Rearrest

the problem with effect sizes3
The Problem with Effect Sizes
  • These densities indicate that there is substantial probability that drug courts will not reduce re-arrests, even though there is nearly a 100% chance that the mean effect is a reduction
  • Telling policymakers that there is a 99% chance of a mean reduction would have left out the critical detail that there is only a 90% chance that they will experience a reduction in arrests
  • The blue lines indicate the general findings from past meta-analyses, which suggest that the mean drug court effect is a 10-15 percent reduction in re-arrests
  • While the distributions match this mean estimate very well, this small interval masks considerable variation in estimated effects
the problem with effect sizes4
The Problem with Effect Sizes
  • The following slides illustrate this problem
  • The blue lines indicate the distribution of the effect size based on the simulations (solid line is mean estimate, dashed lines are the 95% confidence interval of the estimate)
  • These lines are horizontal because the effect size estimate is not based on sample size (the x-axis)
the problem with effect sizes5
The Problem with Effect Sizes
  • For each population size displayed on the x-axis, based on the distribution of the effect size, we simulated the number of arrests prevented 10,000 times
  • Rather than the effect size, these simulations represent the reduction that courts of different sizes would likely experience
  • This information is displayed by the black lines
  • Again, the mean is the solid line and the dashed lines are the 95% confidence interval
the problem with effect sizes6
The Problem with Effect Sizes
  • The results indicate the problem with presenting effect sizes to policymakers
  • While the means are identical (as is to be expected) the 95% interval is considerably different
  • While the simulations approach the effect size as court size increases, particularly for small courts, the range of expected effects is drastically larger than the range of effect sizes
  • This indicates that the specificity of the effect size estimate inaccurately underestimates the uncertainty in expected outcomes, which can mislead policymakers and make researchers seem less credible
  • While the effect size is virtually guaranteed to produce arrest reducing effects, the black line indicates that there is substantial probability an increase in arrests will actually be experienced in a small court
why effect sizes are misleading
Why Effect Sizes are Misleading
  • To further illustrate, the following plots holds the “true” reduction constant at 15% and randomly simulates reductions, given different sized populations
  • For each population, there is only 1 simulation
  • These simulations are then plotted (the black lines connect each simulation and the blue line is the true 15%)
  • The results display tremendous variability in outcome for smaller courts, and demonstrate that even with 2,000 participants, the “true” reduction will not always be achieved
final results
Final Results
  • The next slide displays potential reductions in the number of arrests on the x-axis and the probability that more than that number of arrests will actually be prevented on the y-axis
  • The blue lines indicate that if, hypothetically, the mayor was only interested if more than 20 arrests were prevented, there is a 60% chance that this will occur
  • That information is far more useful to the Executive Office, because it allows them to determine their own risk aversion
final results1
Final Results
  • Using the distribution of crimes committed from a large multi-site study of drug courts (Rossman, et al, 2011), the distribution of the costs of victimization from Roman (2009), and the costs of arrest from Miller and Cohen (1996), we estimated the benefits of drug court, and subtracted the costs
  • The following plot displays possible target net benefits per participant (x-axis) and the probability (y-axis) that the net benefits will exceed that target
final results2
Final Results
  • The results highlight the importance of full distributions of possible outcomes
  • The mean expected net benefits (not displayed) is highly positive, however there is only a 35% chance that net benefits will exceed zero
  • This is because there’s a small chance of huge benefits; there’s a 15% chance net benefits per participant will exceed $30,000
  • In other words, drug court will probably not be cost effective, but there’s a chance it will be hugely cost effective
final results3
Final Results
  • The results indicate the importance of simulation based modeling
  • The true drug court effect is virtually guaranteed to reduce the likelihood of reoffending
  • However, given finite samples (participants) this does not translate into a guaranteed reduction in arrests
  • Still, in all likelihood, drug court will reduce arrests
  • Whether or not it is cost effective, however, depends on the types of arrests that were prevented, which is determined by chance (it will not always be the same)
  • If big, costly crimes are prevented (robbery, assault, etc.), drug court costs will pay off, but if only small drug arrests are prevented, it will not
  • Simply taking mean net benefits masks this whole story
further examination of final results
Further Examination of Final Results
  • What drives this conclusion (benefits unlikely, but there is some chance of huge payoff)?
  • The crimes committed by drug involved offenders are rarely serious
  • Thus, most crimes prevented, most of the time, will be minor crimes
  • Even when serious crimes (robbery, assault) occur, there’s tremendous variation in the damage done in particular instances
  • Occasionally, a very serious crime is prevented, making drug court completely worth the investment
  • The following plot presents possible costs per prevented crime (x-axis) and the probability (y-axis) that a random prevented crime will exceed that
  • Clearly, most crimes are nearly insignificant, but there’s a moderate probability of huge crimes being prevented
presentation overview8
Presentation Overview

Background

Existing Models

DCPI

Drug Courts

Meta-Analysis

Proposed Model

Analytic Strategy

Results

Next Steps

next steps
Next Steps

Finalize the estimates of costs of drug court in DC (currently based on 15 year old data adjusted for inflation)

Code data on other alternatives to incarceration for meta-analysis and determine what works best in what circumstances

Code other interventions to see other effective crime prevention strategies

Rank policy priorities based on what is expected to yield the greatest benefits

Link to fiscal policy to explore and predict ripple effects throughout city government agencies

the last word
The Last Word

"I would not say that the future is necessarily less predictable than the past. I think the past was not predictable when it started."

- Donald Rumsfeld

“All models are wrong, some are useful."

- George E.P. Box and Norman Draper