Process outcome and cost evaluation seeing the whole elephant
Sponsored Links
This presentation is the property of its rightful owner.
1 / 43

Process, Outcome and Cost Evaluation: Seeing the Whole Elephant PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Process, Outcome and Cost Evaluation: Seeing the Whole Elephant. American Evaluation Association November 3, 2006 Shannon M. Carey, Ph.D. Presenters: Shannon Carey, Ph.D. Kimberly Pukstas, Ph.D. - Maryland Gwen Marchand, M.S. – Michigan Katharina Wiest, Ph.D. - Indiana.

Download Presentation

Process, Outcome and Cost Evaluation: Seeing the Whole Elephant

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Process, Outcome and Cost Evaluation: Seeing the Whole Elephant

American Evaluation Association

November 3, 2006

Shannon M. Carey, Ph.D.

Presenters:Shannon Carey, Ph.D.Kimberly Pukstas, Ph.D. - MarylandGwen Marchand, M.S. – Michigan Katharina Wiest, Ph.D. - Indiana

NPC received grants from federal sources (NIJ and BJA) and from state sources (E.g., California AOC

  • To perform process, outcome and cost evaluations of 5 drug courts in OR and a statewide drug court cost evaluation in CA

  • Since then we’ve added drug courts in Maryland, Michigan, Indiana, New York, Nevada, Minnesota and Guam

What is a Drug Court?

  • The purpose of drug courts is to guide offenders identified as drug-addicted into treatment that will reduce drug dependence and improve the quality of life for offenders and their families.

What is a Drug Court?

  • Participants are closely supervised by a judge who is supported by a team of agency representatives that operate outside of their traditional adversarial roles including addiction treatment providers, prosecuting attorneys, public defenders, law enforcement officers, and parole and probation officers who work together to provide needed services

What are the main goals of Drug Court?

  • Reduce recidivism

  • Reduce substance use

  • Improve family/community/individual functioning

Drug Courts

10 Key Components (NDCI-1997)

16 Strategies (NDCI/BJA 2003)

(See Handout)



Portland, OR

(n= 600)


(El Monte) (n=127)

Orange (Santa Ana) (n=289)

San Joaquin (n=202)



Average Age








72% Male

74% Male

75% Male

71% Male

61% Male

66% Male



76% Chamorro

24% F, W, K

74% White

9% Hisp

68% Hisp

28% White

45% Hisp

43% White

43% White

24% Hisp

31% AfAm

80% White

16% Hisp

Drug of Choice

90% Meth/(“Ice”)

Alc, MJ

29% Meth

21% Coc

49% Coc

33% Meth

38% Meth

26% Her

26% Coc

29% Coc

25% Meth

76% Meth

11% MJ

Who Participates in Drug Court?

Most common average: 2 arrests in the two years prior

  • Needed to create a methodology that:

  • Includes rigorous research design

  • Is reproducible (providing the same type of

  • information for cross-site comparison)

  • Is flexible across multiple sites (method is

  • responsive to different court characteristics)

  • Produces information useful to program

  • staff, policy makers and the research field

Cross-Site Evaluation

  • To see the whole elephant:

  • (to produce useful information)

  • Process Evaluation

  • Outcome/Impact Evaluation

  • Cost Assessment (Cost-Benefit Analysis)

Cross-Site Evaluation

  • Purpose:

  • To examine program policies and procedures

  • Determine if:

  • the program was implemented as intended

  • the program is serving its target population

  • To explain/interpret outcome and cost results

Process Evaluation

  • Benefit:

  • Useful Information about program functioning

  • Contribute to program improvement

  • Increasing effectiveness for participants

  • Better Outcomes, Better Cost-Benefits

Process Evaluation

  • How do we do it?

  • Interviews and focus groups about drug court

    process, policies, and staff and client successes

    and challenges (learn from the program!)

    (Drug Court Typology Interview Guide –

  • Document review (program policy manuals,

    budgets and grant proposals)

  • Databases, Paper files - Program participant

    characteristics: demographics, referral dates, exit


Process Evaluation


Determine whether the program has improved participant outcomes as intended

Outcome Evaluation: Within Program

(services received, grad rate, completion in intended time-frame, factors that lead to graduation)

Impact Evaluation: Outside/After Program

(recidivism, subsequent treatment, social services, health


Outcome/Impact Evaluation

  • Benefit:

  • Information about program effectiveness

  • Intended outcomes/impacts achieved

  • 2. What program characteristics (process) led to successful outcomes

Outcome/Impact Evaluation

  • How do we do it?

  • Collect administrative data from databases

    (preferred) and paper files

  • Ask questions – Search for any available

    databases that track individuals (keep asking/ask

    in different ways)

  • Participant interviews over time? (Not usually)

Outcome/Impact Evaluation

Type of Session



Medium Intensity



Expert Opinion

Stated Policy

Administrative data

Intensive Tracking
















Court Sessions





Outcome/Impact EvaluationNIJ – STOP DC Study

  • Data Needed:

  • Identifiers

  • Demographics

  • Drug Court entry and exit dates

  • Date of arrest and court case number

  • Date of referral to drug court program

  • Drug Court status on exit

  • If terminated, reason for termination

  • Dates of entry into each phase

  • Criminal justice status on exit

  • Dates of UAs (and other drug tests)

  • Dates of positive UAs (and other drug tests)

  • Dates of drug court sessions

  • Drugs of Choice (Primary and secondary)

  • Attitude toward treatment

Outcome/Impact Evaluation

  • Data Needed (Cont.):

  • Dates of services received

  • General treatment issues

  • Rewards and Sanctions (Dates, types and duration)

  • Non-compliant behavior

  • Aftercare services (Dates and types)

  • (For Juvenile) School attendance status at entry and exit

    Impact Data:

  • Subsequent treatment episodes

  • Dates of re-arrest after entering the drug court program*

  • Probation start and end dates

  • Jail/Detention entry and exit dates

  • Prison start and end dates

  • Social and health services information

  • (For juveniles) School related data such as completion status

Outcome/Impact Evaluation

Why Do Cost Analysis?

Policymakers face tremendous challenges in providing cost-effective public services. Limited financial resources require difficult decisions to be made about resource allocation. Cost-benefit analysis is an effective tool to help with these decisions.

Policy Questions Cost-Benefit Analysis Can Help Answer

1. Which policies and programs are cost beneficial to the taxpayers?

2. How cost effective are alternative programs?

3. What are the hidden costs in existing programs?

Policy Questions Cost-Benefit Analysis Can Help Answer

4. Which expenditures provide taxpayers with the best return on their money?

5. How much does everyday “business as usual” actually cost in time and resources?

Cost Research Strategies

  • Costs and Benefits

    (Opportunity Resources)

  • Cost to taxpayer approach

    (Public Funds)

  • Transactional Cost Analysis

NPC Cost Methods

Step 1: Determine the flow/process

Step 2: Identify the transactions

Step 3: Identify the agencies involved

Step 4: Determine the resources used

Step 5: Identify costs associated

Step 6: Calculate cost results

  • Step 1: Determine the flow/process

  • (Process Evaluation)

  • DC program and “business-as-usual”

    • Interviews

    • Observation

    • Document review

NPC Cost Methods

  • Step 2: Identify the transactions

  • Examine the process description from Step 1

  • Examples of transactions:

    • Drug court hearings

    • Treatment sessions

    • Drug Tests

    • Re-arrests

    • Jail Time

NPC Cost Methods

Step 3: Identify the agencies involved

Interviews and Observations

NPC Cost Methods

  • Step 4: Determine the resources used

  • (Outcome Evaluation)

  • Interviews, Observations, Admin Data, Files

  • Do this for each transaction

  • Example: court hearings

    • Time spent in court hearing

    • Time spent preparing for court hearing

    • Number of court hearings for each


NPC Cost Methods

  • Step 5: Identify costs associated

  • Interviews and Budget Reviews

  • Direct Costs

  • Support Costs (% of direct costs)

  • Institutional Overhead Costs (% direct costs)

    (“Fully-Loaded Cost”)

NPC Cost Methods

  • Step 6: Calculate cost results

    • Investment Cost

    • Net Investment

    • Outcome Costs

    • Net Outcome Costs

    • Total Difference (Savings – or not)

NPC Cost Methods

Investment Costs

Investment Costs - Costs for the case that led (or could have led) to participation in drug court

Net Investment – Cost for case that led to drug court for drug court participants subtracted by the cost for same kind of case for comparison group members.

Net Investment by Transaction

Portland, Oregon



Net Investment by Agency


Average Net Investment

Per Participant

Superior Court


($79) – ($898)

District Attorney


$103 – ($523)

Public Defender


($76) – ($448)



$2,143 – ($632)

Treatment Agencies


$706 - $3,808

Law Enforcement


$1,060 – ($1,033)




Net Investment


Outcome Costs

Costs that occurred after drug court entry that were not associated with the program or the “eligible” case.

Net Outcome Benefits – Cost of drug court participants subtracted from the cost of comparison group members.


Net Outcome Benefits

Portland, OR

Net Outcome Costs By Agency(per participant over 2 years)

Promising Practices

  • A single (or overseeing) treatment provider

  • High drug court team attendance at staffings

  • Court sessions start 1 every 2-3 weeks (start)

  • Treatment 2-3 times per week (start)

  • Drug tests 3 times per week (start)

  • Judges voluntary with no fixed term (or at least two years)

  • Minimum 6 months clean before graduation


Net Outcome Benefits by Agency


Average Net Outcome Benefit

Per Participant

Superior Court


$342 – ($227)

District Attorney


$148 – ($106)


Public Defender


$171 – ($103)



$474 – ($650)

Treatment Agencies


$336 – ($59)

Law Enforcement


$620 – ($3,619)



($541) – ($5,377)

Baltimore, Maryland ($5000)

Juvenile Drug Court (OR)

Savings over 2 years (per participant)

All Drug Court Youth = $1000 per participant

Graduates = $10,958 per graduate

Net Outcome Benefits

(Subtract investment costs from outcome costs)

  • Portland, OR$1.2 million

  • Baltimore City, MD$758,000

  • El Monte, CA$150,777

  • Monterey(- $300,000)

  • Laguna Niguel, CA$107,652

  • Santa Ana, CA$75,000

  • San Joaquin, CA$1.2 million

  • Stanislaus, CA$800,000

Overall Net Benefits Per Year

  • Login