Process outcome and cost evaluation seeing the whole elephant
This presentation is the property of its rightful owner.
Sponsored Links
1 / 43

Process, Outcome and Cost Evaluation: Seeing the Whole Elephant PowerPoint PPT Presentation


  • 74 Views
  • Uploaded on
  • Presentation posted in: General

Process, Outcome and Cost Evaluation: Seeing the Whole Elephant. American Evaluation Association November 3, 2006 Shannon M. Carey, Ph.D. Presenters: Shannon Carey, Ph.D. Kimberly Pukstas, Ph.D. - Maryland Gwen Marchand, M.S. – Michigan Katharina Wiest, Ph.D. - Indiana.

Download Presentation

Process, Outcome and Cost Evaluation: Seeing the Whole Elephant

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Process outcome and cost evaluation seeing the whole elephant

Process, Outcome and Cost Evaluation: Seeing the Whole Elephant

American Evaluation Association

November 3, 2006

Shannon M. Carey, Ph.D.


Process outcome and cost evaluation seeing the whole elephant

Presenters:Shannon Carey, Ph.D.Kimberly Pukstas, Ph.D. - MarylandGwen Marchand, M.S. – Michigan Katharina Wiest, Ph.D. - Indiana


Npc received grants from federal sources nij and bja and from state sources e g california aoc

NPC received grants from federal sources (NIJ and BJA) and from state sources (E.g., California AOC

  • To perform process, outcome and cost evaluations of 5 drug courts in OR and a statewide drug court cost evaluation in CA

  • Since then we’ve added drug courts in Maryland, Michigan, Indiana, New York, Nevada, Minnesota and Guam


What is a drug court

What is a Drug Court?

  • The purpose of drug courts is to guide offenders identified as drug-addicted into treatment that will reduce drug dependence and improve the quality of life for offenders and their families.


What is a drug court1

What is a Drug Court?

  • Participants are closely supervised by a judge who is supported by a team of agency representatives that operate outside of their traditional adversarial roles including addiction treatment providers, prosecuting attorneys, public defenders, law enforcement officers, and parole and probation officers who work together to provide needed services


What are the main goals of drug court

What are the main goals of Drug Court?

  • Reduce recidivism

  • Reduce substance use

  • Improve family/community/individual functioning


Process outcome and cost evaluation seeing the whole elephant

Drug Courts

10 Key Components (NDCI-1997)

16 Strategies (NDCI/BJA 2003)

(See Handout)


Process outcome and cost evaluation seeing the whole elephant

Guam

(N=100)

Portland, OR

(n= 600)

L.A.

(El Monte) (n=127)

Orange (Santa Ana) (n=289)

San Joaquin (n=202)

Stanislaus

(n=399)

Average Age

38

34

32

32

36

33

Gender

72% Male

74% Male

75% Male

71% Male

61% Male

66% Male

Race/

Ethnicity

76% Chamorro

24% F, W, K

74% White

9% Hisp

68% Hisp

28% White

45% Hisp

43% White

43% White

24% Hisp

31% AfAm

80% White

16% Hisp

Drug of Choice

90% Meth/(“Ice”)

Alc, MJ

29% Meth

21% Coc

49% Coc

33% Meth

38% Meth

26% Her

26% Coc

29% Coc

25% Meth

76% Meth

11% MJ

Who Participates in Drug Court?

Most common average: 2 arrests in the two years prior


Cross site evaluation

  • Needed to create a methodology that:

  • Includes rigorous research design

  • Is reproducible (providing the same type of

  • information for cross-site comparison)

  • Is flexible across multiple sites (method is

  • responsive to different court characteristics)

  • Produces information useful to program

  • staff, policy makers and the research field

Cross-Site Evaluation


Cross site evaluation1

  • To see the whole elephant:

  • (to produce useful information)

  • Process Evaluation

  • Outcome/Impact Evaluation

  • Cost Assessment (Cost-Benefit Analysis)

Cross-Site Evaluation


Process evaluation

  • Purpose:

  • To examine program policies and procedures

  • Determine if:

  • the program was implemented as intended

  • the program is serving its target population

  • To explain/interpret outcome and cost results

Process Evaluation


Process evaluation1

  • Benefit:

  • Useful Information about program functioning

  • Contribute to program improvement

  • Increasing effectiveness for participants

  • Better Outcomes, Better Cost-Benefits

Process Evaluation


Process evaluation2

  • How do we do it?

  • Interviews and focus groups about drug court

    process, policies, and staff and client successes

    and challenges (learn from the program!)

    (Drug Court Typology Interview Guide –

    www.npcresearch.com)

  • Document review (program policy manuals,

    budgets and grant proposals)

  • Databases, Paper files - Program participant

    characteristics: demographics, referral dates, exit

    status

Process Evaluation


Outcome impact evaluation

Purpose:

Determine whether the program has improved participant outcomes as intended

Outcome Evaluation: Within Program

(services received, grad rate, completion in intended time-frame, factors that lead to graduation)

Impact Evaluation: Outside/After Program

(recidivism, subsequent treatment, social services, health

care)

Outcome/Impact Evaluation


Outcome impact evaluation1

  • Benefit:

  • Information about program effectiveness

  • Intended outcomes/impacts achieved

  • 2. What program characteristics (process) led to successful outcomes

Outcome/Impact Evaluation


Outcome impact evaluation2

  • How do we do it?

  • Collect administrative data from databases

    (preferred) and paper files

  • Ask questions – Search for any available

    databases that track individuals (keep asking/ask

    in different ways)

  • Participant interviews over time? (Not usually)

Outcome/Impact Evaluation


Outcome impact evaluation nij stop dc study

Type of Session

Low

Intensity

Medium Intensity

High

Intensity

Expert Opinion

Stated Policy

Administrative data

Intensive Tracking

Group

69

103

47.5

45

Individual

25

25

26

24

Acupuncture

51

51

44/31

29

Court Sessions

11

11

14

19

Outcome/Impact EvaluationNIJ – STOP DC Study


Outcome impact evaluation3

  • Data Needed:

  • Identifiers

  • Demographics

  • Drug Court entry and exit dates

  • Date of arrest and court case number

  • Date of referral to drug court program

  • Drug Court status on exit

  • If terminated, reason for termination

  • Dates of entry into each phase

  • Criminal justice status on exit

  • Dates of UAs (and other drug tests)

  • Dates of positive UAs (and other drug tests)

  • Dates of drug court sessions

  • Drugs of Choice (Primary and secondary)

  • Attitude toward treatment

Outcome/Impact Evaluation


Outcome impact evaluation4

  • Data Needed (Cont.):

  • Dates of services received

  • General treatment issues

  • Rewards and Sanctions (Dates, types and duration)

  • Non-compliant behavior

  • Aftercare services (Dates and types)

  • (For Juvenile) School attendance status at entry and exit

    Impact Data:

  • Subsequent treatment episodes

  • Dates of re-arrest after entering the drug court program*

  • Probation start and end dates

  • Jail/Detention entry and exit dates

  • Prison start and end dates

  • Social and health services information

  • (For juveniles) School related data such as completion status

Outcome/Impact Evaluation


Why do cost analysis

Why Do Cost Analysis?

Policymakers face tremendous challenges in providing cost-effective public services. Limited financial resources require difficult decisions to be made about resource allocation. Cost-benefit analysis is an effective tool to help with these decisions.


Policy questions cost benefit analysis can help answer

Policy Questions Cost-Benefit Analysis Can Help Answer

1. Which policies and programs are cost beneficial to the taxpayers?

2. How cost effective are alternative programs?

3. What are the hidden costs in existing programs?


Policy questions cost benefit analysis can help answer1

Policy Questions Cost-Benefit Analysis Can Help Answer

4. Which expenditures provide taxpayers with the best return on their money?

5. How much does everyday “business as usual” actually cost in time and resources?


Cost research strategies

Cost Research Strategies

  • Costs and Benefits

    (Opportunity Resources)

  • Cost to taxpayer approach

    (Public Funds)

  • Transactional Cost Analysis


Npc cost methods

NPC Cost Methods

Step 1: Determine the flow/process

Step 2: Identify the transactions

Step 3: Identify the agencies involved

Step 4: Determine the resources used

Step 5: Identify costs associated

Step 6: Calculate cost results


Npc cost methods1

  • Step 1: Determine the flow/process

  • (Process Evaluation)

  • DC program and “business-as-usual”

    • Interviews

    • Observation

    • Document review

NPC Cost Methods


Npc cost methods2

  • Step 2: Identify the transactions

  • Examine the process description from Step 1

  • Examples of transactions:

    • Drug court hearings

    • Treatment sessions

    • Drug Tests

    • Re-arrests

    • Jail Time

NPC Cost Methods


Npc cost methods3

Step 3: Identify the agencies involved

Interviews and Observations

NPC Cost Methods


Npc cost methods4

  • Step 4: Determine the resources used

  • (Outcome Evaluation)

  • Interviews, Observations, Admin Data, Files

  • Do this for each transaction

  • Example: court hearings

    • Time spent in court hearing

    • Time spent preparing for court hearing

    • Number of court hearings for each

      participant

NPC Cost Methods


Npc cost methods5

  • Step 5: Identify costs associated

  • Interviews and Budget Reviews

  • Direct Costs

  • Support Costs (% of direct costs)

  • Institutional Overhead Costs (% direct costs)

    (“Fully-Loaded Cost”)

NPC Cost Methods


Npc cost methods6

  • Step 6: Calculate cost results

    • Investment Cost

    • Net Investment

    • Outcome Costs

    • Net Outcome Costs

    • Total Difference (Savings – or not)

NPC Cost Methods


Investment costs

Investment Costs

Investment Costs - Costs for the case that led (or could have led) to participation in drug court

Net Investment – Cost for case that led to drug court for drug court participants subtracted by the cost for same kind of case for comparison group members.


Process outcome and cost evaluation seeing the whole elephant

Net Investment by Transaction

Portland, Oregon


Process outcome and cost evaluation seeing the whole elephant

Agency

Range

Net Investment by Agency

California

Average Net Investment

Per Participant

Superior Court

($464)

($79) – ($898)

District Attorney

($235)

$103 – ($523)

Public Defender

($279)

($76) – ($448)

Probation

$697

$2,143 – ($632)

Treatment Agencies

$1918

$706 - $3,808

Law Enforcement

($44)

$1,060 – ($1,033)

Corrections

$0

$0


Process outcome and cost evaluation seeing the whole elephant

Net Investment

California


Outcome costs

Outcome Costs

Costs that occurred after drug court entry that were not associated with the program or the “eligible” case.

Net Outcome Benefits – Cost of drug court participants subtracted from the cost of comparison group members.


Net outcome benefits

California

Net Outcome Benefits


Net outcome costs by agency per participant over 2 years

Portland, OR

Net Outcome Costs By Agency(per participant over 2 years)


Promising practices

Promising Practices

  • A single (or overseeing) treatment provider

  • High drug court team attendance at staffings

  • Court sessions start 1 every 2-3 weeks (start)

  • Treatment 2-3 times per week (start)

  • Drug tests 3 times per week (start)

  • Judges voluntary with no fixed term (or at least two years)

  • Minimum 6 months clean before graduation


Process outcome and cost evaluation seeing the whole elephant

Agency

Net Outcome Benefits by Agency

California

Average Net Outcome Benefit

Per Participant

Superior Court

($46)

$342 – ($227)

District Attorney

($12)

$148 – ($106)

Range

Public Defender

($19)

$171 – ($103)

Probation

($53)

$474 – ($650)

Treatment Agencies

$637

$336 – ($59)

Law Enforcement

($1,525)

$620 – ($3,619)

Corrections

($3,292)

($541) – ($5,377)


Process outcome and cost evaluation seeing the whole elephant

Baltimore, Maryland ($5000)


Net outcome benefits1

Juvenile Drug Court (OR)

Savings over 2 years (per participant)

All Drug Court Youth = $1000 per participant

Graduates = $10,958 per graduate

Net Outcome Benefits


Overall net benefits per year

(Subtract investment costs from outcome costs)

  • Portland, OR$1.2 million

  • Baltimore City, MD$758,000

  • El Monte, CA$150,777

  • Monterey(- $300,000)

  • Laguna Niguel, CA$107,652

  • Santa Ana, CA$75,000

  • San Joaquin, CA$1.2 million

  • Stanislaus, CA$800,000

Overall Net Benefits Per Year


  • Login