Process outcome and cost evaluation seeing the whole elephant
Download
1 / 43

Process, Outcome and Cost Evaluation: Seeing the Whole Elephant - PowerPoint PPT Presentation


  • 98 Views
  • Uploaded on

Process, Outcome and Cost Evaluation: Seeing the Whole Elephant. American Evaluation Association November 3, 2006 Shannon M. Carey, Ph.D. Presenters: Shannon Carey, Ph.D. Kimberly Pukstas, Ph.D. - Maryland Gwen Marchand, M.S. – Michigan Katharina Wiest, Ph.D. - Indiana.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Process, Outcome and Cost Evaluation: Seeing the Whole Elephant' - mostyn


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Process outcome and cost evaluation seeing the whole elephant

Process, Outcome and Cost Evaluation: Seeing the Whole Elephant

American Evaluation Association

November 3, 2006

Shannon M. Carey, Ph.D.


Presenters: ElephantShannon Carey, Ph.D.Kimberly Pukstas, Ph.D. - MarylandGwen Marchand, M.S. – Michigan Katharina Wiest, Ph.D. - Indiana


Npc received grants from federal sources nij and bja and from state sources e g california aoc
NPC received grants from federal sources (NIJ and BJA) and from state sources (E.g., California AOC

  • To perform process, outcome and cost evaluations of 5 drug courts in OR and a statewide drug court cost evaluation in CA

  • Since then we’ve added drug courts in Maryland, Michigan, Indiana, New York, Nevada, Minnesota and Guam


What is a drug court
What is a Drug Court? from state sources (E.g., California AOC

  • The purpose of drug courts is to guide offenders identified as drug-addicted into treatment that will reduce drug dependence and improve the quality of life for offenders and their families.


What is a drug court1
What is a Drug Court? from state sources (E.g., California AOC

  • Participants are closely supervised by a judge who is supported by a team of agency representatives that operate outside of their traditional adversarial roles including addiction treatment providers, prosecuting attorneys, public defenders, law enforcement officers, and parole and probation officers who work together to provide needed services


What are the main goals of drug court
What are the main goals of Drug Court? from state sources (E.g., California AOC

  • Reduce recidivism

  • Reduce substance use

  • Improve family/community/individual functioning


Drug Courts from state sources (E.g., California AOC

10 Key Components (NDCI-1997)

16 Strategies (NDCI/BJA 2003)

(See Handout)


Guam from state sources (E.g., California AOC

(N=100)

Portland, OR

(n= 600)

L.A.

(El Monte) (n=127)

Orange (Santa Ana) (n=289)

San Joaquin (n=202)

Stanislaus

(n=399)

Average Age

38

34

32

32

36

33

Gender

72% Male

74% Male

75% Male

71% Male

61% Male

66% Male

Race/

Ethnicity

76% Chamorro

24% F, W, K

74% White

9% Hisp

68% Hisp

28% White

45% Hisp

43% White

43% White

24% Hisp

31% AfAm

80% White

16% Hisp

Drug of Choice

90% Meth/(“Ice”)

Alc, MJ

29% Meth

21% Coc

49% Coc

33% Meth

38% Meth

26% Her

26% Coc

29% Coc

25% Meth

76% Meth

11% MJ

Who Participates in Drug Court?

Most common average: 2 arrests in the two years prior


Cross site evaluation

  • Needed to create a methodology that: from state sources (E.g., California AOC

  • Includes rigorous research design

  • Is reproducible (providing the same type of

  • information for cross-site comparison)

  • Is flexible across multiple sites (method is

  • responsive to different court characteristics)

  • Produces information useful to program

  • staff, policy makers and the research field

Cross-Site Evaluation


Cross site evaluation1

  • To see the whole elephant: from state sources (E.g., California AOC

  • (to produce useful information)

  • Process Evaluation

  • Outcome/Impact Evaluation

  • Cost Assessment (Cost-Benefit Analysis)

Cross-Site Evaluation


Process evaluation

  • Purpose: from state sources (E.g., California AOC

  • To examine program policies and procedures

  • Determine if:

  • the program was implemented as intended

  • the program is serving its target population

  • To explain/interpret outcome and cost results

Process Evaluation


Process evaluation1

  • Benefit: from state sources (E.g., California AOC

  • Useful Information about program functioning

  • Contribute to program improvement

  • Increasing effectiveness for participants

  • Better Outcomes, Better Cost-Benefits

Process Evaluation


Process evaluation2

  • How do we do it? from state sources (E.g., California AOC

  • Interviews and focus groups about drug court

    process, policies, and staff and client successes

    and challenges (learn from the program!)

    (Drug Court Typology Interview Guide –

    www.npcresearch.com)

  • Document review (program policy manuals,

    budgets and grant proposals)

  • Databases, Paper files - Program participant

    characteristics: demographics, referral dates, exit

    status

Process Evaluation


Outcome impact evaluation

Purpose: from state sources (E.g., California AOC

Determine whether the program has improved participant outcomes as intended

Outcome Evaluation: Within Program

(services received, grad rate, completion in intended time-frame, factors that lead to graduation)

Impact Evaluation: Outside/After Program

(recidivism, subsequent treatment, social services, health

care)

Outcome/Impact Evaluation


Outcome impact evaluation1

  • Benefit: from state sources (E.g., California AOC

  • Information about program effectiveness

  • Intended outcomes/impacts achieved

  • 2. What program characteristics (process) led to successful outcomes

Outcome/Impact Evaluation


Outcome impact evaluation2

  • How do we do it? from state sources (E.g., California AOC

  • Collect administrative data from databases

    (preferred) and paper files

  • Ask questions – Search for any available

    databases that track individuals (keep asking/ask

    in different ways)

  • Participant interviews over time? (Not usually)

Outcome/Impact Evaluation


Outcome impact evaluation nij stop dc study

Type of Session from state sources (E.g., California AOC

Low

Intensity

Medium Intensity

High

Intensity

Expert Opinion

Stated Policy

Administrative data

Intensive Tracking

Group

69

103

47.5

45

Individual

25

25

26

24

Acupuncture

51

51

44/31

29

Court Sessions

11

11

14

19

Outcome/Impact EvaluationNIJ – STOP DC Study


Outcome impact evaluation3

  • Data Needed: from state sources (E.g., California AOC

  • Identifiers

  • Demographics

  • Drug Court entry and exit dates

  • Date of arrest and court case number

  • Date of referral to drug court program

  • Drug Court status on exit

  • If terminated, reason for termination

  • Dates of entry into each phase

  • Criminal justice status on exit

  • Dates of UAs (and other drug tests)

  • Dates of positive UAs (and other drug tests)

  • Dates of drug court sessions

  • Drugs of Choice (Primary and secondary)

  • Attitude toward treatment

Outcome/Impact Evaluation


Outcome impact evaluation4

  • Data Needed (Cont.): from state sources (E.g., California AOC

  • Dates of services received

  • General treatment issues

  • Rewards and Sanctions (Dates, types and duration)

  • Non-compliant behavior

  • Aftercare services (Dates and types)

  • (For Juvenile) School attendance status at entry and exit

    Impact Data:

  • Subsequent treatment episodes

  • Dates of re-arrest after entering the drug court program*

  • Probation start and end dates

  • Jail/Detention entry and exit dates

  • Prison start and end dates

  • Social and health services information

  • (For juveniles) School related data such as completion status

Outcome/Impact Evaluation


Why do cost analysis

Why Do Cost Analysis? from state sources (E.g., California AOC

Policymakers face tremendous challenges in providing cost-effective public services. Limited financial resources require difficult decisions to be made about resource allocation. Cost-benefit analysis is an effective tool to help with these decisions.


Policy questions cost benefit analysis can help answer

Policy Questions Cost-Benefit Analysis Can Help Answer from state sources (E.g., California AOC

1. Which policies and programs are cost beneficial to the taxpayers?

2. How cost effective are alternative programs?

3. What are the hidden costs in existing programs?


Policy questions cost benefit analysis can help answer1

Policy Questions Cost-Benefit Analysis Can Help Answer from state sources (E.g., California AOC

4. Which expenditures provide taxpayers with the best return on their money?

5. How much does everyday “business as usual” actually cost in time and resources?


Cost research strategies
Cost Research Strategies from state sources (E.g., California AOC

  • Costs and Benefits

    (Opportunity Resources)

  • Cost to taxpayer approach

    (Public Funds)

  • Transactional Cost Analysis


Npc cost methods

NPC Cost Methods from state sources (E.g., California AOC

Step 1: Determine the flow/process

Step 2: Identify the transactions

Step 3: Identify the agencies involved

Step 4: Determine the resources used

Step 5: Identify costs associated

Step 6: Calculate cost results


Npc cost methods1

  • Step 1: from state sources (E.g., California AOCDetermine the flow/process

  • (Process Evaluation)

  • DC program and “business-as-usual”

    • Interviews

    • Observation

    • Document review

NPC Cost Methods


Npc cost methods2

  • Step 2: from state sources (E.g., California AOCIdentify the transactions

  • Examine the process description from Step 1

  • Examples of transactions:

    • Drug court hearings

    • Treatment sessions

    • Drug Tests

    • Re-arrests

    • Jail Time

NPC Cost Methods


Npc cost methods3

Step 3: from state sources (E.g., California AOCIdentify the agencies involved

Interviews and Observations

NPC Cost Methods


Npc cost methods4

  • Step 4: from state sources (E.g., California AOCDetermine the resources used

  • (Outcome Evaluation)

  • Interviews, Observations, Admin Data, Files

  • Do this for each transaction

  • Example: court hearings

    • Time spent in court hearing

    • Time spent preparing for court hearing

    • Number of court hearings for each

      participant

NPC Cost Methods


Npc cost methods5

  • Step 5: from state sources (E.g., California AOCIdentify costs associated

  • Interviews and Budget Reviews

  • Direct Costs

  • Support Costs (% of direct costs)

  • Institutional Overhead Costs (% direct costs)

    (“Fully-Loaded Cost”)

NPC Cost Methods


Npc cost methods6

  • Step 6: from state sources (E.g., California AOCCalculate cost results

    • Investment Cost

    • Net Investment

    • Outcome Costs

    • Net Outcome Costs

    • Total Difference (Savings – or not)

NPC Cost Methods


Investment costs

Investment Costs from state sources (E.g., California AOC

Investment Costs - Costs for the case that led (or could have led) to participation in drug court

Net Investment – Cost for case that led to drug court for drug court participants subtracted by the cost for same kind of case for comparison group members.


Net Investment by Transaction from state sources (E.g., California AOC

Portland, Oregon


Agency from state sources (E.g., California AOC

Range

Net Investment by Agency

California

Average Net Investment

Per Participant

Superior Court

($464)

($79) – ($898)

District Attorney

($235)

$103 – ($523)

Public Defender

($279)

($76) – ($448)

Probation

$697

$2,143 – ($632)

Treatment Agencies

$1918

$706 - $3,808

Law Enforcement

($44)

$1,060 – ($1,033)

Corrections

$0

$0


Net Investment from state sources (E.g., California AOC

California


Outcome costs

Outcome Costs from state sources (E.g., California AOC

Costs that occurred after drug court entry that were not associated with the program or the “eligible” case.

Net Outcome Benefits – Cost of drug court participants subtracted from the cost of comparison group members.


Net outcome benefits

California from state sources (E.g., California AOC

Net Outcome Benefits


Net outcome costs by agency per participant over 2 years

Portland from state sources (E.g., California AOC, OR

Net Outcome Costs By Agency(per participant over 2 years)


Promising practices
Promising Practices from state sources (E.g., California AOC

  • A single (or overseeing) treatment provider

  • High drug court team attendance at staffings

  • Court sessions start 1 every 2-3 weeks (start)

  • Treatment 2-3 times per week (start)

  • Drug tests 3 times per week (start)

  • Judges voluntary with no fixed term (or at least two years)

  • Minimum 6 months clean before graduation


Agency from state sources (E.g., California AOC

Net Outcome Benefits by Agency

California

Average Net Outcome Benefit

Per Participant

Superior Court

($46)

$342 – ($227)

District Attorney

($12)

$148 – ($106)

Range

Public Defender

($19)

$171 – ($103)

Probation

($53)

$474 – ($650)

Treatment Agencies

$637

$336 – ($59)

Law Enforcement

($1,525)

$620 – ($3,619)

Corrections

($3,292)

($541) – ($5,377)


Baltimore, Maryland ($5000) from state sources (E.g., California AOC


Net outcome benefits1

Juvenile Drug Court (OR) from state sources (E.g., California AOC

Savings over 2 years (per participant)

All Drug Court Youth = $1000 per participant

Graduates = $10,958 per graduate

Net Outcome Benefits


Overall net benefits per year

(Subtract investment costs from outcome costs) from state sources (E.g., California AOC

  • Portland, OR $1.2 million

  • Baltimore City, MD $758,000

  • El Monte, CA $150,777

  • Monterey (- $300,000)

  • Laguna Niguel, CA $107,652

  • Santa Ana, CA $75,000

  • San Joaquin, CA $1.2 million

  • Stanislaus, CA $800,000

Overall Net Benefits Per Year


ad