1 / 34

Measuring Outcomes and Designing Research

Measuring Outcomes and Designing Research. Characteristic of the target population or the social condition that a program or policy is expected to change Does the use of body worn cameras by the police lead to an increase in individuals’ level of trust in the police ?

gpage
Download Presentation

Measuring Outcomes and Designing Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Measuring Outcomes and Designing Research

  2. Characteristic of the target population or the social condition that a program or policy is expected to change • Does the use of body worn cameras by the police lead to an increase in individuals’ level of trust in the police? • Does increasing police patrol at crime hot spots lead to a reduction in crime? What is an Outcome?

  3. Express outcomes as part of a logic model connecting program activities to proximal (intermediate) outcomes that, in turn, are expected to lead to other, more distal, outcomes Identifying Outcomes: Logic Models with Proximal and Distal Effects

  4. Example of a Logic Model

  5. Outcome: Reduce youth involvement in crime • How to measure? • Whether youth engages in any delinquent behavior (prevalence) • Number of delinquent acts (frequency) • Severity of delinquent acts • Time to first new delinquent act Measuring Outcomes

  6. Outcome: Increase police effectiveness • How to measure? • Crime rate(s) • Arrest rates • Clearance rates • Response time • Number of citizen complaints about police • Satisfaction with police • Trust in police Measuring Outcomes

  7. Is program/policy change designed to affect individuals, organization, community, or society as a whole? Are the measures reliable and valid? Things to Consider in Developing Outcome Measures

  8. 1st Issue: Who/What is Program/Policy Designed to Affect?

  9. Program designed to affect individuals’ • Attitudes • Knowledge • Skills • Behavior Effects on Individuals

  10. Example: training program on effective strategies for interviewing sexual assault victims that is designed to • Change attitudes about victims of sexual assault • Provide information about post-incident behavior of sexual assault victims • Teach skills that officers can use in interviewing victims • Increase arrests of suspects in sexual assault cases Effects on Individuals

  11. Example: anti-graffiti program designed to reduce social disorganization and empower community residents Example: community policing initiative designed to reduce overall crime by targeting public order (nuisance) crimes, such as prostitution, public drunkenness, loitering Effects on the Community

  12. 2nd Issue: Are Outcome Measures Reliable and Valid

  13. Reliability: extent to which measure produces the same results when used repeatedly to measure the same thing • A doctor’s scale is reliable to the extent that it reports the same weight for persons weighing exactly the same • Variation = measurement error Reliability

  14. Not all crimes are reported Not all reported crimes are recorded There are differences in the meaning of “crime,” the definitions of individual crimes, and the rules for classifying crimes Are Crime Statistics a Reliable Measure of the Amount of Crime?

  15. Test-retest using same subjects • If measure is reliable, should produce the same results for each subject each time • Internal Consistency Reliability • Consistency of responses to similar items in a multi-item measure administered at same time • Inter-rater Reliability • Degree to which two individuals record data on the outcome in the same way Assessing Measures for Reliability

  16. Validity: extent to which the indicator measures what it is intended to measure • Are we measuring what we think we are measuring? • Are arrests for drug offenses a valid indicator of level of community drug use? • Are arrest rates or crime rates or clearance rates valid indicators of the effectiveness of the police? Validity

  17. Expect roughly same results if two measures of same outcome are applied at same time • Two different measures of trust in the police should produce similar results • Expect different results when measure is applied to individuals or situations that are different • Measure of trust in the police should produce different scores for individuals in high-crime and low-crime communities Assessing Measures for Validity

  18. Reliability Versus Validity

  19. Experimental • Randomized control trial—the “gold standard” • Quasi-Experimental • Non-Experimental Types of Research Designs

  20. Ideal Design for Measuring Impact • Key elements: • Random Assignment to groups • Program group • Control group • Before and After Measurement Experimental Design

  21. R O1 X O2 RO1 O2 R indicates Random assignment O is the Observation or measure X is the Program or the intervention Experimental Design

  22. Classic Experimental Design: Testing for Program/Policy Impact

  23. Complaint regarding domestic violence led to one of three randomly assigned responses by the police • Send assailant away from home for 8 hours • Mediate/arbitrate dispute/counsel the parties • Arrest the assailant Example: Minneapolis Domestic Violence Experiment (1983)

  24. Outcome measure: rearrest (same suspect/same victim) within 6 months • Results • Arrest—10% • Mediate/Counsel—19% • Send assailant away—24% Example: Minneapolis Domestic Violence Experiment

  25. Other evaluation designs • Quasi-experimental • Non-experimental • Quasi-experimental designs are weaker than experimental and non-experimental designs are weaker than quasi-experimental • More vulnerable to incorrect interpretations of project impacts Other Research Designs

  26. O1 X O2 Program Group O1 O2Control Group • Program Group and Control Group but • No random assignment • Matched pairs • Non-equivalent comparison groups Quasi-Experimental Design

  27. Study of the effect of increased police patrol on violent crime rates in a city’s neighborhoods • Each neighborhood matched with another neighborhood on crime rate, poverty rate, unemployment rate and other relevant factors • One neighborhood received increased police patrol for six months, other did not Quasi-Experimental Design

  28. Before and After Design(lacks control group): O X O • Measure violent crime rate 6 months before and 6 months after an increase in police patrol • Interrupted Time-Series Analysis OOOOOOX O OOOOO • Measure violent crime over time to determine if an increase in police patrol led to a decline in violent crime Two Examples of Non-Experimental Designs

  29. Collecting information from individuals by asking them standardized questions or asking them to respond to or evaluate hypothetical situations • In person interviews • Questionnaires • Telephone or mail/online surveys Survey Research

  30. Question: Do men and women (or youth and adults) have different levels of trust in the police? Survey a sample of individuals and compare responses of men and women (or of youth and adults) Survey Research

  31. Trust in the Police and Age

  32. Longitudinal survey of delinquent behavior and alcohol/drug use among American youth (11 to 17 years of age) • Self-report questionnaires National Youth Survey (NYS)

  33. National Youth Survey: Drug Use Over Time

More Related