Evaluating road safety treatments

1. Evaluating road safety treatments Blair Turner, Principal Research Scientist, Australian Road Research Board (ARRB)

2. Workshop agenda • Overview of road safety evaluation (40 minutes) • Case studies (1 hour) • Evaluating the Road Safety Education Program in Malaysia – Raymond Teoh Joo Han (Malaysia Road Safety Department) • Beijing Vulnerable Road User Safety at Intersections – Ann Yuan (GRSP) • Evaluating Helmet Wearing in Cambodia – Jeroen Stol (HIB) • Evaluating Helmet Wearing in Vietnam – Jon Passmore (WHO) • Panel discussion – questions (15 minutes)

3. Overview of road safety evaluation • Why evaluate? • Evaluation techniques • Sources of evaluation data • Common traps (e.g. confounding factors) • Resources (e.g. documents and websites for further info)

4. Why evaluate? • Need to know that what we are doing is effective • Maximise benefits from limited budgets • Assess whether targets have been met • Need to know how the process can be improved • Use evaluation data to put forward the business case for funding • Fill gaps in knowledge – especially in LMICs • Need to make sure we do no harm!

5. Treatments do not always work • Some treatments shown to have no benefit • Scared straight programs • Some driver education programs have shown an increase in crashes following training – e.g. skid control

6. Barriers to evaluation • Cost • Knowledge • Access to data

7. Types of evaluation • Process evaluation: • how was the treatment delivered? • did things go to plan? • were the methods effective? • were resources wasted (staff time, other costs)? • how can the process be improved? • Outcome evaluation: • how effective was the treatment? • was there a change in crashes or improvement in behaviour? • helps to determine whether the benefits outweigh the costs • Both process and outcome evaluation should be used

8. Evaluation techniques • Randomised controlled trial • Before and after • with Empirical Bayes adjustment • with control • without control • Interrupted time series • Cross-sectional analysis

9. Randomised Controlled Trial • ‘Gold standard’ in evaluation • Rarely used in road safety evaluation – some use in behavioural programs, almost never in infrastructure • Randomly allocate individuals or groups to receive treatment • Balance out all known and unknown variables between groups – only difference will be due to treatment • Removes bias Before After Treatment Treatment Population Control Control

10. Before and after design – with control group • Most commonly used in road safety evaluation • Treatment group receive some intervention, and are measured before and after this occurs • Control group do not receive this intervention, and also measured before and after Before After Treatment Treatment Control Control

11. Before and after design – with control group • Both groups need to be closely matched • General trends are taken into account by use of control group • treatment effect = (change in treatment group) – (change in control group) • Empirical Bayes often used in infrastructure evaluation to address regression to mean issue

12. Before and after design – without control • Often used in road safety evaluation • Provides only weak evidence • Measure the outcome of interest both before and after treatment • Cheap • Can’t attribute the change to the treatment – may be due to some external factor • May provide the wrong result

13. Interrupted time series • Compare situation before and after an intervention • Distortions due to other factors that occur simultaneously • Statistical analysis can help control for these factors

15. Cross-sectional analysis • Used often with road infrastructure • e.g. compare crash performance of roads with a specific feature (e.g. sealed shoulder) with similar roads without this feature • Confounding factors (e.g. roads with sealed shoulders may have other higher quality road features) • Can try to minimise using statistical techniques

16. Sample size • Need to determine an appropriate sample size • Too small, and you won’t be able to detect a difference, even if it exists • Too large, may increase the cost of the study, and produce a significant result even if this is minute. • Sample size depends on effect size, variability of a measure, frequency at which event occurs • Sample size calculators are available online.

17. Sources of data • Crash data • Crash costs (requires information on value of statistical life) • Intermediate outcomes • behavioural measures • surveys • road and roadside asset • Output or process indicators • e.g. number of hours of speed enforcement • Other data requirements • e.g. population, traffic volumes

18. Crash data • e.g. all fatal and injury crashes • e.g. speed related crashes • Useful to express as a rate • e.g. speed crashes per 100,000 population • Severity ratio (e.g. % fatal crashes) • Requires a reliable crash data system • Under-reporting issue

19. What about when there is no crash data? • Proxy measures: • conflict analysis • intermediate measures (e.g. speeds, helmet wearing rates)

20. Performance indicators and data • Speed monitoring • Seat belt wearing rates (front and back seat) • Child restraints • Helmet wearing rates • Per cent of vehicles with 5 star safety rating • Ambulance response rates

21. Road and Roadside data • Road and roadside data useful in evaluation, especially: • When crash data not available • When information is needed after a short duration • Key features that contribute to crashes are known

22. roadside barriers speed environment lane and shoulder width clear zone width road surface condition (skid resistance) linemarking warning signs median barriers delineation overtaking opportunities/facilities street lighting sight distance horizontal alignment vertical alignment pedestrian facilities (footpaths and crossing points) Motorcycle / bicycle facilities What road and roadside features?

23. Risk assessment tools • e.g. iRAP

24. FT60, Manjung, Perak Improved pavement Widened shoulder Installed vibra lines

25. F50, Muar, Johor Installed pedestrian overpass

26. Confounding factors • Change in traffic volume • Crash migration • General trends in crash numbers • Coinciding events • Regression to mean

27. Changes in traffic volumes • Common for traffic volume to change over time • naturally • as a result of a treatment • Pedestrian volume may also change • Increased exposure = increase risk of crashes • Crash risk migration • Need to ensure wide enough geographic coverage • Use control group to counter this

28. General trends in crash numbers • Safety changes over time due to e.g.: • improved vehicle safety through introduction of new safety features • widespread changes to driver behaviour, e.g. from education, enforcement, fuel prices, changes in economy • Use a control group to measure this

29. Coinciding events • Often more than intervention used • Engineering treatments – typically two or more measures • Enforcement • Campaigns

30. Regression to mean • A major issue for evaluation of infrastructure improvements • Crash number fluctuate from year to year • If a site is selected for treatment based on a high number of crashes in one year, the site may revert back to its normal number of crashes in the following year, regardless of treatment • Over-state the benefit of treatments • Around half of the benefit from regression to mean • Differs by treatment type • Use RCT, Empirical Bayes, a larger number of years before and after.

31. Statistical analysis of data • Main applications of statistical testing in road safety: • Comparison of accident frequencies: chi-squared test, or a paired t-test if the distribution of accidents is normal. • Comparison of accident rates: paired t-test. • Comparison of proportions: z-test. • In all statistical analysis of crash reductions, the‘95% confidence level’ is typically used

32. Crash costing and economic analysis • Benefit-cost analysis • “An economic technique for gauging the value of economic decisions in terms of their capacity to satisfy the … wants of all members of society” • Presenting a case for further funding • Allocating existing funding

33. Basic reporting • Nature of the intervention studied • Types of locations / populations treated • Numbers in the treatment and control groups • Time over which evaluation took place, including before and after periods • Intervention effect, including change in crashes or behaviour broken down by crash type and severity

34. Resources • Good Practice Guides • Evaluation methodology in road safety, Cairney, Zivanovic & Turner (in press) www.austroads.com.au • The Magenta Book: Guidance notes for policy evaluation and analysis, UKGovernment Social Research Unit http://www.civilservice.gov.uk/Assets/complete_Magenta_tcm6-8611.pdf

35. Key points • Evaluation is important but often forgotten or not done well • Need to ensure that what we do works • Need to know how effective treatments are • Various evaluation methods available • Randomised Controlled Trial is best • Before and after with control group is okay if done well • Guidance available on how to conduct evaluation

36. Questions

37. Case studies • Evaluating the Road Safety Education Program in Malaysia – Raymond Teoh Joo Han (Malaysia Road Safety Department) • Beijing Vulnerable Road User Safety at Intersections – Ann Yuan (GRSP) • Evaluating Helmet Wearing in Cambodia – Jeroen Stol (HIB) • Evaluating Helmet Wearing in Vietnam – Jon Passmore (WHO)