1 / 8

The challenge of evidence-based policy

The challenge of evidence-based policy. Professor Peter Urwin. Mediation and Alternative Dispute Resolution – Outcomes and Impacts Swansea University 14 th February 2013. The Policy Context.

varuna
Download Presentation

The challenge of evidence-based policy

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The challenge of evidence-based policy Professor Peter Urwin Mediation and Alternative Dispute Resolution – Outcomes and Impacts Swansea University 14th February 2013

  2. The Policy Context • Government allocates monies to competing publicly funded activities to ensure “that public funds are spent on activities that provide the greatest benefits to society” (HMT Green Book, 2003; 2011). • The main way that this is achieved is through the process of Cost Benefit Analysis (CBA). • This approach allocates financial values to costs and benefits arising from a certain activity, which can then be valued according to any net benefit it is seen to provide to society.

  3. Consider the example of Acas services • Meadows (2007) and the subsequent BIS (2010) review for the Public Value Programme consider, • estimates of the value of time saved by employers and third parties as a result of early dispute resolution, • case studies of collective dispute resolution, to estimate the wider benefits of resolution. • However, question still remains of how much of these costs and benefits can be attributed to ADR intervention. • Some cases would have been resolved early even without the help of Acas (deadweight). • Meadows (2007) and the BIS PVP (2010) methods for estimating the proportion of early-resolved cases that can be attributed to intervention have faced some criticism [from Economists and Statisticians].

  4. So, is this a straightforward challenge? • Challenge is to obtain an estimate of the early resolved cases that would have occurred in the absence of ADR intervention. • Observe (i) no. of cases resolved through ADR and (ii) have an estimate of how many of these would have been resolved in the absence of ADR. • Difference between the two gives an estimate of the value-added or Additionality of ADR. • (ii) is the counterfactual and, as the name suggests, it is counter to the fact – it does not exist. • Create a control group that are identical in all respects, other than ‘treatment’ – ‘close enough’ to counterfactual.

  5. So, is this a straightforward challenge? • Gold standard in evaluation literature is random allocation with large numbers to treatment and control. Matches on observables and unobservables. • Problems applying to ADR: Statutory duty; willingness of participants to implement random allocation (JM example); voluntary nature of ADR, and (related) numbers required. • Not insurmountable, but the question then arises of the metrics we might wish to capture. • E.g. Judicial Mediation in ETs (Urwin, et. al. 2010). Early resolution rates (proportion resolved t+1, t+2 etc.), Hearing days vacated and Satisfaction.

  6. Which metrics to capture? • Left to their own devices the parties to conflict will arrive at a ‘solution’. • Journey to such a solution is often costly and protracted. • Any ‘solution’ is potentially short-lived, as resolution through conflict will not necessarily tackle the issues at the heart of the conflict (and they are therefore likely to resurface). • Game Theory: left to own devices, two (seemingly rational) parties to a dispute will often arrive at outcomes that are sub-optimal. • The skill of ADR is often the use and strategic deployment of information to arrive at enhanced (pareto-improved) outcomes • IMPLICATION: need to capture both the earlier resolution and the potential for more enduring outcomes arising from ADR – quality of outcome is important and quality may take longer to achieve.

  7. Final thoughts on evaluation • Early resolution, ‘immediate’ satisfaction and cost savings have the potential to miss much. ADR is not ‘same outcome, lower cost’. • However, there is a question of proportionality V acceptable level of rigour. In context of fight for public resource, where is everybody else’s evaluation evidence? Additional Challenges • Quantifying of outcomes from resolution of Collective V Individual disputes – very different approaches to metrics required. For instance, heterogeneity of Collective disputes. • Variety in the context of intervention, nature of ADR and adding more to existing systems. • Positive externalities [spillovers] V wider displacement effects. • There are ways of moving forward using [for instance] group-randomised trials (GRT) but it involves resource and commitment.

  8. [If time] Does Judicial Mediation (JM) reduce the length of Employment Tribunals • JM service piloted in Newcastle, Central London and Birmingham by the Employment Tribunal Service: June 2006-March 2007. • Evaluation of treatment group (receive judicial mediation) and control group (do not receive JM) carried out (Urwin, et. al. 2010). • Finding of JM evaluation: no statistically significant effect of JM on early resolution or satisfaction. • Outcome: JM rolled out nationally by the ETS.

More Related