What is a rapid evidence assessment what s involved
1 / 29

What is a Rapid Evidence Assessment? What’s involved? - PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

What is a Rapid Evidence Assessment? What’s involved?. Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information Unit. Evidence reviews: The what, why and how… Evidence reviews in practice: Examples of recent Rapid Evidence Assessments.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

What is a Rapid Evidence Assessment? What’s involved?

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

What is a Rapid Evidence Assessment? What’s involved?

Evidence Base Camp 2013

Levin Wheller

Practice Development Team

Research Analysis and Information Unit

Evidence reviews:

The what, why and how…

Evidence reviews in practice:

Examples of recent Rapid Evidence Assessments

This session

Why review evidence?

“The sheer amount of potential research evidence in most substantive areas of social science and public policy… make[s] it almost impossible to keep abreast of the research literature in any one area”.

Davies, 2003

Evidence reviews:the what, why and how…

Methods for reviewing evidence…

Literature reviews

Look! The breadcrumbs lead here, this MUST be the answer!

Literature reviews

AHAHAHAHA! I have tricked you into only reviewing only *some* of the available evidence!

Are we happy for professionals to only have some of the evidence when making decisions?

Antman et al, 1992.

Study comparing recommendations for treating heart attacks based on literature reviews with recommendations based on a systematic meta-analysis.

We need to look at all the evidence…

  • Literature reviews often failed to mention important advances or exhibited delays in recommending effective preventive measures.

  • In some cases, treatments that have no effect on mortality or are potentially harmful continued to be recommended by several clinical experts.

Systematic Reviews

Systematic reviews

Overall, legitimacy interventions resulted in a large, significant increase in positive perceptions of police.

Taken from: Mazerolle, L., Bennett, S., Davis, J., Sargeant, E. and Manning, M. (2013) Procedural Justice and Police Legitimacy: A Systematic Review of the Research Evidence. Campbell Systematic Reviews 2013:1.

Of those that had heard of them, some had concerns:




Some had difficulty distinguishing them from literature reviews, even when explained

Campbell S et al (2007) Analysis for policy: evidence-based policy in practice

Policy opinions of systematic reviews

Use systematic principles and the same process as a systematic review…

…but make compromises given available time and resources

Pragmatic and transparent approach

Rapid Evidence Assessments

Draft search terms

Draft sift criteria

Sift received abstracts

Request relevant papers

Read and ‘grade’ papers

Write it up (‘synthesis’)

The process (in a nutshell)

Demonstrate consistency in searching/ sifting

Document search and sift process

Process should be transparent and repeatable

Specify required quality of evidence

Systematic reviews only?

Pre-post studies only?

All ‘empirical’ papers?

Be explicit/ transparent about the limitations of the approach

Some key principles

Some examplesWhat it is like in practice?

Review of Police Leadership and Training commissioned by the Home Secretary

CC Peter Neyroud needed the best evidence he could get on “What works in training and behaviour change?”

Three weeks to deliver an evidence review

Not a full REA, but…

More than literature or scoping reviews

Used systematic principles

Due to time limit, search restricted to evaluations and systematic reviews only

One practical example

Search terms

training OR learning OR development

AND evaluat* OR assess* OR what works OR impact

AND systematic review


Searched 11 databases and 2 websites

English language only

Systematic review



Searching (Training)

Initial search identified 1,015 abstracts to sift

Sifting (Training)

  • Q1: Is the study about adult training, learning, or development?

  • Q2: Is the study: An evaluation (at least pre & post level) OR a systematic review?

  • Secondary sift to remove papers related to inappropriate populations and specific medical conditions





Read, Appraise and Synthesise papers – What works in training?


Availability of papers (10/22)

Unable to pilot search terms

Available databases

Only those available to the NPL

English language only

Literature focussed in different areas

Almost nothing on policing

Papers mostly from healthcare


Forces are adopting business improvement techniques to examine current practices and explore scope to change processes to release savings

Techniques include QUEST, CI, Lean, Six Sigma, Kaizen, etc.

Organisational change and business improvement

  • So – are these techniques the answer?

  • Is that magic potion? Or is it snake oil?

Rapid Evidence Assessment

Step 1

Systematic search

(11,960 abstracts)

Step 2

Quality assess & critically appraise

(181 empirical studies)

Step 3

Synthesise findings

(41 studies with useful findings)

Draw conclusions

Example of the two searches in the organisational change and business improvement REA

Organisational change and business improvement

Total of approx 12,000 abstracts; 181 full papers were requested; 41 were included…

  • Why so many papers?

    • Better search terms needed?

    • Dodgy descriptions of papers in abstracts?

    • Problems with searching?

Organisational change – what works?

Potential success factors for organisational change

Organisational change: potential success factors


stability of supervision throughout implementation

direct support from supervisors - ‘on-the job’ training

staff involvement in decisions

transformational leadership behaviour = reduced employee cynicism


staff active participation in decision making & ‘room to experiment’

degree to which staff understood rationale for change

communication found to influence self reports of job performance

Piloting your searches is critical (we’ll do this tomorrow) to give you an idea of the size of the job, and if there is much available material

It’s important to focus on the end result and how will the findings be used – make it relevant…

Recognise (and accept) there are limitations of the approach

Make sure you are researching the right question

Follow the key principles:

Demonstrate consistency in searching/ sifting

Document search and sift process

Specify required quality of evidence

Be explicit about the limitations of the approach

REAs should be replicable

Lessons learned...

The process (for Evidence Base Camp)

  • Search terms are hugely important.

  • Tomorrow is all about developing search terms and seeing the impact of using different terms.

  • We will be running live pilot searches in your groups with College librarians.

Civil service REA toolkit:


Campbell Collaboration (social interventions, e.g. crime and justice)


Cochrane Collaboration (medical interventions)


EPPI centre (education)


Useful links

  • Login