What is a rapid evidence assessment what s involved
This presentation is the property of its rightful owner.
Sponsored Links
1 / 29

What is a Rapid Evidence Assessment? What’s involved? PowerPoint PPT Presentation


  • 112 Views
  • Uploaded on
  • Presentation posted in: General

What is a Rapid Evidence Assessment? What’s involved?. Evidence Base Camp 2013 Levin Wheller Practice Development Team Research Analysis and Information Unit. Evidence reviews: The what, why and how… Evidence reviews in practice: Examples of recent Rapid Evidence Assessments.

Download Presentation

What is a Rapid Evidence Assessment? What’s involved?

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


What is a rapid evidence assessment what s involved

What is a Rapid Evidence Assessment? What’s involved?

Evidence Base Camp 2013

Levin Wheller

Practice Development Team

Research Analysis and Information Unit


This session

Evidence reviews:

The what, why and how…

Evidence reviews in practice:

Examples of recent Rapid Evidence Assessments

This session


Evidence reviews the what why and how

Why review evidence?

“The sheer amount of potential research evidence in most substantive areas of social science and public policy… make[s] it almost impossible to keep abreast of the research literature in any one area”.

Davies, 2003

Evidence reviews:the what, why and how…


Methods for reviewing evidence

Methods for reviewing evidence…


Literature reviews

Literature reviews

Look! The breadcrumbs lead here, this MUST be the answer!


Literature reviews1

Literature reviews

AHAHAHAHA! I have tricked you into only reviewing only *some* of the available evidence!


Are we happy for professionals to only have some of the evidence when making decisions

Are we happy for professionals to only have some of the evidence when making decisions?


We need to look at all the evidence

Antman et al, 1992.

Study comparing recommendations for treating heart attacks based on literature reviews with recommendations based on a systematic meta-analysis.

We need to look at all the evidence…

  • Literature reviews often failed to mention important advances or exhibited delays in recommending effective preventive measures.

  • In some cases, treatments that have no effect on mortality or are potentially harmful continued to be recommended by several clinical experts.


Systematic reviews

Systematic Reviews


Systematic reviews1

Systematic reviews

Overall, legitimacy interventions resulted in a large, significant increase in positive perceptions of police.

Taken from: Mazerolle, L., Bennett, S., Davis, J., Sargeant, E. and Manning, M. (2013) Procedural Justice and Police Legitimacy: A Systematic Review of the Research Evidence. Campbell Systematic Reviews 2013:1.


Policy opinions of systematic reviews

Of those that had heard of them, some had concerns:

Timeliness

Relevance

Usefulness

Some had difficulty distinguishing them from literature reviews, even when explained

Campbell S et al (2007) Analysis for policy: evidence-based policy in practice

Policy opinions of systematic reviews


Rapid evidence assessments

Use systematic principles and the same process as a systematic review…

…but make compromises given available time and resources

Pragmatic and transparent approach

Rapid Evidence Assessments


The process in a nutshell

Draft search terms

Draft sift criteria

Sift received abstracts

Request relevant papers

Read and ‘grade’ papers

Write it up (‘synthesis’)

The process (in a nutshell)


Some key principles

Demonstrate consistency in searching/ sifting

Document search and sift process

Process should be transparent and repeatable

Specify required quality of evidence

Systematic reviews only?

Pre-post studies only?

All ‘empirical’ papers?

Be explicit/ transparent about the limitations of the approach

Some key principles


Some examples what it is like in practice

Some examplesWhat it is like in practice?


One practical example

Review of Police Leadership and Training commissioned by the Home Secretary

CC Peter Neyroud needed the best evidence he could get on “What works in training and behaviour change?”

Three weeks to deliver an evidence review

Not a full REA, but…

More than literature or scoping reviews

Used systematic principles

Due to time limit, search restricted to evaluations and systematic reviews only

One practical example


Searching training

Search terms

training OR learning OR development

AND evaluat* OR assess* OR what works OR impact

AND systematic review

Limitations

Searched 11 databases and 2 websites

English language only

Systematic review

Training

Evaluation

Searching (Training)

Initial search identified 1,015 abstracts to sift


Sifting training

Sifting (Training)

  • Q1: Is the study about adult training, learning, or development?

  • Q2: Is the study: An evaluation (at least pre & post level) OR a systematic review?

  • Secondary sift to remove papers related to inappropriate populations and specific medical conditions

38

32

22

10


Read appraise and synthesise papers what works in training

Read, Appraise and Synthesise papers – What works in training?


Limitations

Time

Availability of papers (10/22)

Unable to pilot search terms

Available databases

Only those available to the NPL

English language only

Literature focussed in different areas

Almost nothing on policing

Papers mostly from healthcare

Limitations


Organisational change and business improvement

Forces are adopting business improvement techniques to examine current practices and explore scope to change processes to release savings

Techniques include QUEST, CI, Lean, Six Sigma, Kaizen, etc.

Organisational change and business improvement

  • So – are these techniques the answer?

  • Is that magic potion? Or is it snake oil?


Rapid evidence assessment

Rapid Evidence Assessment

Step 1

Systematic search

(11,960 abstracts)

Step 2

Quality assess & critically appraise

(181 empirical studies)

Step 3

Synthesise findings

(41 studies with useful findings)

Draw conclusions

Example of the two searches in the organisational change and business improvement REA


Organisational change and business improvement1

Organisational change and business improvement

Total of approx 12,000 abstracts; 181 full papers were requested; 41 were included…

  • Why so many papers?

    • Better search terms needed?

    • Dodgy descriptions of papers in abstracts?

    • Problems with searching?


Organisational change what works

Organisational change – what works?

Potential success factors for organisational change


Organisational change potential success factors

Organisational change: potential success factors

Leadership

stability of supervision throughout implementation

direct support from supervisors - ‘on-the job’ training

staff involvement in decisions

transformational leadership behaviour = reduced employee cynicism

Engagement

staff active participation in decision making & ‘room to experiment’

degree to which staff understood rationale for change

communication found to influence self reports of job performance


Lessons learned

Piloting your searches is critical (we’ll do this tomorrow) to give you an idea of the size of the job, and if there is much available material

It’s important to focus on the end result and how will the findings be used – make it relevant…

Recognise (and accept) there are limitations of the approach

Make sure you are researching the right question

Follow the key principles:

Demonstrate consistency in searching/ sifting

Document search and sift process

Specify required quality of evidence

Be explicit about the limitations of the approach

REAs should be replicable

Lessons learned...


The process for evidence base camp

The process (for Evidence Base Camp)


What is a rapid evidence assessment what s involved

  • Search terms are hugely important.

  • Tomorrow is all about developing search terms and seeing the impact of using different terms.

  • We will be running live pilot searches in your groups with College librarians.


Useful links

Civil service REA toolkit:

http://www.civilservice.gov.uk/networks/gsr/resources-and-guidance/rapid-evidence-assessment

Campbell Collaboration (social interventions, e.g. crime and justice)

http://www.campbellcollaboration.org/

Cochrane Collaboration (medical interventions)

http://www.cochrane.org/

EPPI centre (education)

http://eppi.ioe.ac.uk/cms/

Useful links


  • Login