What can systematic reviewing tell us about labour markets?

1. What can systematic reviewing tell us about labour markets? Dr Sally-Anne Barnes Warwick Institute for Employment Research University of Warwick Sally-Anne.Barnes@warwick.ac.uk 4th ESRC Research Methods Festival 5-8 July 2010

2. Aim of presentation • To give an overview of the systematic literature review process • To highlight advantages and disadvantages of the process • To examine the lessons learnt and what the process can tell us about the labour market

3. Brief history of the systematic literature review • History in evaluations of health interventions (Cochrane approach) • Move to applications in public policy evaluations (Campbell Collaboration) • Applied to fields of education and social welfare (EPPI Centre)

4. Phases of the systematic review

5. Advantages of systematic review process • Comprehensive and objective assessment of available research • Consistent approach by research team • Reduction of reviewer bias • Selection and screening of articles monitored and recorded – transparency • Provides a sound evidence base • Identification of potential gaps in existing research

6. Disadvantages of the process for researching the labour market • Downplays the role of reviewer and professional judgement • Exclusion of unpublished reports and thesis • Does not build on other reviews that can be useful • Does not search for online materials • Regimented approach which could exclude interesting research

7. Skills diagnostics and screening tools: A literature review • Based on two review questions • Q1. Approximately 30,000 references identified, 579 screened, 29 used for study • Q2. Approximately 82,000 references identified, 1,131 screened, 40 used for study • Team of 11 • 8 months to conduct and write-up review

8. Adapting the methodology

9. Phase 1: searching Aim: the systematic identification of potentially relevant studies • Identify keywords • Develop search strategy • Search electronic databases for articles • Record search results

10. Phase 2: screening Aim: the application of pre-determined inclusion and exclusion criteria to articles • Initial screening online • Transfer results into a bibliographic software programme • Application of inclusion and exclusion criteria • Record screening results

11. Phase 3: data extraction Aim: examination of studies to assess quality and extract evidence • Develop and use data extraction instrument • Obtain full document • Complete data extraction for each document • Record results

12. Phase 4: synthesis Aim: synthesis of findings and identification of key themes • Develop framework for data analysis • Assess quality of studies • Synthesise findings using data extractions • Identify key themes and conclusions

13. Phase 5: reporting and dissemination Aim: presentation of review findings • Identify studies that could not be obtained • Present record of searches, screening and studies used in review • Identify potential gaps in research and possible areas of future research

14. Lesson learnt • Not following methodology uncritically • Exercising professional judgement • Importance of drawing upon own knowledge and networks • Acknowledging researcher is an integral part of process

15. Conclusions… • Adds to our understanding • Real world perspective • Historical changes and trends • Identifies gaps in LM research • But with adapted methodology… • Identificationof unpublished research • Inclusion of ongoing studies and evaluations • Current view