Literature Reviews

1. Literature Reviews PRA August 6, 2009

2. What is a lit review? • Thematic description (in prose) organized around and related directly to • the issue rationale and relevance in an evaluation framework • a survey of a specific question or policy issue • Synthesis of all relevant information, a coherent summary of what is and is not known • Identification of areas of debate, controversy, and uncertainty

3. A literature review… • Involves “the use of ideas in the literature to justify the particular approach to the topic, the selection of methods, and demonstration that this research contributes something new” • Quality (in a literature review) means appropriate breadth and depth, rigor and consistency, clarity and brevity, and effective analysis and synthesis

4. Four features of a literature review • “methodologically analyze and synthesize quality literature, • provide a firm foundation to a research topic, • provide a firm foundation to the selection of research methodology, and • demonstrate that the proposed research contributes something new to the overall body of knowledge or advances the research field’s knowledge-base” {Levy, Yair, and Ellis, Timothy J. A Systems Approach to Conduct an Effective Literature Review in Support of Information Systems Research. Informing Science Journal. Volume 9, 2006}

5. Varieties of review conducted at PRA (1) • Survey of the state of the art (big picture) • Lessons Learned Background Report - Lessons Learned on Employment, Labour Market, and Economic Development Policies, Programs, and Services for Aboriginal Peoples (Published) - March 1999http://www.hrsdc.gc.ca/eng/cs/sp/hrsdc/edd/reports/1999-000383/page00.shtml • Reconnecting Social Assistance Recipients to the Labour Market - March 2000 http://www.hrsdc.gc.ca/eng/cs/sp/hrsdc/edd/reports/2000-000437/page00.shtml • Gender Equality in the Labour Market - October 2002http://www.hrsdc.gc.ca/eng/cs/sp/hrsdc/edd/reports/2002-002440/page00.shtml • These are stand-alone pieces, designed to review a major policy or idea and are commissioned independently of any evaluation. • They often have external experts to provide context and advice.

6. Varieties of review conducted at PRA (2) • Literature reviews as part of evaluations These can range from short (10-page) reviews to major surveys • Short Review ExampleEvaluation Assessment of the Sport Canada Program F:\PROJECTS\CANADA\HERITAGE\034(Eval Assessment Sport Canada)\REPORTS\Lit Review\rptLiterature Review_V5_Logo.wpd • Long Review Example Business Risk Management in Canadian Agriculture: History and Foundations F:\PROJECTS\CANADA\HERITAGE\034(Eval Assessment Sport Canada)\REPORTS\Lit Review\rptLiterature Review_V5_Logo.wpd • Most literature reviews at PRA explore the theoretical foundations of a policy or program

7. Varieties of review conducted at PRA (3) • Literature reviews help define an effective methodology • This can optimise methods and identify new, more efficient techniques • “Conducting an effective literature review that will yield a solid theoretical foundation should also provide a firm foundation to the selection of the methodology for the study” • Levy, Yair and Ellis, Timothy J. (2006)

8. Steps in writing the lit review • Define the scope and role of the review • Perform the initial scan • Create the framework or paradigm • Complete a detailed search of material • Archive and organize the retrieved material • Develop a detailed outline • Draft - redraft

9. 1. Define the scope and role of the review • Establish expectations with the client • Reviews that are part of evaluations speak directly to rationale and design • Rationale discusses alternatives for dealing with a problem. Confirm with the client that this fundamental questioning of programming or policy is desired. • Design will often look at varieties of implementation – experiences of other countries/provinces are often required. • Budget defines scope • A stand-alone review with experts will cost in excess of $100K • An evaluation with a $50,000 budget can only undertake a cursory literature summary (15 – 20 pages drawing on a specific issue, or summarizing at a high level). Make sure proposals are clear on this.

10. Ensure that the client confirms the scope • Review of academic and theoretical foundations for a policy • Review of the history of a policy • Review of practices/experience elsewhere • Review of the options, with a discussion of pros and cons

11. 2. Perform the initial scan(Orientation) • This step identifies articles that generally survey the field. • You need to identify about 4 – 5 decent general reviews to become familiar with the issues. • Extract the bibliographies to identify seminal articles, journals, and institutes. • Sometimes the client has material from the initial program design. • This familiarization can also be supported by consultations with experts, partners, and consulting associates. • Identify keywords and the framework (see below). • Wikipedia can serve as a source of information, but never cite it as a reference.

12. Reading an article • Never read the article/book/report in its entirety. • Read the abstract, introduction, methodology, and conclusion. • Two tricks: • Copy the citation and abstract into Word to offer a summary of the core ideas. This can become a useful source document for the client and you, but make sure you note that the abstracts are quoted verbatim. • Use Zoterra to make annotations and create tabs to allow you to classify the material. But, make sure you still store the original material in an accessible format. • Skim the article for interesting facts and quotes.

13. 3. Create the framework or paradigm • Every literature review must rest on a framework • This framework connects the different schools of thought or theories • The contrast between various elements or theories can be seen as variations along a spectrum, or as different branches • Each discipline has several frameworks that can support the literature review • Create a table of contents early and “dump” interesting ideas, quotes, and references into the evolving structure

14. Key points to remember at an early stage • If you cut and paste text, always highlight in a consistent colour (red) to remind yourself. • Build the bibliography as you proceed. You can always delete material as you go. • Always use full citations in the APA style, with page numbers and volume/issue. • In most journal articles, you can cut the citation from the header into a Word document. • VIP – Store the articles under a folder entitled “Literature Review”, with thematic subfolders and informative titles. • You can cut and paste titles or title fragments from PDF documents into the file name.

15. Examples of frameworks • Labour market policy Operates on the demand (inducing firms to hire) or supply (training) or market (increasing information on jobs) • Greenhouse gas policy • Regulation on emissions • Incentives (subsidies) to use less energy (cash for clunkers, purchase hybrids, wind power generation) • Taxes to increase the prices of fossil fuels relative to renewable sources • Moral suasion (exhortations to be better)

16. 4. Complete a detailed search of material Google Scholar is the first step • Use keywords (these are listed in many articles) • Vary the keywords, based on the initial scan • Use the Google proxy server to locate PDF articles quickly http://scholar.google.ca.proxy1.lib.umanitoba.ca/ Boolean logic refines the search Example: • Greenhouse gas remediation • “Greenhouse gas” “carbon tax” • “Greenhouse gas” “cap and trade” not “United States” Limit the search to articles after 2000 – key material prior to this will be frequently cited in the recent material. This is just the first step in shaking the literature tree!

17. Follow key journals and publishers • Once you find a journal that is actively publishing material, search the journal online • Other sources include Health (PubMed) http://www.pubmed.com Blackwell (Wiley Inter-science) http://www.wiley.com Sage Publications http://www.sagepub.com

18. Best practices elsewhere • Use international institutes OECD (Organization for Economic Cooperation and Development) http://www.oecd.org World Health Organization http://www.who.org International Labour Organization http://www.ilo.org United Nations Educational Scientific and Cultural Organization (UNESCO) http://www.unesco.org

19. Forward and backward searches • Backward searches • backward references search (search the bibliography to the articles, especially seminal material) • backward authors search (search for authors cited frequently in the bibliography of the seminal material) • previously used keywords (search the keywords used in the seminal articles) • Forward searches • forward reference search (search for articles that have cited the seminal reference) • forward author search (search for articles that have cited the authors of the seminal reference)

20. When is the searching done? • As you find fewer new articles • As you find the articles are repeating familiar arguments However …. Be alert for new material at all stages of the study, and be prepared to augment the literature review.

21. Archive and organize the retrieved material • Create a “literature review” folder under “PROGDATA” • Create meaningful subheadings to group the literature F:\PROJECTS\CANADA\DIAND\032(SINED)\PROGDATA\Lit Review References\SINED\Regional Development • Be prepared to revise your groupings and categories of articles

22. Develop a detailed outline • A detailed table of contents is mandatory • This must consist of sections and subsections with a point form discussion of the contents of each section • Negotiating this at the start avoids “scope creep” • If peer reviewers are involved, make sure they sign-off on the detailed table of contents

23. Draft - redraft

24. Plagiarism – a last word To plagiarize is to take ideas or words of another person and pass them off as one’s own. In short, it is stealing something intangible rather than an object. Plagiarism applies to any written work, in traditional or electronic format, as well as orally or verbally presented work. (Source: University of Manitoba, Academic Regulations http://webapps.cc.umanitoba.ca/calendar10/regulations/plagiarism.asp) Steps to prevention: • Make sure to tag/highlight all copied material • Reference quotes accurately • It is reasonable to use a general reference such as “This section draws from X and Y... • Reproducing web-based material in any report is plagiarism Exception: When we work on certain documents for a client, such as a manual, RMAF/RBAF, or other internal document, we may lift directly without attribution. But, be safe and consult with the client.