Systematic literature reviews

What is a systematic literature review?

A systematic literature review (SLR) is a type of literature review that uses repeated analytical methods to collect and analyse primary research studies. SLRs are:

…syntheses of primary research studies that use (and describe) specific, explicit and therefore reproducible methodology strategies to identify, assemble, critically appraise and synthesise all relevant issues on a specific topic (Carney and Geddes 2002)”

SLRs began in health care research to compare different interventions (for example, a new drug or treatment) across different applications and jurisdictions. SLRs have now been adopted as a method across a wide range of different disciplines and areas of interest. Below are some examples of SLRs in environmental research.

Conservation – Brooks et al. (2013) – Assessing community-based conservation projects: A systematic review and multilevel analysis of attitudinal, behavioral, ecological, and economic outcomes.

Forest Ecology – Häkkilä et al. (2021) – Are small protected habitat patches within boreal production forests effective in conserving species richness, abundance and community composition? A systematic review.

Soil Science – Beckhanova et al. (2021) – Biochar’s effect on the ecosystem services provided by sandy-textured and contaminated sandy soils: a systematic review protocol.

Climate Change – Galama and Scholtens (2021) – A meta-analysis of the relationship between companies’ greenhouse gas emissions and financial performance.

Environmental Economics - Garrett et al. (2021) – Have food supply chain policies improved forest conservation and rural livelihoods? A systematic review.

What type of systematic literature review should I use?

In a SLR, researchers create a codebook (also known as a Systematic Literature Review Protocol) which contains a list of data you intend to collect from each paper included in your review. There are a range of different approaches to codebooks in SLRs. If you plan on publishing your SLR in a journal, make sure you know the codebook they prefer to use before starting your review. Some codebook approaches include:

PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) – the standard method for collecting and reporting data in an SLR. Used by papers published in the journal Environmental Research Letters.

ROSES (Reporting standards for Systematic Evidence Syntheses) - an SLR codebook method devised specifically for environmental research. Papers published in the journal Environmental Evidence use ROSES exclusively.

Steps to conduct a Systematic Literature Review

Step 1: Develop your research question

SLRs are designed to answer a clearly formulated question. The typical permutation of a reviewable question is: Does intervention x on subject y produce outcome z?

For example, in a recent SLR researchers asked: “Are there empirical examples of agricultural land use change as an adaptation response?”. In this example, the intervention is agricultural land use change, the subject is farmers, and the outcome is agricultural land use change as an adaptation response.

Step 2: Devise article screening criteria and SLR codebook

To select relevant papers, create inclusion and exclusion criteria for your search. See below for an example of inclusion and exclusion criteria given for the SLR on agricultural land use change as an adaptation response.

A codebook contains a list of data you intend to collect from each paper. Choose from methods such as PRISMA and ROSES presented earlier.

Step 3: Develop a search chain

Various search chains are often developed and tested before the final search chains are confirmed. Use wildcard and truncation symbols if the database you are search permits their use.

• (?) is useful for retrieving alternate spelling and simple plurals (wom?n will find woman or women)
• () is useful for finding different word endings based on the root of a word (teen will find teen, teens, teenager, teenagers)
• (“ “) is useful for finding exact phrases (“electronic cigarettes” will find exact uses of electronic cigarettes. These exact phrases can be linked with AND or NOT)

In the agricultural land use change example, two search chains were deployed:

Search 1: Land use chang adapt agricultur

Search 2: “land use” AND “chang” AND “adapt” AND “agriculture”

Step 4: Select and search databases

It is best practice to deploy search chains in multiple databases. Potential databases include ISI Web of Science, Scopus, Google Scholar, JSTOR, ResearchGate.

However, some recent SLRs have justified the use of ISI Web of Science as the sole database. In the agricultural land use change SLR, we deployed our two search chains in ISI Web of Science. These two searches yielded 3898 citations.

Step 5: Export searches to endnote or other citation/abstract library

When your search yields multiple hundreds or thousands of citations, it can be difficult to keep a track of them. We suggest exporting searches to a citation library for: (1) ease of control, (2) offline storage, (3) ability to quickly delete duplicate citations, (4) ability to select and delete publication types (e.g. conference proceedings, book sections, books, etc.).

Step 6: Delete duplicates, search titles and abstracts for appropriate papers

When deploying multiple searches, or searching multiple databases, the search results will often contain duplicate citations. These can be easily identified and deleted using endnote. Of the 3898 citations found during the agricultural land use change search, 1610 were duplicates that were deleted.

After deleting duplicates, researcher’s must then scan the titles and abstracts of the remaining citations, and using the screening criteria in step 2, choose what papers to include and to exclude.

In the agricultural land use change example, we excluded 2262 studies from consideration during this step. This left 62 potentially relevant studies.

Step 7: Read remaining papers and delete those that are not relevant to the research question

At this step, all remaining studies should be closely reviewed to see if they meet the inclusion criteria. In our agricultural land use example, the 62 remaining citations were split between two researchers. Each read 31 papers and decided if they should be excluded or included. They initially agreed to include 33 papers, exclude 15, and were unable to agree on the final 14. These final 14 papers were re-read by the researchers who agreed to add five to the final review.

Following the screening, 38 studies were selected for the review. The process is detailed in the image below which is known as a PRISMA flowchart. All SLRs conducted using the PRISMA guidelines should create a similar flowchart to visually detail the process.

Step 8: Fill in SLR codebook

The final step in completing a SLR. The data that you will need to collect to complete the SLR codebook depends on the reporting guidelines that you have chosen (e.g. PRISMA, ROSES etc.).

Once this is complete, contemplate an analysis strategy. For example, SLRs that collect scientific results on similar studies could be analysed through a meta-analyses. SLRs can also be analysed qualitatively using content analysis or similar methods. What analysis methods you choose will depend on the research question as well as the skills of the researchers involved.

Further information

Download this presentation outlining How to complete a Systematic Literature review (Powerpoint).

References

Bekchanova, M., Campion, L., Bruns, S., Kuppens, T., Jozefczak, M., Cuypers, A., & Malina, R. (2021). Biochar's effect on the ecosystem services provided by sandy-textured and contaminated sandy soils: a systematic review protocol. Environmental Evidence, 10(1), 1-12.

Brooks, J., Waylen, K. A., & Mulder, M. B. (2013). Assessing community-based conservation projects: a systematic review and multilevel analysis of attitudinal, behavioral, ecological, and economic outcomes. Environmental Evidence, 2(1), 1-34.

Carney, S., & Geddes, J. (2002). Systematic reviews and meta-analyses. Evidence in mental health care, 73-80.

Deubelli, T. M., & Mechler, R. (2021). Perspectives on transformational change in climate risk management and adaptation. Environmental Research Letters, 16(5), 053002.

Galama, J. T., & Scholtens, B. (2021). A meta-analysis of the relationship between companies' greenhouse gas emissions and financial performance. Environmental Research Letters, 16(4), 043006.

Garrett, R. D., Levy, S., Gollnow, F., Hodel, L., & Rueda, X. (2021). Have food supply chain policies improved forest conservation and rural livelihoods? A systematic review. Environmental Research Letters.

Häkkilä, M., Johansson, A., Sandgren, T., Uusitalo, A., Mönkkönen, M., Puttonen, P., & Savilaakso, S. (2021). Are small protected habitat patches within boreal production forests effective in conserving species richness, abundance and community composition? A systematic review. Environmental Evidence, 10(1), 1-20.

Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009). Systematic literature reviews in software engineering–a systematic literature review. Information and software technology, 51(1), 7-15.

Scully, M. J., Norris, G. A., Falconi, T. M. A., & MacIntosh, D. L. (2021). Carbon intensity of corn ethanol in the United States: state of the science. Environmental Research Letters.

Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS quarterly, xiii-xxiii.