Conduct a systematic review: step-by-step guide: Home

You may already have a clear scope of what your research is about and what is expected to be included and excluded in your project. However, a more promising way of starting a research project would be to not only have a topic, scope and a rough question in mind. Rather, your research team really needs an answerable research question to start with. Check out the tool box below for examples of frameworks used to formulate answerable research questions. 

Toolbox

Some frameworks are more easily applied to certain types of research than others. However, It is not the case that one framework has to be rigidly applied to a particular type of research without any flexibility. The examples given are just for guidance.  Which framework can be helpful for building your research question is highly dependant on your research objective(s) and the focus of your review.

Next, you will have to select the appropriate resources to search for the available evidence. This tab shows you some tips and tricks on how to build your search strategies and how conduct database searches.

  When applying the above steps, It is important to note that building a good search strategy would not be a straightforward process. Along the way, you will need to try out different combinations and review the number of search results and quickly scan through article titles. Then, if necessary, adjusting your keywords and combinations to make the search results more relevant to your research question.

As explained above, subject headings and limits setting are vary from database to database and search platform to search platform. Therefore, the first developed search strategy upon one database via a specific search platform would need to be modified to make it applicable for the others.

Need more help? 

  Book a Training Session

  Request a Literature Search

After you are able to clearly define your research topic, objectives, scope, type of review and research question, you will need to write a protocol of your planned systematic review, which should provide the research methods that are reproducible.

JBI. (2020,September 25). Pre-planning and protocol development for systematic reviews [video]. Youtube. https://youtu.be/K8Jasx0FLys

You may also read this BioMedCentral blog on why protocols are important.

The structural requirements of a protocol can be vary depending on the publishers. Generally, the protocol should consist of the following content / sections: 

(also see the Toolbox at the bottom of this tab to see the structure requirements of protocol based on the publishing partners)

• Title and Authors
• Abstract
• Introduction
• Review objectives
• Research methods
• Acknowledgements
• Conflict of Interest
• Funding / Ethical approval (if any)

Your research methods should at least include:

(1) keywords / key terms / key concepts and selection criteria (inclusion and exclusion criteria) of literature as well as databases and /or platforms your team would use for searching. At this stage, you may not have a detailed search strategy for each database you planned to use but you would still need to include a search strategy section. (The first example below shows you an example with a search strategy section and the second example paper below shows you a search strategy section with a detailed search strategy in Appendix 1.)

(2) how you will validate the findings of the included studies - What are the research design measurements that will minimise the bias of your research? (see the examples of systematic reviews with different risks of bias provided by KSE Evidence). You will need validators (ideally, at least 2-3) to review the findings of the included studies so the systematic review can be produced with as little bias as it can be. The validators are expected to have knowledge of the fields / be experienced workers or, even better, experts of the relevant fields. The validators should be able to review the findings of the included studies independently.

(3) how is your team going to synthesis evidence / findings in a systematic way? This may be as a thematic approach and/ or a meta-ethnographic analysis for qualitative reviews, and meta-analysis for quantitative reviews or a mixed-method reviews etc.(see Step 5 Data Extraction section for more details)

Examples of systematic review protocol:

Pohontsch, N. J., Meyer, T., Eisenmann, Y., Metzendorf, M. I., Leve, V., & Lentsch, V. (2021). Study protocol of a systematic review and qualitative evidence synthesis using two different approaches: Healthcare related needs and desires of older people with post-stroke aphasia. BMJ Open, 11(4), e039348.

Gallant, A. J., Steenbeek, A., & Curran, J. A. (2022). Identifying COVID-19 and H1N1 vaccination hesitancy or refusal among health care providers across North America, the United Kingdom, Europe, and Australia: a scoping review protocol. JBI Evidence Synthesis, 20(1), 173-180.

Toolbox

• The suggested structure of a Cochrane protocol for submission ; an example from the Cochrane Handbook for Systematic Reviews of Interventions (updated February 2022)

• Methodological Expectations of Cochrane Intervention Reviews - The MECIR manual

• PRISMA Protocol Guidance including PRISMA for systematic review protocols (PRISMA-P) 

• PRISMA Flow Diagram generator  - click on "Create flow diagram" on the horizontal manual will bring you to the generator. If you are using R for meta-analysis. There is also a R package link provided by Evidence Synthesis Hackathon.

• Template for Scoping Review Protocols / JBI

• Guide on Registering a review on PROSPERO (does not accept scoping reviews)

References:

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane, 2022. Available from www.training.cochrane.org/handbook.

At this stage, your team can apply the set of search terms / search strategies that have been tested, modified, and then finalised, to conduct the search, review the search results, and well-document the review and selection process. Here is a step-by-step guide:

1. Conduct the search using your final set of search terms / search strategies. It is recommended that your team should do the search in at least two to three publication databases. As explained under the Step 3 tab, each set of search terms / search strategies may not be identical from database to database. However, they should be similar in covering all key concepts and related terms as possible for each database.

2. To review the search results from each database, it is recommended to start with scanning all articles where the title seems relevant. Then import those citations and abstracts into a citation management software (such as EndNote, Mendeley, Zotero or similar) for further review. (Looking for EndNote training? Visit our Referencing and Citation Management Tools Guide for more details.)

3. As there is overlap in the coverage between databases, and you may possibly retrieve a large number of records from each database, citation management software can also help to store a large number of citations from each database and to de-duplicate records.  

4. To continue the review and selection process: Read the abstract for all articles and keep the articles that seem relevant. To further shortlist highly relevant research, retrieve and read the full-text of articles of those with abstracts that seemed relevant. It needs to be at least 2 members of the SR team to review papers for inclusion and exclusion, to reduce bias in the process (learn more about critical appraisal from the next tab - Step 6). The members will need to review papers independently. Papers where there is disagreement on whether to include or exclude are then reconsidered.

5. Through out the process of review and selection, your team will need to well-document: (1) the initial number of articles your retrieved from each database after de-duplication; (2) the number of  articles, from each database, that ended up included and excluded and (3) why some titles are excluded from your paper (generally, the reasons of including or excluding an article should be matched with your proposal). All those numbers will need to be summarised in the PRISMA flow chart and included in your paper. The process will need to be reported under the Method section of your paper.

6. Review the reference lists within the remaining relevant articles in case there are a few more relevant articles listed that were not found through your database searches. If any additional articles are found, Include them in the final list of relevant articles and document it.

In addition to citation management software there are free online platforms for team collaboration. These platforms usually allow reviewers to upload articles and provide functions such as labelling articles (with "included", "excluded" or "maybe") and enable reviewers to provide feedback or comments to help facilitate the selection process and keep the all reviewers of the research team on the same page. Again, the reviewers will need to review papers independently. Papers where there is disagreement on whether to include or exclude are then reconsidered

Free Open Sources for systematic review team collaboration:

• Colandr​ (sign up required)
• HAWC (Health Assessment Workplace Collaborative)
• LitStream (from ICF International)
• PICO Portal (free version available - registration requested)
• Rayyan (registration requested)
• Systematic Review Data Repository (SRDR+) (registration required) (from AHRQ, US Government)

  It is important to note that while there are a number of free open source tools available, you may be restricted to the tools / evidence synthesis software for completing this data extraction process by the requirements of publishing partners. Campbell Collaboration is an example. Therefore, you should always check with the publisher / organisation you are going to submit your review with before you plan the screening, data extraction and analysis process of your review.

To appraise research articles, there are 2 overarching questions to keep in mind:

Critical Appraisal

Questions for assessing how serious the risk of bias is , and how precise the results are, will vary according to the different research design aspects and statistical measurements used in a research study. Unfold the following tabs to access to the list of questions and the Cochrane videos relevant to the different types of research studies.

Toolbox:

The guidelines for reporting a systematic review that you should follow will vary depending on the publisher and the type of review. It is recommended that your team check the relevant publisher's website once your team has decided on where you intend to publish.

Example : "Instructions for Authors" from JAMA (scroll down to the row: "Meta-analysis" or the table "Clinical Review" to see what type of reviews and what reporting guidelines your team should follow).

The following toolbox includes a reporting checklist, a template for flow diagrams and some common reporting guidelines.

Toolbox

• PRISMA 2020 Checklist for guiding the structure of your paper.
• PRISMA Flow Diagram which provides templates, depending on the type of review and databases used, for building the flowchart / diagram you may want to include in your paper.
• Article about PRISMA 2020 and PRISMA-S: common questions on tracking records and the flow diagram.

Rethlefsen, M. L., & Page, M. J. (2022). PRISMA 2020 and PRISMA-S: common questions on tracking records and the flow diagram. Journal of the Medical Library Association: JMLA, 110(2), 253.

       Reporting guidelines apply for systematic review with a meta-analysis:

• PRISMA Reporting Guidelines
• PRISMA Flow Diagram generator - click on "Create flow diagram" on the horizontal manual will bring you to the generator. If you are using R for meta-analysis, there is also a R package link provided by Evidence Synthesis Hackathon.
• MOOSE (Meta-analysis of Observational Studies in Epidemiology) Reporting Guidelines

       Reporting guidelines apply for synthesis without meta-analysis (SWiM):

• Synthesis without meta-analysis (SWiM): in systematic reviews: reporting guideline
• EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Reporting Guidelines