systematic plan in research

Systematic Review

• Library Help
• What is a Systematic Review (SR)?

Steps of a Systematic Review

• Framing a Research Question
• Developing a Search Strategy
• Searching the Literature
• Managing the Process
• Meta-analysis
• Publishing your Systematic Review

Forms and templates

Image: David Parmenter's Shop

• PICO Template
• Inclusion/Exclusion Criteria
• Database Search Log
• Review Matrix
• Cochrane Tool for Assessing Risk of Bias in Included Studies

   • PRISMA Flow Diagram  - Record the numbers of retrieved references and included/excluded studies. You can use the Create Flow Diagram tool to automate the process.

   •  PRISMA Checklist - Checklist of items to include when reporting a systematic review or meta-analysis

PRISMA 2020 and PRISMA-S: Common Questions on Tracking Records and the Flow Diagram

• PROSPERO Template
• Manuscript Template
• Steps of SR (text)
• Steps of SR (visual)
• Steps of SR (PIECES)

Adapted from  A Guide to Conducting Systematic Reviews: Steps in a Systematic Review by Cornell University Library

Source: Cochrane Consumers and Communications  (infographics are free to use and licensed under Creative Commons )

Check the following visual resources titled " What Are Systematic Reviews?"

• Video  with closed captions available
• Animated Storyboard
• << Previous: What is a Systematic Review (SR)?
• Next: Framing a Research Question >>
• Last Updated: Mar 4, 2024 12:09 PM
• URL: https://lib.guides.umd.edu/SR

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Methodology
• Systematic Review | Definition, Examples & Guide

Systematic Review | Definition, Examples & Guide

Published on 15 June 2022 by Shaun Turney . Revised on 17 October 2022.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesise all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question ‘What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?’

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs meta-analysis, systematic review vs literature review, systematic review vs scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce research bias . The methods are repeatable , and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesise the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesising all available evidence and evaluating the quality of the evidence. Synthesising means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Prevent plagiarism, run a free check.

Systematic reviews often quantitatively synthesise the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesise results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarise and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimise bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimise research b ias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinised by others.
• They’re thorough : they summarise all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fourth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomised control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective(s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesise the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Grey literature: Grey literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of grey literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of grey literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Grey literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarise what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgement of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomised into the control and treatment groups.

Step 6: Synthesise the data

Synthesising the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesising the data:

• Narrative ( qualitative ): Summarise the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarise and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analysed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Turney, S. (2022, October 17). Systematic Review | Definition, Examples & Guide. Scribbr. Retrieved 2 April 2024, from https://www.scribbr.co.uk/research-methods/systematic-reviews/

Is this article helpful?

Shaun Turney

Other students also liked, what is a literature review | guide, template, & examples, exploratory research | definition, guide, & examples, what is peer review | types & examples.

Get insightful educational articles from the world of academia for researchers, students and authors. Visit Editage Blog for helpful content and tips on getting published and writing articles that are up to international journal publication standards. Click here to find out more!

• Get Published

Conducting and Reporting Systematic Reviews: A Guide for Medical and Life Sciences Researchers

• Pin it

In the world of medical and life sciences research, staying up to date on the latest evidence-based research is crucial. Systematic reviews are a powerful tool that can help researchers identify gaps in current research and plan future studies. In this article, we’ll provide a comprehensive guide to conducting a systematic review, including what it is, how to plan and conduct a systematic review, and how to report the results. We’ll also provide several examples to help illustrate the key concepts and steps involved in conducting a systematic review. 

What is a Systematic Review?

A systematic review is a scholarly synthesis of existing research on a specific topic, wherein such research is identified through a comprehensive search strategy developed a priori.  

Systematic reviews use a comprehensive and systematic approach to identify, evaluate, and synthesize evidence on a specific research question. This approach helps ensure that the review is unbiased and that all relevant studies are included. The end result is a clear and comprehensive summary of the available evidence, which can inform the design and conduct of future studies. 

Step 1: Define the Research Question

The first step in conducting a systematic review is to clearly define the research question. This should be a clear and concise statement that specifies the population, intervention, comparator, outcome, and study design (PICOS) of interest. For example, a research question could be “What is the effectiveness of drug A compared to placebo in reducing blood pressure in patients with hypertension?” 

Step 2: Develop the Protocol

Once the research question has been defined, the next step is to develop a protocol for the systematic review. This should include a detailed plan for how the review will be conducted, including the search strategy, inclusion and exclusion criteria, and methods for data extraction and analysis. The protocol should also specify the quality criteria that will be used to evaluate the included studies and the methods for assessing the risk of bias. 

Step 3: Conduct the Search

The next step is to conduct a comprehensive search of the literature to identify all relevant studies. This should include a search of relevant databases, such as PubMed, Embase, and Cochrane Library, as well as a hand search of relevant journals and conference proceedings. The search strategy should be developed in consultation with a librarian or information specialist to ensure that it is comprehensive and reproducible. 

Step 4: Screen the Studies

Once the search has been completed, the next step is to screen the studies to determine whether they meet the inclusion criteria. This typically involves a two-stage process, with an initial screening of titles and abstracts followed by a full-text review of the remaining studies. The inclusion criteria should be clearly defined in the protocol and should specify the PICOS of interest. 

Step 5: Extract Data and Assess Quality

Once the studies have been selected, the next step is to extract data from the included studies and assess their quality. This should be done using a standardized data extraction form and quality assessment tool, which should be developed as part of the protocol. The data extraction form should include information on the study design, population, intervention, comparator, outcome, and results, as well as any other relevant information. 

Step 6: Synthesize the Results

The final step is to synthesize the results of the included studies. This typically involves a quantitative synthesis, such as a meta-analysis, if the studies are sufficiently homogeneous. If the studies are too heterogeneous, a qualitative synthesis may be more appropriate. The results should be presented in a clear and concise manner, with appropriate measures of effect size and confidence intervals. 

Reporting the Results

The results of the systematic review should be reported in a transparent and comprehensive manner. The report should follow relevant reporting guidelines, such as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement, and should include a description of the methods used, the results of the review, and a discussion of the implications of the findings. 

Systematic Review Examples

To help illustrate the key concepts and steps involved in conducting a systematic review, we’ll provide several examples of research questions and how they might be approached using a systematic review. 

Example 1: What is the Effectiveness of Drug A Compared to Placebo in Reducing Blood Pressure in Patients with Hypertension?

To answer this research question using a systematic review, the following steps might be taken: 

• Define the research question : What is the effectiveness of drug A compared to placebo in reducing blood pressure in patients with hypertension? 
• Develop the protocol: The protocol might include a comprehensive search of relevant databases, such as PubMed, Embase, and Cochrane Library, using appropriate search terms. The inclusion criteria might specify that only randomized controlled trials (RCTs) are included and that the population of interest is patients with hypertension who have not previously been treated with drug A. The data extraction form might include information on study design, population, intervention, comparator, outcome, and adverse events, as well as information on the quality of the studies. 
• Conduct the search: The search might include a combination of relevant search terms, such as “hypertension,” “drug A,” and “placebo,” as well as appropriate filters to limit the results to RCTs. 

Screen the studies : The studies would be screened based on the inclusion criteria, with an initial screening of titles and abstracts followed by a full-text review of the remaining studies. 

Extract data and assess quality : The data extraction form would be used to extract data from the included studies, including information on the population, intervention, comparator, outcome, and adverse events. The quality of the studies would be assessed using an appropriate tool, such as the Cochrane Risk of Bias tool. 

Synthesize the results: The results of the included studies would be synthesized using appropriate statistical methods, such as a meta-analysis if the studies are sufficiently homogeneous. 

Example 2: What is the Effectiveness of Mindfulness-Based Stress Reduction in Reducing Anxiety in Cancer Patients?

Define the research question : What is the effectiveness of mindfulness-based stress reduction in reducing anxiety in cancer patients? 

Develop the protocol: The protocol might include a comprehensive search of relevant databases, such as PubMed, Embase, and Cochrane Library, using appropriate search terms. The inclusion criteria might specify that only RCTs are included and that the population of interest is cancer patients with symptoms of anxiety. The data extraction form might include information on study design, population, intervention, comparator, outcome, and adverse events, as well as information on the quality of the studies. 

Conduct the search: The search might include a combination of relevant search terms, such as “mindfulness-based stress reduction,” “anxiety,” and “cancer,” as well as appropriate filters to limit the results to RCTs. 

Screen the studies: The studies would be screened based on the inclusion criteria, with an initial screening of titles and abstracts followed by a full-text review of the remaining studies. 

Extract data and assess quality: The data extraction form would be used to extract data from the included studies, including information on the population, intervention, comparator, outcome, and adverse events. The quality of the studies would be assessed using an appropriate tool, such as the Cochrane Risk of Bias tool. 

Example 3: What is the Effectiveness of Different Types of Exercise in Improving Cognitive Function in Older Adults?

Define the research question: What is the effectiveness of different types of exercise in improving cognitive function in older adults? 

Develop the protocol: The protocol might include a comprehensive search of relevant databases, such as PubMed, Embase, and Cochrane Library, using appropriate search terms. The inclusion criteria might specify that only RCTs are included and that the population of interest is older adults (65 years and older) with cognitive impairment. The data extraction form might include information on study design, population, intervention, comparator, outcome, and adverse events, as well as information on the quality of the studies. 

Conduct the search: The search might include a combination of relevant search terms, such as “exercise,” “cognitive function,” and “older adults,” as well as appropriate filters to limit the results to RCTs. 

Systematic reviews are a powerful tool that can help medical and life sciences researchers identify gaps in current research and plan future studies. By utilizing a comprehensive and systematic approach for identifying, evaluating, and synthesizing evidence, systematic reviews can help researchers make informed decisions about the design and conduct of future studies. While conducting a systematic review can be a complex process, following the steps outlined in this guide can help ensure that the review is conducted in a rigorous and transparent manner and that the results are reported in a clear and comprehensive way.

Would you like guidance from an expert statistician on how to define your study variables and conduct your analysis? Check out Editage’s  Statistical Analysis & Review Services !

Data Collection Methods for Medical and Life Sciences Researchers: A Comprehensive Guide

Top 5 Editing Services For Academic Researchers: Importance of Professional Editing

Related post

What is anova (analysis of variance), types, assumptions and uses – a quick guide for biomedical researchers, 5 essential steps to optimize your grant budget to include professional statistical analysis, retrospective chart reviews: best practices for biomedical researchers, cool biology resources to fall in love with – handy tips for academic writing, what is chi-square test and types of chi-square explained for biomedical researchers, a handy guide to random forests for big biomedical data, featured post, what is literature review definition, types and examples, how ai is changing the way research is consumed, conducted, and promoted: saudi vision 2030  , saudi vision 2030 and the kingdom’s research landscape , empowering research excellence: editage supports grantees for global research advancement , data heroes: how biostatisticians can power open science in biomedical research .

• 0 Trackbacks

There are no comment yet.

There are no trackback yet.

This website is intended for healthcare professionals

• { $refs.search.focus(); })" aria-controls="searchpanel" :aria-expanded="open" class="hidden lg:inline-flex justify-end text-gray-800 hover:text-primary py-2 px-4 lg:px-0 items-center text-base font-medium"> Search

Search menu

Bashir Y, Conlon KC. Step by step guide to do a systematic review and meta-analysis for medical professionals. Ir J Med Sci. 2018; 187:(2)447-452 https://doi.org/10.1007/s11845-017-1663-3

Bettany-Saltikov J. How to do a systematic literature review in nursing: a step-by-step guide.Maidenhead: Open University Press; 2012

Bowers D, House A, Owens D. Getting started in health research.Oxford: Wiley-Blackwell; 2011

Hierarchies of evidence. 2016. http://cjblunt.com/hierarchies-evidence (accessed 23 July 2019)

Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology. 2008; 3:(2)37-41 https://doi.org/10.1191/1478088706qp063oa

Developing a framework for critiquing health research. 2005. https://tinyurl.com/y3nulqms (accessed 22 July 2019)

Cognetti G, Grossi L, Lucon A, Solimini R. Information retrieval for the Cochrane systematic reviews: the case of breast cancer surgery. Ann Ist Super Sanita. 2015; 51:(1)34-39 https://doi.org/10.4415/ANN_15_01_07

Dixon-Woods M, Cavers D, Agarwal S Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006; 6:(1) https://doi.org/10.1186/1471-2288-6-35

Guyatt GH, Sackett DL, Sinclair JC Users' guides to the medical literature IX. A method for grading health care recommendations. JAMA. 1995; 274:(22)1800-1804 https://doi.org/10.1001/jama.1995.03530220066035

Hanley T, Cutts LA. What is a systematic review? Counselling Psychology Review. 2013; 28:(4)3-6

Cochrane handbook for systematic reviews of interventions. Version 5.1.0. 2011. https://handbook-5-1.cochrane.org (accessed 23 July 2019)

Jahan N, Naveed S, Zeshan M, Tahir MA. How to conduct a systematic review: a narrative literature review. Cureus. 2016; 8:(11) https://doi.org/10.7759/cureus.864

Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1997; 33:(1)159-174

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014; 14:(1) https://doi.org/10.1186/s12913-014-0579-0

Moher D, Liberati A, Tetzlaff J, Altman DG Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 2009; 6:(7) https://doi.org/10.1371/journal.pmed.1000097

Mueller J, Jay C, Harper S, Davies A, Vega J, Todd C. Web use for symptom appraisal of physical health conditions: a systematic review. J Med Internet Res. 2017; 19:(6) https://doi.org/10.2196/jmir.6755

Murad MH, Asi N, Alsawas M, Alahdab F. New evidence pyramid. Evid Based Med. 2016; 21:(4)125-127 https://doi.org/10.1136/ebmed-2016-110401

National Institute for Health and Care Excellence. Methods for the development of NICE public health guidance. 2012. http://nice.org.uk/process/pmg4 (accessed 22 July 2019)

Sambunjak D, Franic M. Steps in the undertaking of a systematic review in orthopaedic surgery. Int Orthop. 2012; 36:(3)477-484 https://doi.org/10.1007/s00264-011-1460-y

Siddaway AP, Wood AM, Hedges LV. How to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses. Annu Rev Psychol. 2019; 70:747-770 https://doi.org/0.1146/annurev-psych-010418-102803

Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008; 8:(1) https://doi.org/10.1186/1471-2288-8-45

Wallace J, Nwosu B, Clarke M. Barriers to the uptake of evidence from systematic reviews and meta-analyses: a systematic review of decision makers' perceptions. BMJ Open. 2012; 2:(5) https://doi.org/10.1136/bmjopen-2012-001220

Carrying out systematic literature reviews: an introduction

Alan Davies

Lecturer in Health Data Science, School of Health Sciences, University of Manchester, Manchester

View articles · Email Alan

Systematic reviews provide a synthesis of evidence for a specific topic of interest, summarising the results of multiple studies to aid in clinical decisions and resource allocation. They remain among the best forms of evidence, and reduce the bias inherent in other methods. A solid understanding of the systematic review process can be of benefit to nurses that carry out such reviews, and for those who make decisions based on them. An overview of the main steps involved in carrying out a systematic review is presented, including some of the common tools and frameworks utilised in this area. This should provide a good starting point for those that are considering embarking on such work, and to aid readers of such reviews in their understanding of the main review components, in order to appraise the quality of a review that may be used to inform subsequent clinical decision making.

Since their inception in the late 1970s, systematic reviews have gained influence in the health professions ( Hanley and Cutts, 2013 ). Systematic reviews and meta-analyses are considered to be the most credible and authoritative sources of evidence available ( Cognetti et al, 2015 ) and are regarded as the pinnacle of evidence in the various ‘hierarchies of evidence’. Reviews published in the Cochrane Library ( https://www.cochranelibrary.com) are widely considered to be the ‘gold’ standard. Since Guyatt et al (1995) presented a users' guide to medical literature for the Evidence-Based Medicine Working Group, various hierarchies of evidence have been proposed. Figure 1 illustrates an example.

Systematic reviews can be qualitative or quantitative. One of the criticisms levelled at hierarchies such as these is that qualitative research is often positioned towards or even is at the bottom of the pyramid, thus implying that it is of little evidential value. This may be because of traditional issues concerning the quality of some qualitative work, although it is now widely recognised that both quantitative and qualitative research methodologies have a valuable part to play in answering research questions, which is reflected by the National Institute for Health and Care Excellence (NICE) information concerning methods for developing public health guidance. The NICE (2012) guidance highlights how both qualitative and quantitative study designs can be used to answer different research questions. In a revised version of the hierarchy-of-evidence pyramid, the systematic review is considered as the lens through which the evidence is viewed, rather than being at the top of the pyramid ( Murad et al, 2016 ).

Both quantitative and qualitative research methodologies are sometimes combined in a single review. According to the Cochrane review handbook ( Higgins and Green, 2011 ), regardless of type, reviews should contain certain features, including:

• Clearly stated objectives
• Predefined eligibility criteria for inclusion or exclusion of studies in the review
• A reproducible and clearly stated methodology
• Validity assessment of included studies (eg quality, risk, bias etc).

The main stages of carrying out a systematic review are summarised in Box 1 .

Formulating the research question

Before undertaking a systemic review, a research question should first be formulated ( Bashir and Conlon, 2018 ). There are a number of tools/frameworks ( Table 1 ) to support this process, including the PICO/PICOS, PEO and SPIDER criteria ( Bowers et al, 2011 ). These frameworks are designed to help break down the question into relevant subcomponents and map them to concepts, in order to derive a formalised search criterion ( Methley et al, 2014 ). This stage is essential for finding literature relevant to the question ( Jahan et al, 2016 ).

It is advisable to first check that the review you plan to carry out has not already been undertaken. You can optionally register your review with an international register of prospective reviews called PROSPERO, although this is not essential for publication. This is done to help you and others to locate work and see what reviews have already been carried out in the same area. It also prevents needless duplication and instead encourages building on existing work ( Bashir and Conlon, 2018 ).

A study ( Methley et al, 2014 ) that compared PICO, PICOS and SPIDER in relation to sensitivity and specificity recommended that the PICO tool be used for a comprehensive search and the PICOS tool when time/resources are limited.

The use of the SPIDER tool was not recommended due to the risk of missing relevant papers. It was, however, found to increase specificity.

These tools/frameworks can help those carrying out reviews to structure research questions and define key concepts in order to efficiently identify relevant literature and summarise the main objective of the review ( Jahan et al, 2016 ). A possible research question could be: Is paracetamol of benefit to people who have just had an operation? The following examples highlight how using a framework may help to refine the question:

• What form of paracetamol? (eg, oral/intravenous/suppository)
• Is the dosage important?
• What is the patient population? (eg, children, adults, Europeans)
• What type of operation? (eg, tonsillectomy, appendectomy)
• What does benefit mean? (eg, reduce post-operative pyrexia, analgesia).

An example of a more refined research question could be: Is oral paracetamol effective in reducing pain following cardiac surgery for adult patients? A number of concepts for each element will need to be specified. There will also be a number of synonyms for these concepts ( Table 2 ).

Table 2 shows an example of concepts used to define a search strategy using the PICO statement. It is easy to see even with this dummy example that there are many concepts that require mapping and much thought required to capture ‘good’ search criteria. Consideration should be given to the various terms to describe the heart, such as cardiac, cardiothoracic, myocardial, myocardium, etc, and the different names used for drugs, such as the equivalent name used for paracetamol in other countries and regions, as well as the various brand names. Defining good search criteria is an important skill that requires a lot of practice. A high-quality review gives details of the search criteria that enables the reader to understand how the authors came up with the criteria. A specific, well-defined search criterion also aids in the reproducibility of a review.

Search criteria

Before the search for papers and other documents can begin it is important to explicitly define the eligibility criteria to determine whether a source is relevant to the review ( Hanley and Cutts, 2013 ). There are a number of database sources that are searched for medical/health literature including those shown in Table 3 .

The various databases can be searched using common Boolean operators to combine or exclude search terms (ie AND, OR, NOT) ( Figure 2 ).

Although most literature databases use similar operators, it is necessary to view the individual database guides, because there are key differences between some of them. Table 4 details some of the common operators and wildcards used in the databases for searching. When developing a search criteria, it is a good idea to check concepts against synonyms, as well as abbreviations, acronyms and plural and singular variations ( Cognetti et al, 2015 ). Reading some key papers in the area and paying attention to the key words they use and other terms used in the abstract, and looking through the reference lists/bibliographies of papers, can also help to ensure that you incorporate relevant terms. Medical Subject Headings (MeSH) that are used by the National Library of Medicine (NLM) ( https://www.nlm.nih.gov/mesh/meshhome.html) to provide hierarchical biomedical index terms for NLM databases (Medline and PubMed) should also be explored and included in relevant search strategies.

Searching the ‘grey literature’ is also an important factor in reducing publication bias. It is often the case that only studies with positive results and statistical significance are published. This creates a certain bias inherent in the published literature. This bias can, to some degree, be mitigated by the inclusion of results from the so-called grey literature, including unpublished work, abstracts, conference proceedings and PhD theses ( Higgins and Green, 2011 ; Bettany-Saltikov, 2012 ; Cognetti et al, 2015 ). Biases in a systematic review can lead to overestimating or underestimating the results ( Jahan et al, 2016 ).

An example search strategy from a published review looking at web use for the appraisal of physical health conditions can be seen in Box 2 . High-quality reviews usually detail which databases were searched and the number of items retrieved from each.

A balance between high recall and high precision is often required in order to produce the best results. An oversensitive search, or one prone to including too much noise, can mean missing important studies or producing too many search results ( Cognetti et al, 2015 ). Following a search, the exported citations can be added to citation management software (such as Mendeley or Endnote) and duplicates removed.

Title and abstract screening

Initial screening begins with the title and abstracts of articles being read and included or excluded from the review based on their relevance. This is usually carried out by at least two researchers to reduce bias ( Bashir and Conlon, 2018 ). After screening any discrepancies in agreement should be resolved by discussion, or by an additional researcher casting the deciding vote ( Bashir and Conlon, 2018 ). Statistics for inter-rater reliability exist and can be reported, such as percentage of agreement or Cohen's kappa ( Box 3 ) for two reviewers and Fleiss' kappa for more than two reviewers. Agreement can depend on the background and knowledge of the researchers and the clarity of the inclusion and exclusion criteria. This highlights the importance of providing clear, well-defined criteria for inclusion that are easy for other researchers to follow.

Full-text review

Following title and abstract screening, the remaining articles/sources are screened in the same way, but this time the full texts are read in their entirety and included or excluded based on their relevance. Reasons for exclusion are usually recorded and reported. Extraction of the specific details of the studies can begin once the final set of papers is determined.

Data extraction

At this stage, the full-text papers are read and compared against the inclusion criteria of the review. Data extraction sheets are forms that are created to extract specific data about a study (12 Jahan et al, 2016 ) and ensure that data are extracted in a uniform and structured manner. Extraction sheets can differ between quantitative and qualitative reviews. For quantitative reviews they normally include details of the study's population, design, sample size, intervention, comparisons and outcomes ( Bettany-Saltikov, 2012 ; Mueller et al, 2017 ).

Quality appraisal

The quality of the studies used in the review should also be appraised. Caldwell et al (2005) discussed the need for a health research evaluation framework that could be used to evaluate both qualitative and quantitative work. The framework produced uses features common to both research methodologies, as well as those that differ ( Caldwell et al, 2005 ; Dixon-Woods et al, 2006 ). Figure 3 details the research critique framework. Other quality appraisal methods do exist, such as those presented in Box 4 . Quality appraisal can also be used to weight the evidence from studies. For example, more emphasis can be placed on the results of large randomised controlled trials (RCT) than one with a small sample size. The quality of a review can also be used as a factor for exclusion and can be specified in inclusion/exclusion criteria. Quality appraisal is an important step that needs to be undertaken before conclusions about the body of evidence can be made ( Sambunjak and Franic, 2012 ). It is also important to note that there is a difference between the quality of the research carried out in the studies and the quality of how those studies were reported ( Sambunjak and Franic, 2012 ).

The quality appraisal is different for qualitative and quantitative studies. With quantitative studies this usually focuses on their internal and external validity, such as how well the study has been designed and analysed, and the generalisability of its findings. Qualitative work, on the other hand, is often evaluated in terms of trustworthiness and authenticity, as well as how transferable the findings may be ( Bettany-Saltikov, 2012 ; Bashir and Conlon, 2018 ; Siddaway et al, 2019 ).

Reporting a review (the PRISMA statement)

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) provides a reporting structure for systematic reviews/meta-analysis, and consists of a checklist and diagram ( Figure 4 ). The stages of identifying potential papers/sources, screening by title and abstract, determining eligibility and final inclusion are detailed with the number of articles included/excluded at each stage. PRISMA diagrams are often included in systematic reviews to detail the number of papers included at each of the four main stages (identification, screening, eligibility and inclusion) of the review.

Data synthesis

The combined results of the screened studies can be analysed qualitatively by grouping them together under themes and subthemes, often referred to as meta-synthesis or meta-ethnography ( Siddaway et al, 2019 ). Sometimes this is not done and a summary of the literature found is presented instead. When the findings are synthesised, they are usually grouped into themes that were derived by noting commonality among the studies included. Inductive (bottom-up) thematic analysis is frequently used for such purposes and works by identifying themes (essentially repeating patterns) in the data, and can include a set of higher-level and related subthemes (Braun and Clarke, 2012). Thomas and Harden (2008) provide examples of the use of thematic synthesis in systematic reviews, and there is an excellent introduction to thematic analysis by Braun and Clarke (2012).

The results of the review should contain details on the search strategy used (including search terms), the databases searched (and the number of items retrieved), summaries of the studies included and an overall synthesis of the results ( Bettany-Saltikov, 2012 ). Finally, conclusions should be made about the results and the limitations of the studies included ( Jahan et al, 2016 ). Another method for synthesising data in a systematic review is a meta-analysis.

Limitations of systematic reviews

Apart from the many advantages and benefits to carrying out systematic reviews highlighted throughout this article, there remain a number of disadvantages. These include the fact that not all stages of the review process are followed rigorously or even at all in some cases. This can lead to poor quality reviews that are difficult or impossible to replicate. There also exist some barriers to the use of evidence produced by reviews, including ( Wallace et al, 2012 ):

• Lack of awareness and familiarity with reviews
• Lack of access
• Lack of direct usefulness/applicability.

Meta-analysis

When the methods used and the analysis are similar or the same, such as in some RCTs, the results can be synthesised using a statistical approach called meta-analysis and presented using summary visualisations such as forest plots (or blobbograms) ( Figure 5 ). This can be done only if the results can be combined in a meaningful way.

Meta-analysis can be carried out using common statistical and data science software, such as the cross-platform ‘R’ ( https://www.r-project.org), or by using standalone software, such as Review Manager (RevMan) produced by the Cochrane community ( https://tinyurl.com/revman-5), which is currently developing a cross-platform version RevMan Web.

Carrying out a systematic review is a time-consuming process, that on average takes between 6 and 18 months and requires skill from those involved. Ideally, several reviewers will work on a review to reduce bias. Experts such as librarians should be consulted and included where possible in review teams to leverage their expertise.

Systematic reviews should present the state of the art (most recent/up-to-date developments) concerning a specific topic and aim to be systematic and reproducible. Reproducibility is aided by transparent reporting of the various stages of a review using reporting frameworks such as PRISMA for standardisation. A high-quality review should present a summary of a specific topic to a high standard upon which other professionals can base subsequent care decisions that increase the quality of evidence-based clinical practice.

• Systematic reviews remain one of the most trusted sources of high-quality information from which to make clinical decisions
• Understanding the components of a review will help practitioners to better assess their quality
• Many formal frameworks exist to help structure and report reviews, the use of which is recommended for reproducibility
• Experts such as librarians can be included in the review team to help with the review process and improve its quality

CPD reflective questions

• Where should high-quality qualitative research sit regarding the hierarchies of evidence?
• What background and expertise should those conducting a systematic review have, and who should ideally be included in the team?
• Consider to what extent inter-rater agreement is important in the screening process

Easy guide to conducting a systematic review

Affiliations.

• 1 Discipline of Child and Adolescent Health, University of Sydney, Sydney, New South Wales, Australia.
• 2 Department of Nephrology, The Children's Hospital at Westmead, Sydney, New South Wales, Australia.
• 3 Education Department, The Children's Hospital at Westmead, Sydney, New South Wales, Australia.
• PMID: 32364273
• DOI: 10.1111/jpc.14853

A systematic review is a type of study that synthesises research that has been conducted on a particular topic. Systematic reviews are considered to provide the highest level of evidence on the hierarchy of evidence pyramid. Systematic reviews are conducted following rigorous research methodology. To minimise bias, systematic reviews utilise a predefined search strategy to identify and appraise all available published literature on a specific topic. The meticulous nature of the systematic review research methodology differentiates a systematic review from a narrative review (literature review or authoritative review). This paper provides a brief step by step summary of how to conduct a systematic review, which may be of interest for clinicians and researchers.

Keywords: research; research design; systematic review.

© 2020 Paediatrics and Child Health Division (The Royal Australasian College of Physicians).

Publication types

• Systematic Review
• Research Design*

Retirement planning – a systematic review of literature and future research directions

• Published: 28 October 2023

Cite this article

• Kavita Karan Ingale   ORCID: orcid.org/0000-0003-3570-4211 1 &
• Ratna Achuta Paluri 2  

506 Accesses

Explore all metrics

Rising life expectancy and an aging population across nations are leading to an increased need for long-term financial savings and a focus on the financial well-being of retired individuals amidst changing policy framework. This study is a systematic review based on a scientific way of producing high-quality evidence based on 191 articles from the Scopus and Web of Science databases. It adopts the Theory, Context, Characteristics, and Method (TCCM) framework to analyze literature. This study provides collective insights into financial decision-making for retirement savings and identifies constructs for operationalizing and measuring financial behavior for retirement planning. Further, it indicates the need for an interdisciplinary approach. Though cognitive areas were studied extensively, the non-cognitive areas received little attention. Qualitative research design is gaining prominence in research over other methods, with the sparse application of mixed methods design. The study’s TCCM framework explicates several areas for further research. Furthermore, it guides the practice and policy by integrating empirical evidence and concomitant findings. Coherent synthesis of the extant literature reconciles the highly fragmented field of retirement planning. No research reports prospective areas for further analysis based on the TCCM framework on retirement planning, which highlights the uniqueness of the study.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

A Research Proposal to Examine Psychological Factors Influence on Financial Planning for Retirement in China

Domains and determinants of retirement timing: A systematic review of longitudinal studies

Micky Scharn, Ranu Sewdas, … Allard J. van der Beek

Reinventing Retirement

Deanna L. Sharpe

Data Availability

The research data will be made available on request.

Acknowledgment.

Elderly population is defined as a population aged 65 years and over.

Defined benefit plan guarantees benefits to the employee, while defined contribution plan requires employees to decide on their own investment and bear the financial risks identified with it.

“The old-age dependency ratio is defined as the number of individuals aged 65 and over per 100 people of working age defined as those at ages 20 to 64”(OECD 2023 ).

Adams GA, Rau BL (2011) Putting off tomorrow to do what you want today: planning for Retirement. Am Psychol 66(3):180–192. https://doi.org/10.1037/a0022131

Article   Google Scholar  

Aegon Cfor, Longevity, Retirement ICR (2016) The Aegon Retirement Readiness Survey 2016. In The Aegon Retirement Readiness Survey 2016 . https://www.aegon.com/contentassets/c6a4b1cdded34f1b85a4f21d4c66e5d3/2016-aegon-retirement-readiness-report-india.pdf

Agarwalla SK, Barua SK, Jacob J, Varma JR (2015) Financial Literacy among Working Young in Urban India. World Development , 67 (2013), 101–109. https://doi.org/10.1016/j.worlddev.2014.10.004

Ajzen I (1991) The theory of Planned Behavior. Organ Behav Hum Decis Process 50:179–211. https://doi.org/10.47985/dcidj.475

Anderson A, Baker F, Robinson DT (2017) Precautionary savings, retirement planning, and misperceptions of financial literacy. J Financ Econ 126(2):383–398. https://doi.org/10.1016/j.jfineco.2017.07.008

Atkinson A, Messy FA (2011) Assessing financial literacy in 12 countries: an OECD/INFE international pilot exercise. J Pension Econ Finance 10(4):657–665. https://doi.org/10.1017/S1474747211000539`

Aydin AE, Akben Selcuk E (2019) An investigation of financial literacy, money ethics, and time preferences among college students: a structural equation model. Int J Bank Mark 37(3):880–900. https://doi.org/10.1108/IJBM-05-2018-0120

Bapat D (2020) Antecedents to responsible financial management behavior among young adults: the moderating role of financial risk tolerance. Int J Bank Mark 38(5):1177–1194. https://doi.org/10.1108/IJBM-10-2019-0356

Beckett A, Hewer P, Howcroft B (2000) An exposition of consumer behaviour in the financial services industry. Int J Bank Mark 18(1):15–26. https://doi.org/10.1108/02652320010315325

Białowolski P (2019) Economic sentiment as a driver for household financial behavior. J Behav Experimental Econ 80(August 2017):59–66. https://doi.org/10.1016/j.socec.2019.03.006

Binswanger J, Carman KG (2012) How real people make long-term decisions: the case of retirement preparation. J Economic Behav Organ 81(1):39–60. https://doi.org/10.1016/j.jebo.2011.08.010

Brounen D, Koedijk KG, Pownall RAJ (2016) Household financial planning and savings behavior. J Int Money Finance 69:95–107. https://doi.org/10.1016/j.jimonfin.2016.06.011

Brown R, Jones M (2015) Mapping and exploring the topography of contemporary financial accounting research. Br Acc Rev 47(3):237–261. https://doi.org/10.1016/j.bar.2014.08.006

Brown S, Gray D (2016) Household finances and well-being in Australia: an empirical analysis of comparison effects. J Econ Psychol 53:17–36. https://doi.org/10.1016/j.joep.2015.12.006

Brown S, Taylor K (2014) Household finances and the big five personality traits. J Econ Psychol 45:197–212. https://doi.org/10.1016/j.joep.2014.10.006

Brown S, Taylor K (2016) Early influences on saving behaviour: analysis of British panel data. J Bank Finance 62:1–14. https://doi.org/10.1016/j.jbankfin.2015.09.011

Brüggen EC, Post T, Schmitz K (2019) Interactivity in online pension planners enhances engagement with retirement planning – but not for everyone. J Serv Mark 33(4):488–501. https://doi.org/10.1108/JSM-02-2018-0082

Bruggen E, Post T, Katharina S (2019) Interactivity in online pension planners enhances engagement with retirement planning but not for everyone. J Serv Mark 33(4):488–501

Calcagno R, Monticone C (2015) Financial literacy and the demand for financial advice. J Bank Finance 50:363–380. https://doi.org/10.1016/j.jbankfin.2014.03.013

Campbell JY (2006) Household finance. J Finance 61(4):1553–1604. https://doi.org/10.1111/j.1540-6261.2006.00883.x

Choudhury K (2015) Service quality and customers’ behavioural intentions: class and mass banking and implications for the consumer and society. Asia Pac J Mark Logistics 27(5):735–757

Chowdhry N, Jung J, Dholakia U (2018) Association for consumer research. Adv Consum Res 42:42–46

Google Scholar  

Clark GL, Knox-Hayes J, Strauss K (2009) Financial sophistication, salience, and the scale of deliberation in UK retirement planning. Environ Plann A 41(10):2496–2515. https://doi.org/10.1068/a41265

Clark R, Lusardi A, Mitchell OS (2017) Employee Financial Literacy and Retirement Plan Behavior: a case study. Econ Inq 55(1):248–259. https://doi.org/10.1111/ecin.12389

Collins JM, Urban C (2016) The role of information on Retirement Planning: evidence from a field study. Econ Inq 54(4):1860–1872. https://doi.org/10.1111/ecin.12349

Creswell J (2009) Research Design Qualitative Quantitative and Mixed Methods Approaches. In Sage Publishing: Vol. Third edit . https://doi.org/10.1002/tl.20234

Csorba L (2020) The determining factors of financial culture, financial literacy, and financial behavior. Public Finance Q 65:67–83. https://doi.org/10.35551/PFQ_2020_1_6

Davidoff T, Gerhard P, Post T (2017) Reverse mortgages: what homeowners (don’t) know and how it matters. J Economic Behav Organ 133:151–171. https://doi.org/10.1016/j.jebo.2016.11.007

Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly: Management Information Systems 13(3):319–339. https://doi.org/10.2307/249008

Devlin J (2001) Consumer evaluation and competitive advantage in retail financial services - a research agenda. Eur J Mark 35(5/6):639–660

Dholakia U, Tam L, Yoon S, Wong N (2016) The ant and the grasshopper: understanding personal saving orientation of consumers. J Consum Res 43(1):134–155. https://doi.org/10.1093/jcr/ucw004

Dolls M, Doerrenberg P, Peichl A, Stichnoth H (2018) Do retirement savings increase in response to information about retirement and expected pensions? J Public Econ 158(July 2017):168–179. https://doi.org/10.1016/j.jpubeco.2017.12.014

Dragos SL, Dragos CM, Muresan GM (2020) From intention to the decision in purchasing life insurance and private pensions: different effects of knowledge and behavioural factors. J Behav Experimental Econ 87(March):101555. https://doi.org/10.1016/j.socec.2020.101555

Drever AI, Odders-white E, Kalish CW, Hoagland EM, Nelms EN, Drever AI, Odders-white E, Charles W, Else-quest NM, Hoagland EM, Nelms EN (2015) Foundations of Financial Weil-Being: Insights into the Role of Executive Function, Financial Socialization, and Experience-Based Learning in Childhood and Youth Source : The Journal of Consumer Affairs, Vol. 49, No. 1, Special Issue on Starting Ea. The Journal of Consumer Affairs , 49 (1)

Duflo E, Saez E (2002) Participation and investment decisions in a retirement plan: the influence of colleagues’ choices. J Public Econ 85(1):121–148. https://doi.org/10.1016/S0047-2727(01)00098-6

Duxbury D, Summers B, Hudson R, Keasey K (2013) How people evaluate defined contribution, annuity-based pension arrangements: a behavioral exploration. J Econ Psychol 34:256–269. https://doi.org/10.1016/j.joep.2012.10.008

Earl J, Bednall T, Muratore A (2015) A matter of time: why some people plan for retirement and others do not. Work Aging and Retirement 1(2):181–189. https://doi.org/10.1093/workar/wau005

Employees Benefits Research Institute (2020) EBRI Retirement Confidence Survey Report (Issue 202)

Engel JF, Kollat DT, Blackwell RD (1968) A model of consumer motivation and behavior. In: Research in consumer behavior. Holt, Rinehart and Winston, Inc., New York, pp 3–20

Erasmus A, Boshoff E, Rousseau G (2001) Consumer decision-making models within the discipline of consumer science: a critical approach. J Family Ecol Consumer Sci /Tydskrif Vir Gesinsekologie En Verbruikerswetenskappe 29(1):82–90. https://doi.org/10.4314/jfecs.v29i1.52799

Farrell L, Fry TRL, Risse L (2016) The significance of financial self-efficacy in explaining women’s personal finance behaviour. J Econ Psychol 54:85–99

Fernandes D, Lynch JG, Netemeyer RG (2014) Financial literacy, financial education, and downstream financial behaviors. Manage Sci 60(8):1861–1883. https://doi.org/10.1287/mnsc.2013.1849

Filbec G, Ricciardi V, Evensky H, Fan S, Holzhauer H, Spieler A (2017) Behavioral finance: a panel discussion. J Behav Experimental Finance 15:52–58. https://doi.org/10.1016/j.jbef.2015.07.003

Fishbein M (1979) A theory of reasoned action: some applications and implications. Nebraska Symposium on Motivation 27:65–116

Fisher PJ, Montalto CP (2010) Effect of saving motives and horizon on saving behaviors. J Econ Psychol 31(1):92–105. https://doi.org/10.1016/j.joep.2009.11.002

Flores SAM, Vieira KM (2014) Propensity toward indebtedness: an analysis using behavioral factors. J Behav Exp Finance 3:1–10

Foxall GR, Pallister JG (1998) Measuring purchase decision involvement for financial services: comparison of the Zaichkowsky and Mittal scales. Int J Bank Mark 16(5):180–194. https://doi.org/10.1108/02652329810228181

Friedman M (1957) Introduction to “A theory of the consumption function”. In: A theory of the consumption function. Princeton University Press, pp 1–6

Frydman C, Camerer CF (2016) The psychology and neuroscience of financial decision making. Trends Cogn Sci 20(9):661–675. https://doi.org/10.1016/j.tics.2016.07.003

Gardarsdóttir RB, Dittmar H (2012) The relationship of materialism to debt and financial well-being: the case of Iceland’s perceived prosperity. J Econ Psychol 33(3):471–481. https://doi.org/10.1016/j.joep.2011.12.008

Gathergood J (2012) Self-control, financial literacy and consumer over-indebtedness. J Econ Psychol 33(3):590–602

Gerhard P, Gladstone JJ, Hoffmann AOI (2018) Psychological characteristics and household savings behavior: the importance of accounting for latent heterogeneity. J Economic Behav Organ 148:66–82. https://doi.org/10.1016/j.jebo.2018.02.013

Gibbs PT (2009) Time, temporality, and

Goedde-Menke M, Lehmensiek-Starke M, Nolte S (2014) An empirical test of competing hypotheses for the annuity puzzle. J Econ Psychol 43:75–91

Gough O, Nurullah M (2009) Understanding what drives the purchase decision in pension and investment products. J Financial Serv Mark 14(2):152–172. https://doi.org/10.1057/fsm.2009.14

Griffin B, Loe D, Hesketh B (2012) Using Proactivity, Time Discounting, and the theory of Planned Behavior to identify predictors of Retirement Planning. Educ Gerontol 38(12):877–889. https://doi.org/10.1080/03601277.2012.660857

Gritten A (2011) New insights into consumer confidence in financial services. Int J Bank Mark 29(2):90–106. https://doi.org/10.1108/02652321111107602

Grohmann A (2018) Financial literacy and financial behavior: Evidence from the emerging Asian middle class. Pacific Basin Finance Journal , 48 (November 2017), 129–143. https://doi.org/10.1016/j.pacfin.2018.01.007

Grohmann A, Kouwenberg R, Menkhoff L (2015) Childhood roots of financial literacy. J Econ Psychol 51:114–133. https://doi.org/10.1016/j.joep.2015.09.002

Hair JF, Sarstedt M, Ringle CM, Mena JA (2012) An assessment of the use of partial least squares structural equation modeling in marketing research . 414–433. https://doi.org/10.1007/s11747-011-0261-6

Hanna SD, Kim KT, Chen SCC (2016) Retirement savings. In: Handbook of consumer finance research, pp 33–43

Harrison T, Waite K, White P (2006) Analysis by paralysis: the pension purchase decision process. Int J Bank Mark 24(1):5–23. https://doi.org/10.1108/02652320610642317

Hastings J, Mitchell O (2011) How financial literact and impatience shape retirement wealth and investment behaviors. Pengaruh Harga Diskon Dan Persepsi Produk Terhadap Nilai Belanja Serta Perilaku Pembelian Konsumen, NBER Working paper, 1–28

Hauff J, Carlander A, Amelie G, Tommy G, Holmen M (2016) Breaking the ice of low financial involvement: does narrative information format from a trusted sender increase savings in mutual funds? Int J Bank Mark 34(2):151–170

Hentzen JK, Hoffmann A, Dolan R, Pala E (2021) Artificial intelligence in customer-facing financial services: a systematic literature review and agenda for future research. Int J Bank Mark. https://doi.org/10.1108/IJBM-09-2021-0417

Hershey DA, Mowen JC (2000) Psychological determinants of financial preparedness for retirement. Gerontologist 40(6):687–697. https://doi.org/10.1093/geront/40.6.687

Hershey DA, Henkens K, Van Dalen HP (2007) Mapping the minds of retirement planners: a cross-cultural perspective. J Cross-Cult Psychol 38(3):361–382. https://doi.org/10.1177/0022022107300280

Hershey DA, Jacobs-Lawson JM, McArdle JJ, Hamagami F (2007b) Psychological foundations of financial planning for retirement. J Adult Dev 14(1–2):26–36. https://doi.org/10.1007/s10804-007-9028-1

Hershey DA, Jacobs-Lawson JM, McArdle JJ, Hamagami F (2008) Psychological foundations of financial planning for retirement. J Adult Dev 14(1–2):26–36. https://doi.org/10.1007/s10804-007-9028-1

Hershfield H, Goldstein D, Sharpe W, Fox J, Yeykelis L, Carstensen L, Bailenson J (2011) Increasing saving behavior through age-progressed renderings of the future self. J Mark Res 48:23–37

Hoffmann AOI, Broekhuizen TLJ (2009) Susceptibility to and impact of interpersonal influence in an investment context. J Acad Mark Sci 37:488–503

Hoffmann AOI, Broekhuizen TLJ (2010) Understanding investors’ decisions to purchase innovative products: drivers of adoption timing and range. Int J Res Mark 27(4):342–355. https://doi.org/10.1016/j.ijresmar.2010.08.002

Hoffmann AOI, Plotkina D (2020a) Positive framing when assessing the personal resources to manage one’s finances increases consumers’ retirement self-efficacy and improves retirement goal clarity. Psychol Mark 38(12):2286–2304. https://doi.org/10.1002/mar.21563

Hoffmann AOI, Plotkina D (2020b) Why and when does financial information affect retirement planning intentions and which consumers are more likely to act on them? Journal of Business Research , 117 (September 2019), 411–431. https://doi.org/10.1016/j.jbusres.2020.06.023

Hoffmann AOI, Plotkina D (2021) Let your past define your future. How recalling successful financial experiences can increase beliefs of self-efficacy in financial planning. J Consum Aff 55(3):847–871. https://doi.org/10.1111/joca.12378

Hoffmann AOI, Risse L (2020) Do good things come in pairs? How personality traits help explain individuals’ simultaneous pursuit of a healthy lifestyle and financially responsible behavior. J Consum Aff 54(3):1082–1120. https://doi.org/10.1111/joca.12317

Hsiao YJ, Tsai WC (2018) Financial literacy and participation in the derivatives markets. J Bank Finance 88:15–29

Huhmann BA, McQuitty S (2009) A model of consumer financial numeracy. Int J Bank Mark 27(4):270–293. https://doi.org/10.1108/02652320910968359

Huston SJ (2010) Measuring financial literacy. J Consum Aff 44(2):296–316. https://doi.org/10.1111/j.1745-6606.2010.01170.x

Ijevleva K, Arefjevs I (2014) Analysis of the Aggregate Financial Behaviour of customers using the Transtheoretical Model of Change. Procedia - Social and Behavioral Sciences 156(April):435–438. https://doi.org/10.1016/j.sbspro.2014.11.217

Ingale KK, Paluri RA (2020) Financial literacy and financial behavior: a bibliometric analysis. Rev Behav Finance. https://doi.org/10.1108/RBF-06-2020-0141

Jacobs-Lawson J, Hershey D (2005) Influence of future time perspective, financial knowledge, and financial risk tolerance on retirement savings behavior. Financial Serv Rev 14:331–344. https://doi.org/10.1088/1751-8113/44/8/085201

Jappelli T, Padula M (2013) Investment in financial literacy and saving decisions. J Bank Finance 37(8):2779–2792. https://doi.org/10.1016/j.jbankfin.2013.03.019

Kadoya Y, Rahim Khan MS (2020) Financial literacy in Japan: new evidence using financial knowledge, behavior, and attitude. Sustain (Switzerland) 12(9). https://doi.org/10.3390/su12093683

Kamil NSSN, Musa R, Sahak SZ (2014) Examining the Role of Financial Intelligence Quotient (FiQ) in explaining credit card usage behavior: a conceptual Framework. Procedia - Social and Behavioral Sciences 130:568–576. https://doi.org/10.1016/j.sbspro.2014.04.066

Kerry MJ (2018) Psychological antecedents of retirement planning: a systematic review. Front Psychol 9(OCT). https://doi.org/10.3389/fpsyg.2018.01870

Kerry MJ, Embretson SE (2018) An experimental evaluation of competing age predictions of future time perspective between workplace and retirement domains. Front Psychol 8(JAN):1–9. https://doi.org/10.3389/fpsyg.2017.02316

Kiliyanni AL, Sivaraman S (2016) The perception-reality gap in financial literacy: evidence from the most literate state in India. Int Rev Econ Educ 23:47–64. https://doi.org/10.1016/j.iree.2016.07.001

Kimiyaghalam F, Mansori S, Safari M, Yap S (2017) Parents’ influence on retirement planning in Malaysia. Family Consumer Sci Res J 45(3):315–325

Klapper L, Lusardi A, Panos GA (2013) Financial literacy and its consequences: evidence from Russia during the financial crisis. J Bank Finance 37(10):3904–3923

Koehler DJ, Langstaff J, Liu WQ (2015) A simulated financial savings task for studying consumption and retirement decision-making. J Econ Psychol 46:89–97. https://doi.org/10.1016/j.joep.2014.12.004

Kramer MM (2016) Financial literacy, confidence, and financial advice seeking. Journal of Economic Behavior and Organization , 131 (June 2015), 198–217. https://doi.org/10.1016/j.jebo.2016.08.016

Kumar S, Tomar S, Verma D (2019) Women’s financial planning for retirement: systematic literature review and future research agenda. Int J Bank Mark 37(1):120–141. https://doi.org/10.1108/IJBM-08-2017-0165

Kwon KN, Lee J (2009) The effects of reference point, knowledge, and risk propensity on the evaluation of financial products. J Bus Res 62(7):719–725. https://doi.org/10.1016/j.jbusres.2008.07.002

Landerretche OM, Martínez C (2013) Voluntary savings, financial behavior, and pension finance literacy: evidence from Chile. J Pension Econ Finance 12(3):251–297. https://doi.org/10.1017/S1474747212000340

Lee T (2017) (David). Clear, conspicuous, and improving: US corporate websites for critical financial literacy in retirement. International Journal of Bank Marketing , 35 (5), 761–780. https://doi.org/10.1108/IJBM-01-2016-0010

Liang C-J, Wang Wen‐Hung, Farquhar JD (2009) (2009). The influence of customer perceptions on financial performance in financial services. International Journal of Bank Marketing , 27 (2), 129–149

Liberman N, Trope Y (2003) Construal level theory of intertemporal judgment and decision. In: Loewenstein G, Read D, Baumeister R (eds) Time and decision: economic and psychological perspectives on intertemporal choice, pp 245–276

Lim KL, Soutar GN, Lee JA (2013) Factors affecting investment intentions: a consumer behaviour perspective. J Financ Serv Mark 18:301–315

Lin C, Hsiao YJ, Yeh CY (2017) Financial literacy, financial advisors, and information sources on demand for life insurance. Pac Basin Finance J 43(March):218–237. https://doi.org/10.1016/j.pacfin.2017.04.002

Lown JM (2011) Development and validation of a Financial Self-Efficacy Scale. J Financial Couns Plann 22(2):54–63

Lusardi A, Mitchell OS (2007) Baby Boomer retirement security: the roles of planning, financial literacy, and housing wealth. J Monet Econ 54(1):205–224. https://doi.org/10.1016/j.jmoneco.2006.12.001

Maloney M, McCarthy A (2017) Understanding pension communications at the organizational level: insights from bounded rationality theory & implications for HRM. Hum Resource Manage Rev 27(2):338–352. https://doi.org/10.1016/j.hrmr.2016.08.001

Marjanovic Z, Fiksenbaum L, Greenglass E (2018) Financial threat correlates with acute economic hardship and behavioral intentions that can improve one’s personal finances and health. J Behav Experimental Econ 77(April):151–157. https://doi.org/10.1016/j.socec.2018.09.012

Marques S, Mariano J, Lima ML, Abrams D (2018) Are you talking to the future me? The moderator role of future self-relevance on the effects of aging salience in retirement savings. J Appl Soc Psychol 48(7):360–368. https://doi.org/10.1111/jasp.12516

McKechnie S (1992) Consumer buying behaviour in financial services: an overview. Int J Bank Mark 10(5):5–39. https://doi.org/10.1108/02652329210016803

Milner T, Rosenstreich D (2013a) A review of consumer decision-making models and development of a new model for financial services. J Financial Serv Mark 18(2):106–120. https://doi.org/10.1057/fsm.2013.7

Milner T, Rosenstreich D (2013b) Insights into mature consumers of financial services. J Consumer Mark 30(3):248–257. https://doi.org/10.1108/07363761311328919

Mitchell OS, Mukherjee A (2017) Assessing the demand for micro pensions among India’s poor. J Econ Ageing 9:30–40. https://doi.org/10.1016/j.jeoa.2016.05.004

Mitchell O, Utkus S (2003) Lessons from Behavioral Finance for Retirement Plan Design (PRC WP 2003-6). http://prc.wharton.upenn.edu/prc/prc.html

Modigliani F, Brumberg RH (1954) Utility analysis and the consumption function: an interpretation of cross-section data. In: Kurihara KK (ed) Post-Keynesian economics. Rutgers University Press, New Brunswick, pp 388–436

Moher D, Liberati A, Tetzlaff J, Altman DG, Altman D, Antes G, Atkins D, Barbour V, Barrowman N, Berlin JA, Clark J, Clarke M, Cook D, D’Amico R, Deeks JJ, Devereaux PJ, Dickersin K, Egger M, Ernst E, …, Tugwell P (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6(7). https://doi.org/10.1371/journal.pmed.1000097

Monti M, Pelligra V, Martignon L, Berg N (2014) Retail investors and financial advisors: new evidence on trust and advice taking heuristics. J Bus Res 67(8):1749–1757. https://doi.org/10.1016/j.jbusres.2014.02.022

Mouna A, Anis J (2017) Financial literacy in Tunisia: its determinants and its implications on investment behavior. Res Int Bus Finance 39:568–577

Mullainathan S, Thaler R (2000) Massachusetts Institute of Technology Department of Economics Working Paper Series . September

Nga KH, Yeoh KK (2018) An exploratory model on retirement savings behaviour: a Malaysian study. Int J Bus Soc 19(3):637–659

OECD (2023) Old-age dependency ratio (indicator). https://doi.org/10.1787/e0255c98-en . Accessed 13 Oct 2023

Onwuegbuzie AJ, Collins KM (2007) A typology of mixed methods sampling designs in social science research. Qualitative Rep 12(2):474–498

Pallister JG, Wang HC, Foxall GR (2007) An application of the style/involvement model to financial services. Technovation 27(1–2):78–88. https://doi.org/10.1016/j.technovation.2005.10.001

Pan L, Pezzuti T, Lu W, Pechmann C (2019) Hyperopia and frugality: different motivational drivers and yet similar effects on consumer spending. J Bus Res 95(August 2018):347–356

Parise G, Peijnenburg K (2017) Understanding the Determinants of Financial Outcomes and Choices: The Role of Noncognitive Abilities. BIS Working Papers

Paul J, Rosado-Serrano A (2019) Gradual internationalization vs Born-Global/International new venture models: a review and research agenda. Int Mark Rev 36(6):830–858. https://doi.org/10.1108/IMR-10-2018-0280

Paul J, Criado AR (2020) The art of writing literature review: what do we know and what do we need to know? Int Bus Rev 29(4):101717. https://doi.org/10.1016/j.ibusrev.2020.101717

Paul J, Khatri P, Kaur Duggal H (2023) Frameworks for developing impactful systematic literature reviews and theory building: what, why and how? J Decis Syst 00(00):1–14. https://doi.org/10.1080/12460125.2023.2197700

Petkoska J, Earl JK (2009) Understanding the influence of demographic and psychological variables on Retirement Planning. Psychol Aging 24(1):245–251. https://doi.org/10.1037/a0014096

Piotrowska M (2019) The importance of personality characteristics and behavioral constraints for retirement saving. Econ Anal Policy 64:194–220

Plath DA, Stevenson TH (2005) Financial services consumption behavior across Hispanic American consumers. J Bus Res 58(8):1089–1099. https://doi.org/10.1016/j.jbusres.2004.03.003

Poterba JM (2015) Saver heterogeneity and the challenge of assessing retirement saving adequacy. Natl Tax J 68(2):377–388. https://doi.org/10.17310/ntj.2015.2.06

Potrich ACG, Vieira KM, Kirch G (2018) How well do women do when it comes to financial literacy? Proposition of an indicator and analysis of gender differences. J Behav Experimental Finance 17:28–41. https://doi.org/10.1016/j.jbef.2017.12.005

Rai D, Lin CW (2019) (Wilson). The influence of implicit self-theories on consumer financial decision making. Journal of Business Research , 95 (August 2018), 316–325. https://doi.org/10.1016/j.jbusres.2018.08.016

Ramalho TB, Forte D (2019) Financial literacy in Brazil – do knowledge and self-confidence relate with behavior? RAUSP Manage J 54(1):77–95. https://doi.org/10.1108/RAUSP-04-2018-0008

Rana J, Paul J (2017) Consumer behavior and purchase intention for organic food: a review and research agenda. J Retailing Consumer Serv 38(June):157–165. https://doi.org/10.1016/j.jretconser.2017.06.004

Ranyard R, McNair S, Nicolini G, Duxbury D (2020) An item response theory approach to constructing and evaluating brief and in-depth financial literacy scales. J Consum Aff 54(3):1121–1156. https://doi.org/10.1111/joca.12322

RBI Household Finance Committee (2017) Indian household finance. Reserve Bank of India, Mumbai

Ruefenacht M, Schlager T, Maas P, Puustinen P (2015) Drivers of long-term savings behavior from consumer’s perspective. Electron Libr 34(1):1–5

Scholz JK, Seshadri A, Khitatrakun S (2006) Are Americans saving “optimally” for retirement? J Polit Econ 114(4):607–643

Schuabb T, França LH, Amorim SM (2019) Retirement savings model tested with Brazilian private health care workers. Front Psychol 10(JULY):1–11. https://doi.org/10.3389/fpsyg.2019.01701

Schuhen M, Schurkmann S (2014) International Review of Economics Education. Int Rev Econ Educ 16:1–11

Segel-Karpas D, Werner P (2014) Perceived financial retirement preparedness and its correlates: a national study in Israel. Int J Aging Hum Dev 79(4):279–301. https://doi.org/10.1177/0091415015574177

Seth H, Talwar S, Bhatia A, Saxena A, Dhir A (2020) Consumer resistance and inertia of retail investors: Development of the resistance adoption inertia continuance (RAIC) framework. Journal of Retailing and Consumer Services , 55 (August 2019), 102071. https://doi.org/10.1016/j.jretconser.2020.102071

Sewell M (2008) Behavioural finance. Economist 389(8604):1–13. https://doi.org/10.1057/9780230280786_5

Shefrin HM, Thaler RH (1988) The behavioral life‐cycle hypothesis. Econ Inq 26(4):609–643

Shim S, Serido J, Tang C (2012) The ant and the grasshopper revisited: the present psychological benefits of saving and future oriented financial behavior. J Econ Psychol 33(1):155–165

Simon HA (1978) Information-processing theory of human problem solving. In: Handbook of learning and cognitive processes, vol 5, pp 271–295

Sivaramakrishnan S, Srivastava M, Rastogi A (2017) Attitudinal factors, financial literacy, and stock market participation. Int J Bank Mark 34(1):1–5

Snyder H (2019) Literature review as a research methodology: an overview and guidelines. J Bus Res 104(August):333–339. https://doi.org/10.1016/j.jbusres.2019.07.039

Stawski RS, Hershey DA, Jacobs-Lawson JM (2007) Goal clarity and financial planning activities as determinants of retirement savings contributions. Int J Aging Hum Dev 64(1):13–32. https://doi.org/10.2190/13GK-5H72-H324-16P2

Steinert JI, Zenker J, Filipiak U, Movsisyan A, Cluver LD, Shenderovich Y (2018) Do saving promotion interventions increase household savings, consumption, and investments in Sub-saharan Africa? A systematic review and meta-analysis. World Dev 104:238–256. https://doi.org/10.1016/j.worlddev.2017.11.018

Steinhart Y, Mazursky D (2010) Purchase availability and involvement antecedents among financial products. Int J Bank Mark 28(2):113–135. https://doi.org/10.1108/02652321011018314

Strömbäck C, Lind T, Skagerlund K, Västfjäll D, Tinghög G (2017) Does self-control predict financial behavior and financial well-being? J Behav Experimental Finance 14:30–38. https://doi.org/10.1016/j.jbef.2017.04.002

Strömbäck C, Skagerlund K, Västfjäll D, Tinghög G (2020) Subjective self-control but not objective measures of executive functions predict financial behavior and well-being. Journal of Behavioral and Experimental Finance , 27 . https://doi.org/10.1016/j.jbef.2020.100339

Tam L, Dholakia U (2014) Saving in cycles: how to get people to save more money. Psychol Sci 25(2):531–537. https://doi.org/10.1177/0956797613512129

Tang N, Baker A (2016) Self-esteem, financial knowledge and financial behavior. J Econ Psychol 54:164–176

Tate M, Evermann J, Gable G (2015) An integrated framework for theories of individual attitudes toward technology. Inform Manage 52(6):710–727. https://doi.org/10.1016/j.im.2015.06.005

Taylor MP, Jenkins SP, Sacker A (2011) Financial capability and psychological health. J Econ Psychol 32(5):710–723. https://doi.org/10.1016/j.joep.2011.05.006

Tennyson S, Yang HK (2014) The role of life experience in long-term care insurance decisions. Journal of Economic Psychology , 42 (2014), 175–188. https://doi.org/10.1016/j.joep.2014.04.002

Thaler BRH (1994) Psychology and savings policies. Am Econ Rev 84(2):175–179. http://www.jstor.org/stable/3132220

Thaler R (1980) Toward a positive theory of consumer choice. J Econ Behav Organ 1:39–60

Thaler RH (2005) Advances in behavioral finance. Adv Behav Finance 2:1–694. https://doi.org/10.2307/2329257

Thaler R, Shefrin H (1981) An economic theory of self-control. J Polit Econ 89(2):392–406

Tomar S, Kent Baker H, Kumar S, Hoffmann AOI (2021) Psychological determinants of retirement financial planning behavior. Journal of Business Research , 133 (November 2020), 432–449. https://doi.org/10.1016/j.jbusres.2021.05.007

Topa G, Moriano JA, Depolo M, Alcover CM, Morales JF (2009) Antecedents and consequences of retirement planning and decision-making: a meta-analysis and model. J Vocat Behav 75(1):38–55. https://doi.org/10.1016/j.jvb.2009.03.002

Topa G, Moriano JA, Depolo M, Alcover CM, Moreno A (2011) Retirement and wealth relationships: Meta-analysis and SEM. Res Aging 33(5):501–528. https://doi.org/10.1177/0164027511410549

Tranfield D, Denyer D, Smart P (2003) Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br J Manag 14(3):207–222. https://doi.org/10.1111/1467-8551.00375

Ülkümen G, Cheema A (2011) Framing goals to influence personal savings: the role of specificity and construal level. J Mark Res 48(6):958–969. https://doi.org/10.1509/jmr.09.0516

United Nations, Department of Economic and Social, Affairs PD (2020) (2019). World Population Ageing 2019. In United Nations . http://link.springer.com/chapter/10.1007/ 978-94-007-5204-7_6

Utkarsh, Pandey A, Ashta A, Spiegelman E, Sutan A (2020) Catch them young: impact of financial Socialization, financial literacy and attitude towards money on the financial well-being of young adults. Int J Consumer Stud 44(6):531–541. https://doi.org/10.1111/ijcs.12583

Valente TW, Paredes P, Poppe P (1998) Matching the message to the process: the relative ordering of knowledge, attitudes, and practices in behavior change research. Hum Commun Res 24(3):366–385

Van Rooij M, Teppa F (2014) Personal traits and individual choices: taking action in economic and non-economic decisions. J Econ Behav Organ 100:33–43

van Rooij M, Lusardi A, Alessie R (2011) Financial literacy and stock market participation. J Financ Econ 101(2):449–472. https://doi.org/10.1016/j.jfineco.2011.03.006

Van Rooij MCJ, Lusardi A, Alessie RJM (2011a) Financial literacy and retirement planning in the Netherlands. J Econ Psychol 32(4):593–608. https://doi.org/10.1016/j.joep.2011.02.004

van Schie RJG, Dellaert BGC, Donkers B (2015) Promoting later planned retirement: construal level intervention impact reverses with age. J Econ Psychol 50:124–131. https://doi.org/10.1016/j.joep.2015.06.010

Venkatesh V, Morris M, Davis G, Davis F (2003) Factors influencing the Use of M-Banking by academics: Case Study sms-based M-Banking. MIS Q 27(3):425–478

Vitt LA (2004) Consumers’ financial decisions and the psychology of values. J Financial Service Professionals 58(November):68–77. http://search.ebscohost.com/login.aspx?direct=true &db=bth&AN=14888952&site=ehost-live

Wang L, Lu W, Malhotra NK (2011) Demographics, attitude, personality, and credit card features correlate with credit card debt: a view from China. J Econ Psychol 32(1):179–193. https://doi.org/10.1016/j.joep.2010.11.006

World Economic Forum (2019) Investing in (and for) our future. Issue June. www.weforum.org

Xia T, Wang Z, Li K (2014) Financial literacy overconfidence and stock market participation. Soc Indic Res 119(3):1233–1245. https://doi.org/10.1007/s11205-013-0555-9

Xiao JJ, Chen C, Chen F (2014) Consumer financial capability and financial satisfaction. Soc Indic Res 118(1):415–432. https://doi.org/10.1007/s11205-013-0414-8

Yeung DY, Zhou X (2017) Planning for retirement: longitudinal effect on retirement resources and post-retirement well-being. Front Psychol 8:1300

Zhou R, Pham MT (2004) Promotion and prevention across mental accounts: when financial products dictate consumers’ investment goals. J Consum Res 31(1):125–135. https://doi.org/10.1086/383429

Download references

Acknowledgements

Authors would like to acknowledge the academicians and researchers who guided the search of the article and would like to thank the experts for the valuable inputs to refine the work.

There is no funding received for this research.

Author information

Authors and affiliations.

Symbiosis International (Deemed University), Pune, India

Kavita Karan Ingale

Symbiosis Institute of Operations Management, Symbiosis International (Deemed) University, Pune, India

Ratna Achuta Paluri

You can also search for this author in PubMed   Google Scholar

Contributions

Both authors contributed to the conceptualization, research design, methodology, analysis of the data,writing of the manuscript and its revision.

Corresponding author

Correspondence to Kavita Karan Ingale .

Ethics declarations

Conflict of interest.

The authors declare that there is no conflict of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Ingale, K.K., Paluri, R.A. Retirement planning – a systematic review of literature and future research directions. Manag Rev Q (2023). https://doi.org/10.1007/s11301-023-00377-x

Download citation

Received : 14 December 2022

Accepted : 04 October 2023

Published : 28 October 2023

DOI : https://doi.org/10.1007/s11301-023-00377-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Retirement planning
• Systematic literature review
• Financial behavior
• Household finance
• Long-term savings
• Pension plan
• Financial literacy
• TCCM framework
• Find a journal
• Publish with us
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Wiley-Blackwell Online Open

An overview of methodological approaches in systematic reviews

Prabhakar veginadu.

1 Department of Rural Clinical Sciences, La Trobe Rural Health School, La Trobe University, Bendigo Victoria, Australia

Hanny Calache

2 Lincoln International Institute for Rural Health, University of Lincoln, Brayford Pool, Lincoln UK

Akshaya Pandian

3 Department of Orthodontics, Saveetha Dental College, Chennai Tamil Nadu, India

Mohd Masood

Associated data.

APPENDIX B: List of excluded studies with detailed reasons for exclusion

APPENDIX C: Quality assessment of included reviews using AMSTAR 2

The aim of this overview is to identify and collate evidence from existing published systematic review (SR) articles evaluating various methodological approaches used at each stage of an SR.

The search was conducted in five electronic databases from inception to November 2020 and updated in February 2022: MEDLINE, Embase, Web of Science Core Collection, Cochrane Database of Systematic Reviews, and APA PsycINFO. Title and abstract screening were performed in two stages by one reviewer, supported by a second reviewer. Full‐text screening, data extraction, and quality appraisal were performed by two reviewers independently. The quality of the included SRs was assessed using the AMSTAR 2 checklist.

The search retrieved 41,556 unique citations, of which 9 SRs were deemed eligible for inclusion in final synthesis. Included SRs evaluated 24 unique methodological approaches used for defining the review scope and eligibility, literature search, screening, data extraction, and quality appraisal in the SR process. Limited evidence supports the following (a) searching multiple resources (electronic databases, handsearching, and reference lists) to identify relevant literature; (b) excluding non‐English, gray, and unpublished literature, and (c) use of text‐mining approaches during title and abstract screening.

The overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process, as well as some methodological modifications currently used in expedited SRs. Overall, findings of this overview highlight the dearth of published SRs focused on SR methodologies and this warrants future work in this area.

1. INTRODUCTION

Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the “gold standard” of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search, appraise, and synthesize the available evidence. 3 Several guidelines, developed by various organizations, are available for the conduct of an SR; 4 , 5 , 6 , 7 among these, Cochrane is considered a pioneer in developing rigorous and highly structured methodology for the conduct of SRs. 8 The guidelines developed by these organizations outline seven fundamental steps required in SR process: defining the scope of the review and eligibility criteria, literature searching and retrieval, selecting eligible studies, extracting relevant data, assessing risk of bias (RoB) in included studies, synthesizing results, and assessing certainty of evidence (CoE) and presenting findings. 4 , 5 , 6 , 7

The methodological rigor involved in an SR can require a significant amount of time and resource, which may not always be available. 9 As a result, there has been a proliferation of modifications made to the traditional SR process, such as refining, shortening, bypassing, or omitting one or more steps, 10 , 11 for example, limits on the number and type of databases searched, limits on publication date, language, and types of studies included, and limiting to one reviewer for screening and selection of studies, as opposed to two or more reviewers. 10 , 11 These methodological modifications are made to accommodate the needs of and resource constraints of the reviewers and stakeholders (e.g., organizations, policymakers, health care professionals, and other knowledge users). While such modifications are considered time and resource efficient, they may introduce bias in the review process reducing their usefulness. 5

Substantial research has been conducted examining various approaches used in the standardized SR methodology and their impact on the validity of SR results. There are a number of published reviews examining the approaches or modifications corresponding to single 12 , 13 or multiple steps 14 involved in an SR. However, there is yet to be a comprehensive summary of the SR‐level evidence for all the seven fundamental steps in an SR. Such a holistic evidence synthesis will provide an empirical basis to confirm the validity of current accepted practices in the conduct of SRs. Furthermore, sometimes there is a balance that needs to be achieved between the resource availability and the need to synthesize the evidence in the best way possible, given the constraints. This evidence base will also inform the choice of modifications to be made to the SR methods, as well as the potential impact of these modifications on the SR results. An overview is considered the choice of approach for summarizing existing evidence on a broad topic, directing the reader to evidence, or highlighting the gaps in evidence, where the evidence is derived exclusively from SRs. 15 Therefore, for this review, an overview approach was used to (a) identify and collate evidence from existing published SR articles evaluating various methodological approaches employed in each of the seven fundamental steps of an SR and (b) highlight both the gaps in the current research and the potential areas for future research on the methods employed in SRs.

An a priori protocol was developed for this overview but was not registered with the International Prospective Register of Systematic Reviews (PROSPERO), as the review was primarily methodological in nature and did not meet PROSPERO eligibility criteria for registration. The protocol is available from the corresponding author upon reasonable request. This overview was conducted based on the guidelines for the conduct of overviews as outlined in The Cochrane Handbook. 15 Reporting followed the Preferred Reporting Items for Systematic reviews and Meta‐analyses (PRISMA) statement. 3

2.1. Eligibility criteria

Only published SRs, with or without associated MA, were included in this overview. We adopted the defining characteristics of SRs from The Cochrane Handbook. 5 According to The Cochrane Handbook, a review was considered systematic if it satisfied the following criteria: (a) clearly states the objectives and eligibility criteria for study inclusion; (b) provides reproducible methodology; (c) includes a systematic search to identify all eligible studies; (d) reports assessment of validity of findings of included studies (e.g., RoB assessment of the included studies); (e) systematically presents all the characteristics or findings of the included studies. 5 Reviews that did not meet all of the above criteria were not considered a SR for this study and were excluded. MA‐only articles were included if it was mentioned that the MA was based on an SR.

SRs and/or MA of primary studies evaluating methodological approaches used in defining review scope and study eligibility, literature search, study selection, data extraction, RoB assessment, data synthesis, and CoE assessment and reporting were included. The methodological approaches examined in these SRs and/or MA can also be related to the substeps or elements of these steps; for example, applying limits on date or type of publication are the elements of literature search. Included SRs examined or compared various aspects of a method or methods, and the associated factors, including but not limited to: precision or effectiveness; accuracy or reliability; impact on the SR and/or MA results; reproducibility of an SR steps or bias occurred; time and/or resource efficiency. SRs assessing the methodological quality of SRs (e.g., adherence to reporting guidelines), evaluating techniques for building search strategies or the use of specific database filters (e.g., use of Boolean operators or search filters for randomized controlled trials), examining various tools used for RoB or CoE assessment (e.g., ROBINS vs. Cochrane RoB tool), or evaluating statistical techniques used in meta‐analyses were excluded. 14

2.2. Search

The search for published SRs was performed on the following scientific databases initially from inception to third week of November 2020 and updated in the last week of February 2022: MEDLINE (via Ovid), Embase (via Ovid), Web of Science Core Collection, Cochrane Database of Systematic Reviews, and American Psychological Association (APA) PsycINFO. Search was restricted to English language publications. Following the objectives of this study, study design filters within databases were used to restrict the search to SRs and MA, where available. The reference lists of included SRs were also searched for potentially relevant publications.

The search terms included keywords, truncations, and subject headings for the key concepts in the review question: SRs and/or MA, methods, and evaluation. Some of the terms were adopted from the search strategy used in a previous review by Robson et al., which reviewed primary studies on methodological approaches used in study selection, data extraction, and quality appraisal steps of SR process. 14 Individual search strategies were developed for respective databases by combining the search terms using appropriate proximity and Boolean operators, along with the related subject headings in order to identify SRs and/or MA. 16 , 17 A senior librarian was consulted in the design of the search terms and strategy. Appendix A presents the detailed search strategies for all five databases.

2.3. Study selection and data extraction

Title and abstract screening of references were performed in three steps. First, one reviewer (PV) screened all the titles and excluded obviously irrelevant citations, for example, articles on topics not related to SRs, non‐SR publications (such as randomized controlled trials, observational studies, scoping reviews, etc.). Next, from the remaining citations, a random sample of 200 titles and abstracts were screened against the predefined eligibility criteria by two reviewers (PV and MM), independently, in duplicate. Discrepancies were discussed and resolved by consensus. This step ensured that the responses of the two reviewers were calibrated for consistency in the application of the eligibility criteria in the screening process. Finally, all the remaining titles and abstracts were reviewed by a single “calibrated” reviewer (PV) to identify potential full‐text records. Full‐text screening was performed by at least two authors independently (PV screened all the records, and duplicate assessment was conducted by MM, HC, or MG), with discrepancies resolved via discussions or by consulting a third reviewer.

Data related to review characteristics, results, key findings, and conclusions were extracted by at least two reviewers independently (PV performed data extraction for all the reviews and duplicate extraction was performed by AP, HC, or MG).

2.4. Quality assessment of included reviews

The quality assessment of the included SRs was performed using the AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews). The tool consists of a 16‐item checklist addressing critical and noncritical domains. 18 For the purpose of this study, the domain related to MA was reclassified from critical to noncritical, as SRs with and without MA were included. The other six critical domains were used according to the tool guidelines. 18 Two reviewers (PV and AP) independently responded to each of the 16 items in the checklist with either “yes,” “partial yes,” or “no.” Based on the interpretations of the critical and noncritical domains, the overall quality of the review was rated as high, moderate, low, or critically low. 18 Disagreements were resolved through discussion or by consulting a third reviewer.

2.5. Data synthesis

To provide an understandable summary of existing evidence syntheses, characteristics of the methods evaluated in the included SRs were examined and key findings were categorized and presented based on the corresponding step in the SR process. The categories of key elements within each step were discussed and agreed by the authors. Results of the included reviews were tabulated and summarized descriptively, along with a discussion on any overlap in the primary studies. 15 No quantitative analyses of the data were performed.

From 41,556 unique citations identified through literature search, 50 full‐text records were reviewed, and nine systematic reviews 14 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 were deemed eligible for inclusion. The flow of studies through the screening process is presented in Figure  1 . A list of excluded studies with reasons can be found in Appendix B .

Study selection flowchart

3.1. Characteristics of included reviews

Table  1 summarizes the characteristics of included SRs. The majority of the included reviews (six of nine) were published after 2010. 14 , 22 , 23 , 24 , 25 , 26 Four of the nine included SRs were Cochrane reviews. 20 , 21 , 22 , 23 The number of databases searched in the reviews ranged from 2 to 14, 2 reviews searched gray literature sources, 24 , 25 and 7 reviews included a supplementary search strategy to identify relevant literature. 14 , 19 , 20 , 21 , 22 , 23 , 26 Three of the included SRs (all Cochrane reviews) included an integrated MA. 20 , 21 , 23

Characteristics of included studies

SR = systematic review; MA = meta‐analysis; RCT = randomized controlled trial; CCT = controlled clinical trial; N/R = not reported.

The included SRs evaluated 24 unique methodological approaches (26 in total) used across five steps in the SR process; 8 SRs evaluated 6 approaches, 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 while 1 review evaluated 18 approaches. 14 Exclusion of gray or unpublished literature 21 , 26 and blinding of reviewers for RoB assessment 14 , 23 were evaluated in two reviews each. Included SRs evaluated methods used in five different steps in the SR process, including methods used in defining the scope of review ( n  = 3), literature search ( n  = 3), study selection ( n  = 2), data extraction ( n  = 1), and RoB assessment ( n  = 2) (Table  2 ).

Summary of findings from review evaluating systematic review methods

There was some overlap in the primary studies evaluated in the included SRs on the same topics: Schmucker et al. 26 and Hopewell et al. 21 ( n  = 4), Hopewell et al. 20 and Crumley et al. 19 ( n  = 30), and Robson et al. 14 and Morissette et al. 23 ( n  = 4). There were no conflicting results between any of the identified SRs on the same topic.

3.2. Methodological quality of included reviews

Overall, the quality of the included reviews was assessed as moderate at best (Table  2 ). The most common critical weakness in the reviews was failure to provide justification for excluding individual studies (four reviews). Detailed quality assessment is provided in Appendix C .

3.3. Evidence on systematic review methods

3.3.1. methods for defining review scope and eligibility.

Two SRs investigated the effect of excluding data obtained from gray or unpublished sources on the pooled effect estimates of MA. 21 , 26 Hopewell et al. 21 reviewed five studies that compared the impact of gray literature on the results of a cohort of MA of RCTs in health care interventions. Gray literature was defined as information published in “print or electronic sources not controlled by commercial or academic publishers.” Findings showed an overall greater treatment effect for published trials than trials reported in gray literature. In a more recent review, Schmucker et al. 26 addressed similar objectives, by investigating gray and unpublished data in medicine. In addition to gray literature, defined similar to the previous review by Hopewell et al., the authors also evaluated unpublished data—defined as “supplemental unpublished data related to published trials, data obtained from the Food and Drug Administration  or other regulatory websites or postmarketing analyses hidden from the public.” The review found that in majority of the MA, excluding gray literature had little or no effect on the pooled effect estimates. The evidence was limited to conclude if the data from gray and unpublished literature had an impact on the conclusions of MA. 26

Morrison et al. 24 examined five studies measuring the effect of excluding non‐English language RCTs on the summary treatment effects of SR‐based MA in various fields of conventional medicine. Although none of the included studies reported major difference in the treatment effect estimates between English only and non‐English inclusive MA, the review found inconsistent evidence regarding the methodological and reporting quality of English and non‐English trials. 24 As such, there might be a risk of introducing “language bias” when excluding non‐English language RCTs. The authors also noted that the numbers of non‐English trials vary across medical specialties, as does the impact of these trials on MA results. Based on these findings, Morrison et al. 24 conclude that literature searches must include non‐English studies when resources and time are available to minimize the risk of introducing “language bias.”

3.3.2. Methods for searching studies

Crumley et al. 19 analyzed recall (also referred to as “sensitivity” by some researchers; defined as “percentage of relevant studies identified by the search”) and precision (defined as “percentage of studies identified by the search that were relevant”) when searching a single resource to identify randomized controlled trials and controlled clinical trials, as opposed to searching multiple resources. The studies included in their review frequently compared a MEDLINE only search with the search involving a combination of other resources. The review found low median recall estimates (median values between 24% and 92%) and very low median precisions (median values between 0% and 49%) for most of the electronic databases when searched singularly. 19 A between‐database comparison, based on the type of search strategy used, showed better recall and precision for complex and Cochrane Highly Sensitive search strategies (CHSSS). In conclusion, the authors emphasize that literature searches for trials in SRs must include multiple sources. 19

In an SR comparing handsearching and electronic database searching, Hopewell et al. 20 found that handsearching retrieved more relevant RCTs (retrieval rate of 92%−100%) than searching in a single electronic database (retrieval rates of 67% for PsycINFO/PsycLIT, 55% for MEDLINE, and 49% for Embase). The retrieval rates varied depending on the quality of handsearching, type of electronic search strategy used (e.g., simple, complex or CHSSS), and type of trial reports searched (e.g., full reports, conference abstracts, etc.). The authors concluded that handsearching was particularly important in identifying full trials published in nonindexed journals and in languages other than English, as well as those published as abstracts and letters. 20

The effectiveness of checking reference lists to retrieve additional relevant studies for an SR was investigated by Horsley et al. 22 The review reported that checking reference lists yielded 2.5%–40% more studies depending on the quality and comprehensiveness of the electronic search used. The authors conclude that there is some evidence, although from poor quality studies, to support use of checking reference lists to supplement database searching. 22

3.3.3. Methods for selecting studies

Three approaches relevant to reviewer characteristics, including number, experience, and blinding of reviewers involved in the screening process were highlighted in an SR by Robson et al. 14 Based on the retrieved evidence, the authors recommended that two independent, experienced, and unblinded reviewers be involved in study selection. 14 A modified approach has also been suggested by the review authors, where one reviewer screens and the other reviewer verifies the list of excluded studies, when the resources are limited. It should be noted however this suggestion is likely based on the authors’ opinion, as there was no evidence related to this from the studies included in the review.

Robson et al. 14 also reported two methods describing the use of technology for screening studies: use of Google Translate for translating languages (for example, German language articles to English) to facilitate screening was considered a viable method, while using two computer monitors for screening did not increase the screening efficiency in SR. Title‐first screening was found to be more efficient than simultaneous screening of titles and abstracts, although the gain in time with the former method was lesser than the latter. Therefore, considering that the search results are routinely exported as titles and abstracts, Robson et al. 14 recommend screening titles and abstracts simultaneously. However, the authors note that these conclusions were based on very limited number (in most instances one study per method) of low‐quality studies. 14

3.3.4. Methods for data extraction

Robson et al. 14 examined three approaches for data extraction relevant to reviewer characteristics, including number, experience, and blinding of reviewers (similar to the study selection step). Although based on limited evidence from a small number of studies, the authors recommended use of two experienced and unblinded reviewers for data extraction. The experience of the reviewers was suggested to be especially important when extracting continuous outcomes (or quantitative) data. However, when the resources are limited, data extraction by one reviewer and a verification of the outcomes data by a second reviewer was recommended.

As for the methods involving use of technology, Robson et al. 14 identified limited evidence on the use of two monitors to improve the data extraction efficiency and computer‐assisted programs for graphical data extraction. However, use of Google Translate for data extraction in non‐English articles was not considered to be viable. 14 In the same review, Robson et al. 14 identified evidence supporting contacting authors for obtaining additional relevant data.

3.3.5. Methods for RoB assessment

Two SRs examined the impact of blinding of reviewers for RoB assessments. 14 , 23 Morissette et al. 23 investigated the mean differences between the blinded and unblinded RoB assessment scores and found inconsistent differences among the included studies providing no definitive conclusions. Similar conclusions were drawn in a more recent review by Robson et al., 14 which included four studies on reviewer blinding for RoB assessment that completely overlapped with Morissette et al. 23

Use of experienced reviewers and provision of additional guidance for RoB assessment were examined by Robson et al. 14 The review concluded that providing intensive training and guidance on assessing studies reporting insufficient data to the reviewers improves RoB assessments. 14 Obtaining additional data related to quality assessment by contacting study authors was also found to help the RoB assessments, although based on limited evidence. When assessing the qualitative or mixed method reviews, Robson et al. 14 recommends the use of a structured RoB tool as opposed to an unstructured tool. No SRs were identified on data synthesis and CoE assessment and reporting steps.

4. DISCUSSION

4.1. summary of findings.

Nine SRs examining 24 unique methods used across five steps in the SR process were identified in this overview. The collective evidence supports some current traditional and modified SR practices, while challenging other approaches. However, the quality of the included reviews was assessed to be moderate at best and in the majority of the included SRs, evidence related to the evaluated methods was obtained from very limited numbers of primary studies. As such, the interpretations from these SRs should be made cautiously.

The evidence gathered from the included SRs corroborate a few current SR approaches. 5 For example, it is important to search multiple resources for identifying relevant trials (RCTs and/or CCTs). The resources must include a combination of electronic database searching, handsearching, and reference lists of retrieved articles. 5 However, no SRs have been identified that evaluated the impact of the number of electronic databases searched. A recent study by Halladay et al. 27 found that articles on therapeutic intervention, retrieved by searching databases other than PubMed (including Embase), contributed only a small amount of information to the MA and also had a minimal impact on the MA results. The authors concluded that when the resources are limited and when large number of studies are expected to be retrieved for the SR or MA, PubMed‐only search can yield reliable results. 27

Findings from the included SRs also reiterate some methodological modifications currently employed to “expedite” the SR process. 10 , 11 For example, excluding non‐English language trials and gray/unpublished trials from MA have been shown to have minimal or no impact on the results of MA. 24 , 26 However, the efficiency of these SR methods, in terms of time and the resources used, have not been evaluated in the included SRs. 24 , 26 Of the SRs included, only two have focused on the aspect of efficiency 14 , 25 ; O'Mara‐Eves et al. 25 report some evidence to support the use of text‐mining approaches for title and abstract screening in order to increase the rate of screening. Moreover, only one included SR 14 considered primary studies that evaluated reliability (inter‐ or intra‐reviewer consistency) and accuracy (validity when compared against a “gold standard” method) of the SR methods. This can be attributed to the limited number of primary studies that evaluated these outcomes when evaluating the SR methods. 14 Lack of outcome measures related to reliability, accuracy, and efficiency precludes making definitive recommendations on the use of these methods/modifications. Future research studies must focus on these outcomes.

Some evaluated methods may be relevant to multiple steps; for example, exclusions based on publication status (gray/unpublished literature) and language of publication (non‐English language studies) can be outlined in the a priori eligibility criteria or can be incorporated as search limits in the search strategy. SRs included in this overview focused on the effect of study exclusions on pooled treatment effect estimates or MA conclusions. Excluding studies from the search results, after conducting a comprehensive search, based on different eligibility criteria may yield different results when compared to the results obtained when limiting the search itself. 28 Further studies are required to examine this aspect.

Although we acknowledge the lack of standardized quality assessment tools for methodological study designs, we adhered to the Cochrane criteria for identifying SRs in this overview. This was done to ensure consistency in the quality of the included evidence. As a result, we excluded three reviews that did not provide any form of discussion on the quality of the included studies. The methods investigated in these reviews concern supplementary search, 29 data extraction, 12 and screening. 13 However, methods reported in two of these three reviews, by Mathes et al. 12 and Waffenschmidt et al., 13 have also been examined in the SR by Robson et al., 14 which was included in this overview; in most instances (with the exception of one study included in Mathes et al. 12 and Waffenschmidt et al. 13 each), the studies examined in these excluded reviews overlapped with those in the SR by Robson et al. 14

One of the key gaps in the knowledge observed in this overview was the dearth of SRs on the methods used in the data synthesis component of SR. Narrative and quantitative syntheses are the two most commonly used approaches for synthesizing data in evidence synthesis. 5 There are some published studies on the proposed indications and implications of these two approaches. 30 , 31 These studies found that both data synthesis methods produced comparable results and have their own advantages, suggesting that the choice of the method must be based on the purpose of the review. 31 With increasing number of “expedited” SR approaches (so called “rapid reviews”) avoiding MA, 10 , 11 further research studies are warranted in this area to determine the impact of the type of data synthesis on the results of the SR.

4.2. Implications for future research

The findings of this overview highlight several areas of paucity in primary research and evidence synthesis on SR methods. First, no SRs were identified on methods used in two important components of the SR process, including data synthesis and CoE and reporting. As for the included SRs, a limited number of evaluation studies have been identified for several methods. This indicates that further research is required to corroborate many of the methods recommended in current SR guidelines. 4 , 5 , 6 , 7 Second, some SRs evaluated the impact of methods on the results of quantitative synthesis and MA conclusions. Future research studies must also focus on the interpretations of SR results. 28 , 32 Finally, most of the included SRs were conducted on specific topics related to the field of health care, limiting the generalizability of the findings to other areas. It is important that future research studies evaluating evidence syntheses broaden the objectives and include studies on different topics within the field of health care.

4.3. Strengths and limitations

To our knowledge, this is the first overview summarizing current evidence from SRs and MA on different methodological approaches used in several fundamental steps in SR conduct. The overview methodology followed well established guidelines and strict criteria defined for the inclusion of SRs.

There are several limitations related to the nature of the included reviews. Evidence for most of the methods investigated in the included reviews was derived from a limited number of primary studies. Also, the majority of the included SRs may be considered outdated as they were published (or last updated) more than 5 years ago 33 ; only three of the nine SRs have been published in the last 5 years. 14 , 25 , 26 Therefore, important and recent evidence related to these topics may not have been included. Substantial numbers of included SRs were conducted in the field of health, which may limit the generalizability of the findings. Some method evaluations in the included SRs focused on quantitative analyses components and MA conclusions only. As such, the applicability of these findings to SR more broadly is still unclear. 28 Considering the methodological nature of our overview, limiting the inclusion of SRs according to the Cochrane criteria might have resulted in missing some relevant evidence from those reviews without a quality assessment component. 12 , 13 , 29 Although the included SRs performed some form of quality appraisal of the included studies, most of them did not use a standardized RoB tool, which may impact the confidence in their conclusions. Due to the type of outcome measures used for the method evaluations in the primary studies and the included SRs, some of the identified methods have not been validated against a reference standard.

Some limitations in the overview process must be noted. While our literature search was exhaustive covering five bibliographic databases and supplementary search of reference lists, no gray sources or other evidence resources were searched. Also, the search was primarily conducted in health databases, which might have resulted in missing SRs published in other fields. Moreover, only English language SRs were included for feasibility. As the literature search retrieved large number of citations (i.e., 41,556), the title and abstract screening was performed by a single reviewer, calibrated for consistency in the screening process by another reviewer, owing to time and resource limitations. These might have potentially resulted in some errors when retrieving and selecting relevant SRs. The SR methods were grouped based on key elements of each recommended SR step, as agreed by the authors. This categorization pertains to the identified set of methods and should be considered subjective.

5. CONCLUSIONS

This overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process. Limited evidence was also identified on some methodological modifications currently used to expedite the SR process. Overall, findings highlight the dearth of SRs on SR methodologies, warranting further work to confirm several current recommendations on conventional and expedited SR processes.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.

Supporting information

APPENDIX A: Detailed search strategies

ACKNOWLEDGMENTS

The first author is supported by a La Trobe University Full Fee Research Scholarship and a Graduate Research Scholarship.

Open Access Funding provided by La Trobe University.

Veginadu P, Calache H, Gussy M, Pandian A, Masood M. An overview of methodological approaches in systematic reviews . J Evid Based Med . 2022; 15 :39–54. 10.1111/jebm.12468 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

Systematic Sampling | A Step-by-Step Guide with Examples

Published on October 2, 2020 by Lauren Thomas . Revised on December 18, 2023.

Systematic sampling is a probability sampling method in which researchers select members of the population at a regular interval (or k ) determined in advance.

If the population order is random or random-like (e.g., alphabetical), then this method will give you a representative sample that can be used to draw conclusions about your population of interest.

Table of contents

When to use systematic sampling, step 1: define your population, step 2: decide on your sample size, step 3: calculate sampling interval k, step 4: select the sample and collect data, other interesting articles, frequently asked questions about systematic sampling.

Systematic sampling is a method that imitates many of the randomization benefits of simple random sampling , but is slightly easier to conduct.

You can use systematic sampling with a list of the entire population , like you would in simple random sampling. However, unlike with simple random sampling, you can also use this method when you’re unable to access a list of your population in advance.

Order of the population

When using systematic sampling with a population list, it’s essential to consider the order in which your population is listed to ensure that your sample is valid .

If your population is in ascending or descending order, using systematic sampling should still give you a fairly representative sample, as it will include participants from both the bottom and top ends of the population.

For example, if you are sampling from a list of individuals ordered by age, systematic sampling will result in a population drawn from the entire age spectrum. If you instead used simple random sampling, it is possible (although unlikely) that you would end up with only younger or older individuals.

You should not use systematic sampling if your population is ordered cyclically or periodically, as your resulting sample cannot be guaranteed to be representative.

Systematic sampling without a population list

You can use systematic sampling to imitate the randomization of simple random sampling when you don’t have access to a full list of the population in advance.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Like other methods of sampling, you must decide upon the population that you are studying.

In systematic sampling, you have two choices for data collection :

• You can select your sample ahead of time from a list and then approach the selected subjects to collect data, or
• You can approach every k th member of your target population to ask them to participate in your study.

Listing the population in advance

Ensure that your list contains the entire population and is not in a periodic or cyclic order. Ideally, it should be in a random or random-like (such as alphabetical) order, which will allow you to imitate the randomization benefits of simple random sampling .

Selecting your sample on the spot

If you cannot access a list in advance, but you are able to physically observe the population, you can also use systematic sampling to select subjects at the moment of data collection.

In this case, ensure that the timing and location of your sampling procedure covers the full population to avoid bias in the results.

Before you choose your interval, you must first decide on your sample size. It’s important to choose a representative number in order to avoid sampling bias . There are several different ways to choose a sample size, but one of the most common involves using a sample size calculator .

Once you have chosen your desired margin of error and confidence level , estimated total size of the population, and the standard deviation of the variables you are attempting to measure, this calculator will provide you with the sample size you should aim for.

When you know your target sample size, you can calculate your interval, k , by dividing your total estimated population size by your sample size. This can be a rough estimate rather than an exact calculation.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

If you already have a list of your population, randomly select a starting point on your list, and from there, select every k th member of the population to include in your sample.

If you don’t have a list, you choose every k th member of the population for your sample at the same time as collecting the data for your study.

As in simple random sampling , you should try to make sure every individual you have chosen for your sample actually participates in your study. If those who decide to participate do so for reasons connected with the variables that you are collecting, this could cause research bias to affect your study.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

Probability sampling means that every member of the target population has a known chance of being included in the sample.

Probability sampling methods include simple random sampling , systematic sampling , stratified sampling , and cluster sampling .

Systematic sampling is a probability sampling method where researchers select members of the population at a regular interval – for example, by selecting every 15th person on a list of the population. If the population is in a random order, this can imitate the benefits of simple random sampling .

There are three key steps in systematic sampling :

• Define and list your population , ensuring that it is not ordered in a cyclical or periodic order.
• Decide on your sample size and calculate your interval, k , by dividing your population by your target sample size.
• Choose every k th member of the population as your sample.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Thomas, L. (2023, December 18). Systematic Sampling | A Step-by-Step Guide with Examples. Scribbr. Retrieved April 2, 2024, from https://www.scribbr.com/methodology/systematic-sampling/

Is this article helpful?

Lauren Thomas

Other students also liked, simple random sampling | definition, steps & examples, stratified sampling | definition, guide & examples, cluster sampling | a simple step-by-step guide with examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”