methodologies systematic literature review

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Wiley-Blackwell Online Open

An overview of methodological approaches in systematic reviews

Prabhakar veginadu.

1 Department of Rural Clinical Sciences, La Trobe Rural Health School, La Trobe University, Bendigo Victoria, Australia

Hanny Calache

2 Lincoln International Institute for Rural Health, University of Lincoln, Brayford Pool, Lincoln UK

Akshaya Pandian

3 Department of Orthodontics, Saveetha Dental College, Chennai Tamil Nadu, India

Mohd Masood

Associated data.

APPENDIX B: List of excluded studies with detailed reasons for exclusion

APPENDIX C: Quality assessment of included reviews using AMSTAR 2

The aim of this overview is to identify and collate evidence from existing published systematic review (SR) articles evaluating various methodological approaches used at each stage of an SR.

The search was conducted in five electronic databases from inception to November 2020 and updated in February 2022: MEDLINE, Embase, Web of Science Core Collection, Cochrane Database of Systematic Reviews, and APA PsycINFO. Title and abstract screening were performed in two stages by one reviewer, supported by a second reviewer. Full‐text screening, data extraction, and quality appraisal were performed by two reviewers independently. The quality of the included SRs was assessed using the AMSTAR 2 checklist.

The search retrieved 41,556 unique citations, of which 9 SRs were deemed eligible for inclusion in final synthesis. Included SRs evaluated 24 unique methodological approaches used for defining the review scope and eligibility, literature search, screening, data extraction, and quality appraisal in the SR process. Limited evidence supports the following (a) searching multiple resources (electronic databases, handsearching, and reference lists) to identify relevant literature; (b) excluding non‐English, gray, and unpublished literature, and (c) use of text‐mining approaches during title and abstract screening.

The overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process, as well as some methodological modifications currently used in expedited SRs. Overall, findings of this overview highlight the dearth of published SRs focused on SR methodologies and this warrants future work in this area.

1. INTRODUCTION

Evidence synthesis is a prerequisite for knowledge translation. 1 A well conducted systematic review (SR), often in conjunction with meta‐analyses (MA) when appropriate, is considered the “gold standard” of methods for synthesizing evidence related to a topic of interest. 2 The central strength of an SR is the transparency of the methods used to systematically search, appraise, and synthesize the available evidence. 3 Several guidelines, developed by various organizations, are available for the conduct of an SR; 4 , 5 , 6 , 7 among these, Cochrane is considered a pioneer in developing rigorous and highly structured methodology for the conduct of SRs. 8 The guidelines developed by these organizations outline seven fundamental steps required in SR process: defining the scope of the review and eligibility criteria, literature searching and retrieval, selecting eligible studies, extracting relevant data, assessing risk of bias (RoB) in included studies, synthesizing results, and assessing certainty of evidence (CoE) and presenting findings. 4 , 5 , 6 , 7

The methodological rigor involved in an SR can require a significant amount of time and resource, which may not always be available. 9 As a result, there has been a proliferation of modifications made to the traditional SR process, such as refining, shortening, bypassing, or omitting one or more steps, 10 , 11 for example, limits on the number and type of databases searched, limits on publication date, language, and types of studies included, and limiting to one reviewer for screening and selection of studies, as opposed to two or more reviewers. 10 , 11 These methodological modifications are made to accommodate the needs of and resource constraints of the reviewers and stakeholders (e.g., organizations, policymakers, health care professionals, and other knowledge users). While such modifications are considered time and resource efficient, they may introduce bias in the review process reducing their usefulness. 5

Substantial research has been conducted examining various approaches used in the standardized SR methodology and their impact on the validity of SR results. There are a number of published reviews examining the approaches or modifications corresponding to single 12 , 13 or multiple steps 14 involved in an SR. However, there is yet to be a comprehensive summary of the SR‐level evidence for all the seven fundamental steps in an SR. Such a holistic evidence synthesis will provide an empirical basis to confirm the validity of current accepted practices in the conduct of SRs. Furthermore, sometimes there is a balance that needs to be achieved between the resource availability and the need to synthesize the evidence in the best way possible, given the constraints. This evidence base will also inform the choice of modifications to be made to the SR methods, as well as the potential impact of these modifications on the SR results. An overview is considered the choice of approach for summarizing existing evidence on a broad topic, directing the reader to evidence, or highlighting the gaps in evidence, where the evidence is derived exclusively from SRs. 15 Therefore, for this review, an overview approach was used to (a) identify and collate evidence from existing published SR articles evaluating various methodological approaches employed in each of the seven fundamental steps of an SR and (b) highlight both the gaps in the current research and the potential areas for future research on the methods employed in SRs.

An a priori protocol was developed for this overview but was not registered with the International Prospective Register of Systematic Reviews (PROSPERO), as the review was primarily methodological in nature and did not meet PROSPERO eligibility criteria for registration. The protocol is available from the corresponding author upon reasonable request. This overview was conducted based on the guidelines for the conduct of overviews as outlined in The Cochrane Handbook. 15 Reporting followed the Preferred Reporting Items for Systematic reviews and Meta‐analyses (PRISMA) statement. 3

2.1. Eligibility criteria

Only published SRs, with or without associated MA, were included in this overview. We adopted the defining characteristics of SRs from The Cochrane Handbook. 5 According to The Cochrane Handbook, a review was considered systematic if it satisfied the following criteria: (a) clearly states the objectives and eligibility criteria for study inclusion; (b) provides reproducible methodology; (c) includes a systematic search to identify all eligible studies; (d) reports assessment of validity of findings of included studies (e.g., RoB assessment of the included studies); (e) systematically presents all the characteristics or findings of the included studies. 5 Reviews that did not meet all of the above criteria were not considered a SR for this study and were excluded. MA‐only articles were included if it was mentioned that the MA was based on an SR.

SRs and/or MA of primary studies evaluating methodological approaches used in defining review scope and study eligibility, literature search, study selection, data extraction, RoB assessment, data synthesis, and CoE assessment and reporting were included. The methodological approaches examined in these SRs and/or MA can also be related to the substeps or elements of these steps; for example, applying limits on date or type of publication are the elements of literature search. Included SRs examined or compared various aspects of a method or methods, and the associated factors, including but not limited to: precision or effectiveness; accuracy or reliability; impact on the SR and/or MA results; reproducibility of an SR steps or bias occurred; time and/or resource efficiency. SRs assessing the methodological quality of SRs (e.g., adherence to reporting guidelines), evaluating techniques for building search strategies or the use of specific database filters (e.g., use of Boolean operators or search filters for randomized controlled trials), examining various tools used for RoB or CoE assessment (e.g., ROBINS vs. Cochrane RoB tool), or evaluating statistical techniques used in meta‐analyses were excluded. 14

2.2. Search

The search for published SRs was performed on the following scientific databases initially from inception to third week of November 2020 and updated in the last week of February 2022: MEDLINE (via Ovid), Embase (via Ovid), Web of Science Core Collection, Cochrane Database of Systematic Reviews, and American Psychological Association (APA) PsycINFO. Search was restricted to English language publications. Following the objectives of this study, study design filters within databases were used to restrict the search to SRs and MA, where available. The reference lists of included SRs were also searched for potentially relevant publications.

The search terms included keywords, truncations, and subject headings for the key concepts in the review question: SRs and/or MA, methods, and evaluation. Some of the terms were adopted from the search strategy used in a previous review by Robson et al., which reviewed primary studies on methodological approaches used in study selection, data extraction, and quality appraisal steps of SR process. 14 Individual search strategies were developed for respective databases by combining the search terms using appropriate proximity and Boolean operators, along with the related subject headings in order to identify SRs and/or MA. 16 , 17 A senior librarian was consulted in the design of the search terms and strategy. Appendix A presents the detailed search strategies for all five databases.

2.3. Study selection and data extraction

Title and abstract screening of references were performed in three steps. First, one reviewer (PV) screened all the titles and excluded obviously irrelevant citations, for example, articles on topics not related to SRs, non‐SR publications (such as randomized controlled trials, observational studies, scoping reviews, etc.). Next, from the remaining citations, a random sample of 200 titles and abstracts were screened against the predefined eligibility criteria by two reviewers (PV and MM), independently, in duplicate. Discrepancies were discussed and resolved by consensus. This step ensured that the responses of the two reviewers were calibrated for consistency in the application of the eligibility criteria in the screening process. Finally, all the remaining titles and abstracts were reviewed by a single “calibrated” reviewer (PV) to identify potential full‐text records. Full‐text screening was performed by at least two authors independently (PV screened all the records, and duplicate assessment was conducted by MM, HC, or MG), with discrepancies resolved via discussions or by consulting a third reviewer.

Data related to review characteristics, results, key findings, and conclusions were extracted by at least two reviewers independently (PV performed data extraction for all the reviews and duplicate extraction was performed by AP, HC, or MG).

2.4. Quality assessment of included reviews

The quality assessment of the included SRs was performed using the AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews). The tool consists of a 16‐item checklist addressing critical and noncritical domains. 18 For the purpose of this study, the domain related to MA was reclassified from critical to noncritical, as SRs with and without MA were included. The other six critical domains were used according to the tool guidelines. 18 Two reviewers (PV and AP) independently responded to each of the 16 items in the checklist with either “yes,” “partial yes,” or “no.” Based on the interpretations of the critical and noncritical domains, the overall quality of the review was rated as high, moderate, low, or critically low. 18 Disagreements were resolved through discussion or by consulting a third reviewer.

2.5. Data synthesis

To provide an understandable summary of existing evidence syntheses, characteristics of the methods evaluated in the included SRs were examined and key findings were categorized and presented based on the corresponding step in the SR process. The categories of key elements within each step were discussed and agreed by the authors. Results of the included reviews were tabulated and summarized descriptively, along with a discussion on any overlap in the primary studies. 15 No quantitative analyses of the data were performed.

From 41,556 unique citations identified through literature search, 50 full‐text records were reviewed, and nine systematic reviews 14 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 were deemed eligible for inclusion. The flow of studies through the screening process is presented in Figure  1 . A list of excluded studies with reasons can be found in Appendix B .

Study selection flowchart

3.1. Characteristics of included reviews

Table  1 summarizes the characteristics of included SRs. The majority of the included reviews (six of nine) were published after 2010. 14 , 22 , 23 , 24 , 25 , 26 Four of the nine included SRs were Cochrane reviews. 20 , 21 , 22 , 23 The number of databases searched in the reviews ranged from 2 to 14, 2 reviews searched gray literature sources, 24 , 25 and 7 reviews included a supplementary search strategy to identify relevant literature. 14 , 19 , 20 , 21 , 22 , 23 , 26 Three of the included SRs (all Cochrane reviews) included an integrated MA. 20 , 21 , 23

Characteristics of included studies

SR = systematic review; MA = meta‐analysis; RCT = randomized controlled trial; CCT = controlled clinical trial; N/R = not reported.

The included SRs evaluated 24 unique methodological approaches (26 in total) used across five steps in the SR process; 8 SRs evaluated 6 approaches, 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 while 1 review evaluated 18 approaches. 14 Exclusion of gray or unpublished literature 21 , 26 and blinding of reviewers for RoB assessment 14 , 23 were evaluated in two reviews each. Included SRs evaluated methods used in five different steps in the SR process, including methods used in defining the scope of review ( n  = 3), literature search ( n  = 3), study selection ( n  = 2), data extraction ( n  = 1), and RoB assessment ( n  = 2) (Table  2 ).

Summary of findings from review evaluating systematic review methods

There was some overlap in the primary studies evaluated in the included SRs on the same topics: Schmucker et al. 26 and Hopewell et al. 21 ( n  = 4), Hopewell et al. 20 and Crumley et al. 19 ( n  = 30), and Robson et al. 14 and Morissette et al. 23 ( n  = 4). There were no conflicting results between any of the identified SRs on the same topic.

3.2. Methodological quality of included reviews

Overall, the quality of the included reviews was assessed as moderate at best (Table  2 ). The most common critical weakness in the reviews was failure to provide justification for excluding individual studies (four reviews). Detailed quality assessment is provided in Appendix C .

3.3. Evidence on systematic review methods

3.3.1. methods for defining review scope and eligibility.

Two SRs investigated the effect of excluding data obtained from gray or unpublished sources on the pooled effect estimates of MA. 21 , 26 Hopewell et al. 21 reviewed five studies that compared the impact of gray literature on the results of a cohort of MA of RCTs in health care interventions. Gray literature was defined as information published in “print or electronic sources not controlled by commercial or academic publishers.” Findings showed an overall greater treatment effect for published trials than trials reported in gray literature. In a more recent review, Schmucker et al. 26 addressed similar objectives, by investigating gray and unpublished data in medicine. In addition to gray literature, defined similar to the previous review by Hopewell et al., the authors also evaluated unpublished data—defined as “supplemental unpublished data related to published trials, data obtained from the Food and Drug Administration  or other regulatory websites or postmarketing analyses hidden from the public.” The review found that in majority of the MA, excluding gray literature had little or no effect on the pooled effect estimates. The evidence was limited to conclude if the data from gray and unpublished literature had an impact on the conclusions of MA. 26

Morrison et al. 24 examined five studies measuring the effect of excluding non‐English language RCTs on the summary treatment effects of SR‐based MA in various fields of conventional medicine. Although none of the included studies reported major difference in the treatment effect estimates between English only and non‐English inclusive MA, the review found inconsistent evidence regarding the methodological and reporting quality of English and non‐English trials. 24 As such, there might be a risk of introducing “language bias” when excluding non‐English language RCTs. The authors also noted that the numbers of non‐English trials vary across medical specialties, as does the impact of these trials on MA results. Based on these findings, Morrison et al. 24 conclude that literature searches must include non‐English studies when resources and time are available to minimize the risk of introducing “language bias.”

3.3.2. Methods for searching studies

Crumley et al. 19 analyzed recall (also referred to as “sensitivity” by some researchers; defined as “percentage of relevant studies identified by the search”) and precision (defined as “percentage of studies identified by the search that were relevant”) when searching a single resource to identify randomized controlled trials and controlled clinical trials, as opposed to searching multiple resources. The studies included in their review frequently compared a MEDLINE only search with the search involving a combination of other resources. The review found low median recall estimates (median values between 24% and 92%) and very low median precisions (median values between 0% and 49%) for most of the electronic databases when searched singularly. 19 A between‐database comparison, based on the type of search strategy used, showed better recall and precision for complex and Cochrane Highly Sensitive search strategies (CHSSS). In conclusion, the authors emphasize that literature searches for trials in SRs must include multiple sources. 19

In an SR comparing handsearching and electronic database searching, Hopewell et al. 20 found that handsearching retrieved more relevant RCTs (retrieval rate of 92%−100%) than searching in a single electronic database (retrieval rates of 67% for PsycINFO/PsycLIT, 55% for MEDLINE, and 49% for Embase). The retrieval rates varied depending on the quality of handsearching, type of electronic search strategy used (e.g., simple, complex or CHSSS), and type of trial reports searched (e.g., full reports, conference abstracts, etc.). The authors concluded that handsearching was particularly important in identifying full trials published in nonindexed journals and in languages other than English, as well as those published as abstracts and letters. 20

The effectiveness of checking reference lists to retrieve additional relevant studies for an SR was investigated by Horsley et al. 22 The review reported that checking reference lists yielded 2.5%–40% more studies depending on the quality and comprehensiveness of the electronic search used. The authors conclude that there is some evidence, although from poor quality studies, to support use of checking reference lists to supplement database searching. 22

3.3.3. Methods for selecting studies

Three approaches relevant to reviewer characteristics, including number, experience, and blinding of reviewers involved in the screening process were highlighted in an SR by Robson et al. 14 Based on the retrieved evidence, the authors recommended that two independent, experienced, and unblinded reviewers be involved in study selection. 14 A modified approach has also been suggested by the review authors, where one reviewer screens and the other reviewer verifies the list of excluded studies, when the resources are limited. It should be noted however this suggestion is likely based on the authors’ opinion, as there was no evidence related to this from the studies included in the review.

Robson et al. 14 also reported two methods describing the use of technology for screening studies: use of Google Translate for translating languages (for example, German language articles to English) to facilitate screening was considered a viable method, while using two computer monitors for screening did not increase the screening efficiency in SR. Title‐first screening was found to be more efficient than simultaneous screening of titles and abstracts, although the gain in time with the former method was lesser than the latter. Therefore, considering that the search results are routinely exported as titles and abstracts, Robson et al. 14 recommend screening titles and abstracts simultaneously. However, the authors note that these conclusions were based on very limited number (in most instances one study per method) of low‐quality studies. 14

3.3.4. Methods for data extraction

Robson et al. 14 examined three approaches for data extraction relevant to reviewer characteristics, including number, experience, and blinding of reviewers (similar to the study selection step). Although based on limited evidence from a small number of studies, the authors recommended use of two experienced and unblinded reviewers for data extraction. The experience of the reviewers was suggested to be especially important when extracting continuous outcomes (or quantitative) data. However, when the resources are limited, data extraction by one reviewer and a verification of the outcomes data by a second reviewer was recommended.

As for the methods involving use of technology, Robson et al. 14 identified limited evidence on the use of two monitors to improve the data extraction efficiency and computer‐assisted programs for graphical data extraction. However, use of Google Translate for data extraction in non‐English articles was not considered to be viable. 14 In the same review, Robson et al. 14 identified evidence supporting contacting authors for obtaining additional relevant data.

3.3.5. Methods for RoB assessment

Two SRs examined the impact of blinding of reviewers for RoB assessments. 14 , 23 Morissette et al. 23 investigated the mean differences between the blinded and unblinded RoB assessment scores and found inconsistent differences among the included studies providing no definitive conclusions. Similar conclusions were drawn in a more recent review by Robson et al., 14 which included four studies on reviewer blinding for RoB assessment that completely overlapped with Morissette et al. 23

Use of experienced reviewers and provision of additional guidance for RoB assessment were examined by Robson et al. 14 The review concluded that providing intensive training and guidance on assessing studies reporting insufficient data to the reviewers improves RoB assessments. 14 Obtaining additional data related to quality assessment by contacting study authors was also found to help the RoB assessments, although based on limited evidence. When assessing the qualitative or mixed method reviews, Robson et al. 14 recommends the use of a structured RoB tool as opposed to an unstructured tool. No SRs were identified on data synthesis and CoE assessment and reporting steps.

4. DISCUSSION

4.1. summary of findings.

Nine SRs examining 24 unique methods used across five steps in the SR process were identified in this overview. The collective evidence supports some current traditional and modified SR practices, while challenging other approaches. However, the quality of the included reviews was assessed to be moderate at best and in the majority of the included SRs, evidence related to the evaluated methods was obtained from very limited numbers of primary studies. As such, the interpretations from these SRs should be made cautiously.

The evidence gathered from the included SRs corroborate a few current SR approaches. 5 For example, it is important to search multiple resources for identifying relevant trials (RCTs and/or CCTs). The resources must include a combination of electronic database searching, handsearching, and reference lists of retrieved articles. 5 However, no SRs have been identified that evaluated the impact of the number of electronic databases searched. A recent study by Halladay et al. 27 found that articles on therapeutic intervention, retrieved by searching databases other than PubMed (including Embase), contributed only a small amount of information to the MA and also had a minimal impact on the MA results. The authors concluded that when the resources are limited and when large number of studies are expected to be retrieved for the SR or MA, PubMed‐only search can yield reliable results. 27

Findings from the included SRs also reiterate some methodological modifications currently employed to “expedite” the SR process. 10 , 11 For example, excluding non‐English language trials and gray/unpublished trials from MA have been shown to have minimal or no impact on the results of MA. 24 , 26 However, the efficiency of these SR methods, in terms of time and the resources used, have not been evaluated in the included SRs. 24 , 26 Of the SRs included, only two have focused on the aspect of efficiency 14 , 25 ; O'Mara‐Eves et al. 25 report some evidence to support the use of text‐mining approaches for title and abstract screening in order to increase the rate of screening. Moreover, only one included SR 14 considered primary studies that evaluated reliability (inter‐ or intra‐reviewer consistency) and accuracy (validity when compared against a “gold standard” method) of the SR methods. This can be attributed to the limited number of primary studies that evaluated these outcomes when evaluating the SR methods. 14 Lack of outcome measures related to reliability, accuracy, and efficiency precludes making definitive recommendations on the use of these methods/modifications. Future research studies must focus on these outcomes.

Some evaluated methods may be relevant to multiple steps; for example, exclusions based on publication status (gray/unpublished literature) and language of publication (non‐English language studies) can be outlined in the a priori eligibility criteria or can be incorporated as search limits in the search strategy. SRs included in this overview focused on the effect of study exclusions on pooled treatment effect estimates or MA conclusions. Excluding studies from the search results, after conducting a comprehensive search, based on different eligibility criteria may yield different results when compared to the results obtained when limiting the search itself. 28 Further studies are required to examine this aspect.

Although we acknowledge the lack of standardized quality assessment tools for methodological study designs, we adhered to the Cochrane criteria for identifying SRs in this overview. This was done to ensure consistency in the quality of the included evidence. As a result, we excluded three reviews that did not provide any form of discussion on the quality of the included studies. The methods investigated in these reviews concern supplementary search, 29 data extraction, 12 and screening. 13 However, methods reported in two of these three reviews, by Mathes et al. 12 and Waffenschmidt et al., 13 have also been examined in the SR by Robson et al., 14 which was included in this overview; in most instances (with the exception of one study included in Mathes et al. 12 and Waffenschmidt et al. 13 each), the studies examined in these excluded reviews overlapped with those in the SR by Robson et al. 14

One of the key gaps in the knowledge observed in this overview was the dearth of SRs on the methods used in the data synthesis component of SR. Narrative and quantitative syntheses are the two most commonly used approaches for synthesizing data in evidence synthesis. 5 There are some published studies on the proposed indications and implications of these two approaches. 30 , 31 These studies found that both data synthesis methods produced comparable results and have their own advantages, suggesting that the choice of the method must be based on the purpose of the review. 31 With increasing number of “expedited” SR approaches (so called “rapid reviews”) avoiding MA, 10 , 11 further research studies are warranted in this area to determine the impact of the type of data synthesis on the results of the SR.

4.2. Implications for future research

The findings of this overview highlight several areas of paucity in primary research and evidence synthesis on SR methods. First, no SRs were identified on methods used in two important components of the SR process, including data synthesis and CoE and reporting. As for the included SRs, a limited number of evaluation studies have been identified for several methods. This indicates that further research is required to corroborate many of the methods recommended in current SR guidelines. 4 , 5 , 6 , 7 Second, some SRs evaluated the impact of methods on the results of quantitative synthesis and MA conclusions. Future research studies must also focus on the interpretations of SR results. 28 , 32 Finally, most of the included SRs were conducted on specific topics related to the field of health care, limiting the generalizability of the findings to other areas. It is important that future research studies evaluating evidence syntheses broaden the objectives and include studies on different topics within the field of health care.

4.3. Strengths and limitations

To our knowledge, this is the first overview summarizing current evidence from SRs and MA on different methodological approaches used in several fundamental steps in SR conduct. The overview methodology followed well established guidelines and strict criteria defined for the inclusion of SRs.

There are several limitations related to the nature of the included reviews. Evidence for most of the methods investigated in the included reviews was derived from a limited number of primary studies. Also, the majority of the included SRs may be considered outdated as they were published (or last updated) more than 5 years ago 33 ; only three of the nine SRs have been published in the last 5 years. 14 , 25 , 26 Therefore, important and recent evidence related to these topics may not have been included. Substantial numbers of included SRs were conducted in the field of health, which may limit the generalizability of the findings. Some method evaluations in the included SRs focused on quantitative analyses components and MA conclusions only. As such, the applicability of these findings to SR more broadly is still unclear. 28 Considering the methodological nature of our overview, limiting the inclusion of SRs according to the Cochrane criteria might have resulted in missing some relevant evidence from those reviews without a quality assessment component. 12 , 13 , 29 Although the included SRs performed some form of quality appraisal of the included studies, most of them did not use a standardized RoB tool, which may impact the confidence in their conclusions. Due to the type of outcome measures used for the method evaluations in the primary studies and the included SRs, some of the identified methods have not been validated against a reference standard.

Some limitations in the overview process must be noted. While our literature search was exhaustive covering five bibliographic databases and supplementary search of reference lists, no gray sources or other evidence resources were searched. Also, the search was primarily conducted in health databases, which might have resulted in missing SRs published in other fields. Moreover, only English language SRs were included for feasibility. As the literature search retrieved large number of citations (i.e., 41,556), the title and abstract screening was performed by a single reviewer, calibrated for consistency in the screening process by another reviewer, owing to time and resource limitations. These might have potentially resulted in some errors when retrieving and selecting relevant SRs. The SR methods were grouped based on key elements of each recommended SR step, as agreed by the authors. This categorization pertains to the identified set of methods and should be considered subjective.

5. CONCLUSIONS

This overview identified limited SR‐level evidence on various methodological approaches currently employed during five of the seven fundamental steps in the SR process. Limited evidence was also identified on some methodological modifications currently used to expedite the SR process. Overall, findings highlight the dearth of SRs on SR methodologies, warranting further work to confirm several current recommendations on conventional and expedited SR processes.

CONFLICT OF INTEREST

The authors declare no conflicts of interest.

Supporting information

APPENDIX A: Detailed search strategies

ACKNOWLEDGMENTS

The first author is supported by a La Trobe University Full Fee Research Scholarship and a Graduate Research Scholarship.

Open Access Funding provided by La Trobe University.

Veginadu P, Calache H, Gussy M, Pandian A, Masood M. An overview of methodological approaches in systematic reviews . J Evid Based Med . 2022; 15 :39–54. 10.1111/jebm.12468 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]

The ABC of systematic literature review: the basic methodological guidance for beginners

• Published: 23 October 2020
• Volume 55 , pages 1319–1346, ( 2021 )

Cite this article

• Hayrol Azril Mohamed Shaffril 1 ,
• Samsul Farid Samsuddin 2 &
• Asnarulkhadi Abu Samah 1 , 3  

17k Accesses

125 Citations

2 Altmetric

Explore all metrics

There is a need for more methodological-based articles on systematic literature review (SLR) for non-health researchers to address issues related to the lack of methodological references in SLR and less suitability of existing methodological guidance. With that, this study presented a beginner's guide to basic methodological guides and key points to perform SLR, especially for those from non-health related background. For that, a total of 75 articles that passed the minimum quality were retrieved using systematic searching strategies. Seven main points of SLR were discussed, namely (1) the development and validation of the review protocol/publication standard/reporting standard/guidelines, (2) the formulation of research questions, (3) systematic searching strategies, (4) quality appraisal, (5) data extraction, (6) data synthesis, and (7) data demonstration.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

What is Qualitative in Qualitative Research

Qualitative Research: Ethical Considerations

Criteria for Good Qualitative Research: A Comprehensive Review

Athukorala, K., Głowacka, D., Jacucci, G., Oulasvirta, A., Vreeken, J.: Is exploratory search different?: A comparison of information search behavior for exploratory and lookup tasks. J. Assoc. Inf. Sci. Technol. 67 (11), 2635–2651 (2016). https://doi.org/10.1002/asi.23617

Article   Google Scholar  

Barnett-Page, E., Thomas, J.: Methods for the synthesis of qualitative research: a critical review. BMC Med. Res. Methodol. (2009). https://doi.org/10.1186/1471-2288-9-59

Bates, J., Best, P., McQuilkin, J., Taylor, B.: Will web search engines replace bibliographic databases in the systematic identification of research? J. Acad. Librariansh. 43 (1), 8–17 (2017)

Google Scholar  

Berrang-Ford, L., Pearce, T., Ford, J.D.: Systematic review approaches for climate change adaptation research. Reg. Environ. Change 15 (5), 755–769 (2015). https://doi.org/10.1007/s10113-014-0708-7

Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol. 3 (2), 77–101 (2006). https://doi.org/10.1191/1478088706qp063oa

Burgers, C., Brugman, B.C., Boeynaems, A.: Systematic literature reviews: four applications for interdisciplinary research. J. Pragmat. 145 , 102–109 (2019)

Brunton, G., Oliver, S., Oliver, K., Lorenc, T.: A Synthesis of Research Addressing Children’s, Young People’s and Parents’ Views of Walking and Cycling for Transport. EPPI-Centre, Social. Science Research Unit, Institute of Education, University of London, London (2006)

Cañón, M., Buitrago-Gómez, Q.: The research question in clinical practice: a guideline for its formulation. Rev Colomb Psiquiatr. 47 (3), 193–200 (2018). https://doi.org/10.1016/j.rcp.2016.06.004

Centre for Reviews and Dissemination.: Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care. University of York, York (2006)

Charrois, T.L.: Systematic reviews: What do you get to know to get started? Can. J. Hosp. Pharm. 68 (2), 144–148 (2015)

Cooke, A., Smith, D., Booth, A.: Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual. Health Res. 22 (10), 1435–1443 (2012)

Cooper, C., Booth, A., Campbell, J., Britten, N., Garside, R.: Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med. Res. Methodol. 18 , 85 (2018)

Creswell, J.: Qualitative Inquiry & Research Design: Choosing Among Five Approaches, 3rd edn. Sage Publications, Inc., Thousand Oaks, CA (2013)

del Amo, I.F., Erkoyuncu, J.A., Roy, R., Palmarini, R., Onoufriou, D.: A systematic review of augmented reality content-related techniques for knowledge transfer in maintenance applications. Comput. Ind. 103 , 47–71 (2018)

Delaney, A., Tamás, P.A.: Searching for evidence or approval? A commentary on database search in systematic reviews and alternative information retrieval methodologies. Res. Synth. Method 9 (1), 124–131 (2018)

Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., Sutton, A.: Synthesising qualitative and quantitative evidence: a review of possible methods. J. Health Serv. Res. Policy 10 (1), 45–53 (2005)

Doody, O., Bailey, M.E.: Setting a research question, aim and objective. Nurse Res. 23 (4), 19–23 (2016)

Durach, C.F., Kembro, J., Wieland, A.: A new paradigm for systematic literature reviews in supply chain management. J. Supply Chain Manag. 53 (4), 67–85 (2017)

Fagan, J.C.: An evidence-based review of academic web search engines, 2014–2016: implications for Librarians’ Practice and Research Agenda. Inf. Technol. Libr. 36 (2), 7–47 (2017)

Flemming, K., Booth, A., Garside, R., Tunc¸alp, O., Noyes J.: Qualitative evidence synthesis for complex interventions and guideline development: clarification of the purpose, designs and relevant methods. BMJ Global Health (2018)

Gomersall, J.S., Jadotte, Y.T., Xue, Y., Lockwood, S., Riddle, D., Preda, A.: Conducting systematic reviews of economic evaluations. Int. J. Evid.-Based Healthc. 13 (3), 170–178 (2015). https://doi.org/10.1097/XEB.0000000000000063

Green, B.N., Johnson, C.D., Adams, A.: Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J. Chiropr. Med. 5 (3), 101–117 (2006). https://doi.org/10.1016/S0899-3467(07)60142-6

Greyson, D., Rafferty, E., Slater, L., MacDonald, N., Bettinger, J.A., Dubé, È., MacDonald, S.E.: Systematic review searches must be systematic, comprehensive, and transparent: a critique of Perman et al. BMC Public Health 19 (1), 1–6 (2019). https://doi.org/10.1186/s12889-018-6275-y

Gusenbauer, M.: Google Scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases. Sciencetometrics 118 (1), 177–214 (2019)

Gusenbauer M, Haddaway NR (2020) Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res. Synth. Methods. 11(2):181–217. https://doi.org/10.1002/jrsm.1378

Haddaway, N.R., Collins, A.M., Coughlin, D., Kirk, S.: The role of google scholar in evidence reviews and its applicability to grey literature searching. PLoS ONE 10 (9), e0138237 (2015). https://doi.org/10.1371/journal.pone.0138237

Haddaway, N.R., Macura, B., Whaley, P., Pulin, A.S.: ROSES Reporting standards for Systematic Evidence Syntheses: pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps. Environ. Evid 7 , 7 (2018). https://doi.org/10.1186/s13750-018-0121-7

Halevi, G., Moed, H., Bar-Illan, J.: Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation. Revi Lit 11 (3), 823–834 (2017)

Hannes, K.: Critical appraisal of qualitative research. In: Noyes, J., Hannes, K., Harden, A., Harris, J., Lewin, S., Lockwood, C. (eds.) Supplementary Guidance for Inclusion of Qualitative Research in Cochrane Systematic Reviews of Interventions. Cochrane Collaboration Qualitative Methods Group, London (2011)

Higgins, J.P.T., Altman, D.G., Gotzsche, P.C., Juni, P., Moher, D., Oxman, A.D., Savovic, J., Schulz, K.F., Weeks, L., Sterne, J.A.C.: The cochrane collaboration's tool for assessing risk of bias in randomised trials. BMJ 343 (7829), 1–9 (2011). https://doi.org/10.1136/bmj.d5928

Higgins, J.P.T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M.J., Welch, V.A. (eds.): Cochrane Handbook for Systematic Reviews of Interventions, 2nd edn. Wiley, Chichester (UK) (2019)

Hong, Q.N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M-P., Griffiths, F., Nicolau, B., O’Cathain, A., Rousseau, M-C., Vedel, I.: Mixed Methods Appraisal Tool (MMAT), version 2018. Registration of Copyright (#1148552), Canadian Intellectual Property Office, Industry Canada (2018)

Housyar, M., Sotudeh, H.: A reflection on the applicability of Google Scholar as a tool for comprehensive retrieval in bibliometric research and systematic reviews. Int. J. Inf. Sci. Manag. 16 (2), 1–17 (2018)

Hopia, H., Latvala, E., Liimatainen, L.: Reviewing the methodology of an integrative review. Scand. J. Caring Sci. 30 (4), 662–669 (2016)

Johnson, B.T., Hennessy, E.A.: Systematic reviews and meta-analyses in the health sciences: best practice methods for research syntheses. Soc. Sci. Med. 233 , 237–251 (2019)

Kastner, M., Straus, S., Goldsmith, C.H.: Estimating the horizon of articles to decide when to stop searching in systematic reviews: an example using a systematic review of RCTs evaluating osteoporosis clinical decision support tools. AMIA Annu. Symp. Proc. Arch. 2007 , 389–393 (2007)

Kitchenham, B.A., Charters, S.M.: Guidelines for performing systematic literature reviews in software engineering. EBSE Technical Report (2007)

Kraus, S., Breier, M., Dasí-Rodríguez, S.: The art of crafting a systematic literature review in entrepreneurship research. Int. Entrep. Manag. J. 16 (3), 1023–1042 (2020). https://doi.org/10.1007/s11365-020-00635-4

Kushwah, S., Dhir, A., Sagar, M., Gupta, B.: Determinants of organic food consumption. A systematic literature review on motives and barriers. Appetite 143 , 104402 (2019)

Levy, Y., Ellis, T.J.: A systems approach to conduct an effective literature review in supports of information system research. Inf. Sci. J. 9 , 181–212 (2006)

Liberati, A., Altman, D.G., Tetzlaff, J., Mulrow, C., Gøtzsche, P.C., Ioannidis, J.P., Clarke, M., Devereaux, P.J., Kleijnen, J., Moher, D.: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med 6 (7), e1000100 (2009)

Linares-Espinós, E., Hernández, V., Domínguez-Escrig, J.L., Fernández-Pello, S., Hevia, V., Mayor, J., Padilla-Fernández, B., Ribal, M.J.: Methodology of systematic review. Actas urologicas españolas 42 (8), 499–506 (2018)

Lockwood, C., Munn, Z., Porritt, K.: Qualitative research synthesis: methodological guidance for systematic reviewers utilizing meta-aggregation. Int. J. Evid. Based Healthc. 13(3), 179–187 (2015). https://doi.org/10.1097/XEB.0000000000000062

Long, A.F., Godfrey, M.: An evaluation tool to assess the quality of qualitative research studies. Int. J. Soc. Res. Methodol. 7 (2), 181–196 (2004)

Mallet, R., Hagen-Zanker, J., Slater, R., Duvendack, M.: The benefits and challenges of using systematic reviews in international development research. J. Dev. Eff. 4 , 445–455 (2012)

Mantzoukas, S.: Facilitating research students in formulating qualitative research questions. Nurse Educ. Today 28 (3), 371–377 (2008)

Mays, N., Pope, C., Popay, J.: Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J. Health Serv. Res. Policy 10 (1), 6–20 (2005)

Mengist, W., Soromessa, T., Legese, G.: Method for conducting systematic literature review and meta-analysis for environmental science research. MethodsX 7 , 100777 (2020). https://doi.org/10.1016/j.mex.2019.100777

Methley, A.M., Campbell, S., Chew-Graham, C., McNally, R., Cheraghi-Sohi, S.: PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv. Res. (2014). https://doi.org/10.1186/s12913-014-0579-0

Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., PRISMA Group: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 21;6 (7), e1000097 (2009)

Morton SC, Murad MH, O’Connor E, Lee CS, Booth M, Vandermeer BW, Snowden JM, D’Anci KE, Fu R, Gartlehner G, Wang Z, Steele DW (2018) Quantitative synthesis—an update. methods guide for comparative effectiveness reviews. (Prepared by the Scientific Resource Center under Contract No. 290-2012-0004-C). AHRQ Publication No. 18-EHC007- EF. Rockville, MD: Agency for Healthcare Research and Quality. Posted final reports are located on the Effective Health Care Program search page. https://doi.org/10.23970/AHRQEPCMETHGUIDE3

Noble, H., Mitchell, M.: What is grounded theory? Evid. Based Nurs. 19 (2), 34–35 (2016). https://doi.org/10.1136/eb-2016-102306

Okoli, C.: A guide to conducting a standalone systematic literature review. Commun. Assoc. Inf. Syst. 37 , 879–910 (2015)

Okoli, C., Schabram, K.: A guide to conducting a systematic literature review of information systems research. Philos. Methodol. Econ. eJ. (2010). https://doi.org/10.2139/ssrn.1954824

Onwuegbuzie, A.J., Leech, N.L.: Sampling designs in qualitative research: making the sampling process more public. Qual. Rep. 12 (2), 238–254 (2007)

Pace, R., Pluye, P., Bartlett, G., Macaulay, A.C., Salsberg, J., Jagosh, J., Seller, R.: Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review. Int. J. Nurs Stud. 49 (1), 47–53 (2012). https://doi.org/10.1016/j.ijnurstu.2011.07.002

Page, M.J., McKenzie, J.E., Bossuyt, P.M., Boutron, I., Hoffmann, T., Mulrow, C.D., Shamseer, L., Moher, D.: Mapping of reporting guidance for systematic reviews and meta-analyses generated a comprehensive item bank for future reporting guidelines. J. Clin. Epidemiol. 118 , 60–68 (2020)

Paterson, B.L., Thorne, S.E., Canam, C., Jillings, C.: Meta-Study of Qualitative Health Research. A Practical Guide to Meta-Analysis and Meta-Synthesis. Sage Publications, Thousand Oaks (2001)

Palaskar, J.N.: Framing the research question using PICO strategy. J. Dent. Allied Sci. 6 (2), 55 (2017)

Patino, C.M., Ferreira, J.C.: Inclusion and exclusion criteria in research studies: definitions and why they matter. J. Bras. Pneumol. 44 (2), 84 (2018)

Peters, M.D., Godfrey, C.M., Khalil, H., McInerney, P., Parker, D., Soares, C.B.: Guidance for conducting systematic scoping reviews. Int. J. Evid.-Based Healthc. 13 (3), 141–146 (2015). https://doi.org/10.1097/XEB.0000000000000050

Petticrew, M., Roberts, H.: Systematic Reviews in the Social Sciences: A Practical Guide. Blackwell Publishing Ltd, Oxford (2006)

Prieto and Rumbo-Prieto: The systematic review: plurality of approaches and methodologies. Enferm. Clín. (English Edition) 28 (6), 387–393 (2018)

Reim, W., Parida, V., Örtqvist, D.: Product-Service Systems (PSS) business models and tactics – a systematic literature review. J. Clean. Prod. 97 , 61–75 (2015). https://doi.org/10.1016/j.jclepro.2014.07.003

Robinson, P., Lowe, J.: Literature reviews vs systematic reviews. Aust. N. Z. J. Public Health 39 (2), 103 (2015)

Rousseau, D. M., Manning, J., Denyer, D.: Evidence in management and organizational science: assembling the field’s full weight of scientific knowledge through syntheses. In: AIM Research Working Paper Series: Advanced Institute of Management Research (2008)

Sandelowski, M., Barroso, J., Voils, C.I.: Using qualitative metasummary to synthesize qualitative and quantitative descriptive findings. Res. Nuurs. Health 30 (1), 99–111 (2007). https://doi.org/10.1002/nur.20176

Sandelowski, M., Voils, C.I., Barroso, J.: Defining and designing mixed research synthesis studies. Res. Schools: Nat. Ref. J. Spons. Mid-South Educ. Res. Assoc. Univ. Alabama 13 (1), 29 (2006)

Schardt, C., Adams, M.B., Owens, T., Keitz, S., Fontelo, P.: Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Med. Inf. Decis. Mak. 7 , 16 (2007). https://doi.org/10.1186/1472-6947-7-16

Seehra, J., Pandis, N., Koletsi, D.: Use of quality assessment tools in systematic reviews was varied and inconsistent. J. Clin. Epidemiol. 69 , 179–184 (2016)

Shaffril, H.A.M., Krauss, S.E., Samsuddin, S.F.: A systematic review on Asian’s farmers’ adaptation practices towards climate change. Sci. Total Environ. 644 , 683–695 (2018)

Shaffril, H.A.M., Abu Samah, A., Samsuddin, S.F., Ali, Z.: Mirror-mirror on the wall, what climate change adaptation strategies are practiced by the Asian’s fishermen of all? J. Clean. Prod. 232 , 104–117 (2019)

Shorten, A., Shorten, B.: What is meta-analysis. Evid. Based Nurs. 16 (1), 3–4 (2013)

Siering, U., Eikermann, M., Hausner, E., Hoffmann-Eßer, W., Neugebauer, E.A.: Appraisal tools for clinical practice guidelines: a systematic review. PLoS ONE 8 (12), e82915 (2013)

Soares, C.B., Hoga, L., Sangaleti, C., Yonekura, T., Peduzzi, M., Silva, D.: Integrative review in nursing research and EBP: a type of systematic review? Int. J.Evid.-Based Healthcare 11 (3), 246–247 (2013)

Thomas, J., Noel-Storrb, A., Marshall, I., Wallace, B., McDonald, S., Mavergames, C., Glasziou, P., Shemilta, I., Synnote, A., Turnere, T., Elliott, J.: Living systematic reviews: 2. Combining human and machine effort. J. Clin. Epidemiol. 91 , 31–37 (2017)

Thomas, J., Kneale, D., McKenzie, J.E., Brennan, S.E., Bhaumik, S.: Chapter 2: Determining the scope of the review and the questions it will address. In: Higgins, J.P.T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M.J., Welch, V.A. (eds.) Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (2020). Cochrane (2020). https://www.training.cochrane.org/handbook

Wanden-Berghe, C., Sanz-Valero, J.: Systematic reviews in nutrition: standardized methodology. Br. J. Nutr. 107 , S3–S7 (2012)

Whittemore, R., Knafl, K.: The integrative review: updated methodology. J. Adv. Nurs. 52 (5), 546–553 (2005)

Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., Pawson, R.: RAMESES publication standards: realist syntheses. BMC Med. 11 , 21 (2013). https://doi.org/10.1186/1741-7015-11-21

Xiao, Y., Watson, M.: Guidance on conducting a systematic literature review. J. Plan. Educ. 39 (1), 93–112 (2019)

Younger, P.: Using Google Scholar to conduct a literature search. Nurs. Stand. 24 (45), 40–46 (2010)

Download references

Author information

Authors and affiliations.

Institute for Social Science Studies, Universiti Putra Malaysia, Putra Infoport, 43400, Serdang, Selangor, Malaysia

Hayrol Azril Mohamed Shaffril & Asnarulkhadi Abu Samah

Faculty of Computer Science and Information Technology, University of Malaya, 50603, Kuala Lumpur, Malaysia

Samsul Farid Samsuddin

Faculty of Human Ecology, Universiti Putra Malaysia, 43400, Serdang, Selangor, Malaysia

Asnarulkhadi Abu Samah

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Hayrol Azril Mohamed Shaffril .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Mohamed Shaffril, H.A., Samsuddin, S.F. & Abu Samah, A. The ABC of systematic literature review: the basic methodological guidance for beginners. Qual Quant 55 , 1319–1346 (2021). https://doi.org/10.1007/s11135-020-01059-6

Download citation

Accepted : 12 October 2020

Published : 23 October 2020

Issue Date : August 2021

DOI : https://doi.org/10.1007/s11135-020-01059-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Systematic literature review
• Basic methodology
• Non-health related studies
• Find a journal
• Publish with us
• Track your research

Covidence website will be inaccessible as we upgrading our platform on Monday 23rd August at 10am AEST, / 2am CEST/1am BST (Sunday, 15th August 8pm EDT/5pm PDT) 

How to write the methods section of a systematic review

Home | Blog | How To | How to write the methods section of a systematic review

Covidence breaks down how to write a methods section

The methods section of your systematic review describes what you did, how you did it, and why. Readers need this information to interpret the results and conclusions of the review. Often, a lot of information needs to be distilled into just a few paragraphs. This can be a challenging task, but good preparation and the right tools will help you to set off in the right direction 🗺️🧭.

Systematic reviews are so-called because they are conducted in a way that is rigorous and replicable. So it’s important that these methods are reported in a way that is thorough, clear, and easy to navigate for the reader – whether that’s a patient, a healthcare worker, or a researcher. 

Like most things in a systematic review, the methods should be planned upfront and ideally described in detail in a project plan or protocol. Reviews of healthcare interventions follow the PRISMA guidelines for the minimum set of items to report in the methods section. But what else should be included? It’s a good idea to consider what readers will want to know about the review methods and whether the journal you’re planning to submit the work to has expectations on the reporting of methods. Finding out in advance will help you to plan what to include.

Describe what happened

While the research plan sets out what you intend to do, the methods section is a write-up of what actually happened. It’s not a simple case of rewriting the plan in the past tense – you will also need to discuss and justify deviations from the plan and describe the handling of issues that were unforeseen at the time the plan was written. For this reason, it is useful to make detailed notes before, during, and after the review is completed. Relying on memory alone risks losing valuable information and trawling through emails when the deadline is looming can be frustrating and time consuming! 

Keep it brief

The methods section should be succinct but include all the noteworthy information. This can be a difficult balance to achieve. A useful strategy is to aim for a brief description that signposts the reader to a separate section or sections of supporting information. This could include datasets, a flowchart to show what happened to the excluded studies, a collection of search strategies, and tables containing detailed information about the studies.This separation keeps the review short and simple while enabling the reader to drill down to the detail as needed. And if the methods follow a well-known or standard process, it might suffice to say so and give a reference, rather than describe the process at length. 

Follow a structure

A clear structure provides focus. Use of descriptive headings keeps the writing on track and helps the reader get to key information quickly. What should the structure of the methods section look like? As always, a lot depends on the type of review but it will certainly contain information relating to the following areas:

• Selection criteria ⭕
• Data collection and analysis 👩‍💻
• Study quality and risk of bias ⚖️

Let’s look at each of these in turn.

1. Selection criteria ⭕

The criteria for including and excluding studies are listed here. This includes detail about the types of studies, the types of participants, the types of interventions and the types of outcomes and how they were measured. 

2. Search 🕵🏾‍♀️

Comprehensive reporting of the search is important because this means it can be evaluated and replicated. The search strategies are included in the review, along with details of the databases searched. It’s also important to list any restrictions on the search (for example, language), describe how resources other than electronic databases were searched (for example,  non-indexed journals), and give the date that the searches were run. The PRISMA-S extension provides guidance on reporting literature searches. 

Systematic reviewer pro-tip:

 Copy and paste the search strategy to avoid introducing typos

3. Data collection and analysis 👩‍💻

This section describes:

• how studies were selected for inclusion in the review
• how study data were extracted from the study reports
• how study data were combined for analysis and synthesis

To describe how studies were selected for inclusion , review teams outline the screening process. Covidence uses reviewers’ decision data to automatically populate a PRISMA flow diagram for this purpose. Covidence can also calculate Cohen’s kappa to enable review teams to report the level of agreement among individual reviewers during screening.

To describe how study data were extracted from the study reports , reviewers outline the form that was used, any pilot-testing that was done, and the items that were extracted from the included studies. An important piece of information to include here is the process used to resolve conflict among the reviewers. Covidence’s data extraction tool saves reviewers’ comments and notes in the system as they work. This keeps the information in one place for easy retrieval ⚡.

To describe how study data were combined for analysis and synthesis, reviewers outline the type of synthesis (narrative or quantitative, for example), the methods for grouping data, the challenges that came up, and how these were dealt with. If the review includes a meta-analysis, it will detail how this was performed and how the treatment effects were measured.

4. Study quality and risk of bias ⚖️

Because the results of systematic reviews can be affected by many types of bias, reviewers make every effort to minimise it and to show the reader that the methods they used were appropriate. This section describes the methods used to assess study quality and an assessment of the risk of bias across a range of domains. 

Steps to assess the risk of bias in studies include looking at how study participants were assigned to treatment groups and whether patients and/or study assessors were blinded to the treatment given. Reviewers also report their assessment of the risk of bias due to missing outcome data, whether that is due to participant drop-out or non-reporting of the outcomes by the study authors.

Covidence’s default template for assessing study quality is Cochrane’s risk of bias tool but it is also possible to start from scratch and build a tool with a set of custom domains if you prefer.

Careful planning, clear writing, and a structured approach are key to a good methods section. A methodologist will be able to refer review teams to examples of good methods reporting in the literature. Covidence helps reviewers to screen references, extract data and complete risk of bias tables quickly and efficiently. Sign up for a free trial today!

Laura Mellor. Portsmouth, UK

Perhaps you'd also like....

Top 5 Tips for High-Quality Systematic Review Data Extraction

Data extraction can be a complex step in the systematic review process. Here are 5 top tips from our experts to help prepare and achieve high quality data extraction.

How to get through study quality assessment Systematic Review

Find out 5 tops tips to conducting quality assessment and why it’s an important step in the systematic review process.

How to extract study data for your systematic review

Learn the basic process and some tips to build data extraction forms for your systematic review with Covidence.

Better systematic review management

Head office, working for an institution or organisation.

Find out why over 350 of the world’s leading institutions are seeing a surge in publications since using Covidence!

Request a consultation with one of our team members and start empowering your researchers:

By using our site you consent to our use of cookies to measure and improve our site’s performance. Please see our Privacy Policy for more information. 

• Open access
• Published: 19 April 2024

Person-centered care assessment tool with a focus on quality healthcare: a systematic review of psychometric properties

• Lluna Maria Bru-Luna 1 ,
• Manuel Martí-Vilar 2 ,
• César Merino-Soto 3 ,
• José Livia-Segovia 4 ,
• Juan Garduño-Espinosa 5 &
• Filiberto Toledano-Toledano 5 , 6 , 7  

BMC Psychology volume  12 , Article number:  217 ( 2024 ) Cite this article

268 Accesses

Metrics details

The person-centered care (PCC) approach plays a fundamental role in ensuring quality healthcare. The Person-Centered Care Assessment Tool (P-CAT) is one of the shortest and simplest tools currently available for measuring PCC. The objective of this study was to conduct a systematic review of the evidence in validation studies of the P-CAT, taking the “Standards” as a frame of reference.

First, a systematic literature review was conducted following the PRISMA method. Second, a systematic descriptive literature review of validity tests was conducted following the “Standards” framework. The search strategy and information sources were obtained from the Cochrane, Web of Science (WoS), Scopus and PubMed databases. With regard to the eligibility criteria and selection process, a protocol was registered in PROSPERO (CRD42022335866), and articles had to meet criteria for inclusion in the systematic review.

A total of seven articles were included. Empirical evidence indicates that these validations offer a high number of sources related to test content, internal structure for dimensionality and internal consistency. A moderate number of sources pertain to internal structure in terms of test-retest reliability and the relationship with other variables. There is little evidence of response processes, internal structure in measurement invariance terms, and test consequences.

The various validations of the P-CAT are not framed in a structured, valid, theory-based procedural framework like the “Standards” are. This can affect clinical practice because people’s health may depend on it. The findings of this study show that validation studies continue to focus on the types of validity traditionally studied and overlook interpretation of the scores in terms of their intended use.

Peer Review reports

Person-centered care (PCC)

Quality care for people with chronic diseases, functional limitations, or both has become one of the main objectives of medical and care services. The person-centered care (PCC) approach is an essential element not only in achieving this goal but also in providing high-quality health maintenance and medical care [ 1 , 2 , 3 ]. In addition to guaranteeing human rights, PCC provides numerous benefits to both the recipient and the provider [ 4 , 5 ]. Additionally, PCC includes a set of necessary competencies for healthcare professionals to address ongoing challenges in this area [ 6 ]. PCC includes the following elements [ 7 ]: an individualized, goal-oriented care plan based on individuals’ preferences; an ongoing review of the plan and the individual’s goals; support from an interprofessional team; active coordination among all medical and care providers and support services; ongoing information exchange, education and training for providers; and quality improvement through feedback from the individual and caregivers.

There is currently a growing body of literature on the application of PCC. A good example of this is McCormack’s widely known mid-range theory [ 8 ], an internationally recognized theoretical framework for PCC and how it is operationalized in practice. This framework forms a guide for care practitioners and researchers in hospital settings. This framework is elaborated in PCC and conceived of as “an approach to practice that is established through the formation and fostering of therapeutic relationships between all care providers, service users, and others significant to them, underpinned by values of respect for persons, [the] individual right to self-determination, mutual respect, and understanding” [ 9 ].

Thus, as established by PCC, it is important to emphasize that reference to the person who is the focus of care refers not only to the recipient but also to everyone involved in a care interaction [ 10 , 11 ]. PCC ensures that professionals are trained in relevant skills and methodology since, as discussed above, carers are among the agents who have the greatest impact on the quality of life of the person in need of care [ 12 , 13 , 14 ]. Furthermore, due to the high burden of caregiving, it is essential to account for caregivers’ well-being. In this regard, studies on professional caregivers are beginning to suggest that the provision of PCC can produce multiple benefits for both the care recipient and the caregiver [ 15 ].

Despite a considerable body of literature and the frequent inclusion of the term in health policy and research [ 16 ], PCC involves several complications. There is no standard consensus on the definition of this concept [ 17 ], which includes problematic areas such as efficacy assessment [ 18 , 19 ]. In addition, the difficulty of measuring the subjectivity involved in identifying the dimensions of the CPC and the infrequent use of standardized measures are acute issues [ 20 ]. These limitations and purposes motivated the creation of the Person-Centered Care Assessment Tool (P-CAT; [ 21 ]), which emerged from the need for a brief, economical, easily applied, versatile and comprehensive assessment instrument to provide valid and reliable measures of PCC for research purposes [ 21 ].

Person-centered care assessment tool (P-CAT)

There are several instruments that can measure PCC from different perspectives (i.e., the caregiver or the care recipient) and in different contexts (e.g., hospitals and nursing homes). However, from a practical point of view, the P-CAT is one of the shortest and simplest tools and contains all the essential elements of PCC described in the literature. It was developed in Australia to measure the approach of long-term residential settings to older people with dementia, although it is increasingly used in other healthcare settings, such as oncology units [ 22 ] and psychiatric hospitals [ 23 ].

Due to the brevity and simplicity of its application, the versatility of its use in different medical and care contexts, and its potential emic characteristics (i.e., constructs that can be cross-culturally applicable with reasonable and similar structure and interpretation; [ 24 ]), the P-CAT is one of the most widely used tests by professionals to measure PCC [ 25 , 26 ]. It has expanded to several countries with cultural and linguistic differences. Since its creation, it has been adapted in countries separated by wide cultural and linguistic differences, such as Norway [ 27 ], Sweden [ 28 ], China [ 29 ], South Korea [ 30 ], Spain [ 25 ], and Italy [ 31 ].

The P-CAT comprises 13 items rated on a 5-point ordinal scale (from “strongly disagree” to “strongly agree”), with high scores indicating a high degree of person-centeredness. The scale consists of three dimensions: person-centered care (7 items), organizational support (4 items) and environmental accessibility (2 items). In the original study ( n  = 220; [ 21 ]), the internal consistency of the instrument yielded satisfactory values for the total scale ( α  = 0.84) and good test-retest reliability ( r  =.66) at one-week intervals. A reliability generalization study conducted in 2021 [ 32 ] that estimated the internal consistency of the P-CAT and analyzed possible factors that could affect the it revealed that the mean α value for the 25 meta-analysis samples (some of which were part of the validations included in this study) was 0.81, and the only variable that had a statistically significant relationship with the reliability coefficient was the mean age of the sample. With respect to internal structure validity, three factors (56% of the total variance) were obtained, and content validity was assessed by experts, literature reviews and stakeholders [ 33 ].

Although not explicitly stated, the apparent commonality between validation studies of different versions of the P-CAT may be influenced by an influential decades-old validity framework that differentiates three categories: content validity, construct validity, and criterion validity [ 34 , 35 ]. However, a reformulation of the validity of the P-CAT within a modern framework, which would provide a different definition of validity, has not been performed.

Scale validity

Traditionally, validation is a process focused on the psychometric properties of a measurement instrument [ 36 ]. In the early 20th century, with the frequent use of standardized measurement tests in education and psychology, two definitions emerged: the first defined validity as the degree to which a test measures what it intends to measure, while the second described the validity of an instrument in terms of the correlation it presents with a variable [ 35 ].

However, in the past century, validity theory has evolved, leading to the understanding that validity should be based on specific interpretations for an intended purpose. It should not be limited to empirically obtained psychometric properties but should also be supported by the theory underlying the construct measured. Thus, to speak of classical or modern validity theory suggests an evolution in the classical or modern understanding of the concept of validity. Therefore, a classical approach (called classical test theory, CTT) is specifically differentiated from a modern approach. In general, recent concepts associated with a modern view of validity are based on (a) a unitary conception of validity and (b) validity judgments based on inferences and interpretations of the scores of a measure [ 37 , 38 ]. This conceptual advance in the concept of validity led to the creation of a guiding framework to for obtaining evidence to support the use and interpretation of the scores obtained by a measure [ 39 ].

This purpose is addressed by the Standards for Educational and Psychological Testing (“Standards”), a guide created by the American Educational Research Association (AERA), the American Psychological Association (APA) and the National Council on Measurement in Education (NCME) in 2014 with the aim of providing guidelines to assess the validity of the interpretations of scores of an instrument based on their intended use. Two conceptual aspects stand out in this modern view of validity: first, validity is a unitary concept centered on the construct; second, validity is defined as “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests” [ 37 ]. Thus, the “Standards” propose several sources that serve as a reference for assessing different aspects of validity. The five sources of valid evidence are as follows [ 37 ]: test content, response processes, internal structure, relations to other variables and consequences of testing. According to AERA et al. [ 37 ], test content validity refers to the relationship of the administration process, subject matter, wording and format of test items to the construct they are intended to measure. It is measured predominantly with qualitative methods but without excluding quantitative approaches. The validity of the responses is based on analysis of the cognitive processes and interpretation of the items by respondents and is measured with qualitative methods. Internal structure validity is based on the interrelationship between the items and the construct and is measured by quantitative methods. Validity in terms of the relationship with other variables is based on comparison between the variable that the instrument intends to measure and other theoretically relevant external variables and is measured by quantitative methods. Finally, validity based on the results of the test analyses consequences, both intended and unintended, that may be due to a source of invalidity. It is measured mainly by qualitative methods.

Thus, although validity plays a fundamental role in providing a strong scientific basis for interpretations of test scores, validation studies in the health field have traditionally focused on content validity, criterion validity and construct validity and have overlooked the interpretation and use of scores [ 34 ].

“Standards” are considered a suitable validity theory-based procedural framework for reviewing the validity of questionnaires due to its ability to analyze sources of validity from both qualitative and quantitative approaches and its evidence-based method [ 35 ]. Nevertheless, due to a lack of knowledge or the lack of a systematic description protocol, very few instruments to date have been reviewed within the framework of the “Standards” [ 39 ].

Current study

Although the P-CAT is one of the most widely used instruments by professionals and has seven validations [ 25 , 27 , 28 , 29 , 30 , 31 , 40 ], no analysis has been conducted of its validity within the framework of the “Standards”. That is, empirical evidence of the validity of the P-CAT has not been obtained in a way that helps to develop a judgment based on a synthesis of the available information.

A review of this type is critical given that some methodological issues seem to have not been resolved in the P-CAT. For example, although the multidimensionality of the P-CAT was identified in the study that introduced it, Bru-Luna et al. [ 32 ] recently stated that in adaptations of the P-CAT [ 25 , 27 , 28 , 29 , 30 , 40 ], the total score is used for interpretation and multidimensionality is disregarded. Thus, the multidimensionality of the original study was apparently not replicated. Bru-Luna et al. [ 32 ] also indicated that the internal structure validity of the P-CAT is usually underreported due to a lack of sufficiently rigorous approaches to establish with certainty how its scores are calculated.

The validity of the P-CAT, specifically its internal structure, appears to be unresolved. Nevertheless, substantive research and professional practice point to this measure as relevant to assessing PCC. This perception is contestable and judgment-based and may not be sufficient to assess the validity of the P-CAT from a cumulative and synthetic angle based on preceding validation studies. An adequate assessment of validity requires a model to conceptualize validity followed by a review of previous studies of the validity of the P-CAT using this model.

Therefore, the main purpose of this study was to conduct a systematic review of the evidence provided by P-CAT validation studies while taking the “Standards” as a framework.

The present study comprises two distinct but interconnected procedures. First, a systematic literature review was conducted following the PRISMA method ( [ 41 ]; Additional file 1; Additional file 2) with the aim of collecting all validations of the P-CAT that have been developed. Second, a systematic description of the validity evidence for each of the P-CAT validations found in the systematic review was developed following the “Standards” framework [ 37 ]. The work of Hawkins et al. [ 39 ], the first study to review validity sources according to the guidelines proposed by the “Standards”, was also used as a reference. Both provided conceptual and pragmatic guidance for organizing and classifying validity evidence for the P-CAT.

The procedure conducted in the systematic review is described below, followed by the procedure for examining the validity studies.

Systematic review

Search strategy and information sources.

Initially, the Cochrane database was searched with the aim of identifying systematic reviews of the P-CAT. When no such reviews were found, subsequent preliminary searches were performed in the Web of Science (WoS), Scopus and PubMed databases. These databases play a fundamental role in recent scientific literature since they are the main sources of published articles that undergo high-quality content and editorial review processes [ 42 ]. The search formula was as follows. The original P-CAT article [ 21 ] was located, after which all articles that cited it through 2021 were identified and analyzed. This approach ensured the inclusion of all validations. No articles were excluded on the basis of language to avoid language bias [ 43 ]. Moreover, to reduce the effects of publication bias, a complementary search in Google Scholar was also performed to allow the inclusion of “gray” literature [ 44 ]. Finally, a manual search was performed through a review of the references of the included articles to identify other articles that met the search criteria but were not present in any of the aforementioned databases.

This process was conducted by one of the authors and corroborated by another using the Covidence tool [ 45 ]. A third author was consulted in case of doubt.

Eligibility criteria and selection process

The protocol was registered in PROSPERO, and the search was conducted according to these criteria. The identification code is CRD42022335866.

The articles had to meet the following criteria for inclusion in the systematic review: (a) a methodological approach to P-CAT validations, (b) an experimental or quasiexperimental studies, (c) studies with any type of sample, and (d) studies in any language. We discarded studies that met at least one of the following exclusion criteria: (a) systematic reviews or bibliometric reviews of the instrument or meta-analyses or (b) studies published after 2021.

Data collection process

After the articles were selected, the most relevant information was extracted from each article. Fundamental data were recorded in an Excel spreadsheet for each of the sections: introduction, methodology, results and discussion. Information was also recorded about the limitations mentioned in each article as well as the practical implications and suggestions for future research.

Given the aim of the study, information was collected about the sources of validity of each study, including test content (judges’ evaluation, literature review and translation), response processes, internal structure (factor analysis, design, estimator, factor extraction method, factors and items, interfactor R, internal replication, effect of the method, and factor loadings), and relationships with other variables (convergent, divergent, concurrent and predictive validity) and consequences of measurement.

Description of the validity study

To assess the validity of the studies, an Excel table was used. Information was recorded for the seven articles included in the systematic review. The data were extracted directly from the texts of the articles and included information about the authors, the year of publication, the country where each P-CAT validation was produced and each of the five standards proposed in the “Standards” [ 37 ].

The validity source related to internal structure was divided into three sections to record information about dimensionality (e.g., factor analysis, design, estimator, factor extraction method, factors and items, interfactor R, internal replication, effect of the method, and factor loadings), reliability expression (i.e., internal consistency and test-retest) and the study of factorial invariance according to the groups into which it was divided (e.g., sex, age, profession) and the level of study (i.e., metric, intercepts). This approach allowed much more information to be obtained than relying solely on source validity based on internal structure. This division was performed by the same researcher who performed the previous processes.

Study selection and study characteristics

The systematic review process was developed according to the PRISMA methodology [ 41 ].

The WoS, Scopus, PubMed and Google Scholar databases were searched on February 12, 2022 and yielded a total of 485 articles. Of these, 111 were found in WoS, 114 in Scopus, 43 in PubMed and 217 in Google Scholar. In the first phase, the title and abstracts of all the articles were read. In this first screening, 457 articles were eliminated because they did not include studies with a methodological approach to P-CAT validation and one article was excluded because it was the original P-CAT article. This resulted in a total of 27 articles, 19 of which were duplicated in different databases and, in the case of Google Scholar, within the same database. This process yielded a total of eight articles that were evaluated for eligibility by a complete reading of the text. In this step, one of the articles was excluded due to a lack of access to the full text of the study [ 31 ] (although the original manuscript was found, it was impossible to access the complete content; in addition, the authors of the manuscript were contacted, but no reply was received). Finally, a manual search was performed by reviewing the references of the seven studies, but none were considered suitable for inclusion. Thus, the review was conducted with a total of seven articles.

Of the seven studies, six were original validations in other languages. These included Norwegian [ 27 ], Swedish [ 28 ], Chinese (which has two validations [ 29 , 40 ]), Spanish [ 25 ], and Korean [ 30 ]. The study by Selan et al. [ 46 ] included a modification of the Swedish version of the P-CAT and explored the psychometric properties of both versions (i.e., the original Swedish version and the modified version).

The item selection and screening process are illustrated in detail in Fig.  1 .

PRISMA 2020 flow diagram for new systematic reviews including database searches

Validity analysis

To provide a clear overview of the validity analyses, Table  1 descriptively shows the percentages of items that provide information about the five standards proposed by the “Standards” guide [ 37 ].

The table shows a high number of validity sources related to test content and internal structure in relation to dimensionality and internal consistency, followed by a moderate number of sources for test-retest and relationship with other variables. A rate of 0% is observed for validity sources related to response processes, invariance and test consequences. Below, different sections related to each of the standards are shown, and the information is presented in more detail.

Evidence based on test content

The first standard, which focused on test content, was met for all items (100%). Translation, which refers to the equivalence of content between the original language and the target language, was met in the six articles that conducted validation in another language and/or culture. These studies reported that the validations were translated by bilingual experts and/or experts in the area of care. In addition, three studies [ 25 , 29 , 40 ] reported that the translation process followed International Test Commission guidelines, such as those of Beaton et al. [ 47 ], Guillemin [ 48 ], Hambleton et al. [ 49 ], and Muñiz et al. [ 50 ]. Evaluation by judges, who referred to the relevance, clarity and importance of the content, was divided into two categories: expert evaluation (a panel of expert judges for each of the areas to consider in the evaluation instrument) and experiential evaluation (potential participants testing the test). The first type of evaluation occurred in three of the articles [ 28 , 29 , 46 ], while the other occurred in two [ 25 , 40 ]. Only one of the items [ 29 ] reported that the scale contained items that reflected the dimension described in the literature. The validity evidence related to the test content presented in each article can be found in Table  2 .

Evidence based on response processes

The second standard, related to the validity of the response process, was obtained according to the “Standards” from the analysis of individual responses: “questioning test takers about their performance strategies or response to particular items (…), maintaining records that monitor the development of a response to a writing task (…), documentation of other aspects of performance, like eye movement or response times…” [ 37 ] (p. 15). According to the analysis of the validity of the response processes, none of the articles complied with this evidence.

Evidence based on internal structure

The third standard, validity related to internal structure, was divided into three sections. First, the dimensionality of each study was examined in terms of factor analysis, design, estimator, factor extraction method, factors and items, interfactor R, internal replication, effect of the method, and factor loadings. Le et al. [ 40 ] conducted an exploratory-confirmatory design while Sjögren et al. [ 28 ] conducted a confirmatory-exploratory design to assess construct validity using confirmatory factor analysis (CFA) and investigated it further using exploratory factor analysis (EFA). The remaining articles employed only a single form of factor analysis: three employed EFA, and two employed CFA. Regarding the next point, only three of the articles reported the factor extraction method used, including Kaiser’s eigenvalue, criterion, scree plot test, parallel analysis and Velicer’s MAP test. Instrument validations yielded a total of two factors in five of the seven articles, while one yielded a single dimension [ 25 ] and the other yielded three dimensions [ 29 ], as in the original instrument. The interfactor R was reported only in the study by Zhong and Lou [ 29 ], whereas in the study by Martínez et al. [ 25 ], it could be easily obtained since it consisted of only one dimension. Internal replication was also calculated in the Spanish validation by randomly splitting the sample into two to test the correlations between factors. The effectiveness of the method was not reported in any of the articles. This information is presented in Table  3 in addition to a summary of the factor loadings.

The second section examined reliability. All the studies presented measures of internal consistency conducted in their entirety with Cronbach’s α coefficient for both the total scale and the subscales. The ω coefficient of McDonald was not used in any case. Four of the seven articles performed a test-retest test. Martínez et al. [ 25 ] conducted a test-retest after a period of seven days, while Le et al. [ 40 ] and Rokstad et al. [ 27 ] performed it between one and two weeks later and Sjögren et al. [ 28 ] allowed approximately two weeks to pass after the initial test.

The third section analyzes the calculation of invariance, which was not reported in any of the studies.

Evidence based on relationships with other variables

In the fourth standard, based on validity according to the relationship with other variables, the articles that reported it used only convergent validity (i.e., it was hypothesized that the variables related to the construct measured by the test—in this case, person-centeredness—were positively or negatively related to another construct). Discriminant validity hypothesizes that the variables related to the PCC construct are not correlated in any way with any other variable studied. No article (0%) measured discriminant evidence, while four (57%) measured convergent evidence [ 25 , 29 , 30 , 46 ]. Convergent validity was obtained through comparisons with instruments such as the Person-Centered Climate Questionnaire–Staff Version (PCQ-S), the Staff-Based Measures of Individualized Care for Institutionalized Persons with Dementia (IC), the Caregiver Psychological Elder Abuse Behavior Scale (CPEAB), the Organizational Climate (CLIOR) and the Maslach Burnout Inventory (MBI). In the case of Selan et al. [ 46 ], convergent validity was assessed on two items considered by the authors as “crude measures of person-centered care (i.e., external constructs) giving an indication of the instruments’ ability to measure PCC” (p. 4). Concurrent validity, which measures the degree to which the results of one test are or are not similar to those of another test conducted at more or less the same time with the same participants, and predictive validity, which allows predictions to be established regarding behavior based on comparison between the values of the instrument and the criterion, were not reported in any of the studies.

Evidence based on the consequences of testing

The fifth and final standard was related to the consequences of the test. It analyzed the consequences, both intended and unintended, of applying the test to a given sample. None of the articles presented explicit or implicit evidence of this.

The last two sources of validity can be seen in Table  4 .

Table  5 shows the results of the set of validity tests for each study according to the described standards.

The main purpose of this article is to analyze the evidence of validity in different validation studies of the P-CAT. To gather all existing validations, a systematic review of all literature citing this instrument was conducted.

The publication of validation studies of the P-CAT has been constant over the years. Since the publication of the original instrument in 2010, seven validations have been published in other languages (taking into account the Italian version by Brugnolli et al. [ 31 ], which could not be included in this study) as well as a modification of one of these versions. The very unequal distribution of validations between languages and countries is striking. A recent systematic review [ 51 ] revealed that in Europe, the countries where the PCC approach is most widely used are the United Kingdom, Sweden, the Netherlands, Northern Ireland, and Norway. It has also been shown that the neighboring countries seem to exert an influence on each other due to proximity [ 52 ] such that they tend to organize healthcare in a similar way, as is the case for Scandinavian countries. This favors the expansion of PCC and explains the numerous validations we found in this geographical area.

Although this approach is conceived as an essential element of healthcare for most governments [ 53 ], PCC varies according to the different definitions and interpretations attributed to it, which can cause confusion in its application (e.g., between Norway and the United Kingdom [ 54 ]). Moreover, facilitators of or barriers to implementation depend on the context and level of development of each country, and financial support remains one of the main factors in this regard [ 53 ]. This fact explains why PCC is not globally widespread among all territories. In countries where access to healthcare for all remains out of reach for economic reasons, the application of this approach takes a back seat, as does the validation of its assessment tools. In contrast, in a large part of Europe or in countries such as China or South Korea that have experienced decades of rapid economic development, patients are willing to be involved in their medical treatment and enjoy more satisfying and efficient medical experiences and environments [ 55 ], which facilitates the expansion of validations of instruments such as the P-CAT.

Regarding validity testing, the guidelines proposed by the “Standards” [ 37 ] were followed. According to the analysis of the different validations of the P-CAT instrument, none of the studies used a structured validity theory-based procedural framework for conducting validation. The most frequently reported validity tests were on the content of the test and two of the sections into which the internal structure was divided (i.e., dimensionality and internal consistency).

In the present article, the most cited source of validity in the studies was the content of the test because most of the articles were validations of the P-CAT in other languages, and the authors reported that the translation procedure was conducted by experts in all cases. In addition, several of the studies employed International Test Commission guidelines, such as those by Beaton et al. [ 47 ], Guillemin [ 48 ], Hambleton et al. [ 49 ], and Muñiz et al. [ 50 ]. Several studies also assessed the relevance, clarity and importance of the content.

The third source of validity, internal structure, was the next most often reported, although it appeared unevenly among the three sections into which this evidence was divided. Dimensionality and internal consistency were reported in all studies, followed by test-retest consistency. In relation to the first section, factor analysis, a total of five EFAs and four CFAs were presented in the validations. Traditionally, EFA has been used in research to assess dimensionality and identify key psychological constructs, although this approach involves a number of inconveniences, such as difficulty testing measurement invariance and incorporating latent factors into subsequent analyses [ 56 ] or the major problem of factor loading matrix rotation [ 57 ]. Studies eventually began to employ CFA, a technique that overcame some of these obstacles [ 56 ] but had other drawbacks; for example, the strict requirement of zero cross-loadings often does not fit the data well, and misspecification of zero loadings tends to produce distorted factors [ 57 ]. Recently, exploratory structural equation modeling (ESEM) has been proposed. This technique is widely recommended both conceptually and empirically to assess the internal structure of psychological tools [ 58 ] since it overcomes the limitations of EFA and CFA in estimating their parameters [ 56 , 57 ].

The next section, reliability, reports the total number of items according to Cronbach’s α reliability coefficient. Reliability is defined as a combination of systematic and random influences that determine the observed scores on a psychological test. Reporting the reliability measure ensures that item-based scores are consistent, that the tool’s responses are replicable and that they are not modified solely by random noise [ 59 , 60 ]. Currently, the most commonly employed reliability coefficient in studies with a multi-item measurement scale (MIMS) is Cronbach’s α [ 60 , 61 ].

Cronbach’s α [ 62 ] is based on numerous strict assumptions (e.g., the test must be unidimensional, factor loadings must be equal for all items and item errors should not covary) to estimate internal consistency. These assumptions are difficult to meet, and their violation may produce small reliability estimates [ 60 ]. One of the alternative measures to α that is increasingly recommended by the scientific literature is McDonald’s ω [ 63 ], a composite reliability measure. This coefficient is recommended for congeneric scales in which tau equivalence is not assumed. It has several advantages. For example, estimates of ω are usually robust when the estimated model contains more factors than the true model, even with small samples, or when skewness in univariate item distributions produces lower biases than those found when using α [ 59 ].

The test-retest method was the next most commonly reported internal structure section in these studies. This type of reliability considers the consistency of the scores of a test between two measurements separated by a period [ 64 ]. It is striking that test-retest consistency does not have a prevalence similar to that of internal consistency since, unlike internal consistency, test-retest consistency can be assessed for practically all types of patient-reported outcomes. It is even considered by some measurement experts to report reliability with greater relevance than internal consistency since it plays a fundamental role in the calculation of parameters for health measures [ 64 ]. However, the literature provides little guidance regarding the assessment of this type of reliability.

The internal structure section that was least frequently reported in the studies in this review was invariance. A lack of invariance refers to a difference between scores on a test that is not explained by group differences in the structure it is intended to measure [ 65 ]. The invariance of the measure should be emphasized as a prerequisite in comparisons between groups since “if scale invariance is not examined, item bias may not be fully recognized and this may lead to a distorted interpretation of the bias in a particular psychological measure” [ 65 ].

Evidence related to other variables was the next most reported source of validity in the studies included in this review. Specifically, the four studies that reported this evidence did so according to convergent validity and cited several instruments. None of the studies included evidence of discriminant validity, although this may be because there are currently several obstacles related to the measurement of this type of validity [ 66 ]. On the one hand, different definitions are used in the applied literature, which makes its evaluation difficult; on the other hand, the literature on discriminant validity focuses on techniques that require the use of multiple measurement methods, which often seem to have been introduced without sufficient evidence or are applied randomly.

Validity related to response processes was not reported by any of the studies. There are several methods to analyze this validity. These methods can be divided into two groups: “those that directly access the psychological processes or cognitive operations (think aloud, focus group, and interviews), compared to those which provide indirect indicators which in turn require additional inference (eye tracking and response times)” [ 38 ]. However, this validity evidence has traditionally been reported less frequently than others in most studies, perhaps because there are fewer clear and accepted practices on how to design or report these studies [ 67 ].

Finally, the consequences of testing were not reported in any of the studies. There is debate regarding this source of validity, with two main opposing streams of thought. On the one hand [ 68 , 69 ]) suggests that consequences that appear after the application of a test should not derive from any source of test invalidity and that “adverse consequences only undermine the validity of an assessment if they can be attributed to a problem of fit between the test and the construct” (p. 6). In contrast, Cronbach [ 69 , 70 ] notes that adverse social consequences that may result from the application of a test may call into question the validity of the test. However, the potential risks that may arise from the application of a test should be minimized in any case, especially in regard to health assessments. To this end, it is essential that this aspect be assessed by instrument developers and that the experiences of respondents be protected through the development of comprehensive and informed practices [ 39 ].

This work is not without limitations. First, not all published validation studies of the P-CAT, such as the Italian version by Brugnolli et al. [ 31 ], were available. These studies could have provided relevant information. Second, many sources of validity could not be analyzed because the studies provided scant or no data, such as response processes [ 25 , 27 , 28 , 29 , 30 , 40 , 46 ], relationships with other variables [ 27 , 28 , 40 ], consequences of testing [ 25 , 27 , 28 , 29 , 30 , 40 , 46 ], or invariance [ 25 , 27 , 28 , 29 , 30 , 40 , 46 ] in the case of internal structure and interfactor R [ 27 , 28 , 30 , 40 , 46 ], internal replication [ 27 , 28 , 29 , 30 , 40 , 46 ] or the effect of the method [ 25 , 27 , 28 , 29 , 30 , 40 , 46 ] in the case of dimensionality. In the future, it is hoped that authors will become aware of the importance of validity, as shown in this article and many others, and provide data on unreported sources so that comprehensive validity studies can be performed.

The present work also has several strengths. The search was extensive, and many studies were obtained using three different databases, including WoS, one of the most widely used and authoritative databases in the world. This database includes a large number and variety of articles and is not fully automated due to its human team [ 71 , 72 , 73 ]. In addition, to prevent publication bias, gray literature search engines such as Google Scholar were used to avoid the exclusion of unpublished research [ 44 ]. Finally, linguistic bias was prevented by not limiting the search to articles published in only one or two languages, thus avoiding the overrepresentation of studies in one language and underrepresentation in others [ 43 ].

Conclusions

Validity is understood as the degree to which tests and theory support the interpretations of instrument scores for their intended use [ 37 ]. From this perspective, the various validations of the P-CAT are not presented in a structured, valid, theory-based procedural framework like the “Standards” are. After integration and analysis of the results, it was observed that these validation reports offer a high number of sources of validity related to test content, internal structure in dimensionality and internal consistency, a moderate number of sources for internal structure in terms of test-retest reliability and the relationship with other variables, and a very low number of sources for response processes, internal structure in terms of invariance, and test consequences.

Validity plays a fundamental role in ensuring a sound scientific basis for test interpretations because it provides evidence of the extent to which the data provided by the test are valid for the intended purpose. This can affect clinical practice as people’s health may depend on it. In this sense, the “Standards” are considered a suitable and valid theory-based procedural framework for studying this modern conception of questionnaire validity, which should be taken into account in future research in this area.

Although the P-CAT is one of the most widely used instruments for assessing PCC, as shown in this study, PCC has rarely been studied. The developers of measurement tests applied to the health care setting, on which the health and quality of life of many people may depend, should use this validity framework to reflect the clear purpose of the measurement. This approach is important because the equity of decision making by healthcare professionals in daily clinical practice may depend on the source of validity. Through a more extensive study of validity that includes the interpretation of scores in terms of their intended use, the applicability of the P-CAT, an instrument that was initially developed for long-term care homes for elderly people, could be expanded to other care settings. However, the findings of this study show that validation studies continue to focus on traditionally studied types of validity and overlook the interpretation of scores in terms of their intended use.

Data availability

All data relevant to the study were included in the article or uploaded as additional files. Additional template data extraction forms are available from the corresponding author upon reasonable request.

Abbreviations

American Educational Research Association

American Psychological Association

Confirmatory factor analysis

Organizational Climate

Caregiver Psychological Elder Abuse Behavior Scale

Exploratory factor analysis

Exploratory structural equation modeling

Staff-based Measures of Individualized Care for Institutionalized Persons with Dementia

Maslach Burnout Inventory

Multi-item measurement scale

Maximum likelihood

National Council on Measurement in Education

Person-Centered Care Assessment Tool

• Person-centered care

Person-Centered Climate Questionnaire–Staff Version

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

International Register of Systematic Review Protocols

Standards for Educational and Psychological Testing

weighted least square mean and variance adjusted

Web of Science

Institute of Medicine. Crossing the quality chasm: a new health system for the 21st century. Washington, DC: National Academy; 2001.

Google Scholar  

International Alliance of Patients’ Organizations. What is patient-centred healthcare? A review of definitions and principles. 2nd ed. London, UK: International Alliance of Patients’ Organizations; 2007.

World Health Organization. WHO global strategy on people-centred and integrated health services: interim report. Geneva, Switzerland: World Health Organization; 2015.

Britten N, Ekman I, Naldemirci Ö, Javinger M, Hedman H, Wolf A. Learning from Gothenburg model of person centred healthcare. BMJ. 2020;370:m2738.

Article   PubMed   Google Scholar  

Van Diepen C, Fors A, Ekman I, Hensing G. Association between person-centred care and healthcare providers’ job satisfaction and work-related health: a scoping review. BMJ Open. 2020;10:e042658.

Article   PubMed   PubMed Central   Google Scholar  

Ekman N, Taft C, Moons P, Mäkitalo Å, Boström E, Fors A. A state-of-the-art review of direct observation tools for assessing competency in person-centred care. Int J Nurs Stud. 2020;109:103634.

American Geriatrics Society Expert Panel on Person-Centered Care. Person-centered care: a definition and essential elements. J Am Geriatr Soc. 2016;64:15–8.

Article   Google Scholar  

McCormack B, McCance TV. Development of a framework for person-centred nursing. J Adv Nurs. 2006;56:472–9.

McCormack B, McCance T. Person-centred practice in nursing and health care: theory and practice. Chichester, England: Wiley; 2016.

Nolan MR, Davies S, Brown J, Keady J, Nolan J. Beyond person-centred care: a new vision for gerontological nursing. J Clin Nurs. 2004;13:45–53.

McCormack B, McCance T. Person-centred nursing: theory, models and methods. Oxford, UK: Wiley-Blackwell; 2010.

Book   Google Scholar  

Abraha I, Rimland JM, Trotta FM, Dell’Aquila G, Cruz-Jentoft A, Petrovic M, et al. Systematic review of systematic reviews of non-pharmacological interventions to treat behavioural disturbances in older patients with dementia. The SENATOR-OnTop series. BMJ Open. 2017;7:e012759.

Anderson K, Blair A. Why we need to care about the care: a longitudinal study linking the quality of residential dementia care to residents’ quality of life. Arch Gerontol Geriatr. 2020;91:104226.

Bauer M, Fetherstonhaugh D, Haesler E, Beattie E, Hill KD, Poulos CJ. The impact of nurse and care staff education on the functional ability and quality of life of people living with dementia in aged care: a systematic review. Nurse Educ Today. 2018;67:27–45.

Smythe A, Jenkins C, Galant-Miecznikowska M, Dyer J, Downs M, Bentham P, et al. A qualitative study exploring nursing home nurses’ experiences of training in person centred dementia care on burnout. Nurse Educ Pract. 2020;44:102745.

McCormack B, Borg M, Cardiff S, Dewing J, Jacobs G, Janes N, et al. Person-centredness– the ‘state’ of the art. Int Pract Dev J. 2015;5:1–15.

Wilberforce M, Challis D, Davies L, Kelly MP, Roberts C, Loynes N. Person-centredness in the care of older adults: a systematic review of questionnaire-based scales and their measurement properties. BMC Geriatr. 2016;16:63.

Rathert C, Wyrwich MD, Boren SA. Patient-centered care and outcomes: a systematic review of the literature. Med Care Res Rev. 2013;70:351–79.

Sharma T, Bamford M, Dodman D. Person-centred care: an overview of reviews. Contemp Nurse. 2016;51:107–20.

Ahmed S, Djurkovic A, Manalili K, Sahota B, Santana MJ. A qualitative study on measuring patient-centered care: perspectives from clinician-scientists and quality improvement experts. Health Sci Rep. 2019;2:e140.

Edvardsson D, Fetherstonhaugh D, Nay R, Gibson S. Development and initial testing of the person-centered Care Assessment Tool (P-CAT). Int Psychogeriatr. 2010;22:101–8.

Tamagawa R, Groff S, Anderson J, Champ S, Deiure A, Looyis J, et al. Effects of a provincial-wide implementation of screening for distress on healthcare professionals’ confidence and understanding of person-centered care in oncology. J Natl Compr Canc Netw. 2016;14:1259–66.

Degl’ Innocenti A, Wijk H, Kullgren A, Alexiou E. The influence of evidence-based design on staff perceptions of a supportive environment for person-centered care in forensic psychiatry. J Forensic Nurs. 2020;16:E23–30.

Hulin CL. A psychometric theory of evaluations of item and scale translations: fidelity across languages. J Cross Cult Psychol. 1987;18:115–42.

Martínez T, Suárez-Álvarez J, Yanguas J, Muñiz J. Spanish validation of the person-centered Care Assessment Tool (P-CAT). Aging Ment Health. 2016;20:550–8.

Martínez T, Martínez-Loredo V, Cuesta M, Muñiz J. Assessment of person-centered care in gerontology services: a new tool for healthcare professionals. Int J Clin Health Psychol. 2020;20:62–70.

Rokstad AM, Engedal K, Edvardsson D, Selbaek G. Psychometric evaluation of the Norwegian version of the person-centred Care Assessment Tool. Int J Nurs Pract. 2012;18:99–105.

Sjögren K, Lindkvist M, Sandman PO, Zingmark K, Edvardsson D. Psychometric evaluation of the Swedish version of the person-centered Care Assessment Tool (P-CAT). Int Psychogeriatr. 2012;24:406–15.

Zhong XB, Lou VW. Person-centered care in Chinese residential care facilities: a preliminary measure. Aging Ment Health. 2013;17:952–8.

Tak YR, Woo HY, You SY, Kim JH. Validity and reliability of the person-centered Care Assessment Tool in long-term care facilities in Korea. J Korean Acad Nurs. 2015;45:412–9.

Brugnolli A, Debiasi M, Zenere A, Zanolin ME, Baggia M. The person-centered Care Assessment Tool in nursing homes: psychometric evaluation of the Italian version. J Nurs Meas. 2020;28:555–63.

Bru-Luna LM, Martí-Vilar M, Merino-Soto C, Livia J. Reliability generalization study of the person-centered Care Assessment Tool. Front Psychol. 2021;12:712582.

Edvardsson D, Innes A. Measuring person-centered care: a critical comparative review of published tools. Gerontologist. 2010;50:834–46.

Hawkins M, Elsworth GR, Nolte S, Osborne RH. Validity arguments for patient-reported outcomes: justifying the intended interpretation and use of data. J Patient Rep Outcomes. 2021;5:64.

Sireci SG. On the validity of useless tests. Assess Educ Princ Policy Pract. 2016;23:226–35.

Hawkins M, Elsworth GR, Osborne RH. Questionnaire validation practice: a protocol for a systematic descriptive literature review of health literacy assessments. BMJ Open. 2019;9:e030753.

American Educational Research Association, American Psychological Association. National Council on Measurement in Education. Standards for educational and psychological testing. Washington, DC: American Educational Research Association; 2014.

Padilla JL, Benítez I. Validity evidence based on response processes. Psicothema. 2014;26:136–44.

PubMed   Google Scholar  

Hawkins M, Elsworth GR, Hoban E, Osborne RH. Questionnaire validation practice within a theoretical framework: a systematic descriptive literature review of health literacy assessments. BMJ Open. 2020;10:e035974.

Le C, Ma K, Tang P, Edvardsson D, Behm L, Zhang J, et al. Psychometric evaluation of the Chinese version of the person-centred Care Assessment Tool. BMJ Open. 2020;10:e031580.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Int J Surg. 2021;88:105906.

Falagas ME, Pitsouni EI, Malietzis GA, Pappas G. Comparison of PubMed, Scopus, web of Science, and Google Scholar: strengths and weaknesses. FASEB J. 2008;22:338–42.

Grégoire G, Derderian F, Le Lorier J. Selecting the language of the publications included in a meta-analysis: is there a tower of Babel bias? J Clin Epidemiol. 1995;48:159–63.

Arias MM. Aspectos metodológicos Del metaanálisis (1). Pediatr Aten Primaria. 2018;20:297–302.

Covidence. Covidence systematic review software. Veritas Health Innovation, Australia. 2014. https://www.covidence.org/ . Accessed 28 Feb 2022.

Selan D, Jakobsson U, Condelius A. The Swedish P-CAT: modification and exploration of psychometric properties of two different versions. Scand J Caring Sci. 2017;31:527–35.

Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine (Phila Pa 1976). 2000;25:3186–91.

Guillemin F. Cross-cultural adaptation and validation of health status measures. Scand J Rheumatol. 1995;24:61–3.

Hambleton R, Merenda P, Spielberger C. Adapting educational and psychological tests for cross-cultural assessment. Mahwah, NJ: Lawrence Erlbaum Associates; 2005.

Muñiz J, Elosua P, Hambleton RK. International test commission guidelines for test translation and adaptation: second edition. Psicothema. 2013;25:151–7.

Rosengren K, Brannefors P, Carlstrom E. Adoption of the concept of person-centred care into discourse in Europe: a systematic literature review. J Health Organ Manag. 2021;35:265–80.

Alharbi T, Olsson LE, Ekman I, Carlström E. The impact of organizational culture on the outcome of hospital care: after the implementation of person-centred care. Scand J Public Health. 2014;42:104–10.

Bensbih S, Souadka A, Diez AG, Bouksour O. Patient centered care: focus on low and middle income countries and proposition of new conceptual model. J Med Surg Res. 2020;7:755–63.

Stranz A, Sörensdotter R. Interpretations of person-centered dementia care: same rhetoric, different practices? A comparative study of nursing homes in England and Sweden. J Aging Stud. 2016;38:70–80.

Zhou LM, Xu RH, Xu YH, Chang JH, Wang D. Inpatients’ perception of patient-centered care in Guangdong province, China: a cross-sectional study. Inquiry. 2021. https://doi.org/10.1177/00469580211059482 .

Marsh HW, Morin AJ, Parker PD, Kaur G. Exploratory structural equation modeling: an integration of the best features of exploratory and confirmatory factor analysis. Annu Rev Clin Psychol. 2014;10:85–110.

Asparouhov T, Muthén B. Exploratory structural equation modeling. Struct Equ Model Multidiscip J. 2009;16:397–438.

Cabedo-Peris J, Martí-Vilar M, Merino-Soto C, Ortiz-Morán M. Basic empathy scale: a systematic review and reliability generalization meta-analysis. Healthc (Basel). 2022;10:29–62.

Flora DB. Your coefficient alpha is probably wrong, but which coefficient omega is right? A tutorial on using R to obtain better reliability estimates. Adv Methods Pract Psychol Sci. 2020;3:484–501.

McNeish D. Thanks coefficient alpha, we’ll take it from here. Psychol Methods. 2018;23:412–33.

Hayes AF, Coutts JJ. Use omega rather than Cronbach’s alpha for estimating reliability. But… Commun Methods Meas. 2020;14:1–24.

Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297–334.

McDonald R. Test theory: a unified approach. Mahwah, NJ: Erlbaum; 1999.

Polit DF. Getting serious about test-retest reliability: a critique of retest research and some recommendations. Qual Life Res. 2014;23:1713–20.

Ceylan D, Çizel B, Karakaş H. Testing destination image scale invariance for intergroup comparison. Tour Anal. 2020;25:239–51.

Rönkkö M, Cho E. An updated guideline for assessing discriminant validity. Organ Res Methods. 2022;25:6–14.

Hubley A, Zumbo B. Response processes in the context of validity: setting the stage. In: Zumbo B, Hubley A, editors. Understanding and investigating response processes in validation research. Cham, Switzerland: Springer; 2017. pp. 1–12.

Messick S. Validity of performance assessments. In: Philips G, editor. Technical issues in large-scale performance assessment. Washington, DC: Department of Education, National Center for Education Statistics; 1996. pp. 1–18.

Moss PA. The role of consequences in validity theory. Educ Meas Issues Pract. 1998;17:6–12.

Cronbach L. Five perspectives on validity argument. In: Wainer H, editor. Test validity. Hillsdale, MI: Erlbaum; 1988. pp. 3–17.

Birkle C, Pendlebury DA, Schnell J, Adams J. Web of Science as a data source for research on scientific and scholarly activity. Quant Sci Stud. 2020;1:363–76.

Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. 2017;6:245.

Web of Science Group. Editorial selection process. Clarivate. 2024. https://clarivate.com/webofsciencegroup/solutions/%20editorial-selection-process/ . Accessed 12 Sept 2022.

Download references

Acknowledgements

The authors thank the casual helpers for their aid in information processing and searching.

This work is one of the results of research project HIM/2015/017/SSA.1207, “Effects of mindfulness training on psychological distress and quality of life of the family caregiver”. Main researcher: Filiberto Toledano-Toledano Ph.D. The present research was funded by federal funds for health research and was approved by the Commissions of Research, Ethics and Biosafety (Comisiones de Investigación, Ética y Bioseguridad), Hospital Infantil de México Federico Gómez, National Institute of Health. The source of federal funds did not control the study design, data collection, analysis, or interpretation, or decisions regarding publication.

Author information

Authors and affiliations.

Departamento de Educación, Facultad de Ciencias Sociales, Universidad Europea de Valencia, 46010, Valencia, Spain

Lluna Maria Bru-Luna

Departamento de Psicología Básica, Universitat de València, Blasco Ibáñez Avenue, 21, 46010, Valencia, Spain

Manuel Martí-Vilar

Departamento de Psicología, Instituto de Investigación de Psicología, Universidad de San Martín de Porres, Tomás Marsano Avenue 242, Lima 34, Perú

César Merino-Soto

Instituto Central de Gestión de la Investigación, Universidad Nacional Federico Villarreal, Carlos Gonzalez Avenue 285, 15088, San Miguel, Perú

José Livia-Segovia

Unidad de Investigación en Medicina Basada en Evidencias, Hospital Infantil de México Federico Gómez Instituto Nacional de Salud, Dr. Márquez 162, 06720, Doctores, Cuauhtémoc, Mexico

Juan Garduño-Espinosa & Filiberto Toledano-Toledano

Unidad de Investigación Multidisciplinaria en Salud, Instituto Nacional de Rehabilitación Luis Guillermo Ibarra Ibarra, México-Xochimilco 289, Arenal de Guadalupe, 14389, Tlalpan, Mexico City, Mexico

Filiberto Toledano-Toledano

Dirección de Investigación y Diseminación del Conocimiento, Instituto Nacional de Ciencias e Innovación para la Formación de Comunidad Científica, INDEHUS, Periférico Sur 4860, Arenal de Guadalupe, 14389, Tlalpan, Mexico City, Mexico

You can also search for this author in PubMed   Google Scholar

Contributions

L.M.B.L. conceptualized the study, collected the data, performed the formal anal- ysis, wrote the original draft, and reviewed and edited the subsequent drafts. M.M.V. collected the data and reviewed and edited the subsequent drafts. C.M.S. collected the data, performed the formal analysis, wrote the original draft, and reviewed and edited the subsequent drafts. J.L.S. collected the data, wrote the original draft, and reviewed and edited the subsequent drafts. J.G.E. collected the data and reviewed and edited the subsequent drafts. F.T.T. conceptualized the study and reviewed and edited the subsequent drafts. L.M.B.L. conceptualized the study and reviewed and edited the subsequent drafts. M.M.V. conceptualized the study and reviewed and edited the subsequent drafts. C.M.S. reviewed and edited the subsequent drafts. J.G.E. reviewed and edited the subsequent drafts. F.T.T. conceptualized the study; provided resources, software, and supervision; wrote the original draft; and reviewed and edited the subsequent drafts.

Corresponding author

Correspondence to Filiberto Toledano-Toledano .

Ethics declarations

Ethics approval and consent to participate.

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Commissions of Research, Ethics and Biosafety (Comisiones de Investigación, Ética y Bioseguridad), Hospital Infantil de México Federico Gómez, National Institute of Health. HIM/2015/017/SSA.1207, “Effects of mindfulness training on psychological distress and quality of life of the family caregiver”.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Bru-Luna, L.M., Martí-Vilar, M., Merino-Soto, C. et al. Person-centered care assessment tool with a focus on quality healthcare: a systematic review of psychometric properties. BMC Psychol 12 , 217 (2024). https://doi.org/10.1186/s40359-024-01716-7

Download citation

Received : 17 May 2023

Accepted : 07 April 2024

Published : 19 April 2024

DOI : https://doi.org/10.1186/s40359-024-01716-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Person-centered care assessment tool

BMC Psychology

ISSN: 2050-7283

• General enquiries: [email protected]

• Open access
• Published: 08 November 2023

Policies to prevent zoonotic spillover: a systematic scoping review of evaluative evidence

• Chloe Clifford Astbury 1 , 2 , 3 ,
• Kirsten M. Lee 1 , 2 ,
• Ryan Mcleod 1 ,
• Raphael Aguiar 2 ,
• Asma Atique 1 ,
• Marilen Balolong 4 ,
• Janielle Clarke 1 ,
• Anastassia Demeshko 1 ,
• Ronald Labonté 5 ,
• Arne Ruckert 5 ,
• Priyanka Sibal 6 ,
• Kathleen Chelsea Togño 4 ,
• A. M. Viens 1 , 3 ,
• Mary Wiktorowicz 1 , 2 ,
• Marc K. Yambayamba 7 ,
• Amy Yau 8 &
• Tarra L. Penney 1 , 2 , 3  

Globalization and Health volume  19 , Article number:  82 ( 2023 ) Cite this article

2931 Accesses

24 Altmetric

Metrics details

Emerging infectious diseases of zoonotic origin present a critical threat to global population health. As accelerating globalisation makes epidemics and pandemics more difficult to contain, there is a need for effective preventive interventions that reduce the risk of zoonotic spillover events. Public policies can play a key role in preventing spillover events. The aim of this review is to identify and describe evaluations of public policies that target the determinants of zoonotic spillover. Our approach is informed by a One Health perspective, acknowledging the inter-connectedness of human, animal and environmental health.

In this systematic scoping review, we searched Medline, SCOPUS, Web of Science and Global Health in May 2021 using search terms combining animal health and the animal-human interface, public policy, prevention and zoonoses. We screened titles and abstracts, extracted data and reported our process in line with PRISMA-ScR guidelines. We also searched relevant organisations’ websites for evaluations published in the grey literature. All evaluations of public policies aiming to prevent zoonotic spillover events were eligible for inclusion. We summarised key data from each study, mapping policies along the spillover pathway.

Our review found 95 publications evaluating 111 policies. We identified 27 unique policy options including habitat protection; trade regulations; border control and quarantine procedures; farm and market biosecurity measures; public information campaigns; and vaccination programmes, as well as multi-component programmes. These were implemented by many sectors, highlighting the cross-sectoral nature of zoonotic spillover prevention. Reports emphasised the importance of surveillance data in both guiding prevention efforts and enabling policy evaluation, as well as the importance of industry and private sector actors in implementing many of these policies. Thoughtful engagement with stakeholders ranging from subsistence hunters and farmers to industrial animal agriculture operations is key for policy success in this area.

This review outlines the state of the evaluative evidence around policies to prevent zoonotic spillover in order to guide policy decision-making and focus research efforts. Since we found that most of the existing policy evaluations target ‘downstream’ determinants, additional research could focus on evaluating policies targeting ‘upstream’ determinants of zoonotic spillover, such as land use change, and policies impacting infection intensity and pathogen shedding in animal populations, such as those targeting animal welfare.

The increasing incidence of zoonotic emerging infectious diseases (EIDs) has been attributed to behavioural practices and ecological and socioeconomic change, and is predicted to continue in the coming years [ 1 ]. Higher levels of anthropogenic activity, including agricultural intensification, urbanisation and other forms of land use change, have led to increased interactions between wildlife, humans and livestock, increasing the risk of cross-species transmission [ 2 , 3 , 4 ]. Meanwhile, accelerating rates of globalisation and urbanisation, leading to increased global movement of people and goods and more dense human settlements, have made outbreaks of disease in human populations more difficult to contain [ 5 ]. In response, a call has been issued by leading organisations and experts, including the United Nations Environment Programme, the International Livestock Research Institute and the Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services, to complement reactive policy responses with policies that prevent zoonotic EIDs [ 1 , 6 , 7 , 8 , 9 , 10 ]. This approach, sometimes called deep prevention, would need to target upstream drivers to reduce the risk of outbreaks occuring [ 11 ].

Zoonotic spillover, defined as the transmission of a pathogen from an animal to a human, depends on the alignment of ecological, epidemiological and behavioural factors [ 12 ]. Zoonotic pathogens must be transmitted across a spillover pathway (Fig.  1 ) in order to induce infections in humans [ 12 , 13 ]. This involves meeting a series of conditions including appropriate density and distribution of reservoir hosts, pathogen prevalence, infection intensity and human exposure [ 12 ]. Across this pathway, a number of drivers of zoonotic spillover have been identified, including changes in wildlife and livestock populations [ 14 ]; deforestation, urbanisation and other forms of land use change [ 15 , 16 ]; bushmeat consumption [ 17 , 18 , 19 ]; and a variety of human practices including hunting, farming, animal husbandry, mining, keeping of exotic pets and trade [ 8 , 9 , 20 , 21 , 22 ]. These large-scale changes have repeatedly given rise to spillover events [ 2 , 15 , 23 ], sometimes involving pathogens with epidemic or pandemic potential [ 24 ].

Spillover pathway adapted from Plowright et al. [ 12 , 13 ]

The responsibility for addressing zoonotic disease frequently spans multiple sectors of governance due to its relevance for both animals and humans. A One Health perspective, which recognises the health of humans, animals and the environment as being closely linked and inter-dependent [ 25 ], can be useful in understanding the spillover pathway and drivers of spillover events, as well as informing policy and governance approaches to address this cross-sectoral problem. At the international level, the World Health Organization, the Food and Agriculture Organization, the World Organisation for Animal Health and the United Nations Environment Programme have endorsed a One Health approach to policymaking to respond to zoonotic infectious diseases, emphasising collaboration between agencies [ 26 ].

Operationalising a One Health approach to policy

While One Health is a promising approach to preventing zoonotic EIDs, operationalising this concept remains a challenge. Evaluative evidence exists around the effectiveness of interventions to prevent spillover events [ 13 , 27 , 28 , 29 ], however these have often been implemented as short- to medium-term programmes or academic investigations [ 8 ]. In some cases, zoonoses have re-emerged after successful programmes have ended [ 29 ]. As a result, experts have argued for the incorporation of successful interventions into policy frameworks, providing interventions with the sustainability required for long-term disease control [ 8 , 10 ].

Operationalising a One Health approach to policy involves understanding the policy options, identifying the stakeholders involved and developing insights into how to successfully implement and evaluate these policies. Although the longevity and scope of government actions may make policy an effective vehicle for prevention of emerging diseases, implementing policy is a complex process involving numerous actors with competing views and interests [ 30 ]. This context presents challenges for policy development and implementation. Where relevant policies are designed and implemented in isolation, opportunities for co-benefits may be missed and interventions may produce unintended consequences [ 31 ]. Finally, while evaluative evidence is key to informing future policy decisions, the complex systems in which policies are often implemented make evaluation challenging [ 32 ].

Aims and scope

To provide insights around how to use policy to successfully prevent zoonotic spillover events, it is necessary to synthesise the available evaluative evidence. A One Health perspective allows this evidence synthesis to incorporate a wide range of policy instruments and actors and to identify approaches to successfully implementing and evaluating policies in this complex, multi-sectoral context.

Approaches to managing epidemic and pandemic infectious pathogens when they have entered human populations have been systematically catalogued in the medical literature [ 33 , 34 , 35 , 36 , 37 , 38 , 39 ]. These measures include hand washing, face masks, school closures, contact tracing, vaccination and case isolation. Further upstream, systematic reviews of interventions targeting the spillover pathway have predominantly focused on programmes rather than policies, and have been restricted by various characteristics such as geographic region [ 28 ] or pathogen type [ 29 ], or focused on programmes with an explicit endorsement of a One Health approach [ 27 ]. In consequence, a comprehensive understanding of what policies to prevent zoonotic spillover have been evaluated, what actors are involved, and how to successfully implement and evaluate them, is lacking. To address these research gaps, our objective was to synthesise the existing evaluative evidence around policies that target the determinants of zoonotic spillover.

Our approach to identifying and analysing this literature was informed by a One Health perspective, acknowledging the inter-connectedness of human, animal and environmental health.

We conducted a systematic scoping review of evaluations of policies aimed at preventing zoonotic spillover events, based on a previously published protocol [ 40 ]. Results are reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews [ 41 ]. The scoping review was conducted in line with guidelines published by Arksey and O’Malley and refined by Levac and colleagues [ 42 , 43 , 44 ], which emphasise an iterative approach suited to an exploratory research question.

The One Health perspective guided the development of the review methodology. This included the search strategy and inclusion criteria, which allow for the inclusion of policies focused on human, animal or environmental health (or any combination of these areas) and with leadership from one or more of these sectors, and the research questions, which seek to outline the policies and the range of sectors involved in implementation. While our focus on the spillover pathway meant we only included policies that had been evaluated in terms of their impacts on animal and human population distributions, health and interactions, we explicitly searched for environment-focused policies (e.g., protection of wetlands and other wildlife habitats) that might have been evaluated from this perspective. We also aimed to interrogate the One Health approach to governance, by assessing to what extent cross-sectoral collaboration – a key tenet of One Health practice [ 25 ] – emerged as a reason for policy success.

Stage 1: identifying the research question

Informed by our research objective, our research questions were:

What policies aimed at preventing zoonotic spillover (i.e., policies that target the determinants of zoonotic spillover included in the spillover pathway [ 12 ]: population distribution, health and interactions) have been evaluated?

What are the types of policies?

Which policy actors (single department, multi-sectoral, whole of government) are involved?

What are the reasons for policy success and failure, and the unintended consequences of implementing these policies?

How has evaluation of these policies been approached in the literature?

What are the methods or study designs used?

What are the outcomes?

What are the opportunities and challenges for evaluation?

Stage 2: identifying relevant studies

We systematically searched four electronic databases (Medline, Scopus, Web of Science, Global Health) in May 2021. The search strategy was organized by the main concepts in our research question: the spillover pathway; public policy; prevention; and zoonotic pathogens. The search strategy was developed iteratively, informed by existing systematic reviews focused on related concepts [ 28 , 45 , 46 , 47 , 48 , 49 ] and known indicator papers meeting inclusion criteria. We also searched the websites of 18 organisations involved in the prevention of zoonotic spillover to identify relevant grey literature. The choice of organisations was informed by an actor mapping exercise in which we identified key international organisations working on the prevention of emerging zoonoses using network sampling [ 50 ]. We searched the websites of a subset of these organisations, focusing on inter-governmental organisations and organisations whose main focus was zoonotic disease. See Supplementary File 1 for details of academic database and grey literature search strategies.

Stage 3: study selection

Studies were included if they met the following criteria:

Primary empirical study with an English-language abstract from any country or region (reviews were excluded);

Study reporting empirical findings from an evaluation of any sort; and.

Study focused on a policy implemented by government that targets the determinants of zoonotic spillover.

Academic records identified through the searches were collated and double screened using the online platform Covidence [ 51 ]. Two researchers (CCA and KML) initially screened titles and abstracts. Title and abstract screening of an initial set of 100 papers was undertaken by both researchers independently. Results were compared to ensure consistency in decisions around study eligibility, and discrepancies were resolved through consensus. This process was repeated until an acceptable level of agreement (> 90%) was reached. The remaining papers were then screened by one of the two reviewers. Full-text screening was undertaken by two independent researchers and discrepancies were resolved by consensus. Studies with full-texts in any language were eligible for inclusion if they include an English-language abstract. Full-text studies published in French, Spanish or Chinese were single-screened by a member of the research team fluent in that language (CCA or AY). Studies published in other languages were translated as necessary.

Grey literature was screened by one researcher (CCA) to determine whether it met the inclusion criteria. Publications were initially screened by looking at titles, tables of contents and executive summaries. Where these indicated that the publication might be eligible, documents were read in full to determine if inclusion criteria were met.

In line with published guidelines, the approach to study selection was refined iteratively when reviewing articles for inclusion [ 42 , 43 , 44 ].

Stage 4: charting the data

Data charting was conducted using a form designed to identify the information required to answer the research question and sub-research questions (see Supplementary File 2). Data charting focused on characteristics of the study, the policy, and the evaluation. For each policy, this included identifying which determinant of zoonotic spillover situated along the spillover pathway was being targeted. For the purpose of this study, we used a model of the spillover pathway adapted from Plowright et al.’s work [ 12 , 13 ], in which we differentiated between wildlife and domesticated animals (Fig.  1 ). This differentiation is important in the policy context, as the wildlife-domesticated animal interface is an important site for intervention, as well as the human-animal interface.

The data charting form was piloted with ten records to ensure that it was consistent with the research question, and revised iteratively [ 42 , 43 , 44 ]. Data charting was conducted by one researcher (CCA, RM, JC, AD or PS) and checked by a second researcher (CCA or KML). Discrepancies were resolved by consensus.

Stage 5: collating, summarising and reporting the results

Our protocol stated that we would use the Quality Assessment Tool for Quantitative Studies developed by the Effective Public Health Practice Project [ 52 ] to assess study quality [ 40 ]. However, on reviewing the included studies we selected two tools that were more appropriate to their characteristics: (1) ROBINS-I [ 53 ] for quantitative outcome evaluations and (2) a tool developed by the authors of a previous review [ 54 ] – based on Dixon-Woods et al.’s approach to assessing study credibility and contribution [ 55 ] – for all other study types. Two researchers (CCA and KML) assessed study quality independently for an initial set of 10 studies, before comparing assessments and reaching agreement where discrepancies occurred. This process was repeated until an adequate level of agreement was reached (> 90%). The remaining studies were assessed by a single researcher (CCA or KML). Records were not excluded based on quality assessment. Instead, assessments were primarily used to help synthesize the literature on how policies were evaluated. Quality assessment was not performed on grey literature due to the wide variability in the format and comprehensiveness of included publications.

We analysed the charted data, presenting a numerical summary of the included studies in table form, allowing us to describe the range of policy interventions that have been evaluated, aspects of policy implementation and approaches to evaluation. Based on the charted data, we inductively grouped evaluated policies with similar characteristics into policy types and assigned a policy instrument to each policy type: communication/marketing, guidelines, fiscal, regulation, legislation, environmental/social planning or service provision. We mapped policy types onto the spillover pathway shown in Fig.  1 to outline the policies that have been used to target each of these determinants. Thematic analysis was conducted using the approach described by Braun and Clarke where the focus is guided by the researcher’s analytic interests [ 56 ], with five overarching themes chosen as an a priori coding framework: (1) reasons for policy success; (2) reasons for policy failure; (3) unintended consequences of policy implementation; (4) opportunities for policy evaluation; and (5) challenges for policy evaluation. We selected these themes based on our research questions and previous familiarisation with the included articles during the process of article selection, data extraction and quality assessment. Sub-themes were subsequently identified through close reading and coding of the included articles. Thematic analysis was conducted by one researcher (RM) using the qualitative data analysis software Dedoose [ 57 ] and reviewed by the lead author (CCA).

Study characteristics

After removing duplicates, our searches identified a total of 5064 academic records. After screening titles and abstracts, we considered 330 records for full-text review. We also identified 11 relevant publications through our grey literature search. Grey literature reports were published by five organisations: four organisations focused on health and disease, including an intergovernmental organisation (the World Organisation for Animal Health) and three non-governmental organisations (the One Health Commission, the Global Alliance for Rabies Control and EcoHealth Alliance); and one non-governmental organisation focused on wildlife trade (TRAFFIC). In total, we included 95 publications in this review (PRISMA diagram in Fig.  2 ) [ 58 ].

We excluded studies which assessed the unintended consequences of policies to prevent zoonotic spillover without evaluating their effectiveness. This included studies that looked exclusively at the mental health impacts of mandatory livestock culls on farm workers [ 59 ]; studies which focused on potentially relevant factors, such as the wildlife trade, but with no consideration of outcomes situated on the spillover pathway [ 60 ]; and studies which assessed the detection power of surveillance systems without assessing the impact of associated policy interventions [ 61 , 62 , 63 ].

Policy characteristics

The characteristics of the policies evaluated in the included studies are presented in Supplementary File 3 and summarised in Table  1 . Some studies evaluated more than one policy, particularly modelling studies which compared the impacts of several policy options and process evaluations focused on a range of activities undertaken by a single government. Therefore, the number of evaluated policies (n = 111) is greater than the number of included studies (n = 95).

Most policies were evaluated for their impact on human exposure (21%), pathogen prevalence in domesticated animals (18%), barriers within domesticated animals (15%), and pathogen survival and spread in domesticated animals (9%). There were also a number of multi-component policies studies across multiple stages of the spillover pathway (18%). Fewer studies focused on wildlife health and populations, and none of the included studies evaluated policies for their impact on infection intensity and pathogen release in either domesticated animals or wildlife.

Where the government department responsible for implementing a policy was identified in the paper, most policies were implemented by a single department (35%), although there were a number of multi-sectoral efforts (24%). The range of government sectors responsible for implementing policies to prevent zoonotic spillover included human health, animal health, food safety, agriculture, conservation, national parks, forestry, fisheries, environmental protection, border control and foreign affairs. Policies were predominantly intended to be implemented by private sector actors, including individuals and organisations working in trade, retail, hunting and animal agriculture. However, some policies were also implemented by public sector actors working in public health, veterinary public health and environmental conservation.

Most policies were situated in high-income (49%) and upper middle-income (28%) countries, with studies from East Asia and the Pacific (43%) and Europe and Central Asia (19%) dominating. Publications focused on policies targeting various zoonotic diseases, with the most common being avian influenza (50%), rabies (19%), brucellosis (11%) and Hendra virus (4%).

Most policies were evaluated using process (38%) or outcome (31%) evaluation. The most frequently used policy instrument was legislation (59%), particularly for managing pathogen spread in domesticated animals through measures such as mandatory vaccination, culls or disinfection protocols. Meanwhile, communication and marketing or service provision was more typically used to reduce risk in wildlife and human populations, for example by providing guidance around recommended hygiene protocol, by distributing oral vaccination in wildlife habitat or by offering vaccination to human populations.

PRISMA 2020 diagram [ 58 ]

What policies aimed at preventing zoonotic spillover have been evaluated?

Policy types targeted different determinants across the pathway to zoonotic spillover and used various approaches with different evidence of success (Table  2 ). We identified policy options including culling – both general and targeted – of wild and domesticated animals; habitat protection (limiting activities such as agriculture and animal husbandry in wildlife habitats); supplemental feeding to control wildlife movements; vaccination of both wildlife, domesticated animals and human populations with occupational exposure to animals; policies to improve biosecurity in sites where animals are kept, slaughtered and sold, including mandates and information campaigns; live animal market closures; and bans on hunting and selling wildlife. Where outcomes or impacts were evaluated, most policies saw some level of success (i.e., outcome measures were found to vary in a direction that indicated policy success), though relative effectiveness was not assessed due to variation in study design and outcome measure. Policies with consistent evidence of effectiveness – where outcome measures varied in a direction that indicated policy success in all studies included in the review – included culling and sterilisation of wildlife populations, habitat protection, vaccination in wildlife and domesticated animal populations and mandated disinfection protocols. Policies with equivocal evidence of success (i.e., outcome measures varied in different directions or studies had different findings, some indicating success and some indicating failure) included supplemental feeding of wildlife, pre-emptive livestock culls, live animal market closures and bans on wildlife hunting, trade and consumption. For many policies, there were no impact or outcome evaluations identified in this review.

What are the reasons for policy success?

The evidence from the identified impact and outcome evaluations suggests that most of the policies succeeded to some extent. A range of factors contributed to policy success. First, studies emphasized the importance of effective collaboration and coordination between various agencies, disciplines, and levels of government in the execution of policy directives [ 114 , 115 ], in line with a One Health approach to policy and governance. Policy success was attributed, in part, to strong working relationships that encouraged effective communication between various government agencies, and facilitated timely and appropriate policy responses [ 115 ]. Synergy between agencies responsible for surveillance and the execution of control strategies was also reported to be beneficial. For example, prompt communication and effective collaboration between laboratories testing samples and agencies implementing culls in the field was seen as important in the control of highly pathogenic avian influenza in Nigeria [ 116 ]. Similarly, authors also identified the importance of private-public relations and private sector contributions to implementing policies to prevent zoonotic spillover [ 112 ]. This included stronger government engagement with private veterinarians as a factor for success in reducing the spillover of Hendra virus in Queensland [ 109 ], and with farmers, poultry companies and national farming and poultry processing associations in Ghana as part of a successful campaign to reduce risk from highly pathogenic avian influenza [ 112 ]. Studies suggest that the inclusion of private sector stakeholders in the policy process has the potential to improve compliance through transparent dialogue around disease ecology, risk and risk mitigation [ 90 , 91 , 103 , 117 ]; and highlight the utility of participatory approaches in prompting behaviour changes [ 91 ].

Second, authors emphasised the significance of economic incentives, suggesting that policy impact is dependent on private actors’ appraisal of costs and benefits. Studies illustrated how incentives, including compensation, subsidies, rebates, and fines, have had varying degrees of success [ 91 , 97 , 112 , 115 ]. Compensation levels [ 104 , 114 ] and enforcement practices [ 92 ] were identified as salient factors for compliance and adherence. For example, fear of sanctions for bushmeat hunting while a ban was in place in some parts of West Africa were identified as a stronger incentive to avoid bushmeat hunting than the fear of contracting Ebola virus [ 97 ]. Culls were seen as particularly challenging in this regard: while the long-term benefits for farmers may outweigh the financial loss [ 104 ], authorities need to be conscientious of the substantial economic impacts when considering policies that mandate culling or safe disposal [ 95 ]. The direct losses related to compliance (time, labour and expenses) and indirect losses due to price fluctuations and decreases in trade volume, as well as losses to associated industries, are substantial [ 88 , 96 , 113 , 118 ].

Third, trust in government and public support for implemented policy were specified as critical factors influencing the effectiveness of disease control strategies, and research suggests that strategic engagement to facilitate compliance is a necessary step in the policy process [ 97 ]. Participatory approaches that attempt to identify and understand factors influencing compliance have been consistently used to overcome resistance to policy, as insights from engagement and consultation can lead to solutions that facilitate behaviour change at the population level [ 91 , 103 ]. For example, a World Health Organization initiative to reduce avian influenza transmission in poultry markets in Indonesia worked alongside market vendors to achieve its aims, carrying out repeated consultations with the vendors and implementing market infrastructure (such as energy and running water in the market) in collaboration with local authorities to support vendor behaviour change [ 91 ].

Fourth, studies also demonstrated the importance of public communication. The quality of information, as well as the volume, complexity and delivery of public health messages, were key factors [ 75 , 114 ]. Authors contend that communication strategies must understand the target audience and how they interpret and engage with messages [ 97 ], for example by building on relationships where there is exiting trust, such as between veterinarians advising animal vaccination and animal owners [ 117 ]. Homogenously delivered communication strategies were ineffectual: they limited opportunities for open discourse; discounted contradictory lived experiences and expressions of uncertainty; and ultimately contributed to scepticism surrounding implemented policies [ 97 , 117 ].

Finally, studies underscored the importance of surveillance infrastructure to inform intervention strategies. Surveillance programs with the ability to collect and operationalize relevant data were essential to the development of appropriate interventions that are responsive to each unique context [ 115 , 119 ]. Implementing effective surveillance programmes requires the appropriate evaluation tools [ 120 ] and trained personnel [ 81 ].

What are the reasons for policy failure?

Studies showed that perceptions of acceptability and appropriateness were crucial to the effectiveness of implemented policies [ 101 , 104 ]. Several factors were identified that negatively affected acceptability and appropriateness, including: additional expenses for private sector actors without sufficient support [ 75 , 100 , 104 , 112 , 114 ], particularly were culls were demanded but reimbursement for farmers was slow and inadequate, as in a brucellosis eradication campaign in Macedonia [ 81 ]; lack of affordable alternatives [ 97 ]; impracticality of implemented strategies [ 75 , 101 ]; lack of cultural understanding in designing policy interventions [ 97 , 100 ], for example the distribution of footwear to pig farmers in a Polynesian context where footwear was not traditionally worn [ 100 ]; lack of understanding of viral ecology [ 100 ]; as well as public scepticism and distrust [ 97 , 114 ].

Additionally, policy ineffectiveness was associated with poor planning and execution of intervention strategies, including lack of clear direction [ 114 ]; incomplete or inconsistent implementation of control measures (17); limited scope of intervention [ 114 ]; and poor enforcement [ 92 ]. A lack of adequate resources to implement strategies also contributed to policy failure [ 81 ]. Adequate financial resources were necessary to hire and train staff to run surveillance and control operations [ 81 ]. Financial resources were also necessary to fund compensation mechanisms that facilitate compliance. Willingness to adopt policy-prescribed disposal practices was found to be associated with compensation levels (incentives) as a proportion of production price, dependency on income from activities driving zoonotic risk, and contact with prevention staff [ 92 ].

What are the unintended consequences of implementing policies to prevent zoonotic spillover?

A small number of the included studies collected data on the unintended consequences of policies to prevent zoonotic spillover (n = 18). In some instances, unintended consequences were due to disease ecology or human behaviour as a result of policy failure. For example, a study assessing the impacts of the closure of a live poultry market found that, following the closure, vendors travelled to neighbouring markets to sell their animals [ 94 ]. As a result, while cases of avian influenza decreased in the area surrounding the closed market, cases increased in these neighbouring markets, leading to the wider geographic spread of the disease. In another study, elk were provided with supplementary feeding grounds to discourage them from coming into contact with the livestock who shared their range [ 65 ]. While this intervention had the intended consequence of reducing the transmission of brucellosis between elk and livestock, the spread of brucellosis between the elk using the supplementary feeding grounds – who were gathering in larger, tighter groups for longer periods, resulting in higher within-herd transmission – and other elk populations in the area increased. This resulted in an increasing prevalence of brucellosis among the elk, potentially increasing the risk of spillover to livestock. These examples illustrate the complexity of the social and ecological systems in which these policies are implemented, further suggesting the need for a One Health approach to policies to prevent zoonotic spillover.

A key unintended consequence can be attributed to the loss of profits and livelihoods sometimes associated with policies to prevent zoonotic spillover, as described above. The losses incurred by complying with regulations made farmers, hunters and other private sector actors reluctant to report potential infections, contributing to increased unauthorized or illegal activity, and unrestrained spread of disease [ 90 , 92 , 94 , 98 , 112 , 114 ]. Studies investigated the creative ways policy enforcement was circumvented, including hiding hunting equipment on the outskirts of towns or developing informal trade markets and networks [ 97 , 98 ]. Unintended consequences identified in the included evaluations emphasize an opportunity for policymakers to improve sector compliance through public education, levying the influence of consumer attitudes on industry standards [ 104 , 113 ].

A range of study designs were used to evaluate policies. Outcome evaluations (n = 33) used time series or repeat cross-sectional data to conduct evaluations of natural experiments, though most studies did not include a control group for comparison. Outcome evaluations also used case-control and modelling approaches to assess policy impact on an outcome of interest. Process evaluations (n = 30) used cross-sectional and qualitative approaches, as well as study designs combining multiple sources of data, to understand aspects of policy implementation such as the extent to which the policy was being implemented as designed, and the responses and attitudes of stakeholders involved in policy implementation. Economic evaluations (n = 11) included cost-benefit analyses, risk-benefit analyses and modelling studies. Formative evaluations (n = 17) used modelling approaches to estimate what the impacts of a proposed policy option would be in a specific context.

Outcome variables interpreted as indicators of policy success were also numerous and represented determinants along the spillover pathway. As expected, many studies assessed impact on disease transmission, including disease prevalence and incidence, disease eradication, case numbers, and basic reproduction number in human and animal populations, as well as evidence of disease in environmental samples, such as in live animal markets or at carcass disposal sites. Studies also assessed impacts on intermediate factors indicative of successful implementation of specific policies, such as the availability of wild species in markets where a trade ban had been implemented, or knowledge and practices of stakeholders in response to an educational or information campaign.

While most studies found a reduced risk of zoonotic spillover following policy implementation, comparing the magnitude of these impacts was challenging due to the variety of study designs and outcome measures used in the included studies. However, we identified several studies which used modelling to directly compare the impacts of policy options. These studies evaluated various policy scenarios: different combinations within multi-component policy interventions [ 121 ]; culling versus vaccinating wildlife [ 122 ] and livestock [ 84 , 85 ] populations; targeting strategies to humans exclusively versus targeting humans and livestock [ 108 ]; and altering the parameters for culling and vaccination strategies, for example by modelling different ranges for culling and vaccination near infected farms [ 85 ]. These studies often highlighted trade-offs between the effectiveness of policy measures and their cost. For example, estimates of the number of infected flocks were lower when incorporating a ring cull (cull of animals on farms surrounding an outbreak) into a multi-component control strategy for highly pathogenic avian influenza [ 121 ]. However, livestock vaccination was estimated to be a highly effective strategy, with one study findings livestock vaccination to be as or more effective than a pre-emptive cull for outbreak control purposes (depending on the extent of vaccination coverage), while minimising the number of animals culled [ 85 ]. One study jointly modelled costs and benefits of strategies, and found that livestock vaccination had a higher cost-benefit ratio than a wildlife cull [ 122 ]. A final study highlighted the potential of holistic approaches, with drug administration in humans and livestock having a lower cost per disability-adjusted life year averted than intervention in humans alone [ 108 ].

Study authors noted a number of challenges encountered while evaluating policies to prevent zoonotic spillover. One study noted the difficulty of determining the impact of policies aiming to reduce spillover events between wildlife, livestock and humans, as the number of spillover events is often relatively small [ 65 ]. This highlights the importance of considering upstream determinants and risk factors as outcome measures in attempting to evaluate these policies, particularly where spillover events may happen infrequently or not at all during the period of observation. Studying changes in risk factors for spillover can provide insight on the effectiveness of different policies in tackling spillover risk.

Lack of suitable data was a frequently cited barrier to policy evaluation. As policies to prevent zoonotic spillover are often reactive, being implemented in response to an outbreak in animal populations, accessing data from before a policy was implemented was challenging. Studies highlighted the value of routinely collected data, which was often the only data available and was frequently used for policy evaluation [ 65 , 66 , 94 , 115 , 119 , 123 ]. However, in many contexts routine data on animal health is not collected [ 80 ]. Routine testing data from livestock can sometimes be used for evaluation where it exists, but it does not always provide sufficient detail for examining the potential for a policy to prevent zoonotic spillover. For example, some tests do not differentiate between current and past infection, making it difficult to identify where and when spillover occurred [ 65 ], and animal health data may not be granular enough for policy evaluation, particularly in terms of evaluating local policies [ 94 ]. Studies also highlighted instances where the private sector may own data sets reporting disease prevalence and transmission, but may be reluctant to share the data for evaluation purposes [ 121 ]. In such instances, open communication and good relationships with the private sector may be facilitators to evaluation.

Beyond the lack of baseline data, studies highlighted the difficulty in collecting information about policy compliance. As failing to comply often puts farmers and hunters at risk of fines or imprisonment, they were reluctant to disclose information about non-compliance or participation in illegal trade and sale of animals [ 86 , 92 , 97 , 112 ]. This made it difficult to determine policy effectiveness.

Quality assessment

Of the 44 quantitative evaluations, 37 were evaluated as being at moderate or higher risk of bias (see Supplementary File 4), given the possibility of bias in the assessment of intervention impact due to the presence of confounding effects. A small number of studies were determined to be at serious (n = 6) or critical (n = 1) risk of bias, for two main reasons: only having data from after the intervention was implemented; or using a case-control study model without measuring and adjusting for important potential confounders, such as the prevalence of a targeted disease prior to policy implementation. These limitations may reflect the nature of zoonotic spillover events and policy responses, which can happen quickly and leave little time for baseline data collection. Many of the included studies relied on surveillance data, but where such data sets are not available, post-test and case-control study designs may be the only options.

The quality of studies assessed with the tool developed based on Dixon-Woods’ approach [ 55 ] was high overall (n = 41, see Supplementary file 5). Most studies were rated as high in terms of clearly and comprehensively presenting their results (n = 37), analysis (n = 34), research design (n = 33), aims (n = 32) and research process (n = 28). Most studies also had a high relevance to the research question (n = 31), indicating that the research was embedded in policy, being commissioned, co-designed or conducted in partnership with government stakeholders.

We identified a range of policies targeting different parts of the spillover pathway implemented by various policy and governance sectors, including some multi-sectoral initiatives. Policies tended to rely heavily on private sector actors (including actors ranging from small-scale farmers and hunters to larger commercial operations) for implementation, suggesting that open communication and collaboration with these actors was essential for successful policy implementation. Policy success was undermined by lack of collaboration between government agencies; lack of communication between surveillance and control operations; poor understanding of the context in which policies were implemented; and inadequate financial compensation for private sector actors who lost profits and incurred additional costs by complying with policies. Where policies were ineffective, this tended to be due to unintended consequences relating to complex dynamics within the social and ecological systems where policies were implemented. Lack of appropriate data was a key obstacle to policy evaluation, and studies emphasised the importance of robust surveillance infrastructure in evaluating policies that tended to be implemented reactively, in response to an outbreak of zoonotic disease in animal or human populations.

Implications for policy and practice

The key role that the private sector and industry actors play in implementing policies to prevent zoonotic spillover is an important consideration for policymakers. Our findings suggest that many of these policies must be complied with by farmers – from subsistence and smallholder farmers to large corporations – as well as by other actors, such as hunters. Lack of awareness as well as financial costs of compliance among these groups present key barriers to policy success in this area. This set of stakeholders is complex as some may make very marginal profits, if any, and may struggle to afford the additional costs of implementing preventive policies. However, powerful actors and profitable industries are also involved, including large-scale farms and primary resource extraction enterprises [ 22 ]. Acknowledging the differences across these stakeholder groups, and in particular assessing their capacity to bear some of the costs related to prevention, emerges as crucial in successful policy implementation.

Finally, our findings highlight the importance of disease surveillance in efforts to reduce the risk of spillover events. As well as acting as an early warning system, surveillance provides a source of data to evaluate the impact of preventive policies. We found the availability of surveillance data to be a key enabling factor in evaluating policies. In addition, close collaboration between agencies responsible for disease surveillance and control efforts was key to policy success. National surveillance efforts, as well as cross-country collaboration to support global efforts, such as the United States Agency for International Development’s PREDICT program supporting surveillance in areas at high risk for zoonotic disease outbreaks [ 124 ], must be sustained and expanded. In complex areas such as the prevention of zoonotic spillover, approaches to surveillance which encompass risk factors and transmission pathways [ 125 ], as well as One Health surveillance systems which harmonise and integrate data collection and analysis from across human, animal and environmental sectors [ 126 ], are promising approaches to developing surveillance systems that support risk. This context also involves a need to strengthen surveillance capacity in remote and rural locations, as communities living in these contexts may have exposure to numerous pathogens of wildlife origin. This will require strengthening clinical and diagnostic capacity in these settings, as well as engaging with stakeholders such as community human and animal health workers and wildlife or national park rangers [ 127 ].

Comparison with existing literature

This review sought to map the range of policies implemented to reduce the risk of zoonotic spillover, and the various approaches taken to evaluation, and identify factors behind the success and failure of policy implementation and evaluation. Due to this broad scope, comparing relative effectiveness of policy interventions was challenging. Existing systematic reviews with a more specific focus could apply meta-analysis to determine which interventions were most effective. For example, a review of market-level biosecurity measures aiming to reduce the transmission of avian influenza found that reducing market size, separating poultry species, cleaning and disinfecting premises, closing markets and banning overnight storage were highly effective interventions [ 45 ]. However, our findings suggest that studies focused on the control of avian influenza dominate the literature in this space (55 out of 111 evaluated policies), and many of these are focused on market-level measures. Systematic reviews focused on other approaches to reduce spillover risk, such as on-farm biosecurity [ 47 ]; biosecurity for backyard poultry rearing [ 46 ]; and community-based interventions [ 28 ] comment on the paucity of high-quality evidence around the impacts of such approaches. By taking a broad perspective, we hope our findings will provide policy options for consideration in a number of contexts, and guide researchers in focusing their efforts on areas where evidence is lacking.

Strengths and weaknesses of the study

To our knowledge, this is the first attempt to systematically identify and document evaluations of policies aiming to prevent the spillover of zoonotic pathogens into human populations. However, because of the complex drivers of spillover events, some potentially relevant policy evaluations may be excluded where their outcome measures are too far removed from zoonotic spillover. While relevant, such evaluations will be difficult to systematically identify as they make no reference to zoonotic disease.

In addition, this review focused on policy evaluations that have been reported in the peer-reviewed literature and the grey literature published by international agencies and organisations working on these topics. Policies that have been implemented but not evaluated, or evaluated but not published in these literatures, will therefore be excluded from this review. As a result, potentially effective and important policies in the prevention of zoonotic spillover events may not have been identified. However, we hope that the findings from this review will highlight these gaps in the evaluative evidence. We also hope that this review, by extracting practical dimensions, such as study design, outcome measures and the challenges encountered in the evaluation process, will support policymakers and researchers in carrying out further policy evaluations in this space.

Unanswered questions and future research

Our findings highlight several important gaps in the evidence. First, while observational evidence emphasises the importance of upstream determinants such as environmental and ecosystem health in the increasing rate of zoonotic spillover [ 1 , 15 ], we only identified a single evaluation of a policy attempting to target one of these upstream determinants: an evaluation carried out in China to assess the impact of the Ramstar wetland protection program on avian influenza in migratory waterfowl [ 66 ]. This study found that proximity to protected wetlands reduced outbreak risk. Authors hypothesised that this effect was due to the separation of wild waterfowl and poultry populations and the diversion of wild waterfowl away from human-dominated landscapes and toward protected natural habitats. Our findings support existing calls for more quantitative and mechanistic studies of the impact of interventions supporting environmental and ecosystem health on zoonotic spillover risk [ 128 ], as well as calls for greater integration of the environment into One Health research, policy and practice [ 31 ]. Further evaluations of environment and habitat protection policies would strengthen our understanding of this area. In addition, the impact of policies to reduce deforestation or expand forest coverage, such as China’s Grain-to-Green program [ 129 ], on the spillover pathway could be evaluated. Such evaluations might consider potential unintended consequences, as these policies could promote healthier wildlife populations with better disease resistance, but may also facilitate wildlife population growth and higher rates of wildlife-human encounters [ 130 ].

There is also a lack of evaluation of policies targeting infection intensity and pathogen release in either wildlife or domesticated animals. These could include approaches such as improving animal health and welfare to make these populations more resistant to disease [ 13 ]. While arguments have been made for strengthening legal structures supporting animal welfare in order to reduce the risk of zoonotic pathogen transmission [ 131 ], there is a need to evaluate policies that take this approach.

Our review found publications evaluating a wide range of policy interventions spanning the spillover pathway, including habitat protection; trade regulations; border control and quarantine procedures; farm and market biosecurity measures; public information campaigns; and vaccination programmes for wildlife and domesticated animals, as well as human populations with occupational exposure to animals. A wide range of governance sectors implemented these policies, highlighting the prevention of zoonotic spillover as a cross-sectoral issue, though most policies were implemented by a single sector. Our findings highlight the importance of industry and private actors in implementing policies to prevent zoonotic spillover, and the need for thoughtful and effective engagement with this wide range of actors, from subsistence hunters and farmers through to industrial animal agriculture operations to address their concerns through a range of incentives. We also identified the centrality of surveillance data in evaluating policies that are often implemented reactively, and effective collaboration between surveillance and control operations as a central factor in successful policy implementation.

Data Availability

All data generated or analysed during this study are included in this published article and its supplementary information files. Analysis code for descriptive characteristics of included policies is available on GitHub.

Abbreviations

Emerging infectious disease

Morse SS, Mazet JA, Woolhouse M, Parrish CR, Carroll D, Karesh WB, Zambrana-Torrelio C, Lipkin WI, Daszak P. Prediction and prevention of the next pandemic zoonosis. The Lancet. 2012;380:1956–65.

Article   Google Scholar  

Pulliam JRC, Epstein JH, Dushoff J, Rahman SA, Bunning M, Jamaluddin AA, Hyatt AD, Field HE, Dobson AP, Daszak P. Agricultural intensification, priming for persistence and the emergence of Nipah virus: a lethal bat-borne zoonosis. J Royal Soc Interface. 2012;9:89–101.

IPCC. In: Pörtner H-O, Roberts DC, Tignor M, Poloczanska ES, Mintenbeck K, Alegría A, Craig M, Langsdorf S, Löschke S, Möller V, Okem A, Rama B, editors. Climate Change 2022: impacts, adaptation and vulnerability, contribution of Working Group II to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change. In press ed. Cambridge University Press; 2022.

Brenner N, Ghosh S. Between the colossal and the catastrophic: planetary urbanization and the political ecologies of emergent Infectious Disease. Environ Plan A. 2022;54:867–910.

Gallo-Cajiao E, Lieberman S, Dolšak N, et al. Global governance for pandemic prevention and the wildlife trade. Lancet Planet Health. 2023;7:e336–45.

Article   PubMed   PubMed Central   Google Scholar  

Marco MD, Baker ML, Daszak P, et al. Opinion: sustainable development must account for pandemic risk. PNAS. 2020;117:3888–92.

Heymann DL, Dixon M. Infections at the Animal/Human interface: shifting the paradigm from emergency response to Prevention at source. In: Mackenzie JS, Jeggo M, Daszak P, Richt JA, editors. One health: the human-animal-environment interfaces in Emerging Infectious Diseases: Food Safety and Security, and International and National plans for implementation of one health activities. Berlin, Heidelberg: Springer; 2013. pp. 207–15.

Google Scholar  

United Nations Environment Programme, International Livestock Research Institute. (2020) Preventing the next pandemic: Zoonotic diseases and how to break the chain of transmission. 82.

Intergovernmental Science-Policy Platform On Biodiversity And Ecosystem Services (IPBES). (2020) Workshop Report on Biodiversity and Pandemics of the Intergovernmental Platform on Biodiversity and Ecosystem Services (IPBES). https://doi.org/10.5281/ZENODO.4147317 .

One Health theory of change. https://www.who.int/publications/m/item/one-health-theory-of-change . Accessed 30 Jan 2023.

Vinuales J, Moon S, Moli GL, Burci G-L. A global pandemic treaty should aim for deep prevention. The Lancet. 2021;397:1791–2.

Article   CAS   Google Scholar  

Plowright RK, Parrish CR, McCallum H, Hudson PJ, Ko AI, Graham AL, Lloyd-Smith JO. Pathways to zoonotic spillover. Nat Rev Microbiol. 2017;15:502–10.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Sokolow SH, Nova N, Pepin KM, et al. Ecological interventions to prevent and manage zoonotic pathogen spillover. Philosophical Trans Royal Soc B: Biol Sci. 2019;374:20180342.

Johnson CK, Hitchens PL, Pandit PS, Rushmore J, Evans TS, Young CCW, Doyle MM. Global shifts in mammalian population trends reveal key predictors of virus spillover risk. Proc Royal Soc B: Biol Sci. 2020;287:20192736.

Allen T, Murray KA, Zambrana-Torrelio C, Morse SS, Rondinini C, Di Marco M, Breit N, Olival KJ, Daszak P. Global hotspots and correlates of emerging zoonotic Diseases. Nat Commun. 2017;8:1124.

Gandy M. THE ZOONOTIC CITY: Urban Political Ecology and the pandemic imaginary. Int J Urban Reg Res. 2022;46:202–19.

Article   PubMed   Google Scholar  

Hardi R, Babocsay G, Tappe D, Sulyok M, Bodó I, Rózsa L. Armillifer-infected snakes sold at Congolese Bushmeat Markets Represent an emerging zoonotic threat. EcoHealth. 2017;14:743–9.

Steve A-M, Ahidjo A, Placide M-K, et al. High prevalences and a wide genetic diversity of Simian Retroviruses in non-human Primate Bushmeat in Rural areas of the Democratic Republic of Congo. EcoHealth. 2017;14:100–14.

Weiss S, Nowak K, Fahr J, Wibbelt G, Mombouli J-V, Parra H-J, Wolfe ND, Schneider BS, Leendertz FH. Henipavirus-related sequences in Fruit Bat Bushmeat, Republic of Congo. Emerg Infect Dis. 2012;18:1536–7.

Aguirre AA, Catherina R, Frye H, Shelley L. Illicit Wildlife Trade, Wet Markets, and COVID-19: preventing future pandemics. World Med Health Policy. 2020;12:256–65.

Nadimpalli ML, Pickering AJ. A call for global monitoring of WASH in wet markets. Lancet Planet Health. 2020;4:e439–40.

Viliani F, Edelstein M, Buckley E, Llamas A, Dar O. Mining and emerging infectious Diseases: results of the Infectious Disease Risk Assessment and Management (IDRAM) initiative pilot. The Extractive Industries and Society. 2017;4:251–9.

Wegner GI, Murray KA, Springmann M, Muller A, Sokolow SH, Saylors K, Morens DM. Averting wildlife-borne Infectious Disease epidemics requires a focus on socio-ecological drivers and a redesign of the global food system. eClinicalMedicine. 2022. https://doi.org/10.1016/j.eclinm.2022.101386 .

Daszak P. Anatomy of a pandemic. The Lancet. 2012;380:1883–4.

Joint Tripartite (FAO, OIE, WHO) and UNEP Statement. Tripartite and UNEP support OHHLEP’s definition of one health. ” OIE - World Organisation for Animal Health; 2021.

(2022) One Health Joint Plan of Action, 2022–2026. https://doi.org/10.4060/cc2289en .

Baum SE, Machalaba C, Daszak P, Salerno RH, Karesh WB. Evaluating one health: are we demonstrating effectiveness? One Health. 2017;3:5–10.

Halton K, Sarna M, Barnett A, Leonardo L, Graves N. A systematic review of community-based interventions for emerging zoonotic infectious Diseases in Southeast Asia. JBI Database System Rev Implement Rep. 2013;11:1–235.

Article   PubMed Central   Google Scholar  

Meyer A, Holt HR, Selby R, Guitian J. Past and Ongoing Tsetse and Animal Trypanosomiasis Control Operations in five African countries: a systematic review. PLoS Negl Trop Dis. 2016;10:e0005247.

Howlett M, Cashore B. Conceptualizing Public Policy. In: Engeli I, Allison CR, editors. Comparative Policy studies: conceptual and methodological challenges. London: Palgrave Macmillan UK; 2014. pp. 17–33.

Chapter   Google Scholar  

Barrett MA, Bouley TA. Need for enhanced environmental representation in the implementation of one health. EcoHealth. 2015;12:212–9.

Barbrook-Johnson P, Proctor A, Giorgi S, Phillipson J. How do policy evaluators understand complexity? Evaluation. 2020;26:315–32.

Saunders-Hastings P, Crispo JAG, Sikora L, Krewski D. Effectiveness of personal protective measures in reducing pandemic Influenza transmission: a systematic review and meta-analysis. Epidemics. 2017;20:1–20.

Bin Nafisah S, Alamery AH, Al Nafesa A, Aleid B, Brazanji NA. School closure during novel Influenza: a systematic review. J Infect Public Health. 2018;11:657–61.

Viner RM, Russell SJ, Croker H, Packer J, Ward J, Stansfield C, Mytton O, Bonell C, Booy R. School closure and management practices during coronavirus outbreaks including COVID-19: a rapid systematic review. The Lancet Child & Adolescent Health. 2020;4:397–404.

Juneau C-E, Pueyo T, Bell M, Gee G, Collazzo P, Potvin L. (2020) Evidence-Based, cost-effective interventions to suppress the COVID-19 pandemic: a systematic review. medRxiv 2020.04.20.20054726.

MacIntyre CR, Chughtai AA. Facemasks for the prevention of Infection in healthcare and community settings. BMJ. 2015;350:h694.

Smith SMS, Sonego S, Wallen GR, Waterer G, Cheng AC, Thompson P. Use of non-pharmaceutical interventions to reduce the transmission of Influenza in adults: a systematic review. Respirology. 2015;20:896–903.

Jefferson T, Del Mar CB, Dooley L et al. (2011) Physical interventions to interrupt or reduce the spread of respiratory viruses. Cochrane Database Syst Rev CD006207.

Astbury CC, Lee KM, Aguiar R, et al. Policies to prevent zoonotic spillover: protocol for a systematic scoping review of evaluative evidence. BMJ Open. 2022;12:e058437.

Tricco AC, Lillie E, Zarin W, et al. PRISMA Extension for scoping reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169:467–73.

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32.

Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5:69.

Colquhoun HL, Levac D, O’Brien KK, Straus S, Tricco AC, Perrier L, Kastner M, Moher D. Scoping reviews: time for clarity in definition, methods, and reporting. J Clin Epidemiol. 2014;67:1291–4.

Zhou X, Wang Y, Liu H, Guo F, Doi SA, Smith C, Clements ACA, Edwards J, Huang B, Soares Magalhães RJ. Effectiveness of Market-Level Biosecurity at reducing exposure of Poultry and humans to Avian Influenza: a systematic review and Meta-analysis. J Infect Dis. 2018;218:1861–75.

Conan A, Goutard FL, Sorn S, Vong S. Biosecurity measures for backyard poultry in developing countries: a systematic review. BMC Vet Res. 2012;8:240.

Youssef DM, Wieland B, Knight GM, Lines J, Naylor NR. The effectiveness of biosecurity interventions in reducing the transmission of bacteria from livestock to humans at the farm level: a systematic literature review. Zoonoses Public Health. 2021;68:549–62.

Shi N, Huang J, Zhang X, Bao C, Yue N, Wang Q, Cui T, Zheng M, Huo X, Jin H. Interventions in live poultry markets for the Control of Avian Influenza: a systematic review and Meta-analysis. J Infect Dis. 2020;221:553–60.

Cupertino MC, Resende MB, Mayer NA, Carvalho LM, Siqueira-Batista R. Emerging and re-emerging human infectious Diseases: a systematic review of the role of wild animals with a focus on public health impact. Asian Pac J Trop Med. 2020;13:99.

Clifford Astbury C, Demeshko A, McLeod R, Wiktorowicz M, Gallo Caijao E, Cullerton K, Lee KM, Viens AM, Penney TL. (2023) Governance of the wildlife trade and prevention of emerging zoonoses: a mixed methods network analysis of global organisations. [In preparation].

Covidence - Better. systematic review management. https://www.covidence.org/home . Accessed 17 Jul 2020.

Effective Public Health Practice Project. (2009) Quality Assessment Tool for Quantitative Studies. 4.

Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.

Clifford Astbury C, McGill E, Egan M, Penney TL. Systems thinking and complexity science methods and the policy process in non-communicable Disease prevention: a systematic scoping review protocol. BMJ Open. 2021;11:e049878.

Dixon-Woods M, Cavers D, Agarwal S, et al. Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006;6:35.

Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol. 2006;3:77–101.

(2021) Dedoose Version 8.3.47, web application for managing, analyzing, and presenting qualitative and mixed method research data.

Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

Park H, Chun MS, Joo Y. Traumatic stress of frontline workers in culling livestock animals in South Korea. Animals. 2020;10:1–11.

Programme UNE. Effectiveness of policy interventions relating to the illegal and unsustainable. Wildlife Trade - Policy Brief; 2019.

Cito F, Narcisi V, Danzetta ML, Iannetti S, Sabatino DD, Bruno R, Carvelli A, Atzeni M, Sauro F, Calistri P. Analysis of Surveillance systems in Place in European Mediterranean Countries for West Nile Virus (WNV) and Rift Valley Fever (RVF). Transbound Emerg Dis. 2013;60:40–4.

Schwind JS, Goldstein T, Thomas K, Mazet JA, Smith WA, PREDICT Consortium. Capacity building efforts and perceptions for wildlife surveillance to detect zoonotic pathogens: comparing stakeholder perspectives. BMC Public Health. 2014;14:684.

Reisen WK, Kramer VL, Barker CM. CALIFORNIA STATE MOSQUITO-BORNE VIRUS SURVEILLANCE AND RESPONSE PLAN: A RETROSPECTIVE EVALUATION USING CONDITIONAL SIMULATIONS *. Am J Trop Med Hyg. 2003;68:508–18.

Smith GC, Cheeseman CL. A mathematical model for the control of Diseases in wildlife populations: culling, vaccination and fertility control. Ecol Model. 2002;150:45–53.

Brennan A, Cross PC, Portacci K, Scurlock BM, Edwards WH. Shifting brucellosis risk in livestock coincides with spreading seroprevalence in elk. PLoS ONE. 2017;12:e0178780.

Wu T, Perrings C, Shang C, Collins JP, Daszak P, Kinzig A, Minteer BA. Protection of wetlands as a strategy for reducing the spread of avian Influenza from migratory waterfowl. Ambio. 2020;49:939–49.

Basinski AJ, Nuismer SL, Remien CH. A little goes a long way: weak vaccine transmission facilitates oral vaccination campaigns against zoonotic pathogens. PLoS Negl Trop Dis. 2019;13:e0007251.

Selhorst T. (1999) An evaluation of the efficiency of rabies control strategies in fox (Vulpes 6ulpes) populations using a computer simulation program. Ecol Model 12.

Shwiff SA, Sterner RT, Hale R, Jay MT, Sun B, Slate D. Benefit cost scenarios of potential oral rabies vaccination for Skunks in California. J Wildl Dis. 2009;45:227–33.

García-Díaz P, Ross JV, Woolnough AP, Cassey P. Managing the risk of wildlife Disease introduction: pathway‐level biosecurity for preventing the introduction of alien ranaviruses. J Appl Ecol. 2017;54:234–41.

Hassim A, Dekker EH, Byaruhanga C, Reardon T, van Heerden H. A retrospective study of anthrax on the Ghaap Plateau, Northern Cape province of South Africa, with special reference to the 2007–2008 outbreaks. Onderstepoort J Vet Res. 2017;84:a1414.

Knight-Jones TJD, Gibbens J, Wooldridge M, Staerk KDC. Assessment of Farm-Level Biosecurity Measures after an outbreak of Avian Influenza in the United Kingdom. Transbound Emerg Dis. 2011;58:69–75.

Article   CAS   PubMed   Google Scholar  

Karabozhilova I, Wieland B, Alonso S, Salonen L, Häsler B. Backyard chicken keeping in the Greater London Urban Area: welfare status, biosecurity and Disease control issues. Br Poult Sci. 2012;53:421–30.

Manyweathers J, Field H, Jordan D, Longnecker N, Agho K, Smith C, Taylor M. Risk mitigation of emerging zoonoses: Hendra Virus and Non-vaccinating Horse Owners. Transbound Emerg Dis. 2017;64:1898–911.

Kung N, McLaughlin A, Taylor M, Moloney B, Wright T, Field H. Hendra virus and horse owners - risk perception and management. PLoS ONE. 2013. https://doi.org/10.1371/journal.pone.0080897 .

Rasouli J, Holakoui K, Forouzanfar MH, Salari S, Bahoner, Rashidian A. Cost effectiveness of livestock vaccination for brucellosis in West-Azerbayjan province. Urmia Med J. 2009;20:Pe13–En77.

El Masry I, Rijks J, Peyre M, Taylor N, Lubroth J, Jobre Y. Modelling Influenza A H5N1 vaccination strategy scenarios in the household poultry sector in Egypt. Trop Anim Health Prod. 2014;46:57–63.

Mroz C, Gwida M, El-Ashker M, Ziegler U, Homeier-Bachmann T, Eiden M, Groschup MH. Rift valley Fever virus Infections in Egyptian cattle and their prevention. Transbound Emerg Dis. 2017;64:2049–58.

Pinsent A, Pepin KM, Zhu H, Guan Y, White MT, Riley S. (2017) The persistence of multiple strains of avian influenza in live bird markets. Proceedings of the Royal Society B: Biological Sciences 284:20170715.

Abbas B, Yousif MA, Nur HM. Animal health constraints to livestock exports from the Horn of Africa: -EN- -FR- restrictions sanitaires imposées aux exportations de bétail à partir de la corne de l’Afrique -ES- limitaciones zoosanitarias a las exportaciones de ganado desde El Cuerno De África. Rev Sci Tech OIE. 2014;33:711–21.

Naletoski I, Kirandziski T, Mitrov D, Krstevski K, Dzadzovski I, Acevski S. Gaps in brucellosis eradication campaign in Sheep and goats in Republic of Macedonia: lessons learned. Croat Med J. 2010;51:351–6.

Weaver JT, Malladi S, Bonney PJ, Patyk KA, Bergeron JG, Middleton JL, Alexander CY, Goldsmith TJ, Halvorson DA. A Simulation-based evaluation of Premovement active surveillance Protocol options for the Managed Movement of Turkeys to Slaughter during an outbreak of highly pathogenic avian Influenza in the United States. Avian Dis. 2016;60:132–45.

Andronico A, Courcoul A, Bronner A, Scoizec A, Lebouquin-Leneveu S, Guinat C, Paul MC, Durand B, Cauchemez S. Highly pathogenic avian Influenza H5N8 in south-west France 2016–2017: a modeling study of control strategies. Epidemics. 2019;28:100340.

Backer JA, van Roermund HJW, Fischer EAJ, van Asseldonk MAPM, Bergevoet RHM. Controlling highly pathogenic avian Influenza outbreaks: an epidemiological and economic model analysis. Prev Vet Med. 2015;121:142–50.

Backer JA, Hagenaars TJ, van Roermund HJW, de Jong MCM. Modelling the effectiveness and risks of vaccination strategies to control classical swine Fever epidemics. J R Soc Interface. 2009;6:849–61.

Fournie G, Guitian FJ, Mangtani P, Ghani AC. Impact of the implementation of rest days in live bird markets on the dynamics of H5N1 highly pathogenic avian Influenza. J R Soc Interface. 2011;8:1079–89.

Kung NY, Guan Y, Perkins NR, Bissett L, Ellis T, Sims L, Morris RS, Shortridge KF, Peiris JSM. The impact of a monthly Rest Day on Avian Influenza Virus isolation rates in Retail Live Poultry markets in Hong Kong. Avian Dis. 2003;47:1037–41.

Horigan V, Gale P, Adkin A, Brown I, Clark J, Kelly L. A qualitative risk assessment of cleansing and disinfection requirements after an avian Influenza outbreak in commercial poultry. Br Poult Sci. 2019;60:691–9.

Yuan J, Lau EHY, Li K, et al. Effect of live Poultry Market Closure on Avian Influenza A(H7N9) virus activity in Guangzhou, China, 2014. Emerg Infect Dis. 2015;21:1784–93.

Fournie G, Guitian J, Desvaux S, Cuong VC, Dung DH, Pfeiffer DU, Mangtani P, Ghani AC. Interventions for avian Influenza A (H5N1) risk management in live bird market networks. Proc Natl Acad Sci USA. 2013;110:9177–82.

Samaan G, Hendrawati F, Taylor T, Pitona T, Marmansari D, Rahman R, Lokuge K, Kelly PM. Application of a healthy food markets guide to two Indonesian markets to reduce transmission of avian Flu. Bull World Health Organ. 2012;90:295–300.

Huang Z, Wang J, Zuo A. Chinese farmers’ willingness to accept compensation to practice safe disposal of HPAI infected chicken. Prev Vet Med. 2017;139:67–75.

Graiver DA, Topliff CL, Kelling CL, Bartelt-Hunt SL. Survival of the avian Influenza virus (H6N2) after land disposal. Environ Sci Technol. 2009;43:4063–7.

Li Y, Wang Y, Shen C, Huang J, Kang J, Huang B, Guo F, Edwards J. Closure of live bird markets leads to the spread of H7N9 Influenza in China. PLoS ONE. 2018. https://doi.org/10.1371/journal.pone.0208884 .

Ma J, Yang N, Gu H, Bai L, Sun J, Gu S, Gu J. Effect of closure of live poultry markets in China on prevention and control of human Infection with H7N9 avian Influenza: a case study of four cities in Jiangsu Province. J Public Health Policy. 2019;40:436–47.

Chen Y, Cheng J, Xu Z, Hu W, Lu J. Live poultry market closure and avian Influenza A (H7N9) Infection in cities of China, 2013–2017: an ecological study. BMC Infect Dis. 2020. https://doi.org/10.1186/s12879-020-05091-7 .

Bonwitt J, Dawson M, Kandeh M, Ansumana R, Sahr F, Brown H, Kelly AH. Unintended consequences of the `bushmeat ban’ in West Africa during the 2013–2016 Ebola virus Disease epidemic. Soc Sci Med. 2018;200:166–73.

Brooks-Moizer F, Roberton SI, Edmunds K, Bell D. Avian Influenza H5N1 and the wild Bird Trade in Hanoi, Vietnam. Ecol Soc. 2009;14:28.

Cardador L, Tella JL, Anadon JD, Abellan P, Carrete M. The European trade ban on wild birds reduced invasion risks. Conserv Lett. 2019;12:e12631.

Guerrier G, Foster H, Metge O, Chouvin C, Tui M. Cultural contexts of swine-related Infections in Polynesia. Clin Microbiol Infect. 2013;19:595–9.

Massey PD, Polkinghorne BG, Durrheim DN, Lower T, Speare R. Blood, guts and knife cuts: reducing the risk of swine brucellosis in feral pig hunters in north-west New South Wales, Australia. Rural Remote Health. 2011;11:1793.

CAS   PubMed   Google Scholar  

Lauterbach SE, Nelson SW, Martin AM, Spurck MM, Mathys DA, Mollenkopf DF, Nolting JM, Wittum TE, Bowman AS. (2020) Adoption of recommended hand hygiene practices to limit zoonotic Disease transmission at agricultural fairs. Preventive Veterinary Medicine. https://doi.org/10.1016/j.prevetmed.2020.105116 .

Stewart RJ, Rossow J, Conover JT, et al. Do animal exhibitors support and follow recommendations to prevent transmission of variant Influenza at agricultural fairs? A survey of animal exhibitor households after a variant Influenza virus outbreak in Michigan. Zoonoses Public Health. 2018;65:195–201.

Lin X, Zhang D, Wang X, Huang Y, Du Z, Zou Y, Lu J, Hao Y. Attitudes of consumers and live-poultry workers to central slaughtering in controlling H7N9: a cross-sectional study. BMC Public Health. 2017;17:517.

Huot C, De Serres G, Duval B, Maranda-Aubut R, Ouakki M, Skowronski DM. The cost of preventing rabies at any cost: post-exposure prophylaxis for occult bat contact. Vaccine. 2008;26:4446–50.

De Serres G, Skowronski DM, Mimault P, Ouakki M, Maranda-Aubut R, Duval B. Bats in the bedroom, bats in the Belfry: reanalysis of the rationale for rabies postexposure Prophylaxis. Clin Infect Dis. 2009;48:1493–9.

Vivancos R, Showell D, Keeble B, Goh S, Kroese M, Lipp A, Battersby J. Vaccination of Poultry workers: delivery and uptake of Seasonal Influenza immunization. Zoonoses Public Health. 2011;58:126–30.

Okello AL, Thomas LF. Human taeniasis: current insights into prevention and management strategies in endemic countries. RISK MANAG HEALTHC POLICY. 2017;10:107–16.

Mendez D, Buttner P, Speare R. Hendra virus in Queensland, Australia, during the winter of 2011: veterinarians on the path to better management strategies. Prev Vet Med. 2014;117:40–51.

Häsler B, Howe KS, Hauser R, Stärk KDC. A qualitative approach to measure the effectiveness of active avian Influenza virus surveillance with respect to its cost: a case study from Switzerland. Prev Vet Med. 2012;105:209–22.

Brinkley C, Kingsley JS, Mench J. A Method for Guarding Animal Welfare and Public Health: tracking the rise of Backyard Poultry ordinances. J Community Health. 2018;43:639–46.

Turkson PK, Okike I. Assessment of practices, capacities and incentives of poultry chain actors in implementation of highly pathogenic avian Influenza mitigation measures in Ghana. Vet Med Sci. 2016;2:23–35.

Akunzule AN, Koney EBM, Tiongco M. Economic impact assessment of highly pathogenic avian Influenza on the poultry industry in Ghana. Worlds Poult Sci J. 2009;65:517–27.

Hunter C, Birden HH, Toribio J-A, Booy R, Abdurrahman M, Ambarawati AIGAA, Adiputra N. (2014) Community preparedness for highly pathogenic avian Influenza on Bali and Lombok, Indonesia. Rural Remote Health 14.

Tustin J, Laberge K, Michel P, et al. A National Epidemic of Campylobacteriosis in Iceland, lessons learned. Zoonoses Public Health. 2011;58:440–7.

Oladokun AT, Meseko CA, Ighodalo E, John B. Ekong PS Effect of intervention on the control of highly pathogenic avian Influenza in Nigeria. 8.

Manyweathers J, Field H, Longnecker N, Agho K, Smith C, Taylor M. Why won’t they just vaccinate? Horse owner risk perception and uptake of the Hendra virus vaccine. BMC Vet Res. 2017;13:103.

Zhu G, Kang M, Wei X, Tang T, Liu T, Xiao J, Song T, Ma W. Different intervention strategies toward live poultry markets against avian Influenza A (H7N9) virus: model-based assessment. Environ Res. 2020. https://doi.org/10.1016/j.envres.2020.110465 .

Chowell G, Simonsen L, Towers S, Miller MA, Viboud C. Transmission potential of Influenza A/H7N9, February to May 2013, China. BMC Med. 2013;11:214.

Bodenham RF, Mtui-Malamsha N, Gatei W, et al. Multisectoral cost analysis of a human and livestock anthrax outbreak in Songwe Region, Tanzania (December 2018–January 2019), using a novel Outbreak Costing Tool. One Health. 2021;13:100259.

Lewis N, Dorjee S, Dube C, VanLeeuwen J, Sanchez J. Assessment of Effectiveness of Control Strategies against Simulated Outbreaks of Highly Pathogenic Avian Influenza in Ontario, Canada. Transbound Emerg Dis. 2017;64:938–50.

Anderson A, Shwiff S, Gebhardt K, Ramírez AJ, Shwiff S, Kohler D, Lecuona L. Economic evaluation of Vampire Bat ( Desmodus rotundus) rabies Prevention in Mexico. Transbound Emerg Dis. 2014;61:140–6.

Walker PGT, Cauchemez S, Metras R, Dung DH, Pfeiffer D, Ghani AC. A bayesian Approach to quantifying the effects of Mass Poultry Vaccination upon the spatial and temporal dynamics of H5N1 in Northern Vietnam. PLoS Comput Biol. 2010;6:e1000683.

PREDICT Project. In: PREDICT Project. https://p2.predict.global. Accessed 9 Sep 2022.

Loh EH, Zambrana-Torrelio C, Olival KJ, Bogich TL, Johnson CK, Mazet JAK, Karesh W, Daszak P. Targeting transmission pathways for emerging zoonotic Disease Surveillance and Control. Vector-Borne and Zoonotic Diseases. 2015;15:432–7.

Bordier M, Uea-Anuwong T, Binot A, Hendrikx P, Goutard FL. Characteristics of one health surveillance systems: a systematic literature review. Prev Vet Med. 2020;181:104560.

Worsley-Tonks KEL, Bender JB, Deem SL, et al. Strengthening global health security by improving Disease surveillance in remote rural areas of low-income and middle-income countries. The Lancet Global Health. 2022;10:e579–84.

Reaser JK, Witt A, Tabor GM, Hudson PJ, Plowright RK. Ecological countermeasures for preventing zoonotic Disease outbreaks: when ecological restoration is a human health imperative. Restor Ecol. 2021;29:e13357.

Chen HL, Lewison RL, An L, Tsai YH, Stow D, Shi L, Yang S. Assessing the effects of payments for ecosystem services programs on forest structure and species biodiversity. Biodivers Conserv. 2020;29:2123–40.

Chen Y, Marino J, Tao Q, Sullivan CD, Shi K, Macdonald DW. Predicting hotspots of human-elephant conflict to inform mitigation strategies in Xishuangbanna, Southwest China. PLoS ONE. 2016. https://doi.org/10.1371/journal.pone.0162035 .

Whitfort A. COVID-19 and Wildlife Farming in China: legislating to Protect Wild Animal Health and Welfare in the wake of a global pandemic. J Environ Law. 2021;33:57–84.

Download references

Acknowledgements

Not applicable.

CCA, JC and TLP acknowledge internal research support from York University. MW and CCA acknowledge internal research support from the Dahdaleh Institute for Global Health Research. KML acknowledges funding from the Canadian Institutes of Health Research through a Health System Impact Fellowship. AY is funded by the BBSRC through the Mandala project (grant number BB/V004832/1). AMV acknowledges support from York University through a York Research Chair in Population Health Ethics & Law. This review was undertaken as part of a project funded by the Canadian Institutes of Health Research, Grant Reference Number VR5-172686. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Author information

Authors and affiliations.

School of Global Health, York University, Toronto, ON, Canada

Chloe Clifford Astbury, Kirsten M. Lee, Ryan Mcleod, Asma Atique, Janielle Clarke, Anastassia Demeshko, A. M. Viens, Mary Wiktorowicz & Tarra L. Penney

Dahdaleh Institute for Global Health Research, York University, Toronto, ON, Canada

Chloe Clifford Astbury, Kirsten M. Lee, Raphael Aguiar, Mary Wiktorowicz & Tarra L. Penney

Global Strategy Lab, York University, Toronto, ON, Canada

Chloe Clifford Astbury, A. M. Viens & Tarra L. Penney

Applied Microbiology for Health and Environment Research Group, College of Arts and Sciences, University of the Philippines Manila, Manila, Philippines

Marilen Balolong & Kathleen Chelsea Togño

School of Epidemiology and Public Health, University of Ottawa, Ottawa, ON, Canada

Ronald Labonté & Arne Ruckert

School of Health Policy and Management, York University, Toronto, ON, Canada

Priyanka Sibal

School of Public Health, University of Kinshasa, Kinshasa, Democratic Republic of Congo

Marc K. Yambayamba

Department of Public Health, Environments and Society, London School of Hygiene & Tropical Medicine, London, UK

You can also search for this author in PubMed   Google Scholar

Contributions

Conception and design: CCA, KLM and TLP. Acquisition of data: CCA, KLM and AY. Analysis and interpretation of data: CCA, KML, RM, JC, AD and PS. Drafting of the manuscript: CCA and RM. Critical revision of the manuscript for important intellectual content: KML, RA, AA, MB, JC, AD, RL, AR, PS, KCT, AMV, MW, MKY, AY and TLP. Obtaining funding: TLP and MW.

Corresponding author

Correspondence to Tarra L. Penney .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests. RL is a co-editor-in-chief of Globalization and Health.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, supplementary material 3, supplementary material 4, supplementary material 5, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Clifford Astbury, C., Lee, K.M., Mcleod, R. et al. Policies to prevent zoonotic spillover: a systematic scoping review of evaluative evidence. Global Health 19 , 82 (2023). https://doi.org/10.1186/s12992-023-00986-x

Download citation

Received : 05 May 2023

Accepted : 01 November 2023

Published : 08 November 2023

DOI : https://doi.org/10.1186/s12992-023-00986-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Zoonotic spillover
• Public policy
• Emerging zoonoses
• Deep prevention

Globalization and Health

ISSN: 1744-8603

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• For authors
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Online First
• Interventions to suppress puberty in adolescents experiencing gender dysphoria or incongruence: a systematic review
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0001-5898-0900 Jo Taylor ,
• Alex Mitchell ,
• Ruth Hall ,
• Claire Heathcote ,
• Trilby Langton ,
• Lorna Fraser ,
• http://orcid.org/0000-0002-0415-3536 Catherine Elizabeth Hewitt
• Department of Health Sciences , University of York , York , UK
• Correspondence to Dr Jo Taylor, Health Sciences, University of York, York, North Yorkshire, UK; dohs-gender-research{at}york.ac.uk

Background Treatment to suppress or lessen effects of puberty are outlined in clinical guidelines for adolescents experiencing gender dysphoria/incongruence. Robust evidence concerning risks and benefits is lacking and there is a need to aggregate evidence as new studies are published.

Aim To identify and synthesise studies assessing the outcomes of puberty suppression in adolescents experiencing gender dysphoria/incongruence.

Methods A systematic review and narrative synthesis. Database searches (Medline, Embase, CINAHL, PsycINFO, Web of Science) were performed in April 2022, with results assessed independently by two reviewers. An adapted version of the Newcastle-Ottawa Scale for cohort studies was used to appraise study quality. Only moderate-quality and high-quality studies were synthesised. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting guidelines were used.

Results 11 cohort, 8 cross-sectional and 31 pre-post studies were included (n=50). One cross-sectional study was high quality, 25 studies were moderate quality (including 5 cohort studies) and 24 were low quality. Synthesis of moderate-quality and high-quality studies showed consistent evidence demonstrating efficacy for suppressing puberty. Height increased in multiple studies, although not in line with expected growth. Multiple studies reported reductions in bone density during treatment. Limited and/or inconsistent evidence was found in relation to gender dysphoria, psychological and psychosocial health, body satisfaction, cardiometabolic risk, cognitive development and fertility.

Conclusions There is a lack of high-quality research assessing puberty suppression in adolescents experiencing gender dysphoria/incongruence. No conclusions can be drawn about the impact on gender dysphoria, mental and psychosocial health or cognitive development. Bone health and height may be compromised during treatment. More recent studies published since April 2022 until January 2024 also support the conclusions of this review.

PROSPERO registration number CRD42021289659.

Data availability statement

Data sharing not applicable as no datasets generated and/or analysed for this study.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:  http://creativecommons.org/licenses/by-nc/4.0/ .

https://doi.org/10.1136/archdischild-2023-326669

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

WHAT IS ALREADY KNOWN ON THIS TOPIC

Increasing numbers of children and adolescents experiencing gender dysphoria/incongruence are being referred to specialist gender services.

National and international guidelines have changed over time and outline that medications to suppress puberty can be considered for adolescents experiencing gender dysphoria/incongruence.

Several systematic reviews report a limited evidence base for these treatments, and uncertainty about the benefits, risks and long-term effects.

WHAT THIS STUDY ADDS

No high-quality studies were identified that used an appropriate study design to assess the outcomes of puberty suppression in adolescents experiencing gender dysphoria/incongruence.

There is insufficient and/or inconsistent evidence about the effects of puberty suppression on gender-related outcomes, mental and psychosocial health, cognitive development, cardiometabolic risk, and fertility.

There is consistent moderate-quality evidence, although from mainly pre-post studies, that bone density and height may be compromised during treatment.

HOW THIS STUDY MIGHT AFFECT RESEARCH, POLICY OR PRACTICE

There is a lack of high-quality evidence to support the use of puberty suppression in adolescents experiencing gender dysphoria/incongruence, and large well-designed research is needed.

Introduction

Over the last 10-15 years, increasing numbers of children and adolescents experiencing gender dysphoria/incongruence are being referred to specialist paediatric gender services. 1 2

Gender dysphoria/incongruence in childhood is associated with high rates of co-occurring mental health and psychosocial difficulties, which can affect health and well-being. 3 Clinical guidelines recommend psychosocial care to alleviate gender-related distress and any co-occurring difficulties. For pubertal adolescents, medications to suppress or lessen effects of puberty are also outlined. Gonadotropin-releasing hormone analogues (GnRH-a) are used as first-line treatment, although other drugs with anti-androgenic properties including progestins and spironolactone are used in this population. 4 5 The effects differ depending on whether they are initiated in early puberty or mid-puberty, as well as the type of intervention used, with GnRH-a suppressing puberty when started early or suspending further progression when initiated in mid-puberty, and anti-androgens instead blocking specific downstream effects of sex hormones. 4

Rationales for puberty suppression in the Dutch treatment protocol, which has informed practice internationally, were to alleviate worsening gender dysphoria, allow time for gender exploration, and pause development of secondary sex characteristics to make passing in the desired gender role easier. 6 Practice guidelines propose other indications for puberty suppression, including allowing time and/or capacity for decision-making about masculinising or feminising hormone interventions, and improving quality of life. 4 7 8

Criteria in early treatment protocols for puberty suppression specified adolescents be at least age 12 years, at Tanner stage 2 in puberty, experienced gender dysphoria in childhood which persisted and intensified during puberty and met criteria for diagnosis of gender dysphoria. 6 It was also expected that any psychosocial difficulties that could interfere with treatment were managed. 6 The World Professional Association for Transgender Health standards of care 4 and other practice guidelines 5 8 9 have broadened these criteria, for example, removing minimum age. However, other recent guidelines have taken a more cautious approach and restricted inclusion criteria in response to uncertainties in the evidence base. 7 10

Systematic reviews have consistently found mainly low-quality evidence, limited data on key outcomes or long-term follow-up. 11–16 These reviews report that while puberty suppression may offer some benefit, there are concerns about the impact on bone health, and uncertainty regarding cognitive development, psychosocial outcomes and cardiometabolic health. They conclude there is insufficient evidence to support clinical recommendations.

The proliferation of research in this area and lack of evidence to support practice means there is an ongoing need to aggregate evidence. This systematic review aims to synthesise evidence published to April 2022 that reports outcomes of puberty suppression in adolescents experiencing gender dysphoria/incongruence.

The review forms part of a linked series examining the epidemiology, care pathways, outcomes and experiences for children and adolescents experiencing gender dysphoria/incongruence and is reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. 17 The protocol was registered on PROSPERO (CRD42021289659. 18

Search strategy

A single search strategy was used to identify studies comprising two combined concepts: ‘children’, which included all terms for children and adolescents and ‘gender dysphoria’, which included associated terms such as gender-related distress and gender incongruence, and gender identity terms including transgender, gender diverse and non-binary.

MEDLINE ( online supplemental table S1 ), EMBASE and PsycINFO through OVID, CINAHL Complete through EBSCO, and Web of Science (Social Science Citation Index) were searched (13–23 May 2021 and updated on 27 April 2022).

Supplemental material

Reference lists of included studies and relevant systematic reviews were assessed for inclusion. 11–16 19 20

Inclusion criteria

The review included published research that reported outcomes of interventions used to suppress puberty for children and/or adolescents experiencing gender dysphoria/incongruence ( table 1 ).

• View inline

Inclusion and exclusion criteria

Selection process

The results of database and other searches were uploaded to Covidence 21 and screened independently by two reviewers. Full texts of potentially relevant articles were retrieved and reviewed against inclusion criteria by two reviewers independently. Disagreements were resolved through discussion and inclusion of a third reviewer.

Data extraction

Data on study characteristics, methods and reported outcomes were extracted into prepiloted data extraction templates by one reviewer and second-checked by another.

Study quality

Critical appraisal was undertaken by two reviewers independently, with consensus reached through discussion and involvement of a third reviewer where necessary.

Quality was assessed using a modified version ( online supplemental file 1 ) of the Newcastle-Ottawa Scale for cohort studies, a validated scale of eight items covering three domains: selection, comparability and outcome. 22 Scale modification included not scoring certain question(s) for cross-sectional and single-group designs, or particular outcomes; specification of key confounders to assess comparability of cohorts; guidance regarding sufficiency of follow-up and use of numerical scores for items and overall (maximum score 9 for cohorts, 8 for pre-post and cross-sectional studies with comparator). Total scores were calculated as percentages to account for different total scores (≤50% low quality, >50%–75% moderate quality, >75% high quality).

Narrative synthesis methods were used because of heterogeneity in study design, intervention, comparator, outcome and measurement. Due to high risk of bias in low-quality studies, these were excluded from the synthesis.

When synthesising results by outcome domains, care was taken to differentiate between different study designs, comparators and interventions. Where possible, potential differences in effects by birth-registered sex, treatment duration or treatment in early puberty versus late puberty were examined.

The database search yielded 28 147 records, 3181 of which were identified as potentially relevant for the linked systematic reviews and full texts reviewed. From these, 50 studies met inclusion criteria for this review ( figure 1 ).

• Download figure
• Open in new tab
• Download powerpoint

Study flow diagram.

Study characteristics

Studies were published from 2006 to 2022 with the majority published in 2020–2022 (n=29). Studies were conducted in the Netherlands (n=17), 23–39 the US (n=15), 40–54 the UK (n=6), 55–60 Canada (n=4), 61–64 three in Belgium 65–67 and Israel 68–70 and one in Brazil 71 and Germany 72 ( online supplemental table S2 ).

The 50 studies included 11 cohorts comparing adolescents experiencing gender dysphoria/incongruence receiving puberty suppression with a comparator, 35 39–42 45 49 50 52 56 72 8 cross-sectional with a comparator 23 33 37 47 51 53 60 71 and 31 pre-post single group studies. 24–32 34 36 38 43 44 46 48 54 55 57–59 61–70 More than half of studies (n=29) used retrospective chart review.

All but 4 studies selected adolescents experiencing gender dysphoria/incongruence from specialist gender or endocrinology services: 43 from single services (in Belgium, Israel, the Netherlands and the UK these were large regional or national services) and 3 from multiple US services. 48–50 The other four included three US studies (national survey recruiting via community settings, 53 clinical and community settings, 51 US Military Healthcare Data Repository 54 ) and a study from Brazil recruiting via Facebook. 71

Overall, studies included 10 673 participants: 9404 were adolescents experiencing gender dysphoria/incongruence (4702 received puberty suppression, 4702 did not) and 1269 other comparators. Comparator groups included adolescents or adults experiencing gender dysphoria/incongruence who had not received puberty suppression, 35 39 40 42 51–53 60 71 72 untreated adolescents not experiencing gender dysphoria/incongruence, 36 47 50 both of these comparators 23 33 37 56 or adolescents receiving treatment for a different medical reason. 41 45 49

Most studies (n=39) assessed GnRH-a. In one, some participants received GnRH-a and some (birth-registered males) spironolactone. 62 In another, GnRH-a or progestins/anti-androgens were used but numbers taking each were not reported. 40 Among the other 11 studies, 5 assessed effects of progestins (cyproterone acetate, 66 67 lynestrenol, 65 66 medroxyprogesterone 44 and levonorgestrel-releasing intrauterine system 41 ) as alternatives to GnRH-a, 41 44 65–67 1 assessed bicalutamide 46 and 5 did not specify. 43 52–54 71

Of the 50 studies, 29 reported outcomes for feminising or masculinising hormones as well as for puberty suppression, either by including a mixed sample of those receiving the two different interventions or by assessing those who progressed to hormones following puberty suppression.

The most frequently measured outcomes were puberty suppression (n=30) and physical health outcomes (n=27) ( figure 2 , online supplemental table S3 ). Gender-related outcomes and body image were measured in five and four studies, respectively. Psychological health was measured in 13 studies, psychosocial in 9 studies and cognitive/neurodevelopmental outcomes in 3 studies. Side effects were reported in six, bone health in nine, and one study measured fertility.

Outcome categories by study quality and design.

One cross-sectional study was rated high quality, 37 25 moderate quality 23 24 29–32 34–36 39 48–51 54–59 64 65 67–69 and 24 low quality. 25–28 33 38 40–47 52 53 60–63 66 70–72 Of the 11 cohort studies, which were the only studies to include a comparator and assess outcomes over time, only 5 were rated moderate quality ( figure 2 , online supplemental table S4 ). 35 39 49 50 56

In most studies, there were concerns about sample representativeness due to single site recruitment, inclusion of a selected group and/or poor reporting of the eligible population. In studies including a comparator, most did not report or control for key differences between groups and only four used matched controls. 23 33 41 47 Most studies presented results for birth-registered males and females separately or controlled for this. Few studies controlled for age or Tanner stage or co-interventions that could influence outcomes.

Overall, studies used appropriate methods to ascertain exposure and assess outcomes. Adequacy of follow-up was evident in 18 studies, with multiple studies not reporting treatment duration, including participants receiving treatment at baseline, and not aligning follow-up with treatment initiation. Missing data at follow-up/analysis or poor reporting of this affected many studies.

Four studies did not report separate outcome data for adolescents receiving puberty suppression or masculinising/feminising hormones. 39 54 60 71 Two of these were of moderate quality and not included in the synthesis, 39 54 one of which was the only study to assess fertility outcomes. 39 One moderate-quality study assessed amplitude of click-evoked otoacoustic emissions. 23 This was excluded from the synthesis on the basis of not being clinically relevant.

Synthesis of outcomes

Gender dysphoria and body satisfaction.

Two pre-post studies measured gender dysphoria and body satisfaction (with primary and secondary sex or neutral body characteristics) and reported no change before and after receiving treatment 24 55 ( table 2 ).

Gender-related, body image, psychological, psychosocial, and cognitive/neurodevelopmental outcomes

Psychological health

One cross-sectional 37 and two pre-post studies 24 55 measured symptoms of depression (n=1), anxiety (n=1), anger (n=1), internalising and externalising symptoms (n=3), suicide and/or self-harm (n=2) and psychological functioning (n=2).

Three studies assessed internalising and externalising symptoms with one reporting improvements in both (pre-post 24 ), one improvement in internalising but not externalising symptoms when compared with adolescents under assessment by a gender service (cross-sectional 37 ) and one observed no change in either (pre-post). 55

For other psychological outcomes, there was either a single study, or two studies showing inconsistent results, with studies reporting either a small to moderate significant improvement or no change ( table 2 ).

Psychosocial outcomes

One cohort 56 and two pre-post 24 55 studies measured psychosocial functioning, one pre-post study assessed quality of life 55 and one cross-sectional study measured peer-relations ( table 2 ). 37

For psychosocial functioning, both pre-post studies reported no clinically significant change at follow-up. 24 55 The cohort study compared adolescents who were not immediately eligible for puberty suppression and received psychological support only, and adolescents who additionally received GnRH-a after 6 months. 56 Improvements were seen in both groups after 6 months of psychological support. This improvement was maintained over time for those receiving psychological support only. For those receiving GnRH-a, further improvements were observed at 12 and 18 months. At 18 months, psychosocial functioning in this group was considerably higher than in those still waiting for puberty suppression, and similar to adolescents not experiencing gender dysphoria/incongruence. However, there were considerably fewer participants included at final follow-up.

There was no change in quality of life pre-post, 55 and treated adolescents had better peer-relations compared with adolescents under assessment at a gender service but poorer peer-relations than adolescents not experiencing gender dysphoria/incongruence. 37

Cognitive/neurodevelopmental outcomes

One cross-sectional study measured executive functioning and found no difference between adolescents who were treated for <1 year compared with those not treated, but worse executive functioning in those treated for >1 year compared with those not treated. 51 A pre-post study found no differences in features typically associated with autism spectrum condition after treatment ( table 2 ). 59

Physical health outcomes

Bone health.

Five studies found decreases in bone mineral apparent density and z-scores pre-post treatment; however, absolute measures generally remained stable or increased/decreased slightly. 29 32 34 55 58 Results were similar across birth-registered males and females. 29 32 55 58 One study considered timing of treatment, and found similar decreases among those starting GnRH-a in early or late puberty ( table 3 ). 32

Physical health outcomes and side effects

Cardiometabolic health

Twelve pre-post studies measured body mass index (BMI), and in 10 studies there was no evidence of a clinically significant change in BMI and/or BMI SD score. 29 30 32 34 55 57 65 67–69 In one study, BMI increased for birth-registered males but not females. 58 Another study found BMI increased for birth-registered females who started GnRH-a in early puberty or mid-puberty, and birth-registered males in early puberty. 36

Three studies assessed cholesterol markers, one after GnRH-a (no changes), 34 one after cyproterone acetate (decrease in high-density lipoprotein (HDL) and triglycerides) 67 and one after lynestrenol (decrease in HDL, increase in low-density lipoprotein). 65 Three studies assessing GnRH-a reported blood pressure: two found similar systolic and diastolic blood pressure before and after treatment, 34 68 and one found a non-clinically significant increase in diastolic but not systolic blood pressure. 69 Two studies measured markers of diabetes (fasting glucose, HbA1c and/or insulin) and noted no changes. 65 67

Other physiological parameters

Five pre-post studies assessed other parameters from blood tests undertaken at baseline and follow-up, 30 31 34 65 67 three in those treated with GnRH-a, 30 31 34 one lynestrenol 65 and one cyproterone acetate. 67 Measurements included haemoglobin count (n=3), haematocrit percentage (n=3), creatinine (n=4), aspartate aminotransferase (n=3), alanine aminotransferase (n=3), γ-glutamyl transferase (n=1), alkaline phosphatase (n=2), prolactin (n=2), free thyroxin (n=3), thyroid-stimulating hormone (n=3), sex hormone binding globulin (n=3), vitamin D levels (n=1), dehydroepiandrosterone sulfate (n=3) and androstenedione (n=2). For most outcomes, no changes were reported. Where there were changes, these were not consistent in direction across studies.

One pre-post study assessing GnRH-a reported QTc prolongation, 64 and found no change in mean QTc, with no participants outside normal range.

Side effects

A cohort study of GnRH-a reported side effects including mild headaches or hot flushes (~20%) and moderate/severe headaches or hot flushes, mild fatigue, mood swings, weight gain and sleep problems (<10%) ( table 3 ). 55

Two studies assessed other medications and reported headaches and hot flushes as common and an increase in acne in a sample of birth-registered females receiving lynestrenol, 65 and complaints of fatigue in birth-registered males receiving cyproterone acetate. 67

Puberty suppression

Hormone levels.

Hormone levels were reported in nine studies of GnRH-a (two cohort, 49 50 seven pre-post 30 34 36 48 55 68 69 ), two in birth-registered females, 34 69 one in birth-registered males 68 and six including both ( table 4 ). 30 36 48–50 55

Puberty suppression outcomes

Five studies reported decreases in luteinising hormone, follicle-stimulating hormone, oestradiol and testosterone after receiving GnRH-a. 30 34 48 68 69 Another study, which reported luteinising and follicle-stimulating hormones, also found decreases in both pre-post. 55 One study reported that where baseline levels were high due to puberty starting, decreases were reported in testosterone and oestradiol. 36 One cohort study reporting pre-post data found smaller decreases in luteinising hormone, follicle-stimulating hormone, oestradiol and testosterone compared with other studies; however, it included a younger population, some of who were likely prepubertal. 50 The other cohort study included a comparator of adolescents with precocious puberty and found similar decreases in luteinising hormone and oestradiol. 49

One pre-post study of lynestrenol (birth-registered females) found a decrease in luteinising hormones but not follicle-stimulating hormone, oestradiol or testosterone. 65 One study of cyproterone acetate (birth-registered males) found no changes in luteinising hormone, follicle-stimulating hormone or oestradiol, but a decrease in total testosterone. 67

Pubertal progression

Puberty development was reported in four studies (two cohort, two pre-post). 30 35 49 67 One only included birth-registered males, 67 and three included both birth-registered males and females. 30 35 49

A cohort study assessing GnRH-a reported clinical pubertal escape in 2/21 adolescents treated for gender dysphoria/incongruence, in the form of breast enlargement or testicular enlargement together with deepening of voice, compared with no children treated for precocious puberty. 49 A pre-post study reported a decrease in testicular volume in birth-registered males, but unclear results with regard to breast development in birth-registered females (most started treatment at Tanner stage 4–5). 30 A pre-post study of birth-registered males using cyproterone acetate reported decreases in facial shaving and spontaneous erections. 67

A cohort study assessed whether secondary sex characteristics differed depending on receipt or timing of GnRH-a, and whether this affected which surgical interventions/techniques were later used. 35 The study found breast size was smallest in birth-registered females who received GnRH-a in Tanner stage 2/3 and largest in untreated participants. Those treated early in puberty were less likely to require a mastectomy and when surgery was required it was less burdensome. In birth-registered males, penile length was greater in those who received GnRH-a at Tanner stage 4/5 compared with Tanner stage 2/3, and greatest in untreated participants. 35 Those who received GnRH-a early required more invasive vaginoplasty techniques than those who received it later or not at all.

Menstrual suppression

Three studies (one cohort, two pre-post) measured menstrual suppression in birth-registered females, and found full suppression at follow-up, 30 49 55 which was similar to the effect seen in those with precocious puberty in the cohort study. 49

Height/Growth

Eleven studies (1 cohort, 50 10 pre-post 29 30 32 34 36 55 57 58 65 67 ) reported height, nine after GnRH-a, 29 30 32 34 36 50 55 57 58 one lynestrenol 65 and one cyproterone acetate. 67 The cohort study found a similar height velocity between the GnRH-a group and adolescent controls. 50 Six studies reported height Z or SD score 29 30 34 55 57 67 with two studies finding no change, 34 55 two a decrease for birth-registered males but not females, 29 57 one a decrease across birth-registered males and females 30 and one a decrease in birth-registered males with cyproterone acetate. 67 Absolute measures of height generally increased slightly or remained the same. 29 30 32 34 36 58 65 67

Body composition

Two studies reported changes in body composition pre-post, 30 57 reporting a significant decrease in lean mass SD score 57 and percentage 30 in males and females. One also measured body fat percentage and reported significant increases in both groups. 30

Bone geometry

One pre-post study measured the subperiosteal width and endocortical diameter of the hip bone and found that in birth-registered males these increased in those starting GnRH-a in early puberty and mid-puberty, but only in the early puberty group for birth-registered females. 36

This systematic review identified 50 studies reporting outcomes relating to puberty suppression in adolescents experiencing gender dysphoria/incongruence. No high-quality studies using an appropriate design were identified, and only four measured gender dysphoria as an outcome. Only 5 of the 11 cohort studies, which were the only studies to compare groups over time, were rated as moderate quality. 35 40 49 50 56

There was evidence from multiple mainly pre-post studies that puberty suppression exerts its expected physiological effect, as previously demonstrated in children with precocious puberty. 73 In adolescents experiencing gender dysphoria/incongruence, puberty suppression is initiated at different stages of puberty, 74 and two studies found that the effects on secondary sex characteristics may vary depending on whether treatment is initiated in early puberty versus mid-puberty, with potentially different outcomes for birth-registered males and females. 30 35 Multiple studies also found that bone density is compromised during puberty suppression, and gains in height may lag behind that seen in other adolescents. High-quality research is needed to confirm these findings; however, these potential risks should be explained to adolescents considering puberty suppression.

These findings add to other systematic reviews in concluding there is insufficient and/or inconsistent evidence about the effects of puberty suppression on gender dysphoria, body satisfaction, psychological and psychosocial health, cognitive development, cardiometabolic risk and fertility. 11–16 Regarding psychological health, one recent systematic review 14 reported some evidence of benefit while others have not. The results in this review found no consistent evidence of benefit. Inclusion of only moderate-quality to high-quality studies may explain this difference, as 8 of the 12 studies reporting psychological outcomes were rated as low-quality.

The lack of representativeness of samples and comparability of selected control groups were key concerns across studies. Only one study attempted to compare puberty suppression with psychosocial care, which is the only other treatment offered for gender dysphoria/incongruence in childhood, and this included a small sample, limited analyses, and little detail about the intervention. 56 Other studies lacked information about any psychological care provided to participants, and in studies that included a comparator there was limited information about any differences between groups. Large, well-designed studies with appropriate comparators that enable long-term outcomes of puberty suppression to be measured are needed.

Many studies reported effects of both puberty suppressants and hormones used in later adolescence for feminisation/masculinisation. In adolescents, GnRH-a often continues during hormone treatment, 74 or for adolescents who do not receive puberty suppression, GnRH-a or other anti-androgens may be offered at initiation of hormones. 66 This makes long-term follow-up of puberty suppression difficult to assess, including any differences between the types of interventions that are offered and when these are initiated, and the few studies reporting long-term outcomes either did not control for this or reported overall effects for both interventions. Although recent studies suggest nearly all adolescents who receive puberty suppression go on to feminising/masculinising hormones, 74–76 research is still needed to assess whether suppression will have any lasting effects for those who do not. Aggregation of studies reporting proportions of adolescents who progress to hormones and reasons for discontinuation would also offer useful insights.

Included studies assessed different outcomes across various outcome domains and employed multiple different measures. Agreement about the primary aim and related core outcomes of puberty suppression in this population would help to ensure studies measure key outcomes and facilitate future aggregation of evidence. Expert consensus recommendations to guide the methods and domains for assessing the neurodevelopmental effects of puberty suppression have been developed 77 ; however, there is currently no agreement across other outcome domains.

Strengths and limitations

Strengths include a published protocol with robust search strategies, use of PRISMA guidelines and comprehensive synthesis of moderate and high quality studies. Poor reporting across studies may have resulted in moderate-quality studies being rated low-quality and excluded from synthesis. As searches were conducted up to April 2022, this review does not include more recently published studies. However, the findings are in line with previous reviews despite the inclusion of numerous additional studies. In an update of the National Institute for Health and Care Excellence evidence review of GnRH-a performed in April 2023, 78 nine additional studies were identified, two of which they felt might materially affect their conclusions. 72 74 One was already included in this review, 72 and the other examined treatment trajectories which was not an outcome of interest. 74

Of other studies that we are aware have been published since April 2022 until January 2024, very few used a cohort design or an appropriate comparator and were of a similar low quality to moderate quality as the studies summarised in this review. Of those likely to contribute new data for synthesis, five examined physical growth and development, 79–83 one cardiometabolic health 84 and one psychological health. 85 The latter, a study from the US, found that adolescents who received puberty suppression before assessment for masculinising or feminising hormones had fewer symptoms of depression, anxiety, stress and suicidal thoughts compared with those who had not received puberty suppression. A sensitivity analysis found similar results, although no difference in suicidal thoughts. 85 Adding this study would provide no further clarity about whether puberty suppression improves psychological health due to the inconsistency of results between studies, and the limited high-quality research measuring these outcomes.

Two studies from the Netherlands found that height growth and bone maturation both decelerated during GnRH-a treatment, 80 81 and a third assessing only bone health found the same. 83 A Belgian study found stable height growth in birth-registered females but deceleration in birth-registered males. 82 These studies add strength to the conclusion that bone health and adult height may be compromised during GnRH-a, although like in previous studies the participants went on to receive masculinising or feminising hormones, and therefore the long-term outcomes of puberty suppression alone were not possible to determine.

Another new study, also from the Netherlands, assessed changes in body composition. 79 This found that in both birth-registered males and females lean mass z-scores decreased during puberty suppression and fat mass z-scores increased, although the rate and duration of change differed by birth-registered sex. These changes were also found in the two studies synthesised, 30 57 but as all three included no comparator uncertainty continues about the effect of puberty suppression on body composition.

A large study of adults in the US examined whether receipt of hormone interventions during adolescence was associated with cardiometabolic-related diagnoses, and for GnRH-a found no statistically significant differences for any diagnosis. 84 However, the study uses a retrospective cross-sectional design and is the only study to have examined cardiometabolic diagnoses, so no conclusions can be drawn about these outcomes.

To our knowledge, there are no additional moderate-quality or high-quality studies that have measured psychosocial or fertility outcomes, and only a single study assessing cognitive effects which measured a different outcome (white matter microstructure) to those included in this review. 86

Conclusions

There are no high-quality studies using an appropriate study design that assess outcomes of puberty suppression in adolescents experiencing gender dysphoria/incongruence. No conclusions can be drawn about the effect on gender-related outcomes, psychological and psychosocial health, cognitive development or fertility. Bone health and height may be compromised during treatment. High-quality research and agreement on the core outcomes of puberty suppression are needed.

Ethics statements

Patient consent for publication.

Not applicable.

• Thompson L ,
• Sarovic D ,
• Wilson P , et al
• Kaltiala R ,
• Bergman H ,
• Carmichael P , et al
• Coleman E ,
• Bouman WP , et al
• Hembree WC ,
• Cohen-Kettenis PT ,
• Gooren L , et al
• de Vries ALC ,
• Cohen-Kettenis PT
• The Swedish National Board of Health and Welfare
• Telfer MM ,
• Tollit MA ,
• Pace CC , et al
• University of California San Francisco Gender Affirming Health Program
• ↵ Medical treatment methods for Dysphoria associated with variations in gender identity in minors – recommendation . 2020 . Available : https://palveluvalikoima.fi/en/recommendations#genderidentity
• National Institute for Health and Care Excellence (NICE)
• Pasternack I ,
• Söderström I ,
• Saijonkari M , et al
• Ludvigsson JF ,
• Adolfsson J ,
• Höistad M , et al
• Wilson LM ,
• Sharma R , et al
• Anderson J ,
• Williams K , et al
• McKenzie JE ,
• Bossuyt PM , et al
• Taylor J , et al
• Ramos GGF ,
• Mengai ACS ,
• Daltro CAT , et al
• Veritas Health Innovation
• O’Connell D , et al
• van Heesewijk JO ,
• Menks WM , et al
• Steensma TD ,
• Doreleijers TAH , et al
• McGuire JK ,
• Steensma TD , et al
• Delemarre-van de Waal HA ,
• de Mutsert R ,
• Wiepjes CM , et al
• van der Loos MATC , et al
• Heijboer A , et al
• Schagen SEE ,
• Delemarre-van de Waal HA , et al
• Lustenhouwer P ,
• Cohen-Kettenis PT , et al
• Wouters FM ,
• Staphorsius AS ,
• Kreukels BPC ,
• Stoffers IE ,
• de Vries MC ,
• van de Grift TC ,
• van Gelder ZJ ,
• Mullender MG , et al
• van der Loos MA ,
• Hellinga I ,
• Vlot MC , et al
• van der Miesen AIR ,
• de Vries ALC , et al
• den Heijer M , et al
• Mulder CL ,
• Meißner A , et al
• Achille C ,
• Taggart T ,
• Eaton NR , et al
• Ford N , et al
• Grimstad FW ,
• Jacobson JD
• Stewart S ,
• Preston S , et al
• Khandheria MM ,
• Mejia-Otero JD ,
• Nokoff NJ ,
• Scarbro SL ,
• Moreau KL , et al
• Olson-Kennedy J ,
• Streeter LH ,
• Garofalo R , et al
• Pine-Twaddell E ,
• Newfield RS ,
• Marinkovic M
• Schulmeister C ,
• Millington K ,
• Kaufman M , et al
• Strang JF ,
• Nelson E , et al
• Tordoff DM ,
• Collin A , et al
• Turban JL ,
• Carswell JM , et al
• Hisle-Gorman E ,
• Schvey NA ,
• Adirim TA , et al
• Carmichael P ,
• Masic U , et al
• Dunsford M ,
• Skagerberg E , et al
• Ghelani R ,
• Brain C , et al
• Russell I ,
• Pearson B ,
• Arcelus J ,
• Witcomb GL , et al
• Chiniara LN ,
• Bonifacio HJ ,
• Khatchadourian K ,
• Khatchadourian K , et al
• Waldner RC ,
• Atallah J , et al
• Dhondt K , et al
• Lapauw B , et al
• Craen M , et al
• Elkon-Tamir E ,
• Segev-Becker A , et al
• Segev-Becker A ,
• Israeli G , et al
• Israeli G ,
• Elkon-Tamir E , et al
• Fontanari AMV ,
• Vilanova F ,
• Schneider MA , et al
• Becker-Hebly I ,
• Fahrenkrug S ,
• Campion F , et al
• Hou L , et al
• van der Loos MATC ,
• Hannema SE , et al
• Carruthers P , et al
• Hannema SE ,
• Klink DT , et al
• Kolbuck VD , et al
• NHS England
• Boogers LS ,
• Reijtenbagh SJP ,
• Wiepjes CM ,
• Willemsen LA ,
• Ciancia S ,
• Valentine A ,
• Furniss A , et al
• McGregor K ,
• McKenna JL ,
• Williams CR , et al
• van Heesewijk J ,
• Steenwijk MD ,
• Kreukels BPC , et al

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

• Data supplement 1
• Data supplement 2
• Data supplement 3
• Data supplement 4
• Data supplement 5

Contributors LF, CEH, RH, TL and JT contributed to the conception of this review. RH, CEH, CH, AM and JT contributed to screening and selection. AM and JT completed data extraction. CEH, RH, AM and JT contributed to critical appraisal. CEH, AM and JT completed the synthesis and drafted the manuscript. All authors contributed to interpretation and reviewed and approved the manuscript prior to submission. CEH accepts full responsibility for the finished work and/or the conduct of the study, had access to the data, and controlled the decision to publish.

Funding This work was funded by NHS England to inform the Cass Review (Independent review of gender identity services for children and young people). The funder and Cass Review team had a role in commissioning the research programme but no role in the study conduct, interpretation or conclusion.

Competing interests None declared.

Provenance and peer review Commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Linked Articles

• Original research Clinical guidelines for children and adolescents experiencing gender dysphoria or incongruence: a systematic review of guideline quality (part 1) Jo Taylor Ruth Hall Claire Heathcote Catherine Elizabeth Hewitt Trilby Langton Lorna Fraser Archives of Disease in Childhood 2024; - Published Online First: 09 Apr 2024. doi: 10.1136/archdischild-2023-326499
• Original research Care pathways of children and adolescents referred to specialist gender services: a systematic review Jo Taylor Ruth Hall Trilby Langton Lorna Fraser Catherine Elizabeth Hewitt Archives of Disease in Childhood 2024; - Published Online First: 09 Apr 2024. doi: 10.1136/archdischild-2023-326760
• Original research Psychosocial support interventions for children and adolescents experiencing gender dysphoria or incongruence: a systematic review Claire Heathcote Jo Taylor Ruth Hall Stuart William Jarvis Trilby Langton Catherine Elizabeth Hewitt Lorna Fraser Archives of Disease in Childhood 2024; - Published Online First: 09 Apr 2024. doi: 10.1136/archdischild-2023-326347
• Original research Gender services for children and adolescents across the EU-15+ countries: an online survey Ruth Hall Jo Taylor Claire Heathcote Trilby Langton Catherine Elizabeth Hewitt Lorna Fraser Archives of Disease in Childhood 2024; - Published Online First: 09 Apr 2024. doi: 10.1136/archdischild-2023-326348
• Original research Impact of social transition in relation to gender for children and adolescents: a systematic review Ruth Hall Jo Taylor Catherine Elizabeth Hewitt Claire Heathcote Stuart William Jarvis Trilby Langton Lorna Fraser Archives of Disease in Childhood 2024; - Published Online First: 09 Apr 2024. doi: 10.1136/archdischild-2023-326112
• Original research Characteristics of children and adolescents referred to specialist gender services: a systematic review Jo Taylor Ruth Hall Trilby Langton Lorna Fraser Catherine Elizabeth Hewitt Archives of Disease in Childhood 2024; - Published Online First: 09 Apr 2024. doi: 10.1136/archdischild-2023-326681
• Original research Clinical guidelines for children and adolescents experiencing gender dysphoria or incongruence: a systematic review of recommendations (part 2) Jo Taylor Ruth Hall Claire Heathcote Catherine Elizabeth Hewitt Trilby Langton Lorna Fraser Archives of Disease in Childhood 2024; - Published Online First: 09 Apr 2024. doi: 10.1136/archdischild-2023-326500
• Original research Masculinising and feminising hormone interventions for adolescents experiencing gender dysphoria or incongruence: a systematic review Jo Taylor Alex Mitchell Ruth Hall Trilby Langton Lorna Fraser Catherine Elizabeth Hewitt Archives of Disease in Childhood 2024; - Published Online First: 09 Apr 2024. doi: 10.1136/archdischild-2023-326670
• Editorial Holistic approach to gender questioning children and young people Camilla C Kingdon Archives of Disease in Childhood 2024; - Published Online First: 09 Apr 2024. doi: 10.1136/archdischild-2024-327100

Read the full text or download the PDF: