article review on method study or work study

Methodology of Work Study

• First Online: 31 January 2022

Cite this chapter

• Siegfried Lewark 3  

Part of the book series: Tropical Forestry ((TROPICAL))

197 Accesses

Part II turns to the methods of forest work study and to the large variety of work in tropical forest utilization. This follows logically the general description of its development over time in Part I. It follows also the introduction of the discipline of Forest Work Science and its terminology and roots, including Taylorism, as well as its activities that pointed at the urgent need of improving the working conditions.

First, it is outlined how the views and methods of general work study were perceived and adopted to study work in the forests. Forest work scientists have their own scientific community, on international level within IUFRO, and with their own methods. After outlining the common use of forest work study terminology a plea is made for proper use of the term. Work scientists aim at improvements by work design, based on work study. This may refer to the working human directly, to the technical or to the organizational side, which together constitutes the working environment.

A limited number of work study methods is accounting for the bulk of applications. Methods from general work study are adapted to specific applications. Other methods are specifically developed in forest work study during manual and mechanized work, under three orientations, towards working conditions, operational processes, and work organization. One main use of work study has been assessing the bases for piece-rate wages. They are still widely used in forestry and have to be adjusted regularly after gains in productivity and technical improvements. Performance depends on working conditions, including their technical and operational features, so their impact has to be considered. Stress and strain study always has to take into account performance. Specifics of forest work study are outlined as represented in publications.

The selection of sample studies has been structured according to three scenarios and to typical tasks, with the purpose of structuring the huge variety of work done in tropical forests. The scenarios are industrial work in plantation forests, in natural forests, and third work in non-industrial forest utilization.

This is followed by a definition of the requirements for meaningful sample studies, their approaches, methods, and contents, connected to the pressing issues of work in tropical forests. The sample studies demonstrate the cutting edge of forest work science and demonstrate how this research has been done and can be done.

The intention of the chapter and then of the three following chapters is to cover the scope of typical working situations in tropical forests and to share this knowledge. Together the sample studies are also intended for a demonstration of the state of art regarding work study methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Translated from German.

Title in German: Organisation in der Forstwirtschaft—mit REFA-Methoden zu effizienten Arbeitsprozessen.

IUFRO Division 3 has been reorganized, and has changed the contents, names, and numbers of its research units in earlier years.

In managerial work studies they are sometimes also called study objects: “the data on resource input and produce output must be complemented with descriptions of the study object (workers and equipment)” (Björheden and Thompson 1995 ), which may be considered correct in the sense of language logic, but contrary to ethical positions.

Apud E, Meyer F (2004) Ergonomics. In: Burley J, Evans J, Youngquist J (eds) Encyclopedia of forest sciences, 1st edn. Elsevier, Amsterdam, pp 639–645

Chapter   Google Scholar  

Apud E, Bostrand L, Mobbs LI, Strehlke B (eds) (1989) Guide-lines on ergonomic study in forestry: prepared for research workers in developing countries. ILO, Geneva

Google Scholar  

Backhaus G (1997) 25 Jahre REFA-Fachausschuß Forstwirtschaft: Chronik für die Jahre 1972-1997, Weilburg/Lahn

Björheden R, Thompson MA (1995) An international nomenclature for forest work study. In: Field DB (ed) Misc report. University of Maine, Orono, ME, pp 190–215

Bombosch F (1992) Die kombinierte Aufnahme von ergonomischen Parametern mit Zeitinformationen als Instrument der Arbeitsgestaltung. In: Teutenberg-Raupach A, Gnadt C (eds) Work study – measurement and terminology: International Symposium IUFRO, Göttingen, pp 15–25

Borg G (2005) Scaling experiences during work: perceived exertion and difficulty. In: Stanton NA (ed) Handbook of human factors and ergonomics methods. CRC Press, Boca Raton

Bostrand L (1989) Checking workplaces with an ergonomic checklist. In: Apud E, Bostrand L, Mobbs LI, Strehlke B (eds) Guide-lines on ergonomic study in forestry: prepared for research workers in developing countries. ILO, Geneva, pp 111–144

Eisenhauer G (1992) Hubert Hugo Hilf, Begründer der forstlichen Arbeitswissenschaft. In: Teutenberg-Raupach A, Gnadt C (eds) Work study – measurement and terminology: International Symposium IUFRO, Göttingen, pp 26–33

Hammer W (1997) Wörterbuch der Arbeitswissenschaft. Hanser, München

Harstela P (1991) Work studies in forestry. Silva Carelica, vol 18. University of Joensuu, Joensuu, Finland

Hoß C (1992) Arbeitsorganisation und Ergonomie beim Harvestereinsatz. In: Teutenberg-Raupach A, Gnadt C (eds) Work study – measurement and terminology: International Symposium IUFRO, Göttingen, pp 34–49

ILO (1998) Safety and health in forestry work. ILO code of practice. ILO, Geneva

IUFRO (2021a) IUFRO Division 3 – Forest Operations Engineering and Management. www.iufro.org/science/divisions/division-3/ . Accessed 27 June 2021

IUFRO (2021b) IUFRO: 3.03.00 – Forest ergonomics. www.iufro.org/science/divisions/division-3/30000/30300/ . Accessed 27 June 2021

Jacko JA, Yi JS, Sainfort F, McClellan M (2012) Human factors and ergonomic methods. In: Salvendy G (ed) Handbook of human factors and ergonomics, 4th edn. John Wiley & Sons, Hoboken, pp 298–329

Kanawaty G (1992) Introduction to work study, 4th edn. ILO, Geneva

Kataiamäki W (2019) Promoting decent work and safety and health in forestry. ILO, Geneva

Lewark S (1993a) Der arbeitende Mensch im Forstbetrieb. Allgem Forstz 48:859–861

Lewark S (1993b) Diskussion der Ziele der forstlichen Arbeitswissenschaft. In: Forsttechnik B (ed) 26. International Symposium Mechanisierung der Waldarbeit, Vienna, pp 99–108

Löffler H (1992) Leitfaden zu den Lehrveranstaltungen Arbeitswissenschaft für Studierende der Forstwissenschaft, 3rd edn. Lehrstuhl Forstliche Arbeitswissenschaft Verfahrentstechnik Universität München, Weihenstephan

Magagnotti N, Kanzian C, Schulmeyer F, Spinelli R (2013) A new guide for work studies in forestry. Int J Forest Engineering 24:249–253. https://doi.org/10.1080/14942119.2013.856613

Article   Google Scholar  

Mobbs LI (1989) Guide-lines for work study. In: Apud E, Bostrand L, Mobbs LI, Strehlke B (eds) Guide-lines on ergonomic study in forestry: prepared for research workers in developing countries. ILO, Geneva, pp 209–242

Murrell KFH (1965) Ergonomics: man in his working environment. Chapman and Hall, London

North K, Stapleton C, Vogt C (1982) Ergonomics glossary: terms commonly used in ergonomics. Scheltema & Holkema, Utrecht/Antwerp, Bohn

Poschen P (ed) (1993) Forestry, a safe and healthy profession? Unasylva 44:3–12

REFA (1991) Anleitung für forstliche Arbeitsstudien: Datenermittlung, Arbeitsgestaltung mit Beispielen aus der forstlichen praxis, 3rd edn. REFA, Darmstadt

REFA Fachausschuss Forstwirtschaft (2004) Organisation in der Forstwirtschaft: Mit REFA-Methoden zu effizienten Arbeitsprozessen, 2nd edn. REFA-Fachbuchreihe Arbeitsgestaltung, Darmstadt

Rehschuh D (1988) Comparability of work studies. In: Norsk institutt for skogforskning (ed) Ivar Samset. Report published in honour of professor Dr. H.C. Ivar Samset: Festskrift til ære for professor dr. h.c. Ivar Samset. Norsk institutt for skogforskning, Ås, Norway, pp 337–346

Salvendy G (ed) (2012) Handbook of human factors and ergonomics, 4th edn. John Wiley & Sons, Hoboken

Samset I (1990) Some observations on time and performance studies in forestry. Communications of the Norwegian Forest Institute No. 43.5, Ås, Norway

Sanders MS, McCormick EJ (1993) Human factors in engineering and design, 7th edn. McGraw Hill, New York

Singleton WT (2021) The nature and aims of ergonomics. In: ILO (ed) Encyclopaedia of occupational health and safety. 3rd edn. www.iloencyclopaedia.org/part-iv-66769/ergonomics-52353/goals-principles-and-methods-91538/item/478-the-nature-and-aims-of-ergonomics . Accessed 11 July 2021

Staal Wästerlund D (2001) Heat stress in forestry. Doctoral thesis, Swedish University of Agricultural Sciences

Staal Wästerlund D, Chaseling J, Burström L (2004) The effect of fluid consumption on the forest workers’ performance strategy. Appl Ergonomics 35:29–36. https://doi.org/10.1016/j.apergo.2003.09.002

Stanton NA (ed) (2005a) Handbook of human factors and ergonomics methods. CRC Press, Boca Raton

Stanton NA (2005b) Preface. In: Stanton NA (ed) Handbook of human factors and ergonomics methods. CRC Press, Boca Raton

Strehlke B (1979) Tropical work and working conditions in forestry. In: van Loon J, Staudt FJ, Zander J (eds) Ergonomics in tropical agriculture and forestry. Proceedings Fifth Joint Ergonomic Symposium, Ergonomic Commissions of IAAMRH, CIGR, IUFRO. Centre Agr Publishing Documentation, Wageningen, pp 39–44

Strehlke B (1989) The study of forest accidents. In: Apud E, Bostrand L, Mobbs LI, Strehlke B (eds) Guide-lines on ergonomic study in forestry: prepared for research workers in developing countries. ILO, Geneva, pp 171–207

Strehlke B (1993) Forest management in Indonesia: employment, working conditions and occupational safety. Unasylva 44:25–30

Strehlke B (1996) Asking forest workers about their job. Commonwealth For Rev 75:217–220

Strehlke B (2003) How we work and live: forest workers talk about themselves: a global account of working and living conditions in the forestry sector. Sectoral activities programme working papers, vol 207. ILO, Geneva

Sundberg U (1988) The emergence and establishment of forest operations and techniques as a discipline in forest science. An essay in honour of Ivar Samset. Norsk institutt for skogforskning, Ås, Norway

Taylor F (1911) The principles of scientific management. Harper & Row, New York

Thompson MA (1992) Observation and analysis of performance in forest work. In: Teutenberg-Raupach a, Gnadt C (eds) work study – measurement and terminology: International Symposium IUFRO, Göttingen, pp 202–219

Ulich E (2011) Arbeitspsychologie, 7th edn. Schäffer-Poeschel, Zürich

Download references

Author information

Authors and affiliations.

Faculty of Environment & Natural Resources, Albert–Ludwigs–Universität Freiburg, Freiburg, Baden-Württemberg, Germany

Siegfried Lewark

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Siegfried Lewark .

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer-Verlag GmbH Germany, part of Springer Nature

About this chapter

Lewark, S. (2022). Methodology of Work Study. In: Work in Tropical Forests. Tropical Forestry. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-64444-7_4

Download citation

DOI : https://doi.org/10.1007/978-3-662-64444-7_4

Published : 31 January 2022

Publisher Name : Springer, Berlin, Heidelberg

Print ISBN : 978-3-662-64442-3

Online ISBN : 978-3-662-64444-7

eBook Packages : Biomedical and Life Sciences Biomedical and Life Sciences (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Open access
• Published: 19 April 2024

A scoping review of continuous quality improvement in healthcare system: conceptualization, models and tools, barriers and facilitators, and impact

• Aklilu Endalamaw 1 , 2 ,
• Resham B Khatri 1 , 3 ,
• Tesfaye Setegn Mengistu 1 , 2 ,
• Daniel Erku 1 , 4 , 5 ,
• Eskinder Wolka 6 ,
• Anteneh Zewdie 6 &
• Yibeltal Assefa 1  

BMC Health Services Research volume  24 , Article number:  487 ( 2024 ) Cite this article

674 Accesses

Metrics details

The growing adoption of continuous quality improvement (CQI) initiatives in healthcare has generated a surge in research interest to gain a deeper understanding of CQI. However, comprehensive evidence regarding the diverse facets of CQI in healthcare has been limited. Our review sought to comprehensively grasp the conceptualization and principles of CQI, explore existing models and tools, analyze barriers and facilitators, and investigate its overall impacts.

This qualitative scoping review was conducted using Arksey and O’Malley’s methodological framework. We searched articles in PubMed, Web of Science, Scopus, and EMBASE databases. In addition, we accessed articles from Google Scholar. We used mixed-method analysis, including qualitative content analysis and quantitative descriptive for quantitative findings to summarize findings and PRISMA extension for scoping reviews (PRISMA-ScR) framework to report the overall works.

A total of 87 articles, which covered 14 CQI models, were included in the review. While 19 tools were used for CQI models and initiatives, Plan-Do-Study/Check-Act cycle was the commonly employed model to understand the CQI implementation process. The main reported purposes of using CQI, as its positive impact, are to improve the structure of the health system (e.g., leadership, health workforce, health technology use, supplies, and costs), enhance healthcare delivery processes and outputs (e.g., care coordination and linkages, satisfaction, accessibility, continuity of care, safety, and efficiency), and improve treatment outcome (reduce morbidity and mortality). The implementation of CQI is not without challenges. There are cultural (i.e., resistance/reluctance to quality-focused culture and fear of blame or punishment), technical, structural (related to organizational structure, processes, and systems), and strategic (inadequate planning and inappropriate goals) related barriers that were commonly reported during the implementation of CQI.

Conclusions

Implementing CQI initiatives necessitates thoroughly comprehending key principles such as teamwork and timeline. To effectively address challenges, it’s crucial to identify obstacles and implement optimal interventions proactively. Healthcare professionals and leaders need to be mentally equipped and cognizant of the significant role CQI initiatives play in achieving purposes for quality of care.

Peer Review reports

Continuous quality improvement (CQI) initiative is a crucial initiative aimed at enhancing quality in the health system that has gradually been adopted in the healthcare industry. In the early 20th century, Shewhart laid the foundation for quality improvement by describing three essential steps for process improvement: specification, production, and inspection [ 1 , 2 ]. Then, Deming expanded Shewhart’s three-step model into ‘plan, do, study/check, and act’ (PDSA or PDCA) cycle, which was applied to management practices in Japan in the 1950s [ 3 ] and was gradually translated into the health system. In 1991, Kuperman applied a CQI approach to healthcare, comprising selecting a process to be improved, assembling a team of expert clinicians that understands the process and the outcomes, determining key steps in the process and expected outcomes, collecting data that measure the key process steps and outcomes, and providing data feedback to the practitioners [ 4 ]. These philosophies have served as the baseline for the foundation of principles for continuous improvement [ 5 ].

Continuous quality improvement fosters a culture of continuous learning, innovation, and improvement. It encourages proactive identification and resolution of problems, promotes employee engagement and empowerment, encourages trust and respect, and aims for better quality of care [ 6 , 7 ]. These characteristics drive the interaction of CQI with other quality improvement projects, such as quality assurance and total quality management [ 8 ]. Quality assurance primarily focuses on identifying deviations or errors through inspections, audits, and formal reviews, often settling for what is considered ‘good enough’, rather than pursuing the highest possible standards [ 9 , 10 ], while total quality management is implemented as the management philosophy and system to improve all aspects of an organization continuously [ 11 ].

Continuous quality improvement has been implemented to provide quality care. However, providing effective healthcare is a complicated and complex task in achieving the desired health outcomes and the overall well-being of individuals and populations. It necessitates tackling issues, including access, patient safety, medical advances, care coordination, patient-centered care, and quality monitoring [ 12 , 13 ], rooted long ago. It is assumed that the history of quality improvement in healthcare started in 1854 when Florence Nightingale introduced quality improvement documentation [ 14 ]. Over the passing decades, Donabedian introduced structure, processes, and outcomes as quality of care components in 1966 [ 15 ]. More comprehensively, the Institute of Medicine in the United States of America (USA) has identified effectiveness, efficiency, equity, patient-centredness, safety, and timeliness as the components of quality of care [ 16 ]. Moreover, quality of care has recently been considered an integral part of universal health coverage (UHC) [ 17 ], which requires initiatives to mobilise essential inputs [ 18 ].

While the overall objective of CQI in health system is to enhance the quality of care, it is important to note that the purposes and principles of CQI can vary across different contexts [ 19 , 20 ]. This variation has sparked growing research interest. For instance, a review of CQI approaches for capacity building addressed its role in health workforce development [ 21 ]. Another systematic review, based on random-controlled design studies, assessed the effectiveness of CQI using training as an intervention and the PDSA model [ 22 ]. As a research gap, the former review was not directly related to the comprehensive elements of quality of care, while the latter focused solely on the impact of training using the PDSA model, among other potential models. Additionally, a review conducted in 2015 aimed to identify barriers and facilitators of CQI in Canadian contexts [ 23 ]. However, all these reviews presented different perspectives and investigated distinct outcomes. This suggests that there is still much to explore in terms of comprehensively understanding the various aspects of CQI initiatives in healthcare.

As a result, we conducted a scoping review to address several aspects of CQI. Scoping reviews serve as a valuable tool for systematically mapping the existing literature on a specific topic. They are instrumental when dealing with heterogeneous or complex bodies of research. Scoping reviews provide a comprehensive overview by summarizing and disseminating findings across multiple studies, even when evidence varies significantly [ 24 ]. In our specific scoping review, we included various types of literature, including systematic reviews, to enhance our understanding of CQI.

This scoping review examined how CQI is conceptualized and measured and investigated models and tools for its application while identifying implementation challenges and facilitators. It also analyzed the purposes and impact of CQI on the health systems, providing valuable insights for enhancing healthcare quality.

Protocol registration and results reporting

Protocol registration for this scoping review was not conducted. Arksey and O’Malley’s methodological framework was utilized to conduct this scoping review [ 25 ]. The scoping review procedures start by defining the research questions, identifying relevant literature, selecting articles, extracting data, and summarizing the results. The review findings are reported using the PRISMA extension for a scoping review (PRISMA-ScR) [ 26 ]. McGowan and colleagues also advised researchers to report findings from scoping reviews using PRISMA-ScR [ 27 ].

Defining the research problems

This review aims to comprehensively explore the conceptualization, models, tools, barriers, facilitators, and impacts of CQI within the healthcare system worldwide. Specifically, we address the following research questions: (1) How has CQI been defined across various contexts? (2) What are the diverse approaches to implementing CQI in healthcare settings? (3) Which tools are commonly employed for CQI implementation ? (4) What barriers hinder and facilitators support successful CQI initiatives? and (5) What effects CQI initiatives have on the overall care quality?

Information source and search strategy

We conducted the search in PubMed, Web of Science, Scopus, and EMBASE databases, and the Google Scholar search engine. The search terms were selected based on three main distinct concepts. One group was CQI-related terms. The second group included terms related to the purpose for which CQI has been implemented, and the third group included processes and impact. These terms were selected based on the Donabedian framework of structure, process, and outcome [ 28 ]. Additionally, the detailed keywords were recruited from the primary health framework, which has described lists of dimensions under process, output, outcome, and health system goals of any intervention for health [ 29 ]. The detailed search strategy is presented in the Supplementary file 1 (Search strategy). The search for articles was initiated on August 12, 2023, and the last search was conducted on September 01, 2023.

Eligibility criteria and article selection

Based on the scoping review’s population, concept, and context frameworks [ 30 ], the population included any patients or clients. Additionally, the concepts explored in the review encompassed definitions, implementation, models, tools, barriers, facilitators, and impacts of CQI. Furthermore, the review considered contexts at any level of health systems. We included articles if they reported results of qualitative or quantitative empirical study, case studies, analytic or descriptive synthesis, any review, and other written documents, were published in peer-reviewed journals, and were designed to address at least one of the identified research questions or one of the identified implementation outcomes or their synonymous taxonomy as described in the search strategy. Based on additional contexts, we included articles published in English without geographic and time limitations. We excluded articles with abstracts only, conference abstracts, letters to editors, commentators, and corrections.

We exported all citations to EndNote x20 to remove duplicates and screen relevant articles. The article selection process includes automatic duplicate removal by using EndNote x20, unmatched title and abstract removal, citation and abstract-only materials removal, and full-text assessment. The article selection process was mainly conducted by the first author (AE) and reported to the team during the weekly meetings. The first author encountered papers that caused confusion regarding whether to include or exclude them and discussed them with the last author (YA). Then, decisions were ultimately made. Whenever disagreements happened, they were resolved by discussion and reconsideration of the review questions in relation to the written documents of the article. Further statistical analysis, such as calculating Kappa, was not performed to determine article inclusion or exclusion.

Data extraction and data items

We extracted first author, publication year, country, settings, health problem, the purpose of the study, study design, types of intervention if applicable, CQI approaches/steps if applicable, CQI tools and procedures if applicable, and main findings using a customized Microsoft Excel form.

Summarizing and reporting the results

The main findings were summarized and described based on the main themes, including concepts under conceptualizing, principles, teams, timelines, models, tools, barriers, facilitators, and impacts of CQI. Results-based convergent synthesis, achieved through mixed-method analysis, involved content analysis to identify the thematic presentation of findings. Additionally, a narrative description was used for quantitative findings, aligning them with the appropriate theme. The authors meticulously reviewed the primary findings from each included material and contextualized these findings concerning the main themes1. This approach provides a comprehensive understanding of complex interventions and health systems, acknowledging quantitative and qualitative evidence.

Search results

A total of 11,251 documents were identified from various databases: SCOPUS ( n  = 4,339), PubMed ( n  = 2,893), Web of Science ( n  = 225), EMBASE ( n  = 3,651), and Google Scholar ( n  = 143). After removing duplicates ( n  = 5,061), 6,190 articles were evaluated by title and abstract. Subsequently, 208 articles were assessed for full-text eligibility. Following the eligibility criteria, 121 articles were excluded, leaving 87 included in the current review (Fig.  1 ).

Article selection process

Operationalizing continuous quality improvement

Continuous Quality Improvement (CQI) is operationalized as a cyclic process that requires commitment to implementation, teamwork, time allocation, and celebrating successes and failures.

CQI is a cyclic ongoing process that is followed reflexive, analytical and iterative steps, including identifying gaps, generating data, developing and implementing action plans, evaluating performance, providing feedback to implementers and leaders, and proposing necessary adjustments [ 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 ].

CQI requires committing to the philosophy, involving continuous improvement [ 19 , 38 ], establishing a mission statement [ 37 ], and understanding quality definition [ 19 ].

CQI involves a wide range of patient-oriented measures and performance indicators, specifically satisfying internal and external customers, developing quality assurance, adopting common quality measures, and selecting process measures [ 8 , 19 , 35 , 36 , 37 , 39 , 40 ].

CQI requires celebrating success and failure without personalization, leading each team member to develop error-free attitudes [ 19 ]. Success and failure are related to underlying organizational processes and systems as causes of failure rather than blaming individuals [ 8 ] because CQI is process-focused based on collaborative, data-driven, responsive, rigorous and problem-solving statistical analysis [ 8 , 19 , 38 ]. Furthermore, a gap or failure opens another opportunity for establishing a data-driven learning organization [ 41 ].

CQI cannot be implemented without a CQI team [ 8 , 19 , 37 , 39 , 42 , 43 , 44 , 45 , 46 ]. A CQI team comprises individuals from various disciplines, often comprising a team leader, a subject matter expert (physician or other healthcare provider), a data analyst, a facilitator, frontline staff, and stakeholders [ 39 , 43 , 47 , 48 , 49 ]. It is also important to note that inviting stakeholders or partners as part of the CQI support intervention is crucial [ 19 , 38 , 48 ].

The timeline is another distinct feature of CQI because the results of CQI vary based on the implementation duration of each cycle [ 35 ]. There is no specific time limit for CQI implementation, although there is a general consensus that a cycle of CQI should be relatively short [ 35 ]. For instance, a CQI implementation took 2 months [ 42 ], 4 months [ 50 ], 9 months [ 51 , 52 ], 12 months [ 53 , 54 , 55 ], and one year and 5 months [ 49 ] duration to achieve the desired positive outcome, while bi-weekly [ 47 ] and monthly data reviews and analyses [ 44 , 48 , 56 ], and activities over 3 months [ 57 ] have also resulted in a positive outcome.

Continuous quality improvement models and tools

There have been several models are utilized. The Plan-Do-Study/Check-Act cycle is a stepwise process involving project initiation, situation analysis, root cause identification, solution generation and selection, implementation, result evaluation, standardization, and future planning [ 7 , 36 , 37 , 45 , 47 , 48 , 49 , 50 , 51 , 53 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 ]. The FOCUS-PDCA cycle enhances the PDCA process by adding steps to find and improve a process (F), organize a knowledgeable team (O), clarify the process (C), understand variations (U), and select improvements (S) [ 55 , 71 , 72 , 73 ]. The FADE cycle involves identifying a problem (Focus), understanding it through data analysis (Analyze), devising solutions (Develop), and implementing the plan (Execute) [ 74 ]. The Logic Framework involves brainstorming to identify improvement areas, conducting root cause analysis to develop a problem tree, logically reasoning to create an objective tree, formulating the framework, and executing improvement projects [ 75 ]. Breakthrough series approach requires CQI teams to meet in quarterly collaborative learning sessions, share learning experiences, and continue discussion by telephone and cross-site visits to strengthen learning and idea exchange [ 47 ]. Another CQI model is the Lean approach, which has been conducted with Kaizen principles [ 52 ], 5 S principles, and the Six Sigma model. The 5 S (Sort, Set/Straighten, Shine, Standardize, Sustain) systematically organises and improves the workplace, focusing on sorting, setting order, shining, standardizing, and sustaining the improvement [ 54 , 76 ]. Kaizen principles guide CQI by advocating for continuous improvement, valuing all ideas, solving problems, focusing on practical, low-cost improvements, using data to drive change, acknowledging process defects, reducing variability and waste, recognizing every interaction as a customer-supplier relationship, empowering workers, responding to all ideas, and maintaining a disciplined workplace [ 77 ]. Lean Six Sigma, a CQI model, applies the DMAIC methodology, which involves defining (D) and measuring the problem (M), analyzing root causes (A), improving by finding solutions (I), and controlling by assessing process stability (C) [ 78 , 79 ]. The 5 C-cyclic model (consultation, collection, consideration, collaboration, and celebration), the first CQI framework for volunteer dental services in Aboriginal communities, ensures quality care based on community needs [ 80 ]. One study used meetings involving activities such as reviewing objectives, assigning roles, discussing the agenda, completing tasks, retaining key outputs, planning future steps, and evaluating the meeting’s effectiveness [ 81 ].

Various tools are involved in the implementation or evaluation of CQI initiatives: checklists [ 53 , 82 ], flowcharts [ 81 , 82 , 83 ], cause-and-effect diagrams (fishbone or Ishikawa diagrams) [ 60 , 62 , 79 , 81 , 82 ], fuzzy Pareto diagram [ 82 ], process maps [ 60 ], time series charts [ 48 ], why-why analysis [ 79 ], affinity diagrams and multivoting [ 81 ], and run chart [ 47 , 48 , 51 , 60 , 84 ], and others mentioned in the table (Table  1 ).

Barriers and facilitators of continuous quality improvement implementation

Implementing CQI initiatives is determined by various barriers and facilitators, which can be thematized into four dimensions. These dimensions are cultural, technical, structural, and strategic dimensions.

Continuous quality improvement initiatives face various cultural, strategic, technical, and structural barriers. Cultural dimension barriers involve resistance to change (e.g., not accepting online technology), lack of quality-focused culture, staff reporting apprehensiveness, and fear of blame or punishment [ 36 , 41 , 85 , 86 ]. The technical dimension barriers of CQI can include various factors that hinder the effective implementation and execution of CQI processes [ 36 , 86 , 87 , 88 , 89 ]. Structural dimension barriers of CQI arise from the organization structure, process, and systems that can impede the effective implementation and sustainability of CQI [ 36 , 85 , 86 , 87 , 88 ]. Strategic dimension barriers are, for example, the inability to select proper CQI goals and failure to integrate CQI into organizational planning and goals [ 36 , 85 , 86 , 87 , 88 , 90 ].

Facilitators are also grouped to cultural, structural, technical, and strategic dimensions to provide solutions to CQI barriers. Cultural challenges were addressed by developing a group culture to CQI and other rewards [ 39 , 41 , 80 , 85 , 86 , 87 , 90 , 91 , 92 ]. Technical facilitators are pivotal to improving technical barriers [ 39 , 42 , 53 , 69 , 86 , 90 , 91 ]. Structural-related facilitators are related to improving communication, infrastructure, and systems [ 86 , 92 , 93 ]. Strategic dimension facilitators include strengthening leadership and improving decision-making skills [ 43 , 53 , 67 , 86 , 87 , 92 , 94 , 95 ] (Table  2 ).

Impact of continuous quality improvement

Continuous quality improvement initiatives can significantly impact the quality of healthcare in a wide range of health areas, focusing on improving structure, the health service delivery process and improving client wellbeing and reducing mortality.

Structure components

These are health leadership, financing, workforce, technology, and equipment and supplies. CQI has improved planning, monitoring and evaluation [ 48 , 53 ], and leadership and planning [ 48 ], indicating improvement in leadership perspectives. Implementing CQI in primary health care (PHC) settings has shown potential for maintaining or reducing operation costs [ 67 ]. Findings from another study indicate that the costs associated with implementing CQI interventions per facility ranged from approximately $2,000 to $10,500 per year, with an average cost of approximately $10 to $60 per admitted client [ 57 ]. However, based on model predictions, the average cost savings after implementing CQI were estimated to be $5430 [ 31 ]. CQI can also be applied to health workforce development [ 32 ]. CQI in the institutional system improved medical education [ 66 , 96 , 97 ], human resources management [ 53 ], motivated staffs [ 76 ], and increased staff health awareness [ 69 ], while concerns raised about CQI impartiality, independence, and public accountability [ 96 ]. Regarding health technology, CQI also improved registration and documentation [ 48 , 53 , 98 ]. Furthermore, the CQI initiatives increased cleanliness [ 54 ] and improved logistics, supplies, and equipment [ 48 , 53 , 68 ].

Process and output components

The process component focuses on the activities and actions involved in delivering healthcare services.

Service delivery

CQI interventions improved service delivery [ 53 , 56 , 99 ], particularly a significant 18% increase in the overall quality of service performance [ 48 ], improved patient counselling, adherence to appropriate procedures, and infection prevention [ 48 , 68 ], and optimised workflow [ 52 ].

Coordination and collaboration

CQI initiatives improved coordination and collaboration through collecting and analysing data, onsite technical support, training, supportive supervision [ 53 ] and facilitating linkages between work processes and a quality control group [ 65 ].

Patient satisfaction

The CQI initiatives increased patient satisfaction and improved quality of life by optimizing care quality management, improving the quality of clinical nursing, reducing nursing defects and enhancing the wellbeing of clients [ 54 , 76 , 100 ], although CQI was not associated with changes in adolescent and young adults’ satisfaction [ 51 ].

CQI initiatives reduced medication error reports from 16 to 6 [ 101 ], and it significantly reduced the administration of inappropriate prophylactic antibiotics [ 44 ], decreased errors in inpatient care [ 52 ], decreased the overall episiotomy rate from 44.5 to 33.3% [ 83 ], reduced the overall incidence of unplanned endotracheal extubation [ 102 ], improving appropriate use of computed tomography angiography [ 103 ], and appropriate diagnosis and treatment selection [ 47 ].

Continuity of care

CQI initiatives effectively improve continuity of care by improving client and physician interaction. For instance, provider continuity levels showed a 64% increase [ 55 ]. Modifying electronic medical record templates, scheduling, staff and parental education, standardization of work processes, and birth to 1-year age-specific incentives in post-natal follow-up care increased continuity of care to 74% in 2018 compared to baseline 13% in 2012 [ 84 ].

The CQI initiative yielded enhanced efficiency in the cardiac catheterization laboratory, as evidenced by improved punctuality in procedure starts and increased efficiency in manual sheath-pulls inside [ 78 ].

Accessibility

CQI initiatives were effective in improving accessibility in terms of increasing service coverage and utilization rate. For instance, screening for cigarettes, nutrition counselling, folate prescription, maternal care, immunization coverage [ 53 , 81 , 104 , 105 ], reducing the percentage of non-attending patients to surgery to 0.9% from the baseline 3.9% [ 43 ], increasing Chlamydia screening rates from 29 to 60% [ 45 ], increasing HIV care continuum coverage [ 51 , 59 , 60 ], increasing in the uptake of postpartum long-acting reversible contraceptive use from 6.9% at the baseline to 25.4% [ 42 ], increasing post-caesarean section prophylaxis from 36 to 89% [ 62 ], a 31% increase of kangaroo care practice [ 50 ], and increased follow-up [ 65 ]. Similarly, the QI intervention increased the quality of antenatal care by 29.3%, correct partograph use by 51.7%, and correct active third-stage labour management, a 19.6% improvement from the baseline, but not significantly associated with improvement in contraceptive service uptake [ 61 ].

Timely access

CQI interventions improved the time care provision [ 52 ], and reduced waiting time [ 62 , 74 , 76 , 106 ]. For instance, the discharge process waiting time in the emergency department decreased from 76 min to 22 min [ 79 ]. It also reduced mean postprocedural length of stay from 2.8 days to 2.0 days [ 31 ].

Acceptability

Acceptability of CQI by healthcare providers was satisfactory. For instance, 88% of the faculty, 64% of the residents, and 82% of the staff believed CQI to be useful in the healthcare clinic [ 107 ].

Outcome components

Morbidity and mortality.

CQI efforts have demonstrated better management outcomes among diabetic patients [ 40 ], patients with oral mucositis [ 71 ], and anaemic patients [ 72 ]. It has also reduced infection rate in post-caesarean Sect. [ 62 ], reduced post-peritoneal dialysis peritonitis [ 49 , 108 ], and prevented pressure ulcers [ 70 ]. It is explained by peritonitis incidence from once every 40.1 patient months at baseline to once every 70.8 patient months after CQI [ 49 ] and a 63% reduction in pressure ulcer prevalence within 2 years from 2008 to 2010 [ 70 ]. Furthermore, CQI initiatives significantly reduced in-hospital deaths [ 31 ] and increased patient survival rates [ 108 ]. Figure  2 displays the overall process of the CQI implementations.

The overall mechanisms of continuous quality improvement implementation

In this review, we examined the fundamental concepts and principles underlying CQI, the factors that either hinder or assist in its successful application and implementation, and the purpose of CQI in enhancing quality of care across various health issues.

Our findings have brought attention to the application and implementation of CQI, emphasizing its underlying concepts and principles, as evident in the existing literature [ 31 , 32 , 33 , 34 , 35 , 36 , 39 , 40 , 43 , 45 , 46 ]. Continuous quality improvement has shared with the principles of continuous improvement, such as a customer-driven focus, effective leadership, active participation of individuals, a process-oriented approach, systematic implementation, emphasis on design improvement and prevention, evidence-based decision-making, and fostering partnership [ 5 ]. Moreover, Deming’s 14 principles laid the foundation for CQI principles [ 109 ]. These principles have been adapted and put into practice in various ways: ten [ 19 ] and five [ 38 ] principles in hospitals, five principles for capacity building [ 38 ], and two principles for medication error prevention [ 41 ]. As a principle, the application of CQI can be process-focused [ 8 , 19 ] or impact-focused [ 38 ]. Impact-focused CQI focuses on achieving specific outcomes or impacts, whereas process-focused CQI prioritizes and improves the underlying processes and systems. These principles complement each other and can be utilized based on the objectives of quality improvement initiatives in healthcare settings. Overall, CQI is an ongoing educational process that requires top management’s involvement, demands coordination across departments, encourages the incorporation of views beyond clinical area, and provides non-judgemental evidence based on objective data [ 110 ].

The current review recognized that it was not easy to implement CQI. It requires reasonable utilization of various models and tools. The application of each tool can be varied based on the studied health problem and the purpose of CQI initiative [ 111 ], varied in context, content, structure, and usability [ 112 ]. Additionally, overcoming the cultural, technical, structural, and strategic-related barriers. These barriers have emerged from clinical staff, managers, and health systems perspectives. Of the cultural obstacles, staff non-involvement, resistance to change, and reluctance to report error were staff-related. In contrast, others, such as the absence of celebration for success and hierarchical and rational culture, may require staff and manager involvement. Staff members may exhibit reluctance in reporting errors due to various cultural factors, including lack of trust, hierarchical structures, fear of retribution, and a blame-oriented culture. These challenges pose obstacles to implementing standardized CQI practices, as observed, for instance, in community pharmacy settings [ 85 ]. The hierarchical culture, characterized by clearly defined levels of power, authority, and decision-making, posed challenges to implementing CQI initiatives in public health [ 41 , 86 ]. Although rational culture, a type of organizational culture, emphasizes logical thinking and rational decision-making, it can also create challenges for CQI implementation [ 41 , 86 ] because hierarchical and rational cultures, which emphasize bureaucratic norms and narrow definitions of achievement, were found to act as barriers to the implementation of CQI [ 86 ]. These could be solved by developing a shared mindset and collective commitment, establishing a shared purpose, developing group norms, and cultivating psychological preparedness among staff, managers, and clients to implement and sustain CQI initiatives. Furthermore, reversing cultural-related barriers necessitates cultural-related solutions: development of a culture and group culture to CQI [ 41 , 86 ], positive comprehensive perception [ 91 ], commitment [ 85 ], involving patients, families, leaders, and staff [ 39 , 92 ], collaborating for a common goal [ 80 , 86 ], effective teamwork [ 86 , 87 ], and rewarding and celebrating successes [ 80 , 90 ].

The technical dimension barriers of CQI can include inadequate capitalization of a project and insufficient support for CQI facilitators and data entry managers [ 36 ], immature electronic medical records or poor information systems [ 36 , 86 ], and the lack of training and skills [ 86 , 87 , 88 ]. These challenges may cause the CQI team to rely on outdated information and technologies. The presence of barriers on the technical dimension may challenge the solid foundation of CQI expertise among staff, the ability to recognize opportunities for improvement, a comprehensive understanding of how services are produced and delivered, and routine use of expertise in daily work. Addressing these technical barriers requires knowledge creation activities (training, seminar, and education) [ 39 , 42 , 53 , 69 , 86 , 90 , 91 ], availability of quality data [ 86 ], reliable information [ 92 ], and a manual-online hybrid reporting system [ 85 ].

Structural dimension barriers of CQI include inadequate communication channels and lack of standardized process, specifically weak physician-to-physician synergies [ 36 ], lack of mechanisms for disseminating knowledge and limited use of communication mechanisms [ 86 ]. Lack of communication mechanism endangers sharing ideas and feedback among CQI teams, leading to misunderstandings, limited participation and misinterpretations, and a lack of learning [ 113 ]. Knowledge translation facilitates the co-production of research, subsequent diffusion of knowledge, and the developing stakeholder’s capacity and skills [ 114 ]. Thus, the absence of a knowledge translation mechanism may cause missed opportunities for learning, inefficient problem-solving, and limited creativity. To overcome these challenges, organizations should establish effective communication and information systems [ 86 , 93 ] and learning systems [ 92 ]. Though CQI and knowledge translation have interacted with each other, it is essential to recognize that they are distinct. CQI focuses on process improvement within health care systems, aiming to optimize existing processes, reduce errors, and enhance efficiency.

In contrast, knowledge translation bridges the gap between research evidence and clinical practice, translating research findings into actionable knowledge for practitioners. While both CQI and knowledge translation aim to enhance health care quality and patient outcomes, they employ different strategies: CQI utilizes tools like Plan-Do-Study-Act cycles and statistical process control, while knowledge translation involves knowledge synthesis and dissemination. Additionally, knowledge translation can also serve as a strategy to enhance CQI. Both concepts share the same principle: continuous improvement is essential for both. Therefore, effective strategies on the structural dimension may build efficient and effective steering councils, information systems, and structures to diffuse learning throughout the organization.

Strategic factors, such as goals, planning, funds, and resources, determine the overall purpose of CQI initiatives. Specific barriers were improper goals and poor planning [ 36 , 86 , 88 ], fragmentation of quality assurance policies [ 87 ], inadequate reinforcement to staff [ 36 , 90 ], time constraints [ 85 , 86 ], resource inadequacy [ 86 ], and work overload [ 86 ]. These barriers can be addressed through strengthening leadership [ 86 , 87 ], CQI-based mentoring [ 94 ], periodic monitoring, supportive supervision and coaching [ 43 , 53 , 87 , 92 , 95 ], participation, empowerment, and accountability [ 67 ], involving all stakeholders in decision-making [ 86 , 87 ], a provider-payer partnership [ 64 ], and compensating staff for after-hours meetings on CQI [ 85 ]. The strategic dimension, characterized by a strategic plan and integrated CQI efforts, is devoted to processes that are central to achieving strategic priorities. Roles and responsibilities are defined in terms of integrated strategic and quality-related goals [ 115 ].

The utmost goal of CQI has been to improve the quality of care, which is usually revealed by structure, process, and outcome. After resolving challenges and effectively using tools and running models, the goal of CQI reflects the ultimate reason and purpose of its implementation. First, effectively implemented CQI initiatives can improve leadership, health financing, health workforce development, health information technology, and availability of supplies as the building blocks of a health system [ 31 , 48 , 53 , 68 , 98 ]. Second, effectively implemented CQI initiatives improved care delivery process (counselling, adherence with standards, coordination, collaboration, and linkages) [ 48 , 53 , 65 , 68 ]. Third, the CQI can improve outputs of healthcare delivery, such as satisfaction, accessibility (timely access, utilization), continuity of care, safety, efficiency, and acceptability [ 52 , 54 , 55 , 76 , 78 ]. Finally, the effectiveness of the CQI initiatives has been tested in enhancing responses related to key aspects of the HIV response, maternal and child health, non-communicable disease control, and others (e.g., surgery and peritonitis). However, it is worth noting that CQI initiative has not always been effective. For instance, CQI using a two- to nine-times audit cycle model through systems assessment tools did not bring significant change to increase syphilis testing performance [ 116 ]. This study was conducted within the context of Aboriginal and Torres Strait Islander people’s primary health care settings. Notably, ‘the clinics may not have consistently prioritized syphilis testing performance in their improvement strategies, as facilitated by the CQI program’ [ 116 ]. Additionally, by applying CQI-based mentoring, uptake of facility-based interventions was not significantly improved, though it was effective in increasing community health worker visits during pregnancy and the postnatal period, knowledge about maternal and child health and exclusive breastfeeding practice, and HIV disclosure status [ 117 ]. The study conducted in South Africa revealed no significant association between the coverage of facility-based interventions and Continuous Quality Improvement (CQI) implementation. This lack of association was attributed to the already high antenatal and postnatal attendance rates in both control and intervention groups at baseline, leaving little room for improvement. Additionally, the coverage of HIV interventions remained consistently high throughout the study period [ 117 ].

Regarding health care and policy implications, CQI has played a vital role in advancing PHC and fostering the realization of UHC goals worldwide. The indicators found in Donabedian’s framework that are positively influenced by CQI efforts are comparable to those included in the PHC performance initiative’s conceptual framework [ 29 , 118 , 119 ]. It is clearly explained that PHC serves as the roadmap to realizing the vision of UHC [ 120 , 121 ]. Given these circumstances, implementing CQI can contribute to the achievement of PHC principles and the objectives of UHC. For instance, by implementing CQI methods, countries have enhanced the accessibility, affordability, and quality of PHC services, leading to better health outcomes for their populations. CQI has facilitated identifying and resolving healthcare gaps and inefficiencies, enabling countries to optimize resource allocation and deliver more effective and patient-centered care. However, it is crucial to recognize that the successful implementation of Continuous Quality Improvement (CQI) necessitates optimizing the duration of each cycle, understanding challenges and barriers that extend beyond the health system and settings, and acknowledging that its effectiveness may be compromised if these challenges are not adequately addressed.

Despite abundant literature, there are still gaps regarding the relationship between CQI and other dimensions within the healthcare system. No studies have examined the impact of CQI initiatives on catastrophic health expenditure, effective service coverage, patient-centredness, comprehensiveness, equity, health security, and responsiveness.

Limitations

In conducting this review, it has some limitations to consider. Firstly, only articles published in English were included, which may introduce the exclusion of relevant non-English articles. Additionally, as this review follows a scoping methodology, the focus is on synthesising available evidence rather than critically evaluating or scoring the quality of the included articles.

Continuous quality improvement is investigated as a continuous and ongoing intervention, where the implementation time can vary across different cycles. The CQI team and implementation timelines were critical elements of CQI in different models. Among the commonly used approaches, the PDSA or PDCA is frequently employed. In most CQI models, a wide range of tools, nineteen tools, are commonly utilized to support the improvement process. Cultural, technical, structural, and strategic barriers and facilitators are significant in implementing CQI initiatives. Implementing the CQI initiative aims to improve health system blocks, enhance health service delivery process and output, and ultimately prevent morbidity and reduce mortality. For future researchers, considering that CQI is context-dependent approach, conducting scale-up implementation research about catastrophic health expenditure, effective service coverage, patient-centredness, comprehensiveness, equity, health security, and responsiveness across various settings and health issues would be valuable.

Availability of data and materials

The data used and/or analyzed during the current study are available in this manuscript and/or the supplementary file.

Shewhart WA, Deming WE. Memoriam: Walter A. Shewhart, 1891–1967. Am Stat. 1967;21(2):39–40.

Article   Google Scholar  

Shewhart WA. Statistical method from the viewpoint of quality control. New York: Dover; 1986. ISBN 978-0486652320. OCLC 13822053. Reprint. Originally published: Washington, DC: Graduate School of the Department of Agriculture, 1939.

Moen R, editor Foundation and History of the PDSA Cycle. Asian network for quality conference Tokyo. https://www.deming.org/sites/default/files/pdf/2015/PDSA_History_Ron_MoenPdf . 2009.

Kuperman G, James B, Jacobsen J, Gardner RM. Continuous quality improvement applied to medical care: experiences at LDS hospital. Med Decis Making. 1991;11(4suppl):S60–65.

Article   CAS   PubMed   Google Scholar  

Singh J, Singh H. Continuous improvement philosophy–literature review and directions. Benchmarking: An International Journal. 2015;22(1):75–119.

Goldstone J. Presidential address: Sony, Porsche, and vascular surgery in the 21st century. J Vasc Surg. 1997;25(2):201–10.

Radawski D. Continuous quality improvement: origins, concepts, problems, and applications. J Physician Assistant Educ. 1999;10(1):12–6.

Shortell SM, O’Brien JL, Carman JM, Foster RW, Hughes E, Boerstler H, et al. Assessing the impact of continuous quality improvement/total quality management: concept versus implementation. Health Serv Res. 1995;30(2):377.

CAS   PubMed   PubMed Central   Google Scholar  

Lohr K. Quality of health care: an introduction to critical definitions, concepts, principles, and practicalities. Striving for quality in health care. 1991.

Berwick DM. The clinical process and the quality process. Qual Manage Healthc. 1992;1(1):1–8.

Article   CAS   Google Scholar  

Gift B. On the road to TQM. Food Manage. 1992;27(4):88–9.

CAS   PubMed   Google Scholar  

Greiner A, Knebel E. The core competencies needed for health care professionals. health professions education: A bridge to quality. 2003:45–73.

McCalman J, Bailie R, Bainbridge R, McPhail-Bell K, Percival N, Askew D et al. Continuous quality improvement and comprehensive primary health care: a systems framework to improve service quality and health outcomes. Front Public Health. 2018:6 (76):1–6.

Sheingold BH, Hahn JA. The history of healthcare quality: the first 100 years 1860–1960. Int J Afr Nurs Sci. 2014;1:18–22.

Google Scholar  

Donabedian A. Evaluating the quality of medical care. Milbank Q. 1966;44(3):166–206.

Institute of Medicine (US) Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington (DC): National Academies Press (US). 2001. 2, Improving the 21st-century Health Care System. Available from: https://www.ncbi.nlm.nih.gov/books/NBK222265/ .

Rubinstein A, Barani M, Lopez AS. Quality first for effective universal health coverage in low-income and middle-income countries. Lancet Global Health. 2018;6(11):e1142–1143.

Article   PubMed   Google Scholar  

Agency for Healthcare Reserach and Quality. Quality Improvement and monitoring at your fingertips USA,: Agency for Healthcare Reserach and Quality. 2022. Available from: https://qualityindicators.ahrq.gov/ .

Anderson CA, Cassidy B, Rivenburgh P. Implementing continuous quality improvement (CQI) in hospitals: lessons learned from the International Quality Study. Qual Assur Health Care. 1991;3(3):141–6.

Gardner K, Mazza D. Quality in general practice - definitions and frameworks. Aust Fam Physician. 2012;41(3):151–4.

PubMed   Google Scholar  

Loper AC, Jensen TM, Farley AB, Morgan JD, Metz AJ. A systematic review of approaches for continuous quality improvement capacity-building. J Public Health Manage Pract. 2022;28(2):E354.

Hill JE, Stephani A-M, Sapple P, Clegg AJ. The effectiveness of continuous quality improvement for developing professional practice and improving health care outcomes: a systematic review. Implement Sci. 2020;15(1):1–14.

Candas B, Jobin G, Dubé C, Tousignant M, Abdeljelil AB, Grenier S, et al. Barriers and facilitators to implementing continuous quality improvement programs in colonoscopy services: a mixed methods systematic review. Endoscopy Int Open. 2016;4(02):E118–133.

Peters MD, Marnie C, Colquhoun H, Garritty CM, Hempel S, Horsley T, et al. Scoping reviews: reinforcing and advancing the methodology and application. Syst Reviews. 2021;10(1):1–6.

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

McGowan J, Straus S, Moher D, Langlois EV, O’Brien KK, Horsley T, et al. Reporting scoping reviews—PRISMA ScR extension. J Clin Epidemiol. 2020;123:177–9.

Donabedian A. Explorations in quality assessment and monitoring: the definition of quality and approaches to its assessment. Health Administration Press, Ann Arbor. 1980;1.

World Health Organization. Operational framework for primary health care: transforming vision into action. Geneva: World Health Organization and the United Nations Children’s Fund (UNICEF); 2020 [updated 14 December 2020; cited 2023 Nov Oct 17]. Available from: https://www.who.int/publications/i/item/9789240017832 .

The Joanna Briggs Institute. The Joanna Briggs Institute Reviewers’ Manual :2014 edition. Australia: The Joanna Briggs Institute. 2014:88–91.

Rihal CS, Kamath CC, Holmes DR Jr, Reller MK, Anderson SS, McMurtry EK, et al. Economic and clinical outcomes of a physician-led continuous quality improvement intervention in the delivery of percutaneous coronary intervention. Am J Manag Care. 2006;12(8):445–52.

Ade-Oshifogun JB, Dufelmeier T. Prevention and Management of Do not return notices: a quality improvement process for Supplemental staffing nursing agencies. Nurs Forum. 2012;47(2):106–12.

Rubenstein L, Khodyakov D, Hempel S, Danz M, Salem-Schatz S, Foy R, et al. How can we recognize continuous quality improvement? Int J Qual Health Care. 2014;26(1):6–15.

O’Neill SM, Hempel S, Lim YW, Danz MS, Foy R, Suttorp MJ, et al. Identifying continuous quality improvement publications: what makes an improvement intervention ‘CQI’? BMJ Qual Saf. 2011;20(12):1011–9.

Article   PubMed   PubMed Central   Google Scholar  

Sibthorpe B, Gardner K, McAullay D. Furthering the quality agenda in Aboriginal community controlled health services: understanding the relationship between accreditation, continuous quality improvement and national key performance indicator reporting. Aust J Prim Health. 2016;22(4):270–5.

Bennett CL, Crane JM. Quality improvement efforts in oncology: are we ready to begin? Cancer Invest. 2001;19(1):86–95.

VanValkenburgh DA. Implementing continuous quality improvement at the facility level. Adv Ren Replace Ther. 2001;8(2):104–13.

Loper AC, Jensen TM, Farley AB, Morgan JD, Metz AJ. A systematic review of approaches for continuous quality improvement capacity-building. J Public Health Manage Practice. 2022;28(2):E354–361.

Ryan M. Achieving and sustaining quality in healthcare. Front Health Serv Manag. 2004;20(3):3–11.

Nicolucci A, Allotta G, Allegra G, Cordaro G, D’Agati F, Di Benedetto A, et al. Five-year impact of a continuous quality improvement effort implemented by a network of diabetes outpatient clinics. Diabetes Care. 2008;31(1):57–62.

Wakefield BJ, Blegen MA, Uden-Holman T, Vaughn T, Chrischilles E, Wakefield DS. Organizational culture, continuous quality improvement, and medication administration error reporting. Am J Med Qual. 2001;16(4):128–34.

Sori DA, Debelew GT, Degefa LS, Asefa Z. Continuous quality improvement strategy for increasing immediate postpartum long-acting reversible contraceptive use at Jimma University Medical Center, Jimma, Ethiopia. BMJ Open Qual. 2023;12(1):e002051.

Roche B, Robin C, Deleaval PJ, Marti MC. Continuous quality improvement in ambulatory surgery: the non-attending patient. Ambul Surg. 1998;6(2):97–100.

O’Connor JB, Sondhi SS, Mullen KD, McCullough AJ. A continuous quality improvement initiative reduces inappropriate prescribing of prophylactic antibiotics for endoscopic procedures. Am J Gastroenterol. 1999;94(8):2115–21.

Ursu A, Greenberg G, McKee M. Continuous quality improvement methodology: a case study on multidisciplinary collaboration to improve chlamydia screening. Fam Med Community Health. 2019;7(2):e000085.

Quick B, Nordstrom S, Johnson K. Using continuous quality improvement to implement evidence-based medicine. Lippincotts Case Manag. 2006;11(6):305–15 ( quiz 16 – 7 ).

Oyeledun B, Phillips A, Oronsaye F, Alo OD, Shaffer N, Osibo B, et al. The effect of a continuous quality improvement intervention on retention-in-care at 6 months postpartum in a PMTCT Program in Northern Nigeria: results of a cluster randomized controlled study. J Acquir Immune Defic Syndr. 2017;75(Suppl 2):S156–164.

Nyengerai T, Phohole M, Iqaba N, Kinge CW, Gori E, Moyo K, et al. Quality of service and continuous quality improvement in voluntary medical male circumcision programme across four provinces in South Africa: longitudinal and cross-sectional programme data. PLoS ONE. 2021;16(8):e0254850.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Wang J, Zhang H, Liu J, Zhang K, Yi B, Liu Y, et al. Implementation of a continuous quality improvement program reduces the occurrence of peritonitis in PD. Ren Fail. 2014;36(7):1029–32.

Stikes R, Barbier D. Applying the plan-do-study-act model to increase the use of kangaroo care. J Nurs Manag. 2013;21(1):70–8.

Wagner AD, Mugo C, Bluemer-Miroite S, Mutiti PM, Wamalwa DC, Bukusi D, et al. Continuous quality improvement intervention for adolescent and young adult HIV testing services in Kenya improves HIV knowledge. AIDS. 2017;31(Suppl 3):S243–252.

Le RD, Melanson SE, Santos KS, Paredes JD, Baum JM, Goonan EM, et al. Using lean principles to optimise inpatient phlebotomy services. J Clin Pathol. 2014;67(8):724–30.

Manyazewal T, Mekonnen A, Demelew T, Mengestu S, Abdu Y, Mammo D, et al. Improving immunization capacity in Ethiopia through continuous quality improvement interventions: a prospective quasi-experimental study. Infect Dis Poverty. 2018;7:7.

Kamiya Y, Ishijma H, Hagiwara A, Takahashi S, Ngonyani HAM, Samky E. Evaluating the impact of continuous quality improvement methods at hospitals in Tanzania: a cluster-randomized trial. Int J Qual Health Care. 2017;29(1):32–9.

Kibbe DC, Bentz E, McLaughlin CP. Continuous quality improvement for continuity of care. J Fam Pract. 1993;36(3):304–8.

Adrawa N, Ongiro S, Lotee K, Seret J, Adeke M, Izudi J. Use of a context-specific package to increase sputum smear monitoring among people with pulmonary tuberculosis in Uganda: a quality improvement study. BMJ Open Qual. 2023;12(3):1–6.

Hunt P, Hunter SB, Levan D. Continuous quality improvement in substance abuse treatment facilities: how much does it cost? J Subst Abuse Treat. 2017;77:133–40.

Azadeh A, Ameli M, Alisoltani N, Motevali Haghighi S. A unique fuzzy multi-control approach for continuous quality improvement in a radio therapy department. Qual Quantity. 2016;50(6):2469–93.

Memiah P, Tlale J, Shimabale M, Nzyoka S, Komba P, Sebeza J, et al. Continuous quality improvement (CQI) institutionalization to reach 95:95:95 HIV targets: a multicountry experience from the Global South. BMC Health Serv Res. 2021;21(1):711.

Yapa HM, De Neve JW, Chetty T, Herbst C, Post FA, Jiamsakul A, et al. The impact of continuous quality improvement on coverage of antenatal HIV care tests in rural South Africa: results of a stepped-wedge cluster-randomised controlled implementation trial. PLoS Med. 2020;17(10):e1003150.

Dadi TL, Abebo TA, Yeshitla A, Abera Y, Tadesse D, Tsegaye S, et al. Impact of quality improvement interventions on facility readiness, quality and uptake of maternal and child health services in developing regions of Ethiopia: a secondary analysis of programme data. BMJ Open Qual. 2023;12(4):e002140.

Weinberg M, Fuentes JM, Ruiz AI, Lozano FW, Angel E, Gaitan H, et al. Reducing infections among women undergoing cesarean section in Colombia by means of continuous quality improvement methods. Arch Intern Med. 2001;161(19):2357–65.

Andreoni V, Bilak Y, Bukumira M, Halfer D, Lynch-Stapleton P, Perez C. Project management: putting continuous quality improvement theory into practice. J Nurs Care Qual. 1995;9(3):29–37.

Balfour ME, Zinn TE, Cason K, Fox J, Morales M, Berdeja C, et al. Provider-payer partnerships as an engine for continuous quality improvement. Psychiatric Serv. 2018;69(6):623–5.

Agurto I, Sandoval J, De La Rosa M, Guardado ME. Improving cervical cancer prevention in a developing country. Int J Qual Health Care. 2006;18(2):81–6.

Anderson CI, Basson MD, Ali M, Davis AT, Osmer RL, McLeod MK, et al. Comprehensive multicenter graduate surgical education initiative incorporating entrustable professional activities, continuous quality improvement cycles, and a web-based platform to enhance teaching and learning. J Am Coll Surg. 2018;227(1):64–76.

Benjamin S, Seaman M. Applying continuous quality improvement and human performance technology to primary health care in Bahrain. Health Care Superv. 1998;17(1):62–71.

Byabagambi J, Marks P, Megere H, Karamagi E, Byakika S, Opio A, et al. Improving the quality of voluntary medical male circumcision through use of the continuous quality improvement approach: a pilot in 30 PEPFAR-Supported sites in Uganda. PLoS ONE. 2015;10(7):e0133369.

Hogg S, Roe Y, Mills R. Implementing evidence-based continuous quality improvement strategies in an urban Aboriginal Community Controlled Health Service in South East Queensland: a best practice implementation pilot. JBI Database Syst Rev Implement Rep. 2017;15(1):178–87.

Hopper MB, Morgan S. Continuous quality improvement initiative for pressure ulcer prevention. J Wound Ostomy Cont Nurs. 2014;41(2):178–80.

Ji J, Jiang DD, Xu Z, Yang YQ, Qian KY, Zhang MX. Continuous quality improvement of nutrition management during radiotherapy in patients with nasopharyngeal carcinoma. Nurs Open. 2021;8(6):3261–70.

Chen M, Deng JH, Zhou FD, Wang M, Wang HY. Improving the management of anemia in hemodialysis patients by implementing the continuous quality improvement program. Blood Purif. 2006;24(3):282–6.

Reeves S, Matney K, Crane V. Continuous quality improvement as an ideal in hospital practice. Health Care Superv. 1995;13(4):1–12.

Barton AJ, Danek G, Johns P, Coons M. Improving patient outcomes through CQI: vascular access planning. J Nurs Care Qual. 1998;13(2):77–85.

Buttigieg SC, Gauci D, Dey P. Continuous quality improvement in a Maltese hospital using logical framework analysis. J Health Organ Manag. 2016;30(7):1026–46.

Take N, Byakika S, Tasei H, Yoshikawa T. The effect of 5S-continuous quality improvement-total quality management approach on staff motivation, patients’ waiting time and patient satisfaction with services at hospitals in Uganda. J Public Health Afr. 2015;6(1):486.

PubMed   PubMed Central   Google Scholar  

Jacobson GH, McCoin NS, Lescallette R, Russ S, Slovis CM. Kaizen: a method of process improvement in the emergency department. Acad Emerg Med. 2009;16(12):1341–9.

Agarwal S, Gallo J, Parashar A, Agarwal K, Ellis S, Khot U, et al. Impact of lean six sigma process improvement methodology on cardiac catheterization laboratory efficiency. Catheter Cardiovasc Interv. 2015;85:S119.

Rahul G, Samanta AK, Varaprasad G A Lean Six Sigma approach to reduce overcrowding of patients and improving the discharge process in a super-specialty hospital. In 2020 International Conference on System, Computation, Automation and Networking (ICSCAN) 2020 July 3 (pp. 1-6). IEEE

Patel J, Nattabi B, Long R, Durey A, Naoum S, Kruger E, et al. The 5 C model: A proposed continuous quality improvement framework for volunteer dental services in remote Australian Aboriginal communities. Community Dent Oral Epidemiol. 2023;51(6):1150–8.

Van Acker B, McIntosh G, Gudes M. Continuous quality improvement techniques enhance HMO members’ immunization rates. J Healthc Qual. 1998;20(2):36–41.

Horine PD, Pohjala ED, Luecke RW. Healthcare financial managers and CQI. Healthc Financ Manage. 1993;47(9):34.

Reynolds JL. Reducing the frequency of episiotomies through a continuous quality improvement program. CMAJ. 1995;153(3):275–82.

Bunik M, Galloway K, Maughlin M, Hyman D. First five quality improvement program increases adherence and continuity with well-child care. Pediatr Qual Saf. 2021;6(6):e484.

Boyle TA, MacKinnon NJ, Mahaffey T, Duggan K, Dow N. Challenges of standardized continuous quality improvement programs in community pharmacies: the case of SafetyNET-Rx. Res Social Adm Pharm. 2012;8(6):499–508.

Price A, Schwartz R, Cohen J, Manson H, Scott F. Assessing continuous quality improvement in public health: adapting lessons from healthcare. Healthc Policy. 2017;12(3):34–49.

Gage AD, Gotsadze T, Seid E, Mutasa R, Friedman J. The influence of continuous quality improvement on healthcare quality: a mixed-methods study from Zimbabwe. Soc Sci Med. 2022;298:114831.

Chan YC, Ho SJ. Continuous quality improvement: a survey of American and Canadian healthcare executives. Hosp Health Serv Adm. 1997;42(4):525–44.

Balas EA, Puryear J, Mitchell JA, Barter B. How to structure clinical practice guidelines for continuous quality improvement? J Med Syst. 1994;18(5):289–97.

ElChamaa R, Seely AJE, Jeong D, Kitto S. Barriers and facilitators to the implementation and adoption of a continuous quality improvement program in surgery: a case study. J Contin Educ Health Prof. 2022;42(4):227–35.

Candas B, Jobin G, Dubé C, Tousignant M, Abdeljelil A, Grenier S, et al. Barriers and facilitators to implementing continuous quality improvement programs in colonoscopy services: a mixed methods systematic review. Endoscopy Int Open. 2016;4(2):E118–133.

Brandrud AS, Schreiner A, Hjortdahl P, Helljesen GS, Nyen B, Nelson EC. Three success factors for continual improvement in healthcare: an analysis of the reports of improvement team members. BMJ Qual Saf. 2011;20(3):251–9.

Lee S, Choi KS, Kang HY, Cho W, Chae YM. Assessing the factors influencing continuous quality improvement implementation: experience in Korean hospitals. Int J Qual Health Care. 2002;14(5):383–91.

Horwood C, Butler L, Barker P, Phakathi S, Haskins L, Grant M, et al. A continuous quality improvement intervention to improve the effectiveness of community health workers providing care to mothers and children: a cluster randomised controlled trial in South Africa. Hum Resour Health. 2017;15(1):39.

Hyrkäs K, Lehti K. Continuous quality improvement through team supervision supported by continuous self-monitoring of work and systematic patient feedback. J Nurs Manag. 2003;11(3):177–88.

Akdemir N, Peterson LN, Campbell CM, Scheele F. Evaluation of continuous quality improvement in accreditation for medical education. BMC Med Educ. 2020;20(Suppl 1):308.

Barzansky B, Hunt D, Moineau G, Ahn D, Lai CW, Humphrey H, et al. Continuous quality improvement in an accreditation system for undergraduate medical education: benefits and challenges. Med Teach. 2015;37(11):1032–8.

Gaylis F, Nasseri R, Salmasi A, Anderson C, Mohedin S, Prime R, et al. Implementing continuous quality improvement in an integrated community urology practice: lessons learned. Urology. 2021;153:139–46.

Gaga S, Mqoqi N, Chimatira R, Moko S, Igumbor JO. Continuous quality improvement in HIV and TB services at selected healthcare facilities in South Africa. South Afr J HIV Med. 2021;22(1):1202.

Wang F, Yao D. Application effect of continuous quality improvement measures on patient satisfaction and quality of life in gynecological nursing. Am J Transl Res. 2021;13(6):6391–8.

Lee SB, Lee LL, Yeung RS, Chan J. A continuous quality improvement project to reduce medication error in the emergency department. World J Emerg Med. 2013;4(3):179–82.

Chiang AA, Lee KC, Lee JC, Wei CH. Effectiveness of a continuous quality improvement program aiming to reduce unplanned extubation: a prospective study. Intensive Care Med. 1996;22(11):1269–71.

Chinnaiyan K, Al-Mallah M, Goraya T, Patel S, Kazerooni E, Poopat C, et al. Impact of a continuous quality improvement initiative on appropriate use of coronary CT angiography: results from a multicenter, statewide registry, the advanced cardiovascular imaging consortium (ACIC). J Cardiovasc Comput Tomogr. 2011;5(4):S29–30.

Gibson-Helm M, Rumbold A, Teede H, Ranasinha S, Bailie R, Boyle J. A continuous quality improvement initiative: improving the provision of pregnancy care for Aboriginal and Torres Strait Islander women. BJOG: Int J Obstet Gynecol. 2015;122:400–1.

Bennett IM, Coco A, Anderson J, Horst M, Gambler AS, Barr WB, et al. Improving maternal care with a continuous quality improvement strategy: a report from the interventions to minimize preterm and low birth weight infants through continuous improvement techniques (IMPLICIT) network. J Am Board Fam Med. 2009;22(4):380–6.

Krall SP, Iv CLR, Donahue L. Effect of continuous quality improvement methods on reducing triage to thrombolytic interval for Acute myocardial infarction. Acad Emerg Med. 1995;2(7):603–9.

Swanson TK, Eilers GM. Physician and staff acceptance of continuous quality improvement. Fam Med. 1994;26(9):583–6.

Yu Y, Zhou Y, Wang H, Zhou T, Li Q, Li T, et al. Impact of continuous quality improvement initiatives on clinical outcomes in peritoneal dialysis. Perit Dial Int. 2014;34(Suppl 2):S43–48.

Schiff GD, Goldfield NI. Deming meets Braverman: toward a progressive analysis of the continuous quality improvement paradigm. Int J Health Serv. 1994;24(4):655–73.

American Hospital Association Division of Quality Resources Chicago, IL: The role of hospital leadership in the continuous improvement of patient care quality. American Hospital Association. J Healthc Qual. 1992;14(5):8–14,22.

Scriven M. The Logic and Methodology of checklists [dissertation]. Western Michigan University; 2000.

Hales B, Terblanche M, Fowler R, Sibbald W. Development of medical checklists for improved quality of patient care. Int J Qual Health Care. 2008;20(1):22–30.

Vermeir P, Vandijck D, Degroote S, Peleman R, Verhaeghe R, Mortier E, et al. Communication in healthcare: a narrative review of the literature and practical recommendations. Int J Clin Pract. 2015;69(11):1257–67.

Eljiz K, Greenfield D, Hogden A, Taylor R, Siddiqui N, Agaliotis M, et al. Improving knowledge translation for increased engagement and impact in healthcare. BMJ open Qual. 2020;9(3):e000983.

O’Brien JL, Shortell SM, Hughes EF, Foster RW, Carman JM, Boerstler H, et al. An integrative model for organization-wide quality improvement: lessons from the field. Qual Manage Healthc. 1995;3(4):19–30.

Adily A, Girgis S, D’Este C, Matthews V, Ward JE. Syphilis testing performance in Aboriginal primary health care: exploring impact of continuous quality improvement over time. Aust J Prim Health. 2020;26(2):178–83.

Horwood C, Butler L, Barker P, Phakathi S, Haskins L, Grant M, et al. A continuous quality improvement intervention to improve the effectiveness of community health workers providing care to mothers and children: a cluster randomised controlled trial in South Africa. Hum Resour Health. 2017;15:1–11.

Veillard J, Cowling K, Bitton A, Ratcliffe H, Kimball M, Barkley S, et al. Better measurement for performance improvement in low- and middle-income countries: the primary Health Care Performance Initiative (PHCPI) experience of conceptual framework development and indicator selection. Milbank Q. 2017;95(4):836–83.

Barbazza E, Kringos D, Kruse I, Klazinga NS, Tello JE. Creating performance intelligence for primary health care strengthening in Europe. BMC Health Serv Res. 2019;19(1):1006.

Assefa Y, Hill PS, Gilks CF, Admassu M, Tesfaye D, Van Damme W. Primary health care contributions to universal health coverage. Ethiopia Bull World Health Organ. 2020;98(12):894.

Van Weel C, Kidd MR. Why strengthening primary health care is essential to achieving universal health coverage. CMAJ. 2018;190(15):E463–466.

Download references

Acknowledgements

Not applicable.

The authors received no fund.

Author information

Authors and affiliations.

School of Public Health, The University of Queensland, Brisbane, Australia

Aklilu Endalamaw, Resham B Khatri, Tesfaye Setegn Mengistu, Daniel Erku & Yibeltal Assefa

College of Medicine and Health Sciences, Bahir Dar University, Bahir Dar, Ethiopia

Aklilu Endalamaw & Tesfaye Setegn Mengistu

Health Social Science and Development Research Institute, Kathmandu, Nepal

Resham B Khatri

Centre for Applied Health Economics, School of Medicine, Grifth University, Brisbane, Australia

Daniel Erku

Menzies Health Institute Queensland, Grifth University, Brisbane, Australia

International Institute for Primary Health Care in Ethiopia, Addis Ababa, Ethiopia

Eskinder Wolka & Anteneh Zewdie

You can also search for this author in PubMed   Google Scholar

Contributions

AE conceptualized the study, developed the first draft of the manuscript, and managing feedbacks from co-authors. YA conceptualized the study, provided feedback, and supervised the whole processes. RBK provided feedback throughout. TSM provided feedback throughout. DE provided feedback throughout. EW provided feedback throughout. AZ provided feedback throughout. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Aklilu Endalamaw .

Ethics declarations

Ethics approval and consent to participate.

Not applicable because this research is based on publicly available articles.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Endalamaw, A., Khatri, R.B., Mengistu, T.S. et al. A scoping review of continuous quality improvement in healthcare system: conceptualization, models and tools, barriers and facilitators, and impact. BMC Health Serv Res 24 , 487 (2024). https://doi.org/10.1186/s12913-024-10828-0

Download citation

Received : 27 December 2023

Accepted : 05 March 2024

Published : 19 April 2024

DOI : https://doi.org/10.1186/s12913-024-10828-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Continuous quality improvement
• Quality of Care

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

ORIGINAL RESEARCH article

Impact of the covid-19 pandemic on the work of clinical psychologists in austria: results of a mixed-methods study.

• 1 Department for Psychosomatic Medicine and Psychotherapy, University of Continuing Education Krems, Krems an der Donau, Austria
• 2 Faculty of Psychotherapy Science, Sigmund Freud University, Vienna, Austria
• 3 Division of Psychotherapy, Department of Psychology, Paris Lodron University of Salzburg, Salzburg, Austria
• 4 Division of Pediatric Pulmonology, Allergology and Endocrinology, Department of Pediatrics and Adolescent Medicine, Medical University of Vienna, Vienna, Austria
• 5 Department of Child and Adolescent Psychiatry, Comprehensive Center for Pediatrics, Medical University of Vienna, Vienna, Austria
• 6 Österreichischer Arbeitskreis für Gruppentherapie und Gruppendynamik, Fachsektion Integrative Gestalt Therapy Vienna, Vienna, Austria

Introduction: Clinical psychologists in Austria shouldered a large part of the massive increase in demand for mental health services caused by the COVID-19 pandemic. This study aimed to find out how the pandemic affected their work and to gather information on how best to support the profession in the event of a crisis.

Methods: N = 172 Austrian clinical psychologists participated in a cross-sectional online survey between 11 April 2022 and 31 May 2022, including both closed and open-ended questions about their work. Open-ended questions were analyzed using qualitative content analysis. A mixed-methods analysis was conducted to test correlations between the categories derived from the qualitative analysis and professional variables.

Results: The analyses revealed that clinical psychologists, especially those with more years of experience, perceived an increased need for clinical psychological treatment, especially for children and adolescents, a lack of coverage for clinical psychological treatment by health insurance, a change to remote treatment formats, and a number of burdens associated with complying with COVID-19 measures.

Discussion: Clinical psychologists reported an urgent need to increase resources in both outpatient and inpatient settings and to promote health insurance coverage. To support the clinical psychology profession in providing high-quality work in times of crisis, there is a need to facilitate more opportunities for team and peer exchange, as well as financial support in the event of loss of income.

1 Introduction

SARS-CoV-2 (severe acute respiratory syndrome coronavirus type 2) is a new coronavirus identified in early 2020 as the causative agent of COVID-19. The main transmission route of SARS-CoV-2 is the respiratory ingestion of virus-containing particles produced by breathing, coughing, talking, singing, and sneezing ( Wang et al., 2021 ). In March 2020, the World Health Organization (WHO) declared the infection a pandemic due to the high number of cases, the many deaths, and the large number of countries affected. At the WHO press conference on 11 March 2020, the following points were identified as important regarding COVID-19: Prevention, preparedness, public health, and political leadership. Particular emphasis was placed on the call for the general population to protect each other from the virus ( World Health Organization, 2020 ).

Between March and May 2020, the Austrian government declared restrictive measures that limited freedom and social life to reduce the risk of new infections in the population and, at the same time, protected the health system from possible overload ( Pollak et al., 2020 ). In addition to preventive hygiene measures such as washing hands or sneezing into the crook of the arm, the Austrian government imposed a lockdown with only a few exceptions: Shopping for food or medicine, helping others, going to work, or going for a walk. Individuals were called upon to keep a distance of 2 m from others. Meeting other people who did not live in the same household was generally forbidden. The curfew also meant the closure of shops, schools, and cultural institutions.

With the introduction of the COVID-19 vaccination in Austria in December 2020 and declining infection rates between May and July 2021, COVID-19 restrictions could be relaxed. However, in late summer and fall of 2021, a new variant of the virus spread. Infection numbers rose rapidly, reaching a record high in November 2021 ( Pollak et al., 2021 ). ICU bed occupancy also increased sharply, prompting the federal government to impose another lockdown until January 2022. In February 2022, compulsory vaccination officially went into effect ( Walcherberger et al., 2023 ). Despite high infection numbers, COVID-19 measures for the general population in Austria were relaxed in March 2022 ( Walcherberger et al., 2023 ).

Research has shown that the pandemic and the measures taken to contain the virus have had a negative impact on mental health worldwide. Increases in the prevalence of symptoms of depression, anxiety, insomnia, stress, substance abuse, and eating disorders have been reported ( Roberts et al., 2021 ; Sideli et al., 2021 ; Mahmud et al., 2023 ). Fears of illness, reduced social contact, and financial concerns have been identified as underlying factors in mental health deterioration ( Xiong et al., 2020 ). Feelings of loneliness due to self-isolation have often been cited as a cause of mental health problems ( Killgore et al., 2020 ) and are positively correlated with symptoms of anxiety, depression, and stress ( Gu et al., 2021 ).

In Austria, a study conducted early in the pandemic showed an increase in symptoms of depression, anxiety, and insomnia compared with pre-COVID-19 levels ( Pieh et al., 2020 ). A follow-up study showed that the adverse health effects of the COVID-19 pandemic persisted for several months after the outbreak and the end of the lockdown measures ( Pieh et al., 2021 ). In April 2022, the mental health burden was still high, with rates of depressive symptoms continuing to rise ( Humer et al., 2022 ). Younger adults (<35 years old) and people with a low income (<€ 2,000 net household income per month) were particularly affected by the deterioration in mental health ( Humer et al., 2022 ). Austria is not an isolated case. Studies conducted in countries with high COVID-19 cases and death rates, such as Italy, Spain, the United States and the UK, have shown an increase in depression, anxiety, insomnia and PTSD symptoms among the general population ( Budimir et al., 2021 ). A comparison of severe mental health symptoms in the UK and Austria during the COVID-19 lockdown revealed a higher prevalence of severe depression, anxiety, and insomnia symptoms in the UK than in Austria ( Budimir et al., 2021 ). Vulnerability factors that influence the impact of the COVID-19 pandemic on the mental health of populations were identified in both British and Austrian populations. Signs of vulnerability encompass poverty, existing mental health conditions, and inadequate social support. Women, children, young adults (aged 18–30), the unemployed, and individuals living alone appear to be the most affected groups ( Li and Wang, 2020 ; Simon et al., 2021 ).

Mental health services are urgently needed to be able to professionally counter the negative consequences of the pandemic on the mental health of the population in Austria. Clinical psychological treatment exists on an equal footing with other curative procedures for treating mental disorders and states of suffering, such as psychiatric treatment or psychotherapy. While health insurance funds reimburse all or part of the costs of psychotherapy or psychiatric treatment, clinical psychological treatment and diagnostics costs are covered only in the institutional sector ( Berufsverband Österreicher PsychologInnen, 2023 ). Individuals seeking to become clinical psychologists in Austria, must first complete a university degree in psychology, which entails 300 ECTS credits. After finishing the 5-year master’s program in psychology, aspiring clinicians must then undertake 1–2 years of postgraduate specialized training in both theoretical approaches and practical techniques. When all these requirements have been fulfilled, then registration on the Ministry’s list of clinical psychologists is completed and the license to practice as a clinical psychologist is granted ( Berufsverband Österreicher PsychologInnen, 2023 ). Health insurance providers finance clinical psychological services like assessment and therapy when rendered in institutional settings. However, in private practice, only diagnostic evaluations are covered, while treatment and counseling interventions must be paid for out-of-pocket by patients. No subsidies or cost coverage exist from health insurance for clinical psychologists’ private therapeutic practice. Only in the case of acute stress associated with psychological trauma, up to 10 h of clinical psychological crisis support and treatment are covered by the Federal Social Office according to the Crime Victims Act ( Öffentliches Gesundheitsportal Österreichs, 2019 ; Bundeministerium für Soziales, Gesundheit, Pflege und Konsumentenschutz, 2023 ). Ongoing negotiations are being conducted between the professional body representing clinical psychologists and the primary governing bodies overseeing Austrian social insurance institutions, along with the Federal Ministry of Health. The objective of these negotiations is to integrate clinical psychological treatment into the standard care framework, thereby recognizing it as a covered service under the statutory health and social insurance schemes. This development aims to establish clinical psychological interventions as a mandated benefit, whereby individuals insured under the public health and social security systems would have access to such therapeutic services as an entitlement, without incurring additional out-of-pocket expenses ( Berufsverband Österreicher PsychologInnen, 2023 ).

The measures used to contain the SARS-CoV-2 have massively changed the work and the working conditions in the health and social sectors. Although face-to-face treatments were still permitted in Austria despite lockdown measures, the practice of clinical psychology in Austria was nonetheless affected by the lockdown measures. In May 2020, the Professional Association of Austrian Psychologists published the COVID-19 Guidelines and Fact Sheet, which contained recommendations for clinical psychological work in independent practice ( Berufsverband Österreicher PsychologInnen, 2020 ). In addition to appropriate hygiene or cleaning and disinfection measures, such as paying attention to hand hygiene (no shaking, regular hand washing and/or disinfection) or wearing protective masks, the Federal Ministry of Social Affairs, Health and Consumer Protection also recommended that consultations and treatments be carried out via the internet and/or telephone for patients at high health risk ( Berufsverband Österreicher PsychologInnen, 2020 ). The decision of whether the use of online therapy instead of face-to-face contact would also be suitable in other cases was left to the practitioner ( Berufsverband Österreicher PsychologInnen, 2020 ). The COVID-19 measures in the health services sector remained in place in 2022 despite the relaxation of COVID-19 measures within the general population ( Berufsverband Österreicher PsychologInnen, 2020 ).

As the health care system in general, and the mental health care system in particular, faced significant challenges due to the pandemic, it is important to investigate how clinical psychologists experienced the changes and challenges in carrying out their professional work. To provide quality clinical psychological treatment, it is essential to examine the conditions of this treatment and the burdens and needs of those providing the treatment. For this reason, the present study collected data on the psychological well-being of clinical psychologists during the pandemic and investigated which factors they found stressful and what helped them to cope with stress. It also looked at how clinical psychologists perceived changes in their work and what support they needed. Some of the results have already been published. Humer et al. (2023) showed that clinical psychologists had better mental health than the general population during the COVID-19 pandemic, suggesting that they experienced less stress or had more coping strategies to deal with stress. This question was addressed in another publication based on this study by Jesser et al. (2024) , which looked closer at the burdens and resources of clinical psychologists. The manuscript at hand presents different results than the one published by Humer et al. (2023) and Jesser et al. (2024) although using the same sample. The main research questions (RQs) of the current study are as follows:

RQ 1: Qualitative analysis: What was the impact of the COVID-19 pandemic on the work of clinical psychologists? RQ 2: Qualitative analysis: What support do clinical psychologists wish for during the COVID-19 pandemic? RQ 3: Mixed-methods analysis: Are there statistically significant differences between different subgroups of clinical psychologists in the frequencies of reporting in the main and subcategories resulting from RQ 1 and RQ 2?

2 Materials and methods

2.1 study design.

An internet-based cross-sectional survey was conducted between 11 April and 31 May 2022 using Research Electronic Data Capture (REDCap) (Vanderbilt University, Nashville, TN, USA) ( Harris et al., 2019 ). The survey consisted of a total of 49 items. The link to the survey was sent via e-mail to all clinical psychologists registered in the list of the Austrian Federal Ministry of Social Affairs, Health, Care and Consumer Protection (>11,000 clinical psychologists registered in April 2022), provided they had given a valid e-mail address (≈5,000 clinical psychologists). Parallel to this study, a survey on the same topic was conducted among Austrian psychotherapists. As some respondents ( n  = 139) were registered as both clinical psychologists and psychotherapists, only those registered as clinical psychologists ( n  = 172) were used in this study. The results of the survey among psychotherapists are published elsewhere ( Winter et al., 2023 ). Among the ≈ 5,000 clinical psychologists that could be contacted, around 3,000 were not registered as psychotherapists.

The study was conducted after approval by the data protection officer and the ethics committee of the University of Continuing Education Krems, Austria (ethics number: EK GZ 11/2021–2024). All participating clinical psychologists provided electronic consent to participate. The clinical psychologists were not compensated for their time and effort, and participation was voluntary.

2.2 Measures

All questions used in the online survey for this study are summarized in Supplementary File 2: Table S1 .

2.2.1 Socio-demographic and occupational variables

Data was collected on gender, age, region, and years of employment (the time since the participants were registered in the official list of licensed clinical psychologists). All participants were asked about the type of employment (private practice, outpatient facility, inpatient facility) and whether they received all their income from clinical psychological treatments. They were also asked about the number of patients they provided clinical psychological treatment to on average per week via face-to-face, internet and telephone at the time of the survey. Other occupational variables were the group of patients treated (children and adolescents, adults) and the setting where the treatment occurred (treatment of individuals, partners, families, or groups).

2.2.2 Open-ended questions

To assess the perceived impact of the pandemic on the work of a clinical psychologist, the following open-ended question was asked:

1. What direct or indirect effects did the pandemic have on your work as a clinical psychologist?

A closed and an open-ended question was asked to assess the perceived need for support in working as a clinical psychologist. The questions were as follows:

1. Would you wish support concerning your professional activity as a clinical psychologist? Alternative answer: Yes/No

2. What support would you wish for your professional activity as a clinical psychologist?

Both the questions and the answers were given in German. For the open-ended questions, there were no predefined answer options; respondents were free to describe their experiences in their own words. The answers ranged from one-word answers to whole paragraphs. It was also possible not to fill in the answer field and to skip any of the open-ended questions.

2.3 Data analysis

Socio-demographic and work-related data were analyzed descriptively to describe the socio-demographic and occupational characteristics of the sample.

Responses to the open-ended questions were analyzed using conventional qualitative content analysis ( Hsieh and Shannon, 2005 ) to describe the perceived impact of the pandemic on the work of clinical psychologists and the type of support they desire. All the data was first read by the first author to familiarize herself with the material and gain an overview of all the answers. The answers were read word by word several times. In the process, the categories for the open-ended questions were derived inductively, and category definitions, coding rules and example quotations were documented in a codebook. Subsequently, subcategories similar in content were subsumed into more conceptual main categories. In the next step, the data set was coded with the category list using the software ATLAS.ti ( ATLAS.ti Scientific Software Development GmbH, 2023 ). ATLAS.ti facilitates qualitative data management through systematic organization, coding tools to mark and categorize themes, and mapping functions to diagram linkages between coded concepts. This aids researchers in working with and deriving meaning from unstructured data sources. The software aims to provide an integrated suite of tools for qualitative analysis workflows. Since respondents were free to mention several aspects per question, the assignment of more than one category per answer was possible. After the entire data set was coded, all quotations assigned to a category were read again to check for coding errors. Coding errors were corrected, and the category definitions and coding rules were clarified in case of inaccuracies.

In addition to the qualitative content analysis, multivariable logistic regression analyses were performed to determine the odds for psychologists reporting in the respective main or subcategories in relation to the assessed socio-demographic and occupational characteristics. The response within a main- or subcategory (no, yes) was the dependent variable. The socio-demographic and work-related variables functioned as predictors. The following work-related variables were included as predictors: patient group (adults, children and adolescents, both), setting [working with couples (no, yes), working with families (no, yes), working with groups (no, yes)], professional experience (<10 years, 10–19 years, ≥20 years), source of income (additional income, clinical psychological treatment as the only source of income), and form of employment as a clinical psychologist (private practice, institution, both). Socio-demographic variables included in the statistical analyses were: gender (female, male), age (<40 years, 40–49 years, ≥50 years), region (Eastern Austria, Southern Austria, Western Austria). The first subgroup mentioned for each predictor in brackets served as reference group in the statistical model. Adjusted odds ratios (aOR) were assessed and the significance level was set at 0.05. Statistical analyses were conducted using SPSS version 26 (IBM Corp., Armonk, NY, USA).

3.1 Characteristics of the sample

A total of 172 clinical psychologists participated in the study. Participants were 44.90 ± 7.97 years old and 91.9% female (compared to 85.1% in the list of licensed clinical psychologists). They had been in the profession for 13.91 ± 7.72 years (compared to 12.03 ± 6.91 years for all licensed clinical psychologists), and 74.4% worked in private practice only ( Table 1 ).

Table 1 . Study sample characteristics.

3.2 Qualitative results

Of the N  = 172, 86.0% ( n  = 148) answered the first question about the impact of the pandemic on their work as a clinical psychologist. 41.9% ( n  = 72) answered YES to the closed question about the desire for support related to their professional work as a clinical psychologist, and 58.1% ( n  = 100) answered NO. 40.1% ( n  = 69) responded to the open-ended question about the support they would like in relation to their professional activity as a clinical psychologist.

Qualitative content analysis resulted in five main categories and 16 subcategories related to the question of the impact of the pandemic on the work of clinical psychologists (Question 1). Three main categories and ten subcategories were formed to address the psychologists’ wish for support (Question 3). All main and subcategories are shown in Tables 2 , 3 .

Table 2 . Percentage of respondents reporting each main category (in bold) and subcategory of changes in their work as a clinical psychologist in Austria due to the COVID-19 pandemic.

Table 3 . Percentages of respondents reporting each main category (in bold) and subcategory of support wishes.

3.2.1 Results for RQ 1: impact of the pandemic on work as a clinical psychologist

Main category 1: The largest main category, mentioned by 43.6% of clinical psychologists, is related to the change in the number of patients during the pandemic. This main category was divided into two subcategories. While 25.0% of participants reported an increase in patient numbers, 22.1% reported a loss of patients and a decrease in demand. It was noted that there was a decrease in demand, particularly at the beginning of the pandemic. However, over time, there was an increase in demand for clinical psychological treatment, particularly in private practice. Some clinical psychologists reported an increased need for clinical psychological treatment among children and adolescents. In inpatient and/or group settings, on the other hand, patients terminated treatment due to the COVID-19 measures. Increased cancellations and postponements of appointments were also mentioned. One participant (705) observed: “More cancellations at short notice, due to a positive COVID-19 test or contact person number one (K1) quarantine and lockdown.” Participants stated that patients were less reliable during the pandemic, making scheduling difficult.

Main category 2: The second most frequently mentioned main category, reported by 40.7% of respondents, was a change in the treatment setting. The main category comprises three subcategories. 25.6% of clinical psychologists reported a change to voice or video calls. Most statements within this subcategory conveyed a neutral attitude towards remote treatment. However, both advantages and disadvantages were mentioned. For example, one person (465) said: “More work via Zoom and phone - an advantage is that it is possible everywhere, a disadvantage is that I have to be more attentive otherwise details are lost more easily (especially body language).” Clinical psychologists also noted that in some cases, the introduction of voice and video calls was not well received by their patients, who preferred to continue with face-to-face treatment. For example, one clinical psychologist (626) stated: “My patients have reported being grateful to be able to return to the practice in person under safe circumstances.”

A further 14.5% of clinical psychologists mentioned wearing a mask as one of the effects of the pandemic on their work. Participants described wearing the mask as physically demanding and a barrier to building good relationships because of the loss of facial information. For example, one person (381) said: “Masks are annoying and create distance in conversations”; another participant (656) said: “Wearing a mask was sometimes distracting, e.g., when working out emotions.”

The use of disinfectants, air purification, wearing protective clothing, and increasing distance from patients, home office and outdoor settings were mentioned by 14.5% of respondents as additional pandemic containment strategies that directly affected the work of clinical psychologists.

Main category 3: 33.7% of clinical psychologists reported an impact of the pandemic on working conditions in both private practice and institutional settings. There were five subcategories within this main category. 23.8% of respondents noted an increased workload due to pandemic-related measures and changes - and thus an increased burden. The pandemic resulted in additional administrative and bureaucratic work and time pressure, leading to feelings of strain and overwork. One person ( Berufsverband Österreicher PsychologInnen, 2023 ) expressed: “Work more exhausting, confrontation with resignation.” Another person (491) observed: “High work pressure, frequent case requests, quick appointments are expected.” For another clinical psychologist, compliance with COVID-19 measures was associated with an increased workload. The fact that some patients did not want to comply with the measures was also perceived as a burden. Participants also discussed fear of infection at work.

6.4% of clinical psychologists reported experiencing a financial impact from the pandemic. Respondents experienced a loss of income, particularly at the beginning of the pandemic. One person (633) noted: “I opened my practice on 15 March 2020 (lockdown started on 16 March), i.e., at the beginning I only had costs and no income.” Clinical psychologists in the institutional sector stated that they had to accept a reduction in working hours and, thus, a reduction in their salary.

A further 5.8% of clinical psychologists said they missed the exchange with their colleagues as the COVID-19 measures limited team communication. The working atmosphere was described as tense. One participant (291) reported: “The working atmosphere is much more determined by guidelines and collegial stress.”

3.5% of clinical psychologists reported feeling insecure in the workplace due to repeated changes in COVID-19 regulations. In particular, they raised the issue of uncertainty and insecurity about how long new regulations and policies would be in place and how they would be managed. One participant (90) reported: “Uncertainty, anxiety about how to deal with the measures in practice.”

Another impact of the pandemic on working conditions was the need for more flexible availability (2.9%). One participant (373) reported: “Shifts due to quarantine of clients and their relatives required more flexibility than usual.” Positive aspects of flexible working were also mentioned. One respondent (465) commented that the spatial flexibility provided by remote treatment was an advantage at work.

Main category 4: Another main category, mentioned by 22.7% of respondents, relates to the impact of the pandemic on their work with patients. The main category was divided into three subcategories. 13.4% of clinical psychologists experienced difficulties establishing and maintaining good patient relationships. Reasons given included being unable to meet in person, wearing the mask, and introducing voice and video telephony into treatment. Clinical psychologists reported difficulties in communicating with their patients. One participant stated (281): “By wearing FFP2 masks, an important part of facial expression is lost, and this creates a feeling of not knowing the other person.” Another participant (691) described: “Many topics were lost because telephone conversations were much shorter, more superficial and less psychological.”

7.6% of clinical psychologists experienced a shift in the focus of treatment. They found that pandemic-related issues dominated their conversations with patients. As a result, other matters were pushed into the background. Stabilization, resilience and increasing patients’ resources were identified as the focus of psychological treatment during the pandemic. One participant (489) stated: “The pandemic took up a lot of space in therapy that was meant for the clients’ problems. The reduction in face-to-face contact and the use of the mask meant that clinical psychologists experienced more distance or less closeness between themselves and their patients, thus reducing the depth of treatment. they explained this, among other things, by the lack of opportunity to use psychological interventions on the phone and in video telephony or by the fact that the focus of the conversations was on COVID-19. One person (531) stated: “individual methods difficult to impossible to implement by telephone/online, therefore reduced variety of methods/offers; confidence-building measures for individual persons reduced, therefore less in-depth.”

Clinical psychologists also reported on the financial situation of their patients (4.1%). They stated that the pandemic had increased the need for psychological treatment but also worsened the financial situation of their patients. One person (200) reported: “Many requests from patients who cannot afford the treatment.”

Main category 5: Another main category mentioned by 20.3% of respondents was the perceived impact of the pandemic on patients’ mental health. This category was divided into three subcategories. 11.0% of clinical psychologists reported a deterioration in mental health. They observed an increase in the severity of symptoms and a more pronounced manifestation of symptoms, combined with bottlenecks in the admission of the patients to inpatient psychiatric care. For example, some respondents observed that patients had more varied, severe, and chronic symptoms. Crises in treatment were also more frequent during the pandemic. Another aspect related to the mental state of patients was the variety of distressing feelings mentioned by 9.3% of respondents; for example, insecurity and anxiety, withdrawal and loneliness were more frequently observed in patients. Finally, 8.1% of clinical psychologists specifically mentioned certain disorders they had seen more often than usual during the pandemic, including depression, impulse control disorders, eating disorders, post-COVID syndrome and anxiety disorders. One (245) observed: “Depression, eating disorders, compulsions, anxiety in children and adolescents.”

3.2.2 Results for RQ 2: wishes for support in clinical psychological work

Main category 1: The most frequently mentioned main category was the wish for support at a political and legislative level (19.8%). This main category was divided into three subcategories. 15.7% of clinical psychologists said they wanted to see clinical psychological treatment as a statutory service. 5.2% of clinical psychologists would like to see an expansion of institutional capacity and increase in staff in institutions. Respondents cited the need to expand outpatient and inpatient services for children and adolescents. For example, one respondent ( Sevecke et al., 2023 ) commented: “URGENT: expansion of child and adolescent psychiatry!!!!!!!!! And more services there. Capacities of all counselling centers should be increased [...].”1.2% of clinical psychologists wanted support from professional associations.

Main category 2: Another main category, mentioned by 18% of respondents, addresses the desire to network and share information. This main category was divided into four subcategories. 9.9% of clinical psychologists wished for more supervision and intervision during the pandemic. 5.8% of the participants wished for more team exchange within institutions or more networking and meetings between colleagues working in private practice. 2.9% of clinical psychologists wished for more online training and further education on topics related to the pandemic, for example, long-COVID. Finally, 2.9% of clinical psychologists wished for clear communication of guidelines and measures to prevent the spread of COVID-19. One person (373) reported: “Precise differentiation of measures for the practice and the institutional context.”

Main category 3: “Improving working conditions’ is another main category, mentioned by 16.3%. It is divided into three subcategories. 7.6% of respondents wanted support in terms of human and time resources. They hoped to feel less time pressure in doing their work. Clinical psychologists working in private practice indicated they would like more support in planning, organizing, and managing their work, as well as staff to help with it. Clinical psychologists reported that they would like more time off and holidays. A further 6.4% of clinical psychologists wanted financial support for training, supervision, and self-care. They also wanted to be paid more for their work and generally felt that social and health professionals should earn more. Finally, 4.0% of clinical psychologists said they would like more appreciation and recognition for their work. One respondent (732) stated: “Social recognition, systemic recognition.”

3.3 Results for RQ 3: mixed-methods analysis

3.3.1 impact of the pandemic on the work of clinical psychologists.

Working with different patient groups (adults, children, and adolescents) was associated with the odds for reporting effects of the pandemic on collaboration and working atmosphere. Clinical psychologists working with children and adolescents were more likely to report differences in this subcategory compared to those working solely with adults (children and adolescents vs. solely adults: aOR: 23.79, p  = 0.043; children, adolescents, and adults vs. solely adults: aOR: 29.34, p  = 0.021). Also, the likelihood of reporting a higher diversity of stressful feelings in their patients due to the pandemic was higher in clinical psychologists working with children and adolescents compared to those working exclusively with adults (aOR: 7.30, p  = 0.045).

The setting (partners, families, groups) was associated with the likelihood for reporting an impact of the pandemic on the work with patients. Clinical psychologist working with families compared to those providing no treatments to families were less likely to report in this main category (aOR: 0.249, p  = 0.026).

Significant findings concerning professional experience were observed for the likelihood of clinical psychologists reporting an impact of the pandemic on changes in the number of patients. Clinical psychologist with 10–19 years in the profession and those with at least 20 years in the profession had higher odds for reporting changes in the patient number than those with less than 10 years in the profession (10–19 years vs. <10 years, aOR: 4.24, p  = 0.003; ≥20 years vs. <10 years: aOR: 5.34, p  = 0.009). Similarly, being longer in the profession increased the likelihood of reporting an increase in patient numbers (10–19 years vs. <10 years, aOR: 4.81, p  = 0.009; ≥20 years vs. <10 years: aOR: 6.53, p  = 0.015).

Several differences related to the source of income were identified. Clinical psychologists relying solely on clinical psychological treatment as their source of income were more likely reported on a decrease in the number of patients than those with additional income sources (aOR: 2.53, p  = 0.044). Furthermore, they had a lower chance to report about other COVID-19 measures such as home office (aOR: 0.348, p  = 0.036). Clinical psychologists relying solely on clinical psychological treatment as their source of income more likely reported changes in their working conditions due to the pandemic than those with additional income sources (aOR: 2.23, p  = 0.041).

Differences according to the form of employment as clinical psychologists were found in the subcategory collaboration and working atmosphere , with clinical psychologists working in an institution showing higher odds for reporting changes in collaboration due to the pandemic than clinical psychologists working in private practice (aOR: 10.6, p  = 0.043).

Considering all assessed socio-demographic and work-related variables simultaneously in the statistical model revealed no differences in the likelihood of clinical psychologists reporting on the impact of the pandemic on their work regarding age.

The odds for reporting changes in collaboration and working atmosphere were higher in male then female clinical psychologists (aOR: 10.61, p  = 0.043).

The likelihood for reporting an impact of the pandemic on treatment setting was higher in Western Austria compared to Eastern Austria (aOR: 2.29, p  = 0.032). Similarly, clinical psychologist from Western Austria compared to Eastern Austria had higher odds for reporting an impact of the pandemic on the work with patients (aOR: 2.72, p  = 0.030). Changes in collaboration and working atmosphere were less likely to be reported by clinical psychologists from Western Austria (aOR: 0.060, p  = 0.034) and Southern Austria (aOR: 0.069, p  = 0.043) compared to Eastern Austria.

3.3.2 Support wishes of clinical psychologists

Differences in support wishes were found in the closed-ended question on the general wish for support (yes vs. no) regarding the patient groups the clinical psychologists were working with. Clinical psychologists working with children and adolescents were more likely to report that they wanted support compared to those working solely with adults (aOR: 5.71, p  = 0.005).

The setting (partners, families, groups) was associated with some differences. Clinical psychologists working with families were more likely to express a wish for support compared to those providing no treatments to families (aOR: 2.94, p  = 0.017). These psychologists were also more likely to wish for more human and time resources compared to those not working with families (aOR: 8.33, p  = 0.008). Clinical psychologists treating groups had higher odds for wishing more appreciation and recognition than those not working with groups (aOR: 11.26, p  = 0.032).

No difference in the support wishes regarding what type of support they desired was found with respect to professional experience , source of income, and occupational status .

Among the socio-demographic variables, no differences were observed with respect to gender and region , while age differences were observed for the likelihood to wish for more networking and sharing of information. More specifically, the likelihood for wishing more exchange was higher in clinical psychologists aged 40–49 (aOR: 6.43, p  = 0.016) and ≥50 years (aOR: 6.05, p  = 0.025) compared to those younger than 40 years.

4 Discussion

4.1 impact of the pandemic on the work of clinical psychologists – patient numbers and deterioration of mental health.

This study showed that clinical psychologists perceived changes in patient numbers and treatment settings as the most significant changes caused by the COVID-19 pandemic. Concerning patient numbers, it is crucial to consider a temporal aspect and distinguish the beginning from the later stages of the pandemic. Clinical psychologists found that the number of patients decreased at the beginning of the pandemic in spring 2020. Patients often cancelled appointments at short notice and were not always reliable. After the first lockdown in May 2020, clinical psychologists noticed a deterioration in their patients’ mental health and an increase in their need for clinical psychological treatment. These findings are consistent with the results of the study by Winter et al. (2023) . In their study, after a decrease in the total number of patients treated in the first weeks of the pandemic, an increase in the number of patients was observed in the second and third year of the pandemic. In the study by Winter et al. (2023) , the number of patients even exceeded pre-pandemic levels. In the current study, clinical psychologists with more professional experience were particularly in demand. They may have a more extensive network of contacts through years of practice or may have better referral channels.

In our study, clinical psychologists reported that their patients’ mental health deteriorated throughout the pandemic, with more frequent crises and worsening symptoms, which is consistent with the international research literature ( Nochaiwong et al., 2021 ; Leung et al., 2022 ; Mahmud et al., 2023 ). However, it would be short-sighted to attribute the deterioration in mental health solely to the effects of the pandemic and its measures. In 2022, new geopolitical crises such as the war in Ukraine, the energy crisis and related inflation came to the fore. In a qualitative study, Gächter et al. compared the concerns of a representative sample of the Austrian population at two different points in time (winter 2020/21 and spring 2022) ( Gächter et al., 2023 ). In 2020/21, the COVID-19 restrictions and their effects were the primary concerns. In 2022, the main concerns were inflation, finances, and the war in Ukraine. At both points in time, concerns about mental health remained high. In a survey conducted in Austria in June 2022 on fears about the consequences of the Russian-Ukrainian war, a total of 92% of respondents were very worried about higher food prices ( Mohr, 2022 ). This was the greatest fear associated with the conflict, followed by the fear of higher energy prices, reported by 90% ( Mohr, 2022 ).

4.2 Impact of the pandemic on the work of clinical psychologists – bottlenecks in the treatment of children and adolescents

The increase in demand for clinical psychological treatment was particularly marked among children and adolescents. Respondents working with children and adolescents reported a higher diversity of stressful feelings in their patients. Compared to their colleagues treating adult patients, psychologists working with children and adolescents more frequently reported changes in collaboration and working atmosphere, which may indicate an increased need for networking with other professionals and institutions in order to provide the necessary care for their patients, as well as exhaustion of psychologists due to the effort and difficult conditions, as also found by Fiala-Baumann et al. (2022) . Respondents in our study were also critical of the lack of inpatient psychiatric care. These results confirm findings in the literature that children and adolescents were the group most affected by the negative psychological effects of the COVID-19 pandemic and the accompanying containment measures ( Racine et al., 2021 ). This was also supported by findings from Austria ( Pieh et al., 2021 ; Sevecke et al., 2023 ). It seems quite conclusive that in our study, clinical psychologists working with children and adolescents as well as those working with families were more likely to say that they would appreciate support in their work. Psychosocial care for children and adolescents in Austria has received increasing attention in recent years. Due to the challenges of the COVID-19 pandemic, it has become clear that there is a need for a rapid expansion of child and adolescent mental health care as well as low-threshold and low-cost or free psychological and psychotherapeutic services ( Culen et al., 2020 ).

4.3 Support needs – calls for broader coverage of treatment costs

When asked what support they would like to see, clinical psychologists in our study also named political and legislative support. In particular they stressed the need for clinical psychological services to be covered by health insurance. This demand is reflected in the results of Winter et al.’s study of psychotherapists in Austria ( Winter et al., 2023 ). Psychotherapists also advocated for low-cost psychotherapy and less bureaucracy in reimbursing treatment costs. Given the overall deterioration in patients’ mental health and the observation by clinical psychologists that some patients are struggling with the financial consequences of the pandemic, it seems important from a health policy perspective to take steps in this area. At the time the survey was conducted, clinical psychological treatment costs were not covered by health insurance funds. However, it was deemed a significant achievement when, in July 2023, clinical psychological treatment was included in the benefit catalogue of the General Social Insurance Act (ASVG) ( Berufsverband Österreicher PsychologInnen, 2023 ). Participants in our study would like to see a similar increase in resources for inpatient psychiatric care.

4.4 Impact of the pandemic on the work of clinical psychologists – introduction of remote treatment formats

In terms of changes in the treatment setting, clinical psychologists in our study described a shift towards remote treatment. The effectiveness of clinical psychological or psychotherapeutic treatment via videoconferencing has been extensively studied prior to the pandemic for specific psychotherapeutic approaches, such as cognitive behavioral therapy or interpersonal psychotherapy ( Mohr et al., 2012 ; Carlbring et al., 2018 ). Several studies have shown efficacy for most clinical conditions, including anxiety, depression, and post-traumatic disorder ( Mohr et al., 2008 ; Cuijpers et al., 2019 ). In addition, studies have found no difference between face-to-face and online treatment in terms of the quality of the psychotherapeutic relationship ( Sucala et al., 2012 ; Poletti et al., 2021 ). The implementation of videoconferencing-based healthcare delivery during the COVID-19 pandemic yielded comparable clinical outcomes to traditional in-person care modalities. These real-world observations substantiate the findings of prior randomized controlled trials and meta-analytic investigations, wherein direct comparisons were made between videoconferencing and in-person care approaches concerning their respective clinical efficacies ( Beurs et al., 2022 ). Other research conducted during the pandemic has already shown that the COVID-19 pandemic has prompted psychologists and other mental health professionals to increase their use of web-based applications, particularly videoconferencing tools ( Swartz, 2020 ; Markowitz et al., 2021 ). Winter et al. (2023) showed that the highest proportion of internet-based psychotherapeutic treatments in Austria was observed during the lockdown period in 2020. Although it decreased significantly as the pandemic progressed, it was still higher in 2022 than before the pandemic. Other studies have confirmed that Austrian psychotherapists continued to use internet-based treatment even after the lockdown restrictions were lifted and it was legally possible to offer face-to-face treatment again ( Höfner et al., 2021 ; Stefan et al., 2021 ; Stadler et al., 2023 ). Psychotherapists likely came to appreciate the benefits of online interventions, such as spatial flexibility, reduced travel time and costs, and access for patients who could not otherwise travel to the practice due to long distances ( Stefan et al., 2021 ). However, the willingness of therapists to use remote treatment formats depends on several factors, including the therapist’s familiarity with the tools needed for online treatment and the ability of the patient and therapist to adapt to the online format ( Höfner et al., 2021 ). In the current study, clinical psychologists confirmed the importance of online treatment in reaching groups of patients who would not otherwise have the opportunity to receive clinical psychological treatment. They also cited increased spatial and temporal flexibility as an advantage of virtual clinical psychological treatment. At the time of the survey in the spring of 2022, just under 15% of patients received clinical psychological treatment via telephone or the internet.

4.5 Impact of the pandemic on the work of clinical psychologists – burdens caused by COVID-19 measures

Other experiences made by the clinical psychologists in our study were perceived as more distressing. For example, wearing masks caused communication difficulties, and respondents described exhaustion and fatigue after several hours of mask use at work. Compliance with COVID-19 containment measures, such as wearing protective clothing, keeping a distance, or using air purifiers, was considered an additional burden. Other studies showed similar results for healthcare workers. Prolonged use of filtering respirators can lead to increased symptoms of exertion, shortness of breath, headache, fatigue, and difficulty communicating, as well as depressive and anxiety symptoms ( Cigiloglu et al., 2022 ; Sahebi et al., 2022 ). Our study also showed that the pandemic also resulted in increased administrative work. For example, treatment appointments had to be rescheduled, and information on current measures had to be obtained. In addition, working conditions became more tense with less opportunity for team or colleague interaction, which was particularly mentioned by clinical psychologists working in an institutional setting. While clinical psychologists working in private practice probably had more freedom to decide how to manage their day-to-day work, those working in an institution had to follow a strict COVID-19 protocol. In addition, most institutional settings were run at maximum capacity, whereas clinical psychologists in private practice could decide how many patients they wanted to see. Other research findings support the hypothesis that the institutional setting was also perceived as more stressful than private practice by other mental health professionals during the pandemic. Schaffler et al.’s study of psychotherapists ( Schaffler et al., 2023 ) showed that those working in an institutional setting reported poorer well-being.

4.6 Support needs – provision of structures for work organization and collaboration in times of crisis

The findings described in the previous section highlight the importance of improving working conditions and providing workplace support for clinical psychologists to meet the additional demands of the pandemic. The different needs of clinical psychologists in employment and in private practice should be taken into account. Finding ways to meet clinical psychologists’ needs for more opportunities for regeneration and more team interaction (communication within teams and with colleagues, intervision, and supervision) could help to maintain the resilience and capacity of this professional group in the long term, even in times of crisis. A significant challenge is finding solutions for clinical psychologists working in outpatient settings without a secure income.

While more opportunities for networking and communication with colleagues and improvements in working conditions were also areas of support mentioned by psychotherapists in the Winter et al. study ( Winter et al., 2023 ), skills development and training, which was mentioned by a large proportion of psychotherapists, played a minor role for clinical psychologists. It is possible that training as a clinical psychologist provides better preparation to deal with acute social and health crises or to adapt to settings other than face-to-face. The differentiation of psychotherapy training into many different psychotherapy methods (23 different psychotherapy methods are recognized in Austria) ( Heidegger, 2017 ) may mean a loss in teaching a broader range of interventions and techniques in treating patients in a state of crisis.

There exist a number of analogous findings between the psychotherapists studied by Winter et al. (2023) and the current examination’s sample of clinical psychologists. While the two studies have some parallels in terms of the observed impact of the pandemic and the desired areas of support, there are also notable differences between the outcomes of the clinical psychologist group compared to the psychotherapists.

Both studies found an initial drop in patient volume at the beginning of the pandemic, followed by a rebound over time. Winter et al. showed volumes exceeding pre-pandemic levels over time. Clinical psychologists and psychotherapists surveyed identified similar needs for more networking/communication opportunities and improvements in working conditions. Both groups expressed a desire for more political/legislative support, such as coverage of psychological services by health insurance. The current study highlighted that clinical psychologists underlined the value of online treatments to access more patients. Winter et al. also found similar results regarding the importance of online treatments. The desire for skills development regarding crisis coping due to the pandemic played a more important role for the psychotherapists surveyed by Winter et al. than for the clinical psychologists in the current study, possibly due to differences in crisis preparedness in their training. The current study found a higher demand for clinical psychologists with more professional experience, while Winter et al. did not identify differences based on years of practice.

Mental disorders are widespread in Austria and cause high personal, social, and financial costs ( Löffler-Stastka and Hochgerner, 2021 ). Clinical psychologists, together with psychotherapists, provide the majority of clinical treatment for patients with mental health problems ( Löffler-Stastka and Hochgerner, 2021 ). They also provide help in acute crises, offer counselling, and cooperate with other health professionals. This study highlighted the changes that clinical psychologists experienced during the pandemic and the support wishes they identified for their profession. In view of comparable findings on psychological distress in other countries ( Budimir et al., 2021 ), which indicate a similar shortage of care ( Bundes Psychotherapeuten Kammer, 2023 ), we conclude that the results of our study are also relevant for other national contexts. Expanding psychosocial services, covered by health insurance, and supporting professionals in the field through suitable legal structures, administrative help, financial resources, and networking and training platforms can establish a vital groundwork for effectively coping with future crises.

5 Limitations

This study has several limitations. Firstly, the study was conducted using an online questionnaire, which may have led to a bias towards people who are open to electronic data processing and the use of online tools. Another limitation in analyzing the results is that there is no data on the situation before the COVID-19 pandemic to compare the results with. In addition, recall bias is possible, as participants reported on situations from the start of the pandemic in 2020 until the time of the survey in spring 2022. It is also crucial to acknowledge the inherent limitation of the cross-sectional design of the study, which precludes causal inference. The current sample represents a relatively selective subgroup of the ≈ 3,000 eligible participants. We suspect a response rate around 6%, which is considered quite low. Such a limited response poses an augmented risk of response bias, whereby those with intense or polarized perspectives on the matter disproportionately opted to provide input. Finally, it should be noted that we had no data from patients, only information from clinical psychologists.

6 Conclusion

The results of this study highlighted the importance of mental health services in times of pandemic and crisis. At the same time, the results again showed that participants still perceive significant gaps in the mental health care system, and that they experience that financially disadvantaged patients, children and adolescents in particular, have difficulties in accessing clinical psychological services. At the level of the health care system, clinical psychologists expressed a need for more fully or partially funded contingents for clinical psychological treatment and an increase in beds and staff in inpatient psychiatric care. At the level of professional policy, participants see a need for support for the profession of clinical psychology. This could take the form of assistance in coping with additional organizational and administrative tasks, more opportunities for exchange within teams or between colleagues, or financial support in the event of a loss of income.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by Ethics committee of the University of Continuing Education Krems, Austria. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

PS: Formal analysis, Visualization, Writing – original draft. EH: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Supervision, Validation, Visualization, Writing – original draft. YS: Conceptualization, Methodology, Writing – review & editing. CP: Conceptualization, Writing – review & editing. TP: Writing – review & editing. AF: Writing – review & editing. OK: Writing – review & editing. IN: Writing – review & editing. AJ: Conceptualization, Formal analysis, Methodology, Supervision, Writing – original draft.

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Acknowledgments

The authors thank the clinical psychologists who devoted their time and effort to participate in the survey and acknowledge the support of the University for Continuing Education Krems in providing open access funding.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2024.1302442/full#supplementary-material

ATLAS.ti Scientific Software Development GmbH . ATLAS.Ti (version 23.1.2 windows) [computer program] Berlin, Germany: ATLAS.ti Scientific Software Development GmbH (2023)

Google Scholar

Berufsverband Österreicher PsychologInnen . Praktische Tipps für PsychologInnen in freier Praxis Umgang mit KlientInnen im Zusammenhang mit dem Coronavirus SARS-CoV-2 (COVID-19) (2020). Available at: https://institut-avm.at/wp-content/uploads/2020/06/boc2a8p_praktischetipps_wiederausweitung.pdf

Berufsverband Österreicher PsychologInnen . Klinische Psychologie (2023). Available at: https://www.boep.or.at/berufsverband/fachsektionen/klinische-psychologie

Berufsverband Österreicher PsychologInnen . (2023). Klinisch-Psychologische Behandlung in das ASVG. Available at: https://www.boep.or.at/berufspolitik/zentrale-berufspolitische-ziele/klinpsy-behandlung-krankenschein

Beurs, E., Blankers, M., Penn, J., Rademacher, C., Podgorski, A., and Dekker, J. (2022). Impact of COVID-19 social distancing measures on routine mental health care provision and treatment outcome for common mental disorders in the Netherlands. Clin. Psychol. Psychother. 29, 1342–1354. doi: 10.1002/cpp.2713

PubMed Abstract | Crossref Full Text | Google Scholar

Budimir, S., Pieh, C., Dale, R., and Probst, T. (2021). Severe mental health symptoms during COVID-19: a comparison of the United Kingdom and Austria. Healthcare 9:191. doi: 10.3390/healthcare9020191

Bundeministerium für Soziales, Gesundheit, Pflege und Konsumentenschutz . Klinische Psychologin, klinischer Psychologe. (2023) Available at: https://www.sozialministerium.at/Themen/Gesundheit/Medizin-und-Gesundheitsberufe/Berufe-A-bis-Z/Klinische-Psychologin,-Klinischer-Psychologe.html

Bundes Psychotherapeuten Kammer . Resolutionen des 43. Deutschen Psychotherapeutentages Forderungen an die Politik (2023). Available at: https://www.bptk.de/pressemitteilungen/resolutionen-des-43-deutschen-psychotherapeutentages

Carlbring, P., Andersson, G., Cuijpers, P., Riper, H., and Hedman-Lagerlöf, E. (2018). Internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. Cogn. Behav. Ther. 47, 1–18. doi: 10.1080/16506073.2017.1401115

Cigiloglu, A., Ozturk, E., Ganidagli, S., and Ozturk, Z. A. (2022). Different reflections of the face mask: sleepiness, headache and psychological symptoms. Int. J. Occup. Saf. Ergon. 28, 2278–2283. doi: 10.1080/10803548.2021.1984712

Cuijpers, P., Noma, H., Karyotaki, E., Cipriani, A., and Furukawa, T. A. (2019). Effectiveness and acceptability of cognitive behavior therapy delivery formats in adults with depression: a network meta-analysis. JAMA Psychiatry 76, 700–707. doi: 10.1001/jamapsychiatry.2019.0268

Culen, C., Hierzer, J., and Schaupp, G. (2020). Österreichische Liga für Kinder-und Jugendgesundheit. Available at: https://www.kinderjugendgesundheit.at/site/assets/files/1237/jb_kinderliga_2020_komprimiert.pdf

Fiala-Baumann, B., Ploner, H., Witzmann, D., and Jesser, A. (2022). Säuglings‑, Kinder- und Jugendlichen- (SKJ) Psychotherapien während der COVID-19 Pandemie: Ergebnisse einer Studie unter psychodynamischen Psychotherapeut*innen in Österreich. Forum 26, 144–153. doi: 10.1007/s00729-022-00213-9

Crossref Full Text | Google Scholar

Gächter, A., Zauner, B., Haider, K., Schaffler, Y., Probst, T., Pieh, C., et al. (2023). Areas of concern and support among the Austrian general population. A qualitative content analytic mapping of the shift between Winter 2020/21 and spring 2022. Healthcare 11:2539. doi: 10.3390/healthcare11182539

Gu, S., He, Z., Sun, L., Jiang, Y., Xu, M., Feng, G., et al. (2021). Effects of Coronavirus-19 induced loneliness on mental health: sleep quality and intolerance for uncertainty as mediators. Front. Psych. 12:738003. doi: 10.3389/fpsyt.2021.738003

Harris, P. A., Taylor, R., Minor, B. L., Elliott, V., Fernandez, M., O'Neal, L., et al. (2019). The REDCap consortium: building an international community of software platform partners. J. Biomed. Inform. 95:103208. doi: 10.1016/j.jbi.2019.103208

Heidegger, KE . (2017). The Situation of Psychotherapy in Austria. Available at: https://www.europsyche.org/app/uploads/2019/05/Situation-Psychotherapy-in-Austria-2017-10-20.pdf (Accessed March 8, 2021)

Höfner, C., Hochgerner, M., Mantl, G., Stefan, R., and Stammer, J. (2021). Telepsychotherapie als chance und Herausforderung: Eine longitudinale mixed-methods Studie. Psychother. Forum 25, 37–43. doi: 10.1007/s00729-021-00169-2

Hsieh, H. F., and Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qual. Health Res. 15, 1277–1288. doi: 10.1177/1049732305276687

Humer, E., Pammer, B., Schaffler, Y., Kothgassner, O. D., Felnhofer, A., Jesser, A., et al. (2023). Comparison of mental health indicators in clinical psychologists with the general population during the COVID-19 pandemic. Sci. Rep. 13:5050. doi: 10.1038/s41598-023-32316-x

Humer, E., Schaffler, Y., Jesser, A., Probst, T., and Pieh, C. (2022). Mental health in the Austrian general population during COVID-19: cross-sectional study on the association with sociodemographic factors. Front. Psychiatry :943303:13. doi: 10.3389/fpsyt.2022.943303

Jesser, A., Steinböck, A., Pammer, B., Ghorab, T., Weber, M., Schaffler, Y., et al. (2024). Burdens and resources of Austrian clinical psychologists: results of a qualitative study two years into the COVID-19 pandemic. BMC Psychol.

Killgore, W. D. S., Cloonan, S. A., Taylor, E. C., and Dailey, N. S. (2020). Loneliness: a signature mental health concern in the era of COVID-19. Psychiatry Res. 290:113117. doi: 10.1016/j.psychres.2020

Leung, C. M. C., Ho, M. K., Bharwani, A. A., Cogo-Moreira, H., Wang, Y., Chow, M. S. C., et al. (2022). Mental disorders following COVID-19 and other epidemics: a systematic review and meta-analysis. Transl. Psychiatry 12:205. doi: 10.1038/s41398-022-01946-6

Li, L. Z., and Wang, S. (2020). Prevalence and predictors of general psychiatric disorders and loneliness during COVID-19 in the United Kingdom. Psychiatry Res. 291:113267. doi: 10.1016/j.psychres.2020.113267

Löffler-Stastka, H., and Hochgerner, M. (2021). Versorgungswirksamkeit von Psychotherapie in Österreich. psychopraxis. Neuropraxis 24, 57–61. doi: 10.1007/s00739-020-00686-w

Mahmud, S., Mohsin, M., Dewan, M. N., and Muyeed, A. (2023). The global prevalence of depression, anxiety, stress, and insomnia among general population during COVID-19 pandemic: a systematic review and meta-analysis. Trends Psychol. 31, 143–170. doi: 10.1007/s43076-021-00116-9

Markowitz, J. C., Milrod, B., Heckman, T. G., Bergman, M., Amsalem, D., Zalman, H., et al. (2021). Psychotherapy at a distance. Am. J. Psychiatry 178, 240–246. doi: 10.1176/appi.ajp.2020.20050557

Mohr, M . Ängste vor Auswirkungen des Russland-Ukraine-Krieges in Österreich 2022. (2022). Available at: https://de.statista.com/statistik/daten/studie/1294019/umfrage/aengste-vor-auswirkungen-des-russland-ukraine-krieges-in-oesterreich/

Mohr, D. C., Ho, J., Duffecy, J., Reifler, D., Sokol, L., Burns, M. N., et al. (2012). Effect of telephone-administered vs face-to-face cognitive behavioral therapy on adherence to therapy and depression outcomes among primary care patients: a randomized trial. JAMA 307, 2278–2285. doi: 10.1001/jama.2012.5588

Mohr, D. C., Vella, L., Hart, S., Heckman, T., and Simon, G. (2008). The effect of telephone-administered psychotherapy on symptoms of depression and attrition: a meta-analysis. Clin. Psychol. 15, 243–253. doi: 10.1111/j.1468-2850.2008.00134.x

Nochaiwong, S., Ruengorn, C., Thavorn, K., Hutton, B., Awiphan, R., Phosuya, C., et al. (2021). Global prevalence of mental health issues among the general population during the coronavirus disease-2019 pandemic: a systematic review and meta-analysis. Sci. Rep. 11:10173. doi: 10.1038/s41598-021-89700-8

Öffentliches Gesundheitsportal Österreichs (2019). Klinische Psychologie. Available at: https://www.gesundheit.gv.at/gesundheitsleistungen/berufe/gesundheitsberufe-a-z/diagnose-therapie-beratung/klinische-psychologie.html

Pieh, C., Budimir, S., Humer, E., and Probst, T. (2021). Comparing mental health during the COVID-19 lockdown and 6 months after the lockdown in Austria: a longitudinal study. Front. Psychiatry 12:625973. doi: 10.3389/fpsyt.2021.625973

Pieh, C., Budimir, S., and Probst, T. (2020). The effect of age, gender, income, work, and physical activity on mental health during coronavirus disease (COVID-19) lockdown in Austria. J. Psychosom. Res. 136:110186. doi: 10.1016/j.jpsychores.2020.110186

Pieh, C., Plener, P. L., Probst, T., Dale, R., and Humer, E. (2021). Assessment of mental health of high school students during social distancing and remote schooling during the COVID-19 pandemic in Austria. JAMA Netw. Open 4:e2114866. doi: 10.1001/jamanetworkopen.2021.14866

Poletti, B., Tagini, S., Brugnera, A., Parolin, L., Pievani, L., Ferrucci, R., et al. (2021). Telepsychotherapy: a leaflet for psychotherapists in the age of COVID-19. A review of the evidence. Couns. Psychol. Q. 34, 352–367. doi: 10.1080/09515070.2020.1769557

Pollak, M, Kowarz, N, and Partheymüller, J. Chronologie zur Corona-Krise in Österreich – Teil 1: Vorgeschichte, der Weg in den Lockdown, die akute Phase und wirtschaftliche Folgen. (2020). Available at: https://viecer.univie.ac.at/corona-blog/corona-blog-beitraege/blog51/

Pollak, M, Kowarz, N, and Partheymüller, J. Chronologie zur Corona-Krise in Österreich – Teil 6: En “Sommer wie damals” der Weg in die vierte Welle, ein erneuter Lockdown und die Impfpflicht. (2021). Available at: https://viecer.univie.ac.at/corona-blog/corona-blog-beitraege/blog135/

Racine, N., McArthur, B. A., Cooke, J. E., Eirich, R., Zhu, J., and Madigan, S. (2021). Global prevalence of depressive and anxiety symptoms in children and adolescents during COVID-19: a meta-analysis. JAMA Pediatr. 175, 1142–1150. doi: 10.1001/jamapediatrics.2021.2482

Roberts, A., Rogers, J., Mason, R., Siriwardena, A. N., Hogue, T., Whitley, G. A., et al. (2021). Alcohol and other substance use during the COVID-19 pandemic: a systematic review. Drug Alcohol Depend. 229:109150. doi: 10.1016/j.drugalcdep.2021.109150

Sahebi, A., Hasheminejad, N., Shohani, M., Yousefi, A., Tahernejad, S., and Tahernejad, A. (2022). Personal protective equipment-associated headaches in health care workers during COVID-19: a systematic review and meta-analysis. Front. Public Health 10:942046. doi: 10.3389/fpubh.2022.942046

Schaffler, Y., Bauer, M., Schein, B., Jesser, A., Probst, T., Pieh, C., et al. (2023). Understanding pandemic resilience: a mixed-methods exploration of burdens, resources, and determinants of good or poor well-being among Austrian psychotherapists. Front. Public Health :1216833:11. doi: 10.3389/fpubh.2023.1216833

Sevecke, K., Wenter, A., Schickl, M., Kranz, M., Krstic, N., and Fuchs, M. (2023). Stationäre Versorgungskapazitäten in der Kinder-und Jugendpsychiatrie – Zunahme der Akutaufnahmen während der COVID-19 Pandemie? Neuropsychiatrie 37, 12–21. doi: 10.1007/s40211-022-00423-2

Sideli, L., Lo Coco, G., Bonfanti, R. C., Borsarini, B., Fortunato, L., Sechi, C., et al. (2021). Effects of COVID-19 lockdown on eating disorders and obesity: a systematic review and meta-analysis. Eur. Eat. Disord. Rev. 29, 826–841. doi: 10.1002/erv.2861

Simon, J., Helter, T. M., White, R. G., Van der Boor, C., and Łaszewska, A. (2021). Impacts of the Covid-19 lockdown and relevant vulnerabilities on capability well-being, mental health and social support: an Austrian survey study. BMC Public Health 21:314. doi: 10.1186/s12889-021-10351-5

Stadler, M., Jesser, A., Humer, E., Haid, B., Stippl, P., Schimböck, W., et al. (2023). Remote psychotherapy during the COVID-19 pandemic: a mixed-methods study on the changes experienced by Austrian psychotherapists. Life 13:360. doi: 10.3390/life13020360

Stefan, R., Mantl, G., Höfner, C., Stammer, J., Hochgerner, M., and Petersdorfer, K. (2021). Remote psychotherapy during the COVID-19 pandemic. Experiences with the transition and the therapeutic relationship. A longitudinal mixed-methods study. Front. Psychol. 12:743430. doi: 10.3389/fpsyg.2021

Sucala, M., Schnur, J. B., Constantino, M. J., Miller, S. J., Brackman, E. H., and Montgomery, G. H. (2012). The therapeutic relationship in e-therapy for mental health: a systematic review. J. Med. Res 14:e110. doi: 10.2196/jmir.2084

Swartz, H. A. (2020). The role of psychotherapy during the COVID-19 pandemic. Am. J. Psychother. 73, 41–42. doi: 10.1176/appi.psychotherapy

Walcherberger, C, Holl, F, Pollak, M, Kowarz, N, and Partheymüller, J. Chronologie zur Corona-Krise in Österreich - Teil 7: Der Delta-Lockdown, die Omikron-Welle und das “Frühlingserwachen”. (2023). Available at: https://viecer.univie.ac.at/corona-blog/corona-blog-beitraege/blog157/

Wang, C. C., Prather, K. A., Sznitman, J., Jimenez, J. L., Lakdawala, S. S., Tufekci, Z., et al. (2021). Airborne transmission of respiratory viruses. Science 373:6558. doi: 10.1126/science.abd9149

Winter, S., Jesser, A., Probst, T., Schaffler, Y., Kisler, I. M., Haid, B., et al. (2023). How the COVID-19 pandemic affects the provision of psychotherapy: results from three online surveys on Austrian psychotherapists. Int. J. Environ. Res. Public Health 20:1961. doi: 10.3390/ijerph20031961

World Health Organization . Director-General's opening remarks at the media briefing on COVID-19 - 11 March (2020). Available at: https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-2020

Xiong, J., Lipsitz, O., Nasri, F., Lui, L. M. W., Gill, H., Phan, L., et al. (2020). Impact of COVID-19 pandemic on mental health in the general population: a systematic review. J. Affect. Disord. 277, 55–64. doi: 10.1016/j.jad.2020.08.001

Keywords: COVID-19, clinical psychologist, mixed-methods study, mental health professionals, public health

Citation: Santillan-Ramos P, Humer E, Schaffler Y, Pieh C, Probst T, Felnhofer A, Kothgassner O, Netzer I and Jesser A (2024) Impact of the COVID-19 pandemic on the work of clinical psychologists in Austria: results of a mixed-methods study. Front. Psychol . 15:1302442. doi: 10.3389/fpsyg.2024.1302442

Received: 27 September 2023; Accepted: 09 April 2024; Published: 25 April 2024.

Reviewed by:

Copyright © 2024 Santillan-Ramos, Humer, Schaffler, Pieh, Probst, Felnhofer, Kothgassner, Netzer and Jesser. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Andrea Jesser, [email protected]

† These authors have contributed equally to this work

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Perspect Clin Res
• v.11(2); Apr-Jun 2020

Study designs: Part 7 – Systematic reviews

Priya ranganathan.

Department of Anaesthesiology, Tata Memorial Centre, Homi Bhabha National Institute, Mumbai, Maharashtra, India

Rakesh Aggarwal

1 Director, Jawaharlal Institute of Postgraduate Medical Education and Research, Puducherry, India

In this series on research study designs, we have so far looked at different types of primary research designs which attempt to answer a specific question. In this segment, we discuss systematic review, which is a study design used to summarize the results of several primary research studies. Systematic reviews often also use meta-analysis, which is a statistical tool to mathematically collate the results of various research studies to obtain a pooled estimate of treatment effect; this will be discussed in the next article.

In the previous six articles in this series on study designs, we have looked at different types of primary research study designs which are used to answer research questions. In this article, we describe the systematic review, a type of secondary research design that is used to summarize the results of prior primary research studies. Systematic reviews are considered the highest level of evidence for a particular research question.[ 1 ]

SYSTEMATIC REVIEWS

As defined in the Cochrane Handbook for Systematic Reviews of Interventions , “Systematic reviews seek to collate evidence that fits pre-specified eligibility criteria in order to answer a specific research question. They aim to minimize bias by using explicit, systematic methods documented in advance with a protocol.”[ 2 ]

NARRATIVE VERSUS SYSTEMATIC REVIEWS

Review of available data has been done since times immemorial. However, the traditional narrative reviews (“expert reviews”) do not involve a systematic search of the literature. Instead, the author of the review, usually an expert on the subject, used informal methods to identify (what he or she thinks are) the key studies on the topic. The final review thus is a summary of these “selected” studies. Since studies are chosen at will (haphazardly!) and without clearly defined criteria, such reviews preferentially include those studies that favor the author's views, leading to a potential for subjectivity or selection bias.

In contrast, systematic reviews involve a formal prespecified protocol with explicit, transparent criteria for the inclusion and exclusion of studies, thereby ensuring completeness of coverage of the available evidence, and providing a more objective, replicable, and comprehensive overview it.

META-ANALYSIS

Many systematic reviews use an additional tool, known as meta-analysis, which is a statistical technique for combining the results of multiple studies in a systematic review in a mathematically appropriate way, to create a single (pooled) and more precise estimate of treatment effect. The feasibility of performing a meta-analysis in a systematic review depends on the number of studies included in the final review and the degree of heterogeneity in the inclusion criteria as well as the results between the included studies. Meta-analysis will be discussed in detail in the next article in this series.

THE PROCESS OF A SYSTEMATIC REVIEW

The conduct of a systematic review involves several sequential key steps.[ 3 , 4 ] As in other research study designs, a clearly stated research question and a well-written research protocol are essential before commencing a systematic review.

Step 1: Stating the review question

Systematic reviews can be carried out in any field of medical research, e.g. efficacy or safety of interventions, diagnostics, screening or health economics. In this article, we focus on systematic reviews of studies looking at the efficacy of interventions. As for the other study designs, for a systematic review too, the question is best framed using the Population, Intervention, Comparator, and Outcome (PICO) format.

For example, Safi et al . carried out a systematic review on the effect of beta-blockers on the outcomes of patients with myocardial infarction.[ 5 ] In this review, the Population was patients with suspected or confirmed myocardial infarction, the Intervention was beta-blocker therapy, the Comparator was either placebo or no intervention, and the Outcomes were all-cause mortality and major adverse cardiovascular events. The review question was “ In patients with suspected or confirmed myocardial infarction, does the use of beta-blockers affect mortality or major adverse cardiovascular outcomes? ”

Step 2: Listing the eligibility criteria for studies to be included

It is essential to explicitly define a priori the criteria for selection of studies which will be included in the review. Besides the PICO components, some additional criteria used frequently for this purpose include language of publication (English versus non-English), publication status (published as full paper versus unpublished), study design (randomized versus quasi-experimental), age group (adults versus children), and publication year (e.g. in the last 5 years, or since a particular date). The PICO criteria used may not be very specific, e.g. it is possible to include studies that use one or the other drug belonging to the same group. For instance, the systematic review by Safi et al . included all randomized clinical trials, irrespective of setting, blinding, publication status, publication year, or language, and reported outcomes, that had used any beta-blocker and in a broad range of doses.[ 5 ]

Step 3: Comprehensive search for studies that meet the eligibility criteria

A thorough literature search is essential to identify all articles related to the research question and to ensure that no relevant article is left out. The search may include one or more electronic databases and trial registries; in addition, it is common to hand-search the cross-references in the articles identified through such searches. One could also plan to reach out to experts in the field to identify unpublished data, and to search the grey literature non-peer-reviewednon-peer-reviewed. This last option is particularly helpful non-pharmacologic (theses, conference abstracts, and non-peer-reviewed journals). These sources are particularly helpful when the intervention is relatively new, since data on these may not yet have been published as full papers and hence are unlikely to be found in literature databases. In the review by Safi et al ., the search strategy included not only several electronic databases (Cochrane, MEDLINE, EMBASE, LILACS, etc.) but also other resources (e.g. Google Scholar, WHO International Clinical Trial Registry Platform, and reference lists of identified studies).[ 5 ] It is not essential to include all the above databases in one's search. However, it is mandatory to define in advance which of these will be searched.

Step 4: Identifying and selecting relevant studies

Once the search strategy defined in the previous step has been run to identify potentially relevant studies, a two-step process is followed. First, the titles and abstracts of the identified studies are processed to exclude any duplicates and to discard obviously irrelevant studies. In the next step, full-text papers of the remaining articles are retrieved and closely reviewed to identify studies that meet the eligibility criteria. To minimize bias, these selection steps are usually performed independently by at least two reviewers, who also assign a reason for non-selection to each discarded study. Any discrepancies are then resolved either by an independent reviewer or by mutual consensus of the original reviewers. In the Cochrane review on beta-blockers referred to above, two review authors independently screened the titles for inclusion, and then, four review authors independently reviewed the screen-positive studies to identify the trials to be included in the final review.[ 5 ] Disagreements were resolved by discussion or by taking the opinion of a separate reviewer. A summary of this selection process, showing the degree of agreement between reviewers, and a flow diagram that depicts the numbers of screened, included and excluded (with reason for exclusion) studies are often included in the final review.

Step 5: Data extraction

In this step, from each selected study, relevant data are extracted. This should be done by at least two reviewers independently, and the data then compared to identify any errors in extraction. Standard data extraction forms help in objective data extraction. The data extracted usually contain the name of the author, the year of publication, details of intervention and control treatments, and the number of participants and outcome data in each group. In the review by Safi et al ., four review authors independently extracted data and resolved any differences by discussion.[ 5 ]

Handling missing data

Some of the studies included in the review may not report outcomes in accordance with the review methodology. Such missing data can be handled in two ways – by contacting authors of the original study to obtain the necessary data and by using data imputation techniques. Safi et al . used both these approaches – they tried to get data from the trial authors; however, where that failed, they analyzed the primary outcome (mortality) using the best case (i.e. presuming that all the participants in the experimental arm with missing data had survived and those in the control arm with missing mortality data had died – representing the maximum beneficial effect of the intervention) and the worst case (all the participants with missing data in the experimental arm assumed to have died and those in the control arm to have survived – representing the least beneficial effect of the intervention) scenarios.

Evaluating the quality (or risk of bias) in the included studies

The overall quality of a systematic review depends on the quality of each of the included studies. Quality of a study is inversely proportional to the potential for bias in its design. In our previous articles on interventional study design in this series, we discussed various methods to reduce bias – such as randomization, allocation concealment, participant and assessor blinding, using objective endpoints, minimizing missing data, the use of intention-to-treat analysis, and complete reporting of all outcomes.[ 6 , 7 ] These features form the basis of the Cochrane Risk of Bias Tool (RoB 2), which is a commonly used instrument to assess the risk of bias in the studies included in a systematic review.[ 8 ] Based on this tool, one can classify each study in a review as having low risk of bias, having some concerns regarding bias, or at high risk of bias. Safi et al . used this tool to classify the included studies as having low or high risk of bias and presented these data in both tabular and graphical formats.[ 5 ]

In some reviews, the authors decide to summarize only studies with a low risk of bias and to exclude those with a high risk of bias. Alternatively, some authors undertake a separate analysis of studies with low risk of bias, besides an analysis of all the studies taken together. The conclusions from such analyses of only high-quality studies may be more robust.

Step 6: Synthesis of results

The data extracted from various studies are pooled quantitatively (known as a meta-analysis) or qualitatively (if pooling of results is not considered feasible). For qualitative reviews, data are usually presented in the tabular format, showing the characteristics of each included study, to allow for easier interpretation.

Sensitivity analyses

Sensitivity analyses are used to test the robustness of the results of a systematic review by examining the impact of excluding or including studies with certain characteristics. As referred to above, this can be based on the risk of bias (methodological quality), studies with a specific study design, studies with a certain dosage or schedule, or sample size. If results of these different analyses are more-or-less the same, one can be more certain of the validity of the findings of the review. Furthermore, such analyses can help identify whether the effect of the intervention could vary across different levels of another factor. In the beta-blocker review, sensitivity analysis was performed depending on the risk of bias of included studies.[ 5 ]

IMPORTANT RESOURCES FOR CARRYING OUT SYSTEMATIC REVIEWS AND META-ANALYSES

Cochrane is an organization that works to produce good-quality, updated systematic reviews related to human healthcare and policy, which are accessible to people across the world.[ 9 ] There are more than 7000 Cochrane reviews on various topics. One of its main resources is the Cochrane Library (available at https://www.cochranelibrary.com/ ), which incorporates several databases with different types of high-quality evidence to inform healthcare decisions, including the Cochrane Database of Systematic Reviews, Cochrane Central Register of Controlled Trials (CENTRAL), and Cochrane Clinical Answers.

The Cochrane Handbook for Systematic Reviews of Interventions

The Cochrane handbook is an official guide, prepared by the Cochrane Collaboration, to the process of preparing and maintaining Cochrane systematic reviews.[ 10 ]

Review Manager software

Review Manager (RevMan) is a software developed by Cochrane to support the preparation and maintenance of systematic reviews, including tools for performing meta-analysis.[ 11 ] It is freely available in both online (RevMan Web) and offline (RevMan 5.3) versions.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement is an evidence-based minimum set of items for reporting of systematic reviews and meta-analyses of randomized trials.[ 12 ] It can be used both by authors of such studies to improve the completeness of reporting and by reviewers and readers to critically appraise a systematic review. There are several extensions to the PRISMA statement for specific types of reviews. An update is currently underway.

Meta-analysis of Observational Studies in Epidemiology statement

The Meta-analysis of Observational Studies in Epidemiology statement summarizes the recommendations for reporting of meta-analyses in epidemiology.[ 13 ]

PROSPERO is an international database for prospective registration of protocols for systematic reviews in healthcare.[ 14 ] It aims to avoid duplication of and to improve transparency in reporting of results of such reviews.

Financial support and sponsorship

Conflicts of interest.

There are no conflicts of interest.

• Open access
• Published: 22 April 2024

Artificial intelligence and medical education: application in classroom instruction and student assessment using a pharmacology & therapeutics case study

• Kannan Sridharan 1 &
• Reginald P. Sequeira 1  

BMC Medical Education volume  24 , Article number:  431 ( 2024 ) Cite this article

354 Accesses

1 Altmetric

Metrics details

Artificial intelligence (AI) tools are designed to create or generate content from their trained parameters using an online conversational interface. AI has opened new avenues in redefining the role boundaries of teachers and learners and has the potential to impact the teaching-learning process.

In this descriptive proof-of- concept cross-sectional study we have explored the application of three generative AI tools on drug treatment of hypertension theme to generate: (1) specific learning outcomes (SLOs); (2) test items (MCQs- A type and case cluster; SAQs; OSPE); (3) test standard-setting parameters for medical students.

Analysis of AI-generated output showed profound homology but divergence in quality and responsiveness to refining search queries. The SLOs identified key domains of antihypertensive pharmacology and therapeutics relevant to stages of the medical program, stated with appropriate action verbs as per Bloom’s taxonomy. Test items often had clinical vignettes aligned with the key domain stated in search queries. Some test items related to A-type MCQs had construction defects, multiple correct answers, and dubious appropriateness to the learner’s stage. ChatGPT generated explanations for test items, this enhancing usefulness to support self-study by learners. Integrated case-cluster items had focused clinical case description vignettes, integration across disciplines, and targeted higher levels of competencies. The response of AI tools on standard-setting varied. Individual questions for each SAQ clinical scenario were mostly open-ended. The AI-generated OSPE test items were appropriate for the learner’s stage and identified relevant pharmacotherapeutic issues. The model answers supplied for both SAQs and OSPEs can aid course instructors in planning classroom lessons, identifying suitable instructional methods, establishing rubrics for grading, and for learners as a study guide. Key lessons learnt for improving AI-generated test item quality are outlined.

Conclusions

AI tools are useful adjuncts to plan instructional methods, identify themes for test blueprinting, generate test items, and guide test standard-setting appropriate to learners’ stage in the medical program. However, experts need to review the content validity of AI-generated output. We expect AIs to influence the medical education landscape to empower learners, and to align competencies with curriculum implementation. AI literacy is an essential competency for health professionals.

Peer Review reports

Artificial intelligence (AI) has great potential to revolutionize the field of medical education from curricular conception to assessment [ 1 ]. AIs used in medical education are mostly generative AI large language models that were developed and validated based on billions to trillions of parameters [ 2 ]. AIs hold promise in the incorporation of history-taking, assessment, diagnosis, and management of various disorders [ 3 ]. While applications of AIs in undergraduate medical training are being explored, huge ethical challenges remain in terms of data collection, maintaining anonymity, consent, and ownership of the provided data [ 4 ]. AIs hold a promising role amongst learners because they can deliver a personalized learning experience by tracking their progress and providing real-time feedback, thereby enhancing their understanding in the areas they are finding difficult [ 5 ]. Consequently, a recent survey has shown that medical students have expressed their interest in acquiring competencies related to the use of AIs in healthcare during their undergraduate medical training [ 6 ].

Pharmacology and Therapeutics (P & T) is a core discipline embedded in the undergraduate medical curriculum, mostly in the pre-clerkship phase. However, the application of therapeutic principles forms one of the key learning objectives during the clerkship phase of the undergraduate medical career. Student assessment in pharmacology & therapeutics (P&T) is with test items such as multiple-choice questions (MCQs), integrated case cluster questions, short answer questions (SAQs), and objective structured practical examination (OSPE) in the undergraduate medical curriculum. It has been argued that AIs possess the ability to communicate an idea more creatively than humans [ 7 ]. It is imperative that with access to billions of trillions of datasets the AI platforms hold promise in playing a crucial role in the conception of various test items related to any of the disciplines in the undergraduate medical curriculum. Additionally, AIs provide an optimized curriculum for a program/course/topic addressing multidimensional problems [ 8 ], although robust evidence for this claim is lacking.

The existing literature has evaluated the knowledge, attitude, and perceptions of adopting AI in medical education. Integration of AIs in medical education is the need of the hour in all health professional education. However, the academic medical fraternity facing challenges in the incorporation of AIs in the medical curriculum due to factors such as inadequate grounding in data analytics, lack of high-quality firm evidence favoring the utility of AIs in medical education, and lack of funding [ 9 ]. Open-access AI platforms are available free to users without any restrictions. Hence, as a proof-of-concept, we chose to explore the utility of three AI platforms to identify specific learning objectives (SLOs) related to pharmacology discipline in the management of hypertension for medical students at different stages of their medical training.

Study design and ethics

The present study is observational, cross-sectional in design, conducted in the Department of Pharmacology & Therapeutics, College of Medicine and Medical Sciences, Arabian Gulf University, Kingdom of Bahrain, between April and August 2023. Ethical Committee approval was not sought given the nature of this study that neither had any interaction with humans, nor collection of any personal data was involved.

Study procedure

We conducted the present study in May-June 2023 with the Poe© chatbot interface created by Quora© that provides access to the following three AI platforms:

Sage Poe [ 10 ]: A generative AI search engine developed by Anthropic © that conceives a response based on the written input provided. Quora has renamed Sage Poe as Assistant © from July 2023 onwards.

Claude-Instant [ 11 ]: A retrieval-based AI search engine developed by Anthropic © that collates a response based on pre-written responses amongst the existing databases.

ChatGPT version 3.5 [ 12 ]: A generative architecture-based AI search engine developed by OpenAI © trained on large and diverse datasets.

We queried the chatbots to generate SLOs, A-type MCQs, integrated case cluster MCQs, integrated SAQs, and OSPE test items in the domain of systemic hypertension related to the P&T discipline. Separate prompts were used to generate outputs for pre-clerkship (preclinical) phase students, and at the time of graduation (before starting residency programs). Additionally, we have also evaluated the ability of these AI platforms to estimate the proportion of students correctly answering these test items. We used the following queries for each of these objectives:

Specific learning objectives

Can you generate specific learning objectives in the pharmacology discipline relevant to undergraduate medical students during their pre-clerkship phase related to anti-hypertensive drugs?

Can you generate specific learning objectives in the pharmacology discipline relevant to undergraduate medical students at the time of graduation related to anti-hypertensive drugs?

A-type MCQs

In the initial query used for A-type of item, we specified the domains (such as the mechanism of action, pharmacokinetics, adverse reactions, and indications) so that a sample of test items generated without any theme-related clutter, shown below:

Write 20 single best answer MCQs with 5 choices related to anti-hypertensive drugs for undergraduate medical students during the pre-clerkship phase of which 5 MCQs should be related to mechanism of action, 5 MCQs related to pharmacokinetics, 5 MCQs related to adverse reactions, and 5 MCQs should be related to indications.

The MCQs generated with the above search query were not based on clinical vignettes. We queried again to generate MCQs using clinical vignettes specifically because most medical schools have adopted problem-based learning (PBL) in their medical curriculum.

Write 20 single best answer MCQs with 5 choices related to anti-hypertensive drugs for undergraduate medical students during the pre-clerkship phase using a clinical vignette for each MCQ of which 5 MCQs should be related to the mechanism of action, 5 MCQs related to pharmacokinetics, 5 MCQs related to adverse reactions, and 5 MCQs should be related to indications.

We attempted to explore whether AI platforms can provide useful guidance on standard-setting. Hence, we used the following search query.

Can you do a simulation with 100 undergraduate medical students to take the above questions and let me know what percentage of students got each MCQ correct?

Integrated case cluster MCQs

Write 20 integrated case cluster MCQs with 2 questions in each cluster with 5 choices for undergraduate medical students during the pre-clerkship phase integrating pharmacology and physiology related to systemic hypertension with a case vignette.

Write 20 integrated case cluster MCQs with 2 questions in each cluster with 5 choices for undergraduate medical students during the pre-clerkship phase integrating pharmacology and physiology related to systemic hypertension with a case vignette. Please do not include ‘none of the above’ as the choice. (This modified search query was used because test items with ‘None of the above’ option were generated with the previous search query).

Write 20 integrated case cluster MCQs with 2 questions in each cluster with 5 choices for undergraduate medical students at the time of graduation integrating pharmacology and physiology related to systemic hypertension with a case vignette.

Integrated short answer questions

Write a short answer question scenario with difficult questions based on the theme of a newly diagnosed hypertensive patient for undergraduate medical students with the main objectives related to the physiology of blood pressure regulation, risk factors for systemic hypertension, pathophysiology of systemic hypertension, pathological changes in the systemic blood vessels in hypertension, pharmacological management, and non-pharmacological treatment of systemic hypertension.

Write a short answer question scenario with moderately difficult questions based on the theme of a newly diagnosed hypertensive patient for undergraduate medical students with the main objectives related to the physiology of blood pressure regulation, risk factors for systemic hypertension, pathophysiology of systemic hypertension, pathological changes in the systemic blood vessels in hypertension, pharmacological management, and non-pharmacological treatment of systemic hypertension.

Write a short answer question scenario with questions based on the theme of a newly diagnosed hypertensive patient for undergraduate medical students at the time of graduation with the main objectives related to the physiology of blood pressure regulation, risk factors for systemic hypertension, pathophysiology of systemic hypertension, pathological changes in the systemic blood vessels in hypertension, pharmacological management, and non-pharmacological treatment of systemic hypertension.

Can you generate 5 OSPE pharmacology and therapeutics prescription writing exercises for the assessment of undergraduate medical students at the time of graduation related to anti-hypertensive drugs?

Can you generate 5 OSPE pharmacology and therapeutics prescription writing exercises containing appropriate instructions for the patients for the assessment of undergraduate medical students during their pre-clerkship phase related to anti-hypertensive drugs?

Can you generate 5 OSPE pharmacology and therapeutics prescription writing exercises containing appropriate instructions for the patients for the assessment of undergraduate medical students at the time of graduation related to anti-hypertensive drugs?

Both authors independently evaluated the AI-generated outputs, and a consensus was reached. We cross-checked the veracity of answers suggested by AIs as per the Joint National Commission Guidelines (JNC-8) and Goodman and Gilman’s The Pharmacological Basis of Therapeutics (2023), a reference textbook [ 13 , 14 ]. Errors in the A-type MCQs were categorized as item construction defects, multiple correct answers, and uncertain appropriateness to the learner’s level. Test items in the integrated case cluster MCQs, SAQs and OSPEs were evaluated with the Preliminary Conceptual Framework for Establishing Content Validity of AI-Generated Test Items based on the following domains: technical accuracy, comprehensiveness, education level, and lack of construction defects (Table  1 ). The responses were categorized as complete and deficient for each domain.

The pre-clerkship phase SLOs identified by Sage Poe, Claude-Instant, and ChatGPT are listed in the electronic supplementary materials 1 – 3 , respectively. In general, a broad homology in SLOs generated by the three AI platforms was observed. All AI platforms identified appropriate action verbs as per Bloom’s taxonomy to state the SLO; action verbs such as describe, explain, recognize, discuss, identify, recommend, and interpret are used to state the learning outcome. The specific, measurable, achievable, relevant, time-bound (SMART) SLOs generated by each AI platform slightly varied. All key domains of antihypertensive pharmacology to be achieved during the pre-clerkship (pre-clinical) years were relevant for graduating doctors. The SLOs addressed current JNC Treatment Guidelines recommended classes of antihypertensive drugs, the mechanism of action, pharmacokinetics, adverse effects, indications/contraindications, dosage adjustments, monitoring therapy, and principles of monotherapy and combination therapy.

The SLOs to be achieved by undergraduate medical students at the time of graduation identified by Sage Poe, Claude-Instant, and ChatGPT listed in electronic supplementary materials 4 – 6 , respectively. The identified SLOs emphasize the application of pharmacology knowledge within a clinical context, focusing on competencies needed to function independently in early residency stages. These SLOs go beyond knowledge recall and mechanisms of action to encompass competencies related to clinical problem-solving, rational prescribing, and holistic patient management. The SLOs generated require higher cognitive ability of the learner: action verbs such as demonstrate, apply, evaluate, analyze, develop, justify, recommend, interpret, manage, adjust, educate, refer, design, initiate & titrate were frequently used.

The MCQs for the pre-clerkship phase identified by Sage Poe, Claude-Instant, and ChatGPT listed in the electronic supplementary materials 7 – 9 , respectively, and those identified with the search query based on the clinical vignette in electronic supplementary materials ( 10 – 12 ).

All MCQs generated by the AIs in each of the four domains specified [mechanism of action (MOA); pharmacokinetics; adverse drug reactions (ADRs), and indications for antihypertensive drugs] are quality test items with potential content validity. The test items on MOA generated by Sage Poe included themes such as renin-angiotensin-aldosterone (RAAS) system, beta-adrenergic blockers (BB), calcium channel blockers (CCB), potassium channel openers, and centrally acting antihypertensives; on pharmacokinetics included high oral bioavailability/metabolism in liver [angiotensin receptor blocker (ARB)-losartan], long half-life and renal elimination [angiotensin converting enzyme inhibitors (ACEI)-lisinopril], metabolism by both liver and kidney (beta-blocker (BB)-metoprolol], rapid onset- short duration of action (direct vasodilator-hydralazine), and long-acting transdermal drug delivery (centrally acting-clonidine). Regarding the ADR theme, dry cough, angioedema, and hyperkalemia by ACEIs in susceptible patients, reflex tachycardia by CCB/amlodipine, and orthostatic hypotension by CCB/verapamil addressed. Clinical indications included the drug of choice for hypertensive patients with concomitant comorbidity such as diabetics (ACEI-lisinopril), heart failure and low ejection fraction (BB-carvedilol), hypertensive urgency/emergency (alpha cum beta receptor blocker-labetalol), stroke in patients with history recurrent stroke or transient ischemic attack (ARB-losartan), and preeclampsia (methyldopa).

Almost similar themes under each domain were identified by the Claude-Instant AI platform with few notable exceptions: hydrochlorothiazide (instead of clonidine) in MOA and pharmacokinetics domains, respectively; under the ADR domain ankle edema/ amlodipine, sexual dysfunction and fatigue in male due to alpha-1 receptor blocker; under clinical indications the best initial monotherapy for clinical scenarios such as a 55-year old male with Stage-2 hypertension; a 75-year-old man Stage 1 hypertension; a 35-year-old man with Stage I hypertension working on night shifts; and a 40-year-old man with stage 1 hypertension and hyperlipidemia.

As with Claude-Instant AI, ChatGPT-generated test items on MOA were mostly similar. However, under the pharmacokinetic domain, immediate- and extended-release metoprolol, the effect of food to enhance the oral bioavailability of ramipril, and the highest oral bioavailability of amlodipine compared to other commonly used antihypertensives were the themes identified. Whereas the other ADR themes remained similar, constipation due to verapamil was a new theme addressed. Notably, in this test item, amlodipine was an option that increased the difficulty of this test item because amlodipine therapy is also associated with constipation, albeit to a lesser extent, compared to verapamil. In the clinical indication domain, the case description asking “most commonly used in the treatment of hypertension and heart failure” is controversial because the options listed included losartan, ramipril, and hydrochlorothiazide but the suggested correct answer was ramipril. This is a good example to stress the importance of vetting the AI-generated MCQ by experts for content validity and to assure robust psychometrics. The MCQ on the most used drug in the treatment of “hypertension and diabetic nephropathy” is more explicit as opposed to “hypertension and diabetes” by Claude-Instant because the therapeutic concept of reducing or delaying nephropathy must be distinguished from prevention of nephropathy, although either an ACEI or ARB is the drug of choice for both indications.

It is important to align student assessment to the curriculum; in the PBL curriculum, MCQs with a clinical vignette are preferred. The modification of the query specifying the search to generate MCQs with a clinical vignette on domains specified previously gave appropriate output by all three AI platforms evaluated (Sage Poe; Claude- Instant; Chat GPT). The scenarios generated had a good clinical fidelity and educational fit for the pre-clerkship student perspective.

The errors observed with AI outputs on the A-type MCQs are summarized in Table  2 . No significant pattern was observed except that Claude-Instant© generated test items in a stereotyped format such as the same choices for all test items related to pharmacokinetics and indications, and all the test items in the ADR domain are linked to the mechanisms of action of drugs. This illustrates the importance of reviewing AI-generated test items by content experts for content validity to ensure alignment with evidence-based medicine and up-to-date treatment guidelines.

The test items generated by ChatGPT had the advantage of explanations supplied rendering these more useful for learners to support self-study. The following examples illustrate this assertion: “ A patient with hypertension is started on a medication that works by blocking beta-1 receptors in the heart (metoprolol)”. Metoprolol is a beta blocker that works by blocking beta-1 receptors in the heart, which reduces heart rate and cardiac output, resulting in a decrease in blood pressure. However, this explanation is incomplete because there is no mention of other less important mechanisms, of beta receptor blockers on renin release. Also, these MCQs were mostly recall type: Which of the following medications is known to have a significant first-pass effect? The explanation reads: propranolol is known to have a significant first pass-effect, meaning that a large portion of the drug is metabolized by the liver before it reaches systemic circulation. Losartan, amlodipine, ramipril, and hydrochlorothiazide do not have significant first-pass effect. However, it is also important to extend the explanation further by stating that the first-pass effect of propranolol does not lead to total loss of pharmacological activity because the metabolite hydroxy propranolol also has potent beta-blocking activity. Another MCQ test item had a construction defect: “A patient with hypertension is started on a medication that can cause photosensitivity. Which of the following medications is most likely responsible?” Options included: losartan, amlodipine, ramipril, hydrochlorothiazide, hydrochlorothiazide/triamterene. The explanation states that hydrochlorothiazide (HCTZ) can cause photosensitivity, which is increased sensitivity to sunlight that can cause skin rashes and sunburns. Losartan, amlodipine, ramipril, and HCTZ/triamterene are less likely to cause photosensitivity. However, it is irrational to claim that HCTZ/triamterene, one of the wrong options, is less likely to cause photosensitivity. The difficulty index of such test items is likely unacceptable in high-stake tests to make equitable psychometric decisions.

The responses from Sage Poe, Claude-Instant, and ChatGPT on guidance for standard-setting methods are listed in the electronic supplementary materials 13 – 15 . Sage Poe offered this information with the caveat that “the results of this simulation were purely hypothetical and do not reflect the actual performance of any real students”. The simulation estimates of examinees correctly answering test items ranged from 60 to 90% for the mechanism of action, 65–85% for pharmacokinetics, 60–80% for adverse reactions, and 75–90% for clinical indications of antihypertensives.

In contrast, Claude-Instant admitted that “Performing a full simulation with 100 medical students is beyond the scope of what I can do as an AI assistant. I can generate sample MCQ questions and answers, but I do not have access to a population of medical students to administer the questions. A true simulation that administers questions to real students and analyzes the results would require access to a database of medical students, a platform to deliver the questions, and code to analyze the responses. As an AI assistant, I lack these capabilities. While I cannot perform a full simulation, here are some considerations for administering MCQ questions to medical students: provide details on the purpose and instructions for the test to ensure students understand what is expected out of them; consider including some easier questions in addition to more challenging ones to boost students’ confidence and motivation; avoid ambiguous wording and ensure each question has only one unambiguously correct answer option; when analyzing results, look for questions factors that may have impacted performance, like unclear wording or answer options that are too similar. Focusing on weak questions can help improve future tests; compare performance across different student groups to identify areas where more teaching may be needed; provide detailed answer explanations and review sessions after the test to help students learn from any errors and reinforce correct concepts”. However, ChatGPT offered the simulation result estimates for the 20 MCQ on antihypertensive drugs: mechanism of action ranged from 67 to 92%, pharmacokinetics 63–86%, adverse effects 65–82%, and clinical indications 64–89%. Furthermore, it also stated that “Overall, the performance of the students was quite good, with most questions having a response rate of over 70%. However, there were some questions where the projected correct response rate was rather low, such as Question #5 (Mechanism of Action of Hydrochlorothiazide; see Electronic Supplementary Material 12 ) and Question 18 (Indications for Verapamil; see Electronic Supplementary Material 10 ). This may suggest areas where students need more focused education or review.”

We asked AI assistants to generate 20 integrated case cluster MCQs with 2 test items in each cluster with five options for undergraduate medical students in the pre-clerkship phase integrating pharmacology and physiology related to systemic hypertension with a case vignette and the responses by Sage Poe, Claude-Instant, and ChatGPT are listed in the electronic supplementary materials ( 16 – 18 ). In all instances, the test items generated had focused case descriptions in the form of a clinical vignette, and horizontal integration across the pathophysiology of hypertension and pharmacology of antihypertensive drugs. These test items mostly targeted the ‘knows (knowledge)’ or ‘knows how (competence)’ level on Miller’s pyramid and are suitable for assessing the clinical competence of pre-clerkship medical students, especially in an integrated PBL curriculum. Both the AI assistants generated excellent clinical vignettes and themes; however, most of the cluster MCQs by ChatGPT had “None of the above” as an option, which is often considered a test item construction flaw. Notwithstanding these limitations, case cluster integrated test items are valuable for learners to integrate their knowledge of different basic medical sciences and their application to clinical sciences. This integrated approach can be used for both instructional and student assessment purposes to make the course more meaningful. Indeed, one of the basic tenets of PBL is curriculum integration.

In the next query, we asked AI assistants to write integrated case cluster MCQs with 2 test items in each cluster with 5 options for undergraduate medical students at the time of graduation integrating pharmacology and physiology of hypertension with a case vignette. The responses are listed in the electronic supplementary materials 19 – 21 . Claude-Instant has provided only partial responses to both the queries. Overall, the information provided in case vignettes requires the learner to interpret the clinical information and make therapeutic decisions. Such test items are appropriate for assessing the minimum competency of medical school graduates to start internship or residency programs; these MCQs are generally comparable to test items used in USMLE Part 2. The evaluation of the integrated case cluster test items provided by all the AI platforms is summarized in Table  3 . In general, the AI outputs had at least one construction defect, and did not include all the core sub-topics to a sufficient degree.

When asked to simulate the above-generated test items to 100 undergraduate medical students at the time of graduation and provide a percentage of students who are likely to correctly answer each item, the AI assistant responded:

“ As an AI language model, I cannot simulate the responses of 100 undergraduate medical students likely to obtain the correct answer for each question. However, I can provide some general information about the difficulty level of each question based on the complexity of the concepts being tested and the level of integration between pharmacology and physiology ”.

AI assistant went on to clarify that questions that require knowledge of basic pharmacology principles, such as the mechanism of action of specific drugs, are likely to be easier for students to answer correctly. Test items that require an understanding of the physiological mechanisms underlying hypertension and correlating with symptoms are likely to be more challenging for students. The AI assistant sorted these test items into two categories accordingly. Overall, the difficulty level of the test item is based on the level of integration between pharmacology and pathophysiology. Test items that require an understanding of both pharmacological and physiological mechanisms are likely to be more challenging for students requiring a strong foundation in both pharmacology and physiology concepts to be able to correctly answer integrated case-cluster MCQs.

Short answer questions

The responses to a search query on generating SAQs appropriate to the pre-clerkship phase Sage Poe, Claude-Instant, and ChatGPT generated items are listed in the electronic supplementary materials 22 – 24 for difficult questions and 25–27 for moderately difficult questions.

It is apparent from these case vignette descriptions that the short answer question format varied. Accordingly, the scope for asking individual questions for each scenario is open-ended. In all instances, model answers are supplied which are helpful for the course instructor to plan classroom lessons, identify appropriate instructional methods, and establish rubrics for grading the answer scripts, and as a study guide for students.

We then wanted to see to what extent AI can differentiate the difficulty of the SAQ by replacing the search term “difficult” with “moderately difficult” in the above search prompt: the changes in the revised case scenarios are substantial. Perhaps the context of learning and practice (and the level of the student in the MD/medical program) may determine the difficulty level of SAQ generated. It is worth noting that on changing the search from cardiology to internal medicine rotation in Sage Poe the case description also changed. Thus, it is essential to select an appropriate AI assistant, perhaps by trial and error, to generate quality SAQs. Most of the individual questions tested stand-alone knowledge and did not require students to demonstrate integration.

The responses of Sage Poe, Claude-Instant, and ChatGPT for the search query to generate SAQs at the time of graduation are listed in the electronic supplementary materials 28 – 30 . It is interesting to note how AI assistants considered the stage of the learner while generating the SAQ. The response by Sage Poe is illustrative for comparison. “You are a newly graduated medical student who is working in a hospital” versus “You are a medical student in your pre-clerkship.”

Some questions were retained, deleted, or modified to align with competency appropriate to the context (Electronic Supplementary Materials 28 – 30 ). Overall, the test items at both levels from all AI platforms were technically accurate and thorough addressing the topics related to different disciplines (Table  3 ). The differences in learning objective transition are summarized in Table  4 . A comparison of learning objectives revealed that almost all objectives remained the same except for a few (Table  5 ).

A similar trend was apparent with test items generated by other AI assistants, such as ChatGPT. The contrasting differences in questions are illustrated by the vertical integration of basic sciences and clinical sciences (Table  6 ).

Taken together, these in-depth qualitative comparisons suggest that AI assistants such as Sage Poe and ChatGPT consider the learner’s stage of training in designing test items, learning outcomes, and answers expected from the examinee. It is critical to state the search query explicitly to generate quality output by AI assistants.

The OSPE test items generated by Claude-Instant and ChatGPT appropriate to the pre-clerkship phase (without mentioning “appropriate instructions for the patients”) are listed in the electronic supplementary materials 31 and 32 and with patient instructions on the electronic supplementary materials 33 and 34 . For reasons unknown, Sage Poe did not provide any response to this search query.

The five OSPE items generated were suitable to assess the prescription writing competency of pre-clerkship medical students. The clinical scenarios identified by the three AI platforms were comparable; these scenarios include patients with hypertension and impaired glucose tolerance in a 65-year-old male, hypertension with chronic kidney disease (CKD) in a 55-year-old woman, resistant hypertension with obstructive sleep apnea in a 45-year-old man, and gestational hypertension at 32 weeks in a 35-year-old (Claude-Instant AI). Incorporating appropriate instructions facilitates the learner’s ability to educate patients and maximize safe and effective therapy. The OSPE item required students to write a prescription with guidance to start conservatively, choose an appropriate antihypertensive drug class (drug) based on the patients’ profile, specifying drug name, dose, dosing frequency, drug quantity to be dispensed, patient name, date, refill, and caution as appropriate, in addition to prescribers’ name, signature, and license number. In contrast, ChatGPT identified clinical scenarios to include patients with hypertension and CKD, hypertension and bronchial asthma, gestational diabetes, hypertension and heart failure, and hypertension and gout (ChatGPT). Guidance for dosage titration, warnings to be aware, safety monitoring, and frequency of follow-up and dose adjustment. These test items are designed to assess learners’ knowledge of P & T of antihypertensives, as well as their ability to provide appropriate instructions to patients. These clinical scenarios for writing prescriptions assess students’ ability to choose an appropriate drug class, write prescriptions with proper labeling and dosing, reflect drug safety profiles, and risk factors, and make modifications to meet the requirements of special populations. The prescription is required to state the drug name, dose, dosing frequency, patient name, date, refills, and cautions or instructions as needed. A conservative starting dose, once or twice daily dosing frequency based on the drug, and instructions to titrate the dose slowly if required.

The responses from Claude-Instant and ChatGPT for the search query related to generating OSPE test items at the time of graduation are listed in electronic supplementary materials 35 and 36 . In contrast to the pre-clerkship phase, OSPEs generated for graduating doctors’ competence assessed more advanced drug therapy comprehension. For example, writing a prescription for:

(1) A 65-year- old male with resistant hypertension and CKD stage 3 to optimize antihypertensive regimen required the answer to include starting ACEI and diuretic, titrating the dosage over two weeks, considering adding spironolactone or substituting ACEI with an ARB, and need to closely monitor serum electrolytes and kidney function closely.

(2) A 55-year-old woman with hypertension and paroxysmal arrhythmia required the answer to include switching ACEI to ARB due to cough, adding a CCB or beta blocker for rate control needs, and adjusting the dosage slowly and monitoring for side effects.

(3) A 45-year-old man with masked hypertension and obstructive sleep apnea require adding a centrally acting antihypertensive at bedtime and increasing dosage as needed based on home blood pressure monitoring and refer to CPAP if not already using one.

(4) A 75-year-old woman with isolated systolic hypertension and autonomic dysfunction to require stopping diuretic and switching to an alpha blocker, upward dosage adjustment and combining with other antihypertensives as needed based on postural blood pressure changes and symptoms.

(5) A 35-year-old pregnant woman with preeclampsia at 29 weeks require doubling methyldopa dose and consider adding labetalol or nifedipine based on severity and educate on signs of worsening and to follow-up immediately for any concerning symptoms.

These case scenarios are designed to assess the ability of the learner to comprehend the complexity of antihypertensive regimens, make evidence-based regimen adjustments, prescribe multidrug combinations based on therapeutic response and tolerability, monitor complex patients for complications, and educate patients about warning signs and follow-up.

A similar output was provided by ChatGPT, with clinical scenarios such as prescribing for patients with hypertension and myocardial infarction; hypertension and chronic obstructive pulmonary airway disease (COPD); hypertension and a history of angina; hypertension and a history of stroke, and hypertension and advanced renal failure. In these cases, wherever appropriate, pharmacotherapeutic issues like taking ramipril after food to reduce side effects such as giddiness; selection of the most appropriate beta-blocker such as nebivolol in patients with COPD comorbidity; the importance of taking amlodipine at the same time every day with or without food; preference for telmisartan among other ARBs in stroke; choosing furosemide in patients with hypertension and edema and taking the medication with food to reduce the risk of gastrointestinal adverse effect are stressed.

The AI outputs on OSPE test times were observed to be technically accurate, thorough in addressing core sub-topics suitable for the learner’s level and did not have any construction defects (Table  3 ). Both AIs provided the model answers with explanatory notes. This facilitates the use of such OSPEs for self-assessment by learners for formative assessment purposes. The detailed instructions are helpful in creating optimized therapy regimens, and designing evidence-based regimens, to provide appropriate instructions to patients with complex medical histories. One can rely on multiple AI sources to identify, shortlist required case scenarios, and OSPE items, and seek guidance on expected model answers with explanations. The model answer guidance for antihypertensive drug classes is more appropriate (rather than a specific drug of a given class) from a teaching/learning perspective. We believe that these scenarios can be refined further by providing a focused case history along with relevant clinical and laboratory data to enhance clinical fidelity and bring a closer fit to the competency framework.

In the present study, AI tools have generated SLOs that comply with the current principles of medical education [ 15 ]. AI tools are valuable in constructing SLOs and so are especially useful for medical fraternities where training in medical education is perceived as inadequate, more so in the early stages of their academic career. Data suggests that only a third of academics in medical schools have formal training in medical education [ 16 ] which is a limitation. Thus, the credibility of alternatives, such as the AIs, is evaluated to generate appropriate course learning outcomes.

We observed that the AI platforms in the present study generated quality test items suitable for different types of assessment purposes. The AI-generated outputs were similar with minor variation. We have used generative AIs in the present study that could generate new content from their training dataset [ 17 ]. Problem-based and interactive learning approaches are referred to as “bottom-up” where learners obtain first-hand experience in solving the cases first and then indulge in discussion with the educators to refine their understanding and critical thinking skills [ 18 ]. We suggest that AI tools can be useful for this approach for imparting the core knowledge and skills related to Pharmacology and Therapeutics to undergraduate medical students. A recent scoping review evaluating the barriers to writing quality test items based on 13 studies has concluded that motivation, time constraints, and scheduling were the most common [ 19 ]. AI tools can be valuable considering the quick generation of quality test items and time management. However, as observed in the present study, the AI-generated test items nevertheless require scrutiny by faculty members for content validity. Moreover, it is important to train faculty in AI technology-assisted teaching and learning. The General Medical Council recommends taking every opportunity to raise the profile of teaching in medical schools [ 20 ]. Hence, both the academic faculty and the institution must consider investing resources in AI training to ensure appropriate use of the technology [ 21 ].

The AI outputs assessed in the present study had errors, particularly with A-type MCQs. One notable observation was that often the AI tools were unable to differentiate the differences between ACEIs and ARBs. AI platforms access several structured and unstructured data, in addition to images, audio, and videos. Hence, the AI platforms can commit errors due to extracting details from unauthenticated sources [ 22 ] created a framework identifying 28 factors for reconstructing the path of AI failures and for determining corrective actions. This is an area of interest for AI technical experts to explore. Also, this further iterates the need for human examination of test items before using them for assessment purposes.

There are concerns that AIs can memorize and provide answers from their training dataset, which they are not supposed to do [ 23 ]. Hence, the use of AIs-generated test items for summative examinations is debatable. It is essential to ensure and enhance the security features of AI tools to reduce or eliminate cross-contamination of test items. Researchers have emphasized that AI tools will only reach their potential if developers and users can access full-text non-PDF formats that help machines comprehend research papers and generate the output [ 24 ].

AI platforms may not always have access to all standard treatment guidelines. However, in the present study, it was observed that all three AI platforms generally provided appropriate test items regarding the choice of medications, aligning with recommendations from contemporary guidelines and standard textbooks in pharmacology and therapeutics. The prompts used in the study were specifically focused on the pre-clerkship phase of the undergraduate medical curriculum (and at the time of their graduation) and assessed fundamental core concepts, which were also reflected in the AI outputs. Additionally, the recommended first-line antihypertensive drug classes have been established for several decades, and information regarding their pharmacokinetics, ADRs, and indications is well-documented in the literature.

Different paradigms and learning theories have been proposed to support AI in education. These paradigms include AI- directed (learner as recipient), AI-supported (learner as collaborator), and AI-empowered (learner as leader) that are based on Behaviorism, Cognitive-Social constructivism, and Connectivism-Complex adaptive systems, respectively [ 25 ]. AI techniques have potential to stimulate and advance instructional and learning sciences. More recently a three- level model that synthesizes and unifies existing learning theories to model the roles of AIs in promoting learning process has been proposed [ 26 ]. The different components of our study rely upon these paradigms and learning theories as the theoretical underpinning.

Strengths and limitations

To the best of our knowledge, this is the first study evaluating the utility of AI platforms in generating test items related to a discipline in the undergraduate medical curriculum. We have evaluated the AI’s ability to generate outputs related to most types of assessment in the undergraduate medical curriculum. The key lessons learnt for improving the AI-generated test item quality from the present study are outlined in Table  7 . We used a structured framework for assessing the content validity of the test items. However, we have demonstrated using a single case study (hypertension) as a pilot experiment. We chose to evaluate anti-hypertensive drugs as it is a core learning objective and one of the most common disorders relevant to undergraduate medical curricula worldwide. It would be interesting to explore the output from AI platforms for other common (and uncommon/region-specific) disorders, non-/semi-core objectives, and disciplines other than Pharmacology and Therapeutics. An area of interest would be to look at the content validity of the test items generated for different curricula (such as problem-based, integrated, case-based, and competency-based) during different stages of the learning process. Also, we did not attempt to evaluate the generation of flowcharts, algorithms, or figures for generating test items. Another potential area for exploring the utility of AIs in medical education would be repeated procedural practices such as the administration of drugs through different routes by trainee residents [ 27 ]. Several AI tools have been identified for potential application in enhancing classroom instructions and assessment purposes pending validation in prospective studies [ 28 ]. Lastly, we did not administer the AI-generated test items to students and assessed their performance and so could not comment on the validity of test item discrimination and difficulty indices. Additionally, there is a need to confirm the generalizability of the findings to other complex areas in the same discipline as well as in other disciplines that pave way for future studies. The conceptual framework used in the present study for evaluating the AI-generated test items needs to be validated in a larger population. Future studies may also try to evaluate the variations in the AI outputs with repetition of the same queries.

Notwithstanding ongoing discussions and controversies, AI tools are potentially useful adjuncts to optimize instructional methods, test blueprinting, test item generation, and guidance for test standard-setting appropriate to learners’ stage in the medical program. However, experts need to critically review the content validity of AI-generated output. These challenges and caveats are to be addressed before the use of widespread use of AIs in medical education can be advocated.

Data availability

All the data included in this study are provided as Electronic Supplementary Materials.

Tolsgaard MG, Pusic MV, Sebok-Syer SS, Gin B, Svendsen MB, Syer MD, Brydges R, Cuddy MM, Boscardin CK. The fundamentals of Artificial Intelligence in medical education research: AMEE Guide 156. Med Teach. 2023;45(6):565–73.

Article   Google Scholar  

Sriwastwa A, Ravi P, Emmert A, Chokshi S, Kondor S, Dhal K, Patel P, Chepelev LL, Rybicki FJ, Gupta R. Generative AI for medical 3D printing: a comparison of ChatGPT outputs to reference standard education. 3D Print Med. 2023;9(1):21.

Azer SA, Guerrero APS. The challenges imposed by artificial intelligence: are we ready in medical education? BMC Med Educ. 2023;23(1):680.

Masters K. Ethical use of Artificial Intelligence in Health Professions Education: AMEE Guide 158. Med Teach. 2023;45(6):574–84.

Nagi F, Salih R, Alzubaidi M, Shah H, Alam T, Shah Z, Househ M. Applications of Artificial Intelligence (AI) in Medical Education: a scoping review. Stud Health Technol Inf. 2023;305:648–51.

Google Scholar  

Mehta N, Harish V, Bilimoria K, et al. Knowledge and attitudes on artificial intelligence in healthcare: a provincial survey study of medical students. MedEdPublish. 2021;10(1):75.

Mir MM, Mir GM, Raina NT, Mir SM, Mir SM, Miskeen E, Alharthi MH, Alamri MMS. Application of Artificial Intelligence in Medical Education: current scenario and future perspectives. J Adv Med Educ Prof. 2023;11(3):133–40.

Garg T. Artificial Intelligence in Medical Education. Am J Med. 2020;133(2):e68.

Matheny ME, Whicher D, Thadaney IS. Artificial intelligence in health care: a report from the National Academy of Medicine. JAMA. 2020;323(6):509–10.

Sage Poe. Available at: https://poe.com/Assistant (Accessed on. 3rd June 2023).

Claude-Instant: Available at: https://poe.com/Claude-instant (Accessed on 3rd. June 2023).

ChatGPT: Available at: https://poe.com/ChatGPT (Accessed on 3rd. June 2023).

James PA, Oparil S, Carter BL, Cushman WC, Dennison-Himmelfarb C, Handler J, Lackland DT, LeFevre ML, MacKenzie TD, Ogedegbe O, Smith SC Jr, Svetkey LP, Taler SJ, Townsend RR, Wright JT Jr, Narva AS, Ortiz E. 2014 evidence-based guideline for the management of high blood pressure in adults: report from the panel members appointed to the Eighth Joint National Committee (JNC 8). JAMA. 2014;311(5):507–20.

Eschenhagen T. Treatment of hypertension. In: Brunton LL, Knollmann BC, editors. Goodman & Gilman’s the pharmacological basis of therapeutics. 14th ed. New York: McGraw Hill; 2023.

Shabatura J. September. Using Bloom’s taxonomy to write effective learning outcomes. https://tips.uark.edu/using-blooms-taxonomy/ (Accessed on 19th 2023).

Trainor A, Richards JB. Training medical educators to teach: bridging the gap between perception and reality. Isr J Health Policy Res. 2021;10(1):75.

Boscardin C, Gin B, Golde PB, Hauer KE. ChatGPT and generative artificial intelligence for medical education: potential and opportunity. Acad Med. 2023. https://doi.org/10.1097/ACM.0000000000005439 . (Published ahead of print).

Duong MT, Rauschecker AM, Rudie JD, Chen PH, Cook TS, Bryan RN, Mohan S. Artificial intelligence for precision education in radiology. Br J Radiol. 2019;92(1103):20190389.

Karthikeyan S, O’Connor E, Hu W. Barriers and facilitators to writing quality items for medical school assessments - a scoping review. BMC Med Educ. 2019;19(1):123.

Developing teachers and trainers in undergraduate medical education. Advice supplementary to Tomorrow’s Doctors. (2009). https://www.gmc-uk.org/-/media/documents/Developing_teachers_and_trainers_in_undergraduate_medical_education___guidance_0815.pdf_56440721.pdf (Accessed on 19th September 2023).

Cooper A, Rodman A. AI and Medical Education - A 21st-Century Pandora’s Box. N Engl J Med. 2023;389(5):385–7.

Chanda SS, Banerjee DN. Omission and commission errors underlying AI failures. AI Soc. 2022;17:1–24.

Narayanan A, Kapoor S. ‘GPT-4 and Professional Benchmarks: The Wrong Answer to the Wrong Question’. Substack newsletter. AI Snake Oil (blog). https://aisnakeoil.substack.com/p/gpt-4-and-professional-benchmarks (Accessed on 19th September 2023).

Brainard J. November. As scientists face a flood of papers, AI developers aim to help. Science, 21 2023. doi.10.1126/science.adn0669.

Ouyang F, Jiao P. Artificial intelligence in education: the three paradigms. Computers Education: Artif Intell. 2021;2:100020.

Gibson D, Kovanovic V, Ifenthaler D, Dexter S, Feng S. Learning theories for artificial intelligence promoting learning processes. Br J Edu Technol. 2023;54(5):1125–46.

Guerrero DT, Asaad M, Rajesh A, Hassan A, Butler CE. Advancing Surgical Education: the Use of Artificial Intelligence in Surgical Training. Am Surg. 2023;89(1):49–54.

Lee S. AI tools for educators. EIT InnoEnergy Master School Teachers Conference. 2023. https://www.slideshare.net/ignatia/ai-toolkit-for-educators?from_action=save (Accessed on 24th September 2023).

Download references

Author information

Authors and affiliations.

Department of Pharmacology & Therapeutics, College of Medicine & Medical Sciences, Arabian Gulf University, Manama, Kingdom of Bahrain

Kannan Sridharan & Reginald P. Sequeira

You can also search for this author in PubMed   Google Scholar

Contributions

RPS– Conceived the idea; KS– Data collection and curation; RPS and KS– Data analysis; RPS and KS– wrote the first draft and were involved in all the revisions.

Corresponding author

Correspondence to Kannan Sridharan .

Ethics declarations

Ethics approval and consent to participate.

Not applicable as neither there was any interaction with humans, nor any personal data was collected in this research study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Sridharan, K., Sequeira, R.P. Artificial intelligence and medical education: application in classroom instruction and student assessment using a pharmacology & therapeutics case study. BMC Med Educ 24 , 431 (2024). https://doi.org/10.1186/s12909-024-05365-7

Download citation

Received : 26 September 2023

Accepted : 28 March 2024

Published : 22 April 2024

DOI : https://doi.org/10.1186/s12909-024-05365-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Medical education
• Pharmacology
• Therapeutics

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Open access
• Published: 23 April 2024

A mixed methods evaluation of the impact of ECHO ® telementoring model for capacity building of community health workers in India

• Rajmohan Panda 1 ,
• Supriya Lahoti   ORCID: orcid.org/0000-0001-6826-5273 2 ,
• Nivedita Mishra 2 ,
• Rajath R. Prabhu 3 ,
• Kalpana Singh 4 ,
• Apoorva Karan Rai 2 &
• Kumud Rai 2  

Human Resources for Health volume  22 , Article number:  26 ( 2024 ) Cite this article

61 Accesses

1 Altmetric

Metrics details

Introduction

India has the largest cohort of community health workers with one million Accredited Social Health Activists (ASHAs). ASHAs play vital role in providing health education and promoting accessible health care services in the community. Despite their potential to improve the health status of people, they remain largely underutilized because of their limited knowledge and skills. Considering this gap, Extension for Community Healthcare Outcomes (ECHO) ® India, in collaboration with the National Health System Resource Centre (NHSRC), implemented a 15-h (over 6 months) refresher training for ASHAs using a telementoring interface. The present study intends to assess the impact of the training program for improving the knowledge and skills of ASHA workers.

We conducted a pre–post quasi-experimental study using a convergent parallel mixed-method approach. The quantitative survey ( n  = 490) assessed learning competence, performance, and satisfaction of the ASHAs. In addition to the above, in-depth interviews with ASHAs ( n  = 12) and key informant interviews with other stakeholders ( n  = 9) examined the experience and practical applications of the training. Inferences from the quantitative and qualitative approaches were integrated during the reporting stage and presented using an adapted Moore’s Expanded Outcomes Framework.

There was a statistically significant improvement in learning ( p =  0.038) and competence ( p =  0.01) after attending the training. Participants were satisfied with the opportunity provided by the teleECHO™ sessions to upgrade their knowledge. However, internet connectivity, duration and number of participants in the sessions were identified as areas that needed improvement for future training programs. An improvement in confidence to communicate more effectively with the community was reported. Positive changes in the attitudes of ASHAs towards patient and community members were also reported after attending the training. The peer-to-peer learning through case-based discussion approach helped ensure that the training was relevant to the needs and work of the ASHAs.

Conclusions

The ECHO Model ™ was found effective in improving and updating the knowledge and skills of ASHAs across different geographies in India. Efforts directed towards knowledge upgradation of ASHAs are crucial for strengthening the health system at the community level. The findings of this study can be used to guide future training programs.

Trial registration The study has been registered at the Clinical Trials Registry, India (CTRI/2021/10/037189) dated 08/10/2021.

Peer Review reports

The Alma Ata Declaration of 1978 has recognized primary health care as an essential element for improving community health. Community health workers (CHWs) have the potential to complement an overstrained health workforce and enhance primary healthcare access and quality [ 1 ]. Low- and middle-income countries (LMICs) face a triple burden of low density of doctors and nurse-midwives, low government expenditure on health, and disproportionately larger poor health outcomes [ 2 ]. The roles and responsibilities of CHWs vary across LMICs [ 3 ]. A systematic review has documented that the socio-cultural, economic, health system, and political context in which CHW interventions operate in LMICs influence the implementation and success of interventions [ 4 ].

The National Rural Health Mission (NRHM), India introduced Accredited Social Health Activists (ASHAs) as female CHWs in 2005. The ASHAs are women volunteers selected from the local village and were initially conceptualized with a vision to improve maternal and child health in the country; however, over time, they are now involved in different national health programmes [ 5 , 6 ]. Despite their potential to contribute to preventive and promotive healthcare, they remain largely underutilized because of their limited knowledge and skills [ 1 ]. The World Health Organisation (WHO) has suggested ‘regular training and supervision’ for CHWs to fulfil their role successfully [ 7 ]. In India, the health system lacks methods for continuous education and routine upgradation of the ASHA’s skills [ 8 , 9 , 10 ].

In LMICs, digital training programs can help expand the reach of training to large numbers of healthcare workers at a low cost without interfering with the delivery of routine healthcare services [ 11 , 12 ]. An evidence-mapping study of 88 studies that used technology for training CHWs in LMICs found that the focus of trainings was maternal and child health and other high-burden diseases were neglected [ 13 ]. In India, studies evaluating digital trainings for CHWs have focussed on specific diseases or have been limited to specific states in the last decade [ 10 , 14 ]. This study was conducted across multiple states. More such studies with larger sample size are needed on the evaluation of such training initiatives in India [ 13 , 15 , 16 ].

Project Extension for Community Healthcare Outcomes (ECHO) presents an educational opportunity for capacity-building through a telementoring platform that uses video conferencing to create a continuous loop of learning and peer support. The sessions are facilitated by didactic presentation and case-based learning that allows problem-solving through shared best practices [ 17 ]. ECHO India, in collaboration with National Health System Resource Centre (NHSRC), provided refresher training for ASHAs [ 18 ]. There is increasing evidence of the positive effect of ECHO training on medical provider’s learning and self-efficacy. However, its value as a training platform to CHWs in LMICs is limited. Previous studies that evaluated the use of the ECHO Model ™ for CHWs focussed on specific diseases and were conducted in high-income countries (HICs) [ 19 , 20 , 21 ]. For the adoption of digital technology, CHWs in LMICs encounter challenges such as poor proficiency levels in accessing and using digital platforms, limited access to troubleshooting, poor internet connectivity, and in-house support for resolving issues [ 22 ]. The present study was designed to assess the impact of the ECHO telementoring model for improving the knowledge and skills of ASHA workers in delivering comprehensive health services. This will provide new insights for measuring outcomes of digital training programs for CHWs (ASHA workers).

Study design

A pre–post quasi-experimental design using a convergent parallel mixed-method approach [ 23 ] was employed. The quantitative and qualitative data were collected concurrently. Inferences from both approaches were integrated during the reporting stage. This allowed for a comprehensive understanding of the effect of training on the knowledge and skills of ASHAs.

The ECHO training intervention and curriculum

Project ECHO ® designed a 15-h (over 6 months from October 2021 to March 2022), virtual, refresher training program to enhance the capacity of ASHAs to deliver counselling services for comprehensive healthcare in four states ( n  = 2293). Each session lasted for 90 min. The ECHO NHSRC training used a “hub and spoke” structure in which a multidisciplinary team of experts (trainers) based at a regional academic medical centre (the “hub”) engaged with the ASHAs (the “spokes”) [ 24 ] who attended the sessions from dedicated learning sites (PHCs). Each site also had a coordinator who would help facilitate the discussions and questions. The training curriculum was developed based on the NHSRC ‘ASHA training modules’ [ 18 ] in the regional languages in consultation with partners (hub-leaders and trainers). It comprised 10 sessions covering a range of topics, such as maternal health, new-born care, child health, nutrition, reproductive health, violence against women, tuberculosis, vector-borne diseases, non-communicable diseases, COVID-19, palliative care, and mental health. The training presentations included text with visual learning methods, such as images, videos, and links to training resources.

Study settings

The evaluation study was conducted in four states of India, where training sessions were held. These states represented the four geographical regions—northern (Himachal Pradesh) ( n =  499), southern (Tamil Nadu) ( n =  500), eastern (West Bengal) ( n =  618), and north-eastern (Sikkim) ( n =  676). The intervention (training sessions) was completed in March 2022. The end-point data were collected from March 2022 to May 2022.

Study participants and recruitment

Simple random sampling was used to select the ASHAs from each state for the quantitative survey. The participants were recruited from a list of ASHAs who would be receiving the ECHO NHSRC training. To be included, ASHAs had to be enrolled in the refresher training, planning to continue working for the next 10 months, with available contact details and consenting voluntarily. The ASHAs were contacted through mobile phones in each state. Key informant interviews (KIIs) were conducted with hub leaders who were involved in implementing the training, trainers (faculty) who delivered the lectures, and in-depth interviews (IDIs) with ASHAs.

Sample size

The sample size for the quantitative study was estimated by assuming a 25% improvement in knowledge and skills, 80% power, and a design effect factor of 1.7%. An adjustment of 30% loss to follow up and 20% non-response (from previous experience) led to a sample of 591 participants across four states, i.e., 148 participants from each state. For the qualitative study, purposive sampling with maximum variation across age, education, practice sites, and years of work experience was used for the selection of the participants. A total of 12 IDIs were conducted with ASHAs and nine KIIs with stakeholders (Additional file 2 : Appendix S2).

Study tools and data collection

For quantitative data collection, a structured questionnaire was designed through a collaborative approach with the research and program implementation team. The knowledge of ASHAs was assessed by a combination of 18 technical questions and case vignettes. Learning and competence, performance, and satisfaction were assessed with a 5-point Likert scale, using 1 = Strongly Disagree; 2 = Disagree; 3 = Neither Agree nor Disagree; 4 = Agree; and 5 = Strongly Agree. The face validity of the questionnaire was tested with ten ASHAs, separate from those recruited in the study and five primary care experts. The changes related to language, clarity, and relevance were made in the questionnaire based on the feedback from experts and participants. Separate discussion guides were developed for KIIs with trainers (Additional file 3 : Appendix S3) and hub-leaders (Additional file 4 : Appendix S4) and IDIs with ASHAs (Additional file 5 : Appendix S5). The guide focussed on examining the experience and practical applications of the training and was field tested before being administered in the main study. All study tools were translated into the local languages of the states and back-translated to check discrepancies.

The data were collected on the cell phone by experienced and trained researchers from social sciences backgrounds. Due to telephonic data collection, we were unable to capture non-verbal interview data such as emotions or gestures, particularly important in qualitative data. This may affect the richness of data and interpretation of responses. The quantitative tool was designed in the CS Pro software (version 7.5) and data were collected using its smartphone application. The qualitative interviews lasted around 40–50 min and were audio recorded. All interviews were translated and transcribed verbatim.

Data analysis

We summarized the quantitative data using descriptive statistics. Continuous variables were summarized using mean ± SD, and categorical variables were summarized using percentages and frequencies. The responses recorded using the 5-point Likert scale were recategorized during the analysis into three categories, i.e., ‘agree’ (combining strongly agree and agree), ‘disagree’ (combining strongly disagree and disagree), and ‘neutral [ 25 ]. Paired t test was used to find the difference between the pre- and post-scores of learning and competence and the attitude of participants toward ECHO training. McNemar’s test was used to assess changes in pre- and post-test scores for the technical domain. A p value of less than 0.05 was considered significant. STATA 16.0 statistical software was used for the analysis.

Qualitative data were analyzed according to the principles of the Framework approach [ 26 ], which combines inductive and deductive approaches. As a first step, two authors (SL and NM) familiarized themselves with four randomly selected transcripts and independently coded them using initial codes that were developed based on Moore’s framework levels of participation, satisfaction, learning, competence, and performance [ 27 ]. New codes that emerged while undertaking the analysis were included. The discussion and comparison of the double-coded transcripts enabled the development of an agreed set of codes. Any disagreements were discussed and resolved with the help of the third author (RP) to achieve inter-coder agreement. A final codebook was developed and applied to all the transcripts. The codes were combined and categorized into key emerging themes., The themes, including quotes (respondents’ exact words), were included to represent the main findings. Atlas.ti (version 8) software was used for data analysis.

Moore’s level 1—participation

Table 1 represents the baseline demographics of the recruited participants. From 610 participants who completed the pre-training survey, 490 participants completed the post-training survey, resulting in a follow-up rate of 80% (95% CI 76.6, 83.1). A total of 120 (20%, 95% CI 16.8, 23.3) participants were lost to follow up. This was due to a) contact numbers not being operational ( n =  96) and b) refusal due to time considerations ( n =  24). The field investigators attempted three additional phone calls, coordinated with hubs for participants’ alternate contact information, and offered flexible phone appointments to ensure maximum participation in the post-training survey. The majority (68%) of ASHAs were posted at sub-centres. A sub-centre is the most peripheral unit of contact of the health system with the community [ 28 ]. The majority of the participants (75%) had completed their high school (10th) education.

A hub leader described the efforts made by the ECHO to facilitate the participation of the ASHAs in the training.

“ECHO provided a facility where everyone can gather at the nearest block for the training. Physical and online modes [are] both available” (Hub-leader, Himachal Pradesh).

Moore’s level 2—satisfaction

The end-point survey assessed participants’ satisfaction with the ECHO training. The survey included eight items that measured overall training satisfaction and five items that measured satisfaction with factors specific to the telementoring model using close-ended questions. Satisfaction with the training content and environment was measured with four items. Except for one topic area (sharing of additional resources and training material), over 90% of participants were satisfied with almost all of the different components of the ECHO telementoring intervention (Additional file 1 : Appendix S1, Tables S1.1, S1.2, S1.3). While participants found the overall intervention favourable, 54.5% of all participants were dissatisfied with internet connectivity in the training sessions. Around one fourth of the participants faced challenges with the duration (31.2%), frequency (31.2%), and number of participants (28.4%) in the sessions (Additional file 1 : Appendix S1: Table S1.3).

The qualitative findings also show that most of the trainees were satisfied with the learning opportunity provided by the ECHO training.

“After attending these ECHO sessions, I felt we are constantly learning new techniques and it’s a deep sense of satisfaction” (ASHA, Tamil Nadu).

The ASHAs also shared areas or features of the ECHO model that did not meet their requirements and need improvement. They felt that the duration allotted for a session was not sufficient and some topics were covered very fast.

“They rush a lot while teaching over phone. It will be more helpful if they take more time and explain the things in a more detailed manner” (ASHA, WB)

Another ASHA suggested increasing the duration of training to improve their understanding of some topics.

"Increase the time of the training. Topics can be made deeper, and richer for better explanations" (ASHA, Tamil Nadu)

ASHAs described challenges related to connectivity while attending the training.

“The network connection was a problem and video used to lag” (ASHA, Sikkim)

Trainers shared their opinion about aspects of online trainings that did not meet their expectations.

“The problem is that they only join the meeting [online training] and do their own work, they actually do not listen properly.” (Trainer, WB)

A trainer mentioned that the large number of participants in some sessions affected the interaction among participant ASHAs.

“Sometimes a session has too many participants causing coordination efforts to be a challenge in these sessions” (Trainer, TN)

Difficulties in reaching the PHCs were recorded from the state of Sikkim. The geographical location and lack of transport facilities were mentioned by a trainer.

“We have transportation problem, our ASHA comes from rural area and it’s difficult to get taxi, which makes [it] harder to attend classes” (Trainer, Sikkim)

Many participants regarded organizational support as a facilitator for attending the training program. An ASHA from Tamil Nadu described how the issue of distance was resolved through management interventions from the organization.

“Our Block is 30 km away. There is another Block nearby that is 1 km only from here, they sent us there… so there was no problem” (ASHA, TN)

Moore’s level 3—learning

McNemar’s Chi-square statistic showed a significant difference between pre-ECHO and post-ECHO proportions in various aspects of health-related technical knowledge. Before the training, 1% of participants were aware of the correct schedule to be followed in the first week after the delivery of a child, which increased to 40% of participants post-training (p < 0.001). Overall, a statistically significant increase of 6% (95% CI 0.0003, 0.12; p =  0.038) in participants’ technical knowledge after ECHO training was found. After the training, a 7% increase in knowledge of malaria ( p =  0.002) and its symptoms and a 9% increase in knowledge of the right action to be undertaken (p < 0.001) was reported. Knowledge related to some areas such as recommended duration of physical activity or exercise (p < 0.001), immunisation after child birth ( p =  0.001), family planning in women after child birth ( p =  0.002) showed a decrease after attending the training (Additional file 1 : Appendix S1, Table S2). Post ECHO training, ASHAs reported an improvement in their knowledge of using a smartphone (switch on and off, and navigate) ( p =  0.0005) and navigating a mobile application ( p =  0.59). The ASHAs reported a 2% decrease in their knowledge of downloading content in the mobile ( p =  0.07) (Fig.  1 ).

Self-rated ICT knowledge of ASHAs

The qualitative data show that ASHAs who did not have a smartphone found it difficult to download and save content. One of the participants reported receiving additional training content in the form of a pdf file. She also mentioned that those who do not use a smartphone find it challenging to access this additional resource.

“We get the study material in a pdf so that simplifies our work further. But those who do not have a smartphone, find it difficult to get this opportunity” (ASHA, WB)

3A—Declarative learning

Declarative learning assesses how participants articulate the knowledge that the educational activity intended them to know (knowing what). The qualitative findings show that the training had increased the ASHA’s knowledge in specific domains such as breastfeeding during COVID-19.

“The doubt was whether a mother can breastfeed the baby when suffering from COVID-19. I got clarity about that… many such topics were cleared” (ASHA, Himachal Pradesh)

3B—Procedural learning

Procedural learning assesses the participants' articulation of how to do what the educational activity intended them to know (knowing how).

Participants reported that they had gained new skills related to the approach and identification of healthcare issues after attending the ECHO training.

“Earlier we wouldn’t know if ear related issues had a resolution – But following the ear related training we are aware that such issues can be cured or have treatments” (ASHA, Tamil Nadu).

The qualitative interviews revealed additional themes that described the value of the ECHO training program in improving the learning experience of ASHAs.

ASHA workers felt that the case presentations from their peers enhanced their learning experience.

“One ASHA shared a case of an anaemic mother. Based on this case we learned that this could have been prevented if iron tablets are provided from the adolescent stage” (ASHA, Tamil Nadu).

The interactive nature of the sessions and the discussions benefitted the learning experience of the ASHAs.

“Open discussion helped us so much. We can discuss any topics if we haven’t understood and sir used to explain again” (ASHA, Sikkim)

Moore’s level 4—competence

The participants reported significant improvement in their confidence to identify and manage several health conditions like birth asphyxia (for home deliveries) and management with a mucus extractor ( p =  0.01), screen and refer pregnant women ( p =  0.01), disseminate information on domestic violence and sexual harassment ( p =  0.001). Overall, a statistically significant increase of 6% (95% CI 0.01, 0.10; p =  0.01) in participants’ competence after attending the ECHO training was found. Participants reported a decrease in their confidence to track child immunisation ( p  < 0.001), monitor symptoms of COVID (p < 0.001), and clarify concerns of the community ( p  < 0.001) after attending the training (Additional file 1 : Appendix S1, Table S3).

Participants mentioned an improvement in their confidence while communicating with patients and their families.

“Initially we could not talk to people so comfortably, we hesitated at times but after being trained we can talk to people and their families properly and easily now” (ASHA, West Bengal)

An ASHA described a gap in their ability to talk to mothers in the field and suggested including more training content on efficient communication skills.

“We go on field and talk to mothers. There was no training for these, but I feel it will be good if we can have training on how to talk to mothers comfortably” (ASHA, WB)

Moore’s level 5—performance

The study identified a significant improvement in ASHAs’ positive attitude toward maternal and child health issues. Overall, a 5% improvement (95% CI − 0.009, 0.10; p value = 0.09) in participants’ attitudes post-ECHO training was found. Almost all the participants (99%) reported applying the skills learnt during the training at their workplaces. More than 90% of the participants felt that the ECHO training expanded access to healthcare in their community (Fig.  2 ). The ASHAs reported an improvement in their attitude towards inclusion of HIV patients in the community ( p =  0.01) and home visits for new born babies (p < 0.001) (Additional file 1 : Appendix S1, Table S4).

Self-reported performance of ASHAs

The ASHAs shared specific examples where they made changes in their practice or treatment strategies after attending the training.

“[Earlier] the implementation was not proper [correct]. As an example, if a child’s life has to be saved on the spot, we would take the medicines and syringes separately. Now we take the necessary items section wise including the AFI kit. So that’s the change” (ASHA, Tamil Nadu).

The results of this evaluation suggest that Project ECHO provides a suitable and efficacious platform for training for ASHAs. The participants reported an improvement in their knowledge, skills, and practices. They also described improved confidence to communicate more effectively. Some areas in which the ASHAs reported a lack of knowledge and confidence include newborn immunisation and family planning after pregnancy.

The NRHM guidelines for the recruitment of ASHAs require candidates to have at least eight or 10 completed years of formal education. Low literacy and inadequate training of ASHAs have been observed in different states in India [ 30 , 31 ]. However, with the proper training and support, ASHAs can provide comprehensive preventive and promotive healthcare services [ 29 ]. In this study, the majority (75%) of ASHAs across all states had ten or more years of schooling. The ECHO training will bolster their knowledge, skills, and confidence in providing effective services.

The ASHAs receive 23 days of training in the first year, followed by 12 days of training in every subsequent year to keep them updated with the knowledge and skills needed to effectively perform their roles and responsibilities. Previous studies have identified many challenges in the training of ASHAs, such as lack of regular refresher training [ 32 ], shortage of competent trainers, insufficient funds, and use of obsolete health information [ 33 ]. The training programs have mostly been didactic-based and had limitations in the engagement of participants [ 34 ]. The ECHO NHSRC refresher training addresses these limitations by promoting peer-to-peer learning and through a case-based discussion approach [ 35 ].

Our findings report a significant increase in the knowledge of ASHA workers with respect to specific domains like maternal and child health. A randomized controlled trial in Karnataka, India, found a significant improvement in mental health knowledge, attitude, and practice (KAP) scores amongst ASHAs trained by a hybrid training (traditional 1-day in-person classroom training and seven online sessions using the ECHO Model) against conventional classroom training [ 14 ]. This study findings highlight the improvement in knowledge of ASHAs related to oral health and palliative care post-ECHO training. An improvement in knowledge has also been observed in other studies that have evaluated ECHO telementoring interventions in cancer screening [ 36 ], palliative care [ 37 , 38 ], HIV [ 39 ], and chronic pain [ 40 ] In this study, ASHAs reported poor knowledge of the immunisation schedule for a newborn as well as the confidence to record and track immunisation in the community even after the ECHO training. A critical function of ASHAs is to assist ANMs or nurses with all immunisation activities [ 41 ]. A previous study in Karnataka in 2020 found inadequate knowledge among ASHAs about child immunisation. The above study also documented that by increasing the number of days and focusing on child care the ASHAs had a better understanding of interventions related to child healthcare [ 42 ]. As a part of the course structure, ECHO provides one session on new born and post-partum care. An assessment of the number of sessions needed to cover the topics was beyond the scope of our research but would be beneficial.

Previous studies have identified several shortcomings in ASHAs' communication and counselling abilities [ 43 , 44 , 45 ]. The findings of this study revealed that the ASHAs faced communication issues while discussing health matters related to family planning and COVID-19 with the community. Previous research has found that interpersonal communication of ASHAs are influenced by factors such as health system support and community context [ 46 ]. A study exploring the perspectives of ASHAs on a mobile training course in India also found that they encountered barriers in their interactions with beneficiaries such as resistance from family members, fear of poor quality of care, and financial costs of care [ 44 ]. Training programs must therefore, also incorporate how ASHAs can navigate social behaviours and norms to improve the impact of counselling [ 47 , 48 ]. The extent to which the ECHO training can identify and incorporate community hierarchies to improve communication of the ASHAs needs further exploration. In this study, large batch size ( n =  40) and limited use of video by participants during the training hampered the engagement between ASHAs as well as with the trainers. A previous study in the USA suggested that limiting batch size and ensuring face-to-face interactions on the virtual platform ensured a higher level of accountability and made it easier to engage with others in the ECHO training sessions [ 49 ].

CHWs face significant barriers when using digital technology in LMICs, making it challenging for them to access training on digital platforms [ 50 ]. The ASHAs in this study reported an improvement in their ability to use smartphones and navigate mobile applications. Our findings also suggest that ASHAs should be better oriented for accessing content on hand held devices.

The mentorship by trainers added value to participants’ knowledge and helped improve their skills. In this study, participants’ attitudes towards their work changed after attending the ECHO training suggesting that the learning and confidence developed during the training would be transferable to their work in healthcare settings and communities. The ECHO participants of previous studies have also demonstrated similar changes in their practices [ 35 , 40 ]. Our study findings indicate that the ECHO Model is an effective platform that can help foster a virtual community of practice through case-based learning, shared best practices, and online mentorship by experts.

Future directions

There should be more sessions on topics related to post-natal and newborn care as the ASHAs showed poor knowledge and competence in these areas.

There should be more training on counselling and development of communication skills for ASHAs, specially for maternal and child health and COVID-19.

An orientation for ASHAs should be conducted to facilitate the use of technology and the platform for learning. This may also help overcome some of the challenges described by the ASHAs in this study.

Strengths and limitations

The study used a rigorous quasi-experimental design across four different states of India. Our follow-up rate in the study was 80%, indicating a high response from participants completing the pre–post assessment. The presented study has certain limitations. It was not possible to use randomisation and a pure experimental design in this study, and this affects the internal validity of the study. The inclusion of a control group would have strengthened study validity. The self-reported outcomes can be subject to social desirability bias. We did not document the information on attendance and drop outs from the training program. The qualitative results have to be carefully interpreted because of the small sample size of the qualitative study relative to the study sample.

There is increasing recognition of the importance of CHWs globally for promoting a continuum of care and expanding access to health services. ASHA workers constitute critical human resources in the Indian health system and efforts to empower them are crucial for strengthening the health system at the community level. The encouraging results of this study indicate the effectiveness of Project ECHO in building the capacity of ASHA workers across different geographies in the country.

Availability of data and materials

All data generated or analyzed during this study are included in this published article (as Additional files).

Abbreviations

Community health workers

Sustainable development goals

National Rural Health Mission

Accredited social health activists

Digital infrastructure knowledge sharing

Ministry of Human Resource Development

Coronavirus Disease 2019

National Health System Resource Centre

World Health Organization

High-Income Countries

LMICs: Low- and Middle-Income Countries

Extension for Community Healthcare Outcomes

In-depth interviews

Key informant interviews

Continuing medical education

Institutional Ethics Committee

Participant Information Sheet

Jodhpur School of Public Health

Public Health Foundation of India

Hartzler AL, Tuzzio L, Hsu C, Wagner EH. Roles and functions of community health workers in primary care. Ann Fam Med. 2018;16(3):240–5.

Article   PubMed   PubMed Central   Google Scholar  

Olaniran A, Banke-Thomas A, Bar-Zeev S, Madaj B. Not knowing enough, not having enough, not feeling wanted: challenges of community health workers providing maternal and newborn services in Africa and Asia. PLoS ONE. 2022;17(9): e0274110.

Article   CAS   PubMed   PubMed Central   Google Scholar  

O’Donovan J, O’Donovan C, Kuhn I, Sachs SE, Winters N. Ongoing training of community health workers in low-income and middle-income countries: a systematic scoping review of the literature. BMJ Open. 2018;8(4): e021467.

Kok MC, Kane SS, Tulloch O, Ormel H, Theobald S, Dieleman M, et al. How does context influence performance of community health workers in low- and middle-income countries? Evidence from the literature. Health Res Policy Syst. 2015;13(1):13.

Saprii L, Richards E, Kokho P, Theobald S. Community health workers in rural India: analysing the opportunities and challenges accredited social health activists (ASHAs) face in realising their multiple roles. Hum Resour Health. 2015;13(1):95.

Ministry of Health and Family Welfare, Government of India. Non Communicable Disease Control Programmes: National Health Mission. 2023. https://nhm.gov.in/index1.php?lang=1&level=1&sublinkid=1041&lid=614 . Accessed 13 Feb 2023.

World Health Organization. What do we know about community health workers? A systematic review of existing reviews. 2020. https://www.who.int/publications-detail-redirect/what-do-we-know-about-community-health-workers-a-systematic-review-of-existing-reviews . Accessed 13 Feb 2023.

Bajpai N, Dholakia RH. Improving the performance of accredited social health activists in India. Mumbai: Columbia Global Centres South Asia; 2011.

Google Scholar  

Panwar DS, Naidu V, Das E, Verma S, Khan AA. Strengthening support mechanisms for accredited social health activists in order to improve home-based newborn care in Uttar Pradesh, India. BMC Proc. 2012;6(5):O33.

Article   PubMed Central   Google Scholar  

Yadav D, Singh P, Montague K, Kumar V, Sood D, Balaam M, Sharma D, Duggal M, Bartindale T, Varghese D, Olivier P. Sangoshthi: Empowering community health workers through peer learning in rural india. In: Proceedings of the 26th International Conference on World Wide Web 2017 Apr 3 (pp. 499–508).

Labrique AB, Wadhwani C, Williams KA, Lamptey P, Hesp C, Luk R, et al. Best practices in scaling digital health in low and middle income countries. Glob Health. 2018;14(1):103.

Article   Google Scholar  

Bashingwa JJH, Shah N, Mohan D, Scott K, Chamberlain S, Mulder N, et al. Examining the reach and exposure of a mobile phone-based training programme for frontline health workers (ASHAs) in 13 states across India. BMJ Glob Health. 2021;6(Suppl 5): e005299.

Winters N, Langer L, Nduku P, Robson J, O’Donovan J, Maulik P, et al. Using mobile technologies to support the training of community health workers in low-income and middle-income countries: mapping the evidence. BMJ Glob Health. 2019;4(4): e001421.

Nirisha PL, Malathesh BC, Kulal N, Harshithaa NR, Ibrahim FA, Suhas S, et al. Impact of technology driven mental health task-shifting for accredited social health activists (ASHAs): results from a randomised controlled trial of two methods of training. Commun Ment Health J. 2023;59(1):175–84.

Long LA, Pariyo G, Kallander K. Digital technologies for health workforce development in low- and middle-income countries: a scoping review. Glob Health Sci Pract. 2018;6(Supplement 1):S41–8.

Tyagi V, Khan A, Siddiqui S, KakraAbhilashi M, Dhurve P, Tugnawat D, et al. Development of a digital program for training community health workers in the detection and referral of schizophrenia in rural India. Psychiatr Q. 2023;94(2):141–63.

Article   PubMed   Google Scholar  

Arora S, Thornton K, Murata G, Deming P, Kalishman S, Dion D, et al. Outcomes of treatment for Hepatitis C virus infection by primary care providers. N Engl J Med. 2011;364(23):2199–207.

Article   CAS   PubMed   Google Scholar  

Ministry of Health and Family Welfare, Government of India. ASHA Training Modules: National Health Mission. 2022. http://nhm.gov.in/index1.php?lang=1&level=3&sublinkid=184&lid=257 . Accessed 11 Aug 2022.

Zurawski A, Komaromy M, Ceballos V, McAuley C, Arora S. Project ECHO brings innovation to community health worker training and support. J Health Care Poor Underserved. 2016;27(4):53–61.

Komaromy M, Ceballos V, Zurawski A, Bodenheimer T, Thom DH, Arora S. Extension for community healthcare outcomes (ECHO): a new model for community health worker training and support. J Public Health Policy. 2018;39(2):203–16.

Damian AJ, Robinson S, Manzoor F, Lamb M, Rojas A, Porto A, et al. A mixed methods evaluation of the feasibility, acceptability, and impact of a pilot project ECHO for community health workers (CHWs). Pilot Feasibility Stud. 2020;6(1):132.

Feroz AS, Khoja A, Saleem S. Equipping community health workers with digital tools for pandemic response in LMICs. Arch Public Health. 2021;79(1):1.

Creswell JW, Clark VLP. Designing and conducting mixed methods research. Thousand Oaks: SAGE Publications; 2017. p. 521.

Tran L, Feldman R, Riley T III, Jung J. Association of the extension for community healthcare outcomes project with use of direct-acting antiviral treatment among US adults with hepatitis C. JAMA Netw Open. 2021;4(7): e2115523.

Harpe SE. How to analyze Likert and other rating scale data. Curr Pharm Teach Learn. 2015;7(6):836–50.

Hackett A, Strickland K. Using the framework approach to analyse qualitative data: a worked example. Nurse Res. 2018;26(2):8.

Moore DEJ, Green JS, Gallis HA. Achieving desired results and improved outcomes: Integrating planning and assessment throughout learning activities. J Contin Educ Health Prof. 2009;29(1):1.

Chokshi M, Patil B, Khanna R, Neogi SB, Sharma J, Paul VK, et al. Health systems in India. J Perinatol. 2016;36(Suppl 3):S9-12.

National health systems resource centre. ASHA: which way forward. Evaluation of the ASHA Programme. 2010. https://nhm.gov.in/images/pdf/communitisation/asha/Studies/Evaluation_of_ASHA_Program_2010-11_Report.pdf . Accessed 17 Dec 2022.

National health systems resource centre. Tenth common review mission: National health mission. https://nhm.gov.in/images/pdf/monitoring/crm/10th-crm/Report/10th_CRM_Main_Report.pdf . Accessed 29 Dec 2022.

DeRenzi B, Wacksman J, Dell N, Lee S, Lesh N, Borriello G, Ellner A. Closing the feedback loop: a 12-month evaluation of ASTA, a self-tracking application for ASHAs. In: DeRenzi B, editor. Proceedings of the Eighth international conference on information and communication technologies and development. New York: Association for Computing Machinery; 2016. p. 1–10. https://doi.org/10.1145/2909609.2909652 .

Chapter   Google Scholar  

Yadav D. Low-cost mobile learning solutions for community health workers. In: Yadav D, editor. Proceedings of the 26th international on world wide web companion. Geneva: International World Wide Web Conferences Steering Committee; 2017. p. 729–34. https://doi.org/10.1145/3041021.3053377 .

Molapo M, Marsden G. Health education in rural communities with locally produced and locally relevant multimedia content. In: Molapo M, editor. Proceedings of the 3rd ACM symposium on computing for development. New York: Association for Computing Machinery; 2013. p. 1–2. https://doi.org/10.1145/2442882.2442913 .

Bhowmick S, Sorathia K. Findings of the user study conducted to understand the training of rural ASHAs in India. In: Bhowmick S, editor. Proceedings of the tenth international conference on information and communication technologies and development. New York: Association for Computing Machinery; 2019. p. 1–5. https://doi.org/10.1145/3287098.3287150 .

Bikinesi L, O’Bryan G, Roscoe C, Mekonen T, Shoopala N, Mengistu AT, et al. Implementation and evaluation of a Project ECHO telementoring program for the Namibian HIV workforce. Hum Resour Health. 2020;18(1):61.

Adsul P, Nethan ST, deCortina SH, Dhanasekaran K, Hariprasad R. Implementing cancer screening programs by training primary care physicians in India—findings from the national institute of cancer prevention research project ECHO for cancer prevention. Glob Implement Res Appl. 2022;2(1):34–41.

White C, McIlfatrick S, Dunwoody L, Watson M. Supporting and improving community health services—a prospective evaluation of ECHO technology in community palliative care nursing teams. BMJ Support Palliat Care. 2019;9(2):202–8.

Usher R, Payne C, Real S, Carey L. Project ECHO: Enhancing palliative care for primary care occupational therapists and physiotherapists in Ireland. Health Soc Care Commun. 2022;30(3):1143–53.

Wood BR, Unruh KT, Martinez-Paz N, Annese M, Ramers CB, Harrington RD, et al. Impact of a telehealth program that delivers remote consultation and longitudinal mentorship to community HIV providers. Open Forum Infect Dis. 2016;3(3):123.

Katzman JG, Comerci GJ, Boyle JF, Duhigg D, Shelley B, Olivas C, et al. Innovative telementoring for pain management: project ECHO pain. J Contin Educ Health Prof. 2014;34(1):68–75.

Kalne PS, Kalne PS, Mehendale AM. Acknowledging the role of community health workers in providing essential healthcare services in rural india—a review. Cureus. 2022;14(9): e29372.

PubMed   PubMed Central   Google Scholar  

Rohith M, Angadi MM. Evaluation of knowledge and practice of ASHAs, regarding child health services in Vijyapaura district, Karnataka. J Fam Med Prim Care. 2020;9(7):3272–6.

Article   CAS   Google Scholar  

Shrivastava A, Srivastava A. Measuring communication competence and effectiveness of ASHAs (accredited social health activist) in their leadership role at rural settings of Uttar Pradesh (India). Leadersh Health Serv. 2016;29(1):69–81.

Scott K, Ummer O, Chamberlain S, Sharma M, Gharai D, Mishra B, et al. ’[We] learned how to speak with love’: a qualitative exploration of accredited social health activist (ASHA) community health worker experiences of the Mobile Academy refresher training in Rajasthan, India. BMJ Open. 2022;12(6): e050363.

Goel AD, Gosain M, Amarchand R, Sharma H, Rai S, Kapoor SK, et al. Effectiveness of a quality improvement program using difference-in-difference analysis for home based newborn care—results of a community intervention trial. Indian J Pediatr. 2019;86(11):1028–35.

Ved R, Scott K. Counseling is a relationship not just a skill: re-conceptualizing health behavior change communication by India’s accredited social health activists. Glob Health Sci Pract. 2020;8(3):332–4.

Abdel-All M, Putica B, Praveen D, Abimbola S, Joshi R. Effectiveness of community health worker training programmes for cardiovascular disease management in low-income and middle-income countries: a systematic review. BMJ Open. 2017;7(11): e015529.

Smittenaar P, Ramesh BM, Jain M, Blanchard J, Kemp H, Engl E, et al. Bringing greater precision to interactions between community health workers and households to improve maternal and newborn health outcomes in India. Glob Health Sci Pract. 2020;8(3):358–71.

Shimasaki S, Bishop E, Guthrie M, Thomas JF. Strengthening the health workforce through the ECHO stages of participation: participants’ perspectives on key facilitators and barriers. J Med Educ Curric Dev. 2019;6:2382120518820922.

Medhanyie AA, Moser A, Spigt M, Yebyo H, Little A, Dinant G, et al. Mobile health data collection at primary health care in Ethiopia: a feasible challenge. J Clin Epidemiol. 2015;68(1):80–6.

Download references

Acknowledgements

The authors wish to thank all the healthcare workers who kindly participated in this study giving their time, experience, and insights. We also thank Dr. Sourabh Chakraborty (Professor, JSPH), Mr. Swapnil Gupta, and the JSPH data collection team for their contribution to the collection of good quality data in a short time.

The study was funded by Extension for Community Healthcare Outcomes (ECHO) India.

Author information

Authors and affiliations.

Public Health Foundation of India (PHFI), Gurugram, Haryana, India

Rajmohan Panda

Extension for Community Healthcare Outcomes (ECHO) India, Okhla Phase III, New Delhi, India

Supriya Lahoti, Nivedita Mishra, Apoorva Karan Rai & Kumud Rai

HexaHealth, Gurugram, Haryana, India

Rajath R. Prabhu

Hamad Medical Corporation, Doha, Qatar

Kalpana Singh

You can also search for this author in PubMed   Google Scholar

Contributions

R.M. contributed to the conception and design of the study and significant inputs for data analysis and made a significant contribution to the drafting of the discussion and conclusion of the paper. S.L. wrote the first draft of the manuscript. N.M. and S.L. contributed to the implementation of the study and development of interview guides, analysis, and validation of qualitative data. R.P. and K.S. contributed to the analysis and validation of quantitative data. R.M., N.M., R.P., K.S, A.K.R. and K.R. reviewed the manuscript and gave significant inputs for improving the paper. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Supriya Lahoti .

Ethics declarations

Ethics approval and consent to participate.

Ethical clearance was received from the Institutional Ethical Committee (IEC) of the Public Health Foundation of India (PHFI) (ref: TRC-IEC 472/21, dated 26 August 2021). The study has also been registered at the Clinical Trials Registry, India (CTRI/2021/10/037189). All methods were performed in accordance with the relevant guidelines and regulations. A written Participant Information Sheet (PIS) and informed consent form was provided to the participants before conducting the interviews. Verbal informed consent was taken from all participants, and the process of verbal informed consent was approved by the ethics committee (Institutional Ethics Committee (IEC) of the PHFI). Data confidentiality was maintained by coding with the unique identification (ID) of all the participants. The interviews were audio-recorded, and audio files and transcripts were kept in a password-protected folder.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

: Appendix S1. Table S1.1. Satisfaction with different factors of the training. Table S1.2. Satisfaction with content and environment of the training. Table S1.3. Challenges faced with respect to ECHO tele-mentoring model. Table S2. Technical knowledge and skills. Table S3. Statements assessing competence. Table S4. Statements assessing attitude and performance.

Additional file 2

: Appendix S2. Participants in qualitative interviews.

Additional file 3

: Appendix S3. Key informant Interview Guide for Trainers End line Evaluation.

Additional file 4

: Appendix S4. Key informant interview guide for Hub leaders End line Evaluation.

Additional file 5

: Appendix S5. In-depth Interview Guide for ASHAs End line Evaluation.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Panda, R., Lahoti, S., Mishra, N. et al. A mixed methods evaluation of the impact of ECHO ® telementoring model for capacity building of community health workers in India. Hum Resour Health 22 , 26 (2024). https://doi.org/10.1186/s12960-024-00907-y

Download citation

Received : 14 March 2023

Accepted : 10 April 2024

Published : 23 April 2024

DOI : https://doi.org/10.1186/s12960-024-00907-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Community health workers (CHWs)
• Accredited social health activists (ASHAs)
• Maternal and child health
• Primary healthcare
• Health worker training
• ECHO telementoring
• Mixed-method study

Human Resources for Health

ISSN: 1478-4491

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Open access
• Published: 22 April 2024

Patient mistreatment, social sharing of negative events and emotional exhaustion among Chinese nurses: the combined moderating effect of organizational support and trait resilience

• Wei Yan 1 ,
• Xiu Chen 1 ,
• Di Xiao 2 ,
• Huan Wang 3 , 4 ,
• Chunjuan Xu 7 &
• Caiping Song 8  

BMC Nursing volume  23 , Article number:  260 ( 2024 ) Cite this article

80 Accesses

Metrics details

As a primary form of work-related violence in the healthcare sector, patient mistreatment negatively impacts nurses’ well-being. To date, there has yet reached a definitive conclusion on the mediating mechanism and boundary conditions behind the influence of patient mistreatment on nurses’ emotional exhaustion.

This study employed a convenience sampling method to recruit a sample of 1672 nurses from public hospitals in Western China. The data were collected through anonymous self-report questionnaires and analyzed using hierarchical regression and conditional processes to investigate a theoretical framework encompassing patient mistreatment, emotional exhaustion, social sharing of negative events, organizational support, and trait resilience.

Patient mistreatment led to emotional exhaustion among nurses (β = 0.625, p  <.001), and social sharing of negative events mediated this positive relationship (effect = 0.073, SE = 0.013). The combined effects of organizational support and resilience moderated the mediating effect of the social sharing of negative events between patient mistreatment and emotional exhaustion (β=-0.051, p  <.05). Specifically, nurses with a high level of resilience would benefit from organizational support to alleviate emotional exhaustion caused by patient mistreatment.

Conclusions

This study validated a significant positive association between patient mistreatment and emotional exhaustion, which aligns with previous research findings. Integrating conservation of resources theory and goal progress theory, we addressed previous contradictory findings on the impact of social sharing of negative events on emotional exhaustion. Social sharing of negative events served as a mediator between patient mistreatment and emotional exhaustion. Additionally, the moderating effect of organizational support on the relationship between social sharing of negative events and emotional exhaustion depended on individual trait of resilience.

Peer Review reports

Introduction

Workplace violence is a worldwide concern and a major risk in healthcare work. It was defined as incidents in which staff members are mistreated, threatened, or assaulted in circumstances related to their work [ 1 ]. Over the last decades, it has been well documented that healthcare professionals around the world are at significant risk of violence exposure [ 2 ]. Studies have shown that the most vulnerable healthcare workers victimized are nurses and paramedics [ 3 , 4 ], with the most common perpetrators being patients, their relatives, or visitors [ 5 ]. A recent survey of 4263 nurses in the healthcare sector showed that 54% of respondents had experienced verbal violence by patients [ 6 ], including negative emotional behaviors exhibited by patients or their families, such as anger, swearing, insults, yelling, and speaking rudely toward nurses [ 7 , 8 , 9 ]. All these negative emotional behaviors are known as “patient mistreatment”. A considerable amount of research conducted in healthcare organizations has shown that exposure to patient mistreatment is a strong predictor of stress, emotional exhaustion, turnover intention and obstacles to career development among nurses [ 6 , 7 , 10 , 11 , 12 , 13 ]. Specifically, emotional exhaustion, characterized by intense fatigue, lack of interest, low mood, and less enthusiasm for jobs, is not only a key outcome resulting from patient mistreatment but also serves as a significant predictor of nurse turnover and a decline in nursing job performance. The conservation of resources (COR) theory provides a theoretical framework for understanding emotional exhaustion caused by patient mistreatment. The COR theory indicates that people strive to retain, protect, and build resources which are needed in fulfilling job responsibilities and are threatened by the potential or actual loss of those valued resources [ 14 , 15 , 16 ]. Despite increasing research interest, the existing literature has yet reached a definitive conclusion on the mechanism how patient mistreatment impacts nurses’ emotional exhaustion. Therefore, this study developed and examined a theoretical model regarding the influence of patient mistreatment on nurses’ emotional exhaustion and explored the mechanisms and boundary conditions behind this relationship.

Social sharing of negative events refers to talking to others about negative events and one’s emotional reactions to them and can occur hours to months after the event [ 17 , 18 ]. It is often seen as a response to emotional experiences to release negative emotions, alleviate work-related stress, and restore resources. However, there is no consensus on the impact of this behavior on emotional exhaustion [ 19 , 20 , 21 ]. Social sharing of negative events sometimes fails to bring new insights into emotional experiences, and disrupts nurses’ goal-related cognitive processes. Goal progress theory illustrates that goal failure (e.g. receiving customer mistreatment) [ 22 ] is associated with cognitive rumination [ 23 ], which may lead to the further loss of resources. Therefore, we examined the social sharing of negative events as a mediating mechanism in the relationship between patient mistreatment and emotional exhaustion in this study.

Furthermore, studies of organizational support have shown that it provides a supportive environment for individuals in coping with stress caused by customer mistreatment [ 24 ]. The COR theory also explicates that supportive environments and contexts create fertile ground for creation of individual resources [ 15 ]. However, some evidence has revealed that organizational support is not consistently beneficial, yielding inconclusive findings [ 25 , 26 , 27 ]. Besides, it is crucial to understand why some people are able to handle negative experiences at work more functionally than others. Consistent with COR theory, individual resources may be contained or embodied in traits and capabilities [ 14 ]. Resilience is a personal trait that can help individuals better cope with adversity and stress [ 28 ]. Therefore, this study introduces organizational support as a crucial moderating variable to explore its moderating effect on the mediating pathway of the social sharing of negative events between patient mistreatment and emotional exhaustion and examines whether trait resilience serves as a boundary condition to the effectiveness of organizational support.

In summary, drawing upon the conservation of resources theory and goal progress theory, this study attempts to answer the following questions: Is patient mistreatment related to emotional exhaustion through the social sharing of negative events? Is organizational support always beneficial or not? And who will benefit from it?

Patient mistreatment and emotional exhaustion

Among all occupational groups, healthcare workers are ranked as one of the most likely to experience workplace aggression [ 29 , 30 , 31 , 32 ]. Patient mistreatment refers to negative emotional behaviors such as expressed anger, swearing, insulting, yelling, and speaking rudely directed toward healthcare providers by patients or their families [ 9 , 33 , 34 ]. Existing studies have extensively explored the adverse consequences of patient mistreatment on healthcare staff and found that it can negatively impact their psychological and physical well-being, leading to increased anxiety, burnout, and negative emotions [ 35 , 36 , 37 ]. The psychological harm caused by patient mistreatment can also result in stress, which is defined as a reaction to an environment in which there is a threat or net loss of resources [ 34 ].

The conservation of resources (COR) theory constructs a framework for comprehending the origins and coping strategies of stress and is frequently used to interpret the process of emotional exhaustion. Individual resources are defined as any element that is valuable for an individual’s survival and development. Individuals strive to retain, protect, and build the resources they value [ 14 , 15 , 16 ], and suffer salient impacts when they lose resources. Moreover, the availability of resources determines the impact of workplace stressors (such as unfair treatment) on employees [ 38 , 39 ]. Healthcare professionals may experience emotional exhaustion, which refers to energy depletion or the draining of emotional resources [ 38 ], as a consequence of mistreatment by patients [ 35 ]. Therefore, we propose the following hypothesis:

Patient mistreatment is positively related to emotional exhaustion.

The mediating role of social sharing of negative events

Researchers have identified social sharing of negative events as talking to others about a negative event and one’s emotional reactions to it and can occur hours to months after the event [ 17 , 18 , 40 ]. Individuals voluntarily share their negative emotional experiences and feelings with others in social settings to release negative emotions, alleviate work-related stress, and restore psychological resources. Despite research on this topic, there is no consensus on the impact of social sharing on negative emotions. Delroisse et al. suggested that it can reduce job burnout by helping employees make sense of work situations and reinforcing relationships with others [ 19 ]. By contrast, Nolen-Hoeksema posited that sharing could potentially be detrimental if it involves ruminating on or immersing oneself in negative feelings, potentially exacerbating or prolonging feelings of sadness [ 20 ]. Drawing upon the conservation of resources theory and goal progress theory, we aimed to clarify the effect of social sharing of negative events between patient mistreatment and emotional exhaustion.

COR theory stated that individuals should proactively invest resources to protect themselves against potentially stressful situations, recover from losses, and accumulate additional resources to brace themselves for future challenges [ 14 , 15 , 16 ]. Social sharing of negative events has been conceptualized as a social and interpersonal process of repetitively seeking proactive social opportunities to verbalize experiences of stressful events [ 40 , 41 ]. Strongman et al. argued that social sharing of emotions activates the interconnectedness between individuals and their respective social networks or support systems [ 42 ]. Supportive actions by recipients, such as listening, understanding, and consolation, help sharers replenish depleted resources and foster their ability to cope with stressors in the sharing process [ 43 ], ultimately equipping them with the necessary resources to address adverse situations. For example, Zech highlighted that social sharing of negative events can provide informational support (e.g. advice) and facilitate reevaluation for individuals [ 17 ]. Laurens’s study revealed that nurses are inclined to engage in emotional social sharing with professionals, such as colleagues or counselors, when confronted with emotional issues involving their patients [ 44 ]. Therefore, drawing upon conservation resources theory, we anticipated that nurses who experience resource depletion due to patient mistreatment may seek to obtain the necessary resources through social sharing of negative events to manage stressful events.

Social sharing of negative events can facilitate cognitive-affective processing of shared events [ 45 ]. However, it carries “sharing risks” [ 46 ], particularly when negative emotions are involved. When it comes to repeated negative events, deliberate thoughts oriented towards the implications of a given event may alternate with unwanted, intrusive thoughts [ 40 ]. Martin and Tesser defined a class of conscious thoughts that revolve around a common instrumental theme as cognitive rumination [ 23 ], which is associated with goal progress theory [ 22 ] to illustrate the impact of goal failure (e.g., receiving customer mistreatment) [ 47 ]. Patient mistreatment serves as a pivotal emotional event and an original disruption. It fails to bring new insights into emotional experiences, disrupts nurses’ goal-related cognitive processes, and triggers rumination [ 40 , 47 ] when nurses share negative events with others [ 20 ]. The more nurses ruminate, the longer they experience intrusive thoughts linked to unachieved goals [ 22 ]. Moreover, loss of resources or the threat of such loss is a crucial factor in predicting psychological distress and leading to investing more resources, making those already lacking resources even more vulnerable to loss spirals [ 14 ]. Emotional exhaustion occurs when individuals are confronted with dual stressors of resource depletion and goal failure. Consequently, we propose the following hypothesis:

Social sharing of negative events mediates the relationship between patient mistreatment and emotional exhaustion.

The moderating role of organizational support

Hobfoll et al. further clarified those resources, which are central to survival and goal attainment, operate depending on the ecological context [ 48 , 49 ].They further theorized that resources do not exist individually but travel in packs, or caravans for both individuals and organizations [ 15 , 50 ]. Organizational support, which is the overall belief that the organization values contributions and cares about the well-being of its employees [ 51 ], is a vital aspect of work resources. Crossover acts as one of the mechanisms of resource exchange within resource caravans [ 15 ] and states that organizational support can be effectively transferred from organizational context to individuals. Studies have suggested that the crossover of resources is also very important for gaining spirals because it can increase a partner’s engagement, potentially triggering a chain of crossover of engagement processes [ 52 ]. Moreover, global research has also identified organizational support as a new buffer-type resource that can counter the resource-depleting effect of high workload and high emotional demands in a large sample of Dutch health professionals [ 53 ]. Therefore, these important work resources, including concern, recognition, and respect inherited in organizational support, would compensate for individuals’ resources, foster the accomplishment of personal work objectives [ 54 ], and enhance employees’ self-efficacy and sense of self-worth, consequently elevating their positive emotions [ 55 , 56 ]. Thus, we anticipated that organizational support would not only alleviate the adverse effects of mistreatment experienced by employees within the organization [ 57 , 58 , 59 ], but also effectively moderate the relationship between social sharing of negative events and emotional exhaustion.

The combined effect of organizational support and trait relicense

Conventionally, studies have demonstrated that organizational support constitutes a valuable work resource. However, COR theory posited that the transfer of resources across social entities (individuals and organizations) is slower. Mounting evidence suggested that organizational support may, at times, not be helpful or even worsen situations [ 60 , 61 , 62 ]. Perhaps the effects of crossover depend on certain traits of the individuals or groups. Evidence continued to mount regarding those with greater resources being less vulnerable to resource loss and more capable of gaining resources [ 15 , 63 ]. Luthans and Avolio [ 64 ] pointed out that both psychological capital and organizational support are necessary for employees to achieve high performance. Resilience, an individual’s ability to cope effectively with adversity and stress when facing difficulties and setbacks [ 65 , 66 ], can be a key personal resource for understanding how individuals break loss spirals [ 67 , 68 ]. Resilience enables individuals to adapt better to changing environments [ 69 , 70 ] and shapes their perception of stress [ 71 , 72 ].

This study found that trait resilience acts as a boundary condition for the moderating role of organizational support in the relationship between social sharing of negative events and emotional exhaustion. Furthermore, the interactive effects among various resources, such as psychological and organizational resources [ 73 ], do not simply add up, but rather enhance the assets necessary for individuals to accomplish their objectives. Consequently, it facilitates individuals with higher levels of resilience by employing both personal psychological resources and organizational resources to develop effective strategies to handle challenges like patient mistreatment [ 74 ]. In conclusion, this study proposes the following hypotheses:

The moderating effect of organizational support on the relationship between social sharing of negative events and emotional exhaustion depends on trait resilience.

The interaction between organizational support and trait resilience moderates the indirect effect of patient mistreatment on emotional exhaustion via the social sharing of negative work events.

We summarize our theoretical model in Fig.  1 .

Hypothesized theoretical model

Participants and data collection procedures

Convenience sampling was employed in this study. We initiated a call for nursing mistreatment research based on the Hematology Specialty Alliance platform in Chongqing, a major city in Southwest China. Furthermore, we used one-on-one communication to invite the clinical department nurses to participate in the survey. The inclusion criteria for recruiting participants in our study were as follows: ① Certified nurses; ② Clinical nursing positions; ③ Informed consent and voluntary participation. The exclusion criteria were as follows: ① student nurses in rotation, ② student nurses on internships, ③ nursing residents in training programs, and ④ off-duty nurses (on leave, sick leave, or attending external training).

To minimize the risks posed by the COVID-19 pandemic, this study employed a structured online questionnaire to facilitate ease of participation. To ensure the credibility and fairness of the collected data, all responses were submitted anonymously. The questionnaires were completed anonymously to ensure the acquisition of objective and unbiased data. The initial page of the questionnaire presented a clear statement of the study’s objectives and confidentiality of the responses. All questions were designed to be mandatory, and each unique IP address was allowed a single submission to uphold the integrity of the data and avoid duplicate entries. In preparation for the main study, a preliminary survey was conducted to validate the logic of the questions and the accuracy of their responses. The formal survey was conducted from October 9th, 2022 to November 1st, 2022. (Questionnaire link: https://wj.qq.com/mine.html ), ultimately yielding 1627 valid responses.)

We employed the translation and back-translation processes recommended by Brislin [ 75 ] in both surveys prior to the administration. This was done to ensure the validity and appropriateness of all the scales in the Chinese context.

• Patient mistreatment

We used the 18-item scale developed by Wang et al. [ 21 ] to measure patient mistreatment, replacing the word “customers” with “patients” in each item. The scale divides patient mistreatment into two dimensions: aggressive mistreatment and demand-oriented mistreatment. Participants rated the items on a five-point Likert scale from 1 = never to 5 = frequently. Example items were “Patients demanded special treatment,” “Patients spoke aggressively to you,” and “Patients asked you to do things even if they can do them themselves.” The Cronbach’s alpha of the scale was 0.953.

• Social sharing of negative events

Social sharing of negative events scale was adapted from Gable et al. [ 76 ]. In the past month, participants were asked how often they had talked to significant others, other family members, friends, and colleagues about unpleasant things that had happened at work, creating a four-item scale. Responses ranged from 1 = never to 5 = often. Cronbach’s α coefficient was 0.862.

• Emotional exhaustion

Emotional exhaustion was measured using the Chinese version of the Maslach Burnout Inventory (MBI), which was developed by Maslach and Jackson [ 77 ] and is the most widely used tool for evaluating job burnout. Emotional exhaustion included nine items, with sample items such as, “I feel emotionally drained from my work.” Responses ranged from 1 = strongly disagree to 5 = strongly agree. All the items scored positively, with higher scores indicating greater emotional exhaustion. Cronbach’s α coefficient was 0.925.

• Organizational support

In this study, we employed the Organizational Support Perception Scale originally developed and validated by Shen and Benson in 2016 [ 78 ] to assess the perceptions of organizational support. This scale consists of eight items (e.g. “My organization values my contributions to the organization”) and used a 7-point Likert scale. Among these items, four were positively worded and four were reverse-scored. Respondents indicated their agreement on a scale ranging from 1 = strongly disagree to 7 = strongly agree, with higher scores indicating a stronger perception of organizational support. Cronbach’s alpha for the scale was 0.907.

We used the Brief Resilience Scale (BRS) developed by Smith et al. [ 79 ], which consists of six items. Sample items included statements such as “I tend to bounce back quickly after difficulties.” Responses ranged from 1 = strongly disagree to 5 = strongly agree. Three items scored positively and three scored negatively. It is specifically used to measure an individual’s ability to recover their health or well-being in response to stress. Cronbach’s α coefficient was 0.826.

Control variables

Sex, age, education, marital status, years of work, and sports were included as control variables to control for confounding effects on emotional exhaustion.

Data analysis

SPSS23.0 and Mplus7 were used for the statistical analysis. We adopted confirmatory factor analysis to test validity and common method variance. Additionally, we conducted a descriptive statistical analysis of the variables and analyzed each variable using the Pearson’s correlation test to comprehend the characteristics and correlations between the variables. We performed hierarchical regression analysis and conditional process analysis to examine the mediating and moderating effects. Moderating variables were mean-centered to construct the interaction term, mitigating potential multi-collinearity problems. In this study, patient mistreatment served as a predictor variable, social sharing of negative events as a mediator variable, organizational support and resilience as two moderators, and emotional exhaustion as the outcome variable.

Participants

A total of 1627 valid responses were included after a strict review of the collected survey data. The majority of the participants were female (94.7%), while males accounted for only 5.3% of the sample, which is similar to the composition of nurses in other public hospitals in China. Most nurses (87.7%) were between 20 and 39 years old, with two under 20 years old, and 6.9% were over 40 years old. The participants’ years of work experience ranged from less than one year to 36 years, with an average of 9.26 years (SD = 6.40). The majority of nurses (62.6%) were married, and only 36.5% of the total participants reported exercise habits.

Common method variance

Data collected from a single source require querying for possible interference caused by common method variance (CMV). Harman’s single-factor method was used to detect the common method variance. The results of the exploratory factor analysis of the 45 items showed that there were seven factors with eigenvalues greater than 1, and the variance explanation rate of the first factor was 31.579% (< 50%). Therefore, the results suggested that CMV is not a significant problem in this study [ 80 , 81 ].

Confirmatory factor analysis

We conducted confirmatory factor analysis (CFA) to assess the discriminant validity of the scale. As shown in Table  1 , the five-factor model, consisting of patient mistreatment, social sharing of negative events, organizational support, resilience, and emotional exhaustion, demonstrated satisfactory discriminant validity and good fit (χ²/df = 11.276, RMSEA = 0.079, CFI = 0.819, TL = 0.809, SRMR = 0.057). Each variable had a factor loading greater than 0.600 and the internal consistency was good, indicating satisfactory reliability and validity of the scale.

Descriptive statistics

Table  2 presents the means, standard deviations, and correlation coefficients for the variables used in this study. The correlation coefficients were consistent with our expectations, showing that patient mistreatment was significantly positively correlated with emotional exhaustion ( r  =.361, p  <.01) and with the social sharing of negative events ( r  =.198, p  <.01). Additionally, the social sharing of negative events was positively correlated with emotional exhaustion ( r  =.253, p  <.01). Some of the hypotheses of this study were tentatively supported.

Hierarchical regression was used to test the relevant hypotheses and the results are presented in Table  3 . Model 4 indicated a positive correlation between patient mistreatment and emotional exhaustion (β = 0.625, p  <.001), which supported Hypothesis 1. The test for the mediating effect followed the recommended stepwise approach [ 82 ]. First, Model 2 revealed a significant positive correlation between patient mistreatment and the social sharing of negative events (β = 0.275, p  <.001). Second, Model 5 showed that social sharing of negative events was positively correlated with emotional exhaustion (β = 0.264, p  <.001). Finally, while the effect of patient mistreatment on the dependent variable, emotional exhaustion, remained significant (β = 0.552, p  <.001), it was somewhat weaker (0.552 < 0.625) after introducing the mediating variable, suggesting a partial mediating effect.

Following Preacher and Hayes [ 83 ], this study further tested the mediating effect of the social sharing of negative events on the relationship between patient mistreatment and emotional exhaustion. We employed the bias-corrected method with a sample size of 5000 and a 95% confidence interval to perform multiple mediating effect analysis using Process3.2, a software for conditional process analysis. The test results are presented in Table  4 . The results showed that the indirect effect was 0.073, with a 95% confidence interval of [0.049, 0.100], demonstrating that the social sharing of negative events played a mediating role in the relationship between patient mistreatment and emotional exhaustion. Therefore, H2 was supported.

The combined moderating effect of organizational support and trait resilience

Table  5 presents the results of moderation analysis. In Model 2, both organizational support and resilience were found to be significantly negatively correlated with emotional exhaustion (βos=-0.348, p  <.001; βre = − 0.569, p  <.001). However, in Model 3, neither organizational support nor resilience showed any interaction with social sharing of negative events in predicting emotional exhaustion. Nevertheless, the three-way interaction between social sharing of negative events, organizational support, and resilience was significant in predicting emotional exhaustion and negatively correlated with emotional exhaustion (β=-0.051, p  <.05), thus supporting H3. Figure  2 shows the results of the three-way interaction, in which it is evident that higher levels of organizational support and resilience weaken the positive impact of the social sharing of negative events on emotional exhaustion.

Simple slope test

We also conducted a moderated mediation model in Process 3.2, using 95% bias-corrected bootstrap confidence interval analyses with 5,000 bootstrap samples to examine the moderating effect of the interaction term of organizational support and resilience on the mediating role of social sharing of negative events between patient mistreatment and emotional exhaustion. As shown in Table  6 , the index of moderated moderated mediation was − 0.0152, which was statistically significant, with a 95% bias-corrected confidence interval of [-0.0286, − 0.0031]. Therefore, H4 was supported.

Specifically, the 95% confidence interval for the indices of conditional moderated mediation was [-0.0120, 0.0240] for individuals with high resilience and [-0.0291, -0.0012] for those with low resilience. Therefore, H3 was supported, indicating that individual resilience served as a boundary condition for the moderating effect of organizational support on the relationship between the social sharing of negative events and emotional exhaustion.

This study combined conservation of resources theory with goal progress theory to investigate the mediating role of the social sharing of negative events in the association between patient mistreatment and nurses’ emotional exhaustion. We also explored whether the moderating effect of organizational support on the relationship between the social sharing of negative events and emotional exhaustion depended on individual resilience. First, this study confirmed a significant positive correlation between nurses’ experiences of patient mistreatment and emotional exhaustion, which is consistent with previous studies [ 6 , 7 , 84 , 85 , 86 ]. The findings once again underscore the detrimental impact of patient mistreatment on nurses’ emotional and psychological well-being. Given that the rates of different forms and sources of aggression vary considerably between nations [ 3 , 87 ], it is crucial to direct our attention towards the patient mistreatment experiences of nurses in China, especially in the post-epidemic era.

Second, this study revealed that the social sharing behavior of negative events mediates the relationship between patient mistreatment and emotional exhaustion. Previous studies have produced mixed findings regarding the impact of the social sharing of negative events on emotional exhaustion among employees or nurses [ 19 , 20 , 21 ]. However, limited research has examined the role of social sharing of negative emotions as a mediating mechanism between patient mistreatment and nurses’ emotional exhaustion. This study integrated the conservation of resources theory and goal progress theory to establish a theoretical foundation for the mediating model. It indicated that sharing negative work events was a strategy for nurses to cope with resource loss resulting from patient mistreatment. Meanwhile, rumination about negative events was closely associated with goal failure, thereby triggering emotional exhaustion among nurses.

Third, the interaction between resilience and organizational support served as a moderator in the relationship between the social sharing of negative events and emotional exhaustion. Studies have identified organizational support as a crucial resource for mitigating the negative effects of stressors [ 24 ]. However, our findings demonstrated that there was no significant two-way interaction between social sharing of negative events and organizational support in predicting emotional exhaustion. This finding is in line with some research on organizational support [ 25 , 28 ], which suggested that organizational support may fail to alleviate the adverse effects of work stressors. Furthermore, this study responded to the call for conservation resources theory [ 28 ] to explore whether trait resilience serves as a boundary to the effectiveness of organizational support. The significant three-way interaction between the social sharing of negative events, organizational support, and trait resilience revealed that individuals with high levels of resilience will benefit from organizational support. Specifically, individuals with high resilience and organizational support showed lower levels of emotional exhaustion than those with low resilience and high organizational support. The implication for managers, therefore, was that organizational support alone cannot solve all problems. Instead, individualized organizational support should be considered in the light of nurses’ resilience.

Practical implications

The findings of this study have significant practical implications for medical management. First, the findings of this study once again validated the significant influence of patient mistreatment on nurses’ emotional exhaustion. Consequently, it is imperative for healthcare administrators to prioritize the establishment of a secure working environment for nurses while providing comprehensive training programs that could enhance their ability to react more effectively to navigate complex nurse-patient relationships. Second, the study further showed that the social sharing of negative events predicted emotional exhaustion among nurses. Therefore, finding ways to eliminate negative rumination originating from patient mistreatment is essential for reducing emotional exhaustion among nurses. Mindfulness thinking, meditation or psychological detachment from work are potential means that nurses could adopt to take a different perspective on negative events. Although the current study indicates that organizational support may not always be beneficial, we suggest that management consider developing workplace interventions that facilitate supportive relationships between organizations and nurses. Third, it is noted that the effect of organizational support depended on resilience. Resilience-related training programs may help nurses acquire psychological resources, enabling them to effectively navigate through mistreatment and adverse experiences. For instance, professional provider resilience training (PPRT) conducted by the medical department of the US military provides knowledge and skills to assist in stress management [ 88 ], such as developing positive cognition, emotional regulation, and mind-body techniques, which enhances the psychological resilience of medical professionals and alleviates fatigue and burnout.

Limitations and further study

This study has some limitations worth addressing. First, the study design was cross-sectional, which may have limited its ability to capture unexamined longitudinal associations. Thus, experience-sampling method should be employed to study the fluctuations of the relationship examined in this study on daily or week basis. Second, all variables investigated were self-reported, which may raise concerns regarding common method variance (CMV) [ 89 ]. Therefore, future studies should employ objective measures or measures reported by others to reduce same-source bias. Third, we found that the social sharing of negative events only partially mediated the relationship between patient mistreatment and emotional exhaustion. Further investigations should be conducted to explore other pathways linking patient mistreatment with nurses’ emotional exhaustion, as well as the moderating variables influencing these mediating mechanisms.

This study, involving 1672 healthcare nurses from public hospitals in Western China, revealed a notable prevalence of patient mistreatment, which led to emotional exhaustion among all participants. The findings of this study suggest that the sharing of negative events plays a mediating role in the relationship of patient mistreatment and the subsequent emotional drain experienced by nurses. These results serve as a critical alert to medical managers about the profound impact of negative emotional sharing within healthcare settings. Furthermore, the study highlights the importance of valuing and fostering certain personal traits of nurses, such as resilience, which can buffer the effects of patient mistreatment on emotional exhaustion, particularly when coupled with high levels of organizational support. Consequently, it is suggested to combine a supportive organizational culture in healthcare sector with training programs that aims to enhance nurses’ resilience.

Data availability

Data supporting the findings of this study are available upon request from the corresponding author.

Wynne R, Clarkin N, Cox T, Griffith A. Guidance on the prevention of violence at work. Office for Official Publications of the European Communities; 1997.

Goussinsky R. The moderating role of rumination and social sharing in the relationship between mistreatment and service sabotage and depersonalization: a cross-sectional study of hospital nurses. Int J Nurs Stud. 2020;110:103705. https://doi.org/10.1016/j.ijnurstu.2020.103705 .

Article   PubMed   Google Scholar  

Di Martino V. Workplace violence in the health sector. Country case studies Brazil, Bulgaria, Lebanon, Portugal, South Africa, Thailand and an additional Australian study. Ginebra: Organización Internacional del Trabajo; 2002. pp. 3–42.

Google Scholar  

Lim MC, Jeffree MS, Saupin SS, Giloi N, Lukman KA. Workplace violence in healthcare settings: the risk factors, implications and collaborative preventive measures. Annals Med Surg. 2022;78:103727. https://doi.org/10.1016/j.amsu.2022.10372 .

Article   Google Scholar  

Hadi AA. Bully And Harassment In Healthcare Industry: What Are Our Roles In Prevention, Powerpoint Slides, 2019. https://www.aoemm.org.my/wp-content/uploads/2019/07/Bully-Harassment-in-Healthcare-Industry-What-are-Our-Roles-in-Prevention-.pdf (Accessed 14 February 2022).

Liu B, Zhu N, Wang H, Li F, Men C. Protecting nurses from mistreatment by patients: a cross-sectional study on the roles of Emotional Contagion susceptibility and emotional regulation ability. Int J Environ Res Public Health. 2021;18(12):6331. https://doi.org/10.3390/IJERPH18126331 .

Article   PubMed   PubMed Central   Google Scholar  

Yan W, Bao N, Zheng S, Wang H, Yue D, Chen L. The impacts of patient mistreatment on healthcare workers’ role behaviors: a study in Chinese Fangcang shelter hospitals. BMC Nurs. 2023;22(1):444.

Goussinsky R, Livne Y. (2016). Coping with interpersonal mistreatment: The role of emotion regulation strategies and supervisor support. Journal of nursing management, 24 (8), 1109–1118. https://doi.org/10.1111/jonm.12415 .

Grandey A, Foo SC, Groth M, Goodwin RE. Free to be you and me: a climate of authenticity alleviates burnout from emotional labor. J Occup Health Psychol. 2012;17(1):1. https://doi.org/10.1037/a0025102 .

Bonner G, McLaughlin S. The psychological impact of aggression on nursing staff. Br J Nurs. 2007;16(13):810–4. https://doi.org/10.12968/bjon.2007.16.13.24248 .

Zampieron A, Galeazzo M, Turra S, Buja A. Perceived aggression towards nurses: study in two Italian health institutions. J Clin Nurs. 2010;19(15–16):2329–41. https://doi.org/10.1111/j.1365-2702.2009.03118.x .

Ahmad M, Al-Rimawi R, Masadeh A, Atoum M. Workplace violence by patients and their families against nurses: literature review. Int J ofNursing Health Sci. 2015;2(4):46–55. http://www.openscienceonline.com/journal/ijnhs .

Choi S, Cheong K, Feinberg RA. Moderating effects of supervisor support, monetary rewards, and career paths on the relationship between job burnout and turnover intentions in the context of call centers. Managing Service Qual. 2012;22:492–516. https://doi.org/10.1108/09604521211281396 .

Hobfoll SE. Conservation of resources: a new attempt at conceptualizing stress. Am Psychol. 1989;44(3):513. https://doi.org/10.1037//0003-066x.44.3.513 .

Article   CAS   PubMed   Google Scholar  

Hobfoll SE, Halbesleben J, Neveu JP, Westman M. Conservation of resources in the organizational context: the reality of resources and their consequences. Annual Rev Organizational Psychol Organizational Behav. 2018;5:103–28. https://doi.org/10.1146/annurev-orgpsych-032117-104640 .

Hobfoll SE. The influence of culture, community, and the nested-self in the stress process: advancing conservation of resources theory. Appl Psychol. 2001;50(3):337–421. https://doi.org/10.1111/1464-0597.00062 .

Zech E, Rimé B, Nils F. Social sharing of emotion, emotional recovery, and interpersonal aspects. In: Philippot P, Feldman RS, editors. The regulation of emotion. Lawrence Erlbaum Associates; 2004. pp. 157–85.

Spencer L. Social sharing of negative emotional events: whether or not sharing helps depends on the Listener’s response. The Pennsylvania State University; 2019.

Delroisse S, Rimé B, Stinglhamber F. Quality of social sharing of emotions alleviates job burnout: the role of meaning of work. J Health Psychol. 2023;28(1):61–76. https://doi.org/10.1177/13591053221091039 .

Nolen-Hoeksema S, Larson J, Grayson C. Explaining the gender difference in depressive symptoms. J Personal Soc Psychol. 1999;77(5):1061. https://doi.org/10.1037//0022-3514.77.5.1061 .

Article   CAS   Google Scholar  

Wang M, Liao H, Zhan Y, Shi J. Daily customer mistreatment and employee sabotage against customers: examining emotion and resource perspectives. Acad Manag J. 2011;54(2):312–34. https://doi.org/10.5465/amj.2011.60263093 .

Martin LL, Tesser A. (1989). Toward a motivational and structural theory of ruminative thought.

Martin LL, Tesser A. Some ruminative thoughts. Ruminative Thoughts. 1996;9(1996):1–47.

Zhao S, Liu H, Ma H, Jiao M, Li Y, Hao Y, Qiao H. Coping with workplace violence in healthcare settings: social support and strategies. Int J Environ Res Public Health. 2015;12(11):14429–44. https://doi.org/10.3390/ijerph121114429 .

Mathieu M, Eschleman KJ, Cheng D. Meta-analytic and multiwave comparison of emotional support and instrumental support in the workplace. J Occup Health Psychol. 2019;24(3):387. https://doi.org/10.1037/ocp0000135 .

Nahum-Shani I, Bamberger PA, Bacharach SB. Social support and employee well-being: the conditioning effect of perceived patterns of supportive exchange. J Health Soc Behav. 2011;52(1):123–39. https://doi.org/10.1177/0022146510395024 .

Willemse BM, de Jonge J, Smit D, Depla MF, Pot AM. The moderating role of decision authority and coworker-and supervisor support on the impact of job demands in nursing homes: a cross-sectional study. Int J Nurs Stud. 2012;49(7):822–33. https://doi.org/10.1016/j.ijnurstu.2012.02.003 .

Egozi Farkash H, Lahad M, Hobfoll SE, Leykin D, Aharonson-Daniel L. Conservation of resources, psychological distress, and resilience during the Covid-19 pandemic. Int J Public Health. 2022;67:1604567. https://doi.org/10.3389/ijph.2022.1604567 .

Hahn S, Zeller A, Needham I, Kok G, Dassen T, Halfens RJ. Patient and visitor violence in general hospitals: a systematic review of the literature. Aggress Violent Beh. 2008;13(6):431–41. https://doi.org/10.1016/j.avb.2008.07.001 .

Bourn J. A safer place to work: protecting NHS hospital and ambulance staff from violence and aggression. London: The National Audit Office; 2003.

Chappell D, Di Martino V. Violence at work. International Labour Organization; 2006.

Wells J, Bowers L. How prevalent is violence towards nurses working in general hospitals in the UK? J Adv Nurs. 2002;39(3):230–40. https://doi.org/10.1046/j.1365-2648.2002.02269.x .

Karaeminogullari A, Erdogan B, Bauer TN. Biting the hand that heals: mistreatment by patients and the well-being of healthcare workers. Personnel Rev. 2018;47(2):572–91. https://doi.org/10.1108/PR-03-2016-0054 .

Rupp DE, Spencer S. When customers lash out: the effects of customer interactional injustice on emotional labor and the mediating role of discrete emotions. J Appl Psychol. 2006;91(4):971. https://doi.org/10.1037/0021-9010.91.4.971 .

Groth M, Grandey A. From bad to worse: negative exchange spirals in employee–customer service interactions. Organizational Psychol Rev. 2012;2(3):208–33. https://doi.org/10.1177/2041386612441735 .

Lanctôt N, Guay S. The aftermath of workplace violence among healthcare workers: a systematic literature review of the consequences. Aggress Violent Beh. 2014;19(5):492–501. https://doi.org/10.1016/j.avb.2014.07.010 .

Viotti S, Gil-Monte PR, Converso D. Toward validating the Italian version of the Spanish burnout inventory: a preliminary study. Revista Da Escola De Enfermagem Da USP. 2015;49:819–25. https://doi.org/10.1590/S0080-623420150000500016 .

Demerouti E, Bakker AB, Nachreiner F, Schaufeli WB. A model of burnout and life satisfaction amongst nurses. J Adv Nurs. 2000;32(2):454–64. https://doi.org/10.1046/j.1365-2648.2000.01496.x .

Bakker AB, Demerouti E. The job demands-resources model: state of the art. J Managerial Psychol. 2007;22(3):309–28. https://doi.org/10.1108/02683940710733115 .

Rimé B, Mesquita B, Boca S, Philippot P. Beyond the emotional event: six studies on the social sharing of emotion. Cognition Emot. 1991;5(5–6):435–65. https://doi.org/10.1080/02699939108411052 .

Rose AJ. Co–rumination in the friendships of girls and boys. Child Dev. 2002;73(6):1830–43. https://doi.org/10.1111/cdev.2002.73.issue-6 .

Consedine N, Strongman K, Magai C. Emotions and behaviour: data from a cross-cultural recognition study. Cogn Emot. 2003;17(6):881–902. https://doi.org/10.1080/02699930244000246 .

Burleson BR. The experience and effects of emotional support: what the study of cultural and gender differences can tell us about close relationships, emotion, and interpersonal communication. Personal Relationships. 2003;10(1):1–23. https://doi.org/10.1111/1475-6811.00033 .

Laurens F, Herbette G, Rimé B. (2000). Le partage social des épisodes émotionnels vécus en milieu hospitalier par le personnel infirmier [Social sharing of emotion among nurses working in the hospital]. Unpublished manuscrit, University of Louvain, Belgium.

Lepore SJ, Ragan JD, Jones S. Talking facilitates cognitive–emotional processes of adaptation to an acute stressor. J Personal Soc Psychol. 2000;78(3):499. https://doi.org/10.1037//0022-3514.78.3.499 .

Clark MS, Finkel EJ. (2004). Does expressing emotion promote well-being? It depends on relationship context. The social life of emotions, 105–126. https://doi.org/10.1017/CBO9780511819568.007 .

Baranik LE, Wang M, Gong Y, Shi J. Customer mistreatment, employee health, and job performance: cognitive rumination and social sharing as mediating mechanisms. J Manag. 2017;43(4):1261–82. https://doi.org/10.1177/0149206315580581 .

Hobfoll SE. (1988). The ecology of stress. Taylor & Francis. https://doi.org/10.1002/smi.2460050415 .

Hobfoll SE. Stress, culture, and community: the psychology and physiology of stress. New York: Plenum; 1998. https://doi.org/10.1007/978-1-4899-0115-6 .

Book   Google Scholar  

Hobfoll SE. Conservation of resource caravans and engaged settings. J Occup Organizational Psychol. 2011;84(1):116–22. https://doi.org/10.1111/j.2044-8325.2010.02016.x .

Eisenberger R, Huntington R, Hutchison S, Sowa D. Perceived organizational support. J Appl Psychol. 1986;71(3):500. https://doi.org/10.1037/0021-9010.71.3.500 .

Westman M. Stress and strain crossover. Hum Relat. 2001;54(6):717–51. https://doi.org/10.1177/0018726701546002 .

Van Woerkom M, Mostert K, Els C, Bakker AB, De Beer L, Rothmann S Jr. Strengths use and deficit correction in organizations: development and validation of a questionnaire. Eur J Work Organizational Psychol. 2016;25(6):960–75.

Bakker AB, Demerouti E, Euwema MC. Job resources buffer the impact of job demands on burnout. J Occup Health Psychol. 2005;10(2):170. https://doi.org/10.1037/1076-8998.10.2.170 .

Eisenberger R, Armeli S, Rexwinkel B, Lynch PD, Rhoades L. Reciprocation of perceived organizational support. J Appl Psychol. 2001;86(1):42. https://doi.org/10.1037/0021-9010.86.1.42 .

George JM, Brief AP. Feeling good-doing good: a conceptual analysis of the mood at work-organizational spontaneity relationship. Psychol Bull. 1992;112(2):310. https://doi.org/10.1037/0033-2909.112.2.310 .

Denney D, O’Beirne M. Violence to probation staff: patterns and managerial responses. Social Policy Adm. 2003;37(1):49–64. https://doi.org/10.1111/1467-9515.00323 .

Schat AC, Kelloway EK. Reducing the adverse consequences of workplace aggression and violence: the buffering effects of organizational support. J Occup Health Psychol. 2003;8(2):110. https://doi.org/10.1037/1076-8998.8.2.110 .

Wasco SM, Campbell R, Clark M. A multiple case study of rape victim advocates’ self-care routines: the influence of organizational context. Am J Community Psychol. 2002;30:731–60. https://doi.org/10.1111/1467-9515.00323 .

Verquer ML, Beehr TA, Wagner SH. A meta-analysis of relations between person–organization fit and work attitudes. J Vocat Behav. 2003;63(3):473–89. https://doi.org/10.1016/S0001-8791(02)00036-2 .

Bowling NA, Beehr TA. Workplace harassment from the victim’s perspective: a theoretical model and meta-analysis. J Appl Psychol. 2006;91(5):998. https://doi.org/10.1037/0021-9010.91.5.998 .

Deelstra JT, Peeters MC, Schaufeli WB, Stroebe W, Zijlstra FR, Van Doornen LP. Receiving instrumental support at work: when help is not welcome. J Appl Psychol. 2003;88(2):324. https://doi.org/10.1037/0021-9010.88.2.324 .

Dohrenwend BS. Social stress and community psychology. Am J Community Psychol. 1978;6(1):1. https://doi.org/10.1007/bf00890095 .

Luthens F, Avolio BJ. (2003). Authentic Leadership: a positive Developmental Approach. Posit Organizational Scholarsh San Francisco: Barrettt-Koehler, 241–61.

Youssef CM, Luthans F. Positive organizational behavior in the workplace: the impact of hope, optimism, and resilience. J Manag. 2007;33(5):774–800. https://doi.org/10.1177/0149206307305562 .

Lahad M. From victim to victor: the development of the BASIC PH model of co** and resiliency. Traumatology. 2017;23(1):27. https://doi.org/10.1037/trm0000105 .

Chen S, Westman M, Hobfoll SE. The commerce and crossover of resources: resource conservation in the service of resilience. Stress Health. 2015;31(2):95–105. https://doi.org/10.1002/smi.2574 .

Hobfoll SE, Stevens NR, Zalta AK. Expanding the science of resilience: conserving resources in the aid of adaptation. Psychol Inq. 2015;26(2):174–80. https://doi.org/10.1080/1047840x.2015.1002377 .

Bonanno GA. Loss, trauma, and human resilience: have we underestimated the human capacity to Thrive after extremely aversive events? Am Psychol. 2004;59(1):20–8. https://doi.org/10.1037/0003-066X.59.1.20 .

Wagnild GM, Young HM. Development and psychometric. J Nurs Meas. 1993;1(2):165–17847. https://doi.org/10.1891/1061-3749.25.3.400 .

Abolghasemi A, Varaniyab ST. Resilience and perceived stress: predictors of life satisfaction in the students of success and failure. Procedia-Social Behav Sci. 2010;5:748–52. https://doi.org/10.1016/j.sbspro.2010.07.178 .

Tung KS, Ning WW, Kris LEE. Effect of resilience on self-perceived stress and experiences on stress symptoms a surveillance report. Univers J Public Health. 2014;2(2):64–72. https://doi.org/10.13189/ujph.2014.020205 .

Halbesleben JR, Neveu JP, Paustian-Underdahl SC, Westman M. Getting to the COR understanding the role of resources in conservation of resources theory. J Manag. 2014;40(5):1334–64. https://doi.org/10.1177/0149206314527130 .

Moodley J, Cooper D, Mantell JE, Stern E. Health care provider perspectives on pregnancy and parenting in HIV-positive individuals in South Africa. BMC Health Serv Res. 2014;14(1):1–8. https://doi.org/10.1186/1472-6963-14-384 .

Brislin RW. (1980). Translation and content analysis of oral and written materials. Methodology, 389–444.

Gable SL, Reis HT, Impett EA, Asher ER. What do you do when things go right? The intrapersonal and interpersonal benefits of sharing positive events. J Personal Soc Psychol. 2004;87(2):228. https://doi.org/10.1037/0022-3514.87.2.228 .

Maslach C, Jackson SE, Leiter MP. Maslach burnout inventory. Scarecrow Education; 1997.

Shen J, Benson J. When CSR is a social norm: how socially responsible human resource management affects employee work behavior. J Manag. 2016;42(6):1723–46. https://doi.org/10.1177/0149206314522300 .

Smith SE, Read DJ. Mycorrhizal symbiosis. Academic; 2010.

Fuller CM, Simmering MJ, Atinc G, Atinc Y, Babin BJ. Common methods variance detection in business research. J Bus Res. 2016;69(8):3192–8. https://doi.org/10.1016/j.jbusres.2015.12.008 .

Podsakoff PM, MacKenzie SB, Lee JY, Podsakoff NP. Common method biases in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol. 2003;88(5):879. https://doi.org/10.1037/0021-9010.88.5.879 .

Baron RM, Kenny DA. The moderator–mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. J Personal Soc Psychol. 1986;51(6):1173. https://doi.org/10.1037//0022-3514.51.6.1173 .

Preacher KJ, Hayes AF. Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Behav Res Methods. 2008;40(3):879–91. https://doi.org/10.3758/brm.40.3.879 .

Dormann C, Zapf D. Customer-related social stressors and burnout. J Occup Health Psychol. 2004;9(1):61. https://doi.org/10.1037/1076-8998.9.1.61 .

Anjum MA, Liang D, Durrani DK, Parvez A. Workplace mistreatment and emotional exhaustion: the interaction effects of self-compassion. Curr Psychol. 2020;1–12. https://doi.org/10.1007/s12144-020-00673-9 .

Kaur P, Islam N, Tandon A, Dhir A. Social media users’ online subjective well-being and fatigue: a network heterogeneity perspective. Technol Forecast Soc Chang. 2021;172:121039. https://doi.org/10.1016/j.techfore.2021.121039 .

Estryn-Behar M, Van Der Heijden B, Camerino D, Fry C, Le Nezet O, Conway PM, Hasselhorn HM. Violence risks in nursing—results from the European ‘NEXT’study. Occup Med. 2008;58(2):107–14. https://doi.org/10.1093/occmed/kqm142 .

Adams S, Camarillo C, Lewis S, McNish N. Resiliency training for medical professionals. US Army Medical Department Journal; 2010.

Podsakoff PM, Organ DW. Self-reports in organizational research: problems and prospects. J Manag. 1986;12(4):531–44.https://psycnet.apa.org/doi/10.1177/014920638601200408

Download references

Acknowledgements

The authors thank all participants who showed great patience in answering the questionnaire.

This study was supported by the National Social Science Foundation of China (Grant number: 19BJY052, 22BGL141), National Natural Science Foundation of China (Grant number: 72110107002, 71974021), Natural Science Foundation of Chongqing (Grant number: cstc2021jcyj-msxmX0689), the Fundamental Research Funds for the Central Universities (Grant number: 2022CDJSKJC14), and Chongqing Social Science Planning Project (Grant number: 2018PY76).

Author information

Authors and affiliations.

School of Economics and Business Administration, Chongqing University, Chongqing, China

Wei Yan & Xiu Chen

Medical Insurance Office, Hospital of Chongqing University, Chongqing, China

School of Medicine and Health Management, Tongji Medical College, Huazhong University of Science & Technology, Wuhan, China

Development and Planning Department, Chongqing Medical University, Chongqing, China

Medical Center of Hematology, Xinqiao Hospital, State Key Laboratory of Trauma, Burn and Combined Injury, Army Medical University, Chongqing, China

Human Resources Department, The Second Affiliated Hospital of Chongqing Medical University, Chongqing, China

Department of Burn Plastic and Microsurgery, The No. 987 Hospital of Joint Logistic Support Force of PLA, Baoji, China

Chunjuan Xu

Xinqiao Hospital, Army Medical University, No. 83 Xinqiao Main Street, Shapingba District, Chongqing, China

Caiping Song

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed to the conception and design of the study. Study design, questionnaire collection, data acquisition were performed by WY, HW and CJX. Data analysis and the first draft of the manuscript was written by WY and XC. DX, XD, LL and CPS commented on the manuscript and revised it critically for important intellectual content.

Corresponding authors

Correspondence to Chunjuan Xu or Caiping Song .

Ethics declarations

Ethics approval and consent to participate.

This study was approved by the Ethics Review Committee of the School of Economics and Business Administration, Chongqing University (IRB No. SEBA201906). All methods were carried out in accordance with relevant guidelines and regulation. Informed consent was obtained from all subjects.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Yan, W., Chen, X., Xiao, D. et al. Patient mistreatment, social sharing of negative events and emotional exhaustion among Chinese nurses: the combined moderating effect of organizational support and trait resilience. BMC Nurs 23 , 260 (2024). https://doi.org/10.1186/s12912-024-01924-x

Download citation

Received : 18 November 2023

Accepted : 08 April 2024

Published : 22 April 2024

DOI : https://doi.org/10.1186/s12912-024-01924-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Resilience trait
• Combined effects

BMC Nursing

ISSN: 1472-6955

• General enquiries: [email protected]

• Open access
• Published: 19 April 2024

GbyE: an integrated tool for genome widely association study and genome selection based on genetic by environmental interaction

• Xinrui Liu 1 , 2 ,
• Mingxiu Wang 1 ,
• Jie Qin 1 ,
• Yaxin Liu 1 ,
• Shikai Wang 1 ,
• Shiyu Wu 1 ,
• Ming Zhang 1 ,
• Jincheng Zhong 1 &
• Jiabo Wang 1  

BMC Genomics volume  25 , Article number:  386 ( 2024 ) Cite this article

323 Accesses

Metrics details

The growth and development of organism were dependent on the effect of genetic, environment, and their interaction. In recent decades, lots of candidate additive genetic markers and genes had been detected by using genome-widely association study (GWAS). However, restricted to computing power and practical tool, the interactive effect of markers and genes were not revealed clearly. And utilization of these interactive markers is difficult in the breeding and prediction, such as genome selection (GS).

Through the Power-FDR curve, the GbyE algorithm can detect more significant genetic loci at different levels of genetic correlation and heritability, especially at low heritability levels. The additive effect of GbyE exhibits high significance on certain chromosomes, while the interactive effect detects more significant sites on other chromosomes, which were not detected in the first two parts. In prediction accuracy testing, in most cases of heritability and genetic correlation, the majority of prediction accuracy of GbyE is significantly higher than that of the mean method, regardless of whether the rrBLUP model or BGLR model is used for statistics. The GbyE algorithm improves the prediction accuracy of the three Bayesian models BRR, BayesA, and BayesLASSO using information from genetic by environmental interaction (G × E) and increases the prediction accuracy by 9.4%, 9.1%, and 11%, respectively, relative to the Mean value method. The GbyE algorithm is significantly superior to the mean method in the absence of a single environment, regardless of the combination of heritability and genetic correlation, especially in the case of high genetic correlation and heritability.

Conclusions

Therefore, this study constructed a new genotype design model program (GbyE) for GWAS and GS using Kronecker product. which was able to clearly estimate the additive and interactive effects separately. The results showed that GbyE can provide higher statistical power for the GWAS and more prediction accuracy of the GS models. In addition, GbyE gives varying degrees of improvement of prediction accuracy in three Bayesian models (BRR, BayesA, and BayesCpi). Whatever the phenotype were missed in the single environment or multiple environments, the GbyE also makes better prediction for inference population set. This study helps us understand the interactive relationship between genomic and environment in the complex traits. The GbyE source code is available at the GitHub website ( https://github.com/liu-xinrui/GbyE ).

Peer Review reports

Genetic by environmental interaction (G × E) is crucial of explaining individual traits and has gained increasing attention in research. It refers to the influence of genetic factors on susceptibility to environmental factors. In-depth study of G × E contributes to a deeper understanding of the relationship between individual growth, living environment and phenotypes. Genetic factors play a role in most human diseases at the molecular or cellular level, but environmental factors also contribute significantly. Researchers aim to uncover the mechanisms behind complex diseases and quantitative traits by investigating the interactions between organisms and their environment. Common, complex, or rare human diseases are often considered as outcomes resulting from the interplay of genes, environmental factors, and their interactions. Analyzing the joint effects of genes and the environment can provide valuable insights into the underlying pathway mechanisms of diseases. For instance, researchers have successfully identified potential loci associated with asthma risk through G × E interactions [ 1 ], and have explored predisposing factors for challenging-to-treat diseases like cancer [ 2 , 3 ], rhinitis [ 4 ], and depression [ 5 ].

However, two main methods are currently being used by breeders in agricultural production to increase crop yields and livestock productivity [ 6 ]. The first is to develop varieties with relatively low G × E effect to ensure stable production performance in different environments. The second is to use information from different environments to improve the statistical power of genome-wide association study (GWAS) to reveal potential loci of complex traits. The first method requires long-term commitment, while the second method clearly has faster returns. In GWAS, the use of multiple environments or phenotypes for association studies has become increasingly important. This not only improves the statistical power of environmental susceptibility traits[ 7 ], but also allows to detect signaling loci for G × E. There are significant challenges when using multiple environments or phenotypes for GWAS, mainly because most diseases and quantitative traits have numerous associated loci with minimal impact [ 8 ], and thus it is impossible to determine the effect size regulated by environment in these loci. The current detection strategy for G × E is based on complex statistical model, often requiring the use of a large number of samples to detect important signals [ 9 , 10 ]. In GS, breeders can use whole genome marker data to identify and select target strains in the early stages of animal and plant production [ 11 , 12 , 13 ]. Initially, GS models, similar to GWAS models, could only analyze a single environment or phenotype [ 14 ]. To improve the predictive accuracy of the models, higher marker densities are often required, allowing the proportion of genetic variation explained by these markers to be increased, indirectly obtaining higher predictive accuracy. It is worth mentioning that the consideration of G × E and multiple phenotypes in GS models [ 15 ] has been widely studied in different plant and animal breeding [ 16 ]. GS models that allow G × E have been developed [ 17 ] and most of them have modeled and interpreted G × E using structured covariates [ 18 ]. In these studies, most of the GS models provided more predictive accuracy when combined with G × E compared to single environment (or phenotype) analysis. Hence, there is need to develop models that leverage G × E information for GWAS and GS studies.

This study developed a novel genotype-by-environment method based on R, termed GbyE, which leverages the interaction among multiple environments or phenotypes to enhance the association study and prediction performance of environmental susceptibility traits. The method enables the identification of mutation sites that exhibit G × E interactions in specific environments. To evaluate the performance of the method, simulation experiments were conducted using a dataset comprising 282 corn samples. Importantly, this method can be seamlessly integrated into any GWAS and GS analysis.

Materials and methods

Support packages.

The development purpose of GbyE is to apply it to GWAS and GS research, therefore it uses the genome association and prediction integrated tool (GAPIT) [ 19 ], Bayesian Generalized Linear Regression (BGLR) [ 20 ], and Ridge Regression Best Linear Unbiased Prediction (rrBLUP) [ 21 ]package as support packages, where GbyE only provides conversion of interactive formats and file generation. In order to simplify the operation of the GbyE function package, the basic calculation package is attached to this package to support the operation of GbyE, including four function packages GbyE.Simulation.R (Dual environment phenotype simulation based on heritability, genetic correlation, and QTL quantity), GbyE.Calculate.R (For numerical genotype and phenotype data, this package can be used to process interactive genotype files of GbyE), GbyE.Power.FDR.R (Calculate the statistical power and false discovery rate (FDR) of GWAS), and GbyE.Comparison.Pvalue.R (GbyE generates redundant calculations in GWAS calculations, and SNP effect loci with minimal p -values can be filtered by this package).

Samples and sequencing data

In this study, a small volume of data was used for software simulation analysis, which is widely used in testing tasks of software such as GAPIT, TASSEL, and rMPV. The demonstration data comes from 282 inbred lines of maize, including 4 phenotypic data. In any case, there are no missing phenotypes in these data, and this dataset can be obtained from the website of GAPIT ( https://zzlab.net/GAPIT/index.html , accessed on May 1, 2022). Among them, our phenotype data was simulated using a self-made R simulation function, and the Mean and GbyE phenotype files were calculated. Convert this format to HapMap format using PLINK v1.09 and scripts written by oneself.

Simulated traits

Phenotype simulation was performed by modifying the GAPIT.Phenotype.Simulation function in the GAPIT. Based on the input parameter NQTN, the random selected markers’ genotype from whole genome were used to simulate genetic effect in the simulated trait. The genotype effects of these selected QTNs were randomly sampled from a multivariate normal distribution, the correlation value between these normal distribution was used to define the genetic relationship between each environments. The additive heritability ( \({{\text{h}}}_{{\text{g}}}^{2}\) ) was used to scale the relationship between additive genetic variance and phenotype variance. The simulated phenotype conditions in this paper are set as follows: 1) The three levels of \({{\text{h}}}_{{\text{g}}}^{2}\) were set at 0.8, 0.5, and 0.2, representing high ( \({{\text{h}}}_{{\text{h}}}^{2}\) ), median ( \({{\text{h}}}_{{\text{m}}}^{2}\) ) and low ( \({{\text{h}}}_{{\text{l}}}^{2}\) ) heritability; 2) Genetic correlation were set three levels 0.8, 0.5, 0.2 representing high ( \({{\text{R}}}_{{\text{h}}}\) ), medium ( \({{\text{R}}}_{{\text{m}}}\) ) and low ( \({{\text{R}}}_{{\text{l}}}\) ) genetic correlation; 3) 20 pre-set effect loci of QTL. The phenotype values in each environment were simulated together following above parameters.

Genetic by environment interaction model

The pipeline analysis process of GbyE includes three steps: data preprocessing, production converted, Association analysis. Normalize the phenotype data matrix Y of the dual environment and perform GbyE conversion to generate phenotype data in GbyE.Y format. The genotype data format, such as hapmap, vcf, bed and other formats firstly need to be converted into numerical genotype format (homozygotes were coded as 0 or 2, heterozygotes were coded as 1) using software or scripts such as GAPIT, PLINK, etc. The environment (E) matrix is environment index matrix. The G (n × m) originally of genotype matrix was converted as GbyE.GD(2n × 2 m) \(\left[\begin{array}{cc}G& 0\\ G& G\end{array}\right]\) during the Kronecker product, and the Y vector (n × 1) was also converted as the GbyE.Y vector (2n × 1) after normalization. The duplicated data format indicated different environments, genetic effect, and populations. The genomic data we used in the analysis was still retained the whole genome information. The first column of E is the additive effect, which was the average genetic effect among environments. The others columns of E are the interactive effect, which should be less one column than the number of environments. Because it need to avoid the linear dependent in the model. In the GbyE algorithm, we coded the first environment as background as default, that means the genotype in the first environment are 0, the others are 1. Then the Kronecker product of G and environment index matrix was named as GbyE.GD. The interactive effect part of the GbyE.GD matrix in the GWAS and GS were the relative values based on the first environment (Fig.  1 ). The GbyE environmental interaction matrix can be easily obtained by constructing the interaction matrix E (e.g., Eq. 1 ) such that the genotype matrix G is Kronecker-product with the design interaction matrix E (e.g., Eq. 2 ), in which \(\left[\begin{array}{c}G\\ G\end{array}\right]\) matrix is defined as additive effect and \(\left[\begin{array}{c}0\\ G\end{array}\right]\) matrix is defined as interactive effect. \(\left[\begin{array}{cc}G& 0\\ G& G\end{array}\right]\) matrix is called gene by environment interaction matrix, hereinafter referred to as the GbyE matrix. The phenotype file (GbyE.Y) and genotype file (GbyE.GD) after transformation by GbyE will be inputted into the GWAS and GS models and computed as standard phenotype and genotype files.

where G is the matrix of whole genotype and E is the design matrix for exploring interactive effects. GbyE mainly uses the Kronecker product of the genetic matrix (G) and the environmental matrix (E) as the genotype for subsequent GWAS as a way to distinguish between additive and interactive effects.

The workflow pipeline of GbyE. The GbyE contains three main steps. (Step 1) Preprocessing of phenotype and genotype data,. The phenotype values in each environment was normalized respectively. Meanwhile, all genotype from HapMap, VCF, BED, and other types were converted to numeric genotype; (Step 2) Generate GbyE phenotype and interactive genotype matrix through the transformation of GbyE. In GbyE.GD matrix, the blue characters indicate additive effect, and red ones indicate interactive effect; (Step 3) The MLM and rrBLUP and BGLR were used to perform GWAS and GS

Association analysis model

The mixed linear model (MLM) of GAPIT is used as the basic model for GWAS analysis, and the principal component analysis (PCA) parameter is set to 3. Then the p -values of detection results are sorted and their power and FDR values are calculated. General expression of MLM (Fig.  1 ):

where Y is the vector of phenotypic measures (2n × 1); PCA and SNP i were defined as fixed effects, with a size of (2n × 2 m); Z is the incidence matrix of random effects; μ is the random effect vector, which follows the normal distribution μ ~ N(0, \({\delta }_{G}^{2}\) K) with mean vector of 0 and variance covariance matrix of \({\delta }_{G}^{2}\) K, where the \({\delta }_{G}^{2}\) is the total genetic variance including additive variance and interactive variance, the K is the kinship matrix built with all genotype including additive genotype and interactive genotype; e is a random error vector, and its elements need not be independent and identically distributed, e ~ N(0, \({\delta }_{e}^{2}\) I), where the \({\delta }_{e}^{2}\) is the residual and environment variance, the I is the design matrix.

Detectivity of GWAS

In the GWAS results, the list of markers following the order of P-values was used to evaluate detectivity of GWAS methods. When all simulated QTNs were detected, the power of the GWAS method was considered as 1 (100%). From the list of markers, following increasing of the criterion of real QTN, the power values will be increasing. The FDR indicates the rate between the wrong criterion of real QTNs and the number of all un-QTNs. The mean of 100 cycles was used to consider as the reference value for statistical power comparison. Here, we used a commonly used method in GWAS research with multiple traits or environmental phenotypes as a comparison[ 22 ]. This method obtains the mean of phenotypic values under different conditions as the phenotypic values for GWAS analysis, called the Mean value method, Compare the calculation results of GbyE with the additive and interactive effects of the mean method to evaluate the detection power of the GbyE strategy. Through the comprehensive analysis of these evaluation indicators, we aim to comprehensively evaluate the statistical power of the GbyE strategy in GWAS and provide a reference for future optimization research.

Among them, the formulae for calculating Power and FDR are as follows:

where \({{\text{n}}}_{{\text{i}}}\) indicates whether the i-th detection is true, true is 1, false is 0; \({{\text{m}}}_{{\text{r}}}\) is the total number of all true QTLs in the sample size; the maximum value of Power is 1.

where \({{\text{N}}}_{{\text{i}}}\) represents the i-th true value detected in the pseudogene, true is 1, false is 0. and cumulative calculation; \({{\text{M}}}_{{\text{f}}}\) is the number of all labeled un-QTNs in the total samples; the maximum value of FDR is 1.

Genomic prediction

To comparison the prediction accuracy of different GS models using GbyE, we performed rrBLUP, Bayesian methods using R packages. All phenotype of reference population and genotype of all population were used to train the model and predict genomic estimated breeding value (gEBV) of all individuals. The correlation between real phenotypes and gEBV of inference population was considered as prediction accuracy. fivefold cross-validation and 100 times repeats was performed to avoid over prediction and reduce bias. In order to distinguish the additive and interactive effects in GbyE, we designed two lists of additive and interactive effects in the "ETA" of BGLR, and put the additive and interactive effects into the model as two kinships for random objects. However, it was not possible to load the gene effects of the two lists in rrBLUP, so the additive and interactive genotypes together were used to calculate whole genetic kinship in rrBLUP (Fig.  1 ). Relevant parameters in BGLR are set as follows: 1) model set to "RRB"; 2) nIter is set to "12000"; 3) burnIn is set to "10000". The results of the above operations are averaged over 100 cycles. We also validated the GbyE method using four other Bayesian methods (BayesA, BayesB, BayesCpi, and Bayesian LASSO) in addition to RRB in BGLR.

Partial missing phentoype in the prediction

In this study, we artificially missed phenotype values in the single and double environments in the whole population from 281 inbred maize datasets. In the missing single environment case, the inference set in the cross-validation was selected from whole population, and each individual in the inference were only missed phenotypes in the one environment. The phenotype in the other environment was kept. The genotypes were always kept. In the case of missing double environments, both phenotypes and genotypes of environment 1 and environment 2 are missing, and the model can only predict phenotypic values in the two missing environments through the effects of other markers. In addition, the data were standardized and unstandardized to assess whether standardization had an effect on the estimation of the model. This experiment was tested using the "ML" method in rrBLUP to ensure the efficiency of the model.

GWAS statistical power of models at different heritabilities and genetic correlations

Power-FDR plots were used to demonstrate the detection efficiency of GbyE at three genetic correlation and three genetic power levels, with a total of nine different scenarios simulated (from left to right for high and low genetic correlation and from top to bottom for high and low genetic power). In order to distinguish whether the effect of improving the detection ability of genome-wide association analysis in GbyE is an additive effect or an effect of environmental interactions, we plotted their Power-FDR curves separately and added the traditional Mean method for comparative analysis. As shown in Fig.  2 , GbyE algorithm can detect more statistically significant genetic loci with lower FDR under any genetic background. However, in the combination with low heritability (Fig.  2 A, B, C), the interactive effect detected more real loci than GbyE under low FDR, but with the continued increase of FDR, GbyE detected more real loci than other groups. Under the combination with high heritability, all groups have high statistical power at low FDR, but with the increase of FDR, the statistical effect of GbyE gradually highlights. From the analysis of heritability combinations at all levels, the effect of heritability on interactive effect is not obvious, but GbyE always maintains the highest statistical power. The average detection power of GWAS in GbyE can be increased by about 20%, and with the decrease of genetic correlation, the effect of GbyE gradually highlights, indicating that the G × E plays a role.

The power-FDR testing in simulated traits. Comparing the efficacy of the GbyE algorithm with the conventional mean method in terms of detection power and FDR. From left to right, the three levels of genetic correlation are indicated in order of low, medium and high. From top to bottom, the three levels of heritability, low, medium and high, are indicated in order. (1) Inter: Interactive section extracted from GbyE; (2) AddE: Additive section extracted from GbyE; (3) \({{\text{h}}}_{{\text{l}}}^{2}\) , \({{\text{h}}}_{{\text{m}}}^{2}\) , \({{\text{h}}}_{{\text{g}}}^{2}\) : Low, medium, high heritability; (4) \({{\text{R}}}_{{\text{l}}}\) , \({{\text{R}}}_{{\text{m}}}\) , \({{\text{R}}}_{{\text{l}}}\) : where R stands for genetic correlation, represents three levels of low, medium and high

Resolution of additive and interactive effect

The output results of GbyE could be understood as resolution of additive and interactive genetic effect. Hence, we created a combined Manhattan plots with Mean result from MLM, additive, and interactive results from GbyE. As shown in Fig.  3 , true marker loci were detected on chromosomes 1, 6 and 9 in Mean, and the same loci were detected on chromosomes 1 and 6 for the additive result in GbyE (the common loci detected jointly by the two results were marked as solid gray lines in the figure). All known pseudo QTNs were labeled with gray dots in the circle. Total 20 pseudo QTNs were simulated in such trait (The heritability is set to 0.9, and the genetic correlation is set to 0.1). Although the additive section in GbyE did not catch the locus on chromosome 9 yet (those p-values of markers did not show above the significance threshold (p-value < 3.23 × 10 –6 )), it has shown high significance relative to other markers of the same chromosome. In the reciprocal effect of GbyE, we detected more significant loci on chromosomes 1, 2, 3 and 10, and these loci were not detected in either of the two previous sections. An integrate QQ plot (Fig.  3 D) shows that the overall statistical power of the additive section in Mean and GbyE are close, nevertheless, the interactive section in the GbyE provided a bit of inflation.

Manhattan statistical comparison plot. Manhattan comparison plots of mean ( A ), additive ( B ) and gene-environment interactive sections ( C ) at a heritability of 0.9 and genetic correlation of 0.1. Different colors are used in the diagram to distinguish between different chromosomes (X-axis). Loci with reinforcing circles and centroids are set up as real QTN loci. Consecutive loci found in both parts are shown as id lines, and loci found separately in the reciprocal effect only are shown as dashed lines. Parallel horizontal lines indicate significance thresholds ( p -value < 3.23 × 10 –6 ). D Quantile–quantile plots of simulated phenotypes for demo data from genome-wide association studies. x-axis indicates expected values of log p -values and y-axis is observed values of log p -values. The diagonal coefficients in red are 1. GbyE-inter is the interactive section in GbyE; GbyE-AddE is the additive section in GbyE

Genomic selection in assumption codistribution

The prediction accuracy of GbyE was significantly higher than the Mean value method by model statistics of rrBLUP in most cases of heritability and genetic correlation (Fig.  4 ). The prediction accuracy of the additive effect was close to that of Mean value method, which was consistent with the situation under the low hereditary. The prediction accuracy of interactive sections in GbyE remains at the same level as in GbyE, and interactive section plays an important role in the model. We observed that in \({{\text{h}}}_{{\text{l}}}^{2}{{\text{R}}}_{{\text{h}}}\) (Fig.  4 C), \({{\text{h}}}_{{\text{m}}}^{2}{{\text{R}}}_{{\text{h}}}\) (Fig.  4 F), \({{\text{h}}}_{{\text{h}}}^{2}{{\text{R}}}_{{\text{l}}}\) (Fig.  4 G), the prediction accuracy of GbyE was slightly higher than the Mean value method, but there was no significant difference overall. In addition, we only observed that the prediction accuracy of GbyE was slightly lower than the Mean value method in \({{\text{h}}}_{{\text{h}}}^{2}{{\text{R}}}_{{\text{l}}}\) (Fig.  4 H), but there was still no significant difference between GbyE and Mean value methods. Under the combination of low heritability and genetic correlation, the prediction accuracy of Mean value method and additive effect model remained at a similar level. However, with the continuous increase of heritability and genetic correlation, the difference in prediction accuracy between the two gradually increases. In summary, the GbyE algorithm can improve the accuracy of GS by capturing information on multiple environment or trait effects under the rrBLUP model.

Box-plot of model prediction accuracy. The prediction accuracy (pearson's correlation coefficient) of the GbyE algorithm was compared with the tradition al Mean value method in a simulation experiment of genomic selection under the rrBLUP operating environment. The effect of different levels of heritability and genetic correlation on the prediction accuracy of genomic selection was simulated in this experiment. Each row from top to bottom represents low heritability ( \({{\text{h}}}_{{\text{l}}}^{2}\) ), medium heritability ( \({{\text{h}}}_{{\text{m}}}^{2}\) ) and high heritability ( \({{\text{h}}}_{{\text{h}}}^{2}\) ), respectively; each column from left to right represents low genetic correlation ( \({{\text{R}}}_{{\text{l}}}\) ), medium genetic correlation ( \({{\text{R}}}_{{\text{m}}}\) ) and high genetic correlation ( \({{\text{R}}}_{{\text{h}}}\) ), respectively; The X-axis shows the different test methods and effects, and the Y-axis shows the prediction accuracy

Genomic selection in assumption un-codistribution

The overall performance of GbyE under the 'BRR' statistical model based on the BGLR package remained consistent with rrBLUP, maintaining high predictive accuracy in most cases of heritability and genetic relatedness (Fig. S1 ). However, when the heritability is set to low and medium, the difference between the prediction accuracy of GbyE algorithm and Mean value method gradually decreases with the continuous increase of genetic correlation, and there is no statistically significant difference between the two. The prediction accuracy of the model by GbyE in \({{\text{h}}}_{{\text{h}}}^{2}{{\text{R}}}_{{\text{l}}}\) (Fig. S1 G) and \({{\text{h}}}_{{\text{h}}}^{2}{{\text{R}}}_{{\text{h}}}\) (Fig. S1 I) is significantly higher than that by Mean value method when the heritability is set to be high. On the contrary, when the genetic correlation is set to medium, there is no significant difference between GbyE and Mean value method in improving the prediction accuracy of the model, and the overall mean of GbyE is lower than Mean. When GbyE has relatively high heritability and low genetic correlation, its prediction accuracy is significantly higher than the mean method, such as \({{\text{h}}}_{{\text{m}}}^{2}{{\text{R}}}_{{\text{l}}}\) (Fig. S1 D), \({{\text{h}}}_{{\text{h}}}^{2}{{\text{R}}}_{{\text{l}}}\) (Fig. S1 G), and \({{\text{h}}}_{{\text{h}}}^{2}{{\text{R}}}_{{\text{m}}}\) (Fig. S1 H). Therefore, GbyE is more suitable for situations with high heritability and low genetic correlation.

Adaptability of Bayesian models

Next, we tested a more complex Bayesian model. The GbyE algorithm and Mean value method were combined with five Bayesian algorithms in BGLR for GS analysis, and the computing R script was used for phenotypic simulation test, where heritability and genetic correlation were both set to 0.5. The results indicate that among the three Bayesian models of RRB, BayesA, and BayesLASSO, the predictive accuracy of GbyE is significantly higher than that of Mean value method (Fig.  5 ). In contrast, under the Bayesian models of BayesB and BayesCpi, the prediction accuracy of GbyE is lower than that of the Mean value method. The GbyE algorithm improves the prediction accuracy of the three Bayesian models BRR, BayesA, and BayesLASSO using information from G × E and increases the prediction accuracy by 9.4%, 9.1%, and 11%, respectively, relative to the Mean value method. However, the predictive accuracy of the BayesB model decreased by 11.3%, while the BayescCpi model decreased by 6%.

Relative prediction accuracy histogram for different Bayesian models. The X-axis is the Bayesian approach based on BGLR, and the Y-axis is the relative prediction accuracy. Where we normalize the prediction accuracy of Mean (the prediction accuracy is all adjusted to 1); the prediction accuracy of GbyE is the increase or decrease value relative to Mean in the same group of models

Impact of all and partial environmental missing

We tested missing the environmental by using simulated data. In the case of the simulated data, we simulated a total of nine situations with different heritability and genetic correlations (Fig.  6 ) and conducted tests on single and dual environment missing. The improvement in prediction accuracy by the GbyE algorithm was found to be significantly higher than the Mean value method in single environment deletion, regardless of the combination of heritability and genetic correlation. In the case of \({{\text{h}}}_{{\text{h}}}^{2}{{\text{R}}}_{{\text{h}}}\) , the prediction accuracy of GbyE is higher than 0.5, which is the highest value among all simulated combinations. When GbyE estimates the phenotypic values of Environment 1 and Environment 2 separately, its predictive accuracy seems too accurate. On the other hand, when the phenotypic values of both environments are missing on the same genotype, the predictive accuracy of GbyE does not show a significant decrease, and even maintains accuracy comparable to that of a single environment missing. However, when GbyE estimates Environment 1 and Environment 2 separately, the prediction accuracy significantly decreases compared to when a single environment is missing, and the prediction accuracy of Environment 1 and Environment 2 in \({{\text{h}}}_{{\text{l}}}^{2}{{\text{R}}}_{{\text{m}}}\) is extremely low (Fig.  6 B). In addition, the prediction accuracy of GbyE is lower than Mean values only in \({{\text{h}}}_{{\text{l}}}^{2}{{\text{R}}}_{{\text{h}}}\) , whether it is missing in a single or dual environment.

Prediction accuracy of simulated data in single and dual environment absence. The prediction effect of GbyE was divided into two parts, environment 1 and environment 2, to compare the prediction accuracy of GbyE when predicting these two parts separately. This includes simulations with missing phenotypes and genotypes in environment 1 only ( A ) and simulations with missing in both environments ( B ). The horizontal coordinates of the graph indicate the different combinations of heritabilities and genetic correlations of the simulations

The phenotype of organisms is usually controlled by multiple factors, mainly genetic [ 23 ] and environmental factors [ 24 ], and their interactive factors. The phenotype of quantitative traits is often influenced by these three factors [ 25 , 26 ]. However, based on the computing limitation and lack of special tool, the interactive effect always was ignored in most GWAS and GS research, and it is difficult to distinguish additive and interactive effects. The rate between all additive genetic variance and phenotype variance was named as narrow sense heritability. The accuracy square of prediction of additive GS model is considered that can not surpass narrow sense heritability. In this study, the additive effects in GbyE are essentially equivalent to the detectability of traditional models, the key advantage of GbyE is the interactive section. More significant markers with interactive effects were detected. Detecting two genetic effects (additive and interactive sections) in GWAS and GS is a boost to computational complexity, while obtaining genotypes for genetic interactions by Kronecker product is an efficient means. This allows the estimation of additive and interactive genetic effects separately during the analysis, and ultimately the estimated genetic effects for each GbyE genotype (including additive and interactive genetic effect markers) are placed in a t-distribution for p -value calculation, and the significance of each genotype is considered by multiple testing. The GbyE also expanded the estimated heritability as generalized heritability which could be explained as the rate between total genetics variance and phenotype variance.

The genetic correlation among traits in multiple environments is the major immanent cause of GbyE. When the genetic correlation level is high, then additive genetic effects will play primary impact in the total genetic effect, and interactive genetic effects with different traits or environments are often at lower levels [ 27 ]. Therefore, the statistical power of the GbyE algorithm did not improve significantly compared with the traditional method (Mean value) when simulating high levels of genetic correlation. On the contrary, in the case of low levels of genetic correlation, the genetic variance of additive effects is relatively low and the genetic variance of interactive effects is major. At this time, GbyE utilizes multiple environments or traits to highlight the statistical power. Since the GbyE algorithm obtains additive, environmental, and interactive information by encoding numerical genotypes, it only increases the volume of SNP data and can be applied to any traditional GWAS association statistical model. However, this may slightly increase the correlation operation time of the GWAS model, but compared to other multi environment or trait models [ 28 , 29 ], GbyE only needs to perform a complete traditional GWAS once to obtain the results.

In GS, rrBLUP algorithm is a linear mixed model-based prediction method that assumes all markers provide genetic effects and their values following a normal distribution [ 30 ]. In contrast, the BGLR model is a linear mixed model, which assumes that gene effects are randomly drawn from a multivariate normal distribution and genotype effects are randomly drawn from a multivariate Gaussian process, which takes into account potential pleiotropy and polygenic effects and allows inferring the effects of single gene while estimating genomic values [ 31 ]. The algorithm typically uses Markov Chain Monte Carlo methods for estimation of the ratio between genetic variances and residual variances [ 32 , 33 ]. The model has been able to take into account more biological features and complexity, and therefore the overall improvement of the GbyE algorithm under BGLR is smaller than Mean method. In addition, the length of the Markov chain set on the BGLR package is often above 20,000 to obtain stable parameters and to undergo longer iterations to make the chain stable [ 34 ]. GbyE is effective in improving the statistical power of the model under most Bayesian statistical models. In the case of the phenotypes we simulated, more iterations cannot be provided for the BayesB and BayesCpi models because of the limitation of computation time, which causes low prediction accuracy. It is worth noting that the prediction accuracy of BayesCpi may also be influenced by the number of QTLs [ 35 ], and the prediction accuracy of BayesB is often related to the distribution of different allele frequencies (from rare to common variants) at random loci [ 36 ].

The overall statistical power of GbyE was significantly higher in missing single environment than in missing double environment, because in the case of missing single environment, GbyE can take full advantage of the information from the phenotype in the second environment. And the correlation between two environments can also affect the detectability of the GbyE algorithm in different ways. On the one hand, a high correlation between two environments can improve the predictive accuracy of the GbyE algorithm by using the information from one environment to predict the breeding values in the other environment, even if there is only few relationship with that environment [ 37 , 38 ]. On the other hand, when two environments are extremely uncorrelated, GbyE algorithm trained in one environment may not export well to another environment, which may lead to a decrease in prediction accuracy [ 39 ]. In the testing, we found that when the GbyE algorithm uses a GS model trained in one environment and tested in another environment, the high correlation between environments may result to the model capturing similarities between environments unrelated to G × E information [ 40 ]. However, when estimating the breeding values for each environment separately, GbyE still made effective predictions using the genotypes in that environment and maintained high prediction accuracy. As expected, the additive effect calculates the average genetic effect between environments, and its predictive effect does not differ much from the mean method. The interactive effect, however, has one less column than the number of environments, and it calculates the relative values between environments, a component that has a direct impact on the predictive effect. The correlation between the two environments may have both positive and negative effects on the detectability of the GbyE, so it is important to carefully consider the relationship between the two environments in subsequent in development and testing.

A key advantage of the GbyE algorithm is that it can be applied to almost all current genome-wide association and prediction. However, the focus of GbyE is still on estimating additive and interactive effects separately, so that it is easy to determine which portion of the is playing a role in the computational estimation.. The GbyE algorithm may have implications for the design of future GS studies. For example, the model could be used to identify the best environments or traits to include in GS studies in order to maximize prediction accuracy. It is particularly important to test the model on large datasets with different genetic backgrounds and environmental conditions to ensure that it can accurately predict genome-wide effects in a variety of contexts.

GbyE can simulate the effects of gene-environment interactions by building genotype files for multiple environments or multiple traits, normalizing the effects of multiple environments and multiple traits on marker effects. It also enables higher statistical power and prediction accuracy for GWAS and GS. The additive and interactive effects of genes under genetic roles could be revealed clearly, which makes it possible to utilize environmental information to improve the statistical power and prediction accuracy of traditional models, thus helping us to better understand the interactions between genes and the environment.

Availability of data and materials

The GbyE source code, demo script, and demo data are freely available on the GitHub website ( https://github.com/liu-xinrui/GbyE ).

Abbreviations

• Genome-widely association study

Genome selection

Genetic by environmental interaction

Genome association and prediction integrated tool

Mixed linear model

Bayesian generalized linear regression

Ridge regression best linear unbiased prediction

False discovery rate

Principal component analysis

Genomic estimated breeding value

Maazi H, Hartiala JA, Suzuki Y, Crow AL, Shafiei Jahani P, Lam J, Patel N, Rigas D, Han Y, Huang P. A GWAS approach identifies Dapp1 as a determinant of air pollution-induced airway hyperreactivity. PLoS Genet. 2019;15(12):e1008528.

Article   PubMed   PubMed Central   Google Scholar  

Simonds NI, Ghazarian AA, Pimentel CB, Schully SD, Ellison GL, Gillanders EM, Mechanic LE. Review of the gene-environment interaction literature in cancer: what do we know? Genet Epidemiol. 2016;40(5):356–65.

Wang X, Chen H, Kapoor PM, Su Y-R, Bolla MK, Dennis J, Dunning AM, Lush M, Wang Q, Michailidou K. A Genome-Wide Gene-Based Gene-Environment Interaction Study of Breast Cancer in More than 90,000 Women. Cancer research communications. 2022;2(4):211–9.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Chen R-X, Dai M-D, Zhang Q-Z, Lu M-P, Wang M-L, Yin M, Zhu X-J, Wu Z-F, Zhang Z-D, Cheng L. TLR Signaling Pathway Gene Polymorphisms, Gene-Gene and Gene-Environment Interactions in Allergic Rhinitis. Journal of Inflammation Research. 2022;15:3613–30.

Zhao M-Z, Song X-S, Ma J-S. Gene× environment interaction in major depressive disorder. World Journal of Clinical Cases. 2021;9(31):9368.

Falconer DS. The problem of environment and selection. Am Nat. 1952;86(830):293–8.

Article   Google Scholar  

Kim J, Zhang Y, Pan W. Powerful and adaptive testing for multi-trait and multi-SNP associations with GWAS and sequencing data. Genetics. 2016;203(2):715–31.

Visscher PM, Wray NR, Zhang Q, Sklar P, McCarthy MI, Brown MA, Yang J. 10 years of GWAS discovery: biology, function, and translation. The American Journal of Human Genetics. 2017;101(1):5–22.

Article   CAS   PubMed   Google Scholar  

van Os J, Rutten BP. Gene-environment-wide interaction studies in psychiatry. Am J Psychiatry. 2009;166(9):964–6.

Article   PubMed   Google Scholar  

Winham SJ, Biernacka JM. Gene–environment interactions in genome-wide association studies: current approaches and new directions. Journal of Child Psychology Psychiatry. 2013;54(10):1120–34.

Windhausen VS, Atlin GN, Hickey JM, Crossa J, Jannink J-L, Sorrells ME, Raman B, Cairns JE, Tarekegne A, Semagn K. Effectiveness of genomic prediction of maize hybrid performance in different breeding populations and environments. G3: Genes|Genomes|Genetics. 2012;2(11):1427–36.

Xu S, Zhu D, Zhang Q. Predicting hybrid performance in rice using genomic best linear unbiased prediction. Proc Natl Acad Sci. 2014;111(34):12456–61.

Zhao Y, Mette M, Gowda M, Longin C, Reif J. Bridging the gap between marker-assisted and genomic selection of heading time and plant height in hybrid wheat. Heredity. 2014;112(6):638–45.

Crossa J, Perez P, Hickey J, Burgueno J, Ornella L, Cerón-Rojas J, Zhang X, Dreisigacker S, Babu R, Li Y. Genomic prediction in CIMMYT maize and wheat breeding programs. Heredity. 2014;112(1):48–60.

Crossa J, Pérez-Rodríguez P, Cuevas J, Montesinos-López O, Jarquín D, De Los CG, Burgueño J, González-Camacho JM, Pérez-Elizalde S, Beyene Y. Genomic selection in plant breeding: methods, models, and perspectives. Trends Plant Sci. 2017;22(11):961–75.

Roorkiwal M, Jarquin D, Singh MK, Gaur PM, Bharadwaj C, Rathore A, Howard R, Srinivasan S, Jain A, Garg V. Genomic-enabled prediction models using multi-environment trials to estimate the effect of genotype× environment interaction on prediction accuracy in chickpea. Sci Rep. 2018;8(1):11701.

Burgueño J, de los Campos G, Weigel K, Crossa J. Genomic prediction of breeding values when modeling genotype× environment interaction using pedigree and dense molecular markers. Crop Science. 2012;52(2):707–19.

Jarquín D, Crossa J, Lacaze X, Du Cheyron P, Daucourt J, Lorgeou J, Piraux F, Guerreiro L, Pérez P, Calus M. A reaction norm model for genomic selection using high-dimensional genomic and environmental data. Theoretical applied genetics. 2014;127:595–607.

Wang JB, Zhang ZW. GAPIT Version 3: boosting power and accuracy for genomic association and prediction. Genomics Proteomics Bioinformatics. 2021;19(4):629–40.

Pérez P, de Los CG. Genome-wide regression and prediction with the BGLR statistical package. Genetics. 2014;198(2):483–95.

Endelman JB. Ridge Regression and other kernels for genomic selection with R package rrBLUP. Plant Genome J. 2011;4:250–5.

Turley P, Walters RK, Maghzian O, Okbay A, Lee JJ, Fontana MA, Nguyen-Viet TA, Wedow R, Zacher M. Furlotte NAJNg. Multi-trait analysis of genome-wide association summary statistics using MTAG. 2018;50(2):229–37.

CAS   Google Scholar  

Falconer DS. Introduction to quantitative genetics. Pearson Education India; 1996.

Google Scholar  

Lynch M, Walsh B. Genetics and analysis of quantitative traits, vol. 1: Sinauer Sunderland, MA. 1998.

Mackay TF. The genetic architecture of quantitative traits. Annu Rev Genet. 2001;35(1):303–39.

Visscher PM, Hill WG, Wray NR. Heritability in the genomics era—concepts and misconceptions. Nat Rev Genet. 2008;9(4):255–66.

Van der Sluis S, Posthuma D, Dolan CV. TATES: efficient multivariate genotype-phenotype analysis for genome-wide association studies. PLoS Genet. 2013;9(1):e1003235.

O’Reilly PF, Hoggart CJ, Pomyen Y, Calboli FC, Elliott P, Jarvelin M-R, Coin LJ. MultiPhen: joint model of multiple phenotypes can increase discovery in GWAS. PLoS ONE. 2012;7(5):e34861.

Chung J, Jun GR, Dupuis J, Farrer LA. Comparison of methods for multivariate gene-based association tests for complex diseases using common variants. Eur J Hum Genet. 2019;27(5):811–23.

Pérez-Rodríguez P, Gianola D, González-Camacho JM, Crossa J, Manès Y, Dreisigacker S. Comparison between linear and non-parametric regression models for genome-enabled prediction in wheat. G3: Genes|Genomes|Genetics. 2012;2(12):1595–16605.

VanRaden PM. Efficient methods to compute genomic predictions. J Dairy Sci. 2008;91(11):4414–23.

Meuwissen TH, Hayes BJ, Goddard M. Prediction of total genetic value using genome-wide dense marker maps. Genetics. 2001;157(4):1819–29.

de Los CG, Hickey JM, Pong-Wong R, Daetwyler HD, Calus MP. Whole-genome regression and prediction methods applied to plant and animal breeding. Genetics. 2013;193(2):327–45.

Andrieu C, De Freitas N, Doucet A, Jordan MI. An introduction to MCMC for machine learning. Mach Learn. 2003;50:5–43.

Daetwyler HD, Pong-Wong R, Villanueva B, Woolliams JA. The impact of genetic architecture on genome-wide evaluation methods. Genetics. 2010;185(3):1021–31.

Clark SA, Hickey JM, Van der Werf JH. Different models of genetic variation and their effect on genomic evaluation. Genet Sel Evol. 2011;43(1):1–9.

Yang J, Benyamin B, McEvoy BP, Gordon S, Henders AK, Nyholt DR, Madden PA, Heath AC, Martin NG, Montgomery GW. Common SNPs explain a large proportion of the heritability for human height. Nat Genet. 2010;42(7):565–9.

González-Recio O, Forni S. Genome-wide prediction of discrete traits using Bayesian regressions and machine learning. Genet Sel Evol. 2011;43:1–12.

Korte A, Farlow A. The advantages and limitations of trait analysis with GWAS: a review. Plant Methods. 2013;9(1):1–9.

Gauderman WJ. Sample size requirements for matched case-control studies of gene–environment interaction. Stat Med. 2002;21(1):35–50.

Download references

Acknowledgements

Thank you to all colleagues in the laboratory for their continuous help.

This project was partially funded by the National Key Research and Development Project of China, China (2022YFD1601601), the Heilongjiang Province Key Research and Development Project, China (2022ZX02B09), the Qinghai Science and Technology Program, China (2022-NK-110), Sichuan Science and Technology Program, China (Award #s 2021YJ0269 and 2021YJ0266), the Program of Chinese National Beef Cattle and Yak Industrial Technology System, China (Award #s CARS-37), and Fundamental Research Funds for the Central Universities, China (Southwest Minzu University, Award #s ZYN2023097).

Author information

Authors and affiliations.

Key Laboratory of Qinghai-Tibetan Plateau Animal Genetic Resource Reservation and Utilization, Sichuan Province and Ministry of Education, Southwest Minzu University, Chengdu, 6110041, China

Xinrui Liu, Mingxiu Wang, Jie Qin, Yaxin Liu, Shikai Wang, Shiyu Wu, Ming Zhang, Jincheng Zhong & Jiabo Wang

Nanchong Academy of Agricultural Sciences, Nanchong, 637000, China

You can also search for this author in PubMed   Google Scholar

Contributions

JW and XL conceived and designed the project. XL managed the entire trial, conducted software code development, software testing, and visualization. MW, JQ, YL, SW, MZ and SW helped with data collection and analysis. JQ, and YL assisted with laboratory analyses. JW, and XL had primary responsibility for the content in the final manuscript. JZ supervised the research. JW designed software and project methodology. All authors approved the final manuscript. All authors have reviewed the manuscript.

Corresponding author

Correspondence to Jiabo Wang .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors have declared no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Liu, X., Wang, M., Qin, J. et al. GbyE: an integrated tool for genome widely association study and genome selection based on genetic by environmental interaction. BMC Genomics 25 , 386 (2024). https://doi.org/10.1186/s12864-024-10310-5

Download citation

Received : 27 December 2023

Accepted : 15 April 2024

Published : 19 April 2024

DOI : https://doi.org/10.1186/s12864-024-10310-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Genomic selection

BMC Genomics

ISSN: 1471-2164

• Submission enquiries: [email protected]
• General enquiries: [email protected]