impact of covid 19 case study

• Open access
• Published: 17 April 2021

Impact of the COVID-19 crisis on work and private life, mental well-being and self-rated health in German and Swiss employees: a cross-sectional online survey

• Martin Tušl 1 ,
• Rebecca Brauchli 1 ,
• Philipp Kerksieck 1 &
• Georg Friedrich Bauer 1  

BMC Public Health volume  21 , Article number:  741 ( 2021 ) Cite this article

176k Accesses

63 Citations

10 Altmetric

Metrics details

The COVID-19 crisis has radically changed the way people live and work. While most studies have focused on prevailing negative consequences, potential positive shifts in everyday life have received less attention. Thus, we examined the actual and perceived overall impact of the COVID-19 crisis on work and private life, and the consequences for mental well-being (MWB), and self-rated health (SRH) in German and Swiss employees.

Cross-sectional data were collected via an online questionnaire from 2118 German and Swiss employees recruited through an online panel service (18–65 years, working at least 20 h/week, various occupations). The sample provides a good representation of the working population in both countries. Using logistic regression, we analyzed how sociodemographic factors and self-reported changes in work and private life routines were associated with participants’ perceived overall impact of the COVID-19 crisis on work and private life. Moreover, we explored how the perceived impact and self-reported changes were associated with MWB and SRH.

About 30% of employees reported that their work and private life had worsened, whereas about 10% reported improvements in work and 13% in private life. Mandatory short-time work was strongly associated with perceived negative impact on work life, while work from home, particularly if experienced for the first time, was strongly associated with a perceived positive impact on work life. Concerning private life, younger age, living alone, reduction in leisure time, and changes in quantity of caring duties were strongly associated with perceived negative impact. In contrast, living with a partner or family, short-time work, and increases in leisure time and caring duties were associated with perceived positive impact on private life. Perceived negative impact of the crisis on work and private life and mandatory short-time work were associated with lower MWB and SRH. Moreover, perceived positive impact on private life and an increase in leisure time were associated with higher MWB.

The results of this study show the differential impact of the COVID-19 crisis on people’s work and private life as well as the consequences for MWB and SRH. This may inform target groups and situation-specific interventions to ameliorate the crisis.

Peer Review reports

Key findings

31% of employees perceived a negative impact of the crisis on their work life. Mandatory short-time workers and those who lost their job felt the negative impact the most.

10% of employees perceived a positive impact of the crisis on their work life. Those working in home-office, particularly if experienced for the first time, felt the positive impact the most.

30% of employees perceived a negative impact of the crisis on their private life. Living in a single household, reduction in leisure time, and changes in quantity of caring duties (i.e., increase or decrease) were strongly associated with the negative impact.

13% of employees perceived a positive impact on their private life. Living with a partner or family, mandatory short-time work, increases in leisure time and caring duties were strongly associated with the positive impact.

Perceived negative impact of the crisis on work and private life and mandatory short-time work were strongly associated with lower mental well-being and self-rated health.

Perceived positive impact of the crisis on private life and an increase in leisure time were strongly associated with higher mental well-being and, for leisure time, also with higher self-rated health.

Targeted interventions for vulnerable groups should be established on a company/governmental levels such as psychological first aid accessible online or rapid financial aids for those who have lost their income partially or completely.

Companies may consider offering positive psychology trainings to employees to help them purposefully focus on and make use of the beneficial consequences of the crisis. Such trainings may also include workshops on optimal crafting of their work and leisure time during the pandemic.

On January 30, 2020, the World Health Organization (WHO) declared the outbreak of COVID-19 a Public Health Emergency of International Concern (PHEIC) [ 1 ]. In the following weeks, the virus quickly spread worldwide, forcing the governments of affected countries to implement lockdown measures to decrease transmission rates and prevent the overload of hospital emergency rooms. Switzerland entered full lockdown on March 16th, Germany followed 6 days later on March 22nd. Restrictive measures in both countries were comparable and included border controls, closing of schools, markets, restaurants, nonessential shops, bars, entertainment and leisure facilities, as well as ban on all public and private events and gatherings [ 2 , 3 ]. Such strict measures were in place until the end of April when both governments started to gradually ease the measures [ 4 , 5 ]. Consequently, much of the working population suddenly faced drastic changes to everyday life. People who commuted to work and had rich social lives outside their homes found themselves in a mandatory work from home (WFH) situation, many employees were furloughed or laid off as various businesses and industries had to shut down, and health workers in emergency rooms as well as supermarket staff and other essential employees were faced with a dramatic increase in workload and job strain [ 6 , 7 ].

Regarding the public health impact of the COVID-19 crisis, several studies suggest that working conditions have deteriorated and that employees are more likely to experience mental health problems, such as stress, depression, and anxiety [ 8 , 9 , 10 , 11 ]. In particular, women, young adults, people with chronic diseases, and those who have lost their jobs as a result of the crisis seem to be the most affected [ 11 , 12 , 13 , 14 ]. One of the common stressors that research has highlighted is the fear of losing one’s job and, consequently, one’s income [ 7 ]. Moreover, social isolation, conflicting messages from authorities, and an ongoing state of uncertainty have been described as some of the main factors contributing to emotional distress and negatively affecting mental health and well-being [ 8 , 14 , 15 , 16 , 17 , 18 ].

In the European context, Eurofound [ 12 ] released a report on research in April 2020 involving 85,000 participants across 27 EU member countries. The data indicate that the EU population experienced high levels of loneliness, low levels of optimism, insecurity regarding their jobs and financial future, as well as a decrease in well-being. Germany scored slightly below the EU27 average in well-being, and there is further evidence that it decreased significantly in the early stages of the COVID-19 pandemic, between March 2020 and May 2020 [ 19 ]. The Eurofound report does not discuss Switzerland; however, other studies suggest that there has been an increase in emotional distress in Swiss young adults [ 20 ] and that undergraduate students have experienced higher levels of stress, depression, anxiety, and loneliness compared to the time before the COVID-19 outbreak [ 14 ]. A Swiss social monitor study reports that over 40% of Swiss adults perceive a worsened quality of life compared to before the pandemic, 10% experience feelings of loneliness, 10% report fear of losing their job, and about 1% lost their job as a result of the pandemic. The report also indicates an increase in WFH by 29% compared to before the pandemic [ 21 ].

Accordingly, the data from Eurofound [ 12 ] also suggest that European employees have experienced a dramatic increase in WFH. About 37% of the EU working population transitioned to WFH as a result of the pandemic, and 24% WFH for the first time. Before the pandemic, employees had considered remote working a benefit when it followed their preferences. However, the COVID-19 lockdown changed this by forcing many employees into mandatory WFH [ 6 ]. This posed various challenges for employees without prior WFH experience, such as organizing the workspace, establishing new communication channels with colleagues, coping with work isolation, or managing boundaries between work and non-work [ 22 , 23 , 24 ]. Without proper support from the employer or insufficient resources to manage these challenges, mandatory WFH may become a burden that negatively affects employees’ well-being [ 8 ] and, in turn, their performance [ 22 ]. Furthermore, the increase in WFH has been highlighted as a potential threat to parents with small children at home, as this group is likely to experience difficulties in combining work duties with home schooling and household chores [ 12 , 23 ].

Indisputably, the COVID-19 pandemic has had a strong impact on many aspects of our lives and will continue to do so for months and years to come. However, the consequences of the crisis and societal reactions to the challenges posed by the virus are not deemed solely negative. The new situation also holds opportunities for positive shifts in our work and private lives that were impossible before the COVID-19 crisis. Many may see this crisis as an opportunity to learn how to cope with profound changes in everyday life and even to adopt new pro-active behaviors. For instance, some employees may discover that the new ways of working (e.g., WFH) facilitate more productivity and are more satisfying compared to working in an office [ 25 ]. Data collected from employees in Denmark and Germany between March and May 2020 [ 26 ] suggest that 71% of respondents felt informed and well prepared for the changing work situation and WFH. Participants also reported several advantages of working from home, such as perceived control over the workday, working more efficiently, or saving time previously spent commuting. In contrast, some reported disadvantages of WFH included social isolation, loss of the value of work, and a lack of important work equipment. Nonetheless, respondents reported overall relatively more positive experiences of WFH than negative ones. Thus, we argue that more balanced studies are needed that examine both the negative and positive impact of the COVID-19 crisis on peoples’ lives, health, and well-being, considering differential effects in diverse subgroups. Such studies have the potential to conclude how to diminish the negative and enhance the positive outcomes of the current and future pandemic-related crises in the working population.

Aim and objectives

The overall aim of the present study was to examine the actual and perceived overall impact of the COVID-19 crisis on employees’ work and private life, along with its consequences for mental well-being (MWB) and self-rated health (SRH) in the German and Swiss working populations. Specifically, we pursued the following objectives:

To investigate the perceived positive and negative impact of the COVID-19 crisis on work and private life as well as to assess the self-reported changes in work and private life routines induced by the crisis.

To examine which sociodemographic variables and which self-reported changes in work and private life routines are associated with perceived positive and negative impact of the COVID-19 crisis on work and private life.

To investigate how the self-reported changes and perceived overall impact of the COVID-19 crisis on work and private life are associated with MWB and SRH as relevant health outcomes.

Although SRH has been identified as a relevant predictor of mental distress during the COVID-19 pandemic [ 10 , 27 ], to our knowledge, it has not been studied as an outcome variable in combination with MWB indicators as in our study.

The present study used a cross-sectional online survey design. We report our study following the STROBE guidelines for cross-sectional studies [ 28 ], and the checklist for reporting results of internet e-surveys (CHERRIES) [ 29 ], see ‘Additional file  1 .pdf’ in supplementary material.

Participants were recruited through a panel data service Respondi ( respondi.com ). Cross-sectional data were collected from employees in Germany and Switzerland via an online questionnaire using a web-based survey provider SurveyGizmo. The questionnaire was tested and checked by senior researchers from the field for face validity prior to the administration. The period of data collection was from 9th to 22nd April 2020, when both countries were in full lockdown as part of the control measures relating to COVID-19. Participants received a minimal incentive for completing the survey (i.e., points which could be redeemed towards a given service after participating in several surveys). Participation was voluntary and participant anonymity and confidentiality of their data were assured and emphasized. Each participant in the online panel service database had a unique code which ensured anonymity and prevented multiple submissions from one participant. Important items in the survey were mandatory and participants were informed if they accidently skipped an item. Further, the questionnaire used a logic to avoid asking redundant or non-applicable questions (e.g., participants who indicated that they lost their job were not asked about the change in working time or home-office). Moreover, we included several disqualifying items (i.e., “Please choose number three as an answer to this item”) as a quality check to exclude participants who would give random answers. Participants were able to go back in the survey and review or change their answers.

The eligibility criteria were: being employed (not self-employed), working more than 20 h per week, and being within the age range of 18 to 65 years. The final sample included 2118 participants. Figure  1 shows a flow diagram describing how the final sample was achieved.

Sample flow diagram

Sociodemographic characteristics of the sample are shown in Table  1 : the mean age was 46.51 years ( SD  = 11.28), 5% completed primary, 58% secondary, and 37% tertiary education, Footnote 1 55% were male, 77% were from Germany, and 72% were living with a partner, family, or in a shared housing.

Overall, in terms of age, education, and living situation (i.e., single households), the study sample seems to be a good representation of the target of the working population in Germany ( www.destatis.de ) and Switzerland ( www.bfs.admin.ch ). In general, males were slightly overrepresented in our sample (56%) compared to the general population (52%); however, the proportion of males in both countries did not differ significantly (56% from Germany, 52% from Switzerland), χ 2 (1) = 1.63, p  = 0.201.

Perceived overall impact of COVID-19 on work and private life

Assuming that both improvements and deteriorations can simultaneously occur due to COVID-19, we designed four separate items (see ‘Additional file  2 .pdf’ in supplementary material) to assess participants’ subjective evaluation of the overall impact of the COVID-19 crisis on their work and private lives: “The Corona-crisis has (a) worsened my work life; (b) improved my work life; (c) worsened my private life; (d) improved my private life.” The response scale ranged from 1 =  strongly disagree to 5 =  strongly agree . As a primer to this question, we defined the Corona-crisis as follows:

“The following questions deal directly with the current COVID-19 (Corona) pandemic and the consequent regulations from the government (i.e., business closures, school closures, event bans, contact reduction in public spaces, etc.). Hereafter, we refer to this collectively as the Corona-crisis. Please compare your current situation with the situation as it was before the government regulations.”

Changes in work and private life routines

The following items examined qualitative and quantitative changes in participants’ work and private life routines resulting from the COVID-19 crisis: (a) change in employment contract ( no change ; short-time work Footnote 2 with a reduced contract ; short-time work with a contract reduced to 0 h ; job loss ); (b) proportion of WFH before and after COVID-19 ( 0 to 100% ; participants were grouped into three categories according to their answers: None , Experienced , New Footnote 3 ); (c) changes in quantity of working time,; (d) changes in quantity of leisure time; and (e) changes in quantity of caring duties. The response scale for items c, d, and e ranged from 1 =  strongly decreased to 5 =  strongly increased . For the statistical analysis, responses were grouped into three categories: decreased (1 + 2), unchanged (3), increased (4 + 5).

• Mental well-being

MWB was assessed with the Warwick-Edinburgh Mental Well-Being Scale (WEMWBS) [ 30 ]. Specifically, we used the German translation of the 7-item short version of the WEMWBS [ 31 ]. WEMWBS is a measure of MWB capturing the positive aspects of mental health, namely, positive affect (feelings of optimism, relaxation), satisfying interpersonal relationships, and positive functioning (clear thinking, self-acceptance, competence, autonomy). The response scale ranged from 1 =  never to 5 =  all the time . For the statistical analysis (i.e., ordinal logistic regression model), we grouped participants into six categories according to their overall score in percentiles (10, 25, 50, 75, 90, 99%).

• Self-rated health

SRH was assessed with a single item: “In general, how would you evaluate your health?” [ 32 ]. The response scale ranged from 1 =  very bad to 5 =  very good . The application of single-item measures for self-evaluated health is a gold standard in public health research [ 33 ].

Statistical analysis

Data analysis was carried out using R version 4.0.2. In the first step, four ordinal logistic regression models using polr from the MASS R package [ 34 ] were fitted to assess associations of the perceived overall impact of COVID-19 on work and private life as outcome variables with sociodemographic factors (gender, age, country, living situation) and factors related to changes in work and private life routines (changes in employment contract, WFH, work time, leisure time, caring duties) as independent variables. To verify that there was no multicollinearity, the variables were tested a priori using the variance inflation factor tested vif from the car R package [ 35 ] (VIF < 2). The results are presented as adjusted odds ratio (OR) with 95% confidence intervals (95% CI) interpreted as the OR of reporting a higher level of the impact compared to the reference category.

Further, two additional ordinal logistic regression models were fitted to investigate the association between the perceived overall impact of COVID-19 on work and private life Footnote 4 and the self-reported changes in work and private life routines as independent variables and MWB with SRH as outcome variables. In both models, we also controlled for possible confounders (gender, age, country, living situation). The results are presented as adjusted OR with 95% CI interpreted as the OR of reporting a higher level of MWB/SRH compared to the reference category.

Figure  2 displays the correlations between the analyzed variables. Education was not included in the regression models due to missing data (see details in the Methods section).

Correlation matrix of the analyzed variables. Note: Only correlations with p  < 0.01 displayed; Gender (1 = Female, 2 = Male); Country (1 = Germany, 2 = Switzerland); Education (1 = Primary, 2 = Secondary, 3 = Tertiary); Living situation (1 = Alone, 2 = With partner/family); Contract change (1 = No change, 2 = Short-time reduced, 3 = Short-time 0, 4 = Job loss); Home-office (1 = None, 2 = Experienced, 3 = New)

Perceived overall impact of COVID-19 crisis and self-reported changes in work and private life routines

Figure  3 shows the results for the four items related to the perceived overall impact of the COVID-19 crisis on work and private life. Thirty-one percent of participants (strongly) agreed that their work life had worsened and 30% (strongly) agreed that their private life had worsened. In contrast, 10% (strongly) agreed that their work life had improved and 13% (strongly) agreed that their private life had improved as a result of the COVID-19 crisis.

Perceived impact on work and private life and self-reported changes in work time, leisure time, and caring duties. Note: Total percentage does not always equal 100% due to rounding error

Further, Fig.  3 shows self-reported changes with regard to the quantity of time actually spent in work and private life. Work time decreased for 38%, leisure time increased for 36%, while the amount of caring duties changed for 26% of participants.

Figures  4 and 5 show self-reported changes with regard to contracted working hours and home-office. Twenty-eight percent of participants experienced a change in their employment contract, while 27% were affected by mandatory short-time work, 1% lost their job as a result of the COVID-19 crisis. Fifty-one percent reported to WFH and of those, 20% reported doing so for the first time.

Self-reported changes in home-office. Note: None = 0% WFH before COVID-19, 0% after; Experienced = at least 10% WFH before and at least 10% after COVID-19; New = 0% WFH before and at least 10% after COVID-19

Self-reported changes in contracted working hours. Note: Short-time reduced = work hours temporarily partly reduced by employer; Short time 0 = work hours temporarily reduced to 0 by employer

Factors associated with perceived impact on work life

Table  2 shows OR comparisons between different subgroups concerning their evaluation of the degree to which their work life had worsened or improved due to the COVID-19 crisis, assessed by two separate dependent variables. Regarding perceived negative impact on work life, change in employment contract demonstrated the highest OR of reporting a deterioration of work life. The association was particularly strong in participants who had their contract reduced to mandatory short-time work with 0 working hours (OR = 9.72) and in those who had lost their job (OR = 35.07). Further, participants who reported a change in their work time had a significantly higher OR of reporting a deterioration of work life (OR = 2.95; 2.06). Finally, changes in leisure time and increased caring duties were significantly associated with perceived deterioration of work life. This association was particularly strong for a decrease in leisure time (OR = 1.62) and an increase in caring duties (OR = 1.58).

Regarding perceived positive impact of COVID-19 on work life, WFH had the highest OR of reporting an improvement in work life. The association was particularly strong in those who had started to WFH for the first time (OR = 2.77). Increase in leisure time was also significantly associated with a positive impact on work life. Further, older employees in the 51–60 and 61–65 age groups had significantly lower odds of reporting a positive impact of COVID-19 on work life (OR = 0.71; 0.61), as well as short-time employees, in particular those with a contract reduced to 0 working hours (OR = 0.53), and those who reported a decrease in work time (OR = 0.61).

Factors associated with perceived impact on private life

Table 2 further shows OR comparisons within different subgroups concerning their evaluation of the degree to which their private life had worsened or improved due to the COVID-19 crisis, assessed by two separate dependent variables. Regarding perceived negative impact on private life, the subgroup of participants living with a partner, family, or in a shared housing had significantly lower odds of reporting the deterioration of their private life compared to those living alone (OR = 0.41). The odds of reporting deterioration of private life were lower also for the 61–65 age group (OR = 0.58). Finally, changes in the quantity of leisure time and quantity of caring duties were associated with perceived deterioration of private life, and this association was particularly strong for a decrease in leisure time (OR = 2.62) and a decrease in caring duties (OR = 1.62).

Regarding perceived positive impact on private life, the strongest association was with an increase in leisure time (OR = 2.25), followed by living with a partner, family, or in a shared housing (OR = 1.74); WFH, particularly among those with prior WFH experience (OR = 1.72); and with an increase in caring duties (OR = 1.33). Short-time workers had significantly higher odds of reporting a positive impact on their private life compared to workers without any change, especially those with a contract reduced to 0 working hours (OR = 1.57).

Association between the perceived impact, self-reported changes, mental well-being and self-rated health

Table  3 shows the results of the associations between perceived overall impact, the self-reported changes in work and private life routines, and relevant health outcomes in terms of MWB and SRH, controlled for various sociodemographic variables. Regarding the perceived overall impact, participants who (strongly) agreed that COVID-19 had worsened their work life reported significantly lower MWB (OR = 0.61) compared to those who (strongly) disagreed. In addition, participants who neither agreed nor disagreed that their work life had worsened reported lower MWB (OR = 0.71) compared to those who (strongly) disagreed. A strong negative association could also be seen regarding perceived negative impact on private life: participants who (strongly) agreed that their private life had worsened reported lower MWB (OR = 0.62) and SRH scores (OR = 0.67) compared to those who (strongly) disagreed. Both outcomes were also negatively associated with employees who neither agreed nor disagreed that their private life had worsened (OR = 0.80; 0.66) compared to those who (strongly) disagreed. Finally, participants who (strongly) agreed that their private life had improved as a result of the COVID-19 crisis had higher odds of reporting a higher MWB score (OR = 1.39) compared to those who (strongly) disagreed.

Regarding the impact of the self-reported changes in work and private life routines, mandatory short-time workers with a contract reduced to 0 working hours reported significantly lower MWB (OR = 0.57) and SRH (OR = 0.49) compared to participants without any change in their employment contract. In contrast, an increase in leisure time was positively associated with both better MWB (OR = 1.23) and SRH (OR = 1.45).

The present study aimed to examine the impact of the COVID-19 crisis on employees’ work and private life and the consequences for MWB and SRH in German and Swiss employees. The first objective of the study was to assess the perceived impact and self-reported changes related to COVID-19. Although the research has thus far mostly emphasized the negative impact of the COVID-19 crisis [ 9 , 10 , 11 , 12 , 36 ], our data show that more than 40% of participants perceived no negative changes and over 10% even positive shifts in both life domains. This can be partly explained by the experienced changes in daily routines: 28% of participants were affected by a change in their employment contract and 49% by changes in the quantity of work time, confirming almost identical findings for Germany in the Eurofound report [ 12 ]. Also, quantity of leisure time and of caring duties changed for 58 and 26% respectively. The finding that about half WFH at least part of their working time, and 20% for the first time is also in line with Eurofound’s data where 24% reported WFH for the first time [ 12 ]. Overall, the proportion of people affected by changes in work and private life is comparable but hardly exceeds 50%, similar to the proportion of participants who reported a deterioration in their work and private life.

The second objective was to explore the factors associated with perceived impact on work and private life. A change in contracted work hours (i.e., mandatory short-time work, job loss), and changes in work time were strongly associated with reporting deterioration of work life. Those affected by short-time work experienced a significant disruption in their work routine as well as fear of losing the job, factors associated with increased level of distress and low MWB [ 7 ]. In consequence, employees whose contract had been reduced or terminated due to the lockdown measures are particularly vulnerable to developing mental health problems [ 11 , 13 ]. Further, an increase in caring duties, and, perhaps more surprisingly, increase and decrease in leisure time were strongly associated with perceived deterioration of work life. Such changes in private life routines may require efforts for readjustments that can interfere with work and work-life balance. These readjustments may be particularly difficult for older employees (i.e., age group 61–65) who were more likely to report deterioration of their work life. They may be particularly sensitive to changes in daily structure and less flexible in adapting to a new situation, such as mandatory WFH, less personal contact with colleagues, and an increase in the use of digital technology.

WFH was most strongly associated with perceived positive impact of the COVID-19 crisis on work life, particularly in those reporting WFH for the first time, supporting evidence from Ipsen and colleagues [ 26 ]. This positive impact of WFH may be explained by a reduction or absence of commute time, more job autonomy, more flexible workdays, and ultimately, extra time for leisure. In fact, increased leisure time was another important factor associated with perceived positive impact of the COVID-19 crisis on work life. More time for leisure may allow for better recovery from work and rebuilding of personal resources [ 37 , 38 ], which can then help an individual deal with work demands. In contrast, a change in contracted working hours and a decrease in work time were negatively associated with perceived positive impact on work life. A reduction in work time may not only cause financial problems, but also reduces important daily routines and social interactions at work, and may trigger fear of losing one’s job. Again, older employees may struggle more with the new situation and may be less successful in transforming it to their benefit, explaining why the oldest age groups, 54–60 and 61–65 years, were less likely to report an improvement in their work life.

Regarding the perceived impact on private life, participants living alone were more likely to report a deterioration and less likely to report an improvement of their private life compared to those living with a partner, family, or in a shared housing. The COVID-19 lockdown substantially restricted possibilities for social interactions beyond one’s own household, particularly affecting people living alone. For individuals who live alone, this may lead to feelings of loneliness [ 12 ], which in turn, threatens their MWB [ 39 ], highlighting the importance of having opportunities for direct exchange in such a crisis situation. This could also explain that an increase in caring duties, allowing for more exchange with family members, was associated with perceived positive shifts in private life. Further, an increase in WFH showed to be beneficial also to the private life, particularly to those experienced in WFH who did not need to first establish their workspace and new routines. Increase in leisure time and, more surprisingly, mandatory short-time work were also associated with positive impact on private life, as employees can engage more freely in activities they value. Interestingly, participants over 60 years old were less likely to report a deterioration of their private life. Older employees may be less dependent on the number of social contacts beyond their household, and they may have more mature emotion regulation strategies than the younger generations [ 40 ]. Indeed, mental well-being of the German elderly population (65+) remained largely unaltered during the early COVID-19 lockdown [ 41 ].

Finally, our third objective was to investigate how the perceived overall impact and self-reported changes induced by the crisis were associated with MWB and SRH. Low SRH has been associated with increased odds of depression [ 27 ], displaying the relevance of SRH for psychologically demanding situations, such as the COVID-19 pandemic. Our results suggest a strong negative association between the perceived negative impact on work and private life, MWB and SRH, indicating that this perception by itself is of relevance. It is of note that the perceived negative impact, particularly in private life, had such a strong association with SRH, which is more stable over time than MWB. In contrast, perceived positive impact on private life was associated with higher MWB. It seems that those who were able to cope with the COVID-19 crisis and translate the lockdown measures into some positive shifts in their private life, also benefited in terms of increased MWB.

Looking at the impact of the self-reported changes on MWB and SRH, mandatory short-time work with 0 contracted working hours was strongly associated with a lower MWB and SRH. Short-time work leads to significant losses of financial security and of daily structure and routines. Conversely, an increase in leisure time was positively associated with MWB, and the link was even stronger with SRH. More time for leisure gives extra opportunities for individuals to engage in meaningful activities that provide them with important resources that benefit their MWB and SRH. The overall strength of the associations indicates that MBW may be more affected by the perceived impact, as both are cognitive-emotional domains and are more dependent on the cognitive appraisal of one’s situation and emotional experience. SRH, on the other hand, may be more affected by actual changes in work and private life that increase or decrease opportunities to engage in activities that are perceived as beneficial to health.

Limitations and strengths

A major limitation is the cross-sectional design, which allowed only to infer associations between variables but did not provide evidence of the directions of the associations or potential causality. Furthermore, the online survey created timely data on the immediate impact of the COVID-19 crisis situation. However, the self-reported data may be influenced by common method biases [ 42 ], such as social desirability bias [ 43 ] or self-selection bias, posing potential threats to the validity of our findings. Thus, we hired a professional panel data service that guarantees collection of high quality data. Moreover, we implemented various strategies in the questionnaire such as using disqualifying items to prevent invalid answers. The sociodemographic characteristics of our sample indicate a good representation of the target population. Finally, we did not control for all variables that might have affected the results. For instance, coping with a crisis and MWB differ individually and may be influenced by variables such as personality traits, resilience, or coping style [ 44 , 45 , 46 , 47 ]. However, our study aimed to provide a broad picture of both the negative and positive impacts of the COVID-19 crisis on a large, diverse sample of the working population. Thus, it was beyond the scope of this study to investigate individual differences and characteristics. In addition, a more complete, lengthy survey would have likely reduced the participation rate.

A strength of the present study is the relatively large and heterogeneous sample size that allowed us to conduct a detailed analysis and explore different subgroups within the sample. Another strength is the time point of the data collection launched at the beginning of April 2020, close to the first peak of the COVID-19 outbreak in Germany and Switzerland and onset of the related lockdown measures. This enabled us to capture a valid picture of the immediate impact of the lockdown measures. Moreover, the survey assessed the present situation, adding to the validity compared to a retrospective survey design. Finally, the combination of a subjective evaluation of the impact of the crisis with relevant, standardized public health indicators of MWB and SRH increases the relevance of the results to public health research and for policymaking.

Conclusion and policy recommendations

The present study contributes to our understanding of the impact of the COVID-19 crisis on work and private life. It provides evidence on the covariates of a more negative/positive perceived impact and on the associations with MWB and SRH in the German and Swiss working populations. Employees whose employment contract was affected by the crisis seem to have felt the greatest negative impact on their work life. This highlights the crucial role of (un−/under-)employment in a crisis, as employment is associated with several health-promoting factors that cannot be substituted in any other way [ 48 ]. Moreover, the private life of employees living alone has been affected most negatively due to social isolation. Thus, psychological first aid also accessible online should be established particularly for these vulnerable groups [ 49 ]. Employers need to assure that they keep close social ties with and emotionally support employees with reduced contract or working hours. Moreover, rapid financial aids are needed to those who have lost their income partially or completely.

Nevertheless, we should also foster positive consequences of the crisis. In general, it seems that an increase in WFH was positive for work life. Learning from the beneficial effects of WFH in a crisis can inform future organizational and legislative policies to support this form of working. As employees experienced with WFH had a stronger positive impact on private life than first-timers, future WFH policies should include offering training and exchange of experience between employees on how to establish positive routines compatible with their private life. This will help employees to proactively identify their preferences and craft their work environment accordingly [ 50 ]. Further, an increase in leisure time was particularly positive for private life. More leisure time allows for dedicating extra time to activities one enjoys, and this may be beneficial also for recovery and detachment from work [ 51 ] and for mental health in general [ 52 ]. Thus, employees could also be trained in optimal crafting of their leisure time to strengthen these beneficial effects [ 53 , 54 ].

Finally, we saw that besides the reported actual changes in work and private life, also the perception of the overall positive or negative impact is related to the health outcomes. This suggests to offer positive psychology trainings to employees helping them to purposefully focus on and make use of potential positive consequences of the crisis [ 55 , 56 , 57 ]. From a longitudinal research perspective, it would be interesting to further examine how the actual and perceived impact of the ongoing crisis as well as the associated health outcomes change over time and whether some of the new routines developed during the pandemic will be maintained in the long term.

To conclude, our study adds to recent evidence [ 58 ] that the Covid-19 crisis and related lockdown measures do not have solely negative impact. Rather, it affects vulnerable groups of individuals who need targeted support, while the majority of the population remain healthy or even experience positive shifts in their daily life.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request. The R code used for the statistical analysis is available in the GitHub repository: https://github.com/jesuismartin/covid

Education estimates are based on data from n  = 1194 participants who took part in a subsequent wave of data collection (December 2020), missing values ( n  = 924) were imputed using mice R package (for details see supplementary material). Education was not included in the regression models as the imputed data could potentially threaten the validity of our conclusions.

Short-time work is defined as “public programs that allow firms experiencing economic difficulties to temporarily reduce the hours worked while providing their employees with income support from the State for the hours not worked” (European Commission, 2020, Retrieved from: https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1587138033761&uri=CELEX%3A52020PC0139 ).

None  = 0% WFH before COVID-19, 0% after; Experienced  = at least 10% WFH before and at least 10% after COVID-19; New  = 0% WFH before and at least 10% after COVID-19.

Participants were grouped into three categories according to their answers: disagree (1 + 2), neither/nor (3), agree (4 + 5).

Abbreviations

World Health Organization

Public Health Emergency of International Concern

Work from home

European Union

Confidence interval

World Health Organization. World Health Organization Director-General’s statement on IHR Emergency Committee on Novel Coronavirus (2019-nCoV). 2020. https://www.who.int/dg/speeches/detail/who-director-general-s-statement-on-ihr-emergency-committee-on-novel-coronavirus- (2019-ncov). Accessed 19 May 2020.

Google Scholar  

Federal Government of Switzerland. 22. March 2020: Regeln zum Corona-Virus [Rules about the Corona virus] 2020. https://www.bundesregierung.de/breg-de/leichte-sprache/22-maerz-2020-regeln-zum-corona-virus-1733310 . Accessed 20 Feb 2021.

Federal Council of Switzerland. Federal Council declares “extraordinary situation” and introduces more stringent measures [press release]. 2020. https://www.admin.ch/gov/en/start/documentation/media-releases.msg-id-78454.html . Accessed 20 Feb 2021.

Federal Government of Switzerland. “Wir müssen ganz konzentriert weiter machen”. 2020. [“We have to stay focused”]. 2020. https://www.bundesregierung.de/breg-de/themen/coronavirus/bund-laender-corona-1744306 . Accessed 20 Feb 2021.

Federal Council of Switzerland. Federal Council to gradually ease measures against the new coronavirus [press release]. 2020. https://www.admin.ch/gov/en/start/documentation/media-releases.msg-id-78818.html . Accessed 20 Feb 2021.

Kniffin KM, Narayanan J, Anseel F, Antonakis J, Ashford SP, Bakker A, et al. COVID-19 and the workplace: implications, issues, and insights for future research and action. Am Psychol. 2020;76(1):1–52. https://doi.org/10.1037/amp0000716 .

Article   Google Scholar  

Koh D, Goh HP. Occupational health responses to COVID-19: what lessons can we learn from SARS? J Occup Health. 2020;62(1):1–6. https://doi.org/10.1002/1348-9585.12128 .

Article   CAS   Google Scholar  

Brooks SK, Webster RK, Smith LE, Woodland L, Wessely S, Greenberg N, et al. The psychological impact of quarantine and how to reduce it: rapid review of the evidence. Lancet. 2020;395(10227):912–20. https://doi.org/10.1016/S0140-6736(20)30460-8 .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Rodríguez-Rey R, Garrido-Hernansaiz H, Collado S. Psychological impact and associated factors during the initial stage of the coronavirus (COVID-19) pandemic among the general population in Spain. Front Psychol. 2020;11:1–23. https://doi.org/10.3389/fpsyg.2020.01540 .

Wang C, Pan R, Wan X, Tan Y, Xu L, Ho CS, et al. Immediate psychological responses and associated factors during the initial stage of the 2019 coronavirus disease (COVID-19) epidemic among the general population in China. Int J Environ Res Public Health. 2020;17(5):1–25. https://doi.org/10.3390/ijerph17051729 .

Qiu J, Shen B, Zhao M, Wang Z, Xie B, Xu Y. A nationwide survey of psychological distress among Chinese people in the COVID-19 epidemic: implications and policy recommendations. Gen Psychiatr. 2020;33(2):1–4. https://doi.org/10.1136/gpsych-2020-100213 .

Eurofound. Living, working and COVID-19: First findings – April 2020. 2020. https://www.eurofound.europa.eu/publications/report/2020/living-working-and-covid-19-first-findings-april-2020 . Accessed 1 June 2020.

Ozamiz-Etxebarria N, Idoiaga Mondragon N, Dosil Santamaría M, Picaza GM. Psychological symptoms during the two stages of lockdown in response to the COVID-19 outbreak: an investigation in a sample of citizens in northern Spain. Front Psychol. 2020;11:1–9. https://doi.org/10.3389/fpsyg.2020.01491 .

Elmer T, Mepham K, Stadtfeld C. Students under lockdown: comparisons of students’ social networks and mental health before and during the COVID-19 crisis in Switzerland. PLoS One. 2020;15(7):1–22. https://doi.org/10.1371/journal.pone.0236337 .

Carvalho Aguiar Melo M, de Sousa Soares D. Impact of social distancing on mental health during the COVID-19 pandemic: an urgent discussion. Int J Soc Psychiatry. 2020;66:625–6. https://doi.org/10.1177/0020764020927047 .

Article   PubMed   PubMed Central   Google Scholar  

Venkatesh A, Edirappuli S. Social distancing in covid-19: what are the mental health implications? BMJ. 2020;369:1. https://doi.org/10.1136/bmj.m1379 .

Pfefferbaum B, North CS. Mental health and the Covid-19 pandemic. N Engl J Med. 2020;383(6):510–2. https://doi.org/10.1056/NEJMp2008017 .

Article   CAS   PubMed   Google Scholar  

Benke C, Autenrieth LK, Asselmann E, Pané-Farré CA. Lockdown, quarantine measures, and social distancing: associations with depression, anxiety and distress at the beginning of the COVID-19 pandemic among adults from Germany. Psychiatry Res. 2020;293:1–10. https://doi.org/10.1016/j.psychres.2020.113462 .

Zacher H, Rudolph C. Individual differences and changes in subjective wellbeing during the early stages of the COVID-19 pandemic. Am Psychol. 2020;76(1):50–62. https://doi.org/10.1037/amp0000702 .

Article   PubMed   Google Scholar  

Shanahan L, Steinhoff A, Bechtiger L, Murray AL, Nivette A, Hepp U, et al. Emotional distress in young adults during the COVID-19 pandemic: evidence of risk and resilience from a longitudinal cohort study. Psychol Med. 2020:1–10. https://doi.org/10.1017/S003329172000241X .

Moser A, Carlander M, Wieser S, Hämmig O, Puhan MA, Höglinger M. The COVID-19 social monitor longitudinal online panel: real-time monitoring of social and public health consequences of the COVID-19 emergency in Switzerland. PLoS One. 2020;15(11):1–12. https://doi.org/10.1371/journal.pone.0242129 .

Ozcelik H, Barsade SG. No employee an island: workplace loneliness and job performance. AMJ. 2018;61(6):2343–66. https://doi.org/10.5465/amj.2015.1066 .

Shimazu A, Nakata A, Nagata T, Arakawa Y, Kuroda S, Inamizu N, et al. Psychosocial impact of COVID-19 for general workers. J Occup Health. 2020;62(1):1–2. https://doi.org/10.1002/1348-9585.12132 .

Cho E. Examining boundaries to understand the impact of COVID-19 on vocational behaviors. J Vocat Behav. 2020;119:1–3. https://doi.org/10.1016/j.jvb.2020.103437 .

Kramer A, Kramer K. The potential impact of the Covid-19 pandemic on occupational status, work from home, and occupational mobility. J Vocat Behav. 2020;119:1–4. https://doi.org/10.1016/j.jvb.2020.103442 .

Ipsen C, Kirchner K, Hansen J. Experiences of working from home in times of COVID-19. International survey conducted the first months of the national lockdowns March-May, 2020. https://www.forskningsdatabasen.dk/en/catalog/2595069795 . Accessed 20 Aug 2020.

Gao J, Zheng P, Jia Y, Chen H, Mao Y, Chen S, et al. Mental health problems and social media exposure during COVID-19 outbreak. PLoS One. 2020;15(4):1–10. https://doi.org/10.1371/journal.pone.0231924 .

von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med. 2007;147(8):573–7. https://doi.org/10.7326/0003-4819-147-8-200710160-00010 .

Eysenbach G. Improving the quality of web surveys: the checklist for reporting results of internet E-surveys (CHERRIES). J Med Internet Res. 2004;6(3):1–6. https://doi.org/10.2196/jmir.6.3.e34 .

Tennant R, Hiller L, Fishwick R, Platt S, Joseph S, Weich S, et al. The Warwick-Edinburgh mental well-being scale (WEMWBS): development and UK validation. Health Qual Life Outcomes. 2007;5(1):1–13. https://doi.org/10.1186/1477-7525-5-63 .

Lang G, Bachinger A. Validation of the German Warwick-Edinburgh mental well-being scale (WEMWBS) in a community-based sample of adults in Austria: a bi-factor modelling approach. J Public Health. 2017;25(2):135–46. https://doi.org/10.1007/s10389-016-0778-8 .

Idler EL, Benyamini Y. Self-rated health and mortality: a review of twenty-seven community studies. J Health Soc Behav. 1997;38(1):21–37. https://doi.org/10.2307/2955359 .

Bjorner JB, Fayers P, Idler E. Self-rated health. In: Fayers PM, Hays RD, editors. Assessing quality of life in clinical trials: methods and practice. 2nd ed. Oxford: Oxford Univ. Press; 2005. p. 309–23.

Venables WN, Ripley BD. Modern applied statistics with S. New York: Springer; 2002. https://doi.org/10.1007/978-0-387-21706-2 .

Book   Google Scholar  

Fox J, Weisberg S. An R companion to applied regression. Thousand Oaks: Sage; 2019.

Sibley CG, Greaves LM, Satherley N, Wilson MS, Overall NC, Lee CHJ, et al. Effects of the COVID-19 pandemic and nationwide lockdown on trust, attitudes toward government, and well-being. Am Psychol. 2020;75(5):1–14. https://doi.org/10.1037/amp0000662 .

Hobfoll SE. Conservation of resources: a new attempt at conceptualizing stress. Am Psychol. 1989;44(3):513–24. https://doi.org/10.1037/0003-066X.44.3.513 .

Hobfoll SE, Halbesleben J, Neveu J-P, Westman M. Conservation of resources in the organizational context: the reality of resources and their consequences. Ann Rev Org Psychol Org Behav. 2018;5(1):103–28. https://doi.org/10.1146/annurev-orgpsych-032117-104640 .

Ahmed MZ, Ahmed O, Aibao Z, Hanbin S, Siyu L, Ahmad A. Epidemic of COVID-19 in China and associated psychological problems. Asian J Psychiatr. 2020;51:1–8. https://doi.org/10.1016/j.ajp.2020.102092 .

Carstensen LL, Fung HH, Charles ST. Socioemotional selectivity theory and the regulation of emotion in the second half of life. Motiv Emot. 2003;27(2):103–23. https://doi.org/10.1023/A:1024569803230 .

Röhr S, Reininghaus U, Riedel-Heller SG. Mental wellbeing in the German old age population largely unaltered during COVID-19 lockdown: results of a representative survey. BMC Geriatr. 2020;20(1):1–12. https://doi.org/10.1186/s12877-020-01889-x .

Podsakoff PM, MacKenzie SB, Lee J-Y, Podsakoff NP. Common method biases in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol. 2003;88(5):879–903. https://doi.org/10.1037/0021-9010.88.5.879 .

Larsen MV, Petersen MB, Nyrup J. Do Survey Estimates of the Public’s Compliance with COVID-19 Regulations Suffer from Social Desirability Bias? J Behav Public Admin. 2020;3:1–14. https://doi.org/10.31234/osf.io/cy4hk .

Emmons RA, Diener E. Personality correlates of subjective well-being. Personal Soc Psychol Bull. 1985;11(1):89–97. https://doi.org/10.1177/0146167285111008 .

McCauley M, Minsky S, Viswanath K. The H1N1 pandemic: media frames, stigmatization and coping. BMC Public Health. 2013;13(1):1–16. https://doi.org/10.1186/1471-2458-13-1116 .

Anglim J, Horwood S, Smillie LD, Marrero RJ, Wood JK. Predicting psychological and subjective well-being from personality: a meta-analysis. Psychol Bull. 2020;146(4):279–323. https://doi.org/10.1037/bul0000226 .

Tonkin K, Malinen S, Näswall K, Kuntz JC. Building employee resilience through wellbeing in organizations. Hum Resour Dev Q. 2018;29(2):107–24. https://doi.org/10.1002/hrdq.21306 .

Jahoda M. Employment and unemployment: a social-psychological analysis. Cambridge: Cambridge University Press; 1982.

Zürcher SJ, Kerksieck P, Adamus C, Burr CM, Lehmann AI, Huber FK, et al. Prevalence of mental health problems during virus epidemics in the general public, health care workers and survivors: a rapid review of the evidence. Front Public Health. 2020;8:1–15. https://doi.org/10.3389/fpubh.2020.560389 .

Demerouti E. Design your own job through job crafting. Eur Psychol. 2014;19(4):237–47. https://doi.org/10.1027/1016-9040/a000188 .

Wendsche J, Lohmann-Haislah A. A meta-analysis on antecedents and outcomes of detachment from work. Front Psychol. 2016;7:1–24. https://doi.org/10.3389/fpsyg.2016.02072 .

Demerouti E, Mostert K, Bakker A. Burnout and work engagement: a thorough investigation of the independency of both constructs. J Occup Health Psychol. 2010;15(3):209–22. https://doi.org/10.1037/a0019408 .

Kosenkranius M, Rink FA, de Bloom J, van den Heuvel M. The design and development of a hybrid off-job crafting intervention to enhance needs satisfaction, well-being and performance: a study protocol for a randomized controlled trial. BMC Public Health. 2020;20(1):1–11. https://doi.org/10.1186/s12889-020-8224-9 .

De Bloom J, Vaziri H, Tay L, Kujanpää M. An identity-based integrative needs model of crafting: crafting within and across life domains. J Appl Psychol. 2020;105(12):1423–46. https://doi.org/10.1037/apl0000495 .

Bakker A, van Woerkom M. Strengths use in organizations: a positive approach of occupational health. Can Psychol. 2018;59(1):38–46. https://doi.org/10.1037/cap0000120 .

Peláez MJ, Coo C, Salanova M. Facilitating work engagement and performance through strengths-based micro-coaching: a controlled trial study. J Happiness Stud. 2020;21(4):1265–84. https://doi.org/10.1007/s10902-019-00127-5 .

Waters L, Algoe SB, Dutton J, Emmons R, Fredrickson BL, Heaphy E, et al. Positive psychology in a pandemic: buffering, bolstering, and building mental health. J Posit Psychol. 2021:1–21. https://doi.org/10.1080/17439760.2021.1871945 .

Ahrens KF, Neumann RJ, Kollmann B, Plichta MM, Lieb K, Tüscher O, et al. Differential impact of COVID-related lockdown on mental health in Germany. World Psychiatry. 2021;20(1):140–1. https://doi.org/10.1002/wps.20830 .

Download references

Acknowledgements

The authors would like to thank to Roald Pijpker from Wageningen University for his helpful comments during the final editing of the manuscript.

MT received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 801076, through the SSPH+ Global PhD Fellowship Programme in Public Health Sciences (GlobalP3HS) of the Swiss School of Public Health. RB, PK, and GB received funding from the University of Zurich Foundation. Beyond providing the funding, these funding bodies were not involved at any stage of the study.

Author information

Authors and affiliations.

Public & Organizational Health, Center of Salutogenesis, Epidemiology, Biostatistics and Prevention Institute, University of Zurich, Hirschengraben 84, 8001, Zurich, Switzerland

Martin Tušl, Rebecca Brauchli, Philipp Kerksieck & Georg Friedrich Bauer

You can also search for this author in PubMed   Google Scholar

Contributions

MT planned and carried out data collection and analysis, interpretation of the results, writing and reviewing the manuscript in collaboration with the co-authors. RB contributed to the research concept, data collection, data analysis, and review of the manuscript. PK was involved with the conceptualization of the research, interpretation of the results, writing, and review of the manuscript. GB contributed to the conceptualization of the research, interpretation of results, writing, and review of the manuscript. All authors read and approved the final manuscript before submission.

Corresponding author

Correspondence to Martin Tušl .

Ethics declarations

Ethics approval and consent to participate.

Informed consent was obtained from all participants, the study included adult participants (18+ years) only. Participants voluntarily completed the questionnaires, guaranteeing their anonymity. For anonymous surveys on working/living conditions and self-reported mental well-being and health no ethical review was necessary under national, university, or departmental rules (Department of Data Protection at the University of Zurich, www.dsd.uzh.ch/en/ ). The study was conducted under strict observation of ethical and professional guidelines. The study was not registered prior to the start of the data collection as this is not common in the field of occupational health psychology where this study originated. The study is part of a larger longitudinal data collection on occupational health and individual strategies employee use to craft their work, started already before the Covid-19 pandemic. When the pandemic started, we decided to add the study aim to explore the immediate impact of the Covid-19 crisis on Swiss and German working population presented in this paper. The manuscript is an accurate and transparent account of the study, and no important aspects of the study or any analyses conducted have been omitted.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1., additional file 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Tušl, M., Brauchli, R., Kerksieck, P. et al. Impact of the COVID-19 crisis on work and private life, mental well-being and self-rated health in German and Swiss employees: a cross-sectional online survey. BMC Public Health 21 , 741 (2021). https://doi.org/10.1186/s12889-021-10788-8

Download citation

Received : 06 January 2021

Accepted : 30 March 2021

Published : 17 April 2021

DOI : https://doi.org/10.1186/s12889-021-10788-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Online survey
• Logistic regression
• Private life
• Switzerland

BMC Public Health

ISSN: 1471-2458

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 11 February 2021

Methodological quality of COVID-19 clinical research

• Richard G. Jung   ORCID: orcid.org/0000-0002-8570-6736 1 , 2 , 3   na1 ,
• Pietro Di Santo 1 , 2 , 4 , 5   na1 ,
• Cole Clifford 6 ,
• Graeme Prosperi-Porta 7 ,
• Stephanie Skanes 6 ,
• Annie Hung 8 ,
• Simon Parlow 4 ,
• Sarah Visintini   ORCID: orcid.org/0000-0001-6966-1753 9 ,
• F. Daniel Ramirez   ORCID: orcid.org/0000-0002-4350-1652 1 , 4 , 10 , 11 ,
• Trevor Simard 1 , 2 , 3 , 4 , 12 &
• Benjamin Hibbert   ORCID: orcid.org/0000-0003-0906-1363 2 , 3 , 4  

Nature Communications volume  12 , Article number:  943 ( 2021 ) Cite this article

13k Accesses

96 Citations

238 Altmetric

Metrics details

• Infectious diseases
• Public health

The COVID-19 pandemic began in early 2020 with major health consequences. While a need to disseminate information to the medical community and general public was paramount, concerns have been raised regarding the scientific rigor in published reports. We performed a systematic review to evaluate the methodological quality of currently available COVID-19 studies compared to historical controls. A total of 9895 titles and abstracts were screened and 686 COVID-19 articles were included in the final analysis. Comparative analysis of COVID-19 to historical articles reveals a shorter time to acceptance (13.0[IQR, 5.0–25.0] days vs. 110.0[IQR, 71.0–156.0] days in COVID-19 and control articles, respectively; p  < 0.0001). Furthermore, methodological quality scores are lower in COVID-19 articles across all study designs. COVID-19 clinical studies have a shorter time to publication and have lower methodological quality scores than control studies in the same journal. These studies should be revisited with the emergence of stronger evidence.

Similar content being viewed by others

A meta-analysis on global change drivers and the risk of infectious disease

Long COVID: major findings, mechanisms and recommendations

Refining the impact of genetic evidence on clinical success

Introduction.

The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic spread globally in early 2020 with substantial health and economic consequences. This was associated with an exponential increase in scientific publications related to the coronavirus disease 2019 (COVID-19) in order to rapidly elucidate the natural history and identify diagnostic and therapeutic tools 1 .

While a need to rapidly disseminate information to the medical community, governmental agencies, and general public was paramount—major concerns have been raised regarding the scientific rigor in the literature 2 . Poorly conducted studies may originate from failure at any of the four consecutive research stages: (1) choice of research question relevant to patient care, (2) quality of research design 3 , (3) adequacy of publication, and (4) quality of research reports. Furthermore, evidence-based medicine relies on a hierarchy of evidence, ranging from the highest level of randomized controlled trials (RCT) to the lowest level of case series and case reports 4 .

Given the implications for clinical care, policy decision making, and concerns regarding methodological and peer-review standards for COVID-19 research 5 , we performed a formal evaluation of the methodological quality of published COVID-19 literature. Specifically, we undertook a systematic review to identify COVID-19 clinical literature and matched them to historical controls to formally evaluate the following: (1) the methodological quality of COVID-19 studies using established quality tools and checklists, (2) the methodological quality of COVID-19 studies, stratified by median time to acceptance, geographical regions, and journal impact factor and (3) a comparison of COVID-19 methodological quality to matched controls.

Herein, we show that COVID-19 articles are associated with lower methodological quality scores. Moreover, in a matched cohort analysis with control articles from the same journal, we reveal that COVID-19 articles are associated with lower quality scores and shorter time from submission to acceptance. Ultimately, COVID-19 clinical studies should be revisited with the emergence of stronger evidence.

Article selection

A total of 14787 COVID-19 papers were identified as of May 14, 2020 and 4892 duplicate articles were removed. In total, 9895 titles and abstracts were screened, and 9101 articles were excluded due to the study being pre-clinical in nature, case report, case series <5 patients, in a language other than English, reviews (including systematic reviews), study protocols or methods, and other coronavirus variants with an overall inter-rater study inclusion agreement of 96.7% ( κ  = 0.81; 95% CI, 0.79–0.83). A total number of 794 full texts were reviewed for eligibility. Over 108 articles were excluded for ineligible study design or publication type (such as letter to the editors, editorials, case reports or case series <5 patients), wrong patient population, non-English language, duplicate articles, wrong outcomes and publication in a non-peer-reviewed journal. Ultimately, 686 articles were identified with an inter-rater agreement of 86.5% ( κ  = 0.68; 95% CI, 0.67–0.70) (Fig.  1 ).

A total of 14787 articles were identified and 4892 duplicate articles were removed. Overall, 9895 articles were screened by title and abstract leaving 794 articles for full-text screening. Over 108 articles were excluded, leaving a total of 686 articles that underwent methodological quality assessment.

COVID-19 literature methodological quality

Most studies originated from Asia/Oceania with 469 (68.4%) studies followed by Europe with 139 (20.3%) studies, and the Americas with 78 (11.4%) studies. Of included studies, 380 (55.4%) were case series, 199 (29.0%) were cohort, 63 (9.2%) were diagnostic, 38 (5.5%) were case–control, and 6 (0.9%) were RCTs. Most studies (590, 86.0%) were retrospective in nature, 620 (90.4%) reported the sex of patients, and 7 (2.3%) studies excluding case series calculated their sample size a priori. The method of SARS-CoV-2 diagnosis was reported in 558 studies (81.3%) and ethics approval was obtained in 556 studies (81.0%). Finally, journal impact factor of COVID-19 manuscripts was 4.7 (IQR, 2.9–7.6) with a time to acceptance of 13.0 (IQR, 5.0–25.0) days (Table  1 ).

Overall, when COVID-19 articles were stratified by study design, a mean case series score (out of 5) (SD) of 3.3 (1.1), mean NOS cohort study score (out of 8) of 5.8 (1.5), mean NOS case–control study score (out of 8) of 5.5 (1.9), and low bias present in 4 (6.4%) diagnostic studies was observed (Table  2 and Fig.  2 ). Furthermore, in the 6 RCTs in the COVID-19 literature, there was a high risk of bias with little consideration for sequence generation, allocation concealment, blinding, incomplete outcome data, and selective outcome reporting (Table  2 ).

A Distribution of COVID-19 case series studies scored using the Murad tool ( n  = 380). B Distribution of COVID-19 cohort studies scored using the Newcastle–Ottawa Scale ( n  = 199). C Distribution of COVID-19 case–control studies scored using the Newcastle–Ottawa Scale ( n  = 38). D Distribution of COVID-19 diagnostic studies scored using the QUADAS-2 tool ( n  = 63). In panel D , blue represents low risk of bias and orange represents high risk of bias.

For secondary outcomes, rapid time from submission to acceptance (stratified by median time of acceptance of <13.0 days) was associated with lower methodological quality scores for case series and cohort study designs but not for case–control nor diagnostic studies (Fig.  3A–D ). Low journal impact factor (<10) was associated with lower methodological quality scores for case series, cohort, and case–control designs (Fig.  3E–H ). Finally, studies originating from different geographical regions had no differences in methodological quality scores with the exception of cohort studies (Fig.  3I–L ). When dichotomized by high vs. low methodological quality scores, a similar trend was observed with rapid time from submission to acceptance (34.4% vs. 46.3%, p  = 0.01, Supplementary Fig.  1B ), low impact factor journals (<10) was associated with lower methodological quality score (38.8% vs. 68.0%, p  < 0.0001, Supplementary Fig.  1C ). Finally, studies originating in either Americas or Asia/Oceania was associated with higher methodological quality scores than Europe (Supplementary Fig.  1D ).

A When stratified by time of acceptance (13.0 days), increased time of acceptance was associated with higher case series score ( n  = 186 for <13 days and n  = 193 for >=13 days; p  = 0.02). B Increased time of acceptance was associated with higher NOS cohort score ( n  = 112 for <13 days and n  = 144 for >=13 days; p  = 0.003). C No difference in time of acceptance and case–control score was observed ( n  = 18 for <13 days and n  = 27 for >=13 days; p  = 0.34). D No difference in time of acceptance and diagnostic risk of bias (QUADAS-2) was observed ( n  = 43 for <13 days and n  = 33 for >=13 days; p  = 0.23). E When stratified by impact factor (IF ≥10), high IF was associated with higher case series score ( n  = 466 for low IF and n  = 60 for high IF; p  < 0.0001). F High IF was associated with higher NOS cohort score ( n  = 262 for low IF and n  = 68 for high IF; p  = 0.01). G No difference in IF and case–control score was observed ( n  = 62 for low IF and n  = 2 for high IF; p  = 0.052). H No difference in IF and QUADAS-2 was observed ( n  = 101 for low IF and n  = 2 for high IF; p  = 0.93). I When stratified by geographical region, no difference in geographical region and case series score was observed ( n  = 276 Asia/Oceania, n  = 135 Americas, and n  = 143 Europe/Africa; p  = 0.10). J Geographical region was associated with differences in cohort score ( n  = 177 Asia/Oceania, n  = 81 Americas, and n  = 89 Europe/Africa; p  = 0.01). K No difference in geographical region and case–control score was observed ( n  = 37 Asia/Oceania, n  = 13 Americas, and n  = 14 Europe/Africa; p  = 0.81). L No difference in geographical region and QUADAS-2 was observed ( n  = 49 Asia/Oceania, n  = 28 Americas, and n  = 28 Europe/Africa; p  = 0.34). In panels A – D , orange represents lower median time of acceptance and blue represents high median time of acceptance. In panels E – H , red is low impact factor and blue is high impact factor. In panels I – L , orange represents Asia/Oceania, blue represents Americas, and brown represents Europe. Differences in distributions were analysed by two-sided Kruskal–Wallis test. Differences in diagnostic risk of bias were quantified by Chi-squares test. p  < 0.05 was considered statistically significant.

Methodological quality score differences in COVID-19 versus historical control

We matched 539 historical control articles to COVID-19 articles from the same journal with identical study designs in the previous year for a final analysis of 1078 articles (Table  1 ). Overall, 554 (51.4%) case series, 348 (32.3%) cohort, 64 (5.9%) case–control, 106 (9.8%) diagnostic and 6 (0.6%) RCTs were identified from the 1078 total articles. Differences exist between COVID-19 and historical control articles in geographical region of publication, retrospective study design, and sample size calculation (Table  1 ). Time of acceptance was 13.0 (IQR, 5.0–25.0) days in COVID-19 articles vs. 110.0 (IQR, 71.0–156.0) days in control articles (Table  1 and Fig.  4A , p  < 0.0001). Case-series methodological quality score was lower in COVID-19 articles compared to the historical control (3.3 (1.1) vs. 4.3 (0.8); n  = 554; p  < 0.0001; Table  2 and Fig.  4B ). Furthermore, NOS score was lower in COVID-19 cohort studies (5.8 (1.6) vs. 7.1 (1.0); n  = 348; p  < 0.0001; Table  2 and Fig.  4C ) and case–control studies (5.4 (1.9) vs. 6.6 (1.0); n  = 64; p  = 0.003; Table  2 and Fig.  4D ). Finally, lower risk of bias in diagnostic studies was in 12 COVID-19 articles (23%; n  = 53) compared to 24 control articles (45%; n  = 53; p  = 0.02; Table  2 and Fig.  4E ). A similar trend was observed between COVID-19 and historical control articles when dichotomized by good vs. low methodological quality scores (Supplementary Fig.  2 ).

A Time to acceptance was reduced in COVID-19 articles compared to control articles (13.0 [IQR, 5.0–25.0] days vs. 110.0 [IQR, 71.0–156.0] days, n  = 347 for COVID-19 and n  = 414 for controls; p  < 0.0001). B When compared to historical control articles, COVID-19 articles were associated with lower case series score ( n  = 277 for COVID-19 and n  = 277 for controls; p  < 0.0001). C COVID-19 articles were associated with lower NOS cohort score compared to historical control articles ( n  = 174 for COVID-19 and n  = 174 for controls; p  < 0.0001). D COVID-19 articles were associated with lower NOS case–control score compared to historical control articles ( n  = 32 for COVID-19 and n  = 32 for controls; p  = 0.003). E COVID-19 articles were associated with higher diagnostic risk of bias (QUADAS-2) compared to historical control articles ( n  = 53 for COVID-19 and n  = 53 for controls; p  = 0.02). For panel A , boxplot captures 5, 25, 50, 75 and 95% from the first to last whisker. Orange represents COVID-19 articles and blue represents control articles. Two-sided Mann–Whitney U-test was conducted to evaluate differences in time to acceptance between COVID-19 and control articles. Differences in study quality scores were evaluated by two-sided Kruskal–Wallis test. Differences in diagnostic risk of bias were quantified by Chi-squares test. p  < 0.05 was considered statistically significant.

In this systematic evaluation of methodological quality, COVID-19 clinical research was primarily observational in nature with modest methodological quality scores. Not only were the study designs low in the hierarchy of scientific evidence, we found that COVID-19 articles were associated with a lower methodological quality scores when published with a shorter time of publication and in lower impact factor journals. Furthermore, in a matched cohort analysis with historical control articles identified from the same journal of the same study design, we demonstrated that COVID-19 articles were associated with lower quality scores and shorter time from submission to acceptance.

The present study demonstrates comparative differences in methodological quality scores between COVID-19 literature and historical control articles. Overall, the accelerated publication of COVID-19 research was associated with lower study quality scores compared to previously published historical control studies. Our research highlights major differences in study quality between COVID-19 and control articles, possibly driven in part by a combination of more thorough editorial and/or peer-review process as suggested by the time to publication, and robust study design with questions which are pertinent for clinicians and patient management 3 , 6 , 7 , 8 , 9 , 10 , 11 .

In the early stages of the COVID-19 pandemic, we speculate that an urgent need for scientific data to inform clinical, social and economic decisions led to shorter time to publication and explosion in publication of COVID-19 studies in both traditional peer-reviewed journals and preprint servers 1 , 12 . The accelerated scientific process in the COVID-19 pandemic allowed a rapid understanding of natural history of COVID-19 symptomology and prognosis, identification of tools including RT-PCR to diagnose SARS-CoV-2 13 , and identification of potential therapeutic options such as tocilizumab and convalescent plasma which laid the foundation for future RCTs 14 , 15 , 16 . A delay in publication of COVID-19 articles due to a slower peer-review process may potentially delay dissemination of pertinent information against the pandemic. Despite concerns of slow peer review, major landmark trials (i.e. RECOVERY and ACTT-1 trial) 17 , 18 published their findings in preprint servers and media releases to allow for rapid dissemination. Importantly, the data obtained in these initial studies should be revisited as stronger data emerges as lower quality studies may fundamentally risk patient safety, resource allocation and future scientific research 19 .

Unfortunately, poor evidence begets poor clinical decisions 20 . Furthermore, lower quality scientific evidence potentially undermines the public’s trust in science during this time and has been evident through misleading information and high-profile retractions 12 , 21 , 22 , 23 . For example, the benefits of hydroxychloroquine, which were touted early in the pandemic based on limited data, have subsequently failed to be replicated in multiple observational studies and RCTs 5 , 24 , 25 , 26 , 27 , 28 , 29 , 30 . One poorly designed study combined with rapid publication led to considerable investment of both the scientific and medical community—akin to quinine being sold to the public as a miracle drug during the 1918 Spanish Influenza 31 , 32 . Moreover, as of June 30, 2020, ClinicalTrials.gov listed an astonishing 230 COVID-19 trials with hydroxychloroquine/plaquenil, and a recent living systematic review of observational studies and RCTs of hydroxychloroquine or chloroquine for COVID-19 demonstrated no evidence of benefit nor harm with concerns of severe methodological flaws in the included studies 33 .

Our study has important limitations. We evaluated the methodological quality of existing studies using established checklists and tools. While it is tempting to associate methodological quality scores with reproducibility or causal inferences of the intervention, it is not possible to ascertain the impact on the study design and conduct of research nor results or conclusions in the identified reports 34 . Second, although the methodological quality scales and checklists used for the manuscript are commonly used for quality assessment in systematic reviews and meta-analyses 35 , 36 , 37 , 38 , they can only assess the methodology without consideration for causal language and are prone to limitations 39 , 40 . Other tools such as the ROBINS-I and GRADE exist to evaluate methodological quality of identified manuscripts, although no consensus currently exists for critical appraisal of non-randomized studies 41 , 42 , 43 . Furthermore, other considerations of quality such as sample size calculation, sex reporting or ethics approval are not considered in these quality scores. As such, the quality scores measured using these checklists only reflect the patient selection, comparability, diagnostic reference standard and methods to ascertain the outcome of the study. Third, the 1:1 ratio to identify our historical control articles may affect the precision estimates of our findings. Interestingly, a simulation of an increase from 1:1 to 1:4 control ratio tightened the precision estimates but did not significantly alter the point estimate 44 . Furthermore, the decision for 1:1 ratio in our study exists due to limitations of available historical control articles from the identical journal in the restricted time period combined with a large effect size and sample size in the analysis. Finally, our analysis includes early publications on COVID-19 and there is likely to be an improvement in quality of related studies and study design as the field matures and higher-quality studies. Accordingly, our findings are limited to the early body of research as it pertains to the pandemic and it is likely that over time research quality will improve over time.

In summary, the early body of peer-reviewed COVID-19 literature was composed primarily of observational studies that underwent shorter peer-review evaluation and were associated with lower methodological quality scores than comparable studies. COVID-19 clinical studies should be revisited with the emergence of stronger evidence.

A systematic literature search was conducted on May 14, 2020 (registered on June 3, 2020 at PROSPERO: CRD42020187318) and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses. Furthermore, the cohort study was reported according to the Strengthening The Reporting of Observational Studies in Epidemiology checklist. The data supporting the findings of this study is available as Supplementary Data  1 – 2 .

Data sources and searches

The search was created in MEDLINE by a medical librarian with expertise in systematic reviews (S.V.) using a combination of key terms and index headings related to COVID-19 and translated to the remaining bibliographic databases (Supplementary Tables  1 – 3 ). The searches were conducted in MEDLINE (Ovid MEDLINE(R) ALL 1946–), Embase (Ovid Embase Classic + Embase 1947–) and the Cochrane Central Register of Controlled Trials (from inception). Search results were limited to English-only publications, and a publication date limit of January 1, 2019 to present was applied. In addition, a Canadian Agency for Drugs and Technologies in Health search filter was applied in MEDLINE and Embase to remove animal studies, and commentary, newspaper article, editorial, letter and note publication types were also eliminated. Search results were exported to Covidence (Veritas Health Innovation, Melbourne, Australia) and duplicates were eliminated using the platform’s duplicate identification feature.

Study selection, data extraction and methodological quality assessment

We included all types of COVID-19 clinical studies, including case series, observational studies, diagnostic studies and RCTs. For diagnostic studies, the reference standard for COVID-19 diagnosis was defined as a nasopharyngeal swab followed by reverse transcriptase-polymerase chain reaction in order to detect SARS-CoV-2. We excluded studies that were exploratory or pre-clinical in nature (i.e. in vitro or animal studies), case reports or case series of <5 patients, studies published in a language other than English, reviews, methods or protocols, and other coronavirus variants such as the Middle East respiratory syndrome.

The review team consisted of trained research staff with expertise in systematic reviews and one trainee. Title and abstracts were evaluated by two independent reviewers using Covidence and all discrepancies were resolved by consensus. Articles that were selected for full review were independently evaluated by two reviewers for quality assessment using a standardized case report form following the completion of a training period where all reviewers were trained with the original manuscripts which derived the tools or checklists along with examples for what were deemed high scores 35 , 36 , 37 , 38 . Following this, reviewers completed thirty full-text extractions and the two reviewers had to reach consensus and the process was repeated for the remaining manuscripts independently. When two independent reviewers were not able reach consensus, a third reviewer (principal investigator) provided oversight in the process to resolve the conflicted scores.

First and corresponding author names, date of publication, title of manuscript and journal of publication were collected for all included full-text articles. Journal impact factor was obtained from the 2018 InCites Journal Citation Reports from Clarivate Analytics. Submission and acceptance dates were collected in manuscripts when available. Other information such as study type, prospective or retrospective study, sex reporting, sample size calculation, method of SARS-CoV-2 diagnosis and ethics approval was collected by the authors. Methodological quality assessment was conducted using the Newcastle–Ottawa Scale (NOS) for case–control and cohort studies 37 , QUADAS-2 tool for diagnostic studies 38 , Cochrane risk of bias for RCTs 35 and a score derived by Murad et al. for case series studies 36 .

Identification of historical control from identified COVID-19 articles

Following the completion of full-text extraction of COVID-19 articles, we obtained a historical control group by identifying reports matched in a 1:1 fashion. From the eligible COVID-19 article, historical controls were identified by searching the same journal in a systematic fashion by matching the same study design (“case series”, “cohort”, “case control” or “diagnostic”) starting in the journal edition 12 months prior to the COVID-19 article publication on the publisher website (i.e. COVID-19 article published on April 2020, going backwards to April 2019) and proceeding forward (or backward if a specific article type was not identified) in a temporal fashion until the first matched study was identified following abstract screening by two independent reviewers. If no comparison article was found by either reviewers, the corresponding COVID-19 article was excluded from the comparison analysis. Following the identification of the historical control, data extraction and quality assessment was conducted on the identified articles using the standardized case report forms by two independent reviewers and conflicts resolved by consensus. The full dataset has been made available as Supplementary Data  1 – 2 .

Data synthesis and statistical analysis

Continuous variables were reported as mean (SD) or median (IQR) as appropriate, and categorical variables were reported as proportions (%). Continuous variables were compared using Student t -test or Mann–Whitney U-test and categorical variables including quality scores were compared by χ 2 , Fisher’s exact test, or Kruskal–Wallis test.

The primary outcome of interest was to evaluate the methodological quality of COVID-19 clinical literature by study design using the Newcastle–Ottawa Scale (NOS) for case–control and cohort studies, QUADAS-2 tool for diagnostic studies 38 , Cochrane risk of bias for RCTs 35 , and a score derived by Murad et al. for case series studies 36 . Pre-specified secondary outcomes were comparison of methodological quality scores of COVID-19 articles by (i) median time to acceptance, (ii) impact factor, (iii) geographical region and (iv) historical comparator. Time of acceptance was defined as the time between submission to acceptance which captures peer review and editorial decisions. Geographical region was stratified into continents including Asia/Oceania, Europe/Africa and Americas (North and South America). Post hoc comparison analysis between COVID-19 and historical control article quality scores were evaluated using Kruskal–Wallis test. Furthermore, good quality of NOS was defined as 3+ on selection and 1+ on comparability, and 2+ on outcome/exposure domains and high-quality case series scores was defined as a score ≥3.5. Due to a small sample size of identified RCTs, they were not included in the comparison analysis.

The finalized dataset was collected on Microsoft Excel v16.44. All statistical analyses were performed using SAS v9.4 (SAS Institute, Inc., Cary, NC, USA). Statistical significance was defined as P  < 0.05. All figures were generated using GraphPad Prism v8 (GraphPad Software, La Jolla, CA, USA).

Reporting summary

Further information on research design is available in the  Nature Research Reporting Summary linked to this article.

Data availability

The authors can confirm that all relevant data are included in the paper and in Supplementary Data  1 – 2 . The original search was conducted on MEDLINE, Embase and Cochrane Central Register of Controlled Trials.

Chen, Q., Allot, A. & Lu, Z. Keep up with the latest coronavirus research. Nature 579 , 193 (2020).

Article   ADS   CAS   Google Scholar  

Mahase, E. Covid-19: 146 researchers raise concerns over chloroquine study that halted WHO trial. BMJ https://doi.org/10.1136/bmj.m2197 (2020).

Chalmers, I. & Glasziou, P. Avoidable waste in the production and reporting of research evidence. Lancet 374 , 86–89 (2009).

Article   Google Scholar  

Burns, P. B., Rohrich, R. J. & Chung, K. C. The levels of evidence and their role in evidence-based medicine. Plast. Reconstr. Surg. 128 , 305–310 (2011).

Article   CAS   Google Scholar  

Alexander, P. E. et al. COVID-19 coronavirus research has overall low methodological quality thus far: case in point for chloroquine/hydroxychloroquine. J. Clin. Epidemiol. 123 , 120–126 (2020).

Barakat, A. F., Shokr, M., Ibrahim, J., Mandrola, J. & Elgendy, I. Y. Timeline from receipt to online publication of COVID-19 original research articles. Preprint at medRxiv https://doi.org/10.1101/2020.06.22.20137653 (2020).

Chan, A.-W. et al. Increasing value and reducing waste: addressing inaccessible research. Lancet 383 , 257–266 (2014).

Ioannidis, J. P. A. et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 383 , 166–175 (2014).

Chalmers, I. et al. How to increase value and reduce waste when research priorities are set. Lancet 383 , 156–165 (2014).

Salman, R. A.-S. et al. Increasing value and reducing waste in biomedical research regulation and management. Lancet 383 , 176–185 (2014).

Glasziou, P. et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet 383 , 267–276 (2014).

Bauchner, H. The rush to publication: an editorial and scientific mistake. JAMA 318 , 1109–1110 (2017).

He, X. et al. Temporal dynamics in viral shedding and transmissibility of COVID-19. Nat. Med. 26 , 672–675 (2020).

Guaraldi, G. et al. Tocilizumab in patients with severe COVID-19: a retrospective cohort study. Lancet Rheumatol. 2 , e474–e484 (2020).

Duan, K. et al. Effectiveness of convalescent plasma therapy in severe COVID-19 patients. Proc. Natl Acad. Sci. USA 117 , 9490–9496 (2020).

Shen, C. et al. Treatment of 5 critically Ill patients with COVID-19 with convalescent plasma. JAMA 323 , 1582–1589 (2020).

Beigel, J. H. et al. Remdesivir for the treatment of covid-19—final report. N. Engl. J. Med. 383 , 1813–1826 (2020).

Group, R. C. et al. Dexamethasone in hospitalized patients with Covid-19—preliminary report. N. Engl. J. Med. https://doi.org/10.1056/NEJMoa2021436 (2020).

Ramirez, F. D. et al. Methodological rigor in preclinical cardiovascular studies: targets to enhance reproducibility and promote research translation. Circ. Res 120 , 1916–1926 (2017).

Heneghan, C. et al. Evidence based medicine manifesto for better healthcare. BMJ 357 , j2973 (2017).

Mehra, M. R., Desai, S. S., Ruschitzka, F. & Patel, A. N. RETRACTED: hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet https://doi.org/10.1016/S0140-6736(20)31180-6 (2020).

Servick, K. & Enserink, M. The pandemic’s first major research scandal erupts. Science 368 , 1041–1042 (2020).

Mehra, M. R., Desai, S. S., Kuy, S., Henry, T. D. & Patel, A. N. Retraction: Cardiovascular disease, drug therapy, and mortality in Covid-19. N. Engl. J. Med. 382 , 2582–2582, https://doi.org/10.1056/NEJMoa2007621. (2020).

Article   PubMed   Google Scholar  

Boulware, D. R. et al. A randomized trial of hydroxychloroquine as postexposure prophylaxis for Covid-19. N. Engl. J. Med. 383 , 517–525 (2020).

Gautret, P. et al. Clinical and microbiological effect of a combination of hydroxychloroquine and azithromycin in 80 COVID-19 patients with at least a six-day follow up: a pilot observational study. Travel Med. Infect. Dis. 34 , 101663–101663 (2020).

Geleris, J. et al. Observational study of hydroxychloroquine in hospitalized patients with Covid-19. N. Engl. J. Med. 382 , 2411–2418 (2020).

Borba, M. G. S. et al. Effect of high vs low doses of chloroquine diphosphate as adjunctive therapy for patients hospitalized with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection: a randomized clinical trial. JAMA Netw. Open 3 , e208857–e208857 (2020).

Mercuro, N. J. et al. Risk of QT interval prolongation associated with use of hydroxychloroquine with or without concomitant azithromycin among hospitalized patients testing positive for coronavirus disease 2019 (COVID-19). JAMA Cardiol. 5 , 1036–1041 (2020).

Molina, J. M. et al. No evidence of rapid antiviral clearance or clinical benefit with the combination of hydroxychloroquine and azithromycin in patients with severe COVID-19 infection. Médecine et. Maladies Infectieuses 50 , 384 (2020).

Group, R. C. et al. Effect of hydroxychloroquine in hospitalized patients with Covid-19. N. Engl. J. Med . 383, 2030–2040 (2020).

Shors, T. & McFadden, S. H. 1918 influenza: a Winnebago County, Wisconsin perspective. Clin. Med. Res. 7 , 147–156 (2009).

Stolberg, S. A Mad Scramble to Stock Millions of Malaria Pills, Likely for Nothing (The New York Times, 2020).

Hernandez, A. V., Roman, Y. M., Pasupuleti, V., Barboza, J. J. & White, C. M. Hydroxychloroquine or chloroquine for treatment or prophylaxis of COVID-19: a living systematic review. Ann. Int. Med. 173 , 287–296 (2020).

Glasziou, P. & Chalmers, I. Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers. BMJ 363 , k4645 (2018).

Higgins, J. P. T. et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 343 , d5928 (2011).

Murad, M. H., Sultan, S., Haffar, S. & Bazerbachi, F. Methodological quality and synthesis of case series and case reports. BMJ Evid. Based Med. 23 , 60–63 (2018).

Wells, G. S. B. et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analysis. http://wwwohrica/programs/clinical_epidemiology/oxfordasp (2004).

Whiting, P. F. et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann. Intern. Med. 155 , 529–536 (2011).

Sanderson, S., Tatt, I. D. & Higgins, J. P. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int. J. Epidemiol. 36 , 666–676 (2007).

Stang, A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur. J. Epidemiol. 25 , 603–605 (2010).

Guyatt, G. et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J. Clin. Epidemiol. 64 , 383–394 (2011).

Quigley, J. M., Thompson, J. C., Halfpenny, N. J. & Scott, D. A. Critical appraisal of nonrandomized studies-A review of recommended and commonly used tools. J. Evaluation Clin. Pract. 25 , 44–52 (2019).

Sterne, J. A. et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355 , i4919 (2016).

Hamajima, N. et al. Case-control studies: matched controls or all available controls? J. Clin. Epidemiol. 47 , 971–975 (1994).

Download references

Acknowledgements

This study received no specific funding or grant from any agency in the public, commercial, or not-for-profit sectors. R.G.J. was supported by the Vanier CIHR Canada Graduate Scholarship. F.D.R. was supported by a CIHR Banting Postdoctoral Fellowship and a Royal College of Physicians and Surgeons of Canada Detweiler Travelling Fellowship. The funder/sponsor(s) had no role in design and conduct of the study, collection, analysis and interpretation of the data.

Author information

These authors contributed equally: Richard G. Jung, Pietro Di Santo.

Authors and Affiliations

CAPITAL Research Group, University of Ottawa Heart Institute, Ottawa, Ontario, Canada

Richard G. Jung, Pietro Di Santo, F. Daniel Ramirez & Trevor Simard

Vascular Biology and Experimental Medicine Laboratory, University of Ottawa Heart Institute, Ottawa, Ontario, Canada

Richard G. Jung, Pietro Di Santo, Trevor Simard & Benjamin Hibbert

Department of Cellular and Molecular Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada

Richard G. Jung, Trevor Simard & Benjamin Hibbert

Division of Cardiology, University of Ottawa Heart Institute, Ottawa, Ontario, Canada

Pietro Di Santo, Simon Parlow, F. Daniel Ramirez, Trevor Simard & Benjamin Hibbert

School of Epidemiology and Public Health, University of Ottawa, Ottawa, Ontario, Canada

Pietro Di Santo

Faculty of Medicine, University of Ottawa, Ontario, Canada

Cole Clifford & Stephanie Skanes

Department of Medicine, Cumming School of Medicine, Calgary, Alberta, Canada

Graeme Prosperi-Porta

Division of Internal Medicine, The Ottawa Hospital, Ottawa, Ontario, Canada

Berkman Library, University of Ottawa Heart Institute, Ottawa, Ontario, Canada

Sarah Visintini

Hôpital Cardiologique du Haut-Lévêque, CHU Bordeaux, Bordeaux-Pessac, France

F. Daniel Ramirez

L’Institut de Rythmologie et Modélisation Cardiaque (LIRYC), University of Bordeaux, Bordeaux, France

Department of Cardiovascular Medicine, Mayo Clinic, Rochester, MN, USA

Trevor Simard

You can also search for this author in PubMed   Google Scholar

Contributions

R.G.J., P.D.S., S.V., F.D.R., T.S. and B.H. participated in the study conception and design. Data acquisition, analysis and interpretation were performed by R.G.J., P.D.S., C.C., G.P.P., S.P., S.S., A.H., F.D.R., T.S. and B.H. Statistical analysis was performed by R.G.J., P.D.S. and B.H. The manuscript was drafted by R.G.J., P.D.S., F.D.R., T.S. and B.H. All authors approved the final version of the manuscript and agree to be accountable to all aspects of the work.

Corresponding author

Correspondence to Benjamin Hibbert .

Ethics declarations

Competing interests.

B.H. reports funding as a clinical trial investigator from Abbott, Boston Scientific and Edwards Lifesciences outside of the submitted work. The remaining authors declare no competing interests.

Additional information

Peer review information Nature Communications Ian White and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information, peer review file, description of additional supplementary files, supplementary data 1, supplementary data 2, reporting summary, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Jung, R.G., Di Santo, P., Clifford, C. et al. Methodological quality of COVID-19 clinical research. Nat Commun 12 , 943 (2021). https://doi.org/10.1038/s41467-021-21220-5

Download citation

Received : 16 July 2020

Accepted : 13 January 2021

Published : 11 February 2021

DOI : https://doi.org/10.1038/s41467-021-21220-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

The quality of covid-19 systematic reviews during the coronavirus 2019 pandemic: an exploratory comparison.

• Kevin T. McDermott
• Jos Kleijnen

Systematic Reviews (2024)

Gatekeeping should be conserved in the open science era

• Hugh Desmond

Synthese (2024)

Exploring COVID-19 research credibility among Spanish scientists

• Eduardo Garcia-Garzon
• Ariadna Angulo-Brunet
• Guido Corradi

Current Psychology (2024)

Primary health care research in COVID-19: analysis of the protocols reviewed by the ethics committee of IDIAPJGol, Catalonia

• Anna Moleras-Serra
• Rosa Morros-Pedros
• Ainhoa Gómez-Lumbreras

BMC Primary Care (2023)

Identifying patterns of reported findings on long-term cardiac complications of COVID-19: a systematic review and meta-analysis

• Chenya Zhao

BMC Medicine (2023)

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Impacts of COVID-19 on clinical research in the UK: A multi-method qualitative case study

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Validation, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliations School of Population Health and Environmental Sciences, King’s College London, United Kingdom, National Institute for Health Research Biomedical Research Centre at Guy’s and St. Thomas’ NHS Foundation Trust and King’s College London, United Kingdom

Roles Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Validation, Writing – original draft, Writing – review & editing

Roles Conceptualization, Funding acquisition, Writing – review & editing

• David Wyatt, 
• Rachel Faulkner-Gurstein, 
• Hannah Cowan, 
• Charles D. A. Wolfe

• Published: August 31, 2021
• https://doi.org/10.1371/journal.pone.0256871
• Peer Review
• Reader Comments

Clinical research has been central to the global response to COVID-19, and the United Kingdom (UK), with its research system embedded within the National Health Service (NHS), has been singled out globally for the scale and speed of its COVID-19 research response. This paper explores the impacts of COVID-19 on clinical research in an NHS Trust and how the embedded research system was adapted and repurposed to support the COVID-19 response.

Methods and findings

Using a multi-method qualitative case study of a research-intensive NHS Trust in London UK, we collected data through a questionnaire (n = 170) and semi-structured interviews (n = 24) with research staff working in four areas: research governance; research leadership; research delivery; and patient and public involvement. We also observed key NHS Trust research prioritisation meetings (40 hours) and PPI activity (4.5 hours) and analysed documents produced by the Trust and national organisation relating to COVID-19 research. Data were analysed for a descriptive account of the Trust’s COVID-19 research response and research staff’s experiences. Data were then analysed thematically. Our analysis identifies three core themes: centralisation; pace of work; and new (temporary) work practices. By centralising research prioritisation at both national and Trust levels, halting non-COVID-19 research and redeploying research staff, an increased pace in the setup and delivery of COVID-19-related research was possible. National and Trust-level responses also led to widescale changes in working practices by adapting protocols and developing local processes to maintain and deliver research. These were effective practical solutions borne out of necessity and point to how the research system was able to adapt to the requirements of the pandemic.

The Trust and national COVID-19 response entailed a rapid large-scale reorganisation of research staff, research infrastructures and research priorities. The Trust’s local processes that enabled them to enact national policy prioritising COVID-19 research worked well, especially in managing finite resources, and also demonstrate the importance and adaptability of the research workforce. Such findings are useful as we consider how to adapt our healthcare delivery and research practices both at the national and global level for the future. However, as the pandemic continues, research leaders and policymakers must also take into account the short and long term impact of COVID-19 prioritisation on non-COVID-19 health research and the toll of the emergency response on research staff.

Citation: Wyatt D, Faulkner-Gurstein R, Cowan H, Wolfe CDA (2021) Impacts of COVID-19 on clinical research in the UK: A multi-method qualitative case study. PLoS ONE 16(8): e0256871. https://doi.org/10.1371/journal.pone.0256871

Editor: Quinn Grundy, University of Toronto, CANADA

Received: April 14, 2021; Accepted: August 17, 2021; Published: August 31, 2021

Copyright: © 2021 Wyatt et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: Data from this study take the form of interview transcripts, Hospital Trust and national documents, and observations of closed meetings. These data cannot be shared publicly, but extracts from interviews are presented within the body of the paper that make up the "minimal dataset."

Funding: DW, RFG, HC and CADW are all funded by the National Institute for Health Research ( http://nihr.ac.uk/ ) Biomedical Research Centre at Guy’s and St Thomas’ NHS Foundation Trust and King’s College London (Grant number IS‐BRC‐1215‐20006). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care.

Competing interests: No

Introduction

Clinical research is a core part of the global response to COVID-19. The United Kingdom (UK), with its research system embedded within the National Health Service (NHS), has been singled out by commentators globally for the scale and speed of its COVID-19 research response, particularly in terms of trial recruitment [ 1 – 3 ]. Reporting from within the UK context, Darzi et al. suggest that participating in clinical trials should be part of the clinical pathway for all COVID-19 patients [ 4 ]. To date, 95 nationally prioritised COVID-19 research projects, labelled Urgent Public Health studies, have commenced [ 5 ]. These and a large number of other COVID-19 studies have rapidly been set up and rolled out across UK hospitals. Supporting and facilitating such research has been made possible by the widespread reorganisation of the NHS’ existing embedded research infrastructure. This reorganisation was initiated by the UK’s Department Health and Social Care (DHSC), which on 16 th March 2020 stated that all National Institute for Health Research (NIHR) funded staff should “prioritise nationally-sponsored COVID-19 research activity” [ 6 ]. They later clarified, stating “the NIHR Clinical Research Network is pausing the site set up of any new or ongoing studies at NHS and social care sites that are not nationally prioritised COVID-19 studies [ 6 ].” Such decisions were said to “enable our research workforce to focus on delivering the nationally prioritised COVID-19 studies or enable redeployment to frontline care where necessary [ 6 ].” To date, reports have focused on the outputs of this research, such as the outcomes of vaccine studies or results of treatment trials, and on frontline clinical staffing, healthcare provision and resource strains faced by hospitals and health care systems at national and global levels [ 7 – 12 ]. As yet, there has been no analysis of the organisation of the research response and the broader impact of the reorganisation of hospitals and research facilities that has allowed clinical research and emergency care work to take place during the pandemic.

In this paper we provide a detailed exploration of how the embedded research infrastructure in one NHS Trust in South London. Throughout this paper, we e use the pseudonym South London Acute Trust (SLAT) to avoid direct identification. This Trust was repurposed to support the completion of COVID-19 research and delivery of frontline care. SLAT is one of the UK’s most research-active Trusts, annually recruiting over 19,000 patients to more than 550 studies. Between February and December 2020, SLAT opened over 80 COVID-19 studies, with more than 18 of these classed as Urgent Public Health studies, recruiting over 7,000 participants. Within this context, we ask: what have been the impacts of COVID-19 on SLAT’s clinical research system, and how has the embedded research system been adapted and repurposed to support the COVID-19 response?

Prior to the pandemic, the process of setting up and managing a clinical research study within a UK NHS Trust involved multiple steps and several actors. Decisions on whether or not to open specific studies rested primarily with the relevant clinical directorate who would vet the study for its appropriateness, scientific merit and feasibility. Other processes were centralised by the Trust’s Research and Development (R&D) governance office, like the sponsorship review (that is, deciding whether the Trust will take responsibility for the study and study compliance) or assisting researchers to gain approvals from national regulatory bodies like the Medicines and Healthcare products Regulatory Agency (MHRA) and the Health Research Authority (HRA). With approvals in place, R&D would then assess whether sufficient resources were available to support the study (the capacity and capability review). Completing this process was often both onerous and time consuming. As a result of the COVID-19 pandemic, substantial parts of this process were reconfigured, as we detail below.

This is a case study of how the embedded research infrastructure at one NHS Trust was repurposed to support the delivery of frontline care and COVID-19 research. The case study method allowed us to track how the research system was adapting in real time, and enabled an in-depth look at the processes and mechanisms that have underpinned operational changes [ 13 ]. As an instrumental case study, one that focuses on socially, historically and politically situated issues, we use a single site to examine issues that are also faced by other hospital Trusts [ 14 ]. We employed an online questionnaire of research-involved staff, document analysis of emails and official national and Trust documents, observations of planning meetings and semi-structured interviews. Data were collected from individuals working in four levels of the research infrastructure: (1) central research oversight and governance (including R&D leads and research governance staff); (2) principal investigators (PIs); (3) the research delivery workforce (including research nurses, clinical research practitioners, data analysts and research managers); and (4) Patient and Public Involvement (PPI) managers and PPI representatives. Triangulating these four data sources and four levels allowed us to consider the representativeness of our data across the case. Redeployment figures and wider workforce information were provided through a request to SLAT’s research management office.

Sampling and data collection

Data were collected by DW, RFG and HC over a period of six months, from May to October 2020. In the first stage of research, an online questionnaire was disseminated to all research-involved staff at SLAT (approx. 700) on 18 th May 2020 via pre-existing mailing lists. The questionnaire closed on 10 th June 2020 with 170 responses, yielding a response rate of approximately 24%. Whilst 24% would be an inadequate response rate for statistical analysis [ 15 ], it was not intended as a validated survey, but rather a method to gain a broad understanding of staff’s experiences of the COVID-19 research response, with most questions open-ended. We received completed questionnaires from nearly a quarter of research staff during the pandemic. The questionnaire also enabled us to identify and recruit a maximum variation sample of staff involved in the research response across the four groups to interview. Interviews allowed us to explore in more depth some of the recurring themes first identified in the questionnaire.

Interview participants were also recruited using purposive and snowball sampling with an aim to maximise the representation of a variety of experiences across the case [ 16 ]. Key staff within SLAT were identified based on searching the Trust’s website, reviewing staff lists and by speaking to senior personnel for guidance. Interviews were conducted digitally on Microsoft Teams and were recorded and transcribed verbatim. Interviews focused on participants’ work prior to the pandemic, how this work has changed as a result of COVID-19, and the short and long term impacts of COVID-19 on health research more broadly.

Additionally, we obtained permission to observe the regular research prioritisation meetings convened by the Trust’s Director of R&D. These meetings took place over Microsoft Teams once or twice a week and were attended by an average of 10 senior clinical, research and research delivery leaders per session. We attended the meetings as non-participant observers, taking notes and recording proceedings. Recordings were transcribed verbatim. We also analysed all documents that were produced or circulated in connection to the prioritisation meetings. These included email discussions about specific projects, national directives, Trust protocols as well as the applications submitted by investigators to the prioritisation committee.

Lastly, we attended the handful of PPI meetings that were held by the few active PPI groups during this period. We participated in discussions about specific research projects and heard participants’ experiences of PPI during the pandemic. PPI is a core part of the pre-COVID-19 research and research design process [ 17 ]. It was therefore important that changes to PPI were considered within our study. We were also able to present our research and get feedback from groups about our aims. PPI meetings were not recorded, but detailed notes were taken during each session.

Conducting qualitative research during the COVID-19 pandemic has required us to adapt data collection methods to accommodate restrictions on face-to-face meetings and access to the hospital. Studies note that while video conferencing has many benefits, issues such as the familiarity of participants with online platforms and access to technology and high-speed internet can be barriers to the successful use of these technologies in interviewing [ 18 , 19 ]. We experienced only a handful of technical problems in our interviews. In all but two instances, interviews were conducted with cameras on so that we could observe non-verbal communication [ 20 ].

Our data were managed and analysed through NVivo 12 using a two stage process [ 21 ]. In the first stage, we analysed the data for a descriptive and narrative account, paying attention to the contours of the emerging response to COVID-19, including national and Trust decision-making and action [ 22 ]. In the second stage we used thematic analysis to develop an analytic account based on emerging themes [ 21 , 23 ]. Data were coded for key themes independently by DW, RFG and HC iteratively throughout the data collection process. Codes and core themes were then discussed and verified across the researchers. As part of our analysis process, we also presented initial findings to research staff at SLAT and at another NHS Trust. These methods of challenging our analysis both internally and externally were crucial for ensuring we reflected on our own influences on the data and the data’s utility beyond our specific case [ 24 ].

Ethics approval for the study was granted by North East—Newcastle & North Tyneside 2 REC (reference: 20/NE/0138).

We completed 24 interviews, lasting from 24 to 105 minutes (mean average of 52 minutes), observed approximately 40 hours of research prioritisation meetings and 4.5 hours of PPI meetings, and received 170 responses to the questionnaire. In the results that follow our interview participants are divided into four groups. We identify participants using a letter to denote group and number for interview within this group:—G-n (Governance/R&D staff), R-n (Research leaders/PIs), D-n (Research delivery staff), P-n (PPI managers). 3 participants sit in more than one of these groups due to their multiple roles within the Trust. These participants were interviewed using questions from interview guides for all relevant groups. Questionnaire participants are identified as Q-n, followed by a brief description of their role. See Tables 1 and 2 for a breakdown of participants.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0256871.t001

https://doi.org/10.1371/journal.pone.0256871.t002

Centralisation: Prioritising COVID-19 research and redeploying research staff

Centralisation within the research apparatus occurred across two levels.

National decision-making.

At the outset of the pandemic, DHSC took steps to assert central control over national research priorities in order to coordinate the national response to COVID-19. This included the shut down or partial shutdown of the normal functioning of the research system. A document circulated throughout the NHS on the 13 th March 2020, which included information from 25 separate Trusts, announced that elements of the UK’s national R&D infrastructure, including the UK Clinical Research Facilities (CRF) and NIHR Clinical Research Network (NIHR CRN) Coordinating Centre were “joining up working to ensure consistency of approach” and that “currently UK NIHR/RC and EU research funding bodies are in the process of selecting research that will be prioritised for approval and delivery across the NHS during the pandemic.” On 16 th March 2020 a directive from the DHSC and the Chief Medical Officer (CMO) ordered the suspension of all non-COVID-19-related research and the reorientation of research capacity towards the effort to develop COVID-19 treatments and vaccines [ 6 ]. Only those studies funded by the NIHR and where “discontinuing them will have significant detrimental effects on the ongoing care of individual participants involved in those studies” were allowed to continue [ 6 ]—in short, those studies where research was the standard of care, for example, with experimental cancer treatments. Decisions on which studies met this threshold were decided at the Trust level. Table 3 documents the scale of the pause in the normal research pipeline at SLAT. Participant G-2 saw this DHSC and CMO directive as an effective way to focus research resources:

I think the really helpful bit was the sort of diktat from Chris Whitty and Louise Wood at DH [Department of Health and Social Care] to say, “Stop everything that’s not COVID.” […] So, to actually have something centrally that said, “No, you’re not actually allowed to do that because we’ve got to focus on the COVID stuff,” was very helpful because people just stopped asking–which was great. And we were freed up to change processes as we needed to.

https://doi.org/10.1371/journal.pone.0256871.t003

Following this directive, a new system of badging certain studies as of Urgent Public Health (UPH) was established, run by DHSC and the CMO. All clinical studies including COVID-19 treatment and vaccine trials that hoped to recruit patients within NHS sites were required to apply for UPH status. An Urgent Public Health Group was convened, chaired by Nick Lemoine, the medical director of the NIHR CRN. The group was responsible for deciding which protocols to label UPH, based on evaluations of scientific merit, feasibility and greatest potential patient benefit [ 25 , 26 ]. Of the 1600 research protocols received by the CMO from March 2020 to February 2021, only 83 were considered national priorities [ 5 , 27 ]. Once a study had received UPH badging, hospital sites like SLAT were required to open them, if resources were available.

This centrally-organised prioritisation of COVID-19-related research removed the authority of individual Trusts and directorates to shape their own research portfolios. This was an unprecedented move by the DHSC, but allowed resources to be concentrated on studies deemed to have the greatest potential impact.

Trust-level decision-making.

In order to enact the DHSC mandate to prioritise COVID-19 research, SLAT created a Trust-level prioritisation process. Twice-weekly prioritisation meetings commenced early April 2020 and were attended by research governance managers, research delivery managers and senior clinicians as well as representatives from the local Clinical Research Network and partner hospitals within the network. The aim of the prioritisation meetings was to protect resources and ensure capacity to undertake UPH-badged research. However, it also ensured effective, timely communication with PIs, helped identify local PIs for new COVID-19 studies led elsewhere, and managed the pause and restart of all non-COVID research. A proforma was introduced to facilitate and standardise prioritisation decision-making. Investigators were asked to provide information summarising their projects, resource requirements and whether they had received UPH badging. Proformas were reviewed during these meetings. By the end of February 2021, this group had reviewed 170 research projects using these proformas across 68 meetings, approving over 80 studies for local setup.

During the first wave of the pandemic, prioritisation group meetings focused mainly on how to open UPH-badged studies, as all other new research had been halted. One important exception was COVID-19 studies that require little or no NHS resource and took place within a single NHS site. These studies were also discussed in these prioritisation group meetings, often with a focus placed on clinical and academic merit. Most of the studies that fitted these criteria and were approved by the prioritisation group involved university researchers analysing patient data collected and pooled in the COVID patient ‘data lake’. This enabled the Trust to maintain research activity in areas not explicitly identified as urgent public health. The research reported in this article was approved through this process.

The joined up approach between national and local decision-making however did cause confusion and frustration. The process of determining whether or not a study would be badged UPH and thus allowed to proceed was initially opaque to Trust researchers and R&D, and the national UPH review process often took weeks from application submission to outcome. Furthermore, the decision to grant a study UPH was and remains out of the hands of the sites that are tasked with delivering this research, even when internally questions were raised about the appropriateness, feasibility or scientific merit of the study. Some researchers designing studies to address key issues in relation to COVID-19 struggled to negotiate the system:

In terms of national COVID studies, we tried to get a number of studies up and going, focusing on older patients. And ran into quite a lot of obstacles and barriers. [..P]eople weren’t certain whether this was research or whether it was quality improvement, audit-type, survey-type work. And that was pretty frustrating, not being able to get clear answers on that from the senior team within R&D. And access to data was very difficult. So, despite lots of conversations about why we really needed to be focusing on older patients, the majority of people with COVID, the biggest impact being in care homes, it was quite frustrating getting hold of people who could actually sign off on studies that we would have like to have done (R-7).

At the Trust level, the prioritisation of research was also important because of the reduction in available research delivery staff. As Table 4 documents, the clinical research delivery workforce, which totalled 165 on 14 th April 2020, was reduced by 79% or 131 staff members during the peak of the first wave due to redeployment to frontline care. A further 52 non-clinical research staff were redeployed to support other Trust activity. With such a reduction of staff, the ability to maintain even those studies which had not been halted was not certain and indeed many studies required changes and protocol deviations as a result. A key point of discussion in all prioritisation meetings was the resourcing requirements of proposed studies and how these requirements might be managed alongside existing commitments. In tandem with these discussions, work was done by the research delivery manager to create a central register of research delivery staff within the Trust. The push to centralise oversite of research delivery staff was initially driven by the requirement to rapidly redeploy staff including nurses and clinical trials practitioners to support the Trust’s emergency response but it was also crucial to the prioritisation group’s understanding of the availability of research resources. Prior to the pandemic, there was no central list of all research delivery staff at the Trust, as D-2 discusses:

A benefit was actually establishing who all the staff are. The systems we have in R&D which relate to where staff sit within the Trust system depends on where they’re funded from. And because research teams have lots of mixed types of funding, some of the staff are visible to me through the systems and some aren’t. So, the only way for me to know who all the staff were, was to manually myself, physically ask. There was no system anywhere that listed who the research staff are.

https://doi.org/10.1371/journal.pone.0256871.t004

In addition to being redeployed to the clinical frontline, research staff were also pulled from across the Trust’s many directorates to form a new dedicated COVID-19 research delivery team. This team became responsible for the rapid set up and roll out of COVID studies of national and international importance, like the Oxford AstraZeneca vaccine trial, among others. Centralising oversight and management of the previously dispersed research delivery workforce enabled SLAT’s research system to react quickly and flexibly to the rapidly evolving clinical demands and research requirements of the pandemic.

While research activity was centrally coordinated within SLAT, R&D were initially left out of Trust emergency planning. An organogram produced by the Trust to represent its emergency response plan did not include R&D or any element of the research system, and a briefing document prepared by SLAT R&D for the Trust’s Gold Tactical Command Unit dated 14 th April 2020 noted this absence, and that there was also no “obvious place in the structure for R&D to naturally sit.” Participant G-3 reflected on what was perceived initially as a failure to consider the role of research:

I think […] the Trust essentially, corporately, hadn’t involved the R&D department in what they were thinking. […] We didn’t have a tactical subgroup where everybody else, every other area in the Trust had a tactical subgroup. […] There was nothing in place. You know, we’ve all voiced this, certainly in meetings at the senior management level–is that, and the words used were, “R&D has been forgotten.” We were forgotten. So, what the Trust had set up and which is, I think, probably a policy or a set of actions that they have for crisis management […] was very militarily organised. […] And we didn’t slot in, nor were we invited on to any of those tactical groups. And didn’t have representation on gold or silver command either. So we were left out of that whole process. […] We had to make real efforts to reach out and offer up. We felt that obligation and we did that.

By late April 2020, R&D were fully integrated into the Trust’s Gold Tactical Command Unit. By this time, however, the prioritisation process had been implemented and oversight of research delivery staff had been centralised, facilitating redeployment to frontline care and COVID-19 research. While the research system contributed staff and other resources to the Trust’s emergency response, it did so at its own initiation.

Pace of work: Shifting gears for the COVID-19 response

One of the most striking aspects of the research infrastructure’s response to the pandemic was the sheer pace of activity and change. The sociological literature on pace suggests that demands for faster productivity are common, and indeed this demand can be seen in the health services literature which often criticises clinical research for not moving fast enough [ 28 – 31 ]. However, the sociological literature also notes the importance of considering where things slow down or even halt [ 28 , 32 ]. In this section we document how pace appeared in participants’ accounts, acknowledging both areas where there were rapid increases in the speed of research work as well as how research work slowed down in other areas.

Increasing pace: Redeployment, research set up and research completion.

Particularly within the first wave, it was the “reserve army” (D-3) of the research delivery workforce who were required to act at speed. As per Table 4 , staff were quickly released from research duties and redeployed to the frontlines to help deliver care. In addition, all NIHR funded staff with clinical training who were not completing COVID-19 research were asked to prioritise frontline care if their employer asked [ 6 ]. Within two weeks, more research delivery staff were redeployed to COVID-19 research teams. Staff were called up one day and told to “come in on the next day” (D-8), and managers were told “they’re going tomorrow. This is their last day with you” (D-4).

As pace of redeployment accelerated, so too did the speed of research. The pace with which researchers demanded studies be delivered and set up was “ten times quicker than normal […] as if someone’s taken a time warp machine to it” (R-2). Those already working in the research infrastructure were aware that research was vital to the pandemic response and, as one participant (D-1) explained:

we needed to start the research while we’re right in the middle of the surge in numbers. And so […] you have studies that come, they need to be set up tomorrow, recruit the first patient by the end of the week.

Such shifts in normal timeframes for work were facilitated in part through centralisation, as noted above. “The real step change,” research manager G-4 suggested, “was having a Prioritisation Group and having [the] team agree a fast-track way of doing things.” Alongside streamlined approval and set-up processes, wider research infrastructures and research practices were adapting at great speed:

I was amazed that, for example, by the end of March, there were–I counted them– 13 granting agencies that, some way or another, had calls on urgent COVID-19 research (R-4).

As a result of these rapid research projects, new knowledge was being produced at an unprecedented rate, as one participant succinctly put it, “science doesn’t usually change that quickly” (D-9). This speed was met with enthusiasm by PIs and research delivery staff alike, but also caused some nervousness. Some were concerned, for example, that PPI had “dropped off the radar” (G-3), whilst others were wary of publication prior to peer review:

the […] thing which is a challenge is that we’re pre-printing research, we’re putting pre-prints out when we’re submitting to journals, because–and we’re rushing to get the pre-prints out. […] And I guess that’s good. But it is also a bit of a–a stresser because […] maybe we haven’t quite got the message right yet (R-1).

Others warned that the pace of research during the first wave of the pandemic came at a human cost. Some researchers had vastly increased workloads, “going at max […] for 5 months” (R-1), where in some cases “there’s not been a single day when [they’ve] not been working in the laboratory including all Sundays and Saturdays, Easter and so on” (R-4). Whilst some enjoyed this fast-paced moment, for those closer to the frontline it has caused anxiety. As one participant (G-5) explained, “we’ve been fire-fighting”, and at least one member of staff, another explained, “can’t come near the hospital. She has panic attacks” (D-3). Whilst it has already been documented that critical care staff’s mental health has suffered in the pandemic, these participants suggest there may also be concern for the staff involved in the research response [ 33 ].

Seeing what is possible within the exceptional circumstances of a global pandemic led some researchers and PPI managers to question the normal slower pace of regulatory approvals and assert, “if you can do it during COVID-19, you can do it any other time” (R-6). The often slow processes such as ethical approvals, data sharing guidelines, funding applications, and study set-up was a common comparator to what has been possible during the COVID-19 pandemic. Yet, as G-1 explained: “The reason [research processes have] been quicker is just because there’s been less studies.” This is evident in SLAT’s own R&D data. Table 5 documents the difference in study numbers and timeframes from initial sponsorship review to final capacity and capability approval (allowing the site set up and recruitment to commence) across 3 financial years. While some approval processes were adapted, generally research governance requirements, both internal to the Trust and at the regulators the MHRA and the Health Research Authority, remained the same. The quick approval processes were possible because no new non-COVID-19 studies were reviewed, COVID-19 studies were processed as quickly as possible and almost all non-COVID-19 related research was halted.

https://doi.org/10.1371/journal.pone.0256871.t005

Slowing or halting non-COVID-19 research.

For some investigators, the halting of non-COVID-19 research led to a slower pace where researchers could play catch up. “People have just been writing up their papers” (R-3), and this period of time “gave […] the opportunity, freed up time” (R-6) to apply for grants. Whilst many tried to set up studies so they were ready to go when restrictions were lifted, they also found that “regulatory bodies have been slower” (R-6) due to their focus on COVID-19. It was apparent that these researchers had more time to engage in PPI whilst putting these grants together–one PPI manager working in cancer (P-3) suggested “PPI activity has probably increased” during the pandemic. Whilst many researchers were understanding of the need to halt research, others found it devastating for patients and the reputation of UK research. These researchers (R-3 and R-6) pointed to other international contexts where they saw standard research continuing. Researcher R-6 was surprised “with the UK being such a […] clinical trials powerhouse”, that decision-makers didn’t “do everything it could to retain that reputation even through the COVID-19 crisis.”

On 21 st May 2020 the DHSC and NIHR circulated a framework for restarting new and paused non-COVID-19 research. Stratifying research studies into three levels of priority, this framework made no distinction between commercial and non-commercial research. Using this framework, the Trust implemented its operational Restart Plan the week commencing 1 st June 2020. Recommendations on which research studies were important or urgent to restart within each directorate was managed a directorate level, with the Prioritisation Group acting as the Trust-level decision making body for the restart plan. The Prioritisation Group continued to meet weekly to approve restart plans for research projects. By mid-summer restart was well underway but the pace of resuming all these studies could not match the pace that research stopped, and researchers were concerned that they “haven’t really been able to pick up our trial recruitment in between [waves], because recovery has been so slow” (R-5). The time of “let’s get back to normal quickly because COVID’s over”, participant R-2 explained soon turned to “actually, let’s not rush back into things because we don’t know what’s coming.” At this point the centralisation of research infrastructures hindered speed rather than aided it–one research governance manager (G-4) suggested that “we need to respect the decision-making of the research managers and matron and the R&D leads now”, but instead studies were “number 507 in the queue”, and having to “wait another week for this prioritisation meeting” whilst “people are really scared about their finances […] frightened about not finishing […] patients are waiting.”

Adopting new and virtual working practices

The response to COVID-19 pandemic has resulted in broad shifts in working patterns across the labour market, and will likely lead to longer term transformations to work practices stemming from these temporary changes [ 34 – 36 ]. In health, research highlights the accelerated adoption of digital and virtual working practices as a result of COVID-19, such as the use of telemedicine in secondary care [ 37 – 39 ]. The implementation of new working practices, taking advantage of digital technologies for communication and the adaptation of existing processes so that they can be completed (at least in part) during the pandemic are also crucial elements of the research response to COVID-19, particularly for facilitating the continuation of research.

Reducing patient visits.

Clinical research is a highly regulated domain, with strict oversight on practices and procedures, and reporting requirements overseen by multiple regulators. While research setup and governance processes became more centralised, the successful conduct of research during the pandemic required a degree of flexibility and creative adaptation. The move to more remote or virtual ways of completing, supporting, regulating, and facilitating research relied on the speedy adoption of new technologies and ways of working.

On 12 th March 2020, the MHRA issued guidance to sites and investigators “regarding protocol compliance during exceptional circumstances” [ 40 ]. The guidance stated that the MHRA recognised “the difficult current situation” and advised on how to manage trials during the pandemic [ 40 ]. The MHRA also noted in this guidance and on the MHRA Inspectorate website that a redistribution of human resources during the pandemic:

may mean certain oversight duties, such as monitoring and quality assurance activities might need to be reassessed and alternative proportionate mechanisms of oversight introduced (such as phone calls, video calls) to ensure ongoing subject safety and well-being. We would advise a brief risk assessment and documentation of the impact of this [ 40 ].

While this guidance came before the formal research shutdown, it remained important, especially for the small amount of research which was allowed to continue because it was the best or only treatment option left available for patients. However, research practices and trial protocols needed to be adapted, particularly as there were restrictions on who could physically visit hospital sites, as G-5 highlights:

If a protocol says that a participant will have a visit at week 1, week 2, week 3 and week 4 and those are protocol visits–it’s unacceptable not to do those visits. They are protocol deviations. However, during the real surge of the pandemic, those visits couldn’t be done. They couldn’t come in and have an MRI scan, and ECG and bloods taken. What they did have was someone contacting them by telephone or by Skype or other formats, media format–to say, “How are you doing? Are you okay? Is there anything you need to report? Keep in touch” (G-5).

Through delaying or adapting follow-up appointment requirements so they could be completed over the telephone or through videoconferencing, many studies were able to maintain some level of continuity. For these research participating patients, other parts of the research process needed sensitive negotiation, as one PI explains in relation to changes in the format of patient consultations:

Some [participants] were actually a bit reluctant and felt a bit fobbed off to be called at home [when] they were due a face-to-face consultation. We had to be a bit careful about that, particularly if we were discontinuing treatment or discharging people from our care. That almost always went badly if we tried to do it remotely. And if we were having a really definitive conversation like that, it was worth–we found, in the end, patients coming up. Other patients were reluctant to come and readily accepted our advice that rather than coming for a CT scan, we just do a chest x-ray when we next saw them. So, there is a difference of approach, which is personal–not particular to their circumstance (R-5).

Balancing the need for face-to-face consultations and the protection offered by telephone or video consultations required thoughtful, individualised decision-making. For other studies however, digital consultation was simply not possible, which lead to investment in supporting people to attend the hospital:

A few studies have been done remotely, but the one that I have taken on, patients really have to come in. So, we had to do a lot of logistic development there, like bringing them in by car, paying for whatever is necessary just to make sure that they continue coming in (D-6).

Working from home.

Another crucial step in facilitating research and frontline care was asking large numbers of staff to complete their work from home. For some participants, working from home lead to greater productivity, but for many others it meant the blurring of home and work lives. Numerous factors impacted on participants’ experiences, from juggling work alongside home schooling and caring responsibilities, to feelings of isolation, through to more practical issues, such as having a space to work at home, having sufficient internet bandwidth and having stable access to Trust systems (see Box 1 ).

Box 1. Indicative questionnaire responses to: What, if any, challenges have you had to face working from home?.

https://doi.org/10.1371/journal.pone.0256871.t006

While research staff were transitioning to working from home, research spaces were transformed to facilitate frontline care. By April, two of the four Clinical Research Facilities (CRFs) in the Trust were repurposed to deliver frontline care and training space for frontline staff. The remaining two CRFs were refocused on supporting COVID-19 research. The vacant R&D department’s office spaces were also used by Trust staff to facilitate socially-distanced meetings and computer work for those who needed to be onsite. Careful repurposing of offices and clinical space provided the Trust with additional, flexible physical space to assist in the emergency response to the pandemic.

Digitalising research processes.

Research work still occurred within the normal parameters of how health research is conducted in the NHS. These practices were, however, done differently to adapt to COVID-19 social distancing measures.

Firstly, researchers initially had to find a workaround for consent to research in COVID-19 wards. Because of infection control protocols no materials, including paper consent forms, could be removed from COVID positive wards. As there were no protocols in place to gain consent digitally, staff developed a local workaround, as D-1 explains:

we managed to get some […] work phones so that we could take a picture of the consent [form]. So, the consent [form] was held up to the window [in the COVID ward], the team outside could take a picture of the consent form and send it directly through on the Pando app, because [Pando] could have patient details. So, it could then be turned into a PDF and printed and put in the patient file.

Another example of a slow but necessary digital solution was with site monitoring. Site monitoring allows commercial companies and other trial sponsors to visit research sites to assess the quality of the data and ensure study protocols are being followed. Despite MHRA instruction that this “should not add extra burden to trial sites” [ 40 ] and that monitors could not be justified as an extra body in the building, these activities are crucial not just for validating data but for hospitals to be able to bill sponsors for the completed research. Workarounds were further limited because of data protection regulations that prevent the digital transfer of patient data or remote access to Trust systems by external individuals. Where site monitors would usually work alone on site, it became a long and arduous process:

a member of the research team within the Trust sits at a screen and shared that screen through Microsoft Teams with the external person. So, no data is held, no recordings are being done, no data is transferred. But it’s very, very labour-intensive. (G-5)

Whilst workarounds were quickly found for some research practices, others took longer. Despite the fact that Patient and Public Involvement in research (PPI) is a core element of contemporary UK health research [ 17 ], there was initially “zero PPI” (G-1). Rather PPI group managers focused on care work: “putting them in touch with local services that could do things like pick up prescriptions for them, get shopping, get the food boxes delivered” (P-1). It was only with time that not only did researchers planning non-COVID research begin to engage more than usual with their PPI groups, but that funders and regulators demanded that PPI should still be prioritised even in emergency research [ 41 , 42 ].

While researchers voiced concerns about the equity of shifting online and assumptions about who will and will not engage with online PPI, this did not appear to be a problem in practice:

There’s often a sort of an ageism about who can–it’s like kind of what you were just saying about older people can’t do PPI. Well, bollocks. I mean actually they’ve been as responsive to this pandemic as anybody else. The rates of use of, you know, technology, has like skyrocketed in the over 65s, because of their need to talk to their grandchildren etc. So, you know, they are adaptive (R-1).

R-1’s experience was echoed by PPI representatives. Reflecting on the move online, these representatives noted some disadvantages, such as the absence of many social aspects of attending PPI meetings, and video fatigue. But participants were generally positive about the potential of virtual PPI for involving those who cannot always travel long distances due to their illnesses, those who work full-time but could attend an hour session online in their lunch break, and representatives in different countries.

In short, the process of realigning and digitalising research practices was not simply one that sped up research and productivity, but it involved a set of necessary, labour-intensive workarounds. It did, however, also bring about possibilities for long term positive effects, such as diversifying involvement in PPI groups.

COVID-19 has brought to the fore the critical importance of the UK’s clinical research infrastructure which has over the past 15 years become increasingly embedded within the NHS. It has enabled NHS hospitals to deliver research of global importance at an unprecedented pace while simultaneously providing critical care for record numbers of acutely ill patients. We provide an analysis of how this was possible through an in-depth case study of the transformations and reconfigurations of the research system at one research-intensive Trust. Our data show that a large-scale reorganisation of research staff, research infrastructures and research priorities took place during the first few weeks and months of the pandemic. We have documented many of the changes in organisational structure, national policy and everyday working practices that facilitated the Trust’s response to COVID-19. These rapid changes have brought about new ways of working, and new perspectives on the role of research which may have far reaching consequences for the future of the clinical research system in the UK.

The pandemic occasioned a large-scale mobilisation of research staff as a “reserve army.” Research staff were crucial in supporting the care-function of NHS hospitals during the first wave of the pandemic. At the same time, the embedded research system helped to streamline, facilitate and deliver rapid COVID-19 research.

Our study documented some of the challenges that the research system has faced in seeking to operate in a COVID-safe manner. At the same time, our participants described instances of improvisation in order to adapt protocols to the COVID-19 environment. Research staff developed effective practical solutions borne out of necessity, rather than the result of prior planning. This points to the resourcefulness of research staff, but also highlights the ways in which the research system was initially largely absent from existing emergency planning within the health system.

Our research was conducted while the Trust we were studying enacted national COVID-19 policy, responded to local care needs and supported clinical research during a global pandemic. This allowed us to observe these events unfolding while gathering data in a COVID-safe manner. But the pandemic created limitations as well, especially impacting the range of methods we were able to use. While working digitally did give us a first-hand experience of how a large proportion of the decision-making infrastructure had to move online, it limited our access to frontline care and everyday research activity.

There are also limitations of looking at a research active Trust like SLAT. While research is increasingly becoming a routine component of all NHS settings, SLATs size and existing research portfolio meant there was a large amount of resource available to redeploy towards COVID-19 care and research delivery. This picture may not be representative of all NHS Trusts, particularly those that are smaller, where less research takes place. Such resource, particularly in the form of biomedical research infrastructures embedded within NHS Trusts, have provided what Roope et al. label ‘option value’ in research, additional capacity to support public good, which in normal times may appear an inefficient use of resource [ 43 ]. Roope et al. highlight that, in comparison to funded, individual research studies, funding research infrastructures allows greater flexibility and speed of response when emergencies arise, such as the COVID-19 pandemic. While the research workforce, funds and infrastructures were used to support other research prior to COVID-19 (as opposed to being excess capacity), the ability of such resource to be reallocated to COVID-19 at such pace underpinned much of the UK’s success in its research response and much of the work described in this paper. It is important to acknowledge, however, that research capacity is distributed unevenly throughout the NHS, and resources such as Clinical Research Facilities and Biomedical Research Centres tend to be situated in major teaching hospitals and trauma centres rather than geographically more localised hospitals. More research is needed to understand how this unequal distribution of resources affected outcomes of care and research during the pandemic.

In documenting how the pace of research work changed dramatically during the pandemic, both in terms of increasing the speed of certain activities and decreasing the speed of others, our paper also contributes to broader discussions of pace in clinical research. In particular, the key question—how do we most effectively streamline the research pipeline, from bench to bedside? Hanney et al. highlight the potential to overlap parts of the translational research pathway to speed up the process, and some of the barriers to this, such as ethical approvals and resourcing issues [ 30 , 31 ]. Many of these issues were removed during the pandemic because of the targeting of resources towards COVID-19 research. On a more practical level, however, our analysis suggests some ways that the research system may be adapted in the future. The potential offered by digital communications to facilitate certain research and PPI activities have led some clinical researchers to question the necessity for research participants and patients to always attend hospital sites for consultations. Trust-level research prioritisation has proved positive in managing finite local resources as effectively as possible, enabling a more holistic view of the research portfolio at a local level as well as take into account national priorities. At the same time, it is clear that the new technologies and new ways of working that were developed to cope with the crisis are not automatically more efficient, and there is a danger that some key steps such as adequate PPI might be overlooked when research pace is increased. Further research and planning will be needed to develop suitable governance processes to facilitate research activities both when on a crisis footing, and in more routine practice. Wider investment in networked digital applications and hardware (such as Trust compliant laptop computers) is needed to facilitate better working from home.

Our study suggests a number of additional lessons for future national emergency planning and policy. Research infrastructure must be better included in advanced planning, both in terms of the personnel, equipment and other resources that can be made available for redeployment as well as the direct impact that research can make. The capacity to develop new treatments and vaccines should be treated as a strategic asset that is a central part of any emergency response. This has been recognised at the national level, and internationally [ 1 – 3 ], but our data suggest that it has not fully translated into Trust-level operations. Planning for future emergencies should include protocols for the rapid establishment of strategic research prioritisation and redeployment of research infrastructure and capacity. Our data also show that throughout the pandemic, there remained a demand for public input in research, which should be included in future emergency planning. Public input is vital in clinical research, especially in an emergency response which requires publics to respond to clinical-expert advice, and planners should recognise it as such.

Future emergency planning must, however, take into account the exhaustion and stress faced by research staff who suddenly found themselves on the front line of a national mobilisation. Research staff experienced the same well-documented stresses experienced by other NHS workers [ 33 , 44 ]. Emergency planning should acknowledge this human cost and find ways to mitigate such costs and provide support for staff as a national priority.

At a global level, the UK response and its specific organisation, as described within this case study Trust, demonstrates some of the benefits of embedding research infrastructures within a national health provider, and how this set up not only enabled a coherent national response, but also provided staff resource to facilitate such research at great speed as well as support the delivery of frontline care. As we look to the future, how we integrate healthcare and research at more national and global levels are important areas for further research and discussion.

Acknowledgments

We are grateful to Christopher McKevitt and Nina Fudge for providing astute comments on drafts of this paper and to our participants who shared their experiences and time with us during this period of unprecedented strain on the NHS.

• View Article
• PubMed/NCBI
• Google Scholar
• 5. National Institute for Health Research. Urgent Public Health COVID-19 Studies 2021 [27/07/2021]. Available from: https://www.nihr.ac.uk/covid-studies/ .
• 6. National Institute for Health Research. DHSC issues guidance on the impact of COVID-19 on research funded or supported by NIHR 2020 [01/05/2020]. Available from: https://www.nihr.ac.uk/news/dhsc-issues-guidance-on-the-impact-on-covid-19-on-research-funded-or-supported-by-nihr/24469 .
• 13. Yin R K. Case study research: Design and methods. 3rd Edition ed. London: Sage; 2003.
• 14. Stake RE. The art of case study research. London: Sage; 1995.
• 16. Quinn Patton M. Qualitative research and evaluation methods. 3rd Edition ed. London: Sage; 2002.
• 17. National Institute for Health Research. National Standards for Public Involvement in Research. 2019.
• 25. National Institute for Health Research. Urgent Public Health Designation Guidance Notes 2020 [03/03/2021]. Available from: https://www.nihr.ac.uk/documents/urgent-public-health-designation-guidance-notes/24992 .
• 26. National Institute for Health Research. Prioritised support for urgent COVID-19 research 2020 [03/03/2021]. Available from: https://www.nihr.ac.uk/covid-19/prioritised-support-for-urgent-covid-19-research.htm .
• 27. Pharmaphorum. Coordinating and delivering research in the pandemic: the UK approach 2021 [01/03/2021]. Available from: https://pharmaphorum.com/webinars/uk-covid-coronavirus-research-nihr/ .
• 28. Sharma S. In the meantime: Temporality and cultural politics. Durham: Duke University Press; 2014.
• 32. Baraitser L. Enduring time. London: Bloomsbury Publishing; 2017.
• 40. MHRA Inspectorate. Advice for Management of Clinical trials in relation to Coronavirus 2020 [03/03/2021]. Available from: https://mhrainspectorate.blog.gov.uk/2020/03/12/advice-for-management-of-clinical-trials-in-relation-to-coronavirus/ .
• 41. National Institute for Health Research. NIHR reaffirms its support for patient and public involvement, engagement and participation during the COVID-19 pandemic 2020 [11/03/2021]. Available from: https://www.nihr.ac.uk/news/nihr-reaffirms-its-support-for-patient-and-public-involvement-engagement-and-participation-during-the-covid-19-pandemic/24641 .
• 42. Health Research Authority. Public involvement in a pandemic: lessons from the UK COVID-19 public involvement matching service. 2021.

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Public health impact...

Public health impact of covid-19 vaccines in the US: observational study

Linked editorial.

The benefits of large scale covid-19 vaccination

• Related content
• Peer review
• Amitabh Bipin Suthar , epidemiologist ,
• Jing Wang , epidemiologist ,
• Victoria Seffren , epidemiologist ,
• Ryan E Wiegand , statistician ,
• Sean Griffing , epidemiologist ,
• Elizabeth Zell , statistician
• Coronavirus Disease (COVID-19) Response, Centers for Disease Control and Prevention, Atlanta, GA, USA
• Correspondence to: A B Suthar icf4{at}cdc.gov (or @AmitabhSuthar on Twitter)
• Accepted 11 March 2022

Objective To evaluate the impact of vaccine scale-up on population level covid-19 mortality and incidence in the United States.

Design Observational study.

Setting US county level case surveillance and vaccine administration data reported from 14 December 2020 to 18 December 2021.

Participants Residents of 2558 counties from 48 US states.

Main outcome measures The primary outcome was county covid-19 mortality rates (deaths/100 000 population/county week). The secondary outcome was incidence of covid-19 (cases/100 000 population/county week). Incidence rate ratios were used to compare rates across vaccination coverage levels. The impact of a 10% improvement in county vaccination coverage (defined as at least one dose of a covid-19 vaccine among adults ≥18 years of age) was estimated During the eras of alpha and delta variant predominance, the impact of very low (0-9%), low (10-39%), medium (40-69%), and high (≥70%) vaccination coverage levels was compared.

Results In total, 30 643 878 cases of covid-19 and 439 682 deaths associated with covid-19 occurred over 132 791 county weeks. A 10% improvement in vaccination coverage was associated with an 8% (95% confidence interval 8% to 9%) reduction in mortality rates and a 7% (6% to 8%) reduction in incidence. Higher vaccination coverage levels were associated with reduced mortality and incidence rates during the eras of alpha and delta variant predominance.

Conclusions Higher vaccination coverage was associated with lower rates of population level covid-19 mortality and incidence in the US.

Introduction

As of 11 April 2022, 497 960 492 cases covid-19 and 6 181 850 covid-19 related deaths had been reported globally, and 80 260 092 covid-19 cases and 983 237 covid-19 related deaths had been reported in the United States. 1 2 The US death toll recently surpassed the 1918 Spanish flu as the deadliest pandemic in recent history. 3 In addition to covid-19 related deaths, the pandemic has also had indirect effects on other health conditions. These effects are captured in excess mortality and reduced life expectancy estimates. Domestically, life expectancy decreased by 1.5 years from 2019 to 2020, representing the largest reduction since the second world war. 4

Messenger RNA (mRNA) covid-19 vaccines developed by Pfizer-BioNTech and Moderna and an adenovirus covid-19 vaccine developed by Johnson & Johnson have become valuable tools to combat this pandemic. Clinical trials evaluating efficacy against symptomatic infection found that the Pfizer-BioNTech vaccine was 95.0% effective, the Moderna vaccine was 94.1% effective, and the Janssen vaccine (Johnson & Johnson) was 66.3% effective. 5 6 7 The US Food and Drug Administration (FDA) granted emergency use authorization for mRNA vaccines in December 2020 and the Janssen vaccine in February 2021. FDA approval for the Pfizer and Moderna vaccines was granted in August 2021 and January 2022, respectively. 8 Emergency use authorization was further granted to additional doses of the mRNA vaccines for certain populations. 8 As of 11 April 2022, nearly 570 million vaccine doses have been administered in the US and more than 11 billion vaccine doses have been administered globally. 1 2 The World Health Organization’s target is to vaccinate 70% of the world’s population by mid-2022. 9

Across countries, the real world effectiveness of the covid-19 vaccines has largely been consistent with estimates of efficacy observed in clinical trials. 10 11 12 In addition to the individual level effect on disease risk and progression, vaccines may also have secondary benefits of slowing spread and reducing onward transmission and its associated morbidity and mortality. 13 Population level data and analyses have been limited. 14 15 We aimed to estimate how increasing county coverage of vaccines affected population level mortality and incidence of covid-19.

Study design

Our observational study of the US population used national, county level surveillance data. In the US, counties are a geographic administrative unit below states and territories and include the nation’s capital, Washington DC. The US Centers for Disease Control and Prevention (CDC) receives surveillance data from 3224 US counties (or county equivalents). We included and analyzed county covid-19 cases, deaths, and vaccinations reported to the CDC. We tracked mortality as our primary outcome and incidence (using reported probable and confirmed covid-19 cases) as our secondary outcome. We calculated county level incidence by standardizing reported county cases and deaths per 100 000 population over a week. 2

Study definitions

We defined a case as one that met the Council of State and Territorial Epidemiologists’ surveillance case definitions as confirmed or probable covid-19 and a death as those that were related to covid-19, as determined or reported by jurisdictions. 16 17 Each vaccine dose administered was attributed to the county in which the person resided. 18 We defined the county vaccination coverage as the number of people aged ≥18 years who received at least one dose of covid-19 vaccine among the total number of people aged ≥18 years old residing in that county. 2

Data sources

For case and death counts disaggregated by county and week, we used the CDC’s managed case surveillance dataset, which includes the most recent numbers reported by states, territories, and other jurisdictions. This dataset is populated by routine reporting from jurisdictions to the CDC. 16 To document new cases, jurisdictions may use the date that a case was reported to the health department, a person took a covid-19 test, a laboratory confirmed a covid-19 test as positive, or a person was diagnosed as having covid-19 by a clinician. For death reporting, jurisdictions may use the date when the death was reported to the health department or the date of covid-19 associated death. 2 We retrieved counts of covid-19 vaccine doses administered by week and county from the CDC’s managed vaccine dataset. This dataset includes covid-19 vaccination data (including the date of vaccine administration, the number of doses administered, and county of residence, among other variables) reported to the CDC by jurisdictions, pharmacies, and federal entities through Immunization Information Systems, the Vaccine Administration Management System, or direct submission of vaccination records. 19 The population data, used for denominators to measure vaccination coverage, came from the vintage 2019 US population estimates. 20

Inclusion criteria

We included case surveillance and vaccine administration data from 14 December 2020 to 18 December 2021. We included people at least 18 years of age with a valid county of residence in one of the states or territories who received at least one covid-19 vaccination. Given that population benefits may extend beyond the primary vaccine recipient, we included case and mortality data across all ages. Data completeness was an inclusion criterion for analysis. We used a 70% threshold for data completeness of reporting county of residence across all data sources. Specifically, we excluded a jurisdiction if more than 30% of the case, death, and/or vaccination data for the jurisdiction were contributed by unspecified or unknown counties of residence. We excluded Texas and Hawaii because vaccination data were unavailable at county level. We excluded county equivalents in territories except for Puerto Rico and Guam, either because the county level population data of adults ≥18 years old were unavailable (US Virgin Islands) or because the county equivalent vaccination data were unavailable (all other territories). In addition, we excluded eight counties in California with a population of fewer than 20 000 people, as California does not report the vaccination data of counties with under 20 000 people. We excluded the Kusilvak Census Area in Alaska owing to unavailable vaccination data and the Valdez-Cordova Census Area in Alaska because the case and mortality data were unavailable. We excluded the District of Columbia, villages in Guam, and municipalities in Puerto Rico because of a lack of mobility data. Finally, we excluded Rio Arriba County in New Mexico because the social vulnerability index was missing. In addition, we excluded any county week missing covariate information used in regression models.

Data analysis

County of residence case and first dose covid-19 vaccination data were aggregated by Morbidity and Mortality Weekly Report (MMWR) week beginning with MMWR week 2020-51 (13-19 December 2020) and ending with MMWR week 2021-50 (12-18 December 2021). 21 The CDC and Agency for Toxic Substances and Disease Registry’s social vulnerability index encompasses socioeconomic status (that is, poverty rates, unemployment rates, income levels, and education levels), household composition and disability (that is, ages, disability, and single parent households), minority status, language capability, and housing type and transportation (that is, multi-unit structures, mobile homes, crowding levels, vehicle ownership, and group housing) into a single measure. 22 Google’s community mobility reports help to measure changes in community mobility related to covid-19. 23 To prevent confounding related to the social vulnerability and mobility of communities, we included these variables in the model. 24 25

We used generalized linear mixed models assuming a negative binomial outcome distribution to assess associations between vaccination coverage and rates of deaths and cases by using continuous estimates. 26 We used a first order autoregressive correlation structure to account for multiple observations per county and for potential autocorrelation. We included county level population as an offset and included social vulnerability index categorized into quarters and retail and work mobility data as covariates. To account for cases occurring during the period of developing immunity, a county remained in the lower vaccination category for two weeks before moving to the next vaccination category.

We calculated estimates during the period of alpha variant predominance and the period of delta variant predominance (starting when the national prevalence of delta was estimated to be at least 50%—that is, the week of 20 June 2021 onward) categorically. 27 28 We compared four different categories for county vaccination coverage: very low (0-9% of the county had been vaccinated), low (10-39% of the county had been vaccinated), medium (40-69% of the county had been vaccinated), and high (≥70% of the county had been vaccinated) during the era of alpha variant predominance. As with the continuous analyses, to account for cases occurring during the period of developing immunity, a county remained in the lower vaccination category for two weeks before moving to the next vaccination category. Moreover, we included county level population as an offset and included social vulnerability index categorized by quarter and retail and work mobility data as covariates. Given the inadequate number of county weeks accrued with very low and low vaccination coverage, we compared the mortality and incidence rates for medium and high coverage during the era of delta variant predominance.

Sensitivity analyses

We did three sensitivity analyses with the continuous analyses. The first sensitivity analysis was to compare definitions of vaccination being at least one dose with including only fully vaccinated people (that is, at least two mRNA doses or a single adenovirus dose). The second was to compare use of a stringency level for data completeness of 70% and 90%. The third was to compare estimates with and without the two week lag period.

Patient and public involvement

We used routinely generated covid-19 vaccine and case surveillance data. No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for design or implementation of the study. No patients were asked to advise on interpretation or writing up of results.

First year of vaccine roll-out

We included data from 2558 counties in 48 US states ( fig 1 ). In total, we observed 30 643 878 cases of covid-19 and 439 682 covid-19 related deaths over 132 791 county weeks ( table 1 ). Every 10% improvement in vaccination coverage was associated with an 8% (95% confidence interval 8% to 9%) reduction in mortality rates ( fig 2 ) and with a 7% (6% to 8% reduction in case incidence ( fig 2 ).

County scale-up of vaccinations from 13 December 2020 to 18 December 2021. Light purple lines represent individual counties; dark purple line is aggregated estimate

• Download figure
• Open in new tab
• Download powerpoint

Characteristics of included counties. Values are median (range) unless stated otherwise

• View inline

Effect of vaccination coverage on county covid-19 related mortality (top) and incidence (bottom) during first year of vaccine roll-out. Analyses are from 2558 counties in 48 US jurisdictions. Model controlled for county population size, social vulnerability index, and mobility changes

Era of alpha variant predominance

In total, we observed 15 493 299 cases of covid-19 and 263 873 covid-19 related deaths over 70 189 county weeks during the era of alpha variant predominance. Compared with very low coverage, low (incidence rate ratio 0.40, 95% confidence interval 0.39 to 0.42), medium (0.25, 0.23 to 0.26), and high (0.19, 0.16 to 0.22) vaccination coverage categories had lower rates of mortality ( fig 3 ). Compared with very low coverage, low (incidence rate ratio 0.43, 0.41 to 0.44), medium (0.30, 0.29 to 0.32), and high (0.20, 0.18 to 0.22) vaccination coverage categories had lower incidence rates ( fig 3 ).

Effect of vaccination coverage on county covid-19 mortality and incidence during era of alpha variant predominance. Analyses are from 2557 counties in 48 US jurisdictions. Model controlled for county population size, social vulnerability index, and mobility changes

Era of delta variant predominance

In total, we observed 15 150 579 cases of covid-19 and 175 809 covid-19 related deaths over 62 602 county weeks during the era of delta variant predominance. When comparing high and medium coverage, we observed similar mortality effects during the era of alpha variant predominance (incidence rate ratio 0.77, 0.66 to 0.90) and the era of delta variant predominance (0.75, 0.71 to 0.79). When comparing high and medium coverage, we observed smaller incidence effect sizes during the era of delta variant predominance (incidence rate ratio 0.90, 0.87 to 0.94) compared with the era of alpha variant predominance (0.66, 0.60 to 0.73).

We observed sustained reductions in county mortality and incidence rates when we included only fully vaccinated people in the vaccination coverage categories, when we increased our data stringency level, and when we removed the two week immunity lag period ( fig 4 ).

Sensitivity analyses of including only fully vaccinated people, increasing data stringency requirements, and removing two week immunity lag period. *In baseline group, vaccination coverage refers to coverage of at least one dose of vaccine, 2558 counties and 48 US jurisdictions included had ≥70% completeness rates of reporting county of residence, and study period was 14 December 2020 to 18 December 2021. †Vaccination coverage refers to coverage of fully vaccinated people. ‡2164 counties and 42 US jurisdictions included had ≥90% completeness rates of reporting county of residence. §Two week immunity period was removed

Using data from 2558 counties—representing nearly 300 million people and 80% of the US population—we found that increasing the vaccination coverage in counties was associated with a reduced incidence of covid-19 related mortality and cases. We observed decreasing trends in mortality and case incidence associated with higher levels of vaccination coverage across the eras of both alpha and delta variant predominance. This effect was robust to various sensitivity analyses, which improves prediction and confidence in these findings.

Covid-19 associated mortality remains one of the most important clinical outcomes to guide public health decision making, measure pandemic severity, and evaluate mitigation efforts. It was our primary outcome. In the US, death registration rates, and cause of death ascertainment, remain high. This suggests that US mortality surveillance systems have been, and will continue to be, useful for covid-19 mortality surveillance. Previous vaccine studies have shown survival benefits at the individual level. 29 We observed that these benefits may extend to the population level; counties with high coverage had a greater than 80% reduction in mortality rates compared with largely unvaccinated counties. Given that infection fatality rates for covid-19 increase with age, counties with a higher proportion of older people may have more covid-19 related mortality and stand to benefit from high coverage of covid-19 vaccines. 30

We used reported cases as a proxy for incidence for our secondary outcome. Although reliable, available across jurisdictions, and reported continuously, reported cases may not reflect true transmission rates because of variation in when people seek out testing. 31 For example, people without symptoms may not actively seek out testing of their own accord but may be important to test for gauging disease transmission. Owing to more recent reopening requirements for workplaces, restaurants, entertainment venues, schools, and outgoing international air travel, more people without symptoms may be seeking out testing. 32 These requirements, and their uptake, may vary across states and counties. Nevertheless, the reduction in incidence observed with increasing vaccination coverage is consistent with surveillance data from other countries that have achieved high vaccination coverage and emerging evidence on transmission from contact tracing programs. 1 33

Comparison with other studies

Increasing vaccination coverage may play a role in mitigating the effects of the delta and omicron variants and reduce the emergence of future variants. 34 35 By 27 June 2021 the delta variant made up more than 50% of circulating variants in the US, increasing to almost 100% by 21 September 2021. 2 More recently, the omicron variant was first reported on 1 December 2021 and comprised 95% of circulating variants by 2 January 2022. 2 The delta variant had increased transmissibility and possible increased virulence compared with earlier SARS-CoV-2 strains. 36 37 In our study, by the time the delta variant predominated, counties with lower levels of vaccination coverage (that is, 0-39%) were rare. Nevertheless, our findings of continued population level protection against death and reductions in population level protection against infection during the period of delta variant predominance seem to be consistent with clinical literature on vaccine effectiveness. 29 38 39 40 Additional studies aimed at assessing the population impact of vaccines during the period of delta variant predominance merit consideration for validating our observations. Although our study period did not include the period of omicron variant predominance, data suggesting reduced vaccine effectiveness and the importance of staying up to date on covid-19 vaccinations are emerging and may lead to changes in population level vaccine impact that merit exploration. 41 42 43 Continuing to monitor the delta and omicron variants, and the emergence of other variants of interest, is critical and will require ongoing genomic surveillance.

Clinical studies indicate that a single dose of an mRNA vaccine provides a lower level of protection than two doses. 44 Furthermore, two mRNA doses seem to be more effective than a single adenovirus dose against symptomatic infection. 5 6 7 We defined people with at least one dose of vaccine as being vaccinated for the purposes of vaccination coverage. Given that our study design used population surveillance data, changing our coverage definition to include only fully vaccinated people would place people with a single dose of mRNA vaccine in our referent, the very low coverage category. This may introduce bias in the incidence and mortality estimates. When we changed our definition of vaccination coverage to being fully dosed during sensitivity analysis, we did not find increased effect sizes, as would be expected from clinical studies. 44 Ongoing vaccine studies continue to evaluate the comparative effectiveness of vaccines by manufacturer. 10

Given that only people aged 18 and older were eligible for vaccination across vaccines during most of our study period, we used this age threshold to define vaccination coverage. Pediatric studies will be a welcome contribution to understanding the effects of vaccines on younger age groups, when feasible. As of September 2021 the FDA began recommending a third dose for specific populations. 8 Owing to waning immunity and variant induced changes in vaccine effectiveness, additional doses may be needed in specific populations and scenarios. Further studies may benefit from evaluating the population impact of vaccination coverage by using different definitions and eligibility scenarios.

Limitations of study

Several limitations should be considered when interpreting these data. We chose vaccination coverage thresholds based on programmatic experience; exploring coverage thresholds above 70% may be worth examining in future research once more counties have achieved these levels for extended periods of time. We excluded some jurisdictions because they did not have county level information on immunizations, cases, and deaths for at least 70% of their counties. Additional markers of disease severity, such as hospital admissions, were not explored in this study owing to possible differences in ascertainment and reporting coverage across jurisdictions. Given the limited number of variables that were known to affect mortality and incidence, collected at the county level, and available on a weekly basis, we did not control for masking, physical distancing, or other similar potential confounding variables in this study. Furthermore, given the limited number of county weeks, we lacked power to stratify by time periods and cannot rule out the possibility of temporal confounding. Finally, given that we used aggregate case surveillance data to have the most complete case and death data available, other characteristics of cases, such as demographics and comorbidities, were not available. States, territories, and jurisdictions adapt national guidance on which date to use for case reporting. 17 In this study we collated county data across these geographic areas. A time difference may be present depending on which date a health department uses; however, this is unlikely to be substantial enough to affect which week a case or death occurs. Naturally acquired immunity resulting from SARS-CoV-2 infection may have affected the reduced case incidence observed during the study period. These are limitations of our study. Nevertheless, reductions in incidence and death observed in emerging US data using alternative data sources and study designs give us confidence in the directionality and magnitude of our estimates. 45 46

Conclusions and policy implications

In addition to individual level benefits, we observed that vaccines protect communities against severe disease and infection. Higher coverage of vaccines seemed to confer greater levels of community benefits. Given that community benefits are rooted in individual benefits, for which vaccine effectiveness has been established in countries around the world, these data may be generalizable to other countries. Future research may benefit from evaluating macroeconomic effects of improving population health, such as changes in employment rates and gross domestic product resulting from reopening society. Vaccines should be deployed strategically with public health and social measures based on ongoing levels of transmission.

What is already known on this topic

The public health impact of scaling up covid-19 vaccination remains largely uncharacterized

What this study adds

Higher vaccination coverage was associated with lower rates of population level incidence of covid-19 and mortality related to covid-19

This community level benefit complements the large body of evidence indicating individual level benefits of covid-19 vaccination

Ethics statements

Ethical approval.

Not applicable.

Data availability statement

Centers for Disease Control and Prevention covid-19 vaccination, case, and death data are available at data.cdc.gov .

Acknowledgments

JW is an Oakridge Institute for Science and Education fellow and SG is a lieutenant with the United States Public Health Service. The findings and conclusions in this article are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention. Use of trade names and commercial sources is for identification only and does not imply endorsement by the US Department of Health and Human Services.

Contributors: ABS, JW, VS, REW, SG, and EZ conceived and designed the study. JW, VS, REW, and EZ analyzed the data. ABS, JW, VS, REW, SG, and EZ wrote the manuscript. ABS, JW, VS, REW, SG, and EZ agree with manuscript’s results and conclusions. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. ABS is the guarantor.

Funding: None.

Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: no support from any organization for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

The lead author (the manuscript’s guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained

Dissemination to participants and related patient and public communities: This publication will be shared on appropriate websites and social media platforms and at meetings.

Provenance and peer review: Not commissioned; externally peer reviewed.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/ .

• ↵ World Health Organization. WHO Coronavirus (COVID-19) Dashboard. https://covid19.who.int/ .
• ↵ Centers for Disease Control and Prevention. COVID Data Tracker. https://covid.cdc.gov/covid-data-tracker/#datatracker-home .
• ↵ Centers for Disease Control and Prevention. 1918 Pandemic (H1N1 virus). 2019. https://www.cdc.gov/flu/pandemic-resources/1918-pandemic-h1n1.html .
• ↵ Arias E, Tejada-Vera B, Ahmad F, Kochanek KD. Provisional Life Expectancy Estimates for 2020. Vital Statistics Rapid Release. Report No. 015. 2021. https://stacks.cdc.gov/view/cdc/107201 .
• Polack FP ,
• Thomas SJ ,
• Kitchin N ,
• C4591001 Clinical Trial Group
• El Sahly HM ,
• COVE Study Group
• Vandebosch A ,
• ENSEMBLE Study Group
• ↵ US Food and Drug Administration. COVID-19 Vaccines. https://www.fda.gov/emergency-preparedness-and-response/coronavirus-disease-2019-covid-19/covid-19-vaccines .
• ↵ World Health Organization. Strategy to Achieve Global Covid-19 Vaccination by mid-2022. https://cdn.who.int/media/docs/default-source/immunization/covid-19/strategy-to-achieve-global-covid-19-vaccination-by-mid-2022.pdf .
• Tenforde MW ,
• Rhoads JP ,
• IVY Network
• Foulkes S ,
• SIREN Study Group
• Angulo FJ ,
• McLaughlin JM ,
• ↵ COVID-19 Scenario Modeling Hub Team. COVID-19 Scenario Modeling Hub. https://covid19scenariomodelinghub.org/viz.html .
• McNamara LA ,
• Wiegand RE ,
• ↵ Centers for Disease Control and Prevention. About CDC COVID-19 Data. 2021. https://www.cdc.gov/coronavirus/2019-ncov/covid-data/about-us-cases-deaths.html .
• ↵ Council of State and Territorial Epidemiologists. Update to the standardized surveillance case definition and national notification for 2019 novel coronavirus disease (COVID-19). 2021. https://cdn.ymaws.com/www.cste.org/resource/resmgr/21-ID-01_COVID-19_updated_Au.pdf .
• ↵ Centers for Disease Control and Prevention. Reporting COVID-19 Vaccinations in the United States. https://www.cdc.gov/coronavirus/2019-ncov/vaccines/reporting-vaccinations.html .
• ↵ Centers for Disease Control and Prevention. About COVID-19 Vaccine Delivered and Administration Data. https://www.cdc.gov/coronavirus/2019-ncov/vaccines/distributing/about-vaccine-data.html .
• ↵ United States Census Bureau. Datasets. https://www2.census.gov/programs-surveys/popest/datasets/2010-2019/ .
• ↵ Centers for Disease Control and Prevention. MMWR weeks ending log 2020-2021. https://stacks.cdc.gov/view/cdc/84011/ .
• ↵ Centers for Disease Control and Prevention, Agency for Toxic Substances and Disease Registry. CDC SVI 2018 Documentation – 1/31/2020. https://www.atsdr.cdc.gov/placeandhealth/svi/documentation/SVI_documentation_2018.html .
• ↵ Google. COVID-19 Community Mobility Reports. https://www.google.com/covid19/mobility/ .
• Wellenius GA ,
• Vispute S ,
• Espinosa V ,
• France AM ,
• Lambrou AS ,
• Steele MK ,
• Strain Surveillance and Emerging Variants Bioinformatic Working Group ,
• Strain Surveillance and Emerging Variants NS3 Working Group
• Scobie HM ,
• Johnson AG ,
• Suthar AB ,
• Hanage WP ,
• Owusu-Boaitey N ,
• Cochran KB ,
• Meyerowitz-Katz G
• Schubert S ,
• Couture A ,
• ↵ Centers for Disease Control and Prevention. Community, Work, and School: Information for Where You Live, Work, Learn, and Play. 2021. https://www.cdc.gov/coronavirus/2019-ncov/community/index.html .
• Christie A ,
• Brooks JT ,
• Sauber-Schatz EK ,
• Honein MA ,
• CDC COVID-19 Response Team
• Walensky RP ,
• Twohig KA ,
• COVID-19 Genomics UK (COG-UK) consortium
• Rosenberg ES ,
• Holtgrave DR ,
• Dorabawila V ,
• Fowlkes A ,
• Gaglani M ,
• Groover K ,
• Thiese MS ,
• Ellingson K ,
• HEROES-RECOVER Cohorts
• Lopez Bernal J ,
• Andrews N ,
• Ackerson BK ,
• Lauring AS ,
• Chappell JD ,
• Influenza and Other Viruses in the Acutely Ill (IVY) Network
• Pilishvili T ,
• Fleming-Dutra KE ,
• Farrar JL ,
• Vaccine Effectiveness Among Healthcare Personnel Study Team
• Clemens JD ,
• Zhang Z-F ,
• Swerdlow DL ,

• Browse All Articles
• Newsletter Sign-Up

• 12 Dec 2023
• Research & Ideas

• 15 Aug 2023
• Cold Call Podcast

• 10 Feb 2023

• 01 Feb 2023
• What Do You Think?

• 12 Oct 2022

• 04 Oct 2022

• 06 Sep 2022

• 21 Jul 2022

• 12 Jul 2022

• 09 Jun 2022

• 14 Apr 2022

• 14 Mar 2022

• 24 Feb 2022

• 10 Feb 2022

• 08 Feb 2022

• 20 Jan 2022

• 11 Jan 2022

• 02 Nov 2021

• 22 Oct 2021
• 19 Oct 2021

The Viability of Supply Chains with Interpretable Learning Systems: The Case of COVID-19 Vaccine Deliveries

• ORIGINAL RESEARCH
• Published: 27 September 2023
• Volume 24 , pages 633–657, ( 2023 )

Cite this article

• Samia Zaoui 1 ,
• Clovis Foguem 2 , 3 ,
• Dieudonné Tchuente   ORCID: orcid.org/0000-0002-6752-4269 4 ,
• Samuel Fosso-Wamba 4 &
• Bernard Kamsu-Foguem 1  

369 Accesses

9 Citations

Explore all metrics

The main objective of this research was to examine the instrumental role played by interpretable learning systems, specifically artificial intelligence (AI) technologies, in enhancing supply chain viability and resilience. It seeks to contribute to our understanding of the critical role played by interpretable learning systems in supporting decision-making during emergencies and crises. The research employs an empirical approach to address the research gaps in the application and impact of interpretable learning systems in supply chain management by utilizing the case of COVID-19 vaccine deliveries in France as a descriptive study. The findings highlight the ability to develop a learning system that adeptly predicts vaccine deliveries and vaccination rates. It emphasizes the importance of interpretable learning systems in optimizing supply chain management, navigating the complex landscape of vaccine distribution, establishing effective prioritization strategies, and maximizing the efficient utilization of resources.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Leveraging electronic logistics management information systems to enhance and optimize supply chain response during public health emergencies: lessons from COVID-19 response in Uganda

Formative Learning Design Within Project Evaluation: Case of a Food Bank Disaster Planning and Recovery Tool

Creating supply chain resilience during and post-COVID-19 outbreak: the organizational ambidexterity perspective

Abdulsalam, Y., Gopalakrishnan, M., Maltz, A., & Schneller, E. (2015). Health care matters: supply chains in and of the health sector. Journal of Business Logistics, 36 (4), 335–339. https://doi.org/10.1111/jbl.12111

Article   Google Scholar  

Al Qundus, J., Gupta, S., Abusaimeh, H., Peikert, S., & Paschke, A. (2023). Prescriptive analytics-based SIRM model for predicting Covid-19 outbreak. Global Journal of Flexible Systems Management, 24 (2), 235–246. https://doi.org/10.1007/s40171-023-00337-0

Alam, S. T., Ahmed, S., Ali, S. M., Sarker, S., & Kabir, G. (2021). Challenges to COVID-19 vaccine supply chain: Implications for sustainable development goals. International Journal of Production Economics, 239 , 108193. https://doi.org/10.1016/j.ijpe.2021.108193

Aldrighetti, R., Zennaro, I., Finco, S., & Battini, D. (2019). Healthcare supply chain simulation with disruption considerations: A case study from Northern Italy. Global Journal of Flexible Systems Management, 20 (1), 81–102. https://doi.org/10.1007/s40171-019-00223-8

Baryannis, G., Dani, S., & Antoniou, G. (2019). Predicting supply chain risks using machine learning: The trade-off between performance and interpretability. Future Generation Computer Systems, 101 , 993–1004. https://doi.org/10.1016/j.future.2019.07.059

Bode, C., & Wagner, S. M. (2015). Structural drivers of upstream supply chain complexity and the frequency of supply chain disruptions. Journal of Operations Management, 36 , 215–228. https://doi.org/10.1016/j.jom.2014.12.004

Bozarth, C. C., Warsing, D. P., Flynn, B. B., & Flynn, E. J. (2009). The impact of supply chain complexity on manufacturing plant performance. Journal of Operations Management, 27 (1), 78–93. https://doi.org/10.1016/j.jom.2008.07.003

Chakraborty, S. (2019). Financial deepening. Arthaniti: Journal of Economic Theory and Practice, 18 (2), 111–137. https://doi.org/10.1177/0976747918814031

Chanal, D., Steiner, N. Y., Petrone, R., Chamagne, D., & Péra, M. C. (2022). Online diagnosis of PEM fuel cell by fuzzy C-means clustering. In L. F. Cabeza (Ed.), Encyclopedia of Energy Storage (pp. 359–393). Oxford: Elsevier. https://doi.org/10.1016/B978-0-12-819723-300099-8

Chapter   Google Scholar  

Chatterjee, S., Chakraborty, S., Fulk, H. K., & Sarker, S. (2021). Building a compassionate workplace using information technology: Considerations for information systems research. International Journal of Information Management, 56 , 102261.

Choi, T.-M., Chan, H. K., & Yue, X. (2017). Recent development in big data analytics for business operations and risk management. IEEE Transactions on Cybernetics, 47 (1), 81–92. https://doi.org/10.1109/TCYB.2015.2507599

Chowdhury, M. M. H., & Quaddus, M. (2017). Supply chain resilience: Conceptualization and scale development using dynamic capability theory. International Journal of Production Economics, 188 , 185–204.

Christoph M. (2020). Interpretable machine learning', Lulu.com.

Chu, D. K., Akl, E. A., Duda, S., Solo, K., Yaacoub, S., Schünemann, H. J., & Reinap, M. (2020). Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-CoV-2 and COVID-19: A systematic review and meta-analysis. Lancet (London, England), 395 (10242), 1973–1987. https://doi.org/10.1016/S0140-6736(20)31142-9

Dey, A. (2016). ‘Machine learning algorithms: A review’, International Journal of Computer Science and Information Technologies , 7(3), pp. 1174–1179. Available at: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85054880698&partnerID=40&md5=4b34e902c004fb6bd9744bfbebba794f

Dhakate, N., & Joshi, R. (2020). Analysing process of organ donation and transplantation services in India at hospital level: SAP-LAP model’. Global Journal of Flexible Systems Management . https://doi.org/10.1007/s40171-020-00251-9

Dubey, R. (2022). Unleashing the potential of digital technologies in emergency supply chain: the moderating effect of crisis leadership. Industrial Management & Data Systems . https://doi.org/10.1108/IMDS-05-2022-0307

Dubey, R. (2023). Unleashing the potential of digital technologies in emergency supply chain: The moderating effect of crisis leadership. Industrial Management & Data Systems, 123 (1), 112–132. https://doi.org/10.1108/IMDS-05-2022-0307

Dubey, R., Bryde, D. J., Dwivedi, Y. K., Graham, G., & Foropon, C. (2022). Impact of artificial intelligence-driven big data analytics culture on agility and resilience in humanitarian supply chain: A practice-based view. International Journal of Production Economics . https://doi.org/10.1016/j.ijpe.2022.108618

Er Kara, M., Ghadge, A., & Bititci, U. (2020). Modelling the impact of climate change risk on supply chain performance. International Journal of Production Research . https://doi.org/10.2139/ssrn.3652664

Fan, J., Xue, L., & Yao, J. (2017). Sufficient forecasting using factor models. Journal of Econometrics, 201 (2), 292–306. https://doi.org/10.1016/j.jeconom.2017.08.009

Gilani, H., & Sahebi, H. (2022). A data-driven robust optimization model by cutting hyperplanes on vaccine access uncertainty in COVID-19 vaccine supply chain. Omega, 110 , 102637. https://doi.org/10.1016/j.omega.2022.102637

Gouv, F. (2022). ' French COVID-19 vaccine deliveries datasets'. Available at: https://www.data.gouv.fr/fr/datasets/donnees-relatives-aux-livraisons-de-vaccins-contre-la-covid-19/

Gupta, R., Tanwar, S., Kumar, N., & Tyagi, S. (2020). Blockchain-based security attack resilience schemes for autonomous vehicles in industry 4.0: A systematic review. Computers & Electrical Engineering, 86 , 106717. https://doi.org/10.1016/j.compeleceng.2020.106717

Hasan, F., Bellenstedt, M. F. R., & Islam, M. R. (2023). Demand and supply disruptions during the Covid-19 crisis on firm productivity. Global Journal of Flexible Systems Management, 24 (1), 87–105. https://doi.org/10.1007/s40171-022-00324-x

Hemmati-Sarapardeh, A., Larestani, A., Menad, N. A., & Hajirezaie, S., et al. (2020). Chapter 4 - Application of intelligent models in reservoir and production engineering. In A. Hemmati-Sarapardeh (Ed.), Applications of Artificial Intelligence Techniques in the Petroleum Industry (pp. 79–227). Gulf Professional Publishing. https://doi.org/10.1016/B978-0-12-818680-0.00004-7

Hey, T. (2010). ‘The next scientific revolution. Harvard Business Review, 88 (11), 56–63.

Google Scholar  

Holland, D., Seltzer, T., & Kochigina, A. (2021). Practicing transparency in a crisis: Examining the combined effects of crisis type, response, and message transparency on organizational perceptions. Public Relations Review, 47 (2), 102017. https://doi.org/10.1016/j.pubrev.2021.102017

Ikonomakis, M., Kotsiantis, S., & Tampakas, V. (2005). Text classification using machine learning techniques. WSEAS Transactions on Computers, 4 (8), 966–974.

Ivanov, D. (2020). Predicting the impacts of epidemic outbreaks on global supply chains: A simulation-based analysis on the coronavirus outbreak (COVID-19/SARS-CoV-2) case. Transportation Research Part E: Logistics and Transportation Review, 136 , 101922. https://doi.org/10.1016/j.tre.2020.101922

Ivanov, D. (2020b). ‘Viable supply chain model: Integrating agility, resilience and sustainability perspectives—lessons from and thinking beyond the COVID-19 pandemic.’ Annals of Operations Research . https://doi.org/10.1007/s10479-020-03640-6

Ivanov, D. (2021). Modeling supply chain resilience. Introduction to Supply Chain Resilience: Management, Modelling (pp. 63–92). Springer. https://doi.org/10.1007/978-3-030-70490-2_3

Ivanov, D. (2021). Supply chain viability and the COVID-19 pandemic: A conceptual and formal generalisation of four major adaptation strategies. International Journal of Production Research, 59 (12), 3535–3552. https://doi.org/10.1080/00207543.2021.1890852

Ivanov, D., & Das, A. (2020). Coronavirus (COVID-19/SARS-CoV-2) and supply chain resilience: A research note. International Journal of Integrated Supply Management, 13 , 90. https://doi.org/10.1504/IJISM.2020.107780

Ivanov, D., & Dolgui, A. (2019). New disruption risk management perspectives in supply chains: digital twins, the ripple effect, and resileanness. IFAC-PapersOnLine, 52 (13), 337–342. https://doi.org/10.1016/j.ifacol.2019.11.138

Ivanov, D., & Dolgui, A. (2020). Viability of intertwined supply networks: extending the supply chain resilience angles towards survivability. A position paper motivated by COVID-19 outbreak. International Journal of Production Research, 58 (10), 2904–2915. https://doi.org/10.1080/00207543.2020.1750727

Ivanov, D., & Dolgui, A. (2021). ‘Stress testing supply chains and creating viable ecosystems.’ Operations Management Research . https://doi.org/10.1007/s12063-021-00194-z

Joshi, S. (2022). A review on sustainable supply chain network design: Dimensions, paradigms, concepts, framework and future directions. Sustainable Operations and Computers , 3 , 136–148. https://doi.org/10.1016/j.susoc.2022.01.001

Jüttner, U. (2005). Supply chain risk management. The International Journal of ogistics Management, 16 (1), 120–141. https://doi.org/10.1108/09574090510617385

Kamsu-Foguem, B., Traore, B. B., & Tangara, F. (2018). Deep convolution neural network for image recognition. Ecological Informatics, 48 , 257–268. https://doi.org/10.1016/j.ecoinf.2018.10.002

Karatzoglou, A., Meyer, D., & Hornik, K. (2006). Support vector machines in R. Journal of Statistical Software, 15 , 1–28.

Koh, S. C. L., Gunasekaran, A., & Tseng, C. S. (2012). Cross-tier ripple and indirect effects of directives WEEE and RoHS on greening a supply chain. International Journal of Production Economics, 140 (1), 305–317. https://doi.org/10.1016/j.ijpe.2011.05.008

Kumar, V., Pallathadka, H., Sharma, S. K., Thakar, C. M., Singh, M., & Pallathadka, L. K. (2022). Role of machine learning in green supply chain management and operations management. Materials Today: Proceedings, 51 , 2485–2489. https://doi.org/10.1016/j.matpr.2021.11.625

LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521 , 436–444. https://doi.org/10.1038/nature14539

Lee, H. L., Padmanabhan, V. & Whang, S. (1997). ‘Information distortion in a supply chain: The Bullwhip Effect’, Management Science , 43(4):546–558, http://www.jstor.org/stable/2634565 .

Liu, S., & Chu, H. (2022). Examining the direct and indirect effects of trust in motivating COVID-19 vaccine uptake. Patient Education and Counseling, 105 (7), 2096–2102. https://doi.org/10.1016/j.pec.2022.02.009

Liu-Lastres, B., & Cahyanto, I. P. (2023). Are we always ready? Examining event professionals approaches to risk and crisis management and resilience. Tourism Management Perspectives, 46 , 101073. https://doi.org/10.1016/j.tmp.2023.101073

Merendino, A., & Sarens, G. (2020). Crisis? What crisis? Exploring the cognitive constraints on boards of directors in times of uncertainty. Journal of Business Research, 118 , 415–430. https://doi.org/10.1016/j.jbusres.2020.07.005

Nagurney, A. (2021). Optimization of supply chain networks with inclusion of labor: Applications to COVID-19 pandemic disruptions. International Journal of Production Economics, 235 , 108080. https://doi.org/10.1016/j.ijpe.2021.108080

Nyawa, S., Tchuente, D., & Fosso-Wamba, S. (2022). COVID-19 vaccine hesitancy: A social media analysis using deep learning. Annals of operations research . https://doi.org/10.1007/s10479-022-04792-3

Pagell, M., & Wu, Z. (2009). Building a more complete theory of sustainable supply chain management using case studies of ten exemplars. Journal of Supply Chain Management, 45 , 37–56. https://doi.org/10.1111/j.1745-493X.2009.03162.x

Patri, R., & Suresh, M. (2017). Modelling the enablers of agile performance in healthcare organization: A TISM approach. Global Journal of Flexible Systems Management, 18 (3), 251–272. https://doi.org/10.1007/s40171-017-0160-x

Pettit, T. J., Fiksel, J., & Croxton, K. L. (2010). Ensuring supply chain resilience: Development of a conceptual framework. Journal of Business Logistics, 31 (1), 1–21. https://doi.org/10.1002/j.2158-1592.2010.tb00125.x

Queiroz, M. M., Fosso Wamba, S., Raut, R. D., & Pappas, I. O. (2023). Does resilience matter for supply chain performance in disruptive crises with scarce resources? British Journal of Management . https://doi.org/10.1111/1467-8551.12748

Queiroz, M. M., Ivanov, D., Dolgui, A., & Fosso Wamba, S. (2020). ‘Impacts of epidemic outbreaks on supply chains: Mapping a research agenda amid the COVID-19 pandemic through a structured literature review.’ Annals of Operations Research . https://doi.org/10.1007/s10479-020-03685-7

Queiroz, M. M., Wamba, S. F., Jabbour, C. J. C., & Machado, M. C. (2022). Supply chain resilience in the UK during the coronavirus pandemic: A resource orchestration perspective. International Journal of Production Economics, 245 , 108405. https://doi.org/10.1016/j.ijpe.2021.108405

Rahman, M. H., Rahman, M. A., & Talapatra, S. (2020). The bullwhip effect: Causes, intensity, and mitigation. Production & Manufacturing Research, 8 (1), 406–426. https://doi.org/10.1080/21693277.2020.1862722

Ramanathan, U., Gunasekaran, A., & Subramanian, N. (2011). Supply chain collaboration performance metrics: A conceptual framework. Benchmarking: An International Journal, 18 , 856–872.

Riahi, Y., Saikouk, T., Gunasekaran, A., & Badraoui, I. (2021). Artificial intelligence applications in supply chain: A descriptive bibliometric analysis and future research directions. Expert Systems with Applications, 173 , 114702. https://doi.org/10.1016/J.ESWA.2021.114702

Rudner, T. G. J. & Toner, H. (2021). ‘Key Concepts in AI Safety: Interpretability in Machine Learning’.

Ruel, S., et al. (2021). ‘Supply chain viability: Conceptualization, measurement, and nomological validation.’ Annals of Operations Research . https://doi.org/10.1007/s10479-021-03974-9

Russell Reed, R. J. M. (1999) Neural Smithing: Supervised Learning in Feedforward Artificial Neural Networks .

Saha, P., Talapatra, S., Belal, H. M., & Jackson, V. (2022). Unleashing the potential of the TQM and industry 4.0 to achieve sustainability performance in the context of a developing country. Global Journal of Flexible Systems Management, 23 (4), 495–513. https://doi.org/10.1007/s40171-022-00316-x

Sassanelli, C., & Terzi, S. (2022). The D-BEST reference model: A flexible and sustainable support for the digital transformation of small and medium enterprises. Global Journal of Flexible Systems Management, 23 (3), 345–370. https://doi.org/10.1007/s40171-022-00307-y

Seuring, S., Stella, T., & Stella, M. (2021). ‘Developing and publishing strong empirical research in sustainability management—Addressing the intersection of theory, method, and empirical field.’ Frontiers in Sustainability . https://doi.org/10.3389/frsus.2020.617870

Shahriar, M. M., Parvez, M. S., Islam, M. A., & Talapatra, S. (2022). Implementation of 5S in a plastic bag manufacturing industry: A case study. Cleaner Engineering and Technology, 8 , 100488.

Shalev-Shwartz, S., et al. (2011). Pegasos: Primal estimated sub-gradient solver for SVM. Mathematical Programming, 127 (1), 3–30. https://doi.org/10.1007/s10107-010-0420-4

Sheng, M. L., & Saide, S. (2021). Supply chain survivability in crisis times through a viable system perspective: Big data, knowledge ambidexterity, and the mediating role of virtual enterprise. Journal of Business Research, 137 , 567–578. https://doi.org/10.1016/j.jbusres.2021.08.041

Singh, H. (2020). Big data, industry 4.0 and cyber-physical systems integration: A smart industry context. Materials Today: Proceedings . https://doi.org/10.1016/j.matpr.2020.07.170

Singh, S., Dhir, S., & Sushil, S. (2022). Developing an evidence-based TISM: An application for the success of COVID-19 Vaccination Drive. Annals of Operations Research . https://doi.org/10.1007/s10479-022-05098-0

Sorooshian, S., Salimi, M., Bavani, S., & Aminattaheri, H. (2012). Case report: Experience of 5S implementation. Journal of Applied Sciences Research, 8 (7), 3855–3859.

Spieske, A., Gebhardt, M., Kopyto, M., & Birkel, H. (2022). Improving resilience of the healthcare supply chain in a pandemic: Evidence from Europe during the COVID-19 crisis. Journal of Purchasing and Supply Management, 28 (5), 100748.

Talapatra, S., Santos, G., & Gaine, A. (2022). Factors affecting customer satisfaction in eatery business – An empirical study from Bangladesh. International Journal for Quality Research, 16 , 163–176. https://doi.org/10.24874/IJQR16.01-11

Valluri, A., North, M. J., & Macal, C. M. (2009). Reinforcement learning in supply chains. International Journal of Neural Systems, 19 , 331–344.

Wamba-Taguimdje, S. L., Fosso Wamba, S., Kala Kamdjoug, J. R., & Tchatchouang Wanko, C. E. (2020). Influence of artificial intelligence (AI) on firm performance: The business value of AI-based transformation projects. Business Process Management Journal, 26 (7), 1893–1924.

Wever, M., Shah, M., & O’Leary, N. (2022). Designing early warning systems for detecting systemic risk: A case study and discussion. Futures, 136 , 102882. https://doi.org/10.1016/j.futures.2021.102882

Witten, I. H., et al . (2017). ‘Chapter 9 - Probabilistic methods’, in I.H. Witten et al. (eds) Data Mining . Fourth Edn. Morgan Kaufmann, pp. 335–416. Available at: https://doi.org/10.1016/B978-0-12-804291-5.00009-X .

Younis, H., Sundarakani, B., & Alsharairi, M. (2022). Applications of artificial intelligence and machine learning within supply chains: Systematic review and future research directions. Journal of Modelling in Management, 17 (3), 916–940.

Zhou, Z.-H. (2009). When semi-supervised learning meets ensemble learning. In J. A. Benediktsson, J. Kittler, & F. Roli (Eds.), Multiple Classifier Systems (pp. 529–538). Springer.

Download references

The authors declare no funding. They have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Author information

Authors and affiliations.

LGP-ENIT-INPT, Université de Toulouse, 47 Avenue Azereix, BP 1629, 65016, Tarbes, France

Samia Zaoui & Bernard Kamsu-Foguem

Auban Moët Hospital (Groupement Universitaire de Campagne), 137 Rue de L’hôpital, 51200, Epernay, France

Clovis Foguem

Université de Lille, 1 Place de Verdun, 59045, Lille, France

TBS Business School, Place Alphonse Jourdain, 31068, Toulouse, France

Dieudonné Tchuente & Samuel Fosso-Wamba

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Dieudonné Tchuente .

Ethics declarations

Conflict of interest.

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Zaoui, S., Foguem, C., Tchuente, D. et al. The Viability of Supply Chains with Interpretable Learning Systems: The Case of COVID-19 Vaccine Deliveries. Glob J Flex Syst Manag 24 , 633–657 (2023). https://doi.org/10.1007/s40171-023-00357-w

Download citation

Received : 29 October 2022

Accepted : 05 September 2023

Published : 27 September 2023

Issue Date : December 2023

DOI : https://doi.org/10.1007/s40171-023-00357-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial intelligence
• Flexible supply chain
• Interpretability
• Machine learning
• Risk prediction
• Find a journal
• Publish with us
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• BMJ - PMC COVID-19 Collection

Side effects of COVID-19 vaccines: a systematic review and meta-analysis protocol of randomised trials

Kleyton santos medeiros.

1 Health Sciences Postgraduate Program, Federal University of Rio Grande do Norte, Natal, Rio Grande do Norte, Brazil

2 Instituto de Ensino, Pesquisa e Inovação, Liga Contra o Câncer, Natal, Rio Grande do Norte, Brazil

Ana Paula Ferreira Costa

Ayane cristine alves sarmento, cijara leonice freitas, ana katherine gonçalves.

3 Department of Obstetrics and Gynecology, Universidade Federal do Rio Grande do Norte, Natal, Brazil

Associated Data

Introduction.

SARS-CoV-2 is responsible for a large number of global COVID-19 cases. Strategies such as social isolation, personal hygiene and frequent hand washing have been implemented; however, a protective vaccine is required to achieve sufficient herd immunity to SARS-CoV-2 infection to ultimately control the COVID-19 pandemic. To meet the urgent need for a vaccine, a reduction in the development schedule has been proposed from 10–15 years to 1–2 years. For this reason, this systematic review and meta-analysis protocol aims to compare the side effects, safety and toxicity of COVID-19 vaccines available globally, including their combinations.

Methods and analysis

We will select randomised controlled trial-type studies that evaluate the side effects of the COVID-19 vaccine. PubMed, Web of Science, Embase, CINAHL, PsycINFO, LILACS, SCOPUS, ClinicalTrials.gov, International Clinical Trials Registry Platform (ICTRP), medRxiv.org, biorxiv.org, preprints.org and the Cochrane Library will be searched for eligible studies until December 2021. Three reviewers will independently screen and select studies, assess methodological quality and extract data. A meta-analysis will be performed, if possible, and the Grading of Recommendations, Assessment, Development and Evaluations summary of findings will be presented.

Ethics and dissemination

This study will review published data, and thus it is unnecessary to obtain ethical approval. The findings of this systematic review will be published in a peer-reviewed journal.

PROSPERO registration number

CRD42021231101.

Strengths and limitations of this study

• Four authors (KSM, APFC, ACAS, CLF) will select the articles independently using titles and abstracts.
• To the best of our knowledge, there are no existing reviews regarding the side effects of COVID-19 vaccines.
• The DerSimonian and Laird method may underestimate the true between-study variance, potentially producing overly narrow CIs for the mean effect. This fact is a limitation, so the collection of studies will be done with care and the assumptions of the analytical methods will be assessed.

SARS-CoV-2 is responsible for a large number of global COVID-19 cases. It is a highly transmissible virus among humans that has become a significant public health issue. 1 Symptoms include fever, dry cough, fatigue, shortness of breath, chills, muscle pain, headache, gastric disorders and weight loss, often leading to death. 2

Strategies such as social isolation, personal hygiene and frequent hand washing have been implemented; however, a protective vaccine is required to achieve sufficient herd immunity to SARS-CoV-2 infection to ultimately control the COVID-19 pandemic. 3 To meet the urgent need for a vaccine, a reduction in the development schedule has been proposed from 10–15 years to 1–2 years. 4

SARS-CoV-2 is an RNA virus with a high mutation rate, and that on the envelope surface has three important structural proteins that can be identified: spike protein (S), envelope protein (E) and membrane protein (M). Most innovative vaccines have focused their efforts on inducing an immune response against the S protein. Attenuated virus vaccines are based on weakened microorganisms, effective in stimulating the immune system. The inactivated ones (dead microorganisms) are more stable than the attenuated ones, but they have a short duration of immunological memory that requires the association of adjuvants. mRNA vaccines are stable—and can be easily produced in large quantities. Vaccines against COVID-19 differ in composition and mechanism of action, which may be relevant for their safety and efficacy, being essential for the success and eradication of this infection. 5 6 The viral vector (mRNA) vaccine encodes full-length S protein ectodomains of SARS-CoV-2, which contains both T and B cell epitopes that can induce cellular and humoral immune responses against viral infection. 7

Assessing the safety, efficacy and side effects of the vaccine is urgently needed, and has been heavily scrutinised by the leading medical agencies around the world, like the Centers for Disease Control and Prevention and the Food and Drug Administration. Developing any vaccine needs to ensure that safety risks are identified and quantified against potential benefits. Among the potential risks raised in the context of COVID-19, vaccine development is the security and effectiveness of immune responses elicited by a vaccine. Here, this systematic review protocol aims to assess the side effects, safety and toxicity of vaccines against COVID-19.

This systematic review and meta-analysis protocol aims to compare the side effects, safety and toxicity of COVID-19 vaccines available globally, including their combination.

Review question

What are the rates of adverse reactions (local and systemic) to COVID-19 vaccines?

The meta-analysis protocol follows the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols guidelines. 8 9 This protocol is registered with the International Prospective Register of Systematic Reviews (PROSPERO).

Eligibility criteria

The inclusion criteria involved: (1) randomised controlled trial (RCT)-type studies that evaluated the side effects of the COVID-19 vaccine; (2) experiments involving human beings; (3) studies evaluating the safety, immunogenicity and efficacy parameters of the vaccines; (4) studies that presented similar vaccination protocols; (5) studies published since January 2020 until December 2021; and (6) studies published in any language.

The exclusion criteria were as follows: (1) observational studies, and (2) case reports, meeting abstracts, review papers and commentaries.

Patients, intervention, comparison, outcome strategy and types of studies

• Patients: healthy adults aged 18 years or older who were HIV negative and previously SARS-CoV-2 infection free.
• Intervention: COVID-19 vaccine or a combination of vaccines against COVID-19.
• Comparator/control: placebo.
• Outcome: safety, tolerability and immunogenicity of the COVID-19 vaccine or the combination of vaccines against COVID-19.
• Types of studies: RCTs.

Information sources

The following databases will be searched: Medline / PubMed, Web of Science, Embase, CINAHL, PsycINFO, Latin American and Caribbean Health Sciences Literature (LILACS), SCOPUS, ClinicalTrials.gov, International Clinical Trials Registry Platform (ICTRP), medRxiv.org, biorxiv.org, preprints.org and Cochrane Central Controlled Trials Registry. Furthermore, eligible studies may also be selected from the reference lists of retrieved articles.

Patient and public involvement

The individual patient data will not be presented. A literature search will be carried out from defined databases. No patient will be involved in the study planning and application process during neither the analysis nor the dissemination of results.

Search strategy

Our keyword search will be based on Medical Subject Headings according to the following combination: (COVID-19 OR SARS-CoV-2 OR 2019-nCoV OR coronavirus) AND (vaccines OR vaccination OR COVID-19 vaccine OR SARS-CoV-2 vaccine OR BNT162 vaccine OR mRNA-1273 vaccine OR COVID-19 aAPC vaccine OR INO-4800 vaccine OR LV-SMENP-DC COVID-19 vaccine OR Ad5-nCoV vaccine OR ChAdOx1 COVID-19 vaccine OR MNA SARS-CoV-2 S1 subunit vaccines OR PittCoVacc OR Inactivated novel coronavirus 2019-CoV vaccine Vero cells OR Inactivated Vaccines OR SARS-CoV-2 inactivated vaccines OR Viral Vaccines OR Gam-COVID-Vac vaccine OR Ad26.COV2.S vaccine OR EpiVacCorona vaccine) AND (Toxicity OR Vaccine Immunogenicity OR side effects OR adverse events) AND (randomized controlled trial OR double blind method OR clinical trial) ( table 1 ). A list of vaccines available at WHO was also used.

Medline search strategy

Study records

Four researchers (KSM, APFC, ACAS, CLF) performed the selection of the studies of interest. Titles and abstracts will be read independently, and duplicate studies will be excluded. The same authors analysed the selected texts to assess the compliance with the inclusion criteria. A fifth reviewer, AKG, solves the discrepancies. The flow chart of this study is shown in figure 1 .

Flow diagram of the search for eligible studies on the side effects, safety and toxicity of the COVID-19 vaccine. CENTRAL, Cochrane Central Register of Controlled Trials.

Data collection process and management

A standardised data extraction form was developed and tested. Data from each included study will be extracted independently by two reviewers (ACAS and APFC), and any subsequent discrepancies will be resolved through discussion with a third reviewer (AKG). The data extracted will include information on authors, the year of publication, study location, type of study, main objectives, population, type of vaccine, follow-up of participants, rates of systemic events, gastrointestinal symptoms, injection site-related adverse effects and serious vaccine-related adverse events ( table 2 ). Furthermore, participant characteristics (eg, mean age, gender) and results for immunogenicity will be collected.

Adverse events of COVID-19 vaccines

The study authors will be contacted in case of missing data and/or to resolve any uncertainties. In addition, any additional information will be recorded. All data entries will be checked twice. If we find a set of articles with similar characteristics based on the information in the data extraction table, we will perform a meta-analysis using a random-effects model. If there are data that are not clear in some articles, the corresponding author will be contacted for possible clarification.

Risk of bias in individual studies

Three authors (KSM, ACAS, APFC) will independently assess the risk of bias in the eligible studies using the Cochrane risk-of-bias tool. 10 The Risk of Bias 2 tool 11 will be used to assess the risk of bias. Bias is assessed as a judgement (high, low or unclear) for individual elements from five domains (selection, performance, attrition, reporting and others).

Data will be entered into the Review Manager software (RevMan V.5.2.3). This software allows the user to enter protocols; complete reviews; include text, characteristics of the studies, comparison tables and study data; and perform meta-analyses. For dichotomous outcomes, we extracted or calculated the OR and 95% CI for each study. In case of heterogeneity (I 2 ≥50%), the random-effects model will be used to combine the studies to calculate the OR and 95% CI using the DerSimonian-Laird algorithm 12 .

Data synthesis and analysis

To grade the strength of evidence from the included data, we will use the Grading of Recommendations, Assessment, Development and Evaluation 13 approach. The summary of the assessment will be incorporated into broader measurements to ensure the judgement of the risk of bias, consistency, directness and precision. The quality of the evidence will be assessed based on the risk of bias, indirectness, inconsistency, imprecision and publication bias.

The COVID-19 pandemic represents one of the most significant global public health crises of this generation. Lockdown, quarantine, contact tracing and case isolation are suggested as effective interventions to control the epidemic; however, they may present different results in different contexts because of the specific features of the COVID-19. The lack of implementation of continued interventions or effective treatments further contributes to discovering and using effective and safe vaccines. 14 15

For all these reasons, scientists worldwide entered a race to find a vaccine candidate useful in fighting the new coronavirus pandemic. Nevertheless, it is essential to note that a vaccine’s production is not easy and quick. Before being released to the population, a vaccine must go through three phases of clinical trials that prove its safety and effectiveness. More volunteers are recruited at each stage, and the researchers analyse the test results to ensure that a vaccine can be licensed. 16–18

One hundred and seventy-three vaccines were in preclinical development and 64 in clinical trials until 20 January 2021. On 31 December 2020, the WHO listed the mRNA vaccine against COVID-19 for emergency use, making this Pfizer/BioNTech immuniser the first to receive WHO emergency validation from the beginning outbreak. Already, in January 2021, emergency approval was granted to nine vaccines by regulatory authorities in different parts of the world. 14 19

With the starting vaccination, several studies were carried out to ascertain the safety of these vaccines, since they were produced in record time. 20–22 Currently, one systematic review about the thematic showed that of 11 published clinical trials of COVID-19 vaccines included in the study, adverse reactions reported were considered mild to moderate with few severe reactions which were unrelated to the test vaccine. Common adverse events were pain at the site of injection, fever, myalgia, fatigue and headache. Serious adverse events (SAE) were reported in four trials: COVID-19 Vaccine AstraZeneca (AZD1222)—168 SAEs with only three related to the vaccine; Ad26.COV2.S—four with none related to the testing vaccine; five with Comirnaty (BNT162b1) vaccine and one with Covaxin (BBV152) vaccine. 19

One limitation about the COVID-19 vaccine safety tested until now is that clinical trials of the safety and effectiveness have had low inclusion of vulnerable groups, for example, older persons, the first population to receive the whole vaccine. That’s why pharmacovigilance postmarketing is necessary to surveillance of new drugs, as a critical aspect of evaluating medicine safety and effectiveness, particularly in risk groups.

Other prevention approaches are likely to emerge in the coming months, including antiviral agents, drugs may be to decrease disease progression, monoclonal antibodies, hyperimmune globulin and convalescent titre. If proven effective, these approaches could be used in high-risk individuals, including healthcare workers, other essential workers and older adults. 23–26 It is essential to maintain protective measures such as washing hands frequently with soap and water or gel alcohol and covering the mouth with a forearm when coughing or sneezing.

For all the reasons mentioned above, this review is necessary and essential. The latter is a well-defined protocol registered with PROSPERO, well planned to include the largest possible number of vaccines, a significant number of vaccinated patients, thus providing safe and reliable results regarding the use of vaccines.

Supplementary Material

Contributors: KSM, ACAS and APFC contributed to the design of this review. KSM and ACAS drafted the protocol manuscript. APFC and AKG revised the manuscript. KSM, AKG and APFC developed the search strategies. KSM, CLF and ACAS implemented the search strategies. KSM, CLF, ACAS and APFC tracked the potential studies, extracted the data and assessed the quality. In case of disagreement between the data extractors, AKG advised on the methodology and worked as a referee. KSM completed the data synthesis. All authors approved the final version for publication.

Funding: The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests: None declared.

Patient and public involvement: Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review: Not commissioned; externally peer reviewed.

Ethics statements

Patient consent for publication.

Not applicable.