Log in using your username and password

  • Search More Search for this keyword Advanced search
  • Latest content
  • Current issue
  • BMJ Journals More You are viewing from: Google Indexer

You are here

  • Volume 12, Issue suppl 2
  • The role of structured observational research in health care
  • Article Text
  • Article info
  • Citation Tools
  • Rapid Responses
  • Article metrics

Download PDF

  • Correspondence to:
 Dr J Carthey
 Assistant Director of Patient Safety, Interagency Working Directorate, National Patient Safety Agency, 4–8 Maple Street, London W1T 5HD, UK; jane.carthey{at}npsa.nhs.uk

Structured observational research involves monitoring of healthcare domains by experts to collect data on errors, adverse events, near misses, team performance, and organisational culture. This paper describes some of the results of structured observational studies carried out in health care. It evaluates the strengths, weaknesses, and future challenges facing observational researchers by drawing lessons from the human factors and neonatal arterial switch operation (ASO) study in which two human factors specialists observed paediatric cardiac surgical procedures in 16 UK centres. Lessons learned from the ASO study are germane to other research teams embarking on studies that involve observational data collection. Future research needs robust observer training, clear measurable criteria to assess each researcher’s domain knowledge, and observational competence. Measures of inter-rater reliability are needed where two or more observers participate in data collection. While it is important to understand the factors that lead to error and excellence among healthcare teams, it is also necessary to understand the characteristics of a good observer and the key types of error that can occur during structured observational studies like the human factors and ASO project.

  • structured observational research
  • error measurement


Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Ethnography is the observation and systematic recording of human cultures and the descriptive work produced from such research. 1 It usually involves an individual or team of researchers who “live” alongside a given workforce, tribe or team, observing how they behave and systematically recording these observations. In health care, ethnographic approaches—originally developed by social scientists—have been adapted and extended; as well as collecting qualitative data, a quantitative (statistical) analysis is also carried out. This is called “structured observational research”.


To date, structured observational research in health care has identified the types, frequency, and severity of adverse events that occur in different domains, including drug administration, 2– 5 emergency departments, 6, 7 operating theatres, 8, 9 anaesthesia, 10 obstetrics, 11 and intensive care units. 12– 14 Structured observational studies have also identified the individual, team, and organisational precursors of adverse events. 6– 7, 15, 16 Some examples are given below to illustrate its potential value.

In one study 17 trained observers recorded all adverse events discussed during day shifts, medical ward rounds, nursing shift changes, case conferences, and departmental meetings in three wards. 185/1047 patients had at least one serious adverse event. The likelihood of experiencing an adverse event increased by about 6% for each day of hospital stay.

Structured observational research has also identified the type, frequency, and severity of drug administration errors. Barker et al 3 undertook a study of 36 US hospitals in which observations of drug rounds were carried out on nursing units which used high volumes of drugs. The results showed an overall drug administration error rate of 19% (605/3216). The most common error types were wrong time (43%), omission (30%), and wrong dose (17%); 7% of errors (40 per day per 300 patient unit) were judged by an expert panel of physicians to be harmful. 3

A recent UK study carried out on 10 wards in two hospitals showed that 212/430 observed intravenous drug preparations and administrations had at least one error. An additional 37 had more than one error per drug dose. Giving boluses and preparing drugs that required multiple steps were identified as the drug administration tasks most susceptible to error. 4

Research has also been carried out in the intensive care unit (ICU). One observational study showed an error rate of 1.7 errors per patient per day. 15, 16 Another study based in a multidisciplinary ICU identified 777 adverse events among 1024 consecutive patients admitted in 1 year. 14 There were 241 human errors (31%) in 161 patients. These errors were classified as errors in planning (n = 75), execution (n = 88), and surveillance (n = 78). The most serious errors were caused by planning failures. It was calculated that human errors prolonged ICU stay by 425 patient days or the equivalent of 15% of ICU time. 14

Observational studies have also been carried out in emergency departments. In one US study a trained emergency room nurse and a physician observed cases and rated teamwork behaviours among various teams working in nine US hospitals. 6 Participating teams were then trained how to improve their performance using a framework called MedTeams which trains key communication and coordination skills. The results showed improved teamwork and a significant reduction in errors among trained teams.

Quite often the structure and culture of healthcare organisations acts as a barrier to effective teamwork; structured observational research is good at identifying these problems. During several of the adverse events observed in the MedTeams study, one team member possessed a skill or had doubts about the course of action that was being taken which, if communicated to the rest of the team, would have prevented the adverse event. Emergency room staff did not speak up because there was a culture of not questioning one’s superiors. This behaviour mirrors findings in the aviation field where co-pilots and cabin crew are often reluctant to question the captain’s authority. 18, 19

Similar organisational problems have been identified in the UK accident and emergency (A&E) departments. In one A&E department a rigid horizontal structure, vertical division of labour, and the strict authority dynamics meant that staff who knew important patient safety information could not influence decision making. It was concluded that there was a need to engender a workplace culture in which sapiential authority—that is, derived from experience or availability in an emergency or holding key information—is recognised in addition to authority derived from formal status. 7

In addition to faulty organisational structures and team cultures, structured observational research has also identified interruptions as a problem for emergency department doctors. 15, 16 Emergency department doctors are “interrupt driven”. They are frequently interrupted and many interruptions result in breaks in task which cause them to switch attention to a different task altogether. 16

These (and other) studies have made an important contribution to our understanding of factors that influence patient safety. They illustrate what types of errors and adverse events occur in various healthcare settings and, in doing so, they make it clear why structured observational research should be prioritised on patient safety research agendas. However, research teams usually publish their findings and do not critique their methods. This prevents the broader health care and research communities from learning about the types of methodological problems and pitfalls experienced by observational researchers. The objective of this paper is to critique one structured observational study—the human factors and the arterial switch operation (ASO) study 20– 22 —to learn about the methodological problems typically experienced when carrying out this type of research. These include measuring interobserver reliability, proving the validity of checklists and other data collection tools, the selection and training of observers, and the characteristics of a good observer.


Human factors is an umbrella term for multidisciplinary studies which analyse people at work to identify how equipment design, organisational, environmental, personal, and social factors influence their performance.

The ASO is carried out on babies who are born with the great vessels of the heart connected to the wrong ventricle. Hence, the aorta is connected to the right ventricle and the pulmonary artery to the left ventricle. The surgical procedure involves putting the patient onto a heart lung bypass machine and freezing the heart with a potassium based solution called cardioplegia. The heart lung bypass machine provides mechanical support to the heart and lungs while the surgical procedure is being carried out. Once the heart is frozen, the cardiac surgeon then transects the native aorta (connected to the right ventricle) and excises the coronary arteries (the tiny vessels that carry oxygenated blood to the heart) from the native aorta and re-implants them into a neo-aorta. A neo-pulmonary artery is also reconstructed using the tissue from the trunk of the native aorta and a piece of tissue called pericardium. Given that the coronary arteries are only millimetres wide and are extremely fragile, the surgeon is working at the edge of the safety envelope.

The human factors and ASO study involved 16 UK paediatric cardiac centres. Observational case study data were collected by two trained human factors specialists on 173 ASOs. The observers watched each case from the point of induction of anaesthesia to handover of the patient from the operating theatre to the ICU. Throughout each case the observers noted down any errors, problems and notable aspects of good performance. The observer’s interpretation was checked with the operating theatre team after each case and a summary report was written.

The case reports differentiated between major and minor events. Minor events are errors that disrupt the surgical flow of the procedure and would not be expected in isolation to have serious consequences to the patient. Major events are errors that are likely to have serious consequences for patient safety. In the statistical analysis, researchers generated two baseline regression models for negative outcomes (the first for death, the second for death and/or near miss) based on patient variables. In the first stage of the analysis the total number of major and minor events per case was added separately to the baseline regression models. The total number of major events per case influenced the probability of death (p<0.001) and death and/or near miss (p<0.001) after adjustment for patient variables. Whether or not major events were compensated for—that is, recognised and recovered from—was also a strong predictor of death (p<0.003). Even for the most serious types of errors (major events), appropriate compensation produces a good outcome. 20

In contrast, when minor events were added separately to the baseline regression models it was the overall number per case that had a significant effect on death and death and/or near miss (p<0.001 in both models). For minor events there was no effect of compensation.

In the second stage of the analysis the total number of major and minor events per case was added jointly to the baseline models. Minor events still influenced surgical outcomes even after adjustment for the total number of major events per case and whether or not they were compensated for (p<0.03 for death; p<0.001 for death and/or near miss). It was concluded that seemingly trivial problems can accumulate to have negative effects.

Understanding surgical excellence

The study also identified the behavioural markers which may lead to surgical excellence. 22 Excellence was defined as achieving a standard beyond the norm expected for a particular type of case.

Procedural excellence scores were developed for 16 surgeons. These scores were based on the difference between observed and expected risk measures derived from baseline multivariable logistic regression models. A negative mean difference equalled fewer major and minor events and fewer uncompensated events than expected after adjusting for patient factors. Differences between procedural excellence scores for the best and worst performing surgeons were explained using a framework of behavioural markers of surgical excellence which comprised individual, team, and organisational factors. 22

Learning methodological lessons from the human factors and ASO study

There are several limitations with this study. These relate to (1) observer training and competency assessment, (2) lack of inter-rater reliability measures, (3) the sole reliance on observational data collection, and (4) transfer of learning from an expert to a novice observer late in the data collection period.

Observer training and competency assessment

Observer 1, a PhD level human factors specialist, was trained by watching three cardiac surgeons performing the surgical procedure in two centres, reading surgical textbooks, and shadowing operating theatre staff. Learning was iterative; it also depended on her confidence to ask questions and her ability to assimilate complex medical information. There was no formal competency assessment (in terms of a written or verbal test of her knowledge). Rather, the senior surgeon who carried out the study reviewed the case reports she produced following each ASO and made a judgement about her readiness to collect data from the operating theatre.

Observer 2, a psychology graduate, never developed sufficient understanding of the ASO to make meaningful observations and his data were not used in the final analysis.

Observer 3, a human factors specialist educated to MSc level, was trained by observer 1. She produced a booklet summarising the ASO in layman’s terms. This document also described key errors and problems that occurred during each stage of the surgical procedure and operating theatre etiquette (where to stand, when to ask the team questions, etc). Observer 3 was trained by shadowing observer 1 for 2 months. During this time, both observer 1 and observer 3 took notes for the same case and compared them afterwards. This proved a valuable way to train observer 3 as he could validate his observations against those of a more experienced colleague. Observer 3 proved to be a competent observer but, as with observer 1, there was no formal examination of his knowledge. Rather, observer 1 and a senior surgeon decided when he was ready to observe cases independently.

Lack of inter-rater reliability measurement

Only one observer was present during each case (with the exception of the training period for observer 3 described above). This precluded inter-rater reliability measures to check the consistency of observations between researchers. This is a serious methodological flaw. However, the research team was undertaking a massive logistical task by collecting data from 16 centres; often there was more than one case per day and obtaining a good sample size took precedence over measuring inter-rater reliability. Furthermore, the operating theatre does not easily lend itself to having multiple observers without compromising the theatre team’s access to the patient.

Sole reliance on observational data collection

The research team did not video record ASOs and relied solely on case reports. Other studies have shown that video recording produces reliable information on team performance. 23, 24 Video recordings could have validated and supplemented the information collected by the two observers.

Transfer of learning from an expert to a novice observer

Observer 1 had acquired expertise in observing ASOs by the time observer 3 joined the research team. There were some difficulties in transferring her knowledge to a novice as she had internalised a surgeon’s instinct for the ASO. Difficulties were experienced in verbalising implicit knowledge of “what happens next” and why certain recovery strategies were appropriate in particular circumstances.

The preceding discussion has focused exclusively on why we should carry out structured observational research, citing examples from published literature to illustrate what such studies have contributed to our knowledge about adverse events in health care. The review of the human factors and ASO study has identified the types of problems likely to be faced by ethnographic research teams. This discussion focuses on answering questions pertaining to the “where, how and who” of structured observational research in health care.

Where is structured observational research viable?

Structured observational research may be more suited to some healthcare domains than others. Whereas the operating theatre provides an environment where clinical tasks have a clear start and end point, the type of elective surgical procedure is usually known beforehand, and there are consistent team roles, A&E departments and ICUs are more challenging. Their unpredictable diverse case mix, larger size, and the greater movement of staff around a wider area while treating the patient can create difficulties for observers. 25

Some types of healthcare tasks are easier to observe than others—for example, staff using bar coding devices, 26 drug dispensing, drug administration, 2– 5 and hand offs between teams. In general, tasks that involve verbal communication or a potentially high frequency of omission and commission errors provide good observational settings.

How should observers be trained?

Healthcare studies should learn from research previously carried out in anthropology, human factors, and organisational psychology, all of which have developed assessment instruments in an effort to make observational data collection systematic and to standardise analysis.

Observers might benefit from training that involves videotapes of procedures with simultaneous explanations from healthcare professionals. Future studies should also assess observer competence from two perspectives: (1) their domain knowledge (that is, their technical know how about the specialty) and (2) their observational ability (that is, whether they have acquired the key skills to make meaningful observations). The key skills or attributes of an excellent observer remains an unanswered question and is still open to debate.

Who is the most appropriate observer?

There is an ongoing debate about who is the most appropriate observer—a medical or a non-medical professional? The operating theatre simulator literature presents two viewpoints on observer qualifications. Some studies advocate medical experts as observers, but others support trained non-medical observers. The literature shows that there is not much difference in the assessments of both types of observers except that medical experts are better at assessing content specific attributes, 27, 28 while non-medical observers are better at assessing interpersonal factors. 29, 30 Research in other industries has shown that researchers who develop good domain knowledge can make consistent and meaningful observations.

In the ASO study, observers 1 and 3 were better at identifying minor events than the operating theatre team. For example, interruptions by colleagues who asked the surgeon questions while he was operating were classed as minor events by the researchers. These were accepted as common practice by the operating theatre team who did not appreciate the increased risk to the patient of frequent task distractions. Medical professionals sometimes do not recognise an event as an error or problem and may also be reluctant to report errors if it makes the team being observed look bad. However, medical professionals are more likely to be accepted by those under observation than non-medical professionals who may be perceived as “outsiders”. This was certainly the experience in the human factors and ASO study where it took time to win the trust and confidence of participating operating theatre teams.

The ASO study has shown that creating a good observer requires consideration of factors other than a person’s professional background. It may be equally relevant to consider their interpersonal skills, their ability to reassure healthcare staff afraid of the medicolegal and punitive consequences of the data, to maintain concentration for long periods of time, to keep to the stated objectives, and to cope with the psychological aftermath of witnessing adverse events. We should work towards creating multidisciplinary observational teams which capitalise on the different perspectives contributed by people from diverse specialities.

Key messages

Structured observational research has a key role in identifying the types, frequency, and severity of errors and adverse events in health care.

Important methodological lessons can be learned from structured observational studies that have already been carried out. It is essential that these lessons inform the development of future research methodologies.

Decisions about who is the most appropriate observer to collect data from healthcare domains should be based on an appraisal of each candidate’s personal skills including the ability to win people’s trust, to maintain attention for long time periods, as well as their domain knowledge and observational experience.


Patient safety experts should value learning methodological lessons from successful ethnographic studies as these can inform the design of future research. While it is important to understand the factors that lead to error and excellence among healthcare teams, it is also necessary to understand the characteristics of an excellent observer and the methodological flaws in studies like the human factors and arterial switch operation project.


The human factors and neonatal arterial switch operation study was supported by a research grant from the British Heart Foundation (PG94166).

  • ↵ Anon . Webster’s Dictionary. Harper Collins Publications, 1998 .
  • ↵ Dean B , Barber N. Validity and reliability of observational methods for studying medication administration errors. Am J Health Syst Pharm 2001 ; 58 : 54 –9. OpenUrl Abstract / FREE Full Text
  • ↵ Barker KN , Flynn EA, Pepper GA, et al. Medication errors observed in 36 health care facilities. Arch Intern Med 2002 ; 162 : 1897 –903. OpenUrl CrossRef PubMed Web of Science
  • ↵ Taxis K , Barber N. Ethnographic study of incidence and severity of intravenous drug errors. BMJ 2003 ; 326 : 684 . OpenUrl Abstract / FREE Full Text
  • ↵ Wirtz V , Taxis K, Barber ND. An observational study of intravenous medication errors in the United Kingdom and in Germany. Pharm World Sci 2003 ; 25 : 104 –11. OpenUrl CrossRef PubMed Web of Science
  • ↵ Morey JC , Simon R, Jay GD, et al. Error reduction and performance improvement in the emergency department through formal teamwork training: evaluation results of the MedTeams project. Health Serv Res 2002 ; 37 : 1553 –81. OpenUrl CrossRef PubMed Web of Science
  • ↵ Boreham NC , Shea CE, Mackway-Jones K. Clinical risk and collective competence in the hospital emergency department in the UK. Soc Sci Med 2000 ; 51 : 83 –91.
  • ↵ Schaefer HG , Helmreich RL, Scheidegger D. Safety in the operating theatre—part 1: interpersonal relationships and team performance. Curr Anaesth Crit Care 1995 ; 6 : 48 –53. OpenUrl CrossRef PubMed
  • ↵ Schaefer HG , Helmreich RL. The importance of human factors in the operating room. Anesthesiology 1994 ; 80 : 479 . OpenUrl CrossRef PubMed Web of Science
  • ↵ Gaba DM , Howard SK, Small SD. Situation awareness in anesthesiology. Hum Factors 1995 ; 37 : 20 –31. OpenUrl Abstract / FREE Full Text
  • ↵ Stanhope N , Vincent C, Taylor-Adams SE, et al. Applying human factors methods to clinical risk management in obstetrics. Br J Obstet Gynaecol 1997 ; 104 : 1225 –32. OpenUrl PubMed Web of Science
  • ↵ Donchin Y , Gopher D, Olin M, et al. A look into the nature and causes of human errors in the intensive care unit. Crit Care Med 1995 ; 23 : 294 –300. OpenUrl CrossRef PubMed Web of Science
  • Donchin Y , Gopher D, Olin M, et al. A look into the nature and causes of human errors in the intensive care unit. Qual Saf Health Care 2003 ; 12 : 143 –7. OpenUrl Abstract / FREE Full Text
  • ↵ Bracco D , Favre JB, Bissonnette B, et al. Human errors in a multidisciplinary intensive care unit: a 1-year prospective study. Intensive Care Med 2001 ; 27 : 137 –45. OpenUrl CrossRef PubMed Web of Science
  • ↵ Coiera EW , Jayasuriya RA, Hardy J, et al. Communication loads on clinical staff in the emergency department. Med J Aust 2002 ; 176 : 415 –8. OpenUrl PubMed Web of Science
  • ↵ Chisholm CD , Collison EK, Nelson DR, et al. Emergency department workplace interruptions: are emergency physicians “interrupt-driven” and “multitasking”? Acad Emerg Med 2000 ; 7 : 1239 –43. OpenUrl CrossRef PubMed Web of Science
  • ↵ Andrews LB , Stocking C, Krizek T, et al. An alternative strategy for studying adverse events in medical care. Lancet 1997 ; 349 : 309 –13. OpenUrl CrossRef PubMed Web of Science
  • ↵ Helmreich RL , Merritt AC. Culture at work in aviation and medicine: national, cultural and professional influences. Aldershot, UK: Ashgate Publishers, 1998 .
  • ↵ Helmreich RL . On error management: lessons from aviation. BMJ 2000 ; 320 : 781 –5. OpenUrl FREE Full Text
  • ↵ de Leval MR , Carthey J, Wright DJ, et al. Human factors and cardiac surgery: a multi-center study. J Thorac Cardiovasc Surg 2000 ; 119 : 661 –72. OpenUrl CrossRef PubMed Web of Science
  • Carthey J , de Leval MR, Reason JT. The human factor in cardiac surgery: errors and near misses in a high technology medical domain. Ann Thorac Surg 2001 ; 72 : 300 –5. OpenUrl CrossRef PubMed Web of Science
  • ↵ Carthey J , de Leval MR, Wright DJ, et al. Behavioural markers of surgical excellence. Safety Sci 2003 ; 41 : 409 –25. OpenUrl CrossRef
  • ↵ Xiao Y , MacKenzie C, Orasanu J, et al. Information acquisition from audio-video-data sources: an experimental study on remote diagnosis. The LOTAS Group. Telemed J 1999 ; 5 : 139 –55. OpenUrl CrossRef PubMed Web of Science
  • ↵ Mackenzie CF , Jefferies NJ, Hunter WA, et al. Comparison of self-reporting of deficiencies in airway management with video analyses of actual performance. Level One Trauma Anesthesia Simulation (LOTAS) Group. Hum Factors 1996 ; 38 : 623 –35. OpenUrl Abstract / FREE Full Text
  • ↵ Wears RL . Beyond error. Acad Emerg Med 2000 ; 7 : 1175 –6. OpenUrl CrossRef PubMed Web of Science
  • ↵ Patterson ES , Cook RI, Render ML. Improving patient safety by identifying side effects from introducing bar coding in medication administration. J Am Med Inform Assoc 2002 ; 9 : 540 –53. OpenUrl Abstract / FREE Full Text
  • ↵ Slagle J , Weinger MB, Dinh MT, et al. Assessment of inter-rater reliability of an established clinical task analysis methodology. Anesthesiology 2002 ; 96 : 1129 –39. OpenUrl CrossRef PubMed Web of Science
  • ↵ Fraind DB , Slagle JM, Tubbessing VA, et al. Re-engineering intravenous drug and fluid administration processes in the operating room. Step one: task analysis of existing processes task analysis of existing processes. Anesthesiology 2002 ; 97 : 139 –47. OpenUrl CrossRef PubMed Web of Science
  • ↵ Schaefer HG , Helmreich RL, Scheidegger D. Safety in the operating theatre—part 1: interpersonal relationships and team performance. Curr Anaesth Crit Care 1995 ; 6 : 48 –53.
  • ↵ Schaefer HG , Helmreich RL, Scheidegger D. Human factors and safety in emergency medicine. Resuscitation 1994 ; 28 : 221 –5. OpenUrl CrossRef PubMed

Read the full text or download the PDF:

  • - Google Chrome

Intended for healthcare professionals

  • Access provided by Google Indexer
  • My email alerts
  • BMA member login
  • Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution


Search form

  • Advanced search
  • Search responses
  • Search blogs
  • Qualitative Research:...

Qualitative Research: Observational methods in health care settings

  • Related content
  • Peer review
  • Nicholas Mays a , director of health services research ,
  • Catherine Pope , director of health services research
  • a King's Fund Institute, London W2 4HT
  • b Department of Epidemiology and Public Health, University of Leicester, Leicester LE1 6TP
  • a Correspondence to: Mr Mays

Clinicians used to observing individual patients, and epidemiologists trained to observe the course of disease, may be forgiven for misunderstanding the term observational method as used in qualitative research. In contrast to the clinician or epidemiologist, the qualitative researcher systematically watches people and events to find out about behaviours and interactions in natural settings. Observation, in this sense, epitomises the idea of the researcher as the research instrument. It involves “going into the field”—describing and analysing what has been seen. In health care settings this method has been insightful and illuminating, but it is not without pitfalls for the unprepared researcher.

The term “observational methods” seems to be a source of some confusion in medical research circles. Qualitative observational studies are very different from the category of observational studies (non-experimental research designs) used in epidemiology, nor are they like the clinical observation of a patient. Observational methods used in social science involve the systematic, detailed observation of behaviour and talk: watching and recording what people do and say. Goffman neatly captured this distinct research method with his recommendation that, in order to learn about a social group, one should “submit oneself in the company of the members to the daily round of petty contingencies to which they are subject.” 1 Thus, observational methods can involve asking questions and analysing documents, but the primary focus on observation makes it distinct from a qualitative research interview (see the next paper in this series) or history taking during patient consultation. Another crucial point about qualitative observation is that it takes place in natural settings not experimental ones; hence, this type of work is often described as “naturalistic research.”

  • View inline

Research roles

In an attempt to minimise the impact on the environment being studied the researcher sometimes adopts a “participant observer” role, becoming involved in the activities taking place while also observing them. The degree of participation varies according to the nature of the setting and the research questions, but broadly corresponds to the first two research roles described in Gold's typology (box 1). 2 There are obviously important ethical considerations about the decision to conduct covert research, and for this reason examples of this type of observational study are rare. However, its use may be justified in some settings, and it has been used to research sensitive topics such as homosexuality 3 and difficult to access areas such as fascist organisations 4 and football hooliganism. 5 Overt research—Gold's “participant as observer”—may pose fewer ethical dilemmas, but this may be offset by the group or individuals reacting to being observed. At its most basic, having a researcher observing actions may stimulate modifications in behaviour or action—the so-called “Hawthorne effect,” 6 or encourage introspection or self questioning among those being researched. In his classic study of street gangs in the United States, Whyte recounted how a key group member said, “You've slowed me up plenty since you've been down here. Now when I do something I have to think what Bill Whyte would want to know about it and how I can explain it. Before I used to do things by instinct.” 7

In addition to these potential problems for the subjects of observational research, there are important considerations for researchers “entering the field.” In essence these involve “getting in and getting out.” In the initial phases there may be problems gaining access to a setting, and then in striking up sufficient rapport and empathy with the group to enable research to be conducted. In medical settings, such as a hospital ward, this may involve negotiating with several different staff groups ranging from consultants and junior doctors, to nurse managers, staff nurses, social workers, and auxiliary professions. Once “inside” there is the problem of avoiding “going native”; that is, becoming so immersed in the group culture that the research agenda is lost, or that it becomes extremely difficult or emotionally draining to exit the field and conclude the data collection.

Observation of transactions with patients presenting to casualty departments found that staff classified patients into “normal rubbish” (the inappropriate attenders) and “good” patients, who were viewed as more deserving.


What can observation tell us that other methods cannot?

Given these difficulties, observational methods may seem a peculiar choice for studying health and health services. However, an important advantage of observation is that it can help to overcome the discrepancy between what people say and what they actually do. It circumvents the biases inherent in the accounts people give of their actions caused by factors such as the wish to present themselves in a good light, differences in recall, selectivity, and the influences of the roles they occupy. For these reasons, observational methods are particularly well suited to the study of the working of organisations and how the people within them perform their functions. It may also uncover behaviours or routines of which the participants themselves may be unaware. For example, Jeffery's observation of casualty wards in Edinburgh indicated that, because of the conflicting demands and pressures on staff, some patients, who were seen as inappropriate attenders, were labelled as “normal rubbish” and treated differently from “good” patients, who were viewed as more deserving. 8 A similar picture emerges from Hughes's work on the decisions made by reception clerks when patients present themselves at casualty department. 9 It is unlikely that interviews alone would have elicited these different patterns of care. Indeed the labelling of certain cases as “normal rubbish” may have been so embedded in the culture of the casualty setting that only an outsider or newcomer to the scene would have considered it noteworthy.

Another observational study provides an example of how qualitative work can build on existing quantitative research. 10 Against the background of large variations in rates of common surgical procedures such as hysterectomy, cholecystectomy, and tonsillectomy, Bloor observed ear, nose, and throat outpatient clinics to see how decisions to admit children for surgery were made. He systematically analysed how surgeons made their decisions to operate and discovered that individual doctors had different “rules of thumb” for coming to a decision. While one surgeon might take clinical signs as the chief indication for surgery, another might be prepared to operate in the absence of such indications at the time of consultation if there was evidence that repeated episodes of tonsillitis were severely affecting a child's education. Understanding the behaviour of these surgeons, knowing why they made their decisions, provided considerable insight into how the variation in surgical rates occurred.

Similar variation and patterning occurs in the statistics on inpatient waiting lists: some surgeons have long lists, others do not; some specialties have long waits, others do not. An observational study showed that rules and routines akin to those discovered by Bloor could be discerned in the day to day management of waiting lists. 11 Surgical and administrative preferences were important in deciding who came off the list. Different reasons for admitting a patient might range from case mix demands for teaching juniors, through ensuring a balanced list, to the ease with which a patient could be contacted and offered admission. Thus, observing how waiting lists work can indicate which policy and administrative changes are likely to have an impact in reducing lists and which are not: a policy which assumed that waiting lists operated as first come, first served queues would be unlikely to affect the day to day routines described above.

Some rules about observation

Before any recording and analysis can take place, the setting to be observed has to be chosen. As in other qualitative research, this sampling is seldom statistically based. Instead, it is likely to be purposive, whereby the researcher deliberately samples a particular group or setting (see Mays and Pope 12 in this series for more on this). The idea of this type of sampling is not to generalise to the whole population but to indicate common links or categories shared between the setting observed and others like it. At its most powerful, the single case can demonstrate features or provide categories relevant to a wide number of settings. Goffman's observation of mental hospitals in the 1960s generated the valuable concept of the “total institution,” of which the asylum was one example alongside others such as prisons and monasteries. 1

Qualitative observation involves watching and recording what people say and do. As it is impossible to record everything, this process is inevitably selective and relies heavily on the researcher to act as the research instrument and document the world he or she observes. Therefore it is vital that the observations are systematically recorded and analysed, either through the traditional medium of field notes written during or immediately after the events occur or by using audio or video recording facilities. From his unique position as a patient in a tuberculosis sanatorium, Roth was able to record events as they happened, 13 but such situations are rare and most researchers, whether in covert or more participative roles, find that recording necessitates the development of memory skills and frequent trips to the lavatory to “write up.”

The systematic recording of data in qualitative observation distinguishes it from other types of observation such as a tourist recording with a camcorder or a nosey neighbour peering over the fence. Even with video and sound recording it is impossible to “get everything,” but as far as possible the researcher aims to record exactly what happened, including his or her own feelings and responses to the situations witnessed. The subjective nature of this type of research contrasts with the objective stance aspired to in the experimental method, but in fact it is a crucial component of the process of analysing qualitative observational data. The researcher usually keeps a field diary or record of the research process to detail events, personal reactions to events, and changes in his or her views over time. Frequently this is the basis of tentative hypotheses or the evolution of systems of classification. In developing classifications or hypotheses it is particularly important to detail any contradictory or negative cases—the unusual, out of the ordinary things which often reveal most about the setting or situation. Tentative classifications and the search for negative cases during the data collection are important facets of the analytic technique used in observational research.

The fieldnotes gathered during observational research are likely to be detailed, highly descriptive accounts and are therefore cumbersome. As descriptions alone they cannot provide explanations. The researcher's task is to sift and decode the data to make sense of the situation, events, and interactions observed. Often this analytical process starts during the data collection phase, a quite different model of the research process to that found in quantitative research, where data collection is completed before any analysis begins (box 2).

Just as the data are systematically recorded, so they are also systematically analysed. Various ways of dealing with observational data have been described, including “analytic induction” and “constant comparison.” 14 Stripped of their theoretical trappings, these methods are all variants of content analysis and involve an iterative process of developing categories from the transcripts or fieldnotes, testing them against hypotheses, and refining them. This analytical process is described in detail by Bloor, based on the observational study of ear, nose, and throat clinics described earlier (box 3). 15

Box 3 AnalysisStages in the analysis of field notes in a qualitative study of ear, nose, and throat surgeons' disposal decisions for children referred for possible tonsillectomy and adenoidectomy (T&A) 11

Provisional classification—For each surgeon all cases categorised according to the disposal category used (for example, T&A or tonsillectomy alone)

Identification of provisional case features—Common features of cases in each disposal category identified (for example, most T&A cases found to have three main clinical signs present)

Scrutiny of deviant cases—Include in (2) or modify

to accommodate deviant cases (for example, T&A performed when only two of three signs present)

Identification of shared case features—Features common to other disposal categories (history of several episodes of tonsillitis, for example)

Derivation of surgeons' decision rules—From the common case features (for example, case history more important than physical examination)

Derivation of surgeons' search procedures (for each decision rule)—The particular clinical signs looked for by each surgeon

Repeat (2) to (6) for each disposal category

As with quantitative work, it is important that evidence from the data is presented to support the conclusions reached. This can take the form of examples of specific cases, descriptions of events, or quotations. The validity of observational accounts relies on the truthful and systematic representation of the research; in many ways it is honesty which separates the observational account from a novel. Hughes says that observational studies should communicate the culture and rules of the setting well enough to allow another researcher to learn them and “pass” as a member of the group. 16 This is not an easy task, and observational research is therefore particularly demanding of the individual researcher.

This brief review has indicated how observational methods can be used to “reach the parts that other methods cannot.” Done well, there is no reason why observation should not be as systematic, rigorous, or valid as other research styles and deserve its place in the health researcher's methodological tool box.

Further reading

Fielding N. Researching social life. London: Sage, 1993.

  • Humphreys L
  • Roethlisberger FJ ,

observational research health care

We have a new app!

Take the Access library with you wherever you go—easy access to books, videos, images, podcasts, personalized features, and more.

Download the Access App here: iOS and Android . Learn more here!

  • Remote Access
  • Save figures into PowerPoint
  • Download tables as PDFs

Understanding Clinical Research

Chapter 10. Observational Research

Michaela A. Dinan

  • Download Chapter PDF

Disclaimer: These citations have been automatically generated based on the information we have and it may not be 100% accurate. Please consult the latest official manual style if you have any questions regarding the format accuracy.

Download citation file:

  • Search Book

Jump to a Section

  • Observational Research: Introduction
  • Full Chapter
  • Supplementary Content

Clinical research can be broadly divided into two subsets: experimental research and observational research. The vast majority of new medical treatments and technologies are tested through experimental or interventional research, often in the form of randomized trials, before they are adopted into clinical use. In contrast, observational studies are primarily conducted on technologies after they have already been adopted and are being implemented in some sector of the healthcare community. Observational research occupies a critical niche within healthcare research that is complementary to experimental studies. Understanding the relative strengths, weaknesses, similarities, and differences between observational and experimental research is critical to accurately interpreting clinical research.

A Working Definition of Observational Research

Observational research is a research in which the investigator cannot control the assignment of treatment to subjects because the participants or conditions are not being directly assigned by the researcher. Observational research examines predetermined treatments, interventions, and policies and their effects. In practical terms, observational comparative effectiveness research (CER) is typically conducted within one of two settings, either within registries or as subgroup analyses within randomized clinical trials. Registries are generally created with a specific disease, treatment, or population of interest, and can occur within a specific institution, network of institutions, or geographic region within which clinically relevant outcomes are recorded. Subgroup analyses within clinical trials include any subset for which patients are not randomly assigned. Because subgroups are not randomly assigned, subgroup analyses share all the strengths and weaknesses of conventional observational studies, such as confounding and multiple hypotheses testing, and provide a similar level of evidence.

In contrast to observational research, researchers in experimental studies directly manipulate or assign participants to different interventions or environments. A third type of research involves descriptive studies, which are conducted without a treatment and are neither experimental nor observational ( 1 ). This type of research is used in the initial exploration and characterization of a healthcare issue. Descriptive studies play no direct role in CER, whereas experimental and observational studies are important in both developmental and CER.

Strengths and Weaknesses of Observational Research Within Comparative Effectiveness Research

Recent focus on the importance of CER was reinvigorated with passage of the Patient Protection and Affordable Care Act (PPACA) in 2010 ( 2 ). From a practical research standpoint, this emphasis on CER makes it important to define and understand observational research within the context of CER.

Your Access profile is currently affiliated with '[InstitutionA]' and is in the process of switching affiliations to '[InstitutionB]'. Please click ‘Continue’ to continue the affiliation switch, otherwise click ‘Cancel’ to cancel signing in.

Sign in or create a free Access profile below to access even more exclusive content.

With an Access profile, you can save and manage favorites from your personal dashboard, complete case quizzes, review Q&A, and take these feature on the go with our Access app.

Pop-up div Successfully Displayed

This div only appears when the trigger link is hovered over. Otherwise it is hidden from view.

Please Wait


  • Mission, Vision & Values
  • Areas of Focus
  • Who We Serve
  • Collaborators
  • Titan Awards
  • Regional Chapters
  • Website Privacy Policy Notice
  • Community News & Updates
  • Collaborator Spotlights
  • Software Tools
  • Open-Source Tutorials
  • Ongoing OHDSI Studies
  • OHDSI Authorship Guidelines
  • Forums Homepage
  • OHDSI Forum Best Practices
  • Book of OHDSI (English Version)
  • Book of OHDSI (Korean Version)
  • Book of OHDSI (Chinese Version)
  • Order The Book of OHDSI (Amazon)
  • EHDEN Academy
  • OHDSI2022 Tutorial: An Introductory Journey From Data To Evidence
  • YouTube: Tutorials and Presentations
  • ATLAS Tutorials
  • Past Presentations (2015-2021)
  • Join Our Microsoft Teams Environment
  • Join A Forum Discussion
  • Introduce Yourself!
  • Upcoming Workgroup Calls
  • Video: What Is OHDSI?
  • Learn More: The Book Of OHDSI
  • 2024 Community Calls
  • APAC Community Calls
  • 2023 Community Calls
  • 2022 Community Calls
  • 2021 Community Calls
  • 2023 Global Collaborator Showcase
  • 2023 APAC Symposium
  • 2023 Europe Symposium Collaborator Showcase
  • 2023 SOS Challenge
  • 2023 DevCon
  • 2023 Phenotype Phebruary
  • Tutorial: An Introductory Journey From Data To Evidence
  • 2022 Collaborator Showcase
  • Plenary Videos, Panels, Closing, Lightning Talks
  • All Collaborator Showcase Presentations
  • Links To All Presentation Videos, Slides
  • 2020 Global Symposium Collaborator Showcase
  • ALL Past OHDSI Events
  • Learn About Our Workgroups
  • Join Our Teams Environment
  • Join Our Workgroups
  • Workgroup Call Schedule
  • Best Practices in MS Teams
  • 2023 ‘Our Journey’ Annual Report
  • This Week In OHDSI
  • Support & Sponsorship
  • Phenotype Phebruary 2024
  • 2023 Global Symposium
  • February 2024
  • January 2024
  • December 2023
  • November 2023
  • October 2023
  • September 2023
  • Full Archive

Welcome to OHDSI!

The Observational Health Data Sciences and Informatics (or OHDSI, pronounced "Odyssey") program is a multi-stakeholder, interdisciplinary collaborative to bring out the value of health data through large-scale analytics. All our solutions are open-source.

OHDSI has established an international network of researchers and observational health databases with a central coordinating center housed at Columbia University.

Read more about us , about  our goals , and how you can help support the OHDSI community .

2023 Global Symposium Talks, Posters Now Available

Thank you to everybody who joined us for the 2023 Global Symposium. All talks, posters, demos and more are now available on the symposium homepage.

2023 OHDSI Global Symposium

What OHDSI Is All About

OHDSI’s Mission

Latest news.

  • Weekly OHDSI Digest – February 19, 2024
  • Weekly OHDSI Digest – February 12, 2004
  • Weekly OHDSI Digest – February 5, 2004
  • Weekly OHDSI Digest – January 29, 2004


Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • View all journals
  • My Account Login
  • Explore content
  • About the journal
  • Publish with us
  • Sign up for alerts
  • Open access
  • Published: 22 February 2024

Ethnicity data resource in population-wide health records: completeness, coverage and granularity of diversity

  • Marta Pineda-Moncusí   ORCID: orcid.org/0000-0003-0567-0137 1 ,
  • Freya Allery 2 ,
  • Antonella Delmestri   ORCID: orcid.org/0000-0003-0388-3403 1 ,
  • Thomas Bolton 3 ,
  • John Nolan 3 ,
  • Johan H. Thygesen   ORCID: orcid.org/0000-0002-7479-3459 2 ,
  • Alex Handy 2 ,
  • Amitava Banerjee 2 ,
  • Spiros Denaxas 2 , 3 , 4 ,
  • Christopher Tomlinson   ORCID: orcid.org/0000-0002-0903-5395 2 , 4 , 5 ,
  • Alastair K. Denniston 6 ,
  • Cathie Sudlow 3 ,
  • Ashley Akbari   ORCID: orcid.org/0000-0003-0814-0801 7 ,
  • Angela Wood 3 , 8 , 9 ,
  • Gary S. Collins   ORCID: orcid.org/0000-0002-2772-2316 1 ,
  • Irene Petersen 10 , 11 ,
  • Laura C. Coates 12 ,
  • Kamlesh Khunti 13 ,
  • Daniel Prieto-sAlhambra 1 , 14 &
  • Sara Khalid 1

on behalf of the CVD-COVID-UK/COVID-IMPACT Consortium

Scientific Data volume  11 , Article number:  221 ( 2024 ) Cite this article

Metrics details

  • Epidemiology

Intersectional social determinants including ethnicity are vital in health research. We curated a population-wide data resource of self-identified ethnicity data from over 60 million individuals in England primary care, linking it to hospital records. We assessed ethnicity data in terms of completeness, consistency, and granularity and found one in ten individuals do not have ethnicity information recorded in primary care. By linking to hospital records, ethnicity data were completed for 94% of individuals. By reconciling SNOMED-CT concepts and census-level categories into a consistent hierarchy, we organised more than 250 ethnicity sub-groups including and beyond “White”, “Black”, “Asian”, “Mixed” and “Other, and found them to be distributed in proportions similar to the general population. This large observational dataset presents an algorithmic hierarchy to represent self-identified ethnicity data collected across heterogeneous healthcare settings. Accurate and easily accessible ethnicity data can lead to a better understanding of population diversity, which is important to address disparities and influence policy recommendations that can translate into better, fairer health for all.


Health inequity is described by disparities in health status between individuals, such as prevalence of comorbidities, life expectancy, access to and quality of care services and treatments, and risk behaviours such as smoking and alcohol consumption. These factors can be influenced by age, sex, ethnicity, disability, socio-economic status, geographical location, and education, among others 1 . For example, many of these determinants were risk factors for infection severity, complications, and mortality during the COVID-19 pandemic 2 , 3 , 4 .

“Ethnicity” commonly refers to terms used to self-report an individual’s own perceived ethnic group and cultural background. This multidimensional, evolving concept can comprise physical appearance, race, culture, language, religion, nationality and identity elements, and is not always captured in electronic health records. Additionally, when recorded, ethnicity is often inaccurately coded, especially for groups other than the predominant group(s) in a given population 5 , 6 . Ethnicity classifications also change over time, limiting comparability with population-level census data 7 . In UK health records, although there are hundreds of heterogenous ethnicity groups defined in the form of SNOMED-CT codes or Read codes, among others, they are commonly collapsed into five or six categories, in part due to power considerations where fine-grained or granular categories would have smaller sample sizes 8 , 9 , 10 , 11 . However, these larger groups may not be equivalent or translate across the world due to differences in population demographics 12 , 13 . Nonetheless, this oversimplification of categories can result in loss of diversity and precision in studies using ethnicity. Incorrect or unrepresentative ethnicity records risk introducing bias in insights drawn from health data and ensuing literature, ultimately contributing to inappropriate healthcare. Use of population-wide routinely-collected data offers an opportunity to study diverse ethnicity groups in detail with sufficient power, enabling health research to become more inclusive 4 , 10 , 11 .

Health inequality was highlighted as a significant issue during the COVID-19 pandemic when individuals from ethnically diverse backgrounds in otherwise predominantly White populations were disproportionately affected by SARS-CoV-2 11 . However, this is an ongoing and multifaceted challenge; one underlying source is bias in health data and ensuing technologies. Understanding and addressing biases in health data is a fundamental first step in addressing this challenge. To improve the understanding of how ethnicity is recorded, mapped, and used in the UK, we explored ethnicity records for completeness, consistency, and granularity in National Health Service (NHS) England’s Secure Data Environment (SDE) service for England (UK) accessed via the BHF Data Science Centre’s CVD-COVID-UK/COVID-IMPACT Consortium 14 .

Data sources and linkages

NHS England maintains an SDE for secure access to anonymised patient-level electronic health records for England with linkages to primary-, secondary-, and tertiary-care data sources for research purposes 15 , 16 . NHS England’s Master Person Service facilitates the linkage between the SDE data sources through the NHS number (a unique 10 digit healthcare identifier), date of birth, and sex 15 , 17 .

This study focused on the General Practice Extraction Service (GPES) Data for Pandemic Planning and Research (GDPPR) data sources, a primary-care dataset for England that collects information from all individuals who are currently registered with a general practitioner (GP) practice and any individual who died on or after 1 st November 2019. GDPPR does not include individuals who died before November 2019 for ethical reasons as those individuals are considered out of scope for COVID-19 research. It is accessible through the NHS England SDE 18 (formerly NHS Digital TRE 15 , 16 ). Data include diagnoses, prescriptions, treatments, outcomes, vaccinations, and immunisations 19 , 20 . GDPPR covers 98% of English GP practices across all relevant GP computer system suppliers (TPP, EMIS, Cegedim (formerly called Vision or In Practice Systems), and Microtest) 15 .

Data sets used

To evaluate and curate the ethnicity data available in the NHS England SDE 15 , 16 , we used the following three linked datasets:

Primary care data: the General Practice Extraction Service (GPES) Data for Pandemic Planning and Research (GDPPR) 19 , 20 .

Hospital admissions data: Hospital Episode Statistics for admitted patient care (HES-APC)

Mortality information from the Office for National Statistics (ONS): Civil Registration of Deaths.

GDPPR was used to select the individuals included in the study. It was the main source to obtain ethnicity data, and all variables included in the study except death, which was obtained from the Civil Registration of Deaths. HES-APC was used as a second source to obtain ethnicity data.

Data access

A data sharing agreement issued by NHS England for the CVD-COVID-UK/COVID-IMPACT research programme (ref: DARS-NIC-381078-Y9C5K) enables accredited, approved researchers from institutions party to the agreement to access data held within the NHS England SDE service for England.

Source codes for ethnicity

Ethnicity is recorded in health records using the following medical concepts (Fig.  1 ):

SNOMED concepts: The Systematized Nomenclature of Medicine Clinical Terms is a standardised vocabulary for the recording of patients’ clinical information in electronic health records. It is used across NHS practices and healthcare providers 21 . We focused on GDPPR records containing SNOMED-CT 22 UK Edition ethnicity concepts. Any mention of SNOMED concepts in this paper directly refers to these codes.

NHS ethnicity codes: Standard ethnicity categories defined in the NHS England Data Dictionary 23 , using A-Z notation. Ethnicity fields in the NHS tables may use different census classifications, therefore NHS ethnicity code notation may differ slightly depending on which census it is based on. Table  1 summarises the NHS ethnicity codes available in GDPPR, the corresponding categories in HES-APC, and the 2011 and 2021 UK ONS census categories. Mapping of SNOMED concepts to NHS ethnicity codes was provided by NHS England (Table  S1 ).

figure 1

How ethnicity is collected in the UK and typically used for research. The A-Z letters are the nomenclature observed in the data to represent the NHS ethnicity codes. Abbreviations: High-level ethnicity groups, general ethnicity classification groups from the Office for National Statistics commonly used in research; NHS, National Health Service in the UK; SNOMED, SNOMED-CT records containing ethnicity concepts.

An individual’s ethnicity may be recorded using either SNOMED-CT concepts or NHS ethnicity codes in GDPPR (primary care records), whereas it may only be recorded using the latter in HES-APC (hospital records).

Other ethnicity classifications

High-level ethnicity groups: Asian/Asian British, Black/African/Caribbean/Black British, Mixed, Other Ethnic Groups, Unknown, and White. Based on ONS ethnicity group high-level category descriptions 13 , 24 .

The algorithm used within the SDE to condense NHS ethnicity codes and SNOMED concepts to these classifications is provided in Table  S2 . Figure  2 shows a representation of the hierarchy between the three classification systems, and points where the aggregation is performed by the mapping provided by NHS England or the algorithm used in the SDE.

figure 2

Visual representation of the hierarchy between the three ethnicity classifications, from the broader to the most specific: High-level ethnicity groups, NHS ethnicity codes and SNOMED concepts. The A-Z letters are the nomenclature observed in the data to represent the NHS ethnicity codes. The colours displayed from the High-level ethnicity groups show how the NHS ethnicity concepts and SNOMED-CT can be aggregated into this 6-category classification. The 1 highlights the different colour for the letters C and T, in respect to the colours of their concepts, Chinese and Gypsy/Irish Traveller, respectively. The colours from the concepts represent the current aggregation algorithm available in the NHS England SDE, whilst the colour of the letters show the aggregation suggested by the UK Office of National Statistics. Abbreviations: *, the Unknown category is not always included; NHS, National Health Service in the UK; SNOMED, SNOMED-CT records containing ethnicity codes; SDE, Secure Data Environment.

Settings and participants

We studied all individuals with a unique patient pseudoidentifier in GDPPR 19 , 20 from 1 st Jan 1997 until 23 rd April 2022. Individuals with an invalid age (i.e., age <0 or ≥115 years old) or missing sex were excluded.

Death date was obtained through civil registration death table which is curated by the ONS and records primary and secondary causes of death using ICD-10. All additional characteristics of individuals were extracted from GDPPR data, which included: age at date of death or age on 23rd April 2022 (date of data extraction), sex, most recent record of residence (i.e., geographical region) in England, body mass index (BMI), index of multiple deprivation (IMD), current smoking status, current alcohol use status, and the presence of any clinical record of atrial fibrillation, acute myocardial infarction, chronic kidney disease, chronic obstructive pulmonary disease (COPD), heart failure, pulmonary embolism, cancer, dementia, diabetes, hypertension, liver disease, obesity, or stroke diagnosis. Geographical region is reported using England’s nine official regions: London, North East, North West, Yorkshire, East Midlands, West Midlands, South East, East, and South West; which were mapped from the Lower Layer Super Output Areas (LSOA).

Statistical analysis

Completeness: missing ethnicity data.

To study completeness, individual-level ethnicity data were extracted from GDPPR using SNOMED concepts and/or NHS ethnicity codes, prioritising SNOMED concepts when available. For individuals with missing ethnicity data in GDPPR, we extracted HES-APC-linked ethnicity data using NHS ethnicity codes (Fig.  3 ).

figure 3

Decision tree of preferred source of ethnicity. Solid arrows mark the preferred option whilst dashed arrows indicate the alternative route. Abbreviations: GDPPR, General Practice Extraction Service (GPES) Data for Pandemic Planning and Research; HES-APC, hospital episode statistics; SNOMED, SNOMED-CT records containing ethnicity codes.

Ethnicity missingness was defined as (i) no record in either GDPPR or HES-APC or (ii) an ethnicity code of “not stated” referring to individuals who were asked but preferred not to state their ethnicity and individuals who may not know what to answer. We compared the clinical characteristics of individuals in GDPPR whose ethnicity data were obtained from GDPPR, from the link to HES-APC, or were not recorded.

Inconsistency: multiple records

We study the presence of multiple ethnicity codes for an individual within the GDPPR records and within the HES-APC records, and we compared the prevalence of individuals with multiple codes in GDPPR and HES-APC.

To study it within the GDPPR records, the individual’s SNOMED concepts were converted into the corresponding 19 NHS ethnicity code categories. For individuals who had two co-existing NHS ethnicity codes, the frequency of each co-existing pair was determined. If a patient had more than two co-existing ethnicity codes present, a count of one was added for each pairing.

Inconsistency: potential discrepancies between classifications

To study potential discrepancies and misclassifications between the different ethnicity classifications, we studied the mappings between SNOMED and NHS ethnicity codes and between NHS ethnicity codes and high-level ethnicity groups.

Granularity: from high-level categories to SNOMED concepts

Here “granularity” refers to the degree of detail, i.e., sub-groups within an ethnicity group. Definitions of the most recent SNOMED record in GDPPR individuals were explored.

Data were prepared using Python V.3.7 and Spark SQL (V.2.4.5) on Databricks Runtime V.6.4 for Machine Learning. Data were analysed using Python in Databricks and RStudio (Professional) Version 1.3.1093.1 driven by R Version 4.0.3.

Completeness of ethnicity data

We identified 61,810,570 individuals with unique identifiers in the GDPPR dataset on 23 rd April 2022. We excluded 403 of these individuals for invalid age or missing sex. Of the remaining, 51,135,903 (83.3%) had an ethnicity code recorded, including those whose ethnicity was recorded but as Unknown, whereas 10,674,667(16.7%) did not have any record for ethnicity. The recorded ethnicity groups included White (77.3%), Asian/Asian British (9.8%), Black/Black British (3.6%), Other Ethnic Groups (3.6%, Mixed (2.2%), and Unknown ethnicity (3.2%) (Figure  S1 ). When linked with HES-APC, the proportion of those without any ethnicity record reduced from 16.7% to 6.1% (Fig.  4 ).

figure 4

Flow chart of availability of ethnicity records for individuals present in GDPPR. Abbreviations: GDPPR, General Practice Extraction Service (GPES) Data for Pandemic Planning and Research; HES-APC, hospital episode statistics; NHS, National Health Service in the UK; SNOMED, SNOMED-CT records containing ethnicity concepts; NA, not available ethnicity.

Individuals with missing ethnicity data were generally younger, with a median age [IQR] of 35·0 [22·0, 53·0] years vs 42·0 [24·0, 61·0] years for those with ethnicity from GDPPR and 36·0 [18·0, 58·0] years for those with ethnicity linked from HES-APC. A greater proportion of those with missing ethnicity were male (58·6%) than those with ethnicity from GDPPR (48·9% male) or HES-APC (54·0% male) (Table  2 ). They also had fewer comorbidities (Table  3 ) and a greater proportion came from the South East and South West regions of England (Figure  S2 ). Individuals aged 18–29 years had between 5·7% and 9% more missing ethnicity data than any other age group (Table  2 ).

Assessment of multiple ethnicity records

About 1·4% of individuals with an original NHS ethnicity code record and 16·0% of individuals with a converted SNOMED concept record had multiple different ethnicity codes (Table  S3 ). Excluding the Not stated (Z) code reduced inconsistencies (to 1·2% and 10·3%, respectively). In contrast, 38·0% of individuals in GDPPR with at least one ethnicity record in HES-APC (n = 46,804,958) had multiple inconsistent records, dropping to 19·0% when the Not stated (Z) code was excluded.

Ethnicity codes most frequently found in individuals with more than one reported code in GDPPR were British (A), Any other White background (C), Not stated (Z) , Any other ethnic group (S), and Any other Asian background (L) (Figure  S3 ). The most common ethnicity code combinations were British (A) – Any other White background (C), British (A) – Not stated (Z), and Any other White background (C) – Any other ethnic group (S). When White ethnicity codes (A, B, and C) were excluded, the most common pairs of minority ethnicity codes were Any other Asian background (L) – Any other ethnic group (S), African (N) – Any other Black background (P), and Indian (H) – Any other Asian background (L) (Figure  S4 ).

Granularity of ethnicity data

Figure  S2 maps the distinct levels of ethnicity concepts from the different data sources to one another. SNOMED currently gives the most granular ethnicity records, with 489 SNOMED concepts representing ethnicity within the NHS England SDE (Table  S1 ). However, only 255 (52·1%) of these codes were used at least once in the extracted individuals’ records. The remaining 234 (47·8%) codes were not assigned to any individual. Figure  S5 shows the five most frequently used SNOMED concepts mapped to each NHS ethnicity code. Table  S4 displays the number of individuals per SNOMED concept in GDPPR.

Diversity in SNOMED concepts

SNOMED concepts were substantially diverse. Of the 255 codes in use, 162 (63·5%) contained an ethnicity/race concept, 5 (2·0%) included a religion, 187 (73·3%) referenced a geographical region, and 60 (23·5%) referenced a language. Full list is available at Table  S5 .

Potential discrepancies and misclassifications

Some inconsistency was found in the aggregation of NHS ethnicity categories into the high-level ethnicity groups. According to the 2011 and 2021 England and Wales census classifications, Gypsy/Irish Traveller (T) falls within the higher-level category White , and Chinese (R) within Asian . In contrast, the NHS ethnicity classification included Gypsy/Irish Traveller (T) and Chinese (R) within the higher-level category Other Ethnic Groups , following the ONS Census 2001 classification (see classification algorithm used in the CVD-COVID-UK/COVID-IMPACT Consortium in Table  S2 ).

Further discrepancies were found in the grouping of SNOMED concepts into NHS ethnicity categories. A mapping algorithm could not be traced. Given the lack of documentation on the mapping of SNOMED concepts to NHS ethnicity code, several potential discrepancies were observed which should be carefully (re)considered by researchers in future (Fig.  5 ): for instance, concepts including a variant of “Black East African Asian/Indo-Caribbean” were assigned to Any other Asian background (L). However, there is no clarification as to whether these concepts are mapped more accurately there than others, such as Other ethnic group or Black/African/Caribbean/Black British . Likewise, variants of “Black West Indian” were mapped to Caribbean (M), although a proportion of these individuals may have Asian legacy. Arguably, the three ‘Mixed’ concepts within Any other background (G) may be better grouped in more specific categories, such as “Black - other, mixed” within Black background (P). Several concepts contained by Any other ethnic group (S) could also be placed in more specific categories. For example, the 2001 census category “Asian and Chinese” is linked to Any other ethnic group (S) instead of Chinese (R) or Any other Asian background (L). Similarly, some SNOMED concepts include the concept “Roma”. As the ONS 2021 census included the new category White: Roma , the mapping could be updated to reflect this change.

figure 5

Sankey plot showing potential discrepancies between SNOMED concepts and NHS ethnicity codes mapping. Abbreviations: NHS, National Health Service in the UK; SNOMED, SNOMED-CT records containing ethnicity concepts.

Facilitating data reuse in future research

The curation process has organised ethnicity data into a hierarchical mapping. This allows the data source to be reused by future researchers using ethnicity information. The publicly available R code can be used to extract the most up-to-date ethnicity records for future research. This allows the necessary flexibility for using the data in observational research such as retrospective cohort studies, as well as potentially help with clinical trial selection such as NHS DigiTrials service.

Errors in health care can impact patient care and outcomes as well as increase costs to the care system 25 and affect public trust 26 . Biased ethnicity knowledge could potentially lead to biased healthcare decision-making and to patients receiving inappropriate or no care. Correct identification of ethnicity is an essential first step to understanding inequities between ethnicities. Despite its complexity, researchers should aim to include ethnicity in their analyses. The results presented here can be used to further the use of ethnicity in future research.

Among those whose ethnicity was recorded, the proportion of individuals with White Black/Black British and Mixed ethnicity were −3.7%, −0.6% and −0.7% lower, respectively, whilst Asian/Asian British and Other Ethnic groups were and 0.2% and 1.4% higher, as compared to 2021 census estimates 27 . Therefore we consider GDPPR as a representative data source for the England population.

Completeness of ethnicity records

We found that over 83·3% of individuals in England’s primary care system had at least one ethnicity recorded; increasing to 93·9% when linked to hospitalisation records. This result represents a greater level of completeness than reported in other routinely collected GP records 28 , and highlights the usefulness of linking data across primary and secondary care to maximise ethnicity data completeness. Individuals with missing ethnicity were younger, more likely to be male and living in the southern regions of England, and had fewer comorbidities than individuals with recorded ethnicity. It may be speculated that this group may be representative of generally healthy individuals or those otherwise not inclined to seek healthcare. In other words, most of individuals with Unknown ethnicity might not be using the health care system very often, which decreases the probability to record data. Similar results have been reported in other UK data sources. For instance, Mathur et al . observed higher rates of ethnicity records for individuals aged 40 to 79 years in the Clinical Practice Research Datalink (CPRD) and HES data sources than older or younger individuals 8 . Petersen at al. found that, among people aged 18–65 years, men were less likely to have health indicators recorded than women in the Health Improvement Network (THIN) 29 .

Most studies collapse the available ethnicity concepts into five (e.g., Asian, Black/African/Caribbean, White, Mixed, Other Ethnic Groups) or six (the aforementioned and Unknown ethnicity) categories 11 . However, some studies have accounted for greater distinctions by exploring ethnic minorities such as Bangladeshi in the UK and Hispanic/Latinos in the US 30 . This work describes for the first time more than 250 patient-identified ethnicity sub-groups in England.

Ethnicity data in healthcare and national statistics are captured for different purposes, partly explaining some of the differences among NHS and census ethnicity categories. For instance, GDPPR is directly intended for patient care; HES has a more administrative nature linked to payments; while the ONS collects information from the UK population, including ethnicity, in a census held approximately every 10 years, most recently in 2021 7 , 13 . To allow for the emergence of new ethnicity groups, the census questionnaire allows free-text answers 13 . After pooling all information, the ONS reports the groups that, in their understanding, best represent the existing diversity in the UK. Updated ethnicity groups are then shared with the NHS, which uses this information to update the ethnicity categories used in their data sources.

The 2011 Census published by the ONS was the gold standard for ethnicity recording in England and Wales until the recent publication of the 2021 Census 31 . However, not all NHS sources base their categories on the same census. For example, HES-APC uses 2001 Census categories, whereas GDPPR uses 2011 census categories. This discrepancy creates uncertainties and data mismatching between different datasets. For example, HES-APC does not include the ethnicity categories Arab (W) and Gypsy/Irish Traveller (T) 32 . This highlights once more the importance of linking data across primary and secondary care, in this case, to maximise ethnicity data granularity.

Despite differences, we can compare the prevalence of SNOMED concepts used in GDPPR to the 2019 population estimates in England and Wales (2019) 33 . Fewer individuals self-identified as White British (66·8% in GDPPR vs 78·4% in 2019 estimates), Gypsy/Traveller (0·1% vs· 0·03%), or Arab (0·2% vs 0·4%). Higher proportions of individuals self-identified as Chinese (1·2% vs 0·6%), Indian (3·7% vs· 2·8%), Pakistan (2·9% vs 2·3%), or Any other mixed background (0·8% vs 0·5%). And similar percentages self-identified as Bangladeshi (1·1% vs 1·0%), Caribbean (0·9% vs 1·0%), African (2·4% vs 2·3%), White and Asian (0·5% both), White and Black African (0·4% vs 0·3%), or White and Black Caribbean (0·5% both). The 2019 estimates did not include people who had died or individuals with an unknown ethnicity, which might account for these differences.

Multiple records and potential discrepancies

Our analysis of multiple ethnicity records within GDPPR and HES-APC sources showed relatively similar discrepancy rates when the code for “do not know/refusal” (i.e., Not stated (Z)) was excluded (12% and 19%, respectively). However, GDPPR ethnicity data should be prioritised to ensure inclusion of Arab (W) and Gypsy/Irish Traveller (T) ethnicities, as well as to reduce the inclusion of older notations such as the “Codes from 1995–1996 to 2000–2001”. Using the prioritisation algorithm from our analysis, the impact of this legacy classifications represented less than the 0.13% of individuals with an ethnicity record and less than 0.12% of all individuals registered in GDPPR. Nonetheless, we considered that using an old classification system is preferred, rather than registering it as missing ethnicity, but researchers may decide differently on a project basis.

Within patient records, the most frequently coexisting codes would be placed in the same higher-order group. For example, codes for African (N) and Any other Black background (P) often appear for the same individual and would both be grouped within the high-level category Black/African/Caribbean/Black British. The use of higher-level groupings can therefore resolve some conflicting cases by reducing granularity. However, it cannot resolve conflicts where different Mixed categories coexist in the same record, such as Any Other Mixed Background (G) occurring alongside British (A), Any Other White Background (C), or African (N). Higher-level Mixed groupings may therefore include more ambiguous ethnicity concepts. The British (A) code had frequent conflicting pairings with Indian (H), Any other Asian background (L), and Caribbean (M), suggesting inconsistencies in individuals’ perceptions of their nationality and ethnicity when self-reporting ethnicity.

The grouping algorithm used can also be a source of inconsistencies. For instance, including Chinese (R) and Gypsy/Irish Traveller (T) within Other Ethnic Groups instead of the established high-level ethnicity groups might be preferred for certain studies, but should not be by default.

Uncertainty regarding mapping of international SNOMED-CT ethnicity concepts to NHS ethnicity codes highlighted the need for better documentation of underlying processes. SNOMED concepts available in GDPPR data account for different, more granular ethnicity groups than NHS ethnicity codes, enabling greater diversity in ethnicity groups to be represented. The descriptions of the observed SNOMED concepts included ethnicity, race, religion, and geographic location, among others. However, many concepts require some aggregation due to their limited use within very large datasets, such as the one explored here. Most research based on NHS data uses wider categories, rather than the highly specific concepts captured by the SNOMED concepts. The large variety and complexity of ethnicity codes can make collapsing and comparing codes difficult, regardless of whether NHS ethnicity codes or high-level ethnicity groups are used. Although using these more general groupings allows researchers to achieve a minimum sample size while protecting individual identities, the cost is the uncertainty of how accurate these bigger groups are. In addition to the improvement of mapping quality, having better documentation of how the observed SNOMED concepts were defined could help in the comparison of these groups with classifications from other countries.

Strengths and limitations of this study

To our knowledge, this work is the first attempt to curate and describe the full breadth and depth of patient self-identified ethnicity using more than 250 ethnicities among over 61 million individuals in England.

GDPPR is a collection of de-identified person-level primary care data (linked to secondary and tertiary care) for one of the world’s largest research-ready population-wide electronic health records databases and housed within a trusted research environment, NHS England’s SDE service for England. This extensive observational dataset, with its large number of ethnicity groups and sub-groups, has the potential to deepen our understanding of ethnicity in health data and its potential to improve real-world evidence generation.

Despite the exclusion of individuals who died before 1 st November 2019, GDPPR can provide a reliable picture of the existing ethnic diversity for studies including individuals registered in the England primary care system after November 2019, like ours. We considered GDPPR as a representative data source for the England population diversity when compared to the UK 2021 census 27 . The slight variations that may be observed, with 1.4% higher representation of Other Ethnic groups being the largest difference, may be explained by our decision not to restrict our analysis to only alive individuals like most researchers would use in their research (in other words, we include all individuals registered in GDPPR with valid inclusion criteria, including those individuals who died between November 2019 and April 2022). However, studies aiming to analyse the diversity of the population before this date may be biased and, therefore, not be representative.

This study provides a first, detailed curation of ethnicity data for re/use in research. The observed findings are highly representative of the England population: in England, there was a total of 6,700 GP practices containing a total number of 60,389,925 unique NHS identifier from patients who were not deceased by 24 August 2020 34 . Of them, 6,535 GP practices containing 56,441,600 unique identifiers were included within GDPPR. Nevertheless, we do not disregard the possibility that patients registered at multiple practices with different identifiers could been counted more than once. However, the impact of this is reduced by the Master Person Service algorithm, which increases the quality of the data by matching and linking person-records within and across the different NHS sources 17 In other words, the algorithm links the different NHS identifiers from the same single patient not only within GDPPR but also across other linked datasets such as HES-APC tables, and assigns a unique anonymised identifier (named Person_ID within the NHS England SDE) that is later used by the SDE user. Further studies are required to assess its accuracy in GDPPR records.

Ethnic diversity is better captured in SNOMED concepts than other existing classifications. Future observational research with study-specific sample sizes may need to consider combining smaller ethnicity categories into larger groups for study feasibility. The detailed representation of the England population described here also means the observed ethnicity groups may not be equivalent or transportable to other countries. Additionally, there is no perfect solution for conflicting codes in the same individual, especially for codes that cannot be reconciled (e.g., White, Black). Of the different available approaches, we used the most recent SNOMED concepts in an individual’s record when exploring granularity in GDPPR. This approach may have affected the prevalence of very small minority groups, including the 234 codes that were not linked to any user. An alternative approach would have been to select the most frequently recorded ethnicity category, which could reduce any potential human error when entering the data into the electronic health system. However, using the most recent codes has the advantage to include any new ethnicity definitions within SNOMED, allowing us to observe a more up-to-date representation of the self-perception of ethnicity of the population.

Whilst improvements to data collection at source would be welcome, much more can be done with currently available ethnicity data than is typically seen in the literature. This is important to be done now more than ever, as routinely collected ethnicity data are increasingly used in the era of real-world analytics and large-scale trials. For instance, improvements required in ethnicity mapping between classifications were identified in this paper. Additionally, this study demonstrates the importance of linking data across primary and secondary care to maximise the ascertainment, completeness, and granularity of ethnicity data, and the application of better ethnicity coding in big health data.

The details of using ethnicity provided in this paper may not only help researchers to improve the representation of the population diversity in their research, but can also be used to conduct much more personalised medicine such as tailoring prognostic models to the 19 ethnicity groups. Accurate ethnicity data will lead to a better understanding of individual diversity, which will help to address disparities and influence policy recommendations that can translate into better, fairer health for all. This, in turn, shows that the effort of collecting ethnicity and using it in research is more than worthwhile.

Ethical approval

The North East – Newcastle and North Tyneside 2 research ethics committee provided ethical approval for the CVD-COVID-UK/COVID-IMPACT research programme (REC no: 20/NE/0161) to access, within secure trusted research environments, unconsented, whole-population, de-identified data from electronic health records collected as part of patients’ routine healthcare. Our project (proposal CCU037, short title: Minimising bias in ethnicity data) agreed to the objectives of the consortium’s ethical and regulatory approvals and was authorised by the BHF Data Science Centre’s Approvals and Oversight Board. Approved researchers (MPM, AD, SK) conducted the analyses within the NHS England’s SDE via secure remote access. Ensuring the anonymity of individuals, only summarised-aggregated results that were manually reviewed by the NHS England ‘safe outputs’ escrow service were exported from the SDE.

Patient and public involvement

A panel of four PPI members were recruited to work on the study team and oversee the project. In addition, a wider stakeholder group representing a wide range of different ethnicities was recruited for three online meetings to get input into the study design, review initial results and finally to consider how to disseminate these results to the public. This group led on the design of a poster and infographic to share the results with the public and encourage them to “Be proud of your ethnicity”.

Data availability

The data used in this study are available in NHS England’s Secure Data Environment (SDE) service for England ( https://digital.nhs.uk/services/secure-data-environment-service ). The CVD-COVID-UK/COVID-IMPACT programme led by the BHF Data Science Centre ( https://bhfdatasciencecentre.org/ ) received approval to access data in NHS England’s SDE service for England from the Independent Group Advising on the Release of Data (IGARD) ( https://digital.nhs.uk/about-nhs-digital/corporate-information-and-documents/independent-group-advising-on-the-release-of-data ) via an application made in the Data Access Request Service (DARS) Online system (ref. DARS-NIC-381078-Y9C5K) ( https://digital.nhs.uk/services/data-access-request-service-dars/dars-products-and-services ). The CVD-COVID-UK/COVID-IMPACT Approvals & Oversight Board ( https://bhfdatasciencecentre.org/areas/cvd-covid-uk-covid-impact/ ) subsequently granted approval to this project to access the data within NHS England’s SDE service for England. The de-identified data used in this study were made available to accredited researchers. Those wishing to gain access to the data should contact [email protected] in the first instance.

Code availability

All code for data preparation and analysis are publicly available on GitHub ( https://github.com/BHFDSC/CCU037_01 ).

Arcaya, M. C., Arcaya, A. L. & Subramanian, S. V. Inequalities in health: definitions, concepts, and theories. Glob Health Action 8 , 27106, https://doi.org/10.3402/gha.v8.27106 (2015).

Article   PubMed   Google Scholar  

Chen, N. et al . Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: a descriptive study. Lancet 395 , 507–513, https://doi.org/10.1016/S0140-6736(20)30211-7 (2020).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Williamson, E. J. et al . Factors associated with COVID-19-related death using OpenSAFELY. Nature 584 , 430–436, https://doi.org/10.1038/s41586-020-2521-4 (2020).

Clift, A. K. et al . Living risk prediction algorithm (QCOVID) for risk of hospital admission and mortality from coronavirus 19 in adults: national derivation and validation cohort study. BMJ 371 , m3731, https://doi.org/10.1136/bmj.m3731 (2020).

Saunders, C. L., Abel, G. A., El Turabi, A., Ahmed, F. & Lyratzopoulos, G. Accuracy of routinely recorded ethnic group information compared with self-reported ethnicity: evidence from the English Cancer Patient Experience survey. BMJ Open 3 , https://doi.org/10.1136/bmjopen-2013-002882 (2013).

Gore, A., Truche, P., Iskerskiy, A., Ortega, G. & Peck, G. Inaccurate Ethnicity and Race Classification of Hispanics Following Trauma Admission. J Surg Res 268 , 687–695, https://doi.org/10.1016/j.jss.2021.08.003 (2021).

GOV.UK. List of ethnic groups , https://www.ethnicity-facts-figures.service.gov.uk/style-guide/ethnic-groups .

Mathur, R. et al . Completeness and usability of ethnicity data in UK-based primary care and hospital databases. J Public Health (Oxf) 36 , 684–692, https://doi.org/10.1093/pubmed/fdt116 (2014).

Tippu, Z. et al . Ethnicity Recording in Primary Care Computerised Medical Record Systems: An Ontological Approach. J Innov Health Inform 23 , 920, https://doi.org/10.14236/jhi.v23i4.920 (2017).

Davidson, J. et al . Ethnic differences in the incidence of clinically diagnosed influenza: an England population-based cohort study 2008-2018. Wellcome Open Res 6 , 49, https://doi.org/10.12688/wellcomeopenres.16620.3 (2021).

Article   PubMed   PubMed Central   Google Scholar  

Mathur, R. et al . Ethnic differences in SARS-CoV-2 infection and COVID-19-related hospitalisation, intensive care unit admission, and death in 17 million adults in England: an observational cohort study using the OpenSAFELY platform. Lancet 397 , 1711–1724, https://doi.org/10.1016/S0140-6736(21)00634-6 (2021).

Bureau, U. S. C. Measuring Racial and Ethnic Diversity for the 2020 Census , https://www.census.gov/newsroom/blogs/random-samplings/2021/08/measuring-racial-ethnic-diversity-2020-census.html (2021).

Office for National Statistics. National identity, ethnic group, language and religion question development for Census 2021 , https://www.ons.gov.uk/census/censustransformationprogramme/questiondevelopment/nationalidentityethnicgrouplanguageandreligionquestiondevelopmentforcensus2021 (2022).

British-Heart-Foundation-data-science-centre. CVD-COVID-UK/COVID-IMPACT website , https://bhfdatasciencecentre.org/areas/cvd-covid-uk-covid-impact/ .

Wood, A. et al . Linked electronic health records for research on a nationwide cohort of more than 54 million people in England: data resource. BMJ 373 , n826, https://doi.org/10.1136/bmj.n826 (2021).

NHS-Digital. Trusted Research Environment service for England , https://digital.nhs.uk/coronavirus/coronavirus-data-services-updates/trusted-research-environment-service-for-england .

NHS-Digital. Master Person Service (MPS) , https://digital.nhs.uk/services/master-person-service .

NHS-Digital. Secure Data Environment service , https://digital.nhs.uk/services/secure-data-environment-service .

NHS-Digital. General Practice Extraction Service (GPES) Data for pandemic planning and research: a guide for analysts and users of the data , https://digital.nhs.uk/coronavirus/gpes-data-for-pandemic-planning-and-research/guide-for-analysts-and-users-of-the-data#:~:text=This%20guidance%20provides%20an%20overview,%2D19)%20planning%20and%20research .

NHS-Digital. NHS Digital transparency notice: GPES data for pandemic planning and research (COVID-19) , https://digital.nhs.uk/coronavirus/gpes-data-for-pandemic-planning-and-research/transparency-notice-gpes-data-for-pandemic-planning-and-research-covid-19#types-of-personal-data-we-process (2020).

NHS-Digital. SNOMED CT , https://digital.nhs.uk/services/terminology-and-classifications/snomed-ct .

Chang, E. & Mostafa, J. The use of SNOMED CT, 2013-2020: a literature review. J Am Med Inform Assoc 28 , 2017–2026, https://doi.org/10.1093/jamia/ocab084 (2021).

National-Health-Service. NHS Data Model and Dictionary , https://www.datadictionary.nhs.uk/ .

Khunti, K., Routen, A., Banerjee, A. & Pareek, M. The need for improved collection and coding of ethnicity in health research. J Public Health (Oxf) 43 , e270–e272, https://doi.org/10.1093/pubmed/fdaa198 (2021).

Elliott, R. A., Camacho, E., Jankovic, D., Sculpher, M. J. & Faria, R. Economic analysis of the prevalence and clinical and economic burden of medication error in England. BMJ Qual Saf 30 , 96–105, https://doi.org/10.1136/bmjqs-2019-010206 (2021).

Armstrong, K., Ravenell, K. L., McMurphy, S. & Putt, M. Racial/ethnic differences in physician distrust in the United States. Am J Public Health 97 , 1283–1289, https://doi.org/10.2105/AJPH.2005.080762 (2007).

Office for National Statistics. Ethnic group, England and Wales: Census 2021 , https://www.ons.gov.uk/peoplepopulationandcommunity/culturalidentity/ethnicity/bulletins/ethnicgroupenglandandwales/census2021 (2022).

Shiekh, S. I. et al . Completeness, agreement, and representativeness of ethnicity recording in the United Kingdom’s Clinical Practice Research Datalink (CPRD) and linked Hospital Episode Statistics (HES). Population Health Metrics 21 , 3, https://doi.org/10.1186/s12963-023-00302-0 (2023).

Petersen, I. et al . Health indicator recording in UK primary care electronic health records: key implications for handling missing data. Clin Epidemiol 11 , 157–167, https://doi.org/10.2147/CLEP.S191437 (2019).

Egan, K. J., Knutson, K. L., Pereira, A. C. & von Schantz, M. The role of race and ethnicity in sleep, circadian rhythms and cardiovascular health. Sleep Med Rev 33 , 70–78, https://doi.org/10.1016/j.smrv.2016.05.004 (2017).

Office for National Statistics. Census 2021 outputs , https://www.ons.gov.uk/census/censustransformationprogramme/census2021outputs (2022).

NHS-Digital. Ethnicity https://digital.nhs.uk/data-and-information/areas-of-interest/ethnicity

Office for National Statistics. Population estimates by ethnic group and religion, England and Wales: 2019. (2021).

NHS-Digital. Excel file downoaded from “Data Coverage - Management Information” section in “General Practice Extraction Service (GPES) Data for pandemic planning and research: a guide for analysts and users of the data” ., https://digital.nhs.uk/coronavirus/gpes-data-for-pandemic-planning-and-research/guide-for-analysts-and-users-of-the-data#data-coverage-management-information .

Download references


The British Heart Foundation Data Science Centre (grant No SP/19/3/34678, awarded to Health Data Research (HDR) UK), funded co-development (with NHS England) of the Secure Data Environment service for England, provision of linked datasets, data access, user software licences, computational usage, and data management and wrangling support, with additional contributions from the HDR UK Data and Connectivity component of the UK Government Chief Scientific Adviser’s National Core Studies programme to coordinate national COVID-19 priority research. Consortium partner organisations funded the time of contributing data analysts, biostatisticians, epidemiologists, and clinicians. The authors acknowledge English language editing by Dr Jennifer A de Beyer and Amelia M Doran, Centre for Statistics in Medicine, University of Oxford. This work was carried out with the support of the BHF Data Science Centre led by HDR UK (BHF Grant no. SP/19/3/34678). This study made use of de-identified data held in NHS England’s Secure Data Environment service for England and made available via the BHF Data Science Centre’s CVD-COVID-UK/COVID-IMPACT consortium. This work used data provided by patients and collected by the NHS as part of their care and support. We would like to acknowledge all data providers who make health relevant data available for research. This research is part of the Data and Connectivity National Core Study, led by Health Data Research UK in partnership with the Office for National Statistics and funded by UK Research and Innovation (grant ref MC_PC_20058). This work was also supported by The Alan Turing Institute via ‘Towards Turing 2.0’ EPSRC Grant Funding. The funders had no role in the study design, data collection, data analysis, data interpretation, or report writing.

Author information

Authors and affiliations.

Centre for Statistics in Medicine, Botnar Research Centre, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences (NDORMS), University of Oxford, Oxford, UK

Marta Pineda-Moncusí, Antonella Delmestri, Gary S. Collins, Daniel Prieto-sAlhambra & Sara Khalid

Institute of Health Informatics, 222 Euston Road, London, NW1 2DA, University College London, London, UK

Freya Allery, Johan H. Thygesen, Alex Handy, Amitava Banerjee, Spiros Denaxas & Christopher Tomlinson

British Heart Foundation Data Science Centre, Health Data Research UK, London, UK

Thomas Bolton, John Nolan, Spiros Denaxas, Cathie Sudlow & Angela Wood

University College London Hospitals Biomedical Research Centre, University College London, London, UK

Spiros Denaxas & Christopher Tomlinson

UK Research and Innovation Centre for Doctoral Training in AI-enabled Healthcare Systems, University College London, London, UK

Christopher Tomlinson

University Hospitals Birmingham NHS Foundation Trust (NHSFT), Birmingham, UK

Alastair K. Denniston

Population Data Science, Swansea University Medical School, Faculty of Medicine, Health & Life Science, Swansea University, Swansea, Wales, UK

Ashley Akbari

British Heart Foundation Cardiovascular Epidemiology Unit, Department of Public Health and Primary Care, University of Cambridge, Cambridge, UK

Angela Wood

Victor Phillip Dahdaleh Heart and Lung Research Institute, University of Cambridge, Cambridge, UK

Department of Primary Care and Population Health, UCL, London, NW3 2PF, UK

Irene Petersen

Department of Clinical Epidemiology, Aarhus University, Aarhus N, Aarhus, 8200, Denmark

Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences (NDORMS), University of Oxford, Oxford, UK

Laura C. Coates

Diabetes Research Centre, University of Leicester, Leicester, UK

Kamlesh Khunti

Department of Medical Informatics, Erasmus MC University Medical Centre Rotterdam, Rotterdam, The Netherlands

Daniel Prieto-sAlhambra

You can also search for this author in PubMed   Google Scholar

  • Marta Pineda-Moncusí
  • , Freya Allery
  • , Antonella Delmestri
  • , Thomas Bolton
  • , John Nolan
  • , Johan H. Thygesen
  • , Alex Handy
  • , Amitava Banerjee
  • , Spiros Denaxas
  • , Christopher Tomlinson
  • , Alastair K. Denniston
  • , Cathie Sudlow
  • , Ashley Akbari
  • , Angela Wood
  • , Kamlesh Khunti
  • , Daniel Prieto-sAlhambra
  •  & Sara Khalid


Conceptualisation: S.K., D.P.A., A.D., G.C., I.P. Data curation: M.P.M., F.A. Formal analysis: M.P.M., F.A., T.B. Funding acquisition: S.K. Data interpretation: M.P.M., F.A., S.K. Writing original draft: M.P.M., F.A., S.K. Writing review and editing: all authors. Approving final version of manuscript: all authors. S.K. and M.P.M. takes responsibility for the integrity of the data analysis. C.S. is the Director of the B.H.F. Data Science Centre and coordinated approvals for and access to data within NHS England’s S.D.E. service for England for CVD-COVID-UK/COVID-IMPACT.

Corresponding author

Correspondence to Sara Khalid .

Ethics declarations

Competing interests.

AA is supported by Health Data Research UK (HDR-9006), which receives its funding from the UK Medical Research Council (MRC, grant MR/V028367/1); and Administrative Data Research UK, which is funded by the ESRC (grant ES/S007393/1). SD is supported by: BigData@Heart Consortium, funded by the Innovative Medicines Initiative-2 Joint Undertaking under grant agreement 116074, The British Heart Foundation Data Science Centre (grant No SP/19/3/34678, awarded to Health Data Research (HDR) UK), NIHR Biomedical Research Centre at University College London (UCL) Hospital NHS Trust, the NIHR-UKRI CONVALESCENCE study, BHF Accelerator Award (AA/18/6/24223). CT is supported by a UCL UKRI Centre for Doctoral Training in AI-enabled Healthcare studentship (EP/S021612/1), MRC Clinical Top-Up and a studentship from the NIHR Biomedical Research Centre at University College London Hospital NHS Trust. KK is the director of Centre for Ethnic Health Research, and trustee of South Asian Health Foundation. SK has received research grant funding from the UKRI and Alan Turing Institute for this work, and from Amgen and UCB Biopharma, and Bill & Melinda Gates Foundation outside of this work. DPA’s research group has received grant/s from Amgen, Chiesi-Taylor, Lilly, Janssen, Novartis, and UCB Biopharma. His research group has received consultancy fees from Astra Zeneca and UCB Biopharma. Amgen, Astellas, Janssen, Synapse Management Partners and UCB Biopharma have funded or supported training programmes organised by SK and DPA’s department. The remaining authors have nothing to declare.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information, supplementary information - supplementary tables on excel, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Pineda-Moncusí, M., Allery, F., Delmestri, A. et al. Ethnicity data resource in population-wide health records: completeness, coverage and granularity of diversity. Sci Data 11 , 221 (2024). https://doi.org/10.1038/s41597-024-02958-1

Download citation

Received : 07 August 2023

Accepted : 12 January 2024

Published : 22 February 2024

DOI : https://doi.org/10.1038/s41597-024-02958-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

  • Explore articles by subject
  • Guide to authors
  • Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

observational research health care

observational research health care

  • Career Center
  • Member Login

Assessing Trust in Health Care: A Compendium of Trust Measures

AcademyHealth Trust Scholars in Residence, Jody Platt, M.P.H., Ph.D., and Lauren Taylor, M.Div, M.P.H., Ph.D., present a compendium of measures on trust in health care

Trust measures report

Measuring Trust: Where are we and where do we need to go?

This compendium outlines a framework for addressing the issue of conceptual clarity in trust research, enabling readers to better articulate why they are measuring trust, what key attributes they hope to prioritize in measuring trust, what they hope to gain from measuring trust, as well as clear expectations about the strengths and weaknesses of any measure they choose.

In Measuring Trust: Where are we and where do we need to go ? Drs. Platt and Taylor set out to address three issues which have fueled the propagation of trust measures:

  • The lack of conceptual clarity across measures
  • A lack of consensus around a single measure or set of measures
  • Trust may operate differently depending on who is trusting whom, and what the context is

The intended readers for this guide are (1) health system leaders, organizational leaders and others interested in adopting measures at their institutions, (2) health services researchers who may not be focused on the issue of trust as a primary area of expertise, but see it as an important variable or outcome of interest in their work, and (3) those interested in assessing measures to support a convergence of methods and/or processes for choosing how, when, and what aspects of trust are to be measured.   

Learn more about the Advancing Research on Trust initiative  here .

This paper is in the following e-collection/theme issue:

Published on 22.2.2024 in Vol 8 (2024)

Investigating the Feasibility of Using a Wearable Device to Measure Physiologic Health Data in Emergency Nurses and Residents: Observational Cohort Study

Authors of this article:

Author Orcid Image

Original Paper

  • Anish K Agarwal 1, 2, 3 , MD, MPH, MS   ; 
  • Rachel Gonzales 1, 2, 3 , MPH   ; 
  • Kevin Scott 1, 2 , MD, MSEd   ; 
  • Raina Merchant 1, 2, 3 , MD, MSHP  

1 Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States

2 Department of Emergency Medicine, University of Pennsylvania, Philadelphia, PA, United States

3 Center for Health Care Transformation and Innovation, Penn Medicine, Philadelphia, PA, United States

Corresponding Author:

Anish K Agarwal, MD, MPH, MS

Perelman School of Medicine

University of Pennsylvania

423 Guardian Drive

410 Blockley Hall

Philadelphia, PA, 19104

United States

Phone: 1 2157465610

Email: [email protected]

Background: Emergency departments play a pivotal role in the US health care system, with high use rates and inherent stress placed on patients, patient care, and clinicians. The impact of the emergency department environment on the health and well-being of emergency residents and nurses can be seen in worsening rates of burnout and cardiovascular health. Research on clinician health has historically been completed outside of clinical areas and not personalized to the individual. The expansion of digital technology, specifically wearable devices, may enhance the ability to understand how health care environments impact clinicians.

Objective: The primary objective of this pilot study was to assess the feasibility and acceptability of using wearable devices to measure and record physiologic data from emergency nurses and resident physicians. Understanding strategies that are accepted and used by clinicians is critical prior to launching larger investigations aimed at improving outcomes.

Methods: This was a longitudinal pilot study conducted at an academic, urban, level 1 trauma center. A total of 20 participants, including emergency medicine resident physicians and nurses, were equipped with a wearable device (WHOOP band) and access to a mobile health platform for 6 weeks. Baseline surveys assessed burnout, mental health, and expectations of the device and experience. Participants provided open-ended feedback on the device and platform, contributing to the assessment of acceptance, adoption, and use of the wearable device. Secondary measures explored early signs and variations in heart rate variability, sleep, recovery, burnout, and mental health assessments.

Results: Of the 20 participants, 10 consistently used the wearable device. Feedback highlighted varying experiences with the device, with a preference for more common wearables like the Apple Watch or Fitbit. Resident physicians demonstrated higher engagement with the device and platform as compared with nurses. Baseline mental health assessments indicated mild anxiety and depressive symptoms among participants. The Professional Fulfillment Index revealed low professional fulfillment, moderate workplace exhaustion, and interpersonal disengagement.

Conclusions: This pilot study underscores the potential of wearable devices in monitoring emergency clinicians’ physiologic data but reveals challenges related to device preferences and engagement. The key takeaway is the necessity to optimize device and platform design for clinician use. Larger, randomized trials are recommended to further explore and refine strategies for leveraging wearable technology to support the well-being of the emergency workforce.


Emergency departments (EDs) are a critical and frequently used resource within the US health care infrastructure (1 in 5 adults are treated in an ED annually) [ 1 , 2 ]. EDs are stressful environments [ 3 - 6 ]. Research has demonstrated the negative impact of the physical ED environment on patients [ 7 - 10 ], yet less is known about the effects on the health and well-being of the clinicians working within these EDs. Burnout and cardiovascular (CV) health remain threats to ED clinicians, their careers, and patient care [ 11 - 14 ]. Research on clinician health and burnout has historically been limited by retrospective studies which are often conducted outside of work. The rapid growth of digital technology, such as wearable devices and remote monitoring [ 15 - 17 ], offers new opportunities and challenges to investigate clinician health within health care environments and spaces.

Multiple occupational hazards have been attributed to the practice of emergency medicine (EM) [ 18 ], EM clinicians have a 20% higher morbidity due to coronary artery disease, motor vehicle accidents, and impaired reproductive health [ 19 - 22 ]. Clinicians working night shifts, an essential practice in EM, have less restorative sleep, elevated blood pressure, and lower heart rate variability (HRV) [ 19 , 23 - 26 ]. Before the pandemic, the prevalence of stress, exhaustion, and burnout was alarmingly high in EM [ 27 , 28 ]. COVID-19 worsened these factors, resulting in workforce depletion, and making this an urgent and critical area of focus underscored by the Surgeon General and National Academy of Medicine [ 14 , 29 ].

A gap exists in understanding how clinicians identify and prioritize their health within the workplace. High-performance athletics provides a potential analogous framework whereby athletes track physiologic data (HRV, physical activity, and sleep) to guide their daily performance. There is an unrealized opportunity space for clinicians to understand and enhance care delivery and career longevity (reduce burnout and CV disease). Technological advancements provide a potentially unobtrusive and personalized method to collect individual data using wearable devices. Wearable device use has grown in popularity, with over 30% of Americans reporting they can obtain device ownership [ 30 ]. These wearable devices are typically wrist-worn and provide methods to measure health data such as step count, heart rate, and sleep [ 15 ]. What is less known, is if these devices and associated platforms are appealing to clinicians and can provide actionable insights to help inform strategies to support the workforce.

The objective of this study was to pilot test and evaluate the feasibility and acceptability of a wearable device and associated platform to measure and record emergency nurse and resident physician physiologic measures while they provide emergency care. This was a pilot study, investigating early barriers and facilitators to using these devices within health care settings for emergency nurses and residents.

Eligible EM resident physicians and emergency nurses included those providing 20 or more hours of patient care per week, having regular access to a smartphone, and providing consent to where a wrist-worn wearable device (WHOOP band [ 31 ]). Participants were recruited via email, completed informed consent, and were given a wrist-worn wearable device. Consenting participants completed a baseline survey assessing burnout and mental health (depression and anxiety), asked about their expectations of the study, and followed for 6 weeks. Validated instruments included the Patient Health Questionnaire (PHQ-8), General Anxiety Disorder (GAD-7), and the Professional Fulfillment Index (PFI) [ 32 - 36 ]. Over the 6 weeks, patients were also given access to a web-based platform that allowed participants to see their own physiologic data, access basic coaching videos, and connect to other users on the platform (Arena Strive [ 37 ]). At the completion of 6 weeks, participants were asked to complete a final survey exploring the feasibility and acceptability of the approach. Participants were also asked to provide free text commentary on the general approach and the specific device. The primary outcomes were use of the wearable device, acceptance of the device, and adoption of the device. Secondary measures included burnout, mental health symptoms, and physiologic measures recorded by the device including HRV and sleep.

Ethical Considerations

This was a longitudinal pilot feasibility study conducted at an urban, academic, level 1 trauma center in the northeastern United States. This study was reviewed and approved by the University of Pennsylvania Institutional Review Board (850371). All eligible participants completed informed consent forms and were informed that all data would be deidentified and aggregated for analysis. Participants received the wearable device at no cost and could keep the device following the completion of the study.

Data Analysis

Analysis was conducted using Stata SE (version 18; StataCorp). Descriptive statistics were used to summarize participant demographics and well-being survey results which included the PHQ-8, GAD-7, and the PFI. Single-sample 2-tailed t tests were used to investigate exploratory differences in physiologic measures.

This was a pilot feasibility study and thus was not powered to detect individual health outcomes. A total of 20 participants were enrolled (13/20, 65% were female), 12 were EM resident physicians and 8 were emergency nurses. Of the 20 participants, 10 participants routinely wore the wearable device (6 resident physicians and 4 nurses).

Participants completing baseline mental health assessments reported mild anxiety as measured by the GAD-7 (mean score 5.07, SD 3.7), with 85% (n=17) reporting minimal or mild anxiety. Participants also reported mild depressive symptoms as measured by the PHQ-8 (mean 5.73, SD 2.9), with half reporting mild depressive symptoms. Participants completed the PFI to evaluate burnout and fulfillment. Individuals reported low professional fulfillment (mean 49.4, SD 16.9) moderate workplace exhaustion (mean 57.1, SD 24.4), and moderate interpersonal disengagement (mean 44.7, SD 20.1). Participants were asked via survey to comment on their early thoughts and goals with the pilot and the device. Notable themes emerged reflecting (1) technological features (eg, seeking a device with a watch face), (2) ways to integrate data into their personal lives and clinical roles, and (3) increasing self-awareness of the objective measures of stress related to clinical care.

Among participants who used the band consistently for 6 weeks, variation existed in their experience with the wearable and the data it generated. None of the participants were very likely to recommend the device to others. Two participants found the data interaction helpful and useful and overall, none commented on the platform being easy to use. When asked specifically, participants noted the band to be obtrusive given its lack of daily use features (eg, watch face and activity data) and odd charging mechanics. Several participants did comment positively that the data output and data generated was useful and empowering but needed to be collected using a more user-friendly design such as the more commonly used Apple Watch or FitBit. Participants sought the same application using these devices and expressed enthusiasm for those.

Of the 10 users who routinely used the device, we saw early variation in physiologic measures related to HRV, stress, and sleep ( Table 1 ; Figures 1 and 2 ). While statistically significant differences are identified here, this remained a pilot study in feasibility, user input, and data collection methods. Early insights from these data suggest differences across roles between resident physicians and nurses, as well as across sex.

observational research health care

Principal Findings

The physical and mental toll facing clinicians working in the ED continues to grow. Emergency nurses and resident physicians face a number of challenges impacting their health including, but not limited to, ED crowding, boarding, workplace violence, shifting schedules, and rising patient acuity. In the wake of the COVID-19 pandemic, an emphasis on supporting the workforce remains a priority. The evolving landscape of digital technology, including wearable devices, offers new opportunities for individuals to monitor their own health and potentially proactively identify physical or mental strain. This pilot study begins to examine if and how wearable devices can be used for emergency clinicians.

First, we found mixed enthusiasm for this approach given low interest in completing exit surveys and ongoing data interaction. It appears from this early feasibility study that resident physicians may be more engaged with this strategy and data collection method. Nurses in this pilot tended to be less engaged with the data tracking and follow-up mechanisms. Resident physicians were, in general, more enthusiastic before, during, and after the period ended. Though small in size, this pilot study does shed some initial interest from emergency clinicians and understanding key physiologic metrics such as sleep and physical activity. Key data insights remain physical activity as measured by step count, amount and quality of sleep, and HRV, which is an established physiologic measure relating to CV health [ 38 ]. The next steps to build on this pilot study included bringing it to a larger scale and designing it for nurse and physician preferences which we have learned to date.

The key finding of this pilot study is that the type of device and platform must be optimized for clinician use. In this study, we used a high-performance athletics device, which is designed primarily for physiologic measures [ 39 ]. This device does not have some traditional features that the average person may be accustomed to including a clock and the ability to send or receive messages. More traditional and popular bands such as the Apple Watch or Fitbit offer participants data tracking with features such as message-sending capability and a watch. These features are important when designing for scale, but were not previously known until we pilot-tested it, as other studies in more controlled environments have used the same device [ 39 , 40 ], underscoring the need to optimize design over technological capacity. The high-performance athletic band used in this pilot study offered a longer battery life and the ability to charge while wearing the device, which seemed less important to participants in this study. Future work needs to leverage the existing devices that clinicians wear in their everyday lives and incorporate those devices into this approach.

Finally, these devices and remote surveys highlight the persistent and real variation in professional disengagement, exhaustion, and burnout. In this small cohort, we do not identify significant amounts of anxiety or depression in screening assessments. We do note some variation and physiologic measures across resin physicians and nurses as well as differences across individuals who identify as male vs female. These differences though statistically significant, represent only a small sample size, and follow-up studies need to be scaled at larger populations. Specific interests should investigate physiologic measures such as heart rate, HRV, and sleep.


This study has several limitations. This was a single-center pilot study and had a small sample size, which was intentional by design. We see glimpses toward mechanisms to optimize digital technology and workforce sustainment. The signals identified here represent only early pilot findings, and inherently there is also selection bias in individuals who opted to wear the band and complete surveys. Less is known about individuals who do not want to be part of this pilot, and future studies will need to be larger, randomized, and for a longer duration. Nonetheless, the study is among the first to begin to investigate the feasibility of using digital technology to support emergency physicians and nurses by helping them identify physiologic variations in their own health. The study represents the pilot beginnings to help identify proactive and much-needed new methods to mitigate strain that is related to physical and mental health.

This pilot study of emergency nurses and resident physicians investigating wearable devices to capture physiologic data from a cohort represents early signals toward feasible and acceptable programs. This pilot study identifies opportunities and interest in these mechanisms and a need to leverage more consumer-facing and potentially less sophisticated wearable devices for emergency clinicians. These methods can be further explored and larger, randomized trials can be conducted to investigate these strategies and how we support the workforce.


The authors would like to acknowledge the staff of the University of Pennsylvania Department of Emergency Medicine. This study was funded by the Emergency Medicine Foundation.

Data Availability

The deidentified data sets generated or analyzed during this study are available from the corresponding author on reasonable request.

Conflicts of Interest

None declared.

  • Cairns C, Ashman JJ, Kang K. Emergency department visit rates by selected characteristics: United States, 2018; National Center for Health Statistics Data Brief No. 401 March 2021. US Centers for Disease Control and Prevention. 2021. URL: https://www.cdc.gov/nchs/products/databriefs/db401.htm [accessed 2023-10-09]
  • Lin MP, Baker O, Richardson LD, Schuur JD. Trends in emergency department visits and admission rates among US acute care hospitals. JAMA Intern Med. 2018;178 (12):1708-1710. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Agarwal AK, Hahn L, Rosin R, Merchant RM. Exploring digital methods to capture self-reported on-shift sentiment amongst academic emergency department physicians. Am J Emerg Med. 2021;46:760-762. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Graneto J, Damm T. Perception of noise by emergency department nurses. West J Emerg Med. 2013;14 (5):547-550. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Agarwal AK, Hahn L, Pelullo A, Rosin R, Merchant RM. Capturing real-time emergency department sentiment: a feasibility study using touch-button terminals. Ann Emerg Med. 2020;75 (6):727-732. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Knopp R, Rosenzweig S, Bernstein E, Totten V. Physician-patient communication in the emergency department, part 1. Acad Emerg Med. 1996;3 (11):1065-1069. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Gaieski DF, Agarwal AK, Mikkelsen ME, Drumheller B, Sante SC, Shofer FS, et al. The impact of ED crowding on early interventions and mortality in patients with severe sepsis. Am J Emerg Med. 2017;35 (7):953-960. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Johnson KD, Winkelman C. The effect of emergency department crowding on patient outcomes: a literature review. Adv Emerg Nurs J. 2011;33 (1):39-54. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Twanmoh JR, Cunningham GP. When overcrowding paralyzes an emergency department. Manag Care. 2006;15 (6):54-59. [ Medline ]
  • Wang H, Kline JA, Jackson BE, Robinson RD, Sullivan M, Holmes M, et al. The role of patient perception of crowding in the determination of real-time patient satisfaction at emergency department. Int J Qual Health Care. 2017;29 (5):722-727. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Rotenstein LS, Torre M, Ramos MA, Rosales RC, Guille C, Sen S, et al. Prevalence of burnout among physicians: a systematic review. JAMA. 2018;320 (11):1131-1150. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Rothenberger DA. Physician burnout and well-being: a systematic review and framework for action. Dis Colon Rectum. 2017;60 (6):567-576. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Halbesleben JRB, Rathert C. Linking physician burnout and patient outcomes: exploring the dyadic relationship between physicians and patients. Health Care Manage Rev. 2008;33 (1):29-39. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Murthy VH. Addressing Health Worker Burnout: The U.S. Surgeon General’s Advisory on Building a Thriving Health Workforce. US Department of Health and Human Services. URL: https://www.hhs.gov/sites/default/files/health-worker-wellbeing-advisory.pdf [accessed 2024-01-25]
  • Case MA, Burwick HA, Volpp KG, Patel MS. Accuracy of smartphone applications and wearable devices for tracking physical activity data. JAMA. 2015;313 (6):625-626. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Patel MS, Foschini L, Kurtzman GW, Zhu J, Wang W, Rareshide CAL, et al. Using wearable devices and smartphones to track physical activity: initial activation, sustained use, and step counts across sociodemographic characteristics in a national sample. Ann Intern Med. 2017;167 (10):755-757. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Agarwal A, Patel M. Prescribing behavior change: opportunities and challenges for clinicians to embrace digital and mobile health. JMIR Mhealth Uhealth. 2020;8 (8):e17281. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Arora M, Asha S, Chinnappa J, Diwan AD. Review article: burnout in emergency medicine physicians. Emerg Med Australas. 2013;25 (6):491-495. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Machi MS, Staum M, Callaway CW, Moore C, Jeong K, Suyama J, et al. The relationship between shift work, sleep, and cognition in career emergency physicians. Acad Emerg Med. 2012;19 (1):85-91. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Shea JA, Bellini LM, Dinges DF, Curtis ML, Tao Y, Zhu J, et al. Impact of protected sleep period for internal medicine interns on overnight call on depression, burnout, and empathy. J Grad Med Educ. 2014;6 (2):256-263. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Vetter C, Devore EE, Wegrzyn LR, Massa J, Speizer FE, Kawachi I, et al. Association between rotating night shift work and risk of coronary heart disease among women. JAMA. 2016;315 (16):1726-1734. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Kawachi I, Colditz GA, Stampfer MJ, Willett WC, Manson JE, Speizer FE, et al. Prospective study of shift work and risk of coronary heart disease in women. Circulation. 1995;92 (11):3178-3182. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Goolsby C, Lala V, Gebner R, Dacuyan-Faucher N, Charlton N, Schuler K. Emergency physician risk of occupational mortality: a scoping review. J Am Coll Emerg Physicians Open. 2021;2 (5):e12554. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Heikkilä K, Pentti J, Madsen IEH, Lallukka T, Virtanen M, Alfredsson L, et al. Job strain as a risk factor for peripheral artery disease: a multi-cohort study. J Am Heart Assoc. 2020;9 (9):e013538. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Shi H, Huang T, Schernhammer ES, Sun Q, Wang M. Rotating night shift work and healthy aging after 24 years of follow-up in the nurses' health study. JAMA Netw Open. 2022;5 (5):e2210450. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Torquati L, Mielke GI, Brown WJ, Kolbe-Alexander T. Shift work and the risk of cardiovascular disease. a systematic review and meta-analysis including dose-response relationship. Scand J Work Environ Health. 2018;44 (3):229-238. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Berger E. Physician burnout: emergency physicians see triple risk of career affliction. Ann Emerg Med. 2013;61 (3):A17-A19. [ FREE Full text ] [ CrossRef ]
  • Spickard A, Gabbe SG, Christensen JF. Mid-career burnout in generalist and specialist physicians. JAMA. 2002;288 (12):1447-1450. [ CrossRef ] [ Medline ]
  • National Academies of Sciences, Engineering, and Medicine; Committee on Systems Approaches to Improve Patient Care by Supporting Clinician Well-Being; National Academy of Medicine. Taking Action Against Clinician Burnout: A Systems Approach to Professional Well-Being. Washington, D.C. National Academies Press; 2019. .
  • Condon S. Forrester: nearly 1 in 3 Americans will use a wearable device by 2021. ZDNET. 2018. URL: https://www.zdnet.com/article/forrester-nearly-1-in-3-americans-will-use-a-wearable-device-by-2021/ [accessed 2024-09-18]
  • Your personal digital fitness and health coach. WHOOP. URL: https://www.whoop.com/ [accessed 2023-08-03]
  • An Y, Yang Y, Wang A, Li Y, Zhang Q, Cheung T, et al. Prevalence of depression and its impact on quality of life among frontline nurses in emergency departments during the COVID-19 outbreak. J Affect Disord. 2020;276:312-315. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Kroenke K, Strine TW, Spitzer RL, Williams JBW, Berry JT, Mokdad AH. The PHQ-8 as a measure of current depression in the general population. J Affect Disord. 2009;114 (1-3):163-173. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Alharthy N, Alrajeh OA, Almutairi M, Alhajri A. Assessment of anxiety level of emergency health-care workers by generalized anxiety disorder-7 tool. Int J Appl Basic Med Res. 2017;7 (3):150-154. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Stewart MT, Reed S, Reese J, Galligan MM, Mahan JD. Conceptual models for understanding physician burnout, professional fulfillment, and well-being. Curr Probl Pediatr Adolesc Health Care. 2019;49 (11):100658. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Trockel M, Bohman B, Lesure E, Hamidi MS, Welle D, Roberts L, et al. A brief instrument to assess both burnout and professional fulfillment in physicians: reliability and validity, including correlation with self-reported medical errors, in a sample of resident and practicing physicians. Acad Psychiatry. 2018;42 (1):11-24. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Pioneering the field of high performance medicine. Arena Labs. URL: https://arenalabs.co/ [accessed 2023-08-03]
  • Task Force of the European Society of Cardiology the North American Society of Pacing Electrophysiology. Heart rate variability: standards of measurement, physiological interpretation, and clinical use. Circulation. 1996;93 (5):1043-1065. [ FREE Full text ] [ CrossRef ]
  • Presby DM, Jasinski SR, Capodilupo ER. Wearable derived cardiovascular responses to stressors in free-living conditions. PLoS One. 2023;18 (6):e0285332. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Rowan SP, Lilly CL, Claydon EA, Wallace J, Merryman K. Monitoring one heart to help two: heart rate variability and resting heart rate using wearable technology in active women across the perinatal period. BMC Pregnancy Childbirth. 2022;22 (1):887. [ FREE Full text ] [ CrossRef ] [ Medline ]


Edited by A Mavragani; submitted 03.08.23; peer-reviewed by S Kheirinejad, L Masanneck; comments to author 02.12.23; revised version received 20.12.23; accepted 07.01.24; published 22.02.24.

©Anish K Agarwal, Rachel Gonzales, Kevin Scott, Raina Merchant. Originally published in JMIR Formative Research (https://formative.jmir.org), 22.02.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • HHS Author Manuscripts

Logo of nihpa

Research and Reporting Considerations for Observational Studies Using Electronic Health Record Data

Alison callahan.

Center for Biomedical Informatics Research, School of Medicine, Stanford University

Nigam H. Shah

Jonathan h. chen.

Division of Hospital Medicine, School of Medicine, Stanford University

Dr. Callahan: Room X231, Medical School Office Building, Stanford University, 1265 Welch Road, Stanford, CA 94305.

Dr. Chen: Room X213, Medical School Office Building, Stanford University, 1265 Welch Road, Stanford, CA 94305.

Electronic health records (EHRs) are an increasingly important source of real-world health care data for observational research. Analyses of data collected for purposes other than research require careful consideration of data quality as well as the general research and reporting principles relevant to observational studies. The core principles for observational research in general also apply to observational research using EHR data, and these are well addressed in prior literature and guidelines. This article provides additional recommendations for EHR-based research. Considerations unique to EHR-based studies include assessment of the accuracy of computer-executable cohort definitions that can incorporate unstructured data from clinical notes and management of data challenges, such as irregular sampling, missingness, and variation across time and place. Principled application of existing research and reporting guidelines alongside these additional considerations will improve the quality of EHR-based observational studies.

Observational research helps to advance clinical knowledge and inform the practice of medicine. Electronic health records (EHRs) contain large quantities of health care data that are captured during care and are an increasingly important resource for conducting observational health research ( 1 ). The potential value of these data relates to the large volume of data drawn from real-world practice that may include more diverse patients and conditions than are feasible to include in studies that rely on primary data collection ( 2 , 3 ). Although EHRs typically provide larger quantities of clinical data than are available from surveys, registries, and clinical trials, the quality of these data—which were not collected for research purposes—raises important research and reporting considerations.

The core considerations for observational research are the same whether the research uses data collected primarily for research purposes or EHR data collected during the course of care. These core considerations are well-described in reporting guidelines, such as STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) ( 4 ). The RECORD (REporting of studies Conducted using Observational Routinely-collected health Data) ( 5 ) guidelines extend the STROBE guideline with related recommendations for studies using routinely collected health data, which are directly relevant to EHR-based studies.

This article is intended to complement existing guidelines by describing additional research and reporting issues that should be considered when conducting, reporting, and interpreting EHR-based studies. Issues encountered in our own prior research ( 6 – 8 ) and discussed by collaborative groups, such as the Observational Health Data Sciences and Informatics (OHDSI) ( 6 ) initiative, inform our recommendations. Issues that we address include assessment of the accuracy of algorithmic cohort definitions and electronic phenotyping that can incorporate unstructured data, such as that from clinical notes ( 9 , 10 ), and managing common irregularities of EHR data that can bias study results, such as irregular sampling, missingness, and nonstationarity across time and place ( 11 – 13 ). We use 2 examples to illustrate some of these issues.

Example 1: Identifying Primary Care Patients With High-Risk Opioid Use

We conducted a study to quantify the prevalence of chronic opioid use and determine whether primary care prescribing guidelines could decrease it ( 7 ). Because primary data collection or manual chart abstraction would be prohibitively expensive, we used EHR data collected in the context of routine primary care. We initially sought to identify a cohort of patients with “prescription opioid misuse”; behaviors of interest included breach of opioid pain contracts ( 14 ), medication diversion, premature refills, and chronic use of high dosages. Unfortunately, we quickly found it difficult to develop an accurate computer-executable definition of “prescription opioid misuse” because formal diagnostic codes were sparse and inconsistent. This is commonly the case for clinical data not closely linked to billing or compliance incentives ( 15 , 16 ). For many medical conditions, less than 10% of the affected individuals’ EHRs contain the respective International Classification of Diseases (ICD) diagnosis codes ( 17 ). Diagnostic coding accuracy also varies across settings, provider types, and whether a billing code specialist assigned the code ( 16 , 18 – 20 ).

Documentation and workflow variability introduced additional challenges. Notably, clinicians did not use a standardized electronic note template for screening questionnaires that could facilitate simple text recognition of such terms as “opioid contract” or “prescription drug monitoring program.” Thus, we had to define our cohort on the basis of alternate structured elements, such as total quantity of opioids prescribed within a given time window, while excluding patients with any history of a cancer-related diagnosis. Subsequent studies by other researchers illustrate that using algorithmic natural language processing to refine and validate cohort definitions can identify one-third more patients with opioid misuse than identified by diagnosis codes alone ( 21 , 22 ).

Example 2: Predicting Diagnostic Test Results

In another study, we sought to identify low-yield diagnostic tests by using EHR data available at the time of test ordering to predict whether common inpatient laboratory tests, such as magnesium, sodium, creatinine, and blood cultures, would yield abnormal results ( 23 , 24 ). The values of common vital signs and laboratory tests were identified as important predictors of subsequent test results, but so was the existence and number of such measurements, so our model included these counts.

Issues to Consider When Conducting Observational Studies of EHR Data

Developing “executable” cohorts in ehrs.

Algorithmic approaches to use EHR data to identify patient cohorts expand the feasibility of large-scale observational research, but require validation. These algorithms are referred to using such terms as “cohort definitions,” “health outcomes of interest,” “inclusion/exclusion criteria,” or “phenotypes.” A key step in “electronic phenotyping” is translating human understandable descriptions into computer-executable definitions ( 25 , 26 ). This step may involve simple logic that combines structured elements. Example 1 used this approach when we identified patients receiving chronic pain care as those who received opioid prescriptions from primary care providers while excluding patients with opioid prescriptions from oncology providers because these prescriptions may be for palliative care. Other approaches use probabilistic algorithms to estimate the likelihood that a patient belongs to a cohort of interest on the basis of patterns of data observed in other similar patients.

Augmenting electronic phenotyping algorithms by including additional content from clinical notes is a popular approach, but not a cure-all, because there can be gross documentation inconsistencies from copy-and-paste templates ( 27 , 28 ) and notes may ultimately only provide incremental information beyond deliberate use of more consistently available structured data elements ( 29 ). These additional layers of complexity require their own evaluation, consistent with recommendations 6.1 and 6.2 from RECORD ( 5 ). For a sample of the cases considered, a reference standard must be established for whether they meet the cohort definition. This often requires manual chart review by multiple domain experts, with assessment of interrater reliability (for example, kappa score) ( 30 ). The algorithmic approach can then be evaluated relative to the reference standard in terms of diagnostic and information retrieval metrics ( 31 ) of precision (positive predictive value) and recall (sensitivity). This allows researchers and reviewers to assess whether the algorithmic cohort definition can be extrapolated to larger samples with satisfactory results. Such projects as Phenotype KnowledgeBase ( 32 ) and OHDSI support these efforts by collecting a growing number of publicly available, human-understandable, and computer-executable definitions.

EHR Data Irregularities

Confounding, a well-recognized challenge in all observational research, is magnified when studies use broadly available EHR data collected by individuals providing care rather than by those curating data for research or billing purposes. For example, because sicker patients tend to receive more testing and treatment, confounding by indication ( 33 ) can bias the predictive value of laboratory results ( 13 ). Strategies for addressing such confounding is an important topic that is well covered in existing literature ( 34 – 42 ).

Missing data is another challenge in observational studies that can be magnified when EHR data are used. Data in an EHR are often missing not at random ( 43 , 44 ). Gaps in a patient’s record may be a result of loss to follow-up or transition to another care provider or insurer. Alternatively, data may be missing because of errors in populating a database record or incomplete linkage of different records belonging to one patient. When data are missing related to patient- or provider-specific factors, such as the patient being too sick to seek health care, the missing-at-random assumption is violated. Statistical methods generally used to handle missing data include multiple imputation and inverse probability weighting ( 43 , 45 , 46 ) and have been applied to studies using EHR data ( 11 , 44 ). Another challenge is “nondata” generated by copyand-paste of note information or inappropriate carry-forward of discontinued medications or resolved diagnoses or symptoms. In some situations, it is possible to discern the presence of the workflow that is producing the nondata (such as audit logs for copied text), but defining true data can be challenging.

Temporal Data Complexity

Electronic health records can provide high-resolution, time-stamped longitudinal data. Yet, misinterpretation of such time stamps can inadvertently “leak” future data into predictive models. For example, observational analysis may indicate that hospital length of stay is associated with growth of resistant bacteria in blood cultures, but length of stay would not be useful for point-of-care predictions because it is future information.

More insidious are misleading EHR time stamps, such as clinical progress notes whose contents may have a time stamp corresponding to note initiation rather than to the timing of clinical events. The time between clinical care decisions, note initiation, and note completion may be separated by many hours or even days, and thus the content of the note may reflect knowledge obtained in the future relative to note initiation. Similarly, using a hospital diagnosis-related group (DRG) for sepsis is unlikely to be valid for intrahospital bacteremia predictions, because the DRG codes are routinely assigned posthospitalization by coders after review of completed documentation ( 16 , 47 ). These irregularities warrant clear specification of source and timing of available data elements in EHR-based studies, and whether they would be available in the respective live clinical settings they are intended to apply to.

Data Nonstationarity

Care captured in EHRs changes over time, often rapidly, as a result of the introduction of new tests and therapies, new clinical evidence, changing incentives, and EHR infrastructure alterations (such as changing vendors, modules, or naming standards). In one study predicting future hospital practices, the relevance of EHR data decayed with a half-life of about 4 months for overall practice trends ( 48 ). For individual patient charts, static clinical information can be outdated within a matter of hours ( 49 ). In another example, time variation had a strong effect on the performance of wound healing prediction models ( 50 ). Such change represents nonstationarity, in which the data-generating process changes over time ( 51 ). However, observational studies often report findings from a single snapshot of a data set in time.

Changes in coding and documentation practices or introduction of new EHR software versions also drive data nonstationarity. As a result, study variable definitions developed by using historical data or data from a different source (such as a different health system) may find fewer subjects or the wrong subjects, while associations between treatment and effect may not hold when replicating analyses with different data ( 52 ). Nonstationarity can similarly affect calibration and clinical utility of predictive models ( 53 ). Diagnostics summarizing how longitudinal EHR data sets change over time can support observational study reporting, such as descriptive statistics year-over-year on the prevalence of categories of data (for example laboratory records, procedure records, mortality data) as well as specific data values (for example, the frequency of specific diagnosis codes).

In example 2 (laboratory diagnostic prediction), validation on “future” data may better reflect whether the models will generalize to future data streams than would random cross-validation or hold-out test sets. In other words, researchers should develop models on early years of data while evaluating on later years of data. Furthermore, nonstationarity indicates that models and cohort definitions based on EHR data probably will need to be regularly updated to match current data structures and processes.

Multisite Data Variability and Common Data Models

Reproducibility and replication are well-accepted principles for high-quality observational research but can raise particular challenges for EHR-based studies when different clinical sites use different EHR vendors. Even with a common EHR vendor or otherwise interoperable data structures (for example, Fast Healthcare Interoperability Resources [FHIR] [54]), the idiosyncrasies of local implementation will probably require a laborious, manual, and potentially ambiguous mapping of semantic meaning of data elements. In our laboratory diagnostics example, we wanted to assess reproducibility across multiple sites (Stanford University; University of California, San Francisco; and University of Michigan), requiring manual reconciliation between each site’s slightly different data representations. For example, one site may use the term “WBC,” another “white blood cells,” and yet another “white cells.” Other data have less clear reconciliation options, such as one site consolidating aerobic and anaerobic blood culture tests into a single result while another separates the 2 types of tests, preventing directly comparable results across sites.

Consolidating standard terminologies and common data models (CDMs), such as that used in the Observational Medical Outcomes Partnership (OMOP), can facilitate multisite observational studies ( 55 ). Distributing executable analysis code can in turn provide the most explicit documentation of subtle study design choices and embedded assumptions that may be unclear in the methods sections of study reports. Provision of code enables review by external experts and can promote replication. The use of CDMs can in turn enable researchers to use turnkey tools for EHR data diagnostics and analysis developed within the respective research communities.

Even if CDMs are used, the processes that convert raw EHR data to research variables require careful consideration and documentation because they may introduce unexpected and unquantified variation in data sets, affecting downstream analyses. For example, following OHDSI conventions to convert EHR data to the OMOP CDM involves mapping source diagnosis codes (such as ICD codes) to Systematic Nomenclature of Medicine (SNOMED) codes, but individual sites may define custom mappings such that different ICD codes may be mapped to the same SNOMED code. Such tools as OHDSI Automated Characterization of Health Information at Large-scale Longitudinal Evidence Systems (ACHILLES) ( 56 ) provide a mechanism to generate reports on data quality by flagging potential errors, such as implausible dates or missing data fields. Other research collaboratives, such as the National Patient-Centered Clinical Research Network (PCORnet) or Sentinel Initiative, have developed related approaches and frameworks for data quality assessment ( 57 – 59 ). In cases where site-to-site variability is directly measurable, such as that introduced by different mappings between terminologies, researchers should consider analyses to quantify and report the effect of site variability on measured associations.

In conclusion, EHRs contain large quantities of real-world health care data and are an increasingly important data resource for observational research. Yet, analysis of data collected for nonresearch purposes requires consideration of data quality and observational research and reporting principles. Most of the important considerations for observational research using EHRs are the same as for observational research using other data and are well-addressed by existing recommendations. In the Table , we summarize the considerations for EHR-based observational research that we discussed in this article and provide suggestions for reporting on these issues. Our hope is that the principled application of existing research and reporting guidelines alongside these additional considerations will improve the quality of EHR-based observational studies that drive continuously learning health care systems ( 60 ).

Recommendations for additional research and reporting considerations for observational research conducted using EHR data.


Financial Support: Drs. Callahan and Shah are supported by the National Institutes of Health (NIH) National Library of Medicine under award 5R01LM01136906 and the NIH National Institute of General Medical Sciences under award 5R01GM10143005. Dr. Chen is supported by the NIH Big Data 2 Knowledge initiative via the National Institute of Environmental Health Sciences under award K01ES026837.

Contributor Information

Alison Callahan, Center for Biomedical Informatics Research, School of Medicine, Stanford University.

Nigam H. Shah, Center for Biomedical Informatics Research, School of Medicine, Stanford University.

Jonathan H. Chen, Division of Hospital Medicine, School of Medicine, Stanford University.


  • The Magazine
  • Newsletters
  • Managing Yourself
  • Managing Teams
  • Work-life Balance
  • The Big Idea
  • Data & Visuals
  • Reading Lists
  • Case Selections
  • HBR Learning
  • Topic Feeds
  • Account Settings
  • Email Preferences

Why the Tech Industry Won’t Disrupt Health Care

  • John Glaser,
  • Sara Vaezy,
  • Janet Guptill

observational research health care

Digital transformation will most likely come from established health systems. Here’s what they need to do.

At first glance, it looks like health care in the United States is ripe for disruption. Digital technology advances have the power to help address the shortcomings of care delivery: It costs too much, its quality isn’t what it could and should be, and millions of people live hundreds of miles from the nearest hospital and/or don’t have a primary care doctor. But for many reasons, the incumbents — established health systems — will be extremely hard to displace. Instead, the winners will be health systems that team up with digital tech companies.

We’ve all watched digital innovators demolish certain industries: video and record stores, neighborhood movie theaters, travel agents. A one-time #18 on the Fortune 500, photo film giant Eastman Kodak was felled by digital photography. But some industries and players successfully fend off digital competitors and incorporate their innovations into their daily operations. An Economist article observed that even though most banking has moved online, the average large bank is 138 years old. Walmart, the world’s largest brick-and-mortar retailer, is also the second-largest online retailer.

  • John Glaser is an executive in residence at Harvard Medical School. He previously served as the CIO of Partners Healthcare (now Mass General Brigham), a senior vice president at Cerner, and the CEO of Siemens Health Services. He is co-chair of the HL7 Advisory Council and a board member of the National Committee for Quality Assurance.
  • Sara Vaezy is executive vice president and chief strategy and digital officer at Providence, where she is responsible for corporate strategy, artificial intelligence strategy, marketing, digital, and experience for the integrated delivery network, which includes 51 hospitals and 1,000 clinics serving 5 million patients annually. She also is a member of the National Committee for Quality Assurance’s board of directors and the Harvard Executive Education faculty.
  • Janet Guptill is president and CEO of the Scottsdale Institute, a not-for-profit organization dedicated to helping its more than 60 large, integrated health systems leverage information and technology to create effective, affordable, and equitable health care centered on whole person care.

Partner Center

The role of structured observational research in health care


  • 1 Interagency Working Directorate, National Patient Safety Agency, London, UK. [email protected]
  • PMID: 14645890
  • PMCID: PMC1765776
  • DOI: 10.1136/qhc.12.suppl_2.ii13

Structured observational research involves monitoring of healthcare domains by experts to collect data on errors, adverse events, near misses, team performance, and organisational culture. This paper describes some of the results of structured observational studies carried out in health care. It evaluates the strengths, weaknesses, and future challenges facing observational researchers by drawing lessons from the human factors and neonatal arterial switch operation (ASO) study in which two human factors specialists observed paediatric cardiac surgical procedures in 16 UK centres. Lessons learned from the ASO study are germane to other research teams embarking on studies that involve observational data collection. Future research needs robust observer training, clear measurable criteria to assess each researcher's domain knowledge, and observational competence. Measures of inter-rater reliability are needed where two or more observers participate in data collection. While it is important to understand the factors that lead to error and excellence among healthcare teams, it is also necessary to understand the characteristics of a good observer and the key types of error that can occur during structured observational studies like the human factors and ASO project.

Publication types

  • Multicenter Study
  • Research Support, Non-U.S. Gov't
  • Cardiac Surgical Procedures / standards
  • Delivery of Health Care / standards*
  • Health Services Research / methods*
  • Medical Errors*
  • Observation
  • Organizational Culture*
  • Pediatrics / standards
  • Reproducibility of Results
  • United Kingdom

College of Nursing

Driving change: a case study of a dnp leader in residence program in a gerontological center of excellence.

View as pdf A later version of this article appeared in Nurse Leader , Volume 21, Issue 6 , December 2023 . 

The American Association of Colleges of Nursing (AACN) published the Essentials of Doctoral Education for Advanced Practice Nursing in 2004 identifying the essential curriculum needed for preparing advanced practice nurse leaders to effectively assess organizations, identify systemic issues, and facilitate organizational changes. 1 In 2021, AACN updated the curriculum by issuing The Essentials: Core Competencies for Professional Nursing Education to guide the development of competency-based education for nursing students. 1 In addition to AACN’s competency-based approach to curriculum, in 2015 the American Organization of Nurse Leaders (AONL) released Nurse Leader Core Competencies (updated in 2023) to help provide a competency based model to follow in developing nurse leaders. 2

Despite AACN and AONL competency-based curriculum and model, it is still common for nurse leaders to be promoted to management positions based solely on their work experience or exceptional clinical skills, rather than demonstration of management and leadership competencies. 3 The importance of identifying, training, and assessing executive leaders through formal leadership development programs, within supportive organizational cultures has been discussed by national leaders. As well as the need for nurturing emerging leaders through fostering interprofessional collaboration, mentorship, and continuous development of leadership skills has been identified. 4 As Doctor of Nursing Practice (DNP) nurse leaders assume executive roles within healthcare organizations, they play a vital role within complex systems. Demonstration of leadership competence and participation in formal leadership development programs has become imperative for their success. However, models of competency-based executive leadership development programs can be hard to find, particularly programs outside of health care systems.

The implementation of a DNP Leader in Residence program, such as the one designed for The Barbara and Richard Csomay Center for Gerontological Excellence, addresses many of the challenges facing new DNP leaders and ensures mastery of executive leadership competencies and readiness to practice through exposure to varied experiences and close mentoring. The Csomay Center , based at The University of Iowa, was established in 2000 as one of the five original Hartford Centers of Geriatric Nursing Excellence in the country. Later funding by the Csomay family established an endowment that supports the Center's ongoing work. The current Csomay Center strategic plan and mission aims to develop future healthcare leaders while promoting optimal aging and quality of life for older adults. The Csomay Center Director created the innovative DNP Leader in Residence program to foster the growth of future nurse leaders in non-healthcare systems. The purpose of this paper is to present a case study of the development and implementation of the Leader in Residence program, followed by suggested evaluation strategies, and discussion of future innovation of leadership opportunities in non-traditional health care settings.

Development of the DNP Leader in Residence Program

The Plan-Do-Study-Act (PDSA) cycle has garnered substantial recognition as a valuable tool for fostering development and driving improvement initiatives. 5 The PDSA cycle can function as an independent methodology and as an integral component of broader quality enhancement approaches with notable efficacy in its ability to facilitate the rapid creation, testing, and evaluation of transformative interventions within healthcare. 6 Consequently, the PDSA cycle model was deemed fitting to guide the development and implementation of the DNP Leader in Residence Program at the Csomay Center.

PDSA Cycle: Plan

Existing resources. The DNP Health Systems: Administration/Executive Leadership Program offered by the University of Iowa is comprised of comprehensive nursing administration and leadership curriculum, led by distinguished faculty composed of national leaders in the realms of innovation, health policy, leadership, clinical education, and evidence-based practice. The curriculum is designed to cultivate the next generation of nursing executive leaders, with emphasis on personalized career planning and tailored practicum placements. The DNP Health Systems: Administration/Executive Leadership curriculum includes a range of courses focused on leadership and management with diverse topics such as policy an law, infrastructure and informatics, finance and economics, marketing and communication, quality and safety, evidence-based practice, and social determinants of health. The curriculum is complemented by an extensive practicum component and culminates in a DNP project with additional hours of practicum.

New program. The DNP Leader in Residence program at the Csomay Center is designed to encompass communication and relationship building, systems thinking, change management, transformation and innovation, knowledge of clinical principles in the community, professionalism, and business skills including financial, strategic, and human resource management. The program fully immerses students in the objectives of the DNP Health Systems: Administration/Executive Leadership curriculum and enables them to progressively demonstrate competencies outlined by AONL. The Leader in Residence program also includes career development coaching, reflective practice, and personal and professional accountability. The program is integrated throughout the entire duration of the Leader in Residence’s coursework, fulfilling the required practicum hours for both the DNP coursework and DNP project.

The DNP Leader in Residence program begins with the first semester of practicum being focused on completing an onboarding process to the Center including understanding the center's strategic plan, mission, vision, and history. Onboarding for the Leader in Residence provides access to all relevant Center information and resources and integration into the leadership team, community partnerships, and other University of Iowa College of Nursing Centers associated with the Csomay Center. During this first semester, observation and identification of the Csomay Center Director's various roles including being a leader, manager, innovator, socializer, and mentor is facilitated. In collaboration with the Center Director (a faculty position) and Center Coordinator (a staff position), specific competencies to be measured and mastered along with learning opportunities desired throughout the program are established to ensure a well-planned and thorough immersion experience.

Following the initial semester of practicum, the Leader in Residence has weekly check-ins with the Center Director and Center Coordinator to continue to identify learning opportunities and progression through executive leadership competencies to enrich the experience. The Leader in Residence also undertakes an administrative project for the Center this semester, while concurrently continuing observations of the Center Director's activities in local, regional, and national executive leadership settings. The student has ongoing participation and advancement in executive leadership roles and activities throughout the practicum, creating a well-prepared future nurse executive leader.

After completing practicum hours related to the Health Systems: Administration/Executive Leadership coursework, the Leader in Residence engages in dedicated residency hours to continue to experience domains within nursing leadership competencies like communication, professionalism, and relationship building. During residency hours, time is spent with the completion of a small quality improvement project for the Csomay Center, along with any other administrative projects identified by the Center Director and Center Coordinator. The Leader in Residence is fully integrated into the Csomay Center's Leadership Team during this phase, assisting the Center Coordinator in creating agendas and leading meetings. Additional participation includes active involvement in community engagement activities and presenting at or attending a national conference as a representative of the Csomay Center. The Leader in Residence must mentor a master’s in nursing student during the final year of the DNP Residency.

Implementation of the DNP Leader in Residence Program

PDSA Cycle: Do

Immersive experience. In this case study, the DNP Leader in Residence was fully immersed in a wide range of center activities, providing valuable opportunities to engage in administrative projects and observe executive leadership roles and skills during practicum hours spent at the Csomay Center. Throughout the program, the Leader in Residence observed and learned from multidisciplinary leaders at the national, regional, and university levels who engaged with the Center. By shadowing the Csomay Center Director, the Leader in Residence had the opportunity to observe executive leadership objectives such as fostering innovation, facilitating multidisciplinary collaboration, and nurturing meaningful relationships. The immersive experience within the center’s activities also allowed the Leader in Residence to gain a deep understanding of crucial facets such as philanthropy and community engagement. Active involvement in administrative processes such as strategic planning, budgeting, human resources management, and the development of standard operating procedures provided valuable exposure to strategies that are needed to be an effective nurse leader in the future.

Active participation. The DNP Leader in Residence also played a key role in advancing specific actions outlined in the center's strategic plan during the program including: 1) the creation of a membership structure for the Csomay Center and 2) successfully completing a state Board of Regents application for official recognition as a distinguished center. The Csomay Center sponsored membership for the Leader in Residence in the Midwest Nurse Research Society (MNRS), which opened doors to attend the annual MNRS conference and engage with regional nursing leadership, while fostering socialization, promotion of the Csomay Center and Leader in Residence program, and observation of current nursing research. Furthermore, the Leader in Residence participated in the strategic planning committee and engagement subcommittee for MNRS, collaborating directly with the MNRS president. Additional active participation by the Leader in Residence included attendance in planning sessions and completion of the annual report for GeriatricPain.org , an initiative falling under the umbrella of the Csomay Center. Finally, the Leader in Residence was involved in archiving research and curriculum for distinguished nursing leader and researcher, Dr. Kitty Buckwalter, for the Benjamin Rose Institute on Aging, the University of Pennsylvania Barbara Bates Center for the Study of the History of Nursing, and the University of Iowa library archives.

Suggested Evaluation Strategies of the DNP Leader in Residence Program

PDSA Cycle: Study

Assessment and benchmarking. To effectively assess the outcomes and success of the DNP Leader in Residence Program, a comprehensive evaluation framework should be used throughout the program. Key measures should include the collection and review of executive leadership opportunities experienced, leadership roles observed, and competencies mastered. The Leader in Residence is responsible for maintaining detailed logs of their participation in center activities and initiatives on a semester basis. These logs serve to track the progression of mastery of AONL competencies by benchmarking activities and identifying areas for future growth for the Leader in Residence.

Evaluation. In addition to assessment and benchmarking, evaluations need to be completed by Csomay Center stakeholders (leadership, staff, and community partners involved) and the individual Leader in Residence both during and upon completion of the program. Feedback from stakeholders will identify the contributions made by the Leader in Residence and provide valuable insights into their growth. Self-reflection on experiences by the individual Leader in Residence throughout the program will serve as an important measure of personal successes and identify gaps in the program. Factors such as career advancement during the program, application of curriculum objectives in the workplace, and prospects for future career progression for the Leader in Residence should be considered as additional indicators of the success of the program.

The evaluation should also encompass a thorough review of the opportunities experienced during the residency, with the aim of identifying areas for potential expansion and enrichment of the DNP Leader in Residence program. By carefully examining the logs, reflecting on the acquired executive leadership competencies, and studying stakeholder evaluations, additional experiences and opportunities can be identified to further enhance the program's efficacy. The evaluation process should be utilized to identify specific executive leadership competencies that require further immersion and exploration throughout the program.

Future Innovation of DNP Leader in Residence Programs in Non-traditional Healthcare Settings

PDSA Cycle: Act

As subsequent residents complete the program and their experiences are thoroughly evaluated, it is essential to identify new opportunities for DNP Leader in Residence programs to be implemented in other non-health care system settings. When feasible, expansion into clinical healthcare settings, including long-term care and acute care environments, should be pursued. By leveraging the insights gained from previous Leaders in Residence and their respective experiences, the program can be refined to better align with desired outcomes and competencies. These expansions will broaden the scope and impact of the program and provide a wider array of experiences and challenges for future Leaders in Residency to navigate, enriching their development as dynamic nurse executive leaders within diverse healthcare landscapes.

This case study presented a comprehensive overview of the development and implementation of the DNP Leader in Residence program developed by the Barbara and Richard Csomay Center for Gerontological Excellence. The Leader in Residence program provided a transformative experience by integrating key curriculum objectives, competency-based learning, and mentorship by esteemed nursing leaders and researchers through successful integration into the Center. With ongoing innovation and application of the PDSA cycle, the DNP Leader in Residence program presented in this case study holds immense potential to help better prepare 21 st century nurse leaders capable of driving positive change within complex healthcare systems.


         The author would like to express gratitude to the Barbara and Richard Csomay Center for Gerontological Excellence for the fostering environment to provide an immersion experience and the ongoing support for development of the DNP Leader in Residence program. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

  • American Association of Colleges of Nursing. The essentials: core competencies for professional nursing education. https://www.aacnnursing.org/Portals/42/AcademicNursing/pdf/Essentials-2021.pdf . Accessed June 26, 2023.
  • American Organization for Nursing Leadership. Nurse leader core competencies. https://www.aonl.org/resources/nurse-leader-competencies . Accessed July 10, 2023.
  • Warshawsky, N, Cramer, E. Describing nurse manager role preparation and competency: findings from a national study. J Nurs Adm . 2019;49(5):249-255. DOI:  10.1097/NNA.0000000000000746
  • Van Diggel, C, Burgess, A, Roberts, C, Mellis, C. Leadership in healthcare education. BMC Med. Educ . 2020;20(465). doi: 10.1186/s12909-020-02288-x
  • Institute for Healthcare Improvement. Plan-do-study-act (PDSA) worksheet. https://www.ihi.org/resources/Pages/Tools/PlanDoStudyActWorksheet.aspx . Accessed July 4, 2023.
  • Taylor, M, McNicolas, C, Nicolay, C, Darzi, A, Bell, D, Reed, J. Systemic review of the application of the plan-do-study-act method to improve quality in healthcare. BMJ Quality & Safety. 2014:23:290-298. doi: 10.1136/bmjqs-2013-002703

Return to College of Nursing Winter 23/24 Newsletter

An 8 percent vaccination rate, Long COVID, free treatment options explained by healthcare professionals

observational research health care

AUGUSTA COUNTY  – Since the start of the pandemic, over one million Americans have died due to COVID-19 .

The Center for Disease Control (CDC) public health emergency ended last May , but COVID has not left.

The University of Virginia held a press briefing in January with Dr. Costi Sifri, who outlined what the current protocols and data tell doctors about the virus.

“COVID hasn’t changed,” Sifri said. “Maybe we’ve changed or our attitude toward COVID has changed, but it remains a wickedly transmissible virus. It does not take much to go from one person to another.

The News Leader also spoke to Laura Lee Wight, public information officer for the Central Shenandoah Health District, to find out where the Shenandoah Valley stands in the fourth year of COVID.

Is the pandemic over?

When asked if COVID is still a pandemic, Sifri said, “That’s a complex question.”

Similar to the flu, COVID-19 resembles a seasonal virus. For the past several years, including 2023, there has been a winter spike in case numbers. The seasonal overlap between three different respiratory illnesses means that doctors should be testing for more than just illness when someone comes in with symptoms.

“I can’t tell you the number of patients I’ve met or talked with who are convinced they have one virus, get tested for one virus, then find out they have an infection due to a different virus,” Sifri said. “They think they have the flu, and it’s COVID, or they think they have RSV but it’s influenza.”

When people gather, there is a chance of an outbreak, such as an “August swell” of cases around the same time schools were starting the new year.

When a new variant emerges, the case load “seems” to go up. The current prominent strain of COVID is JN-1, an Omicron descendant, as of the end of January. This variant accounted for 86% of COVID cases tested across the United States. Previous COVID tests can still detect JN-1, so there’s no need to throw out the older tests in the medicine cabinet.

“There's no evidence that the current variants are really causing more severe disease compared to others,” Wight said.

Case data is not what it used to be

Data on the virus is harder to get now than it was in summer 2020. The CDC’s public health emergency allowed the agency to collect more public health data than normal, but the expiration means this expanded access is no longer available.

“We don’t have as much information, in terms of the genomic makeup of strains that are circulating in the United States and worldwide, anymore, in addition to just the raw case counts,” Sifri said. “We don’t have the sort of fidelity of information that we once had in tracking what’s happening with this virus.”

“So many people now do at home tests,” Wight further explained. “The case count numbers that we have really don't accurately show trends.”

Instead, the case counts are determined by hospitalization rates, emergency room, and urgent care visit counts. However, acute symptoms during the initial infection are no longer all doctors are worried about.

Long COVID can fatigue someone for months, even years

Research into Long COVID data has outlined an illness that can follow a person for much longer than a few weeks.

What is Long COVID? Wight told The News Leader, “We're talking about post-COVID conditions that can last for weeks, months, or even years for some individuals who've had COVID.”

About one in four adults with Long COVID report “significant activity limitations,” Wight said, citing the Center for Disease Control. Sifri called the symptoms "really impactful.” A prominent example is Senator Tim Kaine, who spoke with CBS about his symptoms and legislative efforts to get assistance to those with Long COVID in 2022 .

Fatigue , fever, and headaches are common signs of Long COVID, but Wight explained, “It could be related to your lungs, your heart, your GI or gastrointestinal tract, or even your nervous system.” The symptoms can begin immediately after the initial COVID infection, or they could begin weeks afterwards.

A mechanism underlying Long COVID symptoms has not been found. Sifri limited his explanation, “We do know that Long COVID occurs and can occur after bouts of previous COVID infection.”

“There is still a lot of a lot that we don't know with Long COVID,” Wight said.

The National Institute of Health (NIH) created the RECOVER Initiative to study and test treatments against Long COVID. The program offers information about the illness, such as additional symptom descriptions and a guide to talking to a physician about a child's Long COVID symptoms , as well as recruiting for both observational studies and clinical trials.

COVID healthcare is newly commercialized

Day-to-day prevention now looks the same as it did during the pandemic. Sifri explained, “We did some social distancing, we were very cognizant of washing and cleaning our hands. Those are practices that still work. If you don’t want to get these respiratory viruses, those are things to consider.”

The caseload in late January led Sifri to return to the most visible old habit, “I’m masking now. If I’m in a location with 200 or 300 of my closest friends in a shopping mall or at the store, this is maybe not the worst time to don a mask. We know they work. The better the mask, the better the protection.”

This is not the only protection though. Each year, a new version of the COVID vaccine is released, calibrated to the most common variants of the virus than the previous vaccine. Wight emphasized to The News Leader, “The updated vaccine is considered to be the most important protection the most, best protection against really severe COVID-19 symptoms, potential hospitalization, and potential death.”

Vaccination rates have plummeted since the initial vaccine rollout in 2021. In Staunton, Augusta County, and Waynesboro, the initial vaccination rates ranged from 60% to 75%.

The rate has dropped to 8%, as of the end of January.

While Wight could not provide a definite way to explain the drop off, she did point to changes in the 2023/2024 vaccine rollout. The vaccine was “commercialized,” meaning treatments and supplies are no longer being purchased by the government but are handled by the traditional health care marketplace of doctors and insurance. Wight speculated this “could have impacted people's understanding of where they can gain access to the vaccine or how insurances are covering the vaccine.”

Those looking for a vaccine, even if they don’t have health insurance, are able to get one. Wight explained, “I will say that you can receive the COVID-19 vaccine if you're insured, or uninsured, at the at your local health department locations, as well as some pharmacies and some primary care providers.”

Vaccination sites can still be found at vaccines.gov.

When should someone go to the doctor?

“If you’re having high fevers that aren’t breaking or having breathing issues, unrelenting cough, shortness of breath, chest pain, other types of pain, confusion, all of those things would be important to know and I think to seek medical care right away,” Sifri said.

This also applies if someone’s skin, lips, or nail beds are turning blue, due to lack of oxygen, and sleeping disruptions, unable to stay asleep or wake up fully, Wight added.

Treatments are available for COVID and the earlier they can be administered, the better the outcomes.

The Home Test to Treat program from NIH offers “free virtual care and treatment for COVID-19 and flu,” Wight highlighted, saying, “That's a really great resource for individuals who might not have access to a healthcare provider readily.”

Another thing that hasn’t changed are the isolation recommendations. When someone catches COVID-19, they should isolate at home for five days, then wear a well-fitting mask around other people for five more days.

“If you do test positive for COVID, stay home, even if your symptoms are mild,” Wight said. “We want to make sure we're reducing the spread.”

Lyra Bordelon (she/her) is the public transparency and justice reporter at The News Leader. Do you have a story tip or feedback? It’s welcome through email to  [email protected] . Subscribe to us at  newsleader.com .

More: UVA pain research suggests new ways to manage migraine, chronic pain in women

More: Woman convicted of murder in Staunton toddler's death to be sentenced Thursday

observational research health care

Study record managers: refer to the Data Element Definitions if submitting registration or results information.

Search for terms


  • Advanced Search
  • See Studies by Topic
  • See Studies on Map
  • How to Search
  • How to Use Search Results
  • How to Find Results of Studies
  • How to Read a Study Record

About Studies Menu

  • Learn About Studies
  • Other Sites About Studies
  • Glossary of Common Site Terms

Submit Studies Menu

  • Submit Studies to ClinicalTrials.gov PRS
  • Why Should I Register and Submit Results?
  • FDAAA 801 and the Final Rule
  • How to Apply for a PRS Account
  • How to Register Your Study
  • How to Edit Your Study Record
  • How to Submit Your Results
  • Frequently Asked Questions
  • Support Materials
  • Training Materials

Resources Menu

  • Selected Publications
  • Clinical Alerts and Advisories
  • Trends, Charts, and Maps
  • Downloading Content for Analysis

About Site Menu

  • ClinicalTrials.gov Background
  • About the Results Database
  • History, Policies, and Laws
  • ClinicalTrials.gov Modernization
  • Media/Press Resources
  • Linking to This Site
  • Terms and Conditions
  • Search Results
  • Study Record Detail

Maximum Saved Studies Reached

An Open Comparative Study of the Effectiveness and Incomparable Study of the Immunogenicity and Safety of the Vaccine (CoviVac) for Adults Aged 60 Years and Older

  • Study Details
  • Tabular View
  • No Results Posted


Inclusion Criteria:

Volunteers must meet the following inclusion criteria:

Type of participants

• Healthy volunteers or volunteers with a history of stable diseases that do not meet any of the criteria for non-inclusion in the study.

Other inclusion criteria

  • Written informed consent of volunteers to participate in a clinical trial
  • Volunteers who are able to fulfill the Protocol requirements (i.e., fill out a self-observation Diary, come to control visits).

Exclusion Criteria:

SARS-CoV-2 infection • A case of established COVID-19 disease confirmed by PCR and/or ELISA in the last 6 months.

Diseases or medical conditions

  • Serious post-vaccination reaction (temperature above 40 C, hyperemia or edema more than 8 cm in diameter) or complication (collapse or shock-like condition that developed within 48 hours after vaccination; convulsions, accompanied or not accompanied by a feverish state) to any previous vaccination.
  • Burdened allergic history (anaphylactic shock, Quincke's edema, polymorphic exudative eczema, serum sickness in the anamnesis, hypersensitivity or allergic reactions to the introduction of any vaccines in the anamnesis, known allergic reactions to vaccine components, etc.).
  • Guillain-Barre syndrome (acute polyradiculitis) in the anamnesis.
  • The axillary temperature at the time of vaccination is more than 37.0 ° C.
  • Acute infectious diseases (recovery earlier than 4 weeks before vaccination) according to anamnesis.
  • Donation of blood or plasma (in the amount of 450 ml or more) less than 2 months before inclusion in the study.
  • Severe and/or uncontrolled diseases of the cardiovascular, bronchopulmonary, neuroendocrine systems, gastrointestinal tract, liver, kidneys, hematopoietic, immune systems.
  • Is registered at the dispensary for tuberculosis, leukemia, oncological diseases, autoimmune diseases.
  • Any confirmed or suspected immunosuppressive or immunodeficiency condition in the anamnesis.
  • Splenectomy in the anamnesis.
  • Neutropenia (decrease in the absolute number of neutrophils less than 1000/mm3), agranulocytosis, significant blood loss, severe anemia (hemoglobin less than 80 g/l) according to anamnesis.
  • Anorexia according to anamnesis.

Prior or concomitant therapy

  • Vaccination with any vaccine carried out within 30 days before vaccination / the first dose of the studied vaccine or planned administration within 30 days after vaccination / the last dose of the studied vaccine.
  • Prior vaccination with an experimental or registered vaccine that may affect the interpretation of the study data (any coronavirus or SARS vaccines).
  • Long-term use (more than 14 days) of immunosuppressants or other immunomodulatory drugs (immunoregulatory peptides, cytokines, interferons, immune system effector proteins (immunoglobulins), interferon inducers (cycloferon) during the six months preceding the study, according to anamnesis.
  • Treatment with systemic glucocorticosteroids (≥ 20 mg of prednisone, or an analog, for more than 15 days during the last month).
  • Volunteers who received immunoglobulin preparations or blood transfusion during the last 3 months prior to the start of the study according to anamnesis.

Other non-inclusion criteria

• Participation in any other clinical trial within the last 3 months.

Exclusion criteria:

  • Withdrawal of Informed consent by a volunteer;
  • The volunteer was included in violation of the inclusion/non-inclusion criteria of the Protocol;
  • Any condition of a volunteer that requires, in the reasoned opinion of a medical researcher, the withdrawal of a volunteer from the study;
  • Taking unauthorized medications (see section 6.2);
  • The volunteer refuses to cooperate or is undisciplined (for example, failure to attend a scheduled visit without warning the researcher and/or loss of communication with the volunteer), or dropped out of observation;
  • For administrative reasons (termination of the study by the Sponsor or regulatory authorities), as well as in case of gross violations of the Protocol that may affect the results of the study.
  • For Patients and Families
  • For Researchers
  • For Study Record Managers
  • Customer Support
  • Accessibility
  • Viewers and Players
  • Freedom of Information Act
  • HHS Vulnerability Disclosure
  • U.S. National Library of Medicine
  • U.S. National Institutes of Health
  • U.S. Department of Health and Human Services
  • Enter keyword Search

Free Entry-level Health Care Career Training at OHSU

Cna, dental assistant, medical assistant and phlebotomist programs.

Ready to build a career in health care? OHSU’s Building Bridges free job training and certification program is a great way to start. Take classes to learn and work in one of these health professions:

  • Certified nursing assistant (CNA)
  • Dental assistant
  • Medical assistant
  • Phlebotomist

To be eligible, you:

  • Have a high school diploma or the equivalent.
  • Are 18 or older.
  • Have experience working with the public.

Explore entry-level health care training opportunities

Follow your career path to OHSU with free health care job training and certification. Here’s how to participate:

  • Select a role and apply for training, or submit an interest form .
  • If selected, take classes to learn the role and get certified. Classes last a few weeks to several months.
  • After completing training, apply for the job at OHSU as a priority candidate. If hired, we request you work in that role for at least one year.
  • As an OHSU employee, you’ll have access to exceptional benefits .  

Free CNA training and certification

Prepare for a career as a certified nursing assistant and earn your CNA certification in 6 weeks. Check back to apply for a training group, or submit an interest form to learn more. Please note, training dates may change. 

Group 1: July 2024 

Group 2: July 2025

Explore CNA jobs at OHSU

Free dental assistant training and certification

Prepare for a career as a dental assistant and earn your certification in 9 months. Choose a group and apply for training—or sign up to observe a dental assistant . To learn more, please  submit an interest form . Please note, training dates may change. 

Group 1: April  2024 – December 2024

  • Apply now (OHSU employees)

Group 2: October 2024 – June 2025

Dental assistant job observation

Not sure if being a dental assistant is right for you? Sign up to observe a dental assistant before or after you apply for this training opportunity.

Explore dental assistant jobs at OHSU

Free medical assistant training and certification

Prepare for a career as a medical assistant and earn your certification in 9 months. Check back to apply for a training group, or  submit an interest form  to learn more. Please note, training dates may change. 

Group 1: July 2024 – March 2025

Group 2: June 2025 – February 2026

Explore medical assistant jobs at OHSU

Free phlebotomist training and certification

Prepare for a career as a phlebotomist and earn your phlebotomy certification in 6 weeks. Check back to apply for a training group, or  submit an interest form  to learn more. Please note, training dates may change.

Group 1: April 2024 – May 2024

Group 2: July 2024 – August 2024

Group 3: October 2024 – November 2024

Group 4: January 2025 – February 2025

Group 5: April 2025 – May 2025

Group 6: July 2025 – August 2025

Group 7: October 2025 – November 2025

Explore phlebotomist jobs at OHSU

Frequently Asked Questions

How does the building bridges program work.

OHSU partners with  AFSCME Local 328 ,  United We Heal  and  Unite Oregon  to build job pathways and remove financial barriers for people interested in the health care field. Participants receive free training to become a certified nursing assistant, dental assistant, medical assistant or phlebotomists and help fill these high-demand roles. 

Is training actually free?

Yes, training and certifications through this program are free. Program funds may also cover some of your personal costs, like transportation and childcare.

Who is eligible? 

To be eligible, you have a high school diploma or the equivalent, are 18 or older, and have experience working with the public. If you are an OHSU employee, you have worked here for one year and are in good standing. If you are a represented OHSU employee, you have worked in your role for six months and are in good standing

Is there an application deadline?

You must apply for a job training group more than 45 days before the posted start date

Can I do my training online?

You can take most classes online. Some classes may require in-person attendance in Portland

Are there any prerequisites?

No. As long as you meet eligibility requirements, you can apply.

Will I take a final certification exam?

Yes, after completing your training courses, you can take an exam to get licensed or certified in your new role.

What if I fail courses or need to drop out of the program? 

If you fail a course or can’t complete training, in special circumstances you may be able to repeat the course or re-apply to the program if space is available.

Will I have an OHSU job after I pass training? 

After you successfully complete training, you can apply for that role at OHSU as a priority candidate. Successful dental assistant participants will automatically receive a job offer.

How does OHSU pay for this free training and certification program?

OHSU’s Building Bridges program uses funds it received from a  Future Ready Oregon grant  to offer these job training classes and certifications at no cost to participants. The three-year program is one way OHSU is helping more people build a career in health care.

What is Unite Oregon’s part in Building Bridges?

Through this program,  Unite Oregon  is working with United We Heal and OHSU to create a more diverse health care workforce and increase the pool of skilled, trained BIPOC individuals from marginalized low-income communities. Unite Oregon is a membership organization led by BIPOC individuals, immigrants and refugees, rural communities, and people experiencing poverty. 

What is AFSCME Local 328’s part in the program?

OHSU partnered with  AFSCME Local 328  and United We Heal to secure the Future Ready Oregon education and training grant from the state of Oregon. AFSCME Local 328 represents approximately 8,300 employees at Oregon Health & Science University. Together, we bargain for and enforce a good contract, great benefits and a safe working environment. 

How is United We Heal involved? 

United We Heal (UWH)  Training Trust is partnering with OHSU and Unite Oregon to build training opportunities for people and increase Oregon’s workforce in health care. UWH is a non-profit organization led by an equal number of employer and employee trustees. It connects individuals, including those from traditionally underserved communities, with apprenticeships in the mental health and health care fields.  

Who do I contact if I have questions?

If you have questions about this program, please contact OHSU Human Resources at  [email protected] .

Fill out an interest form

If you’re not ready to apply for training or if you want to receive updates, please complete a short form so we can stay in touch.

Attend a recruitment fair

Join us to learn more about free job training pathways at an upcoming recruitment fair.

Thursday, Feb. 22, 2024 4:30-7:30 p.m.    AFSCME   525 N.E. Oregon St., Room 520   Portland, OR 97232  

Thursday, Feb. 29, 2024 4:30-7:30 p.m.   Tabor Space   5441 S.E. Belmont St.   Portland, OR 97215 

Contact OHSU Human Resources.

About Building Bridges

The Building Bridges program is a partnership between OHSU, AFSCME Local 328, United We Heal, and Unite Oregon. The program applies Future Ready Oregon grant funds to offer job training classes and certifications to help people build careers and fill high-demand roles in health care.


  1. PPT

    observational research health care

  2. PPT

    observational research health care

  3. (PDF) Nurse practitioner consultations in primary health care: An

    observational research health care

  4. The one chart you need to understand any health study

    observational research health care

  5. (PDF) The Observational Research Study with the Trends in Healthcare

    observational research health care

  6. Reporting of Observational Studies

    observational research health care



  2. Observational Study| Research Method, #researchmethodology #shortnotes #bba #bcom



  5. Center for Health Discovery and Well Being Case Study

  6. Nursing Research Matters


  1. Informing Healthcare Decisions with Observational Research Assessing Causal Effect. An Official American Thoracic Society Research Statement

    Objectives: The American Thoracic Society (ATS) created a multidisciplinary ad hoc committee to develop a research statement to clarify the role of observational studies—alongside randomized controlled trials (RCTs)—in informing clinical decisions in pulmonary, critical care, and sleep medicine.

  2. Observational studies and their utility for practice

    Observational studies involve the study of participants without any forced change to their circumstances, that is, without any intervention. 1 Although the participants' behaviour may change under observation, the intent of observational studies is to investigate the 'natural' state of risk factors, diseases or outcomes.

  3. The role of structured observational research in health care

    Structured observational research involves monitoring of healthcare domains by experts to collect data on errors, adverse events, near misses, team performance, and organisational culture. This paper describes some of the results of structured observational studies carried out in health care.

  4. Direct observation methods: A practical guide for health researchers

    Abstract Objective To provide health research teams with a practical, methodologically rigorous guide on how to conduct direct observation. Methods Synthesis of authors' observation-based teaching and research experiences in social sciences and health services research. Results

  5. Using observation to better understand the healthcare context

    Observation is a valuable data collection method for health services researchers to identify key components involved in a topic of interest, a vital step in forming relevant questions, measuring appropriate variables, and designing effective interventions.

  6. Qualitative Research: Observational methods in health care settings

    Qualitative Research: Observational methods in health care settings | The BMJ Education And Debate Qualitative Research: Observational methods in health care settings BMJ 1995 ; 311 doi: https://doi.org/10.1136/bmj.311.6998.182 (Published 15 July 1995) Cite this as: BMJ 1995;311:182 Article Related content Metrics Responses

  7. Case Study Observational Research: A Framework for Conducting Case

    Case study observational research offers a promising approach for researchers in a wide range of health care settings seeking more complete understandings of complex topics, where contextual influences are of primary concern. Future research is needed to refine and evaluate the approach. Get full access to this article

  8. Untapped Potential of Observational Research to Inform Clinical

    Observational studies are those "where the investigators did not assign exposure" to an intervention. 1(p2) In observational research focused on cancer care, investigators observe lifestyle and social traits of patients and clinicians, ongoing treatment choices, adherence to treatment plans, related adverse effects, and the utility of novel biomarkers (including genetic and epigenetic ...

  9. Video-based observation research: A systematic review of studies in

    Abstract Objective To examine the use of video-based observation research in outpatient health care encounter research. Methods

  10. The role of structured observational research in health care

    Structured observational research involves monitoring of healthcare domains by experts to collect data on errors, adverse events, near misses, team performance, and organisational culture....

  11. Chapter 10. Observational Research

    Observational research is a research in which the investigator cannot control the assignment of treatment to subjects because the participants or conditions are not being directly assigned by the researcher. ... or ways to deliver health care. We find such evidence in one of two ways: through experimental studies or through observational ...

  12. OHDSI

    The Observational Health Data Sciences and Informatics (or OHDSI, pronounced "Odyssey") program is a multi-stakeholder, interdisciplinary collaborative to bring out the value of health data through large-scale analytics. All our solutions are open-source.

  13. Observational Methods

    Observation is a core method in ethnography and anthropology. Whilst observational methods can be used to study a range of phenomena, they can be used for different purposes, ranging from foundational studies designed to develop a deeper understanding of neglected issues or less well understood areas, through to research that uses these methods ...

  14. PDF Addressing the need for real-world observational research solutions

    effectiveness research (CER) to address these real-world evidence needs. CER is generally defined as research that is intended to compare two or more interventions or approaches to health care with the goal of generating evidence-based information to assist patients, health care providers and other stakeholders with decision-making.2,3

  15. Ethnicity data resource in population-wide health records ...

    Intersectional social determinants including ethnicity are vital in health research. We curated a population-wide data resource of self-identified ethnicity data from over 60 million individuals ...

  16. Assessing Trust in Health Care: A Compendium of Trust Measures

    Drs. Platt and Taylor set out to address three issues which have fueled the propagation of trust measures: A lack of consensus around a single measure or set of measures. Trust may operate differently depending on who is trusting whom, and what the context is. The intended readers for this guide are (1) health system leaders, organizational ...

  17. JMIR Formative Research

    Background: Emergency departments play a pivotal role in the US health care system, with high use rates and inherent stress placed on patients, patient care, and clinicians. The impact of the emergency department environment on the health and well-being of emergency residents and nurses can be seen in worsening rates of burnout and cardiovascular health.

  18. Research and Reporting Considerations for Observational Studies Using

    Observational research helps to advance clinical knowledge and inform the practice of medicine. Electronic health records (EHRs) contain large quantities of health care data that are captured during care and are an increasingly important resource for conducting observational health research ( 1 ).

  19. UC Davis medical school ranked among the nation's ...

    (SACRAMENTO) The UC Davis School of Medicine is among the nation's leading medical schools in National Institutes of Health (NIH) funding, according to the latest Blue Ridge Institute for Medical Research ranking. The medical school broke its record of NIH funding with a total of more than $209 million, placing it 33 rd nationally. "Research is foundational to our mission of delivering ...

  20. Why the Tech Industry Won't Disrupt Health Care

    Sara Vaezy is executive vice president and chief strategy and digital officer at Providence, where she is responsible for corporate strategy, artificial intelligence strategy, marketing, digital ...

  21. Revealing Disparities: Health Care Workers' Observations of

    A recent study found that health care workers who witness racism by other clinical staff often lack options allowing them to discuss and report such experiences. 5 Our research indicates that health care workers also struggle with the added emotional labor of dealing with discrimination in their workplaces — a heightened concern for those ...

  22. The role of structured observational research in health care

    The role of structured observational research in health care Structured observational research involves monitoring of healthcare domains by experts to collect data on errors, adverse events, near misses, team performance, and organisational culture.

  23. Driving change: a case study of a DNP leader in residence program in a

    College of Nursing Building 50 Newton Road Iowa City, Iowa 52242-1121 319-335-7018 [email protected]

  24. UCSF Tops Public Universities in NIH Research Funding in 2023

    UCSF's total funding of $789,196,651 ranked second among all public and private U.S. universities, after Johns Hopkins University. Leidos Biomedical Research, Inc., which administers research contracts for the National Cancer Institute's Frederick National Laboratory, was the largest recipient overall, receiving $866 million.

  25. Long COVID, an 8 percent vaccination rate, free treatment options

    Research into Long COVID data has outlined an illness that can follow a person for much longer than a few weeks. ... as well as recruiting for both observational studies and clinical ... ," meaning treatments and supplies are no longer being purchased by the government but are handled by the traditional health care marketplace of doctors and ...

  26. Press Release: 2024 Peer Reviewed Orthopaedic Research Program

    Applications may propose several types of research including, but not limited to, translational and clinical research. Qualitative, population science, and health care services research specifically for deployed female Service Members are also encouraged. No FY24 PRORP Focus Areas are required to be addressed for this award mechanism.

  27. An Open Comparative Study of the Effectiveness and Incomparable Study

    State Budgetary Healthcare Institution of the Moscow region "Elektrostal Central City Hospital" Elektrostal, Moscow Oblast, Russian Federation, 144000 : Federal State Budgetary Scientific Institution "I.I. Mechnikov Scientific Research Institute of Vaccines and Serums" Moscow, Russian Federation, 105064 : FSBSI Chumakov FSC R&D IBP RAS

  28. Building Bridges to Health Care Jobs at OHSU

    How does the Building Bridges program work? OHSU partners with AFSCME Local 328, United We Heal and Unite Oregon to build job pathways and remove financial barriers for people interested in the health care field. Participants receive free training to become a certified nursing assistant, dental assistant, medical assistant or phlebotomists and help fill these high-demand roles.