thematic analysis in systematic literature review

• Research article
• Open access
• Published: 10 July 2008

Methods for the thematic synthesis of qualitative research in systematic reviews

• James Thomas 1 &
• Angela Harden 1  

BMC Medical Research Methodology volume  8 , Article number:  45 ( 2008 ) Cite this article

379k Accesses

3840 Citations

102 Altmetric

Metrics details

There is a growing recognition of the value of synthesising qualitative research in the evidence base in order to facilitate effective and appropriate health care. In response to this, methods for undertaking these syntheses are currently being developed. Thematic analysis is a method that is often used to analyse data in primary qualitative research. This paper reports on the use of this type of analysis in systematic reviews to bring together and integrate the findings of multiple qualitative studies.

We describe thematic synthesis, outline several steps for its conduct and illustrate the process and outcome of this approach using a completed review of health promotion research. Thematic synthesis has three stages: the coding of text 'line-by-line'; the development of 'descriptive themes'; and the generation of 'analytical themes'. While the development of descriptive themes remains 'close' to the primary studies, the analytical themes represent a stage of interpretation whereby the reviewers 'go beyond' the primary studies and generate new interpretive constructs, explanations or hypotheses. The use of computer software can facilitate this method of synthesis; detailed guidance is given on how this can be achieved.

We used thematic synthesis to combine the studies of children's views and identified key themes to explore in the intervention studies. Most interventions were based in school and often combined learning about health benefits with 'hands-on' experience. The studies of children's views suggested that fruit and vegetables should be treated in different ways, and that messages should not focus on health warnings. Interventions that were in line with these suggestions tended to be more effective. Thematic synthesis enabled us to stay 'close' to the results of the primary studies, synthesising them in a transparent way, and facilitating the explicit production of new concepts and hypotheses.

We compare thematic synthesis to other methods for the synthesis of qualitative research, discussing issues of context and rigour. Thematic synthesis is presented as a tried and tested method that preserves an explicit and transparent link between conclusions and the text of primary studies; as such it preserves principles that have traditionally been important to systematic reviewing.

Peer Review reports

The systematic review is an important technology for the evidence-informed policy and practice movement, which aims to bring research closer to decision-making [ 1 , 2 ]. This type of review uses rigorous and explicit methods to bring together the results of primary research in order to provide reliable answers to particular questions [ 3 – 6 ]. The picture that is presented aims to be distorted neither by biases in the review process nor by biases in the primary research which the review contains [ 7 – 10 ]. Systematic review methods are well-developed for certain types of research, such as randomised controlled trials (RCTs). Methods for reviewing qualitative research in a systematic way are still emerging, and there is much ongoing development and debate [ 11 – 14 ].

In this paper we present one approach to the synthesis of findings of qualitative research, which we have called 'thematic synthesis'. We have developed and applied these methods within several systematic reviews that address questions about people's perspectives and experiences [ 15 – 18 ]. The context for this methodological development is a programme of work in health promotion and public health (HP & PH), mostly funded by the English Department of Health, at the EPPI-Centre, in the Social Science Research Unit at the Institute of Education, University of London in the UK. Early systematic reviews at the EPPI-Centre addressed the question 'what works?' and contained research testing the effects of interventions. However, policy makers and other review users also posed questions about intervention need, appropriateness and acceptability, and factors influencing intervention implementation. To address these questions, our reviews began to include a wider range of research, including research often described as 'qualitative'. We began to focus, in particular, on research that aimed to understand the health issue in question from the experiences and point of view of the groups of people targeted by HP&PH interventions (We use the term 'qualitative' research cautiously because it encompasses a multitude of research methods at the same time as an assumed range of epistemological positions. In practice it is often difficult to classify research as being either 'qualitative' or 'quantitative' as much research contains aspects of both [ 19 – 22 ]. Because the term is in common use, however, we will employ it in this paper).

When we started the work for our first series of reviews which included qualitative research in 1999 [ 23 – 26 ], there was very little published material that described methods for synthesising this type of research. We therefore experimented with a variety of techniques borrowed from standard systematic review methods and methods for analysing primary qualitative research [ 15 ]. In later reviews, we were able to refine these methods and began to apply thematic analysis in a more explicit way. The methods for thematic synthesis described in this paper have so far been used explicitly in three systematic reviews [ 16 – 18 ].

The review used as an example in this paper

To illustrate the steps involved in a thematic synthesis we draw on a review of the barriers to, and facilitators of, healthy eating amongst children aged four to 10 years old [ 17 ]. The review was commissioned by the Department of Health, England to inform policy about how to encourage children to eat healthily in the light of recent surveys highlighting that British children are eating less than half the recommended five portions of fruit and vegetables per day. While we focus on the aspects of the review that relate to qualitative studies, the review was broader than this and combined answering traditional questions of effectiveness, through reviewing controlled trials, with questions relating to children's views of healthy eating, which were answered using qualitative studies. The qualitative studies were synthesised using 'thematic synthesis' – the subject of this paper. We compared the effectiveness of interventions which appeared to be in line with recommendations from the thematic synthesis with those that did not. This enabled us to see whether the understandings we had gained from the children's views helped us to explain differences in the effectiveness of different interventions: the thematic synthesis had enabled us to generate hypotheses which could be tested against the findings of the quantitative studies – hypotheses that we could not have generated without the thematic synthesis. The methods of this part of the review are published in Thomas et al . [ 27 ] and are discussed further in Harden and Thomas [ 21 ].

Qualitative research and systematic reviews

The act of seeking to synthesise qualitative research means stepping into more complex and contested territory than is the case when only RCTs are included in a review. First, methods are much less developed in this area, with fewer completed reviews available from which to learn, and second, the whole enterprise of synthesising qualitative research is itself hotly debated. Qualitative research, it is often proposed, is not generalisable and is specific to a particular context, time and group of participants. Thus, in bringing such research together, reviewers are open to the charge that they de-contextualise findings and wrongly assume that these are commensurable [ 11 , 13 ]. These are serious concerns which it is not the purpose of this paper to contest. We note, however, that a strong case has been made for qualitative research to be valued for the potential it has to inform policy and practice [ 11 , 28 – 30 ]. In our experience, users of reviews are interested in the answers that only qualitative research can provide, but are not able to handle the deluge of data that would result if they tried to locate, read and interpret all the relevant research themselves. Thus, if we acknowledge the unique importance of qualitative research, we need also to recognise that methods are required to bring its findings together for a wide audience – at the same time as preserving and respecting its essential context and complexity.

The earliest published work that we know of that deals with methods for synthesising qualitative research was written in 1988 by Noblit and Hare [ 31 ]. This book describes the way that ethnographic research might be synthesised, but the method has been shown to be applicable to qualitative research beyond ethnography [ 32 , 11 ]. As well as meta-ethnography, other methods have been developed more recently, including 'meta-study' [ 33 ], 'critical interpretive synthesis' [ 34 ] and 'metasynthesis' [ 13 ].

Many of the newer methods being developed have much in common with meta-ethnography, as originally described by Noblit and Hare, and often state explicitly that they are drawing on this work. In essence, this method involves identifying key concepts from studies and translating them into one another. The term 'translating' in this context refers to the process of taking concepts from one study and recognising the same concepts in another study, though they may not be expressed using identical words. Explanations or theories associated with these concepts are also extracted and a 'line of argument' may be developed, pulling corroborating concepts together and, crucially, going beyond the content of the original studies (though 'refutational' concepts might not be amenable to this process). Some have claimed that this notion of 'going beyond' the primary studies is a critical component of synthesis, and is what distinguishes it from the types of summaries of findings that typify traditional literature reviews [e.g. [ 32 ], p209]. In the words of Margarete Sandelowski, "metasyntheses are integrations that are more than the sum of parts, in that they offer novel interpretations of findings. These interpretations will not be found in any one research report but, rather, are inferences derived from taking all of the reports in a sample as a whole" [[ 14 ], p1358].

Thematic analysis has been identified as one of a range of potential methods for research synthesis alongside meta-ethnography and 'metasynthesis', though precisely what the method involves is unclear, and there are few examples of it being used for synthesising research [ 35 ]. We have adopted the term 'thematic synthesis', as we translated methods for the analysis of primary research – often termed 'thematic' – for use in systematic reviews [ 36 – 38 ]. As Boyatzis [[ 36 ], p4] has observed, thematic analysis is "not another qualitative method but a process that can be used with most, if not all, qualitative methods..." . Our approach concurs with this conceptualisation of thematic analysis, since the method we employed draws on other established methods but uses techniques commonly described as 'thematic analysis' in order to formalise the identification and development of themes.

We now move to a description of the methods we used in our example systematic review. While this paper has the traditional structure for reporting the results of a research project, the detailed methods (e.g. precise terms we used for searching) and results are available online. This paper identifies the particular issues that relate especially to reviewing qualitative research systematically and then to describing the activity of thematic synthesis in detail.

When searching for studies for inclusion in a 'traditional' statistical meta-analysis, the aim of searching is to locate all relevant studies. Failing to do this can undermine the statistical models that underpin the analysis and bias the results. However, Doyle [[ 39 ], p326] states that, "like meta-analysis, meta-ethnography utilizes multiple empirical studies but, unlike meta-analysis, the sample is purposive rather than exhaustive because the purpose is interpretive explanation and not prediction" . This suggests that it may not be necessary to locate every available study because, for example, the results of a conceptual synthesis will not change if ten rather than five studies contain the same concept, but will depend on the range of concepts found in the studies, their context, and whether they are in agreement or not. Thus, principles such as aiming for 'conceptual saturation' might be more appropriate when planning a search strategy for qualitative research, although it is not yet clear how these principles can be applied in practice. Similarly, other principles from primary qualitative research methods may also be 'borrowed' such as deliberately seeking studies which might act as negative cases, aiming for maximum variability and, in essence, designing the resulting set of studies to be heterogeneous, in some ways, instead of achieving the homogeneity that is often the aim in statistical meta-analyses.

However you look, qualitative research is difficult to find [ 40 – 42 ]. In our review, it was not possible to rely on simple electronic searches of databases. We needed to search extensively in 'grey' literature, ask authors of relevant papers if they knew of more studies, and look especially for book chapters, and we spent a lot of effort screening titles and abstracts by hand and looking through journals manually. In this sense, while we were not driven by the statistical imperative of locating every relevant study, when it actually came down to searching, we found that there was very little difference in the methods we had to use to find qualitative studies compared to the methods we use when searching for studies for inclusion in a meta-analysis.

Quality assessment

Assessing the quality of qualitative research has attracted much debate and there is little consensus regarding how quality should be assessed, who should assess quality, and, indeed, whether quality can or should be assessed in relation to 'qualitative' research at all [ 43 , 22 , 44 , 45 ]. We take the view that the quality of qualitative research should be assessed to avoid drawing unreliable conclusions. However, since there is little empirical evidence on which to base decisions for excluding studies based on quality assessment, we took the approach in this review to use 'sensitivity analyses' (described below) to assess the possible impact of study quality on the review's findings.

In our example review we assessed our studies according to 12 criteria, which were derived from existing sets of criteria proposed for assessing the quality of qualitative research [ 46 – 49 ], principles of good practice for conducting social research with children [ 50 ], and whether studies employed appropriate methods for addressing our review questions. The 12 criteria covered three main quality issues. Five related to the quality of the reporting of a study's aims, context, rationale, methods and findings (e.g. was there an adequate description of the sample used and the methods for how the sample was selected and recruited?). A further four criteria related to the sufficiency of the strategies employed to establish the reliability and validity of data collection tools and methods of analysis, and hence the validity of the findings. The final three criteria related to the assessment of the appropriateness of the study methods for ensuring that findings about the barriers to, and facilitators of, healthy eating were rooted in children's own perspectives (e.g. were data collection methods appropriate for helping children to express their views?).

Extracting data from studies

One issue which is difficult to deal with when synthesising 'qualitative' studies is 'what counts as data' or 'findings'? This problem is easily addressed when a statistical meta-analysis is being conducted: the numeric results of RCTs – for example, the mean difference in outcome between the intervention and control – are taken from published reports and are entered into the software package being used to calculate the pooled effect size [ 3 , 51 ].

Deciding what to abstract from the published report of a 'qualitative' study is much more difficult. Campbell et al . [ 11 ] extracted what they called the 'key concepts' from the qualitative studies they found about patients' experiences of diabetes and diabetes care. However, finding the key concepts in 'qualitative' research is not always straightforward either. As Sandelowski and Barroso [ 52 ] discovered, identifying the findings in qualitative research can be complicated by varied reporting styles or the misrepresentation of data as findings (as for example when data are used to 'let participants speak for themselves'). Sandelowski and Barroso [ 53 ] have argued that the findings of qualitative (and, indeed, all empirical) research are distinct from the data upon which they are based, the methods used to derive them, externally sourced data, and researchers' conclusions and implications.

In our example review, while it was relatively easy to identify 'data' in the studies – usually in the form of quotations from the children themselves – it was often difficult to identify key concepts or succinct summaries of findings, especially for studies that had undertaken relatively simple analyses and had not gone much further than describing and summarising what the children had said. To resolve this problem we took study findings to be all of the text labelled as 'results' or 'findings' in study reports – though we also found 'findings' in the abstracts which were not always reported in the same way in the text. Study reports ranged in size from a few pages to full final project reports. We entered all the results of the studies verbatim into QSR's NVivo software for qualitative data analysis. Where we had the documents in electronic form this process was straightforward even for large amounts of text. When electronic versions were not available, the results sections were either re-typed or scanned in using a flat-bed or pen scanner. (We have since adapted our own reviewing system, 'EPPI-Reviewer' [ 54 ], to handle this type of synthesis and the screenshots below show this software.)

Detailed methods for thematic synthesis

The synthesis took the form of three stages which overlapped to some degree: the free line-by-line coding of the findings of primary studies; the organisation of these 'free codes' into related areas to construct 'descriptive' themes; and the development of 'analytical' themes.

Stages one and two: coding text and developing descriptive themes

In our children and healthy eating review, we originally planned to extract and synthesise study findings according to our review questions regarding the barriers to, and facilitators of, healthy eating amongst children. It soon became apparent, however, that few study findings addressed these questions directly and it appeared that we were in danger of ending up with an empty synthesis. We were also concerned about imposing the a priori framework implied by our review questions onto study findings without allowing for the possibility that a different or modified framework may be a better fit. We therefore temporarily put our review questions to one side and started from the study findings themselves to conduct an thematic analysis.

There were eight relevant qualitative studies examining children's views of healthy eating. We entered the verbatim findings of these studies into our database. Three reviewers then independently coded each line of text according to its meaning and content. Figure 1 illustrates this line-by-line coding using our specialist reviewing software, EPPI-Reviewer, which includes a component designed to support thematic synthesis. The text which was taken from the report of the primary study is on the left and codes were created inductively to capture the meaning and content of each sentence. Codes could be structured, either in a tree form (as shown in the figure) or as 'free' codes – without a hierarchical structure.

line-by-line coding in EPPI-Reviewer.

The use of line-by-line coding enabled us to undertake what has been described as one of the key tasks in the synthesis of qualitative research: the translation of concepts from one study to another [ 32 , 55 ]. However, this process may not be regarded as a simple one of translation. As we coded each new study we added to our 'bank' of codes and developed new ones when necessary. As well as translating concepts between studies, we had already begun the process of synthesis (For another account of this process, see Doyle [[ 39 ], p331]). Every sentence had at least one code applied, and most were categorised using several codes (e.g. 'children prefer fruit to vegetables' or 'why eat healthily?'). Before completing this stage of the synthesis, we also examined all the text which had a given code applied to check consistency of interpretation and to see whether additional levels of coding were needed. (In grounded theory this is termed 'axial' coding; see Fisher [ 55 ] for further discussion of the application of axial coding in research synthesis.) This process created a total of 36 initial codes. For example, some of the text we coded as "bad food = nice, good food = awful" from one study [ 56 ] were:

'All the things that are bad for you are nice and all the things that are good for you are awful.' (Boys, year 6) [[ 56 ], p74]

'All adverts for healthy stuff go on about healthy things. The adverts for unhealthy things tell you how nice they taste.' [[ 56 ], p75]

Some children reported throwing away foods they knew had been put in because they were 'good for you' and only ate the crisps and chocolate . [[ 56 ], p75]

Reviewers looked for similarities and differences between the codes in order to start grouping them into a hierarchical tree structure. New codes were created to capture the meaning of groups of initial codes. This process resulted in a tree structure with several layers to organize a total of 12 descriptive themes (Figure 2 ). For example, the first layer divided the 12 themes into whether they were concerned with children's understandings of healthy eating or influences on children's food choice. The above example, about children's preferences for food, was placed in both areas, since the findings related both to children's reactions to the foods they were given, and to how they behaved when given the choice over what foods they might eat. A draft summary of the findings across the studies organized by the 12 descriptive themes was then written by one of the review authors. Two other review authors commented on this draft and a final version was agreed.

relationships between descriptive themes.

Stage three: generating analytical themes

Up until this point, we had produced a synthesis which kept very close to the original findings of the included studies. The findings of each study had been combined into a whole via a listing of themes which described children's perspectives on healthy eating. However, we did not yet have a synthesis product that addressed directly the concerns of our review – regarding how to promote healthy eating, in particular fruit and vegetable intake, amongst children. Neither had we 'gone beyond' the findings of the primary studies and generated additional concepts, understandings or hypotheses. As noted earlier, the idea or step of 'going beyond' the content of the original studies has been identified by some as the defining characteristic of synthesis [ 32 , 14 ].

This stage of a qualitative synthesis is the most difficult to describe and is, potentially, the most controversial, since it is dependent on the judgement and insights of the reviewers. The equivalent stage in meta-ethnography is the development of 'third order interpretations' which go beyond the content of original studies [ 32 , 11 ]. In our example, the step of 'going beyond' the content of the original studies was achieved by using the descriptive themes that emerged from our inductive analysis of study findings to answer the review questions we had temporarily put to one side. Reviewers inferred barriers and facilitators from the views children were expressing about healthy eating or food in general, captured by the descriptive themes, and then considered the implications of children's views for intervention development. Each reviewer first did this independently and then as a group. Through this discussion more abstract or analytical themes began to emerge. The barriers and facilitators and implications for intervention development were examined again in light of these themes and changes made as necessary. This cyclical process was repeated until the new themes were sufficiently abstract to describe and/or explain all of our initial descriptive themes, our inferred barriers and facilitators and implications for intervention development.

For example, five of the 12 descriptive themes concerned the influences on children's choice of foods (food preferences, perceptions of health benefits, knowledge behaviour gap, roles and responsibilities, non-influencing factors). From these, reviewers inferred several barriers and implications for intervention development. Children identified readily that taste was the major concern for them when selecting food and that health was either a secondary factor or, in some cases, a reason for rejecting food. Children also felt that buying healthy food was not a legitimate use of their pocket money, which they would use to buy sweets that could be enjoyed with friends. These perspectives indicated to us that branding fruit and vegetables as a 'tasty' rather than 'healthy' might be more effective in increasing consumption. As one child noted astutely, 'All adverts for healthy stuff go on about healthy things. The adverts for unhealthy things tell you how nice they taste.' [[ 56 ], p75]. We captured this line of argument in the analytical theme entitled 'Children do not see it as their role to be interested in health'. Altogether, this process resulted in the generation of six analytical themes which were associated with ten recommendations for interventions.

Six main issues emerged from the studies of children's views: (1) children do not see it as their role to be interested in health; (2) children do not see messages about future health as personally relevant or credible; (3) fruit, vegetables and confectionery have very different meanings for children; (4) children actively seek ways to exercise their own choices with regard to food; (5) children value eating as a social occasion; and (6) children see the contradiction between what is promoted in theory and what adults provide in practice. The review found that most interventions were based in school (though frequently with parental involvement) and often combined learning about the health benefits of fruit and vegetables with 'hands-on' experience in the form of food preparation and taste-testing. Interventions targeted at people with particular risk factors worked better than others, and multi-component interventions that combined the promotion of physical activity with healthy eating did not work as well as those that only concentrated on healthy eating. The studies of children's views suggested that fruit and vegetables should be treated in different ways in interventions, and that messages should not focus on health warnings. Interventions that were in line with these suggestions tended to be more effective than those which were not.

Context and rigour in thematic synthesis

The process of translation, through the development of descriptive and analytical themes, can be carried out in a rigorous way that facilitates transparency of reporting. Since we aim to produce a synthesis that both generates 'abstract and formal theories' that are nevertheless 'empirically faithful to the cases from which they were developed' [[ 53 ], p1371], we see the explicit recording of the development of themes as being central to the method. The use of software as described can facilitate this by allowing reviewers to examine the contribution made to their findings by individual studies, groups of studies, or sub-populations within studies.

Some may argue against the synthesis of qualitative research on the grounds that the findings of individual studies are de-contextualised and that concepts identified in one setting are not applicable to others [ 32 ]. However, the act of synthesis could be viewed as similar to the role of a research user when reading a piece of qualitative research and deciding how useful it is to their own situation. In the case of synthesis, reviewers translate themes and concepts from one situation to another and can always be checking that each transfer is valid and whether there are any reasons that understandings gained in one context might not be transferred to another. We attempted to preserve context by providing structured summaries of each study detailing aims, methods and methodological quality, and setting and sample. This meant that readers of our review were able to judge for themselves whether or not the contexts of the studies the review contained were similar to their own. In the synthesis we also checked whether the emerging findings really were transferable across different study contexts. For example, we tried throughout the synthesis to distinguish between participants (e.g. boys and girls) where the primary research had made an appropriate distinction. We then looked to see whether some of our synthesis findings could be attributed to a particular group of children or setting. In the event, we did not find any themes that belonged to a specific group, but another outcome of this process was a realisation that the contextual information given in the reports of studies was very restricted indeed. It was therefore difficult to make the best use of context in our synthesis.

In checking that we were not translating concepts into situations where they did not belong, we were following a principle that others have followed when using synthesis methods to build grounded formal theory: that of grounding a text in the context in which it was constructed. As Margaret Kearney has noted "the conditions under which data were collected, analysis was done, findings were found, and products were written for each contributing report should be taken into consideration in developing a more generalized and abstract model" [[ 14 ], p1353]. Britten et al . [ 32 ] suggest that it may be important to make a deliberate attempt to include studies conducted across diverse settings to achieve the higher level of abstraction that is aimed for in a meta-ethnography.

Study quality and sensitivity analyses

We assessed the 'quality' of our studies with regard to the degree to which they represented the views of their participants. In doing this, we were locating the concept of 'quality' within the context of the purpose of our review – children's views – and not necessarily the context of the primary studies themselves. Our 'hierarchy of evidence', therefore, did not prioritise the research design of studies but emphasised the ability of the studies to answer our review question. A traditional systematic review of controlled trials would contain a quality assessment stage, the purpose of which is to exclude studies that do not provide a reliable answer to the review question. However, given that there were no accepted – or empirically tested – methods for excluding qualitative studies from syntheses on the basis of their quality [ 57 , 12 , 58 ], we included all studies regardless of their quality.

Nevertheless, our studies did differ according to the quality criteria they were assessed against and it was important that we considered this in some way. In systematic reviews of trials, 'sensitivity analyses' – analyses which test the effect on the synthesis of including and excluding findings from studies of differing quality – are often carried out. Dixon-Woods et al . [ 12 ] suggest that assessing the feasibility and worth of conducting sensitivity analyses within syntheses of qualitative research should be an important focus of synthesis methods work. After our thematic synthesis was complete, we examined the relative contributions of studies to our final analytic themes and recommendations for interventions. We found that the poorer quality studies contributed comparatively little to the synthesis and did not contain many unique themes; the better studies, on the other hand, appeared to have more developed analyses and contributed most to the synthesis.

This paper has discussed the rationale for reviewing and synthesising qualitative research in a systematic way and has outlined one specific approach for doing this: thematic synthesis. While it is not the only method which might be used – and we have discussed some of the other options available – we present it here as a tested technique that has worked in the systematic reviews in which it has been employed.

We have observed that one of the key tasks in the synthesis of qualitative research is the translation of concepts between studies. While the activity of translating concepts is usually undertaken in the few syntheses of qualitative research that exist, there are few examples that specify the detail of how this translation is actually carried out. The example above shows how we achieved the translation of concepts across studies through the use of line-by-line coding, the organisation of these codes into descriptive themes, and the generation of analytical themes through the application of a higher level theoretical framework. This paper therefore also demonstrates how the methods and process of a thematic synthesis can be written up in a transparent way.

This paper goes some way to addressing concerns regarding the use of thematic analysis in research synthesis raised by Dixon-Woods and colleagues who argue that the approach can lack transparency due to a failure to distinguish between 'data-driven' or 'theory-driven' approaches. Moreover they suggest that, "if thematic analysis is limited to summarising themes reported in primary studies, it offers little by way of theoretical structure within which to develop higher order thematic categories..." [[ 35 ], p47]. Part of the problem, they observe, is that the precise methods of thematic synthesis are unclear. Our approach contains a clear separation between the 'data-driven' descriptive themes and the 'theory-driven' analytical themes and demonstrates how the review questions provided a theoretical structure within which it became possible to develop higher order thematic categories.

The theme of 'going beyond' the content of the primary studies was discussed earlier. Citing Strike and Posner [ 59 ], Campbell et al . [[ 11 ], p672] also suggest that synthesis "involves some degree of conceptual innovation, or employment of concepts not found in the characterisation of the parts and a means of creating the whole" . This was certainly true of the example given in this paper. We used a series of questions, derived from the main topic of our review, to focus an examination of our descriptive themes and we do not find our recommendations for interventions contained in the findings of the primary studies: these were new propositions generated by the reviewers in the light of the synthesis. The method also demonstrates that it is possible to synthesise without conceptual innovation. The initial synthesis, involving the translation of concepts between studies, was necessary in order for conceptual innovation to begin. One could argue that the conceptual innovation, in this case, was only necessary because the primary studies did not address our review question directly. In situations in which the primary studies are concerned directly with the review question, it may not be necessary to go beyond the contents of the original studies in order to produce a satisfactory synthesis (see, for example, Marston and King, [ 60 ]). Conceptually, our analytical themes are similar to the ultimate product of meta-ethnographies: third order interpretations [ 11 ], since both are explicit mechanisms for going beyond the content of the primary studies and presenting this in a transparent way. The main difference between them lies in their purposes. Third order interpretations bring together the implications of translating studies into one another in their own terms, whereas analytical themes are the result of interrogating a descriptive synthesis by placing it within an external theoretical framework (our review question and sub-questions). It may be, therefore, that analytical themes are more appropriate when a specific review question is being addressed (as often occurs when informing policy and practice), and third order interpretations should be used when a body of literature is being explored in and of itself, with broader, or emergent, review questions.

This paper is a contribution to the current developmental work taking place in understanding how best to bring together the findings of qualitative research to inform policy and practice. It is by no means the only method on offer but, by drawing on methods and principles from qualitative primary research, it benefits from the years of methodological development that underpins the research it seeks to synthesise.

Chalmers I: Trying to do more good than harm in policy and practice: the role of rigorous, transparent and up-to-date evaluations. Ann Am Acad Pol Soc Sci. 2003, 589: 22-40. 10.1177/0002716203254762.

Article   Google Scholar  

Oakley A: Social science and evidence-based everything: the case of education. Educ Rev. 2002, 54: 277-286. 10.1080/0013191022000016329.

Cooper H, Hedges L: The Handbook of Research Synthesis. 1994, New York: Russell Sage Foundation

Google Scholar  

EPPI-Centre: EPPI-Centre Methods for Conducting Systematic Reviews. 2006, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=89 ]

Higgins J, Green S, (Eds): Cochrane Handbook for Systematic Reviews of Interventions 4.2.6. 2006, Updated September 2006. Accessed 24th January 2007, [ http://www.cochrane.org/resources/handbook/ ]

Petticrew M, Roberts H: Systematic Reviews in the Social Sciences: A practical guide. 2006, Oxford: Blackwell Publishing

Book   Google Scholar  

Chalmers I, Hedges L, Cooper H: A brief history of research synthesis. Eval Health Prof. 2002, 25: 12-37. 10.1177/0163278702025001003.

Article   PubMed   Google Scholar  

Juni P, Altman D, Egger M: Assessing the quality of controlled clinical trials. BMJ. 2001, 323: 42-46. 10.1136/bmj.323.7303.42.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Mulrow C: Systematic reviews: rationale for systematic reviews. BMJ. 1994, 309: 597-599.

White H: Scientific communication and literature retrieval. The Handbook of Research Synthesis. Edited by: Cooper H, Hedges L. 1994, New York: Russell Sage Foundation

Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, Donovan J: Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Soc Sci Med. 2003, 56: 671-684. 10.1016/S0277-9536(02)00064-3.

Dixon-Woods M, Bonas S, Booth A, Jones DR, Miller T, Sutton AJ, Shaw RL, Smith JA, Young B: How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res. 2006, 6: 27-44. 10.1177/1468794106058867.

Sandelowski M, Barroso J: Handbook for Synthesising Qualitative Research. 2007, New York: Springer

Thorne S, Jensen L, Kearney MH, Noblit G, Sandelowski M: Qualitative meta-synthesis: reflections on methodological orientation and ideological agenda. Qual Health Res. 2004, 14: 1342-1365. 10.1177/1049732304269888.

Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, Oakley A: Applying systematic review methods to studies of people's views: an example from public health. J Epidemiol Community Health. 2004, 58: 794-800. 10.1136/jech.2003.014829.

Article   PubMed   PubMed Central   Google Scholar  

Harden A, Brunton G, Fletcher A, Oakley A: Young People, Pregnancy and Social Exclusion: A systematic synthesis of research evidence to identify effective, appropriate and promising approaches for prevention and support. 2006, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=674 ]

Thomas J, Sutcliffe K, Harden A, Oakley A, Oliver S, Rees R, Brunton G, Kavanagh J: Children and Healthy Eating: A systematic review of barriers and facilitators. 2003, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, accessed 4 th July 2008, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=246 ]

Thomas J, Kavanagh J, Tucker H, Burchett H, Tripney J, Oakley A: Accidental Injury, Risk-Taking Behaviour and the Social Circumstances in which Young People Live: A systematic review. 2007, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=1910 ]

Bryman A: Quantity and Quality in Social Research. 1998, London: Unwin

Hammersley M: What's Wrong with Ethnography?. 1992, London: Routledge

Harden A, Thomas J: Methodological issues in combining diverse study types in systematic reviews. Int J Soc Res Meth. 2005, 8: 257-271. 10.1080/13645570500155078.

Oakley A: Experiments in Knowing: Gender and methods in the social sciences. 2000, Cambridge: Polity Press

Harden A, Oakley A, Oliver S: Peer-delivered health promotion for young people: a systematic review of different study designs. Health Educ J. 2001, 60: 339-353. 10.1177/001789690106000406.

Harden A, Rees R, Shepherd J, Brunton G, Oliver S, Oakley A: Young People and Mental Health: A systematic review of barriers and facilitators. 2001, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=256 ]

Rees R, Harden A, Shepherd J, Brunton G, Oliver S, Oakley A: Young People and Physical Activity: A systematic review of barriers and facilitators. 2001, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=260 ]

Shepherd J, Harden A, Rees R, Brunton G, Oliver S, Oakley A: Young People and Healthy Eating: A systematic review of barriers and facilitators. 2001, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, [ http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=258 ]

Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J: Integrating qualitative research with trials in systematic reviews: an example from public health. BMJ. 2004, 328: 1010-1012. 10.1136/bmj.328.7446.1010.

Davies P: What is evidence-based education?. Br J Educ Stud. 1999, 47: 108-121. 10.1111/1467-8527.00106.

Newman M, Thompson C, Roberts AP: Helping practitioners understand the contribution of qualitative research to evidence-based practice. Evid Based Nurs. 2006, 9: 4-7. 10.1136/ebn.9.1.4.

Popay J: Moving Beyond Effectiveness in Evidence Synthesis. 2006, London: National Institute for Health and Clinical Excellence

Noblit GW, Hare RD: Meta-Ethnography: Synthesizing qualitative studies. 1988, Newbury Park: Sage

Britten N, Campbell R, Pope C, Donovan J, Morgan M, Pill R: Using meta-ethnography to synthesise qualitative research: a worked example. J Health Serv Res Policy. 2002, 7: 209-215. 10.1258/135581902320432732.

Paterson B, Thorne S, Canam C, Jillings C: Meta-Study of Qualitative Health Research. 2001, Thousand Oaks, California: Sage

Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, Katbamna S, Olsen R, Smith L, Riley R, Sutton AJ: Conducting a critical interpretative synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006, 6: 35-10.1186/1471-2288-6-35.

Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A: Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005, 10: 45-53. 10.1258/1355819052801804.

Boyatzis RE: Transforming Qualitative Information. 1998, Sage: Cleveland

Braun V, Clarke V: Using thematic analysis in psychology. Qual Res Psychol. 2006, 3: 77-101. 10.1191/1478088706qp063oa. [ http://science.uwe.ac.uk/psychology/drvictoriaclarke_files/thematicanalysis%20.pdf ]

Silverman D, Ed: Qualitative Research: Theory, method and practice. 1997, London: Sage

Doyle LH: Synthesis through meta-ethnography: paradoxes, enhancements, and possibilities. Qual Res. 2003, 3: 321-344. 10.1177/1468794103033003.

Barroso J, Gollop C, Sandelowski M, Meynell J, Pearce PF, Collins LJ: The challenges of searching for and retrieving qualitative studies. Western J Nurs Res. 2003, 25: 153-178. 10.1177/0193945902250034.

Walters LA, Wilczynski NL, Haynes RB, Hedges Team: Developing optimal search strategies for retrieving clinically relevant qualitative studies in EMBASE. Qual Health Res. 2006, 16: 162-8. 10.1177/1049732305284027.

Wong SSL, Wilczynski NL, Haynes RB: Developing optimal search strategies for detecting clinically relevant qualitative studies in Medline. Medinfo. 2004, 11: 311-314.

Murphy E, Dingwall R, Greatbatch D, Parker S, Watson P: Qualitative research methods in health technology assessment: a review of the literature. Health Technol Assess. 1998, 2 (16):

Seale C: Quality in qualitative research. Qual Inq. 1999, 5: 465-478.

Spencer L, Ritchie J, Lewis J, Dillon L: Quality in Qualitative Evaluation: A framework for assessing research evidence. 2003, London: Cabinet Office

Boulton M, Fitzpatrick R, Swinburn C: Qualitative research in healthcare II: a structured review and evaluation of studies. J Eval Clin Pract. 1996, 2: 171-179. 10.1111/j.1365-2753.1996.tb00041.x.

Article   CAS   PubMed   Google Scholar  

Cobb A, Hagemaster J: Ten criteria for evaluating qualitative research proposals. J Nurs Educ. 1987, 26: 138-143.

CAS   PubMed   Google Scholar  

Mays N, Pope C: Rigour and qualitative research. BMJ. 1995, 311: 109-12.

Medical Sociology Group: Criteria for the evaluation of qualitative research papers. Med Sociol News. 1996, 22: 68-71.

Alderson P: Listening to Children. 1995, London: Barnardo's

Egger M, Davey-Smith G, Altman D: Systematic Reviews in Health Care: Meta-analysis in context. 2001, London: BMJ Publishing

Sandelowski M, Barroso J: Finding the findings in qualitative studies. J Nurs Scholarsh. 2002, 34: 213-219. 10.1111/j.1547-5069.2002.00213.x.

Sandelowski M: Using qualitative research. Qual Health Res. 2004, 14: 1366-1386. 10.1177/1049732304269672.

Thomas J, Brunton J: EPPI-Reviewer 3.0: Analysis and management of data for research synthesis. EPPI-Centre software. 2006, London: EPPI-Centre, Social Science Research Unit, Institute of Education

Fisher M, Qureshi H, Hardyman W, Homewood J: Using Qualitative Research in Systematic Reviews: Older people's views of hospital discharge. 2006, London: Social Care Institute for Excellence

Dixey R, Sahota P, Atwal S, Turner A: Children talking about healthy eating: data from focus groups with 300 9–11-year-olds. Nutr Bull. 2001, 26: 71-79. 10.1046/j.1467-3010.2001.00078.x.

Daly A, Willis K, Small R, Green J, Welch N, Kealy M, Hughes E: Hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol. 2007, 60: 43-49. 10.1016/j.jclinepi.2006.03.014.

Popay J: Moving beyond floccinaucinihilipilification: enhancing the utility of systematic reviews. J Clin Epidemiol. 2005, 58: 1079-80. 10.1016/j.jclinepi.2005.08.004.

Strike K, Posner G: Types of synthesis and their criteria. Knowledge Structure and Use: Implications for synthesis and interpretation. Edited by: Ward S, Reed L. 1983, Philadelphia: Temple University Press

Marston C, King E: Factors that shape young people's sexual behaviour: a systematic review. The Lancet. 2006, 368: 1581-86. 10.1016/S0140-6736(06)69662-1.

Pre-publication history

The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-2288/8/45/prepub

Download references

Acknowledgements

The authors would like to thank Elaine Barnett-Page for her assistance in producing the draft paper, and David Gough, Ann Oakley and Sandy Oliver for their helpful comments. The review used an example in this paper was funded by the Department of Health (England). The methodological development was supported by Department of Health (England) and the ESRC through the Methods for Research Synthesis Node of the National Centre for Research Methods. In addition, Angela Harden held a senior research fellowship funded by the Department of Health (England) December 2003 – November 2007. The views expressed in this paper are those of the authors and are not necessarily those of the funding bodies.

Author information

Authors and affiliations.

EPPI-Centre, Social Science Research Unit, Institute of Education, University of London, UK

James Thomas & Angela Harden

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to James Thomas .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors' contributions

Both authors contributed equally to the paper and read and approved the final manuscript.

James Thomas and Angela Harden contributed equally to this work.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2, rights and permissions.

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Thomas, J., Harden, A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol 8 , 45 (2008). https://doi.org/10.1186/1471-2288-8-45

Download citation

Received : 17 April 2008

Accepted : 10 July 2008

Published : 10 July 2008

DOI : https://doi.org/10.1186/1471-2288-8-45

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Qualitative Research
• Primary Study
• Analytical Theme
• Healthy Eating
• Review Question

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Practical thematic...

Practical thematic analysis: a guide for multidisciplinary health services research teams engaging in qualitative analysis

• Related content
• Peer review
• Catherine H Saunders , scientist and assistant professor 1 2 ,
• Ailyn Sierpe , research project coordinator 2 ,
• Christian von Plessen , senior physician 3 ,
• Alice M Kennedy , research project manager 2 4 ,
• Laura C Leviton , senior adviser 5 ,
• Steven L Bernstein , chief research officer 1 ,
• Jenaya Goldwag , resident physician 1 ,
• Joel R King , research assistant 2 ,
• Christine M Marx , patient associate 6 ,
• Jacqueline A Pogue , research project manager 2 ,
• Richard K Saunders , staff physician 1 ,
• Aricca Van Citters , senior research scientist 2 ,
• Renata W Yen , doctoral student 2 ,
• Glyn Elwyn , professor 2 ,
• JoAnna K Leyenaar , associate professor 1 2
• on behalf of the Coproduction Laboratory
• 1 Dartmouth Health, Lebanon, NH, USA
• 2 Dartmouth Institute for Health Policy and Clinical Practice, Geisel School of Medicine at Dartmouth College, Lebanon, NH, USA
• 3 Center for Primary Care and Public Health (Unisanté), Lausanne, Switzerland
• 4 Jönköping Academy for Improvement of Health and Welfare, School of Health and Welfare, Jönköping University, Jönköping, Sweden
• 5 Highland Park, NJ, USA
• 6 Division of Public Health Sciences, Department of Surgery, Washington University School of Medicine, St Louis, MO, USA
• Correspondence to: C H Saunders catherine.hylas.saunders{at}dartmouth.edu
• Accepted 26 April 2023

Qualitative research methods explore and provide deep contextual understanding of real world issues, including people’s beliefs, perspectives, and experiences. Whether through analysis of interviews, focus groups, structured observation, or multimedia data, qualitative methods offer unique insights in applied health services research that other approaches cannot deliver. However, many clinicians and researchers hesitate to use these methods, or might not use them effectively, which can leave relevant areas of inquiry inadequately explored. Thematic analysis is one of the most common and flexible methods to examine qualitative data collected in health services research. This article offers practical thematic analysis as a step-by-step approach to qualitative analysis for health services researchers, with a focus on accessibility for patients, care partners, clinicians, and others new to thematic analysis. Along with detailed instructions covering three steps of reading, coding, and theming, the article includes additional novel and practical guidance on how to draft effective codes, conduct a thematic analysis session, and develop meaningful themes. This approach aims to improve consistency and rigor in thematic analysis, while also making this method more accessible for multidisciplinary research teams.

Through qualitative methods, researchers can provide deep contextual understanding of real world issues, and generate new knowledge to inform hypotheses, theories, research, and clinical care. Approaches to data collection are varied, including interviews, focus groups, structured observation, and analysis of multimedia data, with qualitative research questions aimed at understanding the how and why of human experience. 1 2 Qualitative methods produce unique insights in applied health services research that other approaches cannot deliver. In particular, researchers acknowledge that thematic analysis is a flexible and powerful method of systematically generating robust qualitative research findings by identifying, analysing, and reporting patterns (themes) within data. 3 4 5 6 Although qualitative methods are increasingly valued for answering clinical research questions, many researchers are unsure how to apply them or consider them too time consuming to be useful in responding to practical challenges 7 or pressing situations such as public health emergencies. 8 Consequently, researchers might hesitate to use them, or use them improperly. 9 10 11

Although much has been written about how to perform thematic analysis, practical guidance for non-specialists is sparse. 3 5 6 12 13 In the multidisciplinary field of health services research, qualitative data analysis can confound experienced researchers and novices alike, which can stoke concerns about rigor, particularly for those more familiar with quantitative approaches. 14 Since qualitative methods are an area of specialisation, support from experts is beneficial. However, because non-specialist perspectives can enhance data interpretation and enrich findings, there is a case for making thematic analysis easier, more rapid, and more efficient, 8 particularly for patients, care partners, clinicians, and other stakeholders. A practical guide to thematic analysis might encourage those on the ground to use these methods in their work, unearthing insights that would otherwise remain undiscovered.

Given the need for more accessible qualitative analysis approaches, we present a simple, rigorous, and efficient three step guide for practical thematic analysis. We include new guidance on the mechanics of thematic analysis, including developing codes, constructing meaningful themes, and hosting a thematic analysis session. We also discuss common pitfalls in thematic analysis and how to avoid them.

Summary points

Qualitative methods are increasingly valued in applied health services research, but multidisciplinary research teams often lack accessible step-by-step guidance and might struggle to use these approaches

A newly developed approach, practical thematic analysis, uses three simple steps: reading, coding, and theming

Based on Braun and Clarke’s reflexive thematic analysis, our streamlined yet rigorous approach is designed for multidisciplinary health services research teams, including patients, care partners, and clinicians

This article also provides companion materials including a slide presentation for teaching practical thematic analysis to research teams, a sample thematic analysis session agenda, a theme coproduction template for use during the session, and guidance on using standardised reporting criteria for qualitative research

In their seminal work, Braun and Clarke developed a six phase approach to reflexive thematic analysis. 4 12 We built on their method to develop practical thematic analysis ( box 1 , fig 1 ), which is a simplified and instructive approach that retains the substantive elements of their six phases. Braun and Clarke’s phase 1 (familiarising yourself with the dataset) is represented in our first step of reading. Phase 2 (coding) remains as our second step of coding. Phases 3 (generating initial themes), 4 (developing and reviewing themes), and 5 (refining, defining, and naming themes) are represented in our third step of theming. Phase 6 (writing up) also occurs during this third step of theming, but after a thematic analysis session. 4 12

Key features and applications of practical thematic analysis

Step 1: reading.

All manuscript authors read the data

All manuscript authors write summary memos

Step 2: Coding

Coders perform both data management and early data analysis

Codes are complete thoughts or sentences, not categories

Step 3: Theming

Researchers host a thematic analysis session and share different perspectives

Themes are complete thoughts or sentences, not categories

Applications

For use by practicing clinicians, patients and care partners, students, interdisciplinary teams, and those new to qualitative research

When important insights from healthcare professionals are inaccessible because they do not have qualitative methods training

When time and resources are limited

Steps in practical thematic analysis

• Download figure
• Open in new tab
• Download powerpoint

We present linear steps, but as qualitative research is usually iterative, so too is thematic analysis. 15 Qualitative researchers circle back to earlier work to check whether their interpretations still make sense in the light of additional insights, adapting as necessary. While we focus here on the practical application of thematic analysis in health services research, we recognise our approach exists in the context of the broader literature on thematic analysis and the theoretical underpinnings of qualitative methods as a whole. For a more detailed discussion of these theoretical points, as well as other methods widely used in health services research, we recommend reviewing the sources outlined in supplemental material 1. A strong and nuanced understanding of the context and underlying principles of thematic analysis will allow for higher quality research. 16

Practical thematic analysis is a highly flexible approach that can draw out valuable findings and generate new hypotheses, including in cases with a lack of previous research to build on. The approach can also be used with a variety of data, such as transcripts from interviews or focus groups, patient encounter transcripts, professional publications, observational field notes, and online activity logs. Importantly, successful practical thematic analysis is predicated on having high quality data collected with rigorous methods. We do not describe qualitative research design or data collection here. 11 17

In supplemental material 1, we summarise the foundational methods, concepts, and terminology in qualitative research. Along with our guide below, we include a companion slide presentation for teaching practical thematic analysis to research teams in supplemental material 2. We provide a theme coproduction template for teams to use during thematic analysis sessions in supplemental material 3. Our method aligns with the major qualitative reporting frameworks, including the Consolidated Criteria for Reporting Qualitative Research (COREQ). 18 We indicate the corresponding step in practical thematic analysis for each COREQ item in supplemental material 4.

Familiarisation and memoing

We encourage all manuscript authors to review the full dataset (eg, interview transcripts) to familiarise themselves with it. This task is most critical for those who will later be engaged in the coding and theming steps. Although time consuming, it is the best way to involve team members in the intellectual work of data interpretation, so that they can contribute to the analysis and contextualise the results. If this task is not feasible given time limitations or large quantities of data, the data can be divided across team members. In this case, each piece of data should be read by at least two individuals who ideally represent different professional roles or perspectives.

We recommend that researchers reflect on the data and independently write memos, defined as brief notes on thoughts and questions that arise during reading, and a summary of their impressions of the dataset. 2 19 Memoing is an opportunity to gain insights from varying perspectives, particularly from patients, care partners, clinicians, and others. It also gives researchers the opportunity to begin to scope which elements of and concepts in the dataset are relevant to the research question.

Data saturation

The concept of data saturation ( box 2 ) is a foundation of qualitative research. It is defined as the point in analysis at which new data tend to be redundant of data already collected. 21 Qualitative researchers are expected to report their approach to data saturation. 18 Because thematic analysis is iterative, the team should discuss saturation throughout the entire process, beginning with data collection and continuing through all steps of the analysis. 22 During step 1 (reading), team members might discuss data saturation in the context of summary memos. Conversations about saturation continue during step 2 (coding), with confirmation that saturation has been achieved during step 3 (theming). As a rule of thumb, researchers can often achieve saturation in 9-17 interviews or 4-8 focus groups, but this will vary depending on the specific characteristics of the study. 23

Data saturation in context

Braun and Clarke discourage the use of data saturation to determine sample size (eg, number of interviews), because it assumes that there is an objective truth to be captured in the data (sometimes known as a positivist perspective). 20 Qualitative researchers often try to avoid positivist approaches, arguing that there is no one true way of seeing the world, and will instead aim to gather multiple perspectives. 5 Although this theoretical debate with qualitative methods is important, we recognise that a priori estimates of saturation are often needed, particularly for investigators newer to qualitative research who might want a more pragmatic and applied approach. In addition, saturation based, sample size estimation can be particularly helpful in grant proposals. However, researchers should still follow a priori sample size estimation with a discussion to confirm saturation has been achieved.

Definition of coding

We describe codes as labels for concepts in the data that are directly relevant to the study objective. Historically, the purpose of coding was to distil the large amount of data collected into conceptually similar buckets so that researchers could review it in aggregate and identify key themes. 5 24 We advocate for a more analytical approach than is typical with thematic analysis. With our method, coding is both the foundation for and the beginning of thematic analysis—that is, early data analysis, management, and reduction occur simultaneously rather than as different steps. This approach moves the team more efficiently towards being able to describe themes.

Building the coding team

Coders are the research team members who directly assign codes to the data, reading all material and systematically labelling relevant data with appropriate codes. Ideally, at least two researchers would code every discrete data document, such as one interview transcript. 25 If this task is not possible, individual coders can each code a subset of the data that is carefully selected for key characteristics (sometimes known as purposive selection). 26 When using this approach, we recommend that at least 10% of data be coded by two or more coders to ensure consistency in codebook application. We also recommend coding teams of no more than four to five people, for practical reasons concerning maintaining consistency.

Clinicians, patients, and care partners bring unique perspectives to coding and enrich the analytical process. 27 Therefore, we recommend choosing coders with a mix of relevant experiences so that they can challenge and contextualise each other’s interpretations based on their own perspectives and opinions ( box 3 ). We recommend including both coders who collected the data and those who are naive to it, if possible, given their different perspectives. We also recommend all coders review the summary memos from the reading step so that key concepts identified by those not involved in coding can be integrated into the analytical process. In practice, this review means coding the memos themselves and discussing them during the code development process. This approach ensures that the team considers a diversity of perspectives.

Coding teams in context

The recommendation to use multiple coders is a departure from Braun and Clarke. 28 29 When the views, experiences, and training of each coder (sometimes known as positionality) 30 are carefully considered, having multiple coders can enhance interpretation and enrich findings. When these perspectives are combined in a team setting, researchers can create shared meaning from the data. Along with the practical consideration of distributing the workload, 31 inclusion of these multiple perspectives increases the overall quality of the analysis by mitigating the impact of any one coder’s perspective. 30

Coding tools

Qualitative analysis software facilitates coding and managing large datasets but does not perform the analytical work. The researchers must perform the analysis themselves. Most programs support queries and collaborative coding by multiple users. 32 Important factors to consider when choosing software can include accessibility, cost, interoperability, the look and feel of code reports, and the ease of colour coding and merging codes. Coders can also use low tech solutions, including highlighters, word processors, or spreadsheets.

Drafting effective codes

To draft effective codes, we recommend that the coders review each document line by line. 33 As they progress, they can assign codes to segments of data representing passages of interest. 34 Coders can also assign multiple codes to the same passage. Consensus among coders on what constitutes a minimum or maximum amount of text for assigning a code is helpful. As a general rule, meaningful segments of text for coding are shorter than one paragraph, but longer than a few words. Coders should keep the study objective in mind when determining which data are relevant ( box 4 ).

Code types in context

Similar to Braun and Clarke’s approach, practical thematic analysis does not specify whether codes are based on what is evident from the data (sometimes known as semantic) or whether they are based on what can be inferred at a deeper level from the data (sometimes known as latent). 4 12 35 It also does not specify whether they are derived from the data (sometimes known as inductive) or determined ahead of time (sometimes known as deductive). 11 35 Instead, it should be noted that health services researchers conducting qualitative studies often adopt all these approaches to coding (sometimes known as hybrid analysis). 3

In practical thematic analysis, codes should be more descriptive than general categorical labels that simply group data with shared characteristics. At a minimum, codes should form a complete (or full) thought. An easy way to conceptualise full thought codes is as complete sentences with subjects and verbs ( table 1 ), although full sentence coding is not always necessary. With full thought codes, researchers think about the data more deeply and capture this insight in the codes. This coding facilitates the entire analytical process and is especially valuable when moving from codes to broader themes. Experienced qualitative researchers often intuitively use full thought or sentence codes, but this practice has not been explicitly articulated as a path to higher quality coding elsewhere in the literature. 6

Example transcript with codes used in practical thematic analysis 36

• View inline

Depending on the nature of the data, codes might either fall into flat categories or be arranged hierarchically. Flat categories are most common when the data deal with topics on the same conceptual level. In other words, one topic is not a subset of another topic. By contrast, hierarchical codes are more appropriate for concepts that naturally fall above or below each other. Hierarchical coding can also be a useful form of data management and might be necessary when working with a large or complex dataset. 5 Codes grouped into these categories can also make it easier to naturally transition into generating themes from the initial codes. 5 These decisions between flat versus hierarchical coding are part of the work of the coding team. In both cases, coders should ensure that their code structures are guided by their research questions.

Developing the codebook

A codebook is a shared document that lists code labels and comprehensive descriptions for each code, as well as examples observed within the data. Good code descriptions are precise and specific so that coders can consistently assign the same codes to relevant data or articulate why another coder would do so. Codebook development is iterative and involves input from the entire coding team. However, as those closest to the data, coders must resist undue influence, real or perceived, from other team members with conflicting opinions—it is important to mitigate the risk that more senior researchers, like principal investigators, exert undue influence on the coders’ perspectives.

In practical thematic analysis, coders begin codebook development by independently coding a small portion of the data, such as two to three transcripts or other units of analysis. Coders then individually produce their initial codebooks. This task will require them to reflect on, organise, and clarify codes. The coders then meet to reconcile the draft codebooks, which can often be difficult, as some coders tend to lump several concepts together while others will split them into more specific codes. Discussing disagreements and negotiating consensus are necessary parts of early data analysis. Once the codebook is relatively stable, we recommend soliciting input on the codes from all manuscript authors. Yet, coders must ultimately be empowered to finalise the details so that they are comfortable working with the codebook across a large quantity of data.

Assigning codes to the data

After developing the codebook, coders will use it to assign codes to the remaining data. While the codebook’s overall structure should remain constant, coders might continue to add codes corresponding to any new concepts observed in the data. If new codes are added, coders should review the data they have already coded and determine whether the new codes apply. Qualitative data analysis software can be useful for editing or merging codes.

We recommend that coders periodically compare their code occurrences ( box 5 ), with more frequent check-ins if substantial disagreements occur. In the event of large discrepancies in the codes assigned, coders should revise the codebook to ensure that code descriptions are sufficiently clear and comprehensive to support coding alignment going forward. Because coding is an iterative process, the team can adjust the codebook as needed. 5 28 29

Quantitative coding in context

Researchers should generally avoid reporting code counts in thematic analysis. However, counts can be a useful proxy in maintaining alignment between coders on key concepts. 26 In practice, therefore, researchers should make sure that all coders working on the same piece of data assign the same codes with a similar pattern and that their memoing and overall assessment of the data are aligned. 37 However, the frequency of a code alone is not an indicator of its importance. It is more important that coders agree on the most salient points in the data; reviewing and discussing summary memos can be helpful here. 5

Researchers might disagree on whether or not to calculate and report inter-rater reliability. We note that quantitative tests for agreement, such as kappa statistics or intraclass correlation coefficients, can be distracting and might not provide meaningful results in qualitative analyses. Similarly, Braun and Clarke argue that expecting perfect alignment on coding is inconsistent with the goal of co-constructing meaning. 28 29 Overall consensus on codes’ salience and contributions to themes is the most important factor.

Definition of themes

Themes are meta-constructs that rise above codes and unite the dataset ( box 6 , fig 2 ). They should be clearly evident, repeated throughout the dataset, and relevant to the research questions. 38 While codes are often explicit descriptions of the content in the dataset, themes are usually more conceptual and knit the codes together. 39 Some researchers hypothesise that theme development is loosely described in the literature because qualitative researchers simply intuit themes during the analytical process. 39 In practical thematic analysis, we offer a concrete process that should make developing meaningful themes straightforward.

Themes in context

According to Braun and Clarke, a theme “captures something important about the data in relation to the research question and represents some level of patterned response or meaning within the data set.” 4 Similarly, Braun and Clarke advise against themes as domain summaries. While different approaches can draw out themes from codes, the process begins by identifying patterns. 28 35 Like Braun and Clarke and others, we recommend that researchers consider the salience of certain themes, their prevalence in the dataset, and their keyness (ie, how relevant the themes are to the overarching research questions). 4 12 34

Use of themes in practical thematic analysis

Constructing meaningful themes

After coding all the data, each coder should independently reflect on the team’s summary memos (step 1), the codebook (step 2), and the coded data itself to develop draft themes (step 3). It can be illuminating for coders to review all excerpts associated with each code, so that they derive themes directly from the data. Researchers should remain focused on the research question during this step, so that themes have a clear relation with the overall project aim. Use of qualitative analysis software will make it easy to view each segment of data tagged with each code. Themes might neatly correspond to groups of codes. Or—more likely—they will unite codes and data in unexpected ways. A whiteboard or presentation slides might be helpful to organise, craft, and revise themes. We also provide a template for coproducing themes (supplemental material 3). As with codebook justification, team members will ideally produce individual drafts of the themes that they have identified in the data. They can then discuss these with the group and reach alignment or consensus on the final themes.

The team should ensure that all themes are salient, meaning that they are: supported by the data, relevant to the study objectives, and important. Similar to codes, themes are framed as complete thoughts or sentences, not categories. While codes and themes might appear to be similar to each other, the key distinction is that the themes represent a broader concept. Table 2 shows examples of codes and their corresponding themes from a previously published project that used practical thematic analysis. 36 Identifying three to four key themes that comprise a broader overarching theme is a useful approach. Themes can also have subthemes, if appropriate. 40 41 42 43 44

Example codes with themes in practical thematic analysis 36

Thematic analysis session

After each coder has independently produced draft themes, a carefully selected subset of the manuscript team meets for a thematic analysis session ( table 3 ). The purpose of this session is to discuss and reach alignment or consensus on the final themes. We recommend a session of three to five hours, either in-person or virtually.

Example agenda of thematic analysis session

The composition of the thematic analysis session team is important, as each person’s perspectives will shape the results. This group is usually a small subset of the broader research team, with three to seven individuals. We recommend that primary and senior authors work together to include people with diverse experiences related to the research topic. They should aim for a range of personalities and professional identities, particularly those of clinicians, trainees, patients, and care partners. At a minimum, all coders and primary and senior authors should participate in the thematic analysis session.

The session begins with each coder presenting their draft themes with supporting quotes from the data. 5 Through respectful and collaborative deliberation, the group will develop a shared set of final themes.

One team member facilitates the session. A firm, confident, and consistent facilitation style with good listening skills is critical. For practical reasons, this person is not usually one of the primary coders. Hierarchies in teams cannot be entirely flattened, but acknowledging them and appointing an external facilitator can reduce their impact. The facilitator can ensure that all voices are heard. For example, they might ask for perspectives from patient partners or more junior researchers, and follow up on comments from senior researchers to say, “We have heard your perspective and it is important; we want to make sure all perspectives in the room are equally considered.” Or, “I hear [senior person] is offering [x] idea, I’d like to hear other perspectives in the room.” The role of the facilitator is critical in the thematic analysis session. The facilitator might also privately discuss with more senior researchers, such as principal investigators and senior authors, the importance of being aware of their influence over others and respecting and eliciting the perspectives of more junior researchers, such as patients, care partners, and students.

To our knowledge, this discrete thematic analysis session is a novel contribution of practical thematic analysis. It helps efficiently incorporate diverse perspectives using the session agenda and theme coproduction template (supplemental material 3) and makes the process of constructing themes transparent to the entire research team.

Writing the report

We recommend beginning the results narrative with a summary of all relevant themes emerging from the analysis, followed by a subheading for each theme. Each subsection begins with a brief description of the theme and is illustrated with relevant quotes, which are contextualised and explained. The write-up should not simply be a list, but should contain meaningful analysis and insight from the researchers, including descriptions of how different stakeholders might have experienced a particular situation differently or unexpectedly.

In addition to weaving quotes into the results narrative, quotes can be presented in a table. This strategy is a particularly helpful when submitting to clinical journals with tight word count limitations. Quote tables might also be effective in illustrating areas of agreement and disagreement across stakeholder groups, with columns representing different groups and rows representing each theme or subtheme. Quotes should include an anonymous label for each participant and any relevant characteristics, such as role or gender. The aim is to produce rich descriptions. 5 We recommend against repeating quotations across multiple themes in the report, so as to avoid confusion. The template for coproducing themes (supplemental material 3) allows documentation of quotes supporting each theme, which might also be useful during report writing.

Visual illustrations such as a thematic map or figure of the findings can help communicate themes efficiently. 4 36 42 44 If a figure is not possible, a simple list can suffice. 36 Both must clearly present the main themes with subthemes. Thematic figures can facilitate confirmation that the researchers’ interpretations reflect the study populations’ perspectives (sometimes known as member checking), because authors can invite discussions about the figure and descriptions of findings and supporting quotes. 46 This process can enhance the validity of the results. 46

In supplemental material 4, we provide additional guidance on reporting thematic analysis consistent with COREQ. 18 Commonly used in health services research, COREQ outlines a standardised list of items to be included in qualitative research reports ( box 7 ).

Reporting in context

We note that use of COREQ or any other reporting guidelines does not in itself produce high quality work and should not be used as a substitute for general methodological rigor. Rather, researchers must consider rigor throughout the entire research process. As the issue of how to conceptualise and achieve rigorous qualitative research continues to be debated, 47 48 we encourage researchers to explicitly discuss how they have looked at methodological rigor in their reports. Specifically, we point researchers to Braun and Clarke’s 2021 tool for evaluating thematic analysis manuscripts for publication (“Twenty questions to guide assessment of TA [thematic analysis] research quality”). 16

Avoiding common pitfalls

Awareness of common mistakes can help researchers avoid improper use of qualitative methods. Improper use can, for example, prevent researchers from developing meaningful themes and can risk drawing inappropriate conclusions from the data. Braun and Clarke also warn of poor quality in qualitative research, noting that “coherence and integrity of published research does not always hold.” 16

Weak themes

An important distinction between high and low quality themes is that high quality themes are descriptive and complete thoughts. As such, they often contain subjects and verbs, and can be expressed as full sentences ( table 2 ). Themes that are simply descriptive categories or topics could fail to impart meaningful knowledge beyond categorisation. 16 49 50

Researchers will often move from coding directly to writing up themes, without performing the work of theming or hosting a thematic analysis session. Skipping concerted theming often results in themes that look more like categories than unifying threads across the data.

Unfocused analysis

Because data collection for qualitative research is often semi-structured (eg, interviews, focus groups), not all data will be directly relevant to the research question at hand. To avoid unfocused analysis and a correspondingly unfocused manuscript, we recommend that all team members keep the research objective in front of them at every stage, from reading to coding to theming. During the thematic analysis session, we recommend that the research question be written on a whiteboard so that all team members can refer back to it, and so that the facilitator can ensure that conversations about themes occur in the context of this question. Consistently focusing on the research question can help to ensure that the final report directly answers it, as opposed to the many other interesting insights that might emerge during the qualitative research process. Such insights can be picked up in a secondary analysis if desired.

Inappropriate quantification

Presenting findings quantitatively (eg, “We found 18 instances of participants mentioning safety concerns about the vaccines”) is generally undesirable in practical thematic analysis reporting. 51 Descriptive terms are more appropriate (eg, “participants had substantial concerns about the vaccines,” or “several participants were concerned about this”). This descriptive presentation is critical because qualitative data might not be consistently elicited across participants, meaning that some individuals might share certain information while others do not, simply based on how conversations evolve. Additionally, qualitative research does not aim to draw inferences outside its specific sample. Emphasising numbers in thematic analysis can lead to readers incorrectly generalising the findings. Although peer reviewers unfamiliar with thematic analysis often request this type of quantification, practitioners of practical thematic analysis can confidently defend their decision to avoid it. If quantification is methodologically important, we recommend simultaneously conducting a survey or incorporating standardised interview techniques into the interview guide. 11

Neglecting group dynamics

Researchers should concertedly consider group dynamics in the research team. Particular attention should be paid to power relations and the personality of team members, which can include aspects such as who most often speaks, who defines concepts, and who resolves disagreements that might arise within the group. 52

The perspectives of patient and care partners are particularly important to cultivate. Ideally, patient partners are meaningfully embedded in studies from start to finish, not just for practical thematic analysis. 53 Meaningful engagement can build trust, which makes it easier for patient partners to ask questions, request clarification, and share their perspectives. Professional team members should actively encourage patient partners by emphasising that their expertise is critically important and valued. Noting when a patient partner might be best positioned to offer their perspective can be particularly powerful.

Insufficient time allocation

Researchers must allocate enough time to complete thematic analysis. Working with qualitative data takes time, especially because it is often not a linear process. As the strength of thematic analysis lies in its ability to make use of the rich details and complexities of the data, we recommend careful planning for the time required to read and code each document.

Estimating the necessary time can be challenging. For step 1 (reading), researchers can roughly calculate the time required based on the time needed to read and reflect on one piece of data. For step 2 (coding), the total amount of time needed can be extrapolated from the time needed to code one document during codebook development. We also recommend three to five hours for the thematic analysis session itself, although coders will need to independently develop their draft themes beforehand. Although the time required for practical thematic analysis is variable, teams should be able to estimate their own required effort with these guidelines.

Practical thematic analysis builds on the foundational work of Braun and Clarke. 4 16 We have reframed their six phase process into three condensed steps of reading, coding, and theming. While we have maintained important elements of Braun and Clarke’s reflexive thematic analysis, we believe that practical thematic analysis is conceptually simpler and easier to teach to less experienced researchers and non-researcher stakeholders. For teams with different levels of familiarity with qualitative methods, this approach presents a clear roadmap to the reading, coding, and theming of qualitative data. Our practical thematic analysis approach promotes efficient learning by doing—experiential learning. 12 29 Practical thematic analysis avoids the risk of relying on complex descriptions of methods and theory and places more emphasis on obtaining meaningful insights from those close to real world clinical environments. Although practical thematic analysis can be used to perform intensive theory based analyses, it lends itself more readily to accelerated, pragmatic approaches.

Strengths and limitations

Our approach is designed to smooth the qualitative analysis process and yield high quality themes. Yet, researchers should note that poorly performed analyses will still produce low quality results. Practical thematic analysis is a qualitative analytical approach; it does not look at study design, data collection, or other important elements of qualitative research. It also might not be the right choice for every qualitative research project. We recommend it for applied health services research questions, where diverse perspectives and simplicity might be valuable.

We also urge researchers to improve internal validity through triangulation methods, such as member checking (supplemental material 1). 46 Member checking could include soliciting input on high level themes, theme definitions, and quotations from participants. This approach might increase rigor.

Implications

We hope that by providing clear and simple instructions for practical thematic analysis, a broader range of researchers will be more inclined to use these methods. Increased transparency and familiarity with qualitative approaches can enhance researchers’ ability to both interpret qualitative studies and offer up new findings themselves. In addition, it can have usefulness in training and reporting. A major strength of this approach is to facilitate meaningful inclusion of patient and care partner perspectives, because their lived experiences can be particularly valuable in data interpretation and the resulting findings. 11 30 As clinicians are especially pressed for time, they might also appreciate a practical set of instructions that can be immediately used to leverage their insights and access to patients and clinical settings, and increase the impact of qualitative research through timely results. 8

Practical thematic analysis is a simplified approach to performing thematic analysis in health services research, a field where the experiences of patients, care partners, and clinicians are of inherent interest. We hope that it will be accessible to those individuals new to qualitative methods, including patients, care partners, clinicians, and other health services researchers. We intend to empower multidisciplinary research teams to explore unanswered questions and make new, important, and rigorous contributions to our understanding of important clinical and health systems research.

Acknowledgments

All members of the Coproduction Laboratory provided input that shaped this manuscript during laboratory meetings. We acknowledge advice from Elizabeth Carpenter-Song, an expert in qualitative methods.

Coproduction Laboratory group contributors: Stephanie C Acquilano ( http://orcid.org/0000-0002-1215-5531 ), Julie Doherty ( http://orcid.org/0000-0002-5279-6536 ), Rachel C Forcino ( http://orcid.org/0000-0001-9938-4830 ), Tina Foster ( http://orcid.org/0000-0001-6239-4031 ), Megan Holthoff, Christopher R Jacobs ( http://orcid.org/0000-0001-5324-8657 ), Lisa C Johnson ( http://orcid.org/0000-0001-7448-4931 ), Elaine T Kiriakopoulos, Kathryn Kirkland ( http://orcid.org/0000-0002-9851-926X ), Meredith A MacMartin ( http://orcid.org/0000-0002-6614-6091 ), Emily A Morgan, Eugene Nelson, Elizabeth O’Donnell, Brant Oliver ( http://orcid.org/0000-0002-7399-622X ), Danielle Schubbe ( http://orcid.org/0000-0002-9858-1805 ), Gabrielle Stevens ( http://orcid.org/0000-0001-9001-178X ), Rachael P Thomeer ( http://orcid.org/0000-0002-5974-3840 ).

Contributors: Practical thematic analysis, an approach designed for multidisciplinary health services teams new to qualitative research, was based on CHS’s experiences teaching thematic analysis to clinical teams and students. We have drawn heavily from qualitative methods literature. CHS is the guarantor of the article. CHS, AS, CvP, AMK, JRK, and JAP contributed to drafting the manuscript. AS, JG, CMM, JAP, and RWY provided feedback on their experiences using practical thematic analysis. CvP, LCL, SLB, AVC, GE, and JKL advised on qualitative methods in health services research, given extensive experience. All authors meaningfully edited the manuscript content, including AVC and RKS. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Funding: This manuscript did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Competing interests: All authors have completed the ICMJE uniform disclosure form at https://www.icmje.org/disclosure-of-interest/ and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

Provenance and peer review: Not commissioned; externally peer reviewed.

• Ziebland S ,
• ↵ A Hybrid Approach to Thematic Analysis in Qualitative Research: Using a Practical Example. 2018. https://methods.sagepub.com/case/hybrid-approach-thematic-analysis-qualitative-research-a-practical-example .
• Maguire M ,
• Vindrola-Padros C ,
• Vindrola-Padros B
• ↵ Vindrola-Padros C. Rapid Ethnographies: A Practical Guide . Cambridge University Press 2021. https://play.google.com/store/books/details?id=n80HEAAAQBAJ
• Schroter S ,
• Merino JG ,
• Barbeau A ,
• ↵ Padgett DK. Qualitative and Mixed Methods in Public Health . SAGE Publications 2011. https://play.google.com/store/books/details?id=LcYgAQAAQBAJ
• Scharp KM ,
• Korstjens I
• Barnett-Page E ,
• ↵ Guest G, Namey EE, Mitchell ML. Collecting Qualitative Data: A Field Manual for Applied Research . SAGE 2013. https://play.google.com/store/books/details?id=-3rmWYKtloC
• Sainsbury P ,
• Emerson RM ,
• Saunders B ,
• Kingstone T ,
• Hennink MM ,
• Kaiser BN ,
• Hennink M ,
• O’Connor C ,
• ↵ Yen RW, Schubbe D, Walling L, et al. Patient engagement in the What Matters Most trial: experiences and future implications for research. Poster presented at International Shared Decision Making conference, Quebec City, Canada. July 2019.
• ↵ Got questions about Thematic Analysis? We have prepared some answers to common ones. https://www.thematicanalysis.net/faqs/ (accessed 9 Nov 2022).
• ↵ Braun V, Clarke V. Thematic Analysis. SAGE Publications. 2022. https://uk.sagepub.com/en-gb/eur/thematic-analysis/book248481 .
• Kalpokas N ,
• Radivojevic I
• Campbell KA ,
• Durepos P ,
• ↵ Understanding Thematic Analysis. https://www.thematicanalysis.net/understanding-ta/ .
• Saunders CH ,
• Stevens G ,
• CONFIDENT Study Long-Term Care Partners
• MacQueen K ,
• Vaismoradi M ,
• Turunen H ,
• Schott SL ,
• Berkowitz J ,
• Carpenter-Song EA ,
• Goldwag JL ,
• Durand MA ,
• Goldwag J ,
• Saunders C ,
• Mishra MK ,
• Rodriguez HP ,
• Shortell SM ,
• Verdinelli S ,
• Scagnoli NI
• Campbell C ,
• Sparkes AC ,
• McGannon KR
• Sandelowski M ,
• Connelly LM ,
• O’Malley AJ ,

Jump to navigation

Cochrane Training

Thematic synthesis.

The webinar began by outlining the key features of thematic synthesis and how it relates to other synthesis methods. It then illustrated the steps involved using worked examples. The different ways it has been used were highlighted and the value of descriptive and analytical themes was discussed. The webinar then moved onto a discussion of rigour in thematic synthesis and key issues to consider when planning a thematic synthesis.

The webinar was delivered in February 2022 and below you will find the videos from the webinar, together with accompanying slides to download [PDF].

Part 1: Thematic synthesis: an overview of its use and features Part 2: How to do a thematic synthesis

Presenter bios.

Professor Angela Harden is Professor of Health Sciences at City, University of London. She is a social scientist and currently leads interdisciplinary and interprofessional collaborative research testing interventions tackling the wider determinants of health across the life course. She is interested in the development of both primary and systematic review methods to better develop and evaluate complex interventions and generate evidence for decision-makers. Angela also builds research capacity in applied research through her role as the Academy Director of the NIHR North Thames ARC. She has been a Co-convenor of the Cochrane Qualitative and Implementation Methods group since 2008. 

Part 1: Thematic synthesis: an overview of its use and features

Part 2: How to do a thematic synthesis

Additional materials

Download the slides from the webinar [PDF].

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Supplements
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 6, Issue 6
• Defining global health: findings from a systematic review and thematic analysis of the literature
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0002-3263-9154 Melissa Salm 1 ,
• Mahima Ali 2 ,
• Mairead Minihane 2 ,
• Patricia Conrad 3
• 1 Anthropology , University of California Davis , Davis , California , USA
• 2 University of California Davis , Davis , California , USA
• 3 VM:PMI , University of California Davis , Davis , California , USA
• Correspondence to Melissa Salm; melsalm{at}gmail.com

Introduction Debate around a common definition of global health has seen extensive scholarly interest within the last two decades; however, consensus around a precise definition remains elusive. The objective of this study was to systematically review definitions of global health in the literature and offer grounded theoretical insights into what might be seen as relevant for establishing a common definition of global health.

Method A systematic review was conducted with qualitative synthesis of findings using peer-reviewed literature from key databases. Publications were identified by the keywords of ‘global health’ and ‘define’ or ‘definition’ or ‘defining’. Coding methods were used for qualitative analysis to identify recurring themes in definitions of global health published between 2009 and 2019.

Results The search resulted in 1363 publications, of which 78 were included. Qualitative analysis of the data generated four theoretical categories and associated subthemes delineating key aspects of global health. These included: (1) global health is a multiplex approach to worldwide health improvement taught and pursued at research institutions; (2) global health is an ethically oriented initiative that is guided by justice principles; (3) global health is a mode of governance that yields influence through problem identification, political decision-making, as well as the allocation and exchange of resources across borders and (4) global health is a vague yet versatile concept with multiple meanings, historical antecedents and an emergent future.

Conclusion Extant definitions of global health can be categorised thematically to designate areas of importance for stakeholders and to organise future debates on its definition. Future contributions to this debate may consider shifting from questioning the abstract ‘what’ of global health towards more pragmatic and reflexive questions about ‘who’ defines global health and towards what ends.

• health education and promotion
• health policy
• public health
• qualitative study
• systematic review

Data availability statement

No data are available. All data relevant to the study are included in the article or uploaded as supplementary information. n/a.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:  http://creativecommons.org/licenses/by-nc/4.0/ .

https://doi.org/10.1136/bmjgh-2021-005292

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Key questions

What is already known.

Debate around a common definition of global health has seen extensive scholarly interest within the last two decades; despite the abundance of literature, ambiguity still persists around its precise definition.

No systematic reviews with thematic analysis have been conducted to explore extant definitions of global health nor to contribute to a comprehensive definition of global health.

What are the new findings?

We compile and thematically analyse extant definitions of global health and propose grounded theoretical insights into what might be seen as relevant for establishing a common definition of global health moving forward.

The need for a clear and concise definition of global health has the highest stakes in the domain of global health policy governance.

What do the new findings imply?

Stakeholders tend to define the ‘what’ of global health: its spaces, objects and practices. Our findings suggest that the debate around definition should shift to more pragmatic and reflexive questions regarding ‘who’ defines global health and towards what ends.

Introduction

Debate around a common definition of global health (GH) has seen extensive scholarly interest within the last two decades. In 2009, a widely circulated paper by Koplan and colleagues aimed to establish ‘a common definition of global health’ as distinct from its derivations in public health (PH) and international health (IH). 1 They rooted the definition of PH in the mid-19th century social reform movements of Europe and the USA, the growth of biological and medical knowledge, and the discipline’s emphasis on population-level health management. Similarly, they traced the evolution of IH back to its colonial roots in hygiene and tropical medicine (TM) through to the mid-20th century with its geographic focus on developing countries. GH, they argued, would require a distinctive definition of its own to be ‘more than a rephrasing of a common definition of PH or a politically correct updating of international health’. Their intervention built on prior research noting confusion and overlap among the three terms and thus a need to carefully articulate the important differences between them. 2–5 Additional stakeholders have since elaborated varied definitions of GH, yet consensus around its precise definition remains elusive.

To determine how GH is presently defined and to identify whether a common conceptualisation has been established, we conducted a qualitative systematic literature review (SLR) of the GH literature between 2009 and 2019. SLRs are a methodology used ‘to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a given research question’. 6 Unlike unsystematic narrative reviews, SLRs use formal, repeatable and transparent, procedures for identifying, evaluating and interpreting available research, thus ensuring robust coverage of the current literature while reducing the biased presentation of available evidence. 7–9 Medical researchers and policy-makers have long relied on SLRs because they integrate and critically evaluate current knowledge to support decisions about important issues. 10 However, very few SLRs exploring aspects of GH have yet been published, 11–13 and no SLRs focusing on extant definitions of GH have been conducted. This paper fills that gap by exploring the thematic components of extant definitions and thereby contributes towards a comprehensive definition of GH.

Aims and objectives

The aim of this review is: (a) to examine how GH has been defined in the literature between 2009 and 2019, (b) to systematically analyse the core thematic categories undergirding extant definitions of GH and (c) to offer grounded theoretical insights into what might be seen as relevant for establishing a common definition of GH.

Aiming to capture definitions of GH in literature between 2009 and 2019, our team conducted a systematic review of the peer-reviewed literature following Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines ( figure 1 ). 14 The sequential steps of our review process included the following.

• Download figure
• Open in new tab
• Download powerpoint

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram of citation analysis and systematic literature review. 14

Search strategy: identify papers and relevant databases

Search technique.

The terms ‘global health’ AND ‘define’ OR ‘definition’ OR ‘defining’ were queried when they appeared in the title, abstract or keyword of studies. Published studies were identified through comprehensive searches of electronic databases accessible through the authors’ university library system (Web of Science, Scopus, Embase, PubMed, EBSCO). Citation tracking through Google Scholar was also completed.

Study selection criteria

Articles published in international peer-reviewed journals, including conference papers, book chapters and editorial material, were reviewed. The studies included were written in English and published between 2009 and 2019. The year 2009 was chosen as a starting point because this is the year in which Koplan et al published ‘Towards a Common Definition of Global Health’. For this review, the team excluded news articles, theses, book reviews and published papers that were not written in English.

Assessment strategy: appraise which papers to include in review

The protocol-driven search strategy required that articles included in the review must: (a) contain the keywords ‘global health’ and ‘definition’ and/or ‘define’; (b) be in the English language and (c) be published between 2009 and 2019. The number of articles containing these keywords was recorded, and all the titles uncovered in the search were imported into Mendeley, a software for managing citations. Duplicates were identified and removed, after which abstracts were screened to assess eligibility against the inclusion criteria. Full-text articles were retrieved for those that met the inclusion criteria and three team members read a designated number of the articles selected for full review. To be included in the data extraction sheet, each article needed to: (a) focus on and explicitly name GH, (b) offer an original definition or description of GH and/or (c) cite an already-existing definition of GH. Articles that mentioned the query terms without any relation to these requirements (eg, did not provide a definition of GH or descriptive data to support interpretations of a GH definition) were excluded. Assessment for relevance and content was conducted by two investigators who reviewed all identified articles independently. Disagreements were resolved by consensus with a third investigator.

Synthesis strategy: extract the data

Based on the research goals, the team designed an initial coding template in Google Sheets as a method of documentation, with the following coding variables: author, title, typology, definition(s), conclusions and conceptual dimensions. To achieve a high level of reliability, the review team open-coded the same five articles, compared their coding experiences, and reconciled differences before adopting a final coding template and evenly dividing the remaining articles to be analysed. Extracted data included the type of study or research paradigm of each publication, the location and disciplinary affiliation of each study based on the contact information of the corresponding author, definitions and descriptions of GH and specialised dimensions of GH. Whenever articles contained more than one definition or description of GH, those items were organised line-by-line under the author on the data extraction sheet.

Analysis strategy: analyse the data

The team conducted thematic analysis of the data to understand how GH has been defined since 2009. Our approach to thematic analysis was based on the guidelines described by Thomas and Harden 15 and further informed by principles in grounded theory. 16 Our strategy consisted of three main stages: Initial Coding—remaining open to all possible emergent themes indicated by readings of the data; 16 17 Focused Coding—categorising the data inductively based on thematic similarity at the level of description 17 and finally, Theoretical Coding—integrating thematic categories into core theoretical constructs at a higher level of analysis. 18

In the first cycle, open descriptive codes were generated (eg, differences between PH and IH, GH education requirements, social justice values) directly from the definitions and descriptions of GH found in the articles. Individual sentences defining or describing GH were treated as unique line items on the data extraction sheet and coded accordingly in order to generate a range of ideas and information on which to build.

In the second cycle, a focused thematic analysis was carried out to identify general relationships and patterns among definitions in the literature and to confirm significant links between the openly coded data. Thematic phrases (eg, GH is multidisciplinary, GH promotes equity) were developed and reapplied to coded definitions on the data extraction sheet. Team members wrote and attached analytic memos to each coded datum—reflecting on emergent patterns and further ‘codeweaving’, 18 which is a term for charting possible relationships among the coded data. At this stage, additional coding techniques were utilised. Attribute coding was applied as a management technique for logging information about the characteristics of each publication. 19 Data segments coded in this manner were extracted from the main data extraction form and reassembled together in a separate Google Sheet for further analysis. The team also coded extracted definitions of GH by type: (a) original definition, (b) cited definition, (c) original description to track possible relationships between citational practices and developments in the conceptualisation and definition of GH.

In the third cycle, thematic phrases were ordered according to frequency then commonality and abstracted for overriding significance into theoretical categories. At this stage, the conceptual level of analysis was raised from description to a more abstract, theoretical level leading to a grounded theory. This resulted in the construction of four thematic categories, which are presented below with their supporting subthemes.

Patient and public involvement

Patients and public were not directly involved in this review; we used publicly available data for the analysis.

The search strategy retrieved bibliographic records for 1363 papers. The assessment strategy resulted in the elimination of 1237 papers after the removal of duplicates. Consequently, 78 papers were subjected to our strategies of synthesis (data extraction) and analysis.

Characteristics of study

A variety of studies were included in this review. The majority (27) were commentaries, viewpoints or debates. 1 20–48 Twenty-four were grouped as review/overview articles. 45–68 There were 25 original research articles, of which 13 used qualitative methods, 69–81 11 used mixed-methods 82–92 and one 93 used quantitative data from a survey to proffer definitions of GH. Two studies included in the review were book chapters. 94 95

The typologic, geographic and disciplinary distribution of the studies in this review are shown in table 1 . Most studies were authored in North America (40), 1 20–31 39–41 43 46 47 50 54–58 61 63 66 68 70 73 74 76–80 83 84 86 87 89–91 94 followed by European countries (29), 22 26 28 32 34–38 42 44 45 48 51 52 59 62 64 65 67 71 75 82 85 88 92 93 95 96 countries in Asia (2), 33 72 Latin America and the Caribbean (2), 60 81 and New Zealand (1). 20 Disciplinary fields represented in our sample included health (56), 20 22–27 30–32 34–40 42 43 45–51 54–56 58–61 63–69 72 74 75 77–79 82–84 86 88–91 93 95 law, social and cultural professions (19), 1 20 28 29 33 41 44 52 53 57 62 70 71 73 76 80 81 87 92 94 and education (2). 20 31

• View inline

Summary of characteristics of retrieved publications

Attributes of definitions

All 78 studies under review defined, described and/or cited extant definitions of GH. The 34 papers shown in table 2 included descriptive definitions of GH that were formulated distinctly by its authors, that is, they were presented as original and without direct reference to other definitions.

How global health has been defined by academics since 2009

Several scholars engaged directly with the Koplan et al definition of GH 1 to stipulate definitions of their own. For example, some authors proposed amendments to Koplan et al that would place greater emphasis on inequity reduction and the need for collaboration, 20 particularly with institutional partners from developing countries. 73 Others were more critical of the broad yet weak conceptual idealism 86 of Koplan et al and recommended detaching normative objectives from its definition, 26 such as the value-laden concept of equity, which could compromise the definition’s technical neutrality by rendering it ideological. 91 Other authors sought to analytically clarify the meaning of ‘the global’ 26 in the definition provided by Koplan et al , distinguish it more clearly from IH 78 or dispute their distinction between GH and PH. 27 Indeed, the impact of the definition of GH proposed by Koplan et al has been substantial. It was variously adopted by the Consortium of Universities for Global Health, 47 the Canadian government, 23 Global Health for Family Medicine, 89 the German Academy of Sciences 75 and the Chinese Consortium of Universities for Global Health. 77

In general, GH was defined as a term, 37 51 95 and in particular, an umbrella term 49 75 or a concept; 69 and more broadly as a zone 76 or field 32 48 91 94 or area of research and practice, 1 56 as an achievable goal, 50 an approach, 48 82 a set of principles, 45 83 an organising framework for thinking and action 96 or a collection of problems. 35 94 GH was frequently contrasted to IH 32 35 68 69 94 95 and PH, 20 21 31 32 35 or else seen as indistinguishably from PH and IH. 27 Additionally, several papers explicitly specified and subsequently defined certain dimensions of GH, such as ‘global health governance’ (GHG), 32 33 35 38 42 51 52 58 69 80 81 87 ‘global health diplomacy’ (GHD), 24 28 95 ‘global health education’, 36 39 46–49 59 70 74 75 77 78 82 89 93 ‘global health security’, 26 41 76 88 92 97 98 ‘global health network’, 41 81 ‘global health actor’, 52 ‘global health ethics’, 69 ‘global health academics’ 64 67 and ‘global health social justice’ 61 (see table 3 ).

Frequently defined facets of ‘Global Health’ with exemplary definitions

Grounded theory approach based on thematic analysis

Definitions and descriptions of GH were aggregated into nine thematic codes reflecting the contents and scope of GH definitions, the functionality of those definitions and/or perceptions about defining GH. Codes were: (1) GH is a domain of research, healthcare and education, (2) GH is multifaceted (disciplinary, sectoral, cultural, national), (3) GH is rooted in a commitment to equity, (4) GH is a political field comprising power relations, (5) GH is problem-oriented, (6) GH transcends national borders, (7) GH is determined by globalisation and international interdependence, (8) conceptually, GH is either similar or dissimilar to PH, IH and TM and (9) GH is perceived as definitionally vague.

These codes were grouped selectively into higher analytical categories or theoretical statements as grounded in the literature: (1) GH is a multiplex approach to worldwide health improvement and form of expertise taught and researched through academic institutions, (2) GH is an ethos (ethical orientation and appeal) that is guided by justice principles, (3) GH is a mode of governance that yields degrees of national, international, transnational and supranational influence through political decision-making, problem identification, the allocation and exchange of resources across borders, (4) GH is a polysemous concept with many meanings and historical antecedents, and which has an emergent future ( table 4 ).

Defining global health with grounded theory analysis—table of themes, code categories and quotes from text

Theme: global health is a multiplex approach to worldwide health improvement taught and pursued through research institutions

Subtheme: gh is a domain of research, healthcare, education.

GH was repeatedly defined as an active field of knowledge production that is composed of the following key elements: research, education, training and practice related to health improvement. 1 20 21 23 32 33 35 38 40 44–49 52 55–58 61 63–69 72 74 75 77 78 80 82 90–92 94 Few authors defined GH as a new, independent discipline within the broader domain of medical knowledge, 17 33 38 46 63 74 80 82 90 and some outlined discipline-specific competencies that were considered integral to the definition of GH, at least in curriculum development; for example: clinical literacy, 80 medical humanities, 82 cross-cultural sensitivity, 33 38 46 59 63 80 90 experiential learning 47 and critical thinking skills. 72 82 Several authors defined GH as a diffuse arena of scholarship that spans an array of academic disciplines, including anthropology, engineering, law, agriculture and healthcare administration. 44 56 59 63–65 78 91 94 Others defined GH explicitly as a ‘transdiscipline’ that seeks to transcend the restricted gaze of any single discipline and consequently integrate knowledge from a variety of sources. 67 94 Several authors explicitly defined GH as a necessarily collaborative field. 1 20 22 24 36 43 45 47 57 61 63 68 77 78 80 91

Subtheme: GH is multifaceted (disciplinary, sectoral, cultural, national)

The prefix ‘multi-’ was consistently applied in definitions of GH to describe a perspective that focuses on the multitude of interrelated factors, dimensions, values and features that underpin health as well as efforts to improve and study it. There was broad agreement that multidisciplinarity is a defining characteristic of GH. 1 23 25 32–34 36 38 40 45–47 49 52 55–57 59 60 64–69 72 75 77 78 80 82 91 However, there was some debate whether multiple disciplines are always needed and beneficial—and therefore essential—to the definition of GH. 23 One author argued that the multidisciplinary nature of GH is precisely what differentiates it from PH and IH. 68 Although some claimed that GH, with its focus on social and economic determinants, is inherently ‘predisposed to include aspects of the liberal arts and social sciences’, 75 others critically observed that most GH educational opportunities still cater predominantly to medical students, 32 35 48 72 which suggests that greater efforts will be required to achieve multidisciplinarity in the field moving forward.

There was a correspondence between GH definitions citing multidisciplinarity and cultural competency. 32 33 38 48 49 56 78 82 90 Curiously, multisectorality was less frequently mentioned than multidisciplinarity in definitions of GH, though it was referenced in some papers. 20 22 43 52 66 83 86 95

Theme: global health is an ethical initiative that is guided by justice principles

Subtheme: gh is rooted in values of equity and social justice.

Equity and social justice were the two most commonly and explicitly referenced values undergirding GH definitions and goals. Equity was repeatedly framed as a ‘main objective’ 60 and core component of GH research and practice. 23 25 43 46 48 53 66 67 77 78 84 However, it remains unclear whether the authors in our sample share the same meaning of equity. Velji and Bryant defined equity broadly as ‘ensuring equal opportunities and resources to enable all people to achieve their fullest health potential’. 66 Meanwhile, others rooted their conceptualisation of equity more specifically in the principles of social justice 30 61 69 88 89 or the human rights concept of equality, 54 62 67 83 86 which asserts that ‘all people are equal in regard to dignity and rights, regardless of their origin and all biological, social or other specific differences’. 59 This postwar sensibility echoes the 1978 Alma Ata Declaration of ‘health for all’, 20 24 as well as a traditional humanitarian ideal, even if now associated with principles grounded in national and global security. 24 54 88

Occasionally, the terms ‘equity’ and ‘equality’ were used interchangeably, suggesting they possess a commonly shared valence and reciprocal relationship despite slight differences in signification. Whereas equity refers to the provision of resources and opportunity based on specific needs, equality connotes providing the same level of resources and opportunities for all. 86 Nevertheless, other scholars questioned whether equity or equality should be included in official definitions of GH, at all, 27 48 75 insofar as what counts as ‘equitable’ for one country may be different for another. 26 32 48

Theme: global health is a form of governance that yields national, international, transnational and supranational influence through political decision-making, problem identification, the allocation and exchange of resources across borders

Subtheme: gh is a political field comprising power relations at multiple scales.

Numerous papers defined GH as embedded within a political field comprising power relations at multiple scales. 20 22–24 26 28 29 31–33 35 41 42 45 48 51–54 56 58 60 63 66 70 72 76 79 87 95 ‘Political field’ refers here to a sphere of influence and jurisdiction wherein institutions determine governing modalities (eg, laws, policies, instruments) to assure a range of activities, such as determining priorities, coordinating stakeholders, regulating funding mechanisms, establishing accountability, allocating resources and providing access to health services for the general public. ‘Power relations’ refers to the capacity of institutions, individuals, instruments and ideas to affect the actions of others; and ‘at multiple scales’ refers to levels of analysis (ie, worldwide, regional, national, local, etc.).

Within the literature on GHG and GH security, authors argued the need for a universal definition of GH to shape policy frameworks that ensure compliance with IH law. 32 45 51 88 95 Here, it is important to note that the ability to shape GH policy is, itself, an exercise in power: some GH actors, defined as ‘individuals or organizations that operate transnationally with a primary intent to improve health’, 56 are more capacitated than others to impact the formulation of policies and amount of attention and resources that certain GH issues receive. 32 41 45 52 95 For example, several papers discussed how ‘GH actors’ like the World Bank and the WHO shaped discussions around the response to Ebola, leading to refined definitions of GHG 35 87 88 and GH security. 41 Similarly, definitions of GH in line with the 2015 United Nations Millennium Development Goals, were also commonly referenced, 25 35 45 51 reflecting the influence of certain GH actors on the conceptualisation of GH.

Subtheme: GH is determined by globalisation and international interdependence

Numerous authors linked interdependence and accelerating globalisation (the process of integrating governments and markets, and of connecting people worldwide) with the need for a cohesive definition of GH, particularly to address issues of governance. 24 32 35 45 68 88 GHG and GHD were outlined as two influential subdomains in which the interconnections between globalisation, foreign policy and international relations were viewed as indispensable to definitions of GH. Two articles quoted David P Fidler’s definition of GHG as ‘the use of formal and informal institutions, rules, and processes by states, intergovernmental organizations, and nonstate actors to deal with challenges to health that require cross-border collective action to address effectively’. 35 58 Elsewhere, GHD was described as ‘bringing together the disciplines of public health, international affairs, management, law and economics and focuses on negotiations that shape and manage the global policy environment for health’. 95

Subtheme: GH issues transcend national borders

Across several papers, we observed a common refrain that GH ‘crosses borders’ and ‘transcends national boundaries’. 1 20 23 42 45 52 60 67 68 74 Authors frequently described GH concerns as those exceeding the jurisdictional reaches of any individual nation-state alone. 34 42 45 51 52 54 77 95 One paper claimed that GH is ‘transnational by definition’, 74 and others characterised GH problems as those experienced transnationally. 20 32 48 50 68

Studies focusing on GH research and training frequently referenced specific diseases and health risks that ‘transcend national borders’ alongside parallel recommendations to include an international component in the development of GH curricula. 16 48 49 63 74 93 While crossing national borders to research and promote health for all is widely perceived as an historical condition for GH 24 that has led to GH’s emergence as an academic discipline, 63 several scholars argued that GH should also focus on domestic health disparities 1 27 38 46 and for local issues to be simultaneously understood as universal or worldwide 48 74 75 to the extent they may occur anywhere 22 and are almost always impacted by global phenomena. 56

Subtheme: GH is problem-oriented

Medical anthropologists, Arthur Kleinman and Paul Farmer, described GH as a collection of problems rather than a distinct discipline. 35 94 Several authors in our review delineated GH problems through identification of specific diseases, such as HIV/AIDS, malaria, TB, Zika and Ebola. 24 29 30 35 45 83 Lee and Brumme noted that it has become common for experts to define GH problems by identifying their objects, namely diseases, population groups and locations. 58 Indeed, some authors outlined GH problems as the set of challenges ‘among those most neglected in developing countries’, 86 among them: emerging infectious diseases and maternal and child health; 43 65 diabetes, cardiovascular disease and other noncommunicable diseases in ‘local’ communities 25 63 and even neurological disorders among refugees arriving in Europe. 93 How these types of object-based definitions of GH problems come to shape GH agendum is important to note.

Clark made a compelling argument against the definition of GH problems in terms of specific diseases, writing that such ‘medicalisation’ may ‘prove detrimental for how the world responds and resources actions designed to alleviate poor health and poverty, redress inequities, and save lives’. 72 Brada also argued against defining GH problems geographically and instead urged experts to consider how the processes by which GH and its quintessential spaces, namely ‘resource-limited’ and ‘resource-poor settings’, are actively constituted, reinforced and contested. 70 Several authors similarly suggested that focusing on the social, political, economic and cultural forces contributing to health inequity and diseases of poverty better captured the scope of GH problems than naming any particular set of diseases or places in the world. 33 43 56 58 69 72 73 86 92

Lack of consensus regarding what counts as a ‘true’ GH problem was linked to the lack of a clear and concise definition of GH. Indeed, several scholars argued that the current inability to define GH made it difficult for stakeholders to define precisely what the ‘problem’ is. 44 45 48 86 Furthermore, the diagnosis of GH problems determined what types of GH ‘solutions’ were proposed in response. For example, when GH problems were defined as universally shared and transnational, then cross-border solutions were developed; when GH issues were framed epidemiologically in terms of distributed risk, then actions targeting specific determinants and burdens were proposed. 1 20 23 67 68 92 When GH problems were framed as threats to inter/national security, strategies were formulated to protect borders, economies, health systems and to improve surveillance mechanisms. 41 45 54 76 80 88 When the problem of inequality drove definitions of GH, recommendations to alleviate poverty, food insecurity, poor sanitation, etc. were proposed. 32 53 60 72

Although Kuhlmann suggested that GH tends to over-prioritise problem-identification to the detriment of critical solution-oriented work, 31 our analysis suggests that the type, scope and quality of solutions proposed are contingent on the elaboration of problems. Similarly, Campbell wrote, ‘Unlike a science or an art, the field of global health is very much about providing solutions to current problems. As such, it would be short-sighted not to consider the causes of global health problems in order to better formulate the solutions. The causes ought to be included in a comprehensive and complete definition of the field’. 23

Theme: global health is a polysemous concept with historical antecedents and an emergent future

Subtheme: gh is conceptually dis/similar to ph, ih and tm.

GH was consistently traced back to and compared with PH, IH and TM. 1 20 27 32–34 43 57 69 71 75 84 86 88 Disagreement or confusion regarding the degrees of similarity and difference between these domains seemed to stem from a shared understanding that GH, in fact, evolved to a varying degree from each of these fields and does not, therefore, denote a clear-cut break with nor full-blown departure from any of them. 84 94

Several authors argued that the scope and scale of GH is distinct from PH. 1 20 32 69 71 Some argued that ‘public health is equated primarily with population-wide interventions; global health is concerned with all strategies for health improvement,’ including clinical care; 20 and that ‘public health acknowledges the state as a dominant actor, (while) global health recognizes the rise of other actors like international institutions’. 35 GH was also seen as placing a greater emphasis on multidisciplinarity and promoting a more expansive conceptualisation of ‘health’, itself, compared with PH. 69 Beyond the prevention of and response to biomedicalised health risks at the population level, Rowson defined GH as oriented towards the ‘underlying determinants of those problems, which are social, political and economic in nature.’ 32 It is questionable, however, to assume similar notions of health have not also been pursued in PH. Meanwhile, opposing views found GH and PH conceptually indistinguishable, 27 43 86 either as terms that could be used interchangeably, 95 or else as coconstitutive of one another, such that PH could be understood as a descriptive component of GH. 33 86

Differences between GH and IH echoed those drawn between GH and PH. For example, GH was characterised as more attentive to multidisciplinarity, while IH was said to implement a more limited biomedical approach to healthcare and health research. 1 69 95 Undergirding a major point of distinction between GH and IH was the belief that IH focuses on health problems in developing countries 1 22 32 43 45 48 54 83 86 93 and relies on ‘the flow of resources and knowledge from the developed to the developing world’, 32 whereas GH either is, or should be, more bidirectional. 1 45 84 In other cases, GH was described as comparable to IH, for example, when countries link GH efforts with development aid. 86 This is because the emphasis on delivering aid to poor countries reinforces an image of the world’s poor as needy subjects and, therefore, marks a continuation of IH and its sentiments under the guise of GH. 35

Finally, the field of TM was referenced to describe the evolutionary track of GH, particularly that GH is a modern-day product of the former. 20 25 57 69 75 84 A few authors critically pointed out that although GH has generally replaced TM and IH as terms embedded in histories of colonial power relations, many of the contemporary structures for governing and/or facilitating GH between countries today have remained largely the same, 25 48 54 62 suggesting that distinguishability between these terms too often occurs at the level of semantics.

Subtheme: GH is still vaguely defined

While GH was often described as a popular and well-established term, another key attribute repeated across the literature was its enduring vagueness. 23 25 26 31 33 43 45 48 52 62 74–77 81 86 Indeed, most papers commented on the term’s defiance of easy definition, its ambiguity and the lack of clarity regarding how people and organisations engaged in GH are using (or not using) the term to describe their interests. For example, Beaglehole and Bonita pointed out that research centres in low-income and middle-income countries are often engaged in GH issues but under other labels. 20 Some authors viewed the present lack of a clear and common definition as an obstacle endangering the coherence and maturation of the field. 33 35 45 For others, this indistinctness was thought to be precisely what gives GH such wide applicability, a certain degree of currency and political expediency. 45 76 81 86

A major concern cited was the lack of guidance for defining the term ‘global’ in GH. 26 34 43 48 75 As Bozorgmehr has outlined, the term is often used interchangeably within the GH community to mean ‘worldwide’, ‘everywhere’, ‘holistic’ and/or ‘issues that transcend national boundaries’. 48 This trend was noticeable within our review, as well. Engebretsen emphasised that GH ‘does not only allude to supranational dependency within the health field, but refers to a norm or vision for health with global ambitions’. 26 This view suggests that because the planet is populated by a multiplicity of positionings, perspectives and diverse world views, there can never be a truly a universal definition of ‘the global’ nor a global consensus around the definition of GH.

Finally, among studies that conducted original research into the definition of GH, several reported that study participants could not reach consensus on a definition. 52 74 75 77 Many thought it would be difficult if not impossible to arrive at a single, unified theoretical definition of GH, yet considered it important to formulate an operational definition of GH for guiding emerging activities related to GH. 23 45 77

This is the first study to systematically synthesise the literature defining GH and analyse the definitions found therein. All of the articles included in this study were published in peer-reviewed journals since 2009 indicating recent and steadfast interest in the topic of GH’s definition. This review examined GH definitions in the literature, and our thematic analysis focused on identifying recurrent themes across different definitions of GH.

Of the 78 articles included in this study, approximately one-third utilised empirical research methodologies to posit definitions of GH or else directly contribute towards the establishment of a common definition. Another one-third of papers summarised and discussed previously published definitions of GH (eg, reviews/overviews), while the remaining one-third suggested definitions of GH that were less grounded in analysis of empirical data than in the perspectives of its authors (eg, editorials, viewpoints). This systematic analysis indicated that the question of GH’s precise definition marks a point of controversy across fields of expertise. The variety of GH definitions posited by diverse experts in search of a common definition indicate that GH is multifaceted and polysemous.

In its broadest sense, GH can be defined as an area of research and practice committed to the application of overtly multidisciplinary, multisectoral and culturally sensitive approaches for reducing health disparities that transcend national borders. Indeed, it was most commonly defined across the literature in such general terms.

More specific definitions of GH were, of course, proposed by and considered valuable for many stakeholders in our review. Our analysis indicates that the precise definitions proposed by different experts were devised to serve particular functions. For example, narrow and concise definitions of GH were most frequently sought in the domains of governance and education, primarily for steering the development of policy frameworks and curricula, respectively. The imperative for an exact definition of GH in these subfields may be linked to bureaucratic demands for demarcating a technical term under which to classify specific activities, standardise certain functions, administer funds and direct workflow accordingly. It is also in this domain that authors most vociferously decried the absence of a unified and concise definition of GH, arguing this lack has led to ineffective initiatives, elusive methods for establishing accountability and instances of resource allocation based on ad hoc criteria—attractiveness to donors, public opinion, development agendum, foreign, economic or security policy priorities and so on—rather than via transparent mechanisms for adjudicating health need. 28 54 58 65 83 In contexts where health needs and upstream challenges were articulated, the lack of an agreed-upon definition oft impeded the policy process because stakeholders could not discern which GH issues among the multitude of different problems labelled as important were, in fact, the most pressing. 24 45 52 Because political indecision ramifies disproportionately for publics in countries where reliance on GH aid is a matter of life and death, establishing a clear definition of GH seems most crucial for the domain of governance.

We also found that detailed descriptions of GH’s specific conceptual and functional dimensions tended to reflect the specialisations or discipline-specific priorities of their authors. For example, definitions of GH stipulating the primacy of ‘cultural competency’ and ‘multidisciplinarity’ were more commonly proposed by interdisciplinary professionals in the literature on GH education than in journals of health policy, where definitions of GH were oriented more toward ‘security’ and ‘governance’ concerns. This suggests a correspondence between the subjective, experiential positions of the definers and the vocabulary they used to define or frame the need to define GH.

Unsurprisingly, we found that health professionals proposed the majority of definitions of GH in the literature. Additionally, the majority of publications and their authors were from higher income countries. Several authors in our review critically observed that GH has become institutionalised at a faster rate in higher income countries compared with lower and middle-income countries. 20 48 63 72 77 82 Their observations combined with our findings suggest that extant definitions of GH published in the literature or otherwise circulating in academic and professionalised spaces may unevenly reflect the interests and priorities of stakeholders from higher income countries. This suggests a need for greater diversity and inclusion in the debate on GH’s definition, as well as further reflexivity regarding who is defining GH, their means and motivations for doing so, and what these definitions put into action.

Interestingly, several articles published since 2019 have extended the debate on this topic of GH’s definition by directly engaging questions of geography and positionality: a recent commentary by King and Kolski defining GH ‘as public health somewhere else’ was met with pushback by those who argue that spatial definitions of GH are limited and limiting. 99–102

Limitations

To determine how GH is defined by experts in the literature, we ensured that the selection criteria developed for this study were broad enough to include a wide range of perspectives. Therefore, we included articles with varying degrees of evidentiary support, such as viewpoints, commentaries and editorials. Consequently, the results may be influenced by some of the primary researchers’ assumptions, projections, and biases. Backward citation tracking was used to add relevant articles to the review that had not been initially identified through database searching. This ensured that the review was exhaustive, however it also means that some conclusions drawn in the thematic analysis may have been influenced by this manual search strategy. By applying qualitative methods, this review provided a robust analysis of the thematic categories undergirding extant definitions of GH. A major limitation of this form of analysis is the extensive time required to develop and establish a code book and standardise the three coders’ use of the code book. However, this was deemed necessary to ensure consistency of judgement and intercoder reliability at each stage in the analysis. Another limitation of this study is that only articles written in English were included. To enhance the generalisability of results, future reviews should include data from non-English articles, especially if an inclusive, common definition of GH is to be achieved. Finally, this review was finalised prior to the emergence of the novel coronavirus. As such, future research should take into account new definitions of GH that emerge in light of the pandemic and lessons learnt.

Between 2009 and 2019, GH was most commonly defined in the literature in broad and general terms: as an area of research and practice committed to the application of multidisciplinary, multisectoral and culturally sensitive approaches for reducing health disparities that transcend national borders. More precise definitions exist to serve particular functions and tend to reflect the priorities of its definers. The four key themes that emerged from the present analysis are that GH is: (1) a multiplex approach to worldwide health improvement taught and researched through academic institutions; (2) an ethos that is guided by justice principles; (3) a mode of governance that yields influence through political decision-making, problem identification, the allocation and exchange of resources across borders and (4) a polysemous concept with historical antecedents and an emergent future. Findings from this thematic analysis have the potential to organise future conversations about which definition of GH is most common and/or most useful. Future discussions on the topic might shift from questioning the abstract ‘what’ of GH to more pragmatic and reflexive questions about ‘who’ defines GH and towards what ends.

Acknowledgments

Helpful comments by anonymous reviewers are acknowledged with thanks.

• Koplan JP ,
• Merson MH , et al
• Macfarlane SB ,
• Kickbusch I
• Liberati A ,
• Altman DG ,
• Tetzlaff J , et al
• Mengist W ,
• Soromessa T ,
• Carver JC ,
• Hassler E ,
• Hernandes E
• Missoni E , et al
• DiPace JI , et al
• Shamseer L ,
• Clarke M , et al
• Beaglehole R ,
• Bozorgmehr K
• Campbell RM ,
• De Cock KM ,
• Simone PM ,
• Davison V , et al
• Engebretsen E ,
• Bentley ME ,
• Buekens P , et al
• Johnston BD
• Gureje O , et al
• Kuhlmann AS ,
• Willott C ,
• Hughes R , et al
• Margolis CZ
• Kadandale S ,
• Nordström A , et al
• John CC , et al
• Speakman EM ,
• Joyner PU ,
• Aluttis C ,
• Arthur MAM ,
• Seidman G ,
• Chadi N , et al
• Bozorgmehr K ,
• Tinnemann P
• Sugarman J , Working Group on Ethics Guidelines for Global Health Training (WEIGHT)
• Bhutta ZA , et al
• Haffeld JB ,
• Røttingen J-A
• Hoffman SJ ,
• Stanhope M , et al
• Jogerst K ,
• Callender B ,
• Adams V , et al
• Schuster S ,
• Vajda C , et al
• Nascimento BR ,
• Brant LCC ,
• Moraes DN , et al
• Nemetchek B
• Steenhoff AP ,
• Crouse HL ,
• Lukolyo H , et al
• ↵ The International health regulations (IHR) – 10 years of global public health security . Wkly Epidemiol Rec 2017 ; 92 : 534 – 6 . pmid: http://www.ncbi.nlm.nih.gov/pubmed/28884994 OpenUrl PubMed
• Van Belle S ,
• Kickbusch I , et al
• Mendes IAC ,
• Klopper H , et al
• Benatar S ,
• Batra M , et al
• Havemann M ,
• Xu D , et al
• Peluso MJ ,
• Encandela J ,
• Hafler JP , et al
• Swanson RC ,
• Schensul JJ , et al
• Sienko KH ,
• Kaufmann EE
• Moore J , et al
• Sridharan S
• Erondu NA ,
• Marten R , et al
• Gostin LO ,
• Friedman EA
• Heymann DL ,
• Takemi K , et al
• Redwood-Campbell L ,
• Rouleau K , et al
• Keating EM ,
• Rowthorn V ,
• Schäferhoff M ,
• Kraus J , et al
• Sauerbier A ,
• Macerollo A ,
• Györfi O , et al
• Kleinman A , et al
• Gan Y , et al
• Turcotte-Tremblay A-M ,
• Fregonese F ,
• Kadio K , et al
• Herzig van Wees S ,
• Fischer SE ,
• Zielinski C , et al

Handling editor Seye Abimbola

Contributors MS initiated and designed the project. MS, MA and MM contributed to the implementation of the research, to the collection of data, analysis of the results and to the writing of the manuscript. PC supervised the project and provided feedback on the manuscript.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

Linked Articles

• Editorial The challenges of defining global health research Alberto L Garcia-Basteiro Seye Abimbola BMJ Global Health 2021; 6 - Published Online First: 30 Dec 2021. doi: 10.1136/bmjgh-2021-008169

Read the full text or download the PDF:

Electronic problem lists: a thematic analysis of a systematic literature review to identify aspects critical to success

Affiliations.

• 1 Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA.
• 2 Intermountain Healthcare, Salt Lake City, UT, USA.
• PMID: 29547974
• PMCID: PMC7647009
• DOI: 10.1093/jamia/ocy011

Objective: Problem list data is a driving force for many beneficial clinical tools, yet these data remain underutilized. We performed a systematic literature review, pulling insights from previous research, aggregating insights into themes, and distilling themes into actionable advice. We sought to learn what changes we could make to existing applications, to the clinical workflow, and to clinicians' perceptions that would improve problem list utilization and increase the prevalence of problems data in the electronic medical record.

Materials and methods: We followed Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines to systematically curate a corpus of pertinent articles. We performed a thematic analysis, looking for interesting excerpts and ideas. By aggregating excerpts from many authors, we gained broader, more inclusive insights into what makes a good problem list and what factors are conducive to its success.

Results: Analysis led to a list of 7 benefits of using the problem list, 15 aspects critical to problem list success, and knowledge to help inform policy development, such as consensus on what belongs on the problem list, who should maintain the problem list, and when.

Conclusions: A list of suggestions is made on ways in which the problem list can be improved to increase utilization by clinicians. There is also a need for standard measurements of the problem list, so that lists can be measured, compared, and discussed with rigor and a common vocabulary.

Publication types

• Meta-Analysis
• Systematic Review
• Medical Records Systems, Computerized*
• Medical Records, Problem-Oriented*

• Locations and Hours
• UCLA Library
• Research Guides
• Biomedical Library Guides

Systematic Reviews

• Types of Literature Reviews

What Makes a Systematic Review Different from Other Types of Reviews?

• Planning Your Systematic Review
• Database Searching
• Creating the Search
• Search Filters and Hedges
• Grey Literature
• Managing and Appraising Results
• Further Resources

Reproduced from Grant, M. J. and Booth, A. (2009), A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26: 91–108. doi:10.1111/j.1471-1842.2009.00848.x

• << Previous: Home
• Next: Planning Your Systematic Review >>
• Last Updated: Apr 17, 2024 2:02 PM
• URL: https://guides.library.ucla.edu/systematicreviews

• Open access
• Published: 17 August 2023

Data visualisation in scoping reviews and evidence maps on health topics: a cross-sectional analysis

• Emily South   ORCID: orcid.org/0000-0003-2187-4762 1 &
• Mark Rodgers 1  

Systematic Reviews volume  12 , Article number:  142 ( 2023 ) Cite this article

3614 Accesses

13 Altmetric

Metrics details

Scoping reviews and evidence maps are forms of evidence synthesis that aim to map the available literature on a topic and are well-suited to visual presentation of results. A range of data visualisation methods and interactive data visualisation tools exist that may make scoping reviews more useful to knowledge users. The aim of this study was to explore the use of data visualisation in a sample of recent scoping reviews and evidence maps on health topics, with a particular focus on interactive data visualisation.

Ovid MEDLINE ALL was searched for recent scoping reviews and evidence maps (June 2020-May 2021), and a sample of 300 papers that met basic selection criteria was taken. Data were extracted on the aim of each review and the use of data visualisation, including types of data visualisation used, variables presented and the use of interactivity. Descriptive data analysis was undertaken of the 238 reviews that aimed to map evidence.

Of the 238 scoping reviews or evidence maps in our analysis, around one-third (37.8%) included some form of data visualisation. Thirty-five different types of data visualisation were used across this sample, although most data visualisations identified were simple bar charts (standard, stacked or multi-set), pie charts or cross-tabulations (60.8%). Most data visualisations presented a single variable (64.4%) or two variables (26.1%). Almost a third of the reviews that used data visualisation did not use any colour (28.9%). Only two reviews presented interactive data visualisation, and few reported the software used to create visualisations.

Conclusions

Data visualisation is currently underused by scoping review authors. In particular, there is potential for much greater use of more innovative forms of data visualisation and interactive data visualisation. Where more innovative data visualisation is used, scoping reviews have made use of a wide range of different methods. Increased use of these more engaging visualisations may make scoping reviews more useful for a range of stakeholders.

Peer Review reports

Scoping reviews are “a type of evidence synthesis that aims to systematically identify and map the breadth of evidence available on a particular topic, field, concept, or issue” ([ 1 ], p. 950). While they include some of the same steps as a systematic review, such as systematic searches and the use of predetermined eligibility criteria, scoping reviews often address broader research questions and do not typically involve the quality appraisal of studies or synthesis of data [ 2 ]. Reasons for conducting a scoping review include the following: to map types of evidence available, to explore research design and conduct, to clarify concepts or definitions and to map characteristics or factors related to a concept [ 3 ]. Scoping reviews can also be undertaken to inform a future systematic review (e.g. to assure authors there will be adequate studies) or to identify knowledge gaps [ 3 ]. Other evidence synthesis approaches with similar aims have been described as evidence maps, mapping reviews or systematic maps [ 4 ]. While this terminology is used inconsistently, evidence maps can be used to identify evidence gaps and present them in a user-friendly (and often visual) way [ 5 ].

Scoping reviews are often targeted to an audience of healthcare professionals or policy-makers [ 6 ], suggesting that it is important to present results in a user-friendly and informative way. Until recently, there was little guidance on how to present the findings of scoping reviews. In recent literature, there has been some discussion of the importance of clearly presenting data for the intended audience of a scoping review, with creative and innovative use of visual methods if appropriate [ 7 , 8 , 9 ]. Lockwood et al. suggest that innovative visual presentation should be considered over dense sections of text or long tables in many cases [ 8 ]. Khalil et al. suggest that inspiration could be drawn from the field of data visualisation [ 7 ]. JBI guidance on scoping reviews recommends that reviewers carefully consider the best format for presenting data at the protocol development stage and provides a number of examples of possible methods [ 10 ].

Interactive resources are another option for presentation in scoping reviews [ 9 ]. Researchers without the relevant programming skills can now use several online platforms (such as Tableau [ 11 ] and Flourish [ 12 ]) to create interactive data visualisations. The benefits of using interactive visualisation in research include the ability to easily present more than two variables [ 13 ] and increased engagement of users [ 14 ]. Unlike static graphs, interactive visualisations can allow users to view hierarchical data at different levels, exploring both the “big picture” and looking in more detail ([ 15 ], p. 291). Interactive visualizations are often targeted at practitioners and decision-makers [ 13 ], and there is some evidence from qualitative research that they are valued by policy-makers [ 16 , 17 , 18 ].

Given their focus on mapping evidence, we believe that scoping reviews are particularly well-suited to visually presenting data and the use of interactive data visualisation tools. However, it is unknown how many recent scoping reviews visually map data or which types of data visualisation are used. The aim of this study was to explore the use of data visualisation methods in a large sample of recent scoping reviews and evidence maps on health topics. In particular, we were interested in the extent to which these forms of synthesis use any form of interactive data visualisation.

This study was a cross-sectional analysis of studies labelled as scoping reviews or evidence maps (or synonyms of these terms) in the title or abstract.

The search strategy was developed with help from an information specialist. Ovid MEDLINE® ALL was searched in June 2021 for studies added to the database in the previous 12 months. The search was limited to English language studies only.

The search strategy was as follows:

Ovid MEDLINE(R) ALL

(scoping review or evidence map or systematic map or mapping review or scoping study or scoping project or scoping exercise or literature mapping or evidence mapping or systematic mapping or literature scoping or evidence gap map).ab,ti.

limit 1 to english language

(202006* or 202007* or 202008* or 202009* or 202010* or 202011* or 202012* or 202101* or 202102* or 202103* or 202104* or 202105*).dt.

The search returned 3686 records. Records were de-duplicated in EndNote 20 software, leaving 3627 unique records.

A sample of these reviews was taken by screening the search results against basic selection criteria (Table 1 ). These criteria were piloted and refined after discussion between the two researchers. A single researcher (E.S.) screened the records in EPPI-Reviewer Web software using the machine-learning priority screening function. Where a second opinion was needed, decisions were checked by a second researcher (M.R.).

Our initial plan for sampling, informed by pilot searching, was to screen and data extract records in batches of 50 included reviews at a time. We planned to stop screening when a batch of 50 reviews had been extracted that included no new types of data visualisation or after screening time had reached 2 days. However, once data extraction was underway, we found the sample to be richer in terms of data visualisation than anticipated. After the inclusion of 300 reviews, we took the decision to end screening in order to ensure the study was manageable.

Data extraction

A data extraction form was developed in EPPI-Reviewer Web, piloted on 50 reviews and refined. Data were extracted by one researcher (E. S. or M. R.), with a second researcher (M. R. or E. S.) providing a second opinion when needed. The data items extracted were as follows: type of review (term used by authors), aim of review (mapping evidence vs. answering specific question vs. borderline), number of visualisations (if any), types of data visualisation used, variables/domains presented by each visualisation type, interactivity, use of colour and any software requirements.

When categorising review aims, we considered “mapping evidence” to incorporate all of the six purposes for conducting a scoping review proposed by Munn et al. [ 3 ]. Reviews were categorised as “answering a specific question” if they aimed to synthesise study findings to answer a particular question, for example on effectiveness of an intervention. We were inclusive with our definition of “mapping evidence” and included reviews with mixed aims in this category. However, some reviews were difficult to categorise (for example where aims were unclear or the stated aims did not match the actual focus of the paper) and were considered to be “borderline”. It became clear that a proportion of identified records that described themselves as “scoping” or “mapping” reviews were in fact pseudo-systematic reviews that failed to undertake key systematic review processes. Such reviews attempted to integrate the findings of included studies rather than map the evidence, and so reviews categorised as “answering a specific question” were excluded from the main analysis. Data visualisation methods for meta-analyses have been explored previously [ 19 ]. Figure  1 shows the flow of records from search results to final analysis sample.

Flow diagram of the sampling process

Data visualisation was defined as any graph or diagram that presented results data, including tables with a visual mapping element, such as cross-tabulations and heat maps. However, tables which displayed data at a study level (e.g. tables summarising key characteristics of each included study) were not included, even if they used symbols, shading or colour. Flow diagrams showing the study selection process were also excluded. Data visualisations in appendices or supplementary information were included, as well as any in publicly available dissemination products (e.g. visualisations hosted online) if mentioned in papers.

The typology used to categorise data visualisation methods was based on an existing online catalogue [ 20 ]. Specific types of data visualisation were categorised in five broad categories: graphs, diagrams, tables, maps/geographical and other. If a data visualisation appeared in our sample that did not feature in the original catalogue, we checked a second online catalogue [ 21 ] for an appropriate term, followed by wider Internet searches. These additional visualisation methods were added to the appropriate section of the typology. The final typology can be found in Additional file 1 .

We conducted descriptive data analysis in Microsoft Excel 2019 and present frequencies and percentages. Where appropriate, data are presented using graphs or other data visualisations created using Flourish. We also link to interactive versions of some of these visualisations.

Almost all of the 300 reviews in the total sample were labelled by review authors as “scoping reviews” ( n  = 293, 97.7%). There were also four “mapping reviews”, one “scoping study”, one “evidence mapping” and one that was described as a “scoping review and evidence map”. Included reviews were all published in 2020 or 2021, with the exception of one review published in 2018. Just over one-third of these reviews ( n  = 105, 35.0%) included some form of data visualisation. However, we excluded 62 reviews that did not focus on mapping evidence from the following analysis (see “ Methods ” section). Of the 238 remaining reviews (that either clearly aimed to map evidence or were judged to be “borderline”), 90 reviews (37.8%) included at least one data visualisation. The references for these reviews can be found in Additional file 2 .

Number of visualisations

Thirty-six (40.0%) of these 90 reviews included just one example of data visualisation (Fig.  2 ). Less than a third ( n  = 28, 31.1%) included three or more visualisations. The greatest number of data visualisations in one review was 17 (all bar or pie charts). In total, 222 individual data visualisations were identified across the sample of 238 reviews.

Number of data visualisations per review

Categories of data visualisation

Graphs were the most frequently used category of data visualisation in the sample. Over half of the reviews with data visualisation included at least one graph ( n  = 59, 65.6%). The least frequently used category was maps, with 15.6% ( n  = 14) of these reviews including a map.

Of the total number of 222 individual data visualisations, 102 were graphs (45.9%), 34 were tables (15.3%), 23 were diagrams (10.4%), 15 were maps (6.8%) and 48 were classified as “other” in the typology (21.6%).

Types of data visualisation

All of the types of data visualisation identified in our sample are reported in Table 2 . In total, 35 different types were used across the sample of reviews.

The most frequently used data visualisation type was a bar chart. Of 222 total data visualisations, 78 (35.1%) were a variation on a bar chart (either standard bar chart, stacked bar chart or multi-set bar chart). There were also 33 pie charts (14.9% of data visualisations) and 24 cross-tabulations (10.8% of data visualisations). In total, these five types of data visualisation accounted for 60.8% ( n  = 135) of all data visualisations. Figure  3 shows the frequency of each data visualisation category and type; an interactive online version of this treemap is also available ( https://public.flourish.studio/visualisation/9396133/ ). Figure  4 shows how users can further explore the data using the interactive treemap.

Data visualisation categories and types. An interactive version of this treemap is available online: https://public.flourish.studio/visualisation/9396133/ . Through the interactive version, users can further explore the data (see Fig.  4 ). The unit of this treemap is the individual data visualisation, so multiple data visualisations within the same scoping review are represented in this map. Created with flourish.studio ( https://flourish.studio )

Screenshots showing how users of the interactive treemap can explore the data further. Users can explore each level of the hierarchical treemap ( A Visualisation category >  B Visualisation subcategory >  C Variables presented in visualisation >  D Individual references reporting this category/subcategory/variable permutation). Created with flourish.studio ( https://flourish.studio )

Data presented

Around two-thirds of data visualisations in the sample presented a single variable ( n  = 143, 64.4%). The most frequently presented single variables were themes ( n  = 22, 9.9% of data visualisations), population ( n  = 21, 9.5%), country or region ( n  = 21, 9.5%) and year ( n  = 20, 9.0%). There were 58 visualisations (26.1%) that presented two different variables. The remaining 21 data visualisations (9.5%) presented three or more variables. Figure  5 shows the variables presented by each different type of data visualisation (an interactive version of this figure is available online).

Variables presented by each data visualisation type. Darker cells indicate a larger number of reviews. An interactive version of this heat map is available online: https://public.flourish.studio/visualisation/10632665/ . Users can hover over each cell to see the number of data visualisations for that combination of data visualisation type and variable. The unit of this heat map is the individual data visualisation, so multiple data visualisations within a single scoping review are represented in this map. Created with flourish.studio ( https://flourish.studio )

Most reviews presented at least one data visualisation in colour ( n  = 64, 71.1%). However, almost a third ( n  = 26, 28.9%) used only black and white or greyscale.

Interactivity

Only two of the reviews included data visualisations with any level of interactivity. One scoping review on music and serious mental illness [ 22 ] linked to an interactive bubble chart hosted online on Tableau. Functionality included the ability to filter the studies displayed by various attributes.

The other review was an example of evidence mapping from the environmental health field [ 23 ]. All four of the data visualisations included in the paper were available in an interactive format hosted either by the review management software or on Tableau. The interactive versions linked to the relevant references so users could directly explore the evidence base. This was the only review that provided this feature.

Software requirements

Nine reviews clearly reported the software used to create data visualisations. Three reviews used Tableau (one of them also used review management software as discussed above) [ 22 , 23 , 24 ]. Two reviews generated maps using ArcGIS [ 25 ] or ArcMap [ 26 ]. One review used Leximancer for a lexical analysis [ 27 ]. One review undertook a bibliometric analysis using VOSviewer [ 28 ], and another explored citation patterns using CitNetExplorer [ 29 ]. Other reviews used Excel [ 30 ] or R [ 26 ].

To our knowledge, this is the first systematic and in-depth exploration of the use of data visualisation techniques in scoping reviews. Our findings suggest that the majority of scoping reviews do not use any data visualisation at all, and, in particular, more innovative examples of data visualisation are rare. Around 60% of data visualisations in our sample were simple bar charts, pie charts or cross-tabulations. There appears to be very limited use of interactive online visualisation, despite the potential this has for communicating results to a range of stakeholders. While it is not always appropriate to use data visualisation (or a simple bar chart may be the most user-friendly way of presenting the data), these findings suggest that data visualisation is being underused in scoping reviews. In a large minority of reviews, visualisations were not published in colour, potentially limiting how user-friendly and attractive papers are to decision-makers and other stakeholders. Also, very few reviews clearly reported the software used to create data visualisations. However, 35 different types of data visualisation were used across the sample, highlighting the wide range of methods that are potentially available to scoping review authors.

Our results build on the limited research that has previously been undertaken in this area. Two previous publications also found limited use of graphs in scoping reviews. Results were “mapped graphically” in 29% of scoping reviews in any field in one 2014 publication [ 31 ] and 17% of healthcare scoping reviews in a 2016 article [ 6 ]. Our results suggest that the use of data visualisation has increased somewhat since these reviews were conducted. Scoping review methods have also evolved in the last 10 years; formal guidance on scoping review conduct was published in 2014 [ 32 ], and an extension of the PRISMA checklist for scoping reviews was published in 2018 [ 33 ]. It is possible that an overall increase in use of data visualisation reflects increased quality of published scoping reviews. There is also some literature supporting our findings on the wide range of data visualisation methods that are used in evidence synthesis. An investigation of methods to identify, prioritise or display health research gaps (25/139 included studies were scoping reviews; 6/139 were evidence maps) identified 14 different methods used to display gaps or priorities, with half being “more advanced” (e.g. treemaps, radial bar plots) ([ 34 ], p. 107). A review of data visualisation methods used in papers reporting meta-analyses found over 200 different ways of displaying data [ 19 ].

Only two reviews in our sample used interactive data visualisation, and one of these was an example of systematic evidence mapping from the environmental health field rather than a scoping review (in environmental health, systematic evidence mapping explicitly involves producing a searchable database [ 35 ]). A scoping review of papers on the use of interactive data visualisation in population health or health services research found a range of examples but still limited use overall [ 13 ]. For example, the authors noted the currently underdeveloped potential for using interactive visualisation in research on health inequalities. It is possible that the use of interactive data visualisation in academic papers is restricted by academic publishing requirements; for example, it is currently difficult to incorporate an interactive figure into a journal article without linking to an external host or platform. However, we believe that there is a lot of potential to add value to future scoping reviews by using interactive data visualisation software. Few reviews in our sample presented three or more variables in a single visualisation, something which can easily be achieved using interactive data visualisation tools. We have previously used EPPI-Mapper [ 36 ] to present results of a scoping review of systematic reviews on behaviour change in disadvantaged groups, with links to the maps provided in the paper [ 37 ]. These interactive maps allowed policy-makers to explore the evidence on different behaviours and disadvantaged groups and access full publications of the included studies directly from the map.

We acknowledge there are barriers to use for some of the data visualisation software available. EPPI-Mapper and some of the software used by reviews in our sample incur a cost. Some software requires a certain level of knowledge and skill in its use. However numerous online free data visualisation tools and resources exist. We have used Flourish to present data for this review, a basic version of which is currently freely available and easy to use. Previous health research has been found to have used a range of different interactive data visualisation software, much of which does not required advanced knowledge or skills to use [ 13 ].

There are likely to be other barriers to the use of data visualisation in scoping reviews. Journal guidelines and policies may present barriers for using innovative data visualisation. For example, some journals charge a fee for publication of figures in colour. As previously mentioned, there are limited options for incorporating interactive data visualisation into journal articles. Authors may also be unaware of the data visualisation methods and tools that are available. Producing data visualisations can be time-consuming, particularly if authors lack experience and skills in this. It is possible that many authors prioritise speed of publication over spending time producing innovative data visualisations, particularly in a context where there is pressure to achieve publications.

Limitations

A limitation of this study was that we did not assess how appropriate the use of data visualisation was in our sample as this would have been highly subjective. Simple descriptive or tabular presentation of results may be the most appropriate approach for some scoping review objectives [ 7 , 8 , 10 ], and the scoping review literature cautions against “over-using” different visual presentation methods [ 7 , 8 ]. It cannot be assumed that all of the reviews that did not include data visualisation should have done so. Likewise, we do not know how many reviews used methods of data visualisation that were not well suited to their data.

We initially relied on authors’ own use of the term “scoping review” (or equivalent) to sample reviews but identified a relatively large number of papers labelled as scoping reviews that did not meet the basic definition, despite the availability of guidance and reporting guidelines [ 10 , 33 ]. It has previously been noted that scoping reviews may be undertaken inappropriately because they are seen as “easier” to conduct than a systematic review ([ 3 ], p.6), and that reviews are often labelled as “scoping reviews” while not appearing to follow any established framework or guidance [ 2 ]. We therefore took the decision to remove these reviews from our main analysis. However, decisions on how to classify review aims were subjective, and we did include some reviews that were of borderline relevance.

A further limitation is that this was a sample of published reviews, rather than a comprehensive systematic scoping review as have previously been undertaken [ 6 , 31 ]. The number of scoping reviews that are published has increased rapidly, and this would now be difficult to undertake. As this was a sample, not all relevant scoping reviews or evidence maps that would have met our criteria were included. We used machine learning to screen our search results for pragmatic reasons (to reduce screening time), but we do not see any reason that our sample would not be broadly reflective of the wider literature.

Data visualisation, and in particular more innovative examples of it, is currently underused in published scoping reviews on health topics. The examples that we have found highlight the wide range of methods that scoping review authors could draw upon to present their data in an engaging way. In particular, we believe that interactive data visualisation has significant potential for mapping the available literature on a topic. Appropriate use of data visualisation may increase the usefulness, and thus uptake, of scoping reviews as a way of identifying existing evidence or research gaps by decision-makers, researchers and commissioners of research. We recommend that scoping review authors explore the extensive free resources and online tools available for data visualisation. However, we also think that it would be useful for publishers to explore allowing easier integration of interactive tools into academic publishing, given the fact that papers are now predominantly accessed online. Future research may be helpful to explore which methods are particularly useful to scoping review users.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Organisation formerly known as Joanna Briggs Institute

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Munn Z, Pollock D, Khalil H, Alexander L, McLnerney P, Godfrey CM, Peters M, Tricco AC. What are scoping reviews? Providing a formal definition of scoping reviews as a type of evidence synthesis. JBI Evid Synth. 2022;20:950–952.

Peters MDJ, Marnie C, Colquhoun H, Garritty CM, Hempel S, Horsley T, Langlois EV, Lillie E, O’Brien KK, Tunçalp Ӧ, et al. Scoping reviews: reinforcing and advancing the methodology and application. Syst Rev. 2021;10:263.

Article   PubMed   PubMed Central   Google Scholar  

Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18:143.

Sutton A, Clowes M, Preston L, Booth A. Meeting the review family: exploring review types and associated information retrieval requirements. Health Info Libr J. 2019;36:202–22.

Article   PubMed   Google Scholar  

Miake-Lye IM, Hempel S, Shanman R, Shekelle PG. What is an evidence map? A systematic review of published evidence maps and their definitions, methods, and products. Syst Rev. 2016;5:28.

Tricco AC, Lillie E, Zarin W, O’Brien K, Colquhoun H, Kastner M, Levac D, Ng C, Sharpe JP, Wilson K, et al. A scoping review on the conduct and reporting of scoping reviews. BMC Med Res Methodol. 2016;16:15.

Khalil H, Peters MDJ, Tricco AC, Pollock D, Alexander L, McInerney P, Godfrey CM, Munn Z. Conducting high quality scoping reviews-challenges and solutions. J Clin Epidemiol. 2021;130:156–60.

Lockwood C, dos Santos KB, Pap R. Practical guidance for knowledge synthesis: scoping review methods. Asian Nurs Res. 2019;13:287–94.

Article   Google Scholar  

Pollock D, Peters MDJ, Khalil H, McInerney P, Alexander L, Tricco AC, Evans C, de Moraes ÉB, Godfrey CM, Pieper D, et al. Recommendations for the extraction, analysis, and presentation of results in scoping reviews. JBI Evidence Synthesis. 2022;10:11124.

Google Scholar  

Peters MDJ GC, McInerney P, Munn Z, Tricco AC, Khalil, H. Chapter 11: Scoping reviews (2020 version). In: Aromataris E MZ, editor. JBI Manual for Evidence Synthesis. JBI; 2020. Available from https://synthesismanual.jbi.global . Accessed 1 Feb 2023.

Tableau Public. https://www.tableau.com/en-gb/products/public . Accessed 24 January 2023.

flourish.studio. https://flourish.studio/ . Accessed 24 January 2023.

Chishtie J, Bielska IA, Barrera A, Marchand J-S, Imran M, Tirmizi SFA, Turcotte LA, Munce S, Shepherd J, Senthinathan A, et al. Interactive visualization applications in population health and health services research: systematic scoping review. J Med Internet Res. 2022;24: e27534.

Isett KR, Hicks DM. Providing public servants what they need: revealing the “unseen” through data visualization. Public Adm Rev. 2018;78:479–85.

Carroll LN, Au AP, Detwiler LT, Fu T-c, Painter IS, Abernethy NF. Visualization and analytics tools for infectious disease epidemiology: a systematic review. J Biomed Inform. 2014;51:287–298.

Lundkvist A, El-Khatib Z, Kalra N, Pantoja T, Leach-Kemon K, Gapp C, Kuchenmüller T. Policy-makers’ views on translating burden of disease estimates in health policies: bridging the gap through data visualization. Arch Public Health. 2021;79:17.

Zakkar M, Sedig K. Interactive visualization of public health indicators to support policymaking: an exploratory study. Online J Public Health Inform. 2017;9:e190–e190.

Park S, Bekemeier B, Flaxman AD. Understanding data use and preference of data visualization for public health professionals: a qualitative study. Public Health Nurs. 2021;38:531–41.

Kossmeier M, Tran US, Voracek M. Charting the landscape of graphical displays for meta-analysis and systematic reviews: a comprehensive review, taxonomy, and feature analysis. BMC Med Res Methodol. 2020;20:26.

Ribecca, S. The Data Visualisation Catalogue. https://datavizcatalogue.com/index.html . Accessed 23 November 2021.

Ferdio. Data Viz Project. https://datavizproject.com/ . Accessed 23 November 2021.

Golden TL, Springs S, Kimmel HJ, Gupta S, Tiedemann A, Sandu CC, Magsamen S. The use of music in the treatment and management of serious mental illness: a global scoping review of the literature. Front Psychol. 2021;12: 649840.

Keshava C, Davis JA, Stanek J, Thayer KA, Galizia A, Keshava N, Gift J, Vulimiri SV, Woodall G, Gigot C, et al. Application of systematic evidence mapping to assess the impact of new research when updating health reference values: a case example using acrolein. Environ Int. 2020;143: 105956.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Jayakumar P, Lin E, Galea V, Mathew AJ, Panda N, Vetter I, Haynes AB. Digital phenotyping and patient-generated health data for outcome measurement in surgical care: a scoping review. J Pers Med. 2020;10:282.

Qu LG, Perera M, Lawrentschuk N, Umbas R, Klotz L. Scoping review: hotspots for COVID-19 urological research: what is being published and from where? World J Urol. 2021;39:3151–60.

Article   CAS   PubMed   Google Scholar  

Rossa-Roccor V, Acheson ES, Andrade-Rivas F, Coombe M, Ogura S, Super L, Hong A. Scoping review and bibliometric analysis of the term “planetary health” in the peer-reviewed literature. Front Public Health. 2020;8:343.

Hewitt L, Dahlen HG, Hartz DL, Dadich A. Leadership and management in midwifery-led continuity of care models: a thematic and lexical analysis of a scoping review. Midwifery. 2021;98: 102986.

Xia H, Tan S, Huang S, Gan P, Zhong C, Lu M, Peng Y, Zhou X, Tang X. Scoping review and bibliometric analysis of the most influential publications in achalasia research from 1995 to 2020. Biomed Res Int. 2021;2021:8836395.

Vigliotti V, Taggart T, Walker M, Kusmastuti S, Ransome Y. Religion, faith, and spirituality influences on HIV prevention activities: a scoping review. PLoS ONE. 2020;15: e0234720.

van Heemskerken P, Broekhuizen H, Gajewski J, Brugha R, Bijlmakers L. Barriers to surgery performed by non-physician clinicians in sub-Saharan Africa-a scoping review. Hum Resour Health. 2020;18:51.

Pham MT, Rajić A, Greig JD, Sargeant JM, Papadopoulos A, McEwen SA. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Res Synth Methods. 2014;5:371–85.

Peters MDJ, Marnie C, Tricco AC, Pollock D, Munn Z, Alexander L, McInerney P, Godfrey CM, Khalil H. Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth. 2020;18:2119–26.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, Moher D, Peters MDJ, Horsley T, Weeks L, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169:467–73.

Nyanchoka L, Tudur-Smith C, Thu VN, Iversen V, Tricco AC, Porcher R. A scoping review describes methods used to identify, prioritize and display gaps in health research. J Clin Epidemiol. 2019;109:99–110.

Wolffe TAM, Whaley P, Halsall C, Rooney AA, Walker VR. Systematic evidence maps as a novel tool to support evidence-based decision-making in chemicals policy and risk management. Environ Int. 2019;130:104871.

Digital Solution Foundry and EPPI-Centre. EPPI-Mapper, Version 2.0.1. EPPI-Centre, UCL Social Research Institute, University College London. 2020. https://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3790 .

South E, Rodgers M, Wright K, Whitehead M, Sowden A. Reducing lifestyle risk behaviours in disadvantaged groups in high-income countries: a scoping review of systematic reviews. Prev Med. 2022;154: 106916.

Download references

Acknowledgements

We would like to thank Melissa Harden, Senior Information Specialist, Centre for Reviews and Dissemination, for advice on developing the search strategy.

This work received no external funding.

Author information

Authors and affiliations.

Centre for Reviews and Dissemination, University of York, York, YO10 5DD, UK

Emily South & Mark Rodgers

You can also search for this author in PubMed   Google Scholar

Contributions

Both authors conceptualised and designed the study and contributed to screening, data extraction and the interpretation of results. ES undertook the literature searches, analysed data, produced the data visualisations and drafted the manuscript. MR contributed to revising the manuscript, and both authors read and approved the final version.

Corresponding author

Correspondence to Emily South .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Typology of data visualisation methods.

Additional file 2.

References of scoping reviews included in main dataset.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

South, E., Rodgers, M. Data visualisation in scoping reviews and evidence maps on health topics: a cross-sectional analysis. Syst Rev 12 , 142 (2023). https://doi.org/10.1186/s13643-023-02309-y

Download citation

Received : 21 February 2023

Accepted : 07 August 2023

Published : 17 August 2023

DOI : https://doi.org/10.1186/s13643-023-02309-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Scoping review
• Evidence map
• Data visualisation

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

ChatGPT in higher education - a synthesis of the literature and a future research agenda

• Open access
• Published: 02 May 2024

Cite this article

You have full access to this open access article

• Pritpal Singh Bhullar 1 ,
• Mahesh Joshi 2 &
• Ritesh Chugh   ORCID: orcid.org/0000-0003-0061-7206 3  

1571 Accesses

33 Altmetric

Explore all metrics

ChatGPT has emerged as a significant subject of research and exploration, casting a critical spotlight on teaching and learning practices in the higher education domain. This study examines the most influential articles, leading journals, and productive countries concerning citations and publications related to ChatGPT in higher education, while also shedding light on emerging thematic and geographic clusters within research on ChatGPT’s role and challenges in teaching and learning at higher education institutions. Forty-seven research papers from the Scopus database were shortlisted for bibliometric analysis. The findings indicate that the use of ChatGPT in higher education, particularly issues of academic integrity and research, has been studied extensively by scholars in the United States, who have produced the largest volume of publications, alongside the highest number of citations. This study uncovers four distinct thematic clusters (academic integrity, learning environment, student engagement, and scholarly research) and highlights the predominant areas of focus in research related to ChatGPT in higher education, including student examinations, academic integrity, student learning, and field-specific research, through a country-based bibliographic analysis. Plagiarism is a significant concern in the use of ChatGPT, which may reduce students’ ability to produce imaginative, inventive, and original material. This study offers valuable insights into the current state of ChatGPT in higher education literature, providing essential guidance for scholars, researchers, and policymakers.

Similar content being viewed by others

Mapping the global evidence around the use of ChatGPT in higher education: A systematic scoping review

ChatGPT and the digitisation of writing

The Educational Affordances and Challenges of ChatGPT: State of the Field

Avoid common mistakes on your manuscript.

1 Introduction

ChatGPT, or Chat Generative Pre-trained Transformer, is a popular generative Artificial Intelligence (AI) chatbot developed by OpenAI, employing natural language processing to deliver interactive human-like conversational experiences (Jeon et al., 2023 ; Angelis et al., 2023 ). ChatGPT utilises a pre-trained language learning model, derived from an extensive big-data corpus, to predict outcomes based on a given prompt (Crawford et al., 2023 ; Geerling et al., 2023 ; Li et al., 2023 ). Since its inception, ChatGPT has attracted widespread attention and popularity and has the potential to disrupt the education sector (Rana, 2023 ). According to a research survey of adults conducted by the Pew Research Centre, approximately 60% of adults in the United States and 78% of adults in Asia possess knowledge of ChatGPT; furthermore, men are more familiar with ChatGPT than women (Vogels, 2023 ). The study also found that among ethnic groups globally, individuals of Asian descent have the highest level of familiarity with AI-based large language models (LLMs).

People have found value in using ChatGPT for a wide range of purposes, including generating creative content, answering questions, providing explanations, offering suggestions, and even having casual conversations (Crawford et al., 2023 ; Throp, 2023 ; Wu et al., 2023 ). Furthermore, ChatGPT is an effective digital assistant for facilitating a thorough understanding of diverse and intricate subjects using simple and accessible language. Given these features, ChatGPT has the potential to bring about a paradigm shift in traditional methods of delivering instruction and revolutionise the future of education (Tlili et al., 2023 ). ChatGPT stands out as a promising tool for open education, enhancing the independence and autonomy of autodidactic learners through personalised support, guidance, and feedback, potentially fostering increased motivation and engagement (Firat, 2023 ). Its capabilities encompass facilitating complex learning, asynchronous communication, feedback provision, and cognitive offloading (Memarian & Doleck, 2023 ).

However, the rapid expansion of ChatGPT has also aroused apprehensions in the academic world, particularly after reports surfaced that the New York Department of Education had unexpectedly imposed a ban on access to the tool due to concerns about academic integrity violations (Sun et al., 2023 ; Neumann et al., 2023 ; Crawford et al., 2023 ). Students who use ChatGPT to produce superior written assignments may have an unfair advantage over peers who lack access (Farrokhnia et al., 2023 ; Cotton et al., 2023 ). Ethical concerns about the deployment of LLMs include the potential for bias, effects on employment, misuse and unethical deployment, and loss of integrity. However, there has been little research on the potential dangers that a sophisticated chatbot such as ChatGPT poses in the realm of higher education, particularly through the lens of a systematic literature review and bibliometric techniques.

In this light, this paper explores the literature on the application of ChatGPT in higher education institutions and the obstacles encountered in various disciplines from the perspectives of both faculty and students. The paper aims to analyse the current state of the field by addressing the following overarching research questions using bibliographic coupling, co-occurrence analysis, citation analysis, and co-authorship analysis:

What are the most influential articles in terms of citations in research related to ChatGPT in education?

What are the top journals and countries in terms of publication productivity related to the implications of ChatGPT in higher education institutions?

What are the emerging thematic clusters in research on the role and challenges of ChatGPT in teaching and learning in higher education institutions?

What are the geographic clusters in research on the role and challenges of ChatGPT in teaching and learning in higher education institutions?

2 Methodology

In conducting this study, publications on the impact of ChatGPT on various aspects of higher education institutions were systematically identified through an extensive search using Elsevier’s Scopus database, a comprehensive repository hosting over 20,000 globally ranked, peer-reviewed journals (Mishra et al., 2017 ; Palomo et al., 2017 ; Vijaya & Mathur, 2023 ). Scopus is a widely used database for bibliometric analyses and is considered one of the “largest curated databases covering scientific journals” (pg. 5116) in different subject areas (Singh et al., 2021 ). Widely acclaimed for its comprehensive coverage, Scopus has been extensively employed in bibliometric analyses across diverse disciplines, as evidenced by studies in capital structure theories, business research, entrepreneurial orientation and blockchain security (Bajaj et al., 2020 ; Donthu et al., 2020 ; Gupta et al., 2021 ; Patrício & Ferreira, 2020 ). Notably, despite the “extremely high” correlation between the Web of Science and Scopus databases, Scopus’s status as a superior and versatile data source for literature extraction is reinforced by its broader coverage of subject areas and categories compared to the narrower journal scope of Web of Science, facilitating scholars in locating literature most pertinent to the review area (Archambault et al., 2009 ; Paul et al., 2021 ). To ensure a systematic literature review, we adhered to the preferred reporting items for systematic reviews and meta-analysis (PRISMA) guidelines (Page et al., 2021 ) for the search, identification, selection, reading, and data extraction from the articles retrieved through the Scopus database (Fig.  1 ). Reliance on a single database is acceptable within the PRISMA framework (Moher et al., 2009 ).

Employing Boolean-assisted search queries, we aimed to capture a comprehensive range of topics related to ChatGPT’s impact on higher education institutions. Specific search queries were carefully selected to ensure a broad yet relevant search scope and included the following:

“ChatGPT and Teaching learning in universities” OR “Effect of ChatGPT in higher education institution” OR “ChatGPT and student assessment in higher education” OR “ChatGPT and academic integrity” OR “ChatGPT and teaching pedagogy in higher education institution” OR “ChatGPT and cheating student course assignment” OR “ChatGPT and teaching in higher education” OR “Implications of ChatGPT in higher education institutions” OR “ChatGPT and evaluation criteria in higher education institution” OR “ChatGPT in universities” OR “ChatGPT and student learnings. ”

The study includes papers published and included in the Scopus database on or before May 26, 2023 on the theme of ChatGPT and higher education. This timeframe was chosen to encompass the most recent and relevant literature available up to the point of data retrieval. Papers identified through the search queries underwent inclusion or exclusion based on predetermined criteria. Specifically, only papers published in journals were considered for this study, as these undergo a peer-review process and are subject to stringent selection criteria set by the journals, ensuring their quality and reliability. Papers in conference proceedings were excluded from the start of the search. Only papers written in English were included to maintain consistency and clarity, whereas others were excluded. Of the 48 research papers that were initially identified, 47 were ultimately selected for the bibliometric analysis, which was conducted using VOSviewer, a bibliometric analysis tool.

PRISMA Flowchart

From the identified pool of 47 articles, the analysis uncovered a nuanced distribution of research methodologies. Specifically, 11 studies were grounded in quantitative research methodologies, underscoring a quantitative focus within the literature. In contrast, a substantial majority of 31 articles embraced a qualitative framework, showcasing a diverse spectrum that included pure qualitative research, editorials, letters to the editor, and opinion pieces. Furthermore, the review brought to light four literature reviews, signifying a synthesis of existing knowledge, and identified one study that strategically employed a mixed-methods approach, blending both qualitative and quantitative research techniques.

To address the research questions, the selected publications underwent analysis using various bibliometric techniques. For the first and second research questions, citation analysis was employed. For the third and fourth research questions, bibliographic analysis was performed in VOSviewer software to generate clusters.

3 Findings and discussion

3.1 publication trend.

Information from the Scopus database indicates that academics began focusing on investigating various aspects of ChatGPT’s potential in higher education in 2022, as they published their findings in 2023. All academic articles in reputable publications in the Scopus database were published in 2023.

3.2 Citation analysis

Table  1 presents the top ten articles according to the number of citations. The number of articles increased significantly in 2023, consistent with the emerging nature and growing relevance of the topic. Exploring the ramifications of ChatGPT in higher education is a recent focal point for scholars, with numerous aspects warranting deeper investigation. The limited citation count, as anticipated, underscores that publications from 2023 are in the early stages of gaining visibility and recognition within the academic community.

The article by Thorp ( 2023 ), entitled “ChatGPT is fun, but not an author”, has received the highest number of citations (79). Thorp stresses the risks associated with implementing ChatGPT in the classroom. Although ChatGPT is an innovative AI tool, significant barriers remain to its implementation in the field of education. According to Thorp, using ChatGPT in academic writing is still inefficient. Thorp also expresses concerns about the rising prevalence of ChatGPT in the fabrication of scientific publications. The second most-cited work, “How Does ChatGPT Perform on the United States Medical Licensing Examination?” by Gilson and colleagues, has received 27 citations. Gilson et al. ( 2023 ) evaluated the accuracy, speed and clarity of ChatGPT’s responses to questions on the United States Medical Licensing Examination’s Step 1 and Step 2 tests. The text responses generated by ChatGPT were evaluated using three qualitative metrics: the logical justification of the chosen answer, the inclusion of information relevant to the question, and the inclusion of information extraneous to the question. The model attained a level of proficiency comparable to that of a third-year medical student. The study demonstrates the potential utility of ChatGPT as an interactive educational resource in the field of medicine to facilitate the acquisition of knowledge and skills. Third is Kasneci et al.’s article “ChatGPT for good? On opportunities and challenges of large language models for education”, with 13 citations. This paper examines the benefits and drawbacks of using language models in the classroom from the perspectives of both teachers and students. The authors find that these comprehensive language models can serve as a supplement rather than a replacement for classroom instruction. Each of the remaining top-ten articles mentioned the impact of ChatGPT on academic integrity in education and had received fewer than ten citations at the time of analysis.

Table  2 presents the top 10 journals in terms of the number of citations of publications related to the topic of ChatGPT in higher education. The journal Science , which published “ChatGPT is fun, but not an author,” was deemed most influential because it received the highest number of citations (79). JMIR Medical Education has published two articles that have been cited by 30 other research articles on the same topic. Journal of University Teaching and Learning Practise has published the most articles: three. Innovations in Education and Teaching International has published two articles on this topic, which together have been cited by six articles.

As shown in Table  3 , the majority of research articles pertaining to ChatGPT and higher education have originated from countries in Asia. Six of the top 10 countries for publishing articles on this topic are located in the Asian continent. However, the most influential studies in terms of citations have been produced by the United States, Germany, Australia, and the United Kingdom. Combined, these countries have received a total of 63 citations, with individual counts of 36, 17, 7, and 7, respectively. These four countries have 90% of the total citations of the top 10 most productive countries in the field of research on higher education perspectives on ChatGPT.

3.3 Bibliographic coupling

3.3.1 thematic clusters.

Four thematic clusters (TCs) were identified from the included research articles, as shown in Table  4 . VOSviewer was used to perform clustering based on bibliographic coupling. This method identifies relations between documents by examining publications that cite the same sources (Boyack & Klavans, 2010 ). VOSviewer clusters articles with a common knowledge base, assigning each publication to exactly one cluster. To implement this clustering technique, we assessed the co-occurrence of bibliographic references among articles within our dataset. Co-occurrence was determined by identifying shared references between articles, indicating a thematic connection (Boyack & Klavans, 2010 ). Articles sharing common references were considered to co-occur, enabling us to quantify the extent of thematic relationships based on the frequency of shared references. We identified and categorised thematic clusters within our dataset through the combined approach of VOSviewer clustering and co-occurrence analysis. This method typically results in a distribution of clusters, with a limited number of larger clusters and a more substantial number of smaller clusters.

The clusters were derived through an analysis of subordinate articles extracted from the Scopus database. VOSviewer systematically organised similar articles into distinct clusters based on the shared patterns of bibliographic references (Van Eck & Waltman, 2010 ). To ensure methodological transparency and robustness, we established clear criteria and parameters for clustering. Specifically, keywords with a minimum frequency ( n  = 5) were included in the analysis, and co-occurrence was calculated based on a pairwise comparison method. This systematic approach ensured the meaningful representation of thematic relationships within the dataset, guided by insights from previous literature (Jarneving, 2007 ). Using cluster analysis techniques, the articles were organised into cohesive groups characterised by the degree of thematic homogeneity guided by the nature of the research findings. This approach ensured a robust representation of the underlying thematic structure (Jarneving, 2007 ).

Furthermore, to mitigate the risk of subjective bias in thematic categorisation, a counter-coding approach was employed. A second researcher independently categorised thematic clusters identified by VOSviewer to assess inter-rater agreement. The level of agreement between the two researchers was assessed using Cohen’s kappa coefficient, ensuring the reliability and validity of the thematic classification process. The resulting kappa coefficient (0.69) indicated substantial agreement, suggesting a high level of agreement beyond what would be expected by chance alone (Gisev et al., 2013 ). Furthermore, the nomenclature assigned to each cluster was finalised based on the predominant research theme emerging from the analysis, providing a concise and informative label for each group.

TC1: ChatGPT and Academic Integrity: Cotton et al. ( 2023 ) describe ChatGPT as a double-edged sword that potentially threatens academic integrity. AI essay writing systems are programmed to churn out essays based on specific guidelines or prompts, and it can be difficult to distinguish between human and machine-generated writing. Thus, students could potentially use these systems to cheat by submitting essays that are not their original work (Dehouche, 2021 ). Kasneci et al. ( 2023 ) argue that effective pedagogical practices must be developed in order to implement large language models in classrooms. These skills include not only a deep understanding of the technology but also an appreciation of its constraints and the vulnerability of complex systems in general. In addition, educational institutions need to develop a clearly articulated plan for the successful integration and optimal use of big language models in educational contexts and teaching curricula. In addition, students need to be taught how to verify information through a teaching strategy emphasising critical thinking effectively. Possible bias in the generated output, the need for continuous human supervision, and the likelihood of unforeseen effects are just a few of the challenges that come with the employment of AI systems. Continuous monitoring and transparency are necessary to ensure academic integrity while using ChatGPT. Lim et al. ( 2023 ) report that ChatGPT poses academic integrity challenges for the faculty of higher education institutions, who must verify whether academic work (assignments, research reports, etc.) submitted by students is derived from the fresh perspective of data analysis or plagiarised and recycled (copying and pasting original work) by ChatGPT. ChatGPT may threaten student learning and classroom engagement if students have access to information and course assignments without assessing their integrity. Perkins ( 2023 ) also expresses concerns regarding academic integrity in the use of ChatGPT. Students are utilising ChatGPT to complete their course assignments without attribution rather than producing original work. Higher education institutions must establish clear boundaries regarding academic integrity and plagiarism in light of the growing utilisation of AI tools in academic and research settings. In addition, the challenges posed by AI essay writing systems like ChatGPT necessitate a multifaceted approach to safeguard academic integrity. Educational institutions should invest in comprehensive educational programs that not only teach students the ethical use of technology but also incorporate rigorous assessments of critical thinking skills. Additionally, integrating AI literacy into the curriculum, with a focus on understanding the limitations and potential biases of big language models, can empower students to discern between human and machine-generated content.

TC2: ChatGPT and Learning Environment: According to Crawford et al. ( 2023 ), increased stress levels and peer pressure among university students have created a favourable environment for the use of AI tools. ChatGPT provides enhanced educational opportunities for college-level students. It can help students identify areas they may have overlooked, offer guidance on additional reading materials, and enhance existing peer and teacher connections. In addition, ChatGPT can propose alternative methods of evaluating students beyond conventional assignments. Crawford et al. ( 2023 ) recommend providing practical assignments incorporating ChatGPT as a supplementary tool to reduce plagiarism. Su ( 2023 ) documents that ChatGPT can provide students with a personalised learning experience based on their specific needs. In addition, the ChatGPT platform can be used to create a virtual coaching system that offers prompt feedback to educators during their classroom evaluations. This approach fosters critical thinking and supports early childhood educators in refining their teaching methodologies to optimise interactive learning outcomes for students. Tang ( 2023b ) proposes that bolstering research integrity can be achieved by imposing restrictions on the utilisation of NLP-generated content in research papers. Additionally, the author advocates for transparency from researchers, emphasising the importance of explicitly stating the proportion of NLP-generated content incorporated in their papers. This recommendation prompts a critical examination of the role of AI-generated content in scholarly work, emphasising the importance of nurturing independent research and writing skills for both students and researchers.

TC3: ChatGPT and Student Engagement: Lee ( 2023 ) examines the ability of ChatGPT to provide an interactive learning experience and boost student engagement beyond textbook pedagogy. Iskender ( 2023 ) explains that ChatGPT provides a mechanism for students to generate and investigate diverse concepts expeditiously, thereby helping them engage in imaginative and evaluative thinking on specific subject matter. This approach has the potential to optimise time management for students and allow them to concentrate on more advanced cognitive activities. AI tools such as ChatGPT can potentially enhance the personalisation of learning materials by providing visual aids and summaries that can aid the learning process and significantly improve students’ competencies. Hence, leveraging ChatGPT in education can revolutionise learning by facilitating interactive experiences, nurturing imaginative thinking, and optimising time management for students.

TC4: ChatGPT and Scholarly Research: Ivanov and Soliman ( 2023 ) and Yan ( 2023 ) focus on the practical applications and implications of LLMs like ChatGPT in educational settings and scholarly research within the context of language learning, writing, and tourism. Yan’s investigation into ChatGPT’s application in second-language writing examines its effectiveness in addressing specific writing tasks at the undergraduate level. The findings underscore the nuanced balance between the strengths of ChatGPT and the inherent limitations in handling demanding academic writing tasks. Nevertheless, ChatGPT is also labelled as an ‘all-in-one’ solution for scholarly research and writing (Yan, 2023 ). In parallel, Ivanov and Soliman ( 2023 ) highlight that ChatGPT can assist scholars in the field of tourism research by composing preliminary literature reviews, substantiating their chosen methodologies, and creating visual aids such as tables and charts. Furthermore, the researchers outline that ChatGPT could provide valuable methodological ideas and insights by helping researchers generate questions and corresponding scales for inclusion in questionnaires. Hence, ChatGPT has the potential to become a valuable ally as a facilitator in academic writing processes and has the potential to transform the research workflow.

3.3.2 Geographic clusters

The results of the country-based bibliographic analysis are summarised in Table  5 . The present study utilised the prevailing research theme in the existing literature as a framework for categorising the countries into four distinct clusters on the basis of the number of documents published from different countries.

Cluster 1: Implications of ChatGPT for Student Examinations and Education : Cluster 1 is composed of five countries: Germany, Ireland, South Korea, Taiwan, and the United States. Researchers in these countries have emphasised the potential role of ChatGPT in higher education within the context of AI language models. Eleven research articles related to this theme were published by researchers based in the United States, the most in this cluster. The top three articles in Table  1 are from the United States. The study entitled “Opportunities and Challenges of Large Language Models for Education,” was authored by German researchers (Kasneci et al., 2023 ) and has been widely cited in the academic community (13 citations). The remaining studies were conducted by researchers from South Korea and Taiwan and focused on the impact of ChatGPT on the education sector and its associated opportunities and challenges. This cluster demonstrates that students could benefit greatly from using ChatGPT in performing various academic tasks, such as reviewing and revising their work, verifying the accuracy of homework answers, and improving the quality of their essays. It has also aided postgraduates whose first language is not English improve their writing, as ChatGPT can be instructed to rewrite a paragraph in a scholarly tone from scratch. The outcomes have demonstrated significant efficacy, thereby alleviating the cognitive load associated with translation for these students, enabling them to concentrate on the substance of their writing rather than the intricacies of composing in an unfamiliar language. To harness the potential benefits, future research could focus on developing targeted training programs for students and educators that emphasise the effective utilisation of ChatGPT to enhance not only academic tasks but also language proficiency for non-native English speakers, addressing both cognitive load and language intricacies.

Cluster 2: ChatGPT and Academic Integrity : Cluster 2 comprises research studies conducted by authors from Japan, Bangladesh, Hong Kong, Nigeria, Pakistan, UAE, the UK, Vietnam and the Netherlands. The most influential study in this cluster, “Unlocking the power of ChatGPT: A framework for applying Generative AI in education”, was authored by researchers from Hong Kong (Su & Yang, 2023 ). They document that ChatGPT can be used to respond to student inquiries, reducing the time and effort required of educators and allowing them to focus their resources on other activities, such as scholarly investigations. Farrokhnia et al. ( 2023 ) and Yeadon et al. ( 2023 ) state that ChatGPT can write scientific abstracts with fabricated data and essays that can evade detection by reviewers. According to Liebrenz et al. ( 2023 ), ChatGPT tends to produce erroneous and incoherent responses, thereby raising the potential for disseminating inaccurate information in scholarly literature. The higher-order cognitive abilities of ChatGPT are relatively low, especially in areas related to creativity, critical thinking, reasoning, and problem-solving. ChatGPT could reduce students’ motivation to explore topics independently, draw their own conclusions, and solve problems independently (Kasneci et al., 2023 ). Ibrahim et al. ( 2023 ) find that ChatGPT can engage students in their academic pursuits. ChatGPT can enhance the writing abilities of non-native English speakers to allow them to concentrate on higher-order cognitive processes. This technological development allows faculty members to allocate more attention to conceptualisation and writing rather than focusing on the mechanics of grammar and spelling. However, there is a debate among intellectuals regarding the implications of AI for content creation, with some asserting that it detracts from innovative content development. The possibility that ChatGPT threatens academic honesty by facilitating essay plagiarism is being acknowledged. In addition, in the absence of appropriate citations, this textual content may violate copyright regulations. Cotton et al. ( 2023 ) express concerns about the potential impact of ChatGPT on academic integrity and plagiarism. Their work corroborates Dehouche’s ( 2021 ) assertion that students may use ChatGPT to engage in academic dishonesty by submitting essays that are not their original work. According to Cotton et al. ( 2023 ), ChatGPT users have a competitive advantage over non-users and can achieve higher grades on their coursework assignments by utilising the AI-based language tool. They classify ChatGPT as a versatile instrument with the potential to pose a threat to academic integrity, noting that AI essay writing systems are specifically programmed to generate content based on specific parameters or prompts, thereby challenging the discernment between human-authored and machine-generated content. Distinguishing between the academic work produced by students and the content of ChatGPT when evaluating assignments is a significant challenge for faculty. It is recommended that academic staff continually monitor student assignments for academic misconduct infractions, coupled with transparent communication about the potential risks associated with AI-generated content.

Cluster 3: ChatGPT and Students’ Learning : Cluster 3 comprises Malaysia, China and Australia. This cluster mainly includes studies of the role of AI-based models in student learning. Researchers from Australia (Crawford et al., 2023 ; Lim et al., 2023 ; Lawrie, 2023 ; Li et al., 2023 ; Seth et al., 2023 ; Cingillioglu, 2023 ; Skavronskaya, 2023 ; and Johinke, 2023 ) have contributed the most (8 studies) to this cluster and put their weight behind the role of AI and student learning in various disciplines. One of the most influential papers, “Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators”, was authored by researchers from both Australia and Malaysia (Lim et al., 2023 ) and reflected on the role of AI in classroom learning and teaching. Rather than banning AI tools, the authors advocate for the productive use of these tools in classrooms to facilitate more engaging student learning. Another Australian study titled, “Leadership is needed for ethical ChatGPT: Character, assessment, and learning using artificial intelligence (AI)” (Crawford et al., 2023 ) highlights AI as an alternative path of learning for students. ChatGPT can promptly evaluate students’ assignments and help them identify areas of weakness. Educators have the option to provide innovative assessments to their students instead of adhering solely to conventional assessments. ChatGPT can augment pedagogical approaches, evaluation structures, and the comprehensive educational milieu by reinforcing the trilateral association among instructors, learners, and technology. The implementation of ChatGPT can provide students with a personalised and interactive learning and research experience facilitated by virtual tutors and customised recommendations. In light of the research in this cluster, the integration of ChatGPT into education should inspire a paradigm shift towards a more dynamic and personalised learning environment. Institutions can explore strategic partnerships with AI researchers to develop context-specific applications of ChatGPT that cater to diverse educational needs, promoting a symbiotic relationship between human instructors, students, and technology for an enriched learning experience.

Cluster 4: ChatGPT and Field-specific Research : This cluster includes research by authors in Asian and European countries (India, Oman, Bulgaria and New Zealand) that has emphasised the potential role of ChatGPT in the medical and tourism industries. Authors from India explored the role of ChatGPT in the medical field (Seetharaman, 2023 ; Subramani et al., 2023 ). Seetharaman ( 2023 ) reports that ChatGPT offers supplementary language assistance to students who are not proficient in English, enabling them to enhance their language proficiency and effectively communicate in English, the principal language of instruction in medical establishments. The ChatGPT platform has the potential to serve as a tool for medical students to replicate patient interactions in a simulated environment, such as accurately obtaining medical histories and documenting symptoms. According to Subramani et al. ( 2023 ), ChatGPT is a highly efficient and user-friendly AI technology that can aid healthcare professionals in various aspects, such as diagnosis, critical decision-making, and devising appropriate treatment plans. ChatGPT has demonstrated impressive performance on medical exams, indicating its potential as a valuable resource for enhancing medical education and assessment (Subramani et al., 2023 ) and can support interdisciplinarity in tourism research (Nautiyal et al., 2023 ). Ivanov and Soliman ( 2023 ) note the potential of ChatGPT to serve as a digital instructor to provide students with enhanced and effective learning experiences and outcomes. Digital instructors can impart knowledge in diverse languages and thus can be used to educate individuals of varying nationalities and backgrounds in the field of tourism. Furthermore, LLM-based chatbots, including ChatGPT, can assess written assignments and provide direction on linguistic proficiency, syntax, and composition, ultimately enhancing students’ scholarly writing proficiency. In exploring the intersection of ChatGPT with medical education, institutions can pioneer innovative approaches by using the platform to create immersive, simulated patient interactions that go beyond language assistance, allowing medical students to practice nuanced skills such as medical history gathering and symptom documentation. Simultaneously, leveraging ChatGPT as a versatile digital instructor offers a unique opportunity to provide cross-cultural and multilingual education, contributing to a more inclusive and globally competent workforce within the tourism industry.

3.4 Challenges of ChatGPT in higher education

In addition to some previously mentioned challenges, such as the potential for plagiarism, the investigation also identified other key challenges in implementing ChatGPT within the context of higher education’s teaching and learning environment. Wu and Yu ( 2023 ) found that the benefits of AI-based ChatGPT are more in higher education as compared to primary and secondary education. The study also reported that the novelty effects of AI chatbots may enhance learning outcomes in brief interventions, but their efficacy diminishes in longer interventions.

First, the implementation of ChatGPT within the educational context engenders learning impediments. In the absence of adequate monitoring and regulation, the technology could lead to human unintelligence and unlearning, but teachers will become more adaptive and create authentic assessments to enhance student learning (Alafnan et al., 2023 ; Lawrie, 2023 ). Second, the technology could be used in a manner that violates students’ privacy. If the model is not adequately secured, it could surreptitiously gather confidential data from students without their explicit awareness or authorisation (Kanseci, 2023). Third, the technology could facilitate discrimination against particular students. If the model is not trained on a dataset that accurately represents the entire student population, it has the potential to create disparities in educational access (Cingillioglu, 2023 ; Lin et al., 2023 ). Fourth, according to Ivanov and Soloman (2023), ChatGPT lacks access to real-time data. Therefore, its responses may be inconsequential, inaccurate, or outdated. The information provided in response to a specific query may also be insufficient. Gao et al. (2022) highlight the need for further investigation of the precision and scholarly authenticity of ChatGPT. Fifth, it may be difficult for ChatGPT to comprehend the context and subtleties of complex academic subjects and answer complex questions (Adetayo, 2023 ; Eysenbach, 2023 ; Neumann et al., 2023 ). The system can misinterpret inquiries, offer inadequate or inaccurate responses, or struggle to comprehend the fundamental purpose behind questions (Clark, 2023 ). In particular, ChatGPT may not have the requisite expertise in highly specialised or advanced subjects such as advanced mathematics or specific sciences. Hence, it may not deliver precise and accurate answers (Neumann et al., 2023 ; Fergus et al., 2023 ). Karaali ( 2023 ) claimed that the primary emphasis in the field of AI is currently directed towards the enhancement of advanced cognitive abilities and mental processes associated with quantitative literacy and quantitative reasoning. However, it is important to acknowledge that fundamental skills such as writing, critical thinking, and numeracy continue to serve as essential foundational components among students. Although AI is making significant progress in fundamental domains, it appears that students are experiencing a decline in performance in the context of fundamental skills. Consequently, NLP-based adaptive learner support and education require further investigation (Bauer et al., 2023 ).

In addressing the challenges of ChatGPT in education, educators need to adapt and develop authentic assessments that mitigate the risk of human unlearning, ensuring that technology enhances, rather than hinders, student learning experiences. Simultaneously, recognising the limitations of ChatGPT in comprehending the nuances of highly specialised subjects underscores the importance of balancing advancements in AI’s cognitive abilities with continued emphasis on fundamental skills like critical thinking, writing, and numeracy, urging a reevaluation of priorities in AI-driven educational research towards comprehensive learner support.

4 Conclusion, implications and agenda for future research

This study identified the most influential articles and top journals and countries in terms of citations and publication productivity related to ChatGPT in higher education, as well as highlighted emerging thematic clusters and geographic clusters in research on the role and challenges of ChatGPT in teaching and learning in higher education institutions. Articles on the topic of ChatGPT in higher education published up to May 2023 were identified by searching the Scopus database. Given the emergent nature of ChatGPT starting in late 2022, all the included articles were published in 2023. Thus, this specific research domain remains relatively unexplored. The findings of this analysis reveal that the United States is the most productive country in terms of research on the role of ChatGPT in higher education, especially relating to academic integrity and research. US researchers also emerged as the most influential in terms of number of citations in the literature. Our findings corroborate those of previous research (Crompton & Burke, 2023 ). However, 60% of the articles in our shortlisted literature emanated from Asian countries.

Four thematic clusters (academic integrity, student engagement, learning environment and research) were identified. Furthermore, the country-based bibliographic analysis revealed that research has focused on student examinations, academic integrity, student learning and field-specific research in medical and tourism education (Nautiyal et al., 2023 ; Subramani et al., 2023 ). Plagiarism is recognised as a major challenge that hinders students’ creativity, innovativeness and originality when using ChatGPT in their academic pursuits. To mitigate the potential drawbacks of using ChatGPT in educational and research settings, proactive measures should be taken to educate students and researchers alike on the nature of plagiarism, its negative impacts and academic integrity (Shoufan, 2023 ; Teixeira, 2023 ) Educators may ask students to provide a written acknowledgement of the authenticity of their assignments and their non-reliance on ChatGPT. Such an acknowledgement would discourage students from utilising ChatGPT in their academic and research endeavours and establish accountability for their academic pursuits. In addition, educators should develop authentic assessments that are ChatGPT-proof.

ChatGPT lacks emotional intelligence and empathy, both of which are crucial in effectively addressing the emotional and psychological dimensions of the learning process (Farrokhnia et al., 2023 ; Neumann et al., 2023 ). Higher education institutions may encounter challenges in using ChatGPT to deliver suitable assistance, comprehension, or direction to students needing emotional or mental health support. The significance of human interaction in learning cannot be overstated. Achieving a balance between using AI and the advantages of human guidance and mentorship is a persistent challenge that requires attention (Neumann et al., 2023 ; Rahman et al., 2023 ). Strzelecki ( 2023 ) observed in his research that behavioural intention and personal innovativeness are the two major determinants behind the adoption of ChatGPT among students.

4.1 Implications

The findings of the present study have numerous important implications. This study provides insight into the current state of ChatGPT in higher education and thus can serve as valuable guidance for academics, practitioners, and policymakers. The study’s findings contribute to the literature by providing new insights into the role of ChatGPT and strategies for mitigating its negative aspects and emphasising its positive attributes.

First, the implementation of AI in education can improve academic performance and student motivation, particularly by facilitating personalised learning. Educational institutions should monitor and regulate students’ use of such technologies proactively. Higher education institutions also ought to prioritise the training of their educators in effectively utilising AI technologies, including ChatGPT. Concurrently, it is imperative for these institutions to equip students with comprehensive academic integrity training, shedding light on the appropriate and inappropriate applications of AI tools like ChatGPT. This includes creating awareness about the potential consequences of utilising these technologies for dishonest practices. Furthermore, educational establishments need to urgently revisit and refine their academic integrity policies to address the evolving landscape shaped by the integration of artificial intelligence tools in various academic facets. This proactive approach will foster a learning environment that embraces technological advancements and upholds the principles of honesty and responsible use. Institutional regulations on accountability and transparency should guide the frameworks that govern the use of AI in the campus environment (Pechenkina, 2023 ; Sun & Hoelscher, 2023 ; Dencik & Sanchez-Monedero, 2022 ).

Second, faculty members must proactively replace traditional coursework with modern alternatives that foster elevated levels of critical thinking among students, as suggested by Zhai ( 2022 ). Educators and learners can augment the academic material produced by ChatGPT with their own insights and information obtained from credible scholarly resources (Emenike & Emenike, 2023 ).

Third, ChatGPT should not be considered a threat to the education sector but a supplementary tool for human instruction that can enhance teaching and learning. It is imperative to acknowledge that the vital role of human educators cannot be replaced (Karaali, 2023 ) Moreover, ChatGPT can potentially enhance the accessibility and inclusivity of higher education. Alternative formats, linguistic support, and individualised explanations can help students who are studying English as a second language, are not native English speakers, or have other unique learning needs. Furthermore, Alnaqbi and Fouda ( 2023 ) highlight the implications of AI in evaluating the teaching style of faculty in higher education by collecting the feedback of students through social media and ChatGPT.

Fourth, the faculty in higher education institutions could address ethical concerns by providing students with explicit and comprehensive guidelines about the prescribed structure of academic assignments (Cotton et al., 2023 ; Gardner & Giordano, 2023 ). This practice can facilitate the production of more cohesive assignments. In addition, teachers can use rubrics to assess assignments and blend automated and manual assessment methodologies to evaluate students’ comprehension of the subject matter (Cotton et al., 2023 ; Shoufan, 2023 ).

In summary, using ChatGPT is recommended for enhancing creativity, refining writing proficiency, and improving research abilities. Nonetheless, it is crucial to emphasise that ChatGPT should not be employed as a substitute for critical thinking and producing original work. While it serves as a valuable tool for augmentation, upholding the integrity of independent thought and authentic content creation in academic endeavours is essential.

4.2 Limitations

The present study acknowledges several limitations. Firstly, the reliance on Scopus as the primary data source for bibliometric analysis may have limitations in capturing the full landscape of relevant literature. Future research may consider incorporating additional databases like Web of Science to ensure a comprehensive assessment. Secondly, due to the English language restriction in the review, potentially relevant studies may have been omitted. Future research could enhance inclusivity by extending its scope to encompass papers written in languages other than English. Thirdly, the current study exclusively focused on journal articles. Expanding the scope to include diverse sources, such as conference proceedings or book chapters, could offer a more comprehensive overview.

Additionally, as a rapidly evolving field, literature published after our inclusion dates need capturing, and future studies should consider adjusting their inclusion criteria to accommodate the dynamic nature of the subject matter. Lastly, the specificity of the bibliometric data search, centred around terms like ChatGPT, AI, higher education, and academic integrity, may have excluded certain relevant articles. Future studies should consider employing more generalised search parameters to encompass synonyms associated with these terms.

4.3 Future scope

The findings of the study suggest new avenues for future research. The effectiveness of evaluation criteria for assessments incorporating ChatGPT-generated text needs to be investigated. Specifically, the appropriate level of ChatGPT-produced text that students may use in academic tasks or assessments has not been established. Research on the ethical implications of using AI tools such as ChatGPT in higher education is also needed. Issues pertaining to data confidentiality, bias, and transparency in algorithms used for decision-making remain to be addressed. Feasible approaches for mitigating the excessive reliance of scholars and learners on ChatGPT or similar AI models are needed. Researchers could also explore the implementation of verification processes that go beyond traditional plagiarism detection methods, accounting for the unique challenges posed by AI systems. Future research in this domain could focus on establishing guidelines and best practices for the integration of AI tools like ChatGPT in academic settings, ensuring a balance between technological innovation and the preservation of academic rigour. Finally, the literature on ChatGPT in higher education has largely focused on the medical and tourism sectors. Future researchers must explore applications of ChatGPT in other disciplines.

Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author upon reasonable request.

Adetayo, A. J. (2023). ChatGPT and librarians for reference consultations. Internet Reference Services Quarterly , 27 (3), 131–147.

Article   Google Scholar  

AlAfnan, M. A., Dishari, S., Jovic, M., & Lomidze, K. (2023). ChatGPT as an educational tool: Opportunities, challenges, and recommendations for communication, business writing, and composition courses. Journal of Artificial Intelligence and Technology , 3 (2), 60–68. https://doi.org/10.37965/jait.2023.0184 .

Alnaqbi, N. M., & Fouda, W. (2023). Exploring the role of ChatGPT and social media in enhancing student evaluation of teaching style in higher education using Neutrosophic sets. International Journal of Neutrosophic Science , 20 (4), 181–190.

Angelis, L. D., Baglivo, F., Arzilli, G., Privitera, G. P., Ferragina, P., Tozzi, A. E., & Rizzo, C. (2023). ChatGPT and the rise of large language models: The new AI-driven infodemic threat in public health. Frontiers in Public Health , 11 , 1–8.

Archambault, E., Campbell, D., Gingras, Y., & Larivière, V. (2009). Comparing bibliometric statistics obtained from the web of Science and Scopus. Journal of the American Society for Information Science and Technology , 60 (7), 1320–1326.

Bajaj, Y., Kashiramka, S., & Singh, S. (2020). Application of capital structure theories: A systematic review. Journal of Advances in Management Research , 18 (2), 173–199. https://doi.org/10.1108/JAMR-01-2020-001 .

Bauer, E., Greisel, M., Kuznetsov, I., Berndt, M., Kollar, I., Dresel, M., Fischer, M. R., & Fischer, F. (2023). Using natural language processing to support peer-feedback in the age of artificial intelligence: A cross-disciplinary framework and a research agenda. British Journal of Educational Technology , 54 (5), 1222–1245.

Boyack, K. W., & Klavans, R. (2010). Co-citation analysis, bibliographic coupling, and direct citation: Which citation approach represents the research front most accurately? Journal of the American Society for Information Science and Technology , 61 (12), 2389–2404.

Cingillioglu, I. (2023). Detecting AI-generated essays: The ChatGPT challenge. International Journal of Information and Learning Technology , 40 (3), 259–268. https://doi.org/10.1108/IJILT-03-2023-0043 .

Clark, T. M. (2023). Investigating the Use of an Artificial Intelligence Chatbot with General Chemistry exam questions. Journal of Chemical Education , 100 (5), 1905–1916.

Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2023). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International , 1–13. https://doi.org/10.1080/14703297.2023.2190148 .

Crawford, J., Cowling, M., & Allen, K. (2023). Leadership is needed for ethical ChatGPT: Character, assessment, and learning using artificial intelligence (AI). Journal of University Teaching & Learning Practice , 20 (3), 1–21. https://doi.org/10.53761/1.20.3.02 .

Crompton, H., & Burke, D. (2023). Artificial intelligence and higher education: The state of the field. International Journal of Educational Technology in Higher Education , 20 , 1–22. https://doi.org/10.1186/s41239-023-00392-8 .

Dehouche, N. (2021). Plagiarism in the age of massive generative pre-trained transformers (GPT-3). Ethics in Science and Environmental Politics , 2 , 17–23. https://doi.org/10.3354/esep00195 .

Dencik, L., & Sanchez-Monedero, J. (2022). Data justice. Internet Policy Review , 11 (1), 1–16. https://doi.org/10.14763/2022.1.1615 .

Donthu, N., Kumar, S., & Pattnaik, D. (2020). Forty-five years of Journal of Business Research : A bibliometric analysis. Journal of Business Research , 109 , 1–14. https://doi.org/10.1016/j.jbusres.2019.10.039 .

Emenike, M., & Emenike, B. (2023). Was this title generated by ChatGPT? Considerations for artificial intelligence text-generation software programs for chemists and chemistry educators. Journal of Chemical Education , 100 (4), 1413–1418. https://doi.org/10.1021/acs.jchemed.3c00063 .

Eysenbach, G. (2023). The role of ChatGPT, generative language models, and artificial intelligence in medical education: A conversation with ChatGPT and a call for papers. JMIR Medical Education , 6 (9), 1–13. https://doi.org/10.2196/46885 .

Farrokhnia, M., Banihashem, K. S., Noroozi, O., & Wals, A. (2023). A SWOT analysis of ChatGPT: Implications for educational practice and research. Innovations in Education and Teaching International , 1–15. https://doi.org/10.1080/14703297.2023.2195846 .

Fergus, S., Botha, M., & Ostovar, M. (2023). Evaluating academic answers generated using ChatGPT. Journal of Chemical Education , 100 (4), 1672–1675. https://doi.org/10.1021/acs.jchemed.3c00087 .

Firat, M. (2023). How chat GPT can transform autodidactic experiences and open education. Department of Distance Education, Open Education Faculty , Anadolu University, 1–6. https://doi.org/10.31219/osf.io/9ge8m .

Gardner, D. E., & Giordano, A. E. (2023). The challenges and value of undergraduate oral exams in the physical chemistry classrooms: A useful tool in the assessment toolbox. Journal of Chemical Education , 100 (5), 1705–1709.

Geerling, W., Mateer, G. D., Wooten, J., & Damodaran, N. (2023). ChatGPT has aced the test of understanding in college economics: Now what? The American Economist , 68 (2), 233–245.

Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023). How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Medical Education , 9 , 1–9. https://doi.org/10.2196/45312 .

Gisev, N., Bell, J. S., & Chen, T. F. (2013). Interrater agreement and interrater reliability: Key concepts, approaches, and applications. Research in Social and Administrative Pharmacy , 9 (3), 330–338.

Gupta, R., Pandey, R., & Sebastian, V. J. (2021). International entrepreneurial orientation (IEO): A bibliometric overview of scholarly research. Journal of Business Research , 125 , 74–88. https://doi.org/10.1016/j.jbusres.2020.12.00 .

Ibrahim, H., Asim, R., & Zaffar, F. (2023). Rethinking homework in the age of artificial intelligence. Intelligent Systems , 38 (2), 24–27.

Google Scholar  

Iskender, A. (2023). Holy or Unholy? Interview with Open AI’s ChatGPT. European Journal of Tourism Research , 34 , 3414. https://doi.org/10.54055/ejtr.v34i.3169 .

Ivanov, S., & Soliman, M. (2023). Game of algorithms: ChatGPT implications for the future of tourism education and research. Journal of Tourism Futures , 9 (2), 214–221.

Jarneving, B. (2007). Bibliographic coupling and its application to research-front and other core documents. Journal of Informetrics , 1 (4), 287–307.

Jeon, J., Lee, S., & Cho, S. (2023). A systematic review of research on speech-recognition chatbots for language learning: Implications for future directions in the era of large language models. Interactive Learning Environment , 1–19. https://doi.org/10.1080/10494820.2023.2204343 .

Johinke, R., Cummings, R., & Di Lauro, F. (2023). Reclaiming the technology of higher education for teaching digital writing in a post-pandemic world. Journal of University Teaching and Learning Practice , 20 (2), 1–18. https://doi.org/10.53761/1.20.02.01 .

Karaali, G. (2023). Artificial Intelligence, Basic skills, and quantitative literacy. Numeracy , 16 (1), 1–22. https://doi.org/10.5038/1936-4660.16.1.1438 .

Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., & Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences , 103 (2023), 102274–102271. https://doi.org/10.1016/j.lindif.2023.102274 .

Lawrie, G. (2023). Establishing a delicate balance in the relationship between artificial intelligence and authenticate assessment in student learning. Chemistry Education Research and Practice , 24 (2), 392–393.

Lee, H. (2023). The rise of ChatGPT: Exploring its potential in medical education. Anatomical Science Education , 1–6. https://doi.org/10.1002/ase.2270 .

Li, Y., Sha, L., Yan, L., Lin, J., Raković, M., Galbraith, K., Lyons, K., Gašević, D., & Chen, G. (2023). Can large language models write reflectively. Computer and Education: Artificial Intelligence , 4 (2023), 100140–100141.

Liebrenz, M., Schleifer, R., Buadze, A., Bhugra, D., & Smith, A. (2023). Generating scholarly content with ChatGPT: Ethical challenges for medical publishing. Lancet Digital Health , 5 (3), e105–106. https://doi.org/10.1016/S2589-7500(23)00019-5 .

Lim, W. M., Gunasekara, A., Pallant, J. L., Pallant, J. I., & Pechenkina, E. (2023). Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. The International Journal of Management Education , 21 (2), 1–13.

Lin, C. C., Huang, A. Y. Q., Stephen, J. H., & Yang (2023). A review of AI-Driven conversational chatbots implementation methodologies and challenges (1999–2022). Sustainability , 15 (5), 4012. https://doi.org/10.3390/su15054012 .

Memarian, B., & Doleck, T. (2023). ChatGPT in education: Methods, potentials and limitations. Computers in Human Behavior: Artificial Humans , 100022 , 1–11.

Mishra, D., Gunasekaran, A., Papadopoulos, T., & Hazen, B. (2017). Green supply chain performance measures: A review and bibliometric analysis. Sustainable Production and Consumption , 10 , 85–99. https://doi.org/10.1016/j.spc.2017.01.003

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine , 151 (4), 264–269.

Nautiyal, R., Albrecht, J. N., & Nautiyal, A. (2023). ChatGPT and tourism academia. Annals of Tourism Research , 99 , 1–3.

Neumann, M., Rauschenberger, M., & Schon, E-M. (2023). 2023 IEEE/ACM 5th International Workshop on Software Engineering Education for the Next Generation (SEENG), Melbourne, Australia, 2023, pp. 29–32, https://doi.org/10.1109/SEENG59157.2023.00010 .

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., … & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ , 372 (71), 1–9. https://doi.org/10.1136/bmj.n71

Palomo, J., Figueroa-Domecq, C., & Laguna, P. (2017). Women, peace and security state-of-art: A bibliometric analysis in social sciences based on SCOPUS database. Scientometrics , 113 (1), 123–148. https://doi.org/10.1007/s11192-017-2484-x .

Patrício, L. D., & Ferreira, J. J. (2020). Blockchain security research: Theorizing through bibliographic-coupling analysis. Journal of Advances in Management Research , 18 (1), 1–35. https://doi.org/10.1108/JAMR-04-2020-0051 .

Paul, J., Lim, W. M., O’Cass, A., Hao, A. W., & Bresciani, S. (2021). Scientific procedures and rationales for systematic literature reviews (SPAR-4-SLR). International Journal of Consumer Studies . https://doi.org/10.1111/ijcs.12695 .

Pechenkina, K. (2023). Artificial intelligence for good? Challenges and possibilities of AI in higher education from a data justice perspective. In L. Czerniewicz, & C. Cronin (Eds.), Higher education for good: Teaching and learning futures (#HE4Good) . Open Book.

Perkins, M. (2023). Academic integrity considerations of AI large Language models in the post-pandemic era: ChatGPT and beyond. Journal of University Teaching & Learning Practice , 20 (2), 1–26. https://doi.org/10.53761/1.20.02.07 .

Rahman, M., Mostafizer, & Yutaka Watanobe. (2023). ChatGPT for Education and Research: Opportunities, threats, and strategies. Applied Sciences , 13 (9), 1–21. https://doi.org/10.3390/app13095783 .

Rana, S. (2023). AI and GPT for Management scholars and practitioners: Guidelines and implications. FIIB Business Review , 12 (1), 7–9.

Seetharaman, R. (2023). Revolutionising medical education: Can ChatGPT boost subjective learning and expression? Journal of Medical System , 47 (61), 1–4. https://doi.org/10.1007/s10916-023-01957-w .

Seth, I., Bulloch, G., & Lee, C. H. A. (2023). Redefining academic integrity, authorship, and innovation: The impact of ChatGPT on surgical research. Annals of Surgical Oncology , 30 , 5284–5285. https://doi.org/10.1245/s10434-023-13642-w .

Shoufan, A. (2023). Exploring students’ perceptions of ChatGPT: Thematic analysis and follow-up survey. IEEE Access , 11, 38805–38818, 2023, https://doi.org/10.1109/ACCESS.2023.3268224 .

Singh, V. K., Singh, P., Karmakar, M., Leta, J., & Mayr, P. (2021). The journal coverage of web of Science, Scopus and dimensions: A comparative analysis. Scientometrics , 126 , 5113–5142.

Skavronskaya, L., Hadinejad, A., & Cotterell, D. (2023). Reversing the threat of artificial intelligence to opportunity: A discussion of ChatGPT in tourism education. Journal of Teaching in Travel & Tourism , 23 , 1–6. https://doi.org/10.1080/15313220.2023.2196658 .

Strzelecki, A. (2023). To use or not to use ChatGPT in higher education? A study of students’ acceptance and use of technology. Interactive Learning Environments , 1–15. https://doi.org/10.1080/10494820.2023.2209881 .

Su, J., & Yang, W. (2023). Unlocking the power of ChatGPT: A framework for applying generative AI in education. ECNU Review of Education , 6 (3), 355–366.

Subramani, M., Jallel, I., & Mohan, S. K. (2023). Evaluating the performance of ChatGPT in medical physiology university examination of phase I MBBS. Advances in Physical Education , 47 , 270–271.

Sun, G. H., & Hoelscher, S. H. (2023). The ChatGPT Storm and what Faculty can do. Nurse Educator , 48 , 119–124.

Tang, G. (2023a). Academic journals cannot simply require authors to declare that they used ChatGPT. Irish Journal of Medical Science , 192 (6), 3195–3196.

Tang, G. (2023b). Letter to editor: Academic journals should clarify the proportion of NLP-generated content in papers. Accountability in Research , 1–3. https://doi.org/10.1080/08989621.2023.2180359 .

Teixeira da Silva, J. A. (2023). How are authors’ contributions verified in the ICMJE model? Plant Cell Reports , 42 , 1529–1153. https://doi.org/10.1007/s00299-023-03022-9 .

Thorp, H. H. (2023). ChatGPT is fun, but not an author. Science , 379 (6630), 313–313.

Tlili, A., Shehata, B., Adarkwah, M., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments , 10 (15), 1–24.

Van Eck, N., & Waltman, L. (2010). Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics , 84 (2), 523–538.

Vijaya, V., & Mathur, H. P. (2023). A decade of donation-based crowdfunding: A bibliometric analysis using the SCOPUS Database. Purushartha , 15 (2), 32–51.

Vogels, E. A. (2023). A majority of Americans have heard of ChatGPT, but few have tried it themselves, https://www.pewresearch.org/short-reads/2023/05/24/a-majority-of-americans-have-heard-of-chatgpt-but-few-have-tried-it-themselves/ .

Wu, R., & Yu, Z. (2023). Do AI Chatbots improve students learning outcome? Evidence from a meta analysis. British Journal of Educational Technology . https://doi.org/10.1111/bjet.13334 .

Wu, T. Y., He, S. Z., Liu, J. P., Sun, S. Q., Liu, K., Han, Q. L., & Tang, Y. (2023). A brief overview of ChatGPT: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica , 10 (5), 1122–1136. https://doi.org/10.1109/JAS.2023.123618 .

Yan, D. (2023). Impact of ChatGPT on learners in a L2 writing practicum: An exploratory investigation. Education and Information Technologies , 1–25. https://doi.org/10.1007/s10639-023-11742-4 .

Yeadon, W., Inyang, O-O., Mizouri, A., Peach, A., & Testrow, C. P. (May 2023). The death of the short-term physics essay in the coming AI revolution. Physics Education , 58 , 1–13.

Zhai, X. (2022). ChatGPT user experience: Implications for education. SSRN , 4312418 , 1–18.

Download references

Open Access funding enabled and organized by CAUL and its Member Institutions

Author information

Authors and affiliations.

University Business School, Maharaja Ranjit Singh Punjab Technical University, Punjab, India

Pritpal Singh Bhullar

Department of Financial Planning and Tax, School of Accounting, Information Systems and Supply Chain, RMIT University, Melbourne, Australia

Mahesh Joshi

School of Engineering and Technology, CML‑NET & CREATE Research Centres, Central Queensland University, Queensland, Australia

Ritesh Chugh

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Ritesh Chugh .

Ethics declarations

Conflict of interest.

No Conflict of Interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Bhullar, P.S., Joshi, M. & Chugh, R. ChatGPT in higher education - a synthesis of the literature and a future research agenda. Educ Inf Technol (2024). https://doi.org/10.1007/s10639-024-12723-x

Download citation

Received : 04 October 2023

Accepted : 16 April 2024

Published : 02 May 2024

DOI : https://doi.org/10.1007/s10639-024-12723-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial Intelligence
• Generative AI
• Higher education
• Academic integrity
• Systematic review
• Bibliometric analysis
• Find a journal
• Publish with us
• Track your research