literature review doctoral study

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Jan 4, 2024 10:52 AM
• URL: https://guides.library.yale.edu/YSNDoctoral

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Try for free

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved April 15, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, what is your plagiarism score.

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

• What Is a PhD Literature Review?
• Doing a PhD

A literature review is a critical analysis of published academic literature, mainly peer-reviewed papers and books, on a specific topic. This isn’t just a list of published studies but is a document summarising and critically appraising the main work by researchers in the field, the key findings, limitations and gaps identified in the knowledge.

• The aim of a literature review is to critically assess the literature in your chosen field of research and be able to present an overview of the current knowledge gained from previous work.
• By the conclusion of your literature review, you as a researcher should have identified the gaps in knowledge in your field; i.e. the unanswered research questions which your PhD project will help to answer.
• Quality not quantity is the approach to use when writing a literature review for a PhD but as a general rule of thumb, most are between 6,000 and 12,000 words.

What Is the Purpose of a Literature Review?

First, to be clear on what a PhD literature review is NOT: it is not a ‘paper by paper’ summary of what others have done in your field. All you’re doing here is listing out all the papers and book chapters you’ve found with some text joining things together. This is a common mistake made by PhD students early on in their research project. This is a sign of poor academic writing and if it’s not picked up by your supervisor, it’ll definitely be by your examiners.

The biggest issue your examiners will have here is that you won’t have demonstrated an application of critical thinking when examining existing knowledge from previous research. This is an important part of the research process as a PhD student. It’s needed to show where the gaps in knowledge were, and how then you were able to identify the novelty of each research question and subsequent work.

The five main outcomes from carrying out a good literature review should be:

• An understanding of what has been published in your subject area of research,
• An appreciation of the leading research groups and authors in your field and their key contributions to the research topic,
• Knowledge of the key theories in your field,
• Knowledge of the main research areas within your field of interest,
• A clear understanding of the research gap in knowledge that will help to motivate your PhD research questions .

When assessing the academic papers or books that you’ve come across, you must think about the strengths and weaknesses of them; what was novel about their work and what were the limitations? Are different sources of relevant literature coming to similar conclusions and complementing each other, or are you seeing different outcomes on the same topic by different researchers?

When Should I Write My Literature Review?

In the structure of your PhD thesis , your literature review is effectively your first main chapter. It’s at the start of your thesis and should, therefore, be a task you perform at the start of your research. After all, you need to have reviewed the literature to work out how your research can contribute novel findings to your area of research. Sometimes, however, in particular when you apply for a PhD project with a pre-defined research title and research questions, your supervisor may already know where the gaps in knowledge are.

You may be tempted to skip the literature review and dive straight into tackling the set questions (then completing the review at the end before thesis submission) but we strongly advise against this. Whilst your supervisor will be very familiar with the area, you as a doctoral student will not be and so it is essential that you gain this understanding before getting into the research.

How Long Should the Literature Review Be?

As your literature review will be one of your main thesis chapters, it needs to be a substantial body of work. It’s not a good strategy to have a thesis writing process here based on a specific word count, but know that most reviews are typically between 6,000 and 12,000 words. The length will depend on how much relevant material has previously been published in your field.

A point to remember though is that the review needs to be easy to read and avoid being filled with unnecessary information; in your search of selected literature, consider filtering out publications that don’t appear to add anything novel to the discussion – this might be useful in fields with hundreds of papers.

How Do I Write the Literature Review?

Before you start writing your literature review, you need to be clear on the topic you are researching.

1. Evaluating and Selecting the Publications

After completing your literature search and downloading all the papers you find, you may find that you have a lot of papers to read through ! You may find that you have so many papers that it’s unreasonable to read through all of them in their entirety, so you need to find a way to understand what they’re about and decide if they’re important quickly.

A good starting point is to read the abstract of the paper to gauge if it is useful and, as you do so, consider the following questions in your mind:

• What was the overarching aim of the paper?
• What was the methodology used by the authors?
• Was this an experimental study or was this more theoretical in its approach?
• What were the results and what did the authors conclude in their paper?
• How does the data presented in this paper relate to other publications within this field?
• Does it add new knowledge, does it raise more questions or does it confirm what is already known in your field? What is the key concept that the study described?
• What are the strengths and weaknesses of this study, and in particular, what are the limitations?

2. Identifying Themes

To put together the structure of your literature review you need to identify the common themes that emerge from the collective papers and books that you have read. Key things to think about are:

• Are there common methodologies different authors have used or have these changed over time?
• Do the research questions change over time or are the key question’s still unanswered?
• Is there general agreement between different research groups in the main results and outcomes, or do different authors provide differing points of view and different conclusions?
• What are the key papers in your field that have had the biggest impact on the research?
• Have different publications identified similar weaknesses or limitations or gaps in the knowledge that still need to be addressed?

Structuring and Writing Your Literature Review

There are several ways in which you can structure a literature review and this may depend on if, for example, your project is a science or non-science based PhD.

One approach may be to tell a story about how your research area has developed over time. You need to be careful here that you don’t just describe the different papers published in chronological order but that you discuss how different studies have motivated subsequent studies, how the knowledge has developed over time in your field, concluding with what is currently known, and what is currently not understood.

Alternatively, you may find from reading your papers that common themes emerge and it may be easier to develop your review around these, i.e. a thematic review. For example, if you are writing up about bridge design, you may structure the review around the themes of regulation, analysis, and sustainability.

As another approach, you might want to talk about the different research methodologies that have been used. You could then compare and contrast the results and ultimate conclusions that have been drawn from each.

As with all your chapters in your thesis, your literature review will be broken up into three key headings, with the basic structure being the introduction, the main body and conclusion. Within the main body, you will use several subheadings to separate out the topics depending on if you’re structuring it by the time period, the methods used or the common themes that have emerged.

The important thing to think about as you write your main body of text is to summarise the key takeaway messages from each research paper and how they come together to give one or more conclusions. Don’t just stop at summarising the papers though, instead continue on to give your analysis and your opinion on how these previous publications fit into the wider research field and where they have an impact. Emphasise the strengths of the studies you have evaluated also be clear on the limitations of previous work how these may have influenced the results and conclusions of the studies.

In your concluding paragraphs focus your discussion on how your critical evaluation of literature has helped you identify unanswered research questions and how you plan to address these in your PhD project. State the research problem you’re going to address and end with the overarching aim and key objectives of your work .

When writing at a graduate level, you have to take a critical approach when reading existing literature in your field to determine if and how it added value to existing knowledge. You may find that a large number of the papers on your reference list have the right academic context but are essentially saying the same thing. As a graduate student, you’ll need to take a methodological approach to work through this existing research to identify what is relevant literature and what is not.

You then need to go one step further to interpret and articulate the current state of what is known, based on existing theories, and where the research gaps are. It is these gaps in the literature that you will address in your own research project.

• Decide on a research area and an associated research question.
• Decide on the extent of your scope and start looking for literature.
• Review and evaluate the literature.
• Plan an outline for your literature review and start writing it.

Browse PhDs Now

Join thousands of students.

Join thousands of other students and stay up to date with the latest PhD programmes, funding opportunities and advice.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• CAREER FEATURE
• 04 December 2020
• Correction 09 December 2020

How to write a superb literature review

Andy Tay is a freelance writer based in Singapore.

You can also search for this author in PubMed   Google Scholar

Literature reviews are important resources for scientists. They provide historical context for a field while offering opinions on its future trajectory. Creating them can provide inspiration for one’s own research, as well as some practice in writing. But few scientists are trained in how to write a review — or in what constitutes an excellent one. Even picking the appropriate software to use can be an involved decision (see ‘Tools and techniques’). So Nature asked editors and working scientists with well-cited reviews for their tips.

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

doi: https://doi.org/10.1038/d41586-020-03422-x

Interviews have been edited for length and clarity.

Updates & Corrections

Correction 09 December 2020 : An earlier version of the tables in this article included some incorrect details about the programs Zotero, Endnote and Manubot. These have now been corrected.

Hsing, I.-M., Xu, Y. & Zhao, W. Electroanalysis 19 , 755–768 (2007).

Article   Google Scholar  

Ledesma, H. A. et al. Nature Nanotechnol. 14 , 645–657 (2019).

Article   PubMed   Google Scholar  

Brahlek, M., Koirala, N., Bansal, N. & Oh, S. Solid State Commun. 215–216 , 54–62 (2015).

Choi, Y. & Lee, S. Y. Nature Rev. Chem . https://doi.org/10.1038/s41570-020-00221-w (2020).

Download references

Related Articles

• Research management

Shrouded in secrecy: how science is harmed by the bullying and harassment rumour mill

Career Feature 16 APR 24

‘Shrugging off failure is hard’: the $400-million grant setback that shaped the Smithsonian lead scientist’s career

Career Column 15 APR 24

Citizenship privilege harms science

Comment 15 APR 24

How we landed job interviews for professorships straight out of our PhD programmes

Career Column 08 APR 24

How two PhD students overcame the odds to snag tenure-track jobs

Adopt universal standards for study adaptation to boost health, education and social-science research

Correspondence 02 APR 24

Structure peer review to make it more robust

World View 16 APR 24

Is ChatGPT corrupting peer review? Telltale words hint at AI use

News 10 APR 24

Rwanda 30 years on: understanding the horror of genocide

Editorial 09 APR 24

Associate or Senior Editor (Immunology), Nature Communications

The Editor in Immunology at Nature Communications will handle original research papers and work on all aspects of the editorial process.

London, Beijing or Shanghai - Hybrid working model

Springer Nature Ltd

Assistant Professor - Cell Physiology & Molecular Biophysics

Opportunity in the Department of Cell Physiology and Molecular Biophysics (CPMB) at Texas Tech University Health Sciences Center (TTUHSC)

Lubbock, Texas

Texas Tech University Health Sciences Center, School of Medicine

Postdoctoral Associate- Curing Brain Tumors

Houston, Texas (US)

Baylor College of Medicine (BCM)

Energy AI / Grid Modernization / Hydrogen Energy / Power Semiconductor Concentration / KENTECH College

21, Kentech-gil, Naju-si, Jeollanam-do, Republic of Korea(KR)

Korea Institute of Energy Technology

Professor in Macromolecular Chemistry

The Department of Chemistry - Ångström conducts research and education in Chemistry. The department has 260 employees and has a turnover of 290 mil...

Uppsala (Stad) (SE)

Uppsala University

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Undergraduate courses
• Postgraduate courses
• Foundation courses
• Apprenticeships
• Part-time and short courses
• Apply undergraduate
• Apply postgraduate

Search for a course

Search by course name, subject, and more

• Undergraduate
• Postgraduate
• (suspended) - Available in Clearing Not available in Clearing location-sign UCAS

Fees and funding

• Tuition fees
• Scholarships
• Funding your studies
• Student finance
• Cost of living support

Why study at Kent

Student life.

• Careers and employability
• Student support and wellbeing
• Our locations
• Placements and internships
• Year abroad
• Student stories
• Schools and colleges
• International
• International students
• Your country
• Applicant FAQs
• International scholarships
• University of Kent International College
• Campus Tours
• Applicant Events
• Postgraduate events
• Maps and directions
• Research strengths
• Research centres
• Research impact

Research institutes

• Durrell Institute of Conservation and Ecology
• Institute of Cyber Security for Society
• Institute of Cultural and Creative Industries
• Institute of Health, Social Care and Wellbeing

Research students

• Graduate and Researcher College
• Research degrees
• Find a supervisor
• How to apply

Popular searches

• Visits and Open Days
• Jobs and vacancies
• Accommodation
• Student guide
• Library and IT
• Partner with us
• Student Guide
• Student Help
• Health & wellbeing
• Student voice
• Living at Kent
• Careers & volunteering
• Diversity at Kent
• Finance & funding
• Life after graduation

Literature reviews

Writing a literature review.

The following guide has been created for you by the  Student Learning Advisory Service . For more detailed guidance and to speak to one of our advisers, please book an  appointment  or join one of our  workshops . Alternatively, have a look at our  SkillBuilder  skills videos.   

Preparing a literature review involves:

• Searching for reliable, accurate and up-to-date material on a topic or subject
• Reading and summarising the key points from this literature
• Synthesising these key ideas, theories and concepts into a summary of what is known
• Discussing and evaluating these ideas, theories and concepts
• Identifying particular areas of debate or controversy
• Preparing the ground for the application of these ideas to new research

Finding and choosing material

Ensure you are clear on what you are looking for. ask yourself:.

• What is the specific question, topic or focus of my assignment?
• What kind of material do I need (e.g. theory, policy, empirical data)?
• What type of literature is available (e.g. journals, books, government documents)?

What kind of literature is particularly authoritative in this academic discipline (e.g. psychology, sociology, pharmacy)?

How much do you need?

This will depend on the length of the dissertation, the nature of the subject, and the level of study (undergraduate, Masters, PhD). As a very rough rule of thumb – you may choose 8-10 significant pieces (books and/or articles) for an 8,000 word dissertation, up to 20 major pieces of work for 12-15,000 words, and so on. Bear in mind that if your dissertation is based mainly around an interaction with existing scholarship you will need a longer literature review than if it is there as a prelude to new empirical research. Use your judgement or ask your supervisor for guidance.

Where to find suitable material

Your literature review should include a balance between substantial academic books, journal articles and other scholarly publications. All these sources should be as up-to-date as possible, with the exception of ‘classic texts’ such as major works written by leading scholars setting out formative ideas and theories central to your subject. There are several ways to locate suitable material:

Module bibliography: for undergraduate dissertations, look first at the bibliography provided with the module documentation. Choose one or two likely looking books or articles and then scan through the bibliographies provided by these authors. Skim read some of this material looking for clues: can you use these leads to identify key theories and authors or track down other appropriate material?

Library catalogue search engine: enter a few key words to capture a range of items, but avoid over-generalisations; if you type in something as broad as ‘social theory’ you are likely to get several thousand results. Be more specific: for example, ‘Heidegger, existentialism’. Ideally, you should narrow the field to obtain just a few dozen results. Skim through these quickly to identity texts which are most likely to contribute to your study.

Library bookshelves: browse the library shelves in the relevant subject area and examine the books that catch your eye. Check the contents and index pages, or skim through the introductions (or abstracts, in the case of journal articles) to see if they contain relevant material, and replace them if not. Don’t be afraid to ask one of the subject librarians for further help. Your supervisor may also be able to point you in the direction of some of the important literature , but remember this is your literature search, not theirs.

Online: for recent journal articles you will almost certainly need to use one of the online search engines. These can be found on the ‘Indexing Services’ button on the Templeman Library website. Kent students based at Medway still need to use the Templeman pages to access online journals, although you can get to these pages through the Drill Hall Library catalogue. Take a look as well at the Subject Guides on both the Templeman and DHL websites.

Check that you have made the right selection by asking:

• Has my search been wide enough to ensure that I have identified all the relevant material, but narrow enough to exclude irrelevant material?
• Is there a good enough sample of literature for the level (PhD, Masters, undergraduate) of my dissertation or thesis?
• Have I considered as many alternative points of view as possible?
• Will the reader find my literature review relevant and useful?

Assessing the literature

Read the material you have chosen carefully, considering the following:

• The key point discussed by the author: is this clearly defined
• What evidence has the author produced to support this central idea?
• How convincing are the reasons given for the author’s point of view?
• Could the evidence be interpreted in other ways?
• What is the author's research method (e.g. qualitative, quantitative, experimental, etc.)?
• What is the author's theoretical framework (e.g. psychological, developmental, feminist)?
• What is the relationship assumed by the author between theory and practice?
• Has the author critically evaluated the other literature in the field?
• Does the author include literature opposing their point of view?
• Is the research data based on a reliable method and accurate information?
• Can you ‘deconstruct’ the argument – identify the gaps or jumps in the logic?
• What are the strengths and limitations of this study?
• What does this book or article contribute to the field or topic?
• What does this book or article contribute to my own topic or thesis?

As you note down the key content of each book or journal article (together with the reference details of each source) record your responses to these questions. You will then be able to summarise each piece of material from two perspectives:     

Content: a brief description of the content of the book or article. Remember, an author will often make just one key point; so, what is the point they are making, and how does it relate to your own research project or assignment?

Critical analysis: an assessment of the relative strengths and weaknesses of the evidence used, and the arguments presented. Has anything conveniently been left out or skated over? Is there a counter-argument, and has the author dealt with this adequately? Can the evidence presented be interpreted another way? Does the author demonstrate any obvious bias which could affect their reliability? Overall, based on the above analysis of the author’s work, how do you evaluate its contribution to the scholarly understanding and knowledge surrounding the topic?    

Structuring the literature review

In a PhD thesis, the literature review typically comprises one chapter (perhaps 8-10,000 words), for a Masters dissertation it may be around 2-3,000 words, and for an undergraduate dissertation it may be no more than 2,000 words. In each case the word count can vary depending on a range of factors and it is always best, if in doubt, to ask your supervisor.

The overall structure of the section or chapter should be like any other: it should have a beginning, middle and end. You will need to guide the reader through the literature review, outlining the strategy you have adopted for selecting the books or articles, presenting the topic theme for the review, then using most of the word limit to analyse the chosen books or articles thoroughly before pulling everything together briefly in the conclusion.

Some people prefer a less linear approach. Instead of simply working through a list of 8-20 items on your book review list, you might want to try a thematic approach, grouping key ideas, facts, concepts or approaches together and then bouncing the ideas off each other. This is a slightly more creative (and interesting) way of producing the review, but a little more risky as it is harder to establish coherence and logical sequencing.

Whichever approach you adopt, make sure everything flows smoothly – that one idea or book leads neatly to the next. Take your reader effortlessly through a sequence of thought that is clear, accurate, precise and interesting. 

Writing up your literature review

As with essays generally, only attempt to write up the literature review when you have completed all the reading and note-taking, and carefully planned its content and structure. Find an appropriate way of introducing the review, then guide the reader through the material clearly and directly, bearing in mind the following:

• Be selective in the number of points you draw out from each piece of literature; remember that one of your objectives is to demonstrate that you can use your judgement to identify what is central and what is secondary.
• Summarise and synthesise – use your own words to sum up what you think is important or controversial about the book or article.
• Never claim more than the evidence will support. Too many dissertations and theses are let down by sweeping generalisations. Be tentative and careful in the way you interpret the evidence.
• Keep your own voice – you are entitled to your own point of view provided it is based on evidence and clear argument.
• At the same time, aim to project an objective and tentative tone by using the 3rd person, (for example, ‘this tends to suggest’, ‘it could be argued’ and so on).
• Even with a literature review you should avoid using too many, or overlong, quotes. Summarise material in your own words as much as possible. Save the quotes for ‘punch-lines’ to drive a particular point home.
• Revise, revise, revise: refine and edit the draft as much as you can. Check for fluency, structure, evidence, criticality and referencing, and don’t forget the basics of good grammar, punctuation and spelling.

What’s Included: Literature Review Template

This template is structure is based on the tried and trusted best-practice format for formal academic research projects such as dissertations and theses. The literature review template includes the following sections:

• Before you start – essential groundwork to ensure you’re ready
• The introduction section
• The core/body section
• The conclusion /summary
• Extra free resources

Each section is explained in plain, straightforward language , followed by an overview of the key elements that you need to cover. We’ve also included practical examples and links to more free videos and guides to help you understand exactly what’s required in each section.

The cleanly-formatted Google Doc can be downloaded as a fully editable MS Word Document (DOCX format), so you can use it as-is or convert it to LaTeX.

PS – if you’d like a high-level template for the entire thesis, you can we’ve got that too .

FAQs: Literature Review Template

What format is the template (doc, pdf, ppt, etc.).

The literature review chapter template is provided as a Google Doc. You can download it in MS Word format or make a copy to your Google Drive. You’re also welcome to convert it to whatever format works best for you, such as LaTeX or PDF.

What types of literature reviews can this template be used for?

The template follows the standard format for academic literature reviews, which means it will be suitable for the vast majority of academic research projects (especially those within the sciences), whether they are qualitative or quantitative in terms of design.

Keep in mind that the exact requirements for the literature review chapter will vary between universities and degree programs. These are typically minor, but it’s always a good idea to double-check your university’s requirements before you finalize your structure.

Is this template for an undergrad, Master or PhD-level thesis?

This template can be used for a literature review at any level of study. Doctoral-level projects typically require the literature review to be more extensive/comprehensive, but the structure will typically remain the same.

Can I modify the template to suit my topic/area?

Absolutely. While the template provides a general structure, you should adapt it to fit the specific requirements and focus of your literature review.

What structural style does this literature review template use?

The template assumes a thematic structure (as opposed to a chronological or methodological structure), as this is the most common approach. However, this is only one dimension of the template, so it will still be useful if you are adopting a different structure.

Does this template include the Excel literature catalog?

No, that is a separate template, which you can download for free here . This template is for the write-up of the actual literature review chapter, whereas the catalog is for use during the literature sourcing and sorting phase.

How long should the literature review chapter be?

This depends on your university’s specific requirements, so it’s best to check with them. As a general ballpark, literature reviews for Masters-level projects are usually 2,000 – 3,000 words in length, while Doctoral-level projects can reach multiples of this.

Can I include literature that contradicts my hypothesis?

Yes, it’s important to acknowledge and discuss literature that presents different viewpoints or contradicts your hypothesis. So, don’t shy away from existing research that takes an opposing view to yours.

How do I avoid plagiarism in my literature review?

Always cite your sources correctly and paraphrase ideas in your own words while maintaining the original meaning. You can always check our plagiarism score before submitting your work to help ease your mind. 

Do you have an example of a populated template?

We provide a walkthrough of the template and review an example of a high-quality literature research chapter here .

Can I share this literature review template with my friends/colleagues?

Yes, you’re welcome to share this template in its original format (no editing allowed). If you want to post about it on your blog or social media, all we ask is that you reference this page as your source.

Do you have templates for the other dissertation/thesis chapters?

Yes, we do. You can find our full collection of templates here .

Can Grad Coach help me with my literature review?

Yes, you’re welcome to get in touch with us to discuss our private coaching services , where we can help you work through the literature review chapter (and any other chapters).

• Library Catalogue

Literature reviews for graduate students

On this page, what is a literature review, literature review type definitions, literature review protocols and guidelines, to google scholar, or not to google scholar, subject headings vs. keywords, keeping track of your research, project management software, citation management software, saved searches.

Related guides:

• Systematic, scoping, and rapid reviews: An overview
• Academic writing: what is a literature review , a guide that addresses the writing and composition aspect of a literature review
• Media literature reviews: how to conduct a literature review using news sources
• Literature reviews in the applied sciences
• Start your research here , literature review searching, mainly of interest to newer researchers

For more assistance, please contact the Liaison Librarian in your subject area .

Most generally, a literature review is a search within a defined range of information source types, such as, for instance, journals and books, to discover what has been already written about a specific subject or topic.  A literature review is a key component of almost all research papers.  However, the term is often applied loosely to describe a wide range of methodological approaches. A literature review in a first or second year course may involve browsing the library databases to get a sense of the research landscape in your topic and including 3-4 journal articles in your paper. At the other end of the continuum, the review may involve completing a comprehensive search, complete with documented search strategies and a listing of article inclusion and exclusion criteria. In the most rigorous format - a Systematic Review - a team of researchers may compile and review over 100,000 journal articles in a project spanning one to two years! These are out of scope for most graduate students, but it is important to be aware of the range of types of reviews possible.

One of the first steps in conducting a lit review is thus to clarify what kind of review you are doing, and its associated expectations.

Factors determining review approach are varied, including departmental/discipline conventions, granting agency stipulations, evolving standards for evidence-based research (and the corollary need for documented, replicable search strategies), and available time and resources.

The standards are also continually evolving in light of changing technology and evidence-based research about literature review methodology effectiveness. The availability of new tools such as large-scale library search engines and sophisticated citation management software continues to influence the research process.

Some specific types of lit reviews types include systematic reviews , scoping reviews , realist reviews , narrative reviews , mapping reviews, and qualitative systematic reviews , just to name a few. The protocols and distinctions for review types are particularly delineated in health research fields, but we are seeing conventions quickly establishing themselves in other academic fields.

The below definitions are quoted from the very helpful book, Booth, A., Papaioannou, D., & Sutton, A. (2012). Systematic approaches to a successful literature review . London: SAGE Publications Ltd.

For more definitions, try:

• Grant, M.J. & Booth, A. (2009). A typology of reviews: an analysis of the 14 review types and associated methodologies. Health Information & Libraries Journal , 26(2), 91-108. doi: 10.1111/j.1471-1842.2009.00848.x
• Sage Research Methods Online. A database devoted to research methodology. Includes handbooks, encyclopedia entries, and a research concepts map.
• Research Methods
• Report Writing
• Research--Methodology
• Research--Methodology--Handbooks, manuals, etc.

Note:   There is unfortunately no subject heading specifically for "literature reviews" which brings together all related material.

Mapping Review : "A rapid search of the literature aiming to give a broad overview of the characteristics of a topic area. Mapping of existing research, identification of gaps, and a summary assessment of the quantity and quality of the available evidence helps to decide future areas for research or for systematic reviews." (Booth, Papaioannou & Sutton, 2012, p. 264)

Mixed Method Review : "A literature review that seeks to bring together data from quantitative and qualitative studies integrating them in a way that facilitates subsequent analysis" (Booth et al., p. 265).

Meta-analysis : "The process of combining statistically quantitative studies that have measured the same effect using similar methods and a common outcome measure" (Booth et al., p. 264).

Narrative Review: "A term used to describe a conventional overview of the literature, particularly when contrasted with a systematic review" (Booth et al., p. 265).

Note: this term is often used pejoratively, describing a review that is inadvertently guided by a confirmation bias.

Qualitative Evidence Synthesis : "An umbrella term increasingly used to describe a group of review types that attempt to synthesize and analyze findings from primary qualitative research studies" (Booth et al., p. 267).

Rapid Review : "Assessment of what is already known about a policy or practice issue, by using systematic review methods to search and critically appraise existing research" (Grant & Booth, 2009, p.96).

Note: Rapid reviews are often done when there are insufficient time and/or resources to conduct a systematic review. As stated by Butler et. al, "They aim to be rigorous and explicit in method and thus systematic but make concessions to the breadth or depth of the process by limiting particular aspects of the systematic review process" (as cited in Grant & Booth, 2009, p. 100). 

Scoping Review: "A type of review that has as its primary objective the identification of the size and quality of research in a topic area in order to inform subsequent review" (Booth et al., p. 269).

Systematic Review : "A review of a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research and to collect and analyse data from the studies that are included in the review" (Booth et al., p. 271).

Note : a systematic review (SR) is the most extensive and well-documented type of lit review, as well as potentially the most time-consuming. The idea with SRs  is that the search process becomes a replicable scientific study in itself. This level of review will possibly not be necessary (or desirable) for your research project.

Many lit review types are based on organization-driven specific protocols for conducting the reviews. These protocols provide specific frameworks, checklists, and other guidance to the generic literature review sub-types. Here are a few popular examples:

Cochrane Review - known as the "gold standard" of systematic reviews, designed by the Cochrane Collaboration. Primarily used in health research literature reviews.

• Cochrane Handbook for Systematic Reviews of Interventions . "The official document that describes in detail the process of preparing and maintaining Cochrane systematic reviews".

Campbell Review - the sister organization of the Cochrane Institute which focuses on systematic reviews in the social sciences.

• So you want to write a Campbell Systematic review?
• Campbell Information Retrieval Guide. The details of effective information searching

Literature Reviews in Psychology

A recent article in the  Annual Review of Psychology  provides a very helpful guide to conducting literature reviews specifically in the field of Psychology.

How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses. (2019). Annual Review of Psychology, 70 (1), 747-770. doi: 10.1146/annurev-psych-010418-102803

Rapid Reviews have become increasingly common due to their flexibility, as well as the lack of time and resources available to do a comprehensive systematic review. McMaster University's National Collaborating Centre for Methods and Tools (NCCMT) has created a  Rapid Review Guidebook , which "details each step in the rapid review process, with notes on how to tailor the process given resource limitations."  

Scoping Review

There is no strict protocol for a scoping review (unlike Campbell and Cochrane reviews). The following are some recommended guidelines for scoping reviews:

• Scoping Reviews  from the JBI Manual for Evidence Synthesis
• Current best practices for the conduct of scoping reviews, from the EQUATOR Network

In addition to protocols which provide holistic guidance for conducting specific kinds of reviews, there are also a vast number of frameworks, checklists, and other tools available to help focus your review and ensure comprehensiveness. Some provide broader-level guidance; others are targeted to specific parts of your reviews such as data extraction or reporting out results.

• PICO or PICOC A framework for posing a researchable question (population, intervention, comparisons, outcomes, context/environment)
• PRISMA Minimum items to report upon in a systematic review, as well as its extensions , such as  PRISMA-ScR (for scoping reviews)
• SALSA framework: frames the literature review into four parts: search (S), appraisal(AL), synthesis(S), analysis(A)
• STARLITE Minimum requirements for reporting out on literature reviews.
• Critical Appraisal Skills Program (CASP) Checklists Includes a checklist for evaluating Systematic Reviews.

These are just a sampling of specific guides generated from the ever-growing literature review industry.

Much of the online discussion about the use of Google Scholar in literature reviews seems to focus more on values and ideals, rather than a technical assessment of the search engine's role. Here are some things to keep in mind.

• It's good practice to use both Google Scholar and subject-specific databases (example: PsycINFO) for conducting a lit review of any type. For most graduate-level literature reviews, it is usually recommended to use both.
• You should search Google Scholar through the library's website when off-campus. This way you can avoid being prompted for payment to access articles that the SFU Library already subscribes to.
• Search tips for Google and Google Scholar

Google Advantages:

• Allows you to cast a wide net in your search.
• The most popular articles are revealed
• A high volume of articles are retrieved
• Google's algorithm helps compensate for poorly designed searches
• Full-text indexing of articles is now being done in Google Scholar
• A search feature allow you to search within articles citing your key article
• Excellent for known-item searching or locating a quote/citation
• Helpful when searching for very unique terminology (e.g., places and people)
• Times cited tool can help identify relevant articles
• Extensive searching of non-article, but academic, information items: universities' institutional repositories, US case law, grey literature , academic websites, etc.

Disadvantages:

• The database is not mapped to a specific discipline
• Much less search sophistication and manipulation supported
• Psuedo-Boolean operators
• Missing deep data (e.g., statistics)
• Mysterious algorithms and unknown source coverage at odds with the systematic and transparent requirement of a literature review.
• Searches are optimized (for example, by your location), thwarting the replicability criteria of most literature review types
• Low level of subject and author collocation - that is, bringing together all works by one author or one sub-topic
• Challenging to run searches that involve common words. A search for "art AND time", for example, might bring up results on the art of time management when you are looking for the representation of time in art. In contrast, searching by topic is readily facilitated by use of subject headings in discipline-specific databases. Google Scholar has no subject headings.
• New articles might not be pushed up if the popularity of an article is prioritized
• Indexes articles from predatory publishers , which may be hard to identify if working outside of your field

Unlike Google Scholar, subject specific databases such as  PsycINFO , Medline , or Criminal Justice Abstracts are mapped to a disciplinary perspective. Article citations contain high-quality and detailed metadata. Metadata can be used to build specific searches and apply search limits relevant to your subject area. These databases also often offer access to specialized material in your area such as grey literature , psychological tests, statistics, books and dissertations.

For most graduate-level literature reviews, it is usually recommended to use both. Build careful searches in the subject/academic databases, and check Google Scholar as well.

For most graduate-level lit reviews, you will want to make use of the subject headings (aka descriptors) found in the various databases.

Subject headings are words or phrases assigned to articles, books, and other info items that describe the subject of their content. They are designed to succinctly capture a document's concepts, allowing the researcher to retrieve all articles/info items about that concept using one term. By identifying the subject headings associated with your research areas, and subsequently searching the database for other articles and materials assigned with that same subject heading, you are taking a significant measure to ensure the comprehensiveness of your literature review.

About subject headings:

• They are applied systematically : articles and books will usually have about 3-8 subject headings assigned to their bibliographic record.
• The subject headings come from a finite pool of terms -  one that is updated frequently.
• They are often organized in a hierarchical taxonomy , with subject headings belonging to broader headings, and/or having narrower headings beneath them. Sometimes there are related terms (lateral) as well.
• They provide a standardized way to describe a concept. For instance, a subject heading of "physician" may be used to capture many of the natural language words that describe a physician such as doctor, family doctor, GP, and MD.

One way to identify subject headings (SHs) of interest to you is to start with a keyword search in a database, and see which SHs are associated with the articles of interest.

A. In the below example, we start with a keyword search for "type a" personality in PsycINFO .  A more contemporary term to describe this phenomena is then found in the subject heading field:

B. Another way to identify subject headings related to your topic is to go directly to a database's thesaurus or index. For example, if we are researching depression, the PsycINFO entry for major depression suggests some narrower terms we could focus our search by.

For more in-depth help with using subject headings in a literature review, please contact the Liaison Librarian in your subject area .

• NEW! Covidence . Covidence is a web-based literature review tool that will help you through the process of screening your references, data extraction, and keeping track of your work. Ideal for streamlining systematic reviews, scoping reviews, meta-analyses, and other related methods of evidence synthesis.
• NVivo is a robust software package that helps with management and analysis of qualitative information.The Library's Research Commons offers extensive support for NVivo.
• Research Support Software offered by the Research Commons

Citation management software such as Zotero, Mendeley, or Endnote is essential for completing a substantial lit review. Citation software is a centralized, online location for managing your sources. Specifically, it allows you to:

• Access and manage your sources online, all in one place
• Import references from library databases and websites
• Automatically generate bibliographies and in-text citations within Microsoft Word
• Share your collection of sources with others, and work collaboratively with references
• De-duplicate your search results* (*Note: Mendeley is not recommended for deduplication in systematic reviews.)
• Annotate your citations. Some software allows you to mark up PDFs.
• Note trends in your research such as which journals or authors you cite from the most.

More information on Citation Management Software

Did you know that many databases allow you to save  your search strategies? The advantages of saving and tracking your search strategies online in a literature review include:

• Developing your search strategy in a methodological manner, section by section. For instance, you can run searches for all synonyms and subjects headings associated with one concept, then combine them with different concepts in various combinations.
• Re-running your well-executed search in the future
• Creating search alerts based on a well-designed search, allowing you to stay notified of new research in your area
• Tracking and remember all of the searches you have done. Avoid inadvertently re-doing your searches by being well-documented and systematic as you go along - it's worth the extra effort!

Databases housed on the EBSCO plaform (examples: Business Source Complete, PsycINFO, Medline, Academic Search Premier) allow you to create an free account where you might save your searches:

• Using the EBSCOhost Search History - Tutorial [2:08]
• Creating a Search Alert in EBSCOhost - Tutorial [1:26]

• UWF Libraries

Literature Review: Conducting & Writing

• Sample Literature Reviews
• Steps for Conducting a Lit Review
• Finding "The Literature"
• Organizing/Writing
• APA Style This link opens in a new window
• Chicago: Notes Bibliography This link opens in a new window
• MLA Style This link opens in a new window

Sample Lit Reviews from Communication Arts

Have an exemplary literature review.

• Literature Review Sample 1
• Literature Review Sample 2
• Literature Review Sample 3

Have you written a stellar literature review you care to share for teaching purposes?

Are you an instructor who has received an exemplary literature review and have permission from the student to post?

Please contact Britt McGowan at [email protected] for inclusion in this guide. All disciplines welcome and encouraged.

• << Previous: MLA Style
• Next: Get Help! >>
• Last Updated: Mar 22, 2024 9:37 AM
• URL: https://libguides.uwf.edu/litreview

Wrestling an elephant into a cupboard: how to write a PhD literature review in nine easy steps

Feb 10, 2019

When I was writing my PhD I hated the literature review. I was scared of it. One day, my supervisor took me to one side and told me that I had no choice: ‘It was going to have to be done before you start fieldwork’. I was terrified.

Sound familiar? According to Google, 5,000 people a month search for advice on how to conduct a literature review. And we know from the one-on-one PhD coaching we offer and from the theses we proofread that many students struggle with this part of their thesis. 

If you’re feeling lost, keep reading. In this guide, I’ll walk you through the nine steps involved in conducting and writing a PhD literature review.

You’ll realise what I eventually found out: C onducting a literature review is easy. Okay, perhaps that’s a bit much. Let me rephrase: Conducting a PhD literature review isn’t as hard as you think.

What a PhD literature review isn’t

Let us make one thing very clear. A PhD literature review isn’t just a summary of existing literature. That’s an annotated bibliography and that isn’t what a PhD literature review is about. This is the mistake I see most frequently in the PhDs I proofread.

Not only will your examiners send this back for corrections, but it may mean the whole PhD thesis is problematic because it isn’t grounded in a critical review of the literature.

What a PhD literature review is

A PhD literature review is a critical assessment of the literature in your field and related to your specific research topic. When discussing each relevant piece of literature, the review must highlight where the gaps are and what the strengths and weaknesses are of particular studies, papers, books, etc. Also, different pieces of literature are compared and contrasted with one another so that themes and relationships are highlighted.

The job of a literature review is to show five things (if you’re using our PhD Writing Template , you may recognise these):

1. What has been written on your topic 2. Who the key authors are and what the key works are 3. The main theories and hypotheses 4. The main themes that exist in the literature 5. Gaps and weaknesses that your study will then help fill

Who cares what other people have written and said, or what they haven’t said? Well, you should and your examiners definitely will. For your own study to make sense, it has to be situated in the literature. That means you must relate it to what others are talking about.

If you wanted to build a new mobile phone, you would have to research how other mobile phones are built, find out where they can be improved and then design one that makes those improvements.

The literature review is the same.

But where do I start? Here, we list nine steps. Follow each and you’ll be on your way to literature review greatness.

We’ve made the infographic below to help you on your way. Click the image to download it.

Step One: Pick a Broad Topic

You will be reviewing literature on a particular topic, so knowing what your topic is beforehand means you can narrow down your search. At this stage your topic is broad. You won’t be able to know the specifics until you do the review itself.

For my PhD, which looked at the contributions that local government made to climate change policy, my literature review started with a broad topic of ‘climate change policy’. I didn’t focus in on local government until I had read the literature on climate change policy and realized there was a gap.

So, having a clearly defined purpose is really important. Otherwise you are searching blind. If you refer to your PhD Writing Template, take a look at the box titled ‘Aims & Objectives’ – you’ll need to make sure you have established your aims, scope and research questions.

Step Two: Find the Way In

If you search for your broad topic in Google Scholar, you’ll be presented with millions of results. With my own PhD, a search for ‘climate change policy’ bought up over 3 million results.

  Obviously it’s unfeasible to read through all these.

So where do you start? Easy: choose the biggest names in your field.

There are three ways to find these:

1. Textbooks 2. Review articles 3. Most-cited articles

Read through these seminal texts and you’ll begin to get an idea of the broad topic.

Step Three: Who’s Saying What & When

Your job at this stage is to find out the key debates in the field. 

• Who is making the most significant contribution?
• What are they saying?
• How are they saying it?
• What aren’t they saying?

Step Four: Notes, Notes, Notes.

Whenever you read anything you should be taking notes. Detailed notes. These need to cover the following points: 

• What is the author saying?
• How is it relevant to your research?
• What are the gaps/weaknesses?
• What are the key references that you should read?

The more of these kind of standardised notes you have, the easier it will be when you write your literature review.

Step Five: Narrow Down the Field

As you read the key texts, you will begin to see what the key debates are in your field. There might be a number of ’schools’, for example. When you become aware of them, start to focus your literature review around them.

Step Six: Filter Through Your Growing List of References

Don’t just read everything. You need to find a way to filter through the articles or books that are relevant. For example, scan the abstracts, introduction, keywords, titles and references.

Filter the sources you come across into three separate categories:

• Probably won’t read

Step Seven: Use Snowball Sampling

As you read through these articles, look at their reference list. Collect articles that you think will be relevant and use them in your literature review. This is known as snowball sampling.

Step Eight: Think About the Questions that Haven’t Been Asked

You must be reading critically, which means asking what the weaknesses are and where particular articles or book could be improved.

In order to tease out your own specific research topic, you need to think of the questions that haven’t been asked.

Master your lit review & theory framework.

Learn what goes where (and why), and how it all fit together with this free, interactive guide to the PhD literature review and theory framework.

Step Nine: Writing Up Your Literature Review

  The review will broadly follow the key debates you have spotted in step five above. As you write, focus on putting in more detail about particular sources (i.e. flesh out steps six and seven). The focus when writing is to elaborate upon the key patterns and themes that have emerged.

However, you need to include your own synthesis of the material. I said earlier that you shouldn’t just summarize the literature. Instead you should write critically. You should clearly and precisely present your argument. The argument will focus around the questions that haven’t been asked – step nine above – and will ground the literature review. We’ve written a guide to being critical in your literature review . You should read it if you’re unsure what’s required.

So, write early and write that first draft quickly. The earlier you start writing your literature review the better. You must accept that your first draft is going to be just that: a draft. When you write the first draft, focus on the broad structure first. This means focus on the broad themes you want to discuss in the review.

Something you need to consider is how to structure the chapter. The simple answer is that you can either structure it chronologically or thematically.

The long answer is that chronological literature reviews are restrictive and over-simplify the field. They are useful for very early drafts of the review and can help you to arrange the literature and trace threads and connections within it. However, your supervisors and examiners are looking for thematic reviews (unless they have told you otherwise), where you discuss the literature with reference to the themes that have emerged.

Equally important is knowing when to stop reviewing the literature.

The sooner you go out and do your fieldwork, the better. The literature review is a cruel mistress; you’ll struggle to fully nail down its various components and fully understand how everything you have read is related. But don’t despair; aspects of the literature review will become clearer when you enter the field and start to collect data.

Don’t fall into the trap of spending too long in the library and too little time doing fieldwork.

  It’s natural to be scared of the literature review. To conduct one, you have to read, process and synthesise hundreds of thousands of words. But it’s not impossible. Keep this guide to hand and refer to it when you feel yourself getting lost. Share it with your colleagues so they too can conquer their fear of the literature review.

Now read our guide to being critical in the literature review and, if you haven’t already, download our PhD writing template .

And if you need a little extra support, check out our one-on-one PhD coaching . It’s like having a personal trainer, but for your PhD. 

Hello, Doctor…

Sounds good, doesn’t it?  Be able to call yourself Doctor sooner with our five-star rated How to Write A PhD email-course. Learn everything your supervisor should have taught you about planning and completing a PhD.

Now half price. Join hundreds of other students and become a better thesis writer, or your money back. 

Share this:

24 comments.

Good. Clear guidance

I have read the guidelines and noted numerous tricks of writing a thesis. My understanding of writing literature review has improved a lot. Thanks a lot

You’re welcome 🙂

Another Great piece.

It’s understandable and clearer now. I do appreciate you. Thanks so much

Many thanks for the kind words.

You are just brilliant. Outstanding piece for the literature review.

You’re too kind. Thanks!

Great insights! Thanks indeed.

Thank you very much for your clear advice. I am beginning to see where my early literature review drafts were lacking and my feelings of panic are reducing!

is the process the same a research paper?

Broadly speaking, yes. It’ll follow the same overall structure, but you won’t be going into as much detail.

This is very helpful. Thank you so much for sharing

Great advice. Thank you!

You’re welcome!

Thank you for this! I’m a first-year Ph.D. candidate, and I’m super nervous about writing my first literature review. I’ll be sure to use this for some more insight!

Thanks for the kind words. You’re welcome to join us on a PhD Masterclass. We’re currently putting together the Spring 24 calendar and we always run literature review sessions. You can bookmark this page to be the first to hear when our new programme is ready for bookings: https://www.thephdproofreaders.com/phd-workshops/

I cannot tell you how much more concise this makes everything for my ADHD brain. Thank you!

I’m so glad. Thanks for the kind words Kimberly.

I’m staring down the barrel of my literature review and this article made it much clearer what I’m trying to accomplish and actually feel more doable. Thank you!

You’re welcome. I’m glad it helped. Best of luck with it. If you need any support you can get me at max[at]thephdproofreaders.com

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Search The PhD Knowledge Base

Most popular articles from the phd knowlege base.

The PhD Knowledge Base Categories

• Your PhD and Covid
• Mastering your theory and literature review chapters
• How to structure and write every chapter of the PhD
• How to stay motivated and productive
• Techniques to improve your writing and fluency
• Advice on maintaining good mental health
• Resources designed for non-native English speakers
• PhD Writing Template
• Explore our back-catalogue of motivational advice

• Library databases
• Library website

Library Guide to Capstone Literature Reviews: Find a Research Gap

Find a research gap: tips to get started.

Finding a research gap is not an easy process and there is no one linear path. These tips and suggestions are just examples of possible ways to begin. 

In Ph.D. dissertations, students identify a gap in research. In other programs, students identify a gap in practice. The literature review for a gap in practice will show the context of the problem and the current state of the research. 

Research gap definition

A research gap exists when:

• a question or problem has not been answered by existing studies/research in the field 
• a concept or new idea has not been studied at all
• all the existing literature on a topic is outdated 
• a specific population/location/age group etc has not been studied 

A research gap should be:

• grounded in the literature
• amenable to scientific study
• Litmus Test for a Doctoral-Level Research Problem (Word) This tool helps students determine if they have identified a doctoral level research problem.

Identify a research gap

To find a gap you must become very familiar with a particular field of study. This will involve a lot of research and reading, because a gap is defined by what does (and does not) surround it.

• Search the research literature and dissertations (search all university dissertations, not just Walden!).
• Understand your topic! Review background information in books and encyclopedias . 
• Look for literature reviews, systematic reviews, and meta-analyses.
• Take notes on concepts, themes, and subject terms . 
• Look closely at each article's limitations, conclusions, and recommendations for future research. 
• Organize, analyze, and repeat! 

• Quick Answer: How do I find dissertations on a topic?

Start with broad searches

Use the Library Search (formerly Thoreau)  to do a broad search with just one concept at a time . Broad searches give you an idea of the academic conversation surrounding your topic.

• Try the terms you know (keywords) first.
• Look at the Subject Terms (controlled language) to brainstorm terms. 
• Subject terms help you understand what terms are most used, and what other terms to try.
• No matter what your topic is, not every researcher will be using the same terms. Keep an eye open for additional ways to describe your topic.
• Guide: Subject Terms & Index Searches: Index Overview

Keep a list of terms

• Create a list of terms
• Example list of terms

This list will be a record of what terms are: 

• related to or represent your topic
• synonyms or antonyms
• more or less commonly used
• keywords (natural language) or subject terms (controlled language)
• Synonyms & antonyms (database search skills)
• Turn keywords into subject terms

Term I started with:

culturally aware 

Subject terms I discovered:

cultural awareness (SU) 

cultural sensitivity (SU) 

cultural competence (SU) 

Search with different combinations of terms

• Combine search terms list
• Combine search terms table
• Video: Search by Themes

Since a research gap is defined by the absence of research on a topic, you will search for articles on everything that relates to your topic. 

• List out all the themes related to your gap.
• Search different combinations of the themes as you discover them (include search by theme video at bottom) 

For example, suppose your research gap is on the work-life balance of tenured and tenure-track women in engineering professions. In that case, you might try searching different combinations of concepts, such as: 

• women and STEM 
• STEM or science or technology or engineering or mathematics
• female engineering professors 
• tenure-track women in STEM
• work-life balance and women in STEM
• work-life balance and women professors
• work-life balance and tenure 

Topic adapted from one of the award winning Walden dissertations. 

• Walden University Award Winning Dissertations
• Gossage, Lily Giang-Tien, "Work-Life Balance of Tenured and Tenure-Track Women Engineering Professors" (2019). Walden Dissertations and Doctoral Studies. 6435.

Break your topic into themes and try combining the terms from different themes in different ways. For example: 

Theme 1 and Theme 4

Theme 2 and Theme 1

Theme 3 and Theme 4

Video: Search by Themes (YouTube)

(2 min 40 sec) Recorded April 2014 Transcript

Track where more research is needed

Most research articles will identify where more research is needed. To identify research trends, use the literature review matrix to track where further research is needed. 

• Download or create your own Literature Review Matrix (examples in links below).
• Do some general database searches on broad topics.
• Find an article that looks interesting.
• When you read the article, pay attention to the conclusions and limitations sections.
• Use the Literature Review Matrix to track where  'more research is needed' or 'further research needed'. NOTE:  you might need to add a column to the template.
• As you fill in the matrix you should see trends where more research is needed.

There is no consistent section in research articles where the authors identify where more research is needed. Pay attention to these sections: 

• limitations
• conclusions
• recommendations for future research 
• Literature Review Matrix Templates: learn how to keep a record of what you have read
• Literature Review Matrix (Excel) with color coding Sample template for organizing and synthesizing your research
• Previous Page: Scope
• Next Page: Get & Stay Organized
• Office of Student Disability Services

Walden Resources

Departments.

• Academic Residencies
• Academic Skills
• Career Planning and Development
• Customer Care Team
• Field Experience
• Military Services
• Student Success Advising
• Writing Skills

Centers and Offices

• Center for Social Change
• Office of Academic Support and Instructional Services
• Office of Degree Acceleration
• Office of Research and Doctoral Services
• Office of Student Affairs

Student Resources

• Doctoral Writing Assessment
• Form & Style Review
• Quick Answers
• ScholarWorks
• SKIL Courses and Workshops
• Walden Bookstore
• Walden Catalog & Student Handbook
• Student Safety/Title IX
• Legal & Consumer Information
• Website Terms and Conditions
• Cookie Policy
• Accessibility
• Accreditation
• State Authorization
• Net Price Calculator
• Contact Walden

Walden University is a member of Adtalem Global Education, Inc. www.adtalem.com Walden University is certified to operate by SCHEV © 2024 Walden University LLC. All rights reserved.

• Open access
• Published: 10 April 2024

Development of an index system for the scientific literacy of medical staff: a modified Delphi study in China

• Shuyu Liang 2   na1 ,
• Ziyan Zhai 2   na1 ,
• Xingmiao Feng 2 ,
• Xiaozhi Sun 1 ,
• Jingxuan Jiao 1 ,
• Yuan Gao 1   na2 &
• Kai Meng   ORCID: orcid.org/0000-0003-1467-7904 2 , 3   na2  

BMC Medical Education volume  24 , Article number:  397 ( 2024 ) Cite this article

118 Accesses

Metrics details

Scientific research activity in hospitals is important for promoting the development of clinical medicine, and the scientific literacy of medical staff plays an important role in improving the quality and competitiveness of hospital research. To date, no index system applicable to the scientific literacy of medical staff in China has been developed that can effectively evaluate and guide scientific literacy. This study aimed to establish an index system for the scientific literacy of medical staff in China and provide a reference for improving the evaluation of this system.

In this study, a preliminary indicator pool for the scientific literacy of medical staff was constructed through the nominal group technique ( n  = 16) with medical staff. Then, two rounds of Delphi expert consultation surveys ( n  = 20) were conducted with clinicians, and the indicators were screened, revised and supplemented using the boundary value method and expert opinions. Next, the hierarchical analysis method was utilized to determine the weights of the indicators and ultimately establish a scientific literacy indicator system for medical staff.

Following expert opinion, the index system for the scientific literacy of medical staff featuring 2 first-level indicators, 9 second-level indicators, and 38 third-level indicators was ultimately established, and the weights of the indicators were calculated. The two first-level indicators were research literacy and research ability, and the second-level indicators were research attitude (0.375), ability to identify problems (0.2038), basic literacy (0.1250), ability to implement projects (0.0843), research output capacity (0.0747), professional capacity (0.0735), data-processing capacity (0.0239), thesis-writing skills (0.0217), and ability to use literature (0.0181).

Conclusions

This study constructed a comprehensive scientific literacy index system that can assess medical staff's scientific literacy and serve as a reference for evaluating and improving their scientific literacy.

Peer Review reports

Due to the accelerated aging of the population and the growing global demand for healthcare in the wake of epidemics, there is an urgent need for medicine to provide greater support and protection. Medical scientific research is a critical element in promoting medical science and technological innovation, as well as improving clinical diagnosis and treatment techniques. It is the main driving force for the development of healthcare [ 1 ].

Medical personnel are highly compatible with clinical research. Due to their close interaction with patients, medical staff are better equipped to identify pertinent clinical research issues and actually implement clinical research projects [ 2 ]. Countries have created favorable conditions for the research and development of medical personnel by providing financial support, developing policies, and offering training courses [ 3 , 4 ]. However, some clinical studies have shown that the ability of most medical staff does not match current health needs and cannot meet the challenges posed by the twenty-first century [ 5 ]. It is clear that highly skilled professionals with scientific literacy are essential for national and social development [ 6 ]. Given the importance of scientific research in countries and hospitals, it is crucial to determine the level of scientific research literacy that medical personnel should possess and how to train them to acquire the necessary scientific research skills. These issues have significant practical implications.

Scientific literacy refers to an individual's ability to engage in science-related activities [ 7 ]. Some scholars suggest that the scientific literacy of medical personnel encompasses the fundamental qualities required for scientific research work, encompassing three facets: academic moral accomplishment, scientific research theory accomplishment, and scientific research ability accomplishment [ 8 ]. The existing research has focused primarily on the research capabilities of medical staff. According to Rillero, problem-solving skills, critical thinking, communication skills, and the ability to interpret data are the four core components of scientific literacy [ 9 ]. The ability to perform scientific research in nursing encompasses a range of abilities, including identifying problems, conducting literature reviews, designing and conducting scientific research, practicing scientific research, processing data, and writing papers [ 10 ]. Moule and Goodman proposed a framework of skills that research-literate nurses should possess, such as critical thinking capacity, analytical skills, searching skills, research critique skills, the ability to read and critically appraise research, and an awareness of ethical issues [ 11 ]. Several researchers have developed self-evaluation questionnaires to assess young researchers' scientific research and innovative abilities in the context of university-affiliated hospitals (UHAs) [ 12 ]. The relevant indicators include sensitivity to problems, sensitivity to cutting-edge knowledge, critical thinking, and other aspects. While these indicators cover many factors, they do not consider the issue of scientific research integrity in the medical field. The lack of detailed and targeted indicators, such as clinical resource collection ability and interdisciplinary cooperation ability, hinders the effective measurement of the current status of scientific literacy among medical staff [ 12 ]. In conclusion, the current research on the evaluation indicators of scientific literacy among medical personnel is incomplete, overlooking crucial humanistic characteristics, attitudes, and other moral literacy factors. Therefore, there is an urgent need to establish a comprehensive and systematic evaluation index to effectively assess the scientific literacy of medical staff.

Therefore, this study utilized a literature search and nominal group technique to screen the initial evaluation index and subsequently constructed an evaluation index system for medical staff's scientific research literacy utilizing the Delphi method. This index system would serve as a valuable tool for hospital managers, aiding them in the selection, evaluation, and training of scientific research talent. Additionally, this approach would enable medical personnel to identify their own areas of weakness and implement targeted improvement strategies.

Patient and public involvement

Patients and the public were not involved in this research.

Study design and participants

In this study, an initial evaluation index system was developed through a literature review and nominal group technique. Subsequently, a more comprehensive and scientific index system was constructed by combining qualitative and quantitative analysis utilizing the Delphi method to consult with experts. Finally, the hierarchical analysis method and the percentage weight method were employed to empower the index system.

The program used for this study is shown in Fig.  1 .

Study design. AHP, analytic hierarchy process

Establishing the preliminary indicator pool

Search process.

A literature search was performed in the China National Knowledge Infrastructure (CNKI), WanFang, PubMed, Web of Science and Scopus databases to collect the initial evaluation indicators. The time span ranged from the establishment of the database to July 2022. We used a combination of several MeSH terms in our searches:(("Medical Staff"[Mesh] OR "Nurses"[Mesh] OR "Physicians"[Mesh])) AND (("Literacy"[Mesh]) OR "Aptitude"[Mesh]). We also used several Title/Abstract searches, including keywords such as: Evaluation, scientific literacy, research ability.

The inclusion criteria were as follows: (1)The subjects were nurses, medicial staff and other personnel engaged in the medical industry; (2) Explore topics related to scientific literacy, such as research ability, and literature that can clarify the structure or dependency between indicators of scientific literacy; (3) Select articles published in countries such as China, the United States, the United Kingdom, Australia and Canada; (4) Research published in English or Chinese is considered to be eligible. The exclusion criteria are as follows: (1) indicators not applicable to medical staff; (2) Conference abstracts, case reports or review papers; (3) Articles with repeated descriptions; (4) There are no full-text articles or grey literature. A total of 78 articles were retrieved and 60 were retained after screening according to inclusion and exclusion criteria.

The research was conducted by two graduate students and two undergraduate students who participated in the literature search and screening. The entire research process was supervised and guided by one professor. All five members were from the fields of social medicine and health management. The professor was engaged in hospital management and health policy research for many years.

Nominal group technique

The nominal group technique was introduced at Hospital H in Beijing in July 2022. This hospital, with over 2,500 beds and 3,000 doctors, is a leading comprehensive medical center also known for its educational and research achievements, including numerous national research projects and awards.

The interview questions were based on the research question: What research literacy should medical staff have? 16 clinicians and nurses from Hospital H were divided into 2 equal groups and asked to provide their opinions on important aspects of research literacy based on their positions and experiences. Once all participants had shared their thoughts, similar responses were merged and polished. If anyone had further inputs after this, a second round of interviews was held until no new inputs were given. The entire meeting, including both rounds, was documented by researchers with audio recordings on a tape recorder.

Scientific literacy dimensions

Based on the search process, the research group extracted 58 tertiary indicators. To ensure the practicality and comprehensiveness of the indicators, the Nominal group technique was used on the basis of the literature search. Panelists summarized the entries shown in the interviews and merged similar content to obtain 32 third-level indicators. The indicators obtained from the literature search were compared. Several indicators with similar meanings, such as capture information ability, language expression ability, communication ability, and scientific research integrity, were merged. Additionally, the indicators obtained from the literature search, such as scientific research ethics, database use ability, feasibility and analysis ability, were added to the 15 indicators. A total of 47 third-level indicators were identified.

Fengling Dai and colleagues developed an innovation ability index system with six dimensions covering problem discovery, information retrieval, research design, practice, data analysis, and report writing, which represents the whole of innovative activity. Additionally, the system includes an innovation spirit index focusing on motivation, thinking, emotion, and will, reflecting the core of the innovation process in terms of competence [ 13 ]. Liao et al. evaluated the following five dimensions in their study on scientific research competence: literature processing, experimental manipulation, statistical analysis, manuscript production, and innovative project design [ 14 ]. Mohan claimed that scientific literacy consists of four core components: problem solving, critical thinking, communication skills, and the ability to interpret data [ 15 ].

This study structured scientific literacy into 2 primary indicators (research literacy and research competence) and 9 secondary indicators (basic qualifications, research ethics, research attitude, problem identification, literature use, professional capacity, subject implementation, data processing, thesis writing, and research output).

Using the Delphi method to develop an index system

Expert selection.

This study used the Delphi method to distribute expert consultation questionnaires online, allowing experts to exchange opinions anonymously to ensure that the findings were more desirable and scientific. No fixed number of experts is required for a Delphi study, but the more experts involved, the more stable the results will be [ 16 ]; this method generally includes 15 to 50 experts [ 17 ]. We selected clinicians from several tertiary hospitals in the Beijing area to serve as Delphi study consultants based on the following inclusion criteria: (1) they had a title of senior associate or above; (2) they had more than 10 years of work experience in the field of clinical scientific research, and (3) they were presiding over national scientific research projects. The exclusion criteria were as follows: (1) full-time scientific researchers, and (2) personnel in hospitals who were engaged only in management. To ensure that the selected experts were representative, this study selected 20 experts from 14 tertiary hospitals affiliated with Capital Medical University, Peking University, the Chinese Academy of Medical Sciences and the China Academy of Traditional Chinese Medicine according to the inclusion criteria; the hospitals featured an average of 1,231 beds each, and 9 hospitals were included among the 77 hospitals in the domestic comprehensive hospital ranking (Fudan Hospital Management Institute ranking). The experts represented various specialties and roles from different hospitals, including cardiology, neurosurgery, neurology, ear and throat surgery, head and neck surgery, radiology, imaging, infection, vascular interventional oncology, pediatrics, general practice, hematology, stomatology, nephrology, urology, and other related fields. This diverse group included physicians, nurses, managers, and vice presidents. The selected experts had extensive clinical experience, achieved numerous scientific research accomplishments and possessed profound knowledge and experience in clinical scientific research. This ensured the reliability of the consultation outcomes.

Design of the expert consultation questionnaire

The Delphi survey for experts included sections on their background, familiarity with the indicator system, system evaluation, and opinions. Experts rated indicators on importance, feasibility, and sensitivity using a 1–10 scale and their own familiarity with the indicators on a 1–5 scale. They also scored their judgment basis and impact on a 1–3 scale, considering theoretical analysis, work experience, peer understanding, and intuition. Two rounds of Delphi surveys were carried out via email with 20 experts to evaluate and suggest changes to the indicators. Statistical coefficients were calculated to validate the Delphi process. Feedback from the first round led to modifications and the inclusion of an AHP questionnaire for the second round. After the second round, indicators deemed less important were removed, and expert discussion finalized the indicator weights based on their relative importance scores. This resulted in the development of an index system for medical staff scientific literacy. The questionnaire is included in Additional file 1 (first round) and Additional file 2 (second round).

Using the boundary value method to screen the indicators

In this study, the boundary value method was utilized to screen the indicators of medical staff's scientific literacy, and the importance, feasibility, and sensitivity of each indicator were measured using the frequency of perfect scores, the arithmetic mean, and the coefficient of variation, respectively. When calculating the frequency of perfect scores and arithmetic means, the boundary value was set as "mean-SD," and indicators with scores higher than this value were retained. When calculating the coefficient of variation, the cutoff value was set to "mean + SD," and indicators with values below this threshold were retained.

The principles of indicator screening are as follows:

To evaluate the importance of the indicators, if none of the boundary values of the three statistics met the requirements, the indicators were deleted.

If an indicator has two aspects, importance, feasibility, or sensitivity, and each aspect has two or more boundary values that do not meet the requirements, then the indicator is deleted.

If all three boundary values for an indicator meet the requirements, the research group discusses the modification feedback from the experts and determines whether the indicator should be used.

The results of the two rounds of boundary values are shown in Table  1 .

Using the AHP to assign weights

After the second round of Delphi expert consultations, the analytic hierarchy process (AHP) was used to determine the weights of the two first-level indicators and the nine second-level indicators. The weights of the 37 third-level indicators were subsequently calculated via the percentage weight method. The AHP, developed by Saaty in the 1980s, is used to determine the priority and importance of elements constituting the decision-making hierarchy. It is based on multicriteria decision-making (MCDM) and determines the importance of decision-makers' judgments based on weights derived from pairwise comparisons between elements. In the AHP, pairwise comparisons are based on a comparative evaluation in which each element's weight in the lower tier is compared with that of other lower elements based on the element in the upper tier [ 18 ].

AHP analysis involves the following steps:

Step 1: Establish a final goal and list related elements to construct a hierarchy based on interrelated criteria.

Step 2: Perform a pairwise comparison for each layer to compare the weights of each element. Using a score from 1 to 9, which is the basic scale of the AHP, each pair is compared according to the expert’s judgment, and the importance is judged [ 19 , 20 ].

Yaahp software was employed to analyze data by creating a judgment matrix based on the experts' scores and hierarchical model. The index system weights were obtained by combining the experts' scores. The percentage weight method used experts' importance ratings from the second round to calculate weights, ranking indicators by importance, calculating their scores based on frequency of ranking, and determining weighting coefficients by dividing these scores by the total of all third-level indicators' scores. The third-level indicator weighting coefficients were then calculated by multiplying the coefficients [ 21 ].

Data analysis

Expert positivity coefficient.

The expert positivity coefficient is indicated by the effective recovery rate of the expert consultation questionnaire, which represents the level of expert positivity toward this consultation and determines the credibility and scientific validity of the questionnaire results. Generally, a questionnaire with an effective recovery rate of 70% is considered very good [ 22 ].

In this study, 20 questionnaires were distributed in both rounds of Delphi expert counseling, and all 20 were effectively recovered, resulting in a 100% effective recovery rate. Consequently, the experts provided positive feedback on the Delphi counseling.

Expert authority coefficient (CR)

The expert authority coefficient (Cr) is the arithmetic mean of the judgment coefficient (Ca) and the familiarity coefficient (Cs), namely, Cr =  \(\frac{({\text{Ca}}+{\text{Cs}})}{2}\) . The higher the degree of expert authority is, the greater the predictive accuracy of the indicator. A Cr ≥ 0.70 was considered to indicate an acceptable level of confidence [ 23 ]. Ca represents the basis on which the expert makes a judgment about the scenario in question, while Cs represents the expert's familiarity with the relevant problem [ 24 ].

Ca is calculated on the basis of experts' judgments of each indicator and the magnitude of its influence. In this study, experts used "practical experience (0.4), "theoretical analysis (0.3), "domestic and foreign peers (0.2)" and "intuition (0.1)" as the basis for judgment and assigned points according to the influence of each basis for judgment on the experts' judgment. Ca = 1 when the basis for judgment has a large influence on the experts, and Ca = 0.5 when the influence of the experts' judgment is at a medium level. When no influence on expert judgment was evident, Ca = 0 [ 25 ] (Table  2 ).

Cs refers to the degree to which the expert was familiar with the question. This study used the Likert scale method to score experts’ familiarity with the question on a scale ranging from 0 to 1 (1 = very familiar, 0.75 = more familiar, 0.5 = moderately familiar, 0.25 = less familiar, 0 = unfamiliar). The familiarity coefficient for each expert (the average familiarity for each indicator) was calculated. The average familiarity coefficient was subsequently computed [ 26 ].

The Cr value of the primary indicator in this study was 0.83, and the Cr value of the secondary indicator was 0.82 (> 0.7); hence, the results of the expert consultation were credible and accurate, as shown in Table  3 .

The degree of expert coordination is an important indicator used to judge the consistency among various experts regarding indicator scores. This study used the Kendall W coordination coefficient test to determine the degree of expert coordination. A higher Kendall W coefficient indicates a greater degree of expert coordination and greater consistency in expert opinion, and P  <  0.05 indicates that the difference is significant [ 26 ]. The results of the three-dimensional harmonization coefficient test for each indicator in the two rounds of the expert consultation questionnaire were valid ( p  <  0.01 ), emphasizing the consistency of the experts' scores. The values of the Kendall W coordination coefficients for both rounds are shown in Table  4 .

Basic information regarding the participants

The 20 Delphi experts who participated in this study were predominantly male (80.0%) rather than female (20.0%). In addition, the participants’ ages were mainly concentrated in the range of 41–50 years old (60.0%). The majority of the experts were doctors by profession (85.0%), and their education and titles were mainly doctoral degree (90.0%) and full senior level (17.0%). The experts also exhibited high academic achievement in their respective fields and had many years of working experience, with the majority having between 21 and 25 years of experience (40.0%) (Table  5 ).

Index screening

The boundary value method was applied to eliminate indicators, leading to the removal of 6 third-level indicators in the first round. One of these, the ability to use statistical software, was associated with a more significant second-level indicator involving data processing, which was kept after expert review. Six indicators were merged into three indicators due to duplication, and 5 third-level indicators were added, resulting in 2 primary indicators, 10 secondary indicators, and 43 third-level indicators.

In the second round of Delphi expert consultation, 5 third-level indicators were deleted, as shown in Additional file 3 , and only one third-level indicator, "Scientific spirit", remained under the secondary indicator "research attitude". The secondary indicator "Research attitude" was combined with "Research ethics" and the third-level indicator "Scientific spirit" was also considered part of "Research ethics". After expert discussion, these were merged into a new secondary indicator "Research attitude" with three third-level indicators: "Research ethics", "Research integrity", and "Scientific spirit". The final index system included two primary indicators, nine secondary indicators, and thirty-eight third-level indicators, as shown in Additional File 3 .

Final index system with weights

The weights of the two primary indexes, research literacy and research ability, were equal. This was determined using the hierarchical analysis method and the percentage weight method based on the results of the second round of Delphi expert consultation (Table  6 ). The primary indicator of research literacy encompasses the fundamental qualities and attitudes medical staff develop over time, including basic qualifications and approach to research. The primary indicator of research ability refers to medical professionals' capacity to conduct scientific research in new areas using suitable methods, as well as their skills needed for successful research using scientific methods.

In this study, the Delphi method was employed, and after two rounds of expert consultation, in accordance with the characteristics and scientific research requirements of medical staff in China, an index system for the scientific literacy of medical staff in China was constructed. The index system for medical staff's scientific literacy in this study consists of 2 first-level indicators, 9 second-level indicators, and 38 third-level indicators. Medical institutions at all levels can use this index system to scientifically assess medical staff's scientific literacy.

In 2014, the Joint Task Force for Clinical Trial Competency (JTF) published its Core Competency Framework [ 27 ]. The Framework focuses more on the capacity to conduct clinical research. These include principles such as clinical research and quality practices for drug clinical trials. However, this framework does not apply to the current evaluation of scientific literacy in hospitals. Because these indicators do not apply to all staff members, there is a lack of practical scientific research, such as information about the final paper output. Therefore, the experts who constructed the index system in this study came from different specialties, and the indicators can be better applied to scientific researchers in all fields. This approach not only addresses clinical researchers but also addresses the concerns of hospital managers, and the indicators are more applicable.

The weighted analysis showed that the primary indicators "research literacy" and "research ability" had the same weight (0.50) and were two important components of scientific literacy. Research ability is a direct reflection of scientific literacy and includes the ability to identify problems, the ability to use literature, professional capacity, subject implementation capacity, data-processing capacity, thesis-writing skills, and research output capacity. Only by mastering these skills can medical staff carry out scientific research activities more efficiently and smoothly. The ability to identify problems refers to the ability of medical staff to obtain insights into the frontiers of their discipline and to identify and ask insightful questions. Ratten claimed that only with keen insight and sufficient sensitivity to major scientific issues can we exploit the opportunities for innovation that may lead to breakthroughs [ 28 ]. Therefore, it is suggested that in the process of cultivating the scientific literacy of medical staff, the ability to identify problems, including divergent thinking, innovative sensitivity, and the ability to produce various solutions, should be improved. Furthermore, this study included three subentries of the secondary indicator "research attitude", namely, research ethics, research integrity, and scientific spirit. This is likely because improper scientific research behavior is still prevalent. A study conducted in the United States and Europe showed that the rate of scientific research misconduct was 2% [ 13 ]. A small survey conducted in Indian medical schools and hospitals revealed that 57% of the respondents knew that someone had modified or fabricated data for publication [ 28 ]. The weight of this index ranked first in the secondary indicators, indicating that scientific attitude is an important condition for improving research quality, relevance, and reliability. Countries and hospitals should develop, implement, and optimize policies and disciplinary measures to combat academic misconduct.

In addition, the third-level indicator "scheduling ability" under the second-level indicator "basic qualification" has a high weight, indicating that medical staff attach importance to management and distribution ability in the context of scientific research. Currently, hospitals face several problems, such as a shortage of medical personnel, excessive workload, and an increase in the number of management-related documents [ 29 , 30 ]. These factors result in time conflicts between daily responsibilities and scientific research tasks, thereby presenting significant obstacles to the allocation of sufficient time for scientific inquiry [ 31 ]. Effectively arranging clinical work and scientific research time is crucial to improving the overall efficiency of scientific research. In the earlier expert interviews, most medical staff believed that scientific research work must be combined with clinical work rather than focused only on scientific research. Having the ability to make overall arrangements is essential to solving these problems. The high weight given to the second-level index of 'subject implementation capacity', along with its associated third-level indicators, highlights the challenges faced by young medical staff in obtaining research subjects. Before implementing a project, researchers must thoroughly investigate, analyze, and compare various aspects of the research project, including its technical, economic, and engineering aspects. Moreover, potential financial and economic benefits, as well as social impacts, need to be predicted to determine the feasibility of the project and develop a research plan [ 32 ]. However, for most young medical staff in medical institutions, executing such a project can be challenging due to their limited scientific research experience [ 33 ]. A researcher who possesses these skills can truly carry out independent scientific research.

The weights of the second-level index "research output capacity" cannot be ignored. In Chinese hospitals, the ability to produce scientific research output plays a certain role in employees’ ability to obtain rewards such as high pay, and this ability is also used as a reference for performance appraisals [ 34 ]. The general scientific research performance evaluation includes the number of projects, scientific papers and monographs, scientific and technological achievements, and patents. In particular, the publication of papers is viewed as an indispensable aspect of performance appraisal by Chinese hospitals [ 35 ]. Specifically, scientific research papers are the carriers of scientific research achievements and academic research and thus constitute an important symbol of the level of medical development exhibited by medical research institutions; they are thus used as recognized and important indicators of scientific research output [ 36 ]. This situation is consistent with the weight evaluation results revealed by this study.

The results of this study are important for the training and management of the scientific research ability of medical personnel. First, the index system focuses not only on external characteristics such as scientific knowledge and skills but also on internal characteristics such as individual traits, motivation, and attitudes. Therefore, when building a research team and selecting and employing researchers, hospital managers can use the index system to comprehensively and systematically evaluate the situation of researchers, which is helpful for optimizing the allocation of a research team, learning from each other's strengths, and strengthening the strength of the research team. Second, this study integrates the content of existing research to obtain useful information through in-depth interviews with medical staff and constructs an evaluation index system based on Delphi expert consultation science, which comprehensively includes the evaluation of the whole process of scientific research activities. These findings can provide a basis for medical institutions to formulate scientific research training programs, help medical personnel master and improve scientific research knowledge and skills, and improve their working ability and quality. Moreover, the effectiveness of the training can also be evaluated according to the system.

In China, with the emergence of STEM rankings, hospitals pay more and more attention to the scientific research performance of medical personnel. Scientific literacy not only covers the abilities of medical personnel engaged in scientific research, but also reflects their professional quality in this field. Having high quality medical personnel often means that they have excellent scientific research ability, and their scientific research performance will naturally rise. In view of this,,medical institutions can define the meaning of third-level indicators and create Likert scales to survey medical staff. Based on the weights assigned to each indicator, comprehensive scores can be calculated to evaluate the level of scientific literacy among medical staff. Through detailed data analysis, they can not only reveal their shortcomings in scientific research ability and quality, but also provide a strong basis for subsequent training and promotion. Through targeted inspection, we can not only promote the comprehensive improvement of the ability of medical staff, but also promote the steady improvement of their scientific research performance, and inject new vitality into the scientific research cause of hospitals.

Limitations

This study has several limitations that need to be considered. First, the participants were only recruited from Beijing (a city in China), potentially lacking geographical diversity. We plan to select more outstanding experts from across the country to participate. Second, the index system may be more suitable for countries with medical systems similar to those of China. When applying this system in other countries, some modifications may be necessary based on the local context. Last, While this study has employed scientific methods to establish the indicator system, the index system has yet to be implemented on a large sample of medical staff. Therefore, the reliability and validity of the index system must be confirmed through further research. In conclusion, it is crucial to conduct further detailed exploration of the effectiveness and practical application of the index system in the future.

This study developed an evaluation index system using the Delphi method to assess the scientific literacy of medical staff in China. The system comprises two primary indicators, nine secondary indicators, and thirty-eight third-level indicators, with each index assigned a specific weight. The index system emphasizes the importance of both attitudes and abilities in the scientific research process for medical staff and incorporates more comprehensive evaluation indicators. In the current era of medical innovation, enhancing the scientific literacy of medical staff is crucial for enhancing the competitiveness of individuals, hospitals, and overall medical services in society. This evaluation index system is universally applicable and beneficial for countries with healthcare systems similar to those of China. This study can serve as a valuable reference for cultivating highly qualified and capable research personnel and enhancing the competitiveness of medical research.

Availability of data and materials

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.

Coloma J, Harris E. From construction workers to architects: developing scientific research capacity in low-income countries. PLoS Biol. 2009;7(7):e1000156. https://doi.org/10.1371/journal.pbio.1000156 .

Article   Google Scholar  

Brauer SG, Haines TP, Bew PG. Fostering clinician-led research. Aust J Physiother. 2007;53(3):143–4. https://doi.org/10.1016/s0004-9514(07)70020-x .

The L. China’s research renaissance. Lancet. 2019;393(10179):1385. https://doi.org/10.1016/S0140-6736(19)30797-4 .

Hannay DR. Evaluation of a primary care research network in rural Scotland. Prim Health Care ResDevelop. 2006;7(3):194–200. https://doi.org/10.1191/1463423606pc296oa .

Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010;376:1923–58.

Xie Y, Wang J, Li S, Zheng Y. Research on the Influence Path of Metacognitive Reading Strategies on Scientific Literacy. J Intell. 2023;11(5):78. https://doi.org/10.3390/jintelligence11050078 . PMID: 37233327; PMCID: PMC10218841.

Pang YH, Cheng JL. Revise of scientific research ability self-evaluation rating scales of nursing staff. Chin Nurs Res. 2011;13:1205–8. https://doi.org/10.3969/j.issn.1009-6493.2011.13.040 .

Zhang J, Jianshan MAO, Gu Y. On the cultivation of scientific research literacy of medical graduate students. Continu Med Educ China. 2023;15(3):179–82. https://doi.org/10.3969/j.issn.1674-9308.2023.03.043 .

Rillero P. Process skills and content knowledge. Sci Act. 1998;3:3–4.

Google Scholar  

Liu RS. Study on reliability and validity of self rating scale for scientific research ability of nursing staff. Chinese J Pract Nurs. 2004;9:8–10. https://doi.org/10.3760/cma.j.issn.1672-7088.2004.09.005 .

Moule P, Goodman M. Nursing research: An introduction. London, UK: Sage; 2013.

Xue J, Chen X, Zhang Z, et al. Survey on status quo and development needs of research and innovation capabilities of young researchers at university-affiliated hospitals in China: a cross-sectional survey. Ann Transl Med. 2022;10(18):964. https://doi.org/10.21037/atm-22-3692 .

Fanelli D, Costas R, Fang FC, et al. Testing hypotheses on risk factors for scientific misconduct via matched-control analysis of papers containing problematic image duplications. Sci Eng Ethics. 2019;25(3):771–89. https://doi.org/10.1007/s11948-018-0023-7 .

Liao Y, Zhou H, Wang F, et al. The Impact of Undergraduate Tutor System in Chinese 8-Year Medical Students in Scientific Research. Front Med (Lausanne). 2022;9:854132. https://doi.org/10.3389/fmed.2022.854132 .

Mohan L, Singh Y, Kathrotia R, et al. Scientific literacy and the medical student: A cross-sectional study. Natl Med J India. 2020;33(1):35–7. https://doi.org/10.4103/0970-258X.308242 .

Jorm AF. Using the Delphi expert consensus method in mental health research. Aust N Z J Psychiatry. 2015;49(10):887–97. https://doi.org/10.1177/0004867415600891 .

Xinran S, Heping W, Yule H, et al. Defining the scope and weights of services of a family doctor service project for the functional community using Delphi technique and analytic hierarchy process. Chinese Gen Pract. 2021;24(34):4386–91.

Park S, Kim HK, Lee M. An analytic hierarchy process analysis for reinforcing doctor-patient communication. BMC Prim Care. 2023;24(1):24. https://doi.org/10.1186/s12875-023-01972-3 . Published 2023 Jan 21.

Zhou MLY, Yin H, et al. New screening tool for neonatal nutritional risk in China: a validation study. BMJ Open. 2021;11(4):e042467. https://doi.org/10.1136/bmjopen-2020-042467 .

Wang K, Wang Z, Deng J, et al. Study on the evaluation of emergency management capacity of resilient communities by the AHP-TOPSIS method. Int J Environ Res Public Health. 2022;19(23):16201. https://doi.org/10.3390/ijerph192316201 .

Yuwei Z, Chuanhui Y, Junlong Z, et al. Application of analytic Hierarchy Process and percentage weight method to determine the weight of traditional Chinese medicine appropriate technology evaluation index system. Chin J Tradit Chinese Med. 2017;32(07):3054–6.

Babbie E. The practice of social research. 10th Chinese language edition. Huaxia Publisher, 2005: 253–4.

Liu W, Hu M, Chen W. Identifying the Service Capability of Long-Term Care Facilities in China: an e-Delphi study. Front Public Health. 2022;10:884514. https://doi.org/10.3389/fpubh.2022.884514 .

Zeng G. Modern epidemiological methods and application. Pecking Union Medical College Union Press, 1996.

Geng Y, Zhao L, Wang Y, et al. Competency model for dentists in China: Results of a Delphi study. PLoS One. 2018;13(3):e0194411. https://doi.org/10.1371/journal.pone.0194411 .

Cong C, Liu Y, Wang R. Kendall coordination coefficient W test and its SPSS implementation. Journal of Taishan Medical College. 2010;31(7):487–490. https://doi.org/10.3969/j.issn.1004-7115.2010.07.002 .

Sonstein S, Seltzer J, Li R, et al. Moving from compliance to competency: a harmonized core competency framework for the clinical research professional. Clin Res. 2014;28(3):17–23.

Madan C, Kruger E, Tennant M. 30 Years of dental research in Australia and India: a comparative analysis of published peer review literature. Indian J Dent Res. 2012;23(2):293–4. https://doi.org/10.4103/0970-9290.100447 .

Siemens DR, Punnen S, Wong J, Kanji N. A survey on the attitudes towards research in medical school. BMC Med Educ. 2010;10:4. https://doi.org/10.1186/1472-6920-10-4 .

Solomon SS, Tom SC, Pichert J, Wasserman D, Powers AC. Impact of medical student research in the development of physician-scientists. J Investig Med. 2003;51(3):149–56. https://doi.org/10.1136/jim-51-03-17 .

Misztal-Okonska P, Goniewicz K, Hertelendy AJ, et al. How Medical Studies in Poland Prepare Future Healthcare Managers for Crises and Disasters: Results of a Pilot Study. Healthcare (Basel). 2020;8(3):202. https://doi.org/10.3390/healthcare8030202 .

Xu G. On the declaration of educational scientific research topics. Journal of Henan Radio & TV University. 2013;26(01):98–101.

Ju Y, Zhao X. Top three hospitals clinical nurse scientific research ability present situation and influence factors analysis. J Health Vocational Educ. 2022;40(17):125–8.

Zhu Q, Li T, Li X, et al. Industry gain public hospital medical staff performance distribution mode of integration, exploring. J Health Econ Res. 2022;33(11):6-82–6.

Sun YLL. Analysis of hospital papers published based on performance appraisal. China Contemp Med. 2015;22(31):161–3.

Jian Y, Wu J, Liu Y. Citation analysis of seven tertiary hospitals in Yunnan province from 2008 to 2012. Yunnan Medicine. 2014;(6):700–704.

Download references

Acknowledgements

The authors thank all who participated in the nominal group technique and two rounds of the Delphi study.

This study was supported by the National Natural Science Foundation of China (72074160) and the Natural Science Foundation Project of Beijing (9222004).

Author information

Shuyu Liang and Ziyan Zhai contributed equally to this work and joint first authors.

Kai Meng and Yuan Gao contributed equally to this work and share corresponding author.

Authors and Affiliations

Aerospace Center Hospital, No. 15 Yuquan Road, Haidian District, Beijing, 100049, China

Xiaozhi Sun, Jingxuan Jiao & Yuan Gao

School of Public Health, Capital Medical University, No.10 Xitoutiao, Youanmenwai Street, Fengtai District, Beijing, 100069, China

Shuyu Liang, Ziyan Zhai, Xingmiao Feng & Kai Meng

Beijing Tiantan Hospital, Capital Medical University, No. 119 South Fourth Ring West Road, Fengtai District, Beijing, 100070, China

You can also search for this author in PubMed   Google Scholar

Contributions

S.L. and Z.Z. contributed equally to this paper. S.L. took charge of the nominal group technique, data analysis, writing the first draft and revising the manuscript; Z.Z. was responsible for the Delphi survey, data analysis, and writing of the first draft of the manuscript; XF was responsible for the rigorous revision of Delphi methods; X.S. and J.J. were responsible for the questionnaire survey and data collection; Y.G. contributed to the questionnaire survey, organization of the nominal group interview, supervision, project administration and resources; and K.M. contributed to conceptualization, methodology, writing—review; editing, supervision, and project administration. All the authors read and approved the final manuscript.

Corresponding authors

Correspondence to Yuan Gao or Kai Meng .

Ethics declarations

Ethics approval and consent to participate.

This study involved human participants and was approved by the Ethical Review Committee of the Capital Medical University (No. Z2022SY089). Participation in the survey was completely voluntary, and written informed consent was obtained from the participants.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Liang, S., Zhai, Z., Feng, X. et al. Development of an index system for the scientific literacy of medical staff: a modified Delphi study in China. BMC Med Educ 24 , 397 (2024). https://doi.org/10.1186/s12909-024-05350-0

Download citation

Received : 25 October 2023

Accepted : 26 March 2024

Published : 10 April 2024

DOI : https://doi.org/10.1186/s12909-024-05350-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Medical staff
• Scientific literacy
• Evaluation indicators

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Study Guides
• Homework Questions

Literature Review 4:2

Advertisement

Publics’ views on ethical challenges of artificial intelligence: a scoping review

• Open access
• Published: 19 December 2023

Cite this article

You have full access to this open access article

• Helena Machado   ORCID: orcid.org/0000-0001-8554-7619 1 ,
• Susana Silva   ORCID: orcid.org/0000-0002-1335-8648 2 &
• Laura Neiva   ORCID: orcid.org/0000-0002-1954-7597 3  

2555 Accesses

9 Altmetric

Explore all metrics

This scoping review examines the research landscape about publics’ views on the ethical challenges of AI. To elucidate how the concerns voiced by the publics are translated within the research domain, this study scrutinizes 64 publications sourced from PubMed ® and Web of Science™. The central inquiry revolves around discerning the motivations, stakeholders, and ethical quandaries that emerge in research on this topic. The analysis reveals that innovation and legitimation stand out as the primary impetuses for engaging the public in deliberations concerning the ethical dilemmas associated with AI technologies. Supplementary motives are rooted in educational endeavors, democratization initiatives, and inspirational pursuits, whereas politicization emerges as a comparatively infrequent incentive. The study participants predominantly comprise the general public and professional groups, followed by AI system developers, industry and business managers, students, scholars, consumers, and policymakers. The ethical dimensions most commonly explored in the literature encompass human agency and oversight, followed by issues centered on privacy and data governance. Conversely, topics related to diversity, nondiscrimination, fairness, societal and environmental well-being, technical robustness, safety, transparency, and accountability receive comparatively less attention. This paper delineates the concrete operationalization of calls for public involvement in AI governance within the research sphere. It underscores the intricate interplay between ethical concerns, public involvement, and societal structures, including political and economic agendas, which serve to bolster technical proficiency and affirm the legitimacy of AI development in accordance with the institutional norms that underlie responsible research practices.

Similar content being viewed by others

Artificial intelligence ethics has a black box problem

Jean-Christophe Bélisle-Pipon, Erica Monteferrante, … Vincent Couture

Are we Nearly There Yet? A Desires & Realities Framework for Europe’s AI Strategy

Ariana Polyviou & Efpraxia D. Zamani

Ensuring a ‘Responsible’ AI future in India: RRI as an approach for identifying the ethical challenges from an Indian perspective

Nitika Bhalla, Laurence Brooks & Tonii Leach

Avoid common mistakes on your manuscript.

1 Introduction

Current advances in the research, development, and application of artificial intelligence (AI) systems have yielded a far-reaching discourse on AI ethics that is accompanied by calls for AI technology to be democratically accountable and trustworthy from the publics’ Footnote 1 perspective [ 1 , 2 , 3 , 4 , 5 ]. Consequently, several ethics guidelines for AI have been released in recent years. As of early 2020, there were 167 AI ethics guidelines documents around the world [ 6 ]. Organizations such as the European Commission (EC), the Organization for Economic Co-operation and Development (OECD), and the United Nations Educational, Scientific and Cultural Organization (UNESCO) recognize that public participation is crucial for ensuring the responsible development and deployment of AI technologies, Footnote 2 emphasizing the importance of inclusivity, transparency, and democratic processes to effectively address the societal implications of AI [ 11 , 12 ]. These efforts were publicly announced as aiming to create a common understanding of ethical AI development and foster responsible practices that address societal concerns while maximizing AI’s potential benefits [ 13 , 14 ]. The concept of human-centric AI has emerged as a key principle in many of these regulatory initiatives, with the purposes of ensuring that human values are incorporated into the design of algorithms, that humans do not lose control over automated systems, and that AI is used in the service of humanity and the common good to improve human welfare and human rights [ 15 ]. Using the same rationale, the opacity and rapid diffusion of AI have prompted debate about how such technologies ought to be governed and which actors and values should be involved in shaping governance regimes [ 1 , 2 ].

While industry and business have traditionally tended to be seen as having no or little incentive to engage with ethics or in dialogue, AI leaders currently sponsor AI ethics [ 6 , 16 , 17 ]. However, some concerns call for ethics, public participation, and human-centric approaches in areas such as AI with high economic and political importance to be used within an instrumental rationale by the AI industry. A growing corpus of critical literature has conceived the development of AI ethics as efforts to reduce ethics to another form of industrial capital or to coopt and capture researchers as part of efforts to control public narratives [ 12 , 18 ]. According to some authors, one of the reasons why ethics is so appealing to many AI companies is to calm critical voices from the publics; therefore, AI ethics is seen as a way of gaining or restoring trust, credibility and support, as well as legitimation, while criticized practices are calmed down to maintain the agenda of industry and science [ 12 , 17 , 19 , 20 ].

Critical approaches also point out that despite regulatory initiatives explicitly invoking the need to incorporate human values into AI systems, they have the main objective of setting rules and standards to enable AI-based products and services to circulate in markets [ 20 , 21 , 22 ] and might serve to avoid or delay binding regulation [ 12 , 23 ]. Other critical studies argue that AI ethics fails to mitigate the racial, social, and environmental damage of AI technologies in any meaningful sense [ 24 ] and excludes alternative ethical practices [ 25 , 26 ]. As explained by Su [ 13 ], in a paper that considers the promise and perils of international human rights in AI governance, while human rights can serve as an authoritative source for holding AI developers accountable, its application to AI governance in practice shows a lack of effectiveness, an inability to effect structural change, and the problem of cooptation.

In a value analysis of AI national strategies, Wilson [ 5 ] concludes that the publics are primarily cast as recipients of AI’s abstract benefits, users of AI-driven services and products, a workforce in need of training and upskilling, or an important element for thriving democratic society that unlocks AI's potential. According to the author, when AI strategies articulate a governance role for the publics, it is more like an afterthought or rhetorical gesture than a clear commitment to putting “society-in-the-loop” into AI design and implementation [ 5 , pp. 7–8]. Another study of how public participation is framed in AI policy documents [ 4 ] shows that high expectations are assigned to public participation as a solution to address concerns about the concentration of power, increases in inequality, lack of diversity, and bias. However, in practice, this framing thus far gives little consideration to some of the challenges well known for researchers and practitioners of public participation with science and technology, such as the difficulty of achieving consensus among diverse societal views, the high resource requirements for public participation exercises, and the risks of capture by vested interests [ 4 , pp. 170–171]. These studies consistently reveal a noteworthy pattern: while references to public participation in AI governance are prevalent in the majority of AI national strategies, they tend to remain abstract and are often overshadowed by other roles, values, and policy concerns.

Some authors thus contended that the increasing demand to involve multiple stakeholders in AI governance, including the publics, signifies a discernible transformation within the sphere of science and technology policy. This transformation frequently embraces the framework of “responsible innovation”, Footnote 3 which emphasizes alignment with societal imperatives, responsiveness to evolving ethical, social, and environmental considerations, and the participation of the publics as well as traditionally defined stakeholders [ 3 , 28 ]. When investigating how the conception and promotion of public participation in European science and technology policies have evolved, Macq, Tancoine, and Strasser [ 29 ] distinguish between “participation in decision-making” (pertaining to science policy decisions or decisions on research topics) and “participation in knowledge and innovation-making”. They find that “while public participation had initially been conceived and promoted as a way to build legitimacy of research policy decisions by involving publics into decision-making processes, it is now also promoted as a way to produce better or more knowledge and innovation by involving publics into knowledge and innovation-making processes, and thus building legitimacy for science and technology as a whole” [ 29 , p. 508]. Although this shift in science and technology research policies has been noted, there exists a noticeable void in the literature in regard to understanding how concrete research practices incorporate public perspectives and embrace multistakeholder approaches, inclusion, and dialogue.

While several studies have delved into the framing of the publics’ role within AI governance in several instances (from Big Tech initiatives to hiring ethics teams and guidelines issued from multiple institutions to governments’ national policies related to AI development), discussing the underlying motivations driving the publics’ participation and the ethical considerations resulting from such involvement, there remains a notable scarcity of knowledge concerning how publicly voiced concerns are concretely translated into research efforts [ 30 , pp. 3–4, 31 , p. 8, 6]. To address this crucial gap, our scoping review endeavors to analyse the research landscape about the publics’ views on the ethical challenges of AI. Our primary objective is to uncover the motivations behind involving the publics in research initiatives, identify the segments of the publics that are considered in these studies, and illuminate the ethical concerns that warrant specific attention. Through this scoping review, we aim to enhance the understanding of the political and social backdrop within which debates and prior commitments regarding values and conditions for publics’ participation in matters related to science and technology are formulated and expressed [ 29 , 32 , 33 ] and which specific normative social commitments are projected and performed by institutional science [ 34 , p. 108, [ 35 , p. 856].

We followed the guidance for descriptive systematic scoping reviews by Levac et al. [ 36 ], based on the methodological framework developed by Arksey and O’Malley [ 37 ]. The steps of the review are listed below:

2.1 Stage 1: identifying the research question

The central question guiding this scoping review is the following: What motivations, publics and ethical issues emerge in research addressing the publics’ views on the ethical challenges of AI? We ask:

What motivations for engaging the publics with AI technologies are articulated?

Who are the publics invited?

Which ethical issues concerning AI technologies are perceived as needing the participation of the publics?

2.2 Stage 2: identifying relevant studies

A search of the publications on PubMed® and Web of Science™ was conducted on 19 May 2023, with no restriction set for language or time of publication, using the following search expression: (“AI” OR “artificial intelligence”) AND (“public” OR “citizen”) AND “ethics”. The search was followed by backwards reference tracking, examining the references of the selected publications based on full-text assessment.

2.3 Stage 3: study selection

The inclusion criteria allowed only empirical, peer-reviewed, original full-length studies written in English to explore publics’ views on the ethical challenges of AI as their main outcome. The exclusion criteria disallowed studies focusing on media discourses and texts. The titles of 1612 records were retrieved. After the removal of duplicates, 1485 records were examined. Two authors (HM and SS) independently screened all the papers retrieved initially, based on the title and abstract, and afterward, based on the full text. This was crosschecked and discussed in both phases, and perfect agreement was achieved.

The screening process is summarized in Fig.  1 . Based on title and abstract assessments, 1265 records were excluded because they were neither original full-length peer-reviewed empirical studies nor focused on the publics’ views on the ethical challenges of AI. Of the 220 fully read papers, 54 met the inclusion criteria. After backwards reference tracking, 10 papers were included, and the final review was composed of 64 papers.

Flowchart showing the search results and screening process for the scoping review of publics’ views on ethical challenges of AI

2.4 Stage 4: charting the data

A standardized data extraction sheet was initially developed by two authors (HM and SS) and completed by two coders (SS and LN), including both quantitative and qualitative data (Supplemental Table “Data Extraction”). We used MS Excel to chart the data from the studies.

The two coders independently charted the first 10 records, with any disagreements or uncertainties in abstractions being discussed and resolved by consensus. The forms were further refined and finalized upon consensus before completing the data charting process. Each of the remaining records was charted by one coder. Two meetings were held to ensure consistency in data charting and to verify accuracy. The first author (HM) reviewed the results.

Descriptive data for the characterization of studies included information about the authors and publication year, the country where the study was developed, study aims, type of research (quantitative, qualitative, or other), assessment of the publics’ views, and sample. The types of research participants recruited as publics were coded into 11 categories: developers of AI systems; managers from industry and business; representatives of governance bodies; policymakers; academics and researchers; students; professional groups; general public; local communities; patients/consumers; and other (specify).

Data on the main motivations for researching the publics’ views on the ethical challenges of AI were also gathered. Authors’ accounts of their motivations were synthesized into eight categories according to the coding framework proposed by Weingart and colleagues [ 33 ] concerning public engagement with science and technology-related issues: education (to inform and educate the public about AI, improving public access to scientific knowledge); innovation (to promote innovation, the publics are considered to be a valuable source of knowledge and are called upon to contribute to knowledge production, bridge building and including knowledge outside ‘formal’ ethics); legitimation (to promote public trust in and acceptance of AI, as well as of policies supporting AI); inspiration (to inspire and raise interest in AI, to secure a STEM-educated labor force); politicization (to address past political injustices and historical exclusion); democratization (to empower citizens to participate competently in society and/or to participate in AI); other (specify); and not clearly evident.

Based on the content analysis technique [ 38 ], ethical issues perceived as needing the participation of the publics were identified through quotations stated in the studies. These were then summarized in seven key ethical principles, according to the proposal outlined by the EC's Ethics Guidelines for Trustworthy AI [ 39 ]: human agency and oversight; technical robustness and safety; privacy and data governance; transparency; diversity, nondiscrimination and fairness; societal and environmental well-being; and accountability.

2.5 Stage 5: collating, summarizing, and reporting the results

The main characteristics of the 64 studies included can be found in Table  1 . Studies were grouped by type of research and ordered by the year of publication. The findings regarding the publics invited to participate are presented in Fig.  2 . The main motivations for engaging the publics with AI technologies and the ethical issues perceived as needing the participation of the publics are summarized in Tables  2 and 3 , respectively. The results are presented below in a narrative format, with complimentary tables and figures to provide a visual representation of key findings.

Publics invited to engage with issues framed as ethical challenges of AI

There are some methodological limitations in this scoping review that should be taken into account when interpreting the results. The use of only two search engines may preclude the inclusion of relevant studies, although supplemented by scanning the reference list of eligible studies. An in-depth analysis of the topics explored within each of the seven key ethical principles outlined by the EC's Ethics Guidelines for Trustworthy AI was not conducted. This assessment would lead to a detailed understanding of the publics’ views on ethical challenges of AI.

3.1 Study characteristics

Most of the studies were in recent years, with 35 of the 64 studies being published in 2022 and 2023. Journals were listed either on the databases related to Science Citation Index Expanded (n = 25) or the Social Science Citation Index (n = 23), with fewer journals indexed in the Emerging Sources Citation Index (n = 7) and the Arts and Humanities Citation Index (n = 2). Works covered a wide range of fields, including health and medicine (services, policy, medical informatics, medical ethics, public and environmental health); education; business, management and public administration; computer science; information sciences; engineering; robotics; communication; psychology; political science; and transportation. Beyond the general assessment of publics’ attitudes toward, preferences for, and expectations and concerns about AI, the publics’ views on ethical challenges of AI technologies have been studied mainly concerning healthcare and public services and less frequently regarding autonomous vehicles (AV), education, robotic technologies, and smart homes. Most of the studies (n = 47) were funded by research agencies, with 7 papers reporting conflicts of interest.

Quantitative research approaches have assessed the publics’ views on the ethical challenges of AI mainly through online or web-based surveys and experimental platforms, relying on Delphi studies, moral judgment studies, hypothetical vignettes, and choice-based/comparative conjoint surveys. The 25 qualitative studies collected data mainly by semistructured or in-depth interviews. Analysis of publicly available material reporting on AI-use cases, focus groups, a post hoc self-assessment, World Café, participatory research, and practice-based design research were used once or twice. Multi or mixed-methods studies relied on surveys with open-ended and closed questions, frequently combined with focus groups, in-depth interviews, literature reviews, expert opinions, examinations of relevant curriculum examples, tests, and reflexive writings.

The studies were performed (where stated) in a wide variety of countries, including the USA and Australia. More than half of the studies (n = 38) were conducted in a single country. Almost all studies used nonprobability sampling techniques. In quantitative studies, sample sizes varied from 2.3 M internet users in an online experimental platform study [ 40 ] to 20 participants in a Delphi study [ 41 ]. In qualitative studies, the samples varied from 123 participants in 21 focus groups [ 42 ] to six expert interviews [ 43 ]. In multi or mixed-methods studies, samples varied from 2036 participants [ 44 ] to 21 participants [ 45 ].

3.2 Motivations for engaging the publics

The qualitative synthesis of the motivations for researching the publics’ views on the ethical challenges of AI is presented in Table  2 and ordered by the number of studies referencing them in the scoping review. More than half of the studies (n = 37) addressed a single motivation. Innovation (n = 33) and legitimation (n = 29) proved to have the highest relevance as motivations for engaging the publics in the ethical challenges of AI technologies, as articulated in 15 studies. Additional motivations are rooted in education (n = 13), democratization (n = 11), and inspiration (n = 9). Politicization was mentioned in five studies. Although they were not authors’ motivations, few studies were found to have educational [ 46 , 47 ], democratization [ 48 , 49 ], and legitimation or inspirations effects [ 50 ].

To consider the publics as a valuable source of knowledge that can add real value to innovation processes in both the private and public sectors was the most frequent motivation mentioned in the literature. The call for public participation is rooted in the aspiration to add knowledge outside “formal” ethics at three interrelated levels. First, at a societal level, by asking what kind of AI we want as a society based on novel experiments on public policy preferences [ 51 ] and on the study of public perceptions, values, and concerns regarding AI design, development, and implementation in domains such as health care [ 46 , 52 , 53 , 54 , 55 ], public and social services [ 49 , 56 , 57 , 58 ], AV [ 59 , 60 ] and journalism [ 61 ]. Second, at a practical level, the literature provides insights into the perceived usefulness of AI applications [ 62 , 63 ] and choices between boosting developers’ voluntary adoption of ethical standards or imposing ethical standards via regulation and oversight [ 64 ], as well as suggesting specific guidance for the development and use of AI systems [ 65 , 66 , 67 ]. Finally, at a theoretical level, literature expands the social-technical perspective [ 68 ] and motivated-reasoning theory [ 69 ].

Legitimation was also a frequent motivation for engaging the publics. It was underpinned by the need for public trust in and social licences for implementing AI technologies. To ensure the long-term social acceptability of AI as a trustworthy technology [ 70 , 71 ] was perceived as essential to support its use and to justify its implementation. In one study [ 72 ], the authors developed an AI ethics scale to quantify how AI research is accepted in society and which area of ethical, legal, and social issues (ELSI) people are most concerned with. Public trust in and acceptance of AI is claimed by social institutions such as governments, private sectors, industry bodies, and the science community, behaving in a trustworthy manner, respecting public concerns, aligning with societal values, and involving members of the publics in decision-making and public policy [ 46 , 48 , 73 , 74 , 75 ], as well as in the responsible design and integration of AI technologies [ 52 , 76 , 77 ].

Education, democratization, and inspiration had a more modest presence as motivations to explore the publics’ views on the ethical challenges of AI. Considering the emergence of new roles and tasks related to AI, the literature has pointed to the public need to ensure the safe use of AI technologies by incorporating ethics and career futures into the education, preparation, and training of both middle school and university students and the current and future health workforce. Improvements in education and guidance for developers and older adults were also noticed. The views of the publics on what needs to be learned or how this learning may be supported or assessed were perceived as crucial. In one study [ 78 ], the authors developed strategies that promote learning related to AI through collaborative media production, connecting computational thinking to civic issues and creative expression. In another study [ 79 ], real-world scenarios were successfully used as a novel approach to teaching AI ethics. Rhim et al. [ 76 ] provided AV moral behavior design guidelines for policymakers, developers, and the publics by reducing the abstractness of AV morality.

Studies motivated by democratization promoted broader public participation in AI, aiming to empower citizens both to express their understandings, apprehensions, and concerns about AI [ 43 , 78 , 80 , 81 ] and to address ethical issues in AI as critical consumers, (potential future) developers of AI technologies or would-be participants in codesign processes [ 40 , 43 , 45 , 78 , 82 , 83 ]. Understanding the publics’ views on the ethical challenges of AI is expected to influence companies and policymakers [ 40 ]. In one study [ 45 ], the authors explored how a digital app might support citizens’ engagement in AI governance by informing them, raising public awareness, measuring publics’ attitudes and supporting collective decision-making.

Inspiration revolved around three main motivations: to raise public interest in AI [ 46 , 48 ]; to guide future empirical and design studies [ 79 ]; and to promote developers’ moral awareness through close collaboration between all those involved in the implementation, use, and design of AI technologies [ 46 , 61 , 78 , 84 , 85 ].

Politicization was the less frequent motivation reported in the literature for engaging the publics. Recognizing the need for mitigation of social biases [ 86 ], public participation to address historically marginalized populations [ 78 , 87 ], and promoting social equity [ 79 ] were the highlighted motives.

3.3 The invited publics

Study participants were mostly the general public and professional groups, followed by developers of AI systems, managers from industry and business, students, academics and researchers, patients/consumers, and policymakers (Fig.  2 ). The views of local communities and representatives of governance bodies were rarely assessed.

Representative samples of the general public were used in five papers related to studies conducted in the USA [ 88 ], Denmark [ 73 ], Germany [ 48 ], and Austria [ 49 , 63 ]. The remaining random or purposive samples from the general public comprised mainly adults and current and potential users of AI products and services, with few studies involving informal caregivers or family members of patients (n = 3), older people (n = 2), and university staff (n = 2).

Samples of professional groups included mainly healthcare professionals (19 out of 24 studies). Educators, law enforcement, media practitioners, and GLAM professionals (galleries, libraries, archives, and museums) were invited once.

3.4 Ethical issues

The ethical issues concerning AI technologies perceived as needing the participation of the publics are depicted in Table  3 . They were mapped by measuring the number of studies referencing them in the scoping review. Human agency and oversight (n = 55) was the most frequent ethical aspect that was studied in the literature, followed by those centered on privacy and data governance (n = 43). Diversity, nondiscrimination and fairness (n = 39), societal and environmental well-being (n = 39), technical robustness and safety (n = 38), transparency (n = 35), and accountability (n = 31) were less frequently discussed.

The concerns regarding human agency and oversight were the replacement of human beings by AI technologies and deskilling [ 47 , 55 , 67 , 74 , 75 , 89 , 90 ]; the loss of autonomy, critical thinking, and innovative capacities [ 50 , 58 , 61 , 77 , 78 , 83 , 85 , 90 ]; the erosion of human judgment and oversight [ 41 , 70 , 91 ]; and the potential for (over)dependence on technology and “oversimplified” decisions [ 90 ] due to the lack of publics’ expertise in judging and controlling AI technologies [ 68 ]. Beyond these ethical challenges, the following contributions of AI systems to empower human beings were noted: more fruitful and empathetic social relationships [ 47 , 68 , 90 ]; enhancing human capabilities and quality of life [ 68 , 70 , 74 , 83 , 92 ]; improving efficiency and productivity at work [ 50 , 53 , 62 , 65 , 83 ] by reducing errors [ 77 ], relieving the burden of professionals and/or increasing accuracy in decisions [ 47 , 55 , 90 ]; and facilitating and expanding access to safe and fair healthcare [ 42 , 53 , 54 ] through earlier diagnosis, increased screening and monitoring, and personalized prescriptions [ 47 , 90 ]. To foster human rights, allowing people to make informed decisions, the last say was up to the person themselves [ 42 , 43 , 46 , 55 , 64 , 67 , 73 , 76 ]. People should determine where and when to use automated functions and which functions to use [ 44 , 54 ], developing “job sharing” arrangements with machines and humans complementing and enriching each other [ 56 , 65 , 90 ]. The literature highlights the need to build AI systems that are under human control [ 48 , 70 ] whether to confirm or to correct the AI system’s outputs and recommendations [ 66 , 90 ]. Proper oversight mechanisms were seen as crucial to ensure accuracy and completeness, with divergent views about who should be involved in public participation approaches [ 86 , 87 ].

Data sharing and/or data misuse were considered the major roadblocks regarding privacy and data governance, with some studies pointing out distrust of participants related to commercial interests in health data [ 55 , 90 , 93 , 94 , 95 ] and concerns regarding risks of information getting into the hands of hackers, banks, employers, insurance companies, or governments [ 66 ]. As data are the backbone of AI, secure methods of data storage and protection are understood as needing to be provided from the input to the output data. Recognizing that in contemporary societies, people are aware of the consequences of smartphone use resulting in the minimization of privacy concerns [ 93 ], some studies have focused on the impacts of data breaches and loss of privacy and confidentiality [ 43 , 45 , 46 , 60 , 62 , 80 ] in relation to health-sensitive personal data [ 46 , 93 ], potentially affecting more vulnerable populations, such as senior citizens and mentally ill patients [ 82 , 90 ] as well as those at young ages [ 50 ], and when journalistic organizations collect user data to provide personalized news suggestions [ 61 ]. The need to find a balance between widening access to data and ensuring confidentiality and respect for privacy [ 53 ] was often expressed in three interrelated terms: first, the ability of data subjects to be fully informed about how data will be used and given the option of providing informed consent [ 46 , 58 , 78 ] and controlling personal information about oneself [ 57 ]; second, the need for regulation [ 52 , 65 , 87 ], with one study reporting that AI developers complain about the complexity, slowness, and obstacles created by regulation [ 64 ]; and last, the testing and certification of AI-enabled products and services [ 71 ]. The study by De Graaf et al. [ 91 ] discussed the robots’ right to store and process the data they collect, while Jenkins and Draper [ 42 ] explored less intrusive ways in which the robot could use information to report back to carers about the patient’s adherence to healthcare.

Studies discussing diversity, nondiscrimination, and fairness have pointed to the development of AI systems that reflect and reify social inequalities [ 45 , 78 ] through nonrepresentative datasets [ 55 , 58 , 96 , 97 ] and algorithmic bias [ 41 , 45 , 85 , 98 ] that might benefit some more than others. This could have multiple negative consequences for different groups based on ethnicity, disease, physical disability, age, gender, culture, or socioeconomic status [ 43 , 55 , 58 , 78 , 82 , 87 ], from the dissemination of hate speech [ 79 ] to the exacerbation of discrimination, which negatively impacts peace and harmony within society [ 58 ]. As there were cross-country differences and issue variations in the publics’ views of discriminatory bias [ 51 , 72 , 73 ], fostering diversity, inclusiveness, and cultural plurality [ 61 ] was perceived as crucial to ensure the transferability/effectiveness of AI systems in all social groups [ 60 , 94 ]. Diversity, nondiscrimination, and fairness were also discussed as a means to help reduce health inequalities [ 41 , 67 , 90 ], to compensate for human preconceptions about certain individuals [ 66 ], and to promote equitable distribution of benefits and burdens [ 57 , 71 , 80 , 93 ], namely, supporting access by all to the same updated and high-quality AI systems [ 50 ]. In one study [ 83 ], students provided constructive solutions to build an unbiased AI system, such as using a dataset that includes a diverse dataset engaging people of different ages, genders, ethnicities, and cultures. In another study [ 86 ], participants recommended diverse approaches to mitigate algorithmic bias, from open disclosure of limitations to consumer and patient engagement, representation of marginalized groups, incorporation of equity considerations into sampling methods and legal recourse, and identification of a wide range of stakeholders who may be responsible for addressing AI bias: developers, healthcare workers, manufacturers and vendors, policymakers and regulators, AI researchers and consumers.

Impacts on employment and social relationships were considered two major ethical challenges regarding societal and environmental well-being. The literature has discussed tensions between job creation [ 51 ] and job displacement [ 42 , 90 ], efficiency [ 90 ], and deskilling [ 57 ]. The concerns regarding future social relationships were the loss of empathy, humanity, and/or sensitivity [ 52 , 66 , 90 , 99 ]; isolation and fewer social connections [ 42 , 47 , 90 ]; laziness [ 50 , 83 ]; anxious counterreactions [ 83 , 99 ]; communication problems [ 90 ]; technology dependence [ 60 ]; plagiarism and cheating in education [ 50 ]; and becoming too emotionally attached to a robot [ 65 ]. To overcome social unawareness [ 56 ] and lack of acceptance [ 65 ] due to financial costs [ 56 , 90 ], ecological burden [ 45 ], fear of the unknown [ 65 , 83 ] and/or moral issues [ 44 , 59 , 100 ], AI systems need to provide public benefit sharing [ 55 ], consider discrepancies between public discourse about AI and the utility of the tools in real-world settings and practices [ 53 ], conform to the best standards of sustainability and address climate change and environmental justice [ 60 , 71 ]. Successful strategies in promoting the acceptability of robots across contexts included an approachable and friendly looking as possible, but not too human-like [ 49 , 65 ], and working with, rather than in competition, with humans [ 42 ].

The publics were invited to participate in the following ethical issues related to technical robustness and safety: usability, reliability, liability, and quality assurance checks of AI tools [ 44 , 45 , 55 , 62 , 99 ]; validity of big data analytic tools [ 87 ]; the degree to which an AI system can perform tasks without errors or mistakes [ 50 , 57 , 66 , 84 , 90 , 93 ]; and needed resources to perform appropriate (cyber)security [ 62 , 101 ]. Other studies approached the need to consider both material and normative concerns of AI applications [ 51 ], namely, assuring that AI systems are developed responsibly with proper consideration of risks [ 71 ] and sufficient proof of benefits [ 96 ]. One study [ 64 ] highlighted that AI developers tend to be reluctant to recognize safety issues, bias, errors, and failures, and when they do so, they do so in a selective manner and in their terms by adopting positive-sounding professional jargon as AI robustness.

Some studies recognized the need for greater transparency that reduces the mystery and opaqueness of AI systems [ 71 , 82 , 101 ] and opens its “black box” [ 64 , 71 , 98 ]. Clear insights about “what AI is/is not” and “how AI technology works” (definition, applications, implications, consequences, risks, limitations, weaknesses, threats, rewards, strengths, opportunities) were considered as needed to debunk the myth about AI as an independent entity [ 53 ] and for providing sufficient information and understandable explanations of “what’s happening” to society and individuals [ 43 , 48 , 72 , 73 , 78 , 102 ]. Other studies considered that people, when using AI tools, should be made fully aware that these AI devices are capturing and using their data [ 46 ] and how data are collected [ 58 ] and used [ 41 , 46 , 93 ]. Other transparency issues reported in the literature included the need for more information about the composition of data training sets [ 55 ], how algorithms work [ 51 , 55 , 84 , 94 , 97 ], how AI makes a decision [ 57 ] and the motivations for that decision [ 98 ]. Transparency requirements were also addressed as needing the involvement of multiple stakeholders: one study reported that transparency requirements should be seen as a mediator of debate between experts, citizens, communities, and stakeholders [ 87 ] and cannot be reduced to a product feature, avoiding experiences where people feel overwhelmed by explanations [ 98 ] or “too much information” [ 66 ].

Accountability was perceived by the publics as an important ethical issue [ 48 ], while developers expressed mixed attitudes, from moral disengagement to a sense of responsibility and moral conflict and uncertainty [ 85 ]. The literature has revealed public skepticism regarding accountability mechanisms [ 93 ] and criticism about the shift of responsibility away from tech industries that develop and own AI technologies [ 53 , 68 ], as it opens space for users to assume their own individual responsibility [ 78 ]. This was the case in studies that explored accountability concerns regarding the assignment of fault and responsibility for car accidents using self-driving technology [ 60 , 76 , 77 , 88 ]. Other studies considered that more attention is needed to scrutinize each application across the AI life cycle [ 41 , 71 , 94 ], to explainability of AI algorithms that provide to the publics the cause of AI outcomes [ 58 ], and to regulations that assign clear responsibility concerning litigation and liability [ 52 , 89 , 101 , 103 ].

4 Discussion

Within the realm of research studies encompassed in the scoping review, the contemporary impetus for engaging the publics in ethical considerations related to AI predominantly revolves around two key motivations: innovation and legitimation. This might be explained by the current emphasis on responsible innovation, which values the publics’ participation in knowledge and innovation-making [ 29 ] within a prioritization of the instrumental role of science for innovation and economic return [ 33 ]. Considering the publics as a valuable source of knowledge that should be called upon to contribute to knowledge innovation production is underpinned by the desire for legitimacy, specifically centered around securing the publics’ endorsement of scientific and technological advancements [ 33 , 104 ]. Approaching the publics’ views on the ethical challenges of AI can also be used as a form of risk prevention to reduce conflict and close vital debates in contention areas [ 5 , 34 , 105 ].

A second aspect that stood out in this finding is a shift in the motivations frequently reported as central for engaging the publics with AI technologies. Previous studies analysing AI national policies and international guidelines addressing AI governance [ 3 , 4 , 5 ] and a study analysing science communication journals [ 33 ] highlighted education, inspiration and democratization as the most prominent motivations. Our scoping review did not yield similar findings, which might signal a departure, in science policy related to public participation, from the past emphasis on education associated with the deficit model of public understanding of science and democratization of the model of public engagement with science [ 106 , 107 ].

The underlying motives for the publics’ engagement raise the question of the kinds of publics it addresses, i.e., who are the publics that are supposed to be recruited as research participants [ 32 ]. Our findings show a prevalence of the general public followed by professional groups and developers of AI systems. The wider presence of the general public indicates not only what Hagendijk and Irwin [ 32 , p. 167] describe as a fashionable tendency in policy circles since the late 1990s, and especially in Europe, focused on engaging 'the public' in scientific and technological change but also the avoidance of the issues of democratic representation [ 12 , 18 ]. Additionally, the unspecificity of the “public” does not stipulate any particular action [ 24 ] that allows for securing legitimacy for and protecting the interests of a wide range of stakeholders [ 19 , 108 ] while bringing the risk of silencing the voices of the very publics with whom engagement is sought [ 33 ]. The focus on approaching the publics’ views on the ethical challenges of AI through the general public also demonstrates how seeking to “lay” people’s opinions may be driven by a desire to promote public trust and acceptance of AI developments, showing how science negotiates challenges and reinstates its authority [ 109 ].

While this strategy is based on nonscientific audiences or individuals who are not associated with any scientific discipline or area of inquiry as part of their professional activities, the converse strategy—i.e., involving professional groups and AI developers—is also noticeable in our findings. This suggests that technocratic expert-dominated approaches coexist with a call for more inclusive multistakeholder approaches [ 3 ]. This coexistence is reinforced by the normative principles of the “responsible innovation” framework, in particular the prescription that innovation should include the publics as well as traditionally defined stakeholders [ 3 , 110 ], whose input has become so commonplace that seeking the input of laypeople on emerging technologies is sometimes described as a “standard procedure” [ 111 , p. 153].

In the body of literature included in the scoping review, human agency and oversight emerged as the predominant ethical dimension under investigation. This finding underscores the pervasive significance attributed to human centricity, which is progressively integrated into public discourses concerning AI, innovation initiatives, and market-driven endeavours [ 15 , 112 ]. In our perspective, the importance given to human-centric AI is emblematic of the “techno-regulatory imaginary” suggested by Rommetveit and van Dijk [ 35 ] in their study about privacy engineering applied in the European Union’s General Data Protection Regulation. This term encapsulates the evolving collective vision and conceptualization of the role of technology in regulatory and oversight contexts. At least two aspects stand out in the techno-regulatory imaginary, as they are meant to embed technoscience in societally acceptable ways. First, it reinstates pivotal demarcations between humans and nonhumans while concurrently producing intensified blurring between these two realms. Second, the potential resolutions offered relate to embedding fundamental rights within the structural underpinnings of technological architectures [ 35 ].

Following human agency and oversight, the most frequent ethical issue discussed in the studies contained in our scoping review was privacy and data governance. Our findings evidence additional central aspects of the “techno-regulatory imaginary” in the sense that instead of the traditional regulatory sites, modes of protecting privacy and data are increasingly located within more privatized and business-oriented institutions [ 6 , 35 ] and crafted according to a human-centric view of rights. The focus on secure ways of data storage and protection as in need to be provided from the input to the output data, the testing and certification of AI-enabled products and services, the risks of data breaches, and calls for finding a balance between widening access to data and ensuring confidentiality and respect for privacy, exhibited by many studies in this scoping review, portray an increasing framing of privacy and data protection within technological and standardization sites. This tendency shows how forms of expertise for privacy and data protection are shifting away from traditional regulatory and legal professionals towards privacy engineers and risk assessors in information security and software development. Another salient element to highlight pertains to the distribution of responsibility for privacy and data governance [ 6 , 113 ] within the realm of AI development through engagement with external stakeholders, including users, governmental bodies, and regulatory authorities. It extends from an emphasis on issues derived from data sharing and data misuse to facilitating individuals to exercise control over their data and privacy preferences and to advocating for regulatory frameworks that do not impede the pace of innovation. This distribution of responsibility shared among the contributions and expectations of different actors is usually convoked when the operationalization of ethics principles conflicts with AI deployment [ 6 ]. In this sense, privacy and data governance are reconstituted as a “normative transversal” [ 113 , p. 20], both of which work to stabilize or close controversies, while their operationalization does not modify any underlying operations in AI development.

Diversity, nondiscrimination and fairness, societal and environmental well-being, technical robustness and safety, transparency, and accountability were the ethical issues less frequently discussed in the studies included in this scoping review. In contrast, ethical issues of technical robustness and safety, transparency, and accountability “are those for which technical fixes can be or have already been developed” and “implemented in terms of technical solutions” [ 12 , p. 103]. The recognition of issues related to technical robustness and safety expresses explicit admissions of expert ignorance, error, or lack of control, which opens space for politics of “optimization of algorithms” [ 114 , p. 17] while reinforcing “strategic ignorance” [ 114 , p. 89]. In the words of the sociologist Linsey McGoey, strategic ignorance refers to “any actions which mobilize, manufacture or exploit unknowns in a wider environment to avoid liability for earlier actions” [ 115 , p. 3].

According to the analysis of Jobin et al. [ 11 ] of the global landscape of existing ethics guidelines for AI, transparency comprising efforts to increase explainability, interpretability, or other acts of communication and disclosure is the most prevalent principle in the current literature. Transparency gains high relevance in ethics guidelines because this principle has become a pro-ethical condition “enabling or impairing other ethical practices or principles” [Turilli and Floridi 2009, [ 11 ], p. 14]. Our findings highlight transparency as a crucial ethical concern for explainability and disclosure. However, as emphasized by Ananny and Crawford [ 116 , p. 973], there are serious limitations to the transparency ideal in making black boxes visible (i.e., disclosing and explaining algorithms), since “being able to see a system is sometimes equated with being able to know how it works and governs it—a pattern that recurs in recent work about transparency and computational systems”. The emphasis on transparency mirrors Aradau and Blanke’s [ 114 ] observation that Big Tech firms are creating their version of transparency. They are prompting discussions about their data usage, whether it is for “explaining algorithms” or addressing bias and discrimination openly.

The framing of ethical issues related to accountability, as elucidated by the studies within this scoping review, manifests as a commitment to ethical conduct and the transparent allocation of responsibility and legal obligations in instances where the publics encounters algorithmic deficiencies, glitches, or other imperfections. Within this framework, accountability becomes intricately intertwined with the notion of distributed responsibility, as expounded upon in our examination of how the literature addresses challenges in privacy and data governance. Simultaneously, it converges with our discussion on optimizing algorithms concerning ethical concerns on technical robustness and safety by which AI systems are portrayed as fallible yet eternally evolving towards optimization. As astutely observed by Aradau and Blanke [ 114 , p. 171], “forms of accountability through error enact algorithmic systems as fallible but ultimately correctable and therefore always desirable. Errors become temporary malfunctions, while the future of algorithms is that of indefinite optimization”.

5 Conclusion

This scoping review of how publics' views on ethical challenges of AI are framed, articulated, and concretely operationalized in the research sector shows that ethical issues and publics formation are closely entangled with symbolic and social orders, including political and economic agendas and visions. While Steinhoff [ 6 ] highlights the subordinated nature of AI ethics within an innovation network, drawing on insights from diverse sources beyond Big Tech, we assert that this network is dynamically evolving towards greater hybridity and boundary fusion. In this regard, we extend Steinhoff's argument by emphasizing the imperative for a more nuanced understanding of how this network operates within diverse contexts. Specifically, within the research sector, it operates through a convergence of boundaries, engaging human and nonhuman entities and various disciplines and stakeholders. Concurrently, the advocacy for diversity and inclusivity, along with the acknowledgement of errors and flaws, serves to bolster technical expertise and reaffirm the establishment of order and legitimacy in alignment with the institutional norms underpinning responsible research practices.

Our analysis underscores the growing importance of involving the publics in AI knowledge creation and innovation, both to secure public endorsement and as a tool for risk prevention and conflict mitigation. We observe two distinct approaches: one engaging nonscientific audiences and the other involving professional groups and AI developers, emphasizing the need for inclusivity while safeguarding expert knowledge. Human-centred approaches are gaining prominence, emphasizing the distinction and blending of human and nonhuman entities and embedding fundamental rights in technological systems. Privacy and data governance emerge as the second most prevalent ethical concern, shifting expertise away from traditional regulatory experts to privacy engineers and risk assessors. The distribution of responsibility for privacy and data governance is a recurring theme, especially in cases of ethical conflicts with AI deployment. However, there is a notable imbalance in attention, with less focus on diversity, nondiscrimination, fairness, societal, and environmental well-being, compared to human-centric AI, privacy, and data governance being managed through technical fixes. Last, acknowledging technical robustness and safety, transparency, and accountability as foundational ethics principles reveals an openness to expert limitations, allowing room for the politics of algorithm optimization, framing AI systems as correctable and perpetually evolving.

Data availability

This manuscript has data included as electronic supplementary material. The dataset constructed by the authors, resulting from a search of publications on PubMed ® and Web of Science™, analysed in the current study, is not publicly available. But it can be available from the corresponding author on reasonable request.

In this article, we will employ the term "publics" rather than the singular "public" to delineate our viewpoint concerning public participation in AI. Our option is meant to acknowledge that there are no uniform, monolithic viewpoints or interests. From our perspective, the term "publics" allows for a more nuanced understanding of the various groups, communities, and individuals who may have different attitudes, beliefs, and concerns regarding AI. This choice may differ from the terminology employed in the referenced literature.

The following examples are particularly illustrative of the multiplicity of organizations emphasizing the need for public participation in AI. The OECD Recommendations of the Council on AI specifically emphasizes the importance of empowering stakeholders considering essential their engagement to adoption of trustworthy [ 7 , p. 6]. The UNESCO Recommendation on the Ethics of AI emphasizes that public awareness and understanding of AI technologies should be promoted (recommendation 44) and it encourages governments and other stakeholders to involve the publics in AI decision-making processes (recommendation 47) [ 8 , p. 23]. The European Union (EU) White Paper on AI [ 9 , p. 259] outlines the EU’s approach to AI, including the need for public consultation and engagement. The Ethics Guidelines for Trustworthy AI [ 10 , pp. 19, 239], developed by the High-Level Expert Group on AI (HLEG) appointed by the EC, emphasize the importance of public participation and consultation in the design, development, and deployment of AI systems.

“Responsible Innovation” (RI) and “Responsible Research and Innovation” (RRI) have emerged in parallel and are often used interchangeably, but they are not the same thing [ 27 , 28 ]. RRI is a policy-driven discourse that emerged from the EC in the early 2010s, while RI emerged largely from academic roots. For this paper, we will not consider the distinctive features of each discourse, but instead focus on the common features they share.

Cath, C., Wachter, S., Mittelstadt, B., Taddeo, M., Floridi, L.: Artificial intelligence and the ‘good society’: the US, EU, and UK approach. Sci. Eng. Ethics 24 , 505–528 (2017). https://doi.org/10.1007/s11948-017-9901-7

Article   Google Scholar  

Cussins, J.N.: Decision points in AI governance. CLTC white paper series. Center for Long-term Cybersecurity. https://cltc.berkeley.edu/publication/decision-points-in-ai-governance/ (2020). Accessed 8 July 2023

Ulnicane, I., Okaibedi Eke, D., Knight, W., Ogoh, G., Stahl, B.: Good governance as a response to discontents? Déjà vu, or lessons for AI from other emerging technologies. Interdiscip. Sci. Rev. 46 (1–2), 71–93 (2021). https://doi.org/10.1080/03080188.2020.1840220

Ulnicane, I., Knight, W., Leach, T., Stahl, B., Wanjiku, W.: Framing governance for a contested emerging technology: insights from AI policy. Policy Soc. 40 (2), 158–177 (2021). https://doi.org/10.1080/14494035.2020.1855800

Wilson, C.: Public engagement and AI: a values analysis of national strategies. Gov. Inf. Q. 39 (1), 101652 (2022). https://doi.org/10.1016/j.giq.2021.101652

Steinhoff, J.: AI ethics as subordinated innovation network. AI Soc. (2023). https://doi.org/10.1007/s00146-023-01658-5

Organization for Economic Co-operation and Development. Recommendation of the Council on Artificial Intelligence. https://legalinstruments.oecd.org/en/instruments/oecd-legal-0449 (2019). Accessed 8 July 2023

United Nations Educational, Scientific and Cultural Organization. Recommendation on the Ethics of Artificial Intelligence. https://unesdoc.unesco.org/ark:/48223/pf0000381137 (2021). Accessed 28 June 2023

European Commission. On artificial intelligence – a European approach to excellence and trust. White paper. COM(2020) 65 final. https://commission.europa.eu/publications/white-paper-artificial-intelligence-european-approach-excellence-and-trust_en (2020). Accessed 28 June 2023

European Commission. The ethics guidelines for trustworthy AI. Directorate-General for Communications Networks, Content and Technology, EC Publications Office. https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai (2019). Accessed 10 July 2023

Jobin, A., Ienca, M., Vayena, E.: The global landscape of AI ethics guidelines. Nat. Mach. Intell. 1 , 389–399 (2019). https://doi.org/10.1038/s42256-019-0088-2

Hagendorff, T.: The ethics of AI ethics: an evaluation of guidelines. Minds Mach. 30 , 99–120 (2020). https://doi.org/10.1007/s11023-020-09517-8

Su, A.: The promise and perils of international human rights law for AI governance. Law Technol. Hum. 4 (2), 166–182 (2022). https://doi.org/10.5204/lthj.2332

Article   MathSciNet   Google Scholar  

Ulnicane, I.: Emerging technology for economic competitiveness or societal challenges? Framing purpose in artificial intelligence policy. GPPG. 2 , 326–345 (2022). https://doi.org/10.1007/s43508-022-00049-8

Sigfrids, A., Leikas, J., Salo-Pöntinen, H., Koskimies, E.: Human-centricity in AI governance: a systemic approach. Front Artif. Intell. 6 , 976887 (2023). https://doi.org/10.3389/frai.2023.976887

Benkler, Y.: Don’t let industry write the rules for AI. Nature 569 (7755), 161 (2019). https://doi.org/10.1038/d41586-019-01413-1

Phan, T., Goldenfein, J., Mann, M., Kuch, D.: Economies of virtue: the circulation of ‘ethics’ in Big Tech. Sci. Cult. 31 (1), 121–135 (2022). https://doi.org/10.1080/09505431.2021.1990875

Ochigame, R.: The invention of “ethical AI”: how big tech manipulates academia to avoid regulation. Intercept. https://theintercept.com/2019/12/20/mit-ethical-ai-artificial-intelligence/ (2019). Accessed 10 July 2023

Ferretti, T.: An institutionalist approach to ai ethics: justifying the priority of government regulation over self-regulation. MOPP 9 (2), 239–265 (2022). https://doi.org/10.1515/mopp-2020-0056

van Maanen, G.: AI ethics, ethics washing, and the need to politicize data ethics. DISO 1 (9), 1–23 (2022). https://doi.org/10.1007/s44206-022-00013-3

Gerdes, A.: The tech industry hijacking of the AI ethics research agenda and why we should reclaim it. Discov. Artif. Intell. 2 (25), 1–8 (2022). https://doi.org/10.1007/s44163-022-00043-3

Amariles, D.R., Baquero, P.M.: Promises and limits of law for a human-centric artificial intelligence. Comput. Law Secur. Rev. 48 (105795), 1–10 (2023). https://doi.org/10.1016/j.clsr.2023.105795

Mittelstadt, B.: Principles alone cannot guarantee ethical AI. Nat. Mach. Intell. 1 (11), 501–507 (2019). https://doi.org/10.1038/s42256-019-0114-4

Munn, L.: The uselessness of AI ethics. AI Ethics 3 , 869–877 (2022). https://doi.org/10.1007/s43681-022-00209-w

Heilinger, J.C.: The ethics of AI ethics. A constructive critique. Philos. Technol. 35 (61), 1–20 (2022). https://doi.org/10.1007/s13347-022-00557-9

Roche, C., Wall, P.J., Lewis, D.: Ethics and diversity in artificial intelligence policies, strategies and initiatives. AI Ethics (2022). https://doi.org/10.1007/s43681-022-00218-9

Diercks, G., Larsen, H., Steward, F.: Transformative innovation policy: addressing variety in an emerging policy paradigm. Res. Policy 48 (4), 880–894 (2019). https://doi.org/10.1016/j.respol.2018.10.028

Owen, R., Pansera, M.: Responsible innovation and responsible research and innovation. In: Dagmar, S., Kuhlmann, S., Stamm, J., Canzler, W. (eds.) Handbook on Science and Public Policy, pp. 26–48. Edward Elgar, Cheltenham (2019)

Google Scholar  

Macq, H., Tancoigne, E., Strasser, B.J.: From deliberation to production: public participation in science and technology policies of the European Commission (1998–2019). Minerva 58 (4), 489–512 (2020). https://doi.org/10.1007/s11024-020-09405-6

Cath, C.: Governing artificial intelligence: ethical, legal and technical opportunities and challenges. Philos. Trans. Royal Soc. A. 376 , 20180080 (2018). https://doi.org/10.1098/rsta.2018.0080

Wilson, C.: The socialization of civic participation norms in government?: Assessing the effect of the Open Government Partnership on countries’e-participation. Gov. Inf. Q. 37 (4), 101476 (2020). https://doi.org/10.1016/j.giq.2020.101476

Hagendijk, R., Irwin, A.: Public deliberation and governance: engaging with science and technology in contemporary Europe. Minerva 44 (2), 167–184 (2006). https://doi.org/10.1007/s11024-006-0012-x

Weingart, P., Joubert, M., Connoway, K.: Public engagement with science - origins, motives and impact in academic literature and science policy. PLoS One 16 (7), e0254201 (2021). https://doi.org/10.1371/journal.pone.0254201

Wynne, B.: Public participation in science and technology: performing and obscuring a political–conceptual category mistake. East Asian Sci. 1 (1), 99–110 (2007). https://doi.org/10.1215/s12280-007-9004-7

Rommetveit, K., Van Dijk, N.: Privacy engineering and the techno-regulatory imaginary. Soc. Stud. Sci. 52 (6), 853–877 (2022). https://doi.org/10.1177/03063127221119424

Levac, D., Colquhoun, H., O’Brien, K.: Scoping studies: advancing the methodology. Implement. Sci. 5 (69), 1–9 (2010). https://doi.org/10.1186/1748-5908-5-69

Arksey, H., O’Malley, L.: Scoping studies: towards a methodological framework. Int. J. Soc. Res. Methodol. 8 (1), 19–32 (2005). https://doi.org/10.1080/1364557032000119616

Stemler, S.: An overview of content analysis. Pract. Asses. Res. Eval. 7 (17), 1–9 (2001). https://doi.org/10.7275/z6fm-2e34

European Commission. European Commission's ethics guidelines for trustworthy AI. https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai (2021). Accessed 8 July 2023

Awad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., et al.: The moral machine experiment. Nature 563 (7729), 59–64 (2018). https://doi.org/10.1038/s41586-018-0637-6

Liyanage, H., Liaw, S.T., Jonnagaddala, J., Schreiber, R., Kuziemsky, C., Terry, A.L., de Lusignan, S.: Artificial intelligence in primary health care: perceptions, issues, and challenges. Yearb. Med. Inform. 28 (1), 41–46 (2019). https://doi.org/10.1055/s-0039-1677901

Jenkins, S., Draper, H.: Care, monitoring, and companionship: views on care robots from older people and their carers. Int. J. Soc. Robot. 7 (5), 673–683 (2015). https://doi.org/10.1007/s12369-015-0322-y

Tzouganatou, A.: Openness and privacy in born-digital archives: reflecting the role of AI development. AI Soc. 37 (3), 991–999 (2022). https://doi.org/10.1007/s00146-021-01361-3

Liljamo, T., Liimatainen, H., Pollanen, M.: Attitudes and concerns on automated vehicles. Transp. Res. Part F Traffic Psychol. Behav. 59 , 24–44 (2018). https://doi.org/10.1016/j.trf.2018.08.010

Couture, V., Roy, M.C., Dez, E., Laperle, S., Belisle-Pipon, J.C.: Ethical implications of artificial intelligence in population health and the public’s role in its governance: perspectives from a citizen and expert panel. J. Med. Internet Res. 25 , e44357 (2023). https://doi.org/10.2196/44357

McCradden, M.D., Sarker, T., Paprica, P.A.: Conditionally positive: a qualitative study of public perceptions about using health data for artificial intelligence research. BMJ Open 10 (10), e039798 (2020). https://doi.org/10.1136/bmjopen-2020-039798

Blease, C., Kharko, A., Annoni, M., Gaab, J., Locher, C.: Machine learning in clinical psychology and psychotherapy education: a mixed methods pilot survey of postgraduate students at a Swiss University. Front. Public Health 9 (623088), 1–8 (2021). https://doi.org/10.3389/fpubh.2021.623088

Kieslich, K., Keller, B., Starke, C.: Artificial intelligence ethics by design. Evaluating public perception on the importance of ethical design principles of artificial intelligence. Big Data Soc. 9 (1), 1–15 (2022). https://doi.org/10.1177/20539517221092956

Willems, J., Schmidthuber, L., Vogel, D., Ebinger, F., Vanderelst, D.: Ethics of robotized public services: the role of robot design and its actions. Gov. Inf. Q. 39 (101683), 1–11 (2022). https://doi.org/10.1016/J.Giq.2022.101683

Tlili, A., Shehata, B., Adarkwah, M.A., Bozkurt, A., Hickey, D.T., Huang, R.H., Agyemang, B.: What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learn Environ. 10 (15), 1–24 (2023). https://doi.org/10.1186/S40561-023-00237-X

Ehret, S.: Public preferences for governing AI technology: comparative evidence. J. Eur. Public Policy 29 (11), 1779–1798 (2022). https://doi.org/10.1080/13501763.2022.2094988

Esmaeilzadeh, P.: Use of AI-based tools for healthcare purposes: a survey study from consumers’ perspectives. BMC Med. Inform. Decis. Mak. 20 (170), 1–19 (2020). https://doi.org/10.1186/s12911-020-01191-1

Laïï, M.C., Brian, M., Mamzer, M.F.: Perceptions of artificial intelligence in healthcare: findings from a qualitative survey study among actors in France. J. Transl. Med. 18 (14), 1–13 (2020). https://doi.org/10.1186/S12967-019-02204-Y

Valles-Peris, N., Barat-Auleda, O., Domenech, M.: Robots in healthcare? What patients say. Int. J. Environ. Res. Public Health 18 (9933), 1–18 (2021). https://doi.org/10.3390/ijerph18189933

Hallowell, N., Badger, S., Sauerbrei, A., Nellaker, C., Kerasidou, A.: “I don’t think people are ready to trust these algorithms at face value”: trust and the use of machine learning algorithms in the diagnosis of rare disease. BMC Med. Ethics 23 (112), 1–14 (2022). https://doi.org/10.1186/s12910-022-00842-4

Criado, J.I., de Zarate-Alcarazo, L.O.: Technological frames, CIOs, and artificial intelligence in public administration: a socio-cognitive exploratory study in spanish local governments. Gov. Inf. Q. 39 (3), 1–13 (2022). https://doi.org/10.1016/J.Giq.2022.101688

Isbanner, S., O’Shaughnessy, P.: The adoption of artificial intelligence in health care and social services in Australia: findings from a methodologically innovative national survey of values and attitudes (the AVA-AI Study). J. Med. Internet Res. 24 (8), e37611 (2022). https://doi.org/10.2196/37611

Kuberkar, S., Singhal, T.K., Singh, S.: Fate of AI for smart city services in India: a qualitative study. Int. J. Electron. Gov. Res. 18 (2), 1–21 (2022). https://doi.org/10.4018/Ijegr.298216

Kallioinen, N., Pershina, M., Zeiser, J., Nezami, F., Pipa, G., Stephan, A., Konig, P.: Moral judgements on the actions of self-driving cars and human drivers in dilemma situations from different perspectives. Front. Psychol. 10 (2415), 1–15 (2019). https://doi.org/10.3389/fpsyg.2019.02415

Vrščaj, D., Nyholm, S., Verbong, G.P.J.: Is tomorrow’s car appealing today? Ethical issues and user attitudes beyond automation. AI Soc. 35 (4), 1033–1046 (2020). https://doi.org/10.1007/s00146-020-00941-z

Bastian, M., Helberger, N., Makhortykh, M.: Safeguarding the journalistic DNA: attitudes towards the role of professional values in algorithmic news recommender designs. Digit. Journal. 9 (6), 835–863 (2021). https://doi.org/10.1080/21670811.2021.1912622

Kaur, K., Rampersad, G.: Trust in driverless cars: investigating key factors influencing the adoption of driverless cars. J. Eng. Technol. Manag. 48 , 87–96 (2018). https://doi.org/10.1016/j.jengtecman.2018.04.006

Willems, J., Schmid, M.J., Vanderelst, D., Vogel, D., Ebinger, F.: AI-driven public services and the privacy paradox: do citizens really care about their privacy? Public Manag. Rev. (2022). https://doi.org/10.1080/14719037.2022.2063934

Duke, S.A.: Deny, dismiss and downplay: developers’ attitudes towards risk and their role in risk creation in the field of healthcare-AI. Ethics Inf. Technol. 24 (1), 1–15 (2022). https://doi.org/10.1007/s10676-022-09627-0

Cresswell, K., Cunningham-Burley, S., Sheikh, A.: Health care robotics: qualitative exploration of key challenges and future directions. J. Med. Internet Res. 20 (7), e10410 (2018). https://doi.org/10.2196/10410

Amann, J., Vayena, E., Ormond, K.E., Frey, D., Madai, V.I., Blasimme, A.: Expectations and attitudes towards medical artificial intelligence: a qualitative study in the field of stroke. PLoS One 18 (1), e0279088 (2023). https://doi.org/10.1371/journal.pone.0279088

Aquino, Y.S.J., Rogers, W.A., Braunack-Mayer, A., Frazer, H., Win, K.T., Houssami, N., et al.: Utopia versus dystopia: professional perspectives on the impact of healthcare artificial intelligence on clinical roles and skills. Int. J. Med. Inform. 169 (104903), 1–10 (2023). https://doi.org/10.1016/j.ijmedinf.2022.104903

Sartori, L., Bocca, G.: Minding the gap(s): public perceptions of AI and socio-technical imaginaries. AI Soc. 38 (2), 443–458 (2022). https://doi.org/10.1007/s00146-022-01422-1

Chen, Y.-N.K., Wen, C.-H.R.: Impacts of attitudes toward government and corporations on public trust in artificial intelligence. Commun. Stud. 72 (1), 115–131 (2021). https://doi.org/10.1080/10510974.2020.1807380

Aitken, M., Ng, M., Horsfall, D., Coopamootoo, K.P.L., van Moorsel, A., Elliott, K.: In pursuit of socially ly-minded data-intensive innovation in banking: a focus group study of public expectations of digital innovation in banking. Technol. Soc. 66 (101666), 1–10 (2021). https://doi.org/10.1016/j.techsoc.2021.101666

Choung, H., David, P., Ross, A.: Trust and ethics in AI. AI Soc. 38 (2), 733–745 (2023). https://doi.org/10.1007/s00146-022-01473-4

Hartwig, T., Ikkatai, Y., Takanashi, N., Yokoyama, H.M.: Artificial intelligence ELSI score for science and technology: a comparison between Japan and the US. AI Soc. 38 (4), 1609–1626 (2023). https://doi.org/10.1007/s00146-021-01323-9

Ploug, T., Sundby, A., Moeslund, T.B., Holm, S.: Population preferences for performance and explainability of artificial intelligence in health care: choice-based conjoint survey. J. Med. Internet Res. 23 (12), e26611 (2021). https://doi.org/10.2196/26611

Zheng, B., Wu, M.N., Zhu, S.J., Zhou, H.X., Hao, X.L., Fei, F.Q., et al.: Attitudes of medical workers in China toward artificial intelligence in ophthalmology: a comparative survey. BMC Health Serv. Res. 21 (1067), 1–13 (2021). https://doi.org/10.1186/S12913-021-07044-5

Ma, J., Tojib, D., Tsarenko, Y.: Sex robots: are we ready for them? An exploration of the psychological mechanisms underlying people’s receptiveness of sex robots. J. Bus. Ethics 178 (4), 1091–1107 (2022). https://doi.org/10.1007/s10551-022-05059-4

Rhim, J., Lee, G.B., Lee, J.H.: Human moral reasoning types in autonomous vehicle moral dilemma: a cross-cultural comparison of Korea and Canada. Comput. Hum. Behav. 102 , 39–56 (2020). https://doi.org/10.1016/j.chb.2019.08.010

Dempsey, R.P., Brunet, J.R., Dubljevic, V.: Exploring and understanding law enforcement’s relationship with technology: a qualitative interview study of police officers in North Carolina. Appl. Sci-Basel 13 (6), 1–17 (2023). https://doi.org/10.3390/App13063887

Lee, C.H., Gobir, N., Gurn, A., Soep, E.: In the black mirror: youth investigations into artificial intelligence. ACM Trans. Comput. Educ. 22 (3), 1–25 (2022). https://doi.org/10.1145/3484495

Kong, S.C., Cheung, W.M.Y., Zhang, G.: Evaluating an artificial intelligence literacy programme for developing university students? Conceptual understanding, literacy, empowerment and ethical awareness. Educ. Technol. Soc. 26 (1), 16–30 (2023). https://doi.org/10.30191/Ets.202301_26(1).0002

Street, J., Barrie, H., Eliott, J., Carolan, L., McCorry, F., Cebulla, A., et al.: Older adults’ perspectives of smart technologies to support aging at home: insights from five world cafe forums. Int. J. Environ. Res. Public Health 19 (7817), 1–22 (2022). https://doi.org/10.3390/Ijerph19137817

Ikkatai, Y., Hartwig, T., Takanashi, N., Yokoyama, H.M.: Octagon measurement: public attitudes toward AI ethics. Int J Hum-Comput Int. 38 (17), 1589–1606 (2022). https://doi.org/10.1080/10447318.2021.2009669

Wang, S., Bolling, K., Mao, W., Reichstadt, J., Jeste, D., Kim, H.C., Nebeker, C.: Technology to support aging in place: older adults’ perspectives. Healthcare (Basel) 7 (60), 1–18 (2019). https://doi.org/10.3390/healthcare7020060

Zhang, H., Lee, I., Ali, S., DiPaola, D., Cheng, Y.H., Breazeal, C.: Integrating ethics and career futures with technical learning to promote AI literacy for middle school students: an exploratory study. Int. J. Artif. Intell. Educ. 33 , 290–324 (2022). https://doi.org/10.1007/s40593-022-00293-3

Henriksen, A., Blond, L.: Executive-centered AI? Designing predictive systems for the public sector. Soc. Stud. Sci. (2023). https://doi.org/10.1177/03063127231163756

Nichol, A.A., Halley, M.C., Federico, C.A., Cho, M.K., Sankar, P.L.: Not in my AI: moral engagement and disengagement in health care AI development. Pac. Symp. Biocomput. 28 , 496–506 (2023)

Aquino, Y.S.J., Carter, S.M., Houssami, N., Braunack-Mayer, A., Win, K.T., Degeling, C., et al.: Practical, epistemic and normative implications of algorithmic bias in healthcare artificial intelligence: a qualitative study of multidisciplinary expert perspectives. J. Med. Ethics (2023). https://doi.org/10.1136/jme-2022-108850

Nichol, A.A., Bendavid, E., Mutenherwa, F., Patel, C., Cho, M.K.: Diverse experts’ perspectives on ethical issues of using machine learning to predict HIV/AIDS risk in sub-Saharan Africa: a modified Delphi study. BMJ Open 11 (7), e052287 (2021). https://doi.org/10.1136/bmjopen-2021-052287

Awad, E., Levine, S., Kleiman-Weiner, M., Dsouza, S., Tenenbaum, J.B., Shariff, A., et al.: Drivers are blamed more than their automated cars when both make mistakes. Nat. Hum. Behav. 4 (2), 134–143 (2020). https://doi.org/10.1038/s41562-019-0762-8

Blease, C., Kaptchuk, T.J., Bernstein, M.H., Mandl, K.D., Halamka, J.D., DesRoches, C.M.: Artificial intelligence and the future of primary care: exploratory qualitative study of UK general practitioners’ views. J. Med. Internet Res. 21 (3), e12802 (2019). https://doi.org/10.2196/12802

Blease, C., Locher, C., Leon-Carlyle, M., Doraiswamy, M.: Artificial intelligence and the future of psychiatry: qualitative findings from a global physician survey. Digit. Health 6 , 1–18 (2020). https://doi.org/10.1177/2055207620968355

De Graaf, M.M.A., Hindriks, F.A., Hindriks, K.V.: Who wants to grant robots rights? Front Robot AI 8 , 781985 (2022). https://doi.org/10.3389/frobt.2021.781985

Guerouaou, N., Vaiva, G., Aucouturier, J.-J.: The shallow of your smile: the ethics of expressive vocal deep-fakes. Philos. Trans. R Soc. B Biol. Sci. 377 (1841), 1–11 (2022). https://doi.org/10.1098/rstb.2021.0083

McCradden, M.D., Baba, A., Saha, A., Ahmad, S., Boparai, K., Fadaiefard, P., Cusimano, M.D.: Ethical concerns around use of artificial intelligence in health care research from the perspective of patients with meningioma, caregivers and health care providers: a qualitative study. CMAJ Open 8 (1), E90–E95 (2020). https://doi.org/10.9778/cmajo.20190151

Rogers, W.A., Draper, H., Carter, S.M.: Evaluation of artificial intelligence clinical applications: Detailed case analyses show value of healthcare ethics approach in identifying patient care issues. Bioethics 36 (4), 624–633 (2021). https://doi.org/10.1111/bioe.12885

Tosoni, S., Voruganti, I., Lajkosz, K., Habal, F., Murphy, P., Wong, R.K.S., et al.: The use of personal health information outside the circle of care: consent preferences of patients from an academic health care institution. BMC Med. Ethics 22 (29), 1–14 (2021). https://doi.org/10.1186/S12910-021-00598-3

Allahabadi, H., Amann, J., Balot, I., Beretta, A., Binkley, C., Bozenhard, J., et al.: Assessing trustworthy AI in times of COVID-19: deep learning for predicting a multiregional score conveying the degree of lung compromise in COVID-19 patients. IEEE Trans. Technol. Soc. 3 (4), 272–289 (2022). https://doi.org/10.1109/TTS.2022.3195114

Gray, K., Slavotinek, J., Dimaguila, G.L., Choo, D.: Artificial intelligence education for the health workforce: expert survey of approaches and needs. JMIR Med. Educ. 8 (2), e35223 (2022). https://doi.org/10.2196/35223

Alfrink, K., Keller, I., Doorn, N., Kortuem, G.: Tensions in transparent urban AI: designing a smart electric vehicle charge point. AI Soc. 38 (3), 1049–1065 (2022). https://doi.org/10.1007/s00146-022-01436-9

Bourla, A., Ferreri, F., Ogorzelec, L., Peretti, C.S., Guinchard, C., Mouchabac, S.: Psychiatrists’ attitudes toward disruptive new technologies: mixed-methods study. JMIR Ment. Health 5 (4), e10240 (2018). https://doi.org/10.2196/10240

Kopecky, R., Kosova, M.J., Novotny, D.D., Flegr, J., Cerny, D.: How virtue signalling makes us better: moral preferences with respect to autonomous vehicle type choices. AI Soc. 38 , 937–946 (2022). https://doi.org/10.1007/s00146-022-01461-8

Lam, K., Abramoff, M.D., Balibrea, J.M., Bishop, S.M., Brady, R.R., Callcut, R.A., et al.: A Delphi consensus statement for digital surgery. NPJ Digit. Med. 5 (100), 1–9 (2022). https://doi.org/10.1038/s41746-022-00641-6

Karaca, O., Çalışkan, S.A., Demir, K.: Medical artificial intelligence readiness scale for medical students (MAIRS-MS) – development, validity and reliability study. BMC Med. Educ. 21 (112), 1–9 (2021). https://doi.org/10.1186/s12909-021-02546-6

Papyshev, G., Yarime, M.: The limitation of ethics-based approaches to regulating artificial intelligence: regulatory gifting in the context of Russia. AI Soc. (2022). https://doi.org/10.1007/s00146-022-01611-y

Balaram, B., Greenham, T., Leonard, J.: Artificial intelligence: real public engagement. RSA, London. https://www.thersa.org/globalassets/pdfs/reports/rsa_artificial-intelligence---real-public-engagement.pdf (2018). Accessed 28 June 2023

Hagendorff, T.: A virtue-based framework to support putting AI ethics into practice. Philos Technol. 35 (55), 1–24 (2022). https://doi.org/10.1007/s13347-022-00553-z

Felt, U., Wynne, B., Callon, M., Gonçalves, M. E., Jasanoff, S., Jepsen, M., et al.: Taking european knowledge society seriously. Eur Comm, Brussels, 1–89 (2007). https://op.europa.eu/en/publication-detail/-/publication/5d0e77c7-2948-4ef5-aec7-bd18efe3c442/language-en

Michael, M.: Publics performing publics: of PiGs, PiPs and politics. Public Underst. Sci. 18 (5), 617–631 (2009). https://doi.org/10.1177/09636625080985

Hu, L.: Tech ethics: speaking ethics to power, or power speaking ethics? J. Soc. Comput. 2 (3), 238–248 (2021). https://doi.org/10.23919/JSC.2021.0033

Strasser, B., Baudry, J., Mahr, D., Sanchez, G., Tancoigne, E.: “Citizen science”? Rethinking science and public participation. Sci. Technol. Stud. 32 (2), 52–76 (2019). https://doi.org/10.23987/sts.60425

De Saille, S.: Innovating innovation policy: the emergence of ‘Responsible Research and Innovation.’ J. Responsible Innov. 2 (2), 152–168 (2015). https://doi.org/10.1080/23299460.2015.1045280

Schwarz-Plaschg, C.: Nanotechnology is like… The rhetorical roles of analogies in public engagement. Public Underst. Sci. 27 (2), 153–167 (2018). https://doi.org/10.1177/0963662516655686

Taylor, R.R., O’Dell, B., Murphy, J.W.: Human-centric AI: philosophical and community-centric considerations. AI Soc. (2023). https://doi.org/10.1007/s00146-023-01694-1

van Dijk, N., Tanas, A., Rommetveit, K., Raab, C.: Right engineering? The redesign of privacy and personal data protection. Int. Rev. Law Comput. Technol. 32 (2–3), 230–256 (2018). https://doi.org/10.1080/13600869.2018.1457002

Aradau, C., Blanke, T.: Algorithmic reason. The new government of self and others. Oxford University Press, Oxford (2022)

Book   Google Scholar  

McGoey, L.: The unknowers. How strategic ignorance rules the word. Zed, London (2019)

Ananny, M., Crawford, K.: Seeing without knowing: limitations of the transparency ideal and its application to algorithmic accountability. New Media Soc. 20 (3), 973–989 (2018). https://doi.org/10.1177/1461444816676645

Download references

Acknowledgements

The authors would like to express their gratitude to Rafaela Granja (CECS, University of Minho) for her insightful support in an early stage of preparation of this manuscript, and to the AIDA research netwrok for the inspiring debates.

Open access funding provided by FCT|FCCN (b-on). Helena Machado and Susana Silva did not receive funding to assist in the preparation of this work. Laura Neiva received funding from FCT—Fundação para a Ciência e a Tecnologia, I.P., under a PhD Research Studentships (ref.2020.04764.BD), and under the project UIDB/00736/2020 (base funding) and UIDP/00736/2020 (programmatic funding).

Author information

Authors and affiliations.

Department of Sociology, Institute for Social Sciences, University of Minho, Braga, Portugal

Helena Machado

Department of Sociology and Centre for Research in Anthropology (CRIA), Institute for Social Sciences, University of Minho, Braga, Portugal

Susana Silva

Institute for Social Sciences, Communication and Society Research Centre (CECS), University of Minho, Braga, Portugal

Laura Neiva

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by HM, SS, and LN. The first draft of the manuscript was written by HM and SS. All authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Helena Machado .

Ethics declarations

Conflict of interest.

The authors have no relevant financial or non-financial interests to disclose. The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 20 KB)

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Machado, H., Silva, S. & Neiva, L. Publics’ views on ethical challenges of artificial intelligence: a scoping review. AI Ethics (2023). https://doi.org/10.1007/s43681-023-00387-1

Download citation

Received : 08 October 2023

Accepted : 16 November 2023

Published : 19 December 2023

DOI : https://doi.org/10.1007/s43681-023-00387-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial intelligence
• Public involvement
• Publics’ views
• Responsible research
• Find a journal
• Publish with us
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Int J Qual Stud Health Well-being
• v.13(1); 2018

Doctoral students’ well-being: a literature review

Manuela schmidt.

a Faculty of Health Science, Kristianstad University, Kristianstad, Sweden

b Department of Health Science, Lund University, Lund, Sweden

Erika Hansson

c Faculty of Education, Kristianstad University, Kristianstad, Sweden

Purpose: Doctoral student well-being is an important matter that shapes the well-being of academics throughout their careers. Given that well-being has been found to be closely related to employee productivity and efficiency, strategies associated with maintaining well-being during PhD studies might be crucial for higher education, its outcomes and—just as importantly—for a balanced life of PhD students.

Method: Based on 17 studies, this literature review critically assesses the literature on doctoral student well-being.

Results: Theoretical models, concepts of well-being, and methods applied are discussed, as are the results of the articles. The reviewed studies are then discussed based on a SWOT analysis addressing the strengths and weaknesses of the reviewed research as well as the identified opportunities and threats, which can be used as a basis for future research. Based on the review findings and the SWOT analysis, a multidimensional view of the well-being of doctoral students is proposed.

Conclusions: The study proposes a more student-centred approach to meeting doctoral students’ needs, and the enhancement of doctoral student well-being in order, as a long-term goal, to improve academics’ well-being and productivity.

Introduction

Several studies suggest that academic staff develop strategies to maintain and enhance their well-being early in their academic careers (cf. Agevall, Broberg, & Umans, 2016 ; cf. Lease, 1999 ; Petersen, 2011 ; Salmela-Aro, Tolvanen, & Nurmi, 2011 ). These early years, i.e., the years spent on PhD studies, are usually associated with a transition from dependence to independence, i.e., from the student role to the professional academic role (Laudel & Gläser, 2008 ), and it is in the intersection of this transition, and its associated decisions and uncertainties, the future well-being of aspiring academics possibly develops (Schmidt & Umans, 2014 ; Stubb, Pyhältö, & Lonka, 2011 ).

Understanding the well-being of individuals in the work setting—where they spend most of their adult life in (Greenberg et al., 2003 )—is an emancipating endeavour to pursue (cf. Liu, Siu, & Shi, 2010 ). Usually, individuals’ well-being in work settings is closely related to organizational functioning. Being a key resource in higher education institutions (HEIs), academic staff, including doctoral students, play a major role in achieving the objectives of higher education and their performance affects student learning and success (de Lourdes Machado, Soares, Brites, Ferreira, & Gouveia, 2011 ), significantly influencing the success in any educational programme (Stankovska, Angelkoska, Osmani, & Grncarovska, 2017 ). However, academic staff have been identified over the years as the occupational group in HEIs that experience the most volatile well-being at work (e.g., Abouserie, 1996 ; Craig, Hancock, & Craig, 1996 ; Taris, Schreurs, & Van Iersel-Van Silfhout, 2001 ). Research investigating the well-being of academics is fragmented as well as limited when it comes to explaining the particular factors that contribute to this volatility (Kinman, 2008 ). Understanding the precursors of well-being in this occupational group is important given that the well-being of academics might affect their productivity in both research and teaching, ultimately influencing the quality of higher education (Vera, Salanova, & Martin, 2010 ). Poor well-being among those remaining in academia could be detrimental to their engagement in research and teaching, and might also imprint on the doctoral students they will supervise in the future. The well-being of this occupational group also has both short- and long-term consequences and might be an important enabler not only of educational quality but also of the sustainability of education systems. However, this calls for that the doctoral students, at the beginning of their career, are given the right tools to remain healthy in their work environment. Doctoral studies are often characterized by constant peer pressure, frequent evaluations, low status, high workload, paper deadlines, financial difficulties, pressure to publish, active participation in the scholarly environment, including conferences (Kurtz-Costes, Helmke, & Ulku-Steiner, 2006 ; Maysa & Smith, 2009 ), lack of permanent employment, and an uncertain future (Huisman, de Weert, & Bartelse, 2002 ). Feelings of uncertainty and poor relationships with supervisors (Lovitts, 2001 ) are additional stressors, as are the numerous roles doctoral student are expected to take, e.g., as a student, employee, parent, or researcher (Martinez, Ordu, Della Sala, & McFarlane, 2013 ; Schmidt & Umans, 2014 ). In light of the number of potential stressors and the complex work situation of doctoral students, it is a challenge for them to maintain a healthy work–life balance (Golde, 2005 ). Attrition rates are high, up to 50%, depending on doctoral discipline and country (Gardner, 2008 ; Jiranek, 2010 ; Lovitts & Nelson, 2000 ) and some leave academia after completing their doctoral programme, pursuing other careers. Furthermore, there is evidence that the scholarly communities do not always provide optimal opportunities for doctoral students to participate in. Instead, the milieu is perceived as burdensome by a number of doctoral students, which affects their well-being in a negative way (Stubb et al., 2011 ). There are also indications that doctoral students (especially women) suffer from stress and mental fatigue (Appel & Dahlgren, 2003 ).

Previous studies of the well-being of doctoral students, and of academic staff in general, have primarily concentrated on isolated determinants of well-being instead of taking a multidimensional perspective, which would allow consideration of multiple factors that interact with each other in simultaneously shaping well-being (Moberg, 1979 ). By reviewing the literature, this study aims to critically and systematically assess previous research on doctoral student well-being and give suggestions for future research by performing a SWOT analysis.

The concept of well-being

Well-being is a multifaceted phenomenon that has been studied in a number of different disciplines and thus has been defined in many different ways. Either due to or despite the multiplicity of defini-tions it has been described as a “catch-all category” (Cameron, Mathers, & Parry, 2006 , p. 347) that is still lacking an overall accepted definition (Seedhouse, 1995 ).

It is common to use “health” as a starting point in defining well-being, probably due to the World Health Organization which included well-being in its definition of health by declaring that “Health is a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity” (WHO, 1948 , p. 100). However, the definition of health is also rather problematic as it uses the unclear concept “well-being” in its definition, and has occasionally been criticized as utopian (Larson, 1999 ). Another take on well-being was presented by Galvin and Todres in their conceptual framework consisting of “the Dwelling-mobility lattice’’ (Galvin & Todres, 2011 ) stating that well-being—independent of health and illness—can be experienced spatially, temporally, inter-personally, bodily, in mood and in terms of the experience of personal identity. They state that well-being is more complex than health and is not limited to any setting or role, e.g., work place well-being or the role of being a student, thus their definition focuses on the essence of well-being (Galvin & Todres, 2011 ; Todres & Galvin, 2010 ). In the last decades two dominating perspectives of well-being have emerged: psychological (or eudaimonic) well-being that is concerned with the realization of a person’s true nature and potential; and subjective (or hedonic) well-being that is based on the general idea that happiness and pleasure form the essential goal of human life (Diener, 2018 ; Ryan & Deci, 2001 ; Ryff & Keyes, 1995 ). Both perspectives are relatively distinct and originate from different philosophical views, yet they overlap (Lundqvist, 2011 ).

While acknowledging the debate and the criticism on the definitions of well-being and health, the focus of this review is doctoral students’ well-being, and there is no intention to present yet another definition of the concept. Therefore, the starting point has been Medin and Alexanderson ( 2001 ), definition, which describes well-being as “ the individual’s experience of his or her health ” (p. 75). This comprehensive view of well-being, which highlights the individuals’ constantly changing experiences, is used as a framework for this study. In addition, the multidimensional and pragmatic approach to well-being presented by Ryff ( 1989 ) has been used as guidance, stressing the importance of positive relationships with others, personal growth, environmental autonomy, autonomy, purpose in life, as well as self-acceptance to maintain well-being.

Systematic literature review

For the purpose of this review, a systematic literature search was conducted in March 2018 of the following databases: Web of Science (all databases), ERIC, PsycInfo, and Education Research Complete. The search included the following keywords: well-being OR wellbeing OR “well being” AND “doctoral student*” OR “phd student*” OR “doctoral graduate*” as shown in Table I . The search was limited to the keywords in the abstracts (or topic in Web of Science) and no time limitations were chosen.

Search process and items found.

a Search “Topic” which included title, abstract and keywords.

b Document type: article.

Inclusion criteria and selection process

The eligibility criteria for the publications to be reviewed were: (1) inclusion of an empirical investigation, (2) specific focus on doctoral students and their well-being, (3) peer-reviewed and published in a scientific journal, (4) and written in English. Articles referring to postgraduate students were thoroughly examined because the term “postgraduate” refers to doctoral students in some educational systems but not in others. If it was clear that the author(s) were referring to master’s students, the articles were excluded.

In total, 68 articles were identified in the databases selected, as shown in Table I . After screening the abstracts/articles for relevance and excluding duplicates, 17 articles remained to be included in the literature review, which are presented in Table II . The excluded articles did not satisfy the inclusion criteria, i.e., though they were related (e.g., concerned with coping, resilience, or group writing) they had too little focus on well-being, or involved doctoral students only as a minority in the data collection.

Summary of the literature review.

Description of the findings

Of the reviewed articles, one was published before the year 2000, while the remaining 16 were published after 2010, reflecting growing interest in the field and possibly an increasing occurrence of problems in this occupational group. Most articles (11) were published in education or educational research journals. The remaining six articles were in the fields of psychology, general and internal medicine, management, health (i.e., public, environmental, and occupational), social sciences, and information science.

Data for six of the 17 studies were collected in the USA and/or Canada, another nine had data from Europe, one had data from Asia and one from an unspecified location (Kumar & Cavallaro, 2018 ). Of those 17 studies, four focused explicitly on female doctoral students and one on racio–ethnicity.

Concepts and measurements of well-being used in the studies

The authors of the studies included in the review conceptualized well-being in several different ways. Well-being or lack of it, is typically related to stress, loneliness, psychological distress, depression, and/or social support when viewed through a social/behavioural lens. Yet another perspective on well-being is more clinically based, stressing illness and physical conditions (Cotten, 2008 ). The studies reviewed here mainly emphasized viewing well-being from a social science perspective, educational research being part of it (Kuper & Kuper, 1996 ).

Juniper, Walsh, Richardson, and Morley ( 2012 ) for example, operationalized the concept of well-being prior to data collection, understanding doctoral student well-being in their quantitative study as “ that part of a researcher’s overall well-being that is primarily influenced by their PhD role and can be influenced by university-based interventions ” (p. 565). This definition is a modification of Juniper’s previous clinical work on the health-related quality of life (HRQL) of persons suffering from asthma (Juniper, 2005 ). HRQL, which is understood as a sense of well-being, should include good health, a secure social and occupational environment, financial security, spirituality, self-confidence, and strong, supportive relationships (Juniper, 2005 ). Juniper et al.’s definition of doctoral student well-being was thus derived from her previous definition of well-being as “that part of a patient’s overall well-being that is primarily determined by health and which can be influenced by healthcare interventions” (Juniper et al., 2012 , p. 564; Juniper, 2005 ). In both the clinical and non-clinical work, Juniper stresses the subjective experience of well-being and that all aspects of day to day functional life ought to be taken into consideration (Juniper et al., 2012 ; Juniper, 2005 ). Juniper assesses how work impacts on doctoral student well-being whereas her previous research investigated how disease impacts patient well-being. Based on the previous clinical work, Juniper et al. ( 2012 ) developed and evaluated a questionnaire that ultimately consisted of seven domains: development, facilities, home and health, research, social, supervisor, and university impacting on doctoral student well-being.

Pychyl and Little ( 1998 ) applied a concept of subjective well-being (SWB) which was operationalized in the quantitative part of their study using the Composite Affect Scale developed by Diener, Emmons, Larsen, and Griffin ( 1985 ), the Satisfaction with Life Scale by Diener et al. ( 1985 ), and domain-specific measures of subjective well-being by Palys and Little assessing life satisfaction in seven specific domains (Palys & Little, 1983 ). Diener views SWB as the person’s evaluation of his or her life (Diener, Napa Scollon, & Lucas, 2003 ), and uses SWB as the scientific term for happiness and life satisfaction (Edward Diener, 2018 ). He defines SWB in terms of two separate feelings, positive and negative affect (i.e., the presence of positive emotions and moods, and the absence of unpleasant affect), and satisfaction (e.g., with life, marriage or work) (Diener et al., 2003 ; Diener, Sapyta, & Suh, 1998 ).

Another scale developed by Diener et al. ( 2010 ) was used by Zahniser, Rupert, and Dorociak (Edward Diener, 2018 ). This Flourishing scale—previously referred to as the Psychological Well-being Scale—measures socio-psychological prosperity, focusing on social relationships—which are viewed as a complement of SWB. The term “flourishing” is understood to mean the presence of mental health, which according to Keyes is synonymous with SWB (Keyes, 2002 ) whereas Ryff and Singer ( 2000 ) developed a lifespan theory of human flourishing, understanding well-being as “the striving for perfection that represents the realization of one’s true potential” (Ryff, 1995 , p. 100).

Stubb et al. ( 2011 ) explored the concept of experienced socio-psychological well-being, referring to “doctoral students’ experience of their well-being in their scholarly community” (p. 35), by asking an open-ended question about the PhD student’s role in that community (see Table II ). They did however, similarly to other authors included in this review, adapt a version of the questionnaire NORD MED (Medical Education in Nordic Countries), which was developed for medical students, measuring different theoretical constructs, including motivation, learning and experiences of well-being (Lonka et al., 2008 ). Although well-being was not defined in the original article presenting NORD MED, it was measured by a total of 13 items, including questions regarding stress, exhaustion, lack of regulation, anxiety, and lack of interest (based on Elo, Leppänen, & Jahkola, 2003 ; Mäkinen, Olkinuora, & Lonka, 2004 ; Maslach & Jackson, 1981 ; Vermunt & Van Rijswijk, 1988 ). In the articles included in this review, 10 items were used to investigate doctoral student well-being, including one item question on stress, four item questions on exhaustion, two item questions on anxiety (the question “ I (often) have to force myself to work on my thesis” was reported to belong to different constructs such as a lack of interest scale, anxiety scale and cynicism scale depending on which of the three articles it was used in), and three item questions on lack of interest (Anttila, Lindblom-Ylänne, Lonka, & Pyhältö, 2015 ; Pyhältö & Keskinen, 2012 ; Stubb, Pyhältö, & Lonka, 2012 ). Yet another study investigated experienced well-being in terms of stress, exhaustion, and cynicism in PhD studies (Cornér, Löfström, & Pyhältö, 2017 ). Even though all questions used resemble the same questions used in modified NORD MED, this article refers to a Doctoral Experience Survey, which leads to the assumption that this may be yet another development of the modified NORD MED. Despite the more or less identical exploration of well-being in the above mentioned four articles, it is referred to as experienced socio-psychological well-being (Pyhältö & Keskinen, 2012 ), experienced well-being (Anttila et al., 2015 ; Stubb et al., 2012 ), and lack of well-being, i.e., burnout (Cornér et al., 2017 ). Yet, another perspective on experienced well-being may be Agency well-being , which is an adaptation of Sen’s capability approach based on the dimensions of agency, well-being, freedom and achievement (Sen, 1993 ) and can be understood as “as the success that individuals are having in the pursuit of their core personal projects” (Pychyl & Little, 1998 , p. 458), i.e., paying particular attention to the assessment of individual goals.

Caesens, Stinglhamber, and Luypaert ( 2014 ) used measures of perceived stress, job satisfaction, and sleeping problems to investigate well-being, while Hunter and Devine ( 2016 ) referred to the concept of emotional well-being without defining it, but clearly stating that they were attempting to understand it by examining emotional exhaustion as measured by the emotional exhaustion scale (Maslach & Jackson, 1981 ).

The qualitative study by Schmidt and Umans (Schmidt & Umans, 2014 ) tried to conceptualize doctoral student well-being using the metaphor of “white-water rafting” (Schmidt & Umans, 2014 , p. 10), seeing it as “ cramped in the interaction between self and structural forces " (p. 10), i.e., created through interaction between the self (“agent”) and the external factors (“structures”), which the doctoral student is a subject to. In another study, (Haynes et al., 2012 ) well-being was defined in terms of constitution, force, machine, measurement, and direction, concluding, similar to Schmidt and Umans ( 2014 ), that perceived well-being is an individual and social process that is constantly evolving and unique. The operationalization and conceptualization of well-being in the articles reviewed can be found in Table III .

Examples of the operationalization and conceptualization of well-being from the articles included in the review.

Triggers and outcomes of well-being.

It can be summarized that well-being, as mentioned earlier, is a complex yet well-used concept. It includes both narrow and broad definitions, is interpreted in various ways, and used differently. Furthermore, well-being also often seems to be studied by focusing on the lack of well-being such as stress, burnout and sleep problems.

Theoretical models used in the studies

Theoretical models were found to be used as a basis for the theoretical or analytical frameworks of the reviewed papers. Several studies used theory as a basis for their frameworks. Hunter and Devine ( 2016 ) used the leader–member exchange theoretical perspective to understand the supervisor–doctoral student relationship. Caesens et al. ( 2014 ) used self-determination theory to understand the extrinsic motivation that drives workaholics, applying Higgins’s regulatory focus theory to demonstrate the prevention focus of workaholics. The study further explored the job demands–resources theoretical model, used to describe how job resources, namely, social support, can constitute a positive motivational process that enhances work engagement. Conservation of resources theory was used to understand the relationship between social support and workaholism. The theories used in the paper established a basis for several hypotheses arranged to form a theoretical framework. A conceptual framework was also proposed by Kumar and Cavallaro ( 2018 ) explaining how intertwined the researcher, i.e., the doctoral student, and the research itself are, and how the research process depletes well-being. Pychyl and Little ( 1998 ) and Stubb et al. ( 2012 ) used the social ecological model and the broaden-and-build theory of emotions , respectively, to explain the antecedents of doctoral student well-being.

Juniper et al. ( 2012 ) applied impact analysis —which was previously used to assess well-being in a clinical setting by developing a questionnaire—as a methodological framework when developing a questionnaire for doctoral students covering seven domains (such as home & health, or supervisor), which comprise their well-being.

In the article by Pyhältö and Keskinen ( 2012 ), the concept of relational agency developed by Edwards ( 2005 ) was applied to understand the capacity of doctoral students to work with others to resolve problems by identifying motives, interpreting them and adapting their responses accordingly. The concept is closely related to agency theory, which is mainly used in business studies (Eisenhardt, 1989 ). Agency theory tries to explain relationships between principals and agents, and to show how problems can be resolved by aligning each other’s goals and interests by means of different incentives. However, while relational agency focuses on the interactive nature of the relationship between principle and agent, agency theory also includes non-relational aspects such as opportunistic behaviour, asymmetry of information and risk aspects (Smith, Umans, & Thomasson, 2018 ). However, another article refers to agency in terms of “agency well-being” (Little, 2012 ) when discussing the results of the study.

Another study explicitly stated that they applied theory in discussing their results, namely, Giddens’ structuration theory (Schmidt & Umans, 2014 ) which was used when analyzing the emergence of doctoral student well-being in the interaction between the agent and structure. In addition, other studies used black feminist thought (Shavers & Moore, 2014 ), phenomenological hermeneutics (Schmidt & Umans, 2014 ), constructivism (Haynes et al., 2012 ), the interpretivist paradigm (Martinez et al., 2013 ), and/or grounded theory (Martinez et al., 2013 ; Pychyl & Little, 1998 ) as their overall analytical frameworks. When choosing grounded theory, which two studies did, the ultimate aim was to develop theory based on the findings.

Methods used in the articles

Of the 17 reviewed studies, eight combined qualitative and quantitative data collection methods, whereas five applied a qualitative and four a quantitative design. Studies combining both qualitative and quantitative data used one or two open-ended questions in questionnaires (Zahniser, Rupert, & Dorociak, 2017 ; Anttila et al., 2015 ; Hunter & Devine, 2016 ; Pyhältö & Keskinen, 2012 ; Stubb et al., 2011 , 2012 ). The remainder combined questionnaires with interviews/focus groups (Juniper et al., 2012 ; Pychyl & Little, 1998 ), which was also chosen as the preferred method among the purely qualitative articles (Haynes et al., 2012 ; Martinez et al., 2013 ; Schmidt & Umans, 2014 ; Shavers & Moore, 2014 ) with the exception of one article, which applied auto ethnography (Kumar & Cavallaro, 2018 ).

The qualitative data were analyzed using various analytical tools, such as thematic analysis (Hunter & Devine, 2016 ), the interpretive perspective (Shavers & Moore, 2014 ), the lifeworld concept (Schmidt & Umans, 2014 ), abductive or thematic content analysis (Anttila et al., 2015 ; Hunter & Devine, 2016 ; Pyhältö & Keskinen, 2012 ; Stubb et al., 2011 , 2012 ), grounded theory (Martinez et al., 2013 ; Pychyl & Little, 1998 ), metaphorical analysis (Haynes et al., 2012 ), constant comparative method (Martinez et al., 2013 ) and retrospective analysis of own experiences (Kumar & Cavallaro, 2018 ). Quantitative data was analyzed by applying descriptive or comparative statistical methods (Herrmann & Wichmann-Hansen, 2017 ; Pychyl & Little, 1998 ; Pyhältö & Keskinen, 2012 ; Stubb et al., 2011 , 2012 ) as well as variance analysis such as ANOVA (Hunter & Devine, 2016 ; Juniper et al., 2012 ; Stubb et al., 2012 ; Ziapour, Khatony, Jafari, & Kianipour, 2017 ), correlation (Pychyl & Little, 1998 ; Stubb et al., 2012 ), and regression (Hunter & Devine, 2016 ; Pychyl & Little, 1998 ).

Results of the articles

Several articles described doctoral student well-being as related to terms such as self, agent, being true to oneself (Schmidt & Umans, 2014 ), an individual process (Haynes et al., 2012 ), time for self (Martinez et al., 2013 ) or self-care (Zahniser, et al., 2017 ; Kumar & Cavallaro, 2018 ), the private self and protection of self (Shavers & Moore, 2014 ), and internal reflection or an intuitive process focusing on the individual (Haynes et al., 2012 ), often resulting in various internal battles. These battles or struggles manifest themselves in terms of role conflicts (Pychyl & Little, 1998 ) or internal conflicts (Martinez et al., 2013 ), conflicting responsibilities and priorities (Martinez et al., 2013 ), trade-offs (Martinez et al., 2013 ), or conflicting goals (Haynes et al., 2012 ).

Also the meaning doctoral students attribute to their PhD education,—viewing it as a process or a product or both—affects well-being and has been shown to vary among academic disciplines (Stubb et al., 2012 ). In line with those results, Shavers and Moore ( 2014 ) found that overemphasizing academic growth at the expense of emotional and personal development will lead to a lack of wholeness and centredness. Several studies also reported high frequencies of doctoral students considering interrupting their studies. In one study, 56% considered dropping out at some point during the PhD process, and that decision was influenced by experiences of stress, anxiety, exhaustion, and lack of interest (Anttila et al., 2015 ). Yet another study reported that 43% of the sample considered interrupting their studies (Stubb et al., 2011 ). Experiences of burnout increased the risk of dropping out, while receiving supervision from several supervisors decreased this risk (Cornér et al., 2017 ). The notion of doctoral students’ intending to leave academia after completion of the PhD was supported by the study by Hunter and Devine (Hunter & Devine, 2016 ). About one third of the sample intended leaving academia, which correlated with experiences of well-being in terms of emotional exhaustion during the PhD process. Intention to leave academia after completion of the thesis was higher among students belonging to the hard applied and soft applied disciplines (Hunter & Devine, 2016 ). Variation in well-being were also found to be related to work condition, i.e., full-time students and those partially belonging to a research group reported higher levels of well-being (Stubb et al., 2011 ).

Furthermore, one study identified personality traits as having an impact on doctoral student well-being. Pychyl and Little ( 1998 ) demonstrated that neuroticism correlated positively with a negative affect, and extraversion with a positive affect. Pychyl and Little ( 1998 ) further identified feelings of guilt and anxiety as contributors to stress. The existence of feelings of guilt and frustration was reaffirmed by the study of Schmidt and Umans ( 2014 ).

Coping ability is yet another central aspect of doctoral student well-being. For these students, coping mechanisms are necessary to manage stress and to maintain sanity, physical health, and mental well-being—that is, to remain healthy (Martinez et al., 2013 ). People can respond to stressors in many different ways, for example, working to solve the problem (i.e., problem-focused coping) or reaching out for social support (Carver & Connor-Smith, 2010 ; Lazarus & Folkman, 1984 ). The results of the review indicate a strong emphasis on social support as a way of coping.

Crying, isolation, and social interactions with friends all served as coping strategies for the studied doctoral students (Martinez et al., 2013 ). One study identified “cling[ing] to the spiritual realm” (p. 9) as a coping strategy and found that success in developing coping strategies conferred a certain sense of control (Haynes et al., 2012 ). In addition, planning (i.e., problem-focused coping), and exercise (Martinez et al., 2013 ) were mentioned as coping mechanisms specific to doctoral students.

Shavers and Moore ( 2014 ) found that doctoral students used coping strategies to overcome oppression and to help them persevere academically. An identified coping strategy involved shifting between different selves and using an academic mask; yet, instead of maintaining well-being and fostering optimal, healthy coping, this strategy was categorized as survival-oriented, and using it led to feelings of incompleteness, disconnectedness, and exhaustion. Peer relationships (Schmidt & Umans, 2014 ), passion, and social support (Pychyl & Little, 1998 ) were other identified coping resources used by doctoral students.

Yet another strategy with a particular focus on health prevention was mentioned. Self-care according to the Professional Self-care Scale for Psychologists by Dorociak comprises professional support, cognitive awareness, professional development, life balance, and daily balance. All these aspects have been shown to increase well-being, however the first two aspects are of particular importance (Zahniser, et al., 2017 ). Another study reported self-reflection, yoga, social network support, biking or walking, and compartmentalization as examples of self-care strategies (Kumar & Cavallaro, 2018 ), stressing in their conceptual framework that individual driven self-care and promotion of self-care by the institution are of equal importance.

Several articles described doctoral student well-being as related to structural forces (Schmidt & Umans, 2014 ), outside forces (Haynes et al., 2012 ), external reflection, and social factors (Haynes et al., 2012 ) as well as being in the sphere of others (Schmidt & Umans, 2014 ). It consists of personal and academic social interactions, for example, with the spouse and family (Martinez et al., 2013 ; Schmidt & Umans, 2014 ), supervisors (Caesens et al., 2014 ; Cornér et al., 2017 ; Hunter & Devine, 2016 ; B. Juniper et al., 2012 ; Schmidt & Umans, 2014 ), faculty and the university at large (Caesens et al., 2014 ; Hunter & Devine, 2016 ; Juniper et al., 2012 ; Martinez et al., 2013 ; Schmidt & Umans, 2014 ; Zahniser, et al., 2017), and the scholarly community (Cornér et al., 2017 ; Hunter & Devine, 2016 ; Schmidt & Umans, 2014 ; Stubb et al., 2011 ). Such interactions also relate to social support in general (Juniper et al., 2012 ; Kumar & Cavallaro, 2018 ; Martinez et al., 2013 ; Pychyl & Little, 1998 ). Attention was mainly paid to the social processes created by interacting with external actors. Finally, one study found that organizational support and supervisor support were positively related to work engagement (Caesens et al., 2014 ), which in turn had positive effects on well-being, illustrating once again the complexity of the concepts involved.

Several circumstances mentioned in the studies can be summarized as stressors, some of them chronic. Deadlines, limited finances, time, family issues, and relationships were all mentioned as stressors. Another stressor was the need to take on additional responsibilities to position oneself after graduation, while competing commitments led to less enjoyment, motivation issues, problems finishing the dissertation, and ambiguity (Martinez et al., 2013 ). Managing stress was described as a balancing act, in which the high expectations of various actors, and domestic demands when living a dual life (i.e., being a “superwoman”) had to be balanced to keep stress manageable (Schmidt & Umans, 2014 ). Lack of control (Haynes et al., 2012 ; Schmidt & Umans, 2014 ) was yet another stressor affecting doctoral students’ work, well-being, and health. Pychyl and Little ( 1998 ) identified time pressure, time conflicts, and procrastination as stressors.

Some factors could be attributed a dual function: for example, relationships, supervisors, and the scholarly community could all provide support and function as coping mechanisms at times, yet at other times could also be seen as stressors. Examples of how doctoral students' well-being can be influenced and understood is shown in Table IV .

The interaction between the self and external forces is where one’s unique well-being constantly evolves (Haynes et al., 2012 ; Schmidt & Umans, 2014 ; Stubb et al., 2011 ). Yet, when influenced by external forces, well-being can rapidly develop into an upward or downward spiral. If the work–life balance (Zahniser, et al., 2017 ; Haynes et al., 2012 ; Martinez et al., 2013 ; Pychyl & Little, 1998 ) cannot be maintained, this will ultimately affect the doctoral students’ well-being and produce spill-over effects on their lives more generally.

Inspired by Jackson, Joshi, and Erhardt ( 2003 ), the reviewed studies were subjected to a SWOT analysis, identifying the strengths and weaknesses of the research as well as the opportunities and threats. The analysis will be used as a basis for suggestions for future research (Schmidt, 2018 ).

One strength of the reviewed studies is that most were published after 2010, providing a rather recent view of the situation of doctoral students. Another strength is the number of suggestions made and practical implications identified. Despite Golde’s ( 2005 ) comment that research has failed to address how doctoral education could be improved, almost all the reviewed studies attempted to apply their findings, for example, by developing optimal resistance strategies to enhance well-being, such as teaching doctoral students to affirm themselves daily and develop positive thinking patterns (Shavers & Moore, 2014 ); evaluating and/or developing policies addressing, for example, academic climate or discrimination in PhD programmes (Schmidt & Umans, 2014 ; Shavers & Moore, 2014 ); creating an arena for shared meaning using supervisory contracts (Stubb et al., 2012 ); fostering peer groups as important and meaningful communities for students (Stubb et al., 2011 ); organizing health and wellness biofeedback labs, recreational sports groups and fitness classes, and seminars on time management (Haynes et al., 2012 ); training supervisors in mentoring and supervision, and creating a structured model to help advisers provide feedback, in terms of both academic research and relationship management (Hunter & Devine, 2016 ).

Another strength is the wide sample variation of the articles included (ranging from 2 to 1.760 doctoral students), applying a various number of methodological approaches and study designs. Also, the diverse PhD student body is explored by the inclusion of various academic disciplines such as biology, business administration, health sciences, nursing, informatics, and public health (Schmidt & Umans, 2014 ), humanities, medicine and behavioural sciences (Pyhältö & Keskinen, 2012 ; Stubb et al., 2011 , 2012 ), humanities and theology, natural sciences and engineering, social sciences and law, behavioural sciences, economics, and medicine (Cornér et al., 2017 ), education, chemistry, and agriculture (Hunter & Devine, 2016 ), art and social sciences (Pychyl & Little, 1998 ), and psychology (Zahniser, et al., 2017 ). However, it should be acknowledged that only a few studies address potential differences arising from this diversity.

One weakness of the reviewed literature concerns the problematic matter of defining “well-being”, which may create confusion by referring to different concepts, such as emotional well-being, subjective well-being, psychological well-being, socio-psychological well-being, and agency well-being. Well-being is also operationalized in different ways in the various studies and correlated with various other social or health-related concepts such as social support, work engagement, or personality traits, i.e., well-being is used as an input measure, output measure, mediator, and moderator, making it difficult to discern clear causal relationships. Instead, the intertwined relationships create a spider’s web of interactions between all the elements, indicating the complex nature of doctoral student well-being. Most of the studies are inconsistent when it comes to the use of well-being and health concepts, which at times have similar meanings. While for some, well-being is viewed as a central component of health (Martinez et al., 2013 ), including the World Health Organization’s frequently used definition of 1948 (WHO, 1948 ), for others, well-being is defined as something greater than health (Galvin & Todres, 2011 ; Todres & Galvin, 2010 ). Thus, well-being can be understood as a source of health, and vice versa. The review further showed that meaning and meaningfulness are central attributes of doctoral students’ well-being, as shown by Stubb et al. ( 2012 ) and Pychyl and Little ( 1998 ). Yet another recurrent aspect in the definitions used is the component of social network/support that serves an important function in PhD student life. Overall, the results of the review resonate well with Ryff’s ( 1989 ) holistic definition which highlights the importance of positive relationships with others, personal mastery, autonomy, a feeling of purpose and meaning in life, as well as personal growth and development.

However, some divergence within the definitions remains. For example, Ziapour et al. (Paloutzian & Park, 2015 ; Ziapour et al., 2017 ) view existential well-being as being part of spiritual well-being, which emphasizes the sense of life purpose and life satisfaction (Ellison, 1983 ). Yet the same term is given another meaning by others (Dahlberg, Todres, & Galvin, 2009 ; Todres & Galvin, 2010 ). Todres and Galvin define their existential view of well-being (also referred to as the existential theory of well-being) as well-being as a whole before it is structured or categorized into different domains, and refer to well-being as an essential experience that makes all other kinds of well-being possible in its various forms. Cohen, Mount, Tomas, and Mount ( 1996 ) on the other hand, include existential well-being in their quality of life scale, implying that—in a clinical setting—it is of more importance for patients with a life-threatening illness. Yalom ( 1980 ), who includes death, freedom, isolation and meaning in the existential domain, Cohen et al. ( 1996 ) measure existential well-being by asking six questions (such as whether they have achieved their life goals or how they feel about themselves).

Yet another weakness is that most reviewed studies collected their data in Europe, the USA, and Canada, omitting the perspective of the developing countries.

Opportunities

Much of the reviewed research into doctoral student well-being was conducted in the field of education. One way of further developing the research field would be to expand it to include fields such as psychology, the social sciences, management, and the caring sciences. Theories and models from these fields could help improve our understanding of the complexity of doctoral students’ situations, experiences, and use of suitable coping strategies. They might also improve our understanding of these students’ needs, which, if they are met, would improve their education experience, well-being, and future success and engagement in academia.

Another opportunity would be to use various methods to study all PhD programmes in order to evaluate satisfaction and quality levels from the doctoral student’s perspective. Today, most emphasis is on the academi c progress of the student, measured in numbers of publications or conference appearances. Monitoring not only academic progress but also doctoral students’ well-being could lead to changes at the systemic, institutional level. If doctoral students experience a lack of well-being and cannot maintain a healthy work–life balance during the lengthy period of their PhD studies, and might even consider dropping out, this represents a loss for everyone involved. Related to this attrition are economic costs (i.e., waste of departmental, institutional, state, and personal resources), psychosocial costs (i.e., social and emotional costs to students and faculty from loss of invested time and effort and impaired productivity in research projects) (Golde, 2005 ), and opportunity costs to both the doctoral student and the PhD funder.

Well-developed strategies such as social or problem-focused coping have been shown to be effective for people experiencing stress (Carver & Connor-Smith, 2010 ). One opportunity to advance research in this area would be to investigate how those strategies could be applied by doctoral students, possibly leading to enhanced well-being. Because this review has found that social support as a coping strategy has been greatly emphasized, more research attention should be paid to problem-focused coping.

Majority of the papers reviewed appear to adopt the hedonic perspective of well- being (e.g., subjective well-being) which provides a potential focus of the eudemonic perspective (e.g., psychological well-being) or combination of both for future research.

Finally, a further possible opportunity could be a focus on gender and other socio-demographic diversities. The reviewed studies are dominated by the experiences of female PhD students, with some of the studies only accounting for women (Haynes et al., 2012 ; Kumar & Cavallaro, 2018 ; Schmidt & Umans, 2014 ; Shavers & Moore, 2014 ). In countries such as Sweden, the distribution of gender among PhD students is rather even (46% women) but there are vast differences depending on academic discipline, with the greatest variation in technology and agriculture, where female doctoral students account for 27% and 60% respectively (Statistics Sweden, 2016 ). In countries such as the USA, women account for 44% of the PhDs awarded (Monroe, Ozyurt, Wrigley, & Alexander, 2008 ) while in Finland 66% of the PhD students in humanities are female, 76% in behavioural sciences, and 71% in medicine (Stubb et al., 2011 ). These numbers show differences between countries as well as between academic disciplines. As men and women may react and respond differently to triggers such as supervisor support, loneliness and stress, it could be important to give equal attention to both genders.

The reviewed studies considered many academic disciplines, such as the humanities, medicine, engineering, and law, all of which apply different paradigms and have different research traditions. The experiences of doctoral students vary widely from discipline to discipline (Golde & Dore, 2001 ). This makes comparing PhD programmes difficult because they differ in many ways, for example, in course requirements, supervisor involvement, and teaching assignments. Stubb et al. ( 2012 ) reported that the experienced meaning of PhD studies as well as the reasons for interrupting studies differed between faculties. Although a national-level review including all disciplines might be advisable to eliminate discrepancies in the quality of doctoral research, programmes, and student well-being, it is believed that harmonizing PhD programmes within disciplines, within countries, or worldwide would not necessarily enhance doctoral student well-being.

Well-being is a multifaceted concept and a single generally accepted definition of well-being is lacking (Seedhouse, 1995 ). It is therefore not surprising, although rather problematic, that well-being is described in such different ways in the reviewed studies. There also seems to be confusion in the occupational health field, where well-being is subdivided to the workplace, the social environment and economics, when a more comprehensive approach would be more valuable. Doctoral student well-being might be multidimensional and not limited to a particular setting or role; instead, the present results clearly indicate that it should be studied more comprehensively.

Shavers and Moore ( 2014 ) concluded in their article that well-being and academic perseverance cannot coexist simultaneously. Though this review revealed that doctoral students face multiple challenges, it also identified a need for increased awareness of the basic nature of research as a highly challenging endeavour whose progress is unpredictable and nonlinear (Juniper et al., 2012 )—as is how doctoral student’s emotions and abilities impact their well-being and PhD work process. HEIs are advised to apply a more student-centred approach when interacting their doctoral students, which could increase the likelihood of these students maintaining their well-being during their PhD studies and, in the long term, maintaining the sustainability of the HEIs.

Limitations

This review is not without limitations. First, it must be emphasized that the literature review and the interpretation of the findings are subjective in nature. Second, the search was not limited to any context, location, discipline or time frame, it may be incomplete since four databases were used. Yet, the choice of databases was strategical and was reflected upon prior to conducting the search. Third, a limited number of articles were included in the review. However, a systematic search with inclusion criteria that focused on securing a certain level of quality (e.g., peer review, empirics, English) might have decreased the quantity but as the authors carefully read all abstracts/articles found, excluded articles independently, and compared the individual results until agreement was reached, the validity of the review was increased. The exclusion of dissertations was due to the inclusion criteria that required peer review.

Finally, the choice of keywords was elaborated on, and several different writings were included but the searches may not have been exhaustive. Other possible keywords for use in the keyword search, such as job satisfaction, were rejected because job satisfaction only covers work-related factors not aligned with the purpose of this study, which emphasizes well-being as a multifaceted concept rather than singling out components of well-being or certain settings. Because well-being research addresses diverse concepts such as depression, euphoria, global judgments of life satisfaction (Diener et al., 2003 ) and stress, all negative and positive experiences of well-being are included here to cover as many dimensions as possible of the concept. A key-word search focusing exclusively on the negative aspects, for example, stress, burnout, and exhaustion, was accordingly also rejected.

Acknowledgments

We wish to thank Giuseppe Grossi, Kristianstad University, for comments that improved the manuscript. This research was financially supported by the research environment Governance, Regulation, Internationalization and Performance (GRIP), Kristianstad University.

Disclosure statement

No potential conflict of interest was reported by the authors.

• Abouserie R. (1996). Stress, coping strategies and job satisfaction in university academic staff . Educational Psychology , 16 ( 1 ), 49–14. [ Google Scholar ]
• Agevall L., Broberg P., & Umans T. (2016). The new generation of auditors meeting praxis: Dual learning’s role in audit students’ professional development . Scandinavian Journal of Educational Research , 62 , 1–18. [ Google Scholar ]
• Anttila H., Lindblom-Ylänne S., Lonka K., & Pyhältö K. (2015). The added value of a PhD in medicine-PhD Students’ perceptions of acquired competences . International Journal of Higher Education , 4 ( 2 ), 172. [ Google Scholar ]
• Appel M. L., & Dahlgren L. G. (2003). Swedish doctoral students’ experiences on their journey towards a PhD: Obstacles and opportunities inside and outside the academic building . Scandinavian Journal of Educational Research , 47 ( 1 ), 89–110. [ Google Scholar ]
• Caesens G., Stinglhamber F., & Luypaert G. (2014). The impact of work engagement and workaholism on well-being: The role of work-related social support . Career Development International , 19 ( 7 ), 813–835. [ Google Scholar ]
• Cameron E., Mathers J., & Parry J. (2006). ‘Health and well-being’: Questioning the use of health concepts in public health policy and practice . Critical Public Health , 16 ( 4 ), 347–354. [ Google Scholar ]
• Carver C. S., & Connor-Smith J. (2010). Personality and coping . Annual Review of Psychology , 61 , 679–704. [ PubMed ] [ Google Scholar ]
• Cohen S. R., Mount B. M., Tomas J. J., & Mount L. F. (1996). Existential well-being is an important determinant of quality of life. Evidence from the McGill quality of life questionnaire . Cancer , 77 ( 3 ), 576–586. [ PubMed ] [ Google Scholar ]
• Cornér S., Löfström E., & Pyhältö K. (2017). The relationship between Doctoral students’ perceptions of supervision and burnout . International Journal of Doctoral Studies . doi: 10.28945/3754 [ CrossRef ] [ Google Scholar ]
• Cotten S. R. (2008). Students’ technology use and the impacts on well‐being . New Directions for Student Services , (2008 ( 124 ), 55–70. [ Google Scholar ]
• Craig A., Hancock K., & Craig M. (1996). The lifestyle appraisal questionnaire: A comprehensive assessment of health and stress . Psychology & Health , 11 ( 3 ), 331–343. [ Google Scholar ]
• Dahlberg K., Todres L., & Galvin K. (2009). Lifeworld-led healthcare is more than patient-led care: An existential view of well-being . Medicine, Health Care and Philosophy , 12 ( 3 ), 265–271. [ PubMed ] [ Google Scholar ]
• de Lourdes Machado M., Soares V. M., Brites R., Ferreira J. B., & Gouveia O. M. R. (2011). A look to academics job satisfaction and motivation in Portuguese higher education institutions . Procedia-Social and Behavioral Sciences , 29 , 1715–1724. [ Google Scholar ]
• Diener E. (2018). Happiness: The science of subjective well-being In Biswas-Diener R. & Diener E. (Eds.), Noba textbook series: Psychology . Champaign, IL: DEF publishers. [ Google Scholar ]
• Diener E., Emmons R. A., Larsen R. J., & Griffin S. (1985). The satisfaction with life scale . Journal of Personality Assessment , 49 ( 1 ), 71–75. [ PubMed ] [ Google Scholar ]
• Diener E., Napa Scollon C., & Lucas R. E. (2003). The evolving concept of subjective well-being: The multifaceted nature of happiness . Advances in Cell Aging and Gerontology , 15 , 187–219. [ Google Scholar ]
• Diener E., Sapyta J., & Suh E. (1998). Subjective well-being is essential to wellbeing . Psychological Inquiry , 9 , 33–37. [ Google Scholar ]
• Diener E., Wirtz D., Tov W., Kim-Prieto C., Choi D.-W., Oishi S., & Biswas-Diener R. (2010). New well-being measures: Short scales to assess flourishing and positive and negative feelings . Social Indicators Research , 97 ( 2 ), 143–156. [ Google Scholar ]
• Edwards A. (2005). Relational agency: Learning to be a resourceful practitioner . International Journal of Educa-tional Research , 43 ( 3 ), 168–182. [ Google Scholar ]
• Eisenhardt K. M. (1989). Agency theory: An assessment and review . Academy of Management Review , 14 ( 1 ), 57–74. [ Google Scholar ]
• Ellison C. W. (1983). Spiritual well-being: Conceptualization and measurement . Journal of Psychology and Theology , 11 ( 4 ), 330–338. [ Google Scholar ]
• Elo A.-L., Leppänen A., & Jahkola A. (2003). Validity of a single-item measure of stress symptoms . Scandinavian Journal of Work, Environment & Health , 444–451. doi: 10.5271/sjweh.752 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Galvin K., & Todres L. (2011). Kinds of well-being: A conceptual framework that provides direction for caring . International Journal of Qualitative Studies on Health and Well-Being , 6 ( 4 ), 10362. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Gardner S. K. (2008). “What’s too much and what’s too little?”: The process of becoming an independent researcher in doctoral education . The Journal of Higher Education , 79 ( 3 ), 326–350. [ Google Scholar ]
• Golde C. M. (2005). The role of the department and discipline in doctoral student attrition: Lessons from four departments . The Journal of Higher Education , 76 ( 6 ), 669–670. [ Google Scholar ]
• Golde C. M., & Dore T. M. (2001). At cross purposes: What the experiences of today’s doctoral students reveal about doctoral education . [ Google Scholar ]
• Greenberg P. E., Kessler R. C., Birnbaum H. G., Leong S. A., Lowe S. W., Berglund P. A., & Corey-Lisle P. K. (2003). The economic burden of depression in the United States: How did it change between 1990 and 2000? Journal of Clinical Psychiatry , 64 ( 12 ), 1465–1475. [ PubMed ] [ Google Scholar ]
• Haynes C., Bulosan M., Citty J., Grant-Harris M., Hudson J. C., & Koro-Ljungberg M. (2012). My world is not my doctoral program… or is it?: Female students’ perceptions of well-being . International Journal of Doctoral Studies , 7 , 1–17. [ Google Scholar ]
• Herrmann K. J., & Wichmann-Hansen G. (2017). Validation of the quality in PhD processes questionnaire . Studies in Graduate and Postdoctoral Education , 8 ( 2 ), 189–204. [ Google Scholar ]
• Huisman J., de Weert E., & Bartelse J. (2002). Academic careers from a European perspective: The declining desirability of the faculty position . The Journal of Higher Education , 73 ( 1 ), 141–160. [ Google Scholar ]
• Hunter K. H., & Devine K. (2016). Doctoral students’ emotional exhaustion and intentions to leave academia . International Journal of Doctoral Studies , 11 , 35–61. [ Google Scholar ]
• Jackson S. E., Joshi A., & Erhardt N. L. (2003). Recent research on team and organizational diversity: SWOT analysis and implications . Journal of Management , 29 ( 6 ), 801–830. [ Google Scholar ]
• Jiranek V. (2010). Potential predictors of timely completion among dissertation research students at an Australian faculty of sciences . International Journal of Doctoral Studies , 5 ( 1 ), 1–13. [ Google Scholar ]
• Juniper B., Walsh E., Richardson A., & Morley B. (2012). A new approach to evaluating the well-being of PhD research students . Assessment & Evaluation in Higher Education , 37 ( 5 ), 563–576. [ Google Scholar ]
• Juniper E. (2005). Assessing asthma quality of life: Its role in clinical practice . Breathe , 1 ( 3 ), 192–204. [ Google Scholar ]
• Keyes C. L. (2002). The mental health continuum: From languishing to flourishing in life . Journal of Health and Social Behavior , 207–222. doi: 10.2307/3090197 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Kinman G. (2008). Work stressors, health and sense of coherence in UK academic employees . Educational Psychology , 28 ( 7 ), 823–835. [ Google Scholar ]
• Kumar S., & Cavallaro L. (2018). Researcher self-care in emotionally demanding research: A proposed conceptual framework . Qualitative Health Research , 28 ( 4 ), 648–658. [ PubMed ] [ Google Scholar ]
• Kuper A., & Kuper J. (1996). The social science encyclopedia (2nd ed.). London and New York: Routledge. [ Google Scholar ]
• Kurtz-Costes B., Helmke L. A., & Ulku-Steiner B. (2006). Gender and doctoral studies: The perceptions of Ph.D. students in an American university. Gender and Education , 18 ( 2 ), 137–155. [ Google Scholar ]
• Larson J. S. (1999). The conceptualization of health . Medical Care Research and Review , 56 ( 2 ), 123–136. [ PubMed ] [ Google Scholar ]
• Laudel G., & Gläser J. (2008). From apprentice to colleague: The metamorphosis of early career researchers . Higher Education , 55 ( 3 ), 387–406. [ Google Scholar ]
• Lazarus R., & Folkman S. (1984). Stress, appraisal, and coping . New York: Springer. [ Google Scholar ]
• Lease S. H. (1999). Occupational role stressors, coping, support, and hardiness as predictors of strain in academic faculty: An emphasis on new and female faculty . Research in Higher Education , 40 ( 3 ), 285–307. [ Google Scholar ]
• Little B. R. (2012). Free traits and personal contexts: Expanding a social ecological model of well-being In Walsh W. B., Craik K. H., & Price R. H. (Eds.), Personal-environment psychology-new directions and perspectives (2nd ed.). New York: routledge. [ Google Scholar ]
• Liu J., Siu O. L., & Shi K. (2010). Transformational leadership and employee well‐being: The mediating role of trust in the leader and self‐efficacy . Applied Psychology , 59 ( 3 ), 454–479. [ Google Scholar ]
• Lonka K., Sharafi P., Karlgren K., Masiello I., Nieminen J., Birgegård G., & Josephson A. (2008). MED NORD–A tool for measuring medical students’ well-being and study orientations . Medical Teacher , 30 ( 1 ), 72–79. [ PubMed ] [ Google Scholar ]
• Lovitts B. E. (2001). Leaving the ivory tower: The causes and consequences of departure from doctoral study . Boston, USA: Rowman & Littlefield Publishers, Inc. [ Google Scholar ]
• Lovitts B. E., & Nelson C. (2000). The hidden crisis in graduate education: Attrition from Ph.D. programs . Academe , 86 ( 6 ), 44. [ Google Scholar ]
• Lundqvist C. (2011). Well-being in competitive sports—The feel-good factor? A review of conceptual considerations of well-being . International Review of Sport and Exercise Psychology , 4 ( 2 ), 109–127. [ Google Scholar ]
• Mäkinen J., Olkinuora E., & Lonka K. (2004). Students at risk: Students’ general study orientations and abandoning/prolonging the course of studies . Higher Education , 48 ( 2 ), 173–188. [ Google Scholar ]
• Martinez E., Ordu C., Della Sala M. R., & McFarlane A. (2013). Striving to obtain a school-work-life balance: The full-time doctoral student . International Journal of Doctoral Studies , 8 ( 39–59 ). doi: 10.28945/1765 [ CrossRef ] [ Google Scholar ]
• Maslach C., & Jackson S. E. (1981). The measurement of experienced burnout . Journal of Organizational Behavior , 2 ( 2 ), 99–113. [ Google Scholar ]
• Maysa T. L., & Smith B. T. (2009). Navigating the doctoral journey . Journal of Hospital Librarianship , 9 ( 4 ), 345–361. [ Google Scholar ]
• Medin J., & Alexanderson K. (2001). [Health and healh promotion – A literature study] . Lund, Swedish: Studentlitteratur. [ Google Scholar ]
• Moberg D. O. (1979). The development of social indicators for quality of life research . Sociology of Religion , 40 ( 1 ), 11–26. [ Google Scholar ]
• Monroe K., Ozyurt S., Wrigley T., & Alexander A. (2008). Gender equality in academia: Bad news from the trenches, and some possible solutions . Perspectives on Politics (June , 2008 , 215–233. [ Google Scholar ]
• Preamble to the Constitution of WHO as adopted by the International Health Conference , New York, 19 June – 22 July 1946; signed on 22 July 1946 by the representatives of 61 States (Official Records of WHO, no. 2, p. 100 and entered into force on 7 April 1948) (1948).
• Paloutzian R. F., & Park C. L. (2015). Handbook of the psychology of religion and spirituality (2nd ed.). New York: The Guilford Press. [ Google Scholar ]
• Palys T. S., & Little B. R. (1983). Perceived life satisfaction and the organization of personal project systems . Journal of Personality and Social Psychology , 44 ( 6 ), 1221. [ Google Scholar ]
• Petersen E. B. (2011). Staying or going?: Australian early career researchers’ narratives of academic work, exit options and coping strategies . The Australian Universities’ Review , 53 ( 2 ), 34. [ Google Scholar ]
• Pychyl T. A., & Little B. R. (1998). Dimensional specificity in the prediction of subjective well-being: Personal projects in pursuit of the PhD . Social Indicators Research , 45 ( 1 ), 423–473. [ Google Scholar ]
• Pyhältö K., & Keskinen J. (2012). Doctoral students’ sense of relational agency in their scholarly communities . International Journal of Higher Education , 1 ( 2 ), 136–149. [ Google Scholar ]
• Ryan R. M., & Deci E. L. (2001). To be happy or to be self-fulfilled: A review of research on hedonic and eudaimonic well-being . Annual Review of Psychology , 52 , 141–166. [ PubMed ] [ Google Scholar ]
• Ryff C. D. (1989). Happiness is everything, or is it? Explorations on the meaning of psychological well-being . Journal of Personality and Social Psychology , 57 ( 6 ), 1069–1081. [ Google Scholar ]
• Ryff C. D. (1995). Psychological well-being in adult life . Current Directions in Psychological Science , 4 , 99–104. [ Google Scholar ]
• Ryff C. D., & Keyes C. L. M. (1995). The structure of psychological well-being revisited . Journal of Personality and Social Psychology , 69 ( 4 ), 719. [ PubMed ] [ Google Scholar ]
• Ryff C. D., & Singer B. (2000). Interpersonal flourishing: A positive health agenda for the new millennium . Personality and Social Psychology Review , 4 ( 1 ), 30–44. [ Google Scholar ]
• Salmela-Aro K., Tolvanen A., & Nurmi J.-E. (2011). Social strategies during university studies predict early career work burnout and engagement: 18-year longitudinal study . Journal of Vocational Behavior , 79 ( 1 ), 145–157. [ Google Scholar ]
• Schmidt M. (2018). Frequent visitors at the psychiatric emergency room–A literature review . Psychiatric Quarterly , 89 ( 1 ), 11–32. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Schmidt M., & Umans T. (2014). Experiences of well-being among female doctoral students in Sweden . International Journal of Qualitative Studies on Health and Well-Being , 9 ( 1 ). doi: 10.3402/qhw.v9.23059 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Seedhouse D. (1995). ‘Well-being’: Health promotion’s red herring . Health Promotion International , 10 ( 1 ), 61–67. [ Google Scholar ]
• Sen A. (1993). Capability and well-being In Nussbaum M. & Sen A. (Eds.), The quality of life (pp. 30–53). New York: Oxford University Press. [ Google Scholar ]
• Shavers M. C., & Moore J. L. III (2014). Black female voices: Self-presentation strategies in doctoral programs at predominately White institutions . Journal of College Student Development , 55 ( 4 ), 391–407. [ Google Scholar ]
• Smith E., Umans T., & Thomasson A. (2018). Stages of PPP and principal–Agent conflicts: The Swedish water and sewerage sector . Public Performance & Management Review , 41 ( 1 ), 100–129. [ Google Scholar ]
• Stankovska G., Angelkoska S., Osmani F., & Grncarovska S. P. (2017) Job motivation and job satisfaction among academic staff in higher education . Paper presented at the Paper prepared for the Annual International Conference of the Bulgarian Comparative Education Society (BCES) and the International Partner Conference of the International Research Centre (IRC) Bulgaria. [ Google Scholar ]
• Stubb J., Pyhältö K., & Lonka K. (2011). Balancing between inspiration and exhaustion: PhD students’ experienced socio-psychological well-being . Studies in Continuing Education , 33 ( 1 ), 33–50. [ Google Scholar ]
• Stubb J., Pyhältö K., & Lonka K. (2012). The experienced meaning of working with a PhD Thesis . Scandinavian Journal of Educational Research , 56 ( 4 ), 439–456. [ Google Scholar ]
• Statistics Sweden (2016). Doctoral students and degrees at third cycle studies ( Report: UF 21 SM 1701 )
• Taris T. W., Schreurs P. J. G., & Van Iersel-Van Silfhout I. J. (2001). Job stress, job strain, and psychological withdrawal among Dutch university staff: Towards a dualprocess model for the effects of occupational stress . Work and Stress , 15 ( 4 ), 283–296. [ Google Scholar ]
• Todres L., & Galvin K. (2010). “Dwelling-mobility”: An existential theory of well-being . International Journal of Qualitative Studies on Health and Well-Being , 5 ( 3 ), p 5444. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Vera M., Salanova M., & Martin B. (2010). University faculty and work-related well-being: The importance of the triple work profile . Electronic Journal of Research in Educational Psychology , 8 ( 2 ), 581–602. [ Google Scholar ]
• Vermunt J. D., & Van Rijswijk F. A. (1988). Analysis and development of students’ skill in selfregulated learning . Higher Education , 17 ( 6 ), 647–682. [ Google Scholar ]
• Yalom I. D. (1980). Existential Psychotherapy . New York: Basic Books. [ Google Scholar ]
• Zahniser E., Rupert P. A., & Dorociak K. E. (2017). Self-care in clinical psychology graduate training . Training and Education in Professional Psychology , 11 ( 4 ), 283. [ Google Scholar ]
• Ziapour A., Khatony A., Jafari F., & Kianipour N. (2017). Prediction of the dimensions of the spiritual well-being of students at Kermanshah university of medical sciences, Iran: The roles of demographic variables . Journal of Clinical and Diagnostic Research: JCDR , 11 ( 7 ), VC05. [ PMC free article ] [ PubMed ] [ Google Scholar ]