how to assess research proposal

Organizing Your Social Sciences Research Assignments

• Annotated Bibliography
• Analyzing a Scholarly Journal Article
• Group Presentations
• Dealing with Nervousness
• Using Visual Aids
• Grading Someone Else's Paper
• Types of Structured Group Activities
• Group Project Survival Skills
• Leading a Class Discussion
• Multiple Book Review Essay
• Reviewing Collected Works
• Writing a Case Analysis Paper
• Writing a Case Study
• About Informed Consent
• Writing Field Notes
• Writing a Policy Memo
• Writing a Reflective Paper
• Writing a Research Proposal
• Generative AI and Writing
• Acknowledgments

The goal of a research proposal is twofold: to present and justify the need to study a research problem and to present the practical ways in which the proposed study should be conducted. The design elements and procedures for conducting research are governed by standards of the predominant discipline in which the problem resides, therefore, the guidelines for research proposals are more exacting and less formal than a general project proposal. Research proposals contain extensive literature reviews. They must provide persuasive evidence that a need exists for the proposed study. In addition to providing a rationale, a proposal describes detailed methodology for conducting the research consistent with requirements of the professional or academic field and a statement on anticipated outcomes and benefits derived from the study's completion.

Krathwohl, David R. How to Prepare a Dissertation Proposal: Suggestions for Students in Education and the Social and Behavioral Sciences . Syracuse, NY: Syracuse University Press, 2005.

How to Approach Writing a Research Proposal

Your professor may assign the task of writing a research proposal for the following reasons:

• Develop your skills in thinking about and designing a comprehensive research study;
• Learn how to conduct a comprehensive review of the literature to determine that the research problem has not been adequately addressed or has been answered ineffectively and, in so doing, become better at locating pertinent scholarship related to your topic;
• Improve your general research and writing skills;
• Practice identifying the logical steps that must be taken to accomplish one's research goals;
• Critically review, examine, and consider the use of different methods for gathering and analyzing data related to the research problem; and,
• Nurture a sense of inquisitiveness within yourself and to help see yourself as an active participant in the process of conducting scholarly research.

A proposal should contain all the key elements involved in designing a completed research study, with sufficient information that allows readers to assess the validity and usefulness of your proposed study. The only elements missing from a research proposal are the findings of the study and your analysis of those findings. Finally, an effective proposal is judged on the quality of your writing and, therefore, it is important that your proposal is coherent, clear, and compelling.

Regardless of the research problem you are investigating and the methodology you choose, all research proposals must address the following questions:

• What do you plan to accomplish? Be clear and succinct in defining the research problem and what it is you are proposing to investigate.
• Why do you want to do the research? In addition to detailing your research design, you also must conduct a thorough review of the literature and provide convincing evidence that it is a topic worthy of in-depth study. A successful research proposal must answer the "So What?" question.
• How are you going to conduct the research? Be sure that what you propose is doable. If you're having difficulty formulating a research problem to propose investigating, go here for strategies in developing a problem to study.

Common Mistakes to Avoid

• Failure to be concise . A research proposal must be focused and not be "all over the map" or diverge into unrelated tangents without a clear sense of purpose.
• Failure to cite landmark works in your literature review . Proposals should be grounded in foundational research that lays a foundation for understanding the development and scope of the the topic and its relevance.
• Failure to delimit the contextual scope of your research [e.g., time, place, people, etc.]. As with any research paper, your proposed study must inform the reader how and in what ways the study will frame the problem.
• Failure to develop a coherent and persuasive argument for the proposed research . This is critical. In many workplace settings, the research proposal is a formal document intended to argue for why a study should be funded.
• Sloppy or imprecise writing, or poor grammar . Although a research proposal does not represent a completed research study, there is still an expectation that it is well-written and follows the style and rules of good academic writing.
• Too much detail on minor issues, but not enough detail on major issues . Your proposal should focus on only a few key research questions in order to support the argument that the research needs to be conducted. Minor issues, even if valid, can be mentioned but they should not dominate the overall narrative.

Procter, Margaret. The Academic Proposal.  The Lab Report. University College Writing Centre. University of Toronto; Sanford, Keith. Information for Students: Writing a Research Proposal. Baylor University; Wong, Paul T. P. How to Write a Research Proposal. International Network on Personal Meaning. Trinity Western University; Writing Academic Proposals: Conferences, Articles, and Books. The Writing Lab and The OWL. Purdue University; Writing a Research Proposal. University Library. University of Illinois at Urbana-Champaign.

Structure and Writing Style

Beginning the Proposal Process

As with writing most college-level academic papers, research proposals are generally organized the same way throughout most social science disciplines. The text of proposals generally vary in length between ten and thirty-five pages, followed by the list of references. However, before you begin, read the assignment carefully and, if anything seems unclear, ask your professor whether there are any specific requirements for organizing and writing the proposal.

A good place to begin is to ask yourself a series of questions:

• What do I want to study?
• Why is the topic important?
• How is it significant within the subject areas covered in my class?
• What problems will it help solve?
• How does it build upon [and hopefully go beyond] research already conducted on the topic?
• What exactly should I plan to do, and can I get it done in the time available?

In general, a compelling research proposal should document your knowledge of the topic and demonstrate your enthusiasm for conducting the study. Approach it with the intention of leaving your readers feeling like, "Wow, that's an exciting idea and I can’t wait to see how it turns out!"

Most proposals should include the following sections:

I.  Introduction

In the real world of higher education, a research proposal is most often written by scholars seeking grant funding for a research project or it's the first step in getting approval to write a doctoral dissertation. Even if this is just a course assignment, treat your introduction as the initial pitch of an idea based on a thorough examination of the significance of a research problem. After reading the introduction, your readers should not only have an understanding of what you want to do, but they should also be able to gain a sense of your passion for the topic and to be excited about the study's possible outcomes. Note that most proposals do not include an abstract [summary] before the introduction.

Think about your introduction as a narrative written in two to four paragraphs that succinctly answers the following four questions :

• What is the central research problem?
• What is the topic of study related to that research problem?
• What methods should be used to analyze the research problem?
• Answer the "So What?" question by explaining why this is important research, what is its significance, and why should someone reading the proposal care about the outcomes of the proposed study?

II.  Background and Significance

This is where you explain the scope and context of your proposal and describe in detail why it's important. It can be melded into your introduction or you can create a separate section to help with the organization and narrative flow of your proposal. Approach writing this section with the thought that you can’t assume your readers will know as much about the research problem as you do. Note that this section is not an essay going over everything you have learned about the topic; instead, you must choose what is most relevant in explaining the aims of your research.

To that end, while there are no prescribed rules for establishing the significance of your proposed study, you should attempt to address some or all of the following:

• State the research problem and give a more detailed explanation about the purpose of the study than what you stated in the introduction. This is particularly important if the problem is complex or multifaceted .
• Present the rationale of your proposed study and clearly indicate why it is worth doing; be sure to answer the "So What? question [i.e., why should anyone care?].
• Describe the major issues or problems examined by your research. This can be in the form of questions to be addressed. Be sure to note how your proposed study builds on previous assumptions about the research problem.
• Explain the methods you plan to use for conducting your research. Clearly identify the key sources you intend to use and explain how they will contribute to your analysis of the topic.
• Describe the boundaries of your proposed research in order to provide a clear focus. Where appropriate, state not only what you plan to study, but what aspects of the research problem will be excluded from the study.
• If necessary, provide definitions of key concepts, theories, or terms.

III.  Literature Review

Connected to the background and significance of your study is a section of your proposal devoted to a more deliberate review and synthesis of prior studies related to the research problem under investigation . The purpose here is to place your project within the larger whole of what is currently being explored, while at the same time, demonstrating to your readers that your work is original and innovative. Think about what questions other researchers have asked, what methodological approaches they have used, and what is your understanding of their findings and, when stated, their recommendations. Also pay attention to any suggestions for further research.

Since a literature review is information dense, it is crucial that this section is intelligently structured to enable a reader to grasp the key arguments underpinning your proposed study in relation to the arguments put forth by other researchers. A good strategy is to break the literature into "conceptual categories" [themes] rather than systematically or chronologically describing groups of materials one at a time. Note that conceptual categories generally reveal themselves after you have read most of the pertinent literature on your topic so adding new categories is an on-going process of discovery as you review more studies. How do you know you've covered the key conceptual categories underlying the research literature? Generally, you can have confidence that all of the significant conceptual categories have been identified if you start to see repetition in the conclusions or recommendations that are being made.

NOTE: Do not shy away from challenging the conclusions made in prior research as a basis for supporting the need for your proposal. Assess what you believe is missing and state how previous research has failed to adequately examine the issue that your study addresses. Highlighting the problematic conclusions strengthens your proposal. For more information on writing literature reviews, GO HERE .

To help frame your proposal's review of prior research, consider the "five C’s" of writing a literature review:

• Cite , so as to keep the primary focus on the literature pertinent to your research problem.
• Compare the various arguments, theories, methodologies, and findings expressed in the literature: what do the authors agree on? Who applies similar approaches to analyzing the research problem?
• Contrast the various arguments, themes, methodologies, approaches, and controversies expressed in the literature: describe what are the major areas of disagreement, controversy, or debate among scholars?
• Critique the literature: Which arguments are more persuasive, and why? Which approaches, findings, and methodologies seem most reliable, valid, or appropriate, and why? Pay attention to the verbs you use to describe what an author says/does [e.g., asserts, demonstrates, argues, etc.].
• Connect the literature to your own area of research and investigation: how does your own work draw upon, depart from, synthesize, or add a new perspective to what has been said in the literature?

IV.  Research Design and Methods

This section must be well-written and logically organized because you are not actually doing the research, yet, your reader must have confidence that you have a plan worth pursuing . The reader will never have a study outcome from which to evaluate whether your methodological choices were the correct ones. Thus, the objective here is to convince the reader that your overall research design and proposed methods of analysis will correctly address the problem and that the methods will provide the means to effectively interpret the potential results. Your design and methods should be unmistakably tied to the specific aims of your study.

Describe the overall research design by building upon and drawing examples from your review of the literature. Consider not only methods that other researchers have used, but methods of data gathering that have not been used but perhaps could be. Be specific about the methodological approaches you plan to undertake to obtain information, the techniques you would use to analyze the data, and the tests of external validity to which you commit yourself [i.e., the trustworthiness by which you can generalize from your study to other people, places, events, and/or periods of time].

When describing the methods you will use, be sure to cover the following:

• Specify the research process you will undertake and the way you will interpret the results obtained in relation to the research problem. Don't just describe what you intend to achieve from applying the methods you choose, but state how you will spend your time while applying these methods [e.g., coding text from interviews to find statements about the need to change school curriculum; running a regression to determine if there is a relationship between campaign advertising on social media sites and election outcomes in Europe ].
• Keep in mind that the methodology is not just a list of tasks; it is a deliberate argument as to why techniques for gathering information add up to the best way to investigate the research problem. This is an important point because the mere listing of tasks to be performed does not demonstrate that, collectively, they effectively address the research problem. Be sure you clearly explain this.
• Anticipate and acknowledge any potential barriers and pitfalls in carrying out your research design and explain how you plan to address them. No method applied to research in the social and behavioral sciences is perfect, so you need to describe where you believe challenges may exist in obtaining data or accessing information. It's always better to acknowledge this than to have it brought up by your professor!

V.  Preliminary Suppositions and Implications

Just because you don't have to actually conduct the study and analyze the results, doesn't mean you can skip talking about the analytical process and potential implications . The purpose of this section is to argue how and in what ways you believe your research will refine, revise, or extend existing knowledge in the subject area under investigation. Depending on the aims and objectives of your study, describe how the anticipated results will impact future scholarly research, theory, practice, forms of interventions, or policy making. Note that such discussions may have either substantive [a potential new policy], theoretical [a potential new understanding], or methodological [a potential new way of analyzing] significance.   When thinking about the potential implications of your study, ask the following questions:

• What might the results mean in regards to challenging the theoretical framework and underlying assumptions that support the study?
• What suggestions for subsequent research could arise from the potential outcomes of the study?
• What will the results mean to practitioners in the natural settings of their workplace, organization, or community?
• Will the results influence programs, methods, and/or forms of intervention?
• How might the results contribute to the solution of social, economic, or other types of problems?
• Will the results influence policy decisions?
• In what way do individuals or groups benefit should your study be pursued?
• What will be improved or changed as a result of the proposed research?
• How will the results of the study be implemented and what innovations or transformative insights could emerge from the process of implementation?

NOTE:   This section should not delve into idle speculation, opinion, or be formulated on the basis of unclear evidence . The purpose is to reflect upon gaps or understudied areas of the current literature and describe how your proposed research contributes to a new understanding of the research problem should the study be implemented as designed.

ANOTHER NOTE : This section is also where you describe any potential limitations to your proposed study. While it is impossible to highlight all potential limitations because the study has yet to be conducted, you still must tell the reader where and in what form impediments may arise and how you plan to address them.

VI.  Conclusion

The conclusion reiterates the importance or significance of your proposal and provides a brief summary of the entire study . This section should be only one or two paragraphs long, emphasizing why the research problem is worth investigating, why your research study is unique, and how it should advance existing knowledge.

Someone reading this section should come away with an understanding of:

• Why the study should be done;
• The specific purpose of the study and the research questions it attempts to answer;
• The decision for why the research design and methods used where chosen over other options;
• The potential implications emerging from your proposed study of the research problem; and
• A sense of how your study fits within the broader scholarship about the research problem.

VII.  Citations

As with any scholarly research paper, you must cite the sources you used . In a standard research proposal, this section can take two forms, so consult with your professor about which one is preferred.

• References -- a list of only the sources you actually used in creating your proposal.
• Bibliography -- a list of everything you used in creating your proposal, along with additional citations to any key sources relevant to understanding the research problem.

In either case, this section should testify to the fact that you did enough preparatory work to ensure the project will complement and not just duplicate the efforts of other researchers. It demonstrates to the reader that you have a thorough understanding of prior research on the topic.

Most proposal formats have you start a new page and use the heading "References" or "Bibliography" centered at the top of the page. Cited works should always use a standard format that follows the writing style advised by the discipline of your course [e.g., education=APA; history=Chicago] or that is preferred by your professor. This section normally does not count towards the total page length of your research proposal.

Develop a Research Proposal: Writing the Proposal. Office of Library Information Services. Baltimore County Public Schools; Heath, M. Teresa Pereira and Caroline Tynan. “Crafting a Research Proposal.” The Marketing Review 10 (Summer 2010): 147-168; Jones, Mark. “Writing a Research Proposal.” In MasterClass in Geography Education: Transforming Teaching and Learning . Graham Butt, editor. (New York: Bloomsbury Academic, 2015), pp. 113-127; Juni, Muhamad Hanafiah. “Writing a Research Proposal.” International Journal of Public Health and Clinical Sciences 1 (September/October 2014): 229-240; Krathwohl, David R. How to Prepare a Dissertation Proposal: Suggestions for Students in Education and the Social and Behavioral Sciences . Syracuse, NY: Syracuse University Press, 2005; Procter, Margaret. The Academic Proposal. The Lab Report. University College Writing Centre. University of Toronto; Punch, Keith and Wayne McGowan. "Developing and Writing a Research Proposal." In From Postgraduate to Social Scientist: A Guide to Key Skills . Nigel Gilbert, ed. (Thousand Oaks, CA: Sage, 2006), 59-81; Wong, Paul T. P. How to Write a Research Proposal. International Network on Personal Meaning. Trinity Western University; Writing Academic Proposals: Conferences , Articles, and Books. The Writing Lab and The OWL. Purdue University; Writing a Research Proposal. University Library. University of Illinois at Urbana-Champaign.

• << Previous: Writing a Reflective Paper
• Next: Generative AI and Writing >>
• Last Updated: Mar 6, 2024 1:00 PM
• URL: https://libguides.usc.edu/writingguide/assignments

• Search Menu
• Advance articles
• Author Guidelines
• Submission Site
• Open Access
• Why Publish?
• About Science and Public Policy
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

1. introduction, 2. background, 4. findings, 5. discussion, 6. conclusion and final remarks, supplementary material, data availability, conflict of interest statement., acknowledgements.

• < Previous

Evaluation of research proposals by peer review panels: broader panels for broader assessments?

• Article contents
• Figures & tables
• Supplementary Data

Rebecca Abma-Schouten, Joey Gijbels, Wendy Reijmerink, Ingeborg Meijer, Evaluation of research proposals by peer review panels: broader panels for broader assessments?, Science and Public Policy , Volume 50, Issue 4, August 2023, Pages 619–632, https://doi.org/10.1093/scipol/scad009

• Permissions Icon Permissions

Panel peer review is widely used to decide which research proposals receive funding. Through this exploratory observational study at two large biomedical and health research funders in the Netherlands, we gain insight into how scientific quality and societal relevance are discussed in panel meetings. We explore, in ten review panel meetings of biomedical and health funding programmes, how panel composition and formal assessment criteria affect the arguments used. We observe that more scientific arguments are used than arguments related to societal relevance and expected impact. Also, more diverse panels result in a wider range of arguments, largely for the benefit of arguments related to societal relevance and impact. We discuss how funders can contribute to the quality of peer review by creating a shared conceptual framework that better defines research quality and societal relevance. We also contribute to a further understanding of the role of diverse peer review panels.

Scientific biomedical and health research is often supported by project or programme grants from public funding agencies such as governmental research funders and charities. Research funders primarily rely on peer review, often a combination of independent written review and discussion in a peer review panel, to inform their funding decisions. Peer review panels have the difficult task of integrating and balancing the various assessment criteria to select and rank the eligible proposals. With the increasing emphasis on societal benefit and being responsive to societal needs, the assessment of research proposals ought to include broader assessment criteria, including both scientific quality and societal relevance, and a broader perspective on relevant peers. This results in new practices of including non-scientific peers in review panels ( Del Carmen Calatrava Moreno et al. 2019 ; Den Oudendammer et al. 2019 ; Van den Brink et al. 2016 ). Relevant peers, in the context of biomedical and health research, include, for example, health-care professionals, (healthcare) policymakers, and patients as the (end-)users of research.

Currently, in scientific and grey literature, much attention is paid to what legitimate criteria are and to deficiencies in the peer review process, for example, focusing on the role of chance and the difficulty of assessing interdisciplinary or ‘blue sky’ research ( Langfeldt 2006 ; Roumbanis 2021a ). Our research primarily builds upon the work of Lamont (2009) , Huutoniemi (2012) , and Kolarz et al. (2016) . Their work articulates how the discourse in peer review panels can be understood by giving insight into disciplinary assessment cultures and social dynamics, as well as how panel members define and value concepts such as scientific excellence, interdisciplinarity, and societal impact. At the same time, there is little empirical work on what actually is discussed in peer review meetings and to what extent this is related to the specific objectives of the research funding programme. Such observational work is especially lacking in the biomedical and health domain.

The aim of our exploratory study is to learn what arguments panel members use in a review meeting when assessing research proposals in biomedical and health research programmes. We explore how arguments used in peer review panels are affected by (1) the formal assessment criteria and (2) the inclusion of non-scientific peers in review panels, also called (end-)users of research, societal stakeholders, or societal actors. We add to the existing literature by focusing on the actual arguments used in peer review assessment in practice.

To this end, we observed ten panel meetings in a variety of eight biomedical and health research programmes at two large research funders in the Netherlands: the governmental research funder The Netherlands Organisation for Health Research and Development (ZonMw) and the charitable research funder the Dutch Heart Foundation (DHF). Our first research question focuses on what arguments panel members use when assessing research proposals in a review meeting. The second examines to what extent these arguments correspond with the formal −as described in the programme brochure and assessment form− criteria on scientific quality and societal impact creation. The third question focuses on how arguments used differ between panel members with different perspectives.

2.1 Relation between science and society

To understand the dual focus of scientific quality and societal relevance in research funding, a theoretical understanding and a practical operationalisation of the relation between science and society are needed. The conceptualisation of this relationship affects both who are perceived as relevant peers in the review process and the criteria by which research proposals are assessed.

The relationship between science and society is not constant over time nor static, yet a relation that is much debated. Scientific knowledge can have a huge impact on societies, either intended or unintended. Vice versa, the social environment and structure in which science takes place influence the rate of development, the topics of interest, and the content of science. However, the second part of this inter-relatedness between science and society generally receives less attention ( Merton 1968 ; Weingart 1999 ).

From a historical perspective, scientific and technological progress contributed to the view that science was valuable on its own account and that science and the scientist stood independent of society. While this protected science from unwarranted political influence, societal disengagement with science resulted in less authority by science and debate about its contribution to society. This interdependence and mutual influence contributed to a modern view of science in which knowledge development is valued both on its own merit and for its impact on, and interaction with, society. As such, societal factors and problems are important drivers for scientific research. This warrants that the relation and boundaries between science, society, and politics need to be organised and constantly reinforced and reiterated ( Merton 1968 ; Shapin 2008 ; Weingart 1999 ).

Glerup and Horst (2014) conceptualise the value of science to society and the role of society in science in four rationalities that reflect different justifications for their relation and thus also for who is responsible for (assessing) the societal value of science. The rationalities are arranged along two axes: one is related to the internal or external regulation of science and the other is related to either the process or the outcome of science as the object of steering. The first two rationalities of Reflexivity and Demarcation focus on internal regulation in the scientific community. Reflexivity focuses on the outcome. Central is that science, and thus, scientists should learn from societal problems and provide solutions. Demarcation focuses on the process: science should continuously question its own motives and methods. The latter two rationalities of Contribution and Integration focus on external regulation. The core of the outcome-oriented Contribution rationality is that scientists do not necessarily see themselves as ‘working for the public good’. Science should thus be regulated by society to ensure that outcomes are useful. The central idea of the process-oriented Integration rationality is that societal actors should be involved in science in order to influence the direction of research.

Research funders can be seen as external or societal regulators of science. They can focus on organising the process of science, Integration, or on scientific outcomes that function as solutions for societal challenges, Contribution. In the Contribution perspective, a funder could enhance outside (societal) involvement in science to ensure that scientists take responsibility to deliver results that are needed and used by society. From Integration follows that actors from science and society need to work together in order to produce the best results. In this perspective, there is a lack of integration between science and society and more collaboration and dialogue are needed to develop a new kind of integrative responsibility ( Glerup and Horst 2014 ). This argues for the inclusion of other types of evaluators in research assessment. In reality, these rationalities are not mutually exclusive and also not strictly separated. As a consequence, multiple rationalities can be recognised in the reasoning of scientists and in the policies of research funders today.

2.2 Criteria for research quality and societal relevance

The rationalities of Glerup and Horst have consequences for which language is used to discuss societal relevance and impact in research proposals. Even though the main ingredients are quite similar, as a consequence of the coexisting rationalities in science, societal aspects can be defined and operationalised in different ways ( Alla et al. 2017 ). In the definition of societal impact by Reed, emphasis is placed on the outcome : the contribution to society. It includes the significance for society, the size of potential impact, and the reach , the number of people or organisations benefiting from the expected outcomes ( Reed et al. 2021 ). Other models and definitions focus more on the process of science and its interaction with society. Spaapen and Van Drooge introduced productive interactions in the assessment of societal impact, highlighting a direct contact between researchers and other actors. A key idea is that the interaction in different domains leads to impact in different domains ( Meijer 2012 ; Spaapen and Van Drooge 2011 ). Definitions that focus on the process often refer to societal impact as (1) something that can take place in distinguishable societal domains, (2) something that needs to be actively pursued, and (3) something that requires interactions with societal stakeholders (or users of research) ( Hughes and Kitson 2012 ; Spaapen and Van Drooge 2011 ).

Glerup and Horst show that process and outcome-oriented aspects can be combined in the operationalisation of criteria for assessing research proposals on societal aspects. Also, the funders participating in this study include the outcome—the value created in different domains—and the process—productive interactions with stakeholders—in their formal assessment criteria for societal relevance and impact. Different labels are used for these criteria, such as societal relevance , societal quality , and societal impact ( Abma-Schouten 2017 ; Reijmerink and Oortwijn 2017 ). In this paper, we use societal relevance or societal relevance and impact .

Scientific quality in research assessment frequently refers to all aspects and activities in the study that contribute to the validity and reliability of the research results and that contribute to the integrity and quality of the research process itself. The criteria commonly include the relevance of the proposal for the funding programme, the scientific relevance, originality, innovativeness, methodology, and feasibility ( Abdoul et al. 2012 ). Several studies demonstrated that quality is seen as not only a rich concept but also a complex concept in which excellence and innovativeness, methodological aspects, engagement of stakeholders, multidisciplinary collaboration, and societal relevance all play a role ( Geurts 2016 ; Roumbanis 2019 ; Scholten et al. 2018 ). Another study showed a comprehensive definition of ‘good’ science, which includes creativity, reproducibility, perseverance, intellectual courage, and personal integrity. It demonstrated that ‘good’ science involves not only scientific excellence but also personal values and ethics, and engagement with society ( Van den Brink et al. 2016 ). Noticeable in these studies is the connection made between societal relevance and scientific quality.

In summary, the criteria for scientific quality and societal relevance are conceptualised in different ways, and perspectives on the role of societal value creation and the involvement of societal actors vary strongly. Research funders hence have to pay attention to the meaning of the criteria for the panel members they recruit to help them, and navigate and negotiate how the criteria are applied in assessing research proposals. To be able to do so, more insight is needed in which elements of scientific quality and societal relevance are discussed in practice by peer review panels.

2.3 Role of funders and societal actors in peer review

National governments and charities are important funders of biomedical and health research. How this funding is distributed varies per country. Project funding is frequently allocated based on research programming by specialised public funding organisations, such as the Dutch Research Council in the Netherlands and ZonMw for health research. The DHF, the second largest private non-profit research funder in the Netherlands, provides project funding ( Private Non-Profit Financiering 2020 ). Funders, as so-called boundary organisations, can act as key intermediaries between government, science, and society ( Jasanoff 2011 ). Their responsibility is to develop effective research policies connecting societal demands and scientific ‘supply’. This includes setting up and executing fair and balanced assessment procedures ( Sarewitz and Pielke 2007 ). Herein, the role of societal stakeholders is receiving increasing attention ( Benedictus et al. 2016 ; De Rijcke et al. 2016 ; Dijstelbloem et al. 2013 ; Scholten et al. 2018 ).

All charitable health research funders in the Netherlands have, in the last decade, included patients at different stages of the funding process, including in assessing research proposals ( Den Oudendammer et al. 2019 ). To facilitate research funders in involving patients in assessing research proposals, the federation of Dutch patient organisations set up an independent reviewer panel with (at-risk) patients and direct caregivers ( Patiëntenfederatie Nederland, n.d .). Other foundations have set up societal advisory panels including a wider range of societal actors than patients alone. The Committee Societal Quality (CSQ) of the DHF includes, for example, (at-risk) patients and a wide range of cardiovascular health-care professionals who are not active as academic researchers. This model is also applied by the Diabetes Foundation and the Princess Beatrix Muscle Foundation in the Netherlands ( Diabetesfonds, n.d .; Prinses Beatrix Spierfonds, n.d .).

In 2014, the Lancet presented a series of five papers about biomedical and health research known as the ‘increasing value, reducing waste’ series ( Macleod et al. 2014 ). The authors addressed several issues as well as potential solutions that funders can implement. They highlight, among others, the importance of improving the societal relevance of the research questions and including the burden of disease in research assessment in order to increase the value of biomedical and health science for society. A better understanding of and an increasing role of users of research are also part of the described solutions ( Chalmers et al. 2014 ; Van den Brink et al. 2016 ). This is also in line with the recommendations of the 2013 Declaration on Research Assessment (DORA) ( DORA 2013 ). These recommendations influence the way in which research funders operationalise their criteria in research assessment, how they balance the judgement of scientific and societal aspects, and how they involve societal stakeholders in peer review.

2.4 Panel peer review of research proposals

To assess research proposals, funders rely on the services of peer experts to review the thousands or perhaps millions of research proposals seeking funding each year. While often associated with scholarly publishing, peer review also includes the ex ante assessment of research grant and fellowship applications ( Abdoul et al. 2012 ). Peer review of proposals often includes a written assessment of a proposal by an anonymous peer and a peer review panel meeting to select the proposals eligible for funding. Peer review is an established component of professional academic practice, is deeply embedded in the research culture, and essentially consists of experts in a given domain appraising the professional performance, creativity, and/or quality of scientific work produced by others in their field of competence ( Demicheli and Di Pietrantonj 2007 ). The history of peer review as the default approach for scientific evaluation and accountability is, however, relatively young. While the term was unheard of in the 1960s, by 1970, it had become the standard. Since that time, peer review has become increasingly diverse and formalised, resulting in more public accountability ( Reinhart and Schendzielorz 2021 ).

While many studies have been conducted concerning peer review in scholarly publishing, peer review in grant allocation processes has been less discussed ( Demicheli and Di Pietrantonj 2007 ). The most extensive work on this topic has been conducted by Lamont (2009) . Lamont studied peer review panels in five American research funding organisations, including observing three panels. Other examples include Roumbanis’s ethnographic observations of ten review panels at the Swedish Research Council in natural and engineering sciences ( Roumbanis 2017 , 2021a ). Also, Huutoniemi was able to study, but not observe, four panels on environmental studies and social sciences of the Academy of Finland ( Huutoniemi 2012 ). Additionally, Van Arensbergen and Van den Besselaar (2012) analysed peer review through interviews and by analysing the scores and outcomes at different stages of the peer review process in a talent funding programme. In particular, interesting is the study by Luo and colleagues on 164 written panel review reports, showing that the reviews from panels that included non-scientific peers described broader and more concrete impact topics. Mixed panels also more often connected research processes and characteristics of applicants with impact creation ( Luo et al. 2021 ).

While these studies primarily focused on peer review panels in other disciplinary domains or are based on interviews or reports instead of direct observations, we believe that many of the findings are relevant to the functioning of panels in the context of biomedical and health research. From this literature, we learn to have realistic expectations of peer review. It is inherently difficult to predict in advance which research projects will provide the most important findings or breakthroughs ( Lee et al. 2013 ; Pier et al. 2018 ; Roumbanis 2021a , 2021b ). At the same time, these limitations may not substantiate the replacement of peer review by another assessment approach ( Wessely 1998 ). Many topics addressed in the literature are inter-related and relevant to our study, such as disciplinary differences and interdisciplinarity, social dynamics and their consequences for consistency and bias, and suggestions to improve panel peer review ( Lamont and Huutoniemi 2011 ; Lee et al. 2013 ; Pier et al. 2018 ; Roumbanis 2021a , b ; Wessely 1998 ).

Different scientific disciplines show different preferences and beliefs about how to build knowledge and thus have different perceptions of excellence. However, panellists are willing to respect and acknowledge other standards of excellence ( Lamont 2009 ). Evaluation cultures also differ between scientific fields. Science, technology, engineering, and mathematics panels might, in comparison with panellists from social sciences and humanities, be more concerned with the consistency of the assessment across panels and therefore with clear definitions and uses of assessment criteria ( Lamont and Huutoniemi 2011 ). However, much is still to learn about how panellists’ cognitive affiliations with particular disciplines unfold in the evaluation process. Therefore, the assessment of interdisciplinary research is much more complex than just improving the criteria or procedure because less explicit repertoires would also need to change ( Huutoniemi 2012 ).

Social dynamics play a role as panellists may differ in their motivation to engage in allocation processes, which could create bias ( Lee et al. 2013 ). Placing emphasis on meeting established standards or thoroughness in peer review may promote uncontroversial and safe projects, especially in a situation where strong competition puts pressure on experts to reach a consensus ( Langfeldt 2001 ,2006 ). Personal interest and cognitive similarity may also contribute to conservative bias, which could negatively affect controversial or frontier science ( Luukkonen 2012 ; Roumbanis 2021a ; Travis and Collins 1991 ). Central in this part of literature is that panel conclusions are the outcome of and are influenced by the group interaction ( Van Arensbergen et al. 2014a ). Differences in, for example, the status and expertise of the panel members can play an important role in group dynamics. Insights from social psychology on group dynamics can help in understanding and avoiding bias in peer review panels ( Olbrecht and Bornmann 2010 ). For example, group performance research shows that more diverse groups with complementary skills make better group decisions than homogenous groups. Yet, heterogeneity can also increase conflict within the group ( Forsyth 1999 ). Therefore, it is important to pay attention to power dynamics and maintain team spirit and good communication ( Van Arensbergen et al. 2014a ), especially in meetings that include both scientific and non-scientific peers.

The literature also provides funders with starting points to improve the peer review process. For example, the explicitness of review procedures positively influences the decision-making processes ( Langfeldt 2001 ). Strategic voting and decision-making appear to be less frequent in panels that rate than in panels that rank proposals. Also, an advisory instead of a decisional role may improve the quality of the panel assessment ( Lamont and Huutoniemi 2011 ).

Despite different disciplinary evaluative cultures, formal procedures, and criteria, panel members with different backgrounds develop shared customary rules of deliberation that facilitate agreement and help avoid situations of conflict ( Huutoniemi 2012 ; Lamont 2009 ). This is a necessary prerequisite for opening up peer review panels to include non-academic experts. When doing so, it is important to realise that panel review is a social, emotional, and interactional process. It is therefore important to also take these non-cognitive aspects into account when studying cognitive aspects ( Lamont and Guetzkow 2016 ), as we do in this study.

In summary, what we learn from the literature is that (1) the specific criteria to operationalise scientific quality and societal relevance of research are important, (2) the rationalities from Glerup and Horst predict that not everyone values societal aspects and involve non-scientists in peer review to the same extent and in the same way, (3) this may affect the way peer review panels discuss these aspects, and (4) peer review is a challenging group process that could accommodate other rationalities in order to prevent bias towards specific scientific criteria. To disentangle these aspects, we have carried out an observational study of a diverse range of peer review panel sessions using a fixed set of criteria focusing on scientific quality and societal relevance.

3.1 Research assessment at ZonMw and the DHF

The peer review approach and the criteria used by both the DHF and ZonMw are largely comparable. Funding programmes at both organisations start with a brochure describing the purposes, goals, and conditions for research applications, as well as the assessment procedure and criteria. Both organisations apply a two-stage process. In the first phase, reviewers are asked to write a peer review. In the second phase, a panel reviews the application based on the advice of the written reviews and the applicants’ rebuttal. The panels advise the board on eligible proposals for funding including a ranking of these proposals.

There are also differences between the two organisations. At ZonMw, the criteria for societal relevance and quality are operationalised in the ZonMw Framework Fostering Responsible Research Practices ( Reijmerink and Oortwijn 2017 ). This contributes to a common operationalisation of both quality and societal relevance on the level of individual funding programmes. Important elements in the criteria for societal relevance are, for instance, stakeholder participation, (applying) holistic health concepts, and the added value of knowledge in practice, policy, and education. The framework was developed to optimise the funding process from the perspective of knowledge utilisation and includes concepts like productive interactions and Open Science. It is part of the ZonMw Impact Assessment Framework aimed at guiding the planning, monitoring, and evaluation of funding programmes ( Reijmerink et al. 2020 ). At ZonMw, interdisciplinary panels are set up specifically for each funding programme. Panels are interdisciplinary in nature with academics of a wide range of disciplines and often include non-academic peers, like policymakers, health-care professionals, and patients.

At the DHF, the criteria for scientific quality and societal relevance, at the DHF called societal impact , find their origin in the strategy report of the advisory committee CardioVascular Research Netherlands ( Reneman et al. 2010 ). This report forms the basis of the DHF research policy focusing on scientific and societal impact by creating national collaborations in thematic, interdisciplinary research programmes (the so-called consortia) connecting preclinical and clinical expertise into one concerted effort. An International Scientific Advisory Committee (ISAC) was established to assess these thematic consortia. This panel consists of international scientists, primarily with expertise in the broad cardiovascular research field. The DHF criteria for societal impact were redeveloped in 2013 in collaboration with their CSQ. This panel assesses and advises on the societal aspects of proposed studies. The societal impact criteria include the relevance of the health-care problem, the expected contribution to a solution, attention to the next step in science and towards implementation in practice, and the involvement of and interaction with (end-)users of research (R.Y. Abma-Schouten and I.M. Meijer, unpublished data). Peer review panels for consortium funding are generally composed of members of the ISAC, members of the CSQ, and ad hoc panel members relevant to the specific programme. CSQ members often have a pre-meeting before the final panel meetings to prepare and empower CSQ representatives participating in the peer review panel.

3.2 Selection of funding programmes

To compare and evaluate observations between the two organisations, we selected funding programmes that were relatively comparable in scope and aims. The criteria were (1) a translational and/or clinical objective and (2) the selection procedure consisted of review panels that were responsible for the (final) relevance and quality assessment of grant applications. In total, we selected eight programmes: four at each organisation. At the DHF, two programmes were chosen in which the CSQ did not participate to better disentangle the role of the panel composition. For each programme, we observed the selection process varying from one session on one day (taking 2–8 h) to multiple sessions over several days. Ten sessions were observed in total, of which eight were final peer review panel meetings and two were CSQ meetings preparing for the panel meeting.

After management approval for the study in both organisations, we asked programme managers and panel chairpersons of the programmes that were selected for their consent for observation; none refused participation. Panel members were, in a passive consent procedure, informed about the planned observation and anonymous analyses.

To ensure the independence of this evaluation, the selection of the grant programmes, and peer review panels observed, was at the discretion of the project team of this study. The observations and supervision of the analyses were performed by the senior author not affiliated with the funders.

3.3 Observation matrix

Given the lack of a common operationalisation for scientific quality and societal relevance, we decided to use an observation matrix with a fixed set of detailed aspects as a gold standard to score the brochures, the assessment forms, and the arguments used in panel meetings. The matrix used for the observations of the review panels was based upon and adapted from a ‘grant committee observation matrix’ developed by Van Arensbergen. The original matrix informed a literature review on the selection of talent through peer review and the social dynamics in grant review committees ( van Arensbergen et al. 2014b ). The matrix includes four categories of aspects that operationalise societal relevance, scientific quality, committee, and applicant (see  Table 1 ). The aspects of scientific quality and societal relevance were adapted to fit the operationalisation of scientific quality and societal relevance of the organisations involved. The aspects concerning societal relevance were derived from the CSQ criteria, and the aspects concerning scientific quality were based on the scientific criteria of the first panel observed. The four argument types related to the panel were kept as they were. This committee-related category reflects statements that are related to the personal experience or preference of a panel member and can be seen as signals for bias. This category also includes statements that compare a project with another project without further substantiation. The three applicant-related arguments in the original observation matrix were extended with a fourth on social skills in communication with society. We added health technology assessment (HTA) because one programme specifically focused on this aspect. We tested our version of the observation matrix in pilot observations.

Aspects included in the observation matrix and examples of arguments.

3.4 Observations

Data were primarily collected through observations. Our observations of review panel meetings were non-participatory: the observer and goal of the observation were introduced at the start of the meeting, without further interactions during the meeting. To aid in the processing of observations, some meetings were audiotaped (sound only). Presentations or responses of applicants were not noted and were not part of the analysis. The observer made notes on the ongoing discussion and scored the arguments while listening. One meeting was not attended in person and only observed and scored by listening to the audiotape recording. Because this made identification of the panel members unreliable, this panel meeting was excluded from the analysis of the third research question on how arguments used differ between panel members with different perspectives.

3.5 Grant programmes and the assessment criteria

We gathered and analysed all brochures and assessment forms used by the review panels in order to answer our second research question on the correspondence of arguments used with the formal criteria. Several programmes consisted of multiple grant calls: in that case, the specific call brochure was gathered and analysed, not the overall programme brochure. Additional documentation (e.g. instructional presentations at the start of the panel meeting) was not included in the document analysis. All included documents were marked using the aforementioned observation matrix. The panel-related arguments were not used because this category reflects the personal arguments of panel members that are not part of brochures or instructions. To avoid potential differences in scoring methods, two of the authors independently scored half of the documents that were checked and validated afterwards by the other. Differences were discussed until a consensus was reached.

3.6 Panel composition

In order to answer the third research question, background information on panel members was collected. We categorised the panel members into five common types of panel members: scientific, clinical scientific, health-care professional/clinical, patient, and policy. First, a list of all panel members was composed including their scientific and professional backgrounds and affiliations. The theoretical notion that reviewers represent different types of users of research and therefore potential impact domains (academic, social, economic, and cultural) was leading in the categorisation ( Meijer 2012 ; Spaapen and Van Drooge 2011 ). Because clinical researchers play a dual role in both advancing research as a fellow academic and as a user of the research output in health-care practice, we divided the academic members into two categories of non-clinical and clinical researchers. Multiple types of professional actors participated in each review panel. These were divided into two groups for the analysis: health-care professionals (without current academic activity) and policymakers in the health-care sector. No representatives of the private sector participated in the observed review panels. From the public domain, (at-risk) patients and patient representatives were part of several review panels. Only publicly available information was used to classify the panel members. Members were assigned to one category only: categorisation took place based on the specific role and expertise for which they were appointed to the panel.

In two of the four DHF programmes, the assessment procedure included the CSQ. In these two programmes, representatives of this CSQ participated in the scientific panel to articulate the findings of the CSQ meeting during the final assessment meeting. Two grant programmes were assessed by a review panel with solely (clinical) scientific members.

3.7 Analysis

Data were processed using ATLAS.ti 8 and Microsoft Excel 2010 to produce descriptive statistics. All observed arguments were coded and given a randomised identification code for the panel member using that particular argument. The number of times an argument type was observed was used as an indicator for the relative importance of that argument in the appraisal of proposals. With this approach, a practical and reproducible method for research funders to evaluate the effect of policy changes on peer review was developed. If codes or notes were unclear, post-observation validation of codes was carried out based on observation matrix notes. Arguments that were noted by the observer but could not be matched with an existing code were first coded as a ‘non-existing’ code, and these were resolved by listening back to the audiotapes. Arguments that could not be assigned to a panel member were assigned a ‘missing panel member’ code. A total of 4.7 per cent of all codes were assigned a ‘missing panel member’ code.

After the analyses, two meetings were held to reflect on the results: one with the CSQ and the other with the programme coordinators of both organisations. The goal of these meetings was to improve our interpretation of the findings, disseminate the results derived from this project, and identify topics for further analyses or future studies.

3.8 Limitations

Our study focuses on studying the final phase of the peer review process of research applications in a real-life setting. Our design, a non-participant observation of peer review panels, also introduced several challenges ( Liu and Maitlis 2010 ).

First, the independent review phase or pre-application phase was not part of our study. We therefore could not assess to what extent attention to certain aspects of scientific quality or societal relevance and impact in the review phase influenced the topics discussed during the meeting.

Second, the most important challenge of overt non-participant observations is the observer effect: the danger of causing reactivity in those under study. We believe that the consequences of this effect on our conclusions were limited because panellists are used to external observers in the meetings of these two funders. The observer briefly explained the goal of the study during the introductory round of the panel in general terms. The observer sat as unobtrusively as possible and avoided reactivity to discussions. Similar to previous observations of panels, we experienced that the fact that an observer was present faded into the background during a meeting ( Roumbanis 2021a ). However, a limited observer effect can never be entirely excluded.

Third, our design to only score the arguments raised, and not the responses of the applicant, or information on the content of the proposals, has its positives and negatives. With this approach, we could assure the anonymity of the grant procedures reviewed, the applicants and proposals, panels, and individual panellists. This was an important condition for the funders involved. We took the frequency arguments used as a proxy for the relative importance of that argument in decision-making, which undeniably also has its caveats. Our data collection approach limits more in-depth reflection on which arguments were decisive in decision-making and on group dynamics during the interaction with the applicants as non-verbal and non-content-related comments were not captured in this study.

Fourth, despite this being one of the largest observational studies on the peer review assessment of grant applications with the observation of ten panels in eight grant programmes, many variables might explain differences in arguments used within and beyond our view. Examples of ‘confounding’ variables are the many variations in panel composition, the differences in objectives of the programmes, and the range of the funding programmes. Our study should therefore be seen as exploratory and thus warrants caution in drawing conclusions.

4.1 Overview of observational data

The grant programmes included in this study reflected a broad range of biomedical and health funding programmes, ranging from fellowship grants to translational research and applied health research. All formal documents available to the applicants and to the review panel were retrieved for both ZonMw and the DHF. In total, eighteen documents corresponding to the eight grant programmes were studied. The number of proposals assessed per programme varied from three to thirty-three. The duration of the panel meetings varied between 2 h and two consecutive days. Together, this resulted in a large spread in the number of total arguments used in an individual meeting and in a grant programme as a whole. In the shortest meeting, 49 arguments were observed versus 254 in the longest, with a mean of 126 arguments per meeting and on average 15 arguments per proposal.

We found consistency between how criteria were operationalised in the grant programme’s brochures and in the assessment forms of the review panels overall. At the same time, because the number of elements included in the observation matrix is limited, there was a considerable diversity in the arguments that fall within each aspect (see examples in  Table 1 ). Some of these differences could possibly be explained by differences in language used and the level of detail in the observation matrix, the brochure, and the panel’s instructions. This was especially the case in the applicant-related aspects in which the observation matrix was more detailed than the text in the brochure and assessment forms.

In interpretating our findings, it is important to take into account that, even though our data were largely complete and the observation matrix matched well with the description of the criteria in the brochures and assessment forms, there was a large diversity in the type and number of arguments used and in the number of proposals assessed in the grant programmes included in our study.

4.2 Wide range of arguments used by panels: scientific arguments used most

For our first research question, we explored the number and type of arguments used in the panel meetings. Figure 1 provides an overview of the arguments used. Scientific quality was discussed most. The number of times the feasibility of the aims was discussed clearly stands out in comparison to all other arguments. Also, the match between the science and the problem studied and the plan of work were frequently discussed aspects of scientific quality. International competitiveness of the proposal was discussed the least of all five scientific arguments.

The number of arguments used in panel meetings.

Attention was paid to societal relevance and impact in the panel meetings of both organisations. Yet, the language used differed somewhat between organisations. The contribution to a solution and the next step in science were the most often used societal arguments. At ZonMw, the impact of the health-care problem studied and the activities towards partners were less frequently discussed than the other three societal arguments. At the DHF, the five societal arguments were used equally often.

With the exception of the fellowship programme meeting, applicant-related arguments were not often used. The fellowship panel used arguments related to the applicant and to scientific quality about equally often. Committee-related arguments were also rarely used in the majority of the eight grant programmes observed. In three out of the ten panel meetings, one or two arguments were observed, which were related to personal experience with the applicant or their direct network. In seven out of ten meetings, statements were observed, which were unasserted or were explicitly announced as reflecting a personal preference. The frequency varied between one and seven statements (sixteen in total), which is low in comparison to the other arguments used (see  Fig. 1 for examples).

4.3 Use of arguments varied strongly per panel meeting

The balance in the use of scientific and societal arguments varied strongly per grant programme, panel, and organisation. At ZonMw, two meetings had approximately an equal balance in societal and scientific arguments. In the other two meetings, scientific arguments were used twice to four times as often as societal arguments. At the DHF, three types of panels were observed. Different patterns in the relative use of societal and scientific arguments were observed for each of these panel types. In the two CSQ-only meetings the societal arguments were used approximately twice as often as scientific arguments. In the two meetings of the scientific panels, societal arguments were infrequently used (between zero and four times per argument category). In the combined societal and scientific panel meetings, the use of societal and scientific arguments was more balanced.

4.4 Match of arguments used by panels with the assessment criteria

In order to answer our second research question, we looked into the relation of the arguments used with the formal criteria. We observed that a broader range of arguments were often used in comparison to how the criteria were described in the brochure and assessment instruction. However, arguments related to aspects that were consequently included in the brochure and instruction seemed to be discussed more frequently than in programmes where those aspects were not consistently included or were not included at all. Although the match of the science with the health-care problem and the background and reputation of the applicant were not always made explicit in the brochure or instructions, they were discussed in many panel meetings. Supplementary Fig. S1 provides a visualisation of how arguments used differ between the programmes in which those aspects were, were not, consistently included in the brochure and instruction forms.

4.5 Two-thirds of the assessment was driven by scientific panel members

To answer our third question, we looked into the differences in arguments used between panel members representing a scientific, clinical scientific, professional, policy, or patient perspective. In each research programme, the majority of panellists had a scientific background ( n  = 35), thirty-four members had a clinical scientific background, twenty had a health professional/clinical background, eight members represented a policy perspective, and fifteen represented a patient perspective. From the total number of arguments (1,097), two-thirds were made by members with a scientific or clinical scientific perspective. Members with a scientific background engaged most actively in the discussion with a mean of twelve arguments per member. Similarly, clinical scientists and health-care professionals participated with a mean of nine arguments, and members with a policy and patient perspective put forward the least number of arguments on average, namely, seven and eight. Figure 2 provides a complete overview of the total and mean number of arguments used by the different disciplines in the various panels.

The total and mean number of arguments displayed per subgroup of panel members.

4.6 Diverse use of arguments by panellists, but background matters

In meetings of both organisations, we observed a diverse use of arguments by the panel members. Yet, the use of arguments varied depending on the background of the panel member (see  Fig. 3 ). Those with a scientific and clinical scientific perspective used primarily scientific arguments. As could be expected, health-care professionals and patients used societal arguments more often.

The use of arguments differentiated by panel member background.

Further breakdown of arguments across backgrounds showed clear differences in the use of scientific arguments between the different disciplines of panellists. Scientists and clinical scientists discussed the feasibility of the aims more than twice as often as their second most often uttered element of scientific quality, which was the match between the science and the problem studied . Patients and members with a policy or health professional background put forward fewer but more varied scientific arguments.

Patients and health-care professionals accounted for approximately half of the societal arguments used, despite being a much smaller part of the panel’s overall composition. In other words, members with a scientific perspective were less likely to use societal arguments. The relevance of the health-care problem studied, activities towards partners , and arguments related to participation and diversity were not used often by this group. Patients often used arguments related to patient participation and diversity and activities towards partners , although the frequency of the use of the latter differed per organisation.

The majority of the applicant-related arguments were put forward by scientists, including clinical scientists. Committee-related arguments were very rare and are therefore not differentiated by panel member background, except comments related to a comparison with other applications. These arguments were mainly put forward by panel members with a scientific background. HTA -related arguments were often used by panel members with a scientific perspective. Panel members with other perspectives used this argument scarcely (see Supplementary Figs S2–S4 for the visual presentation of the differences between panel members on all aspects included in the matrix).

5.1 Explanations for arguments used in panels

Our observations show that most arguments for scientific quality were often used. However, except for the feasibility , the frequency of arguments used varied strongly between the meetings and between the individual proposals that were discussed. The fact that most arguments were not consistently used is not surprising given the results from previous studies that showed heterogeneity in grant application assessments and low consistency in comments and scores by independent reviewers ( Abdoul et al. 2012 ; Pier et al. 2018 ). In an analysis of written assessments on nine observed dimensions, no dimension was used in more than 45 per cent of the reviews ( Hartmann and Neidhardt 1990 ).

There are several possible explanations for this heterogeneity. Roumbanis (2021a) described how being responsive to the different challenges in the proposals and to the points of attention arising from the written assessments influenced discussion in panels. Also when a disagreement arises, more time is spent on discussion ( Roumbanis 2021a ). One could infer that unambiguous, and thus not debated, aspects might remain largely undetected in our study. We believe, however, that the main points relevant to the assessment will not remain entirely unmentioned, because most panels in our study started the discussion with a short summary of the proposal, the written assessment, and the rebuttal. Lamont (2009) , however, points out that opening statements serve more goals than merely decision-making. They can also increase the credibility of the panellist, showing their comprehension and balanced assessment of an application. We can therefore not entirely disentangle whether the arguments observed most were also found to be most important or decisive or those were simply the topics that led to most disagreement.

An interesting difference with Roumbanis’ study was the available discussion time per proposal. In our study, most panels handled a limited number of proposals, allowing for longer discussions in comparison with the often 2-min time frame that Roumbanis (2021b) described, potentially contributing to a wider range of arguments being discussed. Limited time per proposal might also limit the number of panellists contributing to the discussion per proposal ( De Bont 2014 ).

5.2 Reducing heterogeneity by improving operationalisation and the consequent use of assessment criteria

We found that the language used for the operationalisation of the assessment criteria in programme brochures and in the observation matrix was much more detailed than in the instruction for the panel, which was often very concise. The exercise also illustrated that many terms were used interchangeably.

This was especially true for the applicant-related aspects. Several panels discussed how talent should be assessed. This confusion is understandable when considering the changing values in research and its assessment ( Moher et al. 2018 ) and the fact that the instruction of the funders was very concise. For example, it was not explicated whether the individual or the team should be assessed. Arensbergen et al. (2014b) described how in grant allocation processes, talent is generally assessed using limited characteristics. More objective and quantifiable outputs often prevailed at the expense of recognising and rewarding a broad variety of skills and traits combining professional, social, and individual capital ( DORA 2013 ).

In addition, committee-related arguments, like personal experiences with the applicant or their institute, were rarely used in our study. Comparisons between proposals were sometimes made without further argumentation, mainly by scientific panel members. This was especially pronounced in one (fellowship) grant programme with a high number of proposals. In this programme, the panel meeting concentrated on quickly comparing the quality of the applicants and of the proposals based on the reviewer’s judgement, instead of a more in-depth discussion of the different aspects of the proposals. Because the review phase was not part of this study, the question of which aspects have been used for the assessment of the proposals in this panel therefore remains partially unanswered. However, weighing and comparing proposals on different aspects and with different inputs is a core element of scientific peer review, both in the review of papers and in the review of grants ( Hirschauer 2010 ). The large role of scientific panel members in comparing proposals is therefore not surprising.

One could anticipate that more consequent language in the operationalising criteria may lead to more clarity for both applicants and panellists and to more consistency in the assessment of research proposals. The trend in our observations was that arguments were used less when the related criteria were not or were consequently included in the brochure and panel instruction. It remains, however, challenging to disentangle the influence of the formal definitions of criteria on the arguments used. Previous studies also encountered difficulties in studying the role of the formal instruction in peer review but concluded that this role is relatively limited ( Langfeldt 2001 ; Reinhart 2010 ).

The lack of a clear operationalisation of criteria can contribute to heterogeneity in peer review as many scholars found that assessors differ in the conceptualisation of good science and to the importance they attach to various aspects of research quality and societal relevance ( Abdoul et al. 2012 ; Geurts 2016 ; Scholten et al. 2018 ; Van den Brink et al. 2016 ). The large variation and absence of a gold standard in the interpretation of scientific quality and societal relevance affect the consistency of peer review. As a consequence, it is challenging to systematically evaluate and improve peer review in order to fund the research that contributes most to science and society. To contribute to responsible research and innovation, it is, therefore, important that funders invest in a more consistent and conscientious peer review process ( Curry et al. 2020 ; DORA 2013 ).

A common conceptualisation of scientific quality and societal relevance and impact could improve the alignment between views on good scientific conduct, programmes’ objectives, and the peer review in practice. Such a conceptualisation could contribute to more transparency and quality in the assessment of research. By involving panel members from all relevant backgrounds, including the research community, health-care professionals, and societal actors, in a better operationalisation of criteria, more inclusive views of good science can be implemented more systematically in the peer review assessment of research proposals. The ZonMw Framework Fostering Responsible Research Practices is an example of an initiative aiming to support standardisation and integration ( Reijmerink et al. 2020 ).

Given the lack of a common definition or conceptualisation of scientific quality and societal relevance, our study made an important decision by choosing to use a fixed set of detailed aspects of two important criteria as a gold standard to score the brochures, the panel instructions, and the arguments used by the panels. This approach proved helpful in disentangling the different components of scientific quality and societal relevance. Having said that, it is important not to oversimplify the causes for heterogeneity in peer review because these substantive arguments are not independent of non-cognitive, emotional, or social aspects ( Lamont and Guetzkow 2016 ; Reinhart 2010 ).

5.3 Do more diverse panels contribute to a broader use of arguments?

Both funders participating in our study have an outspoken public mission that requests sufficient attention to societal aspects in assessment processes. In reality, as observed in several panels, the main focus of peer review meetings is on scientific arguments. Next to the possible explanations earlier, the composition of the panel might play a role in explaining arguments used in panel meetings. Our results have shown that health-care professionals and patients bring in more societal arguments than scientists, including those who are also clinicians. It is, however, not that simple. In the more diverse panels, panel members, regardless of their backgrounds, used more societal arguments than in the less diverse panels.

Observing ten panel meetings was sufficient to explore differences in arguments used by panel members with different backgrounds. The pattern of (primarily) scientific arguments being raised by panels with mainly scientific members is not surprising. After all, it is their main task to assess the scientific content of grant proposals and fit their competencies. As such, one could argue, depending on how one justifies the relationship between science and society, that health-care professionals and patients might be better suited to assess the value for potential users of research results. Scientific panel members and clinical scientists in our study used less arguments that reflect on opening up and connecting science directly to others who can bring it further (being industry, health-care professionals, or other stakeholders). Patients filled this gap since these two types of arguments were the most prevalent type put forward by them. Making an active connection with society apparently needs a broader, more diverse panel for scientists to direct their attention to more societal arguments. Evident from our observations is that in panels with patients and health-care professionals, their presence seemed to increase the attention placed on arguments beyond the scientific arguments put forward by all panel members, including scientists. This conclusion is congruent with the observation that there was a more equal balance in the use of societal and scientific arguments in the scientific panels in which the CSQ participated. This illustrates that opening up peer review panels to non-scientific members creates an opportunity to focus on both the contribution and the integrative rationality ( Glerup and Horst 2014 ) or, in other words, to allow productive interactions between scientific and non-scientific actors. This corresponds with previous research that suggests that with regard to societal aspects, reviews from mixed panels were broader and richer ( Luo et al. 2021 ). In panels with non-scientific experts, more emphasis was placed on the role of the proposed research process to increase the likelihood of societal impact over the causal importance of scientific excellence for broader impacts. This is in line with the findings that panels with more disciplinary diversity, in range and also by including generalist experts, applied more versatile styles to reach consensus and paid more attention to relevance and pragmatic value ( Huutoniemi 2012 ).

Our observations further illustrate that patients and health-care professionals were less vocal in panels than (clinical) scientists and were in the minority. This could reflect their social role and lower perceived authority in the panel. Several guides are available for funders to stimulate the equal participation of patients in science. These guides are also applicable to their involvement in peer review panels. Measures to be taken include the support and training to help prepare patients for their participation in deliberations with renowned scientists and explicitly addressing power differences ( De Wit et al. 2016 ). Panel chairs and programme officers have to set and supervise the conditions for the functioning of both the individual panel members and the panel as a whole ( Lamont 2009 ).

5.4 Suggestions for future studies

In future studies, it is important to further disentangle the role of the operationalisation and appraisal of assessment criteria in reducing heterogeneity in the arguments used by panels. More controlled experimental settings are a valuable addition to the current mainly observational methodologies applied to disentangle some of the cognitive and social factors that influence the functioning and argumentation of peer review panels. Reusing data from the panel observations and the data on the written reports could also provide a starting point for a bottom-up approach to create a more consistent and shared conceptualisation and operationalisation of assessment criteria.

To further understand the effects of opening up review panels to non-scientific peers, it is valuable to compare the role of diversity and interdisciplinarity in solely scientific panels versus panels that also include non-scientific experts.

In future studies, differences between domains and types of research should also be addressed. We hypothesise that biomedical and health research is perhaps more suited for the inclusion of non-scientific peers in panels than other research domains. For example, it is valuable to better understand how potentially relevant users can be well enough identified in other research fields and to what extent non-academics can contribute to assessing the possible value of, especially early or blue sky, research.

The goal of our study was to explore in practice which arguments regarding the main criteria of scientific quality and societal relevance were used by peer review panels of biomedical and health research funding programmes. We showed that there is a wide diversity in the number and range of arguments used, but three main scientific aspects were discussed most frequently. These are the following: is it a feasible approach; does the science match the problem , and is the work plan scientifically sound? Nevertheless, these scientific aspects were accompanied by a significant amount of discussion of societal aspects, of which the contribution to a solution is the most prominent. In comparison with scientific panellists, non-scientific panellists, such as health-care professionals, policymakers, and patients, often use a wider range of arguments and other societal arguments. Even more striking was that, even though non-scientific peers were often outnumbered and less vocal in panels, scientists also used a wider range of arguments when non-scientific peers were present.

It is relevant that two health research funders collaborated in the current study to reflect on and improve peer review in research funding. There are few studies published that describe live observations of peer review panel meetings. Many studies focus on alternatives for peer review or reflect on the outcomes of the peer review process, instead of reflecting on the practice and improvement of peer review assessment of grant proposals. Privacy and confidentiality concerns of funders also contribute to the lack of information on the functioning of peer review panels. In this study, both organisations were willing to participate because of their interest in research funding policies in relation to enhancing the societal value and impact of science. The study provided them with practical suggestions, for example, on how to improve the alignment in language used in programme brochures and instructions of review panels, and contributed to valuable knowledge exchanges between organisations. We hope that this publication stimulates more research funders to evaluate their peer review approach in research funding and share their insights.

For a long time, research funders relied solely on scientists for designing and executing peer review of research proposals, thereby delegating responsibility for the process. Although review panels have a discretionary authority, it is important that funders set and supervise the process and the conditions. We argue that one of these conditions should be the diversification of peer review panels and opening up panels for non-scientific peers.

Supplementary material is available at Science and Public Policy online.

Details of the data and information on how to request access is available from the first author.

Joey Gijbels and Wendy Reijmerink are employed by ZonMw. Rebecca Abma-Schouten is employed by the Dutch Heart Foundation and as external PhD candidate affiliated with the Centre for Science and Technology Studies, Leiden University.

A special thanks to the panel chairs and programme officers of ZonMw and the DHF for their willingness to participate in this project. We thank Diny Stekelenburg, an internship student at ZonMw, for her contributions to the project. Our sincerest gratitude to Prof. Paul Wouters, Sarah Coombs, and Michiel van der Vaart for proofreading and their valuable feedback. Finally, we thank the editors and anonymous reviewers of Science and Public Policy for their thorough and insightful reviews and recommendations. Their contributions are recognisable in the final version of this paper.

Abdoul   H. , Perrey   C. , Amiel   P. , et al.  ( 2012 ) ‘ Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices ’, PLoS One , 7 : 1 – 15 .

Google Scholar

Abma-Schouten   R. Y. ( 2017 ) ‘ Maatschappelijke Kwaliteit van Onderzoeksvoorstellen ’, Dutch Heart Foundation .

Alla   K. , Hall   W. D. , Whiteford   H. A. , et al.  ( 2017 ) ‘ How Do We Define the Policy Impact of Public Health Research? A Systematic Review ’, Health Research Policy and Systems , 15 : 84.

Benedictus   R. , Miedema   F. , and Ferguson   M. W. J. ( 2016 ) ‘ Fewer Numbers, Better Science ’, Nature , 538 : 453 – 4 .

Chalmers   I. , Bracken   M. B. , Djulbegovic   B. , et al.  ( 2014 ) ‘ How to Increase Value and Reduce Waste When Research Priorities Are Set ’, The Lancet , 383 : 156 – 65 .

Curry   S. , De Rijcke   S. , Hatch   A. , et al.  ( 2020 ) ‘ The Changing Role of Funders in Responsible Research Assessment: Progress, Obstacles and the Way Ahead ’, RoRI Working Paper No. 3, London : Research on Research Institute (RoRI) .

De Bont   A. ( 2014 ) ‘ Beoordelen Bekeken. Reflecties op het Werk van Een Programmacommissie van ZonMw ’, ZonMw .

De Rijcke   S. , Wouters   P. F. , Rushforth   A. D. , et al.  ( 2016 ) ‘ Evaluation Practices and Effects of Indicator Use—a Literature Review ’, Research Evaluation , 25 : 161 – 9 .

De Wit   A. M. , Bloemkolk   D. , Teunissen   T. , et al.  ( 2016 ) ‘ Voorwaarden voor Succesvolle Betrokkenheid van Patiënten/cliënten bij Medisch Wetenschappelijk Onderzoek ’, Tijdschrift voor Sociale Gezondheidszorg , 94 : 91 – 100 .

Del Carmen Calatrava Moreno   M. , Warta   K. , Arnold   E. , et al.  ( 2019 ) Science Europe Study on Research Assessment Practices . Technopolis Group Austria .

Google Preview

Demicheli   V. and Di Pietrantonj   C. ( 2007 ) ‘ Peer Review for Improving the Quality of Grant Applications ’, Cochrane Database of Systematic Reviews , 2 : MR000003.

Den Oudendammer   W. M. , Noordhoek   J. , Abma-Schouten   R. Y. , et al.  ( 2019 ) ‘ Patient Participation in Research Funding: An Overview of When, Why and How Amongst Dutch Health Funds ’, Research Involvement and Engagement , 5 .

Diabetesfonds ( n.d. ) Maatschappelijke Adviesraad < https://www.diabetesfonds.nl/over-ons/maatschappelijke-adviesraad > accessed 18 Sept 2022 .

Dijstelbloem   H. , Huisman   F. , Miedema   F. , et al.  ( 2013 ) ‘ Science in Transition Position Paper: Waarom de Wetenschap Niet Werkt Zoals het Moet, En Wat Daar aan te Doen Is ’, Utrecht : Science in Transition .

Forsyth   D. R. ( 1999 ) Group Dynamics , 3rd edn. Belmont : Wadsworth Publishing Company .

Geurts   J. ( 2016 ) ‘ Wat Goed Is, Herken Je Meteen ’, NRC Handelsblad < https://www.nrc.nl/nieuws/2016/10/28/wat-goed-is-herken-je-meteen-4975248-a1529050 > accessed 6 Mar 2022 .

Glerup   C. and Horst   M. ( 2014 ) ‘ Mapping “Social Responsibility” in Science ’, Journal of Responsible Innovation , 1 : 31 – 50 .

Hartmann   I. and Neidhardt   F. ( 1990 ) ‘ Peer Review at the Deutsche Forschungsgemeinschaft ’, Scientometrics , 19 : 419 – 25 .

Hirschauer   S. ( 2010 ) ‘ Editorial Judgments: A Praxeology of “Voting” in Peer Review ’, Social Studies of Science , 40 : 71 – 103 .

Hughes   A. and Kitson   M. ( 2012 ) ‘ Pathways to Impact and the Strategic Role of Universities: New Evidence on the Breadth and Depth of University Knowledge Exchange in the UK and the Factors Constraining Its Development ’, Cambridge Journal of Economics , 36 : 723 – 50 .

Huutoniemi   K. ( 2012 ) ‘ Communicating and Compromising on Disciplinary Expertise in the Peer Review of Research Proposals ’, Social Studies of Science , 42 : 897 – 921 .

Jasanoff   S. ( 2011 ) ‘ Constitutional Moments in Governing Science and Technology ’, Science and Engineering Ethics , 17 : 621 – 38 .

Kolarz   P. , Arnold   E. , Farla   K. , et al.  ( 2016 ) Evaluation of the ESRC Transformative Research Scheme . Brighton : Technopolis Group .

Lamont   M. ( 2009 ) How Professors Think : Inside the Curious World of Academic Judgment . Cambridge : Harvard University Press .

Lamont   M. Guetzkow   J. ( 2016 ) ‘How Quality Is Recognized by Peer Review Panels: The Case of the Humanities’, in M.   Ochsner , S. E.   Hug , and H.-D.   Daniel (eds) Research Assessment in the Humanities , pp. 31 – 41 . Cham : Springer International Publishing .

Lamont   M. Huutoniemi   K. ( 2011 ) ‘Comparing Customary Rules of Fairness: Evaluative Practices in Various Types of Peer Review Panels’, in C.   Charles   G.   Neil and L.   Michèle (eds) Social Knowledge in the Making , pp. 209–32. Chicago : The University of Chicago Press .

Langfeldt   L. ( 2001 ) ‘ The Decision-making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome ’, Social Studies of Science , 31 : 820 – 41 .

——— ( 2006 ) ‘ The Policy Challenges of Peer Review: Managing Bias, Conflict of Interests and Interdisciplinary Assessments ’, Research Evaluation , 15 : 31 – 41 .

Lee   C. J. , Sugimoto   C. R. , Zhang   G. , et al.  ( 2013 ) ‘ Bias in Peer Review ’, Journal of the American Society for Information Science and Technology , 64 : 2 – 17 .

Liu   F. Maitlis   S. ( 2010 ) ‘Nonparticipant Observation’, in A. J.   Mills , G.   Durepos , and E.   Wiebe (eds) Encyclopedia of Case Study Research , pp. 609 – 11 . Los Angeles : SAGE .

Luo   J. , Ma   L. , and Shankar   K. ( 2021 ) ‘ Does the Inclusion of Non-academix Reviewers Make Any Difference for Grant Impact Panels? ’, Science & Public Policy , 48 : 763 – 75 .

Luukkonen   T. ( 2012 ) ‘ Conservatism and Risk-taking in Peer Review: Emerging ERC Practices ’, Research Evaluation , 21 : 48 – 60 .

Macleod   M. R. , Michie   S. , Roberts   I. , et al.  ( 2014 ) ‘ Biomedical Research: Increasing Value, Reducing Waste ’, The Lancet , 383 : 101 – 4 .

Meijer   I. M. ( 2012 ) ‘ Societal Returns of Scientific Research. How Can We Measure It? ’, Leiden : Center for Science and Technology Studies, Leiden University .

Merton   R. K. ( 1968 ) Social Theory and Social Structure , Enlarged edn. [Nachdr.] . New York : The Free Press .

Moher   D. , Naudet   F. , Cristea   I. A. , et al.  ( 2018 ) ‘ Assessing Scientists for Hiring, Promotion, And Tenure ’, PLoS Biology , 16 : e2004089.

Olbrecht   M. and Bornmann   L. ( 2010 ) ‘ Panel Peer Review of Grant Applications: What Do We Know from Research in Social Psychology on Judgment and Decision-making in Groups? ’, Research Evaluation , 19 : 293 – 304 .

Patiëntenfederatie Nederland ( n.d. ) Ervaringsdeskundigen Referentenpanel < https://www.patientenfederatie.nl/zet-je-ervaring-in/lid-worden-van-ons-referentenpanel > accessed 18 Sept 2022.

Pier   E. L. , M.   B. , Filut   A. , et al.  ( 2018 ) ‘ Low Agreement among Reviewers Evaluating the Same NIH Grant Applications ’, Proceedings of the National Academy of Sciences , 115 : 2952 – 7 .

Prinses Beatrix Spierfonds ( n.d. ) Gebruikerscommissie < https://www.spierfonds.nl/wie-wij-zijn/gebruikerscommissie > accessed 18 Sep 2022 .

( 2020 ) Private Non-profit Financiering van Onderzoek in Nederland < https://www.rathenau.nl/nl/wetenschap-cijfers/geld/wat-geeft-nederland-uit-aan-rd/private-non-profit-financiering-van#:∼:text=R%26D%20in%20Nederland%20wordt%20gefinancierd,aan%20wetenschappelijk%20onderzoek%20in%20Nederland > accessed 6 Mar 2022 .

Reneman   R. S. , Breimer   M. L. , Simoons   J. , et al.  ( 2010 ) ‘ De toekomst van het cardiovasculaire onderzoek in Nederland. Sturing op synergie en impact ’, Den Haag : Nederlandse Hartstichting .

Reed   M. S. , Ferré   M. , Marin-Ortega   J. , et al.  ( 2021 ) ‘ Evaluating Impact from Research: A Methodological Framework ’, Research Policy , 50 : 104147.

Reijmerink   W. and Oortwijn   W. ( 2017 ) ‘ Bevorderen van Verantwoorde Onderzoekspraktijken Door ZonMw ’, Beleidsonderzoek Online. accessed 6 Mar 2022.

Reijmerink   W. , Vianen   G. , Bink   M. , et al.  ( 2020 ) ‘ Ensuring Value in Health Research by Funders’ Implementation of EQUATOR Reporting Guidelines: The Case of ZonMw ’, Berlin : REWARD|EQUATOR .

Reinhart   M. ( 2010 ) ‘ Peer Review Practices: A Content Analysis of External Reviews in Science Funding ’, Research Evaluation , 19 : 317 – 31 .

Reinhart   M. and Schendzielorz   C. ( 2021 ) Trends in Peer Review . SocArXiv . < https://osf.io/preprints/socarxiv/nzsp5 > accessed 29 Aug 2022.

Roumbanis   L. ( 2017 ) ‘ Academic Judgments under Uncertainty: A Study of Collective Anchoring Effects in Swedish Research Council Panel Groups ’, Social Studies of Science , 47 : 95 – 116 .

——— ( 2021a ) ‘ Disagreement and Agonistic Chance in Peer Review ’, Science, Technology & Human Values , 47 : 1302 – 33 .

——— ( 2021b ) ‘ The Oracles of Science: On Grant Peer Review and Competitive Funding ’, Social Science Information , 60 : 356 – 62 .

( 2019 ) ‘ Ruimte voor ieders talent (Position Paper) ’, Den Haag : VSNU, NFU, KNAW, NWO en ZonMw . < https://www.universiteitenvannederland.nl/recognitionandrewards/wp-content/uploads/2019/11/Position-paper-Ruimte-voor-ieders-talent.pdf >.

( 2013 ) San Francisco Declaration on Research Assessment . The Declaration . < https://sfdora.org > accessed 2 Jan 2022 .

Sarewitz   D. and Pielke   R. A.  Jr. ( 2007 ) ‘ The Neglected Heart of Science Policy: Reconciling Supply of and Demand for Science ’, Environmental Science & Policy , 10 : 5 – 16 .

Scholten   W. , Van Drooge   L. , and Diederen   P. ( 2018 ) Excellent Is Niet Gewoon. Dertig Jaar Focus op Excellentie in het Nederlandse Wetenschapsbeleid . The Hague : Rathenau Instituut .

Shapin   S. ( 2008 ) The Scientific Life : A Moral History of a Late Modern Vocation . Chicago : University of Chicago press .

Spaapen   J. and Van Drooge   L. ( 2011 ) ‘ Introducing “Productive Interactions” in Social Impact Assessment ’, Research Evaluation , 20 : 211 – 8 .

Travis   G. D. L. and Collins   H. M. ( 1991 ) ‘ New Light on Old Boys: Cognitive and Institutional Particularism in the Peer Review System ’, Science, Technology & Human Values , 16 : 322 – 41 .

Van Arensbergen   P. and Van den Besselaar   P. ( 2012 ) ‘ The Selection of Scientific Talent in the Allocation of Research Grants ’, Higher Education Policy , 25 : 381 – 405 .

Van Arensbergen   P. , Van der Weijden   I. , and Van den Besselaar   P. V. D. ( 2014a ) ‘ The Selection of Talent as a Group Process: A Literature Review on the Social Dynamics of Decision Making in Grant Panels ’, Research Evaluation , 23 : 298 – 311 .

—— ( 2014b ) ‘ Different Views on Scholarly Talent: What Are the Talents We Are Looking for in Science? ’, Research Evaluation , 23 : 273 – 84 .

Van den Brink , G. , Scholten , W. , and Jansen , T. , eds ( 2016 ) Goed Werk voor Academici . Culemborg : Stichting Beroepseer .

Weingart   P. ( 1999 ) ‘ Scientific Expertise and Political Accountability: Paradoxes of Science in Politics ’, Science & Public Policy , 26 : 151 – 61 .

Wessely   S. ( 1998 ) ‘ Peer Review of Grant Applications: What Do We Know? ’, The Lancet , 352 : 301 – 5 .

Supplementary data

Email alerts, citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1471-5430
• Print ISSN 0302-3427
• Copyright © 2024 Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Writing a Research Proposal

• First Online: 10 April 2022

Cite this chapter

• Fahimeh Tabatabaei 3 &
• Lobat Tayebi 3  

799 Accesses

A research proposal is a roadmap that brings the researcher closer to the objectives, takes the research topic from a purely subjective mind, and manifests an objective plan. It shows us what steps we need to take to reach the objective, what questions we should answer, and how much time we need. It is a framework based on which you can perform your research in a well-organized and timely manner. In other words, by writing a research proposal, you get a map that shows the direction to the destination (answering the research question). If the proposal is poorly prepared, after spending a lot of energy and money, you may realize that the result of the research has nothing to do with the initial objective, and the study may end up nowhere. Therefore, writing the proposal shows that the researcher is aware of the proper research and can justify the significance of his/her idea.

• Research proposal
• Research strategy
• Research methodology
• Research design
• Problem formulation
• Sample size
• Random allocation
• Specific aims
• Review grants

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

A. Gholipour, E.Y. Lee, S.K. Warfield, The anatomy and art of writing a successful grant application: A practical step-by-step approach. Pediatr. Radiol. 44 (12), 1512–1517 (2014)

Article   Google Scholar  

L.S. Marshall, Research commentary: Grant writing: Part I first things first …. J. Radiol. Nurs. 31 (4), 154–155 (2012)

E.K. Proctor, B.J. Powell, A.A. Baumann, A.M. Hamilton, R.L. Santens, Writing implementation research grant proposals: Ten key ingredients. Implement. Sci. 7 (1), 96 (2012)

K.C. Chung, M.J. Shauver, Fundamental principles of writing a successful grant proposal. J. Hand Surg. Am. 33 (4), 566–572 (2008)

A.A. Monte, A.M. Libby, Introduction to the specific aims page of a grant proposal. Kline JA, editor. Acad. Emerg. Med. 25 (9), 1042–1047 (2018)

P. Kan, M.R. Levitt, W.J. Mack, R.M. Starke, K.N. Sheth, F.C. Albuquerque, et al., National Institutes of Health grant opportunities for the neurointerventionalist: Preparation and choosing the right mechanism. J. Neurointerv. Surg. 13 (3), 287–289 (2021)

A.M. Goldstein, S. Balaji, A.A. Ghaferi, A. Gosain, M. Maggard-Gibbons, B. Zuckerbraun, et al., An algorithmic approach to an impactful specific aims page. Surgery 169 (4), 816–820 (2021)

S. Engberg, D.Z. Bliss, Writing a grant proposal—Part 1. J. Wound Ostomy Cont. Nurs. 32 (3), 157–162 (2005)

D.Z. Bliss, K. Savik, Writing a grant proposal—Part 2. J. Wound Ostomy Cont. Nurs. 32 (4), 226–229 (2005)

D.Z. Bliss, Writing a grant proposal—Part 6. J. Wound Ostomy Cont. Nurs. 32 (6), 365–367 (2005)

J.C. Liu, M.A. Pynnonen, M. St John, E.L. Rosenthal, M.E. Couch, C.E. Schmalbach, Grant-writing pearls and pitfalls. Otolaryngol. Neck. Surg. 154 (2), 226–232 (2016)

R.J. Santen, E.J. Barrett, H.M. Siragy, L.S. Farhi, L. Fishbein, R.M. Carey, The jewel in the crown: Specific aims section of investigator-initiated grant proposals. J. Endocr. Soc. 1 (9), 1194–1202 (2017)

O.J. Arthurs, Think it through first: Questions to consider in writing a successful grant application. Pediatr. Radiol. 44 (12), 1507–1511 (2014)

M. Monavarian, Basics of scientific and technical writing. MRS Bull. 46 (3), 284–286 (2021)

Additional Resources

https://grants.nih.gov

https://grants.nih.gov/grants/oer.htm

https://www.ninr.nih.gov

https://www.niaid.nih.gov

http://www.grantcentral.com

http://www.saem.org/research

http://www.cfda.gov

http://www.ahrq.gov

http://www.nsf/gov

Download references

Author information

Authors and affiliations.

School of Dentistry, Marquette University, Milwaukee, WI, USA

Fahimeh Tabatabaei & Lobat Tayebi

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this chapter

Tabatabaei, F., Tayebi, L. (2022). Writing a Research Proposal. In: Research Methods in Dentistry. Springer, Cham. https://doi.org/10.1007/978-3-030-98028-3_4

Download citation

DOI : https://doi.org/10.1007/978-3-030-98028-3_4

Published : 10 April 2022

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-98027-6

Online ISBN : 978-3-030-98028-3

eBook Packages : Engineering Engineering (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

11.2 Steps in Developing a Research Proposal

Learning objectives.

• Identify the steps in developing a research proposal.
• Choose a topic and formulate a research question and working thesis.
• Develop a research proposal.

Writing a good research paper takes time, thought, and effort. Although this assignment is challenging, it is manageable. Focusing on one step at a time will help you develop a thoughtful, informative, well-supported research paper.

Your first step is to choose a topic and then to develop research questions, a working thesis, and a written research proposal. Set aside adequate time for this part of the process. Fully exploring ideas will help you build a solid foundation for your paper.

Choosing a Topic

When you choose a topic for a research paper, you are making a major commitment. Your choice will help determine whether you enjoy the lengthy process of research and writing—and whether your final paper fulfills the assignment requirements. If you choose your topic hastily, you may later find it difficult to work with your topic. By taking your time and choosing carefully, you can ensure that this assignment is not only challenging but also rewarding.

Writers understand the importance of choosing a topic that fulfills the assignment requirements and fits the assignment’s purpose and audience. (For more information about purpose and audience, see Chapter 6 “Writing Paragraphs: Separating Ideas and Shaping Content” .) Choosing a topic that interests you is also crucial. You instructor may provide a list of suggested topics or ask that you develop a topic on your own. In either case, try to identify topics that genuinely interest you.

After identifying potential topic ideas, you will need to evaluate your ideas and choose one topic to pursue. Will you be able to find enough information about the topic? Can you develop a paper about this topic that presents and supports your original ideas? Is the topic too broad or too narrow for the scope of the assignment? If so, can you modify it so it is more manageable? You will ask these questions during this preliminary phase of the research process.

Identifying Potential Topics

Sometimes, your instructor may provide a list of suggested topics. If so, you may benefit from identifying several possibilities before committing to one idea. It is important to know how to narrow down your ideas into a concise, manageable thesis. You may also use the list as a starting point to help you identify additional, related topics. Discussing your ideas with your instructor will help ensure that you choose a manageable topic that fits the requirements of the assignment.

In this chapter, you will follow a writer named Jorge, who is studying health care administration, as he prepares a research paper. You will also plan, research, and draft your own research paper.

Jorge was assigned to write a research paper on health and the media for an introductory course in health care. Although a general topic was selected for the students, Jorge had to decide which specific issues interested him. He brainstormed a list of possibilities.

If you are writing a research paper for a specialized course, look back through your notes and course activities. Identify reading assignments and class discussions that especially engaged you. Doing so can help you identify topics to pursue.

• Health Maintenance Organizations (HMOs) in the news
• Sexual education programs
• Hollywood and eating disorders
• Americans’ access to public health information
• Media portrayal of health care reform bill
• Depictions of drugs on television
• The effect of the Internet on mental health
• Popularized diets (such as low-carbohydrate diets)
• Fear of pandemics (bird flu, HINI, SARS)
• Electronic entertainment and obesity
• Advertisements for prescription drugs
• Public education and disease prevention

Set a timer for five minutes. Use brainstorming or idea mapping to create a list of topics you would be interested in researching for a paper about the influence of the Internet on social networking. Do you closely follow the media coverage of a particular website, such as Twitter? Would you like to learn more about a certain industry, such as online dating? Which social networking sites do you and your friends use? List as many ideas related to this topic as you can.

Narrowing Your Topic

Once you have a list of potential topics, you will need to choose one as the focus of your essay. You will also need to narrow your topic. Most writers find that the topics they listed during brainstorming or idea mapping are broad—too broad for the scope of the assignment. Working with an overly broad topic, such as sexual education programs or popularized diets, can be frustrating and overwhelming. Each topic has so many facets that it would be impossible to cover them all in a college research paper. However, more specific choices, such as the pros and cons of sexual education in kids’ television programs or the physical effects of the South Beach diet, are specific enough to write about without being too narrow to sustain an entire research paper.

A good research paper provides focused, in-depth information and analysis. If your topic is too broad, you will find it difficult to do more than skim the surface when you research it and write about it. Narrowing your focus is essential to making your topic manageable. To narrow your focus, explore your topic in writing, conduct preliminary research, and discuss both the topic and the research with others.

Exploring Your Topic in Writing

“How am I supposed to narrow my topic when I haven’t even begun researching yet?” In fact, you may already know more than you realize. Review your list and identify your top two or three topics. Set aside some time to explore each one through freewriting. (For more information about freewriting, see Chapter 8 “The Writing Process: How Do I Begin?” .) Simply taking the time to focus on your topic may yield fresh angles.

Jorge knew that he was especially interested in the topic of diet fads, but he also knew that it was much too broad for his assignment. He used freewriting to explore his thoughts so he could narrow his topic. Read Jorge’s ideas.

Conducting Preliminary Research

Another way writers may focus a topic is to conduct preliminary research . Like freewriting, exploratory reading can help you identify interesting angles. Surfing the web and browsing through newspaper and magazine articles are good ways to start. Find out what people are saying about your topic on blogs and online discussion groups. Discussing your topic with others can also inspire you. Talk about your ideas with your classmates, your friends, or your instructor.

Jorge’s freewriting exercise helped him realize that the assigned topic of health and the media intersected with a few of his interests—diet, nutrition, and obesity. Preliminary online research and discussions with his classmates strengthened his impression that many people are confused or misled by media coverage of these subjects.

Jorge decided to focus his paper on a topic that had garnered a great deal of media attention—low-carbohydrate diets. He wanted to find out whether low-carbohydrate diets were as effective as their proponents claimed.

Writing at Work

At work, you may need to research a topic quickly to find general information. This information can be useful in understanding trends in a given industry or generating competition. For example, a company may research a competitor’s prices and use the information when pricing their own product. You may find it useful to skim a variety of reliable sources and take notes on your findings.

The reliability of online sources varies greatly. In this exploratory phase of your research, you do not need to evaluate sources as closely as you will later. However, use common sense as you refine your paper topic. If you read a fascinating blog comment that gives you a new idea for your paper, be sure to check out other, more reliable sources as well to make sure the idea is worth pursuing.

Review the list of topics you created in Note 11.18 “Exercise 1” and identify two or three topics you would like to explore further. For each of these topics, spend five to ten minutes writing about the topic without stopping. Then review your writing to identify possible areas of focus.

Set aside time to conduct preliminary research about your potential topics. Then choose a topic to pursue for your research paper.

Collaboration

Please share your topic list with a classmate. Select one or two topics on his or her list that you would like to learn more about and return it to him or her. Discuss why you found the topics interesting, and learn which of your topics your classmate selected and why.

A Plan for Research

Your freewriting and preliminary research have helped you choose a focused, manageable topic for your research paper. To work with your topic successfully, you will need to determine what exactly you want to learn about it—and later, what you want to say about it. Before you begin conducting in-depth research, you will further define your focus by developing a research question , a working thesis, and a research proposal.

Formulating a Research Question

In forming a research question, you are setting a goal for your research. Your main research question should be substantial enough to form the guiding principle of your paper—but focused enough to guide your research. A strong research question requires you not only to find information but also to put together different pieces of information, interpret and analyze them, and figure out what you think. As you consider potential research questions, ask yourself whether they would be too hard or too easy to answer.

To determine your research question, review the freewriting you completed earlier. Skim through books, articles, and websites and list the questions you have. (You may wish to use the 5WH strategy to help you formulate questions. See Chapter 8 “The Writing Process: How Do I Begin?” for more information about 5WH questions.) Include simple, factual questions and more complex questions that would require analysis and interpretation. Determine your main question—the primary focus of your paper—and several subquestions that you will need to research to answer your main question.

Here are the research questions Jorge will use to focus his research. Notice that his main research question has no obvious, straightforward answer. Jorge will need to research his subquestions, which address narrower topics, to answer his main question.

Using the topic you selected in Note 11.24 “Exercise 2” , write your main research question and at least four to five subquestions. Check that your main research question is appropriately complex for your assignment.

Constructing a Working ThesIs

A working thesis concisely states a writer’s initial answer to the main research question. It does not merely state a fact or present a subjective opinion. Instead, it expresses a debatable idea or claim that you hope to prove through additional research. Your working thesis is called a working thesis for a reason—it is subject to change. As you learn more about your topic, you may change your thinking in light of your research findings. Let your working thesis serve as a guide to your research, but do not be afraid to modify it based on what you learn.

Jorge began his research with a strong point of view based on his preliminary writing and research. Read his working thesis statement, which presents the point he will argue. Notice how it states Jorge’s tentative answer to his research question.

One way to determine your working thesis is to consider how you would complete sentences such as I believe or My opinion is . However, keep in mind that academic writing generally does not use first-person pronouns. These statements are useful starting points, but formal research papers use an objective voice.

Write a working thesis statement that presents your preliminary answer to the research question you wrote in Note 11.27 “Exercise 3” . Check that your working thesis statement presents an idea or claim that could be supported or refuted by evidence from research.

Creating a Research Proposal

A research proposal is a brief document—no more than one typed page—that summarizes the preliminary work you have completed. Your purpose in writing it is to formalize your plan for research and present it to your instructor for feedback. In your research proposal, you will present your main research question, related subquestions, and working thesis. You will also briefly discuss the value of researching this topic and indicate how you plan to gather information.

When Jorge began drafting his research proposal, he realized that he had already created most of the pieces he needed. However, he knew he also had to explain how his research would be relevant to other future health care professionals. In addition, he wanted to form a general plan for doing the research and identifying potentially useful sources. Read Jorge’s research proposal.

Before you begin a new project at work, you may have to develop a project summary document that states the purpose of the project, explains why it would be a wise use of company resources, and briefly outlines the steps involved in completing the project. This type of document is similar to a research proposal. Both documents define and limit a project, explain its value, discuss how to proceed, and identify what resources you will use.

Writing Your Own Research Proposal

Now you may write your own research proposal, if you have not done so already. Follow the guidelines provided in this lesson.

Key Takeaways

• Developing a research proposal involves the following preliminary steps: identifying potential ideas, choosing ideas to explore further, choosing and narrowing a topic, formulating a research question, and developing a working thesis.
• A good topic for a research paper interests the writer and fulfills the requirements of the assignment.
• Defining and narrowing a topic helps writers conduct focused, in-depth research.
• Writers conduct preliminary research to identify possible topics and research questions and to develop a working thesis.
• A good research question interests readers, is neither too broad nor too narrow, and has no obvious answer.
• A good working thesis expresses a debatable idea or claim that can be supported with evidence from research.
• Writers create a research proposal to present their topic, main research question, subquestions, and working thesis to an instructor for approval or feedback.

Writing for Success Copyright © 2015 by University of Minnesota is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

Prevent plagiarism. Run a free check.

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved April 8, 2024, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, what is your plagiarism score.

Research Proposal Example/Sample

Detailed Walkthrough + Free Proposal Template

If you’re getting started crafting your research proposal and are looking for a few examples of research proposals , you’ve come to the right place.

In this video, we walk you through two successful (approved) research proposals , one for a Master’s-level project, and one for a PhD-level dissertation. We also start off by unpacking our free research proposal template and discussing the four core sections of a research proposal, so that you have a clear understanding of the basics before diving into the actual proposals.

• Research proposal example/sample – Master’s-level (PDF/Word)
• Research proposal example/sample – PhD-level (PDF/Word)
• Proposal template (Fully editable) 

If you’re working on a research proposal for a dissertation or thesis, you may also find the following useful:

• Research Proposal Bootcamp : Learn how to write a research proposal as efficiently and effectively as possible
• 1:1 Proposal Coaching : Get hands-on help with your research proposal

FAQ: Research Proposal Example

Research proposal example: frequently asked questions, are the sample proposals real.

Yes. The proposals are real and were approved by the respective universities.

Can I copy one of these proposals for my own research?

As we discuss in the video, every research proposal will be slightly different, depending on the university’s unique requirements, as well as the nature of the research itself. Therefore, you’ll need to tailor your research proposal to suit your specific context.

You can learn more about the basics of writing a research proposal here .

How do I get the research proposal template?

You can access our free proposal template here .

Is the proposal template really free?

Yes. There is no cost for the proposal template and you are free to use it as a foundation for your research proposal.

Where can I learn more about proposal writing?

For self-directed learners, our Research Proposal Bootcamp is a great starting point.

For students that want hands-on guidance, our private coaching service is recommended.

Psst… there’s more!

This post is an extract from our bestselling Udemy Course, Research Proposal Bootcamp . If you want to work smart, you don't want to miss this .

You Might Also Like:

I am at the stage of writing my thesis proposal for a PhD in Management at Altantic International University. I checked on the coaching services, but it indicates that it’s not available in my area. I am in South Sudan. My proposed topic is: “Leadership Behavior in Local Government Governance Ecosystem and Service Delivery Effectiveness in Post Conflict Districts of Northern Uganda”. I will appreciate your guidance and support

GRADCOCH is very grateful motivated and helpful for all students etc. it is very accorporated and provide easy access way strongly agree from GRADCOCH.

Proposal research departemet management

I am at the stage of writing my thesis proposal for a masters in Analysis of w heat commercialisation by small holders householdrs at Hawassa International University. I will appreciate your guidance and support

please provide a attractive proposal about foreign universities .It would be your highness.

comparative constitutional law

Kindly guide me through writing a good proposal on the thesis topic; Impact of Artificial Intelligence on Financial Inclusion in Nigeria. Thank you

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

Academic Development Centre

Research proposals, what can research proposals assess.

Warwick is a research-intensive university and we claim that our graduates leave with an understanding of, and capability in, research. It is important to recognise that the research undertaken by undergraduate and postgraduate taught students might not produce knowledge which is new to the world, but that researching a topic that is new to them will still enable them to develop their skills, knowledge, and understanding.

One way to support their development as researchers is to ask them to develop a research proposal, certainly as a stage towards their final year project. A possible progression would be an annotated bibliography in year 1, a research proposal in year 2 and a full research project in the final year.

A research proposal requires learners to select a topic [intrinsic motivation as it is their choice] and through analysis and synthesis of knowledge, decide an original research question and then propose a way to answer that problem. The requirement to complete a literature review will deepen their knowledge and understanding of the topic and selecting an appropriate research methodology means they need to know the ‘standard’ approaches used in the discipline.

Research proposals will bear the hallmarks of their discipline, but one might expect:

• background and rationale
• research question(s)
• research methodology
• plan of work & time schedule
• bibliography

The ability to carry out a literature review to develop the research question, and to generate the bibliography and to select a research methodology means this is an ideal synoptic task to follow courses on both topics. Work plans and schedules will get the learners into the realms of time management and estimates and these are valuable generic skills.

Finally, you need to think how the students will present their proposal: written, oral presentation or poster, and whether this will be an individual or group task. All of these are valuable generic skills that need to be developed in our graduates.

Diversity & inclusion

The range of possible topics [home and international] and appropriate choice of time frames to complete the task [particular needs] will help make the task more inclusive.

Academic integrity

Students are likely to be more personally invested in a topic that they have themselves chosen, which increases their intrinsic motivation and enables them to produce work that they are proud of. This promotes academic integrity.

The individuated nature of these research assignments also reduces the risk of 'answers' floating around on the internet which can help to reduce year-to-year plagiarism.

Support for students to manage their time effectively, regular milestones and check-ins - perhaps using the list above to structure stages - will help keep students on track, reducing deadline stress which might be particularly acute given a large piece of work (likely to be heavily weighted and therefore high stakes). This in turn should reduce motivation to take short cuts which might compromise academic integrity.

Generating evidence over time further encourages academic integrity and makes academic misconduct more easily detectable (which may also act as a deterrent). (Click here for further  guidance on academic integrity in assessment .)

Generative AI is already being utilised as a tool to support academic researchers, and may be authentically used to support all stages in the creation of a research proposal. Integrating skills development for productive and ethical use of AI is likely to become an increasingly important aspect of researcher development. Banning the use of AI in this sort of assessment may therefore be undesirable. Adding a reflective piece or requirement to provide commentary on how the research proposal was developed (research journal/diary or blog) might be helpful if you want to place constraints on the use of AI.

Student and staff experience

Learning through research is a great way to engage learners, especially within a culture like ours, and there are opportunities to develop a range of transferable skills.

If you think about your own development in this aspect of academe you will, no doubt, recall the difficulties of the early researcher and the support that you needed. You need to plan for this support and guidance for your students.

We might need to check that students do not spend too long on this sort of task to the detriment of other work (on other modules, as research tends to take as much time as it is given).

Multiple titles can add to the setting, specifying and marking load for staff.

Annotated bibliography

Class participation

Concept maps

Essay variants: essays only with more focus

• briefing / policy papers
• research proposals
• articles and reviews
• essay plans

Film production

Laboratory notebooks and reports

Objective tests

• short-answer
• multiple choice questions

Oral presentations

Patchwork assessment

Creative / artistic performance

• learning logs
• learning blogs

Simulations

Work-based assessment

Reference list

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Research process
• How to Write a Research Proposal | Examples & Templates

How to Write a Research Proposal | Examples & Templates

Published on 30 October 2022 by Shona McCombes and Tegan George. Revised on 13 June 2023.

A research proposal describes what you will investigate, why it’s important, and how you will conduct your research.

The format of a research proposal varies between fields, but most proposals will contain at least these elements:

Introduction

Literature review.

• Research design

Reference list

While the sections may vary, the overall objective is always the same. A research proposal serves as a blueprint and guide for your research plan, helping you get organised and feel confident in the path forward you choose to take.

Table of contents

Research proposal purpose, research proposal examples, research design and methods, contribution to knowledge, research schedule, frequently asked questions.

Academics often have to write research proposals to get funding for their projects. As a student, you might have to write a research proposal as part of a grad school application , or prior to starting your thesis or dissertation .

In addition to helping you figure out what your research can look like, a proposal can also serve to demonstrate why your project is worth pursuing to a funder, educational institution, or supervisor.

Research proposal length

The length of a research proposal can vary quite a bit. A bachelor’s or master’s thesis proposal can be just a few pages, while proposals for PhD dissertations or research funding are usually much longer and more detailed. Your supervisor can help you determine the best length for your work.

One trick to get started is to think of your proposal’s structure as a shorter version of your thesis or dissertation , only without the results , conclusion and discussion sections.

Download our research proposal template

Prevent plagiarism, run a free check.

Writing a research proposal can be quite challenging, but a good starting point could be to look at some examples. We’ve included a few for you below.

• Example research proposal #1: ‘A Conceptual Framework for Scheduling Constraint Management’
• Example research proposal #2: ‘ Medical Students as Mediators of Change in Tobacco Use’

Like your dissertation or thesis, the proposal will usually have a title page that includes:

• The proposed title of your project
• Your supervisor’s name
• Your institution and department

The first part of your proposal is the initial pitch for your project. Make sure it succinctly explains what you want to do and why.

Your introduction should:

• Introduce your topic
• Give necessary background and context
• Outline your  problem statement  and research questions

To guide your introduction , include information about:

• Who could have an interest in the topic (e.g., scientists, policymakers)
• How much is already known about the topic
• What is missing from this current knowledge
• What new insights your research will contribute
• Why you believe this research is worth doing

As you get started, it’s important to demonstrate that you’re familiar with the most important research on your topic. A strong literature review  shows your reader that your project has a solid foundation in existing knowledge or theory. It also shows that you’re not simply repeating what other people have already done or said, but rather using existing research as a jumping-off point for your own.

In this section, share exactly how your project will contribute to ongoing conversations in the field by:

• Comparing and contrasting the main theories, methods, and debates
• Examining the strengths and weaknesses of different approaches
• Explaining how will you build on, challenge, or synthesise prior scholarship

Following the literature review, restate your main  objectives . This brings the focus back to your own project. Next, your research design or methodology section will describe your overall approach, and the practical steps you will take to answer your research questions.

To finish your proposal on a strong note, explore the potential implications of your research for your field. Emphasise again what you aim to contribute and why it matters.

For example, your results might have implications for:

• Improving best practices
• Informing policymaking decisions
• Strengthening a theory or model
• Challenging popular or scientific beliefs
• Creating a basis for future research

Last but not least, your research proposal must include correct citations for every source you have used, compiled in a reference list . To create citations quickly and easily, you can use our free APA citation generator .

Some institutions or funders require a detailed timeline of the project, asking you to forecast what you will do at each stage and how long it may take. While not always required, be sure to check the requirements of your project.

Here’s an example schedule to help you get started. You can also download a template at the button below.

Download our research schedule template

If you are applying for research funding, chances are you will have to include a detailed budget. This shows your estimates of how much each part of your project will cost.

Make sure to check what type of costs the funding body will agree to cover. For each item, include:

• Cost : exactly how much money do you need?
• Justification : why is this cost necessary to complete the research?
• Source : how did you calculate the amount?

To determine your budget, think about:

• Travel costs : do you need to go somewhere to collect your data? How will you get there, and how much time will you need? What will you do there (e.g., interviews, archival research)?
• Materials : do you need access to any tools or technologies?
• Help : do you need to hire any research assistants for the project? What will they do, and how much will you pay them?

Once you’ve decided on your research objectives , you need to explain them in your paper, at the end of your problem statement.

Keep your research objectives clear and concise, and use appropriate verbs to accurately convey the work that you will carry out for each one.

I will compare …

A research aim is a broad statement indicating the general purpose of your research project. It should appear in your introduction at the end of your problem statement , before your research objectives.

Research objectives are more specific than your research aim. They indicate the specific ways you’ll address the overarching aim.

A PhD, which is short for philosophiae doctor (doctor of philosophy in Latin), is the highest university degree that can be obtained. In a PhD, students spend 3–5 years writing a dissertation , which aims to make a significant, original contribution to current knowledge.

A PhD is intended to prepare students for a career as a researcher, whether that be in academia, the public sector, or the private sector.

A master’s is a 1- or 2-year graduate degree that can prepare you for a variety of careers.

All master’s involve graduate-level coursework. Some are research-intensive and intend to prepare students for further study in a PhD; these usually require their students to write a master’s thesis . Others focus on professional training for a specific career.

Critical thinking refers to the ability to evaluate information and to be aware of biases or assumptions, including your own.

Like information literacy , it involves evaluating arguments, identifying and solving problems in an objective and systematic way, and clearly communicating your ideas.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

McCombes, S. & George, T. (2023, June 13). How to Write a Research Proposal | Examples & Templates. Scribbr. Retrieved 8 April 2024, from https://www.scribbr.co.uk/the-research-process/research-proposal-explained/

Is this article helpful?

Shona McCombes

Other students also liked, what is a research methodology | steps & tips, what is a literature review | guide, template, & examples, how to write a results section | tips & examples.

Organizing Academic Research Papers: Writing a Research Proposal

• Purpose of Guide
• Design Flaws to Avoid
• Glossary of Research Terms
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Choosing a Title
• Making an Outline
• Paragraph Development
• Executive Summary
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tertiary Sources
• What Is Scholarly vs. Popular?
• Qualitative Methods
• Quantitative Methods
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Annotated Bibliography
• Dealing with Nervousness
• Using Visual Aids
• Grading Someone Else's Paper
• How to Manage Group Projects
• Multiple Book Review Essay
• Reviewing Collected Essays
• About Informed Consent
• Writing Field Notes
• Writing a Policy Memo
• Writing a Research Proposal
• Acknowledgements

The goal of a research proposal is to present and justify a research idea you have and to present the practical ways in which you think this research should be conducted. The forms and procedures for such research are defined by the field of study, so guidelines for research proposals are generally more exacting and less formal than a project proposal. Research proposals contain extensive literature reviews and must provide persuasive evidence that there is a need for the research study being proposed. In addition to providing rationale for the proposed research, a proposal describes detailed methodology for conducting the research consistent with requirements of the professional or academic field and a statement on anticipated outcomes and/or benefits derived from the study.

Krathwohl, David R. How to Prepare a Dissertation Proposal: Suggestions for Students in Education and the Social and Behavioral Sciences . Syracuse, NY: Syracuse University Press, 2005.

How to Approach Writing a Research Proposal

Your professor may assign the task of writing a research proposal for the following reasons:

• Develop your skills in thinking about and designing a comprehensive research study.
• Help learn how to conduct a comprehensive review of the literature to ensure a research problem has not already been answered [or you may determine the problem has been answered ineffectively] and, in so doing, become familiar with scholarship related to your topic.
• Improve your general research and writing skills.
• Practice identifying what logical steps must be taken to accomplish one's research goals.
• Nurture a sense of inquisitiveness within yourself and to help see yourself as an active participant in the process of doing scholarly research.

A proposal should contain all the key elements involved in designing a complete research study, with sufficient information that allows readers to assess the validity and usefulness of your proposed study. The only elements missing from a research proposal are the results of the study and your analysis of those results. Finally, an effective proposal is judged on the quality of your writing. It is, therefore, important that your writing is coherent, clear, and compelling.

Regardless of the research problem you are investigating and the methodology you choose, all research proposals must address the following questions:

• What do you plan to accomplish? Be clear and succient in defining the research problem and what it is you are proposing to research.
• Why do you want to do it? In addition to detailing your research design, you also must conduct a thorough review of the literature and provide convincing evidence that it is a topic worthy of study. Be sure to answer the "So what? question.
• How are you going to do it? Be sure that what you propose is doable.

Common Mistakes to Avoid

• Failure to be concise; being "all over the map" without a clear sense of purpose.
• Failure to cite landmark works in your literature review.
• Failure to delimit the contextual boundaries of your research [e.g., time, place, people, etc.].
• Failure to develop a coherent and persuasive argument for the proposed research.
• Failure to stay focused on the research question; going off on unrelated tangents.
• Sloppy or imprecise writing. Poor grammar.
• Too much detail on minor issues, but not enough detail on major issues.

Procter, Margaret. The Academic Proposal .  The Lab Report. University College Writing Centre. University of Toronto; Sanford, Keith. Information for Students: Writing a Research Proposal . Baylor University; Wong, Paul T. P. How to Write a Research Proposal . International Network on Personal Meaning. Trinity Western University; Writing Academic Proposals: Conferences, Articles, and Books . The Writing Lab and The OWL. Purdue University; Writing a Research Proposal. University Library. University of Illinois at Urbana-Champaign.

Structure and Writing Style

Beginning the Proposal Process

As with writing a traditional research paper, research proposals are generally organized the same way throughout the social sciences. Most proposals are between ten and fifteen pages in length. However, before you begin, read the assignment carefully and, if anything seems unclear, ask your professor whether there are any specific requirements for organizing and writing the proposal.

A good place to begin is to ask yourself a series of questions:

• What do I want to study, and why?
• How is it significant within the subject areas covered in my class?
• What problems will it help solve?
• How does it build upon [and hopefully go beyond] research already conducted on my topic?
• What exactly should I plan to do, and can I get it done in the time available?

In the end, your research proposal should document your knowledge of the topic and highlight enthusiasm for conducting the study. Approach it with the intention of leaving your readers feeling like--"Wow, that's an exciting idea and I can’t wait to see how it turns out!"

In general your proposal should include the following sections:

I.  Introduction

In the real world of higher education, a research proposal is most often written by scholars seeking grant funding for a research project or it's the first step in getting approval to write your doctoral dissertation. Even if this is just a course assignment, treat your introduction as the initial pitch of an idea. After reading the introduction, your readers should not only have an understanding of what you want to do, but they should also be able to sense your passion for the topic and be excited about its possible outcomes.

Think about your introduction as a narrative written in one to three paragraphs that succinctly answers the following four questions :

• What is the central research problem?
• What is the topic of study related to that problem?
• What methods should be used to analyze the research problem?
• Why is this important research, and why should someone reading the proposal care about the outcomes from the study?

II.  Background and Significance

This section can be melded into your introduction or you can create a separate section to help with the organization and flow of your proposal. This is where you explain the context of your project and outline why it's important. Approach writing this section with the thought that you can’t assume your readers will know as much about the research problem as you do. Note that this section is not an essay going over everything you have learned about the research problem; instead, you must choose what is relevant to help explain your goals for the study.

To that end, while there are no hard and fast rules, you should attempt to deal with some or all of the following:

• State the research problem and give a more detailed explanation about the purpose of the study than what you stated in the introduction.
• Present the rationale of your proposed study and clearly indicate why it is worth doing. Answer the "So what? question [i.e., why should anyone care].
• Describe the major issues or problems to be addressed by your research.
• Explain how you plan to go about conducting your research. Clearly identify the key sources you intend to use and explain how they will contribute to the analysis of your topic.
• Set the boundaries of your proposed research in order to provide a clear focus.
• Provide definitions of key concepts or terms, if necessary.

III.  Literature Review

Connected to the background and significance of your study is a more deliberate review and synthesis of prior studies related to the research problem under investigation . The purpose here is to place your project within the larger whole of what is currently being explored, while demonstrating to your readers that your work is original and innovative. Think about what questions other researchers have asked, what methods they've used, and what is your understanding of their findings. Assess what you believe is still missing, and state how previous research has failed to examine the issue that your study addresses.

Since a literature review is information dense, it is crucial that this section is intelligently structured to enable a reader to grasp the key arguments underpinning your study in relation to that of other researchers. A good strategy is to break the literature into "conceptual categories" [themes] rather than systematically describing materials one at a time.

To help frame your proposal's literature review, here are the "five C’s" of writing a literature review:

• Cite : keep the primary focus on the literature pertinent to your research problem.
• Compare the various arguments, theories, methodologies, and findings expressed in the literature: what do the authors agree on? Who applies similar approaches to analyzing the research problem?
• Contrast the various arguments, themes, methodologies, approaches and controversies expressed in the literature: what are the major areas of disagreement, controversy, or debate?
• Critique the literature: Which arguments are more persuasive, and why? Which approaches, findings, methodologies seem most reliable, valid, or appropriate, and why? Pay attention to the verbs you use to describe what an author says/does [e.g., asserts, demonstrates, etc.] .
• Connect the literature to your own area of research and investigation: how does your own work draw upon, depart from, or synthesize what has been said in the literature?

IV.  Research Design and Methods

This section must be well-written and logically organized because you are not actually doing the research . As a consequence, the reader will never have a study outcome from which to evaluate whether your methodological choices were the correct ones. The objective here is to ensure that the reader is convinced that your overall research design and methods of analysis will correctly address the research problem. Your design and methods should be absolutely and unmistakably tied to the specific aims of your study.

Describe the overall research design by building upon and drawing examples from your review of the literature. Be specific about the methodological approaches you plan to undertake to collect information, about the techniques you will use to analyze it, and about tests of external validity to which you commit yourself [i.e., the trustworthiness by which you can generalize from your study to other people, places or times].

When describing the methods you will use, be sure to cover these issues:

• Specify the research operations you will undertake and the way you will interpret the results of these operations in relation to your research problem. Don't just describe what you intend to achieve from applying the methods you choose, but state how you will spend your time while doing it.
• Keep in mind that a methodology is not just a list of research tasks; it is an argument as to why these tasks add up to the best way to investigate the research problem. This is an important point because the mere listing of tasks to perform does not demonstrate that they add up to the best feasible approach.
• Be sure to anticipate and acknowledge any potential barriers and pitfalls in carrying out your research design and explain how you plan to get around them.

V.  Preliminary Suppositions and Implications

Just because you don't have to actually conduct the study and analyze the results, it doesn't mean that you can skip talking about the process and potential implications . The purpose of this section is to argue how and in what ways you believe your research will refine, revise, or extend existing knowledge in the subject area under investigation. Depending on the aims and objectives of your study, describe how the anticipated results of your study will impact future scholarly research, theory, practice, forms of interventions, or policy. Note that such discussions may have either substantive [a potential new policy], theoretical [a potential new understanding], or methodological [a potential new way of analyzing] significance.   When thinking about the potential implications of your study, ask the following questions:

• What might the results mean in regards to the theoretical framework that frames the study?
• What suggestions for subsequent research could arise from the potential outcomes of the study?
• What will the results mean to practitioners in the "real world"?
• Will the results influence programs, methods, and/or forms of intervention?
• How might the results contribute to the solution of social, economic, or other types of problems?
• Will the results influence policy decisions?
• What will be improved or changed as a result of the proposed research?
• How will the results of the study be implemented, and what innovations will come about?

VI.  Conclusion

The conclusion reiterates the importance or significance of your proposal and provides a brief recap of the entire study . This section should be only one or two paragraphs long, emphasizing why your research study is unique, why it advances knowledge, and why the research problem is worth investigating.

Someone reading this section should come away with an understanding of:

• Why the study was done,
• The specific purpose of the study and the research questions it attempted to answer,
• The research design and methods used,
• The potential implications emerging from your proposed study of the research problem, and
• A sense of how your study fits within the broader scholarship about the research problem.

VII.  Citations

As with any scholarly research paper, you must cite the sources you used in composing your proposal. In a standard research proposal, this section can take two forms, so speak with your professor about which one is preferred.

• References -- lists only the literature that you actually used or cited in your proposal.
• Bibliography -- lists everything you used or cited in your proposal with additional citations of any key sources relevant to understanding the research problem.

In either case, this section should testify to the fact that you did enough preparatory work to make sure the project will complement and not duplicate the efforts of other researchers. Start a new page and use the heading "References" or "Bibliography" at the top of the page. Cited works should always use a standard format that follows the writing style advised by the discipline of your course [i.e., education=APA; history=Chicago, etc]. This section normally does not count towards the total length of your proposal.

Develop a Research Proposal: Writing the Proposal . Office of Library Information Services. Baltimore County Public Schools; Krathwohl, David R. How to Prepare a Dissertation Proposal: Suggestions for Students in Education and the Social and Behavioral Sciences. Syracuse, NY: Syracuse University Press, 2005; Procter, Margaret. The Academic Proposal . The Lab Report. University College Writing Centre. University of Toronto; Punch, Keith and Wayne McGowan. Developing and Writing a Research Proposal. In From Postgraduate to Social Scientist: A Guide to Key Skills. Nigel Gilbert, ed. (Thousand Oaks, CA: Sage, 2006), 59-81; Sanford, Keith. Information for Students: Writing a Research Proposal . Baylor University; Wong, Paul T. P. How to Write a Research Proposal . International Network on Personal Meaning. Trinity Western University; Writing Academic Proposals: Conferences, Articles, and Books . The Writing Lab and The OWL. Purdue University; Writing a Research Proposal . University Library. University of Illinois at Urbana-Champaign.

• << Previous: Writing a Policy Memo
• Next: Acknowledgements >>
• Last Updated: Jul 18, 2023 11:58 AM
• URL: https://library.sacredheart.edu/c.php?g=29803
• QuickSearch
• Library Catalog
• Databases A-Z
• Publication Finder
• Course Reserves
• Citation Linker
• Digital Commons
• Our Website

Research Support

• Ask a Librarian
• Appointments
• Interlibrary Loan (ILL)
• Research Guides
• Databases by Subject
• Citation Help

Using the Library

• Reserve a Group Study Room
• Renew Books
• Honors Study Rooms
• Off-Campus Access
• Library Policies
• Library Technology

User Information

• Grad Students
• Online Students
• COVID-19 Updates
• Staff Directory
• News & Announcements
• Library Newsletter

My Accounts

• Interlibrary Loan
• Staff Site Login

FIND US ON  

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Indian J Anaesth
• v.60(9); 2016 Sep

How to write a research proposal?

Department of Anaesthesiology, Bangalore Medical College and Research Institute, Bengaluru, Karnataka, India

Devika Rani Duggappa

Writing the proposal of a research work in the present era is a challenging task due to the constantly evolving trends in the qualitative research design and the need to incorporate medical advances into the methodology. The proposal is a detailed plan or ‘blueprint’ for the intended study, and once it is completed, the research project should flow smoothly. Even today, many of the proposals at post-graduate evaluation committees and application proposals for funding are substandard. A search was conducted with keywords such as research proposal, writing proposal and qualitative using search engines, namely, PubMed and Google Scholar, and an attempt has been made to provide broad guidelines for writing a scientifically appropriate research proposal.

INTRODUCTION

A clean, well-thought-out proposal forms the backbone for the research itself and hence becomes the most important step in the process of conduct of research.[ 1 ] The objective of preparing a research proposal would be to obtain approvals from various committees including ethics committee [details under ‘Research methodology II’ section [ Table 1 ] in this issue of IJA) and to request for grants. However, there are very few universally accepted guidelines for preparation of a good quality research proposal. A search was performed with keywords such as research proposal, funding, qualitative and writing proposals using search engines, namely, PubMed, Google Scholar and Scopus.

Five ‘C’s while writing a literature review

BASIC REQUIREMENTS OF A RESEARCH PROPOSAL

A proposal needs to show how your work fits into what is already known about the topic and what new paradigm will it add to the literature, while specifying the question that the research will answer, establishing its significance, and the implications of the answer.[ 2 ] The proposal must be capable of convincing the evaluation committee about the credibility, achievability, practicality and reproducibility (repeatability) of the research design.[ 3 ] Four categories of audience with different expectations may be present in the evaluation committees, namely academic colleagues, policy-makers, practitioners and lay audiences who evaluate the research proposal. Tips for preparation of a good research proposal include; ‘be practical, be persuasive, make broader links, aim for crystal clarity and plan before you write’. A researcher must be balanced, with a realistic understanding of what can be achieved. Being persuasive implies that researcher must be able to convince other researchers, research funding agencies, educational institutions and supervisors that the research is worth getting approval. The aim of the researcher should be clearly stated in simple language that describes the research in a way that non-specialists can comprehend, without use of jargons. The proposal must not only demonstrate that it is based on an intelligent understanding of the existing literature but also show that the writer has thought about the time needed to conduct each stage of the research.[ 4 , 5 ]

CONTENTS OF A RESEARCH PROPOSAL

The contents or formats of a research proposal vary depending on the requirements of evaluation committee and are generally provided by the evaluation committee or the institution.

In general, a cover page should contain the (i) title of the proposal, (ii) name and affiliation of the researcher (principal investigator) and co-investigators, (iii) institutional affiliation (degree of the investigator and the name of institution where the study will be performed), details of contact such as phone numbers, E-mail id's and lines for signatures of investigators.

The main contents of the proposal may be presented under the following headings: (i) introduction, (ii) review of literature, (iii) aims and objectives, (iv) research design and methods, (v) ethical considerations, (vi) budget, (vii) appendices and (viii) citations.[ 4 ]

Introduction

It is also sometimes termed as ‘need for study’ or ‘abstract’. Introduction is an initial pitch of an idea; it sets the scene and puts the research in context.[ 6 ] The introduction should be designed to create interest in the reader about the topic and proposal. It should convey to the reader, what you want to do, what necessitates the study and your passion for the topic.[ 7 ] Some questions that can be used to assess the significance of the study are: (i) Who has an interest in the domain of inquiry? (ii) What do we already know about the topic? (iii) What has not been answered adequately in previous research and practice? (iv) How will this research add to knowledge, practice and policy in this area? Some of the evaluation committees, expect the last two questions, elaborated under a separate heading of ‘background and significance’.[ 8 ] Introduction should also contain the hypothesis behind the research design. If hypothesis cannot be constructed, the line of inquiry to be used in the research must be indicated.

Review of literature

It refers to all sources of scientific evidence pertaining to the topic in interest. In the present era of digitalisation and easy accessibility, there is an enormous amount of relevant data available, making it a challenge for the researcher to include all of it in his/her review.[ 9 ] It is crucial to structure this section intelligently so that the reader can grasp the argument related to your study in relation to that of other researchers, while still demonstrating to your readers that your work is original and innovative. It is preferable to summarise each article in a paragraph, highlighting the details pertinent to the topic of interest. The progression of review can move from the more general to the more focused studies, or a historical progression can be used to develop the story, without making it exhaustive.[ 1 ] Literature should include supporting data, disagreements and controversies. Five ‘C's may be kept in mind while writing a literature review[ 10 ] [ Table 1 ].

Aims and objectives

The research purpose (or goal or aim) gives a broad indication of what the researcher wishes to achieve in the research. The hypothesis to be tested can be the aim of the study. The objectives related to parameters or tools used to achieve the aim are generally categorised as primary and secondary objectives.

Research design and method

The objective here is to convince the reader that the overall research design and methods of analysis will correctly address the research problem and to impress upon the reader that the methodology/sources chosen are appropriate for the specific topic. It should be unmistakably tied to the specific aims of your study.

In this section, the methods and sources used to conduct the research must be discussed, including specific references to sites, databases, key texts or authors that will be indispensable to the project. There should be specific mention about the methodological approaches to be undertaken to gather information, about the techniques to be used to analyse it and about the tests of external validity to which researcher is committed.[ 10 , 11 ]

The components of this section include the following:[ 4 ]

Population and sample

Population refers to all the elements (individuals, objects or substances) that meet certain criteria for inclusion in a given universe,[ 12 ] and sample refers to subset of population which meets the inclusion criteria for enrolment into the study. The inclusion and exclusion criteria should be clearly defined. The details pertaining to sample size are discussed in the article “Sample size calculation: Basic priniciples” published in this issue of IJA.

Data collection

The researcher is expected to give a detailed account of the methodology adopted for collection of data, which include the time frame required for the research. The methodology should be tested for its validity and ensure that, in pursuit of achieving the results, the participant's life is not jeopardised. The author should anticipate and acknowledge any potential barrier and pitfall in carrying out the research design and explain plans to address them, thereby avoiding lacunae due to incomplete data collection. If the researcher is planning to acquire data through interviews or questionnaires, copy of the questions used for the same should be attached as an annexure with the proposal.

Rigor (soundness of the research)

This addresses the strength of the research with respect to its neutrality, consistency and applicability. Rigor must be reflected throughout the proposal.

It refers to the robustness of a research method against bias. The author should convey the measures taken to avoid bias, viz. blinding and randomisation, in an elaborate way, thus ensuring that the result obtained from the adopted method is purely as chance and not influenced by other confounding variables.

Consistency

Consistency considers whether the findings will be consistent if the inquiry was replicated with the same participants and in a similar context. This can be achieved by adopting standard and universally accepted methods and scales.

Applicability

Applicability refers to the degree to which the findings can be applied to different contexts and groups.[ 13 ]

Data analysis

This section deals with the reduction and reconstruction of data and its analysis including sample size calculation. The researcher is expected to explain the steps adopted for coding and sorting the data obtained. Various tests to be used to analyse the data for its robustness, significance should be clearly stated. Author should also mention the names of statistician and suitable software which will be used in due course of data analysis and their contribution to data analysis and sample calculation.[ 9 ]

Ethical considerations

Medical research introduces special moral and ethical problems that are not usually encountered by other researchers during data collection, and hence, the researcher should take special care in ensuring that ethical standards are met. Ethical considerations refer to the protection of the participants' rights (right to self-determination, right to privacy, right to autonomy and confidentiality, right to fair treatment and right to protection from discomfort and harm), obtaining informed consent and the institutional review process (ethical approval). The researcher needs to provide adequate information on each of these aspects.

Informed consent needs to be obtained from the participants (details discussed in further chapters), as well as the research site and the relevant authorities.

When the researcher prepares a research budget, he/she should predict and cost all aspects of the research and then add an additional allowance for unpredictable disasters, delays and rising costs. All items in the budget should be justified.

Appendices are documents that support the proposal and application. The appendices will be specific for each proposal but documents that are usually required include informed consent form, supporting documents, questionnaires, measurement tools and patient information of the study in layman's language.

As with any scholarly research paper, you must cite the sources you used in composing your proposal. Although the words ‘references and bibliography’ are different, they are used interchangeably. It refers to all references cited in the research proposal.

Successful, qualitative research proposals should communicate the researcher's knowledge of the field and method and convey the emergent nature of the qualitative design. The proposal should follow a discernible logic from the introduction to presentation of the appendices.

Financial support and sponsorship

Conflicts of interest.

There are no conflicts of interest.

• Privacy Policy

Buy Me a Coffee

Home » How To Write A Proposal – Step By Step Guide [With Template]

How To Write A Proposal – Step By Step Guide [With Template]

Table of Contents

How To Write A Proposal

Writing a Proposal involves several key steps to effectively communicate your ideas and intentions to a target audience. Here’s a detailed breakdown of each step:

Identify the Purpose and Audience

• Clearly define the purpose of your proposal: What problem are you addressing, what solution are you proposing, or what goal are you aiming to achieve?
• Identify your target audience: Who will be reading your proposal? Consider their background, interests, and any specific requirements they may have.

Conduct Research

• Gather relevant information: Conduct thorough research to support your proposal. This may involve studying existing literature, analyzing data, or conducting surveys/interviews to gather necessary facts and evidence.
• Understand the context: Familiarize yourself with the current situation or problem you’re addressing. Identify any relevant trends, challenges, or opportunities that may impact your proposal.

Develop an Outline

• Create a clear and logical structure: Divide your proposal into sections or headings that will guide your readers through the content.
• Introduction: Provide a concise overview of the problem, its significance, and the proposed solution.
• Background/Context: Offer relevant background information and context to help the readers understand the situation.
• Objectives/Goals: Clearly state the objectives or goals of your proposal.
• Methodology/Approach: Describe the approach or methodology you will use to address the problem.
• Timeline/Schedule: Present a detailed timeline or schedule outlining the key milestones or activities.
• Budget/Resources: Specify the financial and other resources required to implement your proposal.
• Evaluation/Success Metrics: Explain how you will measure the success or effectiveness of your proposal.
• Conclusion: Summarize the main points and restate the benefits of your proposal.

Write the Proposal

• Grab attention: Start with a compelling opening statement or a brief story that hooks the reader.
• Clearly state the problem: Clearly define the problem or issue you are addressing and explain its significance.
• Present your proposal: Introduce your proposed solution, project, or idea and explain why it is the best approach.
• State the objectives/goals: Clearly articulate the specific objectives or goals your proposal aims to achieve.
• Provide supporting information: Present evidence, data, or examples to support your claims and justify your proposal.
• Explain the methodology: Describe in detail the approach, methods, or strategies you will use to implement your proposal.
• Address potential concerns: Anticipate and address any potential objections or challenges the readers may have and provide counterarguments or mitigation strategies.
• Recap the main points: Summarize the key points you’ve discussed in the proposal.
• Reinforce the benefits: Emphasize the positive outcomes, benefits, or impact your proposal will have.
• Call to action: Clearly state what action you want the readers to take, such as approving the proposal, providing funding, or collaborating with you.

Review and Revise

• Proofread for clarity and coherence: Check for grammar, spelling, and punctuation errors.
• Ensure a logical flow: Read through your proposal to ensure the ideas are presented in a logical order and are easy to follow.
• Revise and refine: Fine-tune your proposal to make it concise, persuasive, and compelling.

Add Supplementary Materials

• Attach relevant documents: Include any supporting materials that strengthen your proposal, such as research findings, charts, graphs, or testimonials.
• Appendices: Add any additional information that might be useful but not essential to the main body of the proposal.

Formatting and Presentation

• Follow the guidelines: Adhere to any specific formatting guidelines provided by the organization or institution to which you are submitting the proposal.
• Use a professional tone and language: Ensure that your proposal is written in a clear, concise, and professional manner.
• Use headings and subheadings: Organize your proposal with clear headings and subheadings to improve readability.
• Pay attention to design: Use appropriate fonts, font sizes, and formatting styles to make your proposal visually appealing.
• Include a cover page: Create a cover page that includes the title of your proposal, your name or organization, the date, and any other required information.

Seek Feedback

• Share your proposal with trusted colleagues or mentors and ask for their feedback. Consider their suggestions for improvement and incorporate them into your proposal if necessary.

Finalize and Submit

• Make any final revisions based on the feedback received.
• Ensure that all required sections, attachments, and documentation are included.
• Double-check for any formatting, grammar, or spelling errors.
• Submit your proposal within the designated deadline and according to the submission guidelines provided.

Proposal Format

The format of a proposal can vary depending on the specific requirements of the organization or institution you are submitting it to. However, here is a general proposal format that you can follow:

1. Title Page:

• Include the title of your proposal, your name or organization’s name, the date, and any other relevant information specified by the guidelines.

2. Executive Summary:

•  Provide a concise overview of your proposal, highlighting the key points and objectives.
• Summarize the problem, proposed solution, and anticipated benefits.
• Keep it brief and engaging, as this section is often read first and should capture the reader’s attention.

3. Introduction:

• State the problem or issue you are addressing and its significance.
• Provide background information to help the reader understand the context and importance of the problem.
• Clearly state the purpose and objectives of your proposal.

4. Problem Statement:

• Describe the problem in detail, highlighting its impact and consequences.
• Use data, statistics, or examples to support your claims and demonstrate the need for a solution.

5. Proposed Solution or Project Description:

• Explain your proposed solution or project in a clear and detailed manner.
• Describe how your solution addresses the problem and why it is the most effective approach.
• Include information on the methods, strategies, or activities you will undertake to implement your solution.
• Highlight any unique features, innovations, or advantages of your proposal.

6. Methodology:

• Provide a step-by-step explanation of the methodology or approach you will use to implement your proposal.
• Include a timeline or schedule that outlines the key milestones, tasks, and deliverables.
• Clearly describe the resources, personnel, or expertise required for each phase of the project.

7. Evaluation and Success Metrics:

• Explain how you will measure the success or effectiveness of your proposal.
• Identify specific metrics, indicators, or evaluation methods that will be used.
• Describe how you will track progress, gather feedback, and make adjustments as needed.
• Present a detailed budget that outlines the financial resources required for your proposal.
• Include all relevant costs, such as personnel, materials, equipment, and any other expenses.
• Provide a justification for each item in the budget.

9. Conclusion:

•  Summarize the main points of your proposal.
•  Reiterate the benefits and positive outcomes of implementing your proposal.
• Emphasize the value and impact it will have on the organization or community.

10. Appendices:

• Include any additional supporting materials, such as research findings, charts, graphs, or testimonials.
•  Attach any relevant documents that provide further information but are not essential to the main body of the proposal.

Proposal Template

Here’s a basic proposal template that you can use as a starting point for creating your own proposal:

Dear [Recipient’s Name],

I am writing to submit a proposal for [briefly state the purpose of the proposal and its significance]. This proposal outlines a comprehensive solution to address [describe the problem or issue] and presents an actionable plan to achieve the desired objectives.

Thank you for considering this proposal. I believe that implementing this solution will significantly contribute to [organization’s or community’s goals]. I am available to discuss the proposal in more detail at your convenience. Please feel free to contact me at [your email address or phone number].

Yours sincerely,

Note: This template is a starting point and should be customized to meet the specific requirements and guidelines provided by the organization or institution to which you are submitting the proposal.

Proposal Sample

Here’s a sample proposal to give you an idea of how it could be structured and written:

Subject : Proposal for Implementation of Environmental Education Program

I am pleased to submit this proposal for your consideration, outlining a comprehensive plan for the implementation of an Environmental Education Program. This program aims to address the critical need for environmental awareness and education among the community, with the objective of fostering a sense of responsibility and sustainability.

Executive Summary: Our proposed Environmental Education Program is designed to provide engaging and interactive educational opportunities for individuals of all ages. By combining classroom learning, hands-on activities, and community engagement, we aim to create a long-lasting impact on environmental conservation practices and attitudes.

Introduction: The state of our environment is facing significant challenges, including climate change, habitat loss, and pollution. It is essential to equip individuals with the knowledge and skills to understand these issues and take action. This proposal seeks to bridge the gap in environmental education and inspire a sense of environmental stewardship among the community.

Problem Statement: The lack of environmental education programs has resulted in limited awareness and understanding of environmental issues. As a result, individuals are less likely to adopt sustainable practices or actively contribute to conservation efforts. Our program aims to address this gap and empower individuals to become environmentally conscious and responsible citizens.

Proposed Solution or Project Description: Our Environmental Education Program will comprise a range of activities, including workshops, field trips, and community initiatives. We will collaborate with local schools, community centers, and environmental organizations to ensure broad participation and maximum impact. By incorporating interactive learning experiences, such as nature walks, recycling drives, and eco-craft sessions, we aim to make environmental education engaging and enjoyable.

Methodology: Our program will be structured into modules that cover key environmental themes, such as biodiversity, climate change, waste management, and sustainable living. Each module will include a mix of classroom sessions, hands-on activities, and practical field experiences. We will also leverage technology, such as educational apps and online resources, to enhance learning outcomes.

Evaluation and Success Metrics: We will employ a combination of quantitative and qualitative measures to evaluate the effectiveness of the program. Pre- and post-assessments will gauge knowledge gain, while surveys and feedback forms will assess participant satisfaction and behavior change. We will also track the number of community engagement activities and the adoption of sustainable practices as indicators of success.

Budget: Please find attached a detailed budget breakdown for the implementation of the Environmental Education Program. The budget covers personnel costs, materials and supplies, transportation, and outreach expenses. We have ensured cost-effectiveness while maintaining the quality and impact of the program.

Conclusion: By implementing this Environmental Education Program, we have the opportunity to make a significant difference in our community’s environmental consciousness and practices. We are confident that this program will foster a generation of individuals who are passionate about protecting our environment and taking sustainable actions. We look forward to discussing the proposal further and working together to make a positive impact.

Thank you for your time and consideration. Should you have any questions or require additional information, please do not hesitate to contact me at [your email address or phone number].

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Grant Proposal – Example, Template and Guide

How To Write A Business Proposal – Step-by-Step...

Business Proposal – Templates, Examples and Guide

How To Write A Research Proposal – Step-by-Step...

Proposal – Types, Examples, and Writing Guide

How to choose an Appropriate Method for Research?

• How to Write a...

How to Write a Research Proposal: A Comprehensive Guide

• By angelie karin
• Apr 08, 2024

Writing a research proposal is a pivotal step for any researcher aiming to secure funding or approval for their project. 

A well-crafted proposal not only showcases the significance and feasibility of your study but also demonstrates your capability as a researcher to conduct the study. 

Below is a detailed exploration of the process, designed to guide you through each component of crafting a compelling research proposal.

Understanding the Fundamental Elements of a Research Proposal

The journey of your research proposal begins with the title – your first opportunity to make an impression. It should be both descriptive and concise, directly reflecting the essence of your research without any ambiguity. 

This is where you capture the attention of your readers, making them eager to delve into the details of your study.

Following the title, the abstract serves as a snapshot of your research proposal. Although brief, it must encapsulate the aims, methodology, and anticipated outcomes of your study. 

Succinctness and clarity are your allies here, aiding reviewers in quickly grasping the essence of your proposed research.

Introduction

The introduction lays down the foundation of your research proposal. It is here that you articulate your research question, setting the stage by providing background information and stating the rationale behind your study. 

This section should not only highlight the significance of your research but also build a compelling argument for why it is necessary.

Literature Review

A thorough literature review is the backbone of any research proposal . It demonstrates your deep understanding of the field by summarizing existing research and identifying gaps that your study aims to fill. 

This section should underscore the originality of your proposal, making a clear case for the contribution your research will make to the field. 

Integrating a variety of sources, from academic journals to books, you should weave a narrative that supports the premise of your study, emphasizing its necessity and potential impact.

Research Design and Methods

This section is where you detail how you plan to answer your research question. Transparently discuss your research design and methodology, whether it be qualitative, quantitative, or a mixture of both. 

Describe the data collection methods you intend to use, along with your analysis plan, giving special attention to the reproducibility and ethical considerations of your approach. 

Your goal is to convince the reviewers of the soundness of your research methodology and your competence in executing it.

Implications and Contribution to the Field

Explaining the significance of your research, this part of the proposal should articulate how your study will contribute to the existing body of knowledge. 

Discuss the potential implications of your findings for the field and how they might influence future research, policy, or practice. 

This is your argument for the relevance and necessity of your study, highlighting the unique insights it promises to offer.

Budget and Resources

As pragmatic as it is essential, the budget and resources section details the financial aspects of your proposed research. 

It is important to provide a comprehensive budget that justifies each expense, whether it’s for equipment, personnel, travel, or other costs. 

Be realistic and transparent, ensuring that your budget aligns with the scope of your project. Additionally, listing the resources you have and those you need demonstrates thorough planning and feasibility.

Timeline and Milestones

A well-structured timeline not only helps in organizing your research but also reassures funders of your project’s timeliness. 

Break down the research process into phases, assigning realistic deadlines to each. Including milestones allows for a clearer understanding of the project’s progress, making this section a crucial aspect of your proposal.

Ethical Considerations

Ethical considerations are paramount in research. This section of the proposal should address potential ethical issues and describe how they will be managed. 

If applicable, mention the process of obtaining Institutional Review Board (IRB) approval. A thoughtful discussion of ethics not only highlights the integrity of your research design but also builds trust with your reviewers.

Though not always necessary, appendices can be included to provide additional supporting information that is too detailed for the main body of the proposal. This might include charts, graphs, letters of support, or any other documents that strengthen your proposal.

Crafting a compelling research proposal is a meticulous process. Tailoring your proposal to echo the priorities of the funding agency can significantly increase your chances of success. 

Furthermore, ensuring that your writing is clear, jargon-free, and meticulously proofread can make your proposal more accessible and appealing. Remember, clarity and coherence are your best tools for conveying the value of your research.

What is the typical length of a research proposal?

While the length can vary depending on the requirements of the funding agency, a research proposal typically ranges from 10 to 15 pages.

How can I make my research proposal stand out to funders?

Emphasize the originality and significance of your research, clearly articulate your methodology, and ensure your proposal is well-organized and free of jargon. Tailoring the proposal to the funder’s priorities can also make a significant difference.

What are the most common reasons for research proposal rejection?

Common reasons include a lack of originality, unclear or flawed methodology, insufficient literature review, and unrealistic goals or budgets.

How detailed should the methodology section be in a research proposal?

It should be detailed enough to clearly explain how you will conduct your research, including your research design, data collection, and analysis methods, while being concise.

Is it necessary to include a literature review in my research proposal?

Yes, a literature review is crucial as it establishes the context of your research, demonstrates your knowledge of the field, and justifies the necessity of your proposed study.

In the conclusion, succinctly recap the key points of your proposal, reinforcing the significance and feasibility of your research. 

This final pitch should leave no doubt in the reviewers’ minds about the value and necessity of your study, encouraging their support for its execution. 

In crafting your research proposal, it’s essential to communicate not only the significance of your research but also your ability to carry it out successfully. 

By carefully addressing each component of the proposal and ensuring clarity and coherence throughout, you’ll significantly increase your chances of securing the necessary approval or funding. 

Remember, a compelling research proposal is your ticket to embarking on a meaningful research journey, contributing valuable insights to your field of study.

More From The Stock Dork:

Copyright © 2023 THE STOCK DORK

• Affiliate Disclosure
• Cookie Policy

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 02 April 2024

Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation

• Elizabeth C. Stade 1 , 2 , 3 ,
• Shannon Wiltsey Stirman 1 , 2 ,
• Lyle H. Ungar 4 ,
• Cody L. Boland 1 ,
• H. Andrew Schwartz 5 ,
• David B. Yaden 6 ,
• João Sedoc 7 ,
• Robert J. DeRubeis 8 ,
• Robb Willer 9 &
• Johannes C. Eichstaedt 3  

npj Mental Health Research volume  3 , Article number:  12 ( 2024 ) Cite this article

417 Accesses

3 Altmetric

Metrics details

• Psychiatric disorders

Large language models (LLMs) such as Open AI’s GPT-4 (which power ChatGPT) and Google’s Gemini, built on artificial intelligence, hold immense potential to support, augment, or even eventually automate psychotherapy. Enthusiasm about such applications is mounting in the field as well as industry. These developments promise to address insufficient mental healthcare system capacity and scale individual access to personalized treatments. However, clinical psychology is an uncommonly high stakes application domain for AI systems, as responsible and evidence-based therapy requires nuanced expertise. This paper provides a roadmap for the ambitious yet responsible application of clinical LLMs in psychotherapy. First, a technical overview of clinical LLMs is presented. Second, the stages of integration of LLMs into psychotherapy are discussed while highlighting parallels to the development of autonomous vehicle technology. Third, potential applications of LLMs in clinical care, training, and research are discussed, highlighting areas of risk given the complex nature of psychotherapy. Fourth, recommendations for the responsible development and evaluation of clinical LLMs are provided, which include centering clinical science, involving robust interdisciplinary collaboration, and attending to issues like assessment, risk detection, transparency, and bias. Lastly, a vision is outlined for how LLMs might enable a new generation of studies of evidence-based interventions at scale, and how these studies may challenge assumptions about psychotherapy.

Similar content being viewed by others

The imperative for regulatory oversight of large language models (or generative AI) in healthcare

Bertalan Meskó & Eric J. Topol

The future landscape of large language models in medicine

Jan Clusmann, Fiona R. Kolbinger, … Jakob Nikolas Kather

Large language models in medicine

Arun James Thirunavukarasu, Darren Shu Jeng Ting, … Daniel Shu Wei Ting

Introduction

Large language models (LLMs), built on artificial intelligence (AI) – such as Open AI’s GPT-4 (which power ChatGPT) and Google’s Gemini – are breakthrough technologies that can read, summarize, and generate text. LLMs have a wide range of abilities, including serving as conversational agents (chatbots), generating essays and stories, translating between languages, writing code, and diagnosing illness 1 . With these capacities, LLMs are influencing many fields, including education, media, software engineering, art, and medicine. They have started to be applied in the realm of behavioral healthcare, and consumers are already attempting to use LLMs for quasi-therapeutic purposes 2 .

Applications incorporating older forms of AI, including natural language processing (NLP) technology, have existed for decades 3 . For example, machine learning and NLP have been used to detect suicide risk 4 , identify the assignment of homework in psychotherapy sessions 5 , and identify patient emotions within psychotherapy 6 . Current applications of LLMs in the behavioral health field are far more nascent – they include tailoring an LLM to help peer counselors increase their expressions of empathy, which has been deployed with clients both in academic and commercial settings 2 , 7 . As another example, LLM applications have been used to identify therapists’ and clients’ behaviors in a motivational interviewing framework 8 , 9 .

Similarly, while algorithmic intelligence with NLP has been deployed in patient-facing behavioral health contexts, LLMs have not yet been heavily employed in these domains. For example, mental health chatbots Woebot and Tessa, which target depression and eating pathology respectively 10 , 11 , are rule-based and do not use LLMs (i.e., the application’s content is human-generated, and the chatbot’s responds based on predefined rules or decision trees 12 ). However, these and other existing chatbots frequently struggle to understand and respond to unanticipated user responses 10 , 13 , which likely contributes to their low engagement and high dropout rates 14 , 15 . LLMs may hold promise to fill some of these gaps, given their ability to flexibly generate human-like and context-dependent responses. A small number of patient-facing applications incorporating LLMs have been tested, including a research-based application to generate dialog for therapeutic counseling 16 , 17 , and an industry-based mental-health chatbot, Youper, which uses a mix of rule-based and generative AI 18 .

These early applications demonstrate the potential of LLMs in psychotherapy – as their use becomes more widespread, they will change many aspects of psychotherapy care delivery. However, despite the promise they may hold for this purpose, caution is warranted given the complex nature of psychopathology and psychotherapy. Psychotherapy delivery is an unusually complex, high-stakes domain vis-à-vis other LLM use cases. For example, in the productivity realm, with a “LLM co-pilot” summarizing meeting notes, the stakes are failing to maximize efficiency or helpfulness; in behavioral healthcare, the stakes may include improperly handling the risk of suicide or homicide.

While there are other applications of artificial intelligence that may involve high-stakes or life-or death decisions (e.g., self-driving cars), prediction and mitigation of risk in the case of psychotherapy is very nuanced, involving complex case conceptualization, the consideration of social and cultural contexts, and addressing unpredictable human behavior. Poor outcomes or ethical transgressions from clinical LLMs could run the risk of harming individuals, which may also be disproportionately publicized (as has occurred with other AI failures 19 ), which may damage public trust in the field of behavioral healthcare.

Therefore, developers of clinical LLMs need to act with special caution to prevent such consequences. Developing responsible clinical LLMs will be a challenging coordination problem, primarily because the technological developers who are typically responsible for product design and development lack clinical sensitivity and experience. Thus, behavioral health experts will need to play a critical role in guiding development and speaking to the potential limitations, ethical considerations, and risks of these applications.

Presented below is a discussion on the future of LLMs in behavioral healthcare from the perspective of both behavioral health providers and technologists. A brief overview of the technology underlying clinical LLMs is provided for the purposes of both educating clinical providers and to set the stage for further discussion regarding recommendations for development. The discussion then outlines various applications of LLMs to psychotherapy and provides a proposal for the cautious, phased development and evaluation of LLM-based applications for psychotherapy.

Overview of clinical LLMs

Clinical LLMs could take a wide variety of forms, spanning everything from brief interventions or circumscribed tools to augment therapy, to chatbots designed to provide psychotherapy in an autonomous manner. These applications could be patient-facing (e.g., providing psychoeducation to the patient), therapist-facing (e.g., offering options for interventions from which the therapist could select), trainee-facing (e.g., offering feedback on qualities of the trainee’s performance), or supervisor/consultant facing (e.g., summarizing supervisees’ therapy sessions in a high-level manner).

How language models work

Language models, or computational models of the probability of sequences of words, have existed for quite some time. The mathematical formulations date back to 20 and original use cases focused on compressing communication 21 and speech recognition 22 , 23 , 24 . Language modeling became a mainstay for choosing among candidate phrases in speech recognition and automatic translation systems but until recently, using such models for generating natural language found little success beyond abstract poetry 24 .

Large language models

The advent of large language models, enabled by a combination of the deep learning technique transformers 25 and increases in computing power, has opened new possibilities 26 . These models are first trained on massive amounts of data 27 , 28 using “unsupervised” learning in which the model’s task is to predict a given word in a sequence of words. The models can then be tailored to a specific task using methods, including prompting with examples or fine-tuning, some of which use no or small amounts of task-specific data (see Fig. 1 ) 28 , 29 . LLMs hold promise for clinical applications because they can parse human language and generate human-like responses, classify/score (i.e., annotate) text, and flexibly adopt conversational styles representative of different theoretical orientations.

Figure was designed using image components from Flaticon.com.

LLMs and psychotherapy skills

For certain use cases, LLM show a promising ability to conduct tasks or skills needed for psychotherapy, such as conducting assessment, providing psychoeducation, or demonstrating interventions (see Fig. 2 ). Yet to date, clinical LLM products and prototypes have not demonstrated anywhere near the level of sophistication required to take the place of psychotherapy. For example, while an LLM can generate an alternative belief in the style of CBT, it remains to be seen whether it can engage in the type of turn-based, Socratic questioning that would be expected to produce cognitive change. This more generally highlights the gap that likely exists between simulating therapy skills and implementing them effectively to alleviate patient suffering. Given that psychotherapy transcripts are likely poorly represented in the training data for LLMs, and that privacy and ethical concerns make such representation challenging, prompt engineering may ultimately be the most appropriate fine-tuning approach for shaping LLM behavior in this manner.

Note . Figure was designed using image component from Flaticon.com.

Clinical LLMs: stages of integration

The integration of LLMs into psychotherapy could be articulated as occurring along a continuum of stages spanning from assistive AI to fully autonomous AI (see Fig. 3 and Table 1 ). This continuum can be illustrated by models of AI integration in other fields, such as those used in the autonomous vehicle industry. For example, at one end of this continuum is the assistive AI (“machine in the loop”) stage, wherein the vehicle system has no ability to complete the primary tasks – acceleration, braking, and steering – on its own, but provides momentary assistance (e.g., automatic emergency breaking, lane departure warning) to increase driving quality or decrease burden on the driver. In the collaborative AI (“human in the loop”) stage, the vehicle system aids in the primary tasks, but requires human oversight (e.g., adaptive cruise control, lane keeping assistance). Finally, in fully autonomous AI, vehicles are self-driving and do not require human oversight. The stages of LLM integration into psychotherapy and their related functionalities are described below.

Stage 1: assistive LLMs

At the first stage in LLM integration, AI will be used as a tool to assist clinical providers and researchers with tasks that can easily be “offloaded” to AI assistants (Table 1 ; first row). As this is a preliminary step in integration, relevant tasks will be low-level, concrete, and circumscribed, such that they present a low level of risk. Examples of tasks could include assisting with collecting information for patient intakes or assessment, providing basic psychoeducation to patients, suggesting text edits for providers engaging in text-based care, and summarizing patient worksheets. Administratively, systems at this stage could also assist with clinical documentation by drafting session notes.

Stage 2: collaborative LLMs

Further along the continuum, AI systems will take the lead by providing or suggesting options for treatment planning and much of the therapy content, which humans will use their professional judgement to select from or tailor. For example, in the context of a text- or instant-message delivered structured psychotherapeutic intervention, the LLM might generate messages containing session content and assignments, which the therapist would review and adapt as needed before sending (Table 1 ; second row). A more advanced use of AI within the collaborative stage may entail a LLM providing a structured intervention in a semi-independent manner (e.g., as a chatbot), with a provider monitoring the discussion and stepping in to take control of the conversation as needed. The collaborative LLM stage has parallels to “guided self-help” approaches 30 .

Stage 3: fully autonomous LLMs

In the fully autonomous stage, AIs will achieve the greatest degree of scope and autonomy wherein a clinical LLM would perform a full range of clinical skills and interventions in an integrated manner without direct provider oversight (Table 1 ; third row). For example, an application at this stage might theoretically conduct a comprehensive assessment, select an appropriate intervention, and deliver a full course of therapy with no human intervention. In addition to clinical content, applications in this stage could integrate with the electronic health record to complete clinical documentation and report writing, schedule appointments and process billing. Fully autonomous applications offer the most scalable treatment method 30 .

Progression across the stages

Progression across the stages may not be linear; human oversight will be required to ensure that applications at greater stages of integration are safe for real world deployment. As different forms of psychopathology and their accompanying interventions vary in complexity, certain types of interventions will be simpler than others to develop as LLM applications. Interventions that are more concrete and standardized may be easier for models to deliver (and may be available sooner), such as circumscribed behavior change interventions (e.g., activity scheduling), as opposed to applications which include skills that are abstract in nature or emphasize cognitive change (e.g., Socratic questioning). Similarly, when it comes to full therapy protocols, LLM applications for interventions that are highly structured, behavioral, and protocolized (e.g., CBT for insomnia [CBT-I] or exposure therapy for specific phobia) may be available sooner than applications delivering highly flexible or personalized interventions (for example 31 ).

In theory, the final stage in the integration of LLMs into psychotherapy is fully autonomous delivery of psychotherapy which does not require human intervention or monitoring. However, it remains to be seen whether fully autonomous AI systems will reach a point at which they have been evaluated to be safe for deployment by the behavioral health community. Specific concerns include how well these systems are able to carry out case conceptualization on individuals with complex, highly comorbid symptom presentations, including accounting for current and past suicidality, substance use, safety concerns, medical comorbidities, and life circumstances and events (such as court dates and upcoming medical procedures). Similarly, it is unclear whether these systems will prove sufficiently adept at engaging patients over time 32 or accounting for and addressing contextual nuances in treatment (e.g., using exposure to treat a patient experiencing PTSD-related fear of leaving the house, who also lives in a neighborhood with high rates of crime). Furthermore, several skills which may be viewed as central to clinical work currently fall outside the purview of LLM systems, such as interpreting nonverbal behavior (e.g., fidgeting, eye-rolling), appropriately challenging a patient, addressing alliance ruptures, and making decisions about termination. Technological advances, including the approaching advent of multimodal language models that integrate text, images, video, and audio, may eventually begin to fill these gaps.

Beyond technical limitations, it remains to be decided whether complete automation is an appropriate end goal for behavioral healthcare, due to safety, legal, philosophical, and ethical concerns 33 . While some evidence indicates that humans can develop a therapeutic alliance with chatbots 34 , the long-term viability of such alliance building, and whether or not it produces undesirable downstream effects (e.g., altering an individual’s existing relationships or social skills) remains to be seen. Others have documented potentially harmful behavior of LLM chatbots, such as narcissistic tendencies 35 and expressed concerns about the potential for their undue influence on humans in addition to articulating societal risks associated with LLMs more generally 36 , 37 . The field will also need to grapple with questions of accountability and liability in the case of a fully autonomous clinical LLM application causing damage (e.g., identifying the responsible party in an incident of malpractice 38 ). For these and other reasons, some have argued against the implementation of fully autonomous systems in behavioral healthcare and healthcare more broadly 39 , 40 . Taken together, these issues and concerns may suggest that in the short and medium term, assistive or collaborative AI applications will be more appropriate for the provision of behavioral healthcare.

Applications of clinical LLMs

Given the vast nature of behavioral healthcare, there are seemingly endless applications of LLMs. Outlined below are some of the currently existing, imminently feasible, and potential long-term applications of clinical LLMs. Here we focus our discussion on applications directly related to the provision of, training in, and research on psychotherapy. As such, several important aspects of behavioral healthcare, such as initial symptom detection, psychological assessment and brief interventions (e.g., crisis counseling) are not explicitly discussed herein.

Imminent applications

Automating clinical administration tasks.

At the most basic level, LLMs have the potential to automate several time-consuming tasks associated with providing psychotherapy (Table 2 , first row). In addition to using session transcripts to summarize the session for the provider, there is potential for such models to integrate within electronic health records to aid with clinical documentation and conducting chart reviews. Clinical LLMs could also produce a handout for the patient that provides a personalized overview of the session, skills learned and assigned homework or between-session material.

Measuring treatment fidelity

A clinical LLM application could automate measurement of therapist fidelity to evidence-based practices (EBPs; Table 2 , second row), which can include measuring adherence to the treatment as designed, competence in delivering a specific therapy skill, treatment differentiation (whether multiple treatments being compared actually differ from one another), and treatment receipt (patient comprehension of, engagement with, and adherence to the therapy content) 41 , 42 . Measuring fidelity is crucial to the development, testing, dissemination, and implementation of EBPs, yet can be resource intensive and difficult to do reliably. In the future, clinical LLMs could computationally derive adherence and competence ratings, aiding research efforts and reducing therapist drift 43 . Traditional machine-learning models are already being used to assess fidelity to specific modalities 44 and other important constructs like counseling skills 45 and alliance 46 . Given their improved ability to consider context, LLMs will likely increase the accuracy with which these constructs are assessed.

Offering feedback on therapy worksheets and homework

LLM applications could also be developed deliver real-time feedback and support on patients’ between-session homework assignments (Table 2 , third row). For example, an LLM tailored to assist a patient to complete a CBT worksheet might provide clarification or aid in problem solving if the patient experiences difficulty (e.g., the patient was completing a thought log and having trouble differentiating between the thought and the emotion). This could help to “bridge the gap” between sessions and expedite patient skill development. Early evidence outside the AI realm 47 points to increasing worksheet competence as a fruitful clinical target.

Automating aspects of supervision and training

LLMs could be used to provide feedback on psychotherapy or peer support sessions, especially for clinicians with less training and experience (i.e., peer counselors, lay health workers, psychotherapy trainees). For example, an LLM might be used to offer corrections and suggestions to the dialog of peer counselors (Table 2 , fourth row). This application has parallels to “task sharing,” a method used in the global mental health field by which nonprofessionals provide mental health care with the oversight by specialist workers to expand access to mental health services 48 . Some of this work is already underway, for example, as described above, using LLMs to support peer counselors 7 .

LLMs could also support supervision for psychotherapists learning new treatments (Table 2 , fifth row). Gold-standard methods of reviewing trainees’ work, like live observation or review of recorded sessions 49 , are time-consuming. LLMs could analyze entire therapy sessions and identify areas of improvement, offering a scalable approach for supervisors or consultants to review.

Potential long-term applications

It is important to note that many of the potential applications listed below are theoretical and have yet to be developed, let alone thoroughly evaluated. Furthermore, we use the term “clinical LLM” in recognition of the fact that when and under what circumstances the work of an LLM could be called psychotherapy is evolving and depends on how psychotherapy is defined.

Fully autonomous clinical care

As previously described, the final stage of clinical LLM development could involve an LLM that can independently conduct comprehensive behavioral healthcare. This could involve all aspects related to traditional care including conducting assessment, presenting feedback, selecting an appropriate intervention and delivering a course of therapy to the patient. This course of treatment could be delivered in ways consistent with current models of psychotherapy wherein a patient engages with a “chatbot” weekly for a prescribed amount of time, or in more flexible or alternative formats. LLMs used in this manner would ideally be trained using standardized assessment approaches and manualized therapy protocols that have large bodies of evidence.

Decision aid for existing evidence-based practices

Even without full automation, clinical LLMs could be used as a tool to guide a provider on the best course of treatment for a given patient by optimizing the delivery of existing EBPs and therapeutic techniques. In practice, this may look like a LLM that can analyze transcripts from therapy sessions and offer a provider guidance on therapeutic skills, approaches or language, either in real time, or at the end of the therapy session. Furthermore, the LLM could integrate current evidence on the tailoring of specific EBPs to the condition being treated, and to demographic or cultural factors and comorbid conditions. Developing tailored clinical LLM “advisors” based on EBPs could both enhance fidelity to treatment and maximize the possibility of patients achieving clinical improvement in light of updated clinical evidence.

Development of new therapeutic techniques and EBPs

To this point, we have discussed how LLMs could be applied to current approaches to psychotherapy using extant evidence. However, LLMs and other computational methods could greatly enhance the detection and development of new therapeutic skills and EBPs. Historically, EBPs have traditionally been developed using human-derived insights and then evaluated through years of clinical trial research. While EBPs are effective, effect sizes for psychotherapy are typically small 50 , 51 and significant proportions of patients do not respond 52 . There is a great need for more effective treatments, particularly for individuals with complex presentations or comorbid conditions. However, the traditional approach to developing and testing therapeutic interventions is slow, contributing to significant time lags in translational research 53 , and fails to deliver insights at the level of the individual.

Data-driven approaches hold the promise of revealing patterns that are not yet realized by clinicians, thus generating new approaches to psychotherapy; machine learning is already being used, for example, to predict behavioral health treatment outcomes 54 . With their ability to parse and summarize natural language, LLMs could add to existing data-driven approaches. For example, an LLM could be provided with a large historical dataset containing psychotherapy transcripts of different therapeutic orientations, outcome measures and sociodemographic information, and tasked with detecting therapeutic behaviors and techniques associated with objective outcomes (e.g., reduction in depressive symptoms). Using such a process might make it possible for an LLM to yield fine-grained insights about what makes existing therapeutic techniques work best (e.g., Which components of existing EBPs are the most potent? Are there therapist or patient characteristics that moderate the efficacy of intervention X? How does the ordering of interventions effect outcomes?) or even to isolate previously unidentified therapeutic techniques associated with improved clinical outcomes. By identifying what happens in therapy in such a fine-grained manner, LLMs could also play a role in revealing mechanisms of change, which is important for improving existing treatments and facilitating real-world implementation 55 .

However, to realize this possibility, and make sure that LLM-based advances can be integrated and vetted by the clinical community, it is necessary to steer away from the development of “black box,” LLM-identified interventions with low explainability (e.g., interpretability 56 ). To guard against interventions with low interpretability, work to finetune LLMs to improve patient outcomes could include inspectable representations of the techniques employed by the LLM. Clinicians could examine these representations and situate them in the broader psychotherapy literature, which would involve comparing them to existing psychotherapy techniques and theories. Such an approach could speed up the identification of novel mechanisms while guarding against the identification of “novel” interventions which overlap with existing techniques or constructs (thus avoiding the jangle fallacy, the erroneous assumption that two constructs with different names are necessarily distinct 57 ).

In the long run, by combining this information, it might even be possible for an LLM to “reverse-engineer” a new EBP, freed from the constraints of traditional therapeutic protocols and instead maximizing on the delivery of the constituent components shown to produce patient change (in a manner akin to modular approaches, wherein an individualized treatment plan is crafted for each patient by curating and sequencing treatment modules from an extensive menu of all available options based on the unique patient’s presentation 31 ). Eventually, a self-learning clinical LLM might deliver a broad range of psychotherapeutic interventions while measuring patient outcomes and adapting its approach on the fly in response to changes in the patient (or lack thereof).

Toward a precision medicine approach to psychotherapy

Current approaches to psychotherapy often are unable to provide guidance on the best approach to treatment when an individual has a complex presentation, which is often the rule rather than being the exception. For example, providers are likely to have greatly differing treatment plans for a patient with concurrent PTSD, substance use, chronic pain, and significant interpersonal difficulties. Models that use a data-driven approach (rather than a provider’s educated guess) to address an individual’s presenting concern alongside their comorbidities, sociodemographic factors, history, and responses to the current treatment, may ultimately offer the best chance at maximizing patient benefit. While there have been some advances in precision medicine approaches in behavioral healthcare 54 , 58 , these efforts are in their infancy and limited by sample sizes 59 .

The potential applications of clinical LLMs we have outlined above may come together to facilitate a personalized approach to behavioral healthcare, analogous to that of precision medicine. Through optimizing existing EBPs, identifying new therapeutic approaches, and better understanding mechanisms of change, LLMs (and their future descendants) may provide behavioral healthcare with an enhanced ability to identify what works best for whom and under what circumstances.

Recommendations for responsible development and evaluation of clinical LLMs

Focus first on evidence-based practices.

In the immediate future, clinical LLM applications will have the greatest chance of creating meaningful clinical impact if developed based on EBPs or a “common elements” approach (i.e., evidence-based procedures shared across treatments) 60 . Evidence-based treatments and techniques have been identified for specific psychopathologies (e.g., major depressive disorder, posttraumatic stress disorder), stressors (e.g., bereavement, job loss, divorce), and populations (e.g., LGBTQ individuals, older adults) 55 , 61 , 62 . Without an initial focus on EBPs, clinical LLM applications may fail to reflect current knowledge and may even produce harm 63 . Only once LLMs have been fully trained on EBPs can the field start to consider using LLMs in a data-driven manner, such as those outlined in the previous section on potential long-term applications.

Focus next on improvement (engagement is not enough)

Others have highlighted the importance of promoting engagement with digital mental health applications 15 , which is important for achieving an adequate “dose” of the therapeutic intervention. LLM applications hold the promise of improving engagement and retention through their ability to respond to free text, extract key concepts, and address patients’ unique context and concerns during interventions in a timely manner. However, engagement alone is not an appropriate outcome on which to train an LLM, because engagement is not expected to be sufficient for producing change. A focus on such metrics for clinical LLMs will risk losing sight of the primary goals, clinical improvement (e.g., reductions in symptoms or impairment, increases in well-being and functioning) and prevention of risks and adverse events. It will behoove the field to be wary of attempts to optimize clinical LLMs on outcomes that have an explicit relationship with a company’s profit (e.g., length of time using the application). An LLM that optimizes only for engagement (akin to YouTube recommendations) could have high rates of user retention without employing meaningful clinical interventions to reduce suffering and improve quality of life. Previous research has suggested that this may be happening with non-LLM digital mental health interventions. For instance, exposure is a technique with strong support for treating anxiety, yet it is rarely included in popular smartphone applications for anxiety 64 , perhaps because developers fear that the technique will not appeal to users, or have concerns about how exposures going poorly or increasing anxiety in the short term, which may prompt concerns about legal exposure.

Commit to rigorous yet commonsense evaluation

An evaluation approach for clinical LLMs that hierarchically prioritizes risk and safety, followed by feasibility, acceptability, and effectiveness, would be in line with existing recommendations for the evaluation of digital mental health smartphone apps 65 . The first level of evaluation could involve a demonstration that a clinical LLM produces no harm or very minimal harm that is outweighed by its benefits, similar to FDA phase I drug tests. Key risk and safety related constructs include measures of suicidality, non-suicidal self harm, and risk of harm to others.

Next, rigorous examinations of clinical LLM applications will be needed to provide empirical evidence of their utility, using head-to-head comparisons with standard treatments. Key constructs to be assessed in these empirical tests are feasibility and acceptability to the patient and the therapist as well as treatment outcomes (e.g., symptoms, impairment, clinical status, rates of relapse). Other relevant considerations include patients’ user experience with the application, measures of therapist efficiency and burnout, and cost.

Lastly, we note that given that possible benefits of clinical LLMs (including expanding access to care), it will be important for the field to adopt a commonsense approach to evaluation. While rigorous evaluation is important, the comparison conditions on which these evaluations are based should reflect real-world risk and efficacy rates, and perhaps employ a graded hierarchy with which to classify risk and error (i.e., missing a mention of suicidality is unacceptable, but getting a patient’s partner’s name wrong is nonideal but tolerable), rather than holding clinical LLM applications to a standard of perfection which humans do not achieve. Furthermore, developers will need to strike the appropriate balance of prioritizing constructs in a manner expected to be most clinically beneficial, for example, if exposure therapy is indicated for the patient, but the patient does not find this approach acceptable, the clinical LLM could recommend the intervention prioritizing effectiveness before offering second-line interventions which may be more acceptable.

Involve interdisciplinary collaboration

Interdisciplinary collaboration between clinical scientists, engineers, and technologists will be crucial in the development of clinical LLMs. While it is plausible that engineers and technologists could use available therapeutic manuals to develop clinical LLMs without the expertise of a behavioral health expert, this is ill-advised. Manuals are only a first step towards learning a specific intervention, as they do not provide guidance on how the intervention can be applied to specific individuals or presentations, or how to handle specific issues or concerns that may arise through the course of treatment.

Clinicians and clinician-scientists have expertise that bears on these issues, as well as many other aspects of the clinical LLM development process. Their involvement could include a) testing new applications to identify limitations and risks and optimize their integration into clinical practice, b) improving the ability of applications to adequately address the complexity of psychological phenomena, c) ensuring that applications are developed and implemented in an ethical manner, and d) testing and ensuring that applications don’t have iatrogenic effects, such as reinforcing behaviors that perpetuate psychopathology or distress.

Behavioral health experts could also provide guidance on how best to finetune or tailor models, including addressing the question of whether and how real patient data should be used for these purposes. For example, most proximately, behavioral health experts might assist in prompt engineering , or the designing and testing of a series of prompts which provide the LLM framing and context for delivering a specific type of treatment or clinical skill (e.g., “Use cognitive restructuring to help the patient evaluate and reappraise negative thoughts in depression”), or a desired clinical task, such as evaluating therapy sessions for fidelity (e.g., “Analyze this psychotherapy transcript and select sections in which the therapist demonstrated the particularly skillful use of CBT skills, and sections in which the therapist’s delivery of CBT skills could be improved”). Similarly, in few-shot learning , behavioral health experts could be involved in crafting example exchanges which are added to prompts. For example, treatment modality experts might generate examples of clinical skills (e.g., high-quality examples of using cognitive restructuring to address depression) or of a clinical task (e.g., examples of both high- and low-quality delivery of CBT skills). For fine-tuning , in which a large, labeled dataset is used to train the LLM, and reinforcement learning from human feedback (RLHF), in which a human-labeled dataset is used to train a smaller model which is then used for LLM “self-training,” behavioral health experts could build and curate (and ensure informed patient consent for use of) appropriate datasets (e.g., a dataset containing psychotherapy transcripts rated for fidelity to an evidence-based psychotherapy). The expertise that behavioral health experts could draw on to generate instructive examples and curate high-quality datasets holds particular value in light of recent evidence that quality of data trumps quantity of data for training well-performing models 66 .

In the service of facilitating interdisciplinary collaboration, it would benefit clinical scientists to seek out a working knowledge about LLMs, while it would benefit technologists to develop a working knowledge of therapy in general and EBPs in particular. Dedicated venues that bring together behavioral health experts and clinical psychologists for interdisciplinary collaboration and communication will aid in these efforts. Historically, venues of this type have included psychology-focused workshops at NLP conferences (e.g., the Workshop on Computational Linguistics and Clinical Psychology [CLPsych], held at the Annual Conference of the North American Chapter of the Association for Computational Linguistics [NAACL]) and technology-focused conferences or workgroups hosted by psychological organizations (e.g., APA’s Technology, Mind & Society conference; Association for Behavioral and Cognitive Therapies’ [ABCT] Technology and Behavior Change special interest group). This work has also been done at nonprofits centered on technological tools for mental health (e.g., the Society for Digital Mental Health). Beyond these venues, it may be fruitful to develop a gathering that brings together technologists, clinical scientists, and industry partners with a dedicated focus on AI/LLMs, which could routinely publish on its efforts, akin to the efforts of the World Health Organization’s Infodemic Management Conference, which has employed this approach to address misinformation 67 . Finally, given the numerous applications of AI to behavioral health, it is conceivable that a new “computational behavioral health” subfield could emerge, offering specialized training that would bridge the gap between these two domains.

Focus on trust and usability for clinicians and patients

It is important to engage therapists, policymakers, end-users, and experts in human-computer interactions to understand and improve levels of trust that will be necessary for successful and effective implementation. With respect to applications of AI to augment supervision and support for psychotherapy, therapists have expressed concern about privacy, the ability to detect subtle non-verbal cues and cultural responsiveness, and the impact on therapist confidence, but they also see benefits for training and professional growth 68 . Other research suggests that while therapists believe AI can increase access to care, allow individuals to disclose embarrassing information more comfortably, continuously refine therapeutic techniques 69 , they have concerns about privacy and the formation of a strong therapeutic bond with machine-based therapeutic interventions 70 . Involvement of individuals who will be referring their patients and using LLMs in their own practice will be essential to developing solutions they can trust and implement, and to make sure these solutions have the features that support trust and usability (simple interfaces, accurate summaries of AI-patient interactions, etc.).

Regarding how much patients will trust the AI systems, following the stages we outlined in Fig. 3 , initial AI-patient interactions will continue to be supervised by clinicians, and the therapeutic bond between the clinician and the patient will continue to be the primary relationship. During this stage, it is important that clinicians talk to the patients about their experience with the LLMs, and that the field as a whole begins to accumulate an understanding and data on how acceptable interfacing with LLMs is for what kind of patient for what kind of clinical use case, in how clinicians can scaffold the patient-LLM relationship. This data will be critical for developing collaborative LLM applications that have more autonomy, and for ensuring that the transition from assistive to collaborative stage applications is not associated with large unforeseen risk. For example, in the case of CBT for insomnia, once an assistive AI system has been iterated on to reliably collect information about patients’ sleep patterns, it is more conceivable that it could be evolved into a collaborative AI system that does a comprehensive insomnia assessment (i.e., it also collects and interprets data on patients’ clinically significant distress, impairment of functioning, and ruling out of sleep-wake disorders, like narcolepsy) 71 .

Design criteria for effective clinical LLMs

Below, we propose an initial set of desirable design qualities for clinical LLMs.

Detect risk of harm

Accurate risk detection and mandated reporting are crucial aspects that clinical LLMs must prioritize, particularly in the identification of suicidal/homicidal ideation, child/elder abuse, and intimate partner violence. Algorithms for detecting risks are under development 4 . One threat to risk detection is that current LLMs have limited context windows, meaning they only “remember” a limited amount of user input. Functionally, this means a clinical LLM application could “forget” crucial details about a patient, which could impact safety (e.g., an application “forgetting” that the patient owns firearms would threaten its ability to properly assess and intervene around suicide risk). However, context windows have been rapidly expanding with each subsequent model release, so this issue may not be a problem for long. In addition, it is already possible to augment the memory of LLMs with “vector databases,” which would have the added benefit of retaining inspectable learnings and summaries across clinical encounters 72 .

In the future, and especially given much larger context windows, clinical LLMs could prompt clinicians with ethical guidelines, legal requirements (e.g., the Tarasoff rule, which requires clinicians to warn intended victims when a patient presents a serious threat of violence), or evidence-based methods for decreasing risk (e.g., safety planning 73 ), or even provide interventions targeting risk directly to patients. This type of risk monitoring and intervention could be particularly useful in supplementing existing healthcare systems during gaps in clinician coverage like nights and weekends 4 .

b) Be “Healthy.” There is growing concern that AI chat systems can demonstrate undesirable behaviors, including expressions akin to depression or narcissism 35 , 74 . Such poorly understood, undesirable behaviors risk harming already vulnerable patients or interfering with their ability to benefit from treatment. Clinical LLM applications will need training, monitoring, auditing, and guardrails to prevent the expression of undesirable behaviors and maintain healthy interactions with users. These efforts will need to be continually evaluated and updated to prevent or address the emergence of new undesirable or clinically contraindicated behavior.

Aid in psychodiagnostic assessment

Clinical LLMs ought to integrate psychodiagnostic assessment and diagnosis, facilitating intervention selection and outcome monitoring 75 . Recent developments show promise for LLMs in the assessment realm 76 . Down the line, LLMs could be used for diagnostic interviewing (e.g., Structured Clinical Interview for the DSM-5 77 ) using chatbots or voice interfaces. Prioritizing assessment enhances diagnostic accuracy and ensures appropriate intervention, reducing the risk of harmful interventions 63 .

Be responsive and flexible

Given the frequency with which ambivalence and poor patient engagement arise in clinical encounters, clinical LLMs which use evidence-based and patient-centered methods for handling these issues (e.g., motivational enhancement techniques, shared decision making), and have options for second-line interventions for patients not interested in gold-standard treatments, will have the best chance of success.

Stop when not helping or confident

Psychologists are ethically obligated to cease treatment and offer appropriate referrals to the patient if the current course of treatment has not helped or likely will not help. Clinical LLMs can abide by this ethical standard by drawing on integrated assessment (discussed above) to assess the appropriateness of the given intervention and detect cases that need more specialized or intensive intervention.

Be fair, inclusive, and free from bias

As has been written about extensively, LLMs may perpetuate bias, including racism, sexism, and homophobia, given that they are trained on existing text 36 . These biases can contribute to both error disparities – where models are less accurate for particular groups – or outcome disparities – where models tend to over-capture demographic information 78 – which would in turn contribute to the disparities in mental health status and care already experienced by minoritized groups 79 . The integration of bias countermeasures into clinical LLM applications could serve to prevent this 78 , 80 .

Be empathetic–to an extent

Clinical LLMs will likely need to demonstrate empathy and build the therapeutic alliance in order to engage patients. Other skills used by therapists include humor, irreverence, and gentle methods of challenging the patient. Incorporating these into clinical LLMs might be beneficial, as appropriate human likeness may facilitate engagement and interaction with AI 81 . However, this needs to be balanced against associated risks, mentioned above, of incorporating human likeness in systems 36 . Whether and how much human likeness is necessary for a psychological intervention remains a question for future empirical work.

Be transparent about being AIs

Mental illness and mental health care is already stigmatized, and the application of LLMs without transparent consent can erode patient/consumer trust, which reduces trust in the behavioral health profession more generally. Some mental health startups have already faced criticism for employing generative AI in applications without disclosing this information to the end user 2 . As laid out in the White House Blueprint for an AI Bill of Rights, AI applications should be explicitly (and perhaps repeatedly/consistently) labeled as such to allow patients and consumers to “know that an automated system is being used and understand how and why it contributes to outcomes that impact them” 82 .

Unintended consequences may change the clinical profession

The development of clinical LLM applications could lead to unintended consequences, such as changes to the structure of and compensation for mental health services. AI may permit increased staffing by non-professionals or paraprofessionals, causing professional clinicians to supervise large numbers of non-professionals or even semi-autonomous LLM systems. This could reduce clinicians’ direct patient contact and perhaps increase their exposure to challenging or complicated cases not suitable for the LLM, which may lead to burnout and make clinical jobs less attractive. To address this, research could determine the appropriate number of cases for a clinician to oversee safely and guidelines could be published to disseminate these findings. The 24-hour availability of LLM-based intervention may also change consumer expectations of psychotherapy in a way that is at odds with many of the norms of psychotherapy practice (e.g., waiting for a session to discuss stressors, limited or emergency-only contact between sessions).

LLMs could pave the way for a next generation of clinical science

Beyond the imminent applications described in this paper, it is worth considering how the long-term applications of clinical LLMs might also facilitate significant advances in clinical care and clinical science.

Clinical practice

In terms of their effects on therapeutic interventions themselves, clinical LLMs might promote advances in the field by allowing for the pooling of data on what works with the most difficult cases, perhaps through the use of practice research networks 83 . At the level of health systems, they could expedite the implementation and translation of research findings into clinical practice by suggesting therapeutic strategies to psychotherapists, for instance, promoting strategies that enhance inhibitory learning during exposure therapy 84 . Lastly, clinical LLMs could increase access to care if LLM-based psychotherapy chatbots are offered as low intensity, low-cost options in stepped-care models, similar to the existing provision of computerized CBT and guided self-help 85 .

As the utilization of clinical LLMs expands, there may be a shift towards psychologists and other behavioral health experts operating at the top of their degree. Presently, a significant amount of clinician time is consumed by administrative tasks, chart review, and documentation. The shifting of responsibilities afforded by the automation of certain aspects of psychotherapy by clinical LLMs could allow clinicians to pursue leadership roles, contribute to the development, evaluation, and implementation of LLM-based care, or lead policy efforts, or simply to devote more time to direct patient care.

Clinical science

By facilitating supervision, consultation, and fidelity measurement, LLMs could expedite psychotherapist training and increase the capacity of study supervisors, thus making psychotherapy research less expensive and more efficient.

In a world in which fully autonomous LLM applications screen and assess patients, deliver high-fidelity, protocolized psychotherapy, and collect outcome measurements, psychotherapy clinical trials would be limited largely by the number of willing participants eligible for the study, rather than by the resources required to screen, assess, treat, and follow these participants. This could open the door to unprecedentedly large-N clinical trials. This would allow for well-powered, sophisticated dismantling studies to support the search for mechanisms of change in psychotherapy, which are currently only possible using individual participant level meta-analysis (for example, see ref. 86 ). Ultimately, such insights into causal mechanisms of change in psychotherapy could help to refine these treatments and potentially improve their efficacy.

Finally, the emergence of LLM treatment modalities will challenge (or confirm) fundamental assumptions about psychotherapy. Does therapeutic (human) alliance account for a majority of the variance in patient change? To what extent can an alliance be formed with a technological agent? Is lasting and meaningful therapeutic change only possible through working with a human therapist? LLMs hold the promise of empirical answers to these questions.

In summary, large language models hold promise for supporting, augmenting, or even in some cases replacing human-led psychotherapy, which may improve the quality, accessibility, consistency, and scalability of therapeutic interventions and clinical science research. However, LLMs are advancing quickly and will soon be deployed in the clinical domain, with little oversight or understanding of harms that they may produce. While cautious optimism about clinical LLM applications is warranted, it is also crucial for psychologists to approach the integration of LLMs into psychotherapy with caution and to educate the public about the potential risks and limitations of using these technologies for therapeutic purposes. Furthermore, clinical psychologists ought to actively engage with the technologists building these solutions. As the field of AI continues to evolve, it is essential that researchers and clinicians closely monitor the use of LLMs in psychotherapy and advocate for responsible and ethical use to protect the wellbeing of patients.

Data availability

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

Bubeck, S. et al. Sparks of artificial general intelligence: Early experiments with GPT-4. Preprint at http://arxiv.org/abs/2303.12712 (2023).

Broderick, R. People are using AI for therapy, whether the tech is ready for it or not. Fast Company (2023).

Weizenbaum, J. ELIZA—a computer program for the study of natural language communication between man and machine. Commun. ACM 9 , 36–45 (1966).

Article   Google Scholar  

Bantilan, N., Malgaroli, M., Ray, B. & Hull, T. D. Just in time crisis response: Suicide alert system for telemedicine psychotherapy settings. Psychother. Res. 31 , 289–299 (2021).

Peretz, G., Taylor, C. B., Ruzek, J. I., Jefroykin, S. & Sadeh-Sharvit, S. Machine learning model to predict assignment of therapy homework in behavioral treatments: Algorithm development and validation. JMIR Form. Res. 7 , e45156 (2023).

Article   PubMed   PubMed Central   Google Scholar  

Tanana, M. J. et al. How do you feel? Using natural language processing to automatically rate emotion in psychotherapy. Behav. Res. Methods 53 , 2069–2082 (2021).

Sharma, A., Lin, I. W., Miner, A. S., Atkins, D. C. & Althoff, T. Human–AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support. Nat. Mach. Intell . 5 , 46–57 (2023).

Chen, Z., Flemotomos, N., Imel, Z. E., Atkins, D. C. & Narayanan, S. Leveraging open data and task augmentation to automated behavioral coding of psychotherapy conversations in low-resource scenarios. Preprint at https://doi.org/10.48550/arXiv.2210.14254 (2022).

Shah, R. S. et al. Modeling motivational interviewing strategies on an online peer-to-peer counseling platform. Proc. ACM Hum.-Comput. Interact 6 , 1–24 (2022).

Chan, W. W. et al. The challenges in designing a prevention chatbot for eating disorders: Observational study. JMIR Form. Res. 6 , e28003 (2022).

Darcy, A. Why generative AI Is not yet ready for mental healthcare. Woebot Health https://woebothealth.com/why-generative-ai-is-not-yet-ready-for-mental-healthcare/ (2023).

Abd-Alrazaq, A. A. et al. An overview of the features of chatbots in mental health: A scoping review. Int. J. Med. Inf. 132 , 103978 (2019).

Lim, S. M., Shiau, C. W. C., Cheng, L. J. & Lau, Y. Chatbot-delivered psychotherapy for adults with depressive and anxiety symptoms: A systematic review and meta-regression. Behav. Ther. 53 , 334–347 (2022).

Article   PubMed   Google Scholar  

Baumel, A., Muench, F., Edan, S. & Kane, J. M. Objective user engagement with mental health apps: Systematic search and panel-based usage analysis. J. Med. Internet Res. 21 , e14567 (2019).

Torous, J., Nicholas, J., Larsen, M. E., Firth, J. & Christensen, H. Clinical review of user engagement with mental health smartphone apps: Evidence, theory and improvements. Evid. Based Ment. Health 21 , 116–119 (2018b).

Das, A. et al. Conversational bots for psychotherapy: A study of generative transformer models using domain-specific dialogues. in Proceedings of the 21st Workshop on Biomedical Language Processing 285–297 (Association for Computational Linguistics, 2022). https://doi.org/10.18653/v1/2022.bionlp-1.27 .

Liu, H. Towards automated psychotherapy via language modeling. Preprint at http://arxiv.org/abs/2104.10661 (2021).

Hamilton, J. Why generative AI (LLM) is ready for mental healthcare. LinkedIn https://www.linkedin.com/pulse/why-generative-ai-chatgpt-ready-mental-healthcare-jose-hamilton-md/ (2023).

Shariff, A., Bonnefon, J.-F. & Rahwan, I. Psychological roadblocks to the adoption of self-driving vehicles. Nat. Hum. Behav. 1 , 694–696 (2017).

Markov, A. A. Essai d’une recherche statistique sur le texte du roman “Eugene Onegin” illustrant la liaison des epreuve en chain (‘Example of a statistical investigation of the text of “Eugene Onegin” illustrating the dependence between samples in chain’). Izvistia Imperatorskoi Akad. Nauk Bull. L’Academie Imp. Sci. St-Petersbourg 7 , 153–162 (1913).

Google Scholar  

Shannon, C. E. A mathematical theory of communication. Bell Syst. Tech. J. 27 , 379–423 (1948).

Article   MathSciNet   Google Scholar  

Baker, J. K. Stochastic modeling for automatic speech understanding. in Speech recognition: invited papers presented at the 1974 IEEE symposium (ed. Reddy, D. R.) (Academic Press, 1975).

Jelinek, F. Continuous speech recognition by statistical methods. Proc. IEEE 64 , 532–556 (1976).

Jurafsky, D. & Martin, J. H. N-gram language models. in Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition (Pearson Prentice Hall, 2009).

Vaswani, A. et al. Attention is all you need. 31st Conf. Neural Inf. Process. Syst . (2017).

Bommasani, R. et al. On the opportunities and risks of foundation models. Preprint at http://arxiv.org/abs/2108.07258 (2022).

Gao, L. et al. The Pile: An 800GB dataset of diverse text for language modeling. Preprint at http://arxiv.org/abs/2101.00027 (2020).

Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. Preprint at http://arxiv.org/abs/1810.04805 (2019).

Kojima, T., Gu, S. S., Reid, M., Matsuo, Y. & Iwasawa, Y. Large language models are zero-shot reasoners. Preprint at http://arxiv.org/abs/2205.11916 (2023).

Fairburn, C. G. & Patel, V. The impact of digital technology on psychological treatments and their dissemination. Behav. Res. Ther. 88 , 19–25 (2017).

Fisher, A. J. et al. Open trial of a personalized modular treatment for mood and anxiety. Behav. Res. Ther. 116 , 69–79 (2019).

Fan, X. et al. Utilization of self-diagnosis health chatbots in real-world settings: Case study. J. Med. Internet Res. 23 , e19928 (2021).

Coghlan, S. et al. To chat or bot to chat: Ethical issues with using chatbots in mental health. Digit. Health 9 , 1–11 (2023).

Beatty, C., Malik, T., Meheli, S. & Sinha, C. Evaluating the therapeutic alliance with a free-text CBT conversational agent (Wysa): A mixed-methods study. Front. Digit. Health 4 , 847991 (2022).

Lin, B., Bouneffouf, D., Cecchi, G. & Varshney, K. R. Towards healthy AI: Large language models need therapists too. Preprint at http://arxiv.org/abs/2304.00416 (2023).

Weidinger, L. et al. Ethical and social risks of harm from language models. Preprint at http://arxiv.org/abs/2112.04359 (2021).

Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency 610–623 (ACM, 2021). https://doi.org/10.1145/3442188.3445922 .

Chamberlain, J. The risk-based approach of the European Union’s proposed artificial intelligence regulation: Some comments from a tort law perspective. Eur. J. Risk Regul. 14 , 1–13 (2023).

Norden, J. G. & Shah, N. R. What AI in health care can learn from the long road to autonomous vehicles. NEJM Catal. Innov. Care Deliv . https://doi.org/10.1056/CAT.21.0458 (2022).

Sedlakova, J. & Trachsel, M. Conversational artificial intelligence in psychotherapy: A new therapeutic tool or agent? Am. J. Bioeth. 23 , 4–13 (2023).

Gearing, R. E. et al. Major ingredients of fidelity: A review and scientific guide to improving quality of intervention research implementation. Clin. Psychol. Rev. 31 , 79–88 (2011).

Wiltsey Stirman, S. Implementing evidence-based mental-health treatments: Attending to training, fidelity, adaptation, and context. Curr. Dir. Psychol. Sci. 31 , 436–442 (2022).

Waller, G. Evidence-based treatment and therapist drift. Behav. Res. Ther. 47 , 119–127 (2009).

Flemotomos, N. et al. “Am I a good therapist?” Automated evaluation of psychotherapy skills using speech and language technologies. CoRR, Abs, 2102 (10.3758) (2021).

Zhang, X. et al. You never know what you are going to get: Large-scale assessment of therapists’ supportive counseling skill use. Psychotherapy https://doi.org/10.1037/pst0000460 (2022).

Goldberg, S. B. et al. Machine learning and natural language processing in psychotherapy research: Alliance as example use case. J. Couns. Psychol. 67 , 438–448 (2020).

Wiltsey Stirman, S. et al. A novel approach to the assessment of fidelity to a cognitive behavioral therapy for PTSD using clinical worksheets: A proof of concept with cognitive processing therapy. Behav. Ther. 52 , 656–672 (2021).

Raviola, G., Naslund, J. A., Smith, S. L. & Patel, V. Innovative models in mental health delivery systems: Task sharing care with non-specialist providers to close the mental health treatment gap. Curr. Psychiatry Rep. 21 , 44 (2019).

American Psychological Association. Guidelines for clinical supervision in health service psychology. Am. Psychol. 70 , 33–46 (2015).

Cook, S. C., Schwartz, A. C. & Kaslow, N. J. Evidence-based psychotherapy: Advantages and challenges. Neurotherapeutics 14 , 537–545 (2017).

Leichsenring, F., Steinert, C., Rabung, S. & Ioannidis, J. P. A. The efficacy of psychotherapies and pharmacotherapies for mental disorders in adults: An umbrella review and meta‐analytic evaluation of recent meta‐analyses. World Psych. 21 , 133–145 (2022).

Cuijpers, P., van Straten, A., Andersson, G. & van Oppen, P. Psychotherapy for depression in adults: A meta-analysis of comparative outcome studies. J. Consult. Clin. Psychol. 76 , 909–922 (2008).

Morris, Z. S., Wooding, S. & Grant, J. The answer is 17 years, what is the question: Understanding time lags in translational research. J. R. Soc. Med. 104 , 510–520 (2011).

Chekroud, A. M. et al. The promise of machine learning in predicting treatment outcomes in psychiatry. World Psych. 20 , 154–170 (2021).

Kazdin, A. E. Mediators and mechanisms of change in psychotherapy research. Annu. Rev. Clin. Psychol. 3 , 1–27 (2007).

Angelov, P. P., Soares, E. A., Jiang, R., Arnold, N. I. & Atkinson, P. M. Explainable artificial intelligence: An analytical review. WIREs Data Min. Knowl. Discov . 11 , (2021).

Kelley, T. L. Interpretation of Educational Measurements . (World Book, 1927).

van Bronswijk, S. C. et al. Precision medicine for long-term depression outcomes using the Personalized Advantage Index approach: Cognitive therapy or interpersonal psychotherapy? Psychol. Med. 51 , 279–289 (2021).

Scala, J. J., Ganz, A. B. & Snyder, M. P. Precision medicine approaches to mental health care. Physiology 38 , 82–98 (2023).

Article   CAS   Google Scholar  

Chorpita, B. F., Daleiden, E. L. & Weisz, J. R. Identifying and selecting the common elements of evidence based interventions: A distillation and matching model. Ment. Health Serv. Res. 7 , 5–20 (2005).

Chambless, D. L. & Hollon, S. D. Defining empirically supported therapies. J. Consult. Clin. Psychol. 66 , 7–18 (1998).

Article   CAS   PubMed   Google Scholar  

Tolin, D. F., McKay, D., Forman, E. M., Klonsky, E. D. & Thombs, B. D. Empirically supported treatment: Recommendations for a new model. Clin. Psychol. Sci. Pract. 22 , 317–338 (2015).

Lilienfeld, S. O. Psychological treatments that cause harm. Perspect. Psychol. Sci. 2 , 53–70 (2007).

Wasil, A. R., Venturo-Conerly, K. E., Shingleton, R. M. & Weisz, J. R. A review of popular smartphone apps for depression and anxiety: Assessing the inclusion of evidence-based content. Behav. Res. Ther. 123 , 103498 (2019).

Torous, J. B. et al. A hierarchical framework for evaluation and informed decision making regarding smartphone apps for clinical care. Psychiatr. Serv. 69 , 498–500 (2018).

Gunasekar, S. et al. Textbooks are all you need. Preprint at http://arxiv.org/abs/2306.11644 (2023).

Wilhelm, E. et al. Measuring the burden of infodemics: Summary of the methods and results of the Fifth WHO Infodemic Management Conference. JMIR Infodemiology 3 , e44207 (2023).

Creed, T. A. et al. Knowledge and attitudes toward an artificial intelligence-based fidelity measurement in community cognitive behavioral therapy supervision. Adm. Policy Ment. Health Ment. Health Serv. Res. 49 , 343–356 (2022).

Aktan, M. E., Turhan, Z. & Dolu, İ. Attitudes and perspectives towards the preferences for artificial intelligence in psychotherapy. Comput. Hum. Behav. 133 , 107273 (2022).

Prescott, J. & Hanley, T. Therapists’ attitudes towards the use of AI in therapeutic practice: considering the therapeutic alliance. Ment. Health Soc. Incl. 27 , 177–185 (2023).

American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders . (2013).

Yogatama, D., De Masson d’Autume, C. & Kong, L. Adaptive semiparametric language models. Trans. Assoc. Comput. Linguist 9 , 362–373 (2021).

Stanley, B. & Brown, G. K. Safety planning intervention: A brief intervention to mitigate suicide risk. Cogn. Behav. Pract. 19 , 256–264 (2012).

Behzadan, V., Munir, A. & Yampolskiy, R. V. A psychopathological approach to safety engineering in AI and AGI. Preprint at http://arxiv.org/abs/1805.08915 (2018).

Lambert, M. J. & Harmon, K. L. The merits of implementing routine outcome monitoring in clinical practice. Clin. Psychol. Sci. Pract . 25 , (2018).

Kjell, O. N. E., Kjell, K. & Schwartz, H. A. AI-based large language models are ready to transform psychological health assessment. Preprint at https://doi.org/10.31234/osf.io/yfd8g (2023).

First, M. B., Williams, J. B. W., Karg, R. S. & Spitzer, R. L. SCID-5-CV: Structured Clinical Interview for DSM-5 Disorders: Clinician Version . (American Psychiatric Association Publishing, 2016).

Shah, D. S., Schwartz, H. A. & Hovy, D. Predictive biases in natural language processing models: A conceptual framework and overview. in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics 5248–5264 (Association for Computational Linguistics, 2020). https://doi.org/10.18653/v1/2020.acl-main.468 .

Adams, L. M. & Miller, A. B. Mechanisms of mental-health disparities among minoritized groups: How well are the top journals in clinical psychology representing this work? Clin . Psychol. Sci. 10 , 387–416 (2022).

Viswanath, H. & Zhang, T. FairPy: A toolkit for evaluation of social biases and their mitigation in large language models. Preprint at http://arxiv.org/abs/2302.05508 (2023).

von Zitzewitz, J., Boesch, P. M., Wolf, P. & Riener, R. Quantifying the human likeness of a humanoid robot. Int. J. Soc. Robot. 5 , 263–276 (2013).

White House Office of Science and Technology Policy. Blueprint for an AI bill of rights. (2022).

Parry, G., Castonguay, L. G., Borkovec, T. D. & Wolf, A. W. Practice research networks and psychological services research in the UK and USA. in Developing and Delivering Practice-Based Evidence (eds. Barkham, M., Hardy, G. E. & Mellor-Clark, J.) 311–325 (Wiley-Blackwell, 2010). https://doi.org/10.1002/9780470687994.ch12 .

Craske, M. G., Treanor, M., Conway, C. C., Zbozinek, T. & Vervliet, B. Maximizing exposure therapy: An inhibitory learning approach. Behav. Res. Ther. 58 , 10–23 (2014).

Delgadillo, J. et al. Stratified care vs stepped care for depression: A cluster randomized clinical trial. JAMA Psychiatry 79 , 101 (2022).

Furukawa, T. A. et al. Dismantling, optimising, and personalising internet cognitive behavioural therapy for depression: A systematic review and component network meta-analysis using individual participant data. Lancet Psychiatry 8 , 500–511 (2021).

Download references

Acknowledgements

This work was supported by the National Institute of Mental Health under award numbers R01-MH125702 (PI: H.A.S) and RF1-MH128785 (PI: S.W.S.), and by the Institute for Human-Centered A.I. at Stanford University to J.C.E. The authors are grateful to Adam S. Miner and Victor Gomes who provided critical feedback on an earlier version of this manuscript.

Author information

Authors and affiliations.

Dissemination and Training Division, National Center for PTSD, VA Palo Alto Health Care System, Palo Alto, CA, USA

Elizabeth C. Stade, Shannon Wiltsey Stirman & Cody L. Boland

Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA

Elizabeth C. Stade & Shannon Wiltsey Stirman

Institute for Human-Centered Artificial Intelligence & Department of Psychology, Stanford University, Stanford, CA, USA

Elizabeth C. Stade & Johannes C. Eichstaedt

Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA

Lyle H. Ungar

Department of Computer Science, Stony Brook University, Stony Brook, NY, USA

H. Andrew Schwartz

Department of Psychiatry and Behavioral Sciences, Johns Hopkins University School of Medicine, Baltimore, MD, USA

David B. Yaden

Department of Technology, Operations, and Statistics, New York University, New York, NY, USA

Department of Psychology, University of Pennsylvania, Philadelphia, PA, USA

Robert J. DeRubeis

Department of Sociology, Stanford University, Stanford, CA, USA

Robb Willer

You can also search for this author in PubMed   Google Scholar

Contributions

E.C.S., S.W.S., C.L.B., and J.C.E. wrote the main manuscript text. E.C.S., L.H.U., and J.C.E. prepared the figures. All authors reviewed the manuscript.

Corresponding authors

Correspondence to Elizabeth C. Stade or Johannes C. Eichstaedt .

Ethics declarations

Competing interests.

The authors declare the following competing interests: receiving consultation fees from Jimini Health (E.C.S., L.H.U., H.A.S., and J.C.E.).

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Stade, E.C., Stirman, S.W., Ungar, L.H. et al. Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluation. npj Mental Health Res 3 , 12 (2024). https://doi.org/10.1038/s44184-024-00056-z

Download citation

Received : 24 July 2023

Accepted : 30 January 2024

Published : 02 April 2024

DOI : https://doi.org/10.1038/s44184-024-00056-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.