literature review of organizations

Organizing and Creating Information

Citation and Attribution

What Is a Literature Review?

Review the literature, write the literature review, further reading, learning objectives, attribution.

This guide is designed to:

Identify the sections and purpose of a literature review in academic writing
Review practical strategies and organizational methods for preparing a literature review

A literature review is a summary and synthesis of scholarly research on a specific topic. It should answer questions such as:

What research has been done on the topic?
Who are the key researchers and experts in the field?
What are the common theories and methodologies?
Are there challenges, controversies, and contradictions?
Are there gaps in the research that your approach addresses?

The process of reviewing existing research allows you to fine-tune your research question and contextualize your own work. Preparing a literature review is a cyclical process. You may find that the research question you begin with evolves as you learn more about the topic.

Once you have defined your research question , focus on learning what other scholars have written on the topic.

In order to do a thorough search of the literature on the topic, define the basic criteria:

Databases and journals: Look at the subject guide related to your topic for recommended databases. Review the tutorial on finding articles for tips.
Books: Search BruKnow, the Library's catalog. Steps to searching ebooks are covered in the Finding Ebooks tutorial .
What time period should it cover? Is currency important?
Do I know of primary and secondary sources that I can use as a way to find other information?
What should I be aware of when looking at popular, trade, and scholarly resources ?

One strategy is to review bibliographies for sources that relate to your interest. For more on this technique, look at the tutorial on finding articles when you have a citation .

Tip: Use a Synthesis Matrix

As you read sources, themes will emerge that will help you to organize the review. You can use a simple Synthesis Matrix to track your notes as you read. From this work, a concept map emerges that provides an overview of the literature and ways in which it connects. Working with Zotero to capture the citations, you build the structure for writing your literature review.

How do I know when I am done?

A key indicator for knowing when you are done is running into the same articles and materials. With no new information being uncovered, you are likely exhausting your current search and should modify search terms or search different catalogs or databases. It is also possible that you have reached a point when you can start writing the literature review.

Tip: Manage Your Citations

These citation management tools also create citations, footnotes, and bibliographies with just a few clicks:

Zotero Tutorial

Endnote Tutorial

Your literature review should be focused on the topic defined in your research question. It should be written in a logical, structured way and maintain an objective perspective and use a formal voice.

Review the Summary Table you created for themes and connecting ideas. Use the following guidelines to prepare an outline of the main points you want to make.

Synthesize previous research on the topic.
Aim to include both summary and synthesis.
Include literature that supports your research question as well as that which offers a different perspective.
Avoid relying on one author or publication too heavily.
Select an organizational structure, such as chronological, methodological, and thematic.

The three elements of a literature review are introduction, body, and conclusion.

Introduction

Define the topic of the literature review, including any terminology.
Introduce the central theme and organization of the literature review.
Summarize the state of research on the topic.
Frame the literature review with your research question.
Focus on ways to have the body of literature tell its own story. Do not add your own interpretations at this point.
Look for patterns and find ways to tie the pieces together.
Summarize instead of quote.
Weave the points together rather than list summaries of each source.
Include the most important sources, not everything you have read.
Summarize the review of the literature.
Identify areas of further research on the topic.
Connect the review with your research.
DeCarlo, M. (2018). 4.1 What is a literature review? In Scientific Inquiry in Social Work. Open Social Work Education. https://scientificinquiryinsocialwork.pressbooks.com/chapter/4-1-what-is-a-literature-review/
Literature Reviews (n.d.) https://writingcenter.unc.edu/tips-and-tools/literature-reviews/ Accessed Nov. 10, 2021

This guide was designed to:

Identify the sections and purpose of a literature review in academic writing
Review practical strategies and organizational methods for preparing a literature review

Content on this page adapted from:

Frederiksen, L. and Phelps, S. (2017). Literature Reviews for Education and Nursing Graduate Students. Licensed CC BY 4.0

<< Previous: EndNote
Last Updated: Jan 9, 2024 3:05 PM
URL: https://libguides.brown.edu/organize

Library Intranet

Library Catalogue

Doing a literature review: Community organizations and nonprofits

This page is a living document -- we welcome community feedback. If you have comments or questions, please contact us: [email protected].

What is a literature review, and why do one?

Literature reviews summarize the existing conversation on a topic. They can accomplish something similar to environmental scans, researching industry best practices, summarizing evidence-based approaches, or laying the groundwork for a new initiative. A literature review can be done as its own complete product, or it can serve as an introduction for the original research you plan to do to add to the conversation in your field. In academic research contexts, such as for theses and dissertations, they provide a grounding for the new work a scholar is presenting to add to this conversation.

However, most guidance available on conducting literature reviews does not consider how to do a literature review in a nonprofit context, or outside of university settings in general.

What a literature review could do for you

Outside of academia, your motivations for research may be different. For a non-profit, literature reviews might help you:

Gather evidence to assess your organization’s best practices
Provide evidence for grant proposals
Look for new approaches to an issue
Provide background or a methodology for your original research
Synthesize what’s known on a topic, to share with others working in your area
Find out if someone has already studied the topic you are interested in

What are the steps?

What you're looking to create may not be exactly a formal literature review. Most guidance on conducting literature reviews assumes an academic context, with strict guidelines for how a review should be conducted. As a non-profit or NGO worker your resources, needs, and audience may be different from researchers in academic institutions.

Here is a structure to follow based on the literature review process, with questions to consider along the way to help you tailor the process to your context. Each step is described in greater detail below the infographic.

The process doesn't have to be linear! You can return to each step as needed throughout your research.

Preparation

Think through your needs, capacity, and resources to plan your work.

What is your motivation for creating a review on your topic?

Do any literature reviews exist on this topic already?

Do you have fellow staff or volunteers that could contribute to this project?
How much time do you (and others) have to dedicate?
Do any staff or volunteers have relevant experience to contribute?

Information and supports:

You may have more resources available to you than you realize.
See "Where to search" in the "Retrieval" section for more ideas.

Keeping track of sources:

To keep track of the sources you intend to use, you can keep a list in your own document, or you may want to consider using citation management software.
Citation managers are designed to make it quick and easy to track and cite your sources. You can read more at Citation management software and tools .

Keeping track of the information you're collecting:

What kind of note-taking works well for you?
Even if something feels memorable in the moment, it can be difficult to remember where you found a piece of information even the next day.
Many citation management tools allow you to add notes or attach documents.

Consider any possible search terms that relate to your topic -- it's useful to try different combinations of words to see what results you get back. Your early searches may help you find relevant terms to add to your list.

You can also consider what kinds of articles will be relevant to your needs, then use search filters to narrow your results to fit those criteria. For example, if you are looking for recent articles, some databases give you the option to filter by publication date.

It's a good idea to keep a record of your searches: a list of what keywords or filters you've used, on what website, and what you were able to find with each search. This can help you locate sources again later, and have a record of what has worked well so you don't repeat work.

Resource links: The collapsed "Where to search section" below describes some options of where you may be able to find information and documents, as well as support services that could help you in your research. "Research strategies" contains more information on research strategies and approaches.

Open web (search engines like Google, Bing, DuckDuckGo ):

Tools for searching for Open Access articles ( CORE plug-in , OA button )
Advanced Google search tips
Grey literature search tips - grey literature is information produced outside of traditional publishing and distribution channels - things like newsletters, working papers, speeches, reports, and policy literature. It often comes from NGOs , government, industry, and other organizations.

Potentially useful collections:

SFU list of Open Access databases
Directory of Open Access Journals
Academic library databases -- use Open Access filters in search ( SFU guide , UBC guide )
Open content on JSTOR -- a digital library of academic journals, books, and primary sources in the humanities and social sciences
Downtown Eastside Research Access Portal -- contains research and related materials relevant to Vancouver's Downtown Eastside
Cochrane Library -- a collection of databases with evidence to inform healthcare decision-making, including systematic reviews, controlled trials, etc. (not all Open Access content, but many articles are)
Candid. Issue Lab -- an online collection of free research, evaluations, case studies, toolkits , etc.
Frontier Life -- a digital collection of primary source documents relating to colonialism in North America, Australia, Asia, and Africa

Paid databases:

How to effectively search databases for journal articles

There are a few other resources that may assist you in your search or provide you access to additional materials. For example, public libraries may subscribe to databases that would then be free to patrons to access. Academic libraries also sometimes allow guests to visit in person and use university computers to access their electronic resources like databases. Other options include:

Paid searching services ( InfoAction at the Vancouver Public Library, for example)
Professional organizations you are a member of may also have their own independent library collections
JSTOR free account - a digital library of academic journals, books, and primary sources in the humanities and social sciences
You may qualify for the Community Scholars Program
Start your research here -- an overview of the research process
Library Research Skills -- interactive Canvas tutorial from the SFU Library that covers understanding assignments (or in your case, your research needs), narrowing your topic, and finding background sources, scholarly books, and articles. Geared towards students but may still be useful
Finding articles: Advanced search techniques -- overview video of advanced search techniques that are especially useful when searching databases
Search tips for Google, Google Scholar, DuckDuckGo , and other search engines -- advanced search techniques for web search engines
The Beginner’s Guide to Business Research -- from the UBC Small Business Accelerator
How to Conduct Market Research for Nonprofits Like a Pro - Pollfish Resources -- includes suggestions for sources for nonprofit secondary market research

Screen the results you're getting: are they relevant to the topic you've chosen? Do they fit the criteria you've selected for retrieval? You don't need to read the full article at this stage -- you can read the abstract or skim headings. If the article looks relevant, save the full text to read once you reach the synthesis stage.

Evaluating sources:

Evaluating Sources - from the University of the Fraser Valley on scholarly vs. popular sources, news sources, images, and the "filter bubble"
Evaluating Information Sources - from the University of British Columbia on evaluation frameworks and checklists
CASP Checklists - checklists for evaluating methodology and study design

Choosing relevant sources for your topic:

Choosing Sources - Sheridan College resource with questions to reflect about the type of information you're seeking
Choosing the Best Sources and Evidence - University of Arizona Global Campus resource on selecting strong and relevant evidence
How do I choose which sources to use? - Tips from Imperial College London to quickly identify whether information is relevant to your research

You’ll be doing a lot of reading, so keep notes on each source you find. This can be done in the notes section of your citation manager, or your own document.

The other articles your sources are citing can clue you into other important research in the field -- especially if they appear in multiple of your sources.

Recognizing patterns or themes (i.e. in findings, specific aspects of your topic studied, methods of research, conclusions) will be useful when you begin your write-up to help you link articles together or contrast them.

Spying on a conversation [video] - tips and tricks on identifying relevant information in a literature review search, 4:43 - 8:54 is particularly relevant for the synthesis step
Read & Take Notes - Researching the Literature Review - guide from Oregon State University on strategies for note-taking during a literature review

The form of your finished product may differ based on your audience, needs, and purpose. Should the write-up be thorough or brief? Casual or formal tone? Purely text, or with visuals?

Generally, a literature review should contain:

An introduction -- where you explain why you've conducted this research, your context, and give readers a preview what you'll discuss and how that will be structured
You don't need to fully summarize each article, and you also don't need to include every article on a topic -- just what's relevant to you!
Element of analysis/critique -- consider which sources you find particularly useful, and the positives or flaws they may have
Conclusion -- how does it all add together? Does anything seem to be missing from the scholarship as a whole? Did you learn anything that's useful to your organization and its practices?

There are many possibilities to shape the format to your needs. Your literature review could contain an executive summary to explain your project and call out key findings. It could end with recommendations for action in your organization and beyond. It could take the form of an interactive web page. It could have bulleted summaries, photographs, and graphs.

See below in the "Examples" section to see some possible structures that incorporate some of these elements.

Academic writing: What is a literature review? - guide from the SFU Student Learning Commons on what should be included in a literature review and how to organize it
Literature Reviews - guide from the University of North Carolina at Chapel Hill on writing a literature review. "Strategies for writing the literature review" and "Begin composing" sections may be most helpful in how to create your write-up

Revisit: who was your original target or imagined audience?

Now that you've conducted your research, are there any other groups (whether internal or external to your organization) who would be interested in what you've created? These could be community members, local leaders, professional organizations, or partner organizations you work closely with.

What methods are available to reach your target audience and other potential groups you've identified? Trade publications, blogs, email lists, or conferences might be good places to share your work. Internally, newsletters, meetings, and emails could be effective.

Planning for your project's future

Keeping notes on your process (searches, sources, etc.) throughout your research allows you or others in your organization to pick your work back up later, or to replicate your process on a different topic.

Ensure your notes are kept somewhere secure and reliable, where others in your organization can also access it as needed.

The following links contain literature reviews conducted by and for community groups and nonprofits. They provide an example of the different forms your literature review could take.

CityHive - "Enhancing Youth Leadership and Agency - A Toolkit for Successful Leadership Programs"
Ontario Nonprofit Network - "Decent Work for Women - A literature review of women working in Ontario's nonprofit sector"
Downtown Eastside Women's Centre - "Red Women Rising: Indigenous Women Survivors in Vancouver's Downtown Eastside"
The Coalition for Healthy School Food - "The Impact"
Canadian Observatory on Homelessness - "Northern, Rural, and Remote Homelessness: A Review of the Literature"

Further resources

For more detailed advice on literature reviews, see our guide on literature reviews for graduate students .

For more information on citation, see this page on citation and style guides , this guide on citing Indigenous Elders and Knowledge Keepers (APA 7), and this piece on citational justice by Neha Kumar and Naveena Karusala.

Acknowledgments

This guide draws on research supported by the Social Sciences and Humanities Research Council. It was developed as part of the Supporting Transparent and open Research Engagement & Exchange ( STOREE ) research project.

Thank you to the Community Scholars and others who reviewed drafts of this page, including Savannah Swann from the Dr. Peter Centre.

Literature Reviews

"How To" Books
Examples of Literature Reviews
Collecting Resources for a Literature Review
Organizing the Literature Review
Writing the Literature Review
Endnote This link opens in a new window
Evaluating Websites

Organization

Organization of your Literature Review

What is the most effective way of presenting the information? What are the most important topics, subtopics, etc., that your review needs to include? What order should you present them?

Just like most academic papers, literature reviews must contain at least three basic elements: an introduction or background information section; the body of the review containing the discussion of sources; and, finally, a conclusion and/or recommendations section to end the paper.

Introduction: Gives a quick idea of the topic of the literature review, such as the central theme or organizational pattern.

Body: Contains your discussion of sources and is organized either chronologically, thematically, or methodologically (see below for more information on each).

Conclusions/Recommendations: Discuss what you have drawn from reviewing the literature so far. Where might the discussion proceed?

Once you have the basic categories in place, then you must consider how you will present the sources themselves within the body of your paper. Create an organizational method to focus this section even further.

To help you come up with an overall organizational framework for your review, consider the following scenario and then three typical ways of organizing the sources into a review:

You've decided to focus your literature review on materials dealing with sperm whales. This is because you've just finished reading Moby Dick, and you wonder if that whale's portrayal is really real. You start with some articles about the physiology of sperm whales in biology journals written in the 1980's. But these articles refer to some British biological studies performed on whales in the early 18th century. So you check those out. Then you look up a book written in 1968 with information on how sperm whales have been portrayed in other forms of art, such as in Alaskan poetry, in French painting, or on whale bone, as the whale hunters in the late 19th century used to do. This makes you wonder about American whaling methods during the time portrayed in Moby Dick, so you find some academic articles published in the last five years on how accurately Herman Melville portrayed the whaling scene in his novel.

Chronological

If your review follows the chronological method, you could write about the materials above according to when they were published. For instance, first you would talk about the British biological studies of the 18th century, then about Moby Dick, published in 1851, then the book on sperm whales in other art (1968), and finally the biology articles (1980s) and the recent articles on American whaling of the 19th century. But there is relatively no continuity among subjects here. And notice that even though the sources on sperm whales in other art and on American whaling are written recently, they are about other subjects/objects that were created much earlier. Thus, the review loses its chronological focus.

By publication

Order your sources chronologically by publication if the order demonstrates a more important trend. For instance, you could order a review of literature on biological studies of sperm whales if the progression revealed a change in dissection practices of the researchers who wrote and/or conducted the studies.

Another way to organize sources chronologically is to examine the sources under a trend, such as the history of whaling. Then your review would have subsections according to eras within this period. For instance, the review might examine whaling from pre-1600-1699, 1700-1799, and 1800-1899. Using this method, you would combine the recent studies on American whaling in the 19th century with Moby Dick itself in the 1800-1899 category, even though the authors wrote a century apart.

Thematic reviews of literature are organized around a topic or issue, rather than the progression of time. However, progression of time may still be an important factor in a thematic review. For instance, the sperm whale review could focus on the development of the harpoon for whale hunting. While the study focuses on one topic, harpoon technology, it will still be organized chronologically. The only difference here between a "chronological" and a "thematic" approach is what is emphasized the most: the development of the harpoon or the harpoon technology.

More authentic thematic reviews tend to break away from chronological order. For instance, a thematic review of material on sperm whales might examine how they are portrayed as "evil" in cultural documents. The subsections might include how they are personified, how their proportions are exaggerated, and their behaviors misunderstood. A review organized in this manner would shift between time periods within each section according to the point made.

Methodological

A methodological approach differs from the two above in that the focusing factor usually does not have to do with the content of the material. Instead, it focuses on the "methods" of the researcher or writer. For the sperm whale project, one methodological approach would be to look at cultural differences between the portrayal of whales in American, British, and French art work. Or the review might focus on the economic impact of whaling on a community. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Once you've decided on the organizational method for the body of the review, the sections you need to include in the paper should be easy to figure out. They should arise out of your organizational strategy. In other words, a chronological review would have subsections for each vital time period. A thematic review would have subtopics based upon factors that relate to the theme or issue.

Sometimes, though, you might need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. Put in only what is necessary. Here are a few other sections you might want to consider:

Current Situation: Information necessary to understand the topic or focus of the literature review.

History: The chronological progression of the field, the literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.

Methods and/or Standards: The criteria you used to select the sources in your literature review or the way in which you present your information. For instance, you might explain that your review includes only peer-reviewed articles and journals.

Questions for Further Research: What questions about the field has the review sparked? How will you further your research as a result of the review?

<< Previous: Collecting Resources for a Literature Review
Next: Writing the Literature Review >>
Last Updated: Nov 2, 2021 12:11 PM
URL: https://guides.library.stonybrook.edu/literature-review
Request a Class
Hours & Locations
Ask a Librarian
Special Collections
Library Faculty & Staff

Library Administration: 631.632.7100

Stony Brook Home
Campus Maps
Web Accessibility Information
Accessibility Barrier Report Form

Comments or Suggestions? | Library Webmaster

Except where otherwise noted, this work by SBU Libraries is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License .

USC Libraries
Research Guides

Organizing Your Social Sciences Research Paper

5. The Literature Review
Purpose of Guide
Design Flaws to Avoid
Independent and Dependent Variables
Glossary of Research Terms
Reading Research Effectively
Narrowing a Topic Idea
Broadening a Topic Idea
Extending the Timeliness of a Topic Idea
Academic Writing Style
Applying Critical Thinking
Choosing a Title
Making an Outline
Paragraph Development
Research Process Video Series
Executive Summary
The C.A.R.S. Model
Background Information
The Research Problem/Question
Theoretical Framework
Citation Tracking
Content Alert Services
Evaluating Sources
Primary Sources
Secondary Sources
Tiertiary Sources
Scholarly vs. Popular Publications
Qualitative Methods
Quantitative Methods
Insiderness
Using Non-Textual Elements
Limitations of the Study
Common Grammar Mistakes
Writing Concisely
Avoiding Plagiarism
Footnotes or Endnotes?
Further Readings
Generative AI and Writing
USC Libraries Tutorials and Other Guides
Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

Give a new interpretation of old material or combine new with old interpretations,
Trace the intellectual progression of the field, including major debates,
Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

Place each work in the context of its contribution to understanding the research problem being studied.
Describe the relationship of each work to the others under consideration.
Identify new ways to interpret prior research.
Reveal any gaps that exist in the literature.
Resolve conflicts amongst seemingly contradictory previous studies.
Identify areas of prior scholarship to prevent duplication of effort.
Point the way in fulfilling a need for additional research.
Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE : Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews." Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I. Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
An explanation of how each work is similar to and how it varies from the others,
Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
Persuasiveness -- which of the author's theses are most convincing or least convincing?
Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II. Development of the Literature Review

Four Basic Stages of Writing 1. Problem formulation -- which topic or field is being examined and what are its component issues? 2. Literature search -- finding materials relevant to the subject being explored. 3. Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4. Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1. Roughly how many sources would be appropriate to include? 2. What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3. Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4. Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5. Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III. Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

Current Situation : Information necessary to understand the current topic or focus of the literature review.
Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
Standards : Description of the way in which you present your information.
Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV. Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V. Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

Sources in your literature review do not clearly relate to the research problem;
You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
Does not describe the search procedures that were used in identifying the literature to review;
Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

How are they organizing their ideas?
What methods have they used to study the problem?
What theories have been used to explain, predict, or understand their research problem?
What sources have they cited to support their conclusions?
How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research? Does it forge a new path? Or, does is merely add more of the same thing being said?
Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

<< Previous: Theoretical Framework
Next: Citation Tracking >>
Last Updated: Apr 22, 2024 9:12 AM
URL: https://libguides.usc.edu/writingguide

University Library

Research Guides

Organizational Development

Literature Reviews
Books & Media
Articles & Journals

What is a Literature Review?

The scholarly conversation.

A literature review provides an overview of previous research on a topic that critically evaluates, classifies, and compares what has already been published on a particular topic. It allows the author to synthesize and place into context the research and scholarly literature relevant to the topic. It helps map the different approaches to a given question and reveals patterns. It forms the foundation for the author’s subsequent research and justifies the significance of the new investigation.

A literature review can be a short introductory section of a research article or a report or policy paper that focuses on recent research. Or, in the case of dissertations, theses, and review articles, it can be an extensive review of all relevant research.

The format is usually a bibliographic essay; sources are briefly cited within the body of the essay, with full bibliographic citations at the end.
The introduction should define the topic and set the context for the literature review. It will include the author's perspective or point of view on the topic, how they have defined the scope of the topic (including what's not included), and how the review will be organized. It can point out overall trends, conflicts in methodology or conclusions, and gaps in the research.
In the body of the review, the author should organize the research into major topics and subtopics. These groupings may be by subject, (e.g., globalization of clothing manufacturing), type of research (e.g., case studies), methodology (e.g., qualitative), genre, chronology, or other common characteristics. Within these groups, the author can then discuss the merits of each article and analyze and compare the importance of each article to similar ones.
The conclusion will summarize the main findings, make clear how this review of the literature supports (or not) the research to follow, and may point the direction for further research.
The list of references will include full citations for all of the items mentioned in the literature review.

Key Questions for a Literature Review

A literature review should try to answer questions such as

Who are the key researchers on this topic?
What has been the focus of the research efforts so far and what is the current status?
How have certain studies built on prior studies? Where are the connections? Are there new interpretations of the research?
Have there been any controversies or debate about the research? Is there consensus? Are there any contradictions?
Which areas have been identified as needing further research? Have any pathways been suggested?
How will your topic uniquely contribute to this body of knowledge?
Which methodologies have researchers used and which appear to be the most productive?
What sources of information or data were identified that might be useful to you?
How does your particular topic fit into the larger context of what has already been done?
How has the research that has already been done help frame your current investigation ?

Examples of Literature Reviews

Example of a literature review at the beginning of an article: Forbes, C. C., Blanchard, C. M., Mummery, W. K., & Courneya, K. S. (2015, March). Prevalence and correlates of strength exercise among breast, prostate, and colorectal cancer survivors . Oncology Nursing Forum, 42(2), 118+. Retrieved from http://go.galegroup.com.sonoma.idm.oclc.org/ps/i.do?p=HRCA&sw=w&u=sonomacsu&v=2.1&it=r&id=GALE%7CA422059606&asid=27e45873fddc413ac1bebbc129f7649c Example of a comprehensive review of the literature: Wilson, J. L. (2016). An exploration of bullying behaviours in nursing: a review of the literature. British Journal Of Nursing , 25 (6), 303-306. For additional examples, see:

Galvan, J., Galvan, M., & ProQuest. (2017). Writing literature reviews: A guide for students of the social and behavioral sciences (Seventh ed.). [Electronic book]

Pan, M., & Lopez, M. (2008). Preparing literature reviews: Qualitative and quantitative approaches (3rd ed.). Glendale, CA: Pyrczak Pub. [ Q180.55.E9 P36 2008]

Useful Links

Write a Literature Review (UCSC)
Literature Reviews (Purdue)
Literature Reviews: overview (UNC)
Review of Literature (UW-Madison)

Evidence Matrix for Literature Reviews

The Evidence Matrix can help you organize your research before writing your lit review. Use it to identify patterns and commonalities in the articles you have found--similar methodologies ? common theoretical frameworks ? It helps you make sure that all your major concepts covered. It also helps you see how your research fits into the context of the overall topic.

Evidence Matrix Special thanks to Dr. Cindy Stearns, SSU Sociology Dept, for permission to use this Matrix as an example.
<< Previous: Articles & Journals
Next: APA 6th ed >>
Last Updated: Feb 2, 2024 12:47 PM
URL: https://libguides.sonoma.edu/od

The Literature Review: 5. Organizing the Literature Review

1. Introduction
2. Why Do a Literature Review?
3. Methods for Searching the Literature
4. Analysing the Literature
5. Organizing the Literature Review
6. Writing the Review

1. Organizing Principles

A literature review is a piece of discursive prose, not a list describing or summarizing one piece of literature after another. It should have a single organizing principle:

Thematic - organize around a topic or issue
Chronological - sections for each vital time period
Methodological - focus on the methods used by the researchers/writers

4. Selected Online Resources

Literature Review in Education & Behavioral Sciences This is an interactive tutorial from Adelphi University Libraries on how to conduct a literature review in education and the behavioural sciences using library databases
Writing Literature Reviews This tutorial is from the Writing section of Monash University's Language and Learning Online site
The Literature Review: A Few Tips on Conducting It This guide is from the Health Services Writing Centre at the University of Toronto
Learn How to Write a Review of the Literature This guide is part of the Writer's Handbook provided by the Writing Center at the University of Wisconsin-Madison

2. Structure of the Literature Review

Although your literature review will rely heavily on the sources you read for its information, you should dictate the structure of the review. It is important that the concepts are presented in an order that makes sense of the context of your research project.

There may be clear divisions on the sets of ideas you want to discuss, in which case your structure may be fairly clear. This is an ideal situation. In most cases, there will be several different possible structures for your review.

Similarly to the structure of the research report itself, the literature review consists of:

Introduction

Introduction - profile of the study

Define or identify the general topic to provide the context for reviewing the literature
Outline why the topic is important
Identify overall trends in what has been published about the topic
Identify conflicts in theory, methodology, evidence, and conclusions
Identify gaps in research and scholarlship
Explain the criteria to be used in analysing and comparing the literature
Describe the organization of the review (the sequence)
If necessary, state why certain literature is or is not included (scope)

Body - summative, comparative, and evaluative discussion of literature reviewed

For a thematic review:

organize the review into paragraphs that present themes and identify trends relevant to your topic
each paragraph should deal with a different theme - you need to synthesize several of your readings into each paragraph in such a way that there is a clear connection between the sources
don't try to list all the materials you have identified in your literature search

From each of the section summaries:

summarize the main agreements and disagreements in the literature
summarize the general conclusions that have been drawn
establish where your own research fits in the context of the existing literature

5. A Final Checklist

Have you indicated the purpose of the review?
Have you emphasized recent developments?
Is there a logic to the way you organized the material?
Does the amount of detail included on an issue relate to its importance?
Have you been sufficiently critical of design and methodological issues?
Have you indicated when results were conflicting or inconclusive and discussed possible reasons?
Has your summary of the current literature contributed to the reader's understanding of the problems?

3. Tips on Structure

A common error in literature reviews is for writers to present material from one author, followed by information from another, then another.... The way in which you group authors and link ideas will help avoid this problem. To group authors who draw similar conclusions, you can use linking words such as:

additionally

When authors disagree, linking words that indicate contrast will show how you have analysed their work. Words such as:

on the other hand
nonetheless

will indicate to your reader how you have analysed the material. At other times, you may want to qualify an author's work (using such words as specifically, usually, or generally ) or use an example ( thus, namely, to illustrate ). In this way you ensure that you are synthesizing the material, not just describing the work already carried out in your field.

Another major problem is that literature reviews are often written as if they stand alone, without links to the rest of the paper. There needs to be a clear relationship between the literature review and the methodology to follow.

<< Previous: 4. Analysing the Literature
Next: 6. Writing the Review >>
Last Updated: Feb 8, 2022 5:25 PM
URL: https://libguides.uwi.edu/litreviewsoe

UWF Libraries

Literature Review: Conducting & Writing

Organizing/Writing
Steps for Conducting a Lit Review
Finding "The Literature"

Consider Organization

Literature review synthesis matrix, composing your literature review, managing citations / zotero.

APA Style This link opens in a new window
Chicago: Notes Bibliography This link opens in a new window
MLA Style This link opens in a new window
Sample Literature Reviews

Presentation on Synthesizing a Literature Review

You've got a focus, and you've narrowed it down to a thesis statement. Now what is the most effective way of presenting the information? What are the most important topics, subtopics, etc., that your review needs to include? And in what order should you present them? Develop an organization for your review at both a global and local level:

First, cover the basic categories Just like most academic papers, literature reviews also must contain at least three basic elements: an introduction or background information section; the body of the review containing the discussion of sources; and, finally, a conclusion and/or recommendations section to end the paper. Introduction: Gives a quick idea of the topic of the literature review, such as the central theme or organizational pattern. Body: Contains your discussion of sources and is organized either chronologically, thematically, or methodologically (see below for more information on each). Conclusions/Recommendations: Discuss what you have drawn from reviewing literature so far. Where might the discussion proceed? Organizing the body Once you have the basic categories in place, then you must consider how you will present the sources themselves within the body of your paper. Create an organizational method to focus this section even further. To help you come up with an overall organizational framework for your review, consider the following scenario and then three typical ways of organizing the sources into a review: You've decided to focus your literature review on materials dealing with sperm whales. This is because you've just finished reading Moby Dick , and you wonder if that whale's portrayal is really real. You start with some articles about the physiology of sperm whales in biology journals written in the 2020's. But these articles refer to some British biological studies performed on whales in the early 18th century. So you check those out. Then you look up a book written in 2021 with information on how sperm whales have been portrayed in other forms of art, such as in Alaskan poetry, in French painting, or on whale bone, as the whale hunters in the late 19th century used to do. This makes you wonder about American whaling methods during the time portrayed in Moby Dick , so you find some academic articles published in the last five years on how accurately Herman Melville portrayed the whaling scene in his novel. Chronological If your review follows the chronological method, you could write about the materials above according to when they were published. For instance, first you would talk about the British biological studies of the 18th century, then about Moby Dick, published in 1851, then the book on sperm whales in other art (2021), and finally the biology articles (2000s) and the recent articles (last five years) on American whaling of the 19th century. But there is relatively no continuity among subjects here. And notice that even though the sources on sperm whales in other art and on American whaling are written recently, they are about other subjects/objects that were created much earlier. Thus, the review loses its chronological focus. By publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on biological studies of sperm whales if the progression revealed a change in dissection practices of the researchers who wrote and/or conducted the studies. By trend A better way to organize the above sources chronologically is to examine the sources under another trend, such as the history of whaling. Then your review would have subsections according to eras within this period. For instance, the review might examine whaling from pre-1600-1699, 1700-1799, and 1800-1899. Under this method, you would combine the recent studies on American whaling in the 19th century with Moby Dick itself in the 1800-1899 category, even though the authors wrote more than a century apart.

But more authentic thematic reviews tend to break away from chronological order. For instance, a thematic review of material on sperm whales might examine how they are portrayed as "evil" in cultural documents. The subsections might include how they are personified, how their proportions are exaggerated, and their behaviors misunderstood. A review organized in this manner would shift between time periods within each section according to the point made.

Methodological

Sometimes, though, you might need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. Put in only what is necessary. Here are a few other sections you might want to consider:

Current Situation : Information necessary to understand the topic or focus of the literature review.

History : The chronological progression of the field, the literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.

Methods and/or Standards : The criteria you used to select the sources in your literature review or the way in which you present your information. For instance, you might explain that your review includes only peer-reviewed articles and journals.

Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

(Adapted from "Literature Reviews" from The Writing Center, University of North Carolina at Chapel Hill )

This synthesis matrix in Excel can help you get a jumpstart on finding ways in which the literature differs and is the same.

Synthesis Matrix

O nce you've settled on a general pattern of organization, you're ready to write each section. There are a few guidelines you should follow during the writing stage. Here is a sample paragraph from a literature review about sexism and language to illuminate the following discussion:

However, other studies have shown that even gender-neutral antecedents are more likely to produce masculine images than feminine ones (Gastil, 1990). Hamilton (1988) asked students to complete sentences that required them to fill in pronouns that agreed with gender-neutral antecedents such as "writer," "pedestrian," and "persons." The students were asked to describe any image they had when writing the sentence. Hamilton found that people imagined 3.3 men to each woman in the masculine "generic" condition and 1.5 men per woman in the unbiased condition. Thus, while ambient sexism accounted for some of the masculine bias, sexist language amplified the effect. (Source: Erika Falk and Jordan Mills, "Why Sexist Language Affects Persuasion: The Role of Homophily, Intended Audience, and Offense," Women and Language19:2.

Use evidence

In the example above, the writers refer to several other sources when making their point. A literature review in this sense is just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence to show that what you are saying is valid.

Be selective

Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the review's focus, whether it is thematic, methodological, or chronological.

Use quotes sparingly

Falk and Mills do not use any direct quotes. That is because the survey nature of the literature review does not allow for in-depth discussion or detailed quotes from the text. Some short quotes here and there are okay, though if you want to emphasize a point, or if what the author said just cannot be rewritten in your own words. Notice that Falk and Mills do quote certain terms that were coined by the author, not common knowledge, or taken directly from the study. But if you find yourself wanting to put in more quotes, check with your instructor.

Summarize and synthesize

Remember to summarize and synthesize your sources within each paragraph as well as throughout the review. The authors here recapitulate important features of Hamilton's study, but then synthesize it by rephrasing the study's significance and relating it to their own work.

Keep your own voice

While the literature review presents others' ideas, your voice (the writer's) should remain front and center. Notice that Falk and Mills weave references to other sources into their own text, but they still maintain their own voice by starting and ending the paragraph with their own ideas and their own words. The sources support what Falk and Mills are saying.

Use caution when paraphrasing

When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. In the preceding example, Falk and Mills either directly refer in the text to the author of their source, such as Hamilton, or they provide ample notation in the text when the ideas they are mentioning are not their own, for example, Gastil's. For more information, please see our handout on plagiarism .

Use a citation manager to manage citations from journals, books, documents, and internet sites.

A good one to use is Zotero. Instructions on using it can be found in the following guide:

Zotero Guide

Content for this section of the guide was taken from Literature Reviews from The Writing Center, University of North Carolina at Chapel Hill , under the guidelines of their Creative Commons License.

<< Previous: Finding "The Literature"
Next: APA Style >>
Last Updated: Mar 22, 2024 9:37 AM
URL: https://libguides.uwf.edu/litreview

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

Advanced Search
Journal List
Springer Nature - PMC COVID-19 Collection

Systematic Literature Review of E-Learning Capabilities to Enhance Organizational Learning

Michail n. giannakos.

1 Norwegian University of Science and Technology, Trondheim, Norway

Patrick Mikalef

Ilias o. pappas.

2 University of Agder, Kristiansand, Norway

E-learning systems are receiving ever increasing attention in academia, business and public administration. Major crises, like the pandemic, highlight the tremendous importance of the appropriate development of e-learning systems and its adoption and processes in organizations. Managers and employees who need efficient forms of training and learning flow within organizations do not have to gather in one place at the same time or to travel far away to attend courses. Contemporary affordances of e-learning systems allow users to perform different jobs or tasks for training courses according to their own scheduling, as well as to collaborate and share knowledge and experiences that result in rich learning flows within organizations. The purpose of this article is to provide a systematic review of empirical studies at the intersection of e-learning and organizational learning in order to summarize the current findings and guide future research. Forty-seven peer-reviewed articles were collected from a systematic literature search and analyzed based on a categorization of their main elements. This survey identifies five major directions of the research on the confluence of e-learning and organizational learning during the last decade. Future research should leverage big data produced from the platforms and investigate how the incorporation of advanced learning technologies (e.g., learning analytics, personalized learning) can help increase organizational value.

Introduction

E-learning covers the integration of information and communication technology (ICT) in environments with the main goal of fostering learning (Rosenberg and Foshay 2002 ). The term “e-learning” is often used as an umbrella term to portray several modes of digital learning environments (e.g., online, virtual learning environments, social learning technologies). Digitalization seems to challenge numerous business models in organizations and raises important questions about the meaning and practice of learning and development (Dignen and Burmeister 2020 ). Among other things, the digitalization of resources and processes enables flexible ways to foster learning across an organization’s different sections and personnel.

Learning has long been associated with formal or informal education and training. However organizational learning is much more than that. It can be defined as “a learning process within organizations that involves the interaction of individual and collective (group, organizational, and inter-organizational) levels of analysis and leads to achieving organizations’ goals” (Popova-Nowak and Cseh 2015 ) with a focus on the flow of knowledge across the different organizational levels (Oh 2019 ). Flow of knowledge or learning flow is the way in which new knowledge flows from the individual to the organizational level (i.e., feed forward) and vice versa (i.e., feedback) (Crossan et al. 1999 ; March 1991 ). Learning flow and the respective processes constitute the cornerstone of an organization’s learning activities (e.g., from physical training meetings to digital learning resources), they are directly connected to the psycho-social experiences of an organization’s members, and they eventually lead to organizational change (Crossan et al. 2011 ). The overall organizational learning is extremely important in an organization because it is associated with the process of creating value from an organizations’ intangible assets. Moreover, it combines notions from several different domains, such as organizational behavior, human resource management, artificial intelligence, and information technology (El Kadiri et al. 2016 ).

A growing body of literature lies at the intersection of e-learning and organizational learning. However, there is limited work on the qualities of e-learning and the potential of its qualities to enhance organizational learning (Popova-Nowak and Cseh 2015 ). Blockages and disruptions in the internal flow of knowledge is a major reason why organizational change initiatives often fail to produce their intended results (Dee and Leisyte 2017 ). In recent years, several models of organizational learning have been published (Berends and Lammers 2010 ; Oh 2019 ). However, detailed empirical studies indicate that learning does not always proceed smoothly in organizations; rather, the learning meets interruptions and breakdowns (Engeström et al. 2007 ).

Discontinuities and disruptions are common phenomena in organizational learning (Berends and Lammers 2010 ), and they stem from various causes. For example, organizational members’ low self-esteem, unsupportive technology and instructors (Garavan et al. 2019 ), and even crises like the Covid-19 pandemic can result in demotivated learners and overall unwanted consequences for their learning (Broadbent 2017 ). In a recent conceptual article, Popova-Nowak and Cseh ( 2015 ) emphasized that there is a limited use of multidisciplinary perspectives to investigate and explain the processes and importance of utilizing the available capabilities and resources and of creating contexts where learning is “attractive to individual agents so that they can be more engaged in exploring ways in which they can contribute through their learning to the ongoing renewal of organizational routines and practices” (Antonacopoulou and Chiva 2007 , p. 289).

Despite the importance of e-learning, the lack of systematic reviews in this area significantly hinders research on the highly promising value of e-learning capabilities for efficiently supporting organizational learning. This gap leaves practitioners and researchers in uncharted territories when faced with the task of implementing e-learning designs or deciding on their digital learning strategies to enhance the learning flow of their organizations. Hence, in order to derive meaningful theoretical and practical implications, as well as to identify important areas for future research, it is critical to understand how the core capabilities pertinent to e-learning possess the capacity to enhance organizational learning.

In this paper, we define e-learning enhanced organizational learning (eOL) as the utilization of digital technologies to enhance the process of improving actions through better knowledge and understanding in an organization. In recent years, a significant body of research has focused on the intersection of e-learning and organizational learning (e.g., Khandakar and Pangil 2019 ; Lin et al. 2019 ; Menolli et al. 2020 ; Turi et al. 2019 ; Xiang et al. 2020 ). However, there is a lack of systematic work that summarizes and conceptualizes the results in order to support organizations that want to move from being information-based enterprises to being knowledge-based ones (El Kadiri et al. 2016 ). In particular, recent technological advances have led to an increase in research that leverages e-learning capacities to support organizational learning, from virtual reality (VR) environments (Costello and McNaughton 2018 ; Muller Queiroz et al. 2018 ) to mobile computing applications (Renner et al. 2020 ) to adaptive learning and learning analytics (Zhang et al. 2019 ). These studies support different skills, consider different industries and organizations, and utilize various capacities while focusing on various learning objectives (Garavan et al. 2019 ). Our literature review aims to tease apart these particularities and to investigate how these elements have been utilized over the past decade in eOL research. Therefore, in this review we aim to answer the following research questions (RQs):

RQ1: What is the status of research at the intersection of e-learning and organizational learning, seen through the lens of areas of implementation (e.g., industries, public sector), technologies used, and methodologies (e.g., types of data and data analysis techniques employed)?
RQ2: How can e-learning be leveraged to enhance the process of improving actions through better knowledge and understanding in an organization?

Our motivation for this work is based on the emerging developments in the area of learning technologies that have created momentum for their adoption by organizations. This paper provides a review of research on e-learning capabilities to enhance organizational learning with the purpose of summarizing the findings and guiding future studies. This study can provide a springboard for other scholars and practitioners, especially in the area of knowledge-based enterprises, to examine e-learning approaches by taking into consideration the prior and ongoing research efforts. Therefore, in this paper we present a systematic literature review (SLR) (Kitchenham and Charters 2007 ) on the confluence of e-learning and organizational learning that uncovers initial findings on the value of e-learning to support organizational learning while also delineating several promising research streams.

The rest of this paper is organized as follows. In the next section, we present the related background work. The third section describes the methodology used for the literature review and how the studies were selected and analyzed. The fourth section presents the research findings derived from the data analysis based on the specific areas of focus. In the fifth section, we discuss the findings, the implications for practice and research, and the limitations of the selected methodological approach. In the final section, we summarize the conclusions from the study and make suggestions for future work.

Background and Related Work

E-learning systems.

E-learning systems provide solutions that deliver knowledge and information, facilitate learning, and increase performance by developing appropriate knowledge flow inside organizations (Menolli et al. 2020 ). Putting into practice and appropriately managing technological solutions, processes, and resources are necessary for the efficient utilization of e-learning in an organization (Alharthi et al. 2019 ). Examples of e-learning systems that have been widely adopted by various organizations are Canvas, Blackboard, and Moodle. Such systems provide innovative services for students, employees, managers, instructors, institutions, and other actors to support and enhance the learning processes and facilitate efficient knowledge flow (Garavan et al. 2019 ). Functionalities, such as creating modules to organize mini course information and learning materials or communication channels such as chat, forums, and video exchange, allow instructors and managers to develop appropriate training and knowledge exchange (Wang et al. 2011 ). Nowadays, the utilization of various e-learning capabilities is a commodity for supporting organizational and workplace learning. Such learning refers to training or knowledge development (also known in the literature as learning and development, HR development, and corporate training: Smith and Sadler-Smith 2006 ; Garavan et al. 2019 ) that takes place in the context of work.

Previous studies have focused on evaluating e-learning systems that utilize various models and frameworks. In particular, the development of maturity models, such as the e-learning capability maturity model (eLCMM), addresses technology-oriented concerns (Hammad et al. 2017 ) by overcoming the limitations of the domain-specific models (e.g., game-based learning: Serrano et al. 2012 ) or more generic lenses such as the e-learning maturity model (Marshall 2006 ). The aforementioned models are very relevant since they focus on assessing the organizational capabilities for sustainably developing, deploying, and maintaining e-learning. In particular, the eLCMM focuses on assessing the maturity of adopting e-learning systems and adds a feedback building block for improving learners’ experiences (Hammad et al. 2017 ). Our proposed literature review builds on the previously discussed models, lenses, and empirical studies, and it provides a review of research on e-learning capabilities with the aim of enhancing organizational learning in order to complement the findings of the established models and guide future studies.

E-learning systems can be categorized into different types, depending on their functionalities and affordances. One very popular e-learning type is the learning management system (LMS), which includes a virtual classroom and collaboration capabilities and allows the instructor to design and orchestrate a course or a module. An LMS can be either proprietary (e.g., Blackboard) or open source (e.g., Moodle). These two types differ in their features, costs, and the services they provide; for example, proprietary systems prioritize assessment tools for instructors, whereas open-source systems focus more on community development and engagement tools (Alharthi et al. 2019 ). In addition to LMS, e-learning systems can be categorized based on who controls the pace of learning; for example, an institutional learning environment (ILE) is provided by the organization and is usually used for instructor-led courses, while a personal learning environment (PLE) is proposed by the organization and is managed personally (i.e., learner-led courses). Many e-learning systems use a hybrid version of ILE and PLE that allows organizations to have either instructor-led or self-paced courses.

Besides the controlled e-learning systems, organizations have been using environments such as social media (Qi and Chau 2016 ), massive open online courses (MOOCs) (Weinhardt and Sitzmann 2018 ) and other web-based environments (Wang et al. 2011 ) to reinforce their organizational learning potential. These systems have been utilized through different types of technology (e.g., desktop applications, mobile) that leverage the various capabilities offered (e.g., social learning, VR, collaborative systems, smart and intelligent support) to reinforce the learning and knowledge flow potential of the organization. Although there is a growing body of research on e-learning systems for organizational learning due to the increasingly significant role of skills and expertise development in organizations, the role and alignment of the capabilities of the various e-learning systems with the expected competency development remains underexplored.

Organizational Learning

There is a large body of research on the utilization of technologies to improve the process and outcome dimensions of organizational learning (Crossan et al. 1999 ). Most studies have focused on the learning process and on the added value that new technologies can offer by replacing some of the face-to-face processes with virtual processes or by offering new, technology-mediated phases to the process (Menolli et al. 2020 ; Lau 2015 ) highlighted how VR capabilities can enhance organizational learning, describing the new challenges and frameworks needed in order to effectively utilize this potential. In the same vein, Zhang et al. ( 2017 ) described how VR influences reflective thinking and considered its indirect value to overall learning effectiveness. In general, contemporary research has investigated how novel technologies and approaches have been utilized to enhance organizational learning, and it has highlighted both the promises and the limitations of the use of different technologies within organizations.

In many organizations, alignment with the established infrastructure and routines, and adoption by employees are core elements for effective organizational learning (Wang et al. 2011 ). Strict policies, low digital competence, and operational challenges are some of the elements that hinder e-learning adoption by organizations (Garavan et al. 2019 ; Wang 2018 ) demonstrated the importance of organizational, managerial, and job support for utilizing individual and social learning in order to increase the adoption of organizational learning. Other studies have focused on the importance of communication through different social channels to develop understanding of new technology, to overcome the challenges employees face when engaging with new technology, and, thereby, to support organizational learning (Menolli et al. 2020 ). By considering the related work in the area of organizational learning, we identified a gap in aligning an organization’s learning needs with the capabilities offered by the various technologies. Thus, systematic work is needed to review e-learning capabilities and how these capabilities can efficiently support organizational learning.

E-learning Systems to Enhance Organizational Learning

When considering the interplay between e-learning systems and organizational learning, we observed that a major challenge for today’s organizations is to switch from being information-based enterprises to become knowledge-based enterprises (El Kadiri et al. 2016 ). Unidirectional learning flows, such as formal and informal training, are important but not sufficient to cover the needs that enterprises face (Manuti et al. 2015 ). To maintain enterprises’ competitiveness, enterprise staff have to operate in highly intense information and knowledge-oriented environments. Traditional learning approaches fail to substantiate learning flow on the basis of daily evidence and experience. Thus, novel, ubiquitous, and flexible learning mechanisms are needed, placing humans (e.g., employees, managers, civil servants) at the center of the information and learning flow and bridging traditional learning with experiential, social, and smart learning.

Organizations consider lack of skills and competences as being the major knowledge-related factors hampering innovation (El Kadiri et al. 2016 ). Thus, solutions need to be implemented that support informal, day-to-day, and work training (e.g., social learning, collaborative learning, VR/AR solutions) in order to develop individual staff competences and to upgrade the competence affordances at the organizational level. E-learning-enhanced organizational learning has been delivered primarily in the form of web-based learning (El Kadiri et al. 2016 ). More recently, the TEL tools portfolio has rapidly expanded to make more efficient joint use of novel learning concepts, methodologies, and technological enablers to achieve more direct, effective, and lasting learning impacts. Virtual learning environments, mobile-learning solutions, and AR/VR technologies and head-mounted displays have been employed so that trainees are empowered to follow their own training pace, learning topics, and assessment tests that fit their needs (Costello and McNaughton 2018 ; Mueller et al. 2011 ; Muller Queiroz et al. 2018 ). The expanding use of social networking tools has also brought attention to the contribution of social and collaborative learning (Hester et al. 2016 ; Wei and Ram 2016 ).

Contemporary learning systems supporting adaptive, personalized, and collaborative learning expand the tools available in eOL and contribute to the adoption, efficiency, and general prospects of the introduction of TEL in organizations (Cheng et al. 2011 ). In recent years, eOL has emphasized how enterprises share knowledge internally and externally, with particular attention being paid to systems that leverage collaborative learning and social learning functionalities (Qi and Chau 2016 ; Wang 2011 ). This is the essence of computer-supported collaborative learning (CSCL). The CSCL literature has developed a framework that combines individual learning, organizational learning, and collaborative learning, facilitated by establishing adequate learning flows and emerges effective learning in an enterprise learning (Goggins et al. 2013 ), in Fig. 1 .

An external file that holds a picture, illustration, etc.
Object name is 10796_2020_10097_Fig1_HTML.jpg

Representation of the combination of enterprise learning and knowledge flows. (adapted from Goggins et al. 2013 )

Establishing efficient knowledge and learning flows is a primary target for future data-driven enterprises (El Kadiri et al. 2016 ). Given the involved knowledge, the human resources, and the skills required by enterprises, there is a clear need for continuous, flexible, and efficient learning. This can be met by contemporary learning systems and practices that provide high adoption, smooth usage, high satisfaction, and close alignment with the current practices of an enterprise. Because the required competences of an enterprise evolve, the development of competence models needs to be agile and to leverage state-of-the art technologies that align with the organization’s processes and models. Therefore, in this paper we provide a review of the eOL research in order to summarize the findings, identify the various capabilities of eOL, and guide the development of organizational learning in future enterprises as well as in future studies.

Methodology

To answer our research questions, we conducted an SLR, which is a means of evaluating and interpreting all available research relevant to a particular research question, topic area, or phenomenon of interest. A SLR has the capacity to present a fair evaluation of a research topic by using a trustworthy, rigorous, and auditable methodology (Kitchenham and Charters 2007 ). The guidelines used (Kitchenham and Charters 2007 ) were derived from three existing guides adopted by medical researchers. Therefore, we adopted SLR guidelines that follow transparent and widely accepted procedures (especially in the area of software engineering and information systems, as well as in e-learning), minimize potential bias (researchers), and support reproducibility (Kitchenham and Charters 2007 ). Besides the minimization of bias and support for reproducibility, an SLR allows us to provide information about the impact of some phenomenon across a wide range of settings, contexts, and empirical methods. Another important advantage is that, if the selected studies give consistent results, SLRs can provide evidence that the phenomenon is robust and transferable (Kitchenham and Charters 2007 ).

Article Collection

Several procedures were followed to ensure a high-quality review of the literature of eOL. A comprehensive search of peer-reviewed articles was conducted in February 2019 (short papers, posters, dissertations, and reports were excluded), based on a relatively inclusive range of key terms: “organizational learning” & “elearning”, “organizational learning” & “e-learning”, “organisational learning” & “elearning”, and “organisational learning” & “e-learning”. Publications were selected from 2010 onwards, because we identified significant advances since 2010 (e.g., MOOCs, learning analytics, personalized learning) in the area of learning technologies. A wide variety of databases were searched, including SpringerLink, Wiley, ACM Digital Library, IEEE Xplore, Science Direct, SAGE, ERIC, AIS eLibrary, and Taylor & Francis. The selected databases were aligned with the SLR guidelines (Kitchenham and Charters 2007 ) and covered the major venues in IS and educational technology (e.g., a basket of eight IS journals, the top 20 journals in the Google Scholar IS subdiscipline, and the top 20 journals in the Google Scholar Educational Technology subdiscipline). The search process uncovered 2,347 peer-reviewed articles.

Inclusion and Exclusion Criteria

The selection phase determines the overall validity of the literature review, and thus it is important to define specific inclusion and exclusion criteria. As Dybå and Dingsøyr ( 2008 ) specified, the quality criteria should cover three main issues – namely, rigor, credibility, and relevance – that need to be considered when evaluating the quality of the selected studies. We applied eight quality criteria informed by the proposed Critical Appraisal Skills Programme (CASP) and related works (Dybå and Dingsøyr 2008 ). Table Table1 1 presents these criteria.

Quality criteria

Therefore, studies were eligible for inclusion if they were focused on eOL. The aforementioned criteria were applied in stages 2 and 3 of the selection process (see Fig. 2 ), when we assessed the papers based on their titles and abstracts, and read the full papers. From March 2020, we performed an additional search (stage 4) following the same process for papers published after the initial search period (i.e., 2010–February 2019). The additional search returned seven papers. Figure Figure2 2 summarizes the stages of the selection process.

An external file that holds a picture, illustration, etc.
Object name is 10796_2020_10097_Fig2_HTML.jpg

Stages of the selection process

Each collected study was analyzed based on the following elements: study design (e.g., experiment, case study), area (e.g., IT, healthcare), technology (e.g., wiki, social media), population (e.g., managers, employees), sample size, unit of analysis (individual, firm), data collections (e.g., surveys, interviews), research method, data analysis, and the main research objective of the study. It is important to highlight that the articles were coded based on the reported information, that different authors reported information at different levels of granularity (e.g., an online system vs. the name of the system), and that in some cases the information was missing from the paper. Overall, we endeavored to code the articles as accurately and completely as possible.

The coding process was iterative with regular consensus meetings between the two researchers involved. The primary coder prepared the initial coding for a number of articles and both coders reviewed and agreed on the coding in order to reach the final codes presented in the Appendix . Disagreements between the coders and inexplicit aspects of the reviewed papers were discussed and resolved in regular consensus meetings. Although this process did not provide reliability indices (e.g., Cohen’s kappa), it did provide certain reliability in terms of consistency of the coding and what Krippendorff ( 2018 ) stated as the reliability of “the degree to which members of a designated community concur on the readings, interpretations, responses to, or uses of given texts or data”, which is considered acceptable research practice (McDonald et al. 2019 ).

In this section, we present the detailed results of the analysis of the 47 papers. Analysis of the studies was performed using non-statistical methods that considered the variables reported in the Appendix . This section is followed by an analysis and discussion of the categories.

Sample Size and Population Involved

The categories related to the sample of the articles and included the number of participants in each study (size), their position (e.g., managers, employees), and the area/topic covered by the study. The majority of the studies involved employees (29), with few studies involving managers (6), civil servants (2), learning specialists (2), clients, and researchers. Regarding the sample size, approximately half of the studies (20) were conducted with fewer than 100 participants; some (12) can be considered large-scale studies (more than 300 participants); and only a few (9) can be considered small scale (fewer than 20 participants). In relation to the area/topic of the study, most studies (11) were conducted in the context of the IT industry, but there was also good coverage of other important areas (i.e., healthcare, telecommunications, business, public sector). Interestingly, several studies either did not define the area or were implemented in a generic context (sector-agnostic studies, n = 10), and some studies were implemented in a multi-sector context (e.g., participants from different sections or companies, n = 4).

Research Methods

When assessing the status of research for an area, one of the most important aspects is the methodology used. By “method” in the Appendix , we refer to the distinction between quantitative, qualitative, and mixed methods research. In addition to the method, in our categorization protocol we also included “study design” to refer to the distinction between survey studies (i.e., those that gathered data by asking a group of participants), experiments (i.e., those that created situations to record beneficial data), and case studies (i.e., those that closely studied a group of individuals).

Based on this categorization, the Appendix shows that the majority of the papers were quantitative (34) and qualitative (7), with few studies (6) utilizing mixed methods. Regarding the study design, most of the studies were survey studies (26), 13 were case studies, and fewer were experiments (8). For most studies, the individual participant (40) was the unit of analysis, with few studies having the firm as the unit of analysis, and only one study using the training session as a unit of analysis. Regarding the measures used in the studies, most utilized surveys (39), with 11 using interviews, and only a few studies using field notes from focus groups (2) and log files from the systems (2). Only eight studies involved researchers using different measures to triangulate or extend their findings. Most articles used structural equation modeling (SEM) (17) to analyze their data, with 13 studies employing descriptive statistics, seven using content analysis, nine using regression analysis or analyses of variances/covariance, and one study using social network analysis (SNA).

Technologies

Concerning the technology used, most of the studies (17) did not study a specific system, referring instead in their investigation to a generic e-learning or technological solution. Several studies (9) named web-based learning environments, without describing the functionalities of the identified system. Other studies focused on online learning environments (4), collaborative learning systems (3), social learning systems (3), smart learning systems (2), podcasting (2), with the rest of the studies using a specific system (e.g., a wiki, mobile learning, e-portfolios, Second Life, web application).

Research Objectives

The research objectives of the studies could be separated into six main categories. The first category focuses on the intention of the employees to use the technology (9); the second focuses on the performance of the employees (8); the third focuses on the value/outcome for the organization (4); the fourth focuses on the actual usage of the system (7); the fifth focuses on employees’ satisfaction (4); and the sixth focuses on the ability of the proposed system to foster learning (9). In addition to these six categories, we also identified studies that focused on potential barriers for eOL in organizations (Stoffregen et al. 2016 ), the various benefits associated with the successful implementation of eOL (Liu et al. 2012 ), the feasibility of eOL (Kim et al. 2014 ; Mueller et al. 2011 ), and the alignment of the proposed innovation with the other processes and systems in the organization (Costello and McNaughton 2018 ).

E-learning Capabilities in Various Organizations and for Various Objectives

The technology used has an inherent role for both the organization and the expected eOL objective. E-learning systems are categorized based on their functionalities and affordances. Based on the information reported in the selected papers, we ranked them based on the different technologies and functionalities (e.g., collaborative, online, smart). To do so, we focused on the main elements described in the selected paper; for instance, a paper that described the system as wiki-based or indicated that the system was Second Life was ranked as such, rather than being added to collaborative systems or social learning respectively. We did this because we wanted to capture all the available information since it gave us additional insights (e.g., Second Life is both a social and a VR system).

To investigate the connection between the various technologies used to enhance organizational learning and their application in the various organizations, we utilized the coding (see Appendix ) and mapped the various e-learning technologies (or their affordances) with the research industries to which they applied (Fig. 3 ). There was occasionally a lack of detailed information about the capabilities of the e-learning systems applied (e.g., generic, or a web application, or an online system), which limited the insights. Figure Figure3 3 provides a useful mapping of the confluence of e-learning technologies and their application in the various industries.

An external file that holds a picture, illustration, etc.
Object name is 10796_2020_10097_Fig3_HTML.jpg

Association of the different e-learning technologies with the industries to which they are applied in the various studies. Note: The size of the circles depicts the frequency of studies, with the smallest circle representing one study and the largest representing six studies. The mapping is extracted from the data in the Appendix , which outlines the papers that belong in each of the circles

To investigate the connection between the various technologies used to enhance organizational learning and their intended objectives, we utilized the coding of the articles (see Appendix ) and mapped the various e-learning technologies (or their affordances) with the intended objectives, as reported in the various studies (Fig. 4 ). The results in Fig. 4 show the objectives that are central in eOL research (e.g., performance, fostering learning, adoption, and usage) as well as those objectives on which few studies have focused (e.g., alignment, feasibility, behavioral change). In addition, the results also indicate the limited utilization of the various e-learning capabilities (e.g., social, collaborative, smart) to achieve objectives connected with those capabilities (e.g., social learning and behavioral change, collaborative learning, and barriers).

An external file that holds a picture, illustration, etc.
Object name is 10796_2020_10097_Fig4_HTML.jpg

Association of the different e-learning technologies with the objectives investigated in the various studies. Note: The size of the circles depicts the frequency of studies, with the smallest circle representing one study and the largest representing five studies. The mapping is extracted from the data in the Appendix , which outlines the papers that belong in each of the circles

5. Discussion

After reviewing the 47 identified articles in the area of eOL, we can observe that all the works acknowledge the importance of the affordances offered by different e-learning technologies (e.g., remote collaboration, anytime anywhere), the importance of the relationship between eOL and employees’ satisfaction and performance, and the benefits associated with organizational value and outcome. Most of the studies agree that eOL provides employees, managers, and even clients with opportunities to learn in a more differentiated manner, compared to formal and face-to-face learning. However, how the organization adopts and puts into practice these capabilities to leverage them and achieve its goals are complex and challenging procedures that seem to be underexplored.

Several studies (Lee et al. 2015a ; Muller Queiroz et al. 2018 ; Tsai et al. 2010 ) focused on the positive effect of perceived managerial support, perceived usefulness, perceived ease of use, and other technology acceptance model (TAM) constructs of the e-learning system in supporting all three levels of learning (i.e., individual, collaborative, and organizational). Another interesting dimension highlighted by many studies (Choi and Ko 2012 ; Khalili et al. 2012 ; Yanson and Johnson 2016 ) is the role of socialization in the adoption and usage of the e-learning systems that offer these capabilities. Building connections and creating a shared learning space in the e-learning system is challenging but also critical for the learners (Yanson and Johnson 2016 ). This is consistent with the expectancy-theoretical explanation of how social context impacts on employees’ motivation to participate in learning (Lee et al. 2015a ; Muller Queiroz et al. 2018 ).

The organizational learning literature suggests that e-learning may be more appropriate for the acquisition of certain types of knowledge than others (e.g., procedural vs. declarative, or hard-skills vs. soft-skills); however, there is no empirical evidence for this (Yanson and Johnson 2016 ). To advance eOL research, there is a need for a significant move to address complex, strategic skills by including learning and development professionals (Garavan et al. 2019 ) and by developing strategic relationships. Another important element is to utilize e-learning technology that addresses and integrates organizational, individual, and social perspectives in eOL (Wang 2011 ). This is also identified in our literature review since we found only limited specialized e-learning systems in domain areas that have traditionally benefited from such technology. For instance, although there were studies that utilized VR environments (Costello and McNaughton 2018 ; Muller Queiroz et al. 2018 ) and video-based learning systems (Wei et al. 2013 ; Wei and Ram 2016 ), there was limited focus in contemporary eOL research on how specific affordances of the various environments that are used in organizations (e.g., Carnetsoft, Outotec HSC, and Simscale for simulations of working environments; or Raptivity, YouTube, and FStoppers to gain specific skills and how-to knowledge) can benefit the intended goals or be integrated with the unique qualities of the organization (e.g., IT, healthcare).

For the design and the development of the eOL approach, the organization needs to consider the alignment of individual learning needs, organizational objectives, and the necessary resources (Wang 2011 ). To achieve this, it is advisable for organizations to define the expected objectives, catalogue the individual needs, and select technologies that have the capacity to support and enrich learners with self-directed and socially constructed learning practices in the organization (Wang 2011 ). This needs to be done by taking into consideration that on-demand eOL is gradually replacing the classic static eOL curricula and processes (Dignen and Burmeister 2020 ).

Another important dimension of eOL research is the lenses used to approach effectiveness. The selected papers approached effectiveness with various objectives, such as fostering learning, usage of the e-learning system, employees’ performance, and the added organizational value (see Appendix ). To measure these indices, various metrics (quantitative, qualitative, and mixed) have been applied. The qualitative dimensions emphasize employees’ satisfaction and system usage (e.g., Menolli et al. 2020 ; Turi et al. 2019 ), as well as managers’ perceived gained value and benefits (e.g., Lee et al. 2015b ; Xiang et al. 2020 ) and firms’ perceived effective utilization of eOL resources (López-Nicolás and Meroño-Cerdán 2011 ). The quantitative dimensions focus on usage, feasibility, and experience at different levels within an organization, based on interviews, focus groups, and observations (Costello and McNaughton 2018 ; Michalski 2014 ; Stoffregen et al. 2016 ). However, it is not always clear the how eOL effectiveness has been measured, nor the extent to which eOL is well aligned with and is strategically impactful on delivering the strategic agenda of the organization (Garavan et al. 2019 ).

Research on digital technologies is developing rapidly, and big data and business analytics have the potential to pave the way for organizations’ digital transformation and sustainable development (Mikalef et al. 2018 ; Pappas et al. 2018 ); however, our review finds surprisingly limited use of big data and analytics in eOL. Despite contemporary e-learning systems adopting data-driven mechanisms, as well as advances in learning analytics (Siemens and Long 2011 ), the results of our analysis indicate that learner-generated data in the context of eOL are used in only a few studies to extract very limited insights with respect to the effectiveness of eOL and the intended objectives of the respective study (Hung et al. 2015 ; Renner et al. 2020 ; Rober and Cooper 2011 ). Therefore, eOL research needs to focus on data-driven qualities that will allow future researchers to gain deeper insights into which capabilities need to be developed to monitor the effectiveness of the various practices and technologies, their alignment with other functions of the organization, and how eOL can be a strategic and impactful vehicle for materializing the strategic agenda of the organization.

Status of eOL Research

The current review suggests that, while the efficient implementation of eOL entails certain challenges, there is also a great potential for improving employees’ performance as well as overall organizational outcome and value. There are also opportunities for improving organizations’ learning flow, which might not be feasible with formal learning and training. In order to construct the main research dimensions of eOL research and to look more deeply at the research objectives of the studies (the information we coded as objectives in the Appendix ), we performed a content analysis and grouped the research objectives. This enabled us to summarize the contemporary research on eOL according to five major categories, each of which is describes further below. As the research objectives of the published work shows, the research on eOL conducted during the last decade has particularly focused on the following five directions.

Research has particularly focused on how easy the technology is to use, on how useful it is, or on how well aligned/integrated it is with other systems and processes within the organization. In addition, studies have used different learning technologies (e.g., smart, social, personalized) to enhance organizational learning in different contexts and according to different needs. However, most works have focused on affordances such as remote training and the development of static courses or modules to share information with learners. Although a few studies have utilized contemporary e-learning systems (see Appendix ), even in these studies there is a lack of alignment between the capabilities of those systems (e.g., open online course, adaptive support, social and collaborative learning) and the objectives and strategy of the organization (e.g., organizational value, fostering learning).

The reviewed work has emphasized how different factors contribute to different levels of organizational learning, and it has focused on practices that address individual, collaborative, and organizational learning within the structure of the organization. In particular, most of the reviewed studies recognize that organizational learning occurs at multiple levels: individual, team (or group), and organization. In other words, although each of the studies carried out an investigation within a given level (except for Garavan et al. 2019 ), there is a recognition and discussion of the different levels. Therefore, the results align with the 4I framework of organizational learning that recognizes how learning across the different levels is linked by social and psychological processes: intuiting, interpreting, integrating, and institutionalizing (the 4Is) (Crossan et al. 1999 ). However, most of the studies focused on the institutionalizing-intuiting link (i.e., top-down feedback); moreover, no studies focused on contemporary learning technologies and processes that strengthen the learning flow (e.g., self-regulated learning).

There is a considerable amount of predominantly qualitative studies that focus on potential barriers to eOL implementation as well as on the risks and requirements associated with the feasibility and successful implementation of eOL. In the same vein, research has emphasized the importance of alignment of eOL (both in processes and in technologies) within the organization. These critical aspects for effective eOL are sometimes the main objectives of the studies (see Appendix ). However, most of the elements relating to the effectiveness of eOL were measured with questionnaires and interviews with employees and managers, and very little work was conducted on how to leverage the digital technologies employed in eOL, big data, and analytics in order to monitor the effectiveness of eOL.

In most of the studies, the main objective was to increase employees’ adoption, satisfaction, and usage of the e-learning system. In addition, several studies focused on the e-learning system’s ability to improve employees’ performance, increase the knowledge flow in the organization, and foster learning. Most of the approaches were employee-centric, with a small amount of studies focusing on managers and the firm in general. However, employees were seen as static entities within the organization, with limited work investigating how eOL-based training exposes employees to new knowledge, broadens their skills repertoire, and has tremendous potential for fostering innovation (Lin and Sanders 2017 ).

A considerable number of studies utilized the firm (rather than the individual employee) as the unit of analysis. Such studies focused on how the implementation of eOL can increase employee performance, organizational value, and customer value. Although this is extremely helpful in furthering knowledge about eOL technologies and practices, a more granular investigation of the different e-learning systems and processes to address the various goals and strategies of the organization would enable researchers to extract practical insights on the design and implementation of eOL.

Research Agenda

By conducting an SLR and documenting the eOL research of the last decade, we have identified promising themes of research that have the potential to further eOL research and practice. To do so, we define a research agenda consisting of five thematic areas of research, as depicted in the research framework in Fig. 5 , and we provide some suggestions on how researchers could approach these challenges. In this visualization of the framework, on the left side we present the organizations as they were identified from our review (i.e., area/topic category in the Appendix ) and the multiple levels where organizational learning occurs (Costello and McNaughton 2018 ). On the right side, we summarize the objectives as they were identified from our review (i.e., the objectives category in the Appendix ). In the middle, we depict the orchestration that was conducted and how potential future research on eOL can improve the orchestration of the various elements and accelerate the achievement of the intended objectives. In particular, our proposed research agenda includes five research themes discussed in the following subsections.

An external file that holds a picture, illustration, etc.
Object name is 10796_2020_10097_Fig5_HTML.jpg

E-learning capabilities to enhance organizational research agenda

Theme 1: Couple E-learning Capabilities With the Intended Goals

The majority of the eOL studies either investigated a generic e-learning system using the umbrella term “e-learning” or did not provide enough details about the functionalities of the system (in most cases, it was simply defined as an online or web system). This indicates the very limited focus of the eOL research on the various capabilities of e-learning systems. In other words, the literature has been very detailed on the organizational value and employees’ acceptance of the technology, but less detailed on the capabilities of this technology that needs to be put into place to achieve the intended goals and strategic agenda. However, the capabilities of the e-learning systems and their use are not one-size-fits-all, and the intended goals (to obtain certain skills and competences) and employees’ needs and backgrounds play a determining role in the selection of the e-learning system (Al-Fraihat et al. 2020 ).

Only in a very few studies (Mueller et al. 2011 ; Renner et al. 2020 ) were the capabilities of the e-learning solutions (e.g., mobile learning, VR) utilized, and the results were found to significantly contribute to the intended goals. The intended knowledge can be procedural, declarative, general competence (e.g., presentation, communication, or leadership skills) or else, and its particularities and the pedagogical needs of the intended knowledge (e.g., a need for summative/formative feedback or for social learning support) should guide the selection of the e-learning system and the respective capabilities. Therefore, future research needs to investigate how the various capabilities offered by contemporary learning systems (e.g., assessment mechanisms, social learning, collaborative learning, personalized learning) can be utilized to adequately reinforce the intended goals (e.g., to train personnel to use a new tool, to improve presentation skills).

Theme 2: Embrace the Particularities of the Various Industries

Organizational learning entails sharing knowledge and enabling opportunities for growth at the individual, group, team, and organizational levels. Contemporary e-learning systems provide the medium to substantiate the necessary knowledge flow within organizations and to support employees’ overall learning. From the selected studies, we can infer that eOL research is either conducted in an industry-agnostic context (either generic or it was not properly reported) or there is a focus on the IT industry (see Appendix ). However, when looking at the few studies that provide results from different industries (Garavan et al. 2019 ; Lee et al. 2014 ), companies indicate that there are different practices, processes, and expectations, and that employees have different needs and perceptions with regards to e-learning systems and eOL in general. Such particularities influence the perceived dimensions of a learning organization. Some industries noted that eOL promoted the development of their learning organizations, whereas others reported that eOL did not seem to contribute to their development as a learning organization (Yoo and Huang 2016 ). Therefore, it is important that the implementation of organizational learning embraces the particularities of the various industries and future research needs to identify how the industry-specific characteristics can inform the design and development of organizational learning in promoting an organization’s goals and agenda.

Theme 3: Utilize E-learning Capabilities to Implement Employee-centric Approaches

For efficient organizational learning to be implemented, the processes and technologies need to recognize that learning is linked by social and psychological processes (Crossan et al. 1999 ). This allows employees to develop learning in various forms (e.g., social, emotional, personalized) and to develop elements such as self-awareness, self-control, and interpersonal skills that are vital for the organization. Looking at the contemporary eOL research, we notice that the exploration of e-learning capabilities to nurture the aforementioned elements and support employee-centric approaches is very limited (e.g., personalized technologies, adaptive assessment). Therefore, future research needs to collect data to understand how e-learning capabilities can be utilized in relation to employees’ needs and perceptions in order to provide solutions (e.g., collaborative, social, adaptive) that are employee-centric and focused on development, and that have the potential to move away from standard one-size-fits-all e-learning solutions to personalized and customized systems and processes.

Theme 4: Employ Analytics-enabled eOL

There is a lot of emphasis on measuring, via various qualitative and quantitative metrics, the effectiveness of eOL implemented at different levels in organizations. However, most of these metrics come from surveys and interviews that capture employees’ and managers’ perceptions of various aspects of eOL (e.g., fostering of learning, organizational value, employees’ performance), and very few studies utilize analytics (Hung et al. 2015 ; Renner et al. 2020 ; Rober and Cooper 2011 ). Given how digital technologies, big data, and business analytics pave the way towards organizations’ digital transformation and sustainable development (Mikalef et al. 2018 ; Pappas et al. 2018 ), and considering the learning analytics affordances of contemporary e-learning systems (Siemens and Long 2011 ), future work needs to investigate how learner/employee-generated data can be employed to inform practice and devise more accurate and temporal effectiveness metrics when measuring the importance and impact of eOL.

Theme 5: Orchestrate the Employees’ Needs, Resources, and Objectives in eOL Implementation

While considerable effort has been directed towards the various building blocks of eOL implementation, such as resources (intangible, tangible, and human skills) and employees’ needs (e.g., vision, growth, skills development), little is known so far about the processes and structures necessary for orchestrating those elements in order to achieve an organization’s intended goals and to materialize its overall agenda. In other words, eOL research has been very detailed on some of the elements that constitute efficient eOL, but less so on the interplay of those elements and how they need to be put into place. Prior literature on strategic resource planning has shown that competence in orchestrating such elements is a prerequisite to successfully increasing business value (Wang et al. 2012 ). Therefore, future research should not only investigate each of these elements in silos, but also consider their interplay, since it is likely that organizations with similar resources will exert highly varied levels in each of these elements (e.g., analytics-enabled, e-learning capabilities) to successfully materialize their goals (e.g., increase value, improve the competence base of their employees, modernize their organization).

Implications

Several implications for eOL have been revealed in this literature review. First, most studies agree that employees’ or trainees’ experience is extremely important for the successful implementation of eOL. Thus, keeping them in the design and implementation cycle of eOL will increase eOL adoption and satisfaction as well as reduce the risks and barriers. Another important implication addressed by some studies relates to the capabilities of the e-learning technologies, with easy-to-use, useful, and social technologies resulting in more efficient eOL (e.g., higher adoption and performance). Thus, it is important for organizations to incorporate these functionalities in the platform and reinforce them with appropriate content and support. This should not only benefit learning outcomes, but also provide the networking opportunities for employees to broaden their personal networks, which are often lost when companies move from face-to-face formal training to e-learning-enabled organizational learning.

Limitations

This review has some limitations. First, we had to make some methodological decisions (e.g., selection of databases, the search query) that might lead to certain biases in the results. However, tried to avoid such biases by considering all the major databases and following the steps indicated by Kitchenham and Charters ( 2007 ). Second, the selection of empirical studies and coding of the papers might pose another possible bias. However, the focus was clearly on the empirical evidence, the terminology employed (“e-learning”) is an umbrella term that covers the majority of the work in the area, and the coding of papers was checked by two researchers. Third, some elements of the papers were not described accurately, leading to some missing information in the coding of the papers. However, the amount of missing information was very small and could not affect the results significantly. Finally, we acknowledge that the selected methodology (Kitchenham and Charters 2007 ) includes potential biases (e.g., false negatives and false positives), and that different, equally valid methods (e.g., Okoli and Schabram 2010 ) might have been used and have resulted in slightly different outcomes. Nevertheless, despite the limitations of the selected methodology, it is a well-accepted and widely used literature review method in both software engineering and information systems (Boell and Cecez-Kecmanovic 2014 ), providing certain assurance of the results.

Conclusions and Future Work

We have presented an SLR of 47 contributions in the field of eOL over the last decade. With respect to RQ1, we analyzed the papers from different perspectives, such as research methodology, technology, industries, employees, and intended outcomes in terms of organizational value, employees’ performance, usage, and behavioral change. The detailed landscape is depicted in the Appendix and Figs. 3 and and4; 4 ; with the results indicating the limited utilization of the various e-learning capabilities (e.g., social, collaborative) to achieve objectives connected with those capabilities (e.g., social learning and behavioral change, collaborative learning and overcoming barriers).

With respect to RQ2, we categorized the main findings of the selected papers into five areas that reflect the status of eOL research, and we have discussed the challenges and opportunities emerging from the current review. In addition, we have synthesized the extracted challenges and opportunities and proposed a research agenda consisting of five elements that provide suggestions on how researchers could approach these challenges and exploit the opportunities. Such an agenda will strengthen how e-learning can be leveraged to enhance the process of improving actions through better knowledge and understanding in an organization.

A number of suggestions for further research have emerged from reviewing prior and ongoing work on eOL. One recommendation for future researchers is to clearly describe the eOL approach by providing detailed information about the technologies and materials used, as well as the organizations. This will allow meta-analyses to be conducted and it will also identify the potential effects of a firm’s size or area on the performance and other aspects relating to organizational value. Future work should also focus on collecting and triangulating different types of data from different sources (e.g., systems’ logs). The reviewed studies were conducted mainly by using survey data, and they made limited use of data coming from the platforms; thus, the interpretations and triangulation between the different types of collected data were limited.

Biographies

is a Professor of Interaction Design and Learning Technologies at the Department of Computer Science of NTNU, and Head of the Learner-Computer Interaction lab (https://lci.idi.ntnu.no/). His research focuses on the design and study of emerging technologies in online and hybrid education settings, and their connections to student and instructor experiences and practices. Giannakos has co-authored more than 150 manuscripts published in peer-reviewed journals and conferences (including Computers & Education, Computers in Human Behavior, IEEE TLT, Behaviour & Information Technology, BJET, ACM TOCE, CSCL, Interact, C&C, IDC to mention few) and has served as an evaluator for the EC and the US-NSF. He has served/serves in various organization committees (e.g., general chair, associate chair), program committees as well as editor and guest editor on highly recognized journals (e.g., BJET, Computers in Human Behavior, IEEE TOE, IEEE TLT, ACM TOCE). He has worked at several research projects funded by diverse sources like the EC, Microsoft Research, The Research Council of Norway (RCN), US-NSF, the German agency for international academic cooperation (DAAD) and Cheng Endowment; Giannakos is also a recipient of a Marie Curie/ERCIM fellowship, the Norwegian Young Research Talent award and he is one of the outstanding academic fellows of NTNU (2017-2021).

is an Associate Professor in Data Science and Information Systems at the Department of Computer Science. In the past, he has been a Marie Skłodowska-Curie post-doctoral research fellow working on the research project “Competitive Advantage for the Data-driven Enterprise” (CADENT). He received his B.Sc. in Informatics from the Ionian University, his M.Sc. in Business Informatics for Utrecht University, and his Ph.D. in IT Strategy from the Ionian University. His research interests focus the on strategic use of information systems and IT-business value in turbulent environments. He has published work in international conferences and peer-reviewed journals including the Journal of Business Research, British Journal of Management, Information and Management, Industrial Management & Data Systems, and Information Systems and e-Business Management.

Ilias O. Pappasis

an Associate Professor of Information Systems at the Department of Information Systems, University of Agder (UiA), Norway. His research and teaching activities include data science and digital transformation, social innovation and social change, user experience in different contexts,as well as digital marketing, e-services, and information technology adoption. He has published articles in peer reviewed journals and conferences including Journal of Business Research, European Journal of Marketing, Computers in Human Behavior, Information & Management, Psychology & Marketing, International Journal of Information Management, Journal of Systems and Software. Pappas has been a Guest Editor for the journals Information & Management, Technological Forecasting and Social Change, Information Systems Frontiers, Information Technology & People, and Information Systems and e-Business Management. Pappas is a recipient of ERCIM and Marie Skłodowska-Curiefellowships.

Survey = survey study; Exp. = experiment; CaseSt = case study; ND = non-defined; MGM = management; Telec. = telecommunication; Bsn = business; Univ. = university; Cons. = consulting; Public = public sector; Ent. = enterprise; Web = Web-based; KRS = knowledge repository system; OERs = open educational resources; SL = Second Life, Mg, = managers; Empl = employees; Stud = students; Res. = researchers; Learn. = learning specialists; Individ. = individual; Surv. = surveys; Int. = interviews; FG = focus groups; Log = log files; Obs. = observations; Reg. = regression analysis; Descr. = descriptive statistics; A-VA = analysis of variances/covariance; CA = content analysis; ItU = intention to use; Sat. = satisfaction; OV = organizational value; Per. = performance; Flearn = foster learning; Benef. = benefits; Align. = alignment; Feas. = feasibility; Barr. = barriers; Beh. = behavioral change

Open Access funding provided by NTNU Norwegian University of Science and Technology (incl St. Olavs Hospital - Trondheim University Hospital).

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Michail N. Giannakos, Email: on.untn@gliahcim .

Patrick Mikalef, Email: [email protected] .

Ilias O. Pappas, Email: [email protected] .

Al-Fraihat D, Joy M, Sinclair J. Evaluating E-learning systems success: An empirical study. Computers in Human Behavior. 2020; 102 :67–86. doi: 10.1016/j.chb.2019.08.004. [ CrossRef ] [ Google Scholar ]
Alharthi AD, Spichkova M, Hamilton M. Sustainability requirements for eLearning systems: A systematic literature review and analysis. Requirements Engineering. 2019; 24 (4):523–543. doi: 10.1007/s00766-018-0299-9. [ CrossRef ] [ Google Scholar ]
Alsabawy AY, Cater-Steel A, Soar J. IT infrastructure services as a requirement for e-learning system success. Computers & Education. 2013; 69 :431–451. doi: 10.1016/j.compedu.2013.07.035. [ CrossRef ] [ Google Scholar ]
Antonacopoulou E, Chiva R. The social complexity of organizational learning: The dynamics of learning and organizing. Management Learning. 2007; 38 :277–295. doi: 10.1177/1350507607079029. [ CrossRef ] [ Google Scholar ]
Berends H, Lammers I. Explaining discontinuity in organizational learning: A process analysis. Organization Studies. 2010; 31 (8):1045–1068. doi: 10.1177/0170840610376140. [ CrossRef ] [ Google Scholar ]
Boell SK, Cecez-Kecmanovic D. A hermeneutic approach for conducting literature reviews and literature searches. Communications of the Association for Information Systems. 2014; 34 (1):12. [ Google Scholar ]
Bologa R, Lupu AR. Organizational learning networks that can increase the productivity of IT consulting companies. A case study for ERP consultants. Expert Systems with Applications. 2014; 41 (1):126–136. doi: 10.1016/j.eswa.2013.07.016. [ CrossRef ] [ Google Scholar ]
Broadbent J. Comparing online and blended learner’s self-regulated learning strategies and academic performance. The Internet and Higher Education. 2017; 33 :24–32. doi: 10.1016/j.iheduc.2017.01.004. [ CrossRef ] [ Google Scholar ]
Cheng B, Wang M, Moormann J, Olaniran BA, Chen NS. The effects of organizational learning environment factors on e-learning acceptance. Computers & Education. 2012; 58 (3):885–899. doi: 10.1016/j.compedu.2011.10.014. [ CrossRef ] [ Google Scholar ]
Cheng B, Wang M, Yang SJ, Peng J. Acceptance of competency-based workplace e-learning systems: Effects of individual and peer learning support. Computers & Education. 2011; 57 (1):1317–1333. doi: 10.1016/j.compedu.2011.01.018. [ CrossRef ] [ Google Scholar ]
Choi S, Ko I. Leveraging electronic collaboration to promote interorganizational learning. International Journal of Information Management. 2012; 32 (6):550–559. doi: 10.1016/j.ijinfomgt.2012.03.002. [ CrossRef ] [ Google Scholar ]
Costello JT, McNaughton RB. Integrating a dynamic capabilities framework into workplace e-learning process evaluations. Knowledge and Process Management. 2018; 25 (2):108–125. doi: 10.1002/kpm.1565. [ CrossRef ] [ Google Scholar ]
Crossan MM, Lane HW, White RE. An organizational learning framework: From intuition to institution. Academy of Management Review. 1999; 24 :522–537. doi: 10.2307/259140. [ CrossRef ] [ Google Scholar ]
Crossan MM, Maurer CC, White RE. Reflections on the 2009 AMR decade award: Do we have a theory of organizational learning? Academy of Management Review. 2011; 36 (3):446–460. doi: 10.5465/amr.2010.0544. [ CrossRef ] [ Google Scholar ]
Dee, J., & Leisyte, L. (2017). Knowledge sharing and organizational change in higher education. The Learning Organization, 24 (5), 355–365. 10.1108/TLO-04-2017-0034
Dignen B, Burmeister T. Three pillars of organization and leadership in disruptive times. Cham: Springer; 2020. Learning and development in the organizations of the future; pp. 207–232. [ Google Scholar ]
Dybå T, Dingsøyr T. Empirical studies of agile software development: A systematic review. Information and Software Technology. 2008; 50 (9–10):833–859. doi: 10.1016/j.infsof.2008.01.006. [ CrossRef ] [ Google Scholar ]
El Kadiri S, Grabot B, Thoben KD, Hribernik K, Emmanouilidis C, Von Cieminski G, Kiritsis D. Current trends on ICT technologies for enterprise information systems. Computers in Industry. 2016; 79 :14–33. doi: 10.1016/j.compind.2015.06.008. [ CrossRef ] [ Google Scholar ]
Engeström Y, Kerosuo H, Kajamaa A. Beyond discontinuity: Expansive organizational learning remembered. Management Learning. 2007; 38 (3):319–336. doi: 10.1177/1350507607079032. [ CrossRef ] [ Google Scholar ]
Gal E, Nachmias R. Online learning and performance support in organizational environments using performance support platforms. Performance Improvement. 2011; 50 (8):25–32. doi: 10.1002/pfi.20238. [ CrossRef ] [ Google Scholar ]
Garavan TN, Heneghan S, O’Brien F, Gubbins C, Lai Y, Carbery R, Grant K. L&D professionals in organisations: much ambition, unfilled promise. European Journal of Training and Development. 2019; 44 (1):1–86. [ Google Scholar ]
Goggins SP, Jahnke I, Wulf V. Computer-supported collaborative learning at the workplace. New York: Springer; 2013. [ Google Scholar ]
Hammad, R., Odeh, M., & Khan, Z. (2017). ELCMM: An e-learning capability maturity model. In Proceedings of the 15th International Conference (e-Society 2017) (pp. 169–178).
Hester AJ, Hutchins HM, Burke-Smalley LA. Web 2.0 and transfer: Trainers’ use of technology to support employees’ learning transfer on the job. Performance Improvement Quarterly. 2016; 29 (3):231–255. doi: 10.1002/piq.21225. [ CrossRef ] [ Google Scholar ]
Hung YH, Lin CF, Chang RI. Developing a dynamic inference expert system to support individual learning at work. British Journal of Educational Technology. 2015; 46 (6):1378–1391. doi: 10.1111/bjet.12214. [ CrossRef ] [ Google Scholar ]
Iris R, Vikas A. E-Learning technologies: A key to dynamic capabilities. Computers in Human Behavior. 2011; 27 (5):1868–1874. doi: 10.1016/j.chb.2011.04.010. [ CrossRef ] [ Google Scholar ]
Jia H, Wang M, Ran W, Yang SJ, Liao J, Chiu DK. Design of a performance-oriented workplace e-learning system using ontology. Expert Systems with Applications. 2011; 38 (4):3372–3382. doi: 10.1016/j.eswa.2010.08.122. [ CrossRef ] [ Google Scholar ]
Joo YJ, Lim KY, Park SY. Investigating the structural relationships among organisational support, learning flow, learners’ satisfaction and learning transfer in corporate e-learning. British Journal of Educational Technology. 2011; 42 (6):973–984. doi: 10.1111/j.1467-8535.2010.01116.x. [ CrossRef ] [ Google Scholar ]
Kaschig, A., Maier, R., Sandow, A., Lazoi, M., Barnes, S. A., Bimrose, J., … Schmidt, A. (2010). Knowledge maturing activities and practices fostering organisational learning: results of an empirical study. In European Conference on Technology Enhanced Learning (pp. 151–166). Berlin: Springer.
Khalili A, Auer S, Tarasowa D, Ermilov I. International Conference on Knowledge Engineering and Knowledge Management. Berlin: Springer; 2012. SlideWiki: Elicitation and sharing of corporate knowledge using presentations; pp. 302–316. [ Google Scholar ]
Khandakar MSA, Pangil F. Relationship between human resource management practices and informal workplace learning. Journal of Workplace Learning. 2019; 31 (8):551–576. doi: 10.1108/JWL-04-2019-0049. [ CrossRef ] [ Google Scholar ]
Kim MK, Kim SM, Bilir MK. Investigation of the dimensions of workplace learning environments (WLEs): Development of the WLE measure. Performance Improvement Quarterly. 2014; 27 (2):35–57. doi: 10.1002/piq.21170. [ CrossRef ] [ Google Scholar ]
Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE-2007-01, 2007 . https://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=35909B1B280E2032BF116BDC9DCB71EA? .
Krippendorff, K. (2018). Content analysis: an introduction to its methodology. Thousand Oaks: Sage Publications.
Lai HJ. Examining civil servants’ decisions to use Web 2.0 tools for learning, based on the decomposed theory of planned behavior. Interactive Learning Environments. 2017; 25 (3):295–305. doi: 10.1080/10494820.2015.1121879. [ CrossRef ] [ Google Scholar ]
Lau K. Organizational learning goes virtual? A study of employees’ learning achievement in stereoscopic 3D virtual reality. The Learning Organization. 2015; 22 (5):289–303. doi: 10.1108/TLO-11-2014-0063. [ CrossRef ] [ Google Scholar ]
Lee J, Choi M, Lee H. Factors affecting smart learning adoption in workplaces: Comparing large enterprises and SMEs. Information Technology and Management. 2015; 16 (4):291–302. doi: 10.1007/s10799-014-0201-5. [ CrossRef ] [ Google Scholar ]
Lee J, Kim DW, Zo H. Conjoint analysis on preferences of HRD managers and employees for effective implementation of m-learning: The case of South Korea. Telematics and Informatics. 2015; 32 (4):940–948. doi: 10.1016/j.tele.2015.04.010. [ CrossRef ] [ Google Scholar ]
Lee J, Zo H, Lee H. Smart learning adoption in employees and HRD managers. British Journal of Educational Technology. 2014; 45 (6):1082–1096. doi: 10.1111/bjet.12210. [ CrossRef ] [ Google Scholar ]
Lin CH, Sanders K. HRM and innovation: A multi-level organizational learning perspective. Human Resource Management Journal. 2017; 27 (2):300–317. doi: 10.1111/1748-8583.12127. [ CrossRef ] [ Google Scholar ]
Lin CY, Huang CK, Zhang H. Enhancing employee job satisfaction via e-learning: The mediating role of an organizational learning culture. International Journal of Human–Computer Interaction. 2019; 35 (7):584–595. doi: 10.1080/10447318.2018.1480694. [ CrossRef ] [ Google Scholar ]
Liu YC, Huang YA, Lin C. Organizational factors’ effects on the success of e-learning systems and organizational benefits: An empirical study in Taiwan. The International Review of Research in Open and Distributed Learning. 2012; 13 (4):130–151. doi: 10.19173/irrodl.v13i4.1203. [ CrossRef ] [ Google Scholar ]
López-Nicolás C, Meroño-Cerdán ÁL. Strategic knowledge management, innovation and performance. International Journal of Information Management. 2011; 31 (6):502–509. doi: 10.1016/j.ijinfomgt.2011.02.003. [ CrossRef ] [ Google Scholar ]
Manuti A, Pastore S, Scardigno AF, Giancaspro ML, Morciano D. Formal and informal learning in the workplace: A research review. International Journal of Training and Development. 2015; 19 (1):1–17. doi: 10.1111/ijtd.12044. [ CrossRef ] [ Google Scholar ]
March JG. Exploration and exploitation in organizational learning. Organization Science. 1991; 2 (1):71–87. doi: 10.1287/orsc.2.1.71. [ CrossRef ] [ Google Scholar ]
Marshall S. New Zealand Tertiary Institution E-learning Capability: Informing and Guiding eLearning Architectural Change and Development. Report to the ministry of education. NZ: Victoria University of Wellington; 2006. [ Google Scholar ]
McDonald, N., Schoenebeck, S., & Forte, A. (2019). Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice. In Proceedings of the ACM on Human–Computer Interaction, 3(CSCW) (pp. 1–23).
Menolli, A., Tirone, H., Reinehr, S., & Malucelli, A. (2020). Identifying organisational learning needs: An approach to the semi-automatic creation of course structures for software companies. Behaviour & Information Technology, 39 (11), 1140–1155.
Michalski MP. Symbolic meanings and e-learning in the workplace: The case of an intranet-based training tool. Management Learning. 2014; 45 (2):145–166. doi: 10.1177/1350507612468419. [ CrossRef ] [ Google Scholar ]
Mikalef P, Pappas IO, Krogstie J, Giannakos M. Big data analytics capabilities: A systematic literature review and research agenda. Information Systems and e-Business Management. 2018; 16 (3):547–578. doi: 10.1007/s10257-017-0362-y. [ CrossRef ] [ Google Scholar ]
Mitić S, Nikolić M, Jankov J, Vukonjanski J, Terek E. The impact of information technologies on communication satisfaction and organizational learning in companies in Serbia. Computers in Human Behavior. 2017; 76 :87–101. doi: 10.1016/j.chb.2017.07.012. [ CrossRef ] [ Google Scholar ]
Mueller J, Hutter K, Fueller J, Matzler K. Virtual worlds as knowledge management platform—A practice-perspective. Information Systems Journal. 2011; 21 (6):479–501. doi: 10.1111/j.1365-2575.2010.00366.x. [ CrossRef ] [ Google Scholar ]
Muller Queiroz, A. C., Nascimento, M., Tori, A., Alejandro, R. Brashear, Veloso, T., de Melo, V., de Souza Meirelles, F., & da Silva Leme, M. I. (2018). Immersive virtual environments in corporate education and training. In AMCIS. https://aisel.aisnet.org/amcis2018/Education/Presentations/12/ .
Navimipour NJ, Zareie B. A model for assessing the impact of e-learning systems on employees’ satisfaction. Computers in Human Behavior. 2015; 53 :475–485. doi: 10.1016/j.chb.2015.07.026. [ CrossRef ] [ Google Scholar ]
Oh SY. Effects of organizational learning on performance: The moderating roles of trust in leaders and organizational justice. Journal of Knowledge Management. 2019; 23 :313–331. doi: 10.1108/JKM-02-2018-0087. [ CrossRef ] [ Google Scholar ]
Okoli, C., & Schabram, K. (2010). A guide to conducting a systematic literature review of information systems research. Sprouts: Working Papers on Information Systems, 10 (26), 1–46.
Pappas IO, Mikalef P, Giannakos MN, Krogstie J, Lekakos G. Big data and business analytics ecosystems: paving the way towards digital transformation and sustainable societies. Information Systems and e-Business Management. 2018; 16 :479–491. doi: 10.1007/s10257-018-0377-z. [ CrossRef ] [ Google Scholar ]
Popova-Nowak IV, Cseh M. The meaning of organizational learning: A meta-paradigm perspective. Human Resource Development Review. 2015; 14 (3):299–331. doi: 10.1177/1534484315596856. [ CrossRef ] [ Google Scholar ]
Qi, C., & Chau, P. Y. (2016). An empirical study of the effect of enterprise social media usage on organizational learning. In Pacific Asia Conference on Information Systems (PACIS'16). Proceedings , Paper 330. http://aisel.aisnet.org/pacis2016/330 .
Renner B, Wesiak G, Pammer-Schindler V, Prilla M, Müller L, Morosini D, Cress U. Computer-supported reflective learning: How apps can foster reflection at work. Behaviour & Information Technology. 2020; 39 (2):167–187. doi: 10.1080/0144929X.2019.1595726. [ CrossRef ] [ Google Scholar ]
Rober, M. B., & Cooper, L. P. (2011, January). Capturing knowledge via an” Intrapedia”: A case study. In 2011 44th Hawaii International Conference on System Sciences (pp. 1–10). New York: IEEE.
Rosenberg MJ, Foshay R. E-learning: Strategies for delivering knowledge in the digital age. Performance Improvement. 2002; 41 (5):50–51. doi: 10.1002/pfi.4140410512. [ CrossRef ] [ Google Scholar ]
Serrano, Á., Marchiori, E. J., del Blanco, Á., Torrente, J., & Fernández-Manjón, B. (2012). A framework to improve evaluation in educational games. The IEEE Global Engineering Education Conference (pp. 1–8). Marrakesh, Morocco.
Siadaty, M., Jovanović, J., Gašević, D., Jeremić, Z., & Holocher-Ertl, T. (2010). Leveraging semantic technologies for harmonization of individual and organizational learning. In European Conference on Technology Enhanced Learning (pp. 340–356). Berlin: Springer.
Siemens G, Long P. Penetrating the fog: Analytics in learning and education. EDUCAUSE Review. 2011; 46 (5):30. [ Google Scholar ]
Škerlavaj M, Dimovski V, Mrvar A, Pahor M. Intra-organizational learning networks within knowledge-intensive learning environments. Interactive Learning Environments. 2010; 18 (1):39–63. doi: 10.1080/10494820802190374. [ CrossRef ] [ Google Scholar ]
Smith PJ, Sadler-Smith E. Learning in organizations: Complexities and diversities. London: Routledge; 2006. [ Google Scholar ]
Stoffregen JD, Pawlowski JM, Ras E, Tobias E, Šćepanović S, Fitzpatrick D, Friedrich H. Barriers to open e-learning in public administrations: A comparative case study of the European countries Luxembourg, Germany, Montenegro and Ireland. Technological Forecasting and Social Change. 2016; 111 :198–208. doi: 10.1016/j.techfore.2016.06.030. [ CrossRef ] [ Google Scholar ]
Subramaniam, R., & Nakkeeran, S. (2019). Impact of corporate e-learning systems in enhancing the team performance in virtual software teams. In Smart Technologies and Innovation for a Sustainable Future (pp. 195–204). Berlin: Springer.
Tsai CH, Zhu DS, Ho BCT, Wu DD. The effect of reducing risk and improving personal motivation on the adoption of knowledge repository system. Technological Forecasting and Social Change. 2010; 77 (6):840–856. doi: 10.1016/j.techfore.2010.01.011. [ CrossRef ] [ Google Scholar ]
Turi JA, Sorooshian S, Javed Y. Impact of the cognitive learning factors on sustainable organizational development. Heliyon. 2019; 5 (9):e02398. doi: 10.1016/j.heliyon.2019.e02398. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
Wang, M. (2011). Integrating organizational, social, and individual perspectives in Web 2.0-based workplace e-learning. Information Systems Frontiers, 13 (2), 191–205.
Wang, M. (2018). Effects of individual and social learning support on employees’ acceptance of performance-oriented e-learning. In E-Learning in the Workplace (pp. 141–159). Springer. 10.1007/978-3-319-64532-2_13.
Wang M, Ran W, Liao J, Yang SJ. A performance-oriented approach to e-learning in the workplace. Journal of Educational Technology & Society. 2010; 13 (4):167–179. [ Google Scholar ]
Wang M, Vogel D, Ran W. Creating a performance-oriented e-learning environment: A design science approach. Information & Management. 2011; 48 (7):260–269. doi: 10.1016/j.im.2011.06.003. [ CrossRef ] [ Google Scholar ]
Wang N, Liang H, Zhong W, Xue Y, Xiao J. Resource structuring or capability building? An empirical study of the business value of information technology. Journal of Management Information Systems. 2012; 29 (2):325–367. doi: 10.2753/MIS0742-1222290211. [ CrossRef ] [ Google Scholar ]
Wang S, Wang H. Organizational schemata of e-portfolios for fostering higher-order thinking. Information Systems Frontiers. 2012; 14 (2):395–407. doi: 10.1007/s10796-010-9262-0. [ CrossRef ] [ Google Scholar ]
Wei K, Ram J. Perceived usefulness of podcasting in organizational learning: The role of information characteristics. Computers in Human Behavior. 2016; 64 :859–870. doi: 10.1016/j.chb.2016.08.003. [ CrossRef ] [ Google Scholar ]
Wei, K., Sun, H., & Li, H. (2013). On the driving forces of diffusion of podcasting in organizational settings: A case study and propositions. In PACIS 2013. Proceedings , 217. http://aisel.aisnet.org/pacis2013/217 .
Weinhardt, J. M., & Sitzmann, T. (2018). Revolutionizing training and education? Three questions regarding massive open online courses (MOOCs). Human Resource Management Review, 29 (2), 218–225.
Xiang, Q., Zhang, J., & Liu, H. (2020). Organisational improvisation as a path to new opportunity identification for incumbent firms: An organisational learning view. Innovation, 22 (4), 422–446. 10.1080/14479338.2020.1713001.
Yanson R, Johnson RD. An empirical examination of e-learning design: The role of trainee socialization and complexity in short term training. Computers & Education. 2016; 101 :43–54. doi: 10.1016/j.compedu.2016.05.010. [ CrossRef ] [ Google Scholar ]
Yoo SJ, Huang WD. Can e-learning system enhance learning culture in the workplace? A comparison among companies in South Korea. British Journal of Educational Technology. 2016; 47 (4):575–591. doi: 10.1111/bjet.12240. [ CrossRef ] [ Google Scholar ]
Zhang X, Jiang S, Ordóñez de Pablos P, Lytras MD, Sun Y. How virtual reality affects perceived learning effectiveness: A task–technology fit perspective. Behaviour & Information Technology. 2017; 36 (5):548–556. doi: 10.1080/0144929X.2016.1268647. [ CrossRef ] [ Google Scholar ]
Zhang X, Meng Y, de Pablos PO, Sun Y. Learning analytics in collaborative learning supported by Slack: From the perspective of engagement. Computers in Human Behavior. 2019; 92 :625–633. doi: 10.1016/j.chb.2017.08.012. [ CrossRef ] [ Google Scholar ]

Organizational unlearning as a process: What we know, what we don’t know, what we should know

Open access
Published: 23 April 2024

Cite this article

You have full access to this open access article

Adrian Klammer ORCID: orcid.org/0000-0001-9665-0419 1 ,
Thomas Grisold 2 ,
Nhien Nguyen 3 &
Shih-wei Hsu 4

Although the field of organizational unlearning has recently gained increased interest, its conceptual foundations and raison d’être are still debated. In this review, we aim to revisit various discourses and arguments to advance the understanding of organizational unlearning in management and organization studies. Using an integrative literature review approach with systematic elements, we examine the existing body of research on organizational unlearning. We review the literature from different perspectives, focusing on a process-based understanding in terms of why and how organizations intentionally discard knowledge. Based on our review, we develop an integrative framework that portrays organizational unlearning as a dynamically unfolding process over time. We propose implications and offer research directions that will allow future researchers to develop a more profound understanding of the concept.

Avoid common mistakes on your manuscript.

1 Introduction

Organizational unlearning implies that organizations intentionally and deliberately discard undesired, obsolete, or harmful knowledge—often to make room for the creation of new knowledge (Tsang and Zahra 2008 ). To this end, organizational unlearning can target different knowledge structures, such as systems, routines, basic assumptions, values, or norms. Moreover, it can occur in various contexts, such as innovation (Wang et al. 2013 ; Yang et al. 2014 ; Açıkgöz et al. 2021 ), mergers and acquisitions (Tsang 2008 ; Wang et al. 2017 ), organizational change (Grisold et al. 2020 ), and social care (Brook et al. 2016 ), among others.

Despite the considerable uptake of organizational unlearning in research, the concept has sparked controversy, primarily owing to its conceptual ambiguities (see Klein 1989 ; Martin de Holan 2011b ; Howells and Scholderer 2016 ; Tsang 2017a , b ; 1989 ); Klammer et al. 2019b ). Along these lines, it has been argued that the term organizational unlearning conveys the impression that knowledge can be eliminated from organizations, essentially insinuating that targeted knowledge structures can be objectified and selectively erased (Turc and Baumard 2007 ; Howells and Scholderer 2016 ; Grisold et al. 2017 ). The main objection to these claims is that a large share of organizational knowledge is embedded in mental models, practices, and routines, which cannot be removed or taken out in any literal sense (e.g., Cowan et al. 2000 ; Tsoukas and Vladimirou 2001 ).

In response to these claims, emerging arguments emphasize that organizational unlearning should be understood as a process (e.g., Fiol and O’Connor 2017a , b ; Grisold et al. 2017 ; Kluge and Gronau 2018 ; Peschl 2019 ; Burton et al. 2023 ). These arguments depart from the observation that organizational knowledge is deeply embedded in collective beliefs and routines. If some of these knowledge structures are to be unlearned, one has to focus on how they become less dominant over time . In other words, from a process-based perspective, organizational unlearning implies that organizational actors gradually reduce the influence of unwanted or harmful knowledge structures by blocking or preventing their enactment (Grisold et al. 2017 ; Kluge and Gronau 2018 ). As this process progresses, old knowledge becomes less likely to be used (and new knowledge, if any, becomes more likely to be used).

Such process-based views of unlearning evoke considerable interest in the field. They not only resonate with perspectives from other fields, such as psychology and cognitive sciences (e.g., Kluge and Gronau 2018 ; Peschl 2019 ; Haase et al. 2020 ), but also inform practical interventions to enable or support unlearning initiatives (Klammer et al. 2019a ; Grisold et al. 2020 ). However, we lack a systematic understanding of what we know about the process behind organizational unlearning. Some open questions include the following: what does this process imply? How does it evolve? Why and when does it succeed or fail?

Existing reviews of organizational unlearning (e.g., Tsang and Zahra 2008 ; Klammer and Gueldenberg 2019 ; Sharma and Lenka 2022a , b ) highlight various important aspects, but do not establish a process-based understanding of organizational unlearning. Hence, in this review, we pursue the following questions: what do we know about the process of organizational unlearning, and how can we synthesize existing perspectives? To answer these questions, we develop a multi-perspective and integrative view to explain how organizational unlearning evolves over time.

2 Review approach

We followed an integrative review approach, including systematic elements, to search for relevant literature. Due to the field’s fragmented understanding, we deem it necessary and suitable to bring different perspectives together to surface the nature of the concept, develop implications, and provide avenues for future research. This procedure is motivated by the observation that organizational unlearning is discussed within the broader realm of management and organization studies (MOS), but its conceptual assumptions and conversation topics remain within rather insulated communities in specific sub-fields, thereby fostering and reproducing different perspectives on the same concept.

We (the authors) ascribe ourselves as researchers in the broader field of MOS, although each of us has researched organizational unlearning from a different perspective, based on different scholarly communities. This enabled us to adopt different perspectives to examine the same phenomenon. We initially engaged in several rounds of discussion and sensemaking to establish our position and define the scope of our review (Cronin and George 2023 ). In the time between these discussions, each author conducted initial, non-systematic searches (Rojon et al. 2021 ) to bring in different perspectives. We then established our final position that organizational unlearning is a processual phenomenon warranting attention to the antecedents, outcomes, and dynamics of intentionally discarding undesired or outdated knowledge from organizations.

After establishing our position, we applied various systematic steps to build the foundation for our review (Tranfield et al. 2003 ). We searched for literature on organizational unlearning written in English from 1981 (Hedberg’s book chapter as the starting point) to February 2024. Using the keywords [organization* AND unlearn*], we conducted a title and abstract search in Web of Science, EBSCOhost (Business Source Premier), ProQuest (ABI/INFORM), and Elsevier (ScienceDirect) databases (n = 1104). Next, we merged all results from the databases into a list and, subsequently, deleted duplicate results (n = 759). In an initial review, we read all titles and abstracts and applied two specific criteria to exclude false positives. First, we removed literature from research fields that have no connection to the broader domain of MOS (e.g., clinical psychology). Second, we excluded studies that only serendipitously mentioned the term unlearning in the title or abstract (n = 246). Next, we screened and assessed the remaining full texts. At this stage, we identified literature that fell outside our scope. In doing so, we eliminated non-substantive works that use the term “unlearning” in the title or abstract, while not thoroughly addressing or discussing the phenomenon in the remainder of the paper (n = 88).

As an important additional step, we added an integrative dimension to maximize the comprehensiveness of our review. We conducted hand-searching, snow-balling, and citation-tracking to identify relevant literature that did not fit our search criteria and might have been missed (cf. Trullen et al. 2020 ). Additionally, we integrated literature from our respective communities to acquire different perspectives (cf. Cronin and George 2023 ). This approach allowed us to incorporate relevant literature beyond our initial, systematic search strings. In doing so, we illustrated that some works examine, at their core, intentional loss of knowledge in the context of MOS, without actually using the term unlearning (e.g., Polites and Karahanna 2012 ; Pentland et al. 2020 ), but are deemed useful to further the understanding of the phenomenon (n = 124) (Fig. 1 ).

Overview of the search process

We analyzed and synthesized the final sample using an Excel data extraction template to elicit both quantitative (e.g., authors, publication information) and qualitative (e.g., methodology, findings) information. In terms of the content, we identified relevant perspectives that previous researchers have used to empirically investigate and theorize about organizational unlearning, and which are relevant to examining organizational unlearning as a process.

3.1 Organizational unlearning as a process: Definitions and viewpoints

The concept’s raison d’être has been discussed from various perspectives. Starbuck (in: Nguyen 2017 ) explains the origins of unlearning as an organizational phenomenon in MOS. Hedberg and Starbuck observe that organizations find it difficult to adapt to crises and changing environments; they face failure, reluctance, or hesitancy to unlearn (e.g., Hedberg et al. 1976 ; Starbuck et al. 1978 ; Nystrom and Starbuck 1984 ; Starbuck 1996 ). While some assert that unlearning is subsumable under organizational learning (Huber 1991 ), or argue for its inclusion in the wider context of learning dynamics (Visser 2017 ), others recognize the merits of treating organizational unlearning as a distinct, isolated, and stand-alone phenomenon (Tsang 2017a , b ; Becker 2019 ).

While terms, such as knowledge, dominant logics, or routines are loosely used to describe what organizational unlearning entails, existing studies fall short of clearly defining the kinds of knowledge structures being investigated, respectively unlearned. We found that cognitive and behavioral knowledge structures are two of the most widely used perspectives for pinpointing the locus of the unlearning process (Akgün et al. 2007b ; Tsang and Zahra 2008 ). While the cognitive perspective describes how unlearning helps discard knowledge that has been collectively interpreted, the behavioral perspective refers to how routines, habits, or procedures are collectively abandoned (Easterby-Smith and Lyles 2011 ). The collective lens of shared beliefs and assumptions is thought to be a vital part of the unlearning process (Turc and Baumard 2007 ). Sinkula ( 2002 ) suggests that organizational unlearning starts with changing cognitive structures, mental models, dominant logics, and other core assumptions that guide behavior. In turn, organizations can destabilize and eliminate behaviors, such as routines, habits, or procedures (Martin de Holan and Phillips 2004b ; Fiol and O’Connor 2017a , b ).

Visser ( 2017 ) highlights the interplay of complex social processes as organizational unlearning necessitates individuals to let go of part of their identities as enacted practices are strongly connected to social identities (McKeown 2012 ). In addition, unlearning has also been explored from emotional (Pratt and Barnett 1997 ; Rushmer and Davies 2004 ) and normative perspectives (Yildiz and Fey 2010 ). Hence, organizational unlearning is a multi-faceted term yielding multiple associations regarding the dynamics of knowledge loss.

3.2 Organizational unlearning mechanisms and conceptualizations

Several studies aim to shed light on different mechanisms of unlearning explaining how organizations discard existing knowledge. Bowker ( 1997 ), for example, distinguishes between clearance and erasure of organizational knowledge. Similarly, unlearning has been described as the process by which organizational members gradually refrain from enacting existing routines over time by removing cues (Kluge and Gronau 2018 ). Organizations might unlearn through tailored interventions, such as inactivating specific knowledge structures or rivaling enforced enactment (Turc and Baumard 2007 ).

Several quantitative empirical studies investigate the mechanisms of organizational unlearning. For example, the “unlearning context,” introduced by Cegarra-Navarro and Sánchez-Polo ( 2008 ) includes sequential unlearning steps from the individual to the organizational level. This model has been widely used in other studies (e.g., Cegarra-Navarro et al. 2010 , 2011a , b , 2013 , 2014 , 2016 , 2021 ; Cepeda-Carrion et al. 2012a , b ; Cegarra-Navarro and Cepeda-Carrion 2013 ; Ortega-Gutiérrez et al. 2015 , 2022 ; Wensley and Cegarra-Navarro 2015 ; Cegarra-Navarro and Wensley 2019 ; Lyu et al. 2022 ). Akgün et al. ( 2006 , 2007a , b ) operationalize unlearning as changes in beliefs and routines, a conceptualization that has been used in several other studies (e.g., Wang et al. 2013 , 2017 ; Yang et al. 2014 ; Xi et al. 2020 ; Zhao and Wang 2020 ).

Qualitative empirical studies paint a more fine-grained picture of unlearning mechanisms in organizations. Mechanisms to facilitate organizational unlearning might vary, depending on the timing of their occurrence or the desired outcomes of the process (e.g., Grisold et al. 2020 ; Xu et al. 2023 ). Rezazade Mehrizi and Lashkarbolouki ( 2016 ) outline the cognitive and behavioral dynamics of organizational unlearning when discarding troubled business models including the stages of realizing, revitalizing, parallelizing, and marginalizing. Similarly, Tsang ( 2008 ) finds organizational unlearning mechanisms at different stages of knowledge transfer to acquisition joint ventures. Stage-driven process models are often found in practitioner articles that typically provide advice on how managers can help their organizations unlearn as they follow a sequence of steps (Reese 2017 ; Klammer et al. 2019a ; Govindarajan et al. 2020 , 2021 ).

Another way to unpack organizational unlearning mechanisms is to sketch its recursive nature. The key assumption here is that unlearning is a fragile and highly dynamic process wherein discarding and learning activities unfold interchangeably (Nygren et al. 2017 ), or sometimes occur simultaneously (Fiol and O’Connor 2017a , b ). Organizational unlearning cycles (Pratt and Barnett 1997 ; Cegarra-Navarro and Wensley 2019 ; Hamza-Orlinska et al. 2024 ) or spirals (Macdonald 2002 ; Grisold and Kaiser 2017 ) provide additional insights into the recursive nature of the process. Peschl ( 2019 ) argues that the exact process of unlearning cannot be defined; embracing an unknown future means to embark on an uncertain and emergent process.

Further, we identified studies that relate organizational unlearning to learning and relearning, often contextualized in sequential learning-unlearning-relearning steps (e.g., Azmi 2008 ; Rupcic 2019 ; Sharma and Lenka 2019 ; Zhao and Wang 2020 ). This idea stresses that unlearning occurs in relation to existing knowledge (prior learning) and relearning (new learning of knowledge). From this viewpoint, new learning cannot be acquired before established knowledge has been removed. Existing views on mechanisms and conceptualizations share the commonality that organizational unlearning is a process characterized by context-specific dynamics in terms of discarding and/or acquiring knowledge.

3.3 Levels of unlearning

We found different views regarding the levels as well as their interdependence and interplay during unlearning processes. Generally, unlearning is portrayed as an organizational phenomenon that helps describe learning, adaptation, and change, or how firms deal with crises (Nguyen 2017 ; Vu and Nguyen 2022 ). Researching organizational unlearning, however, also requires an understanding of individuals and groups, as organizations do not have cognitive capabilities per se (Hedberg 1981 ; Brooks et al. 2022 ).

For example, awareness and relinquishing capabilities are strongly connected to intentional knowledge loss of individuals (Becker 2008 , 2010 ). Individual unlearning can also be described as a transformative journey of discernment including receptivity, recognition, and grieving (Macdonald 2002 ). Further, individual unlearning in organizational contexts has been typologized into routine unlearning, wiping, and deep unlearning depending on the depth of the discarding process (Rushmer and Davies 2004 ; Hislop et al. 2014 ).

A conceptual attempt to explain the interplay between different levels suggests that individual unlearning first promotes group and, subsequently, organizational unlearning, or vice versa (Zhao et al. 2013 ). We identified two viewpoints on how unlearning transfers across levels: top-down and bottom-up. The idea of unlearning as a top-down activity refers to instances wherein organizational decision-makers introduce changes that require individuals to discard existing assumptions, mental models, behaviors, or routines (e.g., Nystrom and Starbuck 1984 ; Martin de Holan et al. 2004 ; Martin de Holan and Phillips 2004a ; Nguyen 2017 ; Grisold et al. 2020 ; Klammer 2021 ). On the other hand, unlearning as a bottom-up activity describes the effects of individuals’ decisions to discard existing knowledge structures of an organization (e.g., Becker 2008 , 2010 ; Hislop et al. 2014 ; Matsuo 2019a ). Additionally, we found studies that specifically deal with the individual level (Tanaka 2023 ; Yin 2023 ) or group levels (e.g., Akgün et al. 2006 , 2007a ; Klammer and Gueldenberg 2020 ; Açıkgöz et al. 2021 ). The process of organizational unlearning can differ significantly, depending on whether and how unlearning unfolds within or between different organizational levels and entities over time.

3.4 Timing of organizational unlearning

Existing research highlights how the process of unlearning depends on timing-related decisions. To ensure strategic resilience in a world of turbulence and uncertainty, organizations should take action before it is desperately needed, thus unlearning should be a proactive process (Morais-Storz and Nguyen 2017 ). Managers should be able to identify early warning signs of an inflection point, that is, a shift in the external environment causing change that alters the basic assumptions upon which a business is built (McGrath 2019 ; Sharma and Lenka 2024 ). An early warning system may help identify and unlearn basic assumptions that are no longer applicable (McGrath and Euchner 2020 ).

Numerous studies indicate, however, that this approach can be challenging. First, it is difficult to anticipate the exact timing of environmental change (Martignoni and Keil 2021 ) to initiate the process of organizational unlearning. Second, organizations might find it difficult to find and adopt new operating methods because they have become firmly dependent on past methods (Starbuck 2017 ; Snihur 2018 ) and might be stuck in competence traps due to inertia arising from prior success (Leonard-Barton 1992 ). Third, it is not easy to tell whether companies render an old belief obsolete (Nguyen 2017 ), because it can often only be known retrospectively if an organization’s belief has become obsolete and, therefore, should have been discarded (Martignoni and Keil 2021 ). Fourth, unlearning requires a collective decision-making process, challenged by specialized personnel, who see their careers as tied to existing strategies and their core beliefs (Starbuck 2017 ).

We found two conflicting paradigms regarding the timing of organizational unlearning: (i) the reactive paradigm, which suggests that unlearning can only take place after noticeable failures or major interruptions, and (ii) the proactive paradigm, which implies that unlearning should occur prior to inflection points. We observed that many empirical studies empirically investigate organizational unlearning from the perspective of the reactive paradigm. For example, organizations tend to introduce technical and organizational change only after the occurrence of catastrophic failures, as in the case of NASA during the Challenger disaster (Starbuck and Milliken 1988 ). Conversely, only very few studies investigate proactive unlearning approaches at the organizational level. For example, Burt and Nair ( 2020 ) investigate how an organization proactively discards deeply held assumptions about its business logic, and thus initiates strategic change. Hence, the point of initiating the purposeful discarding of knowledge seems vital to navigating unlearning processes in organizations.

3.5 Critical views of organizational unlearning

We also found that critical approaches shed light on the process of organizational unlearning. These approaches are considered “critical” because they fit in with what Fournier and Grey ( 2000 ; cf. Alakavuklar and Alamgir 2018 ) called “non-performative intent,” an important theme in critical management studies. In general, they highlight the importance of unlearning, but reject “the instrumental and performative use of unlearning in the sole service of attaining organizational goals” in the neoliberal system (Visser 2017 , p. 49). In this regard, these views differ from many other MOS approaches to organizational unlearning.

Although Contu et al. ( 2003 , p. 934) do not directly address the concept itself, they offer a useful starting point for the critical understanding of organizational unlearning in MOS and identify two central issues as learning can become “antithetical:” to learn is to disorganize and increase variety, but to organize is to reduce variety. That is, learning can be used as a tool to enhance organizational performance, but it can also have a wider impact beyond managerial concerns and may violate the common social good. These views have important implications for a critical understanding of the organizational unlearning process.

Brook et al. ( 2016 , p. 371) contend that there is a cultural tendency to see learning as an unquestionably “good thing,” which altogether is exacerbating rather than resolving the problems confronting business and societies (cf. Contu et al. 2003 ; Hsu 2013 ). In Brook et al.’s ( 2016 ) account, organizational unlearning is a necessity because it not only problematizes the self-evident, positive views of learning, but also reveals the political nature of learning; they applied the concept of organizational unlearning in the field of (critical) action learning and argue that unlearning is particularly relevant to address “wicked problems,” like global warming.

Drawing upon Foucault’s ( 1991 ) governmentality, Chokr ( 2009 , p. 61) perceives unlearning as a reflective, enduring capability for individuals “not to be governed” by “the illusory world of all the ideas, notions and, beliefs that hem, jostle, whirl, confuse and oppress them.” Ultimately, for Chokr ( 2009 , p. 49), unlearning should generate “well-trained minds and individuals capable of questioning, critical thinking, imagination, creativity and self-reflective deliberation as engaged citizens.” Pedler and Hsu ( 2014 ) apply this approach to MOS and suggest that power is an inseparable, unmanageable, and uncontrollable dimension of learning, and that unlearning implies an individual’s capability to recognize the inevitable power relations in the process of learning, and making ethical judgments over time. Hsu ( 2021 ) articulates three capabilities implied by an on-going attempt of unlearning in the field of management education: the capability to think differently, to approach knowledge autonomously, and to act as self-governed, self-reflective, self-engaged citizens.

Antonacopoulou ( 2009 , p. 424) views unlearning as an on-going practice of “asking different questions by extending the outcomes sought” which is “in sharp contrast to previous conceptualizations” to remove “old knowledge in favour of new knowledge.” Unlearning ought to trigger “difference” (Deleuze 1994 ). Hsu ( 2013 ) contends that unlearning, as a practice, bears liberating and emancipatory implications as it enables individuals to develop a capability to problematize institutionalized ideologies and actions; epistemologically, unlearning assists the rediscovery of what Foucault ( 1980 ) called “subjugated knowledge.” Such subjugated knowledge may include that wisdom has been marginalized within predominant theories and practices, for example, the wisdom of non-action (Hsu 2013 ). Drawing upon a feminist, de-colonial, and arts-based perspective, Krauss ( 2019 ) views unlearning as a collective practice that assists individuals in creating alternative forms of living while breaking with the promise of economic advancement and growth. Taken together, these views suggest that the process of organizational unlearning requires several skills and practices associated with the capability or possibility of individuals and collectives to question and discard knowledge.

3.6 Summary of key findings

The following table provides an overview of the key points of each perspective in the process of organizational unlearning (Table 1 ). Our findings form the foundation of the implications, the integrative framework as well as future research directions.

4 Implications

The current body of literature shares three common underlying assumptions about the concept:

Organizational unlearning is perceived as a process that is based on an organization’s intention to discard—often multiple and intertwined—existing organizational knowledge structures;

Organizational unlearning evolves through mechanisms that assume different shapes and forms, depend significantly on the context, and are mostly introduced reactively to ensure organizational survival during crises, facilitate organizational change and learning, and improve innovativeness; and

Organizational unlearning is regarded as a highly complex organizational phenomenon as it dynamically unfolds within and across multiple levels, such as groups or individuals.

Our review, however, also reveals that the concept of organizational unlearning is imbalanced and fragmented (cf. Martin de Holan and Phillips 2011 ; Klammer and Gueldenberg 2019 ) which has led to its contestation (cf. Klein 1989 ; Howells and Scholderer 2016 ; Tsang 2017a , b ), because our understanding of how unlearning unfolds in organizational settings over time is still vague.

Three issues stand out. First, studies use different underlying assumptions about the concept, each typically arising from and remaining within its own domain. Using different terminologies (e.g., intentional forgetting, unlearning) or using the same terminology to describe different underlying assumptions about unlearning (e.g., unlearning following a sequential, recursive, or dialectic logic) leads to discrepancies and hampers our understanding of the concept. This also pertains to the process of unlearning; for instance, do organizations try to overwrite established knowledge by enacting new knowledge, or is knowledge aimed to be erased? Second, existing literature tends to focus on specific aspects of organizational unlearning (e.g., levels, antecedents, outcomes) without setting studies in a wider context, thereby leading to fragmentation. This perpetuates existing conceptual issues regarding the process of unlearning. Third, and in contrast to the previous point, other studies disregard the clarification of underlying assumptions about organizational unlearning (e.g., problematization or clearly defining levels), fostering a lack of decipherability.

We find that literature lacks an encompassing perspective that synthesizes existing conceptualizations and empirical studies to clarify why unlearning occurs, what it entails, and how the process actually unfolds. We propose and visualize an integrative framework that considers the issues outlined above and incorporates various fragments and streams in the field of organizational unlearning. To build a framework that is applicable across all communities within MOS, we assert that viewing unlearning as a process and making the concept dynamic are key to bringing different perspectives together. In the following, we articulate and discuss four implications that help future studies navigate through the profound and dynamic nature of organizational unlearning.

4.1 Implication 1: Organizational unlearning involves multiple levels

Unlearning entails a profound interdependence and interplay between and within different levels of an organization. However, existing research reflects a distinction between levels, with studies typically focusing on the individual level (Hislop et al. 2014 ; Matsuo 2018 , 2019a , b ), the group level (Akgün et al. 2006 ; Lee and Sukoco 2011 ; Klammer and Gueldenberg 2020 ), or the organizational level (Yang et al. 2014 ; Snihur 2018 ). Whether initiated top-down or bottom-up (Klammer et al. 2019a ; Padan and Nguyen 2020 ; Grisold et al. 2020 ), unlearning cannot be perceived as an isolated phenomenon. It dynamically and sometimes even simultaneously affects all entities including individuals, groups as well as the entire organization. Literature highlights the vital role of individuals and groups in the process of unlearning (Zhao et al. 2013 ; Hislop et al. 2014 ; Kluge 2023 ); since these claims are conceptual, however, we know little about the dynamics that unfold across these levels.

We suggest that the unlearning process manifests at all organizational levels. It is crucial to stress that in order to understand unlearning at the collective level, one cannot aggregate and extrapolate individual-level cognitive processes (Grisold and Kaiser 2017 ). Rather, collective unlearning involves complex feedback mechanisms that either reinforce or diminish the influence of old knowledge on organizational practices, which, in turn, spills over to collective activities (e.g., Crossan et al. 1999 ).

4.2 Implication 2: Motives behind organizational unlearning need to be translated into interventions

Organizational unlearning is enabled by intentional interventions that specifically aim to support the process of discarding obsolete knowledge structures over time. Several empirical studies offer initial insights into the workings and dynamics of interventions as mechanisms of organizational unlearning.

Perhaps the most challenging and complex intervention is to reduce the influence of old knowledge over time. While explicit, codified knowledge, such as written rules and regulations can be discarded relatively easily, implicit knowledge structures, like assumptions, beliefs, values, or norms are unequally harder to be unlearned. For this intervention, it is important to eliminate retrieval cues that make individuals draw less from old knowledge or habits over time (Kluge and Gronau 2018 ). This also holds true when no new knowledge should be implemented; reducing the influence of old knowledge is key in discarding an organization’s obsolete cognitive and behavioral knowledge structures to free up space for future possibilities (Peschl 2019 ). Combining both approaches, appreciative inquiry, for example, can facilitate the discarding of old knowledge while simultaneously addressing the creation of new knowledge (Srithika and Bhattacharyya 2009 ). Additionally, the benefits of the “new” should be constantly reinforced through feedback and clear communication (Grisold et al. 2020 ).

4.3 Implication 3: Processes of organizational unlearning differ in form, antecedents, and outcomes

We suggest that antecedents can be based on reactive and proactive grounds, and that the (desired) outcomes of organizational unlearning can only be fully known once the process has been completed. Generally, scholars promote the understanding that organizational unlearning is a reactive phenomenon (Snihur 2018 ) typically triggered by problems (Hedberg 1981 ; Nystrom and Starbuck 1984 ) or different cues (Sinkula 2002 ). More recent studies show that organizational unlearning also entails a proactive dimension and is advantageous when executed proactively (Morais-Storz and Nguyen 2017 ). In terms of outcomes and consequences, unlearning is generally perceived as a positive phenomenon. It is regarded as a facilitator of organizational change (e.g., Johannessen and Hauan 1994 ; Turc and Baumard 2007 ; Martin de Holan 2011a ; Mull et al. 2023 ; van Oers et al. 2023 ; Hamza-Orlinska et al. 2024 ) and an enabler of innovation and innovative behavior (e.g., Becker 2008 ; Cepeda-Carrion et al. 2012a ; Leal-Rodríguez et al. 2015 ; Zhang et al. 2022 ; Zhao et al. 2022 ; Klammer et al. 2023 ).

Researchers have seldom questioned the positive value of organizational unlearning. However, as knowledge is intertwined throughout the organization and embedded in assumptions, world views, values, habits, routines, processes, etc., unlearning specific knowledge structures might lead to a decrease of value or functioning of other parts (Zahra et al. 2011 ). Therefore, it is difficult to judge the value of (to-be) discarded knowledge. Organizational unlearning prompts a clash between the past, present, and future and involves different elements, such as culture, assumptions, beliefs, structures, strategies, routines, or habits. Hence, and contrary to managerial expectations (Govindarajan et al. 2021 ), the outcome of organizational unlearning can only be fully understood once the process is complete.

4.4 Implication 4: Prevalent organizational contexts highly influence the unlearning process

Researchers need to acknowledge that organizational unlearning comes with different reasons, decisions, and strategies. Studying idiosyncratic features of a given organizational context contrasts with the prevalent focus in organizational unlearning research. Some studies provide in-depth insights about how unlearning unfolds in a specific organizational context (Martin de Holan and Phillips 2004b ; Rezazade Mehrizi and Lashkarbolouki 2016 ; Burt and Nair 2020 ). The contexts or situated features in which unlearning occurs, however, remain elusive as the main interest is often placed on abstract sequences or phases that characterize unlearning (e.g., Nygren et al. 2017 ; Cegarra-Navarro et al. 2021 ; Kim and Park 2022 ). This comes at the cost of understanding how organizational unlearning actually unfolds and what elements it entails.

Empirical studies that embrace the processes through which organizational phenomena unfold typically find that these processes are tied to the specific situated context of organizations (Langley et al. 2013 ). Based on this line of thinking, we argue that the elaboration of an empirically examined unlearning process should be tied to its organizational context and other prevailing situated features.

We summarize and visualize our implications in an integrative framework (Fig. 2 ) to highlight the characteristics of the organizational unlearning process. Unlearning in organizations depends on a variety of factors that can alter the course of the process. In the following, we propose future research avenues that can further our understanding of organizational unlearning.

Process-based framework of organizational unlearning

5 Future research directions

5.1 forging organizational unlearning research as process-based studies.

Discarded knowledge that has once been enacted in organizations is difficult to capture. Researchers have attempted to capture this process using cross-sectional surveys (e.g., Sheaffer and Mano-Negrin 2003 ). We believe that—although efforts to operationalize unlearning are immensely valuable—existing questionnaires fall short of capturing the full extent of the organizational unlearning process; not capturing the full extent of unlearning does not allow for explaining non-linear dynamics that underlie the process (e.g., actors may find it more difficult to unlearn initially, but it becomes significantly easier after knowledge has been used less often). We assert that researchers need to study the concept more profoundly and longitudinally by examining different antecedents, processes, interventions, outcomes, levels, knowledge structures, and so on, from a process-based perspective (Langley and Tsoukas 2017 ). This can be achieved through methods, such as ethnography or case study research, that capture discarded knowledge and allow for a deep observation of the organizational unlearning process.

New research methods for generating insightful data may contribute to a clearer understanding of the phenomenon. One of the issues in survey-based research, for example, is knowledge retrieval; asking subjects if they currently need to unlearn, or have unlearned knowledge recently, might trigger an association with old knowledge. Hence, the process of unlearning could be disturbed. Methods that track the development and paths of knowledge to make it more explicit are especially interesting (Kluge et al. 2019 ).

Turning to research methods in digital environments, for example, may allow researchers to generate fresh insights into organizational unlearning. The increasing availability of digital trace data, i.e., digital footprints that are automatically recorded whenever actors use information technology, such as ERP systems (Pentland et al. 2020 ) or online platforms (Lindberg et al. 2016 ), renders promising opportunities. Digital trace data are considered particularly useful by organizational researchers because they provide an unobtrusive and unbiased way of studying organizational work (e.g., Berente et al. 2019 ). Using digital trace data to study unlearning processes allows researchers to gain an accurate picture of the more and less frequently adopted actions, and how processes change over time (e.g., before and after an unlearning-related intervention). Therefore, using digital trace data could open entirely new avenues for investigating organizational unlearning. Researchers could conduct in-depth analyses to examine whether, and/or how, interventions yield desired outcomes, undesired routines vanish, or single actions disappear over time.

Process-based studies can also shed a more nuanced light on mechanisms, antecedents, or outcomes (Langley et al. 2013 ). Our findings on the timing of unlearning imply that organizations, although seldom investigated empirically, do not always wait until they have no other choice but to unlearn. This challenges the assumption that organizational unlearning is caused exclusively by endogenous or exogenous shocks and, in turn, raises questions about the antecedents and expected outcomes of the process. Diagnosing antecedents and outcomes seems to be a major challenge, often because we can only observe organizational unlearning retrospectively.

If organizations understand how knowledge abandonment can help them achieve specific goals, they can design a setup for the type of unlearning that matches their objectives. For example, for organizations that want to improve gradually and continuously, shallow unlearning would be a good option because it contributes to day-to-day adaptation without destroying operational stability. Organizations that want to challenge their deeply held beliefs or taken-for-granted assumptions might require a proactive and deep unlearning approach. Following this line of thinking, we suggest for future studies to focus on the dynamic nature of the concept to highlight the specific facets and interventions of organizational unlearning processes, and provide in-depth explanations of how organizations intentionally refrain from using old knowledge over time. Focusing on such dynamics might also provide fresh perspectives on the interdependence and interplay at different organizational levels. These insights are needed, from our point of view, to strengthen the conceptual understanding of the organizational unlearning phenomenon, and demarcate it from related concepts, such as organizational learning and change (Howells and Scholderer 2016 ).

5.2 Highlighting contextual features and the nature of the unlearning process

Putting increased focus on the context of an organization may shed light on how or why organizations detach from—or keep adhering to—old routines, assumptions, and beliefs. Foregrounding the idiosyncratic features of old knowledge and how they are tied to the context of an organization might inform the design of effective interventions in a given situation. As such, unlearning interventions have both explanatory and normative value for organizational unlearning research. From an explanatory perspective, focusing on the context of unlearning interventions enables researchers to outline why an unlearning process unfolds the way it does. Differences in the width and depth of unlearning interventions, paired with the desired outcomes of the process, may explain how organizations intentionally remove knowledge from points A to B in a specific context. From a normative perspective, the awareness of contextual features can guide organizations, policy-makers, and other stakeholders in initiating and guiding different unlearning processes.

This also corresponds with emerging claims that MOS researchers should increasingly engage in real-world problem solving (e.g., Hideg et al. 2020 ; Howard-Grenville 2021 ). For example, scholars in the field of MOS have increasingly focused on grand challenges, questioning how organizations can effectively address complex social and environmental threats (e.g., Ambos and Tatarinov 2022 ; Voegtlin et al. 2022 ; Sele et al. 2024 ). One underlying theme in this stream of research is that organizations need to replace their established logics and routines, which are often profit-oriented, with new and more conducive ones. The transition from old to new ways of doing things, however, rarely works smoothly. Several studies have found that organizations tend to fall back on old detrimental knowledge as they tackle grand challenges (e.g., Wright and Nyberg 2017 ; van Wijk et al. 2020 ). Focusing on contextual features and the in-situ nature of unlearning processes helps researchers understand the latent, sub-conscious facets of why knowledge abandonment might or might not unfold in a given situation.

5.3 Spotlighting power, power relations, and politics in unlearning processes

Critical perspectives of unlearning, informed by critical management studies, problematize the predominant managerial understanding of organizational unlearning, because they recognize that the process is highly power-laden. Such views differ from the vast majority of existing unlearning literature. While critical perspectives do not forsake the idea of unlearning and learning, they suggest that these processes may have far-reaching effects, for which organizations and managers purport to take responsibility. However, to date, critical views of unlearning have had little impact on mainstream MOS literature, but may enrich the aforementioned research possibilities.

First, future studies could focus on the power relations embedded in the process of organizational unlearning. For instance, managerial intervention in the unlearning process inevitably reflects different interests and may generate resistance because unlearning, like learning, is also a socially constructed entity with relations of power (Pedler and Hsu 2014 ). It is important to understand the different stakeholders and organizational politics involved in this process, including the beneficiaries and victims of organizational unlearning. Second, the critical views of unlearning may legitimize what Pedler and Hsu ( 2019 ) called an “alternative paradigm” of learning organizations. Future studies could explore how the unlearning process stimulates incompatible organizational purpose that collides with the prevailing one. Researchers may also explore different forms of wisdom and their relationship with organizational unlearning, and how unlearning helps inspire alternative organizational realities.

6 Practical implications

Organizational unlearning, particularly seen as a process that evolves over time, has significant practical implications for how organizations progress, innovate, and adapt to changing environments. By actively unlearning outdated or inefficient practices, organizations can adopt innovative methods and technologies more effectively (Di Maria et al. 2023 ). This process is crucial in rapidly changing industries where clinging to old ways can be a significant disadvantage. Unlearning, when understood as an on-going and persistent effort, helps to create a culture of agility and flexibility. Organizations become more adept at responding to market changes, customer needs, and emerging trends.

Furthermore, leaders and managers play a crucial role in initiating, modelling, and facilitating unlearning. This process calls for adaptable and self-aware leaders capable of challenging the status quo. It also requires them to be effective communicators in guiding their teams through unlearning processes. Organizational unlearning encourages a culture of critical thinking and open-mindedness, which is essential for strategic planning and problem-solving. To summarize, understanding organizational unlearning as an on-going effort requires deliberate strategies and a supportive organizational culture as it involves systematic approaches to identify what needs to be unlearned, mechanisms to facilitate the unlearning process, and the integration of new learning and knowledge into an organization's operations.

7 Conclusion

Our review of the existing literature in the broader context of MOS and its respective domains reveals a fragmented field of organizational unlearning, including studies based on different underlying assumptions about the concept. To bring different viewpoints together and highlight concerns about the phenomenon, we propose implications and possible future research directions that will help researchers navigate through the jungle of different understandings of unlearning. Table 2 presents exemplary research questions that can serve as starting points for future research. Organizational unlearning is best understood and researched as an intentionally initiated and dynamically unfolding process that aims to discard or reduce undesired knowledge structures over time.

Data availability

The authors do not generate new datasets in this literature review. All articles and works included in this literature review can be accessed through the databases mentioned in the text.

References marked with an asterisk indicate works analyzed in the literature review.

*Açıkgöz A, Demirkan I, Latham GP, Kuzey C (2021) The relationship between unlearning and innovation ambidexterity with the performance of new product development teams. Group Decis Negot 30:945–982

Article Google Scholar

*Akgün AE, Lynn GS, Byrne JC (2006) Antecedents and consequences of unlearning in new product development teams. J Prod Innov Manag 23:73–88

*Akgün AE, Byrne JC, Lynn GS, Keskin H (2007a) New product development in turbulent environments: impact of improvisation and unlearning on new product performance. J Eng Technol Manag 24:203–230

*Akgün AE, Byrne JC, Lynn GS, Keskin H (2007b) Organizational unlearning as changes in beliefs and routines in organizations. J Organ Change Manag 20:794–812

*Alakavuklar ON, Alamgir F (2018) Ethics of resistance in organisations: a conceptual proposal. J Bus Ethics 149:31–43

Ambos TC, Tatarinov K (2022) Building responsible innovation in international organizations through intrapreneurship. J Manag Stud 59:92–125

*Antonacopoulou EP (2009) Impact and scholarship: unlearning and practising to co-create actionable knowledge. Manag Learn 40:421–430

*Azmi FT (2008) Mapping the learn-unlearn-relearn model: imperatives for strategic management. Eur Bus Rev 20:240–259

*Becker KL (2008) Unlearning as a driver of sustainable change and innovation: three Australian case studies. Int J Technol Manag 42:89–106

*Becker KL (2010) Facilitating unlearning during implementation of new technology. J Organ Change Manag 23:251–268

*Becker KL (2019) Organizational unlearning: the challenges of a developing phenomenon. Learn Organ 26:534–541

Berente N, Seidel S, Safadi H (2019) Research commentary––data-driven computationally intensive theory development. Inf Syst Res 30:50–64

*Bowker GC (1997) Lest we remember: organizational forgetting and the production of knowledge. Account Manag Inf Technol 7:113–138

Google Scholar

*Brook C, Pedler M, Abbott C, Burgoyne J (2016) On stopping doing those things that are not getting us to where we want to be: unlearning, wicked problems and critical action learning. Hum Relat 69:369–389

*Brooks J, Grugulis I, Cook H (2022) Unlearning and consent in the UK Fire and Rescue Service. Hum Relat 75:2300–2317

*Burt G, Nair AK (2020) Rigidities of imagination in scenario planning: strategic foresight through ‘unlearning.’ Technol Forecast Soc Change 153:119927

*Burton N, Vu MC, Hawkins M (2023) Secular discernment: a process of individual unlearning and collective relearning. Manag Learn 54:680–704

*Cegarra-Navarro JG, Cepeda-Carrion G (2013) Implementing telemedicine technologies through an unlearning context in a homecare setting. Behav Inf Technol 32:80–90

*Cegarra-Navarro JG, Sánchez-Polo MT (2008) Linking the individual forgetting context with customer capital from a seller’s perspective. J Oper Res Soc 59:1614–1623

*Cegarra-Navarro JG, Wensley AKP (2019) Promoting intentional unlearning through an unlearning cycle. J Organ Change Manag 32:67–79

*Cegarra-Navarro JG, Eldridge S, Martinez-Martinez A (2010) Managing environmental knowledge through unlearning in Spanish hospitality companies. J Environ Psychol 30:249–257

*Cegarra-Navarro JG, Cepeda-Carrion G, Eldridge S (2011a) Balancing technology and physician–patient knowledge through an unlearning context. Int J Inf Manag 31:420–427

*Cegarra-Navarro JG, Sánchez-Vidal ME, Cegarra-Leiva D (2011b) Balancing exploration and exploitation of knowledge through an unlearning context: an empirical investigation in SMEs. Manag Decis 49:1099–1119

*Cegarra-Navarro JG, Martinez-Martinez A, Ortega-Gutiérrez J, Leal-Rodríguez AL (2013) Environmental knowledge, unlearning, and performance in hospitality companies. Manag Decis 51:341–360

*Cegarra-Navarro JG, Eldridge S, Wensley AKP (2014) Counter-knowledge and realized absorptive capacity. Eur Manag J 32:165–176

*Cegarra-Navarro JG, Sánchez-Vidal ME, Cegarra-Leiva D (2016) Linking unlearning with work-life balance: an initial empirical investigation into SMEs. J Small Bus Manag 54:373–391

*Cegarra-Navarro JG, Wensley AKP, Batisic S, Evans M, Cubillas Para C (2021) Minimizing the effects of defensive routines on knowledge hiding though unlearning. J Bus Res 137:58–68

*Cepeda-Carrion G, Cegarra-Navarro JG, Jimenez-Jimenez D (2012a) The effect of absorptive capacity on innovativeness: context and information systems capability as catalysts. Brit J Manag 23:110–129

*Cepeda-Carrion G, Cegarra-Navarro JG, Martinez-Caro E (2012b) Improving the absorptive capacity through unlearning context: an empirical investigation in hospital-in-the-home units. Serv Ind J 32:1551–1570

*Chokr NN (2009) Unlearning or ‘How NOT To Be Governed?’ Societas, Charlottesville

*Contu A, Grey C, Örtenblad A (2003) Against learning. Hum Relat 56:931–952

Cowan R, David PA, Foray D (2000) The explicit economics of knowledge codification and tacitness. Ind Corp Change 9:211–253

Cronin MA, George E (2023) The why and how of the integrative review. Organ Res Methods 26:186–192

Crossan MM, Lane HW, White RE (1999) An organizational learning framework: from intuition to institution. Acad Manag Rev 24:522–537

*Deleuze G (1994) Difference and repetition. Athlone Press, London

Di Maria M, Walter D, Schoormann T, Knackstedt R (2023) Practical support for unlearning—a systematic review to organize the field. In: Aenestad M, Klein S, Tarafdar M (eds) Proceedings of the 31st European Conference on Information Systems

*Easterby-Smith M, Lyles MA (2011) In praise of organizational forgetting. J Manag Inq 20:311–316

*Fiol CM, O’Connor EJ (2017a) Unlearning established organizational routines—Part I. Learn Organ 24:13–29

*Fiol CM, O’Connor EJ (2017b) Unlearning established organizational routines—Part II. Learn Organ 24:82–92

*Foucault M (1980) Power/knowledge: selected interviews and other writings 1972–1977. Pantheon Books, New York

Foucault M (1991) Discipline and punish: the birth of the prison. Penguin Books, London

*Fournier V, Grey C (2000) At the critical moment: conditions and prospects from critical management studies. Hum Relat 53:7–32

Govindarajan V, Srivastava A, Grisold T, Klammer A (2020) COVID-imposed opportunity to selectively unlearn past practices. In: California Management Review Insight. https://cmr.berkeley.edu/2020/10/selective-unlearning . Accessed 2 March 2023

Govindarajan V, Srivastava A, Grisold T, Klammer A (2021) Resist old routines when returning to the office. In: Harvard Business Review. https://hbr.org/2021/05/resist-old-routines-when-returning-to-the-office . Accessed 3 March 2023

*Grisold T, Kaiser A (2017) Leaving behind what we are not: applying a systems thinking perspective to present unlearning as an enabler for finding the best version of the self. J Organ Transform Soc Change 14:39–55

*Grisold T, Klammer A, Kragulj F (2020) Two forms of organizational unlearning: insights from engaged scholarship research with change consultants. Manag Learn 51:598–619

*Grisold T, Kaiser A, Hafner J (2017) Unlearning before creating new knowledge: A cognitive process. In: Bui TX, Sprague Jr. R (eds) Proceedings of the 50th Hawaii International Conference on System Sciences, IEEE Computer Society Press, New York, pp 4614–4623

Haase J, Matthiesen J, Schüffler AS, Kluge A (2020) Retentivity beats prior knowledge as predictor for the acquisition and adaptation of new production processes. In: Bui TX, Sprague Jr. R (eds) Proceedings of the 53rd Hawaii International Conference on System Sciences, IEEE Computer Society Press, New York, pp 4796–4805

*Hamza-Orlinska A, Maj J, Shantz A, Vassilopoulou J (2024) Unlearning diversity management. J World Bus 59:101519

*Hedberg BLT (1981) How organizations learn and unlearn. In: Nystrom PC, Starbuck WH (eds) Handbook of organizational design: adapting organizations to their environments. Oxford University Press, Oxford, pp 3–27

*Hedberg BLT, Nystrom PC, Starbuck WH (1976) Camping on seesaws: prescriptions for a self-designing organization. Adm Sci Q 21:41–65

Hideg I, DeCelles KA, Tihanyi L (2020) From the editors: publishing practical and responsible research in AMJ. Acad Manag J 63:1681–1686

*Hislop D, Bosley S, Coombs CR, Holland J (2014) The process of individual unlearning: a neglected topic in an under-researched field. Manag Learn 45:540–560

Howard-Greenville J (2021) Grand challenges, Covid-19 and the future of organizational scholarship. J Manag Stud 58:254–258

*Howells J, Scholderer J (2016) Forget unlearning? How an empirically unwarranted concept from psychology was imported to flourish in management and organization studies. Manag Learn 47:443–463

*Hsu S-w (2013) Alternative learning organization. In: Örtenblad A (ed) Handbook of research on the learning organization: adaptation and context. Edward Elgar, Cheltenham, pp 358–371

*Hsu S-w (2021) Exploring an alternative: Foucault-Chokr’s unlearning approach to management education. Int J Manag Educ 19:100496

Huber GP (1991) Organizational learning: the contributing processes and the literatures. Organ Sci 2:88–115

*Johannessen JA, Hauan A (1994) Organizational unlearning. Create Innov Manag 3:43–53

*Kim EJ, Park S (2022) Unlearning in the workplace: antecedents and outcomes. Hum Resour Dev Q 33:273–296

*Klammer A (2021) Embracing organisational unlearning as a facilitator of business model innovation. Int J Innov Manag 25:2150061

*Klammer A, Gueldenberg S (2019) Unlearning and forgetting in organizations: a systematic review of literature. J Knowl Manag 23:860–888

*Klammer A, Gueldenberg S (2020) Honor the old, welcome the new: an account of unlearning and forgetting in NPD teams. Eur J Innov Manag 23:581–603

*Klammer A, Grisold T, Gueldenberg S (2019a) Introducing a “stop-doing” culture: How to free your organization from rigidity. Bus Horiz 62:451–458

*Klammer A, Grisold T, Nguyen N (2019b) Guest editorial: opportunities and interdisciplinary perspectives for organizational unlearning. Learn Organ 26:445–453

*Klammer A, Hora W, Kailer N (2023) Opposites attract: How incumbents learn and unlearn in coopetitive relationships with start-ups. Ind Mark Manag 112:85–97

*Klein JI (1989) Parenthetic learning in organizations: toward the unlearning of the unlearning model. J Manag Stud 26:291–308

*Kluge A (2023) Recent findings on organizational unlearning and intentional forgetting research (2019–2022). Front Psychol 14:1160173

*Kluge A, Gronau N (2018) Intentional forgetting in organizations: the importance of eliminating retrieval cues for implementing new routines. Front Psychol. https://doi.org/10.3389/fpsyg.2018.00051

*Kluge A, Schüffler AS, Thim C, Haase J, Gronau N (2019) Investigating unlearning and forgetting in organizations: research methods, designs and implications. Learn Organ 26:518–533

*Krauss A (2019) Unlearning institutional habits: an arts-based perspective on organizational unlearning. Learn Organ 26:485–499

Langley A, Tsoukas H (2017) The SAGE handbook of process organization studies. SAGE, Thousand Oaks

Langley A, Smallman C, Tsoukas H, Van de Ven AH (2013) Process studies of change in organization and management: unveiling temporality, activity, and flow. Acad Manag J 56:1–13

*Leal-Rodríguez AL, Eldridge S, Roldán JL, Leal-Millán AG, Ortega-Gutiérrez J (2015) Organizational unlearning, innovation outcomes, and performance: the moderating effect of firm size. J Bus Res 68:803–809

*Lee LTS, Sukoco BM (2011) Reflexivity, stress, and unlearning in the new product development team: the moderating effect of procedural justice. R&D Manag 41:410–423

Leonard-Barton D (1992) Core capabilities and core rigidities: a paradox in managing new product development. Strat Manag J 13:111–125

Lindberg A, Berente N, Gaskin J, Lyytinen K (2016) Coordinating interdependencies in online communities: a study of an open source software project. Inf Syst Res 27:751–772

*Lyu C, Zhang F, Ji J, Teo TSH, Wang T, Liu Z (2022) Competitive intensity and new product development outcomes: the roles of knowledge integration and organizational unlearning. J Bus Res 139:121–133

*Macdonald G (2002) Transformative unlearning: safety, discernment and communities of learning. Nurs Inq 9:170–178

*Martignoni D, Keil T (2021) It did not work? Unlearn and try again—unlearning success and failure beliefs in changing environments. Strat Manag J 42:1057–1082

*Martin de Holan P (2011a) Agency in voluntary organizational forgetting. J Manag Inq 20:317–322

*Martin de Holan P (2011b) Organizational forgetting, unlearning, and memory systems. J Manag Inq 20:302–304

*Martin de Holan P, Phillips N (2004a) Organizational forgetting as strategy. Strat Organ 2:423–433

*Martin de Holan P, Phillips N (2004b) Remembrance of things past? The dynamics of organizational forgetting. Manag Sci 50:1603–1613

*Martin de Holan P, Phillips N (2011) Organizational forgetting. In: Easterby-Smith M, Lyles MA (eds) Handbook of organizational learning and knowledge management. Wiley, Chichester, pp 433–451

*Martin de Holan P, Phillips N, Lawrence TB (2004) Managing organizational forgetting. MIT Sloan Manag Rev 45:45–51

*Matsuo M (2018) Goal orientation, critical reflection, and unlearning: an individual-level study. Hum Resour Dev Q 29:49–66

*Matsuo M (2019a) Critical reflection, unlearning, and engagement. Manag Learn 50:465–481

*Matsuo M (2019b) The unlearning of managerial skills: a qualitative study of executive officers. Eur Manag Rev 16:303–315

McGrath RG (2019) Seeing around corners: How to spot inflection points in business before they happen. Houghton Mifflin Harcourt, Boston

McGrath RG, Euchner J (2020) Seeing around corners: an interview with Rita McGrath. Res-Technol Manag 63:12–17

*McKeown I (2012) Teaching old dogs new tricks: Why unlearning matters in SMEs. Int J Entrep Innov 13:25–34

*Morais-Storz M, Nguyen N (2017) The role of unlearning in metamorphosis and strategic resilience. Learn Organ 24:93–106

*Mull M, Duffy C, Silberman D (2023) Forgetting to learn and learning to forget: the call for organizational unlearning. Eur J Train Dev 47:586–598

*Nguyen N (2017) The journey of organizational unlearning: a conversation with William H. Starbuck Learn Organ 24:58–66

*Nygren NV, Jokinen A, Nikula A (2017) Unlearning in managing wicked biodiversity problems. Landsc Urban Plan 167:473–482

*Nystrom PC, Starbuck WH (1984) To avoid organizational crises, unlearn. Organ Dyn 12:53–65

*Ortega-Gutiérrez J, Cegarra-Navarro JG, Cepeda-Carrion G, Leal-Rodríguez AL (2015) Linking unlearning with quality of health services through knowledge corridors. J Bus Res 68:815–822

*Ortega-Gutiérrez J, Cepeda-Carrión I, Alves H (2022) The role of absorptive capacity and organizational unlearning in the link between social media and service dominant orientation. J Knowl Manag 26:920–942

*Padan T, Nguyen N (2020) Responsible management unlearning. In: Laasch O, Suddaby R, Freeman RE, Jamali D (eds) Research handbook of responsible management. Edward Elgar, Cheltenham, pp 501–515

*Pedler M, Hsu S-w (2014) Unlearning, critical action learning and wicked problems. Action Learn Res Pract 11:296–310

*Pedler M, Hsu S-w (2019) Regenerating the learning organization: towards an alternative paradigm. Learn Organ 26:97–112

*Pentland B, Liu P, Kremser W, Hærem T (2020) The dynamics of drift in digitized processes. MIS Q 44:19–47

*Peschl MF (2019) Unlearning towards an uncertain future: on the back end of future-driven unlearning. Learn Organ 26:454–469

*Polites GL, Karahanna E (2012) Shackled to the status quo: the inhibiting effects of incumbent system habit, switching costs, and inertia on new system acceptance. MIS Q 36:21–42

*Pratt MG, Barnett CK (1997) Emotions and unlearning in Amway recruiting techniques: promoting change through ‘safe’ ambivalence. Manag Learn 28:65–88

*Reese S (2017) Putting organizational unlearning into practice: a few steps for the practitioner. Learn Organ 24:67–69

*Rezazade Mehrizi MH, Lashkarbolouki M (2016) Unlearning troubled business models: from realization to marginalization. Long Range Plan 49:298–323

Rojon C, Okupe A, McDowall A (2021) Utilization and development of systematic reviews in management research: what do we know and where do we go from here? Int J Manag Rev 23:191–223

*Rupcic N (2019) Learning-forgetting-unlearning-relearning—the learning organization’s learning dynamics. Learn Organ 26:542–548

*Rushmer R, Davies HTO (2004) Unlearning in health care. BMJ Qual Saf 13:ii10–ii15

Sele K, Mahringer CA, Danner-Schröder A, Grisold T, Renzl B (2024) We are all pattern makers! How a flat ontology connects organizational routines and grand challenges. Strat Organ. https://doi.org/10.1177/14761270231215685

*Sharma S, Lenka U (2019) Exploring linkages between unlearning and relearning in organizations. Learn Organ 26:500–517

*Sharma S, Lenka U (2022a) Counterintuitive, yet essential: taking stock of organizational unlearning research through a scientometric analysis (1976–2019). Knowl Manag Res Pract 20:152–174

*Sharma S, Lenka U (2022b) On the shoulders of giants: uncovering key themes of organizational unlearning research in mainstream management journals. Rev Manag Sci 16:1599–1695

*Sharma S, Lenka U (2024) How does transformational leadership impact organizational unlearning: insights from persistence theories. J Organ Change Manag 37:150–172

*Sheaffer Z, Mano-Negrin R (2003) Executives’ orientations as indicators of crisis management policies and practices. J Manag Stud 40:573–606

*Sinkula JM (2002) Market-based success, organizational routines, and unlearning. J Bus Ind Mark 17:253–269

*Snihur Y (2018) Responding to business model innovation: organizational unlearning and firm failure. Learn Organ 25:190–198

*Srithika TM, Bhattacharyya S (2009) Facilitating organizational unlearning using appreciative inquiry as an intervention. Vikalpa 34:67–78

*Starbuck WH (1996) Unlearning ineffective or obsolete technologies. Int J Technol Manag 11:725–737

*Starbuck WH (2017) Organizational learning and unlearning. Learn Organ 24:30–38

Starbuck WH, Milliken FJ (1988) Challenger: fine-tuning the odds until something breaks. J Manag Stud 25:319–340

*Starbuck WH, Greve A, Hedberg BLT (1978) Responding to Crises. J Bus Adm 9:111–137

*Tanaka S (2023) Effects of goal orientation and unlearning on individual exploration activities. J Workplace Learn 35:57–74

Tranfield D, Denyer D, Smart P (2003) Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Brit J Manag 14:207–222

Trullen J, Bos-Nehles A, Valverde M (2020) From intended to actual and beyond: a cross-disciplinary view of (human resource management) implementation. Int J Manag Rev 22:150–176

*Tsang EWK (2008) Transferring knowledge to acquisition joint ventures: an organizational unlearning perspective. Manag Learn 39:5–20

*Tsang EWK (2017a) How the concept of organizational unlearning contributes to studies of learning organizations: a personal reflection. Learn Organ 24:39–48

*Tsang EWK (2017b) Stop eulogizing, complicating or straitjacketing the concept of organizational unlearning, please. Learn Organ 24:78–81

*Tsang EWK, Zahra SA (2008) Organizational unlearning. Hum Relat 61:1435–1462

Tsoukas H, Vladimirou E (2001) What is organizational knowledge? J Manag Stud 38:973–993

*Turc E, Baumard P (2007) Can organizations really unlearn? In: McInerney CR, Day RE (eds) Rethinking knowledge management: from knowledge objects to knowledge processes. Springer, Berlin, pp 125–146

Chapter Google Scholar

*van Oers L, Feola G, Runhaar H, Moors E (2023) Unlearning in sustainability transitions: insight from two Dutch community-supported agriculture farms. Environ Innov Soc Transit 46:100693

van Wijk J, van Wijk J, Drost S, Stam W (2020) Challenges in building robust interventions in contexts of poverty: insights from an NGO-driven multi-stakeholder network in Ethiopia. Organ Stud 41:1391–1415

*Visser M (2017) Learning and unlearning: a conceptual note. Learn Organ 24:49–57

Voegtlin C, Scherer AG, Stahl GK, Hawn O (2022) Grand societal challenges and responsible innovation. J Manag Stud 59:1–28

*Vu MC, Nguyen LA (2022) Mindful unlearning in unprecedented times: implications for management and organizations. Manag Learn 53:797–817

*Wang X, Lu Y, Zhao Y, Gong S, Li B (2013) Organizational unlearning, organizational flexibility and innovation capability: an empirical study of SMEs in China. Int J Technol Manag 61:132–155

*Wang X, Xi Y, Xie J, Zhao Y (2017) Organizational unlearning and knowledge transfer in cross-border M&A: the roles of routine and knowledge compatibility. J Knowl Manag 21:1580–1595

*Wensley AKP, Cegarra-Navarro JG (2015) Overcoming knowledge loss through the utilization of an unlearning context. J Bus Res 68:1563–1569

Wright C, Nyberg D (2017) An inconvenient truth: how organizations translate climate change into business as usual. Acad Manag J 60:1633–1661

*Xi Y, Wang X, Zhu Y (2020) Organizational unlearning and knowledge transfer in cross-border M&As: the mediating role of knowledge integration from a routine-based view. J Knowl Manag 24:841–860

*Xu N, Wu X, Du J, Li S (2023) Organizational unlearning: a process of tension imposition, integration, and splitting. Eur Manag Rev. https://doi.org/10.1111/emre.12586

*Yang KP, Chou C, Chiu YJ (2014) How unlearning affects radical innovation: the dynamics of social capital and slack resources. Technol Forecast Soc Change 87:152–163

*Yildiz HE, Fey CF (2010) Compatibility and unlearning in knowledge transfer in mergers and acquisitions. Scand J Manag 26:448–456

*Yin J (2023) Effects of the paradox mindset on work engagement: the mediating role of seeking challenges and individual unlearning. Curr Psychol 42:2708–2718

*Zahra SA, Abdelgawad SG, Tsang EWK (2011) Emerging multinationals venturing into developed economies: Implications for learning, unlearning, and entrepreneurial capability. J Manag Inq 20:323–330

*Zhang F, Lyu C, Zhu L (2022) Organizational unlearning, knowledge generation strategies and radical innovation performance: evidence from a transitional economy. Eur J Mark 56:133–158

*Zhao Y, Wang X (2020) Organisational unlearning, relearning and strategic flexibility: from the perspective of updating routines and knowledge. Technol Anal Strat Manag 32:1251–1263

*Zhao Y, Lu Y, Wang X (2013) Organizational unlearning and organizational relearning: a dynamic process of knowledge management. J Knowl Manag 17:902–912

*Zhao Y, Wang Z, Feng T, Kong T, Zhang Q (2022) Organizational unlearning and inclusive innovation: the moderating role of green control ambidexterity. Sustain Dev 30:539–555

Download references

Open access funding provided by University of Liechtenstein. The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.

Author information

Authors and affiliations.

Liechtenstein Business School, University of Liechtenstein, Vaduz, Principality of Liechtenstein

Adrian Klammer

Institute of Information Management, University of St. Gallen, St. Gallen, Switzerland

Thomas Grisold

Department of Industrial Economics and Technology Management, Norwegian University of Science and Technology, Trondheim, Norway

Nhien Nguyen

Center for General Education, National Yang Ming Chiao Tung University, Taipei, Taiwan

Shih-wei Hsu

You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adrian Klammer .

Ethics declarations

Conflict of interest.

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Klammer, A., Grisold, T., Nguyen, N. et al. Organizational unlearning as a process: What we know, what we don’t know, what we should know. Manag Rev Q (2024). https://doi.org/10.1007/s11301-024-00430-3

Download citation

Received : 12 October 2023

Accepted : 31 March 2024

Published : 23 April 2024

DOI : https://doi.org/10.1007/s11301-024-00430-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Organizational unlearning
Unlearning process
Knowledge loss
Literature review
Integrative literature review

JEL Classification

Find a journal
Publish with us
Track your research

Challenges Affecting the Successful Adoption of DevOps Practices: A Systematic Literature Review

Ieee account.

Change Username/Password
Update Address

Purchase Details

Payment Options
Order History
View Purchased Documents

Profile Information

Communications Preferences
Profession and Education
Technical Interests
US & Canada: +1 800 678 4333
Worldwide: +1 732 981 0060
Contact & Support
About IEEE Xplore
Accessibility
Terms of Use
Nondiscrimination Policy
Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Open access
Published: 23 April 2024

Designing feedback processes in the workplace-based learning of undergraduate health professions education: a scoping review

Javiera Fuentes-Cimma 1 , 2 ,
Dominique Sluijsmans 3 ,
Arnoldo Riquelme 4 ,
Ignacio Villagran ORCID: orcid.org/0000-0003-3130-8326 1 ,
Lorena Isbej ORCID: orcid.org/0000-0002-4272-8484 2 , 5 ,
María Teresa Olivares-Labbe 6 &
Sylvia Heeneman 7

BMC Medical Education volume 24 , Article number: 440 ( 2024 ) Cite this article

Metrics details

Feedback processes are crucial for learning, guiding improvement, and enhancing performance. In workplace-based learning settings, diverse teaching and assessment activities are advocated to be designed and implemented, generating feedback that students use, with proper guidance, to close the gap between current and desired performance levels. Since productive feedback processes rely on observed information regarding a student's performance, it is imperative to establish structured feedback activities within undergraduate workplace-based learning settings. However, these settings are characterized by their unpredictable nature, which can either promote learning or present challenges in offering structured learning opportunities for students. This scoping review maps literature on how feedback processes are organised in undergraduate clinical workplace-based learning settings, providing insight into the design and use of feedback.

A scoping review was conducted. Studies were identified from seven databases and ten relevant journals in medical education. The screening process was performed independently in duplicate with the support of the StArt program. Data were organized in a data chart and analyzed using thematic analysis. The feedback loop with a sociocultural perspective was used as a theoretical framework.

The search yielded 4,877 papers, and 61 were included in the review. Two themes were identified in the qualitative analysis: (1) The organization of the feedback processes in workplace-based learning settings, and (2) Sociocultural factors influencing the organization of feedback processes. The literature describes multiple teaching and assessment activities that generate feedback information. Most papers described experiences and perceptions of diverse teaching and assessment feedback activities. Few studies described how feedback processes improve performance. Sociocultural factors such as establishing a feedback culture, enabling stable and trustworthy relationships, and enhancing student feedback agency are crucial for productive feedback processes.

Conclusions

This review identified concrete ideas regarding how feedback could be organized within the clinical workplace to promote feedback processes. The feedback encounter should be organized to allow follow-up of the feedback, i.e., working on required learning and performance goals at the next occasion. The educational programs should design feedback processes by appropriately planning subsequent tasks and activities. More insight is needed in designing a full-loop feedback process, in which specific attention is needed in effective feedforward practices.

Peer Review reports

The design of effective feedback processes in higher education has been important for educators and researchers and has prompted numerous publications discussing potential mechanisms, theoretical frameworks, and best practice examples over the past few decades. Initially, research on feedback primarily focused more on teachers and feedback delivery, and students were depicted as passive feedback recipients [ 1 , 2 , 3 ]. The feedback conversation has recently evolved to a more dynamic emphasis on interaction, sense-making, outcomes in actions, and engagement with learners [ 2 ]. This shift aligns with utilizing the feedback process as a form of social interaction or dialogue to enhance performance [ 4 ]. Henderson et al. (2019) defined feedback processes as "where the learner makes sense of performance-relevant information to promote their learning." (p. 17). When a student grasps the information concerning their performance in connection to the desired learning outcome and subsequently takes suitable action, a feedback loop is closed so the process can be regarded as successful [ 5 , 6 ].

Hattie and Timperley (2007) proposed a comprehensive perspective on feedback, the so-called feedback loop, to answer three key questions: “Where am I going? “How am I going?” and “Where to next?” [ 7 ]. Each question represents a key dimension of the feedback loop. The first is the feed-up, which consists of setting learning goals and sharing clear objectives of learners' performance expectations. While the concept of the feed-up might not be consistently included in the literature, it is considered to be related to principles of effective feedback and goal setting within educational contexts [ 7 , 8 ]. Goal setting allows students to focus on tasks and learning, and teachers to have clear intended learning outcomes to enable the design of aligned activities and tasks in which feedback processes can be embedded [ 9 ]. Teachers can improve the feed-up dimension by proposing clear, challenging, but achievable goals [ 7 ]. The second dimension of the feedback loop focuses on feedback and aims to answer the second question by obtaining information about students' current performance. Different teaching and assessment activities can be used to obtain feedback information, and it can be provided by a teacher or tutor, a peer, oneself, a patient, or another coworker. The last dimension of the feedback loop is the feedforward, which is specifically associated with using feedback to improve performance or change behaviors [ 10 ]. Feedforward is crucial in closing the loop because it refers to those specific actions students must take to reduce the gap between current and desired performance [ 7 ].

From a sociocultural perspective, feedback processes involve a social practice consisting of intricate relationships within a learning context [ 11 ]. The main feature of this approach is that students learn from feedback only when the feedback encounter includes generating, making sense of, and acting upon the information given [ 11 ]. In the context of workplace-based learning (WBL), actionable feedback plays a crucial role in enabling learners to leverage specific feedback to enhance their performance, skills, and conceptual understandings. The WBL environment provides students with a valuable opportunity to gain hands-on experience in authentic clinical settings, in which students work more independently on real-world tasks, allowing them to develop and exhibit their competencies [ 3 ]. However, WBL settings are characterized by their unpredictable nature, which can either promote self-directed learning or present challenges in offering structured learning opportunities for students [ 12 ]. Consequently, designing purposive feedback opportunities within WBL settings is a significant challenge for clinical teachers and faculty.

In undergraduate clinical education, feedback opportunities are often constrained due to the emphasis on clinical work and the absence of dedicated time for teaching [ 13 ]. Students are expected to perform autonomously under supervision, ideally achieved by giving them space to practice progressively and providing continuous instances of constructive feedback [ 14 ]. However, the hierarchy often present in clinical settings places undergraduate students in a dependent position, below residents and specialists [ 15 ]. Undergraduate or junior students may have different approaches to receiving and using feedback. If their priority is meeting the minimum standards given pass-fail consequences and acting merely as feedback recipients, other incentives may be needed to engage with the feedback processes because they will need more learning support [ 16 , 17 ]. Adequate supervision and feedback have been recognized as vital educational support in encouraging students to adopt a constructive learning approach [ 18 ]. Given that productive feedback processes rely on observed information regarding a student's performance, it is imperative to establish structured teaching and learning feedback activities within undergraduate WBL settings.

Despite the extensive research on feedback, a significant proportion of published studies involve residents or postgraduate students [ 19 , 20 ]. Recent reviews focusing on feedback interventions within medical education have clearly distinguished between undergraduate medical students and residents or fellows [ 21 ]. To gain a comprehensive understanding of initiatives related to actionable feedback in the WBL environment for undergraduate health professions, a scoping review of the existing literature could provide insight into how feedback processes are designed in that context. Accordingly, the present scoping review aims to answer the following research question: How are the feedback processes designed in the undergraduate health professions' workplace-based learning environments?

A scoping review was conducted using the five-step methodological framework proposed by Arksey and O'Malley (2005) [ 22 ], intertwined with the PRISMA checklist extension for scoping reviews to provide reporting guidance for this specific type of knowledge synthesis [ 23 ]. Scoping reviews allow us to study the literature without restricting the methodological quality of the studies found, systematically and comprehensively map the literature, and identify gaps [ 24 ]. Furthermore, a scoping review was used because this topic is not suitable for a systematic review due to the varied approaches described and the large difference in the methodologies used [ 21 ].

Search strategy

With the collaboration of a medical librarian, the authors used the research question to guide the search strategy. An initial meeting was held to define keywords and search resources. The proposed search strategy was reviewed by the research team, and then the study selection was conducted in two steps:

An online database search included Medline/PubMed, Web of Science, CINAHL, Cochrane Library, Embase, ERIC, and PsycINFO.

A directed search of ten relevant journals in the health sciences education field (Academic Medicine, Medical Education, Advances in Health Sciences Education, Medical Teacher, Teaching and Learning in Medicine, Journal of Surgical Education, BMC Medical Education, Medical Education Online, Perspectives on Medical Education and The Clinical Teacher) was performed.

The research team conducted a pilot or initial search before the full search to identify if the topic was susceptible to a scoping review. The full search was conducted in November 2022. One team member (MO) identified the papers in the databases. JF searched in the selected journals. Authors included studies written in English due to feasibility issues, with no time span limitation. After eliminating duplicates, two research team members (JF and IV) independently reviewed all the titles and abstracts using the exclusion and inclusion criteria described in Table 2 and with the support of the screening application StArT [ 25 ]. A third team member (AR) reviewed the titles and abstracts when the first two disagreed. The reviewer team met again at a midpoint and final stage to discuss the challenges related to study selection. Articles included for full-text review were exported to Mendeley. JF independently screened all full-text papers, and AR verified 10% for inclusion. The authors did not analyze study quality or risk of bias during study selection, which is consistent with conducting a scoping review.

The analysis of the results incorporated a descriptive summary and a thematic analysis, which was carried out to clarify and give consistency to the results' reporting [ 22 , 24 , 26 ]. Quantitative data were analyzed to report the characteristics of the studies, populations, settings, methods, and outcomes. Qualitative data were labeled, coded, and categorized into themes by three team members (JF, SH, and DS). The feedback loop framework with a sociocultural perspective was used as the theoretical framework to analyze the results.

The keywords used for the search strategies were as follows:

Clinical clerkship; feedback; formative feedback; health professions; undergraduate medical education; workplace.

Definitions of the keywords used for the present review are available in Appendix 1 .

As an example, we included the search strategy that we used in the Medline/PubMed database when conducting the full search:

("Formative Feedback"[Mesh] OR feedback) AND ("Workplace"[Mesh] OR workplace OR "Clinical Clerkship"[Mesh] OR clerkship) AND (("Education, Medical, Undergraduate"[Mesh] OR undergraduate health profession*) OR (learner* medical education)).

Inclusion and exclusion criteria

The following inclusion and exclusion criteria were used (Table 1 ):

Data extraction

The research group developed a data-charting form to organize the information obtained from the studies. The process was iterative, as the data chart was continuously reviewed and improved as necessary. In addition, following Levac et al.'s recommendation (2010), the three members involved in the charting process (JF, LI, and IV) independently reviewed the first five selected studies to determine whether the data extraction was consistent with the objectives of this scoping review and to ensure consistency. Then, the team met using web-conferencing software (Zoom; CA, USA) to review the results and adjust any details in the chart. The same three members extracted data independently from all the selected studies, considering two members reviewing each paper [ 26 ]. A third team member was consulted if any conflict occurred when extracting data. The data chart identified demographic patterns and facilitated the data synthesis. To organize data, we used a shared Excel spreadsheet, considering the following headings: title, author(s), year of publication, journal/source, country/origin, aim of the study, research question (if any), population/sample size, participants, discipline, setting, methodology, study design, data collection, data analysis, intervention, outcomes, outcomes measure, key findings, and relation of findings to research question.

Additionally, all the included papers were uploaded to AtlasTi v19 to facilitate the qualitative analysis. Three team members (JF, SH, and DS) independently coded the first six papers to create a list of codes to ensure consistency and rigor. The group met several times to discuss and refine the list of codes. Then, one member of the team (JF) used the code list to code all the rest of the papers. Once all papers were coded, the team organized codes into descriptive themes aligned with the research question.

Preliminary results were shared with a number of stakeholders (six clinical teachers, ten students, six medical educators) to elicit their opinions as an opportunity to build on the evidence and offer a greater level of meaning, content expertise, and perspective to the preliminary findings [ 26 ]. No quality appraisal of the studies is considered for this scoping review, which aligns with the frameworks for guiding scoping reviews [ 27 ].

The datasets analyzed during the current study are available from the corresponding author upon request.

A database search resulted in 3,597 papers, and the directed search of the most relevant journals in the health sciences education field yielded 2,096 titles. An example of the results of one database is available in Appendix 2 . Of the titles obtained, 816 duplicates were eliminated, and the team reviewed the titles and abstracts of 4,877 papers. Of these, 120 were selected for full-text review. Finally, 61 papers were included in this scoping review (Fig. 1 ), as listed in Table 2 .

PRISMA flow diagram for included studies, incorporating records identified through the database and direct searching

The selected studies were published between 1986 and 2022, and seventy-five percent (46) were published during the last decade. Of all the articles included in this review, 13% (8) were literature reviews: one integrative review [ 28 ] and four scoping reviews [ 29 , 30 , 31 , 32 ]. Finally, fifty-three (87%) original or empirical papers were included (i.e., studies that answered a research question or achieved a research purpose through qualitative or quantitative methodologies) [ 15 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 ].

Table 2 summarizes the papers included in the present scoping review, and Table 3 describes the characteristics of the included studies.

The thematic analysis resulted in two themes: (1) the organization of feedback processes in WBL settings, and (2) sociocultural factors influencing the organization of feedback processes. Table 4 gives a summary of the themes and subthemes.

Organization of feedback processes in WBL settings.

Setting learning goals (i.e., feed-up dimension).

Feedback that focuses on students' learning needs and is based on known performance standards enhances student response and setting learning goals [ 30 ]. Discussing goals and agreements before starting clinical practice enhances students' feedback-seeking behavior [ 39 ] and responsiveness to feedback [ 83 ]. Farrell et al. (2017) found that teacher-learner co-constructed learning goals enhance feedback interactions and help establish educational alliances, improving the learning experience [ 50 ]. However, Kiger (2020) found that sharing individualized learning plans with teachers aligned feedback with learning goals but did not improve students' perceived use of feedback [ 64 ]

Two papers of this set pointed out the importance of goal-oriented feedback, a dynamic process that depends on discussion of goal setting between teachers and students [ 50 ] and influences how individuals experience, approach, and respond to upcoming learning activities [ 34 ]. Goal-oriented feedback should be embedded in the learning experience of the clinical workplace, as it can enhance students' engagement in safe feedback dialogues [ 50 ]. Ideally, each feedback encounter in the WBL context should conclude, in addition to setting a plan of action to achieve the desired goal, with a reflection on the next goal [ 50 ].

Feedback strategies within the WBL environment. (i.e., feedback dimension)

In undergraduate WBL environments, there are several tasks and feedback opportunities organized in the undergraduate clinical workplace that can enable feedback processes:

Questions from clinical teachers to students are a feedback strategy [ 74 ]. There are different types of questions that the teacher can use, either to clarify concepts, to reach the correct answer, or to facilitate self-correction [ 74 ]. Usually, questions can be used in conjunction with other communication strategies, such as pauses, which enable self-correction by the student [ 74 ]. Students can also ask questions to obtain feedback on their performance [ 54 ]. However, question-and-answer as a feedback strategy usually provides information on either correct or incorrect answers and fewer suggestions for improvement, rendering it less constructive as a feedback strategy [ 82 ].

Direct observation of performance by default is needed to be able to provide information to be used as input in the feedback process [ 33 , 46 , 49 , 86 ]. In the process of observation, teachers can include clarification of objectives (i.e., feed-up dimension) and suggestions for an action plan (i.e., feedforward) [ 50 ]. Accordingly, Schopper et al. (2016) showed that students valued being observed while interviewing patients, as they received feedback that helped them become more efficient and effective as interviewers and communicators [ 33 ]. Moreover, it is widely described that direct observation improves feedback credibility [ 33 , 40 , 84 ]. Ideally, observation should be deliberate [ 33 , 83 ], informal or spontaneous [ 33 ], conducted by a (clinical) expert [ 46 , 86 ], provided immediately after the observation, and clinical teacher if possible, should schedule or be alert on follow-up observations to promote closing the gap between current and desired performance [ 46 ].

Workplace-based assessments (WBAs), by definition, entail direct observation of performance during authentic task demonstration [ 39 , 46 , 56 , 87 ]. WBAs can significantly impact behavioral change in medical students [ 55 ]. Organizing and designing formative WBAs and embedding these in a feedback dialogue is essential for effective learning [ 31 ].

Summative organization of WBAs is a well described barrier for feedback uptake in the clinical workplace [ 35 , 46 ]. If feedback is perceived as summative, or organized as a pass-fail decision, students may be less inclined to use the feedback for future learning [ 52 ]. According to Schopper et al. (2016), using a scale within a WBA makes students shift their focus during the clinical interaction and see it as an assessment with consequences [ 33 ]. Harrison et al. (2016) pointed out that an environment that only contains assessments with a summative purpose will not lead to a culture of learning and improving performance [ 56 ]. The recommendation is to separate the formative and summative WBAs, as feedback in summative instances is often not recognized as a learning opportunity or an instance to seek feedback [ 54 ]. In terms of the design, an organizational format is needed to clarify to students how formative assessments can promote learning from feedback [ 56 ]. Harrison et al. (2016) identified that enabling students to have more control over their assessments, designing authentic assessments, and facilitating long-term mentoring could improve receptivity to formative assessment feedback [ 56 ].

Multiple WBA instruments and systems are reported in the literature. Sox et al. (2014) used a detailed evaluation form to help students improve their clinical case presentation skills. They found that feedback on oral presentations provided by supervisors using a detailed evaluation form improved clerkship students’ oral presentation skills [ 78 ]. Daelmans et al. (2006) suggested that a formal in-training assessment programme composed by 19 assessments that provided structured feedback, could promote observation and verbal feedback opportunities through frequent assessments [ 43 ]. However, in this setting, limited student-staff interactions still hindered feedback follow-up [ 43 ]. Designing frequent WBA improves feedback credibility [ 28 ]. Long et al. (2021) emphasized that students' responsiveness to assessment feedback hinges on its perceived credibility, underlining the importance of credibility for students to effectively engage and improve their performance [ 31 ].

The mini-CEX is one of the most widely described WBA instruments in the literature. Students perceive that the mini-CEX allows them to be observed and encourages the development of interviewing skills [ 33 ]. The mini-CEX can provide feedback that improves students' clinical skills [ 58 , 60 ], as it incorporates a structure for discussing the student's strengths and weaknesses and the design of a written action plan [ 39 , 80 ]. When mini-CEXs are incorporated as part of a system of WBA, such as programmatic assessment, students feel confident in seeking feedback after observation, and being systematic allows for follow-up [ 39 ]. Students suggested separating grading from observation and using the mini-CEX in more informal situations [ 33 ].

Clinical encounter cards allow students to receive weekly feedback and make them request more feedback as the clerkship progresses [ 65 ]. Moreover, encounter cards stimulate that feedback is given by supervisors, and students are more satisfied with the feedback process [ 72 ]. With encounter card feedback, students are responsible for asking a supervisor for feedback before a clinical encounter, and supervisors give students written and verbal comments about their performance after the encounter [ 42 , 72 ]. Encounter cards enhance the use of feedback and add approximately one minute to the length of the clinical encounter, so they are well accepted by students and supervisors [ 72 ]. Bennett (2006) identified that Instant Feedback Cards (IFC) facilitated mid-rotation feedback [ 38 ]. Feedback encounter card comments must be discussed between students and supervisors; otherwise, students may perceive it as impersonal, static, formulaic, and incomplete [ 59 ].

Self-assessments can change students' feedback orientation, transforming them into coproducers of learning [ 68 ]. Self-assessments promote the feedback process [ 68 ]. Some articles emphasize the importance of organizing self-assessments before receiving feedback from supervisors, for example, discussing their appraisal with the supervisor [ 46 , 52 ]. In designing a feedback encounter, starting with a self-assessment as feed-up, discussing with the supervisor, and identifying areas for improvement is recommended, as part of the feedback dialogue [ 68 ].

Peer feedback as an organized activity allows students to develop strategies to observe and give feedback to other peers [ 61 ]. Students can act as the feedback provider or receiver, fostering understanding of critical comments and promoting evaluative judgment for their clinical practice [ 61 ]. Within clerkships, enabling the sharing of feedback information among peers allows for a better understanding and acceptance of feedback [ 52 ]. However, students can find it challenging to take on the peer assessor/feedback provider role, as they prefer to avoid social conflicts [ 28 , 61 ]. Moreover, it has been described that they do not trust the judgment of their peers because they are not experts, although they know the procedures, tasks, and steps well and empathize with their peer status in the learning process [ 61 ].

Bedside-teaching encounters (BTEs) provide timely feedback and are an opportunity for verbal feedback during performance [ 74 ]. Rizan et al. (2014) explored timely feedback delivered within BTEs and determined that it promotes interaction that constructively enhances learner development through various corrective strategies (e.g., question and answers, pauses, etc.). However, if the feedback given during the BTEs was general, unspecific, or open-ended, it could go unnoticed [ 74 ]. Torre et al. (2005) investigated which integrated feedback activities and clinical tasks occurred on clerkship rotations and assessed students' perceived quality in each teaching encounter [ 81 ]. The feedback activities reported were feedback on written clinical history, physical examination, differential diagnosis, oral case presentation, a daily progress note, and bedside feedback. Students considered all these feedback activities high-quality learning opportunities, but they were more likely to receive feedback when teaching was at the bedside than at other teaching locations [ 81 ].

Case presentations are an opportunity for feedback within WBL contexts [ 67 , 73 ]. However, both students and supervisors struggled to identify them as feedback moments, and they often dismissed questions and clarifications around case presentations as feedback [ 73 ]. Joshi (2017) identified case presentations as a way for students to ask for informal or spontaneous supervisor feedback [ 63 ].

Organization of follow-up feedback and action plans (i.e., feedforward dimension).

Feedback that generates use and response from students is characterized by two-way communication and embedded in a dialogue [ 30 ]. Feedback must be future-focused [ 29 ], and a feedback encounter should be followed by planning the next observation [ 46 , 87 ]. Follow-up feedback could be organized as a future self-assessment, reflective practice by the student, and/or a discussion with the supervisor or coach [ 68 ]. The literature describes that a lack of student interaction with teachers makes follow-up difficult [ 43 ]. According to Haffling et al. (2011), follow-up feedback sessions improve students' satisfaction with feedback compared to students who do not have follow-up sessions. In addition, these same authors reported that a second follow-up session allows verification of improved performances or confirmation that the skill was acquired [ 55 ].

Although feedback encounter forms are a recognized way of obtaining information about performance (i.e., feedback dimension), the literature does not provide many clear examples of how they may impact the feedforward phase. For example, Joshi et al. (2016) consider a feedback form with four fields (i.e., what did you do well, advise the student on what could be done to improve performance, indicate the level of proficiency, and personal details of the tutor). In this case, the supervisor highlighted what the student could improve but not how, which is the missing phase of the co-constructed action plan [ 63 ]. Whichever WBA instrument is used in clerkships to provide feedback, it should include a "next steps" box [ 44 ], and it is recommended to organize a long-term use of the WBA instrument so that those involved get used to it and improve interaction and feedback uptake [ 55 ]. RIME-based feedback (Reporting, Interpreting, Managing, Educating) is considered an interesting example, as it is perceived as helpful to students in knowing what they need to improve in their performance [ 44 ]. Hochberg (2017) implemented formative mid-clerkship assessments to enhance face-to-face feedback conversations and co-create an improvement plan [ 59 ]. Apps for structuring and storing feedback improve the amount of verbal and written feedback. In the study of Joshi et al. (2016), a reasonable proportion of students (64%) perceived that these app tools help them improve their performance during rotations [ 63 ].

Several studies indicate that an action plan as part of the follow-up feedback is essential for performance improvement and learning [ 46 , 55 , 60 ]. An action plan corresponds to an agreed-upon strategy for improving, confirming, or correcting performance. Bing-You et al. (2017) determined that only 12% of the articles included in their scoping review incorporated an action plan for learners [ 32 ]. Holmboe et al. (2004) reported that only 11% of the feedback sessions following a mini-CEX included an action plan [ 60 ]. Suhoyo et al. (2017) also reported that only 55% of mini-CEX encounters contained an action plan [ 80 ]. Other authors reported that action plans are not commonly offered during feedback encounters [ 77 ]. Sokol-Hessner et al. (2010) implemented feedback card comments with a space to provide written feedback and a specific action plan. In their results, 96% contained positive comments, and only 5% contained constructive comments [ 77 ]. In summary, although the recommendation is to include a “next step” box in the feedback instruments, evidence shows these items are not often used for constructive comments or action plans.

Sociocultural factors influencing the organization of feedback processes.

Multiple sociocultural factors influence interaction in feedback encounters, promoting or hampering the productivity of the feedback processes.

Clinical learning culture

Context impacts feedback processes [ 30 , 82 ], and there are barriers to incorporating actionable feedback in the clinical learning context. The clinical learning culture is partly determined by the clinical context, which can be unpredictable [ 29 , 46 , 68 ], as the available patients determine learning opportunities. Supervisors are occupied by a high workload, which results in limited time or priority for teaching [ 35 , 46 , 48 , 55 , 68 , 83 ], hindering students’ feedback-seeking behavior [ 54 ], and creating a challenge for the balance between patient care and student mentoring [ 35 ].

Clinical workplace culture does not always purposefully prioritize instances for feedback processes [ 83 , 84 ]. This often leads to limited direct observation [ 55 , 68 ] and the provision of poorly informed feedback. It is also evident that this affects trust between clinical teachers and students [ 52 ]. Supervisors consider feedback a low priority in clinical contexts [ 35 ] due to low compensation and lack of protected time [ 83 ]. In particular, lack of time appears to be the most significant and well-known barrier to frequent observation and workplace feedback [ 35 , 43 , 48 , 62 , 67 , 83 ].

The clinical environment is hierarchical [ 68 , 80 ] and can make students not consider themselves part of the team and feel like a burden to their supervisor [ 68 ]. This hierarchical learning environment can lead to unidirectional feedback, limit dialogue during feedback processes, and hinder the seeking, uptake, and use of feedback [ 67 , 68 ]. In a learning culture where feedback is not supported, learners are less likely to want to seek it and feel motivated and engaged in their learning [ 83 ]. Furthermore, it has been identified that clinical supervisors lack the motivation to teach [ 48 ] and the intention to observe or reobserve performance [ 86 ].

In summary, the clinical context and WBL culture do not fully use the potential of a feedback process aimed at closing learning gaps. However, concrete actions shown in the literature can be taken to improve the effectiveness of feedback by organizing the learning context. For example, McGinness et al. (2022) identified that students felt more receptive to feedback when working in a safe, nonjudgmental environment [ 67 ]. Moreover, supervisors and trainees identified the learning culture as key to establishing an open feedback dialogue [ 73 ]. Students who perceive culture as supportive and formative can feel more comfortable performing tasks and more willing to receive feedback [ 73 ].

Relationships

There is a consensus in the literature that trusting and long-term relationships improve the chances of actionable feedback. However, relationships between supervisors and students in the clinical workplace are often brief and not organized as more longitudinally [ 68 , 83 ], leaving little time to establish a trustful relationship [ 68 ]. Supervisors change continuously, resulting in short interactions that limit the creation of lasting relationships over time [ 50 , 68 , 83 ]. In some contexts, it is common for a student to have several supervisors who have their own standards in the observation of performance [ 46 , 56 , 68 , 83 ]. A lack of stable relationships results in students having little engagement in feedback [ 68 ]. Furthermore, in case of summative assessment programmes, the dual role of supervisors (i.e., assessing and giving feedback) makes feedback interactions perceived as summative and can complicate the relationship [ 83 ].

Repeatedly, the articles considered in this review describe that long-term and stable relationships enable the development of trust and respect [ 35 , 62 ] and foster feedback-seeking behavior [ 35 , 67 ] and feedback-giver behavior [ 39 ]. Moreover, constructive and positive relationships enhance students´ use of and response to feedback [ 30 ]. For example, Longitudinal Integrated Clerkships (LICs) promote stable relationships, thus enhancing the impact of feedback [ 83 ]. In a long-term trusting relationship, feedback can be straightforward and credible [ 87 ], there are more opportunities for student observation, and the likelihood of follow-up and actionable feedback improves [ 83 ]. Johnson et al. (2020) pointed out that within a clinical teacher-student relationship, the focus must be on establishing psychological safety; thus, the feedback conversations might be transformed [ 62 ].

Stable relationships enhance feedback dialogues, which offer an opportunity to co-construct learning and propose and negotiate aspects of the design of learning strategies [ 62 ].

Students as active agents in the feedback processes

The feedback response learners generate depends on the type of feedback information they receive, how credible the source of feedback information is, the relationship between the receiver and the giver, and the relevance of the information delivered [ 49 ]. Garino (2020) noted that students who are most successful in using feedback are those who do not take criticism personally, who understand what they need to improve and know they can do so, who value and feel meaning in criticism, are not surprised to receive it, and who are motivated to seek new feedback and use effective learning strategies [ 52 ]. Successful users of feedback ask others for help, are intentional about their learning, know what resources to use and when to use them, listen to and understand a message, value advice, and use effective learning strategies. They regulate their emotions, find meaning in the message, and are willing to change [ 52 ].

Student self-efficacy influences the understanding and use of feedback in the clinical workplace. McGinness et al. (2022) described various positive examples of self-efficacy regarding feedback processes: planning feedback meetings with teachers, fostering good relationships with the clinical team, demonstrating interest in assigned tasks, persisting in seeking feedback despite the patient workload, and taking advantage of opportunities for feedback, e.g., case presentations [ 67 ].

When students are encouraged to seek feedback aligned with their own learning objectives, they promote feedback information specific to what they want to learn and improve and enhance the use of feedback [ 53 ]. McGinness et al. (2022) identified that the perceived relevance of feedback information influenced the use of feedback because students were more likely to ask for feedback if they perceived that the information was useful to them. For example, if students feel part of the clinical team and participate in patient care, they are more likely to seek feedback [ 17 ].

Learning-oriented students aim to seek feedback to achieve clinical competence at the expected level [ 75 ]; they focus on improving their knowledge and skills and on professional development [ 17 ]. Performance-oriented students aim not to fail and to avoid negative feedback [ 17 , 75 ].

For effective feedback processes, including feed-up, feedback, and feedforward, the student must be feedback-oriented, i.e., active, seeking, listening to, interpreting, and acting on feedback [ 68 ]. The literature shows that feedback-oriented students are coproducers of learning [ 68 ] and are more involved in the feedback process [ 51 ]. Additionally, students who are metacognitively aware of their learning process are more likely to use feedback to reduce gaps in learning and performance [ 52 ]. For this, students must recognize feedback when it occurs and understand it when they receive it. Thus, it is important to organize training and promote feedback literacy so that students understand what feedback is, act on it, and improve the quality of feedback and their learning plans [ 68 ].

Table 5 summarizes those feedback tasks, activities, and key features of organizational aspects that enable each phase of the feedback loop based on the literature review.

The present scoping review identified 61 papers that mapped the literature on feedback processes in the WBL environments of undergraduate health professions. This review explored how feedback processes are organized in these learning contexts using the feedback loop framework. Given the specific characteristics of feedback processes in undergraduate clinical learning, three main findings were identified on how feedback processes are being conducted in the clinical environment and how these processes could be organized to support feedback processes.

First, the literature lacks a balance between the three dimensions of the feedback loop. In this regard, most of the articles in this review focused on reporting experiences or strategies for delivering feedback information (i.e., feedback dimension). Credible and objective feedback information is based on direct observation [ 46 ] and occurs within an interaction or a dialogue [ 62 , 88 ]. However, only having credible and objective information does not ensure that it will be considered, understood, used, and put into practice by the student [ 89 ].

Feedback-supporting actions aligned with goals and priorities facilitate effective feedback processes [ 89 ] because goal-oriented feedback focuses on students' learning needs [ 7 ]. In contrast, this review showed that only a minority of the studies highlighted the importance of aligning learning objectives and feedback (i.e., the feed-up dimension). To overcome this, supervisors and students must establish goals and agreements before starting clinical practice, as it allows students to measure themselves on a defined basis [ 90 , 91 ] and enhances students' feedback-seeking behavior [ 39 , 92 ] and responsiveness to feedback [ 83 ]. In addition, learning goals should be shared, and co-constructed, through a dialogue [ 50 , 88 , 90 , 92 ]. In fact, relationship-based feedback models emphasize setting shared goals and plans as part of the feedback process [ 68 ].

Many of the studies acknowledge the importance of establishing an action plan and promoting the use of feedback (i.e., feedforward). However, there is yet limited insight on how to best implement strategies that support the use of action plans, improve performance and close learning gaps. In this regard, it is described that delivering feedback without perceiving changes, results in no effect or impact on learning [ 88 ]. To determine if a feedback loop is closed, observing a change in the student's response is necessary. In other words, feedback does not work without repeating the same task [ 68 ], so teachers need to observe subsequent tasks to notice changes [ 88 ]. While feedforward is fundamental to long-term performance, it is shown that more research is needed to determine effective actions to be implemented in the WBL environment to close feedback loops.

Second, there is a need for more knowledge about designing feedback activities in the WBL environment that will generate constructive feedback for learning. WBA is the most frequently reported feedback activity in clinical workplace contexts [ 39 , 46 , 56 , 87 ]. Despite the efforts of some authors to use WBAs as a formative assessment and feedback opportunity, in several studies, a summative component of the WBA was presented as a barrier to actionable feedback [ 33 , 56 ]. Students suggest separating grading from observation and using, for example, the mini-CEX in informal situations [ 33 ]. Several authors also recommend disconnecting the summative components of WBAs to avoid generating emotions that can limit the uptake and use of feedback [ 28 , 93 ]. Other literature recommends purposefully designing a system of assessment using low-stakes data points for feedback and learning. Accordingly, programmatic assessment is a framework that combines both the learning and the decision-making function of assessment [ 94 , 95 ]. Programmatic assessment is a practical approach for implementing low-stakes as a continuum, giving opportunities to close the gap between current and desired performance and having the student as an active agent [ 96 ]. This approach enables the incorporation of low-stakes data points that target student learning [ 93 ] and provide performance-relevant information (i.e., meaningful feedback) based on direct observations during authentic professional activities [ 46 ]. Using low-stakes data points, learners make sense of information about their performance and use it to enhance the quality of their work or performance [ 96 , 97 , 98 ]. Implementing multiple instances of feedback is more effective than providing it once because it promotes closing feedback loops by giving the student opportunities to understand the feedback, make changes, and see if those changes were effective [ 89 ].

Third, the support provided by the teacher is fundamental and should be built into a reliable and long-term relationship, where the teacher must take the role of coach rather than assessor, and students should develop feedback agency and be active in seeking and using feedback to improve performance. Although it is recognized that institutional efforts over the past decades have focused on training teachers to deliver feedback, clinical supervisors' lack of teaching skills is still identified as a barrier to workplace feedback [ 99 ]. In particular, research indicates that clinical teachers lack the skills to transform the information obtained from an observation into constructive feedback [ 100 ]. Students are more likely to use feedback if they consider it credible and constructive [ 93 ] and based on stable relationships [ 93 , 99 , 101 ]. In trusting relationships, feedback can be straightforward and credible, and the likelihood of follow-up and actionable feedback improves [ 83 , 88 ]. Coaching strategies can be enhanced by teachers building an educational alliance that allows for trustworthy relationships or having supervisors with an exclusive coaching role [ 14 , 93 , 102 ].

Last, from a sociocultural perspective, individuals are the main actors in the learning process. Therefore, feedback impacts learning only if students engage and interact with it [ 11 ]. Thus, feedback design and student agency appear to be the main features of effective feedback processes. Accordingly, the present review identified that feedback design is a key feature for effective learning in complex environments such as WBL. Feedback in the workplace must ideally be organized and implemented to align learning outcomes, learning activities, and assessments, allowing learners to learn, practice, and close feedback loops [ 88 ]. To guide students toward performances that reflect long-term learning, an intensive formative learning phase is needed, in which multiple feedback processes are included that shape students´ further learning [ 103 ]. This design would promote student uptake of feedback for subsequent performance [ 1 ].

Strengths and limitations

The strengths of this study are (1) the use of an established framework, the Arksey and O'Malley's framework [ 22 ]. We included the step of socializing the results with stakeholders, which allowed the team to better understand the results from another perspective and offer a realistic look. (2) Using the feedback loop as a theoretical framework strengthened the results and gave a more thorough explanation of the literature regarding feedback processes in the WBL context. (3) our team was diverse and included researchers from different disciplines as well as a librarian.

The present scoping review has several limitations. Although we adhered to the recommended protocols and methodologies, some relevant papers may have been omitted. The research team decided to select original studies and reviews of the literature for the present scoping review. This caused some articles, such as guidelines, perspectives, and narrative papers, to be excluded from the current study.

One of the inclusion criteria was a focus on undergraduate students. However, some papers that incorporated undergraduate and postgraduate participants were included, as these supported the results of this review. Most articles involved medical students. Although the authors did not limit the search to medicine, maybe some articles involving students from other health disciplines needed to be included, considering the search in other databases or journals.

The results give insight in how feedback could be organized within the clinical workplace to promote feedback processes. On a small scale, i.e., in the feedback encounter between a supervisor and a learner, feedback should be organized to allow for follow-up feedback, thus working on required learning and performance goals. On a larger level, i.e., in the clerkship programme or a placement rotation, feedback should be organized through appropriate planning of subsequent tasks and activities.

More insight is needed in designing a closed loop feedback process, in which specific attention is needed in effective feedforward practices. The feedback that stimulates further action and learning requires a safe and trustful work and learning environment. Understanding the relationship between an individual and his or her environment is a challenge for determining the impact of feedback and must be further investigated within clinical WBL environments. Aligning the dimensions of feed-up, feedback and feedforward includes careful attention to teachers’ and students’ feedback literacy to assure that students can act on feedback in a constructive way. In this line, how to develop students' feedback agency within these learning environments needs further research.

Boud D, Molloy E. Rethinking models of feedback for learning: The challenge of design. Assess Eval High Educ. 2013;38:698–712.

Article Google Scholar

Henderson M, Ajjawi R, Boud D, Molloy E. Identifying feedback that has impact. In: The Impact of Feedback in Higher Education. Springer International Publishing: Cham; 2019. p. 15–34.

Chapter Google Scholar

Winstone N, Carless D. Designing effective feedback processes in higher education: A learning-focused approach. 1st ed. New York: Routledge; 2020.

Google Scholar

Ajjawi R, Boud D. Assessment & Evaluation in Higher Education Researching feedback dialogue: an interactional analysis approach. 2015. https://doi.org/10.1080/02602938.2015.1102863 .

Carless D. Feedback loops and the longer-term: towards feedback spirals. Assess Eval High Educ. 2019;44:705–14.

Sadler DR. Formative assessment and the design of instructional systems. Instr Sci. 1989;18:119–44.

Hattie J, Timperley H. The Power of Feedback The Meaning of Feedback. Rev Educ Res. 2007;77:81–112.

Zarrinabadi N, Rezazadeh M. Why only feedback? Including feed up and feed forward improves nonlinguistic aspects of L2 writing. Language Teaching Research. 2023;27(3):575–92.

Fisher D, Frey N. Feed up, back, forward. Educ Leadersh. 2009;67:20–5.

Reimann A, Sadler I, Sambell K. What’s in a word? Practices associated with ‘feedforward’ in higher education. Assessment evaluation in higher education. 2019;44:1279–90.

Esterhazy R. Re-conceptualizing Feedback Through a Sociocultural Lens. In: Henderson M, Ajjawi R, Boud D, Molloy E, editors. The Impact of Feedback in Higher Education. Cham: Palgrave Macmillan; 2019. https://doi.org/10.1007/978-3-030-25112-3_5 .

Bransen D, Govaerts MJB, Sluijsmans DMA, Driessen EW. Beyond the self: The role of co-regulation in medical students’ self-regulated learning. Med Educ. 2020;54:234–41.

Ramani S, Könings KD, Ginsburg S, Van Der Vleuten CP. Feedback Redefined: Principles and Practice. J Gen Intern Med. 2019;34:744–53.

Atkinson A, Watling CJ, Brand PL. Feedback and coaching. Eur J Pediatr. 2022;181(2):441–6.

Suhoyo Y, Schonrock-Adema J, Emilia O, Kuks JBM, Cohen-Schotanus JA. Clinical workplace learning: perceived learning value of individual and group feedback in a collectivistic culture. BMC Med Educ. 2018;18:79.

Bowen L, Marshall M, Murdoch-Eaton D. Medical Student Perceptions of Feedback and Feedback Behaviors Within the Context of the “Educational Alliance.” Acad Med. 2017;92:1303–12.

Bok HGJ, Teunissen PW, Spruijt A, Fokkema JPI, van Beukelen P, Jaarsma DADC, et al. Clarifying students’ feedback-seeking behaviour in clinical clerkships. Med Educ. 2013;47:282–91.

Al-Kadri HM, Al-Kadi MT, Van Der Vleuten CPM. Workplace-based assessment and students’ approaches to learning: A qualitative inquiry. Med Teach. 2013;35(SUPPL):1.

Dennis AA, Foy MJ, Monrouxe LV, Rees CE. Exploring trainer and trainee emotional talk in narratives about workplace-based feedback processes. Adv Health Sci Educ. 2018;23:75–93.

Watling C, LaDonna KA, Lingard L, Voyer S, Hatala R. ‘Sometimes the work just needs to be done’: socio-cultural influences on direct observation in medical training. Med Educ. 2016;50:1054–64.

Bing-You R, Hayes V, Varaklis K, Trowbridge R, Kemp H, McKelvy D. Feedback for Learners in Medical Education: What is Known? A Scoping Review Academic Medicine. 2017;92:1346–54.

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8:19–32.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Ann Intern Med. 2018;169:467–73.

Colquhoun HL, Levac D, O’brien KK, Straus S, Tricco AC, Perrier L, et al. Scoping reviews: time for clarity in definition methods and reporting. J Clin Epidemiol. 2014;67:1291–4.

StArt - State of Art through Systematic Review. 2013.

Levac D, Colquhoun H, O’Brien KK. Scoping studies: Advancing the methodology. Implementation Science. 2010;5:1–9.

Peters MDJ, BPharm CMGHK, Parker PMD, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid Based Healthc. 2015;13:141–6.

Bing-You R, Varaklis K, Hayes V, Trowbridge R, Kemp H, McKelvy D, et al. The Feedback Tango: An Integrative Review and Analysis of the Content of the Teacher-Learner Feedback Exchange. Acad Med. 2018;93:657–63.

Ossenberg C, Henderson A, Mitchell M. What attributes guide best practice for effective feedback? A scoping review. Adv Health Sci Educ. 2019;24:383–401.

Spooner M, Duane C, Uygur J, Smyth E, Marron B, Murphy PJ, et al. Self -regulatory learning theory as a lens on how undergraduate and postgraduate learners respond to feedback: A BEME scoping review: BEME Guide No. 66. Med Teach. 2022;44:3–18.

Long S, Rodriguez C, St-Onge C, Tellier PP, Torabi N, Young M. Factors affecting perceived credibility of assessment in medical education: A scoping review. Adv Health Sci Educ. 2022;27:229–62.

Bing-You R, Hayes V, Varaklis K, Trowbridge R, Kemp H, McKelvy D. Feedback for Learners in Medical Education: What is Known? A Scoping Review: Lippincott Williams and Wilkins; 2017.

Schopper H, Rosenbaum M, Axelson R. “I wish someone watched me interview:” medical student insight into observation and feedback as a method for teaching communication skills during the clinical years. BMC Med Educ. 2016;16:286.

Crommelinck M, Anseel F. Understanding and encouraging feedback-seeking behaviour: a literature review. Med Educ. 2013;47:232–41.

Adamson E, Kinga L, Foy L, McLeodb M, Traynor J, Watson W, et al. Feedback in clinical practice: Enhancing the students’ experience through action research. Nurse Educ Pract. 2018;31:48–53.

Al-Mously N, Nabil NM, Al-Babtain SA, et al. Undergraduate medical students’ perceptions on the quality of feedback received during clinical rotations. Med Teach. 2014;36(Supplement 1):S17-23.

Bates J, Konkin J, Suddards C, Dobson S, Pratt D. Student perceptions of assessment and feedback in longitudinal integrated clerkships. Med Educ. 2013;47:362–74.

Bennett AJ, Goldenhar LM, Stanford K. Utilization of a Formative Evaluation Card in a Psychiatry Clerkship. Acad Psychiatry. 2006;30:319–24.

Bok HG, Jaarsma DA, Spruijt A, Van Beukelen P, Van Der Vleuten CP, Teunissen PW, et al. Feedback-giving behaviour in performance evaluations during clinical clerkships. Med Teach. 2016;38:88–95.

Bok HG, Teunissen PW, Spruijt A, Fokkema JP, van Beukelen P, Jaarsma DA, et al. Clarifying students’ feedback-seeking behaviour in clinical clerkships. Med Educ. 2013;47:282–91.

Calleja P, Harvey T, Fox A, Carmichael M, et al. Feedback and clinical practice improvement: A tool to assist workplace supervisors and students. Nurse Educ Pract. 2016;17:167–73.

Carey EG, Wu C, Hur ES, Hasday SJ, Rosculet NP, Kemp MT, et al. Evaluation of Feedback Systems for the Third-Year Surgical Clerkship. J Surg Educ. 2017;74:787–93.

Daelmans HE, Overmeer RM, Van der Hem-Stokroos HH, Scherpbier AJ, Stehouwer CD, van der Vleuten CP. In-training assessment: qualitative study of effects on supervision and feedback in an undergraduate clinical rotation. Medical education. 2006;40(1):51–8.

DeWitt D, Carline J, Paauw D, Pangaro L. Pilot study of a ’RIME’-based tool for giving feedback in a multi-specialty longitudinal clerkship. Med Educ. 2008;42:1205–9.

Dolan BM, O’Brien CL, Green MM. Including Entrustment Language in an Assessment Form May Improve Constructive Feedback for Student Clinical Skills. Med Sci Educ. 2017;27:461–4.

Duijn CC, Welink LS, Mandoki M, Ten Cate OT, Kremer WD, Bok HG. Am I ready for it? Students’ perceptions of meaningful feedback on entrustable professional activities. Perspectives on medical education. 2017;6:256–64.

Elnicki DM, Zalenski D. Integrating medical students’ goals, self-assessment and preceptor feedback in an ambulatory clerkship. Teach Learn Med. 2013;25:285–91.

Embo MP, Driessen EW, Valcke M, Van der Vleuten CP. Assessment and feedback to facilitate self-directed learning in clinical practice of Midwifery students. Medical teacher. 2010;32(7):e263-9.

Eva KW, Armson H, Holmboe E, Lockyer J, Loney E, Mann K, et al. Factors influencing responsiveness to feedback: On the interplay between fear, confidence, and reasoning processes. Adv Health Sci Educ. 2012;17:15–26.

Farrell L, Bourgeois-Law G, Ajjawi R, Regehr G. An autoethnographic exploration of the use of goal oriented feedback to enhance brief clinical teaching encounters. Adv Health Sci Educ. 2017;22:91–104.

Fernando N, Cleland J, McKenzie H, Cassar K. Identifying the factors that determine feedback given to undergraduate medical students following formative mini-CEX assessments. Med Educ. 2008;42:89–95.

Garino A. Ready, willing and able: a model to explain successful use of feedback. Adv Health Sci Educ. 2020;25:337–61.

Garner MS, Gusberg RJ, Kim AW. The positive effect of immediate feedback on medical student education during the surgical clerkship. J Surg Educ. 2014;71:391–7.

Bing-You R, Hayes V, Palka T, Ford M, Trowbridge R. The Art (and Artifice) of Seeking Feedback: Clerkship Students’ Approaches to Asking for Feedback. Acad Med. 2018;93:1218–26.

Haffling AC, Beckman A, Edgren G. Structured feedback to undergraduate medical students: 3 years’ experience of an assessment tool. Medical teacher. 2011;33(7):e349-57.

Harrison CJ, Könings KD, Dannefer EF, Schuwirth LWTT, Wass V, van der Vleuten CPMM. Factors influencing students’ receptivity to formative feedback emerging from different assessment cultures. Perspect Med Educ. 2016;5:276–84.

Harrison CJ, Könings KD, Schuwirth LW, Wass V, Van der Vleuten CP, Konings KD, et al. Changing the culture of assessment: the dominance of the summative assessment paradigm. BMC medical education. 2017;17:1–4.

Harvey P, Radomski N, O’Connor D. Written feedback and continuity of learning in a geographically distributed medical education program. Medical teacher. 2013;35(12):1009–13.

Hochberg M, Berman R, Ogilvie J, Yingling S, Lee S, Pusic M, et al. Midclerkship feedback in the surgical clerkship: the “Professionalism, Reporting, Interpreting, Managing, Educating, and Procedural Skills” application utilizing learner self-assessment. Am J Surg. 2017;213:212–6.

Holmboe ES, Yepes M, Williams F, Huot SJ. Feedback and the mini clinical evaluation exercise. Journal of general internal medicine. 2004;19(5):558–61.

Tai JHM, Canny BJ, Haines TP, Molloy EK. The role of peer-assisted learning in building evaluative judgement: opportunities in clinical medical education. Adv Health Sci Educ. 2016;21:659–76.

Johnson CE, Keating JL, Molloy EK. Psychological safety in feedback: What does it look like and how can educators work with learners to foster it? Med Educ. 2020;54:559–70.

Joshi A, Generalla J, Thompson B, Haidet P. Facilitating the Feedback Process on a Clinical Clerkship Using a Smartphone Application. Acad Psychiatry. 2017;41:651–5.

Kiger ME, Riley C, Stolfi A, Morrison S, Burke A, Lockspeiser T. Use of Individualized Learning Plans to Facilitate Feedback Among Medical Students. Teach Learn Med. 2020;32:399–409.

Kogan J, Shea J. Implementing feedback cards in core clerkships. Med Educ. 2008;42:1071–9.

Lefroy J, Walters B, Molyneux A, Smithson S. Can learning from workplace feedback be enhanced by reflective writing? A realist evaluation in UK undergraduate medical education. Educ Prim Care. 2021;32:326–35.

McGinness HT, Caldwell PHY, Gunasekera H, Scott KM. ‘Every Human Interaction Requires a Bit of Give and Take’: Medical Students’ Approaches to Pursuing Feedback in the Clinical Setting. Teach Learn Med. 2022. https://doi.org/10.1080/10401334.2022.2084401 .

Noble C, Billett S, Armit L, Collier L, Hilder J, Sly C, et al. ``It’s yours to take{’’}: generating learner feedback literacy in the workplace. Adv Health Sci Educ. 2020;25:55–74.

Ogburn T, Espey E. The R-I-M-E method for evaluation of medical students on an obstetrics and gynecology clerkship. Am J Obstet Gynecol. 2003;189:666–9.

Po O, Reznik M, Greenberg L. Improving a medical student feedback with a clinical encounter card. Ambul Pediatr. 2007;7:449–52.

Parkes J, Abercrombie S, McCarty T, Parkes J, Abercrombie S, McCarty T. Feedback sandwiches affect perceptions but not performance. Adv Health Sci Educ. 2013;18:397–407.

Paukert JL, Richards ML, Olney C. An encounter card system for increasing feedback to students. Am J Surg. 2002;183:300–4.

Rassos J, Melvin LJ, Panisko D, Kulasegaram K, Kuper A. Unearthing Faculty and Trainee Perspectives of Feedback in Internal Medicine: the Oral Case Presentation as a Model. J Gen Intern Med. 2019;34:2107–13.

Rizan C, Elsey C, Lemon T, Grant A, Monrouxe L. Feedback in action within bedside teaching encounters: a video ethnographic study. Med Educ. 2014;48:902–20.

Robertson AC, Fowler LC. Medical student perceptions of learner-initiated feedback using a mobile web application. Journal of medical education and curricular development. 2017;4:2382120517746384.

Scheidt PC, Lazoritz S, Ebbeling WL, Figelman AR, Moessner HF, Singer JE. Evaluation of system providing feedback to students on videotaped patient encounters. J Med Educ. 1986;61(7):585–90.

Sokol-Hessner L, Shea J, Kogan J. The open-ended comment space for action plans on core clerkship students’ encounter cards: what gets written? Acad Med. 2010;85:S110–4.

Sox CM, Dell M, Phillipi CA, Cabral HJ, Vargas G, Lewin LO. Feedback on oral presentations during pediatric clerkships: a randomized controlled trial. Pediatrics. 2014;134:965–71.

Spickard A, Gigante J, Stein G, Denny JC. Automatic capture of student notes to augment mentor feedback and student performance on patient write-ups. J Gen Intern Med. 2008;23:979–84.

Suhoyo Y, Van Hell EA, Kerdijk W, Emilia O, Schönrock-Adema J, Kuks JB, et al. nfluence of feedback characteristics on perceived learning value of feedback in clerkships: does culture matter? BMC medical education. 2017;17:1–7.

Torre DM, Simpson D, Sebastian JL, Elnicki DM. Learning/feedback activities and high-quality teaching: perceptions of third-year medical students during an inpatient rotation. Acad Med. 2005;80:950–4.

Urquhart LM, Ker JS, Rees CE. Exploring the influence of context on feedback at medical school: a video-ethnography study. Adv Health Sci Educ. 2018;23:159–86.

Watling C, Driessen E, van der Vleuten C, Lingard L. Learning culture and feedback: an international study of medical athletes and musicians. Med Educ. 2014;48:713–23.

Watling C, Driessen E, van der Vleuten C, Vanstone M, Lingard L. Beyond individualism: Professional culture and its influence on feedback. Med Educ. 2013;47:585–94.

Soemantri D, Dodds A, Mccoll G. Examining the nature of feedback within the Mini Clinical Evaluation Exercise (Mini-CEX): an analysis of 1427 Mini-CEX assessment forms. GMS J Med Educ. 2018;35:Doc47.

Van De Ridder JMMM, Stokking KM, McGaghie WC, ten Cate OTJ, van der Ridder JM, Stokking KM, et al. What is feedback in clinical education? Med Educ. 2008;42:189–97.

van de Ridder JMMM, McGaghie WC, Stokking KM, ten Cate OTJJ. Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review. Med Educ. 2015;49:658–73.

Boud D. Feedback: ensuring that it leads to enhanced learning. Clin Teach. 2015. https://doi.org/10.1111/tct.12345 .

Brehaut J, Colquhoun H, Eva K, Carrol K, Sales A, Michie S, et al. Practice feedback interventions: 15 suggestions for optimizing effectiveness. Ann Intern Med. 2016;164:435–41.

Ende J. Feedback in clinical medical education. J Am Med Assoc. 1983;250:777–81.

Cantillon P, Sargeant J. Giving feedback in clinical settings. Br Med J. 2008;337(7681):1292–4.

Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007;29:855–71.

Watling CJ, Ginsburg S. Assessment, feedback and the alchemy of learning. Med Educ. 2019;53:76–85.

van der Vleuten CPM, Schuwirth LWT, Driessen EW, Dijkstra J, Tigelaar D, Baartman LKJ, et al. A model for programmatic assessment fit for purpose. Med Teach. 2012;34:205–14.

Schuwirth LWT, der Vleuten CPM. Programmatic assessment: from assessment of learning to assessment for learning. Med Teach. 2011;33:478–85.

Schut S, Driessen E, van Tartwijk J, van der Vleuten C, Heeneman S. Stakes in the eye of the beholder: an international study of learners’ perceptions within programmatic assessment. Med Educ. 2018;52:654–63.

Henderson M, Boud D, Molloy E, Dawson P, Phillips M, Ryan T, Mahoney MP. Feedback for learning. Closing the assessment loop. Framework for effective learning. Canberra, Australia: Australian Government, Department for Education and Training; 2018.

Heeneman S, Pool AO, Schuwirth LWT, van der Vleuten CPM, Driessen EW, Oudkerk A, et al. The impact of programmatic assessment on student learning: theory versus practice. Med Educ. 2015;49:487–98.

Lefroy J, Watling C, Teunissen P, Brand P, Watling C. Guidelines: the do’s, don’ts and don’t knows of feedback for clinical education. Perspect Med Educ. 2015;4:284–99.

Ramani S, Krackov SK. Twelve tips for giving feedback effectively in the clinical environment. Med Teach. 2012;34:787–91.

Telio S, Ajjawi R, Regehr G. The, “Educational Alliance” as a Framework for Reconceptualizing Feedback in Medical Education. Acad Med. 2015;90:609–14.

Lockyer J, Armson H, Könings KD, Lee-Krueger RC, des Ordons AR, Ramani S, et al. In-the-Moment Feedback and Coaching: Improving R2C2 for a New Context. J Grad Med Educ. 2020;12:27–35.

Black P, Wiliam D. Developing the theory of formative assessment. Educ Assess Eval Account. 2009;21:5–31.

Download references

Author information

Authors and affiliations.

Department of Health Sciences, Faculty of Medicine, Pontificia Universidad Católica de Chile, Avenida Vicuña Mackenna 4860, Macul, Santiago, Chile

Javiera Fuentes-Cimma & Ignacio Villagran

School of Health Professions Education, Maastricht University, Maastricht, Netherlands

Javiera Fuentes-Cimma & Lorena Isbej

Rotterdam University of Applied Sciences, Rotterdam, Netherlands

Dominique Sluijsmans

Centre for Medical and Health Profession Education, Department of Gastroenterology, Faculty of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile

Arnoldo Riquelme

School of Dentistry, Faculty of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile

Lorena Isbej

Sistema de Bibliotecas UC (SIBUC), Pontificia Universidad Católica de Chile, Santiago, Chile

María Teresa Olivares-Labbe

Department of Pathology, Faculty of Health, Medicine and Health Sciences, Maastricht University, Maastricht, Netherlands

Sylvia Heeneman

You can also search for this author in PubMed Google Scholar

Contributions

J.F-C, D.S, and S.H. made substantial contributions to the conception and design of the work. M.O-L contributed to the identification of studies. J.F-C, I.V, A.R, and L.I. made substantial contributions to the screening, reliability, and data analysis. J.F-C. wrote th e main manuscript text. All authors reviewed the manuscript.

Corresponding author

Correspondence to Javiera Fuentes-Cimma .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1, supplementary material 2., rights and permissions.

Reprints and permissions

About this article

Cite this article.

Fuentes-Cimma, J., Sluijsmans, D., Riquelme, A. et al. Designing feedback processes in the workplace-based learning of undergraduate health professions education: a scoping review. BMC Med Educ 24 , 440 (2024). https://doi.org/10.1186/s12909-024-05439-6

Download citation

Received : 25 September 2023

Accepted : 17 April 2024

Published : 23 April 2024

DOI : https://doi.org/10.1186/s12909-024-05439-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Clinical clerkship
Feedback processes
Feedforward
Formative feedback
Health professions
Undergraduate medical education
Undergraduate healthcare education
Workplace learning

BMC Medical Education

ISSN: 1472-6920

Submission enquiries: [email protected]
General enquiries: [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

View all journals
My Account Login
Explore content
About the journal
Publish with us
Sign up for alerts
Perspective
Open access
Published: 09 April 2024

The potential for artificial intelligence to transform healthcare: perspectives from international health leaders

Christina Silcox 1 ,
Eyal Zimlichmann 2 , 3 ,
Katie Huber ORCID: orcid.org/0000-0003-2519-8714 1 ,
Neil Rowen 1 ,
Robert Saunders 1 ,
Mark McClellan 1 ,
Charles N. Kahn III 3 , 4 ,
Claudia A. Salzberg 3 &
David W. Bates ORCID: orcid.org/0000-0001-6268-1540 5 , 6 , 7

npj Digital Medicine volume 7 , Article number: 88 ( 2024 ) Cite this article

2327 Accesses

39 Altmetric

Metrics details

Health policy
Health services

Artificial intelligence (AI) has the potential to transform care delivery by improving health outcomes, patient safety, and the affordability and accessibility of high-quality care. AI will be critical to building an infrastructure capable of caring for an increasingly aging population, utilizing an ever-increasing knowledge of disease and options for precision treatments, and combatting workforce shortages and burnout of medical professionals. However, we are not currently on track to create this future. This is in part because the health data needed to train, test, use, and surveil these tools are generally neither standardized nor accessible. There is also universal concern about the ability to monitor health AI tools for changes in performance as they are implemented in new places, used with diverse populations, and over time as health data may change. The Future of Health (FOH), an international community of senior health care leaders, collaborated with the Duke-Margolis Institute for Health Policy to conduct a literature review, expert convening, and consensus-building exercise around this topic. This commentary summarizes the four priority action areas and recommendations for health care organizations and policymakers across the globe that FOH members identified as important for fully realizing AI’s potential in health care: improving data quality to power AI, building infrastructure to encourage efficient and trustworthy development and evaluations, sharing data for better AI, and providing incentives to accelerate the progress and impact of AI.

Guiding principles for the responsible development of artificial intelligence tools for healthcare

Kimberly Badal, Carmen M. Lee & Laura J. Esserman

A short guide for medical professionals in the era of artificial intelligence

Bertalan Meskó & Marton Görög

Reporting guidelines in medical artificial intelligence: a systematic review and meta-analysis

Fiona R. Kolbinger, Gregory P. Veldhuizen, … Jakob Nikolas Kather

Introduction

Artificial intelligence (AI), supported by timely and accurate data and evidence, has the potential to transform health care delivery by improving health outcomes, patient safety, and the affordability and accessibility of high-quality care 1 , 2 . AI integration is critical to building an infrastructure capable of caring for an increasingly aging population, utilizing an ever-increasing knowledge of disease and options for precision treatments, and combatting workforce shortages and burnout of medical professionals. However, we are not currently on track to create this future. This is in part because the health data needed to train, test, use, and surveil these tools are generally neither standardized nor accessible. This is true across the international community, although there is variable progress within individual countries. There is also universal concern about monitoring health AI tools for changes in performance as they are implemented in new places, used with diverse populations, and over time as health data may change.

The Future of Health (FOH) is an international community of senior health care leaders representing health systems, health policy, health care technology, venture funding, insurance, and risk management. FOH collaborated with the Duke-Margolis Institute for Health Policy to conduct a literature review, expert convening, and consensus-building exercise. In total, 46 senior health care leaders were engaged in this work, from eleven countries in Europe, North America, Africa, Asia, and Australia. This commentary summarizes the four priority action areas and recommendations for health care organizations and policymakers that FOH members identified as important for fully realizing AI’s potential in health care: improving data quality to power AI, building infrastructure to encourage efficient and trustworthy development and evaluations, sharing data for better AI, and providing incentives to accelerate the progress and impact of AI.

Powering AI through high-quality data

“Going forward, data are going to be the most valuable commodity in health care. Organizations need robust plans about how to mobilize and use their data.”

AI algorithms will only perform as well as the accuracy and completeness of key underlying data, and data quality is dependent on actions and workflows that encourage trust.

To begin to improve data quality, FOH members agreed that an initial priority is identifying and assuring reliable availability of high-priority data elements for promising AI applications: those with the most predictive value, those of the highest value to patients, and those most important for analyses of performance, including subgroup analyses to detect bias.

Leaders should also advocate for aligned policy incentives to improve the availability and reliability of these priority data elements. There are several examples of efforts across the world to identify and standardize high-priority data elements for AI applications and beyond, such as the multinational project STANDING Together, which is developing standards to improve the quality and representativeness of data used to build and test AI tools 3 .

Policy incentives that would further encourage high-quality data collection include (1) aligned payment incentives for measures of health care quality and safety, and ensuring the reliability of the underlying data, and (2) quality measures and performance standards focused on the reliability, completeness, and timeliness of collection and sharing of high-priority data itself.

Trust and verify

“Your AI algorithms are only going to be as good as the data and the real-world evidence used to validate them, and the data are only going to be as good as the trust and privacy and supporting policies.”

FOH members stressed the importance of showing that AI tools are both effective and safe within their specific patient populations.

This is a particular challenge with AI tools, whose performance can differ dramatically across sites and over time, as health data patterns and population characteristics vary. For example, several studies of the Epic Sepsis Model found both location-based differences in performance and degradation in performance over time due to data drift 4 , 5 . However, real-world evaluations are often much more difficult for algorithms that are used for longer-term predictions, or to avert long-term complications from occurring, particularly in the absence of connected, longitudinal data infrastructure. As such, health systems must prioritize implementing data standards and data infrastructure that can facilitate the retraining or tuning of algorithms, test for local performance and bias, and ensure scalability across the organization and longer-term applications 6 .

There are efforts to help leaders and health systems develop consensus-based evaluation techniques and infrastructure for AI tools, including HealthAI: The Global Agency for Responsible AI in Health, which aims to build and certify validation mechanisms for nations and regions to adopt; and the Coalition for Health AI (CHAI), which recently announced plans to build a US-wide health AI assurance labs network 7 , 8 . These efforts, if successful, will assist manufacturers and health systems in complying with new laws, rules, and regulations being proposed and released that seek to ensure AI tools are trustworthy, such as the EU AI Act and the 2023 US Executive Order on AI.

Sharing data for better AI

“Underlying these challenges is the investment required to standardize business processes so that you actually get data that’s usable between institutions and even within an institution.”

While high-quality internal data may enable some types of AI-tool development and testing, this is insufficient to power and evaluate all AI applications. To build truly effective AI-enabled predictive software for clinical care and predictive supports, data often need to be interoperable across health systems to build a diverse picture of patients’ health across geographies, and reliably shared.

FOH members recommended that health care leaders work with researchers and policymakers to connect detailed encounter data with longitudinal outcomes, and pilot opportunities across diverse populations and systems to help assure valid outcome evaluations as well as address potential confounding and population subgroup differences—the ability to aggregate data is a clear rate-limiting step. The South African National Digital Health Strategy outlined interventions to improve the adoption of digital technologies while complying with the 2013 Protection of Personal Information Act 9 . Although challenges remain, the country has made progress on multiple fronts, including building out a Health Patient Registration System as a first step towards a portable, longitudinal patient record system and releasing a Health Normative Standards Framework to improve data flow across institutional and geographic boundaries 10 .

Leaders should adopt policies in their organizations, and encourage adoption in their province and country, that simplify data governance and sharing while providing appropriate privacy protections – including building foundations of trust with patients and the public as previously discussed. Privacy-preserving innovations include ways to “share” data without movement from protected systems using approaches like federated analyses, data sandboxes, or synthetic data. In addition to exploring privacy-preserving approaches to data sharing, countries and health systems may need to consider broad and dynamic approaches to consent 11 , 12 . As we look to a future where a patient may have thousands of algorithms churning away at their data, efforts to improve data quality and sharing should include enabling patients’ access to and engagement with their own data to encourage them to actively partner in their health and provide transparency on how their data are being used to improve health care. For example, the Understanding Patient Data program in the United Kingdom produces research and resources to explain how the National Health Service uses patients’ data 13 . Community engagement efforts can further assist with these efforts by building trust and expanding understanding.

FOH members also stressed the importance of timely data access. Health systems should work together to establish re-usable governance and privacy frameworks that allow stakeholders to clearly understand what data will be shared and how it will be protected to reduce the time needed for data use agreements. Trusted third-party data coordinating centers could also be used to set up “precertification” systems around data quality, testing, and cybersecurity to support health organizations with appropriate data stewardship to form partnerships and access data rapidly.

Incentivizing progress for AI impact

“Unless it’s tied to some kind of compensation to the organization, the drive to help implement those tools and overcome that risk aversion is going to be very high… I do think that business driver needs to be there.”

AI tools and data quality initiatives have not moved as quickly in health care due to the lack of direct payment, and often, misalignment of financial incentives and supports for high-quality data collection and predictive analytics. This affects both the ability to purchase and safely implement commercial AI products as well as the development of “homegrown” AI tools.

FOH members recommended that leaders should advocate for paying for value in health – quality, safety, better health, and lower costs for patients. This better aligns the financial incentives for accelerating the development, evaluation, and adoption of AI as well as other tools designed to either keep patients healthy or quickly diagnose and treat them with the most effective therapies when they do become ill. Effective personalized health care requires high-quality, standardized, interoperable datasets from diverse sources 14 . Within value-based payments themselves, data are critical to measuring quality of care and patient outcomes, adjusted or contextualized for factors outside of clinical control. Value-based payments therefore align incentives for (1) high-quality data collection and trusted use, (2) building effective AI tools, and (3) ensuring that those tools are improving patient outcomes and/or health system operations.

Data have become the most valuable commodity in health care, but questions remain about whether there will be an AI “revolution” or “evolution” in health care delivery. Early AI applications in certain clinical areas have been promising, but more advanced AI tools will require higher quality, real-world data that is interoperable and secure. The steps health care organization leaders and policymakers take in the coming years, starting with short-term opportunities to develop meaningful AI applications that achieve measurable improvements in outcomes and costs, will be critical in enabling this future that can improve health outcomes, safety, affordability, and equity.

Data availability

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

Abernethy, A. et al. The promise of digital health: then, now, and the future. NAM Perspect. 6 (2022).

Akpakwu, E. Four ways AI can make healthcare more efficient and affordable. World Economic Forum https://www.weforum.org/agenda/2018/05/four-ways-ai-is-bringing-down-the-cost-of-healthcare/ (2018).

STANDING Together. https://www.datadiversity.org/home .

Wong, A. et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med 181 , 1065–1070 (2021).

Article PubMed Google Scholar

Ross, C. STAT and MIT rooted out the weaknesses in health care algorithms. Here’s how we did it. STAT https://www.statnews.com/2022/02/28/data-drift-machine-learning/ (2022).

Locke, T., Parker, V., Thoumi, A., Goldstein, B. & Silcox, C. Preventing bias and inequities in AI-enabled health tools . https://healthpolicy.duke.edu/publications/preventing-bias-and-inequities-ai-enabled-health-tools (2022).

Introducing HealthAI. The International Digital Health and AI Research Collaborative (I-DAIR) https://www.i-dair.org/news/introducing-healthai (2023).

Shah, N. H. et al. A nationwide network of health AI assurance laboratories. JAMA 331 , 245 (2024).

Singh, V. AI & Data in South Africa’s Health Sector . https://policyaction.org.za/sites/default/files/PAN_TopicalGuide_AIData6_Health_Elec.pdf (2020).

Zharima, C., Griffiths, F. & Goudge, J. Exploring the barriers and facilitators to implementing electronic health records in a middle-income country: a qualitative study from South Africa. Front. Digit. Health 5 , 1207602 (2023).

Article PubMed PubMed Central Google Scholar

Lee, A. R. et al. Identifying facilitators of and barriers to the adoption of dynamic consent in digital health ecosystems: a scoping review. BMC Med. Ethics 24 , 107 (2023).

Article CAS PubMed PubMed Central Google Scholar

Stoeklé, H. C., Hulier-Ammar, E. & Hervé, C. Data medicine: ‘broad’ or ‘dynamic’ consent? Public Health Ethics 15 , 181–185 (2022).

Article Google Scholar

Understanding Patient Data. Understanding Patient Data http://understandingpatientdata.org.uk/ .

Chén, O. Y. & Roberts, B. Personalized health care and public health in the digital age. Front. Digit. Health 3 , 595704 (2021).

Download references

Acknowledgements

The authors acknowledge Oranit Ido and Jonathan Gonzalez-Smith for their contributions to this work. This study was funded by The Future of Health, LLC. The Future of Health, LLC, was involved in all stages of this research, including study design, data collection, analysis and interpretation of data, and the preparation of this manuscript.

Author information

Authors and affiliations.

Duke-Margolis Institute for Health Policy, Duke University, Washington, DC, USA &, Durham, NC, USA

Christina Silcox, Katie Huber, Neil Rowen, Robert Saunders & Mark McClellan

Sheba Medical Center, Ramat Gan, Israel

Eyal Zimlichmann

Future of Health, Washington, DC, USA

Eyal Zimlichmann, Charles N. Kahn III & Claudia A. Salzberg

Federation of American Hospitals, Washington, DC, USA

Charles N. Kahn III

Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, MA, USA

David W. Bates

Harvard Medical School, Boston, MA, USA

Department of Health Policy and Management, Harvard T. H. Chan School of Public Health, Boston, MA, USA

You can also search for this author in PubMed Google Scholar

Contributions

C.S., K.H., N.R., and R.S. conducted initial background research and analyzed qualitative data from stakeholders. All authors (C.S., E.Z., K.H., N.R., R.S., M.M., C.K., C.A.S., and D.B.) assisted with conceptualization of the project and strategic guidance. C.S., K.H., and N.R. wrote initial drafts of the manuscript. All authors contributed to critical revisions of the manuscript and read and approved the final manuscript.

Corresponding author

Correspondence to David W. Bates .

Ethics declarations

Competing interests.

C.S., K.H., N.R., and C.A.S. declare no competing interests. E.Z. reports personal fees from Arkin Holdings, personal fees from Statista and equity from Valera Health, Profility and Hello Heart. R.S. has been an external reviewer for The John A. Hartford Foundation, and is a co-chair for the Health Evolution Summit Roundtable on Value-Based Care for Specialized Populations. M.M. is an independent director on the boards of Johnson & Johnson, Cigna, Alignment Healthcare, and PrognomIQ; co-chairs the Guiding Committee for the Health Care Payment Learning and Action Network; and reports fees for serving as an adviser for Arsenal Capital Partners, Blackstone Life Sciences, and MITRE. C.K. is a Profility Board member and additionally reports equity from Valera Health and MDClone. D.W.B. reports grants and personal fees from EarlySense, personal fees from CDI Negev, equity from Valera Health, equity from Clew, equity from MDClone, personal fees and equity from AESOP, personal fees and equity from Feelbetter, equity from Guided Clinical Solutions, and grants from IBM Watson Health, outside the submitted work. D.W.B. has a patent pending (PHC-028564 US PCT), on intraoperative clinical decision support.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Silcox, C., Zimlichmann, E., Huber, K. et al. The potential for artificial intelligence to transform healthcare: perspectives from international health leaders. npj Digit. Med. 7 , 88 (2024). https://doi.org/10.1038/s41746-024-01097-6

Download citation

Received : 30 October 2023

Accepted : 29 March 2024

Published : 09 April 2024

DOI : https://doi.org/10.1038/s41746-024-01097-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

Explore articles by subject
Guide to authors
Editorial policies

IMAGES

(PDF) A Literature Review on Organizational Learning and Learning
(PDF) Conceptualizing Digital Transformation in Business Organizations
(PDF) Literature Review Process:Measuring the Effective Usage of
15 Literature Review Examples (2024)
(PDF) Leading Multicultural Teams in Agile Organizations: An
(PDF) A Systematic Review of Literature about Leadership and Organization

VIDEO

3_session2 Importance of literature review, types of literature review, Reference management tool
Literature Review for Research #hazarauniversity #trendingvideo #pakistan
The Literature Review
A literature review on online learning
LITERATURE REVIEW
Writing a Literature Review

COMMENTS

A Systematic Literature Review of Organizational Factors Influencing
They are indispensable in an economy where the knowledge and skills of employees are seen as a measure for economic potential. This systematic literature review summarizes the current academic knowledge about organizational factors that influence 21st-century skills on an individual level. A search was performed in three databases.
Full article: Organizational culture: a systematic review
A literature review: The relationship and measurement of servant leadership, organizational commitment, organizational culture to employee satisfaction, employee engagement, and employee performance. Systematic Reviews in Pharmacy , 11 (12), 2188-2191.
Literature Reviews
Structure. The three elements of a literature review are introduction, body, and conclusion. Introduction. Define the topic of the literature review, including any terminology. Introduce the central theme and organization of the literature review. Summarize the state of research on the topic. Frame the literature review with your research question.
Reactions towards organizational change: a systematic literature review
The change reaction leads to many outcomes and at different organizational levels. The range of literature examining employees' reaction to change is wide. Furthermore, the results of the literature review identified four vital categories: Voice behavior, exit behavior, neglect behavior, and loyalty behavior.
A Systematic Review of Human Resource Management Systems and Their
Literature Review Conceptualizing HR Systems. SHRM can be defined as "the pattern of planned HR deployments and activities intended to enable an organization to achieve its goals" (Wright & McMahan, 1992: 298). Increasingly, the field has emphasized the importance of focusing on whether and how "systems" or "bundles" of HR practices ...
Organizational justice research: A review, synthesis, and research
We conducted a general, semi-systematic literature review on organizational justice. Based on this review, we first explored the various ways in which organizational justice was conceptualized. Second, we explained the importance of organizational justice by reviewing and synthesizing theoretical frameworks in organizational justice research ...
Organisational agility: systematic literature review and future
A systematic literature review (SLR) was carried out, using the Prisma method, in a first phase, and the IraMuteQ software to operationalise the bibliometrics. Therefore, this study presents the development of a conceptual model to interpret organisational agility, that is, an agile, adjusted and flexible response to the market, which suggests ...
How to Write a Literature Review
Examples of literature reviews. Step 1 - Search for relevant literature. Step 2 - Evaluate and select sources. Step 3 - Identify themes, debates, and gaps. Step 4 - Outline your literature review's structure. Step 5 - Write your literature review.
A Systematic Literature Review of GHRM: Organizational Sustainable
This section discusses the results drawn from the systematic literature review on GHRM and sustainable organizational and individual performance. The way business firms apply GHRM practices to address employees' and groups' performance sustainably has been a critical gap that was previously missing and is the current focus of this article.
(PDF) A Literature Review of Organizational Behavior Management
David Wilder (2018): A Literature Review of Organizational Behavior Management Interventions in Human Service Settings from 1990 to 2016, Journal of Organizational Behavior Management, DOI: 10. ...
Doing a literature review: Community organizations and nonprofits
What a literature review could do for you. Outside of academia, your motivations for research may be different. For a non-profit, literature reviews might help you: Gather evidence to assess your organization's best practices. Provide evidence for grant proposals. Look for new approaches to an issue. Provide background or a methodology for ...
Organizing the Literature Review
Create an organizational method to focus this section even further. To help you come up with an overall organizational framework for your review, consider the following scenario and then three typical ways of organizing the sources into a review: You've decided to focus your literature review on materials dealing with sperm whales.
5. The Literature Review
A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories.A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that ...
(PDF) A Literature Review on Organizational Culture ...
The literature review comprised various published sources on the role of organizational culture, such as journals, periodicals, seminal books, and other published materials. The review focused on ...
(PDF) A Literature Review on Organizational Learning and Learning
Citation: Odor HO (2018) A Literature Review on Organizational Learning and Learning Organizations. Int J Econ Manag Sci 7: 494. doi: 10.4172/2162-6359.1000494. Page 2 of 6.
Literature Reviews
A literature review can be a short introductory section of a research article or a report or policy paper that focuses on recent research. Or, in the case of dissertations, theses, and review articles, it can be an extensive review of all relevant research. The format is usually a bibliographic essay; sources are briefly cited within the body ...
A Systematic Literature Review of Organizational Factors Influencing
Skills, such as critical thinking, creativity, communication, problem solving, collaboration, operational skills, and information management, have become increasingly important for 21st-century employees. These skills are often referred to as 21st-century skills and influence how employees handle novel situations.
The Literature Review
Describe the organization of the review (the sequence) If necessary, state why certain literature is or is not included (scope) Body - summative, comparative, and evaluative discussion of literature reviewed. For a thematic review: organize the review into paragraphs that present themes and identify trends relevant to your topic
Literature Review: Conducting & Writing
Introduction: Gives a quick idea of the topic of the literature review, such as the central theme or organizational pattern. Body: Contains your discussion of sources and is organized either chronologically, thematically, or methodologically (see below for more information on each).
Systematic Literature Review of E-Learning Capabilities to Enhance
Our proposed literature review builds on the previously discussed models, lenses, and empirical studies, and it provides a review of research on e-learning capabilities with the aim of enhancing organizational learning in order to complement the findings of the established models and guide future studies.
Organizational unlearning as a process: What we know, what ...
Although the field of organizational unlearning has recently gained increased interest, its conceptual foundations and raison d'être are still debated. In this review, we aim to revisit various discourses and arguments to advance the understanding of organizational unlearning in management and organization studies. Using an integrative literature review approach with systematic elements, we ...
Administrative Sciences
This literature review aims to examine the relationship between Green Human Resource Management (G-HRM) practices and various outcomes, including employee green attitudes, employee green satisfaction, client green satisfaction, employee green behavior, and organizational green performance. We reviewed existing literature on G-HRM practices and their impact on the selected outcomes.
Performance Management: A Scoping Review of the Literature and an
We conducted a scoping review of the PM literature, consistent with a recent Human Resource Development Review (HRDR) publication (Tkachenko et al., 2017), where the research team ". . . met several times to define and clarify the problem statement of the inquiry, the scope of the study, and the inclusion and exclusion of sources for the ...
Implementation of Information Security Management Systems for Data
The findings advocate for a comprehensive and structured approach to information security, urging future research to explore diverse organizational contexts and industries for a nuanced understanding of ISMS practices and their impact on organizational agility. This systematic literature review investigates the implementation of Information Security Management Systems (ISMS) as a pivotal ...
Challenges Affecting the Successful Adoption of DevOps Practices: A
Hence these organizations are willing to implement DevOps practices. DevOps has emerged as a promising approach in software development and operations, aiming to enhance collaboration, automation, and efficiency in delivering high-quality software products. ... This systematic literature review explores the challenges affecting the successful ...
Designing feedback processes in the workplace-based learning of
A scoping review was conducted using the five-step methodological framework proposed by Arksey and O'Malley (2005) [], intertwined with the PRISMA checklist extension for scoping reviews to provide reporting guidance for this specific type of knowledge synthesis [].Scoping reviews allow us to study the literature without restricting the methodological quality of the studies found ...
The potential for artificial intelligence to transform ...
The Future of Health (FOH), an international community of senior health care leaders, collaborated with the Duke-Margolis Institute for Health Policy to conduct a literature review, expert ...
Aspects of Women's Leadership in the Organisation: Systematic
These aspects are barriers, enablers, strengths and weaknesses of Women's Leadership. The research method used for the study is a systematic literature review method. Peer-reviewed, scholarly journal articles and book references between 1990 and 2021 identified from the Scopus and Web of Sciences databases are considered for the study.
Lean, agile, leagile manufacturing systems pros and cons: A literature
A summary of the literature review for some case studies from 2013 to 2022 was used in this paper to define the current manufacturer's product quality goals through various case studies implemented in multiple manufacturing systems. ... which can help organizations achieve their strategic goals, such as competitiveness, productivity ...