Help | Advanced Search

Computer Science > Computation and Language

Title: a literature survey of recent advances in chatbots.

Abstract: Chatbots are intelligent conversational computer systems designed to mimic human conversation to enable automated online guidance and support. The increased benefits of chatbots led to their wide adoption by many industries in order to provide virtual assistance to customers. Chatbots utilise methods and algorithms from two Artificial Intelligence domains: Natural Language Processing and Machine Learning. However, there are many challenges and limitations in their application. In this survey we review recent advances on chatbots, where Artificial Intelligence and Natural Language processing are used. We highlight the main challenges and limitations of current work and make recommendations for future research investigation.

Submission history

Access paper:.

  • Download PDF
  • Other Formats

chatbot research paper 2020

References & Citations

  • Google Scholar
  • Semantic Scholar

DBLP - CS Bibliography

Bibtex formatted citation.

BibSonomy logo

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

  • Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

  • Reference Manager
  • Simple TEXT file

People also looked at

Systematic review article, are we there yet - a systematic literature review on chatbots in education.

www.frontiersin.org

  • 1 Information Center for Education, DIPF | Leibniz Institute for Research and Information in Education, Frankfurt am Main, Germany
  • 2 Educational Science Faculty, Open University of the Netherlands, Heerlen, Netherlands
  • 3 Computer Science Faculty, Goethe University, Frankfurt am Main, Germany

Chatbots are a promising technology with the potential to enhance workplaces and everyday life. In terms of scalability and accessibility, they also offer unique possibilities as communication and information tools for digital learning. In this paper, we present a systematic literature review investigating the areas of education where chatbots have already been applied, explore the pedagogical roles of chatbots, the use of chatbots for mentoring purposes, and their potential to personalize education. We conducted a preliminary analysis of 2,678 publications to perform this literature review, which allowed us to identify 74 relevant publications for chatbots’ application in education. Through this, we address five research questions that, together, allow us to explore the current state-of-the-art of this educational technology. We conclude our systematic review by pointing to three main research challenges: 1) Aligning chatbot evaluations with implementation objectives, 2) Exploring the potential of chatbots for mentoring students, and 3) Exploring and leveraging adaptation capabilities of chatbots. For all three challenges, we discuss opportunities for future research.

Introduction

Educational Technologies enable distance learning models and provide students with the opportunity to learn at their own pace. They have found their way into schools and higher education institutions through Learning Management Systems and Massive Open Online Courses, enabling teachers to scale up good teaching practices ( Ferguson and Sharples, 2014 ) and allowing students to access learning material ubiquitously ( Virtanen et al., 2018 ).

Despite the innovative power of educational technologies, most commonly used technologies do not substantially change teachers’ role. Typical teaching activities like providing students with feedback, motivating them, or adapting course content to specific student groups are still entrusted exclusively to teachers, even in digital learning environments. This can lead to the teacher-bandwidth problem ( Wiley and Edwards, 2002 ), the result of a shortage of teaching staff to provide highly informative and competence-oriented feedback at large scale. Nowadays, however, computers and other digital devices open up far-reaching possibilities that have not yet been fully exploited. For example, incorporating process data can provide students with insights into their learning progress and bring new possibilities for formative feedback, self-reflection, and competence development ( Quincey et al., 2019 ). According to ( Hattie, 2009 ), feedback in terms of learning success has a mean effect size of d = 0.75, while ( Wisniewski et al., 2019 ) even report a mean effect of d = 0.99 for highly informative feedback. Such feedback provides suitable conditions for self-directed learning ( Winne and Hadwin, 2008 ) and effective metacognitive control of the learning process ( Nelson and Narens, 1994 ).

One of the educational technologies designed to provide actionable feedback in this regard is Learning Analytics. Learning Analytics is defined as the research area that focuses on collecting traces that learners leave behind and using those traces to improve learning ( Duval and Verbert, 2012 ; Greller and Drachsler, 2012 ). Learning Analytics can be used both by students to reflect on their own learning progress and by teachers to continuously assess the students’ efforts and provide actionable feedback. Another relevant educational technology is Intelligent Tutoring Systems. Intelligent Tutoring Systems are defined as computerized learning environments that incorporate computational models ( Graesser et al., 2001 ) and provide feedback based on learning progress. Educational technologies specifically focused on feedback for help-seekers, comparable to raising hands in the classroom, are Dialogue Systems and Pedagogical Conversational Agents ( Lester et al., 1997 ). These technologies can simulate conversational partners and provide feedback through natural language ( McLoughlin and Oliver, 1998 ).

Research in this area has recently focused on chatbot technology, a subtype of dialog systems, as several technological platforms have matured and led to applications in various domains. Chatbots incorporate generic language models extracted from large parts of the Internet and enable feedback by limiting themselves to text or voice interfaces. For this reason, they have also been proposed and researched for a variety of applications in education ( Winkler and Soellner, 2018 ). Recent literature reviews on chatbots in education ( Winkler and Soellner, 2018 ; Hobert, 2019a ; Hobert and Meyer von Wolff, 2019 ; Jung et al., 2020 ; Pérez et al., 2020 ; Smutny and Schreiberova, 2020 ; Pérez-Marín, 2021 ) have reported on such applications as well as design guidelines, evaluation possibilities, and effects of chatbots in education.

In this paper, we contribute to the state-of-the-art of chatbots in education by presenting a systematic literature review, where we examine so-far unexplored areas such as implementation objectives, pedagogical roles, mentoring scenarios, the adaptations of chatbots to learners, and application domains. This paper is structured as follows: First, we review related work (section 2), derive research questions from it, then explain the applied method for searching related studies (section 3), followed by the results (section 4), and finally, we discuss the findings (section 5) and point to future research directions in the field (section 5).

Related Work

In order to accurately cover the field of research and deal with the plethora of terms for chatbots in the literature (e.g. chatbot, dialogue system or pedagogical conversational agent) we propose the following definition:

Chatbots are digital systems that can be interacted with entirely through natural language via text or voice interfaces. They are intended to automate conversations by simulating a human conversation partner and can be integrated into software, such as online platforms, digital assistants, or be interfaced through messaging services.

Outside of education, typical applications of chatbots are in customer service ( Xu et al., 2017 ), counseling of hospital patients ( Vaidyam et al., 2019 ), or information services in smart speakers ( Ram et al., 2018 ). One central element of chatbots is the intent classification, also named the Natural Language Understanding (NLU) component, which is responsible for the sense-making of human input data. Looking at the current advances in chatbot software development, it seems that this technology’s goal is to pass the Turing Test ( Saygin et al., 2000 ) one day, which could make chatbots effective educational tools. Therefore, we ask ourselves “ Are we there yet? - Will we soon have an autonomous chatbot for every learner?”

To understand and underline the current need for research in the use of chatbots in education, we first examined the existing literature, focusing on comprehensive literature reviews. By looking at research questions in these literature reviews, we identified 21 different research topics and extracted findings accordingly. To structure research topics and findings in a comprehensible way, a three-stage clustering process was applied. While the first stage consisted of coding research topics by keywords, the second stage was applied to form overarching research categories ( Table 1 ). In the final stage, the findings within each research category were clustered to identify and structure commonalities within the literature reviews. The result is a concept map, which consists of four major categories. Those categories are CAT1. Applications of Chatbots, CAT2. Chatbot Designs, CAT3. Evaluation of Chatbots and CAT4. Educational Effects of Chatbots. To standardize the terminology and concepts applied, we present the findings of each category in a separate sub-section, respectively ( see Figure 1 , Figure 2 , Figure 3 , and Figure 4 ) and extended it with the outcomes of our own literature study that will be reported in the remaining parts of this article. Due to the size of the concept map a full version can be found in Appendix A .

www.frontiersin.org

TABLE 1 . Assignment of coded research topics identified in related literature reviews to research categories.

www.frontiersin.org

FIGURE 1 . Applications of chatbots in related literature reviews (CAT1).

www.frontiersin.org

FIGURE 2 . Chatbot designs in related literature reviews (CAT2).

www.frontiersin.org

FIGURE 3 . Evaluation of chatbots in related literature reviews (CAT3).

www.frontiersin.org

FIGURE 4 . Educational Effects of chatbots in related literature reviews (CAT4).

Regarding the applications of chatbots (CAT1), application clusters (AC) and application statistics (AS) have been described in the literature, which we visualized in Figure 1 . The study of ( Pérez et al., 2020 ) identifies two application clusters, defined through chatbot activities: “service-oriented chatbots” and “teaching-oriented chatbots.” ( Winkler and Soellner, 2018 ) identify applications clusters by naming the domains “health and well-being interventions,” “language learning,” “feedback and metacognitive thinking” as well as “motivation and self-efficacy.” Concerning application statistics (AS), ( Smutny and Schreiberova, 2020 ) found that nearly 47% of the analyzed chatbots incorporate informing actions, and 18% support language learning by elaborating on chatbots integrated into the social media platform Facebook. Besides, the chatbots studied had a strong tendency to use English, at 89%. This high number aligns with results from ( Pérez-Marín, 2021 ), where 75% of observed agents, as a related technology, were designed to interact in the English language. ( Pérez-Marín, 2021 ) also shows that 42% of the analyzed chatbots had mixed interaction modalities. Finally, ( Hobert and Meyer von Wolff, 2019 ) observed that only 25% of examined chatbots were incorporated in formal learning settings, the majority of published material focuses on student-chatbot interaction only and does not enable student-student communication, as well as nearly two-thirds of the analyzed chatbots center only on a single domain. Overall, we can summarize that so far there are six application clusters for chatbots for education categorized by chatbot activities or domains. The provided statistics allow for a clearer understanding regarding the prevalence of chatbots applications in education ( see Figure 1 ).

Regarding chatbot designs (CAT2), most of the research questions concerned with chatbots in education can be assigned to this category. We found three aspects in this category visualized in Figure 2 : Personality (PS), Process Pipeline (PP), and Design Classifications (DC). Within these, most research questions can be assigned to Design Classifications (DC), which are separated into Classification Aspects (DC2) and Classification Frameworks (DC1). One classification framework is defined through “flow chatbots,” “artificially intelligent chatbots,” “chatbots with integrated speech recognition,” as well as “chatbots with integrated context-data” by ( Winkler and Soellner, 2018 ). A second classification framework by ( Pérez-Marín, 2021 ) covers pedagogy, social, and HCI features of chatbots and agents, which themselves can be further subdivided into more detailed aspects. Other Classification Aspects (DC2) derived from several publications, provide another classification schema, which distinguishes between “retrieval vs. generative” based technology, the “ability to incorporate context data,” and “speech or text interface” ( Winkler and Soellner, 2018 ; Smutny and Schreiberova, 2020 ). By specifying text interfaces as “Button-Based” or “Keyword Recognition-Based” ( Smutny and Schreiberova, 2020 ), text interfaces can be subdivided. Furthermore, a comparison of speech and text interfaces ( Jung et al., 2020 ) shows that text interfaces have advantages for conveying information, and speech interfaces have advantages for affective support. The second aspect of CAT2 concerns the chatbot processing pipeline (PP), highlighting user interface and back-end importance ( Pérez et al., 2020 ). Finally, ( Jung et al., 2020 ) focuses on the third aspect, the personality of chatbots (PS). Here, the study derives four guidelines helpful in education: positive or neutral emotional expressions, a limited amount of animated or visual graphics, a well-considered gender of the chatbot, and human-like interactions. In summary, we have found in CAT2 three main design aspects for the development of chatbots. CAT2 is much more diverse than CAT1 with various sub-categories for the design of chatbots. This indicates the huge flexibility to design chatbots in various ways to support education.

Regarding the evaluation of chatbots (CAT3), we found three aspects assigned to this category, visualized in Figure 3 : Evaluation Criteria (EC), Evaluation Methods (EM), and Evaluation Instruments (EI). Concerning Evaluation Criteria, seven criteria can be identified in the literature. The first and most important in the educational field, according to ( Smutny and Schreiberova, 2020 ) is the evaluation of learning success ( Hobert, 2019a ), which can have subcategories such as how chatbots are embedded in learning scenarios ( Winkler and Soellner, 2018 ; Smutny and Schreiberova, 2020 ) and teaching efficiency ( Pérez et al., 2020 ). The second is acceptance, which ( Hobert, 2019a ) names as “acceptance and adoption” and ( Pérez et al., 2020 ) as “students’ perception.” Further evaluation criteria are motivation, usability, technical correctness, psychological, and further beneficial factors ( Hobert, 2019a ). These Evaluation Criteria show broad possibilities for the evaluation of chatbots in education. However, ( Hobert, 2019a ) found that most evaluations are limited to single evaluation criteria or narrower aspects of them. Moreover, ( Hobert, 2019a ) introduces a classification matrix for chatbot evaluations, which consists of the following Evaluation Methods (EM): Wizard-of-Oz approach, laboratory studies, field studies, and technical validations. In addition to this, ( Winkler and Soellner, 2018 ) recommends evaluating chatbots by their embeddedness into a learning scenario, a comparison of human-human and human-chatbot interactions, and comparing spoken and written communication. Instruments to measure these evaluation criteria were identified by ( Hobert, 2019a ) by naming quantitative surveys, qualitative interviews, transcripts of dialogues, and technical log files. Regarding CAT3, we found three main aspects for the evaluation of chatbots. We can conclude that this is a more balanced and structured distribution in comparison to CAT2, providing researchers with guidance for evaluating chatbots in education.

Regarding educational effects of chatbots (CAT4), we found two aspects visualized in Figure 4 : Effect Size (ES) and Beneficial Chatbot Features for Learning Success (BF). Concerning the effect size, ( Pérez et al., 2020 ) identified a strong dependency between learning and the related curriculum, while ( Winkler and Soellner, 2018 ) elaborate on general student characteristics that influence how students interact with chatbots. They state that students’ attitudes towards technology, learning characteristics, educational background, self-efficacy, and self-regulation skills affect these interactions. Moreover, the study emphasizes chatbot features, which can be regarded as beneficial in terms of learning outcomes (BF): “Context-Awareness,” “Proactive guidance by students,” “Integration in existing learning and instant messaging tools,” “Accessibility,” and “Response Time.” Overall, for CAT4, we found two main distinguishing aspects for chatbots, however, the reported studies vary widely in their research design, making high-level results hardly comparable.

Looking at the related work, many research questions for the application of chatbots in education remain. Therefore, we selected five goals to be further investigated in our literature review. Firstly, we were interested in the objectives for implementing chatbots in education (Goal 1), as the relevance of chatbots for applications within education seems to be not clearly delineated. Secondly, we aim to explore the pedagogical roles of chatbots in the existing literature (Goal 2) to understand how chatbots can take over tasks from teachers. ( Winkler and Soellner, 2018 ) and ( Pérez-Marín, 2021 ), identified research gaps for supporting meta-cognitive skills with chatbots such as self-regulation. This requires a chatbot application that takes a mentoring role, as the development of these meta-cognitive skills can not be achieved solely by information delivery. Within our review we incorporate this by reviewing the mentoring role of chatbots as (Goal 3). Another key element for a mentoring chatbot is adaptation to the learners needs. Therefore, (Goal 4) of our review lies in the investigation of the adaptation approaches used by chatbots in education. For (Goal 5), we want to extend the work of ( Winkler and Soellner, 2018 ) and ( Pérez et al., 2020 ) regarding Application Clusters (AC) and map applications by further investigating specific learning domains in which chatbots have been studied.

To delineate and map the field of chatbots in education, initial findings were collected by a preliminary literature search. One of the takeaways is that the emerging field around educational chatbots has seen much activity in the last two years. Based on the experience of this preliminary search, search terms, queries, and filters were constructed for the actual structured literature review. This structured literature review follows the PRISMA framework ( Liberati et al., 2009 ), a guideline for reporting systematic reviews and meta-analyses. The framework consists of an elaborated structure for systematic literature reviews and sets requirements for reporting information about the review process ( see section 3.2 to 3.4).

Research Questions

Contributing to the state-of-the-art, we investigate five aspects of chatbot applications published in the literature. We therefore guided our research with the following research questions:

RQ1: Which objectives for implementing chatbots in education can be identified in the existing literature?

RQ2: Which pedagogical roles of chatbots can be identified in the existing literature?

RQ3: Which application scenarios have been used to mentor students?

RQ4: To what extent are chatbots adaptable to personal students’ needs?

RQ5: What are the domains in which chatbots have been applied so far?

Sources of Information

As data sources, Scopus, Web of Science, Google Scholar, Microsoft Academics, and the educational research database “Fachportal Pädagogik” (including ERIC) were selected, all of which incorporate all major publishers and journals. In ( Martín-Martín et al., 2018 ) it was shown that for the social sciences only 29.8% and for engineering and computer science, 46.8% of relevant literature is included in all of the first three databases. For the topic of chatbots in education, a value between these two numbers can be assumed, which is why an approach of integrating several publisher-independent databases was employed here.

Search Criteria

Based on the findings from the initial related work search, we derived the following search query:

( Education OR Educational OR Learning OR Learner OR Student OR Teaching OR School OR University OR Pedagogical ) AND Chatbot.

It combines education-related keywords with the “chatbot” keyword. Since chatbots are related to other technologies, the initial literature search also considered keywords such as “pedagogical agents,” “dialogue systems,” or “bots” when composing the search query. However, these increased the number of irrelevant results significantly and were therefore excluded from the query in later searches.

Inclusion and Exclusion Criteria

The queries were executed on 23.12.2020 and applied twice to each database, first as a title search query and secondly as a keyword-based search. This resulted in a total of 3.619 hits, which were checked for duplicates resulting in 2.678 candidate publications. The overall search and filtering process is shown in Figure 5 .

www.frontiersin.org

FIGURE 5 . PRISMA flow chart.

In the case of Google Scholar, the number of results sorted by relevance per query was limited to 300, as this database also delivers many less relevant works. The value was determined by looking at the search results in detail using several queries to exclude as few relevant works as possible. This approach showed promising results and, at the same time, did not burden the literature list with irrelevant items.

The further screening consisted of a four-stage filtering process. First, eliminating duplicates in the results of title and keyword queries of all databases independently and second, excluding publications based on the title and abstract that:

• were not available in English

• did not describe a chatbot application

• were not mainly focused on learner-centered chatbots applications in schools or higher education institutions, which is according to the preliminary literature search the main application area within education.

Third, we applied another duplicate filter, this time for the merged set of publications. Finally, a filter based on the full text, excluding publications that were:

• limited to improve chatbots technically (e.g., publications that compare or develop new algorithms), as research questions presented in these publications were not seeking for additional insights on applications in education

• exclusively theoretical in nature (e.g., publications that discuss new research projects, implementation concepts, or potential use cases of chatbots in education), as they either do not contain research questions or hypotheses or do not provide conclusions from studies with learners.

After the first, second, and third filters, we identified 505 candidate publications. We continued our filtering process by reading the candidate publications’ full texts resulting in 74 publications that were used for our review. Compared to 3.619 initial database results, the proportion of relevant publications is therefore about 2.0%.

The final publication list can be accessed under https://bit.ly/2RRArFT .

To analyze the identified publications and derive results according to the research questions, full texts were coded, considering for each publication the objectives for implementing chatbots (RQ1), pedagogical roles of chatbots (RQ2), their mentoring roles (RQ3), adaptation of chatbots (RQ4), as well as their implementation domains in education (RQ5) as separated sets of codes. To this end, initial codes were identified by open coding and iteratively improved through comparison, group discussion among the authors, and subsequent code expansion. Further, codes were supplemented with detailed descriptions until a saturation point was reached, where all included studies could be successfully mapped to codes, suggesting no need for further refinement. As an example, codes for RQ2 (Pedagogical Roles) were adapted and refined in terms of their level of abstraction from an initial set of only two codes, 1 ) a code for chatbots in the learning role and 2 ) a code for chatbots in a service-oriented role. After coding a larger set of publications, it became clear that the code for service-oriented chatbots needed to be further distinguished. This was because it summarized e.g. automation activities with activities related to self-regulated learning and thus could not be distinguished sharply enough from the learning role. After refining the code set in the next iteration into a learning role, an assistance role, and a mentoring role, it was then possible to ensure the separation of the individual codes. In order to avoid defining new codes for singular or a very small number of publications, studies were coded as “other” (RQ1) or “not defined” (RQ2), if their occurrence was less than eight publications, representing less than 10% of the publications in the final paper list.

By grouping the resulting relevant publications according to their date of publication, it is apparent that chatbots in education are currently in a phase of increased attention. The release distribution shows slightly lower publication numbers in the current than in the previous year ( Figure 6 ), which could be attributed to a time lag between the actual publication of manuscripts and their dissemination in databases.

www.frontiersin.org

FIGURE 6 . Identified chatbot publications in education per year.

Applying the curve presented in Figure 6 to Gartner’s Hype Cycle ( Linden and Fenn, 2003 ) suggests that technology around chatbots in education may currently be in the “Innovation Trigger” phase. This phase is where many expectations are placed on the technology, but the practical in-depth experience is still largely lacking.

Objectives for Implementing Chatbots in Education

Regarding RQ1, we extracted implementation objectives for chatbots in education. By analyzing the selected publications we identified that most of the objectives for chatbots in education can be described by one of the following categories: Skill improvement, Efficiency of Education, and Students’ Motivation ( see Figure 7 ). First, the “improvement of a student’s skill” (or Skill Improvement ) objective that the chatbot is supposed to help with or achieve. Here, chatbots are mostly seen as a learning aid that supports students. It is the most commonly cited objective for chatbots. The second objective is to increase the Efficiency of Education in general. It can occur, for example, through the automation of recurring tasks or time-saving services for students and is the second most cited objective for chatbots. The third objective is to increase Students’ Motivation . Finally, the last objective is to increase the Availability of Education . This objective is intended to provide learning or counseling with temporal flexibility or without the limitation of physical presence. In addition, there are other, more diverse objectives for chatbots in education that are less easy to categorize. In cases of a publication indicating more than one objective, the publication was distributed evenly across the respective categories.

www.frontiersin.org

FIGURE 7 . Objectives for implementing chatbots identified in chatbot publications.

Given these results, we can summarize four major implementing objectives for chatbots. Of these, Skill Improvement is the most popular objective, constituting around one-third of publications (32%). Making up a quarter of all publications, Efficiency of Education is the second most popular objective (25%), while addressing Students’ Motivation and Availability of Education are third (13%) and fourth (11%), respectively. Other objectives also make up a substantial amount of these publications (19%), although they were too diverse to categorize in a uniform way. Examples of these are inclusivity ( Heo and Lee, 2019 ) or the promotion of student teacher interactions ( Mendoza et al., 2020 ).

Pedagogical Roles

Regarding RQ2, it is crucial to consider the use of chatbots in terms of their intended pedagogical role. After analyzing the selected articles, we were able to identify four different pedagogical roles: a supporting learning role, an assisting role, and a mentoring role.

In the supporting learning role ( Learning ), chatbots are used as an educational tool to teach content or skills. This can be achieved through a fixed integration into the curriculum, such as conversation tasks (L. K. Fryer et al., 2020 ). Alternatively, learning can be supported through additional offerings alongside classroom teaching, for example, voice assistants for leisure activities at home ( Bao, 2019 ). Examples of these are chatbots simulating a virtual pen pal abroad ( Na-Young, 2019 ). Conversations with this kind of chatbot aim to motivate the students to look up vocabulary, check their grammar, and gain confidence in the foreign language.

In the assisting role ( Assisting ), chatbot actions can be summarized as simplifying the student's everyday life, i.e., taking tasks off the student’s hands in whole or in part. This can be achieved by making information more easily available ( Sugondo and Bahana, 2019 ) or by simplifying processes through the chatbot’s automation ( Suwannatee and Suwanyangyuen, 2019 ). An example of this is the chatbot in ( Sandoval, 2018 ) that answers general questions about a course, such as an exam date or office hours.

In the mentoring role ( Mentoring ), chatbot actions deal with the student’s personal development. In this type of support, the student himself is the focus of the conversation and should be encouraged to plan, reflect or assess his progress on a meta-cognitive level. One example is the chatbot in ( Cabales, 2019 ), which helps students develop lifelong learning skills by prompting in-action reflections.

The distribution of each pedagogical role is shown in Figure 8 . From this, it can be seen that Learning is the most frequently used role of the examined publications (49%), followed by Assisting (20%) and Mentoring (15%). It should be noted that pedagogical roles were not identified for all the publications examined. The absence of a clearly defined pedagogical role (16%) can be attributed to the more general nature of these publications, e.g. focused on students’ small talk behaviors ( Hobert, 2019b ) or teachers’ attitudes towards chatbot applications in classroom teaching (P. K. Bii et al., 2018 ).

www.frontiersin.org

FIGURE 8 . Pedagogical roles identified in chatbot publications.

Looking at pedagogical roles in the context of objectives for implementing chatbots, relations among publications can be inspected in a relations graph ( Figure 9 ). According to our results, the strongest relation in the examined publications can be considered between Skill Improvement objective and the Learning role. This strong relation is partly because both, the Skill Improvement objective and the Learning role, are the largest in their respective categories. In addition, two other strong relations can be observed: Between the Students’ Motivation objective and the Learning role, as well as between Efficiency of Education objective and Assisting role.

www.frontiersin.org

FIGURE 9 . Relations graph of pedagogical roles and objectives for implementing chatbots.

By looking at other relations in more detail, there is surprisingly no relation between Skill Improvement as the most common implementation objective and Assisting , as the 2nd most common pedagogical role. Furthermore, it can be observed that the Mentoring role has nearly equal relations to all of the objectives for implementing chatbots.

The relations graph ( Figure 9 ) can interactively be explored through bit.ly/32FSKQM.

Mentoring Role

Regarding RQ3, we identified eleven publications that deal with chatbots in this regard. The Mentoring role in these publications can be categorized in two dimensions. Starting with the first dimension, the mentoring method, three methods can be observed:

• Scaffolding ( n = 7)

• Recommending ( n = 3)

• Informing ( n = 1)

An example of Scaffolding can be seen in ( Gabrielli et al., 2020 ), where the chatbot coaches students in life skills, while an example of Recommending can be seen in ( Xiao et al., 2019 ), where the chatbot recommends new teammates. Finally, Informing can be seen in ( Kerly et al., 2008 ), where the chatbot informs students about their personal Open Learner Model.

The second dimension is the addressed mentoring topic, where the following topics can be observed:

• Self-Regulated Learning ( n = 5)

• Life Skills ( n = 4)

• Learning Skills ( n = 2)

While Mentoring chatbots to support Self-Regulated Learning are intended to encourage students to reflect on and plan their learning progress, Mentoring chatbots to support Life Skills address general student’s abilities such as self-confidence or managing emotions. Finally, Mentoring chatbots to support Learning Skills , in contrast to Self-Regulated Learning , address only particular aspects of the learning process, such as new learning strategies or helpful learning partners. An example for Mentoring chatbots supporting Life Skill is the Logo counseling chatbot, which promotes healthy self-esteem ( Engel et al., 2020 ). CALMsystem is an example of a Self-Regulated Learning chatbot, which informs students about their data in an open learner model ( Kerly et al., 2008 ). Finally, there is the Learning Skills topic. Here, the MCQ Bot is an example that is designed to introduce students to transformative learning (W. Huang et al., 2019 ).

Regarding RQ4, we identified six publications in the final publication list that address the topic of adaptation. Within these publications, five adaptation approaches are described:

The first approach (A1) is proposed by ( Kerly and Bull, 2006 ) and ( Kerly et al., 2008 ), dealing with student discussions based on success and confidence during a quiz. The improvement of self-assessment is the primary focus of this approach. The second approach (A2) is presented in ( Jia, 2008 ), where the personality of the chatbot is adapted to motivate students to talk to the chatbot and, in this case, learn a foreign language. The third approach (A3), as shown in the work of ( Vijayakumar et al., 2019 ), is characterized by a chatbot that provides personalized formative feedback to learners based on their self-assessment, again in a quiz situation. Here, the focus is on Hattie and Timperley’s three guiding questions: “Where am I going?,” “How am I going?” and “Where to go next?” ( Hattie and Timperley, 2007 ). In the fourth approach (A4), exemplified in ( Ruan et al., 2019 ), the chatbot selects questions within a quiz. Here, the chatbot estimates the student’s ability and knowledge level based on the quiz progress and sets the next question accordingly. Finally, a similar approach (A5) is shown in ( Davies et al., 2020 ). In contrast to ( Ruan et al., 2019 ), this chatbot adapts the amount of question variation and takes psychological features into account which were measured by psychological tests before.

We examined these five approaches by organizing them according to their information sources and extracted learner information. The results can be seen in Table 2 .

www.frontiersin.org

TABLE 2 . Adaptation approaches of chatbots in education.

Four out of five adaptation approaches (A1, A3, A4, and A5) are observed in the context of quizzes. These adaptations within quizzes can be divided into two mainstreams: One is concerned about students’ feedback (A1 and A3), while the other is concerned about learning material selection (A4 and A5). The only different adaptation approach is shown in A2, which focuses on the adaptation of the chatbot personality within a language learning application.

Domains for Chatbots in Education

Regarding RQ5, we identified 20 domains of chatbots in education. These can broadly be divided by their pedagogical role into three domain categories (DC): Learning Chatbots , Assisting Chatbots , and Mentoring Chatbots . The remaining publications are grouped in the Other Research domain category. The complete list of identified domains can be seen in Table 3 .

www.frontiersin.org

TABLE 3 . Domains of chatbots in education.

The domain category Learning Chatbots , which deals with chatbots incorporating the pedagogical role Learning , can be subdivided into seven domains: 1 ) Language Learning , 2 ) Learn to Program , 3 ) Learn Communication Skills , 4 ) Learn about Educational Technologies , 5 ) Learn about Cultural Heritage , 6 ) Learn about Laws , and 7 ) Mathematics Learning . With more than half of publications (53%), chatbots for Language Learning play a prominent role in this domain category. They are often used as chat partners to train conversations or to test vocabulary. An example of this can be seen in the work of ( Bao, 2019 ), which tries to mitigate foreign language anxiety by chatbot interactions in foreign languages.

The domain category Assisting Chatbots , which deals with chatbots incorporating the pedagogical role Assisting , can be subdivided into four domains: 1 ) Administrative Assistance , 2 ) Campus Assistance , 3 ) Course Assistance , and 4 ) Library Assistance . With one-third of publications (33%), chatbots in the Administrative Assistance domain that help to overcome bureaucratic hurdles at the institution, while providing round-the-clock services, are the largest group in this domain category. An example of this can be seen in ( Galko et al., 2018 ), where the student enrollment process is completely shifted to a conversation with a chatbot.

The domain category Mentoring Chatbots , which deals with chatbots incorporating the pedagogical role Mentoring , can be subdivided into three domains: 1 ) Scaffolding Chatbots , 2 ) Recommending Chatbots , and 3 ) Informing Chatbots . An example of a Scaffolding Chatbots is the CRI(S) chatbot ( Gabrielli et al., 2020 ), which supports life skills such as self-awareness or conflict resolution in discussion with the student by promoting helpful ideas and tricks.

The domain category Other Research , which deals with chatbots not incorporating any of these pedagogical roles, can be subdivided into three domains: 1 ) General Chatbot Research in Education , 2 ) Indian Educational System , and 3 ) Chatbot Interfaces . The most prominent domain, General Chatbot Research , cannot be classified in one of the other categories but aims to explore cross-cutting issues. An example for this can be seen in the publication of ( Hobert, 2020 ), which researches the importance of small talk abilities of chatbots in educational settings.

Discussions

In this paper, we investigated the state-of-the-art of chatbots in education according to five research questions. By combining our results with previously identified findings from related literature reviews, we proposed a concept map of chatbots in education. The map, reported in Appendix A , displays the current state of research regarding chatbots in education with the aim of supporting future research in the field.

Answer to Research Questions

Concerning RQ1 (implementation objectives), we identified four major objectives: 1 ) Skill Improvement , 2 ) Efficiency of Education , 3 ) Students’ Motivation, and 4 ) Availability of Education . These four objectives cover over 80% of the analyzed publications ( see Figure 7 ). Based on the findings on CAT3 in section 2, we see a mismatch between the objectives for implementing chatbots compared to their evaluation. Most researchers only focus on narrow aspects for the evaluation of their chatbots such as learning success, usability, and technology acceptance. This mismatch of implementation objectives and suitable evaluation approaches is also well known by other educational technologies such as Learning Analytics dashboards ( Jivet et al., 2017 ). A more structured approach of aligning implementation objectives and evaluation procedures is crucial to be able to properly assess the effectiveness of chatbots. ( Hobert, 2019a ), suggested a structured four-stage evaluation procedure beginning with a Wizard-of-Oz experiment, followed by technical validation, a laboratory study, and a field study. This evaluation procedure systematically links hypotheses with outcomes of chatbots helping to assess chatbots for their implementation objectives. “Aligning chatbot evaluations with implementation objectives” is, therefore, an important challenge to be addressed in the future research agenda.

Concerning RQ2 (pedagogical roles), our results show that chatbots’ pedagogical roles can be summarized as Learning , Assisting , and Mentoring . The Learning role is the support in learning or teaching activities such as gaining knowledge. The Assisting role is the support in terms of simplifying learners’ everyday life, e.g. by providing opening times of the library. The Mentoring role is the support in terms of students’ personal development, e.g. by supporting Self-Regulated Learning. From a pedagogical standpoint, all three roles are essential for learners and should therefore be incorporated in chatbots. These pedagogical roles are well aligned with the four implementation objectives reported in RQ1. While Skill Improvement and Students’ Motivation is strongly related to Learning , Efficiency of Education is strongly related to Assisting . The Mentoring role instead, is evenly related to all of the identified objectives for implementing chatbots. In the reviewed publications, chatbots are therefore primarily intended to 1 ) improve skills and motivate students by supporting learning and teaching activities, 2 ) make education more efficient by providing relevant administrative and logistical information to learners, and 3 ) support multiple effects by mentoring students.

Concerning RQ3 (mentoring role), we identified three main mentoring method categories for chatbots: 1 ) Scaffolding , 2 ) Recommending , and 3 ) Informing . However, comparing the current mentoring of chatbots reported in the literature with the daily mentoring role of teachers, we can summarize that the chatbots are not at the same level. In order to take over mentoring roles of teachers ( Wildman et al., 1992 ), a chatbot would need to fulfill some of the following activities in their mentoring role. With respect to 1 ) Scaffolding , chatbots should provide direct assistance while learning new skills and especially direct beginners in their activities. Regarding 2 ) Recommending , chatbots should provide supportive information, tools or other materials for specific learning tasks to life situations. With respect to 3 ) Informing, chatbots should encourage students according to their goals and achievements, and support them to develop meta-cognitive skills like self-regulation. Due to the mismatch of teacher vs. chatbot mentoring we see here another research challenge, which we call “Exploring the potential of chatbots for mentoring students.”

Regarding RQ4 (adaptation), only six publications were identified that discuss an adaptation of chatbots, while four out of five adaptation approaches (A1, A3, A4, and A5) show similarities by being applied within quizzes. In the context of educational technologies, providing reasonable adaptations for learners requires a high level of experience. Based on our results, the research on chatbots does not seem to be at this point yet. Looking at adaptation literature like ( Brusilovsky, 2001 ) or ( Benyon and Murray, 1993 ), it becomes clear that a chatbot needs to consider the learners’ personal information to fulfill the requirement of the adaptation definition. Personal information must be retrieved and stored at least temporarily, in some sort of learner model. For learner information like knowledge and interest, adaptations seem to be barely explored in the reviewed publications, while the model of ( Brusilovsky and Millán, 2007 ) points out further learner information, which can be used to make chatbots more adaptive: personal goals, personal tasks, personal background, individual traits, and the learner’s context. We identify research in this area as a third future challenge and call it the “Exploring and leveraging adaptation capabilities of chatbots” challenge.

In terms of RQ5 (domains), we identified a detailed map of domains applying chatbots in education and their distribution ( see Table 3 ). By systematically analyzing 74 publications, we identified 20 domains and structured them according to the identified pedagogical role into four domain categories: Learning Chatbots , Assisting Chatbots , Mentoring Chatbots , and Other Research . These results extend the taxonomy of Application Clusters (AC) for chatbots in education, which previously comprised the work from ( Pérez et al., 2020 ), who took the chatbot activity as characteristic, and ( Winkler and Soellner, 2018 ), who characterized the chatbots by domains. It draws relationships between these two types of Application Clusters (AC) and structures them accordingly. Our structure incorporates Mentoring Chatbots and Other Research in addition to the “service-oriented chatbots” (cf. Assisting Chatbots ) and “teaching-oriented chatbots” (cf. Learning Chatbots ) identified by (Perez). Furthermore, the strong tendencies of informing students already mentioned by ( Smutny and Schreiberova, 2020 ) can also be recognized in our results, especially in Assisting Chatbots . Compared to ( Winkler and Soellner, 2018 ), we can confirm the prominent domains of “language learning” within Learning Chatbots and “metacognitive thinking” within Mentoring Chatbots . Moreover, through Table 3 , a more detailed picture of chatbot applications in education is reflected, which could help researchers to find similar works or unexplored application areas.

Limitations

One important limitation to be mentioned here is the exclusion of alternative keywords for our search queries, as we exclusively used chatbot as keyword in order to avoid search results that do not fit our research questions. Though we acknowledge that chatbots share properties with pedagogical agents, dialog systems, and bots, we carefully considered this trade-off between missing potentially relevant work and inflating our search procedure by including related but not necessarily pertinent work. A second limitation may lie in the formation of categories and coding processes applied, which, due to the novelty of the findings, could not be built upon theoretical frameworks or already existing code books. Although we have focused on ensuring that codes used contribute to a strong understanding, the determination of the abstraction level might have affected the level of detail of the resulting data representation.

In this systematic literature review, we explored the current landscape of chatbots in education. We analyzed 74 publications, identified 20 domains of chatbots and grouped them based on their pedagogical roles into four domain categories. These pedagogical roles are the supporting learning role ( Learning ), the assisting role ( Assisting ), and the mentoring role ( Mentoring ). By focusing on objectives for implementing chatbots, we identified four main objectives: 1 ) Skill Improvement , 2 ) Efficiency of Education , 3 ) Students’ Motivation, and 4 ) Availability of Education . As discussed in section 5, these objectives do not fully align with the chosen evaluation procedures. We focused on the relations between pedagogical roles and objectives for implementing chatbots and identified three main relations: 1 ) chatbots to improve skills and motivate students by supporting learning and teaching activities, 2 ) chatbots to make education more efficient by providing relevant administrative and logistical information to learners, and 3 ) chatbots to support multiple effects by mentoring students. We focused on chatbots incorporating the Mentoring role and found that these chatbots are mostly concerned with three mentoring topics 1 ) Self-Regulated Learning , 2 ) Life Skills , and 3 ) Learning Skills and three mentoring methods 1 ) Scaffolding , 2 ) Recommending , and 3 ) Informing . Regarding chatbot adaptations, only six publications with adaptations were identified. Furthermore, the adaptation approaches found were mostly limited to applications within quizzes and thus represent a research gap.

Based on these outcomes we consider three challenges for chatbots in education that offer future research opportunities:

Challenge 1: Aligning chatbot evaluations with implementation objectives . Most chatbot evaluations focus on narrow aspects to measure the tool’s usability, acceptance or technical correctness. If chatbots should be considered as learning aids, student mentors, or facilitators, the effects on the cognitive, and emotional levels should also be taken into account for the evaluation of chatbots. This finding strengthens our conclusion that chatbot development in education is still driven by technology, rather than having a clear pedagogical focus of improving and supporting learning.

Challenge 2: Exploring the potential of chatbots for mentoring students . In order to better understand the potentials of chatbots to mentor students, more empirical studies on the information needs of learners are required. It is obvious that these needs differ from schools to higher education. However, so far there are hardly any studies investigating the information needs with respect to chatbots nor if chatbots address these needs sufficiently.

Challenge 3: Exploring and leveraging adaptation capabilities of chatbots . There is a large literature on adaptation capabilities of educational technologies. However, we have seen very few studies on the effect of adaptation of chatbots for education purposes. As chatbots are foreseen as systems that should personally support learners, the area of adaptable interactions of chatbots is an important research aspect that should receive more attention in the near future.

By addressing these challenges, we believe that chatbots can become effective educational tools capable of supporting learners with informative feedback. Therefore, looking at our results and the challenges presented, we conclude, “No, we are not there yet!” - There is still much to be done in terms of research on chatbots in education. Still, development in this area seems to have just begun to gain momentum and we expect to see new insights in the coming years.

Data Availability Statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding authors.

Author Contributions

SW, JS†, DM†, JW†, MR, and HD.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Abbasi, S., Kazi, H., and Hussaini, N. N. (2019). Effect of Chatbot Systems on Student’s Learning Outcomes. Sylwan 163(10).

CrossRef Full Text

Abbasi, S., and Kazi, H. (2014). Measuring Effectiveness of Learning Chatbot Systems on Student's Learning Outcome and Memory Retention. Asian J. Appl. Sci. Eng. 3, 57. doi:10.15590/AJASE/2014/V3I7/53576

CrossRef Full Text | Google Scholar

Almahri, F. A. J., Bell, D., and Merhi, M. (2020). “Understanding Student Acceptance and Use of Chatbots in the United Kingdom Universities: A Structural Equation Modelling Approach,” in 2020 6th IEEE International Conference on Information Management, ICIM 2020 , London, United Kingdom , March 27–29, 2020 , (IEEE), 284–288. doi:10.1109/ICIM49319.2020.244712

Bao, M. (2019). Can Home Use of Speech-Enabled Artificial Intelligence Mitigate Foreign Language Anxiety - Investigation of a Concept. Awej 5, 28–40. doi:10.24093/awej/call5.3

Benyon, D., and Murray, D. (1993). Applying User Modeling to Human-Computer Interaction Design. Artif. Intell. Rev. 7 (3-4), 199–225. doi:10.1007/BF00849555

Bii, P. K., Too, J. K., and Mukwa, C. W. (2018). Teacher Attitude towards Use of Chatbots in Routine Teaching. Univers. J. Educ. Res. . 6 (7), 1586–1597. doi:10.13189/ujer.2018.060719

Bii, P., Too, J., and Langat, R. (2013). An Investigation of Student’s Attitude Towards the Use of Chatbot Technology in Instruction: The Case of Knowie in a Selected High School. Education Research 4, 710–716. doi:10.14303/er.2013.231

Google Scholar

Bos, A. S., Pizzato, M. C., Vettori, M., Donato, L. G., Soares, P. P., Fagundes, J. G., et al. (2020). Empirical Evidence During the Implementation of an Educational Chatbot with the Electroencephalogram Metric. Creative Education 11, 2337–2345. doi:10.4236/CE.2020.1111171

Brusilovsky, P. (2001). Adaptive Hypermedia. User Model. User-Adapted Interaction 11 (1), 87–110. doi:10.1023/a:1011143116306

Brusilovsky, P., and Millán, E. (2007). “User Models for Adaptive Hypermedia and Adaptive Educational Systems,” in The Adaptive Web: Methods and Strategies of Web Personalization . Editors P. Brusilovsky, A. Kobsa, and W. Nejdl. Berlin: Springer , 3–53. doi:10.1007/978-3-540-72079-9_1

Cabales, V. (2019). “Muse: Scaffolding metacognitive reflection in design-based research,” in CHI EA’19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems , Glasgow, Scotland, United Kingdom , May 4–9, 2019 , (ACM), 1–6. doi:10.1145/3290607.3308450

Carayannopoulos, S. (2018). Using Chatbots to Aid Transition. Int. J. Info. Learn. Tech. 35, 118–129. doi:10.1108/IJILT-10-2017-0097

Chan, C. H., Lee, H. L., Lo, W. K., and Lui, A. K.-F. (2018). Developing a Chatbot for College Student Programme Advisement. in 2018 International Symposium on Educational Technology, ISET 2018 , Osaka, Japan , July 31–August 2, 2018 . Editors F. L. Wang, C. Iwasaki, T. Konno, O. Au, and C. Li, (IEEE), 52–56. doi:10.1109/ISET.2018.00021

Chang, M.-Y., and Hwang, J.-P. (2019). “Developing Chatbot with Deep Learning Techniques for Negotiation Course,” in 2019 8th International Congress on Advanced Applied Informatics, IIAI-AAI 2019 , Toyama, Japan , July 7–11, 2019 , (IEEE), 1047–1048. doi:10.1109/IIAI-AAI.2019.00220

Chen, C.-A., Yang, Y.-T., Wu, S.-M., Chen, H.-C., Chiu, K.-C., Wu, J.-W., et al. (2018). “A Study of Implementing AI Chatbot in Campus Consulting Service”, in TANET 2018-Taiwan Internet Seminar , 1714–1719. doi:10.6861/TANET.201810.0317

Chen, H.-L., Widarso, G. V., and Sutrisno, H. (2020). A ChatBot for Learning Chinese: Learning Achievement and Technology Acceptance. J. Educ. Comput. Res. 58 (6), 1161–1189. doi:10.1177/0735633120929622

Daud, S. H. M., Teo, N. H. I., and Zain, N. H. M. (2020). E-java Chatbot for Learning Programming Language: A post-pandemic Alternative Virtual Tutor. Int. J. Emerging Trends Eng. Res. 8(7). 3290–3298. doi:10.30534/ijeter/2020/67872020

Davies, J. N., Verovko, M., Verovko, O., and Solomakha, I. (2020). “Personalization of E-Learning Process Using Ai-Powered Chatbot Integration,” in Selected Papers of 15th International Scientific-practical Conference, MODS, 2020: Advances in Intelligent Systems and Computing , Chernihiv, Ukraine , June 29–July 01, 2020 . Editors S. Shkarlet, A. Morozov, and A. Palagin, ( Springer ) Vol. 1265, 209–216. doi:10.1007/978-3-030-58124-4_20

Diachenko, A. V., Morgunov, B. P., Melnyk, T. P., Kravchenko, O. I., and Zubchenko, L. V. (2019). The Use of Innovative Pedagogical Technologies for Automation of the Specialists' Professional Training. Int. J. Hydrogen. Energy. 8, 288–295. doi:10.5430/ijhe.v8n6p288

Dibitonto, M., Leszczynska, K., Tazzi, F., and Medaglia, C. M. (2018). “Chatbot in a Campus Environment: Design of Lisa, a Virtual Assistant to Help Students in Their university Life,” in 20th International Conference, HCI International 2018 , Las Vegas, NV, USA , July 15–20, 2018 , Lecture Notes in Computer Science. Editors M. Kurosu, (Springer), 103–116. doi:10.1007/978-3-319-91250-9

Durall, E., and Kapros, E. (2020). “Co-design for a Competency Self-Assessment Chatbot and Survey in Science Education,” in 7th International Conference, LCT 2020, Held as Part of the 22nd HCI International Conference, HCII 2020 , Copenhagen, Denmark , July 19–24, 2020 , Lecture Notes in Computer Science. Editors P. Zaphiris, and A. Ioannou, Berlin: Springer Vol. 12206, 13–23. doi:10.1007/978-3-030-50506-6_2

Duval, E., and Verbert, K. (2012). Learning Analytics. Eleed 8 (1).

Engel, J. D., Engel, V. J. L., and Mailoa, E. (2020). Interaction Monitoring Model of Logo Counseling Website for College Students' Healthy Self-Esteem, I. J. Eval. Res. Educ. 9, 607–613. doi:10.11591/ijere.v9i3.20525

Febriani, G. A., and Agustia, R. D. (2019). Development of Line Chatbot as a Learning Media for Mathematics National Exam Preparation. Elibrary.Unikom.Ac.Id . https://elibrary.unikom.ac.id/1130/14/UNIKOM_GISTY%20AMELIA%20FEBRIANI_JURNAL%20DALAM%20BAHASA%20INGGRIS.pdf .

Ferguson, R., and Sharples, M. (2014). “Innovative Pedagogy at Massive Scale: Teaching and Learning in MOOCs,” in 9th European Conference on Technology Enhanced Learning, EC-TEL 2014 , Graz, Austria , September 16–19, 2014 , Lecture Notes in Computer Science. Editors C. Rensing, S. de Freitas, T. Ley, and P. J. Muñoz-Merino, ( Berlin : Springer) Vol. 8719, 98–111. doi:10.1007/978-3-319-11200-8_8

Fryer, L. K., Ainley, M., Thompson, A., Gibson, A., and Sherlock, Z. (2017). Stimulating and Sustaining Interest in a Language Course: An Experimental Comparison of Chatbot and Human Task Partners. Comput. Hum. Behav. 75, 461–468. doi:10.1016/j.chb.2017.05.045

Fryer, L. K., Nakao, K., and Thompson, A. (2019). Chatbot Learning Partners: Connecting Learning Experiences, Interest and Competence. Comput. Hum. Behav. 93, 279–289. doi:10.1016/j.chb.2018.12.023

Fryer, L. K., Thompson, A., Nakao, K., Howarth, M., and Gallacher, A. (2020). Supporting Self-Efficacy Beliefs and Interest as Educational Inputs and Outcomes: Framing AI and Human Partnered Task Experiences. Learn. Individual Differences , 80. doi:10.1016/j.lindif.2020.101850

Gabrielli, S., Rizzi, S., Carbone, S., and Donisi, V. (2020). A Chatbot-Based Coaching Intervention for Adolescents to Promote Life Skills: Pilot Study. JMIR Hum. Factors 7 (1). doi:10.2196/16762

PubMed Abstract | CrossRef Full Text | Google Scholar

Galko, L., Porubän, J., and Senko, J. (2018). “Improving the User Experience of Electronic University Enrollment,” in 16th IEEE International Conference on Emerging eLearning Technologies and Applications, ICETA 2018 , Stary Smokovec, Slovakia , Nov 15–16, 2018 . Editors F. Jakab, (Piscataway, NJ: IEEE ), 179–184. doi:10.1109/ICETA.2018.8572054

Goda, Y., Yamada, M., Matsukawa, H., Hata, K., and Yasunami, S. (2014). Conversation with a Chatbot before an Online EFL Group Discussion and the Effects on Critical Thinking. J. Inf. Syst. Edu. 13, 1–7. doi:10.12937/EJSISE.13.1

Graesser, A. C., VanLehn, K., Rose, C. P., Jordan, P. W., and Harter, D. (2001). Intelligent Tutoring Systems with Conversational Dialogue. AI Mag. 22 (4), 39–51. doi:10.1609/aimag.v22i4.1591

Greller, W., and Drachsler, H. (2012). Translating Learning into Numbers: A Generic Framework for Learning Analytics. J. Educ. Tech. Soc. 15 (3), 42–57. doi:10.2307/jeductechsoci.15.3.42

Haristiani, N., and Rifa’i, M. M. Combining Chatbot and Social Media: Enhancing Personal Learning Environment (PLE) in Language Learning. Indonesian J Sci Tech. 5 (3), 487–506. doi:10.17509/ijost.v5i3.28687

Hattie, J., and Timperley, H. (2007). The Power of Feedback. Rev. Educ. Res. 77 (1), 81–112. doi:10.3102/003465430298487

Hattie, J. (2009). Visible Learning: A Synthesis of over 800 Meta-Analyses Relating to Achievement . Abingdon, UK: Routledge .

Heller, B., Proctor, M., Mah, D., Jewell, L., and Cheung, B. (2005). “Freudbot: An Investigation of Chatbot Technology in Distance Education,” in Proceedings of ED-MEDIA 2005–World Conference on Educational Multimedia, Hypermedia and Telecommunications , Montréal, Canada , June 27–July 2, 2005 . Editors P. Kommers, and G. Richards, ( AACE ), 3913–3918.

Heo, J., and Lee, J. (2019). “CiSA: An Inclusive Chatbot Service for International Students and Academics,” in 21st International Conference on Human-Computer Interaction, HCII 2019: Communications in Computer and Information Science , Orlando, FL, USA , July 26–31, 2019 . Editors C. Stephanidis, ( Springer ) 11786, 153–167. doi:10.1007/978-3-030-30033-3

Hobert, S. (2019a). “How Are You, Chatbot? Evaluating Chatbots in Educational Settings - Results of a Literature Review,” in 17. Fachtagung Bildungstechnologien, DELFI 2019 - 17th Conference on Education Technologies, DELFI 2019 , Berlin, Germany , Sept 16–19, 2019 . Editors N. Pinkwart, and J. Konert, 259–270. doi:10.18420/delfi2019_289

Hobert, S., and Meyer von Wolff, R. (2019). “Say Hello to Your New Automated Tutor - A Structured Literature Review on Pedagogical Conversational Agents,” in 14th International Conference on Wirtschaftsinformatik , Siegen, Germany , Feb 23–27, 2019 . Editors V. Pipek, and T. Ludwig, ( AIS ).

Hobert, S. (2019b). Say Hello to ‘Coding Tutor’! Design and Evaluation of a Chatbot-Based Learning System Supporting Students to Learn to Program in International Conference on Information Systems (ICIS) 2019 Conference , Munich, Germany , Dec 15–18, 2019 , AIS 2661, 1–17.

Hobert, S. (2020). Small Talk Conversations and the Long-Term Use of Chatbots in Educational Settings ‐ Experiences from a Field Study in 3rd International Workshop on Chatbot Research and Design, CONVERSATIONS 2019 , Amsterdam, Netherlands , November 19–20 : Lecture Notes in Computer Science. Editors A. Folstad, T. Araujo, S. Papadopoulos, E. Law, O. Granmo, E. Luger, and P. Brandtzaeg, ( Springer ) 11970, 260–272. doi:10.1007/978-3-030-39540-7_18

Hsieh, S.-W. (2011). Effects of Cognitive Styles on an MSN Virtual Learning Companion System as an Adjunct to Classroom Instructions. Edu. Tech. Society 2, 161–174.

Huang, J.-X., Kwon, O.-W., Lee, K.-S., and Kim, Y.-K. (2018). Improve the Chatbot Performance for the DB-CALL System Using a Hybrid Method and a Domain Corpus in Future-proof CALL: language learning as exploration and encounters–short papers from EUROCALL 2018 , Jyväskylä, Finland , Aug 22–25, 2018 . Editors P. Taalas, J. Jalkanen, L. Bradley, and S. Thouësny, ( Research-publishing.net ). doi:10.14705/rpnet.2018.26.820

Huang, W., Hew, K. F., and Gonda, D. E. (2019). Designing and Evaluating Three Chatbot-Enhanced Activities for a Flipped Graduate Course. Int. J. Mech. Engineer. Robotics. Research. 813–818. doi:10.18178/ijmerr.8.5.813-818

Ismail, M., and Ade-Ibijola, A. (2019). “Lecturer's Apprentice: A Chatbot for Assisting Novice Programmers,”in Proceedings - 2019 International Multidisciplinary Information Technology and Engineering Conference (IMITEC) , Vanderbijlpark, South Africa , (IEEE), 1–8. doi:10.1109/IMITEC45504.2019.9015857

Jia, J. (2008). “Motivate the Learners to Practice English through Playing with Chatbot CSIEC,” in 3rd International Conference on Technologies for E-Learning and Digital Entertainment, Edutainment 2008 , Nanjing, China , June 25–27, 2008 , Lecture Notes in Computer Science, (Springer) 5093, 180–191. doi:10.1007/978-3-540-69736-7_20

Jia, J. (2004). “The Study of the Application of a Keywords-Based Chatbot System on the Teaching of Foreign Languages,” in Proceedings of SITE 2004--Society for Information Technology and Teacher Education International Conference , Atlanta, Georgia, USA . Editors R. Ferdig, C. Crawford, R. Carlsen, N. Davis, J. Price, R. Weber, and D. Willis, (AACE), 1201–1207.

Jivet, I., Scheffel, M., Drachsler, H., and Specht, M. (2017). “Awareness is not enough: Pitfalls of learning analytics dashboards in the educational practice,” in 12th European Conference on Technology Enhanced Learning, EC-TEL 2017 , Tallinn, Estonia , September 12–15, 2017 , Lecture Notes in ComputerScience. Editors E. Lavoué, H. Drachsler, K. Verbert, J. Broisin, and M. Pérez-Sanagustín, (Springer), 82–96. doi:10.1007/978-3-319-66610-5_7

Jung, H., Lee, J., and Park, C. (2020). Deriving Design Principles for Educational Chatbots from Empirical Studies on Human-Chatbot Interaction. J. Digit. Contents Society , 21, 487–493. doi:10.9728/dcs.2020.21.3.487

Kerly, A., and Bull, S. (2006). “The Potential for Chatbots in Negotiated Learner Modelling: A Wizard-Of-Oz Study,” in 8th International Conference on Intelligent Tutoring Systems, ITS 2006 , Jhongli, Taiwan , June 26–30, 2006 , Lecture Notes in Computer Science. Editors M. Ikeda, K. D. Ashley, and T. W. Chan, ( Springer ) 4053, 443–452. doi:10.1007/11774303

Kerly, A., Ellis, R., and Bull, S. (2008). CALMsystem: A Conversational Agent for Learner Modelling. Knowledge-Based Syst. 21, 238–246. doi:10.1016/j.knosys.2007.11.015

Kerly, A., Hall, P., and Bull, S. (2007). Bringing Chatbots into Education: Towards Natural Language Negotiation of Open Learner Models. Knowledge-Based Syst. , 20, 177–185. doi:10.1016/j.knosys.2006.11.014

Kumar, M. N., Chandar, P. C. L., Prasad, A. V., and Sumangali, K. (2016). “Android Based Educational Chatbot for Visually Impaired People,” in 2016 IEEE International Conference on Computational Intelligence and Computing Research , Chennai, India , December 15–17, 2016 , 1–4. doi:10.1109/ICCIC.2016.7919664

Lee, K., Jo, J., Kim, J., and Kang, Y. (2019). Can Chatbots Help Reduce the Workload of Administrative Officers? - Implementing and Deploying FAQ Chatbot Service in a University in 21st International Conference on Human-Computer Interaction, HCII 2019: Communications in Computer and Information Science , Orlando, FL, USA , July 26–31, 2019 . Editors C. Stephanidis, ( Springer ) 1032, 348–354. doi:10.1007/978-3-030-23522-2

Lester, J. C., Converse, S. A., Kahler, S. E., Barlow, S. T., Stone, B. A., and Bhogal, R. S. (1997). “The Persona Effect: Affective Impact of Animated Pedagogical Agents,” in Proceedings of the ACM SIGCHI Conference on Human factors in computing systems , Atlanta, Georgia, USA , March 22–27, 1997 , (ACM), 359–366.

Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., et al. (2009). The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies that Evaluate Health Care Interventions: Explanation and Elaboration. J. Clin. Epidemiol. 62 (10), e1–e34. doi:10.1016/j.jclinepi.2009.06.006

Lin, M. P.-C., and Chang, D. (2020). Enhancing Post-secondary Writers’ Writing Skills with a Chatbot. J. Educ. Tech. Soc. 23, 78–92. doi:10.2307/26915408

Lin, Y.-H., and Tsai, T. (2019). “A Conversational Assistant on Mobile Devices for Primitive Learners of Computer Programming,” in TALE 2019 - 2019 IEEE International Conference on Engineering, Technology and Education , Yogyakarta, Indonesia , December 10–13, 2019 , (IEEE), 1–4. doi:10.1109/TALE48000.2019.9226015

Linden, A., and Fenn, J. (2003). Understanding Gartner’s Hype Cycles. Strategic Analysis Report No. R-20-1971 8. Stamford, CT: Gartner, Inc .

Liu, Q., Huang, J., Wu, L., Zhu, K., and Ba, S. (2020). CBET: Design and Evaluation of a Domain-specific Chatbot for mobile Learning. Univ. Access Inf. Soc. , 19, 655–673. doi:10.1007/s10209-019-00666-x

Mamani, J. R. C., Álamo, Y. J. R., Aguirre, J. A. A., and Toledo, E. E. G. (2019). “Cognitive Services to Improve User Experience in Searching for Academic Information Based on Chatbot,” in Proceedings of the 2019 IEEE 26th International Conference on Electronics, Electrical Engineering and Computing (INTERCON) , Lima, Peru , August 12–14, 2019 , (IEEE), 1–4. doi:10.1109/INTERCON.2019.8853572

Martín-Martín, A., Orduna-Malea, E., Thelwall, M., and Delgado López-Cózar, E. (2018). Google Scholar, Web of Science, and Scopus: A Systematic Comparison of Citations in 252 Subject Categories. J. Informetrics 12 (4), 1160–1177. doi:10.1016/j.joi.2018.09.002

Matsuura, S., and Ishimura, R. (2017). Chatbot and Dialogue Demonstration with a Humanoid Robot in the Lecture Class, in 11th International Conference on Universal Access in Human-Computer Interaction, UAHCI 2017, held as part of the 19th International Conference on Human-Computer Interaction, HCI 2017 , Vancouver, Canada , July 9–14, 2017 , Lecture Notes in Computer Science. Editors M. Antona, and C. Stephanidis, (Springer) Vol. 10279, 233–246. doi:10.1007/978-3-319-58700-4

Matsuura, S., and Omokawa, R. (2020). Being Aware of One’s Self in the Auto-Generated Chat with a Communication Robot in UAHCI 2020 , 477–488. doi:10.1007/978-3-030-49282-3

McLoughlin, C., and Oliver, R. (1998). Maximising the Language and Learning Link in Computer Learning Environments. Br. J. Educ. Tech. 29 (2), 125–136. doi:10.1111/1467-8535.00054

Mendoza, S., Hernández-León, M., Sánchez-Adame, L. M., Rodríguez, J., Decouchant, D., and Meneses-Viveros, A. (2020). “Supporting Student-Teacher Interaction through a Chatbot,” in 7th International Conference, LCT 2020, Held as Part of the 22nd HCI International Conference, HCII 2020 , Copenhagen, Denmark , July 19–24, 2020 , Lecture Notes in Computer Science. Editors P. Zaphiris, and A. Ioannou, ( Springer ) 12206, 93–107. doi:10.1007/978-3-030-50506-6

Meyer, V., Wolff, R., Nörtemann, J., Hobert, S., and Schumann, M. (2020). “Chatbots for the Information Acquisition at Universities ‐ A Student’s View on the Application Area,“in 3rd International Workshop on Chatbot Research and Design, CONVERSATIONS 2019 , Amsterdam, Netherlands , November 19–20 , Lecture Notes in Computer Science. Editors A. Folstad, T. Araujo, S. Papadopoulos, E. Law, O. Granmo, E. Luger, and P. Brandtzaeg, (Springer) 11970, 231–244. doi:10.1007/978-3-030-39540-7

Na-Young, K. (2018c). A Study on Chatbots for Developing Korean College Students’ English Listening and Reading Skills. J. Digital Convergence 16. 19–26. doi:10.14400/JDC.2018.16.8.019

Na-Young, K. (2019). A Study on the Use of Artificial Intelligence Chatbots for Improving English Grammar Skills. J. Digital Convergence 17, 37–46. doi:10.14400/JDC.2019.17.8.037

Na-Young, K. (2018a). Chatbots and Korean EFL Students’ English Vocabulary Learning. J. Digital Convergence 16. 1–7. doi:10.14400/JDC.2018.16.2.001

Na-Young, K. (2018b). Different Chat Modes of a Chatbot and EFL Students’ Writing Skills Development . 1225–4975. doi:10.16933/sfle.2017.32.1.263

Na-Young, K. (2017). Effects of Different Types of Chatbots on EFL Learners’ Speaking Competence and Learner Perception. Cross-Cultural Studies 48, 223–252. doi:10.21049/ccs.2017.48.223

Nagata, R., Hashiguchi, T., and Sadoun, D. (2020). Is the Simplest Chatbot Effective in English Writing Learning Assistance?, in 16th International Conference of the Pacific Association for Computational Linguistics , PACLING, Hanoi, Vietnam , October 11–13, 2019 , Communications in Computer and Information Science. Editors L.-M. Nguyen, S. Tojo, X.-H. Phan, and K. Hasida, ( Springer ) Vol. 1215, 245–246. doi:10.1007/978-981-15-6168-9

Nelson, T. O., and Narens, L. (1994). Why Investigate Metacognition. in Metakognition: Knowing About Knowing . Editors J. Metcalfe, and P. Shimamura, (MIT Press) 13, 1–25.

Nghi, T. T., Phuc, T. H., and Thang, N. T. (2019). Applying Ai Chatbot for Teaching a Foreign Language: An Empirical Research. Int. J. Sci. Res. 8.

Ondas, S., Pleva, M., and Hládek, D. (2019). How Chatbots Can Be Involved in the Education Process. in ICETA 2019 - 17th IEEE International Conference on Emerging eLearning Technologies and Applications, Proceedings, Stary Smokovec , Slovakia , November 21–22, 2019 . Editors F. Jakab, (IEEE), 575–580. doi:10.1109/ICETA48886.2019.9040095

Pereira, J., Fernández-Raga, M., Osuna-Acedo, S., Roura-Redondo, M., Almazán-López, O., and Buldón-Olalla, A. (2019). Promoting Learners' Voice Productions Using Chatbots as a Tool for Improving the Learning Process in a MOOC. Tech. Know Learn. 24, 545–565. doi:10.1007/s10758-019-09414-9

Pérez, J. Q., Daradoumis, T., and Puig, J. M. M. (2020). Rediscovering the Use of Chatbots in Education: A Systematic Literature Review. Comput. Appl. Eng. Educ. 28, 1549–1565. doi:10.1002/cae.22326

Pérez-Marín, D. (2021). A Review of the Practical Applications of Pedagogic Conversational Agents to Be Used in School and University Classrooms. Digital 1 (1), 18–33. doi:10.3390/digital1010002

Pham, X. L., Pham, T., Nguyen, Q. M., Nguyen, T. H., and Cao, T. T. H. (2018). “Chatbot as an Intelligent Personal Assistant for mobile Language Learning,” in ACM International Conference Proceeding Series doi:10.1145/3291078.3291115

Quincey, E. de., Briggs, C., Kyriacou, T., and Waller, R. (2019). “Student Centred Design of a Learning Analytics System,” in Proceedings of the 9th International Conference on Learning Analytics & Knowledge , Tempe Arizona, USA , March 4–8, 2019 , (ACM), 353–362. doi:10.1145/3303772.3303793

Ram, A., Prasad, R., Khatri, C., Venkatesh, A., Gabriel, R., Liu, Q, et al. (2018). Conversational Ai: The Science behind the Alexa Prize, in 1st Proceedings of Alexa Prize (Alexa Prize 2017) . ArXiv [Preprint]. Available at: https://arxiv.org/abs/1801.03604 .

Rebaque-Rivas, P., and Gil-Rodríguez, E. (2019). Adopting an Omnichannel Approach to Improve User Experience in Online Enrolment at an E-Learning University, in 21st International Conference on Human-Computer Interaction, HCII 2019: Communications in Computer and Information Science , Orlando, FL, USA , July 26–31, 2019 . Editors C. Stephanidis, ( Springer ), 115–122. doi:10.1007/978-3-030-23525-3

Robinson, C. (2019). Impressions of Viability: How Current Enrollment Management Personnel And Former Students Perceive The Implementation of A Chatbot Focused On Student Financial Communication. Higher Education Doctoral Projects.2 . https://aquila.usm.edu/highereddoctoralprojects/2 .

Ruan, S., Jiang, L., Xu, J., Tham, B. J.-K., Qiu, Z., Zhu, Y., Murnane, E. L., Brunskill, E., and Landay, J. A. (2019). “QuizBot: A Dialogue-based Adaptive Learning System for Factual Knowledge,” in 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019 , Glasgow, Scotland, United Kingdom , May 4–9, 2019 , (ACM), 1–13. doi:10.1145/3290605.3300587

Sandoval, Z. V. (2018). Design and Implementation of a Chatbot in Online Higher Education Settings. Issues Inf. Syst. 19, 44–52. doi:10.48009/4.iis.2018.44-52

Sandu, N., and Gide, E. (2019). “Adoption of AI-Chatbots to Enhance Student Learning Experience in Higher Education in india,” in 18th International Conference on Information Technology Based Higher Education and Training , Magdeburg, Germany , September 26–27, 2019 , (IEEE), 1–5. doi:10.1109/ITHET46829.2019.8937382

Saygin, A. P., Cicekli, I., and Akman, V. (2000). Turing Test: 50 Years Later. Minds and Machines 10 (4), 463–518. doi:10.1023/A:1011288000451

Sinclair, A., McCurdy, K., Lucas, C. G., Lopez, A., and Gaševic, D. (2019). “Tutorbot Corpus: Evidence of Human-Agent Verbal Alignment in Second Language Learner Dialogues,” in EDM 2019 - Proceedings of the 12th International Conference on Educational Data Mining .

Smutny, P., and Schreiberova, P. (2020). Chatbots for Learning: A Review of Educational Chatbots for the Facebook Messenger. Comput. Edu. 151, 103862. doi:10.1016/j.compedu.2020.103862

Song, D., Rice, M., and Oh, E. Y. (2019). Participation in Online Courses and Interaction with a Virtual Agent. Int. Rev. Res. Open. Dis. 20, 44–62. doi:10.19173/irrodl.v20i1.3998

Stapić, Z., Horvat, A., and Vukovac, D. P. (2020). Designing a Faculty Chatbot through User-Centered Design Approach, in 22nd International Conference on Human-Computer Interaction,HCII 2020 , Copenhagen, Denmark , July 19–24, 2020 , Lecture Notes in Computer Science. Editors C. Stephanidis, D. Harris, W. C. Li, D. D. Schmorrow, C. M. Fidopiastis, and P. Zaphiris, ( Springer ), 472–484. doi:10.1007/978-3-030-60128-7

Subramaniam, N. K. (2019). Teaching and Learning via Chatbots with Immersive and Machine Learning Capabilities. In International Conference on Education (ICE 2019) Proceedings , Kuala Lumpur, Malaysia , April 10–11, 2019 . Editors S. A. H. Ali, T. T. Subramaniam, and S. M. Yusof, 145–156.

Sugondo, A. F., and Bahana, R. (2019). “Chatbot as an Alternative Means to Access Online Information Systems,” in 3rd International Conference on Eco Engineering Development, ICEED 2019 , Surakarta, Indonesia , November 13–14, 2019 , IOP Conference Series: Earth and Environmental Science, (IOP Publishing) 426. doi:10.1088/1755-1315/426/1/012168

Suwannatee, S., and Suwanyangyuen, A. (2019). “Reading Chatbot” Mahidol University Library and Knowledge Center Smart Assistant,” in Proceedings for the 2019 International Conference on Library and Information Science (ICLIS) , Taipei, Taiwan , July 11–13, 2019 .

Vaidyam, A. N., Wisniewski, H., Halamka, J. D., Kashavan, M. S., and Torous, J. B. (2019). Chatbots and Conversational Agents in Mental Health: A Review of the Psychiatric Landscape. Can. J. Psychiatry 64 (7), 456–464. doi:10.1177/0706743719828977

Vijayakumar, B., Höhn, S., and Schommer, C. (2019). “Quizbot: Exploring Formative Feedback with Conversational Interfaces,” in 21st International Conference on Technology Enhanced Assessment, TEA 2018 , Amsterdam, Netherlands , Dec 10-11, 2018 . Editors S. Draaijer, B. D. Joosten-ten, and E. Ras, ( Springer ), 102–120. doi:10.1007/978-3-030-25264-9

Virtanen, M. A., Haavisto, E., Liikanen, E., and Kääriäinen, M. (2018). Ubiquitous Learning Environments in Higher Education: A Scoping Literature Review. Educ. Inf. Technol. 23 (2), 985–998. doi:10.1007/s10639-017-9646-6

Wildman, T. M., Magliaro, S. G., Niles, R. A., and Niles, J. A. (1992). Teacher Mentoring: An Analysis of Roles, Activities, and Conditions. J. Teach. Edu. 43 (3), 205–213. doi:10.1177/0022487192043003007

Wiley, D., and Edwards, E. K. (2002). Online Self-Organizing Social Systems: The Decentralized Future of Online Learning. Q. Rev. Distance Edu. 3 (1), 33–46.

Winkler, R., and Soellner, M. (2018). Unleashing the Potential of Chatbots in Education: A State-Of-The-Art Analysis. in Academy of Management Annual Meeting Proceedings 2018 2018 (1), 15903. doi:10.5465/AMBPP.2018.15903abstract

Winne, P. H., and Hadwin, A. F. (2008). “The Weave of Motivation and Self-Regulated Learning,” in Motivation and Self-Regulated Learning: Theory, Research, and Applications . Editors D. H. Schunk, and B. J. Zimmerman, (Mahwah, NJ: Lawrence Erlbaum Associates Publishers ), 297–314.

Wisniewski, B., Zierer, K., and Hattie, J. (2019). The Power of Feedback Revisited: A Meta-Analysis of Educational Feedback Research. Front. Psychol. 10, 1664–1078. doi:10.3389/fpsyg.2019.03087

Wolfbauer, I., Pammer-Schindler, V., and Rose, C. P. (2020). “Rebo Junior: Analysis of Dialogue Structure Quality for a Reflection Guidance Chatbot,” in Proceedings of the Impact Papers at EC-TEL 2020, co-located with the 15th European Conference on Technology-Enhanced Learning “Addressing global challenges and quality education” (EC-TEL 2020) , Virtual , Sept 14–18, 2020 . Editors T. Broos, and T. Farrell, 1–14.

Xiao, Z., Zhou, M. X., and Fu, W.-T. (2019). “Who should be my teammates: Using a conversational agent to understand individuals and help teaming,” in IUI’19: Proceedings of the 24th International Conference on Intelligent User Interfaces, Marina del Ray , California, USA , March 17–20, 2019 , (ACM), 437–447. doi:10.1145/3301275.3302264

Xu, A., Liu, Z., Guo, Y., Sinha, V., and Akkiraju, R. (2017). “A New Chatbot for Customer Service on Social media,” in Proceedings of the 2017 CHI conference on human factors in computing systems , Denver, Colorado, USA , May 6–11, 2017 , ACM, 3506–3510. doi:10.1145/3025453.3025496

Yin, J., Goh, T.-T., Yang, B., and Xiaobin, Y. (2020). Conversation Technology with Micro-learning: The Impact of Chatbot-Based Learning on Students' Learning Motivation and Performance. J. Educ. Comput. Res. 59, 154–177. doi:10.1177/0735633120952067

Appendix a aconcept map of chatbots in education

Keywords: chatbots, education, literature review, pedagogical roles, domains

Citation: Wollny S, Schneider J, Di Mitri D, Weidlich J, Rittberger M and Drachsler H (2021) Are We There Yet? - A Systematic Literature Review on Chatbots in Education. Front. Artif. Intell. 4:654924. doi: 10.3389/frai.2021.654924

Received: 17 January 2021; Accepted: 10 June 2021; Published: 15 July 2021.

Reviewed by:

Copyright © 2021 Wollny, Schneider, Di Mitri, Weidlich, Rittberger and Drachsler. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Sebastian Wollny, [email protected] ; Jan Schneider, [email protected]

This article is part of the Research Topic

Intelligent Conversational Agents

Book cover

  • Conference proceedings
  • © 2023

Chatbot Research and Design

6th International Workshop, CONVERSATIONS 2022, Amsterdam, The Netherlands, November 22–23, 2022, Revised Selected Papers

  • Asbjørn Følstad   ORCID: https://orcid.org/0000-0003-2763-0996 0 ,
  • Theo Araujo   ORCID: https://orcid.org/0000-0002-4633-9339 1 ,
  • Symeon Papadopoulos   ORCID: https://orcid.org/0000-0002-5441-7341 2 ,
  • Effie L.-C. Law   ORCID: https://orcid.org/0000-0002-0873-0150 3 ,
  • Ewa Luger   ORCID: https://orcid.org/0000-0001-7882-9415 4 ,
  • Morten Goodwin   ORCID: https://orcid.org/0000-0001-6331-702X 5 ,
  • Petter Bae Brandtzaeg   ORCID: https://orcid.org/0000-0002-9010-0800 6

SINTEF, Oslo, Norway

You can also search for this editor in PubMed   Google Scholar

University of Amsterdam, Amsterdam, The Netherlands

Certh-iti, thessaloniki, greece, durham university, durham, uk, university of edinburgh, edinburgh, uk, university of agder, grimstad, norway, university of oslo, oslo, norway.

Part of the book series: Lecture Notes in Computer Science (LNCS, volume 13815)

Conference series link(s): CONVERSATIONS: International Workshop on Chatbot Research and Design

10k Accesses

7 Citations

Conference proceedings info: CONVERSATIONS 2022.

  • Table of contents
  • Other volumes

About this book

Editors and affiliations, bibliographic information.

  • Publish with us

Buying options

  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Other ways to access

This is a preview of subscription content, log in via an institution to check for access.

Table of contents (12 papers)

Front matter, chatbot users and user experience, hi, i’m cecil(y) the smoking cessation chatbot: the effectiveness of motivational interviewing and confrontational counseling chatbots and the moderating role of the need for autonomy and self-efficacy.

  • Lotte Leeuwis, Linwei He

Interacting with the News Through Voice User Interfaces

  • Oda Elise Nordberg, Frode Guribye

Voice Your Opinion! Young Voters’ Usage and Perceptions of a Text-Based, Voice-Based and Text-Voice Combined Conversational Agent Voting Advice Application (CAVAA)

  • Christine Liebrecht, Naomi Kamoen, Celine Aerts

Value Creation in Gamified Chatbot Interactions and Its Impact on Brand Engagement

  • Susana C. Silva, Roberta De Cicco, Maria Levi, Maik Hammerschmidt

Chatbots as Part of Digital Government Service Provision – A User Perspective

  • Nadia Abbas, Asbjørn Følstad, Cato A. Bjørkli

Understanding the Intention to Use Mental Health Chatbots Among LGBTQIA+ Individuals: Testing and Extending the UTAUT

  • Tanja Henkel, Annemiek J. Linn, Margot J. van der Goot

Chatbot Design and Applications

Enhancing conversational troubleshooting with multi-modality: design and implementation.

  • Giulio Antonio Abbo, Pietro Crovari, Franca Garzotto

A Framework and Content Analysis of Social Cues in the Introductions of Customer Service Chatbots

  • Charlotte van Hooijdonk, Gabriëlla Martijn, Christine Liebrecht

An Affective Multi-modal Conversational Agent for Non Intrusive Data Collection from Patients with Brain Diseases

  • Chloe Chira, Evangelos Mathioudis, Christina Michailidou, Pantelis Agathangelou, Georgia Christodoulou, Ioannis Katakis et al.

Interactive Journaling with AI: Probing into Words and Language as Interaction Design Materials

  • Max Angenius, Maliheh Ghajargar

Increasing the Coverage of Clarification Responses for a Cooking Assistant

  • Gina E. M. Stolwijk, Florian A. Kunneman

Designing Context-Aware Chatbots for Product Configuration

  • Tom Niederer, Daniel Schloss, Noemi Christensen

Back Matter

Other volumes.

  • argumentation
  • artificial intelligence
  • autonomous agents
  • computer networks
  • computer programming
  • engineering
  • formal logic
  • Human-Computer Interaction (HCI)
  • information retrieval
  • linguistics
  • logic programming
  • multiagent system
  • natural languages
  • programming languages
  • signal processing
  • speech analysis

Asbjørn Følstad

Theo Araujo

Symeon Papadopoulos

Effie L.-C. Law

Morten Goodwin

Petter Bae Brandtzaeg

Book Title : Chatbot Research and Design

Book Subtitle : 6th International Workshop, CONVERSATIONS 2022, Amsterdam, The Netherlands, November 22–23, 2022, Revised Selected Papers

Editors : Asbjørn Følstad, Theo Araujo, Symeon Papadopoulos, Effie L.-C. Law, Ewa Luger, Morten Goodwin, Petter Bae Brandtzaeg

Series Title : Lecture Notes in Computer Science

DOI : https://doi.org/10.1007/978-3-031-25581-6

Publisher : Springer Cham

eBook Packages : Computer Science , Computer Science (R0)

Copyright Information : The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2023

Softcover ISBN : 978-3-031-25580-9 Published: 02 February 2023

eBook ISBN : 978-3-031-25581-6 Published: 01 February 2023

Series ISSN : 0302-9743

Series E-ISSN : 1611-3349

Edition Number : 1

Number of Pages : XII, 211

Number of Illustrations : 23 b/w illustrations, 21 illustrations in colour

Topics : Natural Language Processing (NLP) , Logic in AI , User Interfaces and Human Computer Interaction , Computer Communication Networks , Information Storage and Retrieval , Programming Techniques

Policies and ethics

  • Find a journal
  • Track your research

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • Springer Nature - PMC COVID-19 Collection

Logo of phenaturepg

Future directions for chatbot research: an interdisciplinary research agenda

Asbjørn følstad.

1 SINTEF, Oslo, Norway

Theo Araujo

2 University of Amsterdam, Amsterdam, The Netherlands

Effie Lai-Chong Law

3 Durham University, Durham, UK

Petter Bae Brandtzaeg

4 University of Oslo, Oslo, Norway

Symeon Papadopoulos

5 CERTH, Thessaloníki, Greece

6 University of Bamberg, Bamberg, Germany

Marcos Baez

7 Claude Bernard University Lyon 1, Villeurbanne, France

8 University of Glasgow, Glasgow, UK

Patrick McAllister

9 Ulster University, Jordanstown campus, Newtownabbey, UK

Carolin Ischen

Rebecca wald, fabio catania.

10 Politecnico di Milano, Milan, Italy

Raphael Meyer von Wolff

11 University of Goettingen, Göttingen, Germany

Sebastian Hobert

12 University of Edinburgh, Edinburgh, UK

Chatbots are increasingly becoming important gateways to digital services and information—taken up within domains such as customer service, health, education, and work support. However, there is only limited knowledge concerning the impact of chatbots at the individual, group, and societal level. Furthermore, a number of challenges remain to be resolved before the potential of chatbots can be fully realized. In response, chatbots have emerged as a substantial research area in recent years. To help advance knowledge in this emerging research area, we propose a research agenda in the form of future directions and challenges to be addressed by chatbot research. This proposal consolidates years of discussions at the CONVERSATIONS workshop series on chatbot research. Following a deliberative research analysis process among the workshop participants, we explore future directions within six topics of interest: (a) users and implications, (b) user experience and design, (c) frameworks and platforms, (d) chatbots for collaboration, (e) democratizing chatbots, and (f) ethics and privacy. For each of these topics, we provide a brief overview of the state of the art, discuss key research challenges, and suggest promising directions for future research. The six topics are detailed with a 5-year perspective in mind and are to be considered items of an interdisciplinary research agenda produced collaboratively by avid researchers in the field.

Introduction

Chatbots are conversational agents providing access to information and services through interaction in everyday language. While research on conversational agents has been pursued for decades within fields such as social robotics, embodied conversational agents, and dialogue systems, it is only recently that conversational agents have become practical reality [ 77 ]. Key drivers of this development include advances in artificial intelligence (AI) fields, such as natural language processing (NLP) and natural language understanding (NLU), as well as the increased consumer uptake of platforms conductive to conversational interaction [ 38 ].

Chatbots are currently taken up in application areas as diverse as customer service [ 1 ], health [ 105 ], education [ 53 ], and office work [ 78 ]. There has lately been a marked increase of interest in chatbot research within academia and industry, specifically from 2016 and onwards [ 86 ]. Recent research addresses, for example, chatbot use (e.g. 74 ], interaction design (e.g. 57 ] and assessment (e.g. 63 ], as well as specific applications (e.g. 96 ] and technological advances (e.g. 2 ].

The rapidly growing body of chatbot research has a marked interdisciplinary character—spanning fields such as informatics, management and marketing, media and communication science, linguistics and philosophy, psychology and sociology, engineering, design, and human-computer interaction. This broad emerging knowledge base is valuable, but also implies that research of relevance to chatbots is currently fragmented across disciplines and application domains. With a broad and rich range of chatbot applications, it is imperative to understand why certain chatbot usages are working (or not) by referencing in-depth theoretical frameworks. As the current interdisciplinary wave of chatbot research is progressing, there is a need to define overarching research directions for guidance, allowing new studies and initiatives to systematically build on and benefit from existing work.

In this paper, we propose a research agenda which has been distilled through a series of dedicated workshops on chatbot research—CONVERSATIONS—with intensive discussions among researchers and practitioners actively working on chatbots. The research agenda has the overall aim to motivate and guide research to establish requisite knowledge for fully realizing the potential of chatbots as a powerful means of accessing information and services and for understanding the impact of chatbots at the individual, group, and societal level. As the research on chatbots is rapidly evolving, we hold that deriving a research agenda from collaborations and discussions among avid researchers and practitioners, who keep abreast of the ongoing developments of the area, is a more effective approach as compared, for example, with a mapping study or systematic literature review. Furthermore, this collaborative approach enables us to gain insights from different perspectives to address opportunities, challenges, and perceived research needs within the field. The research agenda serves as a concise research roadmap, offering links to pertinent studies for those readers who are interested in delving further into specific fields.

In the following, we first present relevant background on chatbot research before we detail the need for a consolidation of future directions. We then present our approach and proposed set of directions. Finally, we discuss our proposal and the way forward.

Historical roots of chatbot research

The emerging chatbot research area has its historical roots in several research fields addressing different aspects of conversational computer systems—the most prominent of these with decades of research and efforts at industrial applications. Within the field of dialogue systems [ 77 ], researchers have since the sixties and seventies worked on text based [ 12 ] and later spoken [ 59 ] conversational user interface to support users with specific tasks. Other streams of research preceding and relevant to current chatbot research have addressed conversational interaction with physical social robots [ 15 ], and embodied virtual agents [ 18 ]. There has also been a long-term research initiative addressing computer systems for open-domain small talk [ 98 ], including the development of the artificial intelligence markup language [ 111 ] used to power chatbots for social chit-chat. Conversational computer systems have also had a long and, at times, winding path through various commercial applications—particularly automated solutions for customer service, sales and support [ 72 ], including interactive voice response (IVR) systems for phone-based self service [ 23 ].

The recent substantial increase in chatbot research can be seen as a direct response to the uptake of so-called virtual assistants by big tech companies, specifically the inclusion of Siri as part of Apple operating system in 2011, Amazon's promotion of Alexa since 2014 and the conversational turn of Facebook, Microsoft and Google in 2016 [ 25 ]. Piccolo et al. [ 86 ] concluded that chatbot research has followed in the trail of the industrial uptake of conversational computer systems rather than being at the driver seat. In consequence, the contribution of this burgeoning research area is as much to understand the emerging application, uses, and implications of conversational computing systems, as to improve on their technological underpinnings and methods for design and development. Consequently, the chatbot research area has a broader scope and disciplinary coverage than the fields at its historical roots.

Clarification of terminology

As noted by McTear [ 77 ], research streams such as those of dialogue systems, embodied conversational agents, and social robotics, are now converging in a common aim for developing and improving on conversational user interfaces to computer systems. However, there is still a wide variety of terms in use in reference to the object of this converging research interest. Since the recent industrial uptake of conversational computing systems, these have increasingly been referred to as chatbots within industry and media [ 25 ] and also in research. To demarcate the research area driven by the industrial uptake of conversational computer systems, and to signify the attention of this area towards emerging patterns of use, as well as broader business and societal implications, we refer to this area as chatbot research.

In line with this scoping of the research area, we understand chatbots as conversational agents providing access to information and services through interaction in everyday language —an understanding which is in line with the definitions by Følstad and Brandtzaeg [ 39 ] and Hobert and Meyer von Wolff [ 54 ]. This use of the term chatbot encompasses conversational agents for goal-oriented task completion, informational purposes, entertainment, and social chatter. It also encompasses agents supporting interactions through text, voice, or both. The use of the term is in reference to the object of our research interest—current and future design, development, and implications of information and services provided through conversational computer systems—rather than in reference to a specific set of technologies or approaches.

In consequence, our use of the term chatbot is broader than what may be found in other research streams. For example, some distinguish between voice-based and text-based conversational agents, using the term chatbot to refer to the latter, (e.g. Ashktorab et al.  6 ). Others distinguish between conversational agents for goal completion versus social chatter, referring only to the latter as chatbots (e.g. Jurafsky and Martin  61 ). However, in consequence of the rapid evolvement both in technology, services, and patterns of use, we find such attempts at principled scoping of the chatbot term challenging. For example, there is often no clear distinction between social chatter and goal-orientation in conversational agents—as seen by the importance of social responses for customer service chatbots [ 114 ]. Likewise, the distinction between text and voice is less than clear-cut as the same conversational agents may make use of different modalities [ 97 ].

Enablers of current chatbots

Current chatbots are enabled by a large range of technologies and services [ 97 ] at varied levels of sophistication. Dialogue management may be enabled through simple rule-based approaches, statistical data-driven systems, or neural generative end-to-end approaches [ 77 ], and many systems employ hybrid models [ 50 ]. Whereas early chatbots for social chit-chat may exemplify rule-based approaches (e.g., Weizenbaum 112 ) current statistical data-driven systems—such as chatbots for customer service—have user intents and corresponding chatbot responses identified on the basis of training of machine learning models based on example user data [ 66 ]. Generative chatbots based on end-to-end approaches are currently a research topic of substantial interest. A much-cited example is presented by Vinyals and Le [ 109 ]. More recently, Facebook's Blender [ 90 ] and Google’s Meena [ 2 ] have received substantial interest due to their near-human open domain conversational capabilities.

A large number of general-purpose platforms and frameworks are available for chatbot delivery, such as Google's DialogFlow, 1  Microsoft Bot Framework, 2  Pandorabots, 3  and the open-source frameworks Rasa 4 and Mycroft. 5  The platforms range from so-called low-code alternatives [ 26 ], where implementation and maintenance may be conducted with limited or no software engineering skills, to frameworks serving as basis for larger software development projects. Platforms and frameworks for chatbot delivery typically provide integrations with a range of communication channels, including social media and chat, as well as websites and collaborative work support systems. Hence, the same chatbot may reach users across their preferred channels.

Research communities

Chatbot research is currently evolving within and across a range of disciplines and has a strong interdisciplinary character. Ground-breaking research has been presented in fields as diverse as communication (e.g. Go and Sundar 42 ), health (e.g. Fitzpatrick et al.  35 ), informatics (e.g. Adiwardana et al.  2 ), and business (e.g. Adam et al.  1 ). While dedicated workshops and conferences of relevance to chatbot research are emerging—such as CUI, 6  CONVERSATIONS, 7  and CAIR 8 —in addition to established venues—such as SIGDIAL, 9  IVA, 10  IWSDS, 11  and INTERSPEECH 12 —research findings are typically presented in a broad range of journals and conferences. Research related to chatbots is also conducted in multiple communities with varying degrees of exchange among them. These communities may not label their area of interest as chatbot research but, for example, research addressing conversational agents [ 79 ], dialogue systems [ 59 ] or social robotics [ 93 ]. The research objectives within these communities may only be partially overlapping. However, we believe these communities likely will benefit from strengthening their collaboration and mutually inform and support each other's research.

Objective: to propose future research directions

While there is a rapidly expanding body of knowledge relevant to chatbot research, rooted in long-standing research fields, current research and knowledge are fragmented across disciplines, application areas, and communities. Such fragmentation is to be expected in a rapidly expanding field. However, we are now at a point in time where it is beneficial to stake out common directions for future research.

The identification of common research directions is not something that can be achieved by individual researchers or single communities. Rather, it should be seen as a collaborative and continuously evolving process across individuals and communities, where adjustments are made on the basis of new insights and knowledge as it is gathered.

Our objective in presenting this work is therefore to provide a needed interdisciplinary and collaborative basis to initiate and guide a broader discussion on the key future research directions for chatbot research. As such, the work will provide a broader perspective on research directions than what is provided, for example, in current reviews on chatbots within specific domains (e.g. [ 105 , 78 ]), specific aspects of chatbot technology and design (e.g. [ 20 , 86 ]), or user behaviour and experience (e.g. [ 63 , 119 ]).

Furthermore, we address perspectives and topics for chatbot research which may be more broadly scoped than what may be found within, for example, the fields and disciplines in which chatbot research has its roots. As such, we aim for the work to provide a basis for chatbot research that is seem of value to research and practice alike, and which also may serve to bridge relevant research currently embedded in distinct disciplines.

The proposed future research directions are based on the collaborative work conducted as part of the CONVERSATIONS workshops. CONVERSATIONS is an international workshop series for chatbot research, where researchers, students, and practitioners with interest in chatbots gather to present their work, discuss, and collaborate. The first workshop in this series was organised in 2017 and it has since been a yearly event, advancing from being arranged in conjunction with a research conference the two first years to now being a 2-day stand-alone event. The most recent workshop in 2020 [ 41 ], arranged as a virtual event due to the COVID-19 pandemic, involved about 150 registered participants from more than 30 countries and 80 different organizations, including more than 20 paper presentations. The participants represent disciplines such as computer science, information systems, human–computer interaction, communication studies, linguistics, psychology, marketing, and design.

Throughout the CONVERSATIONS workshops, we have discussed chatbot research challenges and how to address these. In the first CONVERSATIONS workshop (2017), approximately half of the overall 30 participants engaged in identifying and clustering key research challenges of the field into overarching research topics. The research challenges within these topics formed the basis for the call for papers to the later CONVERSATIONS workshops (2018, 2019, 2020). At the third CONVERSATIONS workshop (2019), the topics—updated throughout the workshop series—were revisited through in-depth group discussions involving approximately half of the overall 50 workshop participants. The output from these group discussions forms the basis for the presented research directions.

The deliberative process at the workshop series was key to identify and propose research directions in a true interdisciplinary fashion. In the 2019 edition, workshop participants were assigned to groups—each with the mandate to address one of six topics: (a) user and communication studies, (b) user experience and design, (c) frameworks and platforms, (d) chatbots for collaboration, (e) democratizing chatbots, and (f) ethics and privacy. The group work was conducted in two sessions across the 2 days of the workshop. In the first session, each group carefully discussed the research topic in a 5-year time frame, identifying (a) relevant state of the art, (b) key research challenges, and (c) future directions. In the second session, the output of each group was presented to the workshop plenary and discussed.

The collaborative process extended across the following year, taking into account the contributions and discussions of the CONVERSATIONS 2020 workshop as well. As a result, the proposed research directions reflect the interdisciplinary position of a group of collaborating researchers within this emerging field.

Proposed future research directions

Through the CONVERSATIONS workshop series, six overarching topics for future chatbot research have been identified. In the following, we detail each of these based on the CONVERSATIONS output, with particular concern for the state of the art, research challenges, and future research directions. An overview of the six topics and associated future research directions is provided in Table  1 .

Topics and proposed directions for future research on chatbots

Users and implications

Given the current evolving use and emerging use cases for chatbots, important questions to ask concern chatbot users and their contexts of use. This includes investigating antecedents for chatbot use—namely individual characteristics, motivations and boundary conditions for choosing, accepting or even preferring to interact with conversational agents. Furthermore, it is necessary to explore and discuss implications of chatbot use on individuals, groups, organizations and society at large.

State of the art

Chatbot use is becoming commonplace. For example, in 2019, over 50 % of US and German consumers were estimated to have used chatbots at least once—with even higher numbers in the UK or France [ 88 ]. In consequence, chatbot researchers currently have an unprecedented opportunity for real-world study of users [ 106 ], user motivations [ 14 ], and implications at scale. In consequence, knowledge on chatbot use has been gathered for a range of contexts—in the private sphere [ 87 ], at work [ 74 ], and in public spaces [ 17 ].

A substantial body of research of relevance for chatbot use has been developed within broad domains such as health [ 105 ], education [ 84 ], and business [ 8 ], as well as more specific application domains such as polling [ 62 ], information search [ 73 ], libraries [ 92 ], and museums [ 64 ]. Knowledge of relevance for understanding the impact of chatbots on individual users may be found in studies of therapy chatbots (e.g. [ 35 ]), relational agents (e.g. [ 10 ] and chatbots for social relationships [ 103 ]. Specifically, it is of interest to note how such studies address implications of individual long-term use.

Because of this, we have substantial knowledge on potential and actual chatbot users and implications for individuals across a wide variety of contexts, building upon a rich stream of research dating back to the work of Weizenbaum [ 112 ]. Chatbot impact on society has, however, not been comprehensively researched and only tentatively been suggested in studies of chatbots for specific domains—as mentioned above. This may in part be due to the substantial impact on the level of organizations and society is assumed to be seen in the future more so than the present.

Research challenges

While we have substantial knowledge on current chatbot users, important topics lack sufficient coverage. Two warrant particular mention: (a) broader chatbot uses and user groups and (b) implications of chatbot use, both detailed below.

For the broader chatbot uses and user groups, the rich literature needs to be continuously updated, especially when it comes to user motivations and behaviour of emerging user groups. This includes knowledge on specific demographics, for example, vulnerable users, such as children, elderly and users with special needs, as well as user groups within particular application areas. Moreover, research still needs to assess whether there are systematic differences in the adoption and usage of chatbots driven by socio-demographic characteristics.

Implications of chatbot use entail a range of exciting research challenges, as knowledge is needed on how the uptake of chatbots may impact groups, organizations, businesses, and society at large. For example, as chatbots are taken up by different sectors and industries, chatbots may transform service provision and work processes.

Another example is our need for knowledge on how the interaction patterns that emerge between human users and chatbots may spill over to our interaction with other people: Will the demanding communication style we learn to use for virtual assistants, such as Alexa and Siri, impact our communication style with our partners or collaborators? How will the companionship offered by social chatbots influence users' social lives and desires, and how chatbots may enter the social fabric of groups or organizations?

Future research directions

Based on the current state of the art and identified research challenges, two future research directions emerge as particularly promising in the area of chatbot user and communication studies.

  • Emerging chatbot user groups and behaviours. While there exists knowledge on current chatbot user groups, this needs to be updated as technology, services, and patterns of use evolve. Furthermore, there is a need to move from studies of chatbot users in general to studies of chatbot users and behaviours for particular demographics, domains, or contexts. We are beginning to see this for domains such as health, education, and business, but given the uptake of chatbots in new contexts and domains, this is an area of research which will be in continuous need of update.
  • Social implications of chatbots. The study of social implications of chatbots is an area where we expect to see substantial research interest in the near future. Knowledge of the social implications of chatbot use will be of importance to guide also future development and design of chatbot services. Possibly, a string of research on the broader social implications could be motivated from the broader discourse on implications of AI for labour and business (e.g. [ 37 , 76 ]). It will be beneficial to accommodate for research on unintended social consequences of chatbots or how chatbots are shaped in response to its uptake in society.

Chatbot user experience and design

Chatbot user experience and design concerns how users perceive and respond to chatbots, and how chatbot layout, interaction mechanisms and conversational content may be designed so as to manage these perceptions and responses. To gather insight into users' perceptions and responses, and how these are impacted by chatbot design, user-centred evaluations of chatbots is necessary; that is, assessments of users' perceptions and responses to chatbots conducted through established methods.

Chatbot user experience has been a key theme in recent research efforts, for voice-based virtual agents [ 75 ] and text-based applications [ 4 ]. This has helped identify factors contributing to positive or negative user experience [ 118 ] and has addressed specific aspects such as trust [ 119 ], perceived social support [ 71 ], human likeness [ 4 ], and how these aspects are impacted by chatbot design [ 42 ]. There is also a growing base of research to inform design of chatbot interactions, whether this concerns conversational design [ 6 ], personalization of chatbots [ 69 ], the use of interactive elements in chatbots [ 57 ], or the use of social cues to indicate social status and capabilities [ 32 ]. Recently, a number of textbooks (e.g. [ 48 , 79 , 97 ]) and industry guidelines (e.g. by Google 13 and Amazon 14 ) have also been published on chatbot interaction design and conversational design. Textual and acoustic properties of users' dialogue input are gradually being applied as outcomes in empirical research for studying engagement and experience with conversational agents [ 52 , 67 ]. Furthermore, there exists an extensive body of research on emotion detection through speech (e.g. [ 95 ]) and non-verbal behaviour [ 27 ] of high relevance to chatbot user experience and design.

There is also a grown body of knowledge on methods and measures for evaluating chatbot user experience. User-centred evaluation has been key to research within several of the disciplines at the roots of current chatbot research, such as studies of social presence in social robotics [ 82 ] and the use of user satisfaction measures in evaluations of dialogue systems [ 28 ]. Evaluation in chatbot research is conducted by instruments for users' self reports of user experience [ 63 ], through user observation and interviews [ 75 ] and analyses of chatbot interaction [ 66 ], and also by physiological measurements [ 22 ]. A range of evaluation approaches are employed, including experiments by self-administered online studies [ 5 ] or in the lab [ 22 ], observational studies in the wild [ 64 ], and investigations of long-term interactions with established services [ 73 ].

While there is a growing body of research available on chatbot user experience there still is a lack of knowledge on how to leverage the findings from this research in chatbot designs that consistently delight and engage users. Users still experience issues in chatbot interaction, both in terms of pragmatic experiences—where chatbots fail to understand or to help users achieve their intended goals [ 75 ]—and in terms of hedonic experiences—where chatbots fail to engage users over time [ 117 ]. These issues may in part be seen as due to the more general challenge of designing human-AI interaction [ 116 ]. There are indeed indications that these challenges are being mitigated, for example in the case of improvements in customer service chatbots [ 80 ] and in the uptake of social chatbots such as Replika [ 103 ]. However, the strengthening of chatbot user experiences remains a key research challenge.

Related to the challenge of strengthening chatbot user experience, is the challenge of measuring and assessing chatbots in terms of user experience and from a more holistic perspective to determine whether chatbots are actually beneficial. Relevant aspects for this are, for instance, usefulness, efficiency and process support. While there is a large number of studies on chatbot user experience available, there is a lack of common definitions, metrics and validated scales for key aspects of chatbot evaluations [ 63 ]. Furthermore, while a broad range of approaches are employed there is a lack of commonly applied approaches to evaluation.

Future research should be directed at addressing the identified key research challenges. Specifically, the following two directions are proposed.

  • Design for improving chatbot user experience. Future research on chatbot user experience needs to evolve from exploring and assessing aspects of user experience and effects of chatbot design elements, towards studying how this knowledge may impact and improve chatbot user experience in industrial applications. Specifically, to translate findings of theoretical interest to conclusions of practical impact on design. This is not to say that research to build theory on chatbot user experience is not needed, but this research may need to take up also more design-oriented objectives—so as to condense current research and knowledge to guidelines that may directly inform conversational design or interaction design.
  • Modelling and evaluating chatbot user experience. To advance future research on chatbot user experience, there is also a need for convergence of chatbot user experience models, measurements, and approaches to evaluation. While diversity in definitions and operationalizations is to be expected in an emerging field of research interest, there may now be the time for seeking agreement and consistency in the use of terminology and definitions of user experience constructs, and also to identify and apply standardized measurements (benchmarks) for these constructs. While such convergence should not be done in a way that hampers theoretical advancement and method innovation, there is clearly a benefit in including common measurements across studies so as to enable cross-study comparison and aggregation, and to be able to track progress over time. For this purpose, established evaluation approaches from fields such as human-computer interaction or the tradition of dialogue systems may be beneficial.

Chatbot frameworks and platforms

This area of chatbot research concerns the current and future frameworks and platforms for chatbot development and delivery. That is, the technological underpinnings of chatbot implementations such as solutions for natural language processing, data extraction, storage, and access, as well as mechanisms to identify and adapt chatbot interactions to context and user profile.

The advances in chatbot frameworks and platforms are key enablers of the current interest in chatbot applications. As noted in the background Sect. 6 myriad platforms and frameworks are available to support design and development of chatbots. Key advances include the application of supervised machine learning for classification and information retrieval—enabling, for example, intent prediction and identification of user sentiment [ 20 ], which are critical to support task-oriented conversations. Furthermore, the use of generative approaches has seen substantial progress, where end-to-end dialogue systems are applied to predict suitable responses to user input based on models built from large conversational datasets [ 2 , 109 ]. Finally, the introduction of the Transformer [ 107 ] as a dominant and highly effective architecture for natural language processing along with high-quality open-source libraries [ 113 ] have lowered the barrier to entry and make it possible to build conversational models that exhibit high generalization and coherence [ 90 ].

In this regard, large-scale generative models are becoming increasingly impactful, enabling a wide range of tasks that can benefit chatbot development [ 36 ]. Models such as GPT-3 [ 16 ] by OpenAI and BERT (Bidirectional Encoder Representations from Transformers) [ 29 ] by Google leverage massive amounts of data and computational power that would not be available to smaller players. Indeed, GTP-3 currently uses 175 billion parameters, and it is estimated to have cost 12 million US dollars to train [ 36 ]. Thus, opening up these powerful models to the public has the potential to accelerate chatbot development even further. It is important to note, however, that criticism around large models has been growing lately [ 9 ], especially ethical concerns regarding undesirable and often inscrutable societal biases percolating the models [ 9 , 120 ], carbon footprint [ 9 ,  99 ], misuse and misinterpretation [ 9 ], privatization of AI research [ 99 ], and even research opportunity costs [ 49 ].

While substantial advances have been made in chatbot frameworks and platforms, a number of challenges remain. Specifically, we lack the needed technological underpinnings to support some key aspects of chatbot applications. We see four such challenges of particular importance. First, understanding user input remains difficult. While machine learning approaches have strengthened both natural language understanding and intent prediction, chatbot interaction is prone to conversational breakdowns due to interpretation issues—in particular in everyday situations or in the wild [ 87 ]. Second, the challenge of modelling and adapting to the user and conversational context is as important as ever. For example, as chatbots are being increasingly deployed in the health domain, in possibly sensitive scenarios, it becomes of paramount importance for chatbots to adapt the conversation to social, emotional and even the health literacy aspects of users [ 60 ]. These were identified as key challenges already by Weizenbaum [ 112 ] and have remained such ever since. Third, challenges remain in solutions for supporting chatbot development and standardised testing for example in terms of studies simulating production environments and approaches to improve chatbots more easily in production. Last, as chatbots are becoming part of an ecosystem of software systems, supporting chatbot integration in this context is a new emerging challenge—for example by facilitating conversational presentation of information and content also intended for other use [ 7 ].

  • Interpretation capabilities and context understanding. As in recent years, further progress in the field of chatbots will depend on advances in natural language understanding, which will remain a key area of research interest. To enable progress in natural language understanding, more quality training data in open repositories is needed. Also, new techniques supporting the involvement of domain experts in content development, natural language processing, and dialogue management—through low-code or end-user development approaches—may be relevant. Finally, the challenges of context and user understanding, for sustained dialogue and adaptation of conversations, will remain critical aspects of future research.
  • Emerging techniques for chatbot design, development, and deployment. Future research is needed to provide increases support for design, development and deployment. The deployment of conversational interfaces on top of software-enabled services is a promising direction for chatbot research and implementation (e.g. [ 115 ])—enabling digital assistants' access to information and services currently out of their reach, and rendering existing systems more accessible. In terms of design, it is promising to see that general guidelines for human-AI interaction are emerging [ 3 ] and more of these are needed. There is also a need for guidelines drawn from systematic comparative studies and to embed research-derived guidelines into chatbot frameworks.

Chatbots for collaboration

The area of chatbots for collaboration concerns how we may understand and design chatbots in the context of networks that comprise humans and intelligent agents, for example for social networking, teamwork, or service provision. While the current research on chatbots typically addresses dyadic interactions between one chatbot and one user, we foresee that chatbots in collaborative relations involving more people and bots will become more prominent as chatbots mature further. In addition, we consider that collaborative relations can be addressed to a chatbot's relations with external online services in the form of application programming interfaces (APIs) and other artificial agents.

Chatbots for collaboration concerns chatbots involved in interactions with humans and possibly with other chatbots in networks larger than dyads. While not as prominent as chatbots for simpler dyadic interaction, chatbots for collaboration have been developed and implemented in a range of contexts and for various purposes, for example, to support group processes in education [ 43 ], at work [ 11 ] and organizational settings [ 104 ], as well as in gaming communities [ 96 ].

Types of collaboration with chatbots may include (a) one human collaborating with one chatbot as an extension of human abilities, for example for analysis, gaming, as part of a service-related inquiry, or as learning partner (e.g. [ 53 ]), (b) chatbots supporting human collaboration, for example by taking notes, documenting, or task management (e.g. [ 104 ]), and (c) chatbots collaborating with other services for example in multi-agent models, networks of chatbots, or external web services (e.g. [ 108 ]).

Chatbots may be integrated into collaborative processes forming what Grudin and Jacques [ 45 ] refer to as humbots , that is, human-chatbot teams which handle challenging service queries better than chatbots alone and more efficiently than humans alone. The concept of humbots assumes a tiered approach to service provision where the chatbots constitute an initial service contact point, and customers are escalated to human helpers only if the chatbot is unable to help. Such human–chatbot teams draw on the concept of human-in-the-loop [ 24 ] from the human factors literature, sensitizing system managers to the need for a collaborative setup allowing sufficient situation awareness to the human part of the team to provide quality takeover if need be. In health-care context, human-in-the-loop concepts for conversational agents supporting hospital nurse teams has proved beneficial [ 13 ]. Likewise, the notion of escalation in customer service chatbots is a practical application of the human-in-the-loop concept for robust application of chatbots in consumer service provision [ 83 ].

There is an essential challenge in studying and designing chatbots for collaboration due to the multifaceted character of such interaction, and the range of potential theoretical perspectives to apply. For example, collaboration may be framed line with game theory—where an agent can be either a collaborator or an opponent [ 56 ]—or follow joint-intention theory where an agent is always aimed to work together with the user [ 55 , 68 ] or to establish a partnership [ 31 ]. When setting the concept of collaboration within social settings, the agent may be considered a mediator of human actors rather than an established actor within the described social structure (e.g. [ 104 ]). Or collaboration is addressed as merely a technical feature when the agent is collaborating with other artificial agents and external web services (e.g., [ 108 ]).

While a range of chatbots for collaboration have been developed, there is relatively scarce research on the characteristics of collaboration with chatbots. That is, we lack models or theories helping us to conceptualize collaboration involving intelligent conversational agents. While this problem of human–machine collaboration is addressed in more generic terms, for example in actor-network theory [ 70 ], there is a lack of models to characterize conversational collaboration involving agents. Related to the challenge of conceptualizing chatbot collaboration, there is a need for research on the different roles chatbots and humans should take in the human–chatbot collaboration, and what the implications of these may be. Should for example, the relation be based on assistance or mutual collaboration? Should chatbot participation be reactive or active? Should the chatbot be submissive or take charge? And what would the implications of these choices be?

Drawing on the above state of the art and research challenges, the following research directions are found to be particularly promising.

  • Modelling human–chatbot collaboration.  Research is needed to conceptualize and model different forms of human–chatbot collaboration, the roles the collaborative partners may take, and the potential implications these forms and roles may have in the short and long term. Addressing this complex concept within interactions with a novel technology like chatbots may benefit from inductive approaches. Future research may build theory inspired by knowledge on collaboration with humans and other artificial agents than chatbots—such as social robots and embodied virtual agents. Accordingly, the concept of collaboration could be conceptualized in line with chatbots' unique embodiment features, paying particular attention to the possible roles of chatbots in collaboration and identifying properties which express these.
  • Empirical investigations of human–chatbot collaboration. When robust concepts for human–chatbot collaboration are established, a range of exciting empirical research is foreseen—for example involving experimental studies and case studies. As part of such research, it will be valuable to investigate incentive structures in collaboration, instruments for measuring human–chatbot collaboration, task-specific differences in outcomes, and levels of participant engagement and activity across and within tasks. These may also be included as mediators, moderators and covariates in complex behavioural models studying other concepts as outcomes (e.g., customer satisfaction, user experience, or technology adaptation). Thus, collaboration with chatbots could be situated not solely as an outcome or a predictor, but also as an adaptive behaviour that has a substantial role in a variety of settings and applications.

Democratizing chatbots–chatbots for all

The topic of democratizing chatbots concerns how chatbots may be developed, designed, and deployed to improve availability and accessibility to information and services. Furthermore, how chatbots may be beneficial in bridging digital divides across various user populations. By nature, democratizing chatbots is a topic of interest to the human–computer Interaction community, but not limited to it. Any discussion around democratizing chatbots has at least some overlap with larger debates concerning the ethics of artificial intelligence—in particular for issues pertaining to fairness, non-discrimination, and justice [ 47 ].

By allowing simple natural language dialogues, chatbots are potentially a low-threshold means to access information and services and may as such serve to bridge digital divides and strengthen inclusion [ 14 ]. Chatbots have been suggested as accessible interactive systems for visually impaired in need of an easily navigable user interface [ 7 ], as conversational support for users with special needs [ 19 ], and to support youth to engage in societal issues [ 110 ]. Chatbots may improve access to health care services (e.g. [ 105 ]) and support health-promoting behavior change (e.g. [ 85 ]), and supplement educational programs [ 53 ].

Also relevant for the democratization of chatbots is also the relative lowering of thresholds that chatbots may introduce to interactive systems development and design. A number of current chatbot platforms are marketed under the promise of supporting chatbot design without need for coding skills [ 26 ]. Likewise, to involve domain experts in dialogue design, platforms may include dashboards for low-code updates of chatbot content and interaction design [ 66 ] or take up low-code approaches [ 89 ]. However, to our knowledge, there is a lack of research on the usability, accessibility, and effectiveness of such platforms.

However, some studies highlighted critical aspects in using chatbots since they may sustain and even strengthen existing biases in society. For example, a gender bias has been identified in chatbot design [ 33 ], and voice-based conversational agents have been shown to more easily interpret particular English dialects potentially reducing their utility for users of specific areas [ 51 ], and also to be difficult to use for user groups with speech impairments [ 19 ]. Although many major companies, research institutions, and public sector organizations have issued ethical artificial intelligence guidelines, recent work [ 58 ] has discovered substantial divergence in how these are written and interpreted, highlighting the complexity of designing guidelines for systems with complex social impact. In this way, responsibility is placed on designers and developers to cultivate awareness of these issues and how their approaches impact the end-user instead of discussing shared ethical approaches and focusing on agent decision-making.

Recent studies suggest that while chatbots may indeed serve as a low-threshold interface to information, services, and societal participation, they may also face challenges regarding bias and inclusion. Besides, there is a lack of more systematic or structured investigations of universal and inclusive design of chatbots. Inclusive and responsible design of chatbots requires an understanding of various linguistic elements of conversation and an awareness of broader social and contextual factors. For example, studies are needed on barriers to onboarding and barriers to the use of chatbots. The aim of using chatbots for strengthening democratization, reducing bias, and facilitating universal design has been included in the vision of chatbots for social good [ 40 ], which may be a useful scope for addressing this set of challenges.

Furthermore, while available platforms and frameworks are promoted as low-threshold means of chatbot design and development, there is a lack of knowledge regarding how these are actually employed to democratize chatbot development and design. Also, knowledge is needed on what challenges users with limited technology skills meet when trying to use these platforms and frameworks, and how such challenges may be overcome through changes, for example, in design and training of machine learning models.

In light of the background and research challenges mentioned above, the following broad directions of future research are identified.

  • Chatbots for social good.  To realize the potential of chatbots as vehicles for bridging digital divides and strengthening accessibility, availability, and affordability of services and information, chatbots for social good may be leveraged as an alternative perspective on chatbot research and design. In this perspective, systematic studies are needed to gain insight into current barriers in chatbot use how these could be employed for social good. In this way, it will be possible to seek to overcome the existing barriers with standardized solutions and follow user-centered design processes focusing on user needs. Finally, research is needed on the normative and ethical implications of the adoption of chatbots in particular contexts, as also outlined in the next section.
  • Inclusive design with and for diverse user groups. Parallel to the research direction of chatbots for social good, we foresee research and development continuing the work towards making the underlying platforms and frameworks for chatbot design and development more easily applicable also for users without strong technical skills. Here, we foresee studies of current opportunities and challenges faced by chatbot creators, followed by development and design stages, aiming to follow up or mitigate these. Removing the need for complex configuration and simplifying or eliminating coding is probably the easiest way to serve the needs of the small business and research groups—but also the needs of large enterprises that may have domain experts creating chatbots. Furthermore, developing platforms that facilitate the implementation of chatbots and recommend using best practices during the design process will surely raise the quality level of the final products.

Ethics and privacy in chatbots

The final research topic concerns ethical and privacy implications of chatbots. Specifically, how to reflect ethical and privacy concerns in the design of chatbots, recognising the implications that different chatbot use cases and design choices may have for users’ trust in chatbots, and how we may identify and address unethical chatbot use.

AI has recently been the objective of substantial interest from policy-making and regulatory bodies, as well as in discussions and reflections on ethics, privacy management and trust [ 21 ]. This concern for ethics in AI is motivated by its disruptive character and potential for changes in the job market and misuse by malevolent actors, as well as issues pertaining to accountability and bias [ 58 ]. Ethical concerns arising from the design and deployment of AI technology have motivated a number of initiatives [ 47 ]—such as the Ethical Guidelines for Trustworthy AI by the European Commission expert group on AI, and Microsoft’s FATE: Fairness, Accountability, Transparency, and Ethics in AI—addressing issues including mitigating bias and discrimination in AI systems and fairness in the use of AI systems [ 81 ]. Chatbots are a prominent AI-based technology, and as such in principle addressed by the broader concern for ethics and privacy in technology research in general and in AI-based technology in particular. Nevertheless, as noted in a review of the chatbot literature, there has been an initial lack of ethical discussion in chatbot research [ 102 ]—though noteworthy exceptions to this exist, such as the exploration of ethical and social considerations for conversational AI by Ruane et al. [ 91 ]. The ethical discussion in chatbot research may, however, be gaining traction motivated, for example, by Bender et al.’s [ 9 ] critical overview of ethical risks pertaining to large language models.

The interest and discussion concerning ethics and privacy in AI have been particularly impactful in Europe, where the General Data Protection Regulation (GDPR) is now used to govern privacy in technology-based systems and services. Furthermore, based on the advice of a high-level expert group on AI, a European set of ethics guidelines for trustworthy AI has been presented [ 30 ]. According to these guidelines, it is of paramount importance for trustworthy AI to be aligned with (a) legal regulations and (b) ethical principles and values, and also (c) be robust from a technical perspective given its particular social context. From these principles, the European Commission expert group has identified seven key requirements for ethical AI applications, including human agency and oversight, privacy and data governance, and diversity, non-discrimination, and fairness. Finally, a proposed European set of regulations for AI, the AI Act, will help strengthen aspects of ethical concern in AI systems, including legal requirements for human oversight, accuracy, robustness, and security. Of particular relevance for chatbots is the proposed requirement for transparency which will make it an obligation for service providers to ensure users are aware when are interacting with machine agents and not human operators [ 94 ].

Ethical and privacy challenges permeate the field of chatbot research, but specifically where the context is sensitive or high-stakes or the users are marginalised or vulnerable; for example, in designing chatbots for health and education, or in designing chatbots to support asylum seekers or children. There is a large and growing body of ethical and privacy knowledge to draw on, and an emerging set of guidelines and regulations on ethics and privacy for digital systems in general, and AI-based systems in particular. Nevertheless, we lack research and theorising around ethics and privacy specifically for conversational user interfaces. This is problematic, as the conversational character of chatbots may conceivably introduce a range of specific ethical problems, for example the ethical implications of human-like and socially present chatbot interaction, issues of consent, the privacy implications of third-party interactions and the implications of emotional effect on children and vulnerable users. Research is needed to better understand and address these, and other, emergent problems of ethics and privacy.

Drawing on the above, we accentuate the following two directions for future research—though other directions could be possible and maybe equally relevant.

  • Understanding chatbot ethics and privacy. Future research should facilitate reflections on ethical implication of chatbots, for example through identification of ethical and privacy issues in chatbot design and implementation—including design intentions, practical mitigation of known issues and exploration of unforeseen implications. These could be domain-specific issues, such as ethical implications for research and education, media or marketing and commerce, but these could also be general issues such as how interaction with chatbots may motivate oversharing in users, helping spread misinformation and hate speech, or induce potential negative consequences as a result of over-humanizing chatbots.
  • Ethics by design.  In parallel with work on chatbot ethics, there will be a need for research on the pragmatic and material issues of how to honour ethical guidelines and principles in the design of chatbot technology and applications. With reference to the principle of privacy by design , we refer to this as ethics by design —where privacy is subsumed as one of several aspects to consider as part of an ethics discussion and subsequent design challenge. Important challenges may include research on how to avoid biases in chatbots, how to avoid chatbot discrimination and redlining, and how to mitigate the ethical issues introduced by the black-box approach to machine learning underpinning aspects of chatbot functioning, as well as to avoid misuse and weaponization of chatbot technology. A useful starting point for an exploration of ethics by design, could be to refine the generic European expert group requirements for ethical AI [ 30 ] to the context of conversational AI.

Drawing on the involvement of chatbot researchers and practitioners in the CONVERSATIONS workshops, we propose a set of future directions for chatbot research. The directions are motivated by the current state of the art and identified research challenges and structured within six overarching topics. In the following, we discuss the implementation of the future directions, our perspectives on chatbot application areas, and how to continue the discussion and reflection started in this paper.

Implementing the future directions

Two of the identified research directions concern studies of users and implications, as well as how to design for desirable chatbot use. As chatbots become more pervasive in the coming years, and communication with non-human agents increasingly become part of our daily routines, it becomes even more pressing to expand our knowledge on the antecedents, contents and consequences of human–machine communication. In doing so, this stream of research needs to explore the cognitive, affective and behavioural dimensions of engagement with these agents, the extent to which there are systematic differences between individuals, groups or contexts of use, and the individual, group and societal implications of this phenomenon. Moreover, as the field progresses, there is a growing need to consolidate the existing knowledge, updating and extending overarching theoretical frameworks and models. Work within a wide variety of disciplines can serve as an inspiration in that regard, such as the studies of Sundar [ 100 ] on the psychology of human–agent interaction and Guzman and Lewis [ 46 ] on human–machine communication.

This evolution in our understanding of conversational user experiences should be accompanied with the proper support from platforms and frameworks. We can see the support as increasingly creating abstractions that would facilitate the design, testing, integration and development of chatbots, as it has historically happened with other software artefacts. Current efforts are already moving in that direction, providing development resources that promise anyone with enough motivation, regardless of their background, to deliver human-like interactive experiences. While this has the potential to bring substantial value to societies, empowering communities to develop their own solutions, it can also bring unintended consequences, as we cannot expect users of these platforms to have knowledge about complexities of modelling proper Human-AI experiences [ 116 ]. On the other hand, abstractions can also hide underlying information about machine learning models, AI decision-making, as well as latent bias in the training data (e.g., [ 101 ]) that can translate into social biases (e.g., [ 120 ]).

Human–chatbot collaboration is foreseen as an increasingly important aspect of chatbot research and applications. We hold that such collaboration will benefit from being implemented while reflecting on human collaboration, and in line with relevant empirical evidence of chatbot research—in line with reflections by Grudin and Jacques [ 45 ]. Considering the meaningful value of collaboration for decision making and productivity in professional and organizational settings, tasks assigned to chatbots in these collaborative interactions can vary in complexity and involvement. Such tasks can be as simple as providing individual notifications, or as complicated as communicating processed and analysed data to different stakeholders. Using chatbots for automating these tasks should enrich group productivity and quality of work, promoting mutual understanding and diversity of opinions. Research supporting such automation could benefit from seeing this as a service design challenge, where the chatbot is seen as one of multiple agents and user interfaces [ 14 ]. On a societal level, collaborative networks of humans and chatbots may require new online safe spaces, with chatbots demonstrating higher levels of involvement. These can moderate social interactions, facilitating engagement, inclusivity and understanding within the parties involved. This is in stark contrast to the current challenge of software agents or bots in social networks, as for example seen in Twitter bots utilizing COVID-19 content to spread political conspiracies [ 34 ], and the general trend of deploying bots in large scale for political interference and influence [ 44 ].

Chatbots will both raise critical ethical challenges and hold implications for democratization of technology, and implementing research addressing these directions is important. Chatbots permit users to interact through natural language, and consequently are a potential low threshold means to access information and services and promote inclusion. However, due to technological limitations and design choices, they can be the means to perpetuate and even reinforce existing biases in society, exclude or discriminate against some user groups (e.g. [ 33 , 51 ]) and over-represent or enshrine specific values. Future research will need to investigate and demonstrate democratization of chatbots in practice, where conversational technology is to be made easily and widely accessible for various businesses and user groups across the globe so that more people can benefit from conversational interaction. Moreover, as part of chatbot democratization, it will be important to make their development process more accessible as well, without requiring chatbot developers to have in-depth software engineering knowledge—as exemplified in using visual programming approaches such as Blockly to chatbot development [ 89 ]. In this way, chatbots can be created by experts in the domain where they will be used. This aspect is fundamental since chatbots are not conventional technologies, but they are developing as agents operating in social contexts.

Taking a broader ethical perspective, key questions when implementing future research on chatbots may include: What are the ethical implications of chatbots imitating human beings? Whose (and which) values should guide design practice within a global marketplace? What are the ethical implications of replacing humans with chatbots as a means of support for purposes such as commerce, therapy, or social interaction? How to facilitate chatbot support in decision making without risking or compromising agreed ethical principles? Ethical reflections and discussion on chatbots and chatbot applications are already emerging (e.g. [ 65 , 91 ]). We anticipate that advances in the democratization of chatbots will increasingly inspire ethical discourse that ties in with higher-level discussions about chatbot applications.

Perspectives on chatbot application areas

The identified research topics and corresponding future research directions may guide research so as to contribute to the fundamental understanding of the chatbot technology and the corresponding user interaction and engagement. However, to generate added value in specific application areas—such as customer service, health, education, office work, and home applications—further reflections about the respective use cases are needed. In particular, researchers need to analyse how chatbots may be leveraged and taken up in different application areas, how knowledge and research may be transferable across different application areas, and whether distinct research agenda should be established.

Many aspects of the outlined on our future research agenda are valid for any application area. For example, results concerning chatbot communication, user experience, design, and technology are the basis for applying chatbots in specific application areas. However, further analysis efforts are needed to understand the characteristics of each application area in more detail. For instance, requirements in the health sector concerning privacy aspects, ethics, and trust may be significantly more demanding than similar requirements in other sectors as they might have severe impacts on the users and concern highly sensitive personal information. In business contexts such as corporate customer support scenarios, the potential impacts may be less severe, but specific corporate regulations and norms need to be considered. In contrast to that, the use of chatbots in personal settings, e.g. the use of chatbots for social relationship, is often mainly driven by motivations for engagement and meaning-making. In contrast to health and business contexts, personal benefits are often not measured in monetary terms, but the main focus of personal usage is the improvement of daily life or wellbeing.

Regardless of differences among the diverse application areas in application-oriented research, many research studies exist in specific domains that could possibly be transferred to others. For instance, studies focusing on information provision in business contexts can most likely be applied in the health sectors as well, e.g., provision of product information will likely be similar to explaining healthy nutrition. However, to enable a transfer of research results across application areas, commonalities and differences of the involved application areas need to be identified and assessed. If the main characteristics of both are similar, transfer of the research results seems viable. Based on such an analysis and comparison, a generalization of the research across application areas seems possible. This procedure for future research on chatbot application areas could lead to a substantial increase in the body of knowledge as many research results from existing pilot studies and prototypes for specific application areas exist and may be reused as the basis for transfer and generalization (i.e., general design guidelines) to further application areas.

Continuing the discussion and collaboration

The presented challenges may serve as a step in the direction of contributing to the body of knowledge about chatbot usage and challenges, the frameworks and platforms underpinning chatbot applications, as well as needed future work on the broader implications of chatbots to work and society.

The proposed future research directions are intended as a response to the current lack of coherence in the emerging field of chatbot research, which may in part be observed by the broad range of journals and conferences in which findings from chatbot research are presented, and also the lack of commonly agreed key constructs, models, and measurement instruments. While this may be expected in an emerging research area, future research will benefit from a greater degree of coherence and cohesiveness in the field.

Nevertheless, there may be topics that have been omitted in the process leading up to our proposition, and relevant state-of-the-art and current research challenges may have been left out. Furthermore, as the field evolves, it is necessary to update the set of topics and research directions regularly. In consequence, continued interdisciplinary discussion and collaboration are needed to validate and refine the proposed set of future research directions.

One limitation deserving particular mention concerns the context of this discussion. The findings are based on discussions at the CONVERSATIONS workshop and mainly involve researchers from European organizations. While we assume the proposed directions hold broad international relevance and interest, it may be fruitful to test this assumption through discussion in the field—a discussion which we hope this paper will spur.

In further discussion and collaboration on chatbot research directions, care should be taken to involve the broadest possible set of interests and perspectives. For example, it will be beneficial to involve both researchers and practitioners, as well as the emerging and established set of research communities with an interest in conversational computer systems, to make sure that different enabling technologies and knowledge resources needed in future development and design of chatbots are represented. While research on conversational systems and user interfaces has been conducted for decades, chatbot research and design are still in its relative infancy. A consolidation of the field is needed, and we hope the proposed research agenda, with its directions for future research, may serve as a step towards such consolidation.

Open access funding provided by SINTEF AS. Funding supporting the work conducted by the first author was provided by Norges Forskningsråd (Grant No. 270940).

1 DialogFlow, https://cloud.google.com/dialogflow .

2 Microsoft Bot Framework, https://dev.botframework.com/ .

3 Pandorabots, https://home.pandorabots.com/ .

4 Rasa, https://rasa.com/ .

5 Mycroft, https://mycroft.ai/ .

6 CUI 2021—Conversational User Interfaces, https://www.conversationaluserinterfaces.org/2021/ .

7 CONVERSATIONS 2021—international workshop on chatbot research, https://conversations2021.wordpress.com/ .

8 CAIR 2020—Conversational Approaches to Information Retrieval, https://sites.google.com/view/cair-ws/cair-2020 .

9 SIGDIAL—Special interest Group on Discourse and Dialogue, https://www.sigdial.org/ .

10 IVA 2021—21st ACM International Conference on Intelligent Virtual Agents, https://sites.google.com/view/iva2021/ .

11 IWSDS 2021—12th International Workshop on Spoken Dialogue Systems Technology, https://www.iwsds.tech/ .

12 INTERSPEECH 2021—The 21st Annual Conference of the International Speech Communication Association, https://interspeech2021.org .

13 Google Conversation Design, https://designguidelines.withgoogle.com/conversation/ .

14 Alexa Design Guide, https://developer.amazon.com/en-US/docs/alexa/alexa-design/get-started.html .

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

IMAGES

  1. Turn Your Research Paper into a Chatbot in Minutes

    chatbot research paper 2020

  2. (PDF) The Use of Chatbots in Digital Business Transformation: A

    chatbot research paper 2020

  3. A Detailed Guide to Chatbot In 2020

    chatbot research paper 2020

  4. (PDF) A Review on Chatbot Design and Implementation Techniques

    chatbot research paper 2020

  5. Research Paper On Chatbot for Diabetic Patient

    chatbot research paper 2020

  6. (PDF) Implementation of a Chat Bot System using AI and NLP

    chatbot research paper 2020

VIDEO

  1. Chatbot Masterclass 2

  2. The Most Common Chatbot Questions Answered!

  3. What is ChatGPT AI chatbot?

  4. Day 50/100: Meta Blenderbot 3 #ai #meta #chatbot

  5. Intro to Chatbots

  6. Ep-3: Build a Chatbot in under 1 Minute

COMMENTS

  1. Chatbots applications in education: A systematic review

    Chatbots are conversational or interactive agents that provide instant response to the user ( Okonkwo & Ade-Ibijola, 2020; Smutny & Schreiberova, 2020 ). Chatbots are increasingly being used to improve student interaction in this current world of technology where communication and many other activities rely heavily on online platform.

  2. Chatbots: History, technology, and applications

    1. Introduction Artificial intelligence (AI) has influenced how we engage in our every day activities by designing and evaluating advanced applications and devices, called intelligent agents, which can perform various functions. A chatbot is an artificial intelligence program and a Human-computer Interaction (HCI) model ( Bansal & Khan, 2018 ).

  3. (PDF) An Overview of Chatbot Technology

    In this paper, we first present a historical overview of the evolution of the international community's interest in chatbots. Next, we discuss the motivations that drive the use of chatbots,...

  4. Future directions for chatbot research: an interdisciplinary research

    Following a deliberative research analysis process among the workshop participants, we explore future directions within six topics of interest: (a) users and implications, (b) user experience and design, (c) frameworks and platforms, (d) chatbots for collaboration, (e) democratizing chatbots, and (f) ethics and privacy.

  5. An Overview of Chatbot Technology

    Artificial Intelligence Applications and Innovations. 2020; 584: 373-383. Published online 2020 May 6. doi: 10.1007/978-3-030-49186-4_31 PMCID: PMC7256567 An Overview of Chatbot Technology Guest Editor (s): Ilias Maglogiannis, 15 Lazaros Iliadis, 16 and Elias Pimenidis 17 15 Department of Digital Systems, University of Piraeus, Piraeus, Greece

  6. Effectiveness and Safety of Using Chatbots to Improve Mental Health

    Chatbots are systems that are able to converse and interact with human users using spoken, written, and visual language. While numerous studies have assessed the effectiveness and safety of using chatbots in mental health, no reviews have pooled the results of those studies. Objective

  7. The human side of human-chatbot interaction: A ...

    Lorenzo Curti, Arianna Boldi https://doi.org/10.1016/j.ijhcs.2021.102630 Get rights and content • We reviewed 83 articles that investigate how people interact with text-based chatbots • We identified five relevant themes that are recurrent across the reviewed articles •

  8. a Study of Human-Chatbot Relationships

    Previous research has demonstrated discontinued usage of a chatbot due to perceived weaknesses in data protection (Prakash and Das, 2020). Future research could potentially add to this by exploring possible tensions arising from varying levels of trust in the chatbot and trust in the provider.

  9. PDF arXiv:2001.09977v3 [cs.CL] 27 Feb 2020

    2 Evaluating chatbots Evaluating chatbots and natural language gen-eration is a well-known challenge (Liu et al., 2016; Lowe et al., 2017; Novikova et al., 2017; Hashimoto et al., 2019), which we aim to address in this paper. First, we propose a human evalua-tion metric that captures key elements of human-

  10. [2201.06657] A Literature Survey of Recent Advances in Chatbots

    A Literature Survey of Recent Advances in Chatbots. Guendalina Caldarini, Sardar Jaf, Kenneth McGarry. Chatbots are intelligent conversational computer systems designed to mimic human conversation to enable automated online guidance and support. The increased benefits of chatbots led to their wide adoption by many industries in order to provide ...

  11. Users' experiences with chatbots: findings from a ...

    For chatbots to be broadly adopted by users, it is critical that they are experienced as useful and pleasurable. While there is an emerging body of research concerning user uptake and use of chatbots, there is a lack of theoretically grounded studies detailing what constitutes good or poor chatbot user experiences. In this paper, we present findings from a questionnaire study involving more ...

  12. (PDF) Design and Development of CHATBOT: A Review

    A CHATBOT is an artificially created virtual entity that interacts with users using interactive textual or speech skills. This CHATBOT directly chats with the people using artificial...

  13. The Development and Use of Chatbots in Public Health: Scoping Review

    This paper complements this research and addresses a gap in the literature by assessing the breadth and scope of research evidence for the use of chatbots across the domain of public health. ... CONVERSATIONS 2020: Chatbot Research and Design; November 23-24, 2020; Virtual event. 2021. Feb 03, pp. 131-144.

  14. Are We There Yet?

    Chatbots are a promising technology with the potential to enhance workplaces and everyday life. In terms of scalability and accessibility, they also offer unique possibilities as communication and information tools for digital learning. In this paper, we present a systematic literature review investigating the areas of education where chatbots have already been applied, explore the pedagogical ...

  15. An Overview of Chatbot Technology

    1 Introduction Artificial Intelligence (ΑΙ) increasingly integrates our daily lives with the creation and analysis of intelligent software and hardware, called intelligent agents. Intelligent agents can do a variety of tasks ranging from labor work to sophisticated operations.

  16. Chatbots in Education and Research: A Critical Examination of Ethical

    The paper also discusses the ethical challenges related to the use of AI systems and chatbots in research, as well as the potential for misuse and exploitation. ... The paper provides an overview of the tool and a step-by-step tutorial for designing an experiment with a chatbot. Another research paper ... IEEE Access 2020, 8, 75264-75278 ...

  17. (PDF) Introduction to AI Chatbots

    PDF | On Jul 11, 2020, Aishwarya Gupta published Introduction to AI Chatbots | Find, read and cite all the research you need on ResearchGate Home Artificial Intelligence Chatbot ArticlePDF...

  18. Artificially intelligent chatbots in digital mental health

    Areas covered . We summarize the current landscape of DMHIs, with a focus on AI-based chatbots. Happify Health's AI chatbot, Anna, serves as a case study for discussion of potential challenges and how these might be addressed, and demonstrates the promise of chatbots as effective, usable, and adoptable within DMHIs.Finally, we discuss ways in which future research can advance the field ...

  19. Service chatbot: Co-citation and big data analysis toward a review and

    Recently, a number of review papers have been published, mostly based on a systematic literature review (SLR) or other qualitative approaches that examine the research state on chatbots (Luo et al., 2022; Nagarhalli et al., 2020; Rapp et al., 2021). In particular, despite the ongoing, intense endeavors in both academia and industry, the current ...

  20. (PDF) The Impact of Artificial Intelligence on Chatbot Technology: A

    This research paper aims to explore the current state of AI-powered chatbot technology, focusing on the latest advancements and leading innovations.

  21. Beware of Botshit: How to Manage the Epistemic Risks of Generative Chatbots

    This article focuses on how to use chatbots for content generation work while mitigating the epistemic (i.e., the process of producing knowledge) risks associated with botshit. Drawing on risk management research, we introduce a typology framework that orients how chatbots can be used based on two dimensions: response veracity verifiability ...

  22. Chatbot Research and Design

    Chatbot Research and Design 6th International Workshop, CONVERSATIONS 2022, Amsterdam, The Netherlands, November 22-23, 2022, Revised Selected Papers Home Conference proceedings Editors: Asbjørn Følstad, Theo Araujo, Symeon Papadopoulos, Effie L.-C. Law, Ewa Luger, Morten Goodwin, Petter Bae Brandtzaeg

  23. Future directions for chatbot research: an interdisciplinary research

    Chatbot research is currently evolving within and across a range of disciplines and has a strong interdisciplinary character. Ground-breaking research has been presented in fields as diverse as communication (e.g. ... and how Human-AI Interaction is uniquely difficult to design. In: Proceedings of CHI 2020 (paper no. 174). ACM,New York. 117 ...

  24. (PDF) Chatbots: History, technology, and applications

    The rest of the paper is structured as follows. ... more research interest in chatbots is the USA, while the United King- ... Arias-Navarrete, and Palacios (2020), the research focuses on the layers.