types of references for research paper

Buy Me a Coffee

Home » References in Research – Types, Examples and Writing Guide

References in Research – Types, Examples and Writing Guide

Table of Contents

References in Research

Definition:

References in research are a list of sources that a researcher has consulted or cited while conducting their study. They are an essential component of any academic work, including research papers, theses, dissertations, and other scholarly publications.

Types of References

There are several types of references used in research, and the type of reference depends on the source of information being cited. The most common types of references include:

References to books typically include the author’s name, title of the book, publisher, publication date, and place of publication.

Example: Smith, J. (2018). The Art of Writing. Penguin Books.

Journal Articles

References to journal articles usually include the author’s name, title of the article, name of the journal, volume and issue number, page numbers, and publication date.

Example: Johnson, T. (2021). The Impact of Social Media on Mental Health. Journal of Psychology, 32(4), 87-94.

Web sources

References to web sources should include the author or organization responsible for the content, the title of the page, the URL, and the date accessed.

Example: World Health Organization. (2020). Coronavirus disease (COVID-19) advice for the public. Retrieved from https://www.who.int/emergencies/disease/novel-coronavirus-2019/advice-for-public

Conference Proceedings

References to conference proceedings should include the author’s name, title of the paper, name of the conference, location of the conference, date of the conference, and page numbers.

Example: Chen, S., & Li, J. (2019). The Future of AI in Education. Proceedings of the International Conference on Educational Technology, Beijing, China, July 15-17, pp. 67-78.

References to reports typically include the author or organization responsible for the report, title of the report, publication date, and publisher.

Example: United Nations. (2020). The Sustainable Development Goals Report. United Nations.

Formats of References

Some common Formates of References with their examples are as follows:

APA (American Psychological Association) Style

The APA (American Psychological Association) Style has specific guidelines for formatting references used in academic papers, articles, and books. Here are the different reference formats in APA style with examples:

Author, A. A. (Year of publication). Title of book. Publisher.

Example : Smith, J. K. (2005). The psychology of social interaction. Wiley-Blackwell.

Journal Article

Author, A. A., Author, B. B., & Author, C. C. (Year of publication). Title of article. Title of Journal, volume number(issue number), page numbers.

Example : Brown, L. M., Keating, J. G., & Jones, S. M. (2012). The role of social support in coping with stress among African American adolescents. Journal of Research on Adolescence, 22(1), 218-233.

Author, A. A. (Year of publication or last update). Title of page. Website name. URL.

Example : Centers for Disease Control and Prevention. (2020, December 11). COVID-19: How to protect yourself and others. https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/prevention.html

Magazine article

Author, A. A. (Year, Month Day of publication). Title of article. Title of Magazine, volume number(issue number), page numbers.

Example : Smith, M. (2019, March 11). The power of positive thinking. Psychology Today, 52(3), 60-65.

Newspaper article:

Author, A. A. (Year, Month Day of publication). Title of article. Title of Newspaper, page numbers.

Example: Johnson, B. (2021, February 15). New study shows benefits of exercise on mental health. The New York Times, A8.

Edited book

Editor, E. E. (Ed.). (Year of publication). Title of book. Publisher.

Example : Thompson, J. P. (Ed.). (2014). Social work in the 21st century. Sage Publications.

Chapter in an edited book:

Author, A. A. (Year of publication). Title of chapter. In E. E. Editor (Ed.), Title of book (pp. page numbers). Publisher.

Example : Johnson, K. S. (2018). The future of social work: Challenges and opportunities. In J. P. Thompson (Ed.), Social work in the 21st century (pp. 105-118). Sage Publications.

MLA (Modern Language Association) Style

The MLA (Modern Language Association) Style is a widely used style for writing academic papers and essays in the humanities. Here are the different reference formats in MLA style:

Author’s Last name, First name. Title of Book. Publisher, Publication year.

Example : Smith, John. The Psychology of Social Interaction. Wiley-Blackwell, 2005.

Journal article

Author’s Last name, First name. “Title of Article.” Title of Journal, volume number, issue number, Publication year, page numbers.

Example : Brown, Laura M., et al. “The Role of Social Support in Coping with Stress among African American Adolescents.” Journal of Research on Adolescence, vol. 22, no. 1, 2012, pp. 218-233.

Author’s Last name, First name. “Title of Webpage.” Website Name, Publication date, URL.

Example : Centers for Disease Control and Prevention. “COVID-19: How to Protect Yourself and Others.” CDC, 11 Dec. 2020, https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/prevention.html.

Author’s Last name, First name. “Title of Article.” Title of Magazine, Publication date, page numbers.

Example : Smith, Mary. “The Power of Positive Thinking.” Psychology Today, Mar. 2019, pp. 60-65.

Newspaper article

Author’s Last name, First name. “Title of Article.” Title of Newspaper, Publication date, page numbers.

Example : Johnson, Bob. “New Study Shows Benefits of Exercise on Mental Health.” The New York Times, 15 Feb. 2021, p. A8.

Editor’s Last name, First name, editor. Title of Book. Publisher, Publication year.

Example : Thompson, John P., editor. Social Work in the 21st Century. Sage Publications, 2014.

Chapter in an edited book

Author’s Last name, First name. “Title of Chapter.” Title of Book, edited by Editor’s First Name Last name, Publisher, Publication year, page numbers.

Example : Johnson, Karen S. “The Future of Social Work: Challenges and Opportunities.” Social Work in the 21st Century, edited by John P. Thompson, Sage Publications, 2014, pp. 105-118.

Chicago Manual of Style

The Chicago Manual of Style is a widely used style for writing academic papers, dissertations, and books in the humanities and social sciences. Here are the different reference formats in Chicago style:

Example : Smith, John K. The Psychology of Social Interaction. Wiley-Blackwell, 2005.

Author’s Last name, First name. “Title of Article.” Title of Journal volume number, no. issue number (Publication year): page numbers.

Example : Brown, Laura M., John G. Keating, and Sarah M. Jones. “The Role of Social Support in Coping with Stress among African American Adolescents.” Journal of Research on Adolescence 22, no. 1 (2012): 218-233.

Author’s Last name, First name. “Title of Webpage.” Website Name. Publication date. URL.

Example : Centers for Disease Control and Prevention. “COVID-19: How to Protect Yourself and Others.” CDC. December 11, 2020. https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/prevention.html.

Author’s Last name, First name. “Title of Article.” Title of Magazine, Publication date.

Example : Smith, Mary. “The Power of Positive Thinking.” Psychology Today, March 2019.

Author’s Last name, First name. “Title of Article.” Title of Newspaper, Publication date.

Example : Johnson, Bob. “New Study Shows Benefits of Exercise on Mental Health.” The New York Times, February 15, 2021.

Example : Thompson, John P., ed. Social Work in the 21st Century. Sage Publications, 2014.

Author’s Last name, First name. “Title of Chapter.” In Title of Book, edited by Editor’s First Name Last Name, page numbers. Publisher, Publication year.

Example : Johnson, Karen S. “The Future of Social Work: Challenges and Opportunities.” In Social Work in the 21st Century, edited by John P. Thompson, 105-118. Sage Publications, 2014.

Harvard Style

The Harvard Style, also known as the Author-Date System, is a widely used style for writing academic papers and essays in the social sciences. Here are the different reference formats in Harvard Style:

Author’s Last name, First name. Year of publication. Title of Book. Place of publication: Publisher.

Example : Smith, John. 2005. The Psychology of Social Interaction. Oxford: Wiley-Blackwell.

Author’s Last name, First name. Year of publication. “Title of Article.” Title of Journal volume number (issue number): page numbers.

Example: Brown, Laura M., John G. Keating, and Sarah M. Jones. 2012. “The Role of Social Support in Coping with Stress among African American Adolescents.” Journal of Research on Adolescence 22 (1): 218-233.

Author’s Last name, First name. Year of publication. “Title of Webpage.” Website Name. URL. Accessed date.

Example : Centers for Disease Control and Prevention. 2020. “COVID-19: How to Protect Yourself and Others.” CDC. https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/prevention.html. Accessed April 1, 2023.

Author’s Last name, First name. Year of publication. “Title of Article.” Title of Magazine, month and date of publication.

Example : Smith, Mary. 2019. “The Power of Positive Thinking.” Psychology Today, March 2019.

Author’s Last name, First name. Year of publication. “Title of Article.” Title of Newspaper, month and date of publication.

Example : Johnson, Bob. 2021. “New Study Shows Benefits of Exercise on Mental Health.” The New York Times, February 15, 2021.

Editor’s Last name, First name, ed. Year of publication. Title of Book. Place of publication: Publisher.

Example : Thompson, John P., ed. 2014. Social Work in the 21st Century. Thousand Oaks, CA: Sage Publications.

Author’s Last name, First name. Year of publication. “Title of Chapter.” In Title of Book, edited by Editor’s First Name Last Name, page numbers. Place of publication: Publisher.

Example : Johnson, Karen S. 2014. “The Future of Social Work: Challenges and Opportunities.” In Social Work in the 21st Century, edited by John P. Thompson, 105-118. Thousand Oaks, CA: Sage Publications.

Vancouver Style

The Vancouver Style, also known as the Uniform Requirements for Manuscripts Submitted to Biomedical Journals, is a widely used style for writing academic papers in the biomedical sciences. Here are the different reference formats in Vancouver Style:

Author’s Last name, First name. Title of Book. Edition number. Place of publication: Publisher; Year of publication.

Example : Smith, John K. The Psychology of Social Interaction. 2nd ed. Oxford: Wiley-Blackwell; 2005.

Author’s Last name, First name. Title of Article. Abbreviated Journal Title. Year of publication; volume number(issue number):page numbers.

Example : Brown LM, Keating JG, Jones SM. The Role of Social Support in Coping with Stress among African American Adolescents. J Res Adolesc. 2012;22(1):218-233.

Author’s Last name, First name. Title of Webpage. Website Name [Internet]. Publication date. [cited date]. Available from: URL.

Example : Centers for Disease Control and Prevention. COVID-19: How to Protect Yourself and Others [Internet]. 2020 Dec 11. [cited 2023 Apr 1]. Available from: https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/prevention.html.

Author’s Last name, First name. Title of Article. Title of Magazine. Year of publication; month and day of publication:page numbers.

Example : Smith M. The Power of Positive Thinking. Psychology Today. 2019 Mar 1:32-35.

Author’s Last name, First name. Title of Article. Title of Newspaper. Year of publication; month and day of publication:page numbers.

Example : Johnson B. New Study Shows Benefits of Exercise on Mental Health. The New York Times. 2021 Feb 15:A4.

Editor’s Last name, First name, editor. Title of Book. Edition number. Place of publication: Publisher; Year of publication.

Example: Thompson JP, editor. Social Work in the 21st Century. 1st ed. Thousand Oaks, CA: Sage Publications; 2014.

Author’s Last name, First name. Title of Chapter. In: Editor’s Last name, First name, editor. Title of Book. Edition number. Place of publication: Publisher; Year of publication. page numbers.

Example : Johnson KS. The Future of Social Work: Challenges and Opportunities. In: Thompson JP, editor. Social Work in the 21st Century. 1st ed. Thousand Oaks, CA: Sage Publications; 2014. p. 105-118.

Turabian Style

Turabian style is a variation of the Chicago style used in academic writing, particularly in the fields of history and humanities. Here are the different reference formats in Turabian style:

Author’s Last name, First name. Title of Book. Place of publication: Publisher, Year of publication.

Example : Smith, John K. The Psychology of Social Interaction. Oxford: Wiley-Blackwell, 2005.

Author’s Last name, First name. “Title of Article.” Title of Journal volume number, no. issue number (Year of publication): page numbers.

Example : Brown, LM, Keating, JG, Jones, SM. “The Role of Social Support in Coping with Stress among African American Adolescents.” J Res Adolesc 22, no. 1 (2012): 218-233.

Author’s Last name, First name. “Title of Webpage.” Name of Website. Publication date. Accessed date. URL.

Example : Centers for Disease Control and Prevention. “COVID-19: How to Protect Yourself and Others.” CDC. December 11, 2020. Accessed April 1, 2023. https://www.cdc.gov/coronavirus/2019-ncov/prevent-getting-sick/prevention.html.

Author’s Last name, First name. “Title of Article.” Title of Magazine, Month Day, Year of publication, page numbers.

Example : Smith, M. “The Power of Positive Thinking.” Psychology Today, March 1, 2019, 32-35.

Author’s Last name, First name. “Title of Article.” Title of Newspaper, Month Day, Year of publication.

Example : Johnson, B. “New Study Shows Benefits of Exercise on Mental Health.” The New York Times, February 15, 2021.

Editor’s Last name, First name, ed. Title of Book. Place of publication: Publisher, Year of publication.

Example : Thompson, JP, ed. Social Work in the 21st Century. Thousand Oaks, CA: Sage Publications, 2014.

Author’s Last name, First name. “Title of Chapter.” In Title of Book, edited by Editor’s Last name, First name, page numbers. Place of publication: Publisher, Year of publication.

Example : Johnson, KS. “The Future of Social Work: Challenges and Opportunities.” In Social Work in the 21st Century, edited by Thompson, JP, 105-118. Thousand Oaks, CA: Sage Publications, 2014.

IEEE (Institute of Electrical and Electronics Engineers) Style

IEEE (Institute of Electrical and Electronics Engineers) style is commonly used in engineering, computer science, and other technical fields. Here are the different reference formats in IEEE style:

Author’s Last name, First name. Book Title. Place of Publication: Publisher, Year of publication.

Example : Oppenheim, A. V., & Schafer, R. W. Discrete-Time Signal Processing. Upper Saddle River, NJ: Prentice Hall, 2010.

Author’s Last name, First name. “Title of Article.” Abbreviated Journal Title, vol. number, no. issue number, pp. page numbers, Month year of publication.

Example: Shannon, C. E. “A Mathematical Theory of Communication.” Bell System Technical Journal, vol. 27, no. 3, pp. 379-423, July 1948.

Conference paper

Author’s Last name, First name. “Title of Paper.” In Title of Conference Proceedings, Place of Conference, Date of Conference, pp. page numbers, Year of publication.

Example: Gupta, S., & Kumar, P. “An Improved System of Linear Discriminant Analysis for Face Recognition.” In Proceedings of the 2011 International Conference on Computer Science and Network Technology, Harbin, China, Dec. 2011, pp. 144-147.

Author’s Last name, First name. “Title of Webpage.” Name of Website. Date of publication or last update. Accessed date. URL.

Example : National Aeronautics and Space Administration. “Apollo 11.” NASA. July 20, 1969. Accessed April 1, 2023. https://www.nasa.gov/mission_pages/apollo/apollo11.html.

Technical report

Author’s Last name, First name. “Title of Report.” Name of Institution or Organization, Report number, Year of publication.

Example : Smith, J. R. “Development of a New Solar Panel Technology.” National Renewable Energy Laboratory, NREL/TP-6A20-51645, 2011.

Author’s Last name, First name. “Title of Patent.” Patent number, Issue date.

Example : Suzuki, H. “Method of Producing Carbon Nanotubes.” US Patent 7,151,019, December 19, 2006.

Standard Title. Standard number, Publication date.

Example : IEEE Standard for Floating-Point Arithmetic. IEEE Std 754-2008, August 29, 2008

ACS (American Chemical Society) Style

ACS (American Chemical Society) style is commonly used in chemistry and related fields. Here are the different reference formats in ACS style:

Author’s Last name, First name; Author’s Last name, First name. Title of Article. Abbreviated Journal Title Year, Volume, Page Numbers.

Example : Wang, Y.; Zhao, X.; Cui, Y.; Ma, Y. Facile Preparation of Fe3O4/graphene Composites Using a Hydrothermal Method for High-Performance Lithium Ion Batteries. ACS Appl. Mater. Interfaces 2012, 4, 2715-2721.

Author’s Last name, First name. Book Title; Publisher: Place of Publication, Year of Publication.

Example : Carey, F. A. Organic Chemistry; McGraw-Hill: New York, 2008.

Author’s Last name, First name. Chapter Title. In Book Title; Editor’s Last name, First name, Ed.; Publisher: Place of Publication, Year of Publication; Volume number, Chapter number, Page Numbers.

Example : Grossman, R. B. Analytical Chemistry of Aerosols. In Aerosol Measurement: Principles, Techniques, and Applications; Baron, P. A.; Willeke, K., Eds.; Wiley-Interscience: New York, 2001; Chapter 10, pp 395-424.

Author’s Last name, First name. Title of Webpage. Website Name, URL (accessed date).

Example : National Institute of Standards and Technology. Atomic Spectra Database. https://www.nist.gov/pml/atomic-spectra-database (accessed April 1, 2023).

Author’s Last name, First name. Patent Number. Patent Date.

Example : Liu, Y.; Huang, H.; Chen, H.; Zhang, W. US Patent 9,999,999, December 31, 2022.

Author’s Last name, First name; Author’s Last name, First name. Title of Article. In Title of Conference Proceedings, Publisher: Place of Publication, Year of Publication; Volume Number, Page Numbers.

Example : Jia, H.; Xu, S.; Wu, Y.; Wu, Z.; Tang, Y.; Huang, X. Fast Adsorption of Organic Pollutants by Graphene Oxide. In Proceedings of the 15th International Conference on Environmental Science and Technology, American Chemical Society: Washington, DC, 2017; Volume 1, pp 223-228.

AMA (American Medical Association) Style

AMA (American Medical Association) style is commonly used in medical and scientific fields. Here are the different reference formats in AMA style:

Author’s Last name, First name. Article Title. Journal Abbreviation. Year; Volume(Issue):Page Numbers.

Example : Jones, R. A.; Smith, B. C. The Role of Vitamin D in Maintaining Bone Health. JAMA. 2019;321(17):1765-1773.

Author’s Last name, First name. Book Title. Edition number. Place of Publication: Publisher; Year.

Example : Guyton, A. C.; Hall, J. E. Textbook of Medical Physiology. 13th ed. Philadelphia, PA: Saunders; 2015.

Author’s Last name, First name. Chapter Title. In: Editor’s Last name, First name, ed. Book Title. Edition number. Place of Publication: Publisher; Year: Page Numbers.

Example: Rajakumar, K. Vitamin D and Bone Health. In: Holick, M. F., ed. Vitamin D: Physiology, Molecular Biology, and Clinical Applications. 2nd ed. New York, NY: Springer; 2010:211-222.

Author’s Last name, First name. Webpage Title. Website Name. URL. Published date. Updated date. Accessed date.

Example : National Cancer Institute. Breast Cancer Prevention (PDQ®)–Patient Version. National Cancer Institute. https://www.cancer.gov/types/breast/patient/breast-prevention-pdq. Published October 11, 2022. Accessed April 1, 2023.

Author’s Last name, First name. Conference presentation title. In: Conference Title; Conference Date; Place of Conference.

Example : Smith, J. R. Vitamin D and Bone Health: A Meta-Analysis. In: Proceedings of the Annual Meeting of the American Society for Bone and Mineral Research; September 20-23, 2022; San Diego, CA.

Thesis or dissertation

Author’s Last name, First name. Title of Thesis or Dissertation. Degree level [Doctoral dissertation or Master’s thesis]. University Name; Year.

Example : Wilson, S. A. The Effects of Vitamin D Supplementation on Bone Health in Postmenopausal Women [Doctoral dissertation]. University of California, Los Angeles; 2018.

ASCE (American Society of Civil Engineers) Style

The ASCE (American Society of Civil Engineers) style is commonly used in civil engineering fields. Here are the different reference formats in ASCE style:

Author’s Last name, First name. “Article Title.” Journal Title, volume number, issue number (year): page numbers. DOI or URL (if available).

Example : Smith, J. R. “Evaluation of the Effectiveness of Sustainable Drainage Systems in Urban Areas.” Journal of Environmental Engineering, vol. 146, no. 3 (2020): 04020010. https://doi.org/10.1061/(ASCE)EE.1943-7870.0001668.

Example : McCuen, R. H. Hydrologic Analysis and Design. 4th ed. Upper Saddle River, NJ: Pearson Education; 2013.

Author’s Last name, First name. “Chapter Title.” In: Editor’s Last name, First name, ed. Book Title. Edition number. Place of Publication: Publisher; Year: page numbers.

Example : Maidment, D. R. “Floodplain Management in the United States.” In: Shroder, J. F., ed. Treatise on Geomorphology. San Diego, CA: Academic Press; 2013: 447-460.

Author’s Last name, First name. “Paper Title.” In: Conference Title; Conference Date; Location. Place of Publication: Publisher; Year: page numbers.

Example: Smith, J. R. “Sustainable Drainage Systems for Urban Areas.” In: Proceedings of the ASCE International Conference on Sustainable Infrastructure; November 6-9, 2019; Los Angeles, CA. Reston, VA: American Society of Civil Engineers; 2019: 156-163.

Author’s Last name, First name. “Report Title.” Report number. Place of Publication: Publisher; Year.

Example : U.S. Army Corps of Engineers. “Hurricane Sandy Coastal Risk Reduction Program, New York and New Jersey.” Report No. P-15-001. Washington, DC: U.S. Army Corps of Engineers; 2015.

CSE (Council of Science Editors) Style

The CSE (Council of Science Editors) style is commonly used in the scientific and medical fields. Here are the different reference formats in CSE style:

Author’s Last name, First Initial. Middle Initial. “Article Title.” Journal Title. Year;Volume(Issue):Page numbers.

Example : Smith, J.R. “Evaluation of the Effectiveness of Sustainable Drainage Systems in Urban Areas.” Journal of Environmental Engineering. 2020;146(3):04020010.

Author’s Last name, First Initial. Middle Initial. Book Title. Edition number. Place of Publication: Publisher; Year.

Author’s Last name, First Initial. Middle Initial. “Chapter Title.” In: Editor’s Last name, First Initial. Middle Initial., ed. Book Title. Edition number. Place of Publication: Publisher; Year:Page numbers.

Author’s Last name, First Initial. Middle Initial. “Paper Title.” In: Conference Title; Conference Date; Location. Place of Publication: Publisher; Year.

Example : Smith, J.R. “Sustainable Drainage Systems for Urban Areas.” In: Proceedings of the ASCE International Conference on Sustainable Infrastructure; November 6-9, 2019; Los Angeles, CA. Reston, VA: American Society of Civil Engineers; 2019.

Author’s Last name, First Initial. Middle Initial. “Report Title.” Report number. Place of Publication: Publisher; Year.

Bluebook Style

The Bluebook style is commonly used in the legal field for citing legal documents and sources. Here are the different reference formats in Bluebook style:

Case citation

Case name, volume source page (Court year).

Example : Brown v. Board of Education, 347 U.S. 483 (1954).

Statute citation

Name of Act, volume source § section number (year).

Example : Clean Air Act, 42 U.S.C. § 7401 (1963).

Regulation citation

Name of regulation, volume source § section number (year).

Example: Clean Air Act, 40 C.F.R. § 52.01 (2019).

Book citation

Author’s Last name, First Initial. Middle Initial. Book Title. Edition number (if applicable). Place of Publication: Publisher; Year.

Example: Smith, J.R. Legal Writing and Analysis. 3rd ed. New York, NY: Aspen Publishers; 2015.

Journal article citation

Author’s Last name, First Initial. Middle Initial. “Article Title.” Journal Title. Volume number (year): first page-last page.

Example: Garcia, C. “The Right to Counsel: An International Comparison.” International Journal of Legal Information. 43 (2015): 63-94.

Website citation

Author’s Last name, First Initial. Middle Initial. “Page Title.” Website Title. URL (accessed month day, year).

Example : United Nations. “Universal Declaration of Human Rights.” United Nations. https://www.un.org/en/universal-declaration-human-rights/ (accessed January 3, 2023).

Oxford Style

The Oxford style, also known as the Oxford referencing system or the documentary-note citation system, is commonly used in the humanities, including literature, history, and philosophy. Here are the different reference formats in Oxford style:

Author’s Last name, First name. Book Title. Place of Publication: Publisher, Year of Publication.

Example : Smith, John. The Art of Writing. New York: Penguin, 2020.

Author’s Last name, First name. “Article Title.” Journal Title volume, no. issue (year): page range.

Example: Garcia, Carlos. “The Role of Ethics in Philosophy.” Philosophy Today 67, no. 3 (2019): 53-68.

Chapter in an edited book citation

Author’s Last name, First name. “Chapter Title.” In Book Title, edited by Editor’s Name, page range. Place of Publication: Publisher, Year of Publication.

Example : Lee, Mary. “Feminism in the 21st Century.” In The Oxford Handbook of Feminism, edited by Jane Smith, 51-69. Oxford: Oxford University Press, 2018.

Author’s Last name, First name. “Page Title.” Website Title. URL (accessed day month year).

Example : Jones, David. “The Importance of Learning Languages.” Oxford Language Center. https://www.oxfordlanguagecenter.com/importance-of-learning-languages/ (accessed 3 January 2023).

Dissertation or thesis citation

Author’s Last name, First name. “Title of Dissertation/Thesis.” PhD diss., University Name, Year of Publication.

Example : Brown, Susan. “The Art of Storytelling in American Literature.” PhD diss., University of Oxford, 2020.

Newspaper article citation

Author’s Last name, First name. “Article Title.” Newspaper Title, Month Day, Year.

Example : Robinson, Andrew. “New Developments in Climate Change Research.” The Guardian, September 15, 2022.

AAA (American Anthropological Association) Style

The American Anthropological Association (AAA) style is commonly used in anthropology research papers and journals. Here are the different reference formats in AAA style:

Author’s Last name, First name. Year of Publication. Book Title. Place of Publication: Publisher.

Example : Smith, John. 2019. The Anthropology of Food. New York: Routledge.

Author’s Last name, First name. Year of Publication. “Article Title.” Journal Title volume, no. issue: page range.

Example : Garcia, Carlos. 2021. “The Role of Ethics in Anthropology.” American Anthropologist 123, no. 2: 237-251.

Author’s Last name, First name. Year of Publication. “Chapter Title.” In Book Title, edited by Editor’s Name, page range. Place of Publication: Publisher.

Example: Lee, Mary. 2018. “Feminism in Anthropology.” In The Oxford Handbook of Feminism, edited by Jane Smith, 51-69. Oxford: Oxford University Press.

Author’s Last name, First name. Year of Publication. “Page Title.” Website Title. URL (accessed day month year).

Example : Jones, David. 2020. “The Importance of Learning Languages.” Oxford Language Center. https://www.oxfordlanguagecenter.com/importance-of-learning-languages/ (accessed January 3, 2023).

Author’s Last name, First name. Year of Publication. “Title of Dissertation/Thesis.” PhD diss., University Name.

Example : Brown, Susan. 2022. “The Art of Storytelling in Anthropology.” PhD diss., University of California, Berkeley.

Author’s Last name, First name. Year of Publication. “Article Title.” Newspaper Title, Month Day.

Example : Robinson, Andrew. 2021. “New Developments in Anthropology Research.” The Guardian, September 15.

AIP (American Institute of Physics) Style

The American Institute of Physics (AIP) style is commonly used in physics research papers and journals. Here are the different reference formats in AIP style:

Example : Johnson, S. D. 2021. “Quantum Computing and Information.” Journal of Applied Physics 129, no. 4: 043102.

Example : Feynman, Richard. 2018. The Feynman Lectures on Physics. New York: Basic Books.

Example : Jones, David. 2020. “The Future of Quantum Computing.” In The Handbook of Physics, edited by John Smith, 125-136. Oxford: Oxford University Press.

Conference proceedings citation

Author’s Last name, First name. Year of Publication. “Title of Paper.” Proceedings of Conference Name, date and location: page range. Place of Publication: Publisher.

Example : Chen, Wei. 2019. “The Applications of Nanotechnology in Solar Cells.” Proceedings of the 8th International Conference on Nanotechnology, July 15-17, Tokyo, Japan: 224-229. New York: AIP Publishing.

Example : American Institute of Physics. 2022. “About AIP Publishing.” AIP Publishing. https://publishing.aip.org/about-aip-publishing/ (accessed January 3, 2023).

Patent citation

Author’s Last name, First name. Year of Publication. Patent Number.

Example : Smith, John. 2018. US Patent 9,873,644.

References Writing Guide

Here are some general guidelines for writing references:

Follow the citation style guidelines: Different disciplines and journals may require different citation styles (e.g., APA, MLA, Chicago). It is important to follow the specific guidelines for the citation style required.
Include all necessary information : Each citation should include enough information for readers to locate the source. For example, a journal article citation should include the author(s), title of the article, journal title, volume number, issue number, page numbers, and publication year.
Use proper formatting: Citation styles typically have specific formatting requirements for different types of sources. Make sure to follow the proper formatting for each citation.
Order citations alphabetically: If listing multiple sources, they should be listed alphabetically by the author’s last name.
Be consistent: Use the same citation style throughout the entire paper or project.
Check for accuracy: Double-check all citations to ensure accuracy, including correct spelling of author names and publication information.
Use reputable sources: When selecting sources to cite, choose reputable and authoritative sources. Avoid sources that are biased or unreliable.
Include all sources: Make sure to include all sources used in the research, including those that were not directly quoted but still informed the work.
Use online tools : There are online tools available (e.g., citation generators) that can help with formatting and organizing references.

Purpose of References in Research

References in research serve several purposes:

To give credit to the original authors or sources of information used in the research. It is important to acknowledge the work of others and avoid plagiarism.
To provide evidence for the claims made in the research. References can support the arguments, hypotheses, or conclusions presented in the research by citing relevant studies, data, or theories.
To allow readers to find and verify the sources used in the research. References provide the necessary information for readers to locate and access the sources cited in the research, which allows them to evaluate the quality and reliability of the information presented.
To situate the research within the broader context of the field. References can show how the research builds on or contributes to the existing body of knowledge, and can help readers to identify gaps in the literature that the research seeks to address.

Importance of References in Research

References play an important role in research for several reasons:

Credibility : By citing authoritative sources, references lend credibility to the research and its claims. They provide evidence that the research is based on a sound foundation of knowledge and has been carefully researched.
Avoidance of Plagiarism : References help researchers avoid plagiarism by giving credit to the original authors or sources of information. This is important for ethical reasons and also to avoid legal repercussions.
Reproducibility : References allow others to reproduce the research by providing detailed information on the sources used. This is important for verification of the research and for others to build on the work.
Context : References provide context for the research by situating it within the broader body of knowledge in the field. They help researchers to understand where their work fits in and how it builds on or contributes to existing knowledge.
Evaluation : References provide a means for others to evaluate the research by allowing them to assess the quality and reliability of the sources used.

Advantages of References in Research

There are several advantages of including references in research:

Acknowledgment of Sources: Including references gives credit to the authors or sources of information used in the research. This is important to acknowledge the original work and avoid plagiarism.
Evidence and Support : References can provide evidence to support the arguments, hypotheses, or conclusions presented in the research. This can add credibility and strength to the research.
Reproducibility : References provide the necessary information for others to reproduce the research. This is important for the verification of the research and for others to build on the work.
Context : References can help to situate the research within the broader body of knowledge in the field. This helps researchers to understand where their work fits in and how it builds on or contributes to existing knowledge.
Evaluation : Including references allows others to evaluate the research by providing a means to assess the quality and reliability of the sources used.
Ongoing Conversation: References allow researchers to engage in ongoing conversations and debates within their fields. They can show how the research builds on or contributes to the existing body of knowledge.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

Data Collection – Methods Types and Examples

Delimitations in Research – Types, Examples and...

Research Process – Steps, Examples and Tips

Research Design – Types, Methods and Examples

Institutional Review Board – Application Sample...

Evaluating Research – Process, Examples and...

Search This Site All UCSD Sites Faculty/Staff Search Term
Contact & Directions
Climate Statement
Cognitive Behavioral Neuroscience
Cognitive Psychology
Developmental Psychology
Social Psychology
Adjunct Faculty
Non-Senate Instructors
Researchers
Psychology Grads
Affiliated Grads
New and Prospective Students
Honors Program
Experiential Learning
Programs & Events
Psi Chi / Psychology Club
Prospective PhD Students
Current PhD Students
Area Brown Bags
Colloquium Series
Anderson Distinguished Lecture Series
Speaker Videos
Undergraduate Program
Academic and Writing Resources

Writing Research Papers

What Types of References Are Appropriate?

When writing a research paper, there are many different types of sources that you might consider citing. Which are appropriate? Which are less appropriate? Here we discuss the different types of sources that you may wish to use when working on a research paper.

Please note that the following represents a general set of recommended guidelines that is not specific to any class and does not represent department policy. The types of allowable sources may vary by course and instructor.

Highly appropriate: peer-reviewed journal articles

In general, you should primarily cite peer-reviewed journal articles in your research papers. Peer-reviewed journal articles are research papers that have been accepted for publication after having undergone a rigorous editorial review process. During that review process, the article was carefully evaluated by at least one journal editor and a group of reviewers (usually scientists that are experts in the field or topic under investigation). Often the article underwent revisions before it was judged to be satisfactory for publication.

Most articles submitted to high quality journals are not accepted for publication. As such, research that is successfully published in a respected peer-reviewed journal is generally regarded as higher quality than research that is not published or is published elsewhere, such as in a book, magazine, or on a website. However, just because a study was published in a peer-reviewed journal does not mean that it is free from error or that its conclusions are correct. Accordingly, it is important to critically read and carefully evaluate all sources, including peer-reviewed journal articles.

Tips for finding and using peer-reviewed journal articles:

Many databases, such as PsycINFO, can be set to only search for peer-reviewed journal articles. Other search engines, such as Google Scholar, typically include both peer-reviewed and not peer-reviewed articles in search results, and thus should be used with greater caution.
Even though a peer-reviewed journal article is, by definition, a source that has been carefully vetted through an editorial process, it should still be critically evaluated by the reader.

Potentially appropriate: books, encyclopedias, and other scholarly works

Another potential source that you might use when writing a research paper is a book, encyclopedia, or an official online source (such as demographic data drawn from a government website). When relying on such sources, it is important to carefully consider its accuracy and trustworthiness. For example, books vary in quality; most have not undergone any form of review process other than basic copyediting. In many cases, a book’s content is little more than the author’s informed or uninformed opinion.

However, there are books that have been edited prior to publication, as is the case with many reputable encyclopedias; also, many books from academic publishers are comprised of multiple chapters, each written by one or more researchers, with the entire volume carefully reviewed by one or more editors. In those cases, the book has undergone a form of peer review, albeit often not as rigorous as that for a peer-reviewed journal article.

Tips for using books, encyclopedias, and other scholarly works:

When using books, encyclopedias, and other scholarly works (that is, works written or produced by researchers, official agencies, or corporations), it is important to very carefully evaluate the quality of that source.
If the source is an edited volume (in which case in the editor(s) will be listed on the cover), is published by a reputable source (such as Academic Press, MIT Press, and others), or is written by a major expert in the field (such as a researcher with a track record of peer-reviewed journal articles on the subject), then it is more likely to be trustworthy.
For online encyclopedias such as Wikipedia, an instructor may or may not consider that an acceptable source (by default, don’t assume that a non-peer reviewed source will be considered acceptable). It is best to ask the instructor for clarification. 1

Usually inappropriate: magazines, blogs, and websites

Most research papers can be written using only peer-reviewed journal articles as sources. However, for many topics it is possible to find a plethora of sources that have not been peer-reviewed but also discuss the topic. These may include articles in popular magazines or postings in blogs, forums, and other websites. In general, although these sources may be well-written and easy to understand, their scientific value is often not as high as that of peer-reviewed articles. Exceptions include some magazine and newspaper articles that might be cited in a research paper to make a point about public awareness of a given topic, to illustrate beliefs and attitudes about a given topic among journalists, or to refer to a news event that is relevant to a given topic.

Tips for using magazines, blogs, and websites:

Avoid such references if possible. You should primarily focus on peer-reviewed journal articles as sources for your research paper. High quality research papers typically do not rely on non-academic and not peer-reviewed sources.
Refer to non-academic, not peer-reviewed sources sparingly, and if you do, be sure to carefully evaluate the accuracy and scientific merit of the source.

Downloadable Resources

How to Write APA Style Research Papers (a comprehensive guide) [ PDF ]
Tips for Writing APA Style Research Papers (a brief summary) [ PDF ]

Further Resources

How-To Videos

Writing Research Paper Videos

Databases and Search Engines (may require connection to UCSD network)

Google Scholar
PubMed (NIH/NLM)
Web of Science

UCSD Resources on Finding and Evaluating Sources

UCSD Library Databases A-Z
UCSD Library Psychology Research Guide: Start Page
UCSD Library Psychology Research Guide : Finding Articles
UCSD Library Psychology Research Guide : Evaluating Sources

External Resources

Critically Reading Journal Articles from PSU/ Colby College
How to Seriously Read a Journal Article from Science Magazine
How to Read Journal Articles from Harvard University
How to Read a Scientific Paper Infographic from Elsevier Publishing
Tips for searching PsycINFO from UC Berkeley Library
Tips for using PsycINFO effectively from the APA Student Science Council

1 Wikipedia articles vary in quality; the site has a peer review system and the very best articles ( Featured Articles ), which go through a multi-stage review process, rival those in traditional encyclopedias and are considered the highest quality articles on the site.

Prepared by s. c. pan for ucsd psychology, graphic adapted from t-x-generic-apply.svg , a public domain creation by the tango desktop project..

Research Paper Structure
Formatting Research Papers
Using Databases and Finding References
Evaluating References and Taking Notes
Citing References
Writing a Literature Review
Writing Process and Revising
Improving Scientific Writing
Academic Integrity and Avoiding Plagiarism
Writing Research Papers Videos

How to Format a Citation

Examples of apa, mla, and chicago manual of style, citation styles: american psychological association (apa), citation styles: chicago, citation styles: modern language association (mla), example: direct quote cited in a book, example: reference within a journal article.

EndNote This link opens in a new window
Compare Citation Management Software

There are two basic approaches to citation:

In-text citations + a list of references at the end of the paper
Endnotes or footnotes +/- a bibliography at the end of the paper

Scholars writing in the sciences and social sciences typically use in-text citations, while humanities scholars utilize endnotes/footnotes.

While the two basic approaches to citations are simple, there are many different citation styles.

What is a citation style?

The way that citations appear (format) depends on the citation style, which is a set of established rules and conventions for documenting sources.

Citation styles can be defined by an association, such as the Modern Language Association (MLA), publisher, such as the University of Chicago Press, or journal, such as The New England Journal of Medicine .

What citation style should I use?

The citation style that you use depends on the discipline in which you are writing, and where, or by whom, your work will be published or read.

When in doubt, ask your professor if there is a particular style that he/she would like you to use.

Where can I find more information on how to cite a specific type of source in a particular style?

The library has style manuals in print and online for several commonly used styles such as American Psychological Association (APA), Modern Language Association (MLA) and Chicago. In addition, there are several excellent citation style guides on the web. (See below)

For examples of APA and MLA and Chicago Manual of Style, visit Purdue's OWL (Online Writing Lab) site.

Frank, H. (2011). Wolves, Dogs, Rearing and Reinforcement: Complex Interactions Underlying Species Differences in Training and Problem-Solving Performance. Behavior Genetics , 41 (6), 830-839.

Publication Manual of the American Psychological Association Print manual for the APA style, available in the Sciences and Rockefeller libraries.
Purdue University Online Writing Lab Well-organized, easy-to-follow guide, with numerous examples.
APA Style American Psychological Association website for the APA Style. Provides tutorials, answers to frequently asked questions, and more.

Frank, H. 2011. "Wolves, Dogs, Rearing and Reinforcement: Complex Interactions Underlying Species Differences in Training and Problem-Solving Performance." Behavior Genetics 41 (6):830-839.

The Chicago Manual of Style Older (15th edition) print manual, available at the Sciences, Rockefeller and Orwig libraries.
The Chicago Manual of Style Online Current (16th) edition of the Chicago Manual of Style, and answers to frequently asked questions. Off-campus use requires Brown username and password.

Frank, H. "Wolves, Dogs, Rearing and Reinforcement: Complex Interactions Underlying Species Differences in Training and Problem-Solving Performance." Behavior Genetics 41.6 (2011): 830-39. Print.

MLA Style Manual and Guide to Scholarly Publishing Print manual for the MLA style. Available in the Rockefeller Library.
MLA Handbook for Writers of Research Papers Print handbook for the MLA. Available in the Rockefeller Library.

Source: Gabriel, R. A. (2001). Gods of Our Fathers: The Memory of Egypt in Judaism & Christianity . Westport, CT, USA: Greenwood Press.

Source: Bradt, J., Potvin, N., Kesslick, A., Shim, M., Radl, D., Schriver, E., … Komarnicky-Kocher, L. T. (2015). The impact of music therapy versus music medicine on psychological outcomes and pain in cancer patients: a mixed methods study. Supportive Care in Cancer : Official Journal of the Multinational Association of Supportive Care in Cancer , 23 (5), 1261–71.

<< Previous: Home
Next: Plagiarism >>
Last Updated: Oct 24, 2023 9:54 AM
URL: https://libguides.brown.edu/citations

Library Intranet

Search Menu
Browse content in Arts and Humanities
Browse content in Archaeology
Anglo-Saxon and Medieval Archaeology
Archaeological Methodology and Techniques
Archaeology by Region
Archaeology of Religion
Archaeology of Trade and Exchange
Biblical Archaeology
Contemporary and Public Archaeology
Environmental Archaeology
Historical Archaeology
History and Theory of Archaeology
Industrial Archaeology
Landscape Archaeology
Mortuary Archaeology
Prehistoric Archaeology
Underwater Archaeology
Urban Archaeology
Zooarchaeology
Browse content in Architecture
Architectural Structure and Design
History of Architecture
Residential and Domestic Buildings
Theory of Architecture
Browse content in Art
Art Subjects and Themes
History of Art
Industrial and Commercial Art
Theory of Art
Biographical Studies
Byzantine Studies
Browse content in Classical Studies
Classical Literature
Classical Reception
Classical History
Classical Philosophy
Classical Mythology
Classical Art and Architecture
Classical Oratory and Rhetoric
Greek and Roman Archaeology
Greek and Roman Epigraphy
Greek and Roman Law
Greek and Roman Papyrology
Late Antiquity
Religion in the Ancient World
Digital Humanities
Browse content in History
Colonialism and Imperialism
Diplomatic History
Environmental History
Genealogy, Heraldry, Names, and Honours
Genocide and Ethnic Cleansing
Historical Geography
History by Period
History of Agriculture
History of Education
History of Emotions
History of Gender and Sexuality
Industrial History
Intellectual History
International History
Labour History
Legal and Constitutional History
Local and Family History
Maritime History
Military History
National Liberation and Post-Colonialism
Oral History
Political History
Public History
Regional and National History
Revolutions and Rebellions
Slavery and Abolition of Slavery
Social and Cultural History
Theory, Methods, and Historiography
Urban History
World History
Browse content in Language Teaching and Learning
Language Learning (Specific Skills)
Language Teaching Theory and Methods
Browse content in Linguistics
Applied Linguistics
Cognitive Linguistics
Computational Linguistics
Forensic Linguistics
Grammar, Syntax and Morphology
Historical and Diachronic Linguistics
History of English
Language Variation
Language Families
Language Acquisition
Language Evolution
Language Reference
Lexicography
Linguistic Theories
Linguistic Typology
Linguistic Anthropology
Phonetics and Phonology
Psycholinguistics
Sociolinguistics
Translation and Interpretation
Writing Systems
Browse content in Literature
Bibliography
Children's Literature Studies
Literary Studies (Modernism)
Literary Studies (Asian)
Literary Studies (European)
Literary Studies (Eco-criticism)
Literary Studies (Romanticism)
Literary Studies (American)
Literary Studies - World
Literary Studies (1500 to 1800)
Literary Studies (19th Century)
Literary Studies (20th Century onwards)
Literary Studies (African American Literature)
Literary Studies (British and Irish)
Literary Studies (Early and Medieval)
Literary Studies (Fiction, Novelists, and Prose Writers)
Literary Studies (Gender Studies)
Literary Studies (Graphic Novels)
Literary Studies (History of the Book)
Literary Studies (Plays and Playwrights)
Literary Studies (Poetry and Poets)
Literary Studies (Postcolonial Literature)
Literary Studies (Queer Studies)
Literary Studies (Science Fiction)
Literary Studies (Travel Literature)
Literary Studies (War Literature)
Literary Studies (Women's Writing)
Literary Theory and Cultural Studies
Mythology and Folklore
Shakespeare Studies and Criticism
Browse content in Media Studies
Browse content in Music
Applied Music
Dance and Music
Ethics in Music
Ethnomusicology
Gender and Sexuality in Music
Medicine and Music
Music Cultures
Music and Culture
Music and Religion
Music and Media
Music Education and Pedagogy
Music Theory and Analysis
Musical Scores, Lyrics, and Libretti
Musical Structures, Styles, and Techniques
Musicology and Music History
Performance Practice and Studies
Race and Ethnicity in Music
Sound Studies
Browse content in Performing Arts
Browse content in Philosophy
Aesthetics and Philosophy of Art
Epistemology
Feminist Philosophy
History of Western Philosophy
Metaphysics
Moral Philosophy
Non-Western Philosophy
Philosophy of Action
Philosophy of Law
Philosophy of Religion
Philosophy of Science
Philosophy of Language
Philosophy of Mind
Philosophy of Perception
Philosophy of Mathematics and Logic
Practical Ethics
Social and Political Philosophy
Browse content in Religion
Biblical Studies
Christianity
East Asian Religions
History of Religion
Judaism and Jewish Studies
Qumran Studies
Religion and Education
Religion and Health
Religion and Politics
Religion and Science
Religion and Law
Religion and Art, Literature, and Music
Religious Studies
Browse content in Society and Culture
Cookery, Food, and Drink
Cultural Studies
Customs and Traditions
Ethical Issues and Debates
Hobbies, Games, Arts and Crafts
Lifestyle, Home, and Garden
Natural world, Country Life, and Pets
Popular Beliefs and Controversial Knowledge
Sports and Outdoor Recreation
Technology and Society
Travel and Holiday
Visual Culture
Browse content in Law
Arbitration
Browse content in Company and Commercial Law
Commercial Law
Company Law
Browse content in Comparative Law
Systems of Law
Competition Law
Browse content in Constitutional and Administrative Law
Government Powers
Judicial Review
Local Government Law
Military and Defence Law
Parliamentary and Legislative Practice
Construction Law
Contract Law
Browse content in Criminal Law
Criminal Procedure
Criminal Evidence Law
Sentencing and Punishment
Employment and Labour Law
Environment and Energy Law
Browse content in Financial Law
Banking Law
Insolvency Law
History of Law
Human Rights and Immigration
Intellectual Property Law
Browse content in International Law
Private International Law and Conflict of Laws
Public International Law
IT and Communications Law
Jurisprudence and Philosophy of Law
Law and Society
Law and Politics
Browse content in Legal System and Practice
Courts and Procedure
Legal Skills and Practice
Primary Sources of Law
Regulation of Legal Profession
Medical and Healthcare Law
Browse content in Policing
Criminal Investigation and Detection
Police and Security Services
Police Procedure and Law
Police Regional Planning
Browse content in Property Law
Personal Property Law
Study and Revision
Terrorism and National Security Law
Browse content in Trusts Law
Wills and Probate or Succession
Browse content in Medicine and Health
Browse content in Allied Health Professions
Arts Therapies
Clinical Science
Dietetics and Nutrition
Occupational Therapy
Operating Department Practice
Physiotherapy
Radiography
Speech and Language Therapy
Browse content in Anaesthetics
General Anaesthesia
Neuroanaesthesia
Browse content in Clinical Medicine
Acute Medicine
Cardiovascular Medicine
Clinical Genetics
Clinical Pharmacology and Therapeutics
Dermatology
Endocrinology and Diabetes
Gastroenterology
Genito-urinary Medicine
Geriatric Medicine
Infectious Diseases
Medical Oncology
Medical Toxicology
Pain Medicine
Palliative Medicine
Rehabilitation Medicine
Respiratory Medicine and Pulmonology
Rheumatology
Sleep Medicine
Sports and Exercise Medicine
Clinical Neuroscience
Community Medical Services
Critical Care
Emergency Medicine
Forensic Medicine
Haematology
History of Medicine
Medical Ethics
Browse content in Medical Dentistry
Oral and Maxillofacial Surgery
Paediatric Dentistry
Restorative Dentistry and Orthodontics
Surgical Dentistry
Browse content in Medical Skills
Clinical Skills
Communication Skills
Nursing Skills
Surgical Skills
Medical Statistics and Methodology
Browse content in Neurology
Clinical Neurophysiology
Neuropathology
Nursing Studies
Browse content in Obstetrics and Gynaecology
Gynaecology
Occupational Medicine
Ophthalmology
Otolaryngology (ENT)
Browse content in Paediatrics
Neonatology
Browse content in Pathology
Chemical Pathology
Clinical Cytogenetics and Molecular Genetics
Histopathology
Medical Microbiology and Virology
Patient Education and Information
Browse content in Pharmacology
Psychopharmacology
Browse content in Popular Health
Caring for Others
Complementary and Alternative Medicine
Self-help and Personal Development
Browse content in Preclinical Medicine
Cell Biology
Molecular Biology and Genetics
Reproduction, Growth and Development
Primary Care
Professional Development in Medicine
Browse content in Psychiatry
Addiction Medicine
Child and Adolescent Psychiatry
Forensic Psychiatry
Learning Disabilities
Old Age Psychiatry
Psychotherapy
Browse content in Public Health and Epidemiology
Epidemiology
Public Health
Browse content in Radiology
Clinical Radiology
Interventional Radiology
Nuclear Medicine
Radiation Oncology
Reproductive Medicine
Browse content in Surgery
Cardiothoracic Surgery
Gastro-intestinal and Colorectal Surgery
General Surgery
Neurosurgery
Paediatric Surgery
Peri-operative Care
Plastic and Reconstructive Surgery
Surgical Oncology
Transplant Surgery
Trauma and Orthopaedic Surgery
Vascular Surgery
Browse content in Science and Mathematics
Browse content in Biological Sciences
Aquatic Biology
Biochemistry
Bioinformatics and Computational Biology
Developmental Biology
Ecology and Conservation
Evolutionary Biology
Genetics and Genomics
Microbiology
Molecular and Cell Biology
Natural History
Plant Sciences and Forestry
Research Methods in Life Sciences
Structural Biology
Systems Biology
Zoology and Animal Sciences
Browse content in Chemistry
Analytical Chemistry
Computational Chemistry
Crystallography
Environmental Chemistry
Industrial Chemistry
Inorganic Chemistry
Materials Chemistry
Medicinal Chemistry
Mineralogy and Gems
Organic Chemistry
Physical Chemistry
Polymer Chemistry
Study and Communication Skills in Chemistry
Theoretical Chemistry
Browse content in Computer Science
Artificial Intelligence
Computer Architecture and Logic Design
Game Studies
Human-Computer Interaction
Mathematical Theory of Computation
Programming Languages
Software Engineering
Systems Analysis and Design
Virtual Reality
Browse content in Computing
Business Applications
Computer Games
Computer Security
Computer Networking and Communications
Digital Lifestyle
Graphical and Digital Media Applications
Operating Systems
Browse content in Earth Sciences and Geography
Atmospheric Sciences
Environmental Geography
Geology and the Lithosphere
Maps and Map-making
Meteorology and Climatology
Oceanography and Hydrology
Palaeontology
Physical Geography and Topography
Regional Geography
Soil Science
Urban Geography
Browse content in Engineering and Technology
Agriculture and Farming
Biological Engineering
Civil Engineering, Surveying, and Building
Electronics and Communications Engineering
Energy Technology
Engineering (General)
Environmental Science, Engineering, and Technology
History of Engineering and Technology
Mechanical Engineering and Materials
Technology of Industrial Chemistry
Transport Technology and Trades
Browse content in Environmental Science
Applied Ecology (Environmental Science)
Conservation of the Environment (Environmental Science)
Environmental Sustainability
Environmentalist Thought and Ideology (Environmental Science)
Management of Land and Natural Resources (Environmental Science)
Natural Disasters (Environmental Science)
Nuclear Issues (Environmental Science)
Pollution and Threats to the Environment (Environmental Science)
Social Impact of Environmental Issues (Environmental Science)
History of Science and Technology
Browse content in Materials Science
Ceramics and Glasses
Composite Materials
Metals, Alloying, and Corrosion
Nanotechnology
Browse content in Mathematics
Applied Mathematics
Biomathematics and Statistics
History of Mathematics
Mathematical Education
Mathematical Finance
Mathematical Analysis
Numerical and Computational Mathematics
Probability and Statistics
Pure Mathematics
Browse content in Neuroscience
Cognition and Behavioural Neuroscience
Development of the Nervous System
Disorders of the Nervous System
History of Neuroscience
Invertebrate Neurobiology
Molecular and Cellular Systems
Neuroendocrinology and Autonomic Nervous System
Neuroscientific Techniques
Sensory and Motor Systems
Browse content in Physics
Astronomy and Astrophysics
Atomic, Molecular, and Optical Physics
Biological and Medical Physics
Classical Mechanics
Computational Physics
Condensed Matter Physics
Electromagnetism, Optics, and Acoustics
History of Physics
Mathematical and Statistical Physics
Measurement Science
Nuclear Physics
Particles and Fields
Plasma Physics
Quantum Physics
Relativity and Gravitation
Semiconductor and Mesoscopic Physics
Browse content in Psychology
Affective Sciences
Clinical Psychology
Cognitive Neuroscience
Cognitive Psychology
Criminal and Forensic Psychology
Developmental Psychology
Educational Psychology
Evolutionary Psychology
Health Psychology
History and Systems in Psychology
Music Psychology
Neuropsychology
Organizational Psychology
Psychological Assessment and Testing
Psychology of Human-Technology Interaction
Psychology Professional Development and Training
Research Methods in Psychology
Social Psychology
Browse content in Social Sciences
Browse content in Anthropology
Anthropology of Religion
Human Evolution
Medical Anthropology
Physical Anthropology
Regional Anthropology
Social and Cultural Anthropology
Theory and Practice of Anthropology
Browse content in Business and Management
Business History
Business Strategy
Business Ethics
Business and Government
Business and Technology
Business and the Environment
Comparative Management
Corporate Governance
Corporate Social Responsibility
Entrepreneurship
Health Management
Human Resource Management
Industrial and Employment Relations
Industry Studies
Information and Communication Technologies
International Business
Knowledge Management
Management and Management Techniques
Operations Management
Organizational Theory and Behaviour
Pensions and Pension Management
Public and Nonprofit Management
Strategic Management
Supply Chain Management
Browse content in Criminology and Criminal Justice
Criminal Justice
Criminology
Forms of Crime
International and Comparative Criminology
Youth Violence and Juvenile Justice
Development Studies
Browse content in Economics
Agricultural, Environmental, and Natural Resource Economics
Asian Economics
Behavioural Finance
Behavioural Economics and Neuroeconomics
Econometrics and Mathematical Economics
Economic Methodology
Economic Systems
Economic History
Economic Development and Growth
Financial Markets
Financial Institutions and Services
General Economics and Teaching
Health, Education, and Welfare
History of Economic Thought
International Economics
Labour and Demographic Economics
Law and Economics
Macroeconomics and Monetary Economics
Microeconomics
Public Economics
Urban, Rural, and Regional Economics
Welfare Economics
Browse content in Education
Adult Education and Continuous Learning
Care and Counselling of Students
Early Childhood and Elementary Education
Educational Equipment and Technology
Educational Strategies and Policy
Higher and Further Education
Organization and Management of Education
Philosophy and Theory of Education
Schools Studies
Secondary Education
Teaching of a Specific Subject
Teaching of Specific Groups and Special Educational Needs
Teaching Skills and Techniques
Browse content in Environment
Applied Ecology (Social Science)
Climate Change
Conservation of the Environment (Social Science)
Environmentalist Thought and Ideology (Social Science)
Natural Disasters (Environment)
Social Impact of Environmental Issues (Social Science)
Browse content in Human Geography
Cultural Geography
Economic Geography
Political Geography
Browse content in Interdisciplinary Studies
Communication Studies
Museums, Libraries, and Information Sciences
Browse content in Politics
African Politics
Asian Politics
Chinese Politics
Comparative Politics
Conflict Politics
Elections and Electoral Studies
Environmental Politics
European Union
Foreign Policy
Gender and Politics
Human Rights and Politics
Indian Politics
International Relations
International Organization (Politics)
International Political Economy
Irish Politics
Latin American Politics
Middle Eastern Politics
Political Theory
Political Methodology
Political Communication
Political Philosophy
Political Sociology
Political Behaviour
Political Economy
Political Institutions
Politics and Law
Public Administration
Public Policy
Quantitative Political Methodology
Regional Political Studies
Russian Politics
Security Studies
State and Local Government
UK Politics
US Politics
Browse content in Regional and Area Studies
African Studies
Asian Studies
East Asian Studies
Japanese Studies
Latin American Studies
Middle Eastern Studies
Native American Studies
Scottish Studies
Browse content in Research and Information
Research Methods
Browse content in Social Work
Addictions and Substance Misuse
Adoption and Fostering
Care of the Elderly
Child and Adolescent Social Work
Couple and Family Social Work
Developmental and Physical Disabilities Social Work
Direct Practice and Clinical Social Work
Emergency Services
Human Behaviour and the Social Environment
International and Global Issues in Social Work
Mental and Behavioural Health
Social Justice and Human Rights
Social Policy and Advocacy
Social Work and Crime and Justice
Social Work Macro Practice
Social Work Practice Settings
Social Work Research and Evidence-based Practice
Welfare and Benefit Systems
Browse content in Sociology
Childhood Studies
Community Development
Comparative and Historical Sociology
Economic Sociology
Gender and Sexuality
Gerontology and Ageing
Health, Illness, and Medicine
Marriage and the Family
Migration Studies
Occupations, Professions, and Work
Organizations
Population and Demography
Race and Ethnicity
Social Theory
Social Movements and Social Change
Social Research and Statistics
Social Stratification, Inequality, and Mobility
Sociology of Religion
Sociology of Education
Sport and Leisure
Urban and Rural Studies
Browse content in Warfare and Defence
Defence Strategy, Planning, and Research
Land Forces and Warfare
Military Administration
Military Life and Institutions
Naval Forces and Warfare
Other Warfare and Defence Issues
Peace Studies and Conflict Resolution
Weapons and Equipment

Referencing styles

Author-date citations (Harvard) Numbered notes Numbered reference citations (Vancouver) OSCOLA

Introduction

Source references are vital to academic works (both print and digital) and so it is essential that they are clear, complete, and consistently formatted. Online bibliographical material is hyperlinked to provide readers with instant access to relevant sources or additional information.

Reference styles vary greatly across disciplines. This section details the main reference styles supported by OUP (Harvard, Vancouver, and OSCOLA) and provides examples that you can follow. If you are in doubt, your OUP editorial contact will be able to advise you on the best citation system for your text.

Author-date citations (Harvard)

The author-date style is an efficient and clear method of providing citations to published sources, which appear in a reference list at the end of the chapter or book. No superscripts are used, which means that reordering of the text does not require renumbering of notes. Instead of superscript numbers, a parenthetical citation (consisting of author name and date of publication) appears in the text and leads the reader to a full entry in a reference list that appears at the end of the chapter or book.

The method works particularly well when most of your citations are to published books or journal articles. It works less well if you are citing a lot of unauthored material or untraditional sources. Unlike numbered notes, author-date citations cannot accommodate translations or commentary outside the main text, although it is possible to combine author-date citations (for bibliographic citations) with numbered notes (for explanatory text).

In-text citation

References are cited within the text by including the author’s last name and a date parenthetically. A page number can be added if needed. If the author’s name appears in the sentence containing the citation, you need only use the date. Complete bibliographical reference information is listed at the end of the chapter or text.

Up to two author names can be used in the in-text citation. When citing a work with three or more authors, use the first author’s last name plus ‘et al.’

If you cite multiple references by the same author that were published in the same year, distinguish between them by adding labels (e.g. ‘a’ and ‘b’) to the year, in both the citation and the reference list.

Structure of the reference list

The reference list appears at the end of the chapter or text in alphabetical order. The name of the first author is inverted. In science literature, initials are often used in place of author first names.

The bibliographic elements listed below are required for the most common types of reference citations. Additional elements are mentioned that may be optional or to be used in only certain instances (e.g. a page number or other locator that is required if you are quoting a precise part of a large work, but not if the reference is to the work as a whole). Consistency in application is important.

Do not use long dashes (“—") to substitute for the name of an author who is identified in the bibliography due to how that entry will be linked in digital versions. Because the entry may not appear immediately following the entry with the full name, repeat the name in full.

Examples of author-date references in British style

Authored book.

Required elements

Lastname, Firstname/initials. Year of Publication. Title of Work .

With optional elements

Lastname, Firstname/initials, and Firstname/initials Lastname. Year of Publication. Title of Work , 2nd ed. City of Publication: Publisher.

Chapter in an edited book

Lastname, Firstname/initials, Year of Publication. ‘Title of Chapter in an Edited Book’. In Title of Edited Volume , edited by Firstname Lastname.

Lastname, Firstname/initials, and Firstname/initials Lastname. Year of Publication. ‘Title of Chapter in an Edited Book’. In Title of Edited Volume , edited by Firstname Lastname, page number(s) [or alternative locator info]. 2nd ed. City of Publication: Publisher.

Journal article

Lastname, Firstname/initials,Year of Publication. ‘Title of Article’. Name of Journal vol. number: start page.

Lastname, Firstname/initials, and Firstname/initials Lastname. Year of Publication. ‘Title of Article’. Name of Journal vol. number (issue number) (Month or Season): start page–end page. doi: DOI [or stable URL].

Magazine article

Lastname, Firstname/initials, Year of Publication. ‘Title of Article’. Day and Month of Pub. doi: DOI [or stable URL].

Lastname, Firstname/initials, and Firstname/initials Lastname. Year of Publication. ‘Title of Article’. Name of Magazine , Day and Month of Pub. doi: DOI [or stable URL].

Required elements if a magazine article has no stated author

‘Title of Article’. Year of Publication. Name of Magazine , Month of Pub. doi: DOI [or stable URL].

Website or other source

Include as much of the following as possible in your bibliographic entry: author; title or description of the content; owner/publisher; month and/or day of publication, most recent revision (or, failing that, date accessed); and URL. The year of publication should be the second element in the entry.

Some flexibility is acceptable to accommodate the wide variety of content available, particularly online.

Website names are usually set in roman type, but the names of online magazines and books are italicized (like their print counterparts).

As you write ...

Example: author–date citation with a reference list and further reading —british style.

Psychoanalytic studies, along with other literary and cultural texts, not only contribute to the new discourse of the jungle but also reflect the imperialist history that brings West Europeans and Americans into contact with the geographic jungles of India, Africa, and other parts of the world (Rogers et al. 2010, 1). This colonial context needs to be sketched here as well in order to reveal how the birth of the jungle eventually produces new constructions of sexuality in the United States. Billops (1999a) notes that the word ‘jungle’ comes from the Hindi and Marathi word jangal, meaning ‘desert’, ‘waste’, ‘forest’; as well as from the Sanskrit jangala, meaning ‘dry’, ‘dry ground’, or ‘desert’. Its first appearance in English is in 1776, with its meaning already shifted towards what might be more recognizable today: ‘Land overgrown with underwood, long grass, or tangled vegetation; also, the luxuriant and often almost impenetrable growth of vegetation covering such a tract’ (Dreft and Smithers 1978, 87). Brought into English as a result of an imperialist presence in India, ‘jungle’ is intimately related to the larger rise of Western imperialism around the world, particularly in the nineteenth century (Billops 1999b). Western powers such as Britain and France went from controlling 35 per cent of the earth’s surface in 1800 to, by 1914, ‘a grand total of roughly 85 per cent of the earth as colonies, protectorates, dependencies, dominions, and commonwealths’ (Said 1993, ch.2, ‘Colonial impacts’).

Reference list

Billops, Camille. 1999a. ‘Indo-European Loan Words’. Annals of Linguistics 21 (4): pp. 38–44.

Billops, Camille. 1999b. ‘Indo-European Vowel Shift: Evidence and Interpretation’. Annals of Linguistics 21 (4): p. 45.

Dreft, Edward, and Susan Smithers. 1978. ‘Words Working’. International Journal of American Linguistics 62 (3): pp. 227–263. doi: 10.1111/j.1749-6632.1978.tb25475.x.

Rogers, Jason, Millicent Eng, and Rene Woo. 2010. ‘English-Based African Creoles’. In Spreading the People: Colonizing Languages in the Raj , edited by Jason Rogers, pp. 310–330. 2nd ed. London: Verso.

Said, Eleanor. 1993. The European Dream of Africa . New York: Random House.

Examples of author-date references in US style

Lastname, Firstname/initials, Year of Publication. Title of Work .

Lastname, Firstname/initials, and Firstname Lastname/initials. Year of Publication. Title of Work , 2nd ed. City of Publication: Publisher.

Lastname, Firstname/initials, Year of Publication. “Title of Chapter in an Edited Book.” In Title of Edited Volume , edited by Firstname Lastname.

Lastname, Firstname/initials, and Firstname Lastname/initials. Year of Publication. “Title of Chapter in an Edited Book.” In Title of Edited Volume , edited by Firstname Lastname, page number(s) [or alternative locator info]. 2nd ed. City of Publication: Publisher.

Lastname, Firstname/initials,Year of Publication. “Title of Article.” Name of Journal vol. number, start page.

Lastname, Firstname/initials, and Firstname Lastname/initials. Year of Publication. “Title of Article.” Name of Journal vol. number (issue number) (Month or Season Year): start page–end page. doi: DOI [or stable URL].

Lastname, Firstname/initials, Year of Publication. “Title of Article.” Name of Magazine , Month of Pub.

Lastname, Firstname/initials, and Firstname Lastname/initials. Year of Publication. “Title of Article.” Name of Magazine , Month and Day of Pub. doi: DOI [or stable URL].

Required elements If a magazine article has no stated author:

“Title of Article.” Year of Publication. Name of Magazine , Month of Pub.

“Title of Article.” Year of Publication. Name of Magazine , Month and Day of Pub, doi: DOI [or stable URL].

The names of websites are usually set in roman type, but the names of online magazines and books are italicized (like their print counterparts).

Reference list vs. bibliography

Note that a reference list in the author-date system can contain only items that are actually cited in the work. The reference list must contain all of those items. This differs from a bibliography in the numbered-note system, which can contain both cited items and items of interest that have not been specifically cited. If there are uncited works that you would like to draw to the reader’s attention, these can be placed after the references in a separate listed titled ‘Further reading’.

Example: author–date citation with a reference list and further reading—US style

Psychoanalytic studies, along with other literary and cultural texts, not only contribute to the new discourse of the jungle but also reflect the imperialist history that brings West Europeans and Americans into contact with the geographic jungles of India, Africa, and other parts of the world (Rogers et al. 2010, 1). This colonial context needs to be sketched here as well in order to reveal how the birth of the jungle eventually produces new constructions of sexuality in the United States. Billops (1999a) notes that the word “jungle” comes from the Hindi and Marathi word jangal, meaning “desert,” “waste,” “forest”; as well as from the Sanskrit jangala, meaning “dry,” “dry ground,” or “desert.” Its first appearance in English is in 1776, with its meaning already shifted toward what might be more recognizable today: “Land overgrown with underwood, long grass, or tangled vegetation; also, the luxuriant and often almost impenetrable growth of vegetation covering such a tract” (Dreft and Smithers 1978, 87). Brought into English as a result of an imperialist presence in India, “jungle” is intimately related to the larger rise of Western imperialism around the world, particularly in the nineteenth century (Billops 1999b). Western powers such as Britain and France went from controlling 35 percent of the earth’s surface in 1800 to, by 1914, “a grand total of roughly 85 percent of the earth as colonies, protectorates, dependencies, dominions, and commonwealths” (Said 1993, ch.2, “Colonial impacts”).

Billops, Camille. 1999a. “Indo-European Loan Words.” Annals of Linguistics 21 (4): pp. 38–44.

Billops, Camille. 1999b. “Indo-European Vowel Shift: Evidence and Interpretation.” Annals of Linguistics 21 (4): p. 45.

Dreft, Edward, and Susan Smithers. 1978. “Words Working.” International Journal of American Linguistics 62 (3): pp. 227–263. doi: 10.1111/j.1749-6632.1978.tb25475.x.

Rogers, Jason, Millicent Eng, and Rene Woo. 2010. “English-Based African Creoles.” In Spreading the People: Colonizing Languages in the Raj , edited by Jason Rogers, pp. 310–330. 2nd ed. London: Verso.

“Evolutionary Linguistics.” 2012. Wikipedia. Updated November 4. http://en.wikipedia.org/wiki/Evolutionary_linguistics.

Mfuti, Miriam. 2001. “Pidgin Town.” In The Oxford Handbook of Pidgins and Creoles , edited by Alain Smet, pp. 107–112. New York: Oxford University Press.

Rambow, John. 2007. “Will This Demon Fit in My Carry-On?” Bangalore Monkey blog. December 21. http://www.bangaloremonkey. com/2007/12/will-this-demon-fit-in-my-carry-on.html.

Numbered notes

Using numbered notes is a common method of citing sources, particularly in the humanities. Sequential superscript numbers appear in the text to direct the reader to bibliographic or explanatory information that appears in a note.

This is a flexible style that allows authors to combine bibliographic information with annotation, translation, or other commentary. Scholars who frequently cite unpublished material will find numbered notes more useful than author-date citations.

Endnotes or footnotes?

In print publishing, notes can be placed at the bottom of the page as footnotes or at the end of a chapter or book in a separate section as endnotes.

Footnotes are preferred in cases where the information in the note is important enough that readers need it to fully engage with the material. Please note that in a digital context, footnotes in the traditional sense are not possible. Depending on the format, footnotes can appear at the end of a section or chapter, or they may be viewed by clicking or hovering over the superscript numbers in the text to display individual footnotes.

Endnotes are a better choice in print if the material in the notes does not need immediate engagement by the reader. For digital publications where individual chapters may be made available to readers, the notes should appear with the chapter, rather than separately at the end of the work. This varies according to discipline, so please consult your OUP editorial contact if you are unsure.

The formatting of bibliographic information is identical for footnotes and endnotes.

Please use the following guidance:

Numbered notes appear sequentially in the text as superscripts, ideally at the end of a sentence, following the closing punctuation.
Use Arabic numerals.
Numbers should restart at 1 at the beginning of each chapter and run consecutively to the end of each chapter. Do not start renumbering within a chapter (e.g. per page or per double-page spread) or use asterisks, as this will cause confusion in a digital environment.
Do not number the notes continuously throughout a book, because a later change would necessitate extensive renumbering.

Note structure and format

Required bibliographic elements are given below for the most common types of reference citations, along with optional elements that if used, must be consistent.

Page numbers are useful locators when referencing in print publications.
Give page ranges using the fewest number of figures as possible (e.g. pp. 126–27, not pp. 126–127).
When referencing a digital publication, you may not have access to a print page number. Cite a specific locator (e.g. chapter titles and sub-headings). Do not use location numbers from a proprietary e-reader (e.g. Kindle location numbers).
Edition numbers are not required when citing a first edition but are necessary for subsequent editions.

Numbered notes in British style

Firstname Lastname, Title of Work (Year of Publication).

Firstname Lastname, Title of Work , 2nd ed. (City of Publication: Publisher, Year of Publication), page number(s) [or alternative locator info].

Michael Murray, Climate Change at the Poles (New York: Scribner, 2007), p. 9.
Darian Ibrahim and Carol Marche, Financing the Next Silicon Valley , 3rd ed. (San Francisco: Upbeat Press, 2010).

Edited book

Firstname Lastname, ed., Title of Work (Year of Publication).

Firstname Lastname, eds., Title of Work , 2nd ed. (City of Publication: Publisher, Year of Publication), page number(s) [or alternative locator info].

Anton Smirov, ed., Eastern Europe After the Iron Curtain (London: Chatto and Windus, 2012).

Firstname Lastname, ‘Title of Chapter in an Edited Volume’, in Title of Edited Volume , edited by Firstname Lastname (Year of Publication).

Firstname Lastname, ‘Title of Chapter in an Edited Volume’, in Title of Edited Volume , edited by Firstname Lastname (City of Publication: Publisher, Year of Publication), page number(s) [or alternative locator info].

Hanna Growiszc, ‘Far Right Ideologies in Czech Literature’, in Eastern Europe After the Iron Curtain , edited by Anton Smirov (London: Chatto and Windus, 2012), ch. 7.

Authored book with an editor or translator

Firstname Lastname, Title of Work , ed./trans. Firstname Lastname, (Year of Publication).

Firstname Lastname, Title of Work , ed./trans. Firstname Lastname, 2nd ed. (City of Publication: Publisher, Year of Publication), page number(s) [or alternative locator info].

Günter Grass, The Tin Drum , trans. Breon Mitchell (New York: Houghton Mifflin, 2009).

Aristotle, Nicomachean Ethics , ed. and trans. Terence Irwin (Indianapolis: Hackett Publishing, 1999).

Multi-volume work

References to multi-volume book citations can take a variety of forms, depending on whether an individual volume or the entire work is being cited, and the authorship of the work.

Citing one volume of a multi-volume work

Robert Caro, The Path to Power , vol. 1, The Years of Lyndon Johnson (New York: Knopf, 1982), p. 267.

Citing a multi-volume work as a whole

Robert Caro, The Years of Lyndon Johnson , 4 vols (New York: Knopf, 1982–2012).

Allison Wyste, ed. Indian and Tibetan Cooking , vol. 6, Cuisines of Asia, ed. Robert Trautmann (London: Brill Books, 2007).

Multi-volume work with series editor and individual author/editors

Whenever possible, include a DOI (preferred) or a stable URL for citations to journal articles. However, a URL or DOI is not sufficient to stand alone as a reference.

Firstname Lastname, ‘Title of Article’, Name of Journal vol. number, (Year): start page.

Firstname Lastname, ‘Title of Article’, Name of Journal vol. number, issue number (Month or Season Year): start page–end page, doi: DOI [or stable URL].

Barbara Eckstein, ‘The Body, the Word, and the State: J. M. Coetzee’s “Waiting for the Barbarians”’, Novel: A Forum on Fiction 22, no. 2 (Winter 1989): pp. 175–198, http://www.jstor.org/stable/1345802.

David Hyun-Su Kim, ‘The Brahmsian Hairpin’, 19th Century Music 36, no. 1 (Summer 2012): pp. 46–47, doi:10.1525/ncm.2012.36.1.046.

A DOI or URL can be included for articles that you consulted online. The citations for online-only magazines follow the same pattern as print-based magazines, with the addition of URLs. If an online journal or magazine has a stable home page that allows a user to search for articles by title or author, it is acceptable to include the URL for that page (rather than the longer, more specific URL).

‘Title of Article’, Name of Magazine , Month of Pub, Year.

Firstname Lastname, ‘Title of Article’, Name of Magazine , Month and Day of Pub, Year, doi: DOI [or stable URL].

Mary Rose Himler, ‘Religious Books as Best Sellers’, Publishers Weekly , 19 February 1927.

‘Amazon Best Books 2012 Revealed’, Publishers Weekly , 13 November 2012, http://www.publishersweekly.com/pw/by-topic/industry-news/publisher-news/article/54738-amazon-best-books-2012-revealed.html.

Fritz Allhoff, ‘The Paradox of Nonlethal Weapons’, Slate , 13 November 2012, http://www.slate.com.

Law citation styles vary widely depending on jurisdiction. The following examples are for citing law cases in a non-specialist academic context. If you are writing specialist legal content, see ‘Citing of Legal Materials’ for detailed citation information.

Case Number Name of Case [Year] Report VolNo-FirstPageNo

Case C-34/89 P Smith v EC Commission [1993] ECR I-454

Name of Case [Year] VolNo Report, PageNo

Ridge v Baldwin [1964] AC 40, 78

Name of Case , VolNo Reporter SeriesNo (Year)

Name of Case , VolNo Reporter SeriesNo (Name of Court Year)

Bowers v Hardwick 478 US 186 (1986).

Unpublished or informally published content

The titles of unpublished works are set in quotation marks rather than italics. In place of a publisher, location or institutional information can be given.

Troy Thibodeaux, ‘Modernism in Greenwich Village, 1908–1929’ (PhD dissertation, New York University, 1999), p. 59.

Mary Koo, ‘Prakriti and Purusha: Dualism in the Yoga of Patanjali’ (lecture, Theosophical Society, Chennai, India, 17 May 2008).

To cite a website or other source that does not fall within those covered here, include as much of the following as possible (in this order) in your citation: author; title or description of the content; owner/publisher; date of publication or most recent revision (or, failing that, date accessed); and URL. Some flexibility is acceptable to accommodate the wide variety of content available, especially online.

The names of websites are usually set in roman type but the names of online magazines and books are italicized (like their print counterparts).

‘The Board of Directors of the Coca-Cola Company Authorizes New Share Repurchase Program’, Coca- Cola Company, 18 October 2012, http://www.coca-colacompany.com/media-center/press-releases/the-board-of-directors-of-the-coca-cola-company-authorizes-new-share-repurchase-program.
John Rambow, ‘Will This Demon Fit in My Carry-On?’, Bangalore Monkey blog, 21 December 2007, http://www.bangaloremonkey.com/2007/12/will-this-demon-fit-in-my-carry-on.html.
Wikimedia privacy policy, Wikimedia Foundation, accessed 26 November 2010, http://wikimediafoundation.org/wiki/ Privacy policy.

Numbered notes in US style

Firstname Lastname, Title of Work , (Year of Publication).

Firstname Lastname, eds., Title of Work , (Year of Publication).

Hanna Growiszc, “Far Right Ideologies in Czech Literature,” in Eastern Europe After the Iron Curtain , edited by Anton Smirov (London: Chatto and Windus, 2012), ch. 7.
Aristotle, Nicomachean Ethics , ed. and trans. Terence Irwin (Indianapolis: Hackett Publishing, 1999).

Multi-volume book citations can take a variety of forms, depending on whether an individual volume or the work as a whole is being cited, and on how the multi-volume work was authored or edited.

Robert Caro, The Years of Lyndon Johnson , 4 vols. (New York: Knopf, 1982–2012).
Allison Wyste, Indian and Tibetan Cooking , vol. 6, Cuisines of Asia, ed. Robert Trautmann (London: Brill Books, 2007).

Firstname Lastname, “Title of Chapter in an Edited Volume,” in Title of Edited Volume , edited by Firstname Lastname (Year of Publication).

Firstname Lastname, “Title of Chapter in an Edited Volume,” in Title of Edited Volume , edited by Firstname Lastname (City of Publication: Publisher, Year of Publication), page number(s) [or alternative locator info].

Firstname Lastname, “Title of Article,” Name of Journal vol. number, (Year): start page.

Firstname Lastname, “Title of Article,” Name of Journal vol. number, issue number (Month or Season Year): start page–end page, doi: DOI [or stable URL].

Barbara Eckstein, “The Body, the Word, and the State: J. M. Coetzee’s ‘Waiting for the Barbarians,’” Novel: A Forum on Fiction 22, no. 2 (Winter 1989): pp. 175–198, http://www.jstor.org/stable/1345802.
David Hyun-Su Kim, “The Brahmsian Hairpin,” 19th Century Music 36, no. 1 (Summer 2012): pp. 46–47, doi:10.1525/ncm.2012.36.1.046.

A DOI or URL can be included for articles that you consulted online. Online-only magazines follow the same pattern as print-based magazines, with the addition of URLs. If an online journal or magazine has a stable home page that allows a user to search for articles by title or author, it is acceptable to cite that page rather than a longer, more specific URL.

“Title of Article,” Name of Magazine , Month of Pub, Year.

Firstname Lastname, “Title of Article,” Name of Magazine, Month and Day of Pub, Year, doi: DOI [or stable URL].

Mary Rose Himler, “Religious Books as Best Sellers,” Publishers Weekly , February 19, 1927.
“Amazon Best Books 2012 Revealed,” Publishers Weekly , November 13, 2012, http://www.publishersweekly.com/pw/by-topic/industry-news/publisher-news/article/54738-amazon-best-books-2012-revealed.html.
Fritz Allhoff, “The Paradox of Nonlethal Weapons,” Slate , November 13, 2012, http://www.slate.com.

Law - case law

Law citation styles can vary widely depending on jurisdiction. These examples are for citing legal case law in a non-specialist academic context. If you are writing specialist legal content, see ‘Citing of legal materials’ for detailed information on law citation.

Name of Case [Year] VolNo Report PageNo

Ridge v. Baldwin [1964] AC 40, 78

Name of Case , Vol No. Reporter Series No. (Year)

Bowers v Hardwick , 478 U.S. 186 (1986)

Name of Case , Vol No. Reporter Series No. (Name of Court Year)

Bowers v. Hardwick 478 U.S. 186 (1986)

The titles of unpublished works are set in quotation marks rather than italics. Since there is no publisher, location or institutional information can be cited.

Troy Thibodeaux, “Modernism in Greenwich Village, 1908–1929” (PhD dissertation, New York University, 1999), p. 59.
Mary Koo, “Prakriti and Purusha: Dualism in the Yoga of Patanjali’ (lecture, Theosophical Society, Chennai, India, May 17, 2008).

If you need to cite a website or other source that does not fall within those covered here, include as much of the following as possible (in this order): author; title or description of the content; owner/publisher; date of publication or most recent revision (or, failing that, date accessed); and URL. Some flexibility is acceptable to accommodate the wide variety of content available, especially online.

“The Board of Directors of the Coca-Cola Company Authorizes New Share Repurchase Program,” Coca-Cola Company, October 18, 2012, http://www.coca-colacompany.com/media-center/press-releases/the-board-of-directors-of-the-coca-cola-company-authorizes-new-share-repurchase-program.
John Rambow, “Will This Demon Fit in My Carry-On?,” Bangalore Monkey blog, December 21, 2007, http://www.bangaloremonkey. com/2007/12/will-this-demon-fit-in-my-carry-on.html.
Wikimedia privacy policy, Wikimedia Foundation, accessed November 26, 2010, http://wikimediafoundation.org/wiki/ Privacy_policy.

Short citations

When a work is cited for the first time in a chapter, full bibliographic information should be given (for an alternative, see ‘Numbered notes in combination with a bibliography’). Subsequent citations should be shortened as in the following examples.

Legal short citations

Give the first mention of legal cases in full. Subsequent mentions within the same article or chapter can be shortened to the case name alone, given in italics (even if italics are not used in the original citation)

Case C–34/89 P Smith v EC Commission [1993] ECR I–454
P Smith v EC Commission.

Example: short citations in US style

See, for example, Alan Hess, Googie: Fifties Coffee Shop Architecture (San Francisco: Chronicle Books, 1985) and Noah Sheldon, Ranch House (New York: Harry S. Abrams, 2004).
Sheldon, Ranch House , p. 207.
Ashraf Salama, “Evolutionary Paradigms in Mosque Architecture,” Faith & Form 40, no. 1 (2007): pp. 16–17.
Salama, “Evolutionary Paradigms.”
Hess, Googie , p. 21.
Wikimedia privacy policy, para. 16.

Numbered notes in combination with a bibliography

It is possible to combine notes and bibliography so that all the notes, including the first reference, are short citations that lead the reader to a full citation in the bibliography. This system results in shorter notes and less work for the reader, since complete information is easily available in the alphabetical bibliography and need not be hunted for through all the chapter notes. This requires that all cited sources appear in a bibliography, which can also contain works that are not cited but are germane to the topic.

Structure of a bibliography entry

Bibliographies are structured similarly to notes, but there are some important differences. The first author name (and only the first) is inverted for alphabetization. Punctuation format also varies slightly between notes and bibliographic entries.

Do not use long dashes (e.g. “—") to substitute for an author’s name if it is repeated in the bibliography. Repeat the name in full because in a digital version, the shortened entry may not follow the complete one immediately.

Bibliography entries in British Style

Lastname, Firstname, Title of Work , (Year of Publication).

Lastname, Firstname, and Firstname Lastname. Title of Work , 2nd ed. (City of Publication: Publisher, Year of Publication).

Lastname, Firstname. ‘Title of Chapter in an Edited Book’. In Title of Edited Volume , edited by Firstname Lastname (Year of Publication).

Lastname, Firstname, and Firstname Lastname. ‘Title of Chapter in an Edited Book’. In Title of Edited Volume , edited by Firstname Lastname (City of Publication: Publisher, Year of Publication), page number(s) [or alternative locator info].

Lastname, Firstname,‘Title of Article’. Name of Journal vol. number, no. X (Year): start page.

Lastname, Firstname, and Firstname Lastname. ‘Title of Article’. Name of Journal vol. number, no. X (Month or Season Year): start page–end page. doi: DOI [or stable URL].

‘Title of Article’. Name of Magazine , Month Year of Pub.

Lastname, Firstname, and Firstname Lastname. ‘Title of Article’. Name of Magazine , Day Month Year of Pub, doi: DOI [or stable URL].

If you need to cite a website or other source that does not fall within those covered here, include as much of the following as possible (in this order): author; title or description of the content; owner/publisher; date of publication, most recent revision (or, failing that, date accessed); and URL. Some flexibility is acceptable to accommodate the wide variety of content available, especially online.

Sample bibliography

Growiszc, Hanna. ‘Far Right Ideologies in Czech Literature’. In Eastern Europe After the Iron Curtain , edited by Anton Smirov (London: Chatto and Windus, 2012), ch. 7.

Himler, Mary Rose. ‘Religious Books as Best Sellers’. Publishers Weekly , 19 February 1927.

Khan, Imran, and Richard Collins. ‘True Belief: Hindu Metanarratives in Bollywood’. Journal of Cinema Studies 7, no. 4 (2009): pp. 104–115. doi:10.1086/jcs113.3.752.

Murray, Michael. ‘The Antarctic Summer Lengthens’. Journal of Climate Studies 20, no. 9 (2011): p. 203.

Murray, Michael. Climate Change at the Poles (New York: Scribner, 2007).

Rambow, John. ‘Will This Demon Fit in My Carry-On?’ Bangalore Monkey blog. 21 December 2007. http://www.bangaloremonkey.com/2007/12/will-this-demon-fit-in-my-carry-on.html.

Bibliography entries in US style

Lastname, Firstname, “Title of Chapter in an Edited Book.” In Title of Edited Volume , edited by Firstname Lastname (Year of Publication).

Lastname, Firstname, and Firstname Lastname. “Title of Chapter in an Edited Book.” In Title of Edited Volume , edited by Firstname Lastname (City of Publication: Publisher, Year of Publication), page number(s) [or alternative locator info].

Lastname, Firstname,“Title of Article.” Name of Journal vol. number, no. X (Year): start page.

Lastname, Firstname, and Firstname Lastname. “Title of Article.” Name of Journal vol. number, no. X (Month or Season Year): start page–end page. doi: DOI [or stable URL].

“Title of Article.” Name of Magazine , Month of Pub, Year.

Lastname, Firstname, and Firstname Lastname. “Title of Article.” Name of Magazine , Month and Day of Pub, Year, doi: DOI [or stable URL].

Growiszc, Hanna. “Far Right Ideologies in Czech Literature.” In Eastern Europe After the Iron Curtain, edited by Anton Smirov (London: Chatto and Windus, 2012), ch. 7.

Himler, Mary Rose. “Religious Books as Best Sellers.” Publishers Weekly, February 19, 1927.

Khan, Imran, and Richard Collins. “True Belief: Hindu Metanarratives in Bollywood.” Journal of Cinema Studies 7, no. 4 (2009): pp. 104–115. doi:10.1086/jcs113.3.752.

Murray, Michael. “The Antarctic Summer Lengthens.” Journal of Climate Studies 20, no. 9 (2011): p. 203.

Rambow, John. “Will This Demon Fit in My Carry-On?” Bangalore Monkey blog. December 21, 2007. http://www.bangaloremonkey.com/2007/12/will-this-demon-fit-in-my-carry-on.html.

Numbered reference citations (Vancouver)

Numbered reference citations (also known as author–number or Vancouver references) are used in scientific and medical texts. In this system, each reference used is assigned a number. When that reference is cited in the text, its number appears, either in parentheses or brackets or as a superscript. All cited references appear in a numbered reference list at the end of the chapter or book.

An advantage of numbered references over the author–date style is that less space in the main text is required for in-text citations. The system also avoids ambiguity in the case of two works by the same author published the same year, an occasional issue in author–date citations. A disadvantage is that late addition or removal of references usually requires renumbering of both the reference list and the citations. Numbered reference citations cannot be used to provide commentary or other explanatory material to the text.

References are cited within the text by using a number in a superscript, in parentheses, or in square brackets. Although each of these variants is acceptable, only one can be used in a single text. The examples in this guide will enclose citation numbers in parentheses. Note that although citations are numbered in the order of their first appearance in the text, non-consecutive note numbers are possible, to allow references to be cited more than once. Citations can take the form of a range: for example (4–7) would cite references 4, 5, 6, and 7 simultaneously. If it is necessary to cite specific page numbers that are not present in the reference list, page numbers can be inserted into the citation: for example (4p6, 5pp1–11).

Please note the following:

Author first names are usually given as initials only, with no full stops (e.g. “AN” not “A.N.”) between initials. In the case of multiple authors, you can list up to six full names; for more than six authors, list the first three plus ‘et al’. All author names are inverted (i.e. last name, first name).
Names of journals can be abbreviated, as in the examples in this section, but must follow the standard abbreviations used by PubMed. Journal article titles are given without quotation marks and in sentence-style capitalization.
Do not use long dashes (e.g. “—") to substitute for the name of an author whose name is repeated in the bibliography. Repeat the name in full because linking in a digital publication may not immediately follow the entry with the full name.
Citations are numbered in the order in which they first appear in the text.

Required bibliographic elements are given below for the most common types of reference citations, along with optional elements (if used, be consistent). Other elements below are required if applicable (for example, you need a page number or other locator if you are quoting a precise part of a large work, but you can skip it if the reference is to the work as a whole).

Numbered reference citations in British style

Lastname FI, Title of Work , Year of Publication.

Lastname FI, Lastname FI, Lastname FI. Title of Work , 2nd ed. City of Publication: Publisher; Year of Publication: startpage–endpage [or alternative locator info].

Unauthored book (books published by committee, agency, or group)

Title of Work . Year of Publication.

Title of Work . 16th ed. City of Publication: Publisher; Year of Publication: startpage–endpage [or alternative locator info].

Lastname FI. Title of chapter in sentence case. In: Lastname FI, eds. Title of Work. Year of Publication.

Lastname FI, Lastname FI, Lastname FI. Title of chapter in sentence case. In: Lastname FI, Lastname FI, Lastname FI, eds. Title of Work . 2nd ed. City of Publication: Publisher; Year of Publication: startpage–endpage [or alternative locator info].

Lastname FI, Title of article in sentence case. Abbreviated Journal Title . Year of Publication; Volume No.

Lastname FI, Lastname FI, Lastname FI, et al. Title of article in sentence case. Abbreviated Journal Title . Year of Publication; Volume No. (Issue No.) (Supplement No.): startpage–endpage [or alternative locator info]. doi: DOI [or stable URL].

Magazine or newspaper article

Lastname FI. Title of article in sentence case. Magazine or Newspaper Title . Month and Year of Publication.

Lastname FI. Title of article in sentence case. Magazine or Newspaper Title . Day Month and Year of Publication: startpage–endpage. doi: DOI [or stable URL].

If the article has no stated author:

Title of article in sentence case. Magazine or Newspaper Title . Month and Year of Publication.

Title of article in sentence case. Magazine or Newspaper Title . Day Month and Year of Publication: startpage–endpage. doi: DOI [or stable URL].

Include of the following (in this order) in your bibliographic entry: author; title or description of the content; owner/publisher; date of publication or most recent revision (or, failing that, date accessed); and URL. Some flexibility is acceptable to accommodate the wide variety of content available online and in non-traditional formats. Follow the capitalization and italicization patterns of the examples here as much as possible.

If the nature of the material you are citing is not clear from the bibliographic information, you can provide a descriptor in brackets after the first element of the reference.

Example: Numbered reference citations and reference list—British style

Colorectal cancer (cRc) is one of the most common malignancies and the second leading cause of death from cancer in Europe and North America (1). While early stage cRc is associated with an excellent 5-year survival rate (90% for localized disease), approximately 20% of patients present with metastatic disease, and many patients diagnosed with stage ii or iii cancer will experience a recurrence and develop distant metastases (2). At present, established clinico-pathological criteria are used to estimate risks of recurrence in stage ii and iii disease, and this is routinely used in the selection of patients or adjuvant systemic therapy following surgical resection. The clinical outcome of patients who receive such adjuvant treatment can, however, vary widely, when additional molecular factors are taken into consideration. Identification of novel prognostic markers is, therefore, vital in improving the prognosis of this disease (3). One of the recently described substances important for angiogenesis is endoglin. Endoglin, also known as cD105, is a receptor for transforming growth factor-ÃƒÅ¸1 molecule, which binds preferentially to the activated endothelial cells that participate in tumour angiogenesis, with weak or negative expression in vascular endothelium of normal tissues. Endoglin is induced by hypoxia. Therefore, it is very useful for assessment of neo-angiogenesis of malignant neoplasms (4–6). Many reports indicate that endoglin assessed immunohistochemically in colorectal cancer correlates not only with tumour microvessel density, but also with survival. It has also been reported as a valuable parameter predicting patients having an increased risk of developing metastatic disease. Endoglin is expressed not only on cell surfaces since its soluble form (sol-end) can be detected also in blood (4–7). A few studies evaluated the clinical significance of elevated sol-end levels in colorectal cancer patients (7).

1. Ferlay J, Autier P, Boniol M, Heanue M, Colombet M, Boyle P. Estimates of the cancer incidence and mortality in Europe in 2006. Ann Oncol . 2007; 18: pp. 581–592.

2. Meyerhardt JA, Mayer RJ. Systemic therapy for colorectal cancer. In: Boniol M, Smith J, eds. Oncological Research Reviews . 16th ed. New York, NY: Dekker; 2005; pp. 476–487.

3. Allegra CJ, Paik S, Colangelo LH, et al. Prognostic value of thymidylate synthase, Ki-67, and p53 in patients with Dukes’ B and C colon cancer: a National Cancer Institute-National Surgical Adjuvant Breast and Bowel Project collaborative study. J Clin Oncol. 2003; 21: pp. 241–250.

4. Drug Topics Red Book . Montvale, NJ: Thomson Healthcare, 2009: p. 232.

5. FDA approves new treatment for advanced colorectal cancer. 2012. US Food and Drug Administration website. 27 September. http://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm321271.htm.

6. Stivarga [package insert]. Wayne, NJ: Bayer Healthcare Pharmaceuticals, 2012.

7. Mysliwiec P, Pawlak K, Kuklinski A, Kedra B. Combined perioperative plasma endoglin and vegF-a assessment in colorectal cancer patients. Folia Histochem Cytobiol . 2008; 46(2)(suppl. 1): pp. 487–49.

Numbered reference citations and reference list in US style

Lastname FI, Lastname FI, Lastname FI. Title of Work , 2nd ed. City of Publication: Publisher; Year of Publication.

Title of Work. Year of Publication.

Title of Work. 16th ed. City of Publication: Publisher; Year of Publication: startpage–endpage [or alternative locator info].

Lastname FI, Title of chapter in sentence case. In: Lastname FI, ed. Title of Work. Year of Publication.

Lastname FI, Lastname FI, Lastname FI. Title of chapter in sentence case. In: Lastname FI, Lastname FI, Lastname FI, eds. Title of Work. 2nd ed. City of Publication: Publisher; Year of Publication: startpage–endpage [or alternative locator info].

Lastname FI, Title of article in sentence case. Abbreviated Journal Title. Year of Publication; Volume No. (Issue No.)

Lastname FI, Lastname FI, Lastname FI, et al. Title of article in sentence case. Abbreviated Journal Title. Year of Publication; Volume No. (Issue No.)(SupplementNo): startpage–endpage [or alternative locator info]. doi: DOI [or stable URL].

Lastname FI. Title of article in sentence case. Magazine or Newspaper Title. Month, Day, and Year of Publication.

Lastname FI. Title of article in sentence case. Magazine or Newspaper Title. Month, Day, and Year of Publication: startpage–endpage. doi: DOI [or stable URL].

Title of article in sentence case. Magazine or Newspaper Title. Month, Day, and Year of Publication.

Title of article in sentence case. Magazine or Newspaper Title. Month, Day, and Year of Publication: startpage–endpage. doi: DOI [or stable URL].

Include as much of the following as possible in your bibliographic entry (in this order): author; title or description of the content; owner/publisher; date of publication or most recent revision, or, failing that, date accessed; and URL if available. Some flexibility is acceptable to accommodate the wide variety of content available online and in non-traditional formats. Follow the capitalization and italicization patterns of these examples.

Example: Numbered reference citations and reference list—US style

Colorectal cancer (cRc) is one of the most common malignancies and the second leading cause of death from cancer in Europe and North America (1). While early stage cRc is associated with an excellent 5-year survival rate (90% for localized disease), approximately 20% of patients present with metastatic disease, and many patients diagnosed with stage ii or iii cancer will experience a recurrence and develop distant metastases (2). At present, established clinico-pathological criteria are used to estimate risks of recurrence in stage ii and iii disease, and this is routinely used in the selection of patients or adjuvant systemic therapy following surgical resection. The clinical outcome of patients who receive such adjuvant treatment can, however, vary widely, when additional molecular factors are taken into consideration. Identification of novel prognostic markers is, therefore, vital in improving the prognosis of this disease (3). One of the recently described substances important for angiogenesis is endoglin. Endoglin, also known as cD105, is a receptor for transforming growth factor-ÃƒÅ¸1 molecule, which binds preferentially to the activated endothelial cells that participate in tumor angiogenesis, with weak or negative expression in vascular endothelium of normal tissues. Endoglin is induced by hypoxia. Therefore it is very useful for assessment of neo-angiogenesis of malignant neoplasms (4–6). Many reports indicate that endoglin assessed immunohistochemically in colorectal cancer correlates not only with tumor microvessel density, but also with survival. It has also been reported as a valuable parameter predicting patients having an increased risk of developing metastatic disease. Endoglin is expressed not only on cell surfaces, since its soluble form (sol-end) can be detected also in blood (4–7). A few studies evaluated the clinical significance of elevated sol-end levels in colorectal cancer patients (7).

1. Ferlay J, Autier P, Boniol M, Heanue M, Colombet M, Boyle P. Estimates of the cancer incidence and mortality in Europe in 2006. Ann Oncol. 2007; 18: pp. 581–592.

2. Meyerhardt JA, Mayer RJ. Systemic therapy for colorectal cancer. In: Boniol M, Smith J, eds. Oncological Research Reviews. 16th ed. New York, NY: Dekker; 2005; pp. 476–487.

3. Allegra CJ, Paik S, Colangelo LH, et al. Prognostic value of thymidylate synthase, Ki-67, and p. 53 in patients with Dukes’ B and C colon cancer: a National Cancer Institute-National Surgical Adjuvant Breast and Bowel Project collaborative study. J Clin Oncol. 2003; 21: pp. 241–250.

4. Drug Topics Red Book. Montvale, NJ: Thomson Healthcare, 2009: p. 232.

5. FDA approves new treatment for advanced colorectal cancer. US Food and Drug Administration website. September 27, 2012. http://www.fda. gov/NewsEvents/Newsroom/PressAnnouncements/ucm321271.htm.

For legal works, we recommend that you follow The Oxford University Standard for Citation of Legal Authorities (OSCOLA). The fourth edition (published in 2012) covers International Law. The full set of guidance can be found at https://www.law.ox.ac.uk/sites/default/files/migrated/oscola_4th_edn_hart_2012.pdf

Information on how to apply OSCOLA style in EndNote, Latex, Refworks and Zotero can be found at https://www.law.ox.ac.uk/research-subject-groups/publications/oscola-styles-endnote-latek-refworks-and-zotero

About Oxford Academic
Publish journals with us
University press partners
What we publish
New features
Open access
Institutional account management
Rights and permissions
Get help with access
Accessibility
Advertising
Media enquiries
Oxford University Press
Oxford Languages
University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

Copyright © 2024 Oxford University Press
Cookie settings
Cookie policy
Privacy policy
Legal notice

This Feature Is Available To Subscribers Only

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

APA Style 6th Edition: Citing Your Sources

Basics of APA Formatting
In Text Quick View
Block Quotes

The Generic Reference

Books & eBooks
Thesis/Dissertation
Conference Presentations
Course Documents
Social Media
Government Documents
Academic Integrity and Plagiarism
Additional Resources
Sample Reference Page

References appear at the end of your document and follow a Who, When, What and Where format. Only include sources you cited within your research document (*exception for personal communications which are excluded from the reference list).

Who: Identify the creator of the source. Often this is: a single author; multiple authors; an organization or corporation; editor/s; or the director and producer. On some occasions when authorship cannot be attributed then you revert to the title entry formatting.

When: Identify when the source was published. For most source types, simply provide a year of publication. Exceptions as follows: year followed by month for papers and posters presented at conferences; year followed by month and date for blogs, social media, newspaper and magazine publications. When there is no publication date (common for web documents and other content) use the abbreviation n.d. for "no date."

What: Title of the source. This is the title identified for the individual source, rather than where the source is published. For articles, you want to identify the title of the article rather than the Journal name. The same can be said for the title of a newspaper article vs. the newspaper name or the title of a webpage vs. the website name. (ex. "Mission statement" is a page title on the APA website).

Where: Once you identify the Who, When, and What, the rest of the information provided to complete the reference falls under "Where." This portion of the reference sees variations in information included based on source type. For books, Where includes the city and state of publication. For journal articles, it includes the journal name, volume & issue number, page range and DOI or journal URL.

Visit the individual Reference Type pages to see further formatting and examples for your source type.

<< Previous: Block Quotes
Next: Books & eBooks >>
Last Updated: Sep 22, 2022 11:20 AM
URL: https://libguides.usc.edu/APA-citation-style

Capstone Form and Style

References: common reference list examples, article (with doi).

Alvarez. E., & Tippins, S. (2019). Socialization agents that Puerto Rican college students use to make financial decisions. Journal of Social Change , 11 (1), 75–85. https://doi.org/10.5590/JOSC.2019.11.1.07

Laplante, J. P., & Nolin, C. (2014). Consultas and socially responsible investing in Guatemala: A case study examining Maya perspectives on the Indigenous right to free, prior, and informed consent. Society & Natural Resources , 27 , 231–248. https://doi.org/10.1080/08941920.2013.861554

Provide a DOI number if there is one. DOI stands for "digital object identifier," a number specific to the article that can help others locate the source. Use CrossRef.org to locate DOI information. This rule applies regardless of how the source was accessed (e.g., online, paper, etc.; see APA 7, Section 9.34).

In APA 7, format the DOI as a web address. Active hyperlinks for DOIs and URLs should be used for documents meant for screen reading. Present these hyperlinks in blue and underlined text (the default formatting in Microsoft Word), although plain black text is also acceptable. Be consistent in the formatting choice for DOIs and URLs throughout the reference list. (Note that this guidance has changed from APA 6 where all hyperlink formatting was removed and no active links were included. In APA 6, the URLs appeared in plain, black type and did not link out from the document.)

Also see our Quick Answer FAQ, "Can I use the DOI format provided by library databases?"

Jerrentrup, A., Mueller, T., Glowalla, U., Herder, M., Henrichs, N., Neubauer, A., & Schaefer, J. R. (2018). Teaching medicine with the help of “Dr. House.” PLoS ONE , 13 (3), Article e0193972. https://doi.org/10.1371/journal.pone.0193972

For journal articles that are assigned article numbers rather than page ranges, include the article number in place of the page range.

For more on citing electronic resources, see Electronic Sources References .

Article (Without DOI)

Found in a common academic research database or in print.

Casler , T. (2020). Improving the graduate nursing experience through support on a social media platform. MEDSURG Nursing , 29 (2), 83–87.

If an article does not have a DOI and you retrieved it from a common academic research database through the university library, there is no need to include any additional electronic retrieval information. The reference list entry looks like the entry for a print copy of the article. (This format differs from APA 6 guidelines that recommended including the URL of a journal's homepage when the DOI was not available.)

Note that APA 7 has additional guidance on reference list entries for articles found only in specific databases or archives such as Cochrane Database of Systematic Reviews, UpToDate, ProQuest Dissertations and Theses Global, and university archives. See APA 7, Section 9.30 for more information.

Found on an Open Access Website

Eaton, T. V., & Akers, M. D. (2007). Whistleblowing and good governance. CPA Journal , 77 (6), 66–71. http://archives.cpajournal.com/2007/607/essentials/p58.htm

Provide the direct web address/URL to a journal article found on the open web, often on an open access journal's website.

In APA 7, active hyperlinks for DOIs and URLs should be used for documents meant for screen reading. Present these hyperlinks in blue and underlined text (the default formatting in Microsoft Word), although plain black text is also acceptable. Be consistent in your formatting choice for DOIs and URLs throughout your reference list. (Note that this guidance has changed from APA 6 where all hyperlink formatting was removed and no active links were included. In APA 6, the URLs appeared in plain, black type and did not link out from the document.)

Weinstein, J. A. (2010). Social change (3rd ed.). Rowman & Littlefield.

If the book has an edition number, include it in parentheses after the title of the book. If the book does not list any edition information, do not include an edition number. The edition number is not italicized. (Note: In APA 6, the location of the publisher was included. This is no longer the case in APA 7; only the publisher name is provided.) Regarding publisher name, when a publisher is named after a person (as is the case with Lawrence Erlbaum or John Wiley), list only the surname (Erlbaum or Wiley). In addition, exclude “Publishers,” “Inc.,” and “Co.” from publisher names in reference entries.

American Nurses Association. (2010). Nursing: Scope and standards of practice (2nd ed.).

In APA 7, if the author and publisher are the same, only include the author in its regular place and omit the publisher. (Note that this is a change from APA 6, where the term “Author” was used for the publisher instead of repeating the name.)

Lencioni, P. (2012). The advantage: Why organizational health trumps everything else in business . Jossey-Bass. https://amzn.to/343XPSJ

As a change from APA 6 to APA 7, it is no longer necessary to include the ebook format in the title. However, if you listened to an audiobook and the content differs from the text version (e.g., abridged content) or your discussion highlights elements of the audiobook (e.g., narrator's performance), then note that it is an audiobook in the title element in brackets. For ebooks and online audiobooks, also include the DOI number (if available) or nondatabase URL but leave out the electronic retrieval element if the ebook was found in a common academic research database, as with journal articles. APA 7 allows for the shortening of long DOIs and URLs, as shown in this example. See APA 7, Section 9.36 for more information.

Chapter in an Edited Book

Poe, M. (2017). Reframing race in teaching writing across the curriculum. In F. Condon & V. A. Young (Eds.), Performing antiracist pedagogy in rhetoric, writing, and communication (pp. 87–105). University Press of Colorado.

Include the page numbers of the chapter in parentheses after the book title. The page range should not be italicized.

Christensen, L. (2001). For my people: Celebrating community through poetry. In B. Bigelow, B. Harvey, S. Karp, & L. Miller (Eds.), Rethinking our classrooms: Teaching for equity and justice (Vol. 2, pp. 16–17). Rethinking Schools.

Also include volume number and edition numbers in the parenthetical information after the book title where relevant.

Freud, S. (1961). The ego and the id. In J. Strachey (Ed.), The standard edition of the complete psychological works of Sigmund Freud (Vol. 19, pp. 3-66). Hogarth Press. (Original work published 1923)

When a text has been republished as part of an anthology collection, after the author’s name include the date of the version that was read. At the end of the entry, place the date of the original publication inside parenthesis along with the note “original work published.” For in-text citations of republished work, use both dates in the parenthetical citation, original date first with a slash separating the years, as in this example: Freud (1923/1961). For more information on reprinted or republished works, see APA 7, Sections 9.40-9.41.

Dissertations or Theses

Retrieved from a database

Nalumango, K. (2019). Perceptions about the asylum-seeking process in the United States after 9/11 (Publication No. 13879844) [Doctoral dissertation, Walden University]. ProQuest Dissertations and Theses.

Retrieved From an Institutional or Personal Website

Evener. J. (2018). Organizational learning in libraries at for-profit colleges and universities [Doctoral dissertation, Walden University]. ScholarWorks. https://scholarworks.waldenu.edu/cgi/viewcontent.cgi?article=6606&context=dissertations

Unpublished Dissertation or Thesis

Kirwan, J. G. (2005). An experimental study of the effects of small-group, face-to-face facilitated dialogues on the development of self-actualization levels: A movement towards fully functional persons [Unpublished doctoral dissertation]. Saybrook Graduate School and Research Center.

For further examples and information, see APA 7, Section 10.6.

Legal Material

For legal references, APA follows the recommendations of The Bluebook: A Uniform System of Citation , so if you have any questions beyond the examples provided in APA, seek out that resource as well.

Court Decisions

Reference format:

Name v. Name, Volume Reporter Page (Court Date). URL

Sample reference entry:

Brown v. Board of Education, 347 U.S. 483 (1954). https://www.oyez.org/cases/1940-1955/347us483

Sample citation:

In Brown v. Board of Education (1954), the Supreme Court ruled racial segregation in schools unconstitutional.

Note: Italicize the case name when it appears in the text of your paper rather than citing it—for example, “Cases such as Brown v. Board of Education and Parents Involved in Community Schools v. Seattle illustrate ...”

Name of Act, Title Source § Section Number (Year). URL

Sample reference entry for a federal statute:

Individuals With Disabilities Education Act, 20 U.S.C. § 1400 et seq. (2004). https://www.congress.gov/108/plaws/publ446/PLAW-108publ446.pdf

Sample reference entry for a state statute:

Minnesota Nurse Practice Act, Minn. Stat. §§ 148.171 et seq. (2019). https://www.revisor.mn.gov/statutes/cite/148.171

Sample citation: Minnesota nurses must maintain current registration in order to practice (Minnesota Nurse Practice Act, 2010).

Note: The § symbol stands for "section." Use §§ for sections (plural). To find this symbol in Microsoft Word, go to "Insert" and click on Symbol." Look in the Latin 1-Supplement subset.

Note: U.S.C. stands for "United States Code."

Note: The Latin abbreviation " et seq. " means "and what follows" and is used when the act includes the cited section and ones that follow.

Note: List the chapter first followed by the section or range of sections.

Unenacted Bills and Resolutions

(Those that did not pass and become law)

Title [if there is one], bill or resolution number, xxx Cong. (year). URL

Sample reference entry for Senate bill:

Anti-Phishing Act, S. 472, 109th Cong. (2005). https://www.congress.gov/bill/109th-congress/senate-bill/472

Sample reference entry for House of Representatives resolution:

Anti-Phishing Act, H.R. 1099, 109th Cong. (2005). https://www.congress.gov/bill/109th-congress/house-bill/1099

The Anti-Phishing Act (2005) proposed up to 5 years prison time for people running Internet scams.

These are the three legal areas you may be most apt to cite in your scholarly work. For more examples and explanation, see APA 7, Chapter 11.

Magazine Article

Clay, R. (2008, June). Science vs. ideology: Psychologists fight back about the misuse of research. Monitor on Psychology , 39 (6). https://www.apa.org/monitor/2008/06/ideology

Note that for citations, include only the year: Clay (2008). For magazine articles retrieved from a common academic research database, leave out the URL. For magazine articles from an online news website that is not an online version of a print magazine, follow the format for a webpage reference list entry.

Newspaper Article

Baker, A. (2014, May 7). Connecticut students show gains in national tests. New York Times . http://www.nytimes.com/2014/05/08/nyregion/national-assessment-of-educational-progress-results-in-Connecticut-and-New-Jersey.html

Include the full date in the format Year, Month Day. Do not include a retrieval date for periodical sources found on websites. Note that for citations, include only the year: Baker (2014). For newspaper articles retrieved from a common academic research database, leave out the URL. For newspaper articles from an online news website that is not an online version of a print newspaper, follow the format for a webpage reference list entry.

Technical and Research Reports

The general structure for a technical or research report is as follows:

Author, A. A. (Publication Year). Title of work . Publisher Name. DOI or URL

Edwards, C. (2015). Lighting levels for isolated intersections: Leading to safety improvements (Report No. MnDOT 2015-05). Center for Transportation Studies. http://www.cts.umn.edu/Publications/ResearchReports/reportdetail.html?id=2402

Technical and research reports by governmental agencies and other research institutions usually follow a different publication process than scholarly, peer-reviewed journals. However, they present original research and are often useful for research papers. Sometimes, researchers refer to these types of reports as gray literature , and white papers are a type of this literature. See APA 7, Section 10.4 for more information.

American Federation of Teachers. (n.d.). Community schools . http://www.aft.org/issues/schoolreform/commschools/index.cfm

If there is no specified author, then use the organization’s name as the author. In such a case, there is no need to repeat the organization's name after the title.

Vartan, S. (2018, January 30). Why vacations matter for your health . CNN. https://www.cnn.com/travel/article/why-vacations-matter/index.html

For webpages from news websites, include the site name after the title and before the URL. If the source is an online newspaper or magazine, follow the models in the previous sections of this page. In APA 7, active hyperlinks for DOIs and URLs should be used for documents meant for screen reading. Present these hyperlinks in blue and underlined text (the default formatting in Microsoft Word), although plain black text is also acceptable. Be consistent in your formatting choice for DOIs and URLs throughout your reference list. (Note that this guidance has changed from APA 6 where all hyperlink formatting was removed, and no active links were included. In APA 6, the URLs appeared in plain, black type and did not link out from the document.)

Previous Page: Home
Next Page: Common Military Reference List Examples
Office of Student Disability Services

Walden Resources

Departments.

Academic Residencies
Academic Skills
Career Planning and Development
Customer Care Team
Field Experience
Military Services
Student Success Advising
Writing Skills

Centers and Offices

Center for Social Change
Office of Academic Support and Instructional Services
Office of Degree Acceleration
Office of Research and Doctoral Services
Office of Student Affairs

Student Resources

Doctoral Writing Assessment
Form & Style Review
Quick Answers
ScholarWorks
SKIL Courses and Workshops
Walden Bookstore
Walden Catalog & Student Handbook
Student Safety/Title IX
Legal & Consumer Information
Website Terms and Conditions
Cookie Policy
Accessibility
Accreditation
State Authorization
Net Price Calculator
Contact Walden

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Types of Sources

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

We live in an age overflowing with sources of information. With so many information sources at our fingertips, knowing where to start, sorting through it all and finding what we want can be overwhelming! This handout provides answers to the following research-related questions: Where do I begin? Where should I look for information? What types of sources are available?

This section lists the types of sources most frequently used in academic research and describes the sort of information that each commonly offers.

Print Sources

Books and Textbooks: Odds are that at least one book has been written about virtually any research topic you can imagine (and if not, your research could represent the first steps toward a best-selling publication that addresses the gap!). Because of the time it takes to publish a book, books usually contain more dated information than will be found in journals and newspapers. However, because they are usually much longer, they can often cover topics in greater depth than more up-to-date sources.

Newspapers: Newspapers contain very up-to-date information by covering the latest events and trends. Newspapers publish both factual information and opinion-based articles. However, due to journalistic standards of objectivity, news reporting will not always take a “big picture” approach or contain information about larger trends, instead opting to focus mainly on the facts relevant to the specifics of the story. This is exacerbated by the rapid publication cycles most newspapers undergo: new editions must come out frequently, so long, in-depth investigations tend to be rarer than simple fact-reporting pieces.

Academic and Trade Journals: Academic and trade journals contain the most up-to-date information and research in industry, business, and academia. Journal articles come in several forms, including literature reviews that overview current and past research, articles on theories and history, and articles on specific processes or research. While a well-regarded journal represents the cutting-edge knowledge of experts in a particular field, journal articles can often be difficult for non-experts to read, as they tend to incorporate lots of technical jargon and are not written to be engaging or entertaining.

Government Reports and Legal Documents: The government regularly releases information intended for internal and/or public use. These types of documents can be excellent sources of information due to their regularity, dependability, and thoroughness. An example of a government report would be any of the reports the U.S. Census Bureau publishes from census data. Note that most government reports and legal documents can now be accessed online.

Press Releases and Advertising: Companies and special interest groups produce texts to help persuade readers to act in some way or inform the public about some new development. While the information they provide can be accurate, approach them with caution, as these texts' publishers may have vested interests in highlighting particular facts or viewpoints.

Flyers, Pamphlets, Leaflets: While some flyers or pamphlets are created by reputable sources, because of the ease with which they can be created, many less-than-reputable sources also produce these. Pamphlets and leaflets can be useful for quick reference or very general information, but beware of pamphlets that spread propaganda or misleading information.

Digital and Electronic Sources

Multimedia: Printed material is certainly not the only option for finding research. You might also consider using sources such as radio and television broadcasts, interactive talks, and recorded public meetings. Though we often go online to find this sort of information today, libraries and archives offer a wealth of nondigitized media or media that is not available online.

Websites: Most of the information on the Internet is distributed via websites. Websites vary widely in terms of the quality of information they offer. For more information, visit the OWL's page on evaluating digital sources.

Blogs and personal websites: Blogs and personal sites vary widely in their validity as sources for serious research. For example, many prestigious journalists and public figures may have blogs, which may be more credible than most amateur or personal blogs. Note, however, that there are very few standards for impartiality or accuracy when it comes to what can be published on personal sites.

Social media pages and message boards: These types of sources exist for all kinds of disciplines, both in and outside of the university. Some may be useful, depending on the topic you are studying, but, just like personal websites, the information found on social media or message boards is not always credible.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

Knowledge Base
Referencing

A Quick Guide to Harvard Referencing | Citation Examples

Published on 14 February 2020 by Jack Caulfield . Revised on 15 September 2023.

Referencing is an important part of academic writing. It tells your readers what sources you’ve used and how to find them.

Harvard is the most common referencing style used in UK universities. In Harvard style, the author and year are cited in-text, and full details of the source are given in a reference list .

Harvard Reference Generator

Instantly correct all language mistakes in your text

Be assured that you'll submit flawless writing. Upload your document to correct all your mistakes.

Harvard in-text citation, creating a harvard reference list, harvard referencing examples, referencing sources with no author or date, frequently asked questions about harvard referencing.

A Harvard in-text citation appears in brackets beside any quotation or paraphrase of a source. It gives the last name of the author(s) and the year of publication, as well as a page number or range locating the passage referenced, if applicable:

Note that ‘p.’ is used for a single page, ‘pp.’ for multiple pages (e.g. ‘pp. 1–5’).

An in-text citation usually appears immediately after the quotation or paraphrase in question. It may also appear at the end of the relevant sentence, as long as it’s clear what it refers to.

When your sentence already mentions the name of the author, it should not be repeated in the citation:

Sources with multiple authors

When you cite a source with up to three authors, cite all authors’ names. For four or more authors, list only the first name, followed by ‘ et al. ’:

Sources with no page numbers

Some sources, such as websites , often don’t have page numbers. If the source is a short text, you can simply leave out the page number. With longer sources, you can use an alternate locator such as a subheading or paragraph number if you need to specify where to find the quote:

Multiple citations at the same point

When you need multiple citations to appear at the same point in your text – for example, when you refer to several sources with one phrase – you can present them in the same set of brackets, separated by semicolons. List them in order of publication date:

Multiple sources with the same author and date

If you cite multiple sources by the same author which were published in the same year, it’s important to distinguish between them in your citations. To do this, insert an ‘a’ after the year in the first one you reference, a ‘b’ in the second, and so on:

Prevent plagiarism, run a free check.

A bibliography or reference list appears at the end of your text. It lists all your sources in alphabetical order by the author’s last name, giving complete information so that the reader can look them up if necessary.

The reference entry starts with the author’s last name followed by initial(s). Only the first word of the title is capitalised (as well as any proper nouns).

Sources with multiple authors in the reference list

As with in-text citations, up to three authors should be listed; when there are four or more, list only the first author followed by ‘ et al. ’:

Reference list entries vary according to source type, since different information is relevant for different sources. Formats and examples for the most commonly used source types are given below.

Entire book
Book chapter
Translated book
Edition of a book

Journal articles

Print journal
Online-only journal with DOI
Online-only journal with no DOI
General web page
Online article or blog
Social media post

Sometimes you won’t have all the information you need for a reference. This section covers what to do when a source lacks a publication date or named author.

No publication date

When a source doesn’t have a clear publication date – for example, a constantly updated reference source like Wikipedia or an obscure historical document which can’t be accurately dated – you can replace it with the words ‘no date’:

Note that when you do this with an online source, you should still include an access date, as in the example.

When a source lacks a clearly identified author, there’s often an appropriate corporate source – the organisation responsible for the source – whom you can credit as author instead, as in the Google and Wikipedia examples above.

When that’s not the case, you can just replace it with the title of the source in both the in-text citation and the reference list:

Harvard referencing uses an author–date system. Sources are cited by the author’s last name and the publication year in brackets. Each Harvard in-text citation corresponds to an entry in the alphabetised reference list at the end of the paper.

Vancouver referencing uses a numerical system. Sources are cited by a number in parentheses or superscript. Each number corresponds to a full reference at the end of the paper.

A Harvard in-text citation should appear in brackets every time you quote, paraphrase, or refer to information from a source.

The citation can appear immediately after the quotation or paraphrase, or at the end of the sentence. If you’re quoting, place the citation outside of the quotation marks but before any other punctuation like a comma or full stop.

In Harvard referencing, up to three author names are included in an in-text citation or reference list entry. When there are four or more authors, include only the first, followed by ‘ et al. ’

Though the terms are sometimes used interchangeably, there is a difference in meaning:

A reference list only includes sources cited in the text – every entry corresponds to an in-text citation .
A bibliography also includes other sources which were consulted during the research but not cited.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Caulfield, J. (2023, September 15). A Quick Guide to Harvard Referencing | Citation Examples. Scribbr. Retrieved 2 April 2024, from https://www.scribbr.co.uk/referencing/harvard-style/

Is this article helpful?

Jack Caulfield

Other students also liked, harvard in-text citation | a complete guide & examples, harvard style bibliography | format & examples, referencing books in harvard style | templates & examples, scribbr apa citation checker.

An innovative new tool that checks your APA citations with AI software. Say goodbye to inaccurate citations!

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

Advanced Search
Journal List
J Postgrad Med
v.69(1); Jan-Mar 2023

The art of referencing: Well begun is half done!

Department of Pediatrics, College of Medicine and Health Sciences, National University of Science and Technology, Sohar, Sultanate of Oman

Department of Pediatrics, Seth G.S. Medical College and K.E.M. Hospital, Mumbai, Maharashtra, India

Introduction

The value of scientific research lies in its wide visibility and access/availability to others; this is generally achieved by a scientific publication as an original (research) paper. The scientific inquiry typically advances based on previously laid ideas/research, making it essential to acknowledge the contribution of the previous authors. The references list is a catalog of literature sources chosen by the researcher to represent the most relevant documents pertaining to his/her study.[ 1 ] The British Standards Institution defines reference as “a set of data describing a document, sufficiently precise and detailed to identify it and enable it to be located.” [ 2 ] References lay the foundation of the paper, providing context for the hypothesis, methodology, interpretation, and justification of the study.[ 3 ] Using other's ideas/thoughts without due credit amounts to plagiarism, compromising the academic integrity of research. A well-referenced paper is thus accurate and complete, adds value and credibility to both the researcher and the source author, and enhances the scientific prestige of the chosen journal.[ 3 ] A bibliography also lists the sources used during research. However, while references only include those sources (journals, books, web information, etc.) which are actually cited in the publication, bibliography comprises all accessed sources (works consulted), irrespective of whether they are cited in the study publication or not.[ 4 ] Thus, referencing in academic writing is an important research tool to display as well as integrate knowledge on a particular subject or topic.[ 5 ]

Importance of Proper Referencing

Scientific research is usually developed on previously established ideas/scientific knowledge. A meticulous literature review at the beginning of the study enables the researcher to identify the work done in the field, identify the gaps in knowledge, and recognize the need for further research.[ 6 ] The most relevant sources from this literature search (essentially) form the list of references. Use of proper referencing is thus beneficial in many ways, such as the following:

a) It helps the readers to identify and locate the sources used in the research and provides evidence to verify the need/rationale of the study, methodology, inferences, and implications of the study.[ 3 ]
b) It provides an overview of the techniques/tools used, supports/convinces the reader about the appropriateness of the methodology, and offers a proper perspective in which the research findings need to be interpreted.[ 3 , 7 ]
c) It is a proof of the author's in-depth reading and knowledge on the subject pertaining to his/her research. References not only highlight similarities in research, but also differentiate the author's ideas from his sources, indirectly acknowledging the author's own contribution to that topic.[ 4 ]
d) References chosen by a researcher not only credit the individual author/s whose work is cited, but also demonstrate his/her appreciation toward cited authors, at times leading readers toward hitherto lesser-known/unknown author's research.[ 4 , 8 ] By providing acknowledgment to the cited idea/thought, the author also avoids being accused of plagiarism and adds credibility to his/her own work.[ 4 ]
e) Referenced works steer the readers toward literature pertaining to a particular topic, thus advancing the readers’ interest.[ 4 ] It also allows to trace the origins of ideas and integrates newer ideas (from current research) with previous ones, thus building a web of learning about the topic of interest.[ 9 ]
f) The reference list provides a list of experts in a specific field, thus helping editors to identify appropriate reviewers.[ 3 ]
g) It provides peer reviewers with related sources of information to evaluate the manuscript with respect to the cited work.[ 6 ]

Organizing the References

An initial extensive literature search helps in identifying the appropriate research question, drafting the study protocol, supervising ongoing research, analyzing the results, and writing the paper.[ 3 , 7 ] Although references are displayed at the end of the article/after the text of the article, they should not be actually written after completing the text of the manuscript. While drafting the text of the manuscript, the author/s should type the references on a separate MS Word document simultaneously. This preparation allows the writer to choose adequate number of relevant and rational references, avoid bias in his/her research/writing, and limit the reference number as per the target journal for publication.[ 7 ] While citing, it is imperative not to cite broadly, but to do so with respect to the content of the article. Articles which define the topic, lay down background information regarding the study question, give current knowledge about the research, and describe previous studies on a similar study question should be mentioned in the “Introduction” section of the manuscript.[ 10 ] These studies enable to identify existing knowledge, gaps in knowledge, and justify the rationale of the study.[ 6 ] Studies which identify or refer to the method, protocols, or standards (whether new or previously published), elaborate on complex or lesser-known statistical analysis, describe diagnostic criteria, rationalize sample size estimation, or justify use of specific study design/method are best suited as references to the “Methods” section of the manuscript – they help to plan a strong and supported methodology and describe the technique and criteria of the study group.[ 3 , 10 ] Research that reflects on the study findings/results or provides supportive explanation merit mention in the “Discussion” section of the manuscript – they provide information to interpret the study based on existing published data, compare results with those of other studies, and rationalize the implications of the results.[ 10 ]

Though citation analysis treats all references equally, it is important to weigh references in terms of their value to the paper.[ 11 ] While some references are worthy to be mentioned only once in the paper, some are very relevant to the study question and referred to on multiple occasions, and it is important to re-cite only the most relevant articles.[ 3 ] Referencing is not just about stating the publication source (providing relatedness), but also adds value to the paper in terms of representation on the subject and connectivity between knowledge sources (capture the “aboutness”).[ 11 ] References can be books (author/s), legal documents, journal articles, newspaper articles, reports (e.g., official reports from government departments), university working papers, papers presented at conferences, internet sources (including weblogs – blogs and email correspondence), DVD/CD databases, radio/television/videos/audio cassette/CD-ROMs, interview transcripts, and illustrations.[ 12 ]

Choosing Appropriate References

As a rule, whenever one uses an idea, data, diagrams, tables, concepts, methods from a previously published work, it should be cited.[ 12 ] With availability of multiple search engines and abundance of online resources, the task of filtering references may seem daunting.[ 5 ] While choosing references, one should ensure that the original source is completely read and correctly interpreted before its citing.[ 6 ] It is preferable to provide direct references to original article sources as far as possible, choosing a landmark article on the topic.[ 5 ] The choice of references should serve as the most relevant, appropriate, and valuable addition, and one should stick to the most pertinent references that actively support/contradict their conclusions or experience.[ 6 ] It is preferable to use the most recent relevant resources to provide the latest and up-to-date information; however, certain landmark papers may also be cited (even if they are old). Note that very old references may not be available/accessible to reviewers as well as readers.[ 7 ] Often, there are multiple sources for the same information; always prefer references that provide the highest level of evidence (such as meta-analysis), most recent publications, or trustworthy sources such as reputed peer-reviewed journals (with higher impact factor), open access and preferably indexed on reputed databases such as MEDLINE and PubMed.[ 13 , 14 ] Citing works from the journal one wishes to submit demonstrates that author follows that particular journal's publications and values it; however, one should refrain from unethical practices such as coercive citation (when authors are coerced/directed to add irrelevant citations from the editor's journals) or padded citation (when authors pad their reference list with superfluous citations).[ 14 , 15 , 16 ] There should be a judicious combination of original as well as review articles. Review articles summarize a large body of literature and reduce the number of references; however they may be biased and may not reflect the original article accurately.[ 16 ] One should stick to the journal guidelines rigorously (in terms of style and number) to avoid rejection or delay in the processing of the manuscript.[ 6 ] Avoid citing conference abstracts as far as possible, as they provide incomplete or limited information on the subject and often lack an appropriate peer review.[ 16 ] Other sources which lack traditional review and thus may cite inappropriate, unchecked, or promotional content include online sources, such as audio and video presentations, and should therefore be used with caution.[ 17 ] It is also prudent to avoid personal communications and limit their use to situations where essential information is unavailable from a public source (if permission is necessary, then name and date of the communication should be cited in brackets in text).[ 16 ] Limit self-citations to the bare crucial ones that are necessary.[ 18 ] Articles accepted but awaiting publications should be cited as “in press.”[ 16 ] Articles submitted but not yet published should be referenced as “unpublished observations” with written permission from the source; however, since they have not undergone a peer review, they should be (preferably) avoided.[ 16 ] It is prudent to avoid citing articles published in predatory journals.[ 16 ]

There is no need to provide references to facts that are expected to be well known to the journal readers, including historical overviews, own experiences, while outlining previously referenced ideas in conclusions, or while summarizing what is regarded as “common knowledge.”[ 12 ] One should be careful with online sources. There may be errors while copying the uniform resource locator (URL) or the webpage, or the website may change or be closed/inaccessible; hence, cite them only if very essential and check for their reliability and give the date of access.[ 3 ] It is preferable to use online sources with digital object identifiers (DOIs), assuring their permanent presence.[ 13 ] Also, before submission, it is worthwhile to check the US National Library of Medicine's (NLM's) PubMed database ( http://www.pubmed.org ) for any recently published articles related to the manuscript's topic.[ 19 ]

The number of references is determined by the target journal requirements as well as the type of manuscript submitted; for example, the Journal of Postgraduate Medicine allows about 30 references for original articles, up to 15 references for brief reports/grand rounds/clinicopathological forum, 12 references for case series, up to 10 references for case reports/research letter, and five references for a letter to editor ( https://www.jpgmonline.com/contributors.asp#Ref ).

Preparing the References

Citation consists of two components – the “in-text citation” and the “reference list.”[ 7 ] In the in-text citation, quotation marks are used to cite an exact line/phrase from another source, specifically for definitions, examples, or explanations provided by another/earlier author/s.[ 13 ] To prevent plagiarism, it is suitable to interpret and then summarize the cited content in one's own words, referencing the source at the end of the sentence.[ 14 ]

The parts and order in the citation depend on the source which the author is referencing (journal, book, book chapter, or web source) and the journal guidelines. It is imperative to go through the target journal rules and follow the “Instructions to Authors” related to referencing guidelines (the style, punctuation, italics, abbreviations, issue number, volume number, and pages). All the references are generally cited and numbered as per the order in which they are mentioned in the text (and are to be inserted immediately after the source information and not necessarily at the end of the sentence, especially when multiple facts are stated in a single sentence).[ 6 ] In case of a table or a figure, the citation number should be in sequence to that of the preceding text.[ 7 ] The same reference number in which the source is first cited should be used throughout the manuscript (if cited again) as well as in the reference list.[ 7 ] The citation numbers are placed as superscript/in parentheses as per the journal guidelines.[ 7 ] In case of multiple citations, place them immediately after the fact; they should be placed in order of their chronology of publication (or alphabetically if published in the same year) separated by commas.[ 6 , 7 ] If many references are cited consecutively, the numbers can be separated by a hyphen.[ 7 ]

Any documented knowledge (text, audio, or visual) can serve as a source of reference. They can be print based or electronic and include journals, books, doctoral theses, conference papers, newspapers and magazines, web pages, and so on.[ 4 ]

The basic elements while referencing are as follows:[ 13 , 20 ]

Journal/research paper: Name of author/s, title of paper, journal title (often abbreviated according to the style used for MEDLINE [ www.ncbi.nlm.nih.gov/nlmcatalog/journals ]),[ 16 ] publication year, volume number, issue number (issue number is kept optional by many journals), and page numbers (starting and ending page numbers or e-article number if the journal does not allot page numbers but allots e-article numbers only)[ 13 ]
Book/chapter: Chapter author/s, title of chapter, editor/s, name of book, edition, publisher, city of publication, year of publication, and page numbers[ 13 ]
Web sources: Names of author/s, title of webpage, year, weblink, date of access, and other information such as publisher, year of publication, and date of recent update (as may be applicable/available).[ 13 ] When citing a webpage, provide the DOI or URL of the original source as far as possible.[ 1 , 20 ]

Special attention needs to be paid to the punctuations while composing the reference, and the authors must adhere to the style recommended by the journal (that the manuscript will be eventually submitted to). Note that with each revision that the author makes in the manuscript, there may be changes in the order, addition, or deletion of references, and these adjustments should be meticulously ensured to avoid referencing errors.[ 3 ] It is also the author's responsibility to ensure that every citation has a corresponding reference and every reference is cited in the right place and context in the manuscript.[ 6 ] To avoid citation errors, the authors must verify each reference against an electronic bibliographic source like PubMed or print/pdf copies of original resources.[ 16 ] Authors should also verify that none of the cited references is a retracted article; this can be done via MEDLINE by searching PubMed for “Retracted publication [publication type]” or by going directly to the PubMed's list of retracted publications ( https://www.ncbi.nlm.nih.gov/pubmed/?termretractedpublication[publication type] ).[ 16 ]

“Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals” issued by the International Committee of Medical Journal Editors (ICMJE) provides specific information on how to cite sources, which should be followed.[ 16 ] These recommendations by the ICMJE summarize and provide regular updates on how to cite various sources (print documents; unpublished material; audio and visual media; material on CD-ROM, DVD, or disk; and material on the Internet) via Sample References ( www.nlm.nih.gov/bsd/uniform_requirements.html ) on their webpage.[ 16 ] Detailed information is also available in the NLM's Citing Medicine, 2 nd edition ( www.ncbi.nlm.nih.gov/books/NBK7256/ ).[ 20 ] Number of references to be cited should be in accordance with/within the limits as stated in the “Author Guidelines” issued by the target journal.[ 7 ] Authors should take precaution, so as to avoid citing the same reference twice in the list of references.

Types of Referencing Formats/Styles

“Citation style” is the standard format in which the source is documented in the text as well as in the reference list at the end of the manuscript.[ 4 ] In-text citation styles can be broadly classified into numerical referencing style (Numeric style/Vancouver/Institute of Electrical and Electronic Engineers [IEEE] and Running notes style/Modern Humanities Research Association [MHRA]) and name referencing style (Author Date/Harvard, American Psychological Association [APA] and Modern Languages Association [MLA]).[ 12 ] The two major used citation styles are the Vancouver and the Harvard styles, and most other styles are minor modifications of these two styles.[ 4 ] The common citation styles and their examples are summarized in Table 1 .[ 3 , 12 , 13 , 16 , 20 ] Thus, there is a wide variability in the citation style in text as well as reference list; however, the author does not have a choice, but to stick to the style recommended by the journal to which he/she wishes to submit his/her research.[ 3 ]

Citation styles with examples[ 3 , 12 , 13 , 16 , 20 ]

Common Errors in Referencing

Referencing is a tedious task and if not taken seriously and performed diligently, it is prone to many (easily avoidable) errors.[ 7 ] A reference should be accurate, clear, and consistent throughout the manuscript.[ 6 ] An incorrect reference not only questions the credibility of the paper, but also makes it difficult for the reviewers and the readers to seek the cited article, thus denying the source author of due credit for his/her work.[ 3 ] It is the author's responsibility to cite the most relevant and appropriate references in his/her research.[ 3 ] The author should not only locate, read, and understand all sources cited by him/her ( intellectual pleasure ), but also confirm the source and provide all elements of the source correctly ( accuracy ).[ 6 ] The author should be careful not to copy references from an earlier article, but should actually rewrite each selected reference afresh.[ 6 ] Some common errors occurring during referencing are summarized in Table 2 .[ 6 , 7 ]

Common errors in the “in-text citation” and the “reference list”[ 6 , 7 ]

Reference Management Software

As described earlier, there is a wide variation in the journal formatting styles and it is laborious for the researcher to store, organize, and manage the references throughout the process of literature review and protocol planning till the drafting and manuscript submission.[ 21 ] Even more challenging is the addition/deletion or reordering of references (in text as well as in the reference list) with each revision or submission to a newer journal.[ 22 ] There is an increased likelihood of making errors in citing, especially while organizing the references and writing the reference list.[ 23 ] To minimize such errors, reference management software (RMS), also known as citation management software or bibliographic management software, are available to the authors/researchers.[ 21 ] They not only help to search and retrieve the online scientific sources, but also help to import them to their database for storing, organization, and subsequent retrieval.[ 22 ] Many RMS have cloud-based storage, enabling users to be able to access the information from multiple devices as well as collaborate with other researchers.[ 22 ] RMS also allow authors to retrieve citations while writing in the format of desired journal, thus permitting to “cite while you write.”[ 14 ] They also enable addition, deletion, insertion of references in the text and automatic (auto) resequencing of their order in the main manuscript (text) as well as in the reference list.[ 22 ] They can generate reference lists in multiple formats/citation styles to suit the target journal requirements and allow conversion of one format style to another with ease at the click of the mouse.[ 14 ] By linking each citation with a full reference, they ensure each citation in the text is accounted for by a corresponding full reference in the list.[ 12 ] Most of them are compatible for use with common programs such as Microsoft Word and Google Docs.[ 24 ]

There are numerous programs for reference management available in the market – independent applications, those operating within an internet interface, and combination of both these modes.[ 1 ] The most commonly used are Mendeley by Elsevier ( www.mendeley.com ), EndNote ( www.endnote.com ) by Thomson Reuters, and Zotero ( www.zotero.org ).[ 1 ] Some others are RefWorks, F1000 Workspace, JabRef, Citavi, Bibsonomy, ReadCube Papers, Colwiz, Sente, RefME, Connotea, CiteULike, BibTeX, and Microsoft Word.[ 22 , 24 , 25 ] While many of them are free, some are fee based and require a (paid) subscription.[ 13 ]

Despite the use of RMS, one cannot guarantee absence of referencing errors, as there can be errors in details (author names, journal title, dates) or duplication of references when retrieved from different databases.[ 23 ] So, ultimately, the authors (themselves) are responsible for the accuracy of the references cited by them (whether they do the referencing using RMS or manually).

Thus, referencing is an essential part of research and should be assigned due importance, right from the conception of the study question till its delivery as a publication. It plays a vital role throughout the manuscript and appears in almost all sections – from laying down the foundation for study rationale (in the “Introduction” section of the manuscript), describing/justifying the study procedure/s (in the “Methods” section), validating the results (in the “Results” section) and its implications (in the “Discussion” section of the manuscript). References are also utilized by editors to identify subject experts for peer review, by readers to obtain more resources on the subject matter, and by peer reviewers to critically evaluate the manuscript in the light of the available evidence. It is thus essential that references are chosen wisely and carefully as they are representative of the study. It is the author's responsibility to confirm the clarity, accuracy, and appropriateness of the cited sources. One should be careful to avoid common referencing errors to prevent delay/rejection by the journal of interest. As Vancouver style is the commonly preferred citation style by journals of medicine and health sciences, researchers should be well versed with it. Authors should diligently stick to the instructions and style of the target journal. The availability of reference management software such as Mendeley and EndNote has made the authors’ task of collecting, storing, organizing, retrieving, and utilizing the references more efficient and easier; however, it is still the authors’ responsibility to select appropriate references and cite them accurately and correctly.

Financial support and sponsorship

Conflicts of interest.

There are no conflicts of interest.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

View all journals
My Account Login
Explore content
About the journal
Publish with us
Sign up for alerts
Open access
Published: 26 March 2024

Predicting and improving complex beer flavor through machine learning

Michiel Schreurs ORCID: orcid.org/0000-0002-9449-5619 1 , 2 , 3 na1 ,
Supinya Piampongsant 1 , 2 , 3 na1 ,
Miguel Roncoroni ORCID: orcid.org/0000-0001-7461-1427 1 , 2 , 3 na1 ,
Lloyd Cool ORCID: orcid.org/0000-0001-9936-3124 1 , 2 , 3 , 4 ,
Beatriz Herrera-Malaver ORCID: orcid.org/0000-0002-5096-9974 1 , 2 , 3 ,
Christophe Vanderaa ORCID: orcid.org/0000-0001-7443-5427 4 ,
Florian A. Theßeling 1 , 2 , 3 ,
Łukasz Kreft ORCID: orcid.org/0000-0001-7620-4657 5 ,
Alexander Botzki ORCID: orcid.org/0000-0001-6691-4233 5 ,
Philippe Malcorps 6 ,
Luk Daenen 6 ,
Tom Wenseleers ORCID: orcid.org/0000-0002-1434-861X 4 &
Kevin J. Verstrepen ORCID: orcid.org/0000-0002-3077-6219 1 , 2 , 3

Nature Communications volume 15 , Article number: 2368 ( 2024 ) Cite this article

49k Accesses

847 Altmetric

Metrics details

Chemical engineering
Gas chromatography
Machine learning
Metabolomics
Taste receptors

The perception and appreciation of food flavor depends on many interacting chemical compounds and external factors, and therefore proves challenging to understand and predict. Here, we combine extensive chemical and sensory analyses of 250 different beers to train machine learning models that allow predicting flavor and consumer appreciation. For each beer, we measure over 200 chemical properties, perform quantitative descriptive sensory analysis with a trained tasting panel and map data from over 180,000 consumer reviews to train 10 different machine learning models. The best-performing algorithm, Gradient Boosting, yields models that significantly outperform predictions based on conventional statistics and accurately predict complex food features and consumer appreciation from chemical profiles. Model dissection allows identifying specific and unexpected compounds as drivers of beer flavor and appreciation. Adding these compounds results in variants of commercial alcoholic and non-alcoholic beers with improved consumer appreciation. Together, our study reveals how big data and machine learning uncover complex links between food chemistry, flavor and consumer perception, and lays the foundation to develop novel, tailored foods with superior flavors.

BitterSweet: Building machine learning models for predicting the bitter and sweet taste of small molecules

Rudraksh Tuwani, Somin Wadhwa & Ganesh Bagler

Sensory lexicon and aroma volatiles analysis of brewing malt

Xiaoxia Su, Miao Yu, … Tianyi Du

Predicting odor from molecular structure: a multi-label classification approach

Kushagra Saini & Venkatnarayan Ramanathan

Introduction

Predicting and understanding food perception and appreciation is one of the major challenges in food science. Accurate modeling of food flavor and appreciation could yield important opportunities for both producers and consumers, including quality control, product fingerprinting, counterfeit detection, spoilage detection, and the development of new products and product combinations (food pairing) 1 , 2 , 3 , 4 , 5 , 6 . Accurate models for flavor and consumer appreciation would contribute greatly to our scientific understanding of how humans perceive and appreciate flavor. Moreover, accurate predictive models would also facilitate and standardize existing food assessment methods and could supplement or replace assessments by trained and consumer tasting panels, which are variable, expensive and time-consuming 7 , 8 , 9 . Lastly, apart from providing objective, quantitative, accurate and contextual information that can help producers, models can also guide consumers in understanding their personal preferences 10 .

Despite the myriad of applications, predicting food flavor and appreciation from its chemical properties remains a largely elusive goal in sensory science, especially for complex food and beverages 11 , 12 . A key obstacle is the immense number of flavor-active chemicals underlying food flavor. Flavor compounds can vary widely in chemical structure and concentration, making them technically challenging and labor-intensive to quantify, even in the face of innovations in metabolomics, such as non-targeted metabolic fingerprinting 13 , 14 . Moreover, sensory analysis is perhaps even more complicated. Flavor perception is highly complex, resulting from hundreds of different molecules interacting at the physiochemical and sensorial level. Sensory perception is often non-linear, characterized by complex and concentration-dependent synergistic and antagonistic effects 15 , 16 , 17 , 18 , 19 , 20 , 21 that are further convoluted by the genetics, environment, culture and psychology of consumers 22 , 23 , 24 . Perceived flavor is therefore difficult to measure, with problems of sensitivity, accuracy, and reproducibility that can only be resolved by gathering sufficiently large datasets 25 . Trained tasting panels are considered the prime source of quality sensory data, but require meticulous training, are low throughput and high cost. Public databases containing consumer reviews of food products could provide a valuable alternative, especially for studying appreciation scores, which do not require formal training 25 . Public databases offer the advantage of amassing large amounts of data, increasing the statistical power to identify potential drivers of appreciation. However, public datasets suffer from biases, including a bias in the volunteers that contribute to the database, as well as confounding factors such as price, cult status and psychological conformity towards previous ratings of the product.

Classical multivariate statistics and machine learning methods have been used to predict flavor of specific compounds by, for example, linking structural properties of a compound to its potential biological activities or linking concentrations of specific compounds to sensory profiles 1 , 26 . Importantly, most previous studies focused on predicting organoleptic properties of single compounds (often based on their chemical structure) 27 , 28 , 29 , 30 , 31 , 32 , 33 , thus ignoring the fact that these compounds are present in a complex matrix in food or beverages and excluding complex interactions between compounds. Moreover, the classical statistics commonly used in sensory science 34 , 35 , 36 , 37 , 38 , 39 require a large sample size and sufficient variance amongst predictors to create accurate models. They are not fit for studying an extensive set of hundreds of interacting flavor compounds, since they are sensitive to outliers, have a high tendency to overfit and are less suited for non-linear and discontinuous relationships 40 .

In this study, we combine extensive chemical analyses and sensory data of a set of different commercial beers with machine learning approaches to develop models that predict taste, smell, mouthfeel and appreciation from compound concentrations. Beer is particularly suited to model the relationship between chemistry, flavor and appreciation. First, beer is a complex product, consisting of thousands of flavor compounds that partake in complex sensory interactions 41 , 42 , 43 . This chemical diversity arises from the raw materials (malt, yeast, hops, water and spices) and biochemical conversions during the brewing process (kilning, mashing, boiling, fermentation, maturation and aging) 44 , 45 . Second, the advent of the internet saw beer consumers embrace online review platforms, such as RateBeer (ZX Ventures, Anheuser-Busch InBev SA/NV) and BeerAdvocate (Next Glass, inc.). In this way, the beer community provides massive data sets of beer flavor and appreciation scores, creating extraordinarily large sensory databases to complement the analyses of our professional sensory panel. Specifically, we characterize over 200 chemical properties of 250 commercial beers, spread across 22 beer styles, and link these to the descriptive sensory profiling data of a 16-person in-house trained tasting panel and data acquired from over 180,000 public consumer reviews. These unique and extensive datasets enable us to train a suite of machine learning models to predict flavor and appreciation from a beer’s chemical profile. Dissection of the best-performing models allows us to pinpoint specific compounds as potential drivers of beer flavor and appreciation. Follow-up experiments confirm the importance of these compounds and ultimately allow us to significantly improve the flavor and appreciation of selected commercial beers. Together, our study represents a significant step towards understanding complex flavors and reinforces the value of machine learning to develop and refine complex foods. In this way, it represents a stepping stone for further computer-aided food engineering applications 46 .

To generate a comprehensive dataset on beer flavor, we selected 250 commercial Belgian beers across 22 different beer styles (Supplementary Fig. S1 ). Beers with ≤ 4.2% alcohol by volume (ABV) were classified as non-alcoholic and low-alcoholic. Blonds and Tripels constitute a significant portion of the dataset (12.4% and 11.2%, respectively) reflecting their presence on the Belgian beer market and the heterogeneity of beers within these styles. By contrast, lager beers are less diverse and dominated by a handful of brands. Rare styles such as Brut or Faro make up only a small fraction of the dataset (2% and 1%, respectively) because fewer of these beers are produced and because they are dominated by distinct characteristics in terms of flavor and chemical composition.

Extensive analysis identifies relationships between chemical compounds in beer

For each beer, we measured 226 different chemical properties, including common brewing parameters such as alcohol content, iso-alpha acids, pH, sugar concentration 47 , and over 200 flavor compounds (Methods, Supplementary Table S1 ). A large portion (37.2%) are terpenoids arising from hopping, responsible for herbal and fruity flavors 16 , 48 . A second major category are yeast metabolites, such as esters and alcohols, that result in fruity and solvent notes 48 , 49 , 50 . Other measured compounds are primarily derived from malt, or other microbes such as non- Saccharomyces yeasts and bacteria (‘wild flora’). Compounds that arise from spices or staling are labeled under ‘Others’. Five attributes (caloric value, total acids and total ester, hop aroma and sulfur compounds) are calculated from multiple individually measured compounds.

As a first step in identifying relationships between chemical properties, we determined correlations between the concentrations of the compounds (Fig. 1 , upper panel, Supplementary Data 1 and 2 , and Supplementary Fig. S2 . For the sake of clarity, only a subset of the measured compounds is shown in Fig. 1 ). Compounds of the same origin typically show a positive correlation, while absence of correlation hints at parameters varying independently. For example, the hop aroma compounds citronellol, and alpha-terpineol show moderate correlations with each other (Spearman’s rho=0.39 and 0.57), but not with the bittering hop component iso-alpha acids (Spearman’s rho=0.16 and −0.07). This illustrates how brewers can independently modify hop aroma and bitterness by selecting hop varieties and dosage time. If hops are added early in the boiling phase, chemical conversions increase bitterness while aromas evaporate, conversely, late addition of hops preserves aroma but limits bitterness 51 . Similarly, hop-derived iso-alpha acids show a strong anti-correlation with lactic acid and acetic acid, likely reflecting growth inhibition of lactic acid and acetic acid bacteria, or the consequent use of fewer hops in sour beer styles, such as West Flanders ales and Fruit beers, that rely on these bacteria for their distinct flavors 52 . Finally, yeast-derived esters (ethyl acetate, ethyl decanoate, ethyl hexanoate, ethyl octanoate) and alcohols (ethanol, isoamyl alcohol, isobutanol, and glycerol), correlate with Spearman coefficients above 0.5, suggesting that these secondary metabolites are correlated with the yeast genetic background and/or fermentation parameters and may be difficult to influence individually, although the choice of yeast strain may offer some control 53 .

Spearman rank correlations are shown. Descriptors are grouped according to their origin (malt (blue), hops (green), yeast (red), wild flora (yellow), Others (black)), and sensory aspect (aroma, taste, palate, and overall appreciation). Please note that for the chemical compounds, for the sake of clarity, only a subset of the total number of measured compounds is shown, with an emphasis on the key compounds for each source. For more details, see the main text and Methods section. Chemical data can be found in Supplementary Data 1 , correlations between all chemical compounds are depicted in Supplementary Fig. S2 and correlation values can be found in Supplementary Data 2 . See Supplementary Data 4 for sensory panel assessments and Supplementary Data 5 for correlation values between all sensory descriptors.

Interestingly, different beer styles show distinct patterns for some flavor compounds (Supplementary Fig. S3 ). These observations agree with expectations for key beer styles, and serve as a control for our measurements. For instance, Stouts generally show high values for color (darker), while hoppy beers contain elevated levels of iso-alpha acids, compounds associated with bitter hop taste. Acetic and lactic acid are not prevalent in most beers, with notable exceptions such as Kriek, Lambic, Faro, West Flanders ales and Flanders Old Brown, which use acid-producing bacteria ( Lactobacillus and Pediococcus ) or unconventional yeast ( Brettanomyces ) 54 , 55 . Glycerol, ethanol and esters show similar distributions across all beer styles, reflecting their common origin as products of yeast metabolism during fermentation 45 , 53 . Finally, low/no-alcohol beers contain low concentrations of glycerol and esters. This is in line with the production process for most of the low/no-alcohol beers in our dataset, which are produced through limiting fermentation or by stripping away alcohol via evaporation or dialysis, with both methods having the unintended side-effect of reducing the amount of flavor compounds in the final beer 56 , 57 .

Besides expected associations, our data also reveals less trivial associations between beer styles and specific parameters. For example, geraniol and citronellol, two monoterpenoids responsible for citrus, floral and rose flavors and characteristic of Citra hops, are found in relatively high amounts in Christmas, Saison, and Brett/co-fermented beers, where they may originate from terpenoid-rich spices such as coriander seeds instead of hops 58 .

Tasting panel assessments reveal sensorial relationships in beer

To assess the sensory profile of each beer, a trained tasting panel evaluated each of the 250 beers for 50 sensory attributes, including different hop, malt and yeast flavors, off-flavors and spices. Panelists used a tasting sheet (Supplementary Data 3 ) to score the different attributes. Panel consistency was evaluated by repeating 12 samples across different sessions and performing ANOVA. In 95% of cases no significant difference was found across sessions ( p > 0.05), indicating good panel consistency (Supplementary Table S2 ).

Aroma and taste perception reported by the trained panel are often linked (Fig. 1 , bottom left panel and Supplementary Data 4 and 5 ), with high correlations between hops aroma and taste (Spearman’s rho=0.83). Bitter taste was found to correlate with hop aroma and taste in general (Spearman’s rho=0.80 and 0.69), and particularly with “grassy” noble hops (Spearman’s rho=0.75). Barnyard flavor, most often associated with sour beers, is identified together with stale hops (Spearman’s rho=0.97) that are used in these beers. Lactic and acetic acid, which often co-occur, are correlated (Spearman’s rho=0.66). Interestingly, sweetness and bitterness are anti-correlated (Spearman’s rho = −0.48), confirming the hypothesis that they mask each other 59 , 60 . Beer body is highly correlated with alcohol (Spearman’s rho = 0.79), and overall appreciation is found to correlate with multiple aspects that describe beer mouthfeel (alcohol, carbonation; Spearman’s rho= 0.32, 0.39), as well as with hop and ester aroma intensity (Spearman’s rho=0.39 and 0.35).

Similar to the chemical analyses, sensorial analyses confirmed typical features of specific beer styles (Supplementary Fig. S4 ). For example, sour beers (Faro, Flanders Old Brown, Fruit beer, Kriek, Lambic, West Flanders ale) were rated acidic, with flavors of both acetic and lactic acid. Hoppy beers were found to be bitter and showed hop-associated aromas like citrus and tropical fruit. Malt taste is most detected among scotch, stout/porters, and strong ales, while low/no-alcohol beers, which often have a reputation for being ‘worty’ (reminiscent of unfermented, sweet malt extract) appear in the middle. Unsurprisingly, hop aromas are most strongly detected among hoppy beers. Like its chemical counterpart (Supplementary Fig. S3 ), acidity shows a right-skewed distribution, with the most acidic beers being Krieks, Lambics, and West Flanders ales.

Tasting panel assessments of specific flavors correlate with chemical composition

We find that the concentrations of several chemical compounds strongly correlate with specific aroma or taste, as evaluated by the tasting panel (Fig. 2 , Supplementary Fig. S5 , Supplementary Data 6 ). In some cases, these correlations confirm expectations and serve as a useful control for data quality. For example, iso-alpha acids, the bittering compounds in hops, strongly correlate with bitterness (Spearman’s rho=0.68), while ethanol and glycerol correlate with tasters’ perceptions of alcohol and body, the mouthfeel sensation of fullness (Spearman’s rho=0.82/0.62 and 0.72/0.57 respectively) and darker color from roasted malts is a good indication of malt perception (Spearman’s rho=0.54).

Heatmap colors indicate Spearman’s Rho. Axes are organized according to sensory categories (aroma, taste, mouthfeel, overall), chemical categories and chemical sources in beer (malt (blue), hops (green), yeast (red), wild flora (yellow), Others (black)). See Supplementary Data 6 for all correlation values.

Interestingly, for some relationships between chemical compounds and perceived flavor, correlations are weaker than expected. For example, the rose-smelling phenethyl acetate only weakly correlates with floral aroma. This hints at more complex relationships and interactions between compounds and suggests a need for a more complex model than simple correlations. Lastly, we uncovered unexpected correlations. For instance, the esters ethyl decanoate and ethyl octanoate appear to correlate slightly with hop perception and bitterness, possibly due to their fruity flavor. Iron is anti-correlated with hop aromas and bitterness, most likely because it is also anti-correlated with iso-alpha acids. This could be a sign of metal chelation of hop acids 61 , given that our analyses measure unbound hop acids and total iron content, or could result from the higher iron content in dark and Fruit beers, which typically have less hoppy and bitter flavors 62 .

Public consumer reviews complement expert panel data

To complement and expand the sensory data of our trained tasting panel, we collected 180,000 reviews of our 250 beers from the online consumer review platform RateBeer. This provided numerical scores for beer appearance, aroma, taste, palate, overall quality as well as the average overall score.

Public datasets are known to suffer from biases, such as price, cult status and psychological conformity towards previous ratings of a product. For example, prices correlate with appreciation scores for these online consumer reviews (rho=0.49, Supplementary Fig. S6 ), but not for our trained tasting panel (rho=0.19). This suggests that prices affect consumer appreciation, which has been reported in wine 63 , while blind tastings are unaffected. Moreover, we observe that some beer styles, like lagers and non-alcoholic beers, generally receive lower scores, reflecting that online reviewers are mostly beer aficionados with a preference for specialty beers over lager beers. In general, we find a modest correlation between our trained panel’s overall appreciation score and the online consumer appreciation scores (Fig. 3 , rho=0.29). Apart from the aforementioned biases in the online datasets, serving temperature, sample freshness and surroundings, which are all tightly controlled during the tasting panel sessions, can vary tremendously across online consumers and can further contribute to (among others, appreciation) differences between the two categories of tasters. Importantly, in contrast to the overall appreciation scores, for many sensory aspects the results from the professional panel correlated well with results obtained from RateBeer reviews. Correlations were highest for features that are relatively easy to recognize even for untrained tasters, like bitterness, sweetness, alcohol and malt aroma (Fig. 3 and below).

RateBeer text mining results can be found in Supplementary Data 7 . Rho values shown are Spearman correlation values, with asterisks indicating significant correlations ( p < 0.05, two-sided). All p values were smaller than 0.001, except for Esters aroma (0.0553), Esters taste (0.3275), Esters aroma—banana (0.0019), Coriander (0.0508) and Diacetyl (0.0134).

Besides collecting consumer appreciation from these online reviews, we developed automated text analysis tools to gather additional data from review texts (Supplementary Data 7 ). Processing review texts on the RateBeer database yielded comparable results to the scores given by the trained panel for many common sensory aspects, including acidity, bitterness, sweetness, alcohol, malt, and hop tastes (Fig. 3 ). This is in line with what would be expected, since these attributes require less training for accurate assessment and are less influenced by environmental factors such as temperature, serving glass and odors in the environment. Consumer reviews also correlate well with our trained panel for 4-vinyl guaiacol, a compound associated with a very characteristic aroma. By contrast, correlations for more specific aromas like ester, coriander or diacetyl are underrepresented in the online reviews, underscoring the importance of using a trained tasting panel and standardized tasting sheets with explicit factors to be scored for evaluating specific aspects of a beer. Taken together, our results suggest that public reviews are trustworthy for some, but not all, flavor features and can complement or substitute taste panel data for these sensory aspects.

Models can predict beer sensory profiles from chemical data

The rich datasets of chemical analyses, tasting panel assessments and public reviews gathered in the first part of this study provided us with a unique opportunity to develop predictive models that link chemical data to sensorial features. Given the complexity of beer flavor, basic statistical tools such as correlations or linear regression may not always be the most suitable for making accurate predictions. Instead, we applied different machine learning models that can model both simple linear and complex interactive relationships. Specifically, we constructed a set of regression models to predict (a) trained panel scores for beer flavor and quality and (b) public reviews’ appreciation scores from beer chemical profiles. We trained and tested 10 different models (Methods), 3 linear regression-based models (simple linear regression with first-order interactions (LR), lasso regression with first-order interactions (Lasso), partial least squares regressor (PLSR)), 5 decision tree models (AdaBoost regressor (ABR), extra trees (ET), gradient boosting regressor (GBR), random forest (RF) and XGBoost regressor (XGBR)), 1 support vector regression (SVR), and 1 artificial neural network (ANN) model.

To compare the performance of our machine learning models, the dataset was randomly split into a training and test set, stratified by beer style. After a model was trained on data in the training set, its performance was evaluated on its ability to predict the test dataset obtained from multi-output models (based on the coefficient of determination, see Methods). Additionally, individual-attribute models were ranked per descriptor and the average rank was calculated, as proposed by Korneva et al. 64 . Importantly, both ways of evaluating the models’ performance agreed in general. Performance of the different models varied (Table 1 ). It should be noted that all models perform better at predicting RateBeer results than results from our trained tasting panel. One reason could be that sensory data is inherently variable, and this variability is averaged out with the large number of public reviews from RateBeer. Additionally, all tree-based models perform better at predicting taste than aroma. Linear models (LR) performed particularly poorly, with negative R 2 values, due to severe overfitting (training set R 2 = 1). Overfitting is a common issue in linear models with many parameters and limited samples, especially with interaction terms further amplifying the number of parameters. L1 regularization (Lasso) successfully overcomes this overfitting, out-competing multiple tree-based models on the RateBeer dataset. Similarly, the dimensionality reduction of PLSR avoids overfitting and improves performance, to some extent. Still, tree-based models (ABR, ET, GBR, RF and XGBR) show the best performance, out-competing the linear models (LR, Lasso, PLSR) commonly used in sensory science 65 .

GBR models showed the best overall performance in predicting sensory responses from chemical information, with R 2 values up to 0.75 depending on the predicted sensory feature (Supplementary Table S4 ). The GBR models predict consumer appreciation (RateBeer) better than our trained panel’s appreciation (R 2 value of 0.67 compared to R 2 value of 0.09) (Supplementary Table S3 and Supplementary Table S4 ). ANN models showed intermediate performance, likely because neural networks typically perform best with larger datasets 66 . The SVR shows intermediate performance, mostly due to the weak predictions of specific attributes that lower the overall performance (Supplementary Table S4 ).

Model dissection identifies specific, unexpected compounds as drivers of consumer appreciation

Next, we leveraged our models to infer important contributors to sensory perception and consumer appreciation. Consumer preference is a crucial sensory aspects, because a product that shows low consumer appreciation scores often does not succeed commercially 25 . Additionally, the requirement for a large number of representative evaluators makes consumer trials one of the more costly and time-consuming aspects of product development. Hence, a model for predicting chemical drivers of overall appreciation would be a welcome addition to the available toolbox for food development and optimization.

Since GBR models on our RateBeer dataset showed the best overall performance, we focused on these models. Specifically, we used two approaches to identify important contributors. First, rankings of the most important predictors for each sensorial trait in the GBR models were obtained based on impurity-based feature importance (mean decrease in impurity). High-ranked parameters were hypothesized to be either the true causal chemical properties underlying the trait, to correlate with the actual causal properties, or to take part in sensory interactions affecting the trait 67 (Fig. 4A ). In a second approach, we used SHAP 68 to determine which parameters contributed most to the model for making predictions of consumer appreciation (Fig. 4B ). SHAP calculates parameter contributions to model predictions on a per-sample basis, which can be aggregated into an importance score.

A The impurity-based feature importance (mean deviance in impurity, MDI) calculated from the Gradient Boosting Regression (GBR) model predicting RateBeer appreciation scores. The top 15 highest ranked chemical properties are shown. B SHAP summary plot for the top 15 parameters contributing to our GBR model. Each point on the graph represents a sample from our dataset. The color represents the concentration of that parameter, with bluer colors representing low values and redder colors representing higher values. Greater absolute values on the horizontal axis indicate a higher impact of the parameter on the prediction of the model. C Spearman correlations between the 15 most important chemical properties and consumer overall appreciation. Numbers indicate the Spearman Rho correlation coefficient, and the rank of this correlation compared to all other correlations. The top 15 important compounds were determined using SHAP (panel B).

Both approaches identified ethyl acetate as the most predictive parameter for beer appreciation (Fig. 4 ). Ethyl acetate is the most abundant ester in beer with a typical ‘fruity’, ‘solvent’ and ‘alcoholic’ flavor, but is often considered less important than other esters like isoamyl acetate. The second most important parameter identified by SHAP is ethanol, the most abundant beer compound after water. Apart from directly contributing to beer flavor and mouthfeel, ethanol drastically influences the physical properties of beer, dictating how easily volatile compounds escape the beer matrix to contribute to beer aroma 69 . Importantly, it should also be noted that the importance of ethanol for appreciation is likely inflated by the very low appreciation scores of non-alcoholic beers (Supplementary Fig. S4 ). Despite not often being considered a driver of beer appreciation, protein level also ranks highly in both approaches, possibly due to its effect on mouthfeel and body 70 . Lactic acid, which contributes to the tart taste of sour beers, is the fourth most important parameter identified by SHAP, possibly due to the generally high appreciation of sour beers in our dataset.

Interestingly, some of the most important predictive parameters for our model are not well-established as beer flavors or are even commonly regarded as being negative for beer quality. For example, our models identify methanethiol and ethyl phenyl acetate, an ester commonly linked to beer staling 71 , as a key factor contributing to beer appreciation. Although there is no doubt that high concentrations of these compounds are considered unpleasant, the positive effects of modest concentrations are not yet known 72 , 73 .

To compare our approach to conventional statistics, we evaluated how well the 15 most important SHAP-derived parameters correlate with consumer appreciation (Fig. 4C ). Interestingly, only 6 of the properties derived by SHAP rank amongst the top 15 most correlated parameters. For some chemical compounds, the correlations are so low that they would have likely been considered unimportant. For example, lactic acid, the fourth most important parameter, shows a bimodal distribution for appreciation, with sour beers forming a separate cluster, that is missed entirely by the Spearman correlation. Additionally, the correlation plots reveal outliers, emphasizing the need for robust analysis tools. Together, this highlights the need for alternative models, like the Gradient Boosting model, that better grasp the complexity of (beer) flavor.

Finally, to observe the relationships between these chemical properties and their predicted targets, partial dependence plots were constructed for the six most important predictors of consumer appreciation 74 , 75 , 76 (Supplementary Fig. S7 ). One-way partial dependence plots show how a change in concentration affects the predicted appreciation. These plots reveal an important limitation of our models: appreciation predictions remain constant at ever-increasing concentrations. This implies that once a threshold concentration is reached, further increasing the concentration does not affect appreciation. This is false, as it is well-documented that certain compounds become unpleasant at high concentrations, including ethyl acetate (‘nail polish’) 77 and methanethiol (‘sulfury’ and ‘rotten cabbage’) 78 . The inability of our models to grasp that flavor compounds have optimal levels, above which they become negative, is a consequence of working with commercial beer brands where (off-)flavors are rarely too high to negatively impact the product. The two-way partial dependence plots show how changing the concentration of two compounds influences predicted appreciation, visualizing their interactions (Supplementary Fig. S7 ). In our case, the top 5 parameters are dominated by additive or synergistic interactions, with high concentrations for both compounds resulting in the highest predicted appreciation.

To assess the robustness of our best-performing models and model predictions, we performed 100 iterations of the GBR, RF and ET models. In general, all iterations of the models yielded similar performance (Supplementary Fig. S8 ). Moreover, the main predictors (including the top predictors ethanol and ethyl acetate) remained virtually the same, especially for GBR and RF. For the iterations of the ET model, we did observe more variation in the top predictors, which is likely a consequence of the model’s inherent random architecture in combination with co-correlations between certain predictors. However, even in this case, several of the top predictors (ethanol and ethyl acetate) remain unchanged, although their rank in importance changes (Supplementary Fig. S8 ).

Next, we investigated if a combination of RateBeer and trained panel data into one consolidated dataset would lead to stronger models, under the hypothesis that such a model would suffer less from bias in the datasets. A GBR model was trained to predict appreciation on the combined dataset. This model underperformed compared to the RateBeer model, both in the native case and when including a dataset identifier (R 2 = 0.67, 0.26 and 0.42 respectively). For the latter, the dataset identifier is the most important feature (Supplementary Fig. S9 ), while most of the feature importance remains unchanged, with ethyl acetate and ethanol ranking highest, like in the original model trained only on RateBeer data. It seems that the large variation in the panel dataset introduces noise, weakening the models’ performances and reliability. In addition, it seems reasonable to assume that both datasets are fundamentally different, with the panel dataset obtained by blind tastings by a trained professional panel.

Lastly, we evaluated whether beer style identifiers would further enhance the model’s performance. A GBR model was trained with parameters that explicitly encoded the styles of the samples. This did not improve model performance (R2 = 0.66 with style information vs R2 = 0.67). The most important chemical features are consistent with the model trained without style information (eg. ethanol and ethyl acetate), and with the exception of the most preferred (strong ale) and least preferred (low/no-alcohol) styles, none of the styles were among the most important features (Supplementary Fig. S9 , Supplementary Table S5 and S6 ). This is likely due to a combination of style-specific chemical signatures, such as iso-alpha acids and lactic acid, that implicitly convey style information to the original models, as well as the low number of samples belonging to some styles, making it difficult for the model to learn style-specific patterns. Moreover, beer styles are not rigorously defined, with some styles overlapping in features and some beers being misattributed to a specific style, all of which leads to more noise in models that use style parameters.

Model validation

To test if our predictive models give insight into beer appreciation, we set up experiments aimed at improving existing commercial beers. We specifically selected overall appreciation as the trait to be examined because of its complexity and commercial relevance. Beer flavor comprises a complex bouquet rather than single aromas and tastes 53 . Hence, adding a single compound to the extent that a difference is noticeable may lead to an unbalanced, artificial flavor. Therefore, we evaluated the effect of combinations of compounds. Because Blond beers represent the most extensive style in our dataset, we selected a beer from this style as the starting material for these experiments (Beer 64 in Supplementary Data 1 ).

In the first set of experiments, we adjusted the concentrations of compounds that made up the most important predictors of overall appreciation (ethyl acetate, ethanol, lactic acid, ethyl phenyl acetate) together with correlated compounds (ethyl hexanoate, isoamyl acetate, glycerol), bringing them up to 95 th percentile ethanol-normalized concentrations (Methods) within the Blond group (‘Spiked’ concentration in Fig. 5A ). Compared to controls, the spiked beers were found to have significantly improved overall appreciation among trained panelists, with panelist noting increased intensity of ester flavors, sweetness, alcohol, and body fullness (Fig. 5B ). To disentangle the contribution of ethanol to these results, a second experiment was performed without the addition of ethanol. This resulted in a similar outcome, including increased perception of alcohol and overall appreciation.

Adding the top chemical compounds, identified as best predictors of appreciation by our model, into poorly appreciated beers results in increased appreciation from our trained panel. Results of sensory tests between base beers and those spiked with compounds identified as the best predictors by the model. A Blond and Non/Low-alcohol (0.0% ABV) base beers were brought up to 95th-percentile ethanol-normalized concentrations within each style. B For each sensory attribute, tasters indicated the more intense sample and selected the sample they preferred. The numbers above the bars correspond to the p values that indicate significant changes in perceived flavor (two-sided binomial test: alpha 0.05, n = 20 or 13).

In a last experiment, we tested whether using the model’s predictions can boost the appreciation of a non-alcoholic beer (beer 223 in Supplementary Data 1 ). Again, the addition of a mixture of predicted compounds (omitting ethanol, in this case) resulted in a significant increase in appreciation, body, ester flavor and sweetness.

Predicting flavor and consumer appreciation from chemical composition is one of the ultimate goals of sensory science. A reliable, systematic and unbiased way to link chemical profiles to flavor and food appreciation would be a significant asset to the food and beverage industry. Such tools would substantially aid in quality control and recipe development, offer an efficient and cost-effective alternative to pilot studies and consumer trials and would ultimately allow food manufacturers to produce superior, tailor-made products that better meet the demands of specific consumer groups more efficiently.

A limited set of studies have previously tried, to varying degrees of success, to predict beer flavor and beer popularity based on (a limited set of) chemical compounds and flavors 79 , 80 . Current sensitive, high-throughput technologies allow measuring an unprecedented number of chemical compounds and properties in a large set of samples, yielding a dataset that can train models that help close the gaps between chemistry and flavor, even for a complex natural product like beer. To our knowledge, no previous research gathered data at this scale (250 samples, 226 chemical parameters, 50 sensory attributes and 5 consumer scores) to disentangle and validate the chemical aspects driving beer preference using various machine-learning techniques. We find that modern machine learning models outperform conventional statistical tools, such as correlations and linear models, and can successfully predict flavor appreciation from chemical composition. This could be attributed to the natural incorporation of interactions and non-linear or discontinuous effects in machine learning models, which are not easily grasped by the linear model architecture. While linear models and partial least squares regression represent the most widespread statistical approaches in sensory science, in part because they allow interpretation 65 , 81 , 82 , modern machine learning methods allow for building better predictive models while preserving the possibility to dissect and exploit the underlying patterns. Of the 10 different models we trained, tree-based models, such as our best performing GBR, showed the best overall performance in predicting sensory responses from chemical information, outcompeting artificial neural networks. This agrees with previous reports for models trained on tabular data 83 . Our results are in line with the findings of Colantonio et al. who also identified the gradient boosting architecture as performing best at predicting appreciation and flavor (of tomatoes and blueberries, in their specific study) 26 . Importantly, besides our larger experimental scale, we were able to directly confirm our models’ predictions in vivo.

Our study confirms that flavor compound concentration does not always correlate with perception, suggesting complex interactions that are often missed by more conventional statistics and simple models. Specifically, we find that tree-based algorithms may perform best in developing models that link complex food chemistry with aroma. Furthermore, we show that massive datasets of untrained consumer reviews provide a valuable source of data, that can complement or even replace trained tasting panels, especially for appreciation and basic flavors, such as sweetness and bitterness. This holds despite biases that are known to occur in such datasets, such as price or conformity bias. Moreover, GBR models predict taste better than aroma. This is likely because taste (e.g. bitterness) often directly relates to the corresponding chemical measurements (e.g., iso-alpha acids), whereas such a link is less clear for aromas, which often result from the interplay between multiple volatile compounds. We also find that our models are best at predicting acidity and alcohol, likely because there is a direct relation between the measured chemical compounds (acids and ethanol) and the corresponding perceived sensorial attribute (acidity and alcohol), and because even untrained consumers are generally able to recognize these flavors and aromas.

The predictions of our final models, trained on review data, hold even for blind tastings with small groups of trained tasters, as demonstrated by our ability to validate specific compounds as drivers of beer flavor and appreciation. Since adding a single compound to the extent of a noticeable difference may result in an unbalanced flavor profile, we specifically tested our identified key drivers as a combination of compounds. While this approach does not allow us to validate if a particular single compound would affect flavor and/or appreciation, our experiments do show that this combination of compounds increases consumer appreciation.

It is important to stress that, while it represents an important step forward, our approach still has several major limitations. A key weakness of the GBR model architecture is that amongst co-correlating variables, the largest main effect is consistently preferred for model building. As a result, co-correlating variables often have artificially low importance scores, both for impurity and SHAP-based methods, like we observed in the comparison to the more randomized Extra Trees models. This implies that chemicals identified as key drivers of a specific sensory feature by GBR might not be the true causative compounds, but rather co-correlate with the actual causative chemical. For example, the high importance of ethyl acetate could be (partially) attributed to the total ester content, ethanol or ethyl hexanoate (rho=0.77, rho=0.72 and rho=0.68), while ethyl phenylacetate could hide the importance of prenyl isobutyrate and ethyl benzoate (rho=0.77 and rho=0.76). Expanding our GBR model to include beer style as a parameter did not yield additional power or insight. This is likely due to style-specific chemical signatures, such as iso-alpha acids and lactic acid, that implicitly convey style information to the original model, as well as the smaller sample size per style, limiting the power to uncover style-specific patterns. This can be partly attributed to the curse of dimensionality, where the high number of parameters results in the models mainly incorporating single parameter effects, rather than complex interactions such as style-dependent effects 67 . A larger number of samples may overcome some of these limitations and offer more insight into style-specific effects. On the other hand, beer style is not a rigid scientific classification, and beers within one style often differ a lot, which further complicates the analysis of style as a model factor.

Our study is limited to beers from Belgian breweries. Although these beers cover a large portion of the beer styles available globally, some beer styles and consumer patterns may be missing, while other features might be overrepresented. For example, many Belgian ales exhibit yeast-driven flavor profiles, which is reflected in the chemical drivers of appreciation discovered by this study. In future work, expanding the scope to include diverse markets and beer styles could lead to the identification of even more drivers of appreciation and better models for special niche products that were not present in our beer set.

In addition to inherent limitations of GBR models, there are also some limitations associated with studying food aroma. Even if our chemical analyses measured most of the known aroma compounds, the total number of flavor compounds in complex foods like beer is still larger than the subset we were able to measure in this study. For example, hop-derived thiols, that influence flavor at very low concentrations, are notoriously difficult to measure in a high-throughput experiment. Moreover, consumer perception remains subjective and prone to biases that are difficult to avoid. It is also important to stress that the models are still immature and that more extensive datasets will be crucial for developing more complete models in the future. Besides more samples and parameters, our dataset does not include any demographic information about the tasters. Including such data could lead to better models that grasp external factors like age and culture. Another limitation is that our set of beers consists of high-quality end-products and lacks beers that are unfit for sale, which limits the current model in accurately predicting products that are appreciated very badly. Finally, while models could be readily applied in quality control, their use in sensory science and product development is restrained by their inability to discern causal relationships. Given that the models cannot distinguish compounds that genuinely drive consumer perception from those that merely correlate, validation experiments are essential to identify true causative compounds.

Despite the inherent limitations, dissection of our models enabled us to pinpoint specific molecules as potential drivers of beer aroma and consumer appreciation, including compounds that were unexpected and would not have been identified using standard approaches. Important drivers of beer appreciation uncovered by our models include protein levels, ethyl acetate, ethyl phenyl acetate and lactic acid. Currently, many brewers already use lactic acid to acidify their brewing water and ensure optimal pH for enzymatic activity during the mashing process. Our results suggest that adding lactic acid can also improve beer appreciation, although its individual effect remains to be tested. Interestingly, ethanol appears to be unnecessary to improve beer appreciation, both for blond beer and alcohol-free beer. Given the growing consumer interest in alcohol-free beer, with a predicted annual market growth of >7% 84 , it is relevant for brewers to know what compounds can further increase consumer appreciation of these beers. Hence, our model may readily provide avenues to further improve the flavor and consumer appreciation of both alcoholic and non-alcoholic beers, which is generally considered one of the key challenges for future beer production.

Whereas we see a direct implementation of our results for the development of superior alcohol-free beverages and other food products, our study can also serve as a stepping stone for the development of novel alcohol-containing beverages. We want to echo the growing body of scientific evidence for the negative effects of alcohol consumption, both on the individual level by the mutagenic, teratogenic and carcinogenic effects of ethanol 85 , 86 , as well as the burden on society caused by alcohol abuse and addiction. We encourage the use of our results for the production of healthier, tastier products, including novel and improved beverages with lower alcohol contents. Furthermore, we strongly discourage the use of these technologies to improve the appreciation or addictive properties of harmful substances.

The present work demonstrates that despite some important remaining hurdles, combining the latest developments in chemical analyses, sensory analysis and modern machine learning methods offers exciting avenues for food chemistry and engineering. Soon, these tools may provide solutions in quality control and recipe development, as well as new approaches to sensory science and flavor research.

Beer selection

250 commercial Belgian beers were selected to cover the broad diversity of beer styles and corresponding diversity in chemical composition and aroma. See Supplementary Fig. S1 .

Chemical dataset

Sample preparation.

Beers within their expiration date were purchased from commercial retailers. Samples were prepared in biological duplicates at room temperature, unless explicitly stated otherwise. Bottle pressure was measured with a manual pressure device (Steinfurth Mess-Systeme GmbH) and used to calculate CO 2 concentration. The beer was poured through two filter papers (Macherey-Nagel, 500713032 MN 713 ¼) to remove carbon dioxide and prevent spontaneous foaming. Samples were then prepared for measurements by targeted Headspace-Gas Chromatography-Flame Ionization Detector/Flame Photometric Detector (HS-GC-FID/FPD), Headspace-Solid Phase Microextraction-Gas Chromatography-Mass Spectrometry (HS-SPME-GC-MS), colorimetric analysis, enzymatic analysis, Near-Infrared (NIR) analysis, as described in the sections below. The mean values of biological duplicates are reported for each compound.

HS-GC-FID/FPD

HS-GC-FID/FPD (Shimadzu GC 2010 Plus) was used to measure higher alcohols, acetaldehyde, esters, 4-vinyl guaicol, and sulfur compounds. Each measurement comprised 5 ml of sample pipetted into a 20 ml glass vial containing 1.75 g NaCl (VWR, 27810.295). 100 µl of 2-heptanol (Sigma-Aldrich, H3003) (internal standard) solution in ethanol (Fisher Chemical, E/0650DF/C17) was added for a final concentration of 2.44 mg/L. Samples were flushed with nitrogen for 10 s, sealed with a silicone septum, stored at −80 °C and analyzed in batches of 20.

The GC was equipped with a DB-WAXetr column (length, 30 m; internal diameter, 0.32 mm; layer thickness, 0.50 µm; Agilent Technologies, Santa Clara, CA, USA) to the FID and an HP-5 column (length, 30 m; internal diameter, 0.25 mm; layer thickness, 0.25 µm; Agilent Technologies, Santa Clara, CA, USA) to the FPD. N 2 was used as the carrier gas. Samples were incubated for 20 min at 70 °C in the headspace autosampler (Flow rate, 35 cm/s; Injection volume, 1000 µL; Injection mode, split; Combi PAL autosampler, CTC analytics, Switzerland). The injector, FID and FPD temperatures were kept at 250 °C. The GC oven temperature was first held at 50 °C for 5 min and then allowed to rise to 80 °C at a rate of 5 °C/min, followed by a second ramp of 4 °C/min until 200 °C kept for 3 min and a final ramp of (4 °C/min) until 230 °C for 1 min. Results were analyzed with the GCSolution software version 2.4 (Shimadzu, Kyoto, Japan). The GC was calibrated with a 5% EtOH solution (VWR International) containing the volatiles under study (Supplementary Table S7 ).

HS-SPME-GC-MS

HS-SPME-GC-MS (Shimadzu GCMS-QP-2010 Ultra) was used to measure additional volatile compounds, mainly comprising terpenoids and esters. Samples were analyzed by HS-SPME using a triphase DVB/Carboxen/PDMS 50/30 μm SPME fiber (Supelco Co., Bellefonte, PA, USA) followed by gas chromatography (Thermo Fisher Scientific Trace 1300 series, USA) coupled to a mass spectrometer (Thermo Fisher Scientific ISQ series MS) equipped with a TriPlus RSH autosampler. 5 ml of degassed beer sample was placed in 20 ml vials containing 1.75 g NaCl (VWR, 27810.295). 5 µl internal standard mix was added, containing 2-heptanol (1 g/L) (Sigma-Aldrich, H3003), 4-fluorobenzaldehyde (1 g/L) (Sigma-Aldrich, 128376), 2,3-hexanedione (1 g/L) (Sigma-Aldrich, 144169) and guaiacol (1 g/L) (Sigma-Aldrich, W253200) in ethanol (Fisher Chemical, E/0650DF/C17). Each sample was incubated at 60 °C in the autosampler oven with constant agitation. After 5 min equilibration, the SPME fiber was exposed to the sample headspace for 30 min. The compounds trapped on the fiber were thermally desorbed in the injection port of the chromatograph by heating the fiber for 15 min at 270 °C.

The GC-MS was equipped with a low polarity RXi-5Sil MS column (length, 20 m; internal diameter, 0.18 mm; layer thickness, 0.18 µm; Restek, Bellefonte, PA, USA). Injection was performed in splitless mode at 320 °C, a split flow of 9 ml/min, a purge flow of 5 ml/min and an open valve time of 3 min. To obtain a pulsed injection, a programmed gas flow was used whereby the helium gas flow was set at 2.7 mL/min for 0.1 min, followed by a decrease in flow of 20 ml/min to the normal 0.9 mL/min. The temperature was first held at 30 °C for 3 min and then allowed to rise to 80 °C at a rate of 7 °C/min, followed by a second ramp of 2 °C/min till 125 °C and a final ramp of 8 °C/min with a final temperature of 270 °C.

Mass acquisition range was 33 to 550 amu at a scan rate of 5 scans/s. Electron impact ionization energy was 70 eV. The interface and ion source were kept at 275 °C and 250 °C, respectively. A mix of linear n-alkanes (from C7 to C40, Supelco Co.) was injected into the GC-MS under identical conditions to serve as external retention index markers. Identification and quantification of the compounds were performed using an in-house developed R script as described in Goelen et al. and Reher et al. 87 , 88 (for package information, see Supplementary Table S8 ). Briefly, chromatograms were analyzed using AMDIS (v2.71) 89 to separate overlapping peaks and obtain pure compound spectra. The NIST MS Search software (v2.0 g) in combination with the NIST2017, FFNSC3 and Adams4 libraries were used to manually identify the empirical spectra, taking into account the expected retention time. After background subtraction and correcting for retention time shifts between samples run on different days based on alkane ladders, compound elution profiles were extracted and integrated using a file with 284 target compounds of interest, which were either recovered in our identified AMDIS list of spectra or were known to occur in beer. Compound elution profiles were estimated for every peak in every chromatogram over a time-restricted window using weighted non-negative least square analysis after which peak areas were integrated 87 , 88 . Batch effect correction was performed by normalizing against the most stable internal standard compound, 4-fluorobenzaldehyde. Out of all 284 target compounds that were analyzed, 167 were visually judged to have reliable elution profiles and were used for final analysis.

Discrete photometric and enzymatic analysis

Discrete photometric and enzymatic analysis (Thermo Scientific TM Gallery TM Plus Beermaster Discrete Analyzer) was used to measure acetic acid, ammonia, beta-glucan, iso-alpha acids, color, sugars, glycerol, iron, pH, protein, and sulfite. 2 ml of sample volume was used for the analyses. Information regarding the reagents and standard solutions used for analyses and calibrations is included in Supplementary Table S7 and Supplementary Table S9 .

NIR analyses

NIR analysis (Anton Paar Alcolyzer Beer ME System) was used to measure ethanol. Measurements comprised 50 ml of sample, and a 10% EtOH solution was used for calibration.

Correlation calculations

Pairwise Spearman Rank correlations were calculated between all chemical properties.

Sensory dataset

Trained panel.

Our trained tasting panel consisted of volunteers who gave prior verbal informed consent. All compounds used for the validation experiment were of food-grade quality. The tasting sessions were approved by the Social and Societal Ethics Committee of the KU Leuven (G-2022-5677-R2(MAR)). All online reviewers agreed to the Terms and Conditions of the RateBeer website.

Sensory analysis was performed according to the American Society of Brewing Chemists (ASBC) Sensory Analysis Methods 90 . 30 volunteers were screened through a series of triangle tests. The sixteen most sensitive and consistent tasters were retained as taste panel members. The resulting panel was diverse in age [22–42, mean: 29], sex [56% male] and nationality [7 different countries]. The panel developed a consensus vocabulary to describe beer aroma, taste and mouthfeel. Panelists were trained to identify and score 50 different attributes, using a 7-point scale to rate attributes’ intensity. The scoring sheet is included as Supplementary Data 3 . Sensory assessments took place between 10–12 a.m. The beers were served in black-colored glasses. Per session, between 5 and 12 beers of the same style were tasted at 12 °C to 16 °C. Two reference beers were added to each set and indicated as ‘Reference 1 & 2’, allowing panel members to calibrate their ratings. Not all panelists were present at every tasting. Scores were scaled by standard deviation and mean-centered per taster. Values are represented as z-scores and clustered by Euclidean distance. Pairwise Spearman correlations were calculated between taste and aroma sensory attributes. Panel consistency was evaluated by repeating samples on different sessions and performing ANOVA to identify differences, using the ‘stats’ package (v4.2.2) in R (for package information, see Supplementary Table S8 ).

Online reviews from a public database

The ‘scrapy’ package in Python (v3.6) (for package information, see Supplementary Table S8 ). was used to collect 232,288 online reviews (mean=922, min=6, max=5343) from RateBeer, an online beer review database. Each review entry comprised 5 numerical scores (appearance, aroma, taste, palate and overall quality) and an optional review text. The total number of reviews per reviewer was collected separately. Numerical scores were scaled and centered per rater, and mean scores were calculated per beer.

For the review texts, the language was estimated using the packages ‘langdetect’ and ‘langid’ in Python. Reviews that were classified as English by both packages were kept. Reviewers with fewer than 100 entries overall were discarded. 181,025 reviews from >6000 reviewers from >40 countries remained. Text processing was done using the ‘nltk’ package in Python. Texts were corrected for slang and misspellings; proper nouns and rare words that are relevant to the beer context were specified and kept as-is (‘Chimay’,’Lambic’, etc.). A dictionary of semantically similar sensorial terms, for example ‘floral’ and ‘flower’, was created and collapsed together into one term. Words were stemmed and lemmatized to avoid identifying words such as ‘acid’ and ‘acidity’ as separate terms. Numbers and punctuation were removed.

Sentences from up to 50 randomly chosen reviews per beer were manually categorized according to the aspect of beer they describe (appearance, aroma, taste, palate, overall quality—not to be confused with the 5 numerical scores described above) or flagged as irrelevant if they contained no useful information. If a beer contained fewer than 50 reviews, all reviews were manually classified. This labeled data set was used to train a model that classified the rest of the sentences for all beers 91 . Sentences describing taste and aroma were extracted, and term frequency–inverse document frequency (TFIDF) was implemented to calculate enrichment scores for sensorial words per beer.

The sex of the tasting subject was not considered when building our sensory database. Instead, results from different panelists were averaged, both for our trained panel (56% male, 44% female) and the RateBeer reviews (70% male, 30% female for RateBeer as a whole).

Beer price collection and processing

Beer prices were collected from the following stores: Colruyt, Delhaize, Total Wine, BeerHawk, The Belgian Beer Shop, The Belgian Shop, and Beer of Belgium. Where applicable, prices were converted to Euros and normalized per liter. Spearman correlations were calculated between these prices and mean overall appreciation scores from RateBeer and the taste panel, respectively.

Pairwise Spearman Rank correlations were calculated between all sensory properties.

Machine learning models

Predictive modeling of sensory profiles from chemical data.

Regression models were constructed to predict (a) trained panel scores for beer flavors and quality from beer chemical profiles and (b) public reviews’ appreciation scores from beer chemical profiles. Z-scores were used to represent sensory attributes in both data sets. Chemical properties with log-normal distributions (Shapiro-Wilk test, p < 0.05 ) were log-transformed. Missing chemical measurements (0.1% of all data) were replaced with mean values per attribute. Observations from 250 beers were randomly separated into a training set (70%, 175 beers) and a test set (30%, 75 beers), stratified per beer style. Chemical measurements (p = 231) were normalized based on the training set average and standard deviation. In total, three linear regression-based models: linear regression with first-order interaction terms (LR), lasso regression with first-order interaction terms (Lasso) and partial least squares regression (PLSR); five decision tree models, Adaboost regressor (ABR), Extra Trees (ET), Gradient Boosting regressor (GBR), Random Forest (RF) and XGBoost regressor (XGBR); one support vector machine model (SVR) and one artificial neural network model (ANN) were trained. The models were implemented using the ‘scikit-learn’ package (v1.2.2) and ‘xgboost’ package (v1.7.3) in Python (v3.9.16). Models were trained, and hyperparameters optimized, using five-fold cross-validated grid search with the coefficient of determination (R 2 ) as the evaluation metric. The ANN (scikit-learn’s MLPRegressor) was optimized using Bayesian Tree-Structured Parzen Estimator optimization with the ‘Optuna’ Python package (v3.2.0). Individual models were trained per attribute, and a multi-output model was trained on all attributes simultaneously.

Model dissection

GBR was found to outperform other methods, resulting in models with the highest average R 2 values in both trained panel and public review data sets. Impurity-based rankings of the most important predictors for each predicted sensorial trait were obtained using the ‘scikit-learn’ package. To observe the relationships between these chemical properties and their predicted targets, partial dependence plots (PDP) were constructed for the six most important predictors of consumer appreciation 74 , 75 .

The ‘SHAP’ package in Python (v0.41.0) was implemented to provide an alternative ranking of predictor importance and to visualize the predictors’ effects as a function of their concentration 68 .

Validation of causal chemical properties

To validate the effects of the most important model features on predicted sensory attributes, beers were spiked with the chemical compounds identified by the models and descriptive sensory analyses were carried out according to the American Society of Brewing Chemists (ASBC) protocol 90 .

Compound spiking was done 30 min before tasting. Compounds were spiked into fresh beer bottles, that were immediately resealed and inverted three times. Fresh bottles of beer were opened for the same duration, resealed, and inverted thrice, to serve as controls. Pairs of spiked samples and controls were served simultaneously, chilled and in dark glasses as outlined in the Trained panel section above. Tasters were instructed to select the glass with the higher flavor intensity for each attribute (directional difference test 92 ) and to select the glass they prefer.

The final concentration after spiking was equal to the within-style average, after normalizing by ethanol concentration. This was done to ensure balanced flavor profiles in the final spiked beer. The same methods were applied to improve a non-alcoholic beer. Compounds were the following: ethyl acetate (Merck KGaA, W241415), ethyl hexanoate (Merck KGaA, W243906), isoamyl acetate (Merck KGaA, W205508), phenethyl acetate (Merck KGaA, W285706), ethanol (96%, Colruyt), glycerol (Merck KGaA, W252506), lactic acid (Merck KGaA, 261106).

Significant differences in preference or perceived intensity were determined by performing the two-sided binomial test on each attribute.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Data availability

The data that support the findings of this work are available in the Supplementary Data files and have been deposited to Zenodo under accession code 10653704 93 . The RateBeer scores data are under restricted access, they are not publicly available as they are property of RateBeer (ZX Ventures, USA). Access can be obtained from the authors upon reasonable request and with permission of RateBeer (ZX Ventures, USA). Source data are provided with this paper.

Code availability

The code for training the machine learning models, analyzing the models, and generating the figures has been deposited to Zenodo under accession code 10653704 93 .

Tieman, D. et al. A chemical genetic roadmap to improved tomato flavor. Science 355 , 391–394 (2017).

Article ADS CAS PubMed Google Scholar

Plutowska, B. & Wardencki, W. Application of gas chromatography–olfactometry (GC–O) in analysis and quality assessment of alcoholic beverages – A review. Food Chem. 107 , 449–463 (2008).

Article CAS Google Scholar

Legin, A., Rudnitskaya, A., Seleznev, B. & Vlasov, Y. Electronic tongue for quality assessment of ethanol, vodka and eau-de-vie. Anal. Chim. Acta 534 , 129–135 (2005).

Loutfi, A., Coradeschi, S., Mani, G. K., Shankar, P. & Rayappan, J. B. B. Electronic noses for food quality: A review. J. Food Eng. 144 , 103–111 (2015).

Ahn, Y.-Y., Ahnert, S. E., Bagrow, J. P. & Barabási, A.-L. Flavor network and the principles of food pairing. Sci. Rep. 1 , 196 (2011).

Article CAS PubMed PubMed Central Google Scholar

Bartoshuk, L. M. & Klee, H. J. Better fruits and vegetables through sensory analysis. Curr. Biol. 23 , R374–R378 (2013).

Article CAS PubMed Google Scholar

Piggott, J. R. Design questions in sensory and consumer science. Food Qual. Prefer. 3293 , 217–220 (1995).

Article Google Scholar

Kermit, M. & Lengard, V. Assessing the performance of a sensory panel-panellist monitoring and tracking. J. Chemom. 19 , 154–161 (2005).

Cook, D. J., Hollowood, T. A., Linforth, R. S. T. & Taylor, A. J. Correlating instrumental measurements of texture and flavour release with human perception. Int. J. Food Sci. Technol. 40 , 631–641 (2005).

Chinchanachokchai, S., Thontirawong, P. & Chinchanachokchai, P. A tale of two recommender systems: The moderating role of consumer expertise on artificial intelligence based product recommendations. J. Retail. Consum. Serv. 61 , 1–12 (2021).

Ross, C. F. Sensory science at the human-machine interface. Trends Food Sci. Technol. 20 , 63–72 (2009).

Chambers, E. IV & Koppel, K. Associations of volatile compounds with sensory aroma and flavor: The complex nature of flavor. Molecules 18 , 4887–4905 (2013).

Pinu, F. R. Metabolomics—The new frontier in food safety and quality research. Food Res. Int. 72 , 80–81 (2015).

Danezis, G. P., Tsagkaris, A. S., Brusic, V. & Georgiou, C. A. Food authentication: state of the art and prospects. Curr. Opin. Food Sci. 10 , 22–31 (2016).

Shepherd, G. M. Smell images and the flavour system in the human brain. Nature 444 , 316–321 (2006).

Meilgaard, M. C. Prediction of flavor differences between beers from their chemical composition. J. Agric. Food Chem. 30 , 1009–1017 (1982).

Xu, L. et al. Widespread receptor-driven modulation in peripheral olfactory coding. Science 368 , eaaz5390 (2020).

Kupferschmidt, K. Following the flavor. Science 340 , 808–809 (2013).

Billesbølle, C. B. et al. Structural basis of odorant recognition by a human odorant receptor. Nature 615 , 742–749 (2023).

Article ADS PubMed PubMed Central Google Scholar

Smith, B. Perspective: Complexities of flavour. Nature 486 , S6–S6 (2012).

Pfister, P. et al. Odorant receptor inhibition is fundamental to odor encoding. Curr. Biol. 30 , 2574–2587 (2020).

Moskowitz, H. W., Kumaraiah, V., Sharma, K. N., Jacobs, H. L. & Sharma, S. D. Cross-cultural differences in simple taste preferences. Science 190 , 1217–1218 (1975).

Eriksson, N. et al. A genetic variant near olfactory receptor genes influences cilantro preference. Flavour 1 , 22 (2012).

Ferdenzi, C. et al. Variability of affective responses to odors: Culture, gender, and olfactory knowledge. Chem. Senses 38 , 175–186 (2013).

Article PubMed Google Scholar

Lawless, H. T. & Heymann, H. Sensory evaluation of food: Principles and practices. (Springer, New York, NY). https://doi.org/10.1007/978-1-4419-6488-5 (2010).

Colantonio, V. et al. Metabolomic selection for enhanced fruit flavor. Proc. Natl. Acad. Sci. 119 , e2115865119 (2022).

Fritz, F., Preissner, R. & Banerjee, P. VirtualTaste: a web server for the prediction of organoleptic properties of chemical compounds. Nucleic Acids Res 49 , W679–W684 (2021).

Tuwani, R., Wadhwa, S. & Bagler, G. BitterSweet: Building machine learning models for predicting the bitter and sweet taste of small molecules. Sci. Rep. 9 , 1–13 (2019).

Dagan-Wiener, A. et al. Bitter or not? BitterPredict, a tool for predicting taste from chemical structure. Sci. Rep. 7 , 1–13 (2017).

Pallante, L. et al. Toward a general and interpretable umami taste predictor using a multi-objective machine learning approach. Sci. Rep. 12 , 1–11 (2022).

Malavolta, M. et al. A survey on computational taste predictors. Eur. Food Res. Technol. 248 , 2215–2235 (2022).

Lee, B. K. et al. A principal odor map unifies diverse tasks in olfactory perception. Science 381 , 999–1006 (2023).

Mayhew, E. J. et al. Transport features predict if a molecule is odorous. Proc. Natl. Acad. Sci. 119 , e2116576119 (2022).

Niu, Y. et al. Sensory evaluation of the synergism among ester odorants in light aroma-type liquor by odor threshold, aroma intensity and flash GC electronic nose. Food Res. Int. 113 , 102–114 (2018).

Yu, P., Low, M. Y. & Zhou, W. Design of experiments and regression modelling in food flavour and sensory analysis: A review. Trends Food Sci. Technol. 71 , 202–215 (2018).

Oladokun, O. et al. The impact of hop bitter acid and polyphenol profiles on the perceived bitterness of beer. Food Chem. 205 , 212–220 (2016).

Linforth, R., Cabannes, M., Hewson, L., Yang, N. & Taylor, A. Effect of fat content on flavor delivery during consumption: An in vivo model. J. Agric. Food Chem. 58 , 6905–6911 (2010).

Guo, S., Na Jom, K. & Ge, Y. Influence of roasting condition on flavor profile of sunflower seeds: A flavoromics approach. Sci. Rep. 9 , 11295 (2019).

Ren, Q. et al. The changes of microbial community and flavor compound in the fermentation process of Chinese rice wine using Fagopyrum tataricum grain as feedstock. Sci. Rep. 9 , 3365 (2019).

Hastie, T., Friedman, J. & Tibshirani, R. The Elements of Statistical Learning. (Springer, New York, NY). https://doi.org/10.1007/978-0-387-21606-5 (2001).

Dietz, C., Cook, D., Huismann, M., Wilson, C. & Ford, R. The multisensory perception of hop essential oil: a review. J. Inst. Brew. 126 , 320–342 (2020).

CAS Google Scholar

Roncoroni, Miguel & Verstrepen, Kevin Joan. Belgian Beer: Tested and Tasted. (Lannoo, 2018).

Meilgaard, M. Flavor chemistry of beer: Part II: Flavor and threshold of 239 aroma volatiles. in (1975).

Bokulich, N. A. & Bamforth, C. W. The microbiology of malting and brewing. Microbiol. Mol. Biol. Rev. MMBR 77 , 157–172 (2013).

Dzialo, M. C., Park, R., Steensels, J., Lievens, B. & Verstrepen, K. J. Physiology, ecology and industrial applications of aroma formation in yeast. FEMS Microbiol. Rev. 41 , S95–S128 (2017).

Article PubMed PubMed Central Google Scholar

Datta, A. et al. Computer-aided food engineering. Nat. Food 3 , 894–904 (2022).

American Society of Brewing Chemists. Beer Methods. (American Society of Brewing Chemists, St. Paul, MN, U.S.A.).

Olaniran, A. O., Hiralal, L., Mokoena, M. P. & Pillay, B. Flavour-active volatile compounds in beer: production, regulation and control. J. Inst. Brew. 123 , 13–23 (2017).

Verstrepen, K. J. et al. Flavor-active esters: Adding fruitiness to beer. J. Biosci. Bioeng. 96 , 110–118 (2003).

Meilgaard, M. C. Flavour chemistry of beer. part I: flavour interaction between principal volatiles. Master Brew. Assoc. Am. Tech. Q 12 , 107–117 (1975).

Briggs, D. E., Boulton, C. A., Brookes, P. A. & Stevens, R. Brewing 227–254. (Woodhead Publishing). https://doi.org/10.1533/9781855739062.227 (2004).

Bossaert, S., Crauwels, S., De Rouck, G. & Lievens, B. The power of sour - A review: Old traditions, new opportunities. BrewingScience 72 , 78–88 (2019).

Google Scholar

Verstrepen, K. J. et al. Flavor active esters: Adding fruitiness to beer. J. Biosci. Bioeng. 96 , 110–118 (2003).

Snauwaert, I. et al. Microbial diversity and metabolite composition of Belgian red-brown acidic ales. Int. J. Food Microbiol. 221 , 1–11 (2016).

Spitaels, F. et al. The microbial diversity of traditional spontaneously fermented lambic beer. PLoS ONE 9 , e95384 (2014).

Blanco, C. A., Andrés-Iglesias, C. & Montero, O. Low-alcohol Beers: Flavor Compounds, Defects, and Improvement Strategies. Crit. Rev. Food Sci. Nutr. 56 , 1379–1388 (2016).

Jackowski, M. & Trusek, A. Non-Alcohol. beer Prod. – Overv. 20 , 32–38 (2018).

Takoi, K. et al. The contribution of geraniol metabolism to the citrus flavour of beer: Synergy of geraniol and β-citronellol under coexistence with excess linalool. J. Inst. Brew. 116 , 251–260 (2010).

Kroeze, J. H. & Bartoshuk, L. M. Bitterness suppression as revealed by split-tongue taste stimulation in humans. Physiol. Behav. 35 , 779–783 (1985).

Mennella, J. A. et al. A spoonful of sugar helps the medicine go down”: Bitter masking bysucrose among children and adults. Chem. Senses 40 , 17–25 (2015).

Wietstock, P., Kunz, T., Perreira, F. & Methner, F.-J. Metal chelation behavior of hop acids in buffered model systems. BrewingScience 69 , 56–63 (2016).

Sancho, D., Blanco, C. A., Caballero, I. & Pascual, A. Free iron in pale, dark and alcohol-free commercial lager beers. J. Sci. Food Agric. 91 , 1142–1147 (2011).

Rodrigues, H. & Parr, W. V. Contribution of cross-cultural studies to understanding wine appreciation: A review. Food Res. Int. 115 , 251–258 (2019).

Korneva, E. & Blockeel, H. Towards better evaluation of multi-target regression models. in ECML PKDD 2020 Workshops (eds. Koprinska, I. et al.) 353–362 (Springer International Publishing, Cham, 2020). https://doi.org/10.1007/978-3-030-65965-3_23 .

Gastón Ares. Mathematical and Statistical Methods in Food Science and Technology. (Wiley, 2013).

Grinsztajn, L., Oyallon, E. & Varoquaux, G. Why do tree-based models still outperform deep learning on tabular data? Preprint at http://arxiv.org/abs/2207.08815 (2022).

Gries, S. T. Statistics for Linguistics with R: A Practical Introduction. in Statistics for Linguistics with R (De Gruyter Mouton, 2021). https://doi.org/10.1515/9783110718256 .

Lundberg, S. M. et al. From local explanations to global understanding with explainable AI for trees. Nat. Mach. Intell. 2 , 56–67 (2020).

Ickes, C. M. & Cadwallader, K. R. Effects of ethanol on flavor perception in alcoholic beverages. Chemosens. Percept. 10 , 119–134 (2017).

Kato, M. et al. Influence of high molecular weight polypeptides on the mouthfeel of commercial beer. J. Inst. Brew. 127 , 27–40 (2021).

Wauters, R. et al. Novel Saccharomyces cerevisiae variants slow down the accumulation of staling aldehydes and improve beer shelf-life. Food Chem. 398 , 1–11 (2023).

Li, H., Jia, S. & Zhang, W. Rapid determination of low-level sulfur compounds in beer by headspace gas chromatography with a pulsed flame photometric detector. J. Am. Soc. Brew. Chem. 66 , 188–191 (2008).

Dercksen, A., Laurens, J., Torline, P., Axcell, B. C. & Rohwer, E. Quantitative analysis of volatile sulfur compounds in beer using a membrane extraction interface. J. Am. Soc. Brew. Chem. 54 , 228–233 (1996).

Molnar, C. Interpretable Machine Learning: A Guide for Making Black-Box Models Interpretable. (2020).

Zhao, Q. & Hastie, T. Causal interpretations of black-box models. J. Bus. Econ. Stat. Publ. Am. Stat. Assoc. 39 , 272–281 (2019).

Article MathSciNet Google Scholar

Hastie, T., Tibshirani, R. & Friedman, J. The Elements of Statistical Learning. (Springer, 2019).

Labrado, D. et al. Identification by NMR of key compounds present in beer distillates and residual phases after dealcoholization by vacuum distillation. J. Sci. Food Agric. 100 , 3971–3978 (2020).

Lusk, L. T., Kay, S. B., Porubcan, A. & Ryder, D. S. Key olfactory cues for beer oxidation. J. Am. Soc. Brew. Chem. 70 , 257–261 (2012).

Gonzalez Viejo, C., Torrico, D. D., Dunshea, F. R. & Fuentes, S. Development of artificial neural network models to assess beer acceptability based on sensory properties using a robotic pourer: A comparative model approach to achieve an artificial intelligence system. Beverages 5 , 33 (2019).

Gonzalez Viejo, C., Fuentes, S., Torrico, D. D., Godbole, A. & Dunshea, F. R. Chemical characterization of aromas in beer and their effect on consumers liking. Food Chem. 293 , 479–485 (2019).

Gilbert, J. L. et al. Identifying breeding priorities for blueberry flavor using biochemical, sensory, and genotype by environment analyses. PLOS ONE 10 , 1–21 (2015).

Goulet, C. et al. Role of an esterase in flavor volatile variation within the tomato clade. Proc. Natl. Acad. Sci. 109 , 19009–19014 (2012).

Article ADS CAS PubMed PubMed Central Google Scholar

Borisov, V. et al. Deep Neural Networks and Tabular Data: A Survey. IEEE Trans. Neural Netw. Learn. Syst. 1–21 https://doi.org/10.1109/TNNLS.2022.3229161 (2022).

Statista. Statista Consumer Market Outlook: Beer - Worldwide.

Seitz, H. K. & Stickel, F. Molecular mechanisms of alcoholmediated carcinogenesis. Nat. Rev. Cancer 7 , 599–612 (2007).

Voordeckers, K. et al. Ethanol exposure increases mutation rate through error-prone polymerases. Nat. Commun. 11 , 3664 (2020).

Goelen, T. et al. Bacterial phylogeny predicts volatile organic compound composition and olfactory response of an aphid parasitoid. Oikos 129 , 1415–1428 (2020).

Article ADS Google Scholar

Reher, T. et al. Evaluation of hop (Humulus lupulus) as a repellent for the management of Drosophila suzukii. Crop Prot. 124 , 104839 (2019).

Stein, S. E. An integrated method for spectrum extraction and compound identification from gas chromatography/mass spectrometry data. J. Am. Soc. Mass Spectrom. 10 , 770–781 (1999).

American Society of Brewing Chemists. Sensory Analysis Methods. (American Society of Brewing Chemists, St. Paul, MN, U.S.A., 1992).

McAuley, J., Leskovec, J. & Jurafsky, D. Learning Attitudes and Attributes from Multi-Aspect Reviews. Preprint at https://doi.org/10.48550/arXiv.1210.3926 (2012).

Meilgaard, M. C., Carr, B. T. & Carr, B. T. Sensory Evaluation Techniques. (CRC Press, Boca Raton). https://doi.org/10.1201/b16452 (2014).

Schreurs, M. et al. Data from: Predicting and improving complex beer flavor through machine learning. Zenodo https://doi.org/10.5281/zenodo.10653704 (2024).

Download references

Acknowledgements

We thank all lab members for their discussions and thank all tasting panel members for their contributions. Special thanks go out to Dr. Karin Voordeckers for her tremendous help in proofreading and improving the manuscript. M.S. was supported by a Baillet-Latour fellowship, L.C. acknowledges financial support from KU Leuven (C16/17/006), F.A.T. was supported by a PhD fellowship from FWO (1S08821N). Research in the lab of K.J.V. is supported by KU Leuven, FWO, VIB, VLAIO and the Brewing Science Serves Health Fund. Research in the lab of T.W. is supported by FWO (G.0A51.15) and KU Leuven (C16/17/006).

Author information

These authors contributed equally: Michiel Schreurs, Supinya Piampongsant, Miguel Roncoroni.

Authors and Affiliations

VIB—KU Leuven Center for Microbiology, Gaston Geenslaan 1, B-3001, Leuven, Belgium

Michiel Schreurs, Supinya Piampongsant, Miguel Roncoroni, Lloyd Cool, Beatriz Herrera-Malaver, Florian A. Theßeling & Kevin J. Verstrepen

CMPG Laboratory of Genetics and Genomics, KU Leuven, Gaston Geenslaan 1, B-3001, Leuven, Belgium

Leuven Institute for Beer Research (LIBR), Gaston Geenslaan 1, B-3001, Leuven, Belgium

Laboratory of Socioecology and Social Evolution, KU Leuven, Naamsestraat 59, B-3000, Leuven, Belgium

Lloyd Cool, Christophe Vanderaa & Tom Wenseleers

VIB Bioinformatics Core, VIB, Rijvisschestraat 120, B-9052, Ghent, Belgium

Łukasz Kreft & Alexander Botzki

AB InBev SA/NV, Brouwerijplein 1, B-3000, Leuven, Belgium

Philippe Malcorps & Luk Daenen

You can also search for this author in PubMed Google Scholar

Contributions

S.P., M.S. and K.J.V. conceived the experiments. S.P., M.S. and K.J.V. designed the experiments. S.P., M.S., M.R., B.H. and F.A.T. performed the experiments. S.P., M.S., L.C., C.V., L.K., A.B., P.M., L.D., T.W. and K.J.V. contributed analysis ideas. S.P., M.S., L.C., C.V., T.W. and K.J.V. analyzed the data. All authors contributed to writing the manuscript.

Corresponding author

Correspondence to Kevin J. Verstrepen .

Ethics declarations

Competing interests.

K.J.V. is affiliated with bar.on. The other authors declare no competing interests.

Peer review

Peer review information.

Nature Communications thanks Florian Bauer, Andrew John Macintosh and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information, peer review file, description of additional supplementary files, supplementary data 1, supplementary data 2, supplementary data 3, supplementary data 4, supplementary data 5, supplementary data 6, supplementary data 7, reporting summary, source data, source data, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Schreurs, M., Piampongsant, S., Roncoroni, M. et al. Predicting and improving complex beer flavor through machine learning. Nat Commun 15 , 2368 (2024). https://doi.org/10.1038/s41467-024-46346-0

Download citation

Received : 30 October 2023

Accepted : 21 February 2024

Published : 26 March 2024

DOI : https://doi.org/10.1038/s41467-024-46346-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

Explore articles by subject
Guide to authors
Editorial policies

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

Open access
Published: 12 December 2023

Examining the role of community resilience and social capital on mental health in public health emergency and disaster response: a scoping review

C. E. Hall 1 , 2 ,
H. Wehling 1 ,
J. Stansfield 3 ,
J. South 3 ,
S. K. Brooks 2 ,
N. Greenberg 2 , 4 ,
R. Amlôt 1 &
D. Weston 1

BMC Public Health volume 23 , Article number: 2482 ( 2023 ) Cite this article

1772 Accesses

21 Altmetric

Metrics details

The ability of the public to remain psychologically resilient in the face of public health emergencies and disasters (such as the COVID-19 pandemic) is a key factor in the effectiveness of a national response to such events. Community resilience and social capital are often perceived as beneficial and ensuring that a community is socially and psychologically resilient may aid emergency response and recovery. This review presents a synthesis of literature which answers the following research questions: How are community resilience and social capital quantified in research?; What is the impact of community resilience on mental wellbeing?; What is the impact of infectious disease outbreaks, disasters and emergencies on community resilience and social capital?; and, What types of interventions enhance community resilience and social capital?

A scoping review procedure was followed. Searches were run across Medline, PsycInfo, and EMBASE, with search terms covering both community resilience and social capital, public health emergencies, and mental health. 26 papers met the inclusion criteria.

The majority of retained papers originated in the USA, used a survey methodology to collect data, and involved a natural disaster. There was no common method for measuring community resilience or social capital. The association between community resilience and social capital with mental health was regarded as positive in most cases. However, we found that community resilience, and social capital, were initially negatively impacted by public health emergencies and enhanced by social group activities.

Several key recommendations are proposed based on the outcomes from the review, which include: the need for a standardised and validated approach to measuring both community resilience and social capital; that there should be enhanced effort to improve preparedness to public health emergencies in communities by gauging current levels of community resilience and social capital; that community resilience and social capital should be bolstered if areas are at risk of disasters or public health emergencies; the need to ensure that suitable short-term support is provided to communities with high resilience in the immediate aftermath of a public health emergency or disaster; the importance of conducting robust evaluation of community resilience initiatives deployed during the COVID-19 pandemic.

Peer Review reports

For the general population, public health emergencies and disasters (e.g., natural disasters; infectious disease outbreaks; Chemical, Biological, Radiological or Nuclear incidents) can give rise to a plethora of negative outcomes relating to both health (e.g. increased mental health problems [ 1 , 2 , 3 , 4 ]) and the economy (e.g., increased unemployment and decreased levels of tourism [ 4 , 5 , 6 ]). COVID-19 is a current, and ongoing, example of a public health emergency which has affected over 421 million individuals worldwide [ 7 ]. The long term implications of COVID-19 are not yet known, but there are likely to be repercussions for physical health, mental health, and other non-health related outcomes for a substantial time to come [ 8 , 9 ]. As a result, it is critical to establish methods which may inform approaches to alleviate the longer-term negative consequences that are likely to emerge in the aftermath of both COVID-19 and any future public health emergency.

The definition of resilience often differs within the literature, but ultimately resilience is considered a dynamic process of adaptation. It is related to processes and capabilities at the individual, community and system level that result in good health and social outcomes, in spite of negative events, serious threats and hazards [ 10 ]. Furthermore, Ziglio [ 10 ] refers to four key types of resilience capacity: adaptive, the ability to withstand and adjust to unfavourable conditions and shocks; absorptive, the ability to withstand but also to recover and manage using available assets and skills; anticipatory, the ability to predict and minimize vulnerability; and transformative, transformative change so that systems better cope with new conditions.

There is no one settled definition of community resilience (CR). However, it generally relates to the ability of a community to withstand, adapt and permit growth in adverse circumstances due to social structures, networks and interdependencies within the community [ 11 ]. Social capital (SC) is considered a major determinant of CR [ 12 , 13 ], and reflects strength of a social network, community reciprocity, and trust in people and institutions [ 14 ]. These aspects of community are usually conceptualised primarily as protective factors that enable communities to cope and adapt collectively to threats. SC is often broken down into further categories [ 15 ], for example: cognitive SC (i.e. perceptions of community relations, such as trust, mutual help and attachment) and structural SC (i.e. what actually happens within the community, such as participation, socialising) [ 16 ]; or, bonding SC (i.e. connections among individuals who are emotionally close, and result in bonds to a particular group [ 17 ]) and bridging SC (i.e. acquaintances or individuals loosely connected that span different social groups [ 18 ]). Generally, CR is perceived to be primarily beneficial for multiple reasons (e.g. increased social support [ 18 , 19 ], protection of mental health [ 20 , 21 ]), and strengthening community resilience is a stated health goal of the World Health Organisation [ 22 ] when aiming to alleviate health inequalities and protect wellbeing. This is also reflected by organisations such as Public Health England (now split into the UK Health Security Agency and the Office for Health Improvement and Disparities) [ 23 ] and more recently, CR has been targeted through the endorsement of Community Champions (who are volunteers trained to support and to help improve health and wellbeing. Community Champions also reflect their local communities in terms of population demographics for example age, ethnicity and gender) as part of the COVID-19 response in the UK (e.g. [ 24 , 25 ]).

Despite the vested interest in bolstering communities, the research base establishing: how to understand and measure CR and SC; the effect of CR and SC, both during and following a public health emergency (such as the COVID-19 pandemic); and which types of CR or SC are the most effective to engage, is relatively small. Given the importance of ensuring resilience against, and swift recovery from, public health emergencies, it is critically important to establish and understand the evidence base for these approaches. As a result, the current review sought to answer the following research questions: (1) How are CR and SC quantified in research?; (2) What is the impact of community resilience on mental wellbeing?; (3) What is the impact of infectious disease outbreaks, disasters and emergencies on community resilience and social capital?; and, (4) What types of interventions enhance community resilience and social capital?

By collating research in order to answer these research questions, the authors have been able to propose several key recommendations that could be used to both enhance and evaluate CR and SC effectively to facilitate the long-term recovery from COVID-19, and also to inform the use of CR and SC in any future public health disasters and emergencies.

A scoping review methodology was followed due to the ease of summarising literature on a given topic for policy makers and practitioners [ 26 ], and is detailed in the following sections.

Identification of relevant studies

An initial search strategy was developed by authors CH and DW and included terms which related to: CR and SC, given the absence of a consistent definition of CR, and the link between CR and SC, the review focuses on both CR and SC to identify as much relevant literature as possible (adapted for purpose from Annex 1: [ 27 ], as well as through consultation with review commissioners); public health emergencies and disasters [ 28 , 29 , 30 , 31 ], and psychological wellbeing and recovery (derived a priori from literature). To ensure a focus on both public health and psychological research, the final search was carried across Medline, PsycInfo, and EMBASE using OVID. The final search took place on the 18th of May 2020, the search strategy used for all three databases can be found in Supplementary file 1 .

Selection criteria

The inclusion and exclusion criteria were developed alongside the search strategy. Initially the criteria were relatively inclusive and were subject to iterative development to reflect the authors’ familiarisation with the literature. For example, the decision was taken to exclude research which focused exclusively on social support and did not mention communities as an initial title/abstract search suggested that the majority of this literature did not meet the requirements of our research question.

The full and final inclusion and exclusion criteria used can be found in Supplementary file 2 . In summary, authors decided to focus on the general population (i.e., non-specialist, e.g. non-healthcare worker or government official) to allow the review to remain community focused. The research must also have assessed the impact of CR and/or SC on mental health and wellbeing, resilience, and recovery during and following public health emergencies and infectious disease outbreaks which affect communities (to ensure the research is relevant to the review aims), have conducted primary research, and have a full text available or provided by the first author when contacted.

Charting the data

All papers were first title and abstract screened by CH or DW. Papers then were full text reviewed by CH to ensure each paper met the required eligibility criteria, if unsure about a paper it was also full text reviewed by DW. All papers that were retained post full-text review were subjected to a standardised data extraction procedure. A table was made for the purpose of extracting the following data: title, authors, origin, year of publication, study design, aim, disaster type, sample size and characteristics, variables examined, results, restrictions/limitations, and recommendations. Supplementary file 3 details the charting the data process.

Analytical method

Data was synthesised using a Framework approach [ 32 ], a common method for analysing qualitative research. This method was chosen as it was originally used for large-scale social policy research [ 33 ] as it seeks to identify: what works, for whom, in what conditions, and why [ 34 ]. This approach is also useful for identifying commonalities and differences in qualitative data and potential relationships between different parts of the data [ 33 ]. An a priori framework was established by CH and DW. Extracted data was synthesised in relation to each research question, and the process was iterative to ensure maximum saturation using the available data.

Study selection

The final search strategy yielded 3584 records. Following the removal of duplicates, 2191 records remained and were included in title and abstract screening. A PRISMA flow diagram is presented in Fig. 1 .

PRISMA flow diagram

At the title and abstract screening stage, the process became more iterative as the inclusion criteria were developed and refined. For the first iteration of screening, CH or DW sorted all records into ‘include,’ ‘exclude,’ and ‘unsure’. All ‘unsure’ papers were re-assessed by CH, and a random selection of ~ 20% of these were also assessed by DW. Where there was disagreement between authors the records were retained, and full text screened. The remaining papers were reviewed by CH, and all records were categorised into ‘include’ and ‘exclude’. Following full-text screening, 26 papers were retained for use in the review.

Study characteristics

This section of the review addresses study characteristics of those which met the inclusion criteria, which comprises: date of publication, country of origin, study design, study location, disaster, and variables examined.

Date of publication

Publication dates across the 26 papers spanned from 2008 to 2020 (see Fig. 2 ). The number of papers published was relatively low and consistent across this timescale (i.e. 1–2 per year, except 2010 and 2013 when none were published) up until 2017 where the number of papers peaked at 5. From 2017 to 2020 there were 15 papers published in total. The amount of papers published in recent years suggests a shift in research and interest towards CR and SC in a disaster/ public health emergency context.

Graph to show retained papers date of publication

Country of origin

The locations of the first authors’ institutes at the time of publication were extracted to provide a geographical spread of the retained papers. The majority originated from the USA [ 35 , 36 , 37 , 38 , 39 , 40 , 41 ], followed by China [ 42 , 43 , 44 , 45 , 46 ], Japan [ 47 , 48 , 49 , 50 ], Australia [ 51 , 52 , 53 ], The Netherlands [ 54 , 55 ], New Zealand [ 56 ], Peru [ 57 ], Iran [ 58 ], Austria [ 59 ], and Croatia [ 60 ].

There were multiple methodological approaches carried out across retained papers. The most common formats included surveys or questionnaires [ 36 , 37 , 38 , 42 , 46 , 47 , 48 , 49 , 50 , 53 , 54 , 55 , 57 , 59 ], followed by interviews [ 39 , 40 , 43 , 51 , 52 , 60 ]. Four papers used both surveys and interviews [ 35 , 41 , 45 , 58 ], and two papers conducted data analysis (one using open access data from a Social Survey [ 44 ] and one using a Primary Health Organisations Register [ 56 ]).

Study location

The majority of the studies were carried out in Japan [ 36 , 42 , 44 , 47 , 48 , 49 , 50 ], followed by the USA [ 35 , 37 , 38 , 39 , 40 , 41 ], China [ 43 , 45 , 46 , 53 ], Australia [ 51 , 52 ], and the UK [ 54 , 55 ]. The remaining studies were carried out in Croatia [ 60 ], Peru [ 57 ], Austria [ 59 ], New Zealand [ 56 ] and Iran [ 58 ].

Multiple different types of disaster were researched across the retained papers. Earthquakes were the most common type of disaster examined [ 45 , 47 , 49 , 50 , 53 , 56 , 57 , 58 ], followed by research which assessed the impact of two disastrous events which had happened in the same area (e.g. Hurricane Katrina and the Deepwater Horizon oil spill in Mississippi, and the Great East Japan earthquake and Tsunami; [ 36 , 37 , 38 , 42 , 44 , 48 ]). Other disaster types included: flooding [ 51 , 54 , 55 , 59 , 60 ], hurricanes [ 35 , 39 , 41 ], infectious disease outbreaks [ 43 , 46 ], oil spillage [ 40 ], and drought [ 52 ].

Variables of interest examined

Across the 26 retained papers: eight referred to examining the impact of SC [ 35 , 37 , 39 , 41 , 46 , 49 , 55 , 60 ]; eight examined the impact of cognitive and structural SC as separate entities [ 40 , 42 , 45 , 48 , 50 , 54 , 57 , 59 ]; one examined bridging and bonding SC as separate entities [ 58 ]; two examined the impact of CR [ 38 , 56 ]; and two employed a qualitative methodology but drew findings in relation to bonding and bridging SC, and SC generally [ 51 , 52 ]. Additionally, five papers examined the impact of the following variables: ‘community social cohesion’ [ 36 ], ‘neighbourhood connectedness’ [ 44 ], ‘social support at the community level’ [ 47 ], ‘community connectedness’ [ 43 ] and ‘sense of community’ [ 53 ]. Table 1 provides additional details on this.

How is CR and SC measured or quantified in research?

The measures used to examine CR and SC are presented Table 1 . It is apparent that there is no uniformity in how SC or CR is measured across the research. Multiple measures are used throughout the retained studies, and nearly all are unique. Additionally, SC was examined at multiple different levels (e.g. cognitive and structural, bonding and bridging), and in multiple different forms (e.g. community connectedness, community cohesion).

What is the association between CR and SC on mental wellbeing?

To best compare research, the following section reports on CR, and facets of SC separately. Please see Supplementary file 4 for additional information on retained papers methods of measuring mental wellbeing.

Community resilience

CR relates to the ability of a community to withstand, adapt and permit growth in adverse circumstances due to social structures, networks and interdependencies within the community [ 11 ].

The impact of CR on mental wellbeing was consistently positive. For example, research indicated that there was a positive association between CR and number of common mental health (i.e. anxiety and mood) treatments post-disaster [ 56 ]. Similarly, other research suggests that CR is positively related to psychological resilience, which is inversely related to depressive symptoms) [ 37 ]. The same research also concluded that CR is protective of psychological resilience and is therefore protective of depressive symptoms [ 37 ].

Social capital

SC reflects the strength of a social network, community reciprocity, and trust in people and institutions [ 14 ]. These aspects of community are usually conceptualised primarily as protective factors that enable communities to cope and adapt collectively to threats.

There were inconsistencies across research which examined the impact of abstract SC (i.e. not refined into bonding/bridging or structural/cognitive) on mental wellbeing. However, for the majority of cases, research deems SC to be beneficial. For example, research has concluded that, SC is protective against post-traumatic stress disorder [ 55 ], anxiety [ 46 ], psychological distress [ 50 ], and stress [ 46 ]. Additionally, SC has been found to facilitate post-traumatic growth [ 38 ], and also to be useful to be drawn upon in times of stress [ 52 ], both of which could be protective of mental health. Similarly, research has also found that emotional recovery following a disaster is more difficult for those who report to have low levels of SC [ 51 ].

Conversely, however, research has also concluded that when other situational factors (e.g. personal resources) were controlled for, a positive relationship between community resources and life satisfaction was no longer significant [ 60 ]. Furthermore, some research has concluded that a high level of SC can result in a community facing greater stress immediately post disaster. Indeed, one retained paper found that high levels of SC correlate with higher levels of post-traumatic stress immediately following a disaster [ 39 ]. However, in the later stages following a disaster, this relationship can reverse, with SC subsequently providing an aid to recovery [ 41 ]. By way of explanation, some researchers have suggested that communities with stronger SC carry the greatest load in terms of helping others (i.e. family, friends and neighbours) as well as themselves immediately following the disaster, but then as time passes the communities recover at a faster rate as they are able to rely on their social networks for support [ 41 ].

Cognitive and structural social capital

Cognitive SC refers to perceptions of community relations, such as trust, mutual help and attachment, and structural SC refers to what actually happens within the community, such as participation, socialising [ 16 ].

Cognitive SC has been found to be protective [ 49 ] against PTSD [ 54 , 57 ], depression [ 40 , 54 ]) mild mood disorder; [ 48 ]), anxiety [ 48 , 54 ] and increase self-efficacy [ 59 ].

For structural SC, research is again inconsistent. On the one hand, structural SC has been found to: increase perceived self-efficacy, be protective of depression [ 40 ], buffer the impact of housing damage on cognitive decline [ 42 ] and provide support during disasters and over the recovery period [ 59 ]. However, on the other hand, it has been found to have no association with PTSD [ 54 , 57 ] or depression, and is also associated with a higher prevalence of anxiety [ 54 ]. Similarly, it is also suggested by additional research that structural SC can harm women’s mental health, either due to the pressure of expectations to help and support others or feelings of isolation [ 49 ].

Bonding and bridging social capital

Bonding SC refers to connections among individuals who are emotionally close, and result in bonds to a particular group [ 17 ], and bridging SC refers to acquaintances or individuals loosely connected that span different social groups [ 18 ].

One research study concluded that both bonding and bridging SC were protective against post-traumatic stress disorder symptoms [ 58 ]. Bridging capital was deemed to be around twice as effective in buffering against post-traumatic stress disorder than bonding SC [ 58 ].

Other community variables

Community social cohesion was significantly associated with a lower risk of post-traumatic stress disorder symptom development [ 35 ], and this was apparent even whilst controlling for depressive symptoms at baseline and disaster impact variables (e.g. loss of family member or housing damage) [ 36 ]. Similarly, sense of community, community connectedness, social support at the community level and neighbourhood connectedness all provided protective benefits for a range of mental health, wellbeing and recovery variables, including: depression [ 53 ], subjective wellbeing (in older adults only) [ 43 ], psychological distress [ 47 ], happiness [ 44 ] and life satisfaction [ 53 ].

Research has also concluded that community level social support is protective against mild mood and anxiety disorder, but only for individuals who have had no previous disaster experience [ 48 ]. Additionally, a study which separated SC into social cohesion and social participation concluded that at a community level, social cohesion is protective against depression [ 49 ] whereas social participation at community level is associated with an increased risk of depression amongst women [ 49 ].

What is the impact of Infectious disease outbreaks / disasters and emergencies on community resilience?

From a cross-sectional perspective, research has indicated that disasters and emergencies can have a negative effect on certain types of SC. Specifically, cognitive SC has been found to be impacted by disaster impact, whereas structural SC has gone unaffected [ 45 ]. Disaster impact has also been shown to have a negative effect on community relationships more generally [ 52 ].

Additionally, of the eight studies which collected data at multiple time points [ 35 , 36 , 41 , 42 , 47 , 49 , 56 , 60 ], three reported the effect of a disaster on the level of SC within a community [ 40 , 42 , 49 ]. All three of these studies concluded that disasters may have a negative impact on the levels of SC within a community. The first study found that the Deepwater Horizon oil spill had a negative effect on SC and social support, and this in turn explained an overall increase in the levels of depression within the community [ 40 ]. A possible explanation for the negative effect lays in ‘corrosive communities’, known for increased social conflict and reduced social support, that are sometimes created following oil spills [ 40 ]. It is proposed that corrosive communities often emerge due to a loss of natural resources that bring social groups together (e.g., for recreational activities), as well as social disparity (e.g., due to unequal distribution of economic impact) becoming apparent in the community following disaster [ 40 ]. The second study found that SC (in the form of social cohesion, informal socialising and social participation) decreased after the 2011 earthquake and tsunami in Japan; it was suggested that this change correlated with incidence of cognitive decline [ 42 ]. However, the third study reported more mixed effects based on physical circumstances of the communities’ natural environment: Following an earthquake, those who lived in mountainous areas with an initial high level of pre-community SC saw a decrease in SC post disaster [ 49 ]. However, communities in flat areas (which were home to younger residents and had a higher population density) saw an increase in SC [ 49 ]. It was proposed that this difference could be due to the need for those who lived in mountainous areas to seek prolonged refuge due to subsequent landslides [ 49 ].

What types of intervention enhance CR and SC and protect survivors?

There were mixed effects across the 26 retained papers when examining the effect of CR and SC on mental wellbeing. However, there is evidence that an increase in SC [ 56 , 57 ], with a focus on cognitive SC [ 57 ], namely by: building social networks [ 45 , 51 , 53 ], enhancing feelings of social cohesion [ 35 , 36 ] and promoting a sense of community [ 53 ], can result in an increase in CR and potentially protect survivors’ wellbeing and mental health following a disaster. An increase in SC may also aid in decreasing the need for individual psychological interventions in the aftermath of a disaster [ 55 ]. As a result, recommendations and suggested methods to bolster CR and SC from the retained papers have been extracted and separated into general methods, preparedness and policy level implementation.

General methods

Suggested methods to build SC included organising recreational activity-based groups [ 44 ] to broaden [ 51 , 53 ] and preserve current social networks [ 42 ], introducing initiatives to increase social cohesion and trust [ 51 ], and volunteering to increase the number of social ties between residents [ 59 ]. Research also notes that it is important to take a ‘no one left behind approach’ when organising recreational and social community events, as failure to do so could induce feelings of isolation for some members of the community [ 49 ]. Furthermore, gender differences should also be considered as research indicates that males and females may react differently to community level SC (as evidence suggests males are instead more impacted by individual level SC; in comparison to women who have larger and more diverse social networks [ 49 ]). Therefore, interventions which aim to raise community level social participation, with the aim of expanding social connections and gaining support, may be beneficial [ 42 , 47 ].

Preparedness

In order to prepare for disasters, it may be beneficial to introduce community-targeted methods or interventions to increase levels of SC and CR as these may aid in ameliorating the consequences of a public health emergency or disaster [ 57 ]. To indicate which communities have low levels of SC, one study suggests implementing a 3-item scale of social cohesion to map areas and target interventions [ 42 ].

It is important to consider that communities with a high level of SC may have a lower level of risk perception, due to the established connections and supportive network they have with those around them [ 61 ]. However, for the purpose of preparedness, this is not ideal as perception of risk is a key factor when seeking to encourage behavioural adherence. This could be overcome by introducing communication strategies which emphasise the necessity of social support, but also highlights the need for additional measures to reduce residual risk [ 59 ]. Furthermore, support in the form of financial assistance to foster current community initiatives may prove beneficial to rural areas, for example through the use of an asset-based community development framework [ 52 ].

Policy level

At a policy level, the included papers suggest a range of ways that CR and SC could be bolstered and used. These include: providing financial support for community initiatives and collective coping strategies, (e.g. using asset-based community development [ 52 ]); ensuring policies for long-term recovery focus on community sustainable development (e.g. community festival and community centre activities) [ 44 ]; and development of a network amongst cooperative corporations formed for reconstruction and to organise self-help recovery sessions among residents of adjacent areas [ 58 ].

This scoping review sought to synthesise literature concerning the role of SC and CR during public health emergencies and disasters. Specifically, in this review we have examined: the methods used to measure CR and SC; the impact of CR and SC on mental wellbeing during disasters and emergencies; the impact of disasters and emergencies on CR and SC; and the types of interventions which can be used to enhance CR. To do this, data was extracted from 26 peer-reviewed journal articles. From this synthesis, several key themes have been identified, which can be used to develop guidelines and recommendations for deploying CR and SC in a public health emergency or disaster context. These key themes and resulting recommendations are summarised below.

Firstly, this review established that there is no consistent or standardised approach to measuring CR or SC within the general population. This finding is consistent with a review conducted by the World Health Organization which concludes that despite there being a number of frameworks that contain indicators across different determinants of health, there is a lack of consensus on priority areas for measurement and no widely accepted indicator [ 27 ]. As a result, there are many measures of CR and SC apparent within the literature (e.g., [ 62 , 63 ]), an example of a developed and validated measure is provided by Sherrieb, Norris and Galea [ 64 ]. Similarly, the definitions of CR and SC differ widely between researchers, which created a barrier to comparing and summarising information. Therefore, future research could seek to compare various interpretations of CR and to identify any overlapping concepts. However, a previous systemic review conducted by Patel et al. (2017) concludes that there are nine core elements of CR (local knowledge, community networks and relationships, communication, health, governance and leadership, resources, economic investment, preparedness, and mental outlook), with 19 further sub-elements therein [ 30 ]. Therefore, as CR is a multi-dimensional construct, the implications from the findings are that multiple aspects of social infrastructure may need to be considered.

Secondly, our synthesis of research concerning the role of CR and SC for ensuring mental health and wellbeing during, or following, a public health emergency or disaster revealed mixed effects. Much of the research indicates either a generally protective effect on mental health and wellbeing, or no effect; however, the literature demonstrates some potential for a high level of CR/SC to backfire and result in a negative effect for populations during, or following, a public health emergency or disaster. Considered together, our synthesis indicates that cognitive SC is the only facet of SC which was perceived as universally protective across all retained papers. This is consistent with a systematic review which also concludes that: (a) community level cognitive SC is associated with a lower risk of common mental disorders, while; (b) community level structural SC had inconsistent effects [ 65 ].

Further examination of additional data extracted from studies which found that CR/SC had a negative effect on mental health and wellbeing revealed no commonalities that might explain these effects (Please see Supplementary file 5 for additional information)

One potential explanation may come from a retained paper which found that high levels of SC result in an increase in stress level immediately post disaster [ 41 ]. This was suggested to be due to individuals having greater burdens due to wishing to help and support their wide networks as well as themselves. However, as time passes the levels of SC allow the community to come together and recover at a faster rate [ 41 ]. As this was the only retained paper which produced this finding, it would be beneficial for future research to examine boundary conditions for the positive effects of CR/SC; that is, to explore circumstances under which CR/SC may be more likely to put communities at greater risk. This further research should also include additional longitudinal research to validate the conclusions drawn by [ 41 ] as resilience is a dynamic process of adaption.

Thirdly, disasters and emergencies were generally found to have a negative effect on levels of SC. One retained paper found a mixed effect of SC in relation to an earthquake, however this paper separated participants by area in which they lived (i.e., mountainous vs. flat), which explains this inconsistent effect [ 49 ]. Dangerous areas (i.e. mountainous) saw a decrease in community SC in comparison to safer areas following the earthquake (an effect the authors attributed to the need to seek prolonged refuge), whereas participants from the safer areas (which are home to younger residents with a higher population density) saw an increase in SC [ 49 ]. This is consistent with the idea that being able to participate socially is a key element of SC [ 12 ]. Overall, however, this was the only retained paper which produced a variable finding in relation to the effect of disaster on levels of CR/SC.

Finally, research identified through our synthesis promotes the idea of bolstering SC (particularly cognitive SC) and cohesion in communities likely to be affected by disaster to improve levels of CR. This finding provides further understanding of the relationship between CR and SC; an association that has been reported in various articles seeking to provide conceptual frameworks (e.g., [ 66 , 67 ]) as well as indicator/measurement frameworks [ 27 ]. Therefore, this could be done by creating and promoting initiatives which foster SC and create bonds within the community. Papers included in the current review suggest that recreational-based activity groups and volunteering are potential methods for fostering SC and creating community bonds [ 44 , 51 , 59 ]. Similarly, further research demonstrates that feelings of social cohesion are enhanced by general social activities (e.g. fairs and parades [ 18 ]). Also, actively encouraging activities, programs and interventions which enhance connectedness and SC have been reported to be desirable to increase CR [ 68 ]. This suggestion is supported by a recent scoping review of literature [ 67 ] examined community champion approaches for the COVID-19 pandemic response and recovery and established that creating and promoting SC focused initiatives within the community during pandemic response is highly beneficial [ 67 ]. In terms of preparedness, research states that it may be beneficial for levels of SC and CR in communities at risk to be assessed, to allow targeted interventions where the population may be at most risk following an incident [ 42 , 44 ]. Additionally, from a more critical perspective, we acknowledge that ‘resilience’ can often be perceived as a focus on individual capacity to adapt to adversity rather than changing or mitigating the causes of adverse conditions [ 69 , 70 ]. Therefore, CR requires an integrated system approach across individual, community and structural levels [ 17 ]. Also, it is important that community members are engaged in defining and agreeing how community resilience is measured [ 27 ] rather than it being imposed by system leads or decision-makers.

In the aftermath of the pandemic, is it expected that there will be long-term repercussions both from an economic [ 8 ] and a mental health perspective [ 71 ]. Furthermore, the findings from this review suggest that although those in areas with high levels of SC may be negatively affected in the acute stage, as time passes, they have potential to rebound at a faster rate than those with lower levels of SC. Ongoing evaluation of the effectiveness of current initiatives as the COVID-19 pandemic progresses into a recovery phase will be invaluable for supplementing the evidence base identified through this review.

Recommendations

As a result of this review, a number of recommendations are suggested for policy and practice during public health emergencies and recovery.

Future research should seek to establish a standardised and validated approach to measuring and defining CR and SC within communities. There are ongoing efforts in this area, for example [ 72 ]. Additionally, community members should be involved in the process of defining how CR is measured.

There should be an enhanced effort to improve preparedness for public health emergencies and disasters in local communities by gauging current levels of SC and CR within communities using a standardised measure. This approach could support specific targeting of populations with low levels of CR/SC in case of a disaster or public health emergency, whilst also allowing for consideration of support for those with high levels of CR (as these populations can be heavily impacted initially following a disaster). By distinguishing levels of SC and CR, tailored community-centred approaches could be implemented, such as those listed in a guide released by PHE in 2015 [ 73 ].

CR and SC (specifically cognitive SC) should be bolstered if communities are at risk of experiencing a disaster or public health emergency. This can be achieved by using interventions which aim to increase a sense of community and create new social ties (e.g., recreational group activities, volunteering). Additionally, when aiming to achieve this, it is important to be mindful of the risk of increased levels of CR/SC to backfire, as well as seeking to advocate an integrated system approach across individual, community and structural levels.

It is necessary to be aware that although communities with high existing levels of resilience / SC may experience short-term negative consequences following a disaster, over time these communities might be able to recover at a faster rate. It is therefore important to ensure that suitable short-term support is provided to these communities in the immediate aftermath of a public health emergency or disaster.

Robust evaluation of the community resilience initiatives deployed during the COVID-19 pandemic response is essential to inform the evidence base concerning the effectiveness of CR/ SC. These evaluations should continue through the response phase and into the recovery phase to help develop our understanding of the long-term consequences of such interventions.

Limitations

Despite this review being the first in this specific topic area, there are limitations that must be considered. Firstly, it is necessary to note that communities are generally highly diverse and the term ‘community’ in academic literature is a subject of much debate (see: [ 74 ]), therefore this must be considered when comparing and collating research involving communities. Additionally, the measures of CR and SC differ substantially across research, including across the 26 retained papers used in the current review. This makes the act of comparing and collating research findings very difficult. This issue is highlighted as a key outcome from this review, and suggestions for how to overcome this in future research are provided. Additionally, we acknowledge that there will be a relationship between CR & SC even where studies measure only at individual or community level. A review [ 75 ] on articulating a hypothesis of the link to health inequalities suggests that wider structural determinants of health need to be accounted for. Secondly, despite the final search strategy encompassing terms for both CR and SC, only one retained paper directly measured CR; thus, making the research findings more relevant to SC. Future research could seek to focus on CR to allow for a comparison of findings. Thirdly, the review was conducted early in the COVID-19 pandemic and so does not include more recent publications focusing on resilience specifically in the context of COVID-19. Regardless of this fact, the synthesis of, and recommendations drawn from, the reviewed studies are agnostic to time and specific incident and contain critical elements necessary to address as the pandemic moves from response to recovery. Further research should review the effectiveness of specific interventions during the COVID-19 pandemic for collation in a subsequent update to this current paper. Fourthly, the current review synthesises findings from countries with individualistic and collectivistic cultures, which may account for some variation in the findings. Lastly, despite choosing a scoping review method for ease of synthesising a wide literature base for use by public health emergency researchers in a relatively tight timeframe, there are disadvantages of a scoping review approach to consider: (1) quality appraisal of retained studies was not carried out; (2) due to the broad nature of a scoping review, more refined and targeted reviews of literature (e.g., systematic reviews) may be able to provide more detailed research outcomes. Therefore, future research should seek to use alternative methods (e.g., empirical research, systematic reviews of literature) to add to the evidence base on CR and SC impact and use in public health practice.

This review sought to establish: (1) How CR and SC are quantified in research?; (2) The impact of community resilience on mental wellbeing?; (3) The impact of infectious disease outbreaks, disasters and emergencies on community resilience and social capital?; and, (4) What types of interventions enhance community resilience and social capital?. The chosen search strategy yielded 26 relevant papers from which we were able extract information relating to the aims of this review.

Results from the review revealed that CR and SC are not measured consistently across research. The impact of CR / SC on mental health and wellbeing during emergencies and disasters is mixed (with some potential for backlash), however the literature does identify cognitive SC as particularly protective. Although only a small number of papers compared CR or SC before and after a disaster, the findings were relatively consistent: SC or CR is negatively impacted by a disaster. Methods suggested to bolster SC in communities were centred around social activities, such as recreational group activities and volunteering. Recommendations for both research and practice (with a particular focus on the ongoing COVID-19 pandemic) are also presented.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Social Capital

Zortea TC, Brenna CT, Joyce M, McClelland H, Tippett M, Tran MM, et al. The impact of infectious disease-related public health emergencies on suicide, suicidal behavior, and suicidal thoughts. Crisis. 2020;42(6):474–87.

Article PubMed PubMed Central Google Scholar

Davis JR, Wilson S, Brock-Martin A, Glover S, Svendsen ER. The impact of disasters on populations with health and health care disparities. Disaster Med Pub Health Prep. 2010;4(1):30.

Article Google Scholar

Francescutti LH, Sauve M, Prasad AS. Natural disasters and healthcare: lessons to be learned. Healthc Manage Forum. 2017;30(1):53–5.

Article PubMed Google Scholar

Jones L, Palumbo D, Brown D. Coronavirus: How the pandemic has changed the world economy. BBC News; 2021. Accessible at: https://www.bbc.co.uk/news/business-51706225 .

Below R, Wallemacq P. Annual disaster statistical review 2017. Brussels: CRED, Centre for Research on the Epidemiology of Disasters; 2018.

Google Scholar

Qiu W, Chu C, Mao A, Wu J. The impacts on health, society, and economy of SARS and H7N9 outbreaks in China: a case comparison study. J Environ Public Health. 2018;2018:2710185.

Worldometer. COVID-19 coronavirus pandemic. 2021.

Harari D, Keep M. Coronavirus: economic impact house of commons library. Briefing Paper (Number 8866); 2021. Accessible at: https://commonslibrary.parliament.uk/research-briefings/cbp-8866/ .

Nabavi N. Covid-19: pandemic will cast a long shadow on mental health, warns England’s CMO. BMJ. 2021;373:n1655.

Ziglio E. Strengthening resilience: a priority shared by health 2020 and the sustainable development goals. No. WHO/EURO: 2017-6509-46275-66939. World Health Organization; Regional Office for Europe; 2017.

Asadzadeh A, Kotter T, Salehi P, Birkmann J. Operationalizing a concept: the systematic review of composite indicator building for measuring community disaster resilience. Int J Disaster Risk Reduct. 2017;25:147.

Sherrieb K, Norris F, Galea S. Measuring capacities for community resilience. Soc Indicators Res. 2010;99(2):227.

Poortinga W. Community resilience and health: the role of bonding, bridging, and linking aspects of social capital. Health Place. 2011;18(2):286–95.

Ferlander S. The importance of different forms of social capital for health. Acta Sociol. 2007;50(2):115–28.

Nakagawa Y, Shaw R. Social capital: a missing link to disaster recovery. Int J Mass Emerge Disasters. 2004;22(1):5–34.

Grootaert C, Narayan D, Jones VN, Woolcock M. Measuring social capital: an integrated questionnaire. Washington, DC: World Bank Working Paper, No. 18; 2004.

Adler PS, Kwon SW. Social capital: prospects for a new concept. Acad Manage Rev. 2002;27(1):17–40.

Aldrich DP, Meyer MA. Social capital and community resilience. Am Behav Sci. 2015;59(2):254–69.

Rodriguez-Llanes JM, Vos F, Guha-Sapir D. Measuring psychological resilience to disasters: are evidence-based indicators an achievable goal? Environ Health. 2013;12(1):115.

De Silva MJ, McKenzie K, Harpham T, Huttly SR. Social capital and mental Illness: a systematic review. J Epidemiol Community Health. 2005;59(8):619–27.

Bonanno GA, Galea S, Bucciarelli A, Vlahov D. Psychological resilience after disaster: New York City in the aftermath of the september 11th terrorist attack. Psychol Sci. 2006;17(3):181.

World Health Organization. Health 2020: a European policy framework and strategy for the 21st century. World Health Organization. Regional Office for Europe; 2013.

Public Health England. Community-Centred Public Health: Taking a Whole System Approach. 2020.

SPI-B. The role of Community Champion networks to increase engagement in the context of COVID19: Evidence and best practice. 2021.

Public Health England. Community champions: A rapid scoping review of community champion approaches for the pandemic response and recovery. 2021.

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

World Health Organisation. WHO health evidence network synthesis report: what quantitative and qualitative methods have been developed to measure health-related community resilience at a national and local level. 2018.

Hall C, Williams N, Gauntlett L, Carter H, Amlôt R, Peterson L et al. Findings from systematic review of public perceptions and responses. PROACTIVE EU. Deliverable 1.1. 2019. Accessible at: https://proactive-h2020.eu/wp-content/uploads/2021/04/PROACTIVE_20210312_D1.1_V5_PHE_Systematic-Review-of-Public-Perceptions-and-Responses_revised.pdf .

Weston D, Ip A, Amlôt R. Examining the application of behaviour change theories in the context of Infectious disease outbreaks and emergency response: a review of reviews. BMC Public Health. 2020;20(1):1483.

Article PubMed PubMed Central CAS Google Scholar

Patel SS, Rogers MB, Amlôt R, Rubin GJ. What do we mean by ‘community resilience’? A systematic literature review of how it is defined in the literature. PLoS Curr. 2017;9:ecurrents.dis.db775aff25efc5ac4f0660ad9c9f7db2.

PubMed PubMed Central Google Scholar

Brooks SK, Weston D, Wessely S, Greenberg N. Effectiveness and acceptability of brief psychoeducational interventions after potentially traumatic events: a systematic review. Eur J Psychotraumatology. 2021;12(1):1923110.

Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review-a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10(1_suppl):21–34.

Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13:1–8.

Bearman M, Dawson P. Qualitative synthesis and systematic review in health professions education. Med Educ. 2013;47(3):252–60.

Heid AR, Pruchno R, Cartwright FP, Wilson-Genderson M. Exposure to Hurricane Sandy, neighborhood collective efficacy, and post-traumatic stress symptoms in older adults. Aging Ment Health. 2017;21(7):742–50.

Hikichi H, Aida J, Tsuboya T, Kondo K, Kawachi I. Can community social cohesion prevent posttraumatic stress disorder in the aftermath of a disaster? A natural experiment from the 2011 Tohoku Earthquake and tsunami. Am J Epidemiol. 2016;183(10):902–10.

Lee J, Blackmon BJ, Cochran DM, Kar B, Rehner TA, Gunnell MS. Community resilience, psychological resilience, and depressive symptoms: an examination of the Mississippi Gulf Coast 10 years after Hurricane Katrina and 5 years after the Deepwater Horizon oil spill. Disaster med. 2018;12(2):241–8.

Lee J, Blackmon BJ, Lee JY, Cochran DM Jr, Rehner TA. An exploration of posttraumatic growth, loneliness, depression, resilience, and social capital among survivors of Hurricane Katrina and the deepwater Horizon oil spill. J Community Psychol. 2019;47(2):356–70.

Lowe SR, Sampson L, Gruebner O, Galea S. Psychological resilience after Hurricane Sandy: the influence of individual- and community-level factors on mental health after a large-scale natural disaster. PLoS One. 2015;10(5):e0125761.

Rung AL, Gaston S, Robinson WT, Trapido EJ, Peters ES. Untangling the disaster-depression knot: the role of social ties after deepwater Horizon. Soc Sci Med. 2017;177:19–26.

Weil F, Lee MR, Shihadeh ES. The burdens of social capital: how socially-involved people dealt with stress after Hurricane Katrina. Soc Sci Res. 2012;41(1):110–9.

Hikichi H, Aida J, Matsuyama Y, Tsuboya T, Kondo K, Kawachi I. Community-level social capital and cognitive decline after a Natural Disaster: a natural experiment from the 2011 Great East Japan Earthquake and Tsunami. Soc Sci Med. 2018;257:111981.

Lau AL, Chi I, Cummins RA, Lee TM, Chou KL, Chung LW. The SARS (severe acute respiratory syndrome) pandemic in Hong Kong: effects on the subjective wellbeing of elderly and younger people. Aging Ment Health. 2008;12(6):746–60.

Sun Y, Yan T. The use of public health indicators to assess individual happiness in post-disaster recovery. Int J Environ Res Public Health. 2019;16(21):4101.

Wong H, Huang Y, Fu Y, Zhang Y. Impacts of structural social capital and cognitive social capital on the psychological status of survivors of the yaan Earthquake. Appl Res Qual Life. 2018;14:1411–33.

Xiao H, Zhang Y, Kong D, Li S, Yang N. Social capital and sleep quality in individuals who self-isolated for 14 days during the coronavirus disease 2019 (COVID-19) outbreak in January 2020 in China. Med Sci Monit. 2020;26:e923921.

PubMed PubMed Central CAS Google Scholar

Matsuyama Y, Aida J, Hase A, Sato Y, Koyama S, Tsuboya T, et al. Do community- and individual-level social relationships contribute to the mental health of disaster survivors? A multilevel prospective study after the great East Japan earthquake. Soc Sci Med. 2016;151:187–95.

Ozaki A, Horiuchi S, Kobayashi Y, Inoue M, Aida J, Leppold C, Yamaoka K. Beneficial roles of social support for mental health vary in the Japanese population depending on disaster experience: a nationwide cross-sectional study. Tohoku J Exp Med. 2018;246(4):213–23.

Sato K, Amemiya A, Haseda M, Takagi D, Kanamori M, Kondo K, et al. Post-disaster changes in Social Capital and Mental Health: a natural experiment from the 2016 Kumamoto Earthquake. Am J Epidemiol. 2020;189(9):910–21.

Tsuchiya N, Nakaya N, Nakamura T, Narita A, Kogure M, Aida J, Tsuji I, Hozawa A, Tomita H. Impact of social capital on psychological distress and interaction with house destruction and displacement after the great East Japan earthquake of 2011. J Neuropsychiatry Clin Neurosci. 2017;71(1):52–60.

Brockie L, Miller E. Understanding older adults’ resilience during the Brisbane floods: social capital, life experience, and optimism. Disaster Med Pub Health Prep. 2017;11(1):72–9.

Caldwell K, Boyd CP. Coping and resilience in farming families affected by drought. Rural Remote Health. 2009;9(2):1088.

PubMed Google Scholar

Huang Y, Tan NT, Liu J. Support, sense of community, and psychological status in the survivors of the Yaan earthquake. J Community Psychol. 2016;44(7):919–36.

Wind T, Fordham M, Komproe H. Social capital and post-disaster mental health. Glob Health Action. 2011;4(1):6351.

Wind T, Komproe IH. The mechanisms that associate community social capital with post-disaster mental health: a multilevel model. Soc Sci Med. 2012;75(9):1715–20.

Hogg D, Kingham S, Wilson TM, Ardagh M. The effects of spatially varying earthquake impacts on mood and anxiety symptom treatments among long-term Christchurch residents following the 2010/11 Canterbury Earthquakes, New Zealand. Health Place. 2016;41:78–88.

Flores EC, Carnero AM, Bayer AM. Social capital and chronic post-traumatic stress disorder among survivors of the 2007 earthquake in Pisco, Peru. Soc Sci Med. 2014;101:9–17.

Rafiey H, Alipour F, LeBeau R, Salimi Y, Ahmadi S. Exploring the buffering role of social capital in the development of posttraumatic stress symptoms among Iranian earthquake survivors. Psychol Trauma. 2019;14(6):1040–6.

Babcicky P, Seebauer S. The two faces of social capital in private Flood mitigation: opposing effects on risk perception, self-efficacy and coping capacity. J Risk Res. 2017;20(8):1017–37.

Bakic H, Ajdukovic D. Stability and change post-disaster: dynamic relations between individual, interpersonal and community resources and psychosocial functioning. Eur J Psychotraumatol. 2019;10(1):1614821.

Rogers RW. A protection motivation theory of fear appeals and attitude change. J Psychol. 1975;91(1):93–114.

Lindberg K, Swearingen T. A reflective thrive-oriented community resilience scale. Am J Community Psychol. 2020;65(3–4):467–78.

Leykin D, Lahad M, Cohen O, Goldberg A, Aharonson-Daniel L. Conjoint community resiliency assessment measure-28/10 items (CCRAM28 and CCRAM10): a self-report tool for assessing community resilience. Am J Community Psychol. 2013;52:313–23.

Sherrieb K, Norris FH, Galea S. Measuring capacities for community resilience. Soc Indic Res. 2010;99:227–47.

Ehsan AM, De Silva MJ. Social capital and common mental disorder: a systematic review. J Epidemiol Community Health. 2015;69(10):1021–8.

Pfefferbaum B, Van Horn RL, Pfefferbaum RL. A conceptual framework to enhance community resilience using social capital. Clin Soc Work J. 2017;45(2):102–10.

Carmen E, Fazey I, Ross H, Bedinger M, Smith FM, Prager K, et al. Building community resilience in a context of climate change: the role of social capital. Ambio. 2022;51(6):1371–87.

Humbert C, Joseph J. Introduction: the politics of resilience: problematising current approaches. Resilience. 2019;7(3):215–23.

Tanner T, Bahadur A, Moench M. Challenges for resilience policy and practice. Working paper: 519. 2017.

Vadivel R, Shoib S, El Halabi S, El Hayek S, Essam L, Bytyçi DG. Mental health in the post-COVID-19 era: challenges and the way forward. Gen Psychiatry. 2021;34(1):e100424.

Article CAS Google Scholar

Pryor M. Social Capital Harmonised Standard. London: Government Statistical Service. 2021. Accessible at: https://gss.civilservice.gov.uk/policystore/social-capital/ .

Public Health England NE. A guide to community-centred approaches for health and wellbeing. 2015.

Hawe P. Capturing the meaning of ‘community’ in community intervention evaluation: some contributions from community psychology. Health Promot Int. 1994;9(3):199–210.

Uphoff EP, Pickett KE, Cabieses B, Small N, Wright J. A systematic review of the relationships between social capital and socioeconomic inequalities in health: a contribution to understanding the psychosocial pathway of health inequalities. Int J Equity Health. 2013;12:1–12.

Download references

Acknowledgements

Not applicable.

This study was supported by the National Institute for Health Research Research Unit (NIHR HPRU) in Emergency Preparedness and Response, a partnership between Public Health England, King’s College London and the University of East Anglia. The views expressed are those of the author(s) and not necessarily those of the NIHR, Public Health England, the UK Health Security Agency or the Department of Health and Social Care [Grant number: NIHR20008900]. Part of this work has been funded by the Office for Health Improvement and Disparities, Department of Health and Social Care, as part of a Collaborative Agreement with Leeds Beckett University.

Author information

Authors and affiliations.

Behavioural Science and Insights Unit, Evaluation & Translation Directorate, Science Group, UK Health Security Agency, Porton Down, Salisbury, SP4 0JG, UK

C. E. Hall, H. Wehling, R. Amlôt & D. Weston

Health Protection Research Unit, Institute of Psychology, Psychiatry and Neuroscience, King’s College London, 10 Cutcombe Road, London, SE5 9RJ, UK

C. E. Hall, S. K. Brooks & N. Greenberg

School of Health and Community Studies, Leeds Beckett University, Portland Building, PD519, Portland Place, Leeds, LS1 3HE, UK

J. Stansfield & J. South

King’s Centre for Military Health Research, Institute of Psychology, Psychiatry and Neuroscience, King’s College London, 10 Cutcombe Road, London, SE5 9RJ, UK

N. Greenberg

You can also search for this author in PubMed Google Scholar

Contributions

DW, JSo and JSt had the main idea for the review. The search strategy and eligibility criteria were devised by CH, DW, JSo and JSt. CH conducted the database searches. CH and DW conducted duplicate, title and abstract and full text screening in accordance with inclusion criteria. CH conducted data extraction, CH and DW carried out the analysis and drafted the initial manuscript. All authors provided critical revision of intellectual content. All authors approved the final manuscript.

Authors’ information

Corresponding author.

Correspondence to D. Weston .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1., additional file 2., additional file 3., additional file 4., additional file 5., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Hall, C.E., Wehling, H., Stansfield, J. et al. Examining the role of community resilience and social capital on mental health in public health emergency and disaster response: a scoping review. BMC Public Health 23 , 2482 (2023). https://doi.org/10.1186/s12889-023-17242-x

Download citation

Received : 04 April 2022

Accepted : 16 November 2023

Published : 12 December 2023

DOI : https://doi.org/10.1186/s12889-023-17242-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Mental health
Community cohesion
Public health emergency

BMC Public Health

ISSN: 1471-2458

Submission enquiries: [email protected]
General enquiries: [email protected]

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

Knowledge Base
Working with sources
What Are Credible Sources & How to Spot Them | Examples

What Are Credible Sources & How to Spot Them | Examples

Published on August 26, 2021 by Tegan George . Revised on May 31, 2023.

A credible source is free from bias and backed up with evidence. It is written by a trustworthy author or organization.

There are a lot of sources out there, and it can be hard to tell what’s credible and what isn’t at first glance.

Evaluating source credibility is an important information literacy skill. It ensures that you collect accurate information to back up the arguments you make and the conclusions you draw.

Types of sources, how to identify a credible source, the craap test, where to find credible sources, evaluating web sources, other interesting articles, frequently asked questions.

There are many different types of sources , which can be divided into three categories: primary sources , secondary sources , and tertiary sources .

Primary sources are often considered the most credible in terms of providing evidence for your argument, as they give you direct evidence of what you are researching. However, it’s up to you to ensure the information they provide is reliable and accurate.

You will likely use a combination of the three types over the course of your research process .

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Try for free

There are a few criteria to look at right away when assessing a source. Together, these criteria form what is known as the CRAAP test .

The information should be up-to-date and current.
The source should be relevant to your research.
The author and publication should be a trusted authority on the subject you are researching.
The sources the author cited should be easy to find, clear, and unbiased.
For web sources, the URL and layout should signify that it is trustworthy.

The CRAAP test is a catchy acronym that will help you evaluate the credibility of a source you are thinking about using. California State University developed it in 2004 to help students remember best practices for evaluating content.

C urrency: Is the source up-to-date?
R elevance: Is the source relevant to your research?
A uthority: Where is the source published? Who is the author? Are they considered reputable and trustworthy in their field?
A ccuracy: Is the source supported by evidence? Are the claims cited correctly?
P urpose: What was the motive behind publishing this source?

The criteria for evaluating each point depend on your research topic .

For example, if you are researching cutting-edge scientific technology, a source from 10 years ago will not be sufficiently current . However, if you are researching the Peloponnesian War, a source from 200 years ago would be reasonable to refer to.

Be careful when ascertaining purpose . It can be very unclear (often by design!) what a source’s motive is. For example, a journal article discussing the efficacy of a particular medication may seem credible, but if the publisher is the manufacturer of the medication, you can’t be sure that it is free from bias. As a rule of thumb, if a source is even passively trying to convince you to purchase something, it may not be credible.

Newspapers can be a great way to glean first-hand information about a historical event or situate your research topic within a broader context. However, the veracity and reliability of online news sources can vary enormously—be sure to pay careful attention to authority here.

When evaluating academic journals or books published by university presses, it’s always a good rule of thumb to ensure they are peer-reviewed and published in a reputable journal.

What is peer review?

The peer review process evaluates submissions to academic journals. A panel of reviewers in the same subject area decide whether a submission should be accepted for publication based on a set of criteria.

For this reason, academic journals are often considered among the most credible sources you can use in a research project– provided that the journal itself is trustworthy and well-regarded.

What sources you use depend on the kind of research you are conducting.

For preliminary research and getting to know a new topic, you could use a combination of primary, secondary, and tertiary sources.

Encyclopedias
Websites with .edu or .org domains
News sources with first-hand reporting
Research-oriented magazines like ScienceMag or Nature Weekly .

As you dig deeper into your scholarly research, books and academic journals are usually your best bet.

Academic journals are often a great place to find trustworthy and credible content, and are considered one of the most reliable sources you can use in academic writing.

Is the journal indexed in academic databases?
Has the journal had to retract many articles?
Are the journal’s policies on copyright and peer review easily available?
Are there solid “About” and “ Scope ” pages detailing what sorts of articles they publish?
Has the author of the article published other articles? A quick Google Scholar search will show you.
Has the author been cited by other scholars? Google Scholar also has a function called “Cited By” that can show you where the author has been cited. A high number of “Cited By” results can often be a measurement of credibility.

Google Scholar is a search engine for academic sources. This is a great place to kick off your research. You can also consider using an academic database like LexisNexis or government open data to get started.

Open Educational Resources , or OERs, are materials that have been licensed for “free use” in educational settings. Legitimate OERs can be a great resource. Be sure they have a Creative Commons license allowing them to be duplicated and shared, and meet the CRAAP test criteria, especially in the authority section. The OER Commons is a public digital library that is curated by librarians, and a solid place to start.

It can be especially challenging to verify the credibility of online sources. They often do not have single authors or publication dates, and their motivation can be more difficult to ascertain.

Websites are not subject to the peer-review and editing process that academic journals or books go through, and can be published by anyone at any time.

When evaluating the credibility of a website, look first at the URL. The domain extension can help you understand what type of website you’re dealing with.

Educational resources end in .edu, and are generally considered the most credible in academic settings.
Advocacy or non-profit organizations end in .org.
Government-affiliated websites end in .gov.
Websites with some sort of commercial aspect end in .com (or .co.uk, or another country-specific domain).

In general, check for vague terms, buzzwords, or writing that is too emotive or subjective . Beware of grandiose claims, and critically analyze anything not cited or backed up by evidence.

How does the website look and feel? Does it look professional to you?
Is there an “About Us” page, or a way to contact the author or organization if you need clarification on a claim they have made?
Are there links to other sources on the page, and are they trustworthy?
Can the information you found be verified elsewhere, even via a simple Google search?
When was the website last updated? If it hasn’t been updated recently, it may not pass the CRAAP test.
Does the website have a lot of advertisements or sponsored content? This could be a sign of bias.
Is a source of funding disclosed? This could also give you insight into the author and publisher’s motivations.

Social media posts, blogs, and personal websites can be good resources for a situational analysis or grounding of your preliminary ideas, but exercise caution here. These highly personal and subjective sources are seldom reliable enough to stand on their own in your final research product.

Similarly, Wikipedia is not considered a reliable source due to the fact that it can be edited by anyone at any time. However, it can be a good starting point for general information and finding other sources.

Checklist: Is my source credible?

My source is relevant to my research topic.

My source is recent enough to contain up-to-date information on my topic.

There are no glaring grammatical or orthographic errors.

The author is an expert in their field.

The information provided is accurate to the best of my knowledge. I have checked that it is supported by evidence and/or verifiable elsewhere.

My source cites or links to other sources that appear relevant and trustworthy.

There is a way to contact the author or publisher of my source.

The purpose of my source is to educate or inform, not to sell a product or push a particular opinion.

My source is unbiased, and offers multiple perspectives fairly.

My source avoids vague or grandiose claims, and writing that is too emotive or subjective.

[For academic journals]: My source is peer-reviewed and published in a reputable and established journal.

[For web sources]: The layout of my source is professional and recently updated. Backlinks to other sources are up-to-date and not broken.

[For web sources]: My source’s URL suggests the domain is trustworthy, e.g. a .edu address.

Your sources are likely to be credible!

If you want to know more about ChatGPT, AI tools , citation , and plagiarism , make sure to check out some of our other articles with explanations and examples.

ChatGPT vs human editor
ChatGPT citations
Is ChatGPT trustworthy?
Using ChatGPT for your studies
What is ChatGPT?
Chicago style
Paraphrasing

Plagiarism

Types of plagiarism
Self-plagiarism
Avoiding plagiarism
Academic integrity
Consequences of plagiarism
Common knowledge

A credible source should pass the CRAAP test and follow these guidelines:

The information should be up to date and current.
For a web source, the URL and layout should signify that it is trustworthy.

Peer review is a process of evaluating submissions to an academic journal. Utilizing rigorous criteria, a panel of reviewers in the same subject area decide whether to accept each submission for publication. For this reason, academic journals are often considered among the most credible sources you can use in a research project– provided that the journal itself is trustworthy and well-regarded.

The CRAAP test is an acronym to help you evaluate the credibility of a source you are considering using. It is an important component of information literacy .

The CRAAP test has five main components:

Currency: Is the source up to date?
Relevance: Is the source relevant to your research?
Authority: Where is the source published? Who is the author? Are they considered reputable and trustworthy in their field?
Accuracy: Is the source supported by evidence? Are the claims cited correctly?
Purpose: What was the motive behind publishing this source?

Academic dishonesty can be intentional or unintentional, ranging from something as simple as claiming to have read something you didn’t to copying your neighbor’s answers on an exam.

You can commit academic dishonesty with the best of intentions, such as helping a friend cheat on a paper. Severe academic dishonesty can include buying a pre-written essay or the answers to a multiple-choice test, or falsifying a medical emergency to avoid taking a final exam.

To determine if a source is primary or secondary, ask yourself:

Was the source created by someone directly involved in the events you’re studying (primary), or by another researcher (secondary)?
Does the source provide original information (primary), or does it summarize information from other sources (secondary)?
Are you directly analyzing the source itself (primary), or only using it for background information (secondary)?

Some types of source are nearly always primary: works of art and literature, raw statistical data, official documents and records, and personal communications (e.g. letters, interviews ). If you use one of these in your research, it is probably a primary source.

Always make sure to properly cite your sources to avoid plagiarism .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

George, T. (2023, May 31). What Are Credible Sources & How to Spot Them | Examples. Scribbr. Retrieved April 5, 2024, from https://www.scribbr.com/working-with-sources/credible-sources/

Is this article helpful?

Tegan George

Other students also liked, applying the craap test & evaluating sources, how to cite a wikipedia article | apa, mla & chicago, primary vs. secondary sources | difference & examples, unlimited academic ai-proofreading.

✔ Document error-free in 5minutes ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

This paper is in the following e-collection/theme issue:

Published on 5.4.2024 in Vol 26 (2024)

Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study

Authors of this article:

Original Paper

Joseph Mugaanyi 1 * , MBBS, MD ;
Liuying Cai 2 * , MPhil ;
Sumei Cheng 2 , PhD ;
Caide Lu 1 , MD, PhD ;
Jing Huang 1 , MD, PhD

1 Department of Hepato-Pancreato-Biliary Surgery, Ningbo Medical Center Lihuili Hospital, Health Science Center, Ningbo University, Ningbo, China

2 Institute of Philosophy, Shanghai Academy of Social Sciences, Shanghai, China

*these authors contributed equally

Corresponding Author:

Jing Huang, MD, PhD

Department of Hepato-Pancreato-Biliary Surgery, Ningbo Medical Center Lihuili Hospital

Health Science Center

Ningbo University

No 1111 Jiangnan Road

Ningbo, 315000

Phone: 86 13819803591

Email: [email protected]

Background: Large language models (LLMs) have gained prominence since the release of ChatGPT in late 2022.

Objective: The aim of this study was to assess the accuracy of citations and references generated by ChatGPT (GPT-3.5) in two distinct academic domains: the natural sciences and humanities.

Methods: Two researchers independently prompted ChatGPT to write an introduction section for a manuscript and include citations; they then evaluated the accuracy of the citations and Digital Object Identifiers (DOIs). Results were compared between the two disciplines.

Results: Ten topics were included, including 5 in the natural sciences and 5 in the humanities. A total of 102 citations were generated, with 55 in the natural sciences and 47 in the humanities. Among these, 40 citations (72.7%) in the natural sciences and 36 citations (76.6%) in the humanities were confirmed to exist ( P =.42). There were significant disparities found in DOI presence in the natural sciences (39/55, 70.9%) and the humanities (18/47, 38.3%), along with significant differences in accuracy between the two disciplines (18/55, 32.7% vs 4/47, 8.5%). DOI hallucination was more prevalent in the humanities (42/55, 89.4%). The Levenshtein distance was significantly higher in the humanities than in the natural sciences, reflecting the lower DOI accuracy.

Conclusions: ChatGPT’s performance in generating citations and references varies across disciplines. Differences in DOI standards and disciplinary nuances contribute to performance variations. Researchers should consider the strengths and limitations of artificial intelligence writing tools with respect to citation accuracy. The use of domain-specific models may enhance accuracy.

Introduction

In the ever-evolving landscape of scholarly research and academic discourse, the role of technology in aiding and enhancing the research process has grown exponentially. One of the most notable advancements in this regard is the emergence of large language models (LLMs) such as GPT-3.5, which have demonstrated impressive capabilities in generating written content across various domains, including academic writing. These LLMs, powered by vast corpora of text data and sophisticated machine-learning algorithms, have offered researchers and writers a new tool for assistance in crafting scholarly documents [ 1 - 3 ]. LLMs were initially designed and developed to primarily assist in natural language writing. However, since the release of ChatGPT in late 2022, the tool has been adopted in a wide range of scenarios, including customer care, expert systems, as well as literature searches and academic writing. Researchers have already used LLMs to write their academic papers, as demonstrated by Kishony and Ifargan [ 4 ]. While the potential of these tools is evident, it is essential to critically assess their performance, especially in the intricate domains of citations and references, which are the foundation of academic discourse and credibility.

Citations and references serve as the backbone of scholarly communication, providing the necessary context, evidence, and credit to prior works, thus fostering intellectual dialogue and ensuring the integrity of the research process. Accuracy in generating citations and the inclusion of Digital Object Identifiers (DOIs) [ 5 ] are paramount, as they directly influence the traceability and accessibility of cited works. Despite the promise of LLMs, concerns have emerged regarding the reliability and precision of their generated citations and references, raising questions about their suitability as academic writing assistants. Studies on the viability of LLMs as writing assistants in scholarly writing [ 6 - 8 ] underscore the significance of this body of research within the broader academic landscape. Although prior works are quite informative [ 9 - 12 ], there is a lack of an interdisciplinary perspective on citations and references generated by LLMs, which is vital for understanding how LLMs perform across different disciplines.

An increasing number of academics and researchers, especially in countries where English is not a first language (eg, China), are relying on ChatGPT to translate their work into English, research the existing published literature, and even generate citations and references to published literature. Therefore, the aim of this study was to evaluate LLM performance in generating citations and references across two distinct domains, the natural sciences and humanities, by assessing both the presence and accuracy of citations, the existence and accuracy of DOIs, and the potential for hallucination. We aim to provide valuable insights into the strengths and limitations of LLMs in supporting academic writing in diverse research contexts.

The outcomes of this study will contribute to a nuanced understanding of the capabilities and limitations of LLMs as academic writing assistants. Moreover, our findings may inform best practices for researchers and writers who employ these tools in their work, fostering transparency and accuracy in scholarly communication.

LLM Concepts

An LLM is a catch-all term for a machine-learning model designed and trained to understand and generate natural language. LLMs are considered “large” language models due to the sheer number of parameters in the model. A parameter in machine learning is a numerical variable or weight that is optimized through training to map a relationship between the input and the output. LLMs have millions to billions of parameters.

Current LLMs are mostly based on the transformer architecture ( Figure 1 ). However, before transformers were introduced in 2017 [ 13 ], recurrent neural nets (RNNs) were mostly used for natural language processing. One key limitation of RNNs was the length of text they could handle. In 2015, Bahdanau et al [ 14 ] proposed accounting for attention to improve RNN performance with long text. Drawing inspiration for the RNN’s encoder-decoder design, the transformer consists of an encoder and a decoder; however, unlike the RNN, the transformer does not perform sequential data processing and each layer can address all other layers. This allows the transformer model to handle different parts of the input as it processes each part at different stages. This is the mechanism that allows for self-attention in the transformer model.

The way attention works in a transformer model is by computing attention weights for each token, and then the relevance of the token is determined based on the weights. This allows the model to track and assign hierarchical values to each token. Fundamentally, this is similar to how humans process language by extracting the key details out of a chunk of text. This architecture is the linchpin for the majority of LLMs, including the GPT model [ 15 ] that is the basis of OpenAI’s ChatGPT or the bidirectional encoder representations from transformers (BERT) algorithm [ 16 ]. These are broadly categorized into encoder-style and decoder-style transformers, with the former mostly applying to predictive tasks and the latter applying to generative tasks.

Irrespective of the architecture, as an encoder-style or decoder-style transformer, the model is trained on a vast volume of data. The objective is to train a model capable of applying the knowledge gained from the training data to unseen data or situations. This is referred to as generalization. If the model is capable of precise recall of data it has previously been exposed to, this would be memorization and overfitting is said to have occurred. However, this does not mean that memorization is in itself a negative feature. Indeed, there are situations where memorization is preferable to generation such as in the task of information cataloging.

LLMs in Academia

LLMs can handle tasks such as text classification, translation, summarization, and text generation. Since the advent of the internet, and with it the publication of scientific information online, the amount of global academic output exploded, with more than 5 million articles published in 2022 ( Table 1 ). Given the pressure in academia to keep up with developments in one’s field, it is increasingly becoming more difficult to track, prioritize, and keep up with scientific information. It is against this backdrop that LLMs offer an opportunity. Perhaps the most obvious use case is in literature reviews and summarization, reference lookup, and data generation.

However, there are still several questions that need to be answered. First, machine-learning models are inherently probabilistic, meaning that they are not deterministic. Therefore, for the same user input, the model may give different results due to the variability baked into the model. While this can be a valuable trait for creative endeavors, in academic and scientific works, there is a need for reproducibility and reliability, and it remains unclear how well this can be achieved. Second, LLMs are constrained to the information they are trained on. This can be affected by selection bias, the quality of data used, artifacts resulting from data cleaning, and other factors. In essence, we rely on trusting the trainer to provide accurate and unbiased training data to the models.

There is potential for LLMs to be useful tools for delivering academic and scientific information to various audiences, including—but not limited to—students and other academics. However, for this use case, a degree of memorization of the underlying content is necessary. Where information is unviable, it would be better to state so rather than to interpolate. In the current iteration of LLMs, since the training is geared toward generalization and the models are probabilistic, they tend to interpolate and fill in the missing information with synthetic text. There is still a need to explore this process deeper to find solutions.

Data Collection and Validation

Topics were selected and categorized as either natural sciences or humanities. Topics were included if they were: (1) clinical or biomedical–related research in the natural sciences category and philosophy/psychology-related research in the humanities category, and (2) published in English. Topics were excluded if they were: (1) not in English, (2) related to a highly specialized or niche field, and (3) sensitive or controversial in nature. Two researchers independently prompted ChatGPT (GPT-3.5) to write sections of a manuscript while adhering to the American Psychological Association style [ 17 ] for citations and including the DOI of each reference. Citations and references generated by ChatGPT were collected for subsequent analysis. The researchers then independently validated the references by conducting searches on Google Scholar, PubMed, and Google Search for each cited reference. The primary objective was to confirm the existence and accuracy of the cited literature. DOI existence and validation were confirmed using the DOI Foundation website [ 18 ]. DOIs that did not exist or were matched to a different source were considered hallucinations [ 19 ]. Data collected by both researchers were aggregated and compared. Independent validation was performed to ensure agreement between the two researchers regarding the existence, validity, and accuracy of the citations and DOIs. Any disagreements or discrepancies were resolved through discussion and consensus.

In this study, hallucination refers to instances where ChatGPT 3.5 generates DOIs and/or citations that do not correspond to actual, valid DOIs/citations for scholarly references. In these instances, the model may produce DOIs and/or citations that seem authentic but are in fact incorrect or nonexistent. The Levenshtein distance, also known as the edit distance, is a measure of the similarity between two strings by calculating the minimum number of single-character edits (insertions, deletions, or substitutions) required to transform one string into the other. In other words, this metric quantifies the “distance” between two strings in terms of the minimum number of operations needed to make them identical. We used the Levenshtein distance to compare the DOI generated by ChatGPT with the correct DOI. This comparison helps to measure how closely the artificial intelligence (AI)–generated DOI aligns with the expected DOI for a given citation. By calculating the Levenshtein distance, we can quantify the differences between the AI-generated DOI and the correct DOI. Larger Levenshtein distance values suggest greater dissimilarity, indicating potential inaccuracies in the AI-generated DOI.

Statistical Analysis

Data analysis was conducted using SPSS 26 and Python. The Levenshtein distance [ 20 ] between the generated DOI and the actual DOI was calculated using the thefuzz package in Python to quantitatively assess the DOI accuracy. Continuous variables are reported as mean (SD) and categorical variables are presented as absolute numbers and percentages. An independent-sample t test was used to compare continuous variables, whereas the Fisher exact test was used for comparisons of categorical variables. A P value <.05 was considered statistically significant in all tests.

Ethical Considerations

This study was exempt from ethical review since no animal or human participants were involved.

Included Topics and Citations

Ten manuscript topics were selected and included in the study, with 5 in the natural sciences group and 5 in the humanities group. ChatGPT 3.5 was prompted to write an introduction section for each topic between July 10 and August 15, 2023. A total of 102 citations were generated by ChatGPT. Of these, 55 were in the natural sciences group and 47 in the humanities group. The existence, validity, and relevance of citations were examined irrespective of the corresponding DOIs. The results are summarized in Table 2 . A list of the included topics and a sample of prompts to ChatGPT are provided in Multimedia Appendix 1 .

a Categorical variables were compared using the Fisher exact test; the continuous variable (Levenshtein distance) was compared using the independent-sample t test.

b DOI: Digital Object Identifier.

Citation Existence and Accuracy

Of the 102 generated citations, 76 (74.5%) were found to be real and exist in the published literature, with 72.7% and 76.6% of the citations verified in the natural and humanities group, respectively. There was no significant difference between the two groups ( P =.42), indicating that the validity of the citations was relatively consistent between the two domains. Similarly, when assessing the accuracy of the citations, no significant difference was observed ( Table 2 ).

Citation Relevance

The relevance of citations generated by ChatGPT was evaluated by assessing whether they were appropriate and contextually meaningful within the research topics. Our analysis indicated that 70.9% and 74.5% of citations in the natural sciences and humanities categories were deemed relevant, respectively ( Table 2 ). The difference was not statistically significant ( P =.43), suggesting that ChatGPT demonstrated a similar ability to generate contextually relevant citations in both domains.

DOI Existence, Accuracy, and Hallucination

Our analysis revealed significant differences between the two domains with respect to DOIs. In the natural sciences, 70.9% of the included DOIs were real, whereas in the humanities, only 38.3% of the DOIs generated were real ( P =.001; Table 2 ). Similarly, the level of DOI accuracy was significantly higher for the natural sciences than for the humanities ( P =.003). Moreover, the occurrence of DOI hallucination, where ChatGPT generates DOIs that do not correspond with the existing literature, was more prevalent in the humanities than in the natural sciences ( P =.001). The mean Levenshtein distance, which measures the deviation between the generated DOI and the actual DOI, was significantly higher in the natural sciences group than in the humanities ( P =.009; Table 2 ).

Principal Findings

The results of this study shed light on the performance of ChatGPT (GPT-3.5) as an academic writing assistant in generating citations and references in natural sciences and humanities topics. Our findings reveal notable differences in the accuracy and reliability of the citations and references generated by ChatGPT when applied to natural sciences and humanities topics. Hallucination in the context of LLMs such as ChatGPT refers to a phenomenon where the model generates content that is incorrect, fabricated, or not grounded in reality. Hallucination occurs when the model produces information that appears plausible or contextually relevant but lacks accuracy or fidelity to real-world knowledge.

The most striking observation was the significant disparity in the existence and accuracy of the DOIs between the two domains. In natural sciences topics, DOIs were real in 70.9% of the generated citations, representing a significantly higher rate compared to the low rate of 38.3% real DOIs in the humanities topics. The discrepancies in the DOI existence and accuracy in the two domains may be attributed to the differential adoption and availability of DOIs across academic disciplines, where the natural sciences literature has often been more proactive in adopting the DOI system of referencing and linking to scholarly works than the humanities. It is a general practice that journals publishing on the natural sciences frequently mandate DOI inclusion, whereas publishers in the humanities have been slower to adopt such standards [ 21 , 22 ]. Consequently, the performance of the ChatGPT LLM in generating accurate DOIs appears to reflect these disciplinary disparities.

LLMs may generate fictional “facts” presented as true “real-world facts,” which is referred to as hallucination [ 19 , 23 ]. In this study, we considered hallucination to have occurred if the DOI of the generated citation was not real or was real but was linked to a different source. DOI hallucination was more frequent in the humanities (89.4%) than in the natural sciences (61.8%). This finding may be explained by the broader and less structured nature of the humanities literature. There is also a high tendency to provide citations from books and other media that do not use DOIs in the humanities. Therefore, researchers in the humanities should not consider DOIs generated by ChatGPT. Even when ChatGPT generates DOIs for humanities citations, they are more likely to deviate from the correct DOI, potentially leading to the inability to access the cited sources and use the DOIs in citation management tools such as EndNote.

In contrast to the disparities observed in DOI-related metrics, our study found a remarkable consistency in the existence, validity, and relevance of the generated citations in the natural sciences and humanities, with real citations found 72.7% and 76.6% of the time and accurate citations confirmed in 67.3% and 61.7% of cases, respectively. This suggests that the citations generated by ChatGPT can be expected to be reliable approximately 60% of the time.

The divergent performance of ChatGPT between the natural sciences and humanities underscores the importance of considering disciplinary nuances when implementing AI-driven writing assistants in academic contexts. Researchers and writers in both domains should be aware of the strengths and limitations of such tools, particularly in relation to citation practices and DOI accuracy. Future research could delve deeper into the factors influencing DOI accuracy and explore strategies for improving DOI generation by LLMs in the humanities literature. Additionally, the development of domain-specific AI writing models may offer tailored solutions to enhance citation and reference accuracy in various academic disciplines.

In this study, we focused only on the potential use of LLMs in citations and references in scholarly writing; however, the scope to which these models are going to be adopted in academic works is much broader. We believe that these models will be improved over time and that they are here to stay. As such, our argument in this paper is not that LLMs should not be used in scholarly writing, but rather that in their iteration, we ought to be aware of their limitations, primarily concerning the reliability of not only the text they generate but also how they interpret that text.

Although the transformer models that are the foundation of LLMs are very capable of handling a significant amount of information, they still do have context-window limitations. The context window is the textual range or span of the input that the LLM can evaluate to generate a response at any given moment. As an example, GPT-3 has a context window of 2000 tokens, whereas GPT-4’s context window is 32,000 tokens. As such, since the size of the context window impacts model performance (larger is better), GPT-4 outperforms GPT-3 (at the cost of more computation and memory). In scientific knowledge, context is key. Removing a word from the context may greatly affect the information being conveyed. Therefore, we believe that the future of LLMs in academia will rely on fine-tuning the LLMs to capitalize on memorization where necessary, reproducibility and stability of the models, as well as access to the latest information rather than only the training data.

Limitations

There were several limitations to this study. The study included a limited number of topics (10 in total), which can only offer insight but cannot possibly cover the full spectrum of complexity and diversity within the two disciplines. Only ChatGPT 3.5 was prompted since it is the most widely used LLM for this purpose and has a free tier that the majority of users rely on. Newer models, including GPT-4, Claude+, and Google’s Gemini, may give significantly different results. Our study focused on the accuracy of citations and DOIs without an exploration of potential user feedback or subjective assessment of the overall quality and coherence of the generated content. These limitations can be addressed in future research.

In conclusion, our study provides valuable insights into the performance of ChatGPT in generating citations and references across interdisciplinary domains. These findings contribute to the ongoing discourse on the use of LLMs in scholarly writing, emphasizing the need for nuanced consideration of discipline-specific challenges and the importance of robust validation processes to ensure the accuracy and reliability of generated content.

Acknowledgments

This work was supported by the Municipal Key Technical Research and Development Program of Ningbo (2023Z160).

Data Availability

The data sets generated during and/or analyzed during this study are available from the corresponding author upon reasonable request.

Conflicts of Interest

None declared.

List of included topics and ChatGPT 3.5 prompt structure.

Golan R, Reddy R, Muthigi A, Ramasamy R. Artificial intelligence in academic writing: a paradigm-shifting technological advance. Nat Rev Urol. Jun 2023;20(6):327-328. [ CrossRef ] [ Medline ]
Checco A, Bracciale L, Loreti P, Pinfield S, Bianchi G. AI-assisted peer review. Humanit Soc Sci Commun. Jan 25, 2021;8(1):25. [ CrossRef ]
Hutson M. Could AI help you to write your next paper? Nature. Nov 31, 2022;611(7934):192-193. [ CrossRef ] [ Medline ]
Conroy G. Scientists used ChatGPT to generate an entire paper from scratch - but is it any good? Nature. Jul 2023;619(7970):443-444. [ CrossRef ] [ Medline ]
Paskin N. Digital object identifier (DOI®) system. In: Encyclopedia of Library and Information Sciences. Milton Park. Taylor and Francis; 2010;1586-1592.
Hosseini M, Horbach SPJM. Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other large language models in scholarly peer review. Res Integr Peer Rev. May 18, 2023;8(1):4. [ FREE Full text ] [ CrossRef ] [ Medline ]
Santra PP, Majhi D. Scholarly communication and machine-generated text: is it finally AI vs AI in plagiarism detection? J Inf Knowl. Jul 01, 2023;60(3):175-183. [ CrossRef ]
Esplugas M. The use of artificial intelligence (AI) to enhance academic communication, education and research: a balanced approach. J Hand Surg Eur Vol. Sep 07, 2023;48(8):819-822. [ CrossRef ] [ Medline ]
Kumar AHS. Analysis of ChatGPT tool to assess the potential of its utility for academic writing in biomedical domain. BEMS Reports. Jan 05, 2023;9(1):24-30. [ CrossRef ]
Marchandot B, Matsushita K, Carmona A, Trimaille A, Morel O. ChatGPT: the next frontier in academic writing for cardiologists or a pandora's box of ethical dilemmas. Eur Heart J Open. Mar 2023;3(2):oead007. [ FREE Full text ] [ CrossRef ] [ Medline ]
Zheng H, Zhan H. ChatGPT in scientific writing: a cautionary tale. Am J Med. Aug 2023;136(8):725-726. [ CrossRef ] [ Medline ]
Alkaissi H, McFarlane S. Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus. Feb 2023;15(2):e35179. [ FREE Full text ] [ CrossRef ] [ Medline ]
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A, et al. Attention is all you need. arXiv. [ FREE Full text ]
Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. arXiv. [ FREE Full text ]
Radford A, Narasimhan K, Salimans T, Sutskever I. Improving language understanding by generative pre-training. OpenAI. 2018. URL: https://openai.com/research/language-unsupervised [accessed 2024-04-01]
Devlin J, Chang M, Lee K, Toutanova K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv. Preprint posted online on October 11, 2018. [ CrossRef ]
Degelman D, Harris M. APA style essentials. Vanguard University. 2000. URL: https://www.vanguard.edu/uploaded/research/apa_style_guide/apastyleessentials.pdf [accessed 2024-04-01]
DOI Foundation. URL: https://doi.org/ [accessed 2024-04-01]
Salvagno M, Taccone FS, Gerli AG. Artificial intelligence hallucinations. Crit Care. May 10, 2023;27(1):180. [ CrossRef ] [ Medline ]
Yujian L, Bo L. A normalized Levenshtein distance metric. IEEE Trans Pattern Anal Machine Intell. Jun 2007;29(6):1091-1095. [ CrossRef ]
Eve MP. Open Access and the Humanities. Cambridge, UK. Cambridge University Press; 2014.
Narayan B, Luca E, Tiffen B, England A, Booth M, Boateng H. Scholarly communication practices in humanities and social sciences: a study of researchers' attitudes and awareness of open access. Open Inf Sci. 2018;2(1):168-180. [ CrossRef ]
Zhang Y, Li Y, Cui L, Cai D, Liu L, Fu T, et al. Siren's song in the AI ocean: a survey on hallucination in large language models. arXiv. [ FREE Full text ]

Abbreviations

Edited by A Mavragani; submitted 19.09.23; peer-reviewed by Y Bu, W Li, I Liu, A Mihalache; comments to author 08.12.23; revised version received 14.12.23; accepted 12.03.24; published 05.04.24.

©Joseph Mugaanyi, Liuying Cai, Sumei Cheng, Caide Lu, Jing Huang. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 05.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

IMAGES

List of Sources APA: References Page
APA Basics: Fundamentals of Formatting Research Papers in APA Style
APA Reference Page Examples and Format Guide
Harvard Style
CSE 8th Edition
Essay Basics: Format a References Page in APA Style

VIDEO

Citation and Reference Definition,example, formats, explained in Urdu and Hindi|Research Methodology
Plagiarism l UGC NET Paper 1 l ln Telugu
Mendeley Cite-Inserting a Citation/Thesis References/Research Paper References
Citation and referencing in academic writing
Learn Zotero Citation & Reference Management
Different citation style#citation#researchaptitude

COMMENTS

References in Research
References in Research. Definition: References in research are a list of sources that a researcher has consulted or cited while conducting their study. They are an essential component of any academic work, including research papers, theses, dissertations, and other scholarly publications. Types of References
What Types of References Are Appropriate?
Potentially appropriate: books, encyclopedias, and other scholarly works. Another potential source that you might use when writing a research paper is a book, encyclopedia, or an official online source (such as demographic data drawn from a government website). When relying on such sources, it is important to carefully consider its accuracy and ...
Citation Styles Guide
There are three main approaches: Parenthetical citations: You include identifying details of the source in parentheses in the text—usually the author's last name and the publication date, plus a page number if relevant ( author-date ). Sometimes the publication date is omitted ( author-page ). Numerical citations: You include a number in ...
Academic Guides: Reference List: Common Reference List Examples
These instructional pages offer examples of reference list entries for different types of sources as well as guidance on the variations for citing online materials using doi numbers and URLs. ... peer-reviewed journals. However, they present original research and are often useful for research papers. Sometimes, researchers refer to these types ...
Reference List: Basic Rules
Reference List: Basic Rules. This resourse, revised according to the 7 th edition APA Publication Manual, offers basic guidelines for formatting the reference list at the end of a standard APA research paper. Most sources follow fairly straightforward rules. However, because sources obtained from academic journals carry special weight in research writing, these sources are subject to special ...
How to Cite in APA Format (7th edition)
APA in-text citations The basics. In-text citations are brief references in the running text that direct readers to the reference entry at the end of the paper. You include them every time you quote or paraphrase someone else's ideas or words to avoid plagiarism.. An APA in-text citation consists of the author's last name and the year of publication (also known as the author-date system).
References
References provide the information necessary for readers to identify and retrieve each work cited in the text. Consistency in reference formatting allows readers to focus on the content of your reference list, discerning both the types of works you consulted and the important reference elements with ease.
LibGuides: Citation: Citation Styles (APA, MLA, Chicago, etc.)
How to Format a Citation. There are two basic approaches to citation: In-text citations + a list of references at the end of the paper. Endnotes or footnotes +/- a bibliography at the end of the paper. Scholars writing in the sciences and social sciences typically use in-text citations, while humanities scholars utilize endnotes/footnotes.
Referencing styles
When citing a work with three or more authors, use the first author's last name plus 'et al.'. If you cite multiple references by the same author that were published in the same year, distinguish between them by adding labels (e.g. 'a' and 'b') to the year, in both the citation and the reference list.
Reference Types
References appear at the end of your document and follow a Who, When, What and Where format. Only include sources you cited within your research document (*exception for personal communications which are excluded from the reference list). Who: Identify the creator of the source. Often this is: a single author; multiple authors; an organization ...
A Quick Guide to Referencing
In-text citations are quick references to your sources. In Harvard referencing, you use the author's surname and the date of publication in brackets. Up to three authors are included in a Harvard in-text citation. If the source has more than three authors, include the first author followed by ' et al. '.
References: Common Reference List Examples
However, they present original research and are often useful for research papers. Sometimes, researchers refer to these types of reports as gray literature, and white papers are a type of this literature. See APA 7, Section 10.4 for more information.
How to Cite Sources
At college level, you must properly cite your sources in all essays, research papers, and other academic texts (except exams and in-class exercises). Add a citation whenever you quote, paraphrase, or summarize information or ideas from a source. You should also give full source details in a bibliography or reference list at the end of your text.
Types of Sources
This section lists the types of sources most frequently used in academic research and describes the sort of information that each commonly offers. Print Sources Books and Textbooks: Odds are that at least one book has been written about virtually any research topic you can imagine (and if not, your research could represent the first steps ...
Formatting References for Scientific Manuscripts
Other resources for citing references include the PubMed section 2 which gives samples of formatting of different reference types and the eBook: Citing Medicine, 3 published by the U. S. National Library of Medicine, which gives assistance and rules to authors, editors, publishers and librarians for formatting of references for different ...
A Quick Guide to Harvard Referencing
When you cite a source with up to three authors, cite all authors' names. For four or more authors, list only the first name, followed by ' et al. ': Number of authors. In-text citation example. 1 author. (Davis, 2019) 2 authors. (Davis and Barrett, 2019) 3 authors.
How to Write References in Research Papers
One of the the most common type s of reference s in research papers is other research papers published in journals; other common sources include technical reports, handbooks, presentations at conferences, and books. Increasingly, the sources in research papers are digital and include web pages, databases, blog posts, and even tweets and emails.
How To Cite a Research Paper in 2024: Citation Styles Guide
There are two main kinds of titles. Firstly, titles can be the name of the standalone work like books and research papers. In this case, the title of the work should appear in the title element of the reference. Secondly, they can be a part of a bigger work, such as edited chapters, podcast episodes, and even songs.
(PDF) A Manual for Referencing Styles in Research
The present manual has been designed in a way to elaborate the major differences that exist among various types of referencing styles so the conversion from one style to the other becomes easier ...
The art of referencing: Well begun is half done!
Organizing the References. An initial extensive literature search helps in identifying the appropriate research question, drafting the study protocol, supervising ongoing research, analyzing the results, and writing the paper.[3,7] Although references are displayed at the end of the article/after the text of the article, they should not be actually written after completing the text of the ...
Predicting and improving complex beer flavor through machine ...
The perception and appreciation of food flavor depends on many interacting chemical compounds and external factors, and therefore proves challenging to understand and predict. Here, we combine ...
Examining the role of community resilience and social capital on mental
The ability of the public to remain psychologically resilient in the face of public health emergencies and disasters (such as the COVID-19 pandemic) is a key factor in the effectiveness of a national response to such events. Community resilience and social capital are often perceived as beneficial and ensuring that a community is socially and psychologically resilient may aid emergency ...
What Are Credible Sources & How to Spot Them
Types of sources. There are many different types of sources, which can be divided into three categories: primary sources, secondary sources, and tertiary sources. Primary sources are often considered the most credible in terms of providing evidence for your argument, as they give you direct evidence of what you are researching.
Journal of Medical Internet Research
Background: Large language models (LLMs) have gained prominence since the release of ChatGPT in late 2022. Objective: The aim of this study was to assess the accuracy of citations and references generated by ChatGPT (GPT-3.5) in two distinct academic domains: the natural sciences and humanities. Methods: Two researchers independently prompted ChatGPT to write an introduction section for a ...

References in Research – Types, Examples and Writing Guide

References in Research

Types of References

Journal Articles

Web sources

Formats of References

APA (American Psychological Association) Style

MLA (Modern Language Association) Style

Chicago Manual of Style

Harvard Style

Vancouver Style

Turabian Style

IEEE (Institute of Electrical and Electronics Engineers) Style

ACS (American Chemical Society) Style

AMA (American Medical Association) Style

ASCE (American Society of Civil Engineers) Style

CSE (Council of Science Editors) Style

Bluebook Style

Oxford Style

AAA (American Anthropological Association) Style

AIP (American Institute of Physics) Style

References Writing Guide

Purpose of References in Research

Importance of References in Research

Advantages of References in Research

About the author

Muhammad Hassan

You may also like

Data Collection – Methods Types and Examples

Delimitations in Research – Types, Examples and...

Research Process – Steps, Examples and Tips

Research Design – Types, Methods and Examples

Institutional Review Board – Application Sample...

Evaluating Research – Process, Examples and...

Writing Research Papers

Highly appropriate: peer-reviewed journal articles

Potentially appropriate: books, encyclopedias, and other scholarly works

Usually inappropriate: magazines, blogs, and websites

Downloadable Resources

Further Resources

1 Wikipedia articles vary in quality; the site has a peer review system and the very best articles ( Featured Articles ), which go through a multi-stage review process, rival those in traditional encyclopedias and are considered the highest quality articles on the site.

How to Format a Citation

Referencing styles

Introduction

Author-date citations (Harvard)

In-text citation

Structure of the reference list

Examples of author-date references in British style

Chapter in an edited book

Journal article

Magazine article

Website or other source

As you write ...

Reference list

Further reading

Examples of author-date references in US style

Reference list vs. bibliography

Example: author–date citation with a reference list and further reading—US style

Numbered notes

Endnotes or footnotes?

Note structure and format

Numbered notes in British style

Edited book

Authored book with an editor or translator

Multi-volume work

Unpublished or informally published content

Numbered notes in US style

Law - case law

Short citations

Legal short citations

Example: short citations in US style

Numbered notes in combination with a bibliography

Structure of a bibliography entry

Bibliography entries in British Style

Sample bibliography

Bibliography entries in US style

Numbered reference citations (Vancouver)

Numbered reference citations in British style

Unauthored book (books published by committee, agency, or group)

Magazine or newspaper article