original article in research paper

• SpringerLink shop

Types of journal articles

It is helpful to familiarise yourself with the different types of articles published by journals. Although it may appear there are a large number of types of articles published due to the wide variety of names they are published under, most articles published are one of the following types; Original Research, Review Articles, Short reports or Letters, Case Studies, Methodologies.

Original Research:

This is the most common type of journal manuscript used to publish full reports of data from research. It may be called an  Original Article, Research Article, Research, or just  Article, depending on the journal. The Original Research format is suitable for many different fields and different types of studies. It includes full Introduction, Methods, Results, and Discussion sections.

Short reports or Letters:

These papers communicate brief reports of data from original research that editors believe will be interesting to many researchers, and that will likely stimulate further research in the field. As they are relatively short the format is useful for scientists with results that are time sensitive (for example, those in highly competitive or quickly-changing disciplines). This format often has strict length limits, so some experimental details may not be published until the authors write a full Original Research manuscript. These papers are also sometimes called Brief communications .

Review Articles:

Review Articles provide a comprehensive summary of research on a certain topic, and a perspective on the state of the field and where it is heading. They are often written by leaders in a particular discipline after invitation from the editors of a journal. Reviews are often widely read (for example, by researchers looking for a full introduction to a field) and highly cited. Reviews commonly cite approximately 100 primary research articles.

TIP: If you would like to write a Review but have not been invited by a journal, be sure to check the journal website as some journals to not consider unsolicited Reviews. If the website does not mention whether Reviews are commissioned it is wise to send a pre-submission enquiry letter to the journal editor to propose your Review manuscript before you spend time writing it.  

Case Studies:

These articles report specific instances of interesting phenomena. A goal of Case Studies is to make other researchers aware of the possibility that a specific phenomenon might occur. This type of study is often used in medicine to report the occurrence of previously unknown or emerging pathologies.

Methodologies or Methods

These articles present a new experimental method, test or procedure. The method described may either be completely new, or may offer a better version of an existing method. The article should describe a demonstrable advance on what is currently available.

Back │ Next

• Main Library
• Digital Fabrication Lab
• Data Visualization Lab
• Business Learning Center
• Klai Juba Wald Architectural Studies Library
• NDSU Nursing at Sanford Health Library
• Research Assistance
• Special Collections
• Digital Collections
• Collection Development Policy
• Course Reserves
• Request Library Instruction
• Main Library Services
• Alumni & Community
• Academic Support Services in the Library
• Libraries Resources for Employees
• Book Equipment or Study Rooms
• Librarians by Academic Subject
• Germans from Russia Heritage Collection
• NDSU Archives
• Mission, Vision, and Strategic Plan 2022-2024
• Staff Directory
• Floor Plans
• The Libraries Magazine
• Accommodations for People with Disabilities
• Annual Report
• Donate to the Libraries
• Equity, Diversity and Inclusion
• Faculty Senate Library Committee
• Undergraduate Research Award

What is an original research article?

An original research article is a report of research activity that is written by the researchers who conducted the research or experiment. Original research articles may also be referred to as: “primary research articles” or “primary scientific literature.” In science courses, instructors may also refer to these as “peer-reviewed articles” or “refereed articles.”

Original research articles in the sciences have a specific purpose, follow a scientific article format, are peer reviewed, and published in academic journals.

Identifying Original Research: What to Look For

An "original research article" is an article that is reporting original research about new data or theories that have not been previously published. That might be the results of new experiments, or newly derived models or simulations. The article will include a detailed description of the methods used to produce them, so that other researchers can verify them. This description is often found in a section called "methods" or "materials and methods" or similar. Similarly, the results will generally be described in great detail, often in a section called "results."

Since the original research article is reporting the results of new research, the authors should be the scientists who conducted that research. They will have expertise in the field, and will usually be employed by a university or research lab.

In comparison, a newspaper or magazine article (such as in  The New York Times  or  National Geographic ) will usually be written by a journalist reporting on the actions of someone else.

An original research article will be written by and for scientists who study related topics. As such, the article should use precise, technical language to ensure that other researchers have an exact understanding of what was done, how to do it, and why it matters. There will be plentiful citations to previous work, helping place the research article in a broader context. The article will be published in an academic journal, follow a scientific format, and undergo peer-review.

Original research articles in the sciences follow the scientific format. ( This tutorial from North Carolina State University illustrates some of the key features of this format.)

Look for signs of this format in the subject headings or subsections of the article. You should see the following:

Scientific research that is published in academic journals undergoes a process called "peer review."

The peer review process goes like this:

• A researcher writes a paper and sends it in to an academic journal, where it is read by an editor
• The editor then sends the article to other scientists who study similar topics, who can best evaluate the article
• The scientists/reviewers examine the article's research methodology, reasoning, originality, and sginificance
• The scientists/reviewers then make suggestions and comments to impove the paper
• The original author is then given these suggestions and comments, and makes changes as needed
• This process repeats until everyone is satisfied and the article can be published within the academic journal

For more details about this process see the Peer Reviewed Publications guide.

This journal article  is an example. It was published in the journal  Royal Society Open Science  in 2015. Clicking on the button that says "Review History" will show the comments by the editors, reviewers and the author as it went through the peer review process. The "About Us" menu provides details about this journal; "About the journal" under that tab includes the statement that the journal is peer reviewed.

Review articles

There are a variety of article types published in academic, peer-reviewed journals, but the two most common are original research articles and review articles . They can look very similar, but have different purposes and structures.

Like original research articles, review articles are aimed at scientists and undergo peer-review. Review articles often even have “abstract,” “introduction,” and “reference” sections. However, they will not (generally) have a “methods” or “results” section because they are not reporting new data or theories. Instead, they review the current state of knowledge on a topic.

Press releases, newspaper or magazine articles

These won't be in a formal scientific format or be peer reviewed. The author will usually be a journalist, and the audience will be the general public. Since most readers are not interested in the precise details of the research, the language will usually be nontechnical and broad. Citations will be rare or nonexistent.

Tips for Finding Original research Articles

Search for articles in one of the library databases recommend for your subject area . If you are using Google, try searching in Google Scholar instead and you will get results that are more likely to be original research articles than what will come up in a regular Google search!

For tips on using library databases to find articles, see our Library DIY guides .

Tips for Finding the Source of a News Report about Science

If you've seen or heard a report about a new scientific finding or claim, these tips can help you find the original source:

• Often, the report will mention where the original research was published; look for sentences like "In an article published yesterday in the journal  Nature ..." You can use this to find the issue of the journal where the research was published, and look at the table of contents to find the original article.
• The report will often name the researchers involved. You can search relevant databases for their name and the topic of the report to find the original research that way.
• Sometimes you may have to go through multiple articles to find the original source. For example, a video or blog post may be based on a newspaper article, which in turn is reporting on a scientific discovery published in another journal; be sure to find the original research article.
• Don't be afraid to ask a librarian for help!

Search The Site

Find Your Librarian  

Phone:  Circulation:  (701) 231-8888 Reference:  (701) 231-8888 Administration:  (701) 231-8753

Email:  Administration InterLibrary Loan (ILL)

• Online Services
• Phone/Email Directory
• Registration And Records
• Government Information
• Library DIY
• Subject and Course Guides
• Special Topics
• Collection Highlights
• Digital Horizons
• NDSU Repository (IR)
• Libraries Hours
• News & Events

• Enroll & Pay
• Prospective Students
• Current Students
• Degree Programs

Original Research

An original research paper should present a unique argument of your own. In other words, the claim of the paper should be debatable and should be your (the researcher’s) own original idea. Typically an original research paper builds on the existing research on a topic, addresses a specific question, presents the findings according to a standard structure (described below), and suggests questions for further research and investigation. Though writers in any discipline may conduct original research, scientists and social scientists in particular are interested in controlled investigation and inquiry. Their research often consists of direct and indirect observation in the laboratory or in the field. Many scientists write papers to investigate a hypothesis (a statement to be tested).

Although the precise order of research elements may vary somewhat according to the specific task, most include the following elements:

• Table of contents
• List of illustrations
• Body of the report
• References cited

Check your assignment for guidance on which formatting style is required. The Complete Discipline Listing Guide (Purdue OWL)  provides information on the most common style guide for each discipline, but be sure to check with your instructor.

The title of your work is important. It draws the reader to your text. A common practice for titles is to use a two-phrase title where the first phrase is a broad reference to the topic to catch the reader’s attention. This phrase is followed by a more direct and specific explanation of your project. For example:

“Lions, Tigers, and Bears, Oh My!: The Effects of Large Predators on Livestock Yields.”

The first phrase draws the reader in – it is creative and interesting. The second part of the title tells the reader the specific focus of the research.

In addition, data base retrieval systems often work with  keywords  extracted from the title or from a list the author supplies. When possible, incorporate them into the title. Select these words with consideration of how prospective readers might attempt to access your document. For more information on creating keywords, refer to this  Springer research publication guide.

See the KU Writing Center Writing Guide on Abstracts for detailed information about creating an abstract.

Table of Contents

The table of contents provides the reader with the outline and location of specific aspects of your document. Listings in the table of contents typically match the headings in the paper. Normally, authors number any pages before the table of contents as well as the lists of illustrations/tables/figures using lower-case roman numerals. As such, the table of contents will use lower-case roman numbers to identify the elements of the paper prior to the body of the report, appendix, and reference page. Additionally, because authors will normally use Arabic numerals (e.g., 1, 2, 3) to number the pages of the body of the research paper (starting with the introduction), the table of contents will use Arabic numerals to identify the main sections of the body of the paper (the introduction, literature review, methods, results, discussion, conclusion, references, and appendices).

Here is an example of a table of contents:

ABSTRACT..................................................iii

TABLE OF CONTENTS...............................iv

LIST OF ILLUSTRATIONS...........................v

LIST OF TABLES.........................................vii

INTRODUCTION..........................................1

LITERATURE REVIEW.................................6

METHODS....................................................9

RESULTS....................................................10

DISCUSSION..............................................16

CONCLUSION............................................18

REFERENCES............................................20

APPENDIX................................................. 23

More information on creating a table of contents can be found in the Table of Contents Guide (SHSU)  from the Newton Gresham Library at Sam Houston State University.

List of Illustrations

Authors typically include a list of the illustrations in the paper with longer documents. List the number (e.g., Illustration 4), title, and page number of each illustration under headings such as "List of Illustrations" or "List of Tables.”

Body of the Report

The tone of a report based on original research will be objective and formal, and the writing should be concise and direct. The structure will likely consist of these standard sections:  introduction, methods, results, discussion, and conclusion . Typically, authors identify these sections with headings and may use subheadings to identify specific themes within these sections (such as themes within the literature under the literature review section).

Introduction

Given what the field says about this topic, here is my contribution to this line of inquiry.

The introduction often consists of the rational for the project. What is the phenomenon or event that inspired you to write about this topic? What is the relevance of the topic and why is it important to study it now? Your introduction should also give some general background on the topic – but this should not be a literature review. This is the place to give your readers and necessary background information on the history, current circumstances, or other qualities of your topic generally. In other words, what information will a layperson need to know in order to get a decent understanding of the purpose and results of your paper? Finally, offer a “road map” to your reader where you explain the general order of the remainder of your paper. In the road map, do not just list the sections of the paper that will follow. You should refer to the main points of each section, including the main arguments in the literature review, a few details about your methods, several main points from your results/analysis, the most important takeaways from your discussion section, and the most significant conclusion or topic for further research.   

Literature Review

This is what other researchers have published about this topic.

In the literature review, you will define and clarify the state of the topic by citing key literature that has laid the groundwork for this investigation. This review of the literature will identify relations, contradictions, gaps, and inconsistencies between previous investigations and this one, and suggest the next step in the investigation chain, which will be your hypothesis. You should write the literature review in the  present tense  because it is ongoing information.

Methods (Procedures)

This is how I collected and analyzed the information.

This section recounts the procedures of the study. You will write this in  past tense  because you have already completed the study. It must include what is necessary to replicate and validate the hypothesis. What details must the reader know in order to replicate this study? What were your purposes in this study? The challenge in this section is to understand the possible readers well enough to include what is necessary without going into detail on “common-knowledge” procedures. Be sure that you are specific enough about your research procedure that someone in your field could easily replicate your study. Finally, make sure not to report any findings in this section.

This is what I found out from my research.

This section reports the findings from your research. Because this section is about research that is completed, you should write it primarily in the  past tense . The form and level of detail of the results depends on the hypothesis and goals of this report, and the needs of your audience. Authors of research papers often use visuals in the results section, but the visuals should enhance, rather than serve as a substitute, for the narrative of your results. Develop a narrative based on the thesis of the paper and the themes in your results and use visuals to communicate key findings that address your hypothesis or help to answer your research question. Include any unusual findings that will clarify the data. It is a good idea to use subheadings to group the results section into themes to help the reader understand the main points or findings of the research. 

This is what the findings mean in this situation and in terms of the literature more broadly.

This section is your opportunity to explain the importance and implications of your research. What is the significance of this research in terms of the hypothesis? In terms of other studies? What are possible implications for any academic theories you utilized in the study? Are there any policy implications or suggestions that result from the study? Incorporate key studies introduced in the review of literature into your discussion along with your own data from the results section. The discussion section should put your research in conversation with previous research – now you are showing directly how your data complements or contradicts other researchers’ data and what the wider implications of your findings are for academia and society in general. What questions for future research do these findings suggest? Because it is ongoing information, you should write the discussion in the  present tense . Sometimes the results and discussion are combined; if so, be certain to give fair weight to both.

These are the key findings gained from this research.

Summarize the key findings of your research effort in this brief final section. This section should not introduce new information. You can also address any limitations from your research design and suggest further areas of research or possible projects you would complete with a new and improved research design.

References/Works Cited

See KU Writing Center  writing guides  to learn more about different citation styles like APA, MLA, and Chicago.  Make an appointment  at the KU Writing Center for more help. Be sure to format the paper and references based on the citation style that your professor requires or based on the requirements of the academic journal or conference where you hope to submit the paper.

The appendix includes attachments that are pertinent to the main document but are too detailed to be included in the main text. These materials should be titled and labeled (for example Appendix A: Questionnaire). You should refer to the appendix in the text with in-text references so the reader understands additional useful information is available elsewhere in the document. Examples of documents to include in the appendix include regression tables, tables of text analysis data, and interview questions.

Updated June 2022  

Scientific Manuscript Writing: Original Research, Case Reports, Review Articles

• First Online: 02 March 2024

Cite this chapter

• Kimberly M. Rathbun 5  

Manuscripts are used to communicate the findings of your work with other researchers. Writing your first manuscript can be a challenge. Journals provide guidelines to authors which should be followed closely. The three major types of articles (original research, case reports, and review articles) all generally follow the IMRAD format with slight variations in content. With planning and thought, manuscript writing does not have to be a daunting task.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Suggested Readings

Alsaywid BS, Abdulhaq NM. Guideline on writing a case report. Urol Ann. 2019;11(2):126–31.

Article   PubMed   PubMed Central   Google Scholar  

Cohen H. How to write a patient case report. Am J Health Syst Pharm. 2006;63(19):1888–92.

Article   PubMed   Google Scholar  

Cooper ID. How to write an original research paper (and get it published). J Med Lib Assoc. 2015;103:67–8.

Article   Google Scholar  

Gemayel R. How to write a scientific paper. FEBS J. 2016;283(21):3882–5.

Article   CAS   PubMed   Google Scholar  

Gülpınar Ö, Güçlü AG. How to write a review article? Turk J Urol. 2013;39(Suppl 1):44–8.

PubMed   PubMed Central   Google Scholar  

Huth EJ. Structured abstracts for papers reporting clinical trials. Ann Intern Med. 1987;106:626–7.

International Committee of Medical Journal Editors. Recommendations for the Conduct, Reporting, Editing and Publication of Scholarly Work in Medical Journals. http://www.ICMJE.org . Accessed 23 Aug 2022.

Liumbruno GM, Velati C, Pasqualetti P, Franchini M. How to write a scientific manuscript for publication. Blood Transfus. 2013;11:217–26.

McCarthy LH, Reilly KE. How to write a case report. Fam Med. 2000;32(3):190–5.

CAS   PubMed   Google Scholar  

Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, McKenzie JE. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160.

The Biosemantics Group. Journal/author name estimator. https://jane.biosemantics.org/ . Accessed 24 Aug 2022.

Weinstein R. How to write a manuscript for peer review. J Clin Apher. 2020;35(4):358–66.

Download references

Author information

Authors and affiliations.

Department of Emergency Medicine, AU/UGA Medical Partnership, Athens, GA, USA

Kimberly M. Rathbun

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Kimberly M. Rathbun .

Editor information

Editors and affiliations.

Emergency Medicine, Penn State Milton S. Hershey Medical Center, Hershey, PA, USA

Robert P. Olympia

Elizabeth Barrall Werley

Jeffrey S. Lubin

MD Anderson Cancer Center at Cooper, Cooper Medical School of Rowan University, Camden, NJ, USA

Kahyun Yoon-Flannery

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Rathbun, K.M. (2023). Scientific Manuscript Writing: Original Research, Case Reports, Review Articles. In: Olympia, R.P., Werley, E.B., Lubin, J.S., Yoon-Flannery, K. (eds) An Emergency Physician’s Path. Springer, Cham. https://doi.org/10.1007/978-3-031-47873-4_80

Download citation

DOI : https://doi.org/10.1007/978-3-031-47873-4_80

Published : 02 March 2024

Publisher Name : Springer, Cham

Print ISBN : 978-3-031-47872-7

Online ISBN : 978-3-031-47873-4

eBook Packages : Medicine Medicine (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

How to write an original article

Affiliations.

• 1 Servicio de Urología, Fundació Puigvert, Universidad Autónoma de Barcelona, Barcelona, España. Electronic address: [email protected].
• 2 Servicio de Urología, Fundació Puigvert, Universidad Autónoma de Barcelona, Barcelona, España.
• PMID: 29779648
• DOI: 10.1016/j.acuro.2018.02.011

Context: A correctly drafted original article gives information on what was done, why it was done, how it was done, the result of what was done, and the significance of what was done. Many articles fail to report their results effectively.

Objective: To describe the characteristics of an original article and to give practical recommendations to prevent the most common errors in our environment.

Evidence acquisition: We performed a systematic search of the terms "how to write a scientific article", "structure of the original article" and "publishing an article" in the databases PubMed and SCOPUS. We analysed the structure of an original article and the characteristics of its parts and prepared advice on the publication of an article.

Evidence synthesis: The journal's guidelines for authors should be read. It is usual for the original article to follow the IMRAD structure: Introduction, Methods, Results and Discussion. The introduction states briefly why the study was performed. The methods' section should give a detailed explanation of how the study was performed. The results should be clearly presented, with the help of tables, without repeating information. The discussion explains the relevance of the results and contrasts them with those of other authors. Any limitations and a conclusion supported by the results must be included.

Conclusions: Writing an original article correctly requires practice and it must be supported by a good research work in order to be published.

Keywords: Escribir un artículo; Estructura del artículo original; Publicar un artículo; Publishing an article; Structure of the original article; Writing an article.

Copyright © 2018 AEU. Publicado por Elsevier España, S.L.U. All rights reserved.

Publication types

• Guidelines as Topic
• Publishing / standards*
• Writing / standards*

• West Coast University Libraries
• Research Guides
• Find Materials
• West Coast University
• Research Help

WCU FAQs: Research Help

• APA and Citations
• General Education Classes
• Library Login and Information
• Program-Specific Classes
• Technology Help
• WCU General Information
• Writing and Tutoring Help
• 26 A-Z Database List
• 3 Academic Search Complete
• 1 Acland's Video Atlas of Human Anatomy
• 1 Alexander Street Academic Video
• 1 Authority
• 1 Business Administration
• 2 Business Source Complete
• 1 Credible or Reliable
• 31 Databases
• 2 Dentistry Oral Sciences
• 1 Evaluating Resources
• 3 Evidence based
• 13 Full Text
• 1 Hein Online or HeinOnline
• 1 Integrative Review
• 3 Interlibrary Loan or ILL
• 21 Journal Articles
• 13 Library Catalog
• 1 Library ID
• 1 Library Information
• 1 Link or URL
• 1 Literature Review
• 1 Mergent Intellect
• 2 MESH or mesh
• 2 NURS 350 or N350
• 2 Nursing Reference Center Plus
• 2 Off-campus
• 8 Opposing Viewpoints
• 5 Original Research
• 8 Peer reviewed
• 4 Primary Source
• 14 Proquest
• 3 Qualitative
• 3 Quantitative
• 10 Reference
• 1 Relevance
• 28 Research
• 13 Research Guides
• 1 Research Paper
• 4 Search or Searching
• 1 Secondary Source
• 4 Statistics
• 2 Subject Headings or Subject Terms
• 1 Up To Date

Research Help Question

Research help answer.

Many original research articles are not labeled as original research articles.  Original research articles include a research question or hypothesis.  They usually contain most of the following sections: methods, results, discussion, conclusion and references.  An original research article is written by the person or people that conducted the experiment or observations.  Original research articles are considered empirical or primary sources and present an original study.

Articles that look at multiple studies are not considered original research articles.  Search library databases using keywords like “study or “case study” to increase your chances of locating original research articles. 

For information on how to find an original research article that is not meta-analysis, not meta-synthesis and not mixed method, go to  https://westcoastuniversitylibrary.libanswers.com/research/faq/291851 .

Related Topics

• A-Z Database List
• Journal Articles
• Library Catalog
• Peer reviewed
• Primary Source
• Qualitative
• Quantitative
• Original Research
• Opposing Viewpoints

Have a Question?

• Share on Facebook

Was this helpful? Yes 0 No 0

Original Research Articles

Preparing your manuscript.

The title page should:

• present a title that includes, if appropriate, the study design
• if a collaboration group should be listed as an author, please list the Group name as an author. If you would like the names of the individual members of the Group to be searchable through their individual PubMed records, please include this information in the “Acknowledgements” section in accordance with the instructions below
• Large Language Models (LLMs), such as ChatGPT , do not currently satisfy our authorship criteria . Notably an attribution of authorship carries with it accountability for the work, which cannot be effectively applied to LLMs. Use of an LLM should be properly documented in the Methods section (and if a Methods section is not available, in a suitable alternative part) of the manuscript
• indicate the corresponding author

The abstract should not exceed 350 words. Please minimize the use of abbreviations and do not cite references in the abstract. The abstract must include the following separate sections:

• Background : the context and purpose of the study
• Results : the main findings
• Conclusions : a brief summary and potential implications

Three to ten keywords representing the main content of the article.

The Background section should explain the background to the study, its aims, a summary of the existing literature and why this study was necessary.

This should include the findings of the study including, if appropriate, results of statistical analysis which must be included either in the text or as tables and figures.

For research articles this section should discuss the implications of the findings in context of existing research and highlight limitations of the study. For study protocols and methodology manuscripts this section should include a discussion of any practical or operational issues involved in performing the study and any issues not covered in other sections.

Conclusions

This should state clearly the main conclusions and provide an explanation of the importance and relevance of the study to the field.

Methods (can also be placed after Background)

The methods section should include:

• the aim, design and setting of the study
• the characteristics of participants or description of materials
• a clear description of all processes, interventions and comparisons. Generic names should generally be used. When proprietary brands are used in research, include the brand names in parentheses
• the type of statistical analysis used, including a power calculation if appropriate

List of abbreviations

If abbreviations are used in the text they should be defined in the text at first use, and a list of abbreviations should be provided.

Declarations

All manuscripts must contain the following sections under the heading 'Declarations':

Ethics approval and consent to participate

Consent for publication.

• Availability of data and material

Competing interests

Authors' contributions, acknowledgements.

• Authors' information (optional)

Please see below for details on the information to be included in these sections.

If any of the sections are not relevant to your manuscript, please include the heading and write 'Not applicable' for that section.

Manuscripts reporting studies involving human participants, human data or human tissue must:

• include a statement on ethics approval and consent (even where the need for approval was waived)
• include the name of the ethics committee that approved the study and the committee’s reference number if appropriate

Studies involving animals must include a statement on ethics approval.

See our  editorial policies  for more information.

If your manuscript does not report on or involve the use of any animal or human data or tissue, please state “Not applicable” in this section.

If your manuscript contains any individual person’s data in any form (including individual details, images or videos), consent to publish must be obtained from that person, or in the case of children, their parent or legal guardian. All presentations of case reports must have consent to publish.

You can use your institutional consent form if you prefer. You should not send the form to us on submission, but we may request to see a copy at any stage (including after publication).

See our  editorial policies  for more information on consent for publication.

If your manuscript does not contain data from any individual person, please state “Not applicable” in this section.

Availability of data and materials

All manuscripts must include an ‘Availability of data and materials’ statement. Data availability statements should include information on where data supporting the results reported in the article can be found including, where applicable, hyperlinks to publicly archived datasets analysed or generated during the study. By data we mean the minimal dataset that would be necessary to interpret, replicate and build upon the findings reported in the article. We recognise it is not always possible to share research data publicly, for instance when individual privacy could be compromised, and in such instances data availability should still be stated in the manuscript along with any conditions for access.

Data availability statements can take one of the following forms (or a combination of more than one if required for multiple datasets):

• The datasets generated and/or analysed during the current study are available in the [NAME] repository, [PERSISTENT WEB LINK TO DATASETS]
• The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
• All data generated or analysed during this study are included in this published article [and its supplementary information files].
• The datasets generated and/or analysed during the current study are not publicly available due [REASON WHY DATA ARE NOT PUBLIC] but are available from the corresponding author on reasonable request.
• Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.
• The data that support the findings of this study are available from [third party name] but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of [third party name].
• Not applicable. If your manuscript does not contain any data, please state 'Not applicable' in this section.

More examples of template data availability statements, which include examples of openly available and restricted access datasets, are available  here .

SpringerOpen  also requires that authors cite any publicly available data on which the conclusions of the paper rely in the manuscript. Data citations should include a persistent identifier (such as a DOI) and should ideally be included in the reference list. Citations of datasets, when they appear in the reference list, should include the minimum information recommended by DataCite and follow journal style. Dataset identifiers including DOIs should be expressed as full URLs. For example:

Hao Z, AghaKouchak A, Nakhjiri N, Farahmand A. Global integrated drought monitoring and prediction system (GIDMaPS) data sets. figshare. 2014.  http://dx.doi.org/10.6084/m9.figshare.853801

With the corresponding text in the Availability of data and materials statement:

The datasets generated during and/or analysed during the current study are available in the [NAME] repository, [PERSISTENT WEB LINK TO DATASETS]. [Reference number]

All financial and non-financial competing interests must be declared in this section.

See our  editorial policies  for a full explanation of competing interests. If you are unsure whether you or any of your co-authors have a competing interest please contact the editorial office.

Please use the authors’ initials to refer to each authors' competing interests in this section.

If you do not have any competing interests, please state "The authors declare that they have no competing interests" in this section.

All sources of funding for the research reported should be declared. The role of the funding body in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript should be declared.

The individual contributions of authors to the manuscript should be specified in this section. Guidance and criteria for authorship can be found in our  editorial policies .

Please use initials to refer to each author's contribution in this section, for example: "FC analyzed and interpreted the patient data regarding the hematological disease and the transplant. RH performed the histological examination of the kidney, and was a major contributor in writing the manuscript. All authors read and approved the final manuscript."

Please acknowledge anyone who contributed towards the article who does not meet the criteria for authorship including anyone who provided professional writing services or materials.

Authors should obtain permission to acknowledge from all those mentioned in the Acknowledgements section.

See our  editorial policies  for a full explanation of acknowledgements and authorship criteria.

If you do not have anyone to acknowledge, please write "Not applicable" in this section.

Group authorship (for manuscripts involving a collaboration group): if you would like the names of the individual members of a collaboration Group to be searchable through their individual PubMed records, please ensure that the title of the collaboration Group is included on the title page and in the submission system and also include collaborating author names as the last paragraph of the “Acknowledgements” section. Please add authors in the format First Name, Middle initial(s) (optional), Last Name. You can add institution or country information for each author if you wish, but this should be consistent across all authors.

Please note that individual names may not be present in the PubMed record at the time a published article is initially included in PubMed as it takes PubMed additional time to code this information.

Authors' information

This section is optional.

You may choose to use this section to include any relevant information about the author(s) that may aid the reader's interpretation of the article, and understand the standpoint of the author(s). This may include details about the authors' qualifications, current positions they hold at institutions or societies, or any other relevant background information. Please refer to authors using their initials. Note this section should not be used to describe any competing interests.

Footnotes should be designated within the text using a superscript number. It is not allowed to use footnotes for references/citations.

Radiochemistry Nomenclature Guidelines

The EJNMMI Journal Family endorses the application of the International Consensus Radiochemistry Nomenclature Guidelines for contributions to the journals. A three page summary of the guidelines, highlighting the most relevant issues used in the notation of radiopharmaceuticals and related terms, is available at EJNMMI Radiopharmacy and Chemistry . The Editors-in-Chief of the EJNMMI Journal Family strongly recommend all manuscripts meet these guidelines submission, and all reviewers are asked to check, wherever possible, that the guidelines are followed. Their endorsement can be found here . 

The full paper with all recommendations is published in “Consensus nomenclature rules for radiopharmaceutical chemistry — setting the record straight”, Coenen and Gee et al. (2017), Nuclear Medicine and Biology   here .

Examples of the Vancouver reference style are shown below. 

See our editorial policies for author guidance on good citation practice.

Web links and URLs: All web links and URLs, including links to the authors' own websites, should be given a reference number and included in the reference list rather than within the text of the manuscript. They should be provided in full, including both the title of the site and the URL, as well as the date the site was accessed, in the following format: The Mouse Tumor Biology Database. http://tumor.informatics.jax.org/mtbwi/index.do . Accessed 20 May 2013. If an author or group of authors can clearly be associated with a web link, such as for weblogs, then they should be included in the reference.

Example reference style:

Article within a journal

Smith JJ. The world of science. Am J Sci. 1999;36:234-5.

Article within a journal (no page numbers)

Rohrmann S, Overvad K, Bueno-de-Mesquita HB, Jakobsen MU, Egeberg R, Tjønneland A, et al. Meat consumption and mortality - results from the European Prospective Investigation into Cancer and Nutrition. BMC Medicine. 2013;11:63.

Article within a journal by DOI

Slifka MK, Whitton JL. Clinical implications of dysregulated cytokine production. Dig J Mol Med. 2000; doi:10.1007/s801090000086.

Article within a journal supplement

Frumin AM, Nussbaum J, Esposito M. Functional asplenia: demonstration of splenic activity by bone marrow scan. Blood 1979;59 Suppl 1:26-32.

Book chapter, or an article within a book

Wyllie AH, Kerr JFR, Currie AR. Cell death: the significance of apoptosis. In: Bourne GH, Danielli JF, Jeon KW, editors. International review of cytology. London: Academic; 1980. p. 251-306.

OnlineFirst chapter in a series (without a volume designation but with a DOI)

Saito Y, Hyuga H. Rate equation approaches to amplification of enantiomeric excess and chiral symmetry breaking. Top Curr Chem. 2007. doi:10.1007/128_2006_108.

Complete book, authored

Blenkinsopp A, Paxton P. Symptoms in the pharmacy: a guide to the management of common illness. 3rd ed. Oxford: Blackwell Science; 1998.

Online document

Doe J. Title of subordinate document. In: The dictionary of substances and their effects. Royal Society of Chemistry. 1999. http://www.rsc.org/dose/title of subordinate document. Accessed 15 Jan 1999.

Online database

Healthwise Knowledgebase. US Pharmacopeia, Rockville. 1998. http://www.healthwise.org. Accessed 21 Sept 1998.

Supplementary material/private homepage

Doe J. Title of supplementary material. 2000. http://www.privatehomepage.com. Accessed 22 Feb 2000.

University site

Doe, J: Title of preprint. http://www.uni-heidelberg.de/mydata.html (1999). Accessed 25 Dec 1999.

Doe, J: Trivial HTTP, RFC2169. ftp://ftp.isi.edu/in-notes/rfc2169.txt (1999). Accessed 12 Nov 1999.

Organization site

ISSN International Centre: The ISSN register. http://www.issn.org (2006). Accessed 20 Feb 2007.

Dataset with persistent identifier

Zheng L-Y, Guo X-S, He B, Sun L-J, Peng Y, Dong S-S, et al. Genome data from sweet and grain sorghum (Sorghum bicolor). GigaScience Database. 2011. http://dx.doi.org/10.5524/100012 .

General formatting information

Manuscripts must be written in concise English. For help on scientific writing, or preparing your manuscript in English, please see Springer's  Author Academy .

Quick points:

• Use double line spacing
• Include line and page numbering
• Use SI units: Please ensure that all special characters used are embedded in the text, otherwise they will be lost during conversion to PDF
• Do not use page breaks in your manuscript

File formats

The following word processor file formats are acceptable for the main manuscript document:

• Microsoft word (DOC, DOCX)
• Rich text format (RTF)
• TeX/LaTeX 

Please note: editable files are required for processing in production. If your manuscript contains any non-editable files (such as PDFs) you will be required to re-submit an editable file if your manuscript is accepted.

For more information, see ' Preparing figures ' below.

Additional information for TeX/LaTeX users

You are encouraged to use the Springer Nature LaTeX template when preparing a submission. A PDF of your manuscript files will be compiled during submission using pdfLaTeX and TexLive 2021. All relevant editable source files must be uploaded during the submission process. Failing to submit these source files will cause unnecessary delays in the production process.  

Style and language

For editors and reviewers to accurately assess the work presented in your manuscript you need to ensure the English language is of sufficient quality to be understood. If you need help with writing in English you should consider:

• Getting a fast, free online grammar check .
• Visiting the English language tutorial which covers the common mistakes when writing in English.
• Asking a colleague who is proficient in English to review your manuscript for clarity.
• Using a professional language editing service where editors will improve the English to ensure that your meaning is clear and identify problems that require your review. Two such services are provided by our affiliates Nature Research Editing Service and American Journal Experts . SpringerOpen authors are entitled to a 10% discount on their first submission to either of these services. To claim 10% off English editing from Nature Research Editing Service, click here . To claim 10% off American Journal Experts, click here .

Please note that the use of a language editing service is not a requirement for publication in EJNMMI Research and does not imply or guarantee that the article will be selected for peer review or accepted.  为便于编辑和评审专家准确评估您稿件中陈述的研究工作，您需要确保文稿英语语言质量足以令人理解。如果您需要英文写作方面的帮助，您可以考虑：

• 获取快速、免费的在线  语法检查 。
• 查看一些有关英语写作中常见语言错误的 教程 。
• 请一位以英语为母语的同事审阅您的稿件是否表意清晰。
• 使用专业语言编辑服务，编辑人员会对英语进行润色，以确保您的意思表达清晰，并提出需要您复核的问题。例如我们的附属机构 Nature Research Editing Service 以及合作伙伴 American Journal Experts 都可以提供此类专业服务。SpringerOpen作者享受首次订单10%优惠，该优惠同时适用于两家公司。您只需点击以下链接即可开始。使用 Nature Research Editing Service的编辑润色10%的优惠服务，请点击 这里 。使用 American Journal Experts的10%优惠服务，请点击 这里 。

请注意，使用语言编辑服务并非在期刊上发表文章的必要条件，这也并不意味或保证文章将被选中进行同行评议或被接受。 エディターと査読者があなたの論文を正しく評価するには、使用されている英語の質が十分であることが必要とされます。英語での論文執筆に際してサポートが必要な場合には、次のオプションがあります：

• 高速なオンライン  文法チェック  を無料で受ける。
• 英語で執筆する際のよくある間違いに関する 英語のチュートリアル を参照する。
• 英語を母国語とする同僚に、原稿内の英語が明確であるかをチェックしてもらう。
• プロの英文校正サービスを利用する。校正者が原稿の意味を明確にしたり、問題点を指摘し、英語を向上させます。 Nature Research Editing Service と American Journal Experts の2つは弊社と提携しているサービスです。SpringerOpenのジャーナルの著者は、いずれかのサービスを初めて利用する際に、10%の割引を受けることができます。Nature Research Editing Serviceの10%割引を受けるには、 こちらをクリックしてください 。. American Journal Expertsの10%割引を受けるには、 こちらをクリックしてください 。

英文校正サービスの利用は、このジャーナルに掲載されるための条件ではないこと、また論文審査や受理を保証するものではないことに留意してください。 영어 원고의 경우, 에디터 및 리뷰어들이 귀하의 원고에 실린 결과물을 정확하게 평가할 수 있도록, 그들이 충분히 이해할 수 있을 만한 수준으로 작성되어야 합니다. 만약 영작문과 관련하여 도움을 받기를 원하신다면 다음의 사항들을 고려하여 주십시오:

• 영어 튜토리얼 페이지 에 방문하여 영어로 글을 쓸 때 자주하는 실수들을 확인합니다.
• 귀하의 원고의 표현을 명확히 해줄 영어 원어민 동료를 찾아서 리뷰를 의뢰합니다
• 리뷰에 대비하여, 원고의 의미를 명확하게 해주고 리뷰에서 요구하는 문제점들을 식별해서 영문 수준을 향상시켜주는 전문 영문 교정 서비스를 이용합니다. Nature Research Editing Service 와 American Journal Experts 에서 저희와 협약을 통해 서비스를 제공하고 있습니다. SpringerOpen에서는 위의 두 가지의 서비스를 첫 논문 투고를 위해 사용하시는 경우, 10%의 할인을 제공하고 있습니다. Nature Research Editing Service이용시 10% 할인을 요청하기 위해서는 여기 를 클릭해 주시고, American Journal Experts 이용시 10% 할인을 요청하기 위해서는 여기 를 클릭해 주십시오.

영문 교정 서비스는 게재를 위한 요구사항은 아니며, 해당 서비스의 이용이 피어 리뷰에 논문이 선택되거나 게재가 수락되는 것을 의미하거나 보장하지 않습니다.

Data and materials

For all journals, SpringerOpen strongly encourages all datasets on which the conclusions of the manuscript rely to be either deposited in publicly available repositories (where available and appropriate) or presented in the main paper or additional supporting files, in machine-readable format (such as spread sheets rather than PDFs) whenever possible. Please see the list of recommended repositories in our editorial policies.

For some journals, deposition of the data on which the conclusions of the manuscript rely is an absolute requirement. Please check the Instructions for Authors for the relevant journal and article type for journal specific policies.

For all manuscripts, information about data availability should be detailed in an ‘Availability of data and materials’ section. For more information on the content of this section, please see the Declarations section of the relevant journal’s Instruction for Authors. For more information on SpringerOpen's policies on data availability, please see our editorial policies .

Formatting the 'Availability of data and materials' section of your manuscript

The following format for the 'Availability of data and materials section of your manuscript should be used:

"The dataset(s) supporting the conclusions of this article is(are) available in the [repository name] repository, [unique persistent identifier and hyperlink to dataset(s) in http:// format]."

The following format is required when data are included as additional files:

"The dataset(s) supporting the conclusions of this article is(are) included within the article (and its additional file(s))."

For databases, this section should state the web/ftp address at which the database is available and any restrictions to its use by non-academics.

For software, this section should include:

• Project name: e.g. My bioinformatics project
• Project home page: e.g. http://sourceforge.net/projects/mged
• Archived version: DOI or unique identifier of archived software or code in repository (e.g. enodo)
• Operating system(s): e.g. Platform independent
• Programming language: e.g. Java
• Other requirements: e.g. Java 1.3.1 or higher, Tomcat 4.0 or higher
• License: e.g. GNU GPL, FreeBSD etc.
• Any restrictions to use by non-academics: e.g. licence needed

Information on available repositories for other types of scientific data, including clinical data, can be found in our editorial policies .

What should be cited?

Only articles, clinical trial registration records and abstracts that have been published or are in press, or are available through public e-print/preprint servers, may be cited.

Unpublished abstracts, unpublished data and personal communications should not be included in the reference list, but may be included in the text and referred to as "unpublished observations" or "personal communications" giving the names of the involved researchers. Obtaining permission to quote personal communications and unpublished data from the cited colleagues is the responsibility of the author. Either footnotes or endnotes are permitted. Journal abbreviations follow Index Medicus/MEDLINE.

Any in press articles cited within the references and necessary for the reviewers' assessment of the manuscript should be made available if requested by the editorial office.

Preparing figures

When preparing figures, please follow the formatting instructions below.

• Figure titles (max 15 words) and legends (max 300 words) should be provided in the main manuscript, not in the graphic file.
• Tables should NOT be submitted as figures but should be included in the main manuscript file.
• Multi-panel figures (those with parts a, b, c, d etc.) should be submitted as a single composite file that contains all parts of the figure.
• Figures should be numbered in the order they are first mentioned in the text, and uploaded in this order.
• Figures should be uploaded in the correct orientation.
• Figure keys should be incorporated into the graphic, not into the legend of the figure.
• Each figure should be closely cropped to minimize the amount of white space surrounding the illustration. Cropping figures improves accuracy when placing the figure in combination with other elements when the accepted manuscript is prepared for publication on our site. For more information on individual figure file formats, see our detailed instructions.
• Individual figure files should not exceed 10 MB. If a suitable format is chosen, this file size is adequate for extremely high quality figures.
• Please note that it is the responsibility of the author(s) to obtain permission from the copyright holder to reproduce figures (or tables) that have previously been published elsewhere. In order for all figures to be open access, authors must have permission from the rights holder if they wish to include images that have been published elsewhere in non open access journals. Permission should be indicated in the figure legend, and the original source included in the reference list.

Figure file types

We accept the following file formats for figures:

• EPS (suitable for diagrams and/or images)
• PDF (suitable for diagrams and/or images)
• Microsoft Word (suitable for diagrams and/or images, figures must be a single page)
• PowerPoint (suitable for diagrams and/or images, figures must be a single page)
• TIFF (suitable for images)
• JPEG (suitable for photographic images, less suitable for graphical images)
• PNG (suitable for images)
• BMP (suitable for images)
• CDX (ChemDraw - suitable for molecular structures)

Figure size and resolution

Figures are resized during publication of the final full text and PDF versions to conform to the SpringerOpen standard dimensions, which are detailed below.

Figures on the web:

• width of 600 pixels (standard), 1200 pixels (high resolution).

Figures in the final PDF version:

• width of 85 mm for half page width figure
• width of 170 mm for full page width figure
• maximum height of 225 mm for figure and legend
• image resolution of approximately 300 dpi (dots per inch) at the final size

Figures should be designed such that all information, including text, is legible at these dimensions. All lines should be wider than 0.25 pt when constrained to standard figure widths. All fonts must be embedded.

Figure file compression

Vector figures should if possible be submitted as PDF files, which are usually more compact than EPS files.

• TIFF files should be saved with LZW compression, which is lossless (decreases file size without decreasing quality) in order to minimize upload time.
• JPEG files should be saved at maximum quality.
• Conversion of images between file types (especially lossy formats such as JPEG) should be kept to a minimum to avoid degradation of quality.

If you have any questions or are experiencing a problem with figures, please contact the customer service team at [email protected] .

Preparing tables

When preparing tables, please follow the formatting instructions below.

• Tables should be numbered and cited in the text in sequence using Arabic numerals (i.e. Table 1, Table 2 etc.).
• Tables less than one A4 or Letter page in length can be placed in the appropriate location within the manuscript.
• Tables larger than one A4 or Letter page in length can be placed at the end of the document text file. Please cite and indicate where the table should appear at the relevant location in the text file so that the table can be added in the correct place during production.
• Larger datasets, or tables too wide for A4 or Letter landscape page can be uploaded as additional files. Please see [below] for more information.
• Tabular data provided as additional files can be uploaded as an Excel spreadsheet (.xls ) or comma separated values (.csv). Please use the standard file extensions.
• Table titles (max 15 words) should be included above the table, and legends (max 300 words) should be included underneath the table.
• Tables should not be embedded as figures or spreadsheet files, but should be formatted using ‘Table object’ function in your word processing program.
• Color and shading may not be used. Parts of the table can be highlighted using superscript, numbering, lettering, symbols or bold text, the meaning of which should be explained in a table legend.
• Commas should not be used to indicate numerical values.

If you have any questions or are experiencing a problem with tables, please contact the customer service team at [email protected] .

Preparing additional files

As the length and quantity of data is not restricted for many article types, authors can provide datasets, tables, movies, or other information as additional files.

All Additional files will be published along with the accepted article. Do not include files such as patient consent forms, certificates of language editing, or revised versions of the main manuscript document with tracked changes. Such files, if requested, should be sent by email to the journal’s editorial email address, quoting the manuscript reference number.

Results that would otherwise be indicated as "data not shown" should be included as additional files. Since many web links and URLs rapidly become broken, SpringerOpen requires that supporting data are included as additional files, or deposited in a recognized repository. Please do not link to data on a personal/departmental website. Do not include any individual participant details. The maximum file size for additional files is 20 MB each, and files will be virus-scanned on submission. Each additional file should be cited in sequence within the main body of text.

Submit manuscript

• Editorial Board
• Sign up for article alerts and news from this journal

Annual Journal Metrics

2022 Citation Impact 3.2 - 2-year Impact Factor 3.3 - 5-year Impact Factor 0.932 - SNIP (Source Normalized Impact per Paper) 0.853 - SJR (SCImago Journal Rank)

2023 Speed 12 days submission to first editorial decision for all manuscripts (Median) 87 days submission to accept (Median)

2023 Usage  604,325 downloads 175 Altmetric mentions 

• More about our metrics
• Follow us on Twitter
• Follow us on Facebook
• ISSN: 2191-219X (electronic)

Affiliated with

European Association of Nuclear Medicine

Visit and follow the  EANM  on their social media channels:

ENGL 101/102

• Background information and exploring your topic
• Make a great research question

Finding original research articles

• Searching Syntax This link opens in a new window
• Citations Guide
• Does NJIT have it?
• Interlibrary Loan This link opens in a new window
• Assignment Tips

Finding Research Articles About Your Topic

What kind of research has been done on your topic? Where can you find original research articles that have been published in peer-reviewed journals?  Use this page to learn about sources for finding peer-reviewed original research articles . Also, learn about  the nature and characteristics of peer-reviewed and original research articles.

Best bets for Original Research Articles-- Library databases for ENGL102

<p>Also explore the <a href="https://researchguides.njit.edu/az.php">Library&#39;s AZ Database list </a>to find scholarly resources dedicated to specific subjects (Psychology, Education, Transportation, etc.)</p>

Also explore the Library's A-Z Database list to find scholarly resources dedicated to specific subjects (Psychology, Education, Transportation, etc.)

Understanding Peer Review and Original Research

Understanding what peer review is all about.

What is a peer-reviewed article anyways? What is the big deal about peer review? What is it? How does it work? Why do my instructors ask me to find peer-reviewed articles to use in my paper? If an article has been "Peer reviewed" it has earned a kind of authority and credibility from an academic/scientific community. It's a process that research has to go through before it is published. When we're talking about peer review, we're usually talking about scholarly and academic publications. They are usually original research -- research that's never been done before.  

Consult these sources to learn more about it:

Learn about peer review -- and its imperfections -- by watching All About Peer Review , a video from the CSUDH Library. Consult the Research Guide on Peer Review from NJIT Library to learn more. Watch Peer Review in 5 minutes from NCSU Library to learn more about the process of peer review and how to identify peer reviewed research. Read the information sheet What's the Difference? from Purdue University. Look over the peer-review process infographic on the LibGuide from UC San Diego. Want to learn more? Read Peer Review in Scientific Publications: Benefits, Critiques & A Survival Guide for an extensive and in-depth treatment of peer review including its history and problems.

What do peer-reviewed articles look like?

What do peer reviewed articles look like.

Probably the most known peer-reviewed journal is Nature . Take a look at the current issue of Nature -- notice that it has all kinds of articles of interest to a lot of people -- news, editorials, book reviews. Take a look at the "Research Articles" section. There you will see the peer reviewed articles from Nature such as: Physiological measurements in social acceptance of self driving technologies Modeling innovation in the cryptocurrency ecosystem Human preferences toward algorithmic bias in a word association task Notice that the articles have a "received" or "submitted", "accepted by" and "published" dates. These are the marking of peer reviewed articles -- finding these dates can be a quick and easy way of identifying peer-reviewed research. These articles also describe an original scientific study or experiment. They follow the scientific method and have sections with names like "Introduction", "Methodology", "Results", etc. Peer-reviewed articles often, but don't always, have multiple authors whose affiliations are given in the article.

• << Previous: Make a great research question
• Next: Searching Syntax >>
• Last Updated: Mar 26, 2024 1:50 PM
• URL: https://researchguides.njit.edu/engl101-2

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• Research paper

How to Write a Research Paper | A Beginner's Guide

A research paper is a piece of academic writing that provides analysis, interpretation, and argument based on in-depth independent research.

Research papers are similar to academic essays , but they are usually longer and more detailed assignments, designed to assess not only your writing skills but also your skills in scholarly research. Writing a research paper requires you to demonstrate a strong knowledge of your topic, engage with a variety of sources, and make an original contribution to the debate.

This step-by-step guide takes you through the entire writing process, from understanding your assignment to proofreading your final draft.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

Understand the assignment, choose a research paper topic, conduct preliminary research, develop a thesis statement, create a research paper outline, write a first draft of the research paper, write the introduction, write a compelling body of text, write the conclusion, the second draft, the revision process, research paper checklist, free lecture slides.

Completing a research paper successfully means accomplishing the specific tasks set out for you. Before you start, make sure you thoroughly understanding the assignment task sheet:

• Read it carefully, looking for anything confusing you might need to clarify with your professor.
• Identify the assignment goal, deadline, length specifications, formatting, and submission method.
• Make a bulleted list of the key points, then go back and cross completed items off as you’re writing.

Carefully consider your timeframe and word limit: be realistic, and plan enough time to research, write, and edit.

Scribbr Citation Checker New

The AI-powered Citation Checker helps you avoid common mistakes such as:

• Missing commas and periods
• Incorrect usage of “et al.”
• Ampersands (&) in narrative citations
• Missing reference entries

There are many ways to generate an idea for a research paper, from brainstorming with pen and paper to talking it through with a fellow student or professor.

You can try free writing, which involves taking a broad topic and writing continuously for two or three minutes to identify absolutely anything relevant that could be interesting.

You can also gain inspiration from other research. The discussion or recommendations sections of research papers often include ideas for other specific topics that require further examination.

Once you have a broad subject area, narrow it down to choose a topic that interests you, m eets the criteria of your assignment, and i s possible to research. Aim for ideas that are both original and specific:

• A paper following the chronology of World War II would not be original or specific enough.
• A paper on the experience of Danish citizens living close to the German border during World War II would be specific and could be original enough.

Note any discussions that seem important to the topic, and try to find an issue that you can focus your paper around. Use a variety of sources , including journals, books, and reliable websites, to ensure you do not miss anything glaring.

Do not only verify the ideas you have in mind, but look for sources that contradict your point of view.

• Is there anything people seem to overlook in the sources you research?
• Are there any heated debates you can address?
• Do you have a unique take on your topic?
• Have there been some recent developments that build on the extant research?

In this stage, you might find it helpful to formulate some research questions to help guide you. To write research questions, try to finish the following sentence: “I want to know how/what/why…”

A thesis statement is a statement of your central argument — it establishes the purpose and position of your paper. If you started with a research question, the thesis statement should answer it. It should also show what evidence and reasoning you’ll use to support that answer.

The thesis statement should be concise, contentious, and coherent. That means it should briefly summarize your argument in a sentence or two, make a claim that requires further evidence or analysis, and make a coherent point that relates to every part of the paper.

You will probably revise and refine the thesis statement as you do more research, but it can serve as a guide throughout the writing process. Every paragraph should aim to support and develop this central claim.

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Try for free

A research paper outline is essentially a list of the key topics, arguments, and evidence you want to include, divided into sections with headings so that you know roughly what the paper will look like before you start writing.

A structure outline can help make the writing process much more efficient, so it’s worth dedicating some time to create one.

Your first draft won’t be perfect — you can polish later on. Your priorities at this stage are as follows:

• Maintaining forward momentum — write now, perfect later.
• Paying attention to clear organization and logical ordering of paragraphs and sentences, which will help when you come to the second draft.
• Expressing your ideas as clearly as possible, so you know what you were trying to say when you come back to the text.

You do not need to start by writing the introduction. Begin where it feels most natural for you — some prefer to finish the most difficult sections first, while others choose to start with the easiest part. If you created an outline, use it as a map while you work.

Do not delete large sections of text. If you begin to dislike something you have written or find it doesn’t quite fit, move it to a different document, but don’t lose it completely — you never know if it might come in useful later.

Paragraph structure

Paragraphs are the basic building blocks of research papers. Each one should focus on a single claim or idea that helps to establish the overall argument or purpose of the paper.

Example paragraph

George Orwell’s 1946 essay “Politics and the English Language” has had an enduring impact on thought about the relationship between politics and language. This impact is particularly obvious in light of the various critical review articles that have recently referenced the essay. For example, consider Mark Falcoff’s 2009 article in The National Review Online, “The Perversion of Language; or, Orwell Revisited,” in which he analyzes several common words (“activist,” “civil-rights leader,” “diversity,” and more). Falcoff’s close analysis of the ambiguity built into political language intentionally mirrors Orwell’s own point-by-point analysis of the political language of his day. Even 63 years after its publication, Orwell’s essay is emulated by contemporary thinkers.

Citing sources

It’s also important to keep track of citations at this stage to avoid accidental plagiarism . Each time you use a source, make sure to take note of where the information came from.

You can use our free citation generators to automatically create citations and save your reference list as you go.

APA Citation Generator MLA Citation Generator

The research paper introduction should address three questions: What, why, and how? After finishing the introduction, the reader should know what the paper is about, why it is worth reading, and how you’ll build your arguments.

What? Be specific about the topic of the paper, introduce the background, and define key terms or concepts.

Why? This is the most important, but also the most difficult, part of the introduction. Try to provide brief answers to the following questions: What new material or insight are you offering? What important issues does your essay help define or answer?

How? To let the reader know what to expect from the rest of the paper, the introduction should include a “map” of what will be discussed, briefly presenting the key elements of the paper in chronological order.

The major struggle faced by most writers is how to organize the information presented in the paper, which is one reason an outline is so useful. However, remember that the outline is only a guide and, when writing, you can be flexible with the order in which the information and arguments are presented.

One way to stay on track is to use your thesis statement and topic sentences . Check:

• topic sentences against the thesis statement;
• topic sentences against each other, for similarities and logical ordering;
• and each sentence against the topic sentence of that paragraph.

Be aware of paragraphs that seem to cover the same things. If two paragraphs discuss something similar, they must approach that topic in different ways. Aim to create smooth transitions between sentences, paragraphs, and sections.

The research paper conclusion is designed to help your reader out of the paper’s argument, giving them a sense of finality.

Trace the course of the paper, emphasizing how it all comes together to prove your thesis statement. Give the paper a sense of finality by making sure the reader understands how you’ve settled the issues raised in the introduction.

You might also discuss the more general consequences of the argument, outline what the paper offers to future students of the topic, and suggest any questions the paper’s argument raises but cannot or does not try to answer.

You should not :

• Offer new arguments or essential information
• Take up any more space than necessary
• Begin with stock phrases that signal you are ending the paper (e.g. “In conclusion”)

There are four main considerations when it comes to the second draft.

• Check how your vision of the paper lines up with the first draft and, more importantly, that your paper still answers the assignment.
• Identify any assumptions that might require (more substantial) justification, keeping your reader’s perspective foremost in mind. Remove these points if you cannot substantiate them further.
• Be open to rearranging your ideas. Check whether any sections feel out of place and whether your ideas could be better organized.
• If you find that old ideas do not fit as well as you anticipated, you should cut them out or condense them. You might also find that new and well-suited ideas occurred to you during the writing of the first draft — now is the time to make them part of the paper.

The goal during the revision and proofreading process is to ensure you have completed all the necessary tasks and that the paper is as well-articulated as possible. You can speed up the proofreading process by using the AI proofreader .

Global concerns

• Confirm that your paper completes every task specified in your assignment sheet.
• Check for logical organization and flow of paragraphs.
• Check paragraphs against the introduction and thesis statement.

Fine-grained details

Check the content of each paragraph, making sure that:

• each sentence helps support the topic sentence.
• no unnecessary or irrelevant information is present.
• all technical terms your audience might not know are identified.

Next, think about sentence structure , grammatical errors, and formatting . Check that you have correctly used transition words and phrases to show the connections between your ideas. Look for typos, cut unnecessary words, and check for consistency in aspects such as heading formatting and spellings .

Finally, you need to make sure your paper is correctly formatted according to the rules of the citation style you are using. For example, you might need to include an MLA heading  or create an APA title page .

Scribbr’s professional editors can help with the revision process with our award-winning proofreading services.

Discover our paper editing service

Checklist: Research paper

I have followed all instructions in the assignment sheet.

My introduction presents my topic in an engaging way and provides necessary background information.

My introduction presents a clear, focused research problem and/or thesis statement .

My paper is logically organized using paragraphs and (if relevant) section headings .

Each paragraph is clearly focused on one central idea, expressed in a clear topic sentence .

Each paragraph is relevant to my research problem or thesis statement.

I have used appropriate transitions  to clarify the connections between sections, paragraphs, and sentences.

My conclusion provides a concise answer to the research question or emphasizes how the thesis has been supported.

My conclusion shows how my research has contributed to knowledge or understanding of my topic.

My conclusion does not present any new points or information essential to my argument.

I have provided an in-text citation every time I refer to ideas or information from a source.

I have included a reference list at the end of my paper, consistently formatted according to a specific citation style .

I have thoroughly revised my paper and addressed any feedback from my professor or supervisor.

I have followed all formatting guidelines (page numbers, headers, spacing, etc.).

You've written a great paper. Make sure it's perfect with the help of a Scribbr editor!

Open Google Slides Download PowerPoint

Is this article helpful?

Other students also liked.

• Writing a Research Paper Introduction | Step-by-Step Guide
• Writing a Research Paper Conclusion | Step-by-Step Guide
• Research Paper Format | APA, MLA, & Chicago Templates

More interesting articles

• Academic Paragraph Structure | Step-by-Step Guide & Examples
• Checklist: Writing a Great Research Paper
• How to Create a Structured Research Paper Outline | Example
• How to Write a Discussion Section | Tips & Examples
• How to Write Recommendations in Research | Examples & Tips
• How to Write Topic Sentences | 4 Steps, Examples & Purpose
• Research Paper Appendix | Example & Templates
• Research Paper Damage Control | Managing a Broken Argument
• What Is a Theoretical Framework? | Guide to Organizing

What is your plagiarism score?

• Step 1: Sections in a Research Paper
• Step 2: Order for Preparation
• Step 3: Conceptualizing an Attractive Title
• Step 4: Effectively Reviewing Literature
• Step 5: Drafting the Abstract
• Step 6: Drafting Introduction
• Step 7: Drafting Materials and Methods
• Step 8: Drafting Results
• Step 9: Drafting Discussion
• Step 10: Drafting the Conclusion
• Step 11: Citing and Referencing
• Step 12: Preparing Figures
• Step 13: Preparing Tables
• Step 14: Assigning Authorship
• Step 15: Acknowledgements Section
• Step 16: Checking the Author Guidelines
• Step 17: Proofreading and Editing
• Step 18: Pre-submission Peer-Review
• Step 1: How to Structure a Research Paper?
• Step 3: How to Conceptualize an Attractive Research Paper Title?
• Step 4: How to Conduct an Effective Literature Review
• Step 5: How to Write a Good Research Paper Abstract
• Step 6: How to Write a Compelling Introduction for a Research Paper
• Step 7: How to Write the Materials and Methods Section of a Research Paper
• Step 8: How to Write the Results Section of a Research Paper
• Step 9: How to Write the Discussion Section of a Research Paper
• Step 10: How to Write the Conclusion of a Research Paper
• Step 15: How to Write an Acknowledgment Section for a Research Paper

How to Write a Research Paper – A to Z of Academic Writing

Part of a scientist’s job is to publish research. In fact, some would argue that your experiment is only complete once you have published the results. This makes it available to the scientific community for authentication and the advancement of science. In addition, publishing is essential for a researcher’s career as it validates the research and opens doors for funding and employment. In this section, we give you a step-by-step guide to help you write an effective research paper. So, remember to set aside half an hour each day to write. This habit will make your writing manageable and keep you focused.

There are different types of research papers. The most common ones include:

Original research paper, rapid communication or letter, review article, meeting abstract, paper, and proceedings.

This is a full report written by researchers covering the analysis of their experimental study from start to finish. It is the most common type research manuscript that is published in academic journals. Original articles are expected to follow the IMRAD format.

These are usually written to publish results urgently in rapidly changing or highly competitive fields. They will be brief and may not be separated by headings.It consists of original preliminary results that are likely to have a significant impact in the respective field.

This is a comprehensive summary of a certain topic. It is usually requested by a journal editor and written by a leader in the field. It includes current assessment, latest findings, and future directions of the field. It is a massive undertaking in which approximately 100 research articles are cited. Uninvited reviews are published too, but it is best to send a pre-submission enquiry letter to the journal editor first.

This is mostly used in the medical field to report interesting occurrences such as previously unknown or emerging pathologies. It could be a report of a single case or multiple cases and will include a short introduction, methods, results, and discussion.

This is a brief report of research presented at an organized meeting such as a conference. These range from an abstract to a full report of the research. It needs to be focused and clear in explaining your topic and the main points of the study that will be shared with the audience.

• STEP 1: How to Structure a Research Paper?
• STEP 2: Order for Preparation of the Manuscript
• STEP 3: How to Conceptualize an Attractive Research Paper Title?
• STEP 4: How to Conduct an Effective Literature Review
• STEP 5: How to Write a Good Research Paper Abstract
• STEP 6: How to Write a Compelling Introduction for a Research Paper
• STEP 7: How to Write the Materials and Methods Section of a Research Paper
• STEP 8: How to Write the Results Section of a Research Paper
• STEP 9: How to Write the Discussion Section of a Research Paper
• STEP 10: How to Write the Conclusion of a Research Paper
• STEP 11: Effectively Citing and Referencing Your Sources
• STEP 12: Preparing Figures
• STEP 13: Preparing Tables
• STEP 14: Assigning Authorship
• STEP 15: How to Write an Acknowledgment Section for a Research Paper
• STEP 16: Checking the Author Guidelines Before Preparing the Manuscript
• STEP 17: Proofreading and Editing Your Manuscript
• STEP 18: Pre-submission Peer-Review

How to Structure a Research Paper?

Your research paper should tell a story of how you began your research, what you found, and how it advances your research field. It is important to structure your research paper so that editors and readers can easily find information. The widely adopted structure that research papers mostly follow is the IMRaD format . IMRaD stands for Introduction, Methods, Results, and Discussion. Additional requirements from journals include an abstract, keywords, acknowledgements, and references. This format helps scientists to tell their story in an organized manner. Authors often find it easier to write the IMRaD sections in a different order. However, the final paper should be collated in the IMRaD format as follows:

Case studies follow a slightly different format to the traditional IMRAD format. They include the following extra sections:

• History and physical examination: Details of the patient’s history. It provides the story of when a patient first sought medical care.
• Diagnostic focus and assessment : Describe the steps taken that lead to a diagnosis and any test results.
• Therapeutic focus and assessment: Explain therapies tried and any other recommendations from consultants. Assess the efficacy of the treatments given.
• Follow-up and outcome: Provide results and state the patient adhered to treatment. Include any side effects.
• Patient perspective: Describe the patient’s experience.
• Patient consent: State that informed consent was obtained from the patient.

Order for Preparation of the Manuscript

As mentioned above, most research publications follow the IMRAD format. However, it is often easier to write each section in a different order than that of the final paper.

Authors recommend you organize the data first and then write the sections as follows:

• Figures and tables: Decide how your data should be presented. You can use graphics, tables or describe it in the text.
• Methods: It is important that anyone can use your methods to reproduce your experiments.
• Results: Here you write only what the results of your experiments were. You do not discuss them here.
• Discussion: This section requires analysis, thought, and a thorough understanding of the literature. You need to discuss your results without repeating the results section.
• Conclusion: This section can either be under a sub-heading or the last paragraph of the discussion. It should inform the reader how your results advance the field.
• Introduction: Now that you have thought about your results in the context of the literature, you can write your introduction.
• Abstract: This is an overview of your paper. Give a concise background of the problem and how you tried to solve it. Next state your main findings.
• Title: As discussed above, this needs to be concise as well as informative. Ensure that it makes sense.
• Keywords: These are used for indexing. Keywords need to be specific. Often you are not allowed to use words that appear in the journal name. Use abbreviations with care and only well-established ones.
• Acknowledgements: This section is to thank anyone involved in the research that does not qualify as an author.
• References: Check the “Guide for authors” for the formatting style. Be accurate and do not include unnecessary references.

How to Conceptualize an Attractive Research Paper Title?

Your research title is the first impression of your paper. A good research paper title is a brief description of the topic, method, sample, and results of your study. A useful formula you could use is:

There are different ways to write a research paper title :

Declarative

State the main conclusions. Example: Mixed strains of probiotics improve antibiotic associated diarrhea.

Descriptive

Describe the subject. Example: Effects of mixed strains of probiotics on antibiotic associated diarrhea.

Interrogative

Use a question for the subject. Example: Do mixed strains of probiotics improve antibiotic associated diarrhea?

We recommend the following five top tips to conceptualize an attractive research title:

• Be descriptive
• Use a low word count (5-15 words)
• Check journal guidelines
• Avoid jargon and symbols

How to Conduct an Effective Literature Review

The process of conducting a literature review can be overwhelming. However, if you start with a clear research question, you can stay focused.

• Literature search: Search for articles related to your research question. Keep notes of the search terms and keywords you use. A list of databases to search and notes of the ones you have searched will prevent duplicate searches.

- What is their research question?

- Are there potential conflicts of interest such as funders who may want a particular result?

- Are their methods sufficient to test the objectives?

- Can you identify any flaws in the research?

- Do their results make sense, or could there be other reasons for their conclusion?

- Are the authors respected in the field?

- Has the research been cited?

- Introduction: Here you introduce the topic. The introduction describes the problem and identifies gaps in knowledge. It also rationalizes your research.

- Discussion: Here you support and compare your results. Use the literature to put your research in context with the current state of knowledge. Furthermore, show how your research has advanced the field.

How to Write a Good Research Paper Abstract

The importance of research paper abstracts  cannot be emphasized enough.

• They are used by online databases to index large research works. Therefore, critical keywords must be used.
• Editors and reviewers read an abstract to decide whether an article is worth considering for publication.
• Readers use an abstract to decide whether the research is relevant to them.

A good research paper abstract is a concise and appealing synopsis of your research. There are two ways to write an abstract:  structured and unstructured research abstracts . The author guidelines of the journal you are submitting your research to will tell you the format they require.

• The structured abstract has distinct sections with headings. This style enables a reader to easily find the relevant information under clear headings (objective, methods, results, and conclusion). Think of each section as a question and provide a concise but detailed answer under each heading.
• The unstructured abstract is a narrative paragraph of your research. It is similar to the structured abstract but does not contain headings. It gives the context, findings, conclusion, and implications of your paper.

How to Write a Compelling Introduction for a Research Paper

The Introduction section of your research paper introduces your research  in the context of the knowledge in the field. First introduce the topic including the problem you are addressing, the importance of solving this problem, and known research and gaps in the knowledge. Then narrow it down to your research questions and hypothesis.

Tips to write an effective introduction for your research paper :

• Give broad background information about the problem.
• Write it in a logical manner so that the reader can follow your thought process.
• Focus on the problem you intend to solve with your research
• Note any solutions in the literature thus far.
• Propose your solution to the problem with reasons.

Done with drafting your research paper?

With enago’s english editing & proofreading service your success is just a step away.

How to Write the Materials and Methods Section of a Research Paper

When writing the Materials and Methods section of a research paper, you need to give enough detail in your methods  so that others can reproduce your experiments. However, there is no need to detail established experiments. Readers can find these details in the previously published references you refer to in the methods. Follow these tips to write the Materials and Methods section of your research paper: :

• Write in the past tense because you are reporting on procedures you carried out.
• Avoid unnecessary details that disrupts the flow.
• Materials and equipments should be mentioned throughout the procedure, rather than listed at the beginning of a section.
• Detail any ethics or consent requirements if your study included humans or animal subjects.
• Use standard nomenclature and numbers.
• Ensure you have the correct control experiments.
• Methods should be listed logically.
• Detail statistical methods used to analyze your data.

Here is a checklist of things that should be in your Materials and Methods:

• References of previously published methods.
• Study settings : If the research involves studying a population, give location and context of the site.
• Cell lines : Give their source and detail any contamination tests performed.
• Antibodies : Give details such as catalogue numbers, citations, dilutions used, and batch numbers.
• Animal models : Species, age, and sex of animals as well as ethical compliance information.
• Human subjects : Ethics committee requirements and a statement confirming you received informed consent. If relevant, clinical trial registration numbers and selection criteria.
• Data accession codes for data you deposited in a repository.
• Software : Where you obtained the programs and their version numbers.
• Statistics : Criteria for including or excluding samples or subjects, randomisation methods, details of investigator blinding to avoid bias, appropriateness of statistical tests used for your study.
• Timeframes if relevant.

How to Write the Results Section of a Research Paper

Some journals combine the results and discussion section, whereas others have separate headings for each section. If the two sections are combined, you state the results of your research   and discuss them immediately afterwards, before presenting your next set of results.  The challenge is to present your data in a way that is logical and accurate. Set out your results in the same order as you set out your methods.

When writing the Results section of your research paper remember to include:

• Control group data.
• Relevant statistical values such as p-values.
• Visual illustrations of your results such as figures and tables.

Things that do not belong in the results section:

• Speculation or commentary about the results.
• References – you are reporting your own data.
• Do not repeat data in text if it has been presented in a table or graph.

Keep the discussion section separate . Keep explanations, interpretations, limitations, and comparisons to the literature for the discussion.

How to Write the Discussion Section of a Research Paper

The discussion section of your research paper answers several questions such as: did you achieve your objectives? How do your results compare to other studies? Were there any limitations to your research? Start discussing your data specifically and then broaden out to how it furthers your field of interest.

Questions to get you started:

• How do your results answer your objectives?
• Why do you think your results are different to published data?
• Do you think further research would help clarify any issues with your data?

The aim is to tell the reader what your results mean. Structure the discussion section of your research paper  in a logical manner. Start with an introductory paragraph where you set out the context and main aims of the study. Do this without repeating the introduction. Some authors prefer starting with the major findings first to keep the readers interested.

The next paragraph should discuss what you found, how it compares to other studies, any limitations, your opinion, and what they mean for the field.

The concluding paragraph should talk about the major outcomes of the study. Be careful not to write your conclusion here. Merely highlight the main themes emerging from your data.

Tips to write an effective discussion:

• It is not a literature review. Keep your comments relevant to your results.
• Interpret your results.
• Be concise and remove unnecessary words.
• Do not include results not presented in the result section.
• Ensure your conclusions are supported by your data.

How to Write the Conclusion of a Research Paper

While writing the conclusion for your research paper, give a summary of your research with emphasis on your findings. Again, structuring the conclusion section of your research paper  will make it easier to draft this section. Here are some tips when writing the conclusion of your paper:

• State what you set out to achieve.
• Tell the reader what your major findings were.
• How has your study contributed to the field?
• Mention any limitations.
• End with recommendations for future research.

Having difficulties with understanding concepts on academic writing?

Enago learn can guide you through the manuscript preparation process and help you achieve success.

Effectively Citing and Referencing Your Sources

You need to acknowledge the original work  that you talk about in your write-up. There are two reasons for this. First, cite someone’s idea  to avoid plagiarism. Plagiarism is when you use words or ideas of others without acknowledging them and this is a serious offence. Second, readers will be able to source the literature you cited easily.

This is done by citing works  in your text and providing the full reference for this citation in a reference list at the end of your document.

Tips for effective refencing/citations:

• Keep a detailed list of your references including author(s), publication, year of publication, title, and page numbers.
• Insert a citation (either a number or author name) in-text as you write.
• List the full reference in a reference list according to the style required by the publication.
• Pay attention to details as mistakes will misdirect readers.

Try referencing software tools “cite while you write”. Examples of such referencing software programs include: Mendeley , Endnote , Refworks  and Zotero .

Preparing Figures

Some quick tips about figures:

• Legends of graphs and tables must be self-explanatory.
• Use easily distinguishable symbols.
• Place long tables of data in the supplementary material.
• Include a scale bar in photographs.

Preparing Tables

Important pointers for tables:

• Check the author guidelines for table formatting requirements.
• Tables do not have vertical lines in publications.
• Legends must be self-explanatory.

Assigning Authorship

To qualify as an author  on a paper, an individual must:

• Make substantial contributions to all stages of the research.
• Draft or revise the manuscript.
• Approve the final version of the article.
• Be accountable for the accuracy and integrity of the research.

Unethical and unprofessional authorships  have emerged over the years. These include:

• Gift authorship : An individual is listed as a co-author in lieu of funding or supervision.
• Ghost authorship : An author is paid to write an article but does not contribute to the article in any other way.
• Guest authorship : An individual who is given authorship because they are well known and respected in the field, or they are senior members of staff.

These authors pose a threat to research. Readers may override their concerns with an article if it includes a well-respected co-author. This is especially problematic when decisions about medical interventions are concerned.

How to Write an Acknowledgment Section for a Research Paper

Those who do not qualify as authors but have contributed to the research should be given credit in the acknowledgements section of your research paper . These include funders, supervisors, administrative supporters, writing, editing, and proofreading assistance .

The contributions made by these individuals should be stated and sometimes their written permission to be acknowledged is required by editors.

Has your target journal's author guidelines left you confused?

With enago consult you can talk to our experts through live 1-to-1 video calls.

Points to Note from the Author Instructions Before Preparing the Manuscript

Check the author guidelines for your chosen publication before submission. Publishers mostly have a “House Style” that ensures all their manuscripts are consistent with regards to language, formatting, and style. For example, these guidelines will tell you whether to use UK or US English, which abbreviations are allowed, and how to format figures and tables. They are also especially important for the references section as each journal has their own style.

Proofreading/Editing your Manuscript

Ensure that your manuscript is structured correctly, clearly written, contains the correct technical language, and supports your claims with proper evidence. To ensure the structure is correct, it is essential to edit your paper .

Once you are happy with the manuscript, proofread for small errors. These could be spelling, consistency, spacing, and so forth. Importantly, check that figures and tables include all the necessary data and statistical values. Seek assistance from colleagues or professional editing companies to edit and proofread your manuscript too.

Pre-submission Peer-Review of Your Manuscript

A pre-submission peer-review  could improve the quality of articles submitted to journals in general. The benefits include:

• A fresh eye to spot gaps or errors.
• Receiving constructive feedback on your work and writing.
• Improves the clarity of your paper.

You could ask experienced colleagues, supervisors or even professional editing services to review your article.

I am looking for Editing/ Proofreading services for my manuscript Tentative date of next journal submission:

What should universities' stance be on AI tools in research and academic writing?

Research articles

Predicting the risks of kidney failure and death in adults with moderate to severe chronic kidney disease, impact of large scale, multicomponent intervention to reduce proton pump inhibitor overuse, esketamine after childbirth for mothers with prenatal depression, glucagon-like peptide 1 receptor agonist use and risk of thyroid cancer, use of progestogens and the risk of intracranial meningioma, delirium and incident dementia in hospital patients, derivation and external validation of a simple risk score for predicting severe acute kidney injury after intravenous cisplatin, quality and safety of artificial intelligence generated health information, large language models and the generation of health disinformation, 25 year trends in cancer incidence and mortality among adults in the uk, cervical pessary versus vaginal progesterone in women with a singleton pregnancy, comparison of prior authorization across insurers, diagnostic accuracy of magnetically guided capsule endoscopy with a detachable string for detecting oesophagogastric varices in adults with cirrhosis, ultra-processed food exposure and adverse health outcomes, added benefit and revenues of oncology drugs approved by the ema, exposure to air pollution and hospital admission for cardiovascular diseases, short term exposure to low level ambient fine particulate matter and natural cause, cardiovascular, and respiratory morbidity, optimal timing of influenza vaccination in young children, effect of exercise for depression, association of non-alcoholic fatty liver disease with cardiovascular disease and all cause death in patients with type 2 diabetes, duration of cpr and outcomes for adults with in-hospital cardiac arrest, clinical effectiveness of an online physical and mental health rehabilitation programme for post-covid-19 condition, atypia detected during breast screening and subsequent development of cancer, publishers’ and journals’ instructions to authors on use of generative ai in academic and scientific publishing, effectiveness of glp-1 receptor agonists on glycaemic control, body weight, and lipid profile for type 2 diabetes, neurological development in children born moderately or late preterm, invasive breast cancer and breast cancer death after non-screen detected ductal carcinoma in situ, all cause and cause specific mortality in obsessive-compulsive disorder, acute rehabilitation following traumatic anterior shoulder dislocation, perinatal depression and risk of mortality, undisclosed financial conflicts of interest in dsm-5-tr, effect of risk mitigation guidance opioid and stimulant dispensations on mortality and acute care visits, update to living systematic review on sars-cov-2 positivity in offspring and timing of mother-to-child transmission, perinatal depression and its health impact, christmas 2023: common healthcare related instruments subjected to magnetic attraction study, using autoregressive integrated moving average models for time series analysis of observational data, demand for morning after pill following new year holiday, christmas 2023: christmas recipes from the great british bake off, effect of a doctor working during the festive period on population health: experiment using doctor who episodes, christmas 2023: analysis of barbie medical and science career dolls, christmas 2023: effect of chair placement on physicians’ behavior and patients’ satisfaction, management of chronic pain secondary to temporomandibular disorders, christmas 2023: projecting complete redaction of clinical trial protocols, christmas 2023: a drug target for erectile dysfunction to help improve fertility, sexual activity, and wellbeing, christmas 2023: efficacy of cola ingestion for oesophageal food bolus impaction, conservative management versus laparoscopic cholecystectomy in adults with gallstone disease, social media use and health risk behaviours in young people, untreated cervical intraepithelial neoplasia grade 2 and cervical cancer, air pollution deaths attributable to fossil fuels, implementation of a high sensitivity cardiac troponin i assay and risk of myocardial infarction or death at five years, covid-19 vaccine effectiveness against post-covid-19 condition, association between patient-surgeon gender concordance and mortality after surgery, intravascular imaging guided versus coronary angiography guided percutaneous coronary intervention, treatment of lower urinary tract symptoms in men in primary care using a conservative intervention, autism intervention meta-analysis of early childhood studies, effectiveness of the live zoster vaccine during the 10 years following vaccination, effects of a multimodal intervention in primary care to reduce second line antibiotic prescriptions for urinary tract infections in women, pyrotinib versus placebo in combination with trastuzumab and docetaxel in patients with her2 positive metastatic breast cancer, association of dcis size and margin status with risk of developing breast cancer post-treatment, racial differences in low value care among older patients in the us, pharmaceutical industry payments and delivery of low value cancer drugs, rosuvastatin versus atorvastatin in adults with coronary artery disease, clinical effectiveness of septoplasty versus medical management for nasal airways obstruction, ultrasound guided lavage with corticosteroid injection versus sham lavage with and without corticosteroid injection for calcific tendinopathy of shoulder, early versus delayed antihypertensive treatment in patients with acute ischaemic stroke, mortality risks associated with floods in 761 communities worldwide, interactive effects of ambient fine particulate matter and ozone on daily mortality in 372 cities, association between changes in carbohydrate intake and long term weight changes, future-case control crossover analysis for adjusting bias in case crossover studies, association between recently raised anticholinergic burden and risk of acute cardiovascular events, suboptimal gestational weight gain and neonatal outcomes in low and middle income countries: individual participant data meta-analysis, efficacy and safety of an inactivated virus-particle vaccine for sars-cov-2, effect of invitation letter in language of origin on screening attendance: randomised controlled trial in breastscreen norway, visits by nurse practitioners and physician assistants in the usa, non-erosive gastro-oesophageal reflux disease and oesophageal adenocarcinoma, venous thromboembolism with use of hormonal contraception and nsaids, food additive emulsifiers and risk of cardiovascular disease, balancing risks and benefits of cannabis use, promoting activity, independence, and stability in early dementia and mild cognitive impairment, effect of home cook interventions for salt reduction in china, cancer mortality after low dose exposure to ionising radiation, effect of a smartphone intervention among university students with unhealthy alcohol use, long term risk of death and readmission after hospital admission with covid-19 among older adults, mortality rates among patients successfully treated for hepatitis c, association between antenatal corticosteroids and risk of serious infection in children, the proportions of term or late preterm births after exposure to early antenatal corticosteroids, and outcomes, safety of ba.4-5 or ba.1 bivalent mrna booster vaccines, comparative effectiveness of booster vaccines among adults aged ≥50 years, third dose vaccine schedules against severe covid-19 during omicron predominance in nordic countries, private equity ownership and impacts on health outcomes, costs, and quality, healthcare disruption due to covid-19 and avoidable hospital admission, educational inequalities in mortality and their mediators among generations across four decades, prevalence and predictors of data and code sharing in the medical and health sciences, medicare eligibility and in-hospital treatment patterns and health outcomes for patients with trauma, therapeutic value of first versus supplemental indications of drugs in us and europe, hospital admissions linked to sars-cov-2 infection in children and adolescents, vitamin d supplementation and major cardiovascular events, menopausal hormone therapy and dementia, associations between modest reductions in kidney function and adverse outcomes in young adults, follow us on, content links.

• Collections
• Health in South Asia
• Women’s, children’s & adolescents’ health
• News and views
• BMJ Opinion
• Rapid responses
• Editorial staff
• BMJ in the USA
• BMJ in South Asia
• Submit your paper
• BMA members
• Subscribers
• Advertisers and sponsors

Explore BMJ

• Our company
• BMJ Careers
• BMJ Learning
• BMJ Masterclasses
• BMJ Journals
• BMJ Student
• Academic edition of The BMJ
• BMJ Best Practice
• The BMJ Awards
• Email alerts
• Activate subscription

Information

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• PMC10306213

Content and form of original research articles in general major medical journals

Nicole heßler.

1 Institut für Medizinische Biometrie und Statistik (IMBS), Universität zu Lübeck, Universitätsklinikum-Schleswig-Holstein, Campus Lübeck, Lübeck, Germany

Andreas Ziegler

2 Cardio-CARE, Medizincampus Davos, Davos, Switzerland

3 School of Mathematics, Statistics and Computer Science, University of KwaZulu Natal, Pietermaritzburg, South Africa

4 Department of Cardiology, University Heart and Vascular Center Hamburg, University Medical Center Hamburg-Eppendorf, Hamburg, Germany

5 Centre for Population Health Innovation (POINT), University Heart and Vascular Center Hamburg, University Medical Center Hamburg-Eppendorf, Hamburg, Germany

6 Swiss Institute of Bioinformatics, Lausanne, Switzerland

Associated Data

All relevant data are within the paper and its Supporting Information files.

The title of an article is the main entrance for reading the full article. The aim of our work therefore is to examine differences of title content and form between original research articles and its changes over time. Using PubMed we examined title properties of 500 randomly chosen original research articles published in the general major medical journals BMJ, JAMA, Lancet, NEJM and PLOS Medicine between 2011 and 2020. Articles were manually evaluated with two independent raters. To analyze differences between journals and changes over time, we performed random effect meta-analyses and logistic regression models. Mentioning of results, providing any quantitative or semi-quantitative information, using a declarative title, a dash or a question mark were rarely used in the title in all considered journals. The use of a subtitle, methods-related items, such as mentioning of methods, clinical context or treatment increased over time (all p < 0.05), while the use of phrasal tiles decreased over time (p = 0.044). Not a single NEJM title contained a study name, while the Lancet had the highest usage of it (45%). The use of study names increased over time (per year odds ratio: 1.13 (95% CI: [1.03‒1.24]), p = 0.008). Investigating title content and form was time-consuming because some criteria could only be adequately evaluated by hand. Title content changed over time and differed substantially between the five major medical journals. Authors are advised to carefully study titles of journal articles in their target journal prior to manuscript submission.

Introduction

Researchers have the duty to make the results of their research on human subjects publicly available according to the declaration of Helsinki [ 1 ], and many recommendations for the reporting of studies have been developed. An overview on these reporting guidelines is provided by the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) network, which aims to tackle the problems of poor reporting [ 2 ]. One consequence of systematic reporting is that many scientific articles are organized in the same way [ 3 , 4 ], and they generally follow the IMRAD structure, which stands for Introduction, Methods, Results, And Discussion. The IMRAD structure is also standard for the writing of abstracts. It is therefore of interest to researchers how they can individualize their reports to increase the citation counts, which is one important measure for career advancement [ 5 ].

Approximately 30 factors affecting citation frequency have already been identified [ 6 – 9 ]. While journal- and author related factors are generally not modifiable, some article-specific factors are subject to active modification by the authors. Especially the title has been proposed as a modifiable component of a research article [ 9 – 11 ]. Researchers should use titles that accurately reflect the content of their work and allow others easily to find and re-use their research [ 12 ]. Most research has focused on the form of article titles because these analyses could be performed automatically and are not very time-consuming [ 9 , 13 , 14 ].

While the article content has been studied well both in features, such as tense, voice and personal pronouns, and in the IMRAD sections between different research disciplines, title content has received less attention, and the main focus was title length [ 15 , 16 ]. One reason could be the lack of automated internet searches until approximately 25 years ago. For example, PubMed was first released in 1996, Web of Science is online since 1997 and Google Scholar started not earlier than in 2004. With the advent of automated internet-based searches the importance of the title has changed, and it is now the “billboard” of a research article [ 17 ]. Another reason could be that these evaluations have to be made manually, and they are thus time-consuming [ 18 ]. An additional time-consuming factor could be that guidelines such as the Standards for Reporting of Diagnostic Accuracy (STARD) statement [ 19 ] strongly recommend that at least two observers should do an independent evaluation where applicable.

Most articles investigating the form of the title compared whether the title was a full sentence [ 20 ], descriptive, indicative, or a question [ 18 , 21 ], or whether the title included non-alphanumeric characters, such as a colon or dash [ 22 ]. Very few publications looked at other title components of a research article. Specifically, Kerans, Marshall [ 23 ] compared the frequency of Methods mentioning or Results mentioning for the general major medical journals, specifically the New England Journal of Medicine (NEJM), the BMJ, the Journal of the American Medical Association (JAMA), and the Lancet by analyzing the first approximately 60 articles published either in 2015 or 2017 in each of the journals. Both articles investigated only a few months from a single publication year per journal. The development of title content over time was thus not considered.

The aim of our work therefore was to examine properties of title content for original research articles published in one of the five major clinical journals (BMJ, JAMA, Lancet, NEJM, and PLOS Medicine (PLOS)) over the 10-year period from 2011 until 2020. Specifically, we aimed at identifying differences between the five journals and changes over time regarding title content and title form. We also compared our findings to those of Kerans et al. [ 15 , 23 ].

Materials and methods

Search in medline and web of science.

The search strategy has been described in detail elsewhere [ 9 ]. In brief, we first extracted all original research articles finally published between 2011 and 2020 in the five major clinical journals BMJ, JAMA, Lancet, NEJM, and PLOS. The restriction to the publication year 2011 allows for proper comparisons between journals because PLOS was reshaped in 2009 [ 24 ].

The variables PubMed identifier (PMID), journal name, article title, author names, publication year, citation, PubMed Central identifier (PMCID) and digital object identifier (DOI) were extracted from the Medline search. From the Web of Science, we reduced available information to journal name, article title, PMID, abstract for the identification of original research articles, DOI and publication date. Both PMID and DOI were used to merge articles identified in Medline (n = 8396) and the Web of Science (n = 10267). Articles being listed with an abstract remained in the data set, while articles only listed in the Web of Science were excluded. Articles being only downloaded in the Medline files were checked whether they were indeed original research articles. If not, they were excluded as well. After data cleaning, a set of 8096 articles was available.

Evaluation of title content and form

To investigate title content and form, we randomly selected 500 original articles from the years 2011 to 2020. The random selection was done with stratification by journal and year so that ten original articles per year (100 articles per journal) were randomly chosen. To avoid a priori information on the specific journal article, only the title and the PMID were presented in the database. In addition, the order of the 500 articles was randomized prior to evaluation. All article titles were evaluated manually by two raters/authors. Both raters performed a training and independently evaluated 25 randomly selected journal articles—five per journal—prior to the evaluation of the 500 articles. These training articles were excluded from the main evaluation. Conflicts in ratings were solved by agreement.

Items for title content and form are displayed in Table 1 and were inspired by other works [ 15 , 25 , 26 ]. One reviewer asked for the discoverability in each of the title items, therefore, we provided two examples of article titles with the result of our evaluation in Table 1 .

The first block of Table 1 reports results on title content. Title content was divided into the topics Methods and Results. The former is concerned with the mentioning of methods in the title, such as the study design or a novel technique used in the paper [ 15 ]. Other elements from the methods concern the mentioning of a patient population, the geography, the clinical context, an intervention, and the use of study names in the title. The latter examines results mentioned in the manuscript. The first question was whether results were stated in the title at all. More detailed were the questions whether quantitative information or semiquantitative or ordinal information was provided [ 26 ]. It was also noted whether the title reported on a relation between two or more variables [ 26 ].

The second block of Table 1 is related to the form of a title divided into the topics Methods, and Conclusion/Discussion. The use of abbreviations, dashes and subtitles was investigated for the Methods. The three single items for Conclusion/Discussion were whether the title was declarative, phrasal, or formulated as a question.

Recently, we performed an analysis after an automatic search for country and city mentioning in the title by the use of the R package maps [ 9 ], and we did not expect substantial differences to our hand search.

Sample size considerations

The main aim of our work was to investigate trends over time by a regression model. In general, regression models have a sufficient sample for a single independent variable, such as time, if n ≈ 50 [ 27 , 28 ]. Specifically, for a weak effect size of R 2 = 0.14 [ 29 ], the required sample size is 51. In case of a weak effect size of Cohen’s f [ 29 ] with f 2 = R 2 / (1 –R 2 ) = 0.14, the required sample size is 403 to achieve a power of 80%. A sample size of 500 as used in our work yields a power of 87.75% at a significance level of 5%.

Descriptive statistics for the specified title properties, i.e., absolute and relative frequencies were reported for each journal over time, refraining of descriptive p-values for investigating journal differences. Fisher’s exact tests were performed at a significance level of 5% to compare the findings of this study with those of Kerans et al. [ 15 , 23 ] regarding methods mentioning, patient population, geography, clinical context, and treatment. Corresponding 95% confidence intervals (CI) were provided. Furthermore, overall tests were performed to compare frequencies of these items between all journals. Bias-corrected Cramérs V effect measures were estimated with corresponding parametric bootstrapped CIs. The DerSimonian and Laird [ 30 ] (DSL) approach was used to perform random effect (RE) meta-analyses, which allows for variability in the variables of interest properties between journals and over time. The logit transformation was used for estimating the pooled proportions [ 31 ], and standard errors were not back-transformed.

The effect of time regarding the specific title properties was investigated by logistic regression models, if appropriate. Post hoc comparisons for the identification of homogeneous subgroups were performed using Tukey’s HSD. Associations between title properties and the journals were analyzed using likelihood ratio tests. Effect estimates, i.e., odds ratios and corresponding 95% CI were reported for all analyses, and the journal BMJ was used as reference category. An odds ratio of x.x being greater than 1 indicates an x.x fold increased chance containing the specific item for an one-year difference adjusted for the variable journal.

Data and R code for all analyses are provided in S1 and S2 Files , respectively.

A total of 500 randomly selected original research articles from 5 medical journals were analyzed regarding the selected title items (see Table 1 ) . In Table 2 , the descriptive statistics, i.e., absolute and relative frequencies for all title properties over the years are shown, respectively for each journal. Results of the meta-analyses are provided in detail in S3 File , sections 4 and 5 .

Absolute and relative frequencies (parenthesis) are shown.

JAMA: The Journal of the American Medical Association, NEJM: The New England Journal of Medicine, PLOS: PLOS Medicine.

Items–Content

In terms of the title content topic methods, the NEJM deviated from the other journals regarding the methods mentioning. While methods were mentioned in at least 93% of the article titles in BMJ, Lancet and PLOS, about the half (47%) was in JAMA and 11% in NEJM article titles. Similar results were reported by Kerans et al. [ 15 , 23 ] for BMJ, JAMA and Lancet, but proportions differed between Lancet titles ( Table 3 ). The mentioning of methods increased over time (OR: 1.12 (95% CI: [1.01‒1.24]), p = 0.025, Fig 1 and S3 File , section 6.1.1 ), i.e., methods were mentioned more frequently in the article titles more recently.

Displayed are odds ratios (square) per increase by one year, corresponding 95% confidence intervals (whisker) and p-values (numbers).

Corresponding 95% confidence intervals (CI) are shown in brackets. Results of PLOS Medicine are missing because Kerans et al. did not examine article titles of this journal.

JAMA: The Journal of the American Medical Association, NEJM: The New England Journal of Medicine. 1 p-values from Fisher exact test; frequencies were compared within a journal for the respective variable. 2 Cramérs V (bias-corrected); CIs calculated by bootstrapping (normal approximation, 100 replications), 3 p-values from Fisher-Freeman-Halton exact test; frequencies were compared between all journals within the respective study

Lowest and highest numbers for the mentioning of the patient population were in the BMJ (62%) and the NEJM (78%), respectively. For the mentioning of the patient population, neither an increase over time (OR: 1.06 (95% CI: [0.99‒1.13]), p = 0.100, Fig 1 ) nor substantial differences between the journals ( S3 File , section 6.1.2 ) could be observed.

About half of the PLOS titles (52%) contained any geographic information, but only 31% of the BMJ titles (see Table 2 ). Frequencies were only 16% and 17% for JAMA and Lancet, respectively, and 9% for NEJM titles. These findings are in line with Kerans et al. [ 15 , 23 ], except for the BMJ, where Kerans et al. observed that 15.8% of the articles mentioned geographic information ( Table 3 ). Mentioning of geographic information varied over time both within each journal ( S3 File , section 4.1.3.1) and over the journals ( S3 File , section 4.1.3.2 ). This is consistent with the results from the logistic regression analysis (OR: 1.07 (95% CI: [0.99‒1.16]), p = 0.072, Fig 1 and S3 File , section 6.1.3 ).

The clinical context was mentioned in 73% of BMJ titles, while it was mentioned at least 80% in the other four journals. This is in line with Kerans et al. [ 15 , 23 ] ( Table 3 ). Additionally, we observed an increase of clinical context mentioning over time (OR: 1.10 (95% CI: [1.01‒1.19]), p = 0.025, Fig 1 and S3 File , section 6.1.4 ).

Only 27% in PLOS and 30% in BMJ provided some treatment information in the title, while for the other three journals at least 50% of the article titles mentioned a treatment. Our results did not show any differences from those of Kerans et al. [ 15 , 23 ] ( Table 3 ). Over time the naming of treatments in the title increased (OR: 1.08 (95% CI: [1.02‒1.16]), p = 0.015, Fig 1 and S3 File , section 6.1.5 ).

There was no NEJM title containing a study name while Lancet had the highest usage of it (45%). The analysis over time showed a trend over time (OR: 1.13 (95% CI: [1.03‒1.24]), p = 0.008) and substantial differences between the journals ( S3 File , section 6.1.6 ).

Regarding the title topic results, only 6 out of the total of 500 articles mentioned results in their titles. This is in line with the findings of Kerans et al. [ 15 , 23 ], who reported that 1.9% of NEJM titles mentioned results. No article provided any quantitative information in its title, and only 4 of 500 articles provided semi-quantitative information in their title. Because of very low numbers, no further analyses were performed for these criteria.

A relation between variables was used least frequently in the NEJM (23%), followed by the Lancet (35%). The other three journals mentioned a relation in more than half of the articles ( Table 2 ). These differences between journals were confirmed in regression analysis ( S3 File , section 6.2.4 ). However, an increase over time could not be observed (p = 0.858, Fig 1 ).

Items–Form

In terms of the title form topic methods, abbreviations were less used in NEJM titles and most used in Lancet titles, 24% and 55 respectively (see Table 2 ). An increase use over time was observed (OR: 1.13 (95% CI: [1.05‒1.20]), p < 0.001, Fig 1 ) as well as differences between journals ( S3 File , section 7.1.1) .

Dashes were rarely used. Only three articles in BMJ and two articles in NEJM used a dash ( Table 2 ). Further analyses were not performed because of these low frequencies.

A subtitle was used in at least 98% of the articles in BMJ, Lancet, and PLOS, while only 41% of JAMA titles and only 2% of NEJM titles used subtitles. These clear differences between the journals were confirmed with the regression analysis ( S3 File , section 7.1.2 ). Moreover, the usage of subtitles increased over time (OR: 1.22 (95% CI: [1.07‒1.38]), p < 0.003 , Fig 1 ).

Finally, regarding the title form topic discussion, not a single article had a declarative title in our analyses ( Table 2 ). Phrasal titles were present in 3% of JAMA, 7% of NEJM, 11% of BMJ, 12% of Lancet, and 15% of PLOS titles. Significant differences between journals could not be observed ( S3 File , section 7.2.2 ). A decrease of phrasal titles over time was observed in the regression analysis (OR: 0.90 (95% CI: [0.81‒1.00]), p < 0.044, Fig 1 and S3 File , section 7.2.2 ).

Only three of 500 article titles were written as a question ( Table 2 ). Kerans et al. [ 15 , 23 ] observed similar low frequencies; and they reported 3.9% for the BMJ and 1.3% for Lancet articles with a question symbol, and none for both JAMA and NEJM ( Table 3 ).

Geographic information–Manual versus automated search with the maps package

The comparison of our hand search on the mentioning of geographic information revealed substantial differences to the automated search with the R package maps [ 9 ].

In detail, respectively, 31% vs. 13% for BMJ, 16% vs. 3% for JAMA, 17% vs. 9% for the Lancet, 9% vs. 3% for the NEJM and 52% vs. 29% for PLOS articles contained any geographical information in their titles for the hand and automatic search. The automated search thus led to fewer titles with any geographic information.

Title content properties varied substantially between original research articles published in the general major medical journals. Furthermore, title content and form changed over time. Differences between journals were specifically observed in the use of subtitles. While almost all articles from the BMJ and PLOS had subtitles, only two of the NEJM articles had a subtitle. Previously, we and others showed that the colon was most used in titles to split a title into multiple parts rather than any other separator [ 9 , 15 , 23 ]. Here, we furthermore showed that the proportion of paper with subtitles increased over time.

Substantial differences between journals were also observed for the mentioning of methods, the patient population, the geography, the interventional treatment, and the use of an abbreviation in the title. In addition, there were substantial differences in the use of a study name in the title. For example, while no article published in the NEJM used a study name, almost half (45%) of the studies in the Lancet used one. Some content criteria were mainly not or rarely used in all considered journals, such as a dash, mentioning of results, using a declarative title, or a question mark. This was in contrast to Paiva, Lima [ 32 ] who showed for PLOS and BMC journals that approximately 40% of the articles mentioned the results, and such articles were more frequently cited than work mentioning methods. In our study, only 6 articles out of 500 mentioned results in the title, while 344 out of the 500 articles mentioned of methods. Our findings are in line with general guidelines that declamatory titles, i.e., titles that give study results should be avoided [ 33 ]; see, e.g., instructions to authors for the Lancet. Authors should thus avoid providing quantitative or semi-quantitative information in the title. In fact, since the title is a one-line summary, the conclusions could be spread out into the world without reading at least the abstract or the full text of the article. Aleixandre-Benavent and colleagues go a step further and provide recommendations what a title should contain, and how it should not be constructed [ 16 ].

Our work focused on the general major medical journals plus the online only journal PLOS. Between the printed journals, there were substantial differences regarding the content of article titles [ 9 ]. One of the reasons could be in the instructions for authors, which differ in the provided information on the construction of a title. Specifically, the NEJM title had the lowest number of frequencies for a couple of criteria, such as the subtitle, methods mentioning, geography, abbreviations, and relation. No NEJM title contained a study name. However, the clinical context and the patient population was most frequently described in NEJM article titles. Differences between printed and online journals were obvious using geographic information in the title or usage of a phrasal title occurring more often in the online journal PLOS.

Subtitles are now more frequently used than a decade ago. Furthermore, the mentioning of methods increased in the 10 years from 2011 to 2020. This change in the title may be caused by the increased use of reporting guidelines, such as the CONSORT statement [ 34 ], which states that a randomized controlled trial should be identifiable as randomized in the title. The instructions for authors of all considered journals state that subtitles should be used for reporting the study design and/or authors should follow the respective reporting guidelines of their study. In fact, authors should look out a copy of the target journal and identify its preferences [ 35 ].

Our results are in line with the recommendations from the journal-specific instructions for authors, except NEJM. The NEJM does not follow the CONSORT statement using subtitles for randomized controlled trials, see also [ 1 ]. For the other four journals, the mentioning of the study design or the type of analysis is almost always done using subtitles as recommended. Furthermore, our results for JAMA using no declarative titles, no results mentioning or using questions in the title match with its recommendations.

Research has so far concentrated on the form of article titles rather than its content. While some authors investigated title content in BMJ, JAMA, Lancet and NEJM for a specific time, generally a single year [ 15 , 23 , 36 ], the development of title content over time has rarely been studied [ 37 ]. A strength of our work thus is the availability of all original articles over a time span of 10 years [ 9 ]. From this database, we randomly selected a subset of articles for manual assessment. These articles were evaluated by two raters according to a pre-specified coding plan with examples and training. Title evaluations were then done blinded by year and journal.

We did not expect different journal-specific frequencies regarding the geographic information in the title compared to our recent work [ 9 ], in which we performed an automatic search for country and city mentioning in the title by the use of the R package maps [ 9 ]. However, frequencies differed substantially. The automated search led to fewer titles with any geographic information. For example, the maps package did not contain countries, such as ‘England’, continents, abbreviation, such as ‘U.S.’, or terms, such as ‘English’. The main reasons for the discrepancies were for the use of country-specific abbreviations and additional country-specific terms. However, other tools or packages might have been more appropriate for the geographical query than the maps package.

One limitation of our study is that we relied on the quality of the data provided by the PubMed database [ 38 ]. Another limitation of our work is that additional variables could have been considered, e.g., more complex title content [ 12 , 16 , 22 ].

A further limitation is the sample size of 500 articles, i.e., 10 articles per journal and year. With a sample size substantially larger than 1000 articles we would have been able to study the association of title characteristics with citation counts. For example, the total sample size of our previous study, which was based on an automated search was 8096 articles [ 9 ]. With 500 articles, 95% confidence intervals are approximately 4 times larger (√8096 / √500 = 4.02), and many results, such as the association between the number of citations would not have been significant. The sample size used in this study is approximately twice that of [ 15 , 23 ], and this study with 500 articles was powered to reliably detect trends over time.

In future research, it would be of interest to analyze the effect of title content properties on citation frequencies. It would also be interesting to compare specific journals with general medical journals.

In conclusion, title content differed substantially between the five major medical journals BMJ, JAMA, Lancet, NEJM and PLOS. Furthermore, title content changed over time. We recommend that authors study titles of articles recently published in their target journal when formulating the manuscript title. Analyses of title content may generally require manual time-consuming inspections.

Supporting information

Funding statement.

The authors received no specific funding for this work.

Data Availability

• PLoS One. 2023; 18(6): e0287677.

Decision Letter 0

10 Apr 2023

PONE-D-23-07021Title Content and Form of Original Research Articles in High-Ranked Medical JournalsPLOS ONE

Dear Dr. Ziegler,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by May 25 2023 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at  gro.solp@enosolp . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

• A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
• A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
• An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols .

We look forward to receiving your revised manuscript.

Kind regards,

Boyen Huang, DDS, MHA, PhD

Academic Editor

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Thank you for stating the following in the Competing Interests section: 

   "There are no patents, products in development or marketed products to declare. AZ is a licensed Tim Albert trainer and has held several courses in the past based on Albert’s concept."

Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: "This does not alter our adherence to  PLOS ONE policies on sharing data and materials.” (as detailed online in our guide for authors http://journals.plos.org/plosone/s/competing-interests ).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared. 

Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

3. In your Data Availability statement, you have not specified where the minimal data set underlying the results described in your manuscript can be found. PLOS defines a study's minimal data set as the underlying data used to reach the conclusions drawn in the manuscript and any additional data required to replicate the reported study findings in their entirety. All PLOS journals require that the minimal data set be made fully available. For more information about our data policy, please see http://journals.plos.org/plosone/s/data-availability .

"Upon re-submitting your revised manuscript, please upload your study’s minimal underlying data set as either Supporting Information files or to a stable, public repository and include the relevant URLs, DOIs, or accession numbers within your revised cover letter. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories . Any potentially identifying patient information must be fully anonymized.

Important: If there are ethical or legal restrictions to sharing your data publicly, please explain these restrictions in detail. Please see our guidelines for more information on what we consider unacceptable restrictions to publicly sharing data: http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions . Note that it is not acceptable for the authors to be the sole named individuals responsible for ensuring data access.

We will update your Data Availability statement to reflect the information you provide in your cover letter.

4. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information. 

Additional Editor Comments:

The major concerns from the reviewers include the sample size, sample selection, and writing style.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Partly

Reviewer #4: Yes

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: I Don't Know

Reviewer #3: Yes

Reviewer #4: I Don't Know

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #3: No

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This article analyses the titles of articles published in a series of medical journals over time. It is interesting for a consideration of how naming practices affect discoverability and use of research material.

There are, though, a few aspects that need revision:

First, the title of this paper makes reference to “high ranked” medical journals, but there is no definition anywhere of how this ranking is constructed or to what ranking you are referring.

Second, the sample size of 500 is relatively small and limits the general applicability of the findings, as the paper notes. I am unclear as to whether this is too small to be useful/generalizable.

Third, some of the limitations could have been overcome with different computational methods. For example, on page 14 you state that the maps package that you used was not able correctly to identify many locations in article headings. However, other named-entity recognition tools would certainly do a better job of this. For instance, Amazon Comprehend or SageMaker could be appropriate tools here.

Fourth, the language needs careful checking throughout. For instance: “can only adequately [be] measured”; “articles meaning no sentence and no question”; “we did neither observe” → “we observed neither”; “almost the half” → “almost half”.

Fifth, and perhaps most significantly, it would be helpful for the conclusions of this paper to interpret the findings more closely. Why have these changes that you find occurred? What does it mean that subtitles are now more common? How does discoverability work in each of the title types to which you refer?

Finally, you open by stating that the prime driver of picking a good title is so that you can pick up citations and have career progression. This seems a very cynical way of thinking about how to title articles. Scientists and medics should use titles that accurately reflect the content of the work and allow others easily to find and re-use their research. I would suggest amending this opening to incorporate such a stance.

Reviewer #2: The paper titled “Title Content and Form of Original Research Articles in High-Ranked Medical Journals” investigates the differences of title content and form between papers in the medical field and their changes over time. Overall, the paper is well-written and well-argued. The methodology is adequate, and there are an overall coherence and relation to the scope of publication in the Plos One journal. In addition, this manuscript addresses a very interesting issue about the analysis of titles and does so in a very competent technical way. It Is worth mentioning that data and R code are shared.

I do however have a major issue (which in fact, is a minor one). The authors did a huge effort in sharing all the data; however, the results section sometimes it's difficult to follow (the reader should go back and forth checking the tables). I think some (introductory) sentences in some parts of the manuscript will benefit the readability of the text (see my suggestions below).

I will go slightly more into detail with them in the position-specific comments below.

I think the paper would benefit by including some keywords related to the titles (e.g. research article titles or titles). I have reservations about the use of ‘impact’ (I think the authors are not analysing the impact of the papers, e.g. citation impact).

The introduction and background is ok, providing the necessary information leading to the purpose of the study. However, I think the authors could expand a bit more (there are more studies on the topic). See my suggestions below.

P10|Line 56. Indicate that the EQUATOR Network is referred to the reporting health research (e.g. Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network).

P10|Line 33 (and P11|Line 83). “content has rarely been investigated beyond title length”. I slightly disagree with this statement. From a bibliometric perspective, there are many articles that analyse impact (e.g. effect on citations, downloads), sentence types (e.g. informative), the information the author wants to include, and in which order, among others. I would like to suggest the following papers (not included by the authors):

• Aleixandre-Benavent, R., Montalt-Resurecció, V., & Valderrama-Zurián, J. C. (2014). A descriptive study of inaccuracy in article titles on bibliometrics published in biomedical journals. Scientometrics, 101(1), 781–791. https://doi.org/10.1007/s11192-014-1296-5 .

• Ball, R. (2009). Scholarly communication in transition: The use of question marks in the titles of scientific articles in medicine, life sciences and physics 1966–2005. Scientometrics, 79(3), 667–679. https://doi.org/10.1007/s11192-007-1984-5 .

• Busch-Lauer, I.-A. (2000). Titles of English and German research papers in medicine and linguistics theses and research articles. In A. Trosborg (Ed.), Analysing professional genres (pp. 77–94). John Benjamins Publishing Company. https://doi.org/10.1075/pbns.74.08bus

• Buter, R. K., & van Raan, A. F. J. (2011). Non-alphanumeric characters in titles of scientific publications: An analysis of their occurrence and correlation with citation impact. Journal of Informetrics, 5(4), 608–617. https://doi.org/10.1016/j.joi.2011.05.008 .

• Haggan, M. (2004). Research paper titles in literature, linguistics and science: Dimensions of attraction. Journal of Pragmatics, 36(2), 293–317. https://doi.org/10.1016/S0378-2166 (03)00090-0

• Nagano, R. L. (2015). Research article titles and disciplinary conventions: A corpus study of eight disciplines. Journal of Academic Writing, 5(1), 133–144. https://doi.org/10.18552/joaw.v5i1.168

• Pearson, W. S. (2021). Quoted speech in linguistics research article titles: patterns of use and effects on citations. Scientometrics, 126(4), 3421-3442.

P10|Line 75. Worth mentioning the Web of Science (1997) which includes the title field tag.

P11|Line 79. Indicate the acronym (Standards for Reporting of Diagnostic Accuracy (STARD)). When the authors mention ‘at least two observers should do an independent evaluation where applicable”. Are referring to the article title? (not clear)

P11|Line 84. Correct typo “(2020)compared”.

P12|106-108. The authors mention “Articles being listed with an abstract remained in the data set, while articles only listed in the Web of Science were excluded.”. I suggest indicating the number of papers. Was the abstract used for any purpose?

P12|111. Indicate in this section that ten original articles per year (100 articles per journal) were randomly chosen.

There is a lot of information in this section (Tables and supplementary material), which allows the reproducibility of the findings. However, I think some introductory sentences will benefit the readability of the text (see my suggestions below).

P14|159-160. Although the information is in the Supplementary Material, I suggest introducing a few words (just one or two sentences) about Table 2 (or Descriptive Statistics).

In Table 3, the Plos Medicine journal information is missing in the table.

P14|159-160 “About half of the PLOS titles (52%) contained any geographic information”: missing this information in Table 3 (or indicate the Supplementary table in which this information is displayed).

P16|203. Correct typo ‘ofKerans’.

P16|207. Here, the authors mention the Results/Relation (and not the previous ones, i.e. Results mention, Quantitative information or Semi-quantitative information). I suggest an introductory sentence indicating that ‘In terms of results, etc’. And also pointing out that the other previous items were rarely used.

P16|214. Regarding this information (24%), indicate in brackets Table 2 (or 3.2. Supplementary Table)

P16|224. Indicate that refers to the discussion/conclusion part.

Discussion/Conclusions

Line 19|278-279. Another aspect that should be considered is the title length allowed by each journal (number maximum of words). Also there is of interest the recommendations from the journals (e.g. in the author guidelines). In some journals there is some criteria such as ‘Specific, descriptive, concise, and comprehensible to readers outside the field’ (Plos One), whereas in others it is suggested to include a subtitle (e.g. https://jamanetwork.com/journals/jamanetworkopen/pages/instructions-for-authors ).

Line 20 |321. Another limitation is the variables considered (e.g. some other studies analyse other non-alphanumeric elements such as exclamations, other criteria for the content, etc.)

Line21|333. A sentence about further research could be included.

Reviewer #3: • There was a desire to look at characteristics within journals over time but use of only 10 articles per year seems subject to selection bias for this research question. What was the power consideration here?

• It is not clear how the titles were evaluated. Was it an automated program or each one manually? The samples size is small enough that manual adjudication is possible.

• Line 115. This is confusing. Results are not given for harmonization of classification of various title attributes.

• Reporting of ORs is confusing. For example on line 170, what is the OR for? The proportion listing method per year? Similar in line 188 – you say the rate varied over time, but you used logistic regression assuming an increase over time?

• The results section includes discussion points (like line 202).

• Were any findings linked back to author instructions for each journal? These often dictate title content.

• Linking the metrics assessed to citation counts would add an important dimension to the significance of this research.

• There is too much repetition of p values in the Discussion. I assume these were not presenting new analyses not shown in the results. It is not appropriate. Line 312 – they do report new analyses. It should be part of the study of not (unless published elsewhere).

• The Discussion is too long.

• The recommendation near the end that authors study titles in their target journals before submission is unfounded. Title is often dictated by author guidelines or changed during peer review. They did not study this particular question – in other words, they did not study title of rejected compared to accepted articles.

Reviewer #4: This study applies what are in my opinion very sound methodologies to analyze the titles of prestigious, general medical journals. The paper is well-written, and its significance lays on going beyond other studies in investigating titles’ form & content and the development of title content over time. To do so, they had to select a representative sample of articles over a period of 10 years. Having two (trained) raters made the methodology strong, as it was the methodology followed in the “Evaluation of title content and form” section. (I have to admit, however, that I lack the expertise to say that the statistical analyses have been performed appropriately and rigorously. So in the following I assume these have been done correctly.) By following this well-crafted methodology, and providing all the relevant data, the code to analyze it, and in detail results in the supplementary materials, the conclusions arrived at are well supported (see some comments below, though) and could be replicated by others.

I do have some specific concerns or comments that I would like the authors to address:

1. I think the authors should stick to the wording “general major medical journals” instead of “highly ranked” as they don’t define which “rank” that is or where it can be found or calculated.

2. Mentioning of guidelines for authors writing the papers in the journals analyzed was not mentioned at all ---even as it is mentioned in the literature they quote. I think this is important as to it may be determining why authors use a particular way to phrase the title. The reader is left to assume that no guidance was provided by the journal that could have biased title wording. I think this to be particularly important for the use or avoidance of abbreviations, dashes, and/or subtitles.

3. The authors recommendation “We recommend that authors study titles of articles recently published in their target journal when formulating the manuscript title” does not seem supported other than by their results implying this is what you find in them already. So, why should you follow the same? Would that make it more likely to be published? The paper’s introduction makes reference to increasing citation frequency in databases, and so does at least one of the authors’ previous papers, yet it’s never mentioned explicitly as a possible outcome of choosing title according to the journal to submit.

4. Regarding their recommendation “In our opinion, authors should avoid providing quantitative or semi-quantitative information in the title. In fact, since the title is a one-line summary, the conclusions could be spread out into the world without reading at least the abstract or the full text of the article. ”I think this argument should expand as to what the consequences are in following this behavior e.g. propagation of misinformation.

5. In their statement “Another limitation of our study is that we relied on the quality of the data provided by the database of PubMed. Specifically, we may have missed some original articles in our database search. And we have previously identified a couple of errors in the database (Heßler and Ziegler, 2022).” One shouldn’t expect the reader to go their paper for finding out what was wrong with those hits/articles.

6. Finally, author AZ declares, in the competing interests field, that he's a "licensed Tim Albert trainer and has held several courses in the past based on Albert’s concept." Please consider adding the statement that (at least some of the) Tim Albert trainings deal with advising people how to write medical papers.

P.S. There are a few typos, like missing words and letters, that need to be corrected throughout the manuscript.

6. PLOS authors have the option to publish the peer review history of their article ( what does this mean? ). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy .

Reviewer #2: No

Reviewer #4: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool,  https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at  gro.solp@serugif . Please note that Supporting Information files do not need this step.

Author response to Decision Letter 0

22 May 2023

See separate document

Submitted filename: Response_to_Reviewers_V01.pdf

Decision Letter 1

12 Jun 2023

Title Content and Form of Original Research Articles in General Major Medical Journals

PONE-D-23-07021R1

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/ , click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at gro.solp@gnillibrohtua .

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact gro.solp@sserpeno .

Additional Editor Comments (optional):

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

Reviewer #4: All comments have been addressed

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #4: (No Response)

3. Has the statistical analysis been performed appropriately and rigorously?

4. Have the authors made all data underlying the findings in their manuscript fully available?

5. Is the manuscript presented in an intelligible fashion and written in standard English?

6. Review Comments to the Author

Reviewer #2: I have read the second revision of this manuscript and, in my opinion, the authors have addressed the main issues and provided a satisfactory answer in their response.

7. PLOS authors have the option to publish the peer review history of their article ( what does this mean? ). If published, this will include your full peer review and any attached files.

Acceptance letter

15 Jun 2023

Dear Dr. Ziegler:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact gro.solp@sserpeno .

If we can help with anything else, please email us at gro.solp@enosolp .

Thank you for submitting your work to PLOS ONE and supporting open access.

PLOS ONE Editorial Office Staff

on behalf of

Dr Boyen Huang

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 15 April 2024

Revealed: the ten research papers that policy documents cite most

• Dalmeet Singh Chawla 0

Dalmeet Singh Chawla is a freelance science journalist based in London.

You can also search for this author in PubMed   Google Scholar

Policymakers often work behind closed doors — but the documents they produce offer clues about the research that influences them. Credit: Stefan Rousseau/Getty

When David Autor co-wrote a paper on how computerization affects job skill demands more than 20 years ago, a journal took 18 months to consider it — only to reject it after review. He went on to submit it to The Quarterly Journal of Economics , which eventually published the work 1 in November 2003.

Autor’s paper is now the third most cited in policy documents worldwide, according to an analysis of data provided exclusively to Nature . It has accumulated around 1,100 citations in policy documents, show figures from the London-based firm Overton (see ‘The most-cited papers in policy’), which maintains a database of more than 12 million policy documents, think-tank papers, white papers and guidelines.

“I thought it was destined to be quite an obscure paper,” recalls Autor, a public-policy scholar and economist at the Massachusetts Institute of Technology in Cambridge. “I’m excited that a lot of people are citing it.”

The most-cited papers in policy

Economics papers dominate the top ten papers that policy documents reference most.

Data from Sage Policy Profiles as of 15 April 2024

The top ten most cited papers in policy documents are dominated by economics research. When economics studies are excluded, a 1997 Nature paper 2 about Earth’s ecosystem services and natural capital is second on the list, with more than 900 policy citations. The paper has also garnered more than 32,000 references from other studies, according to Google Scholar. Other highly cited non-economics studies include works on planetary boundaries, sustainable foods and the future of employment (see ‘Most-cited papers — excluding economics research’).

These lists provide insight into the types of research that politicians pay attention to, but policy citations don’t necessarily imply impact or influence, and Overton’s database has a bias towards documents published in English.

Interdisciplinary impact

Overton usually charges a licence fee to access its citation data. But last year, the firm worked with the London-based publisher Sage to release a free web-based tool that allows any researcher to find out how many times policy documents have cited their papers or mention their names. Overton and Sage said they created the tool, called Sage Policy Profiles, to help researchers to demonstrate the impact or influence their work might be having on policy. This can be useful for researchers during promotion or tenure interviews and in grant applications.

Autor thinks his study stands out because his paper was different from what other economists were writing at the time. It suggested that ‘middle-skill’ work, typically done in offices or factories by people who haven’t attended university, was going to be largely automated, leaving workers with either highly skilled jobs or manual work. “It has stood the test of time,” he says, “and it got people to focus on what I think is the right problem.” That topic is just as relevant today, Autor says, especially with the rise of artificial intelligence.

Most-cited papers — excluding economics research

When economics studies are excluded, the research papers that policy documents most commonly reference cover topics including climate change and nutrition.

Walter Willett, an epidemiologist and food scientist at the Harvard T.H. Chan School of Public Health in Boston, Massachusetts, thinks that interdisciplinary teams are most likely to gain a lot of policy citations. He co-authored a paper on the list of most cited non-economics studies: a 2019 work 3 that was part of a Lancet commission to investigate how to feed the global population a healthy and environmentally sustainable diet by 2050 and has accumulated more than 600 policy citations.

“I think it had an impact because it was clearly a multidisciplinary effort,” says Willett. The work was co-authored by 37 scientists from 17 countries. The team included researchers from disciplines including food science, health metrics, climate change, ecology and evolution and bioethics. “None of us could have done this on our own. It really did require working with people outside our fields.”

Sverker Sörlin, an environmental historian at the KTH Royal Institute of Technology in Stockholm, agrees that papers with a diverse set of authors often attract more policy citations. “It’s the combined effect that is often the key to getting more influence,” he says.

Has your research influenced policy? Use this free tool to check

Sörlin co-authored two papers in the list of top ten non-economics papers. One of those is a 2015 Science paper 4 on planetary boundaries — a concept defining the environmental limits in which humanity can develop and thrive — which has attracted more than 750 policy citations. Sörlin thinks one reason it has been popular is that it’s a sequel to a 2009 Nature paper 5 he co-authored on the same topic, which has been cited by policy documents 575 times.

Although policy citations don’t necessarily imply influence, Willett has seen evidence that his paper is prompting changes in policy. He points to Denmark as an example, noting that the nation is reformatting its dietary guidelines in line with the study’s recommendations. “I certainly can’t say that this document is the only thing that’s changing their guidelines,” he says. But “this gave it the support and credibility that allowed them to go forward”.

Broad brush

Peter Gluckman, who was the chief science adviser to the prime minister of New Zealand between 2009 and 2018, is not surprised by the lists. He expects policymakers to refer to broad-brush papers rather than those reporting on incremental advances in a field.

Gluckman, a paediatrician and biomedical scientist at the University of Auckland in New Zealand, notes that it’s important to consider the context in which papers are being cited, because studies reporting controversial findings sometimes attract many citations. He also warns that the list is probably not comprehensive: many policy papers are not easily accessible to tools such as Overton, which uses text mining to compile data, and so will not be included in the database.

The top 100 papers

“The thing that worries me most is the age of the papers that are involved,” Gluckman says. “Does that tell us something about just the way the analysis is done or that relatively few papers get heavily used in policymaking?”

Gluckman says it’s strange that some recent work on climate change, food security, social cohesion and similar areas hasn’t made it to the non-economics list. “Maybe it’s just because they’re not being referred to,” he says, or perhaps that work is cited, in turn, in the broad-scope papers that are most heavily referenced in policy documents.

As for Sage Policy Profiles, Gluckman says it’s always useful to get an idea of which studies are attracting attention from policymakers, but he notes that studies often take years to influence policy. “Yet the average academic is trying to make a claim here and now that their current work is having an impact,” he adds. “So there’s a disconnect there.”

Willett thinks policy citations are probably more important than scholarly citations in other papers. “In the end, we don’t want this to just sit on an academic shelf.”

doi: https://doi.org/10.1038/d41586-024-00660-1

Autor, D. H., Levy, F. & Murnane, R. J. Q. J. Econ. 118 , 1279–1333 (2003).

Article   Google Scholar  

Costanza, R. et al. Nature 387 , 253–260 (1997).

Willett, W. et al. Lancet 393 , 447–492 (2019).

Article   PubMed   Google Scholar  

Steffen, W. et al. Science 347 , 1259855 (2015).

Rockström, J. et al. Nature 461 , 472–475 (2009).

Download references

Reprints and permissions

Related Articles

Use game theory for climate models that really help reach net zero goals

Correspondence 16 APR 24

Female academics need more support — in China as elsewhere

The world needs a COP for water like the one for climate change

Last-mile delivery increases vaccine uptake in Sierra Leone

Article 13 MAR 24

Global supply chains amplify economic costs of future extreme heat risk

How science is helping farmers to find a balance between agriculture and solar farms

Spotlight 19 FEB 24

Postdoctoral Research Associate position at University of Oklahoma Health Sciences Center

Postdoctoral Research Associate position at University of Oklahoma Health Sciences Center   The Kamiya Mehla lab at the newly established Departmen...

Oklahoma City, Oklahoma

University of Oklahoma Health Sciences Center

Computational Postdoctoral Fellow with a Strong Background in Bioinformatics

Houston, Texas (US)

The University of Texas MD Anderson Cancer Center

Locum Associate or Senior Editor (Immunology), Nature Communications

The Editor in Immunology at Nature Communications will handle original research papers and work on all aspects of the editorial process.

London, Beijing or Shanghai - Hybrid working model

Springer Nature Ltd

Assistant Professor - Cell Physiology & Molecular Biophysics

Opportunity in the Department of Cell Physiology and Molecular Biophysics (CPMB) at Texas Tech University Health Sciences Center (TTUHSC)

Lubbock, Texas

Texas Tech University Health Sciences Center, School of Medicine

Postdoctoral Associate- Curing Brain Tumors

Baylor College of Medicine (BCM)

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

To revisit this article, visit My Profile, then View saved stories .

• Backchannel
• Newsletters
• WIRED Insider
• WIRED Consulting

Amanda Hoover

Students Are Likely Writing Millions of Papers With AI

Students have submitted more than 22 million papers that may have used generative AI in the past year, new data released by plagiarism detection company Turnitin shows.

A year ago, Turnitin rolled out an AI writing detection tool that was trained on its trove of papers written by students as well as other AI-generated texts. Since then, more than 200 million papers have been reviewed by the detector, predominantly written by high school and college students. Turnitin found that 11 percent may contain AI-written language in 20 percent of its content, with 3 percent of the total papers reviewed getting flagged for having 80 percent or more AI writing. (Turnitin is owned by Advance, which also owns Condé Nast, publisher of WIRED.) Turnitin says its detector has a false positive rate of less than 1 percent when analyzing full documents.

ChatGPT’s launch was met with knee-jerk fears that the English class essay would die . The chatbot can synthesize information and distill it near-instantly—but that doesn’t mean it always gets it right. Generative AI has been known to hallucinate , creating its own facts and citing academic references that don’t actually exist. Generative AI chatbots have also been caught spitting out biased text on gender and race . Despite those flaws, students have used chatbots for research, organizing ideas, and as a ghostwriter . Traces of chatbots have even been found in peer-reviewed, published academic writing .

Teachers understandably want to hold students accountable for using generative AI without permission or disclosure. But that requires a reliable way to prove AI was used in a given assignment. Instructors have tried at times to find their own solutions to detecting AI in writing, using messy, untested methods to enforce rules , and distressing students. Further complicating the issue, some teachers are even using generative AI in their grading processes.

Detecting the use of gen AI is tricky. It’s not as easy as flagging plagiarism, because generated text is still original text. Plus, there’s nuance to how students use gen AI; some may ask chatbots to write their papers for them in large chunks or in full, while others may use the tools as an aid or a brainstorm partner.

Students also aren't tempted by only ChatGPT and similar large language models. So-called word spinners are another type of AI software that rewrites text, and may make it less obvious to a teacher that work was plagiarized or generated by AI. Turnitin’s AI detector has also been updated to detect word spinners, says Annie Chechitelli, the company’s chief product officer. It can also flag work that was rewritten by services like spell checker Grammarly, which now has its own generative AI tool . As familiar software increasingly adds generative AI components, what students can and can’t use becomes more muddled.

Detection tools themselves have a risk of bias. English language learners may be more likely to set them off; a 2023 study found a 61.3 percent false positive rate when evaluating Test of English as a Foreign Language (TOEFL) exams with seven different AI detectors. The study did not examine Turnitin’s version. The company says it has trained its detector on writing from English language learners as well as native English speakers. A study published in October found that Turnitin was among the most accurate of 16 AI language detectors in a test that had the tool examine undergraduate papers and AI-generated papers.

Julian Chokkattu

Eric Geller

Caroline Haskins

Schools that use Turnitin had access to the AI detection software for a free pilot period, which ended at the start of this year. Chechitelli says a majority of the service’s clients have opted to purchase the AI detection. But the risks of false positives and bias against English learners have led some universities to ditch the tools for now. Montclair State University in New Jersey announced in November that it would pause use of Turnitin’s AI detector. Vanderbilt University and Northwestern University did the same last summer.

“This is hard. I understand why people want a tool,” says Emily Isaacs, executive director of the Office of Faculty Excellence at Montclair State. But Isaacs says the university is concerned about potentially biased results from AI detectors, as well as the fact that the tools can’t provide confirmation the way they can with plagiarism. Plus, Montclair State doesn’t want to put a blanket ban on AI, which will have some place in academia. With time and more trust in the tools, the policies could change. “It’s not a forever decision, it’s a now decision,” Isaacs says.

Chechitelli says the Turnitin tool shouldn’t be the only consideration in passing or failing a student. Instead, it’s a chance for teachers to start conversations with students that touch on all of the nuance in using generative AI. “People don’t really know where that line should be,” she says.

You Might Also Like …

In your inbox: The best and weirdest stories from WIRED’s archive

Jeffrey Epstein’s island visitors exposed by data broker

8 Google employees invented modern AI. Here’s the inside story

The crypto fraud kingpin who almost got away

It's shadow time! How to view the solar eclipse, online and in person

Steven Levy

Will Knight

Lauren Goode

Matt Burgess

Benj Edwards, Ars Technica

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Browse by collection
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 110, Issue 9
• The role of COVID-19 vaccines in preventing post-COVID-19 thromboembolic and cardiovascular complications
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Núria Mercadé-Besora 1 , 2 , 3 ,
• Xintong Li 1 ,
• Raivo Kolde 4 ,
• Nhung TH Trinh 5 ,
• Maria T Sanchez-Santos 1 ,
• Wai Yi Man 1 ,
• Elena Roel 3 ,
• Carlen Reyes 3 ,
• http://orcid.org/0000-0003-0388-3403 Antonella Delmestri 1 ,
• Hedvig M E Nordeng 6 , 7 ,
• http://orcid.org/0000-0002-4036-3856 Anneli Uusküla 8 ,
• http://orcid.org/0000-0002-8274-0357 Talita Duarte-Salles 3 , 9 ,
• Clara Prats 2 ,
• http://orcid.org/0000-0002-3950-6346 Daniel Prieto-Alhambra 1 , 9 ,
• http://orcid.org/0000-0002-0000-0110 Annika M Jödicke 1 ,
• Martí Català 1
• 1 Pharmaco- and Device Epidemiology Group, Health Data Sciences, Botnar Research Centre, NDORMS , University of Oxford , Oxford , UK
• 2 Department of Physics , Universitat Politècnica de Catalunya , Barcelona , Spain
• 3 Fundació Institut Universitari per a la recerca a l'Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol) , IDIAP Jordi Gol , Barcelona , Catalunya , Spain
• 4 Institute of Computer Science , University of Tartu , Tartu , Estonia
• 5 Pharmacoepidemiology and Drug Safety Research Group, Department of Pharmacy, Faculty of Mathematics and Natural Sciences , University of Oslo , Oslo , Norway
• 6 School of Pharmacy , University of Oslo , Oslo , Norway
• 7 Division of Mental Health , Norwegian Institute of Public Health , Oslo , Norway
• 8 Department of Family Medicine and Public Health , University of Tartu , Tartu , Estonia
• 9 Department of Medical Informatics, Erasmus University Medical Center , Erasmus University Rotterdam , Rotterdam , Zuid-Holland , Netherlands
• Correspondence to Prof Daniel Prieto-Alhambra, Pharmaco- and Device Epidemiology Group, Health Data Sciences, Botnar Research Centre, NDORMS, University of Oxford, Oxford, UK; daniel.prietoalhambra{at}ndorms.ox.ac.uk

Objective To study the association between COVID-19 vaccination and the risk of post-COVID-19 cardiac and thromboembolic complications.

Methods We conducted a staggered cohort study based on national vaccination campaigns using electronic health records from the UK, Spain and Estonia. Vaccine rollout was grouped into four stages with predefined enrolment periods. Each stage included all individuals eligible for vaccination, with no previous SARS-CoV-2 infection or COVID-19 vaccine at the start date. Vaccination status was used as a time-varying exposure. Outcomes included heart failure (HF), venous thromboembolism (VTE) and arterial thrombosis/thromboembolism (ATE) recorded in four time windows after SARS-CoV-2 infection: 0–30, 31–90, 91–180 and 181–365 days. Propensity score overlap weighting and empirical calibration were used to minimise observed and unobserved confounding, respectively.

Fine-Gray models estimated subdistribution hazard ratios (sHR). Random effect meta-analyses were conducted across staggered cohorts and databases.

Results The study included 10.17 million vaccinated and 10.39 million unvaccinated people. Vaccination was associated with reduced risks of acute (30-day) and post-acute COVID-19 VTE, ATE and HF: for example, meta-analytic sHR of 0.22 (95% CI 0.17 to 0.29), 0.53 (0.44 to 0.63) and 0.45 (0.38 to 0.53), respectively, for 0–30 days after SARS-CoV-2 infection, while in the 91–180 days sHR were 0.53 (0.40 to 0.70), 0.72 (0.58 to 0.88) and 0.61 (0.51 to 0.73), respectively.

Conclusions COVID-19 vaccination reduced the risk of post-COVID-19 cardiac and thromboembolic outcomes. These effects were more pronounced for acute COVID-19 outcomes, consistent with known reductions in disease severity following breakthrough versus unvaccinated SARS-CoV-2 infection.

• Epidemiology
• PUBLIC HEALTH
• Electronic Health Records

Data availability statement

Data may be obtained from a third party and are not publicly available. CPRD: CPRD data were obtained under the CPRD multi-study license held by the University of Oxford after Research Data Governance (RDG) approval. Direct data sharing is not allowed. SIDIAP: In accordance with current European and national law, the data used in this study is only available for the researchers participating in this study. Thus, we are not allowed to distribute or make publicly available the data to other parties. However, researchers from public institutions can request data from SIDIAP if they comply with certain requirements. Further information is available online ( https://www.sidiap.org/index.php/menu-solicitudesen/application-proccedure ) or by contacting SIDIAP ([email protected]). CORIVA: CORIVA data were obtained under the approval of Research Ethics Committee of the University of Tartu and the patient level data sharing is not allowed. All analyses in this study were conducted in a federated manner, where analytical code and aggregated (anonymised) results were shared, but no patient-level data was transferred across the collaborating institutions.

This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See:  https://creativecommons.org/licenses/by/4.0/ .

https://doi.org/10.1136/heartjnl-2023-323483

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

WHAT IS ALREADY KNOWN ON THIS TOPIC

COVID-19 vaccines proved to be highly effective in reducing the severity of acute SARS-CoV-2 infection.

While COVID-19 vaccines were associated with increased risk for cardiac and thromboembolic events, such as myocarditis and thrombosis, the risk of complications was substantially higher due to SARS-CoV-2 infection.

WHAT THIS STUDY ADDS

COVID-19 vaccination reduced the risk of heart failure, venous thromboembolism and arterial thrombosis/thromboembolism in the acute (30 days) and post-acute (31 to 365 days) phase following SARS-CoV-2 infection. This effect was stronger in the acute phase.

The overall additive effect of vaccination on the risk of post-vaccine and/or post-COVID thromboembolic and cardiac events needs further research.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

COVID-19 vaccines proved to be highly effective in reducing the risk of post-COVID cardiovascular and thromboembolic complications.

Introduction

COVID-19 vaccines were approved under emergency authorisation in December 2020 and showed high effectiveness against SARS-CoV-2 infection, COVID-19-related hospitalisation and death. 1 2 However, concerns were raised after spontaneous reports of unusual thromboembolic events following adenovirus-based COVID-19 vaccines, an association that was further assessed in observational studies. 3 4 More recently, mRNA-based vaccines were found to be associated with a risk of rare myocarditis events. 5 6

On the other hand, SARS-CoV-2 infection can trigger cardiac and thromboembolic complications. 7 8 Previous studies showed that, while slowly decreasing over time, the risk for serious complications remain high for up to a year after infection. 9 10 Although acute and post-acute cardiac and thromboembolic complications following COVID-19 are rare, they present a substantial burden to the affected patients, and the absolute number of cases globally could become substantial.

Recent studies suggest that COVID-19 vaccination could protect against cardiac and thromboembolic complications attributable to COVID-19. 11 12 However, most studies did not include long-term complications and were conducted among specific populations.

Evidence is still scarce as to whether the combined effects of COVID-19 vaccines protecting against SARS-CoV-2 infection and reducing post-COVID-19 cardiac and thromboembolic outcomes, outweigh any risks of these complications potentially associated with vaccination.

We therefore used large, representative data sources from three European countries to assess the overall effect of COVID-19 vaccines on the risk of acute and post-acute COVID-19 complications including venous thromboembolism (VTE), arterial thrombosis/thromboembolism (ATE) and other cardiac events. Additionally, we studied the comparative effects of ChAdOx1 versus BNT162b2 on the risk of these same outcomes.

Data sources

We used four routinely collected population-based healthcare datasets from three European countries: the UK, Spain and Estonia.

For the UK, we used data from two primary care databases—namely, Clinical Practice Research Datalink, CPRD Aurum 13 and CPRD Gold. 14 CPRD Aurum currently covers 13 million people from predominantly English practices, while CPRD Gold comprises 3.1 million active participants mostly from GP practices in Wales and Scotland. Spanish data were provided by the Information System for the Development of Research in Primary Care (SIDIAP), 15 which encompasses primary care records from 6 million active patients (around 75% of the population in the region of Catalonia) linked to hospital admissions data (Conjunt Mínim Bàsic de Dades d’Alta Hospitalària). Finally, the CORIVA dataset based on national health claims data from Estonia was used. It contains all COVID-19 cases from the first year of the pandemic and ~440 000 randomly selected controls. CORIVA was linked to the death registry and all COVID-19 testing from the national health information system.

Databases included sociodemographic information, diagnoses, measurements, prescriptions and secondary care referrals and were linked to vaccine registries, including records of all administered vaccines from all healthcare settings. Data availability for CPRD Gold ended in December 2021, CPRD Aurum in January 2022, SIDIAP in June 2022 and CORIVA in December 2022.

All databases were mapped to the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) 16 to facilitate federated analytics.

Multinational network staggered cohort study: study design and participants

The study design has been published in detail elsewhere. 17 Briefly, we used a staggered cohort design considering vaccination as a time-varying exposure. Four staggered cohorts were designed with each cohort representing a country-specific vaccination rollout phase (eg, dates when people became eligible for vaccination, and eligibility criteria).

The source population comprised all adults registered in the respective database for at least 180 days at the start of the study (4 January 2021 for CPRD Gold and Aurum, 20 February 2021 for SIDIAP and 28 January 2021 for CORIVA). Subsequently, each staggered cohort corresponded to an enrolment period: all people eligible for vaccination during this time were included in the cohort and people with a history of SARS-CoV-2 infection or COVID-19 vaccination before the start of the enrolment period were excluded. Across countries, cohort 1 comprised older age groups, whereas cohort 2 comprised individuals at risk for severe COVID-19. Cohort 3 included people aged ≥40 and cohort 4 enrolled people aged ≥18.

In each cohort, people receiving a first vaccine dose during the enrolment period were allocated to the vaccinated group, with their index date being the date of vaccination. Individuals who did not receive a vaccine dose comprised the unvaccinated group and their index date was assigned within the enrolment period, based on the distribution of index dates in the vaccinated group. People with COVID-19 before the index date were excluded.

Follow-up started from the index date until the earliest of end of available data, death, change in exposure status (first vaccine dose for those unvaccinated) or outcome of interest.

COVID-19 vaccination

All vaccines approved within the study period from January 2021 to July 2021—namely, ChAdOx1 (Oxford/AstraZeneca), BNT162b2 (BioNTech/Pfizer]) Ad26.COV2.S (Janssen) and mRNA-1273 (Moderna), were included for this study.

Post-COVID-19 outcomes of interest

Outcomes of interest were defined as SARS-CoV-2 infection followed by a predefined thromboembolic or cardiac event of interest within a year after infection, and with no record of the same clinical event in the 6 months before COVID-19. Outcome date was set as the corresponding SARS-CoV-2 infection date.

COVID-19 was identified from either a positive SARS-CoV-2 test (polymerase chain reaction (PCR) or antigen), or a clinical COVID-19 diagnosis, with no record of COVID-19 in the previous 6 weeks. This wash-out period was imposed to exclude re-recordings of the same COVID-19 episode.

Post-COVID-19 outcome events were selected based on previous studies. 11–13 Events comprised ischaemic stroke (IS), haemorrhagic stroke (HS), transient ischaemic attack (TIA), ventricular arrhythmia/cardiac arrest (VACA), myocarditis/pericarditis (MP), myocardial infarction (MI), heart failure (HF), pulmonary embolism (PE) and deep vein thrombosis (DVT). We used two composite outcomes: (1) VTE, as an aggregate of PE and DVT and (2) ATE, as a composite of IS, TIA and MI. To avoid re-recording of the same complication we imposed a wash-out period of 90 days between records. Phenotypes for these complications were based on previously published studies. 3 4 8 18

All outcomes were ascertained in four different time periods following SARS-CoV-2 infection: the first period described the acute infection phase—that is, 0–30 days after COVID-19, whereas the later periods - which are 31–90 days, 91–180 days and 181–365 days, illustrate the post-acute phase ( figure 1 ).

• Download figure
• Open in new tab
• Download powerpoint

Study outcome design. Study outcomes of interest are defined as a COVID-19 infection followed by one of the complications in the figure, within a year after infection. Outcomes were ascertained in four different time windows after SARS-CoV-2 infection: 0–30 days (namely the acute phase), 31–90 days, 91–180 days and 181–365 days (these last three comprise the post-acute phase).

Negative control outcomes

Negative control outcomes (NCOs) were used to detect residual confounding. NCOs are outcomes which are not believed to be causally associated with the exposure, but share the same bias structure with the exposure and outcome of interest. Therefore, no significant association between exposure and NCO is to be expected. Our study used 43 different NCOs from previous work assessing vaccine effectiveness. 19

Statistical analysis

Federated network analyses.

A template for an analytical script was developed and subsequently tailored to include the country-specific aspects (eg, dates, priority groups) for the vaccination rollout. Analyses were conducted locally for each database. Only aggregated data were shared and person counts <5 were clouded.

Propensity score weighting

Large-scale propensity scores (PS) were calculated to estimate the likelihood of a person receiving the vaccine based on their demographic and health-related characteristics (eg, conditions, medications) prior to the index date. PS were then used to minimise observed confounding by creating a weighted population (overlap weighting 20 ), in which individuals contributed with a different weight based on their PS and vaccination status.

Prespecified key variables included in the PS comprised age, sex, location, index date, prior observation time in the database, number of previous outpatient visits and previous SARS-CoV-2 PCR/antigen tests. Regional vaccination, testing and COVID-19 incidence rates were also forced into the PS equation for the UK databases 21 and SIDIAP. 22 In addition, least absolute shrinkage and selection operator (LASSO) regression, a technique for variable selection, was used to identify additional variables from all recorded conditions and prescriptions within 0–30 days, 31–180 days and 181-any time (conditions only) before the index date that had a prevalence of >0.5% in the study population.

PS were then separately estimated for each staggered cohort and analysis. We considered covariate balance to be achieved if absolute standardised mean differences (ASMDs) were ≤0.1 after weighting. Baseline characteristics such as demographics and comorbidities were reported.

Effect estimation

To account for the competing risk of death associated with COVID-19, Fine-and-Grey models 23 were used to calculate subdistribution hazard ratios (sHRs). Subsequently, sHRs and confidence intervals were empirically calibrated from NCO estimates 24 to account for unmeasured confounding. To calibrate the estimates, the empirical null distribution was derived from NCO estimates and was used to compute calibrated confidence intervals. For each outcome, sHRs from the four staggered cohorts were pooled using random-effect meta-analysis, both separately for each database and across all four databases.

Sensitivity analysis

Sensitivity analyses comprised 1) censoring follow-up for vaccinated people at the time when they received their second vaccine dose and 2) considering only the first post-COVID-19 outcome within the year after infection ( online supplemental figure S1 ). In addition, comparative effectiveness analyses were conducted for BNT162b2 versus ChAdOx1.

Supplemental material

Data and code availability.

All analytic code for the study is available in GitHub ( https://github.com/oxford-pharmacoepi/vaccineEffectOnPostCovidCardiacThromboembolicEvents ), including code lists for vaccines, COVID-19 tests and diagnoses, cardiac and thromboembolic events, NCO and health conditions to prioritise patients for vaccination in each country. We used R version 4.2.3 and statistical packages survival (3.5–3), Empirical Calibration (3.1.1), glmnet (4.1-7), and Hmisc (5.0–1).

Patient and public involvement

Owing to the nature of the study and the limitations regarding data privacy, the study design, analysis, interpretation of data and revision of the manuscript did not involve any patients or members of the public.

All aggregated results are available in a web application ( https://dpa-pde-oxford.shinyapps.io/PostCovidComplications/ ).

We included over 10.17 million vaccinated individuals (1 618 395 from CPRD Gold; 5 729 800 from CPRD Aurum; 2 744 821 from SIDIAP and 77 603 from CORIVA) and 10.39 million unvaccinated individuals (1 640 371; 5 860 564; 2 588 518 and 302 267, respectively). Online supplemental figures S2-5 illustrate study inclusion for each database.

Adequate covariate balance was achieved after PS weighting in most studies: CORIVA (all cohorts) and SIDIAP (cohorts 1 and 4) did not contribute to ChAdOx1 subanalyses owing to sample size and covariate imbalance. ASMD results are accessible in the web application.

NCO analyses suggested residual bias after PS weighting, with a majority of NCOs associated positively with vaccination. Therefore, calibrated estimates are reported in this manuscript. Uncalibrated effect estimates and NCO analyses are available in the web interface.

Population characteristics

Table 1 presents baseline characteristics for the weighted populations in CPRD Aurum, for illustrative purposes. Online supplemental tables S1-25 summarise baseline characteristics for weighted and unweighted populations for each database and comparison. Across databases and cohorts, populations followed similar patterns: cohort 1 represented an older subpopulation (around 80 years old) with a high proportion of women (57%). Median age was lowest in cohort 4 ranging between 30 and 40 years.

• View inline

Characteristics of weighted populations in CPRD Aurum database, stratified by staggered cohort and exposure status. Exposure is any COVID-19 vaccine

COVID-19 vaccination and post-COVID-19 complications

Table 2 shows the incidence of post-COVID-19 VTE, ATE and HF, the three most common post-COVID-19 conditions among the studied outcomes. Outcome counts are presented separately for 0–30, 31–90, 91–180 and 181–365 days after SARS-CoV-2 infection. Online supplemental tables S26-36 include all studied complications, also for the sensitivity and subanalyses. Similar pattern for incidences were observed across all databases: higher outcome rates in the older populations (cohort 1) and decreasing frequency with increasing time after infection in all cohorts.

Number of records (and risk per 10 000 individuals) for acute and post-acute COVID-19 cardiac and thromboembolic complications, across cohorts and databases for any COVID-19 vaccination

Forest plots for the effect of COVID-19 vaccines on post-COVID-19 cardiac and thromboembolic complications; meta-analysis across cohorts and databases. Dashed line represents a level of heterogeneity I 2 >0.4. ATE, arterial thrombosis/thromboembolism; CD+HS, cardiac diseases and haemorrhagic stroke; VTE, venous thromboembolism.

Results from calibrated estimates pooled in meta-analysis across cohorts and databases are shown in figure 2 .

Reduced risk associated with vaccination is observed for acute and post-acute VTE, DVT, and PE: acute meta-analytic sHR are 0.22 (95% CI, 0.17–0.29); 0.36 (0.28–0.45); and 0.19 (0.15–0.25), respectively. For VTE in the post-acute phase, sHR estimates are 0.43 (0.34–0.53), 0.53 (0.40–0.70) and 0.50 (0.36–0.70) for 31–90, 91–180, and 181–365 days post COVID-19, respectively. Reduced risk of VTE outcomes was observed in vaccinated across databases and cohorts, see online supplemental figures S14–22 .

Similarly, the risk of ATE, IS and MI in the acute phase after infection was reduced for the vaccinated group, sHR of 0.53 (0.44–0.63), 0.55 (0.43–0.70) and 0.49 (0.38–0.62), respectively. Reduced risk associated with vaccination persisted for post-acute ATE, with sHR of 0.74 (0.60–0.92), 0.72 (0.58–0.88) and 0.62 (0.48–0.80) for 31–90, 91–180 and 181–365 days post-COVID-19, respectively. Risk of post-acute MI remained lower for vaccinated in the 31–90 and 91–180 days after COVID-19, with sHR of 0.64 (0.46–0.87) and 0.64 (0.45–0.90), respectively. Vaccination effect on post-COVID-19 TIA was seen only in the 181–365 days, with sHR of 0.51 (0.31–0.82). Online supplemental figures S23-31 show database-specific and cohort-specific estimates for ATE-related complications.

Risk of post-COVID-19 cardiac complications was reduced in vaccinated individuals. Meta-analytic estimates in the acute phase showed sHR of 0.45 (0.38–0.53) for HF, 0.41 (0.26–0.66) for MP and 0.41 (0.27–0.63) for VACA. Reduced risk persisted for post-acute COVID-19 HF: sHR 0.61 (0.51–0.73) for 31–90 days, 0.61 (0.51–0.73) for 91–180 days and 0.52 (0.43–0.63) for 181–365 days. For post-acute MP, risk was only lowered in the first post-acute window (31–90 days), with sHR of 0.43 (0.21–0.85). Vaccination showed no association with post-COVID-19 HS. Database-specific and cohort-specific results for these cardiac diseases are shown in online supplemental figures S32-40 .

Stratified analyses by vaccine showed similar associations, except for ChAdOx1 which was not associated with reduced VTE and ATE risk in the last post-acute window. Sensitivity analyses were consistent with main results ( online supplemental figures S6-13 ).

Figure 3 shows the results of comparative effects of BNT162b2 versus ChAdOx1, based on UK data. Meta-analytic estimates favoured BNT162b2 (sHR of 0.66 (0.46–0.93)) for VTE in the 0–30 days after infection, but no differences were seen for post-acute VTE or for any of the other outcomes. Results from sensitivity analyses, database-specific and cohort-specific estimates were in line with the main findings ( online supplemental figures S41-51 ).

Forest plots for comparative vaccine effect (BNT162b2 vs ChAdOx1); meta-analysis across cohorts and databases. ATE, arterial thrombosis/thromboembolism; CD+HS, cardiac diseases and haemorrhagic stroke; VTE, venous thromboembolism.

Key findings

Our analyses showed a substantial reduction of risk (45–81%) for thromboembolic and cardiac events in the acute phase of COVID-19 associated with vaccination. This finding was consistent across four databases and three different European countries. Risks for post-acute COVID-19 VTE, ATE and HF were reduced to a lesser extent (24–58%), whereas a reduced risk for post-COVID-19 MP and VACA in vaccinated people was seen only in the acute phase.

Results in context

The relationship between SARS-CoV-2 infection, COVID-19 vaccines and thromboembolic and/or cardiac complications is tangled. Some large studies report an increased risk of VTE and ATE following both ChAdOx1 and BNT162b2 vaccination, 7 whereas other studies have not identified such a risk. 25 Elevated risk of VTE has also been reported among patients with COVID-19 and its occurrence can lead to poor prognosis and mortality. 26 27 Similarly, several observational studies have found an association between COVID-19 mRNA vaccination and a short-term increased risk of myocarditis, particularly among younger male individuals. 5 6 For instance, a self-controlled case series study conducted in England revealed about 30% increased risk of hospital admission due to myocarditis within 28 days following both ChAdOx1 and BNT162b2 vaccines. However, this same study also found a ninefold higher risk for myocarditis following a positive SARS-CoV-2 test, clearly offsetting the observed post-vaccine risk.

COVID-19 vaccines have demonstrated high efficacy and effectiveness in preventing infection and reducing the severity of acute-phase infection. However, with the emergence of newer variants of the virus, such as omicron, and the waning protective effect of the vaccine over time, there is a growing interest in understanding whether the vaccine can also reduce the risk of complications after breakthrough infections. Recent studies suggested that COVID-19 vaccination could potentially protect against acute post-COVID-19 cardiac and thromboembolic events. 11 12 A large prospective cohort study 11 reports risk of VTE after SARS-CoV-2 infection to be substantially reduced in fully vaccinated ambulatory patients. Likewise, Al-Aly et al 12 suggest a reduced risk for post-acute COVID-19 conditions in breakthrough infection versus SARS-CoV-2 infection without prior vaccination. However, the populations were limited to SARS-CoV-2 infected individuals and estimates did not include the effect of the vaccine to prevent COVID-19 in the first place. Other studies on post-acute COVID-19 conditions and symptoms have been conducted, 28 29 but there has been limited reporting on the condition-specific risks associated with COVID-19, even though the prognosis for different complications can vary significantly.

In line with previous studies, our findings suggest a potential benefit of vaccination in reducing the risk of post-COVID-19 thromboembolic and cardiac complications. We included broader populations, estimated the risk in both acute and post-acute infection phases and replicated these using four large independent observational databases. By pooling results across different settings, we provided the most up-to-date and robust evidence on this topic.

Strengths and limitations

The study has several strengths. Our multinational study covering different healthcare systems and settings showed consistent results across all databases, which highlights the robustness and replicability of our findings. All databases had complete recordings of vaccination status (date and vaccine) and are representative of the respective general population. Algorithms to identify study outcomes were used in previous published network studies, including regulatory-funded research. 3 4 8 18 Other strengths are the staggered cohort design which minimises confounding by indication and immortal time bias. PS overlap weighting and NCO empirical calibration have been shown to adequately minimise bias in vaccine effectiveness studies. 19 Furthermore, our estimates include the vaccine effectiveness against COVID-19, which is crucial in the pathway to experience post-COVID-19 complications.

Our study has some limitations. The use of real-world data comes with inherent limitations including data quality concerns and risk of confounding. To deal with these limitations, we employed state-of-the-art methods, including large-scale propensity score weighting and calibration of effect estimates using NCO. 19 24 A recent study 30 has demonstrated that methodologically sound observational studies based on routinely collected data can produce results similar to those of clinical trials. We acknowledge that results from NCO were positively associated with vaccination, and estimates might still be influenced by residual bias despite using calibration. Another limitation is potential under-reporting of post-COVID-19 complications: some asymptomatic and mild COVID-19 infections might have not been recorded. Additionally, post-COVID-19 outcomes of interest might be under-recorded in primary care databases (CPRD Aurum and Gold) without hospital linkage, which represent a large proportion of the data in the study. However, results in SIDIAP and CORIVA, which include secondary care data, were similar. Also, our study included a small number of young men and male teenagers, who were the main population concerned with increased risks of myocarditis/pericarditis following vaccination.

Conclusions

Vaccination against SARS-CoV-2 substantially reduced the risk of acute post-COVID-19 thromboembolic and cardiac complications, probably through a reduction in the risk of SARS-CoV-2 infection and the severity of COVID-19 disease due to vaccine-induced immunity. Reduced risk in vaccinated people lasted for up to 1 year for post-COVID-19 VTE, ATE and HF, but not clearly for other complications. Findings from this study highlight yet another benefit of COVID-19 vaccination. However, further research is needed on the possible waning of the risk reduction over time and on the impact of booster vaccination.

Ethics statements

Patient consent for publication.

Not applicable.

Ethics approval

The study was approved by the CPRD’s Research Data Governance Process, Protocol No 21_000557 and the Clinical Research Ethics committee of Fundació Institut Universitari per a la recerca a l’Atenció Primària de Salut Jordi Gol i Gurina (IDIAPJGol) (approval number 4R22/133) and the Research Ethics Committee of the University of Tartu (approval No. 330/T-10).

Acknowledgments

This study is based in part on data from the Clinical Practice Research Datalink (CPRD) obtained under licence from the UK Medicines and Healthcare products Regulatory Agency. We thank the patients who provided these data, and the NHS who collected the data as part of their care and support. All interpretations, conclusions and views expressed in this publication are those of the authors alone and not necessarily those of CPRD. We would also like to thank the healthcare professionals in the Catalan healthcare system involved in the management of COVID-19 during these challenging times, from primary care to intensive care units; the Institut de Català de la Salut and the Program d’Analítica de Dades per a la Recerca i la Innovació en Salut for providing access to the different data sources accessible through The System for the Development of Research in Primary Care (SIDIAP).

• Pritchard E ,
• Matthews PC ,
• Stoesser N , et al
• Lauring AS ,
• Tenforde MW ,
• Chappell JD , et al
• Pistillo A , et al
• Duarte-Salles T , et al
• Hansen JV ,
• Fosbøl E , et al
• Chen A , et al
• Hippisley-Cox J ,
• Mei XW , et al
• Duarte-Salles T ,
• Fernandez-Bertolin S , et al
• Ip S , et al
• Bowe B , et al
• Prats-Uribe A ,
• Feng Q , et al
• Campbell J , et al
• Herrett E ,
• Gallagher AM ,
• Bhaskaran K , et al
• Raventós B ,
• Fernández-Bertolín S ,
• Aragón M , et al
• Makadia R ,
• Matcho A , et al
• Mercadé-Besora N ,
• Kolde R , et al
• Ostropolets A ,
• Makadia R , et al
• Rathod-Mistry T , et al
• Thomas LE ,
• ↵ Coronavirus (COVID-19) in the UK . 2022 . Available : https://coronavirus.data.gov.uk/
• Generalitat de Catalunya
• Schuemie MJ ,
• Hripcsak G ,
• Ryan PB , et al
• Houghton DE ,
• Wysokinski W ,
• Casanegra AI , et al
• Katsoularis I ,
• Fonseca-Rodríguez O ,
• Farrington P , et al
• Jehangir Q ,
• Li P , et al
• Byambasuren O ,
• Stehlik P ,
• Clark J , et al
• Brannock MD ,
• Preiss AJ , et al
• Schneeweiss S , RCT-DUPLICATE Initiative , et al

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

• Data supplement 1

AMJ and MC are joint senior authors.

Contributors DPA and AMJ led the conceptualisation of the study with contributions from MC and NM-B. AMJ, TD-S, ER, AU and NTHT adapted the study design with respect to the local vaccine rollouts. AD and WYM mapped and curated CPRD data. MC and NM-B developed code with methodological contributions advice from MTS-S and CP. DPA, MC, NTHT, TD-S, HMEN, XL, CR and AMJ clinically interpreted the results. NM-B, XL, AMJ and DPA wrote the first draft of the manuscript, and all authors read, revised and approved the final version. DPA and AMJ obtained the funding for this research. DPA is responsible for the overall content as guarantor: he accepts full responsibility for the work and the conduct of the study, had access to the data, and controlled the decision to publish.

Funding The research was supported by the National Institute for Health and Care Research (NIHR) Oxford Biomedical Research Centre (BRC). DPA is funded through a NIHR Senior Research Fellowship (Grant number SRF-2018–11-ST2-004). Funding to perform the study in the SIDIAP database was provided by the Real World Epidemiology (RWEpi) research group at IDIAPJGol. Costs of databases mapping to OMOP CDM were covered by the European Health Data and Evidence Network (EHDEN).

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

This paper is in the following e-collection/theme issue:

Published on 12.4.2024 in Vol 26 (2024)

Application of AI in in Multilevel Pain Assessment Using Facial Images: Systematic Review and Meta-Analysis

Authors of this article:

• Jian Huo 1 * , MSc   ; 
• Yan Yu 2 * , MMS   ; 
• Wei Lin 3 , MMS   ; 
• Anmin Hu 2, 3, 4 , MMS   ; 
• Chaoran Wu 2 , MD, PhD  

1 Boston Intelligent Medical Research Center, Shenzhen United Scheme Technology Company Limited, Boston, MA, United States

2 Department of Anesthesia, Shenzhen People's Hospital, The First Affiliated Hospital of Southern University of Science and Technology, Shenzhen Key Medical Discipline, Shenzhen, China

3 Shenzhen United Scheme Technology Company Limited, Shenzhen, China

4 The Second Clinical Medical College, Jinan University, Shenzhen, China

*these authors contributed equally

Corresponding Author:

Chaoran Wu, MD, PhD

Department of Anesthesia

Shenzhen People's Hospital, The First Affiliated Hospital of Southern University of Science and Technology

Shenzhen Key Medical Discipline

No 1017, Dongmen North Road

Shenzhen, 518020

Phone: 86 18100282848

Email: [email protected]

Background: The continuous monitoring and recording of patients’ pain status is a major problem in current research on postoperative pain management. In the large number of original or review articles focusing on different approaches for pain assessment, many researchers have investigated how computer vision (CV) can help by capturing facial expressions. However, there is a lack of proper comparison of results between studies to identify current research gaps.

Objective: The purpose of this systematic review and meta-analysis was to investigate the diagnostic performance of artificial intelligence models for multilevel pain assessment from facial images.

Methods: The PubMed, Embase, IEEE, Web of Science, and Cochrane Library databases were searched for related publications before September 30, 2023. Studies that used facial images alone to estimate multiple pain values were included in the systematic review. A study quality assessment was conducted using the Quality Assessment of Diagnostic Accuracy Studies, 2nd edition tool. The performance of these studies was assessed by metrics including sensitivity, specificity, log diagnostic odds ratio (LDOR), and area under the curve (AUC). The intermodal variability was assessed and presented by forest plots.

Results: A total of 45 reports were included in the systematic review. The reported test accuracies ranged from 0.27-0.99, and the other metrics, including the mean standard error (MSE), mean absolute error (MAE), intraclass correlation coefficient (ICC), and Pearson correlation coefficient (PCC), ranged from 0.31-4.61, 0.24-2.8, 0.19-0.83, and 0.48-0.92, respectively. In total, 6 studies were included in the meta-analysis. Their combined sensitivity was 98% (95% CI 96%-99%), specificity was 98% (95% CI 97%-99%), LDOR was 7.99 (95% CI 6.73-9.31), and AUC was 0.99 (95% CI 0.99-1). The subgroup analysis showed that the diagnostic performance was acceptable, although imbalanced data were still emphasized as a major problem. All studies had at least one domain with a high risk of bias, and for 20% (9/45) of studies, there were no applicability concerns.

Conclusions: This review summarizes recent evidence in automatic multilevel pain estimation from facial expressions and compared the test accuracy of results in a meta-analysis. Promising performance for pain estimation from facial images was established by current CV algorithms. Weaknesses in current studies were also identified, suggesting that larger databases and metrics evaluating multiclass classification performance could improve future studies.

Trial Registration: PROSPERO CRD42023418181; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=418181

Introduction

The definition of pain was revised to “an unpleasant sensory and emotional experience associated with, or resembling that associated with, actual or potential tissue damage” in 2020 [ 1 ]. Acute postoperative pain management is important, as pain intensity and duration are critical influencing factors for the transition of acute pain to chronic postsurgical pain [ 2 ]. To avoid the development of chronic pain, guidelines were promoted and discussed to ensure safe and adequate pain relief for patients, and clinicians were recommended to use a validated pain assessment tool to track patients’ responses [ 3 ]. However, these tools, to some extent, depend on communication between physicians and patients, and continuous data cannot be provided [ 4 ]. The continuous assessment and recording of patient pain intensity will not only reduce caregiver burden but also provide data for chronic pain research. Therefore, automatic and accurate pain measurements are necessary.

Researchers have proposed different approaches to measuring pain intensity. Physiological signals, for example, electroencephalography and electromyography, have been used to estimate pain [ 5 - 7 ]. However, it was reported that current pain assessment from physiological signals has difficulties isolating stress and pain with machine learning techniques, as they share conceptual and physiological similarities [ 8 ]. Recent studies have also investigated pain assessment tools for certain patient subgroups. For example, people with deafness or an intellectual disability may not be able to communicate well with nurses, and an objective pain evaluation would be a better option [ 9 , 10 ]. Measuring pain intensity from patient behaviors, such as facial expressions, is also promising for most patients [ 4 ]. As the most comfortable and convenient method, computer vision techniques require no attachments to patients and can monitor multiple participants using 1 device [ 4 ]. However, pain intensity, which is important for pain research, is often not reported.

With the growing trend of assessing pain intensity using artificial intelligence (AI), it is necessary to summarize current publications to determine the strengths and gaps of current studies. Existing research has reviewed machine learning applications for acute postoperative pain prediction, continuous pain detection, and pain intensity estimation [ 10 - 14 ]. Input modalities, including facial recordings and physiological signals such as electroencephalography and electromyography, were also reviewed [ 5 , 8 ]. There have also been studies focusing on deep learning approaches [ 11 ]. AI was applied in children and infant pain evaluation as well [ 15 , 16 ]. However, no study has focused on pain intensity measurement, and no comparison of test accuracy results has been made.

Current AI applications in pain research can be categorized into 3 types: pain assessment, pain prediction and decision support, and pain self-management [ 14 ]. We consider accurate and automatic pain assessment to be the most important area and the foundation of future pain research. In this study, we performed a systematic review and meta-analysis to assess the diagnostic performance of current publications for multilevel pain evaluation.

This study was registered with PROSPERO (International Prospective Register of Systematic Reviews; CRD42023418181) and carried out strictly following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [ 17 ] .

Study Eligibility

Studies that reported AI techniques for multiclass pain intensity classification were eligible. Records including nonhuman or infant participants or 2-class pain detection were excluded. Only studies using facial images of the test participants were accepted. Clinically used pain assessment tools, such as the visual analog scale (VAS) and numerical rating scale (NRS), and other pain intensity indicators, were rejected in the meta-analysis. Textbox 1 presents the eligibility criteria.

Study characteristics and inclusion criteria

• Participants: children and adults aged 12 months or older
• Setting: no restrictions
• Index test: artificial intelligence models that measure pain intensity from facial images
• Reference standard: no restrictions for systematic review; Prkachin and Solomon pain intensity score for meta-analysis
• Study design: no need to specify

Study characteristics and exclusion criteria

• Participants: infants aged 12 months or younger and animal subjects
• Setting: no need to specify
• Index test: studies that use other information such as physiological signals
• Reference standard: other pain evaluation tools, e.g., NRS, VAS, were excluded from meta-analysis
• Study design: reviews

Report characteristics and inclusion criteria

• Year: published between January 1, 2012, and September 30, 2023
• Language: English only
• Publication status: published
• Test accuracy metrics: no restrictions for systematic reviews; studies that reported contingency tables were included for meta-analysis

Report characteristics and exclusion criteria

• Year: no need to specify
• Language: no need to specify
• Publication status: preprints not accepted
• Test accuracy metrics: studies that reported insufficient metrics were excluded from meta-analysis

Search Strategy

In this systematic review, databases including PubMed, Embase, IEEE, Web of Science, and the Cochrane Library were searched until December 2022, and no restrictions were applied. Keywords were “artificial intelligence” AND “pain recognition.” Multimedia Appendix 1 shows the detailed search strategy.

Data Extraction

A total of 2 viewers screened titles and abstracts and selected eligible records independently to assess eligibility, and disagreements were solved by discussion with a third collaborator. A consentient data extraction sheet was prespecified and used to summarize study characteristics independently. Table S5 in Multimedia Appendix 1 shows the detailed items and explanations for data extraction. Diagnostic accuracy data were extracted into contingency tables, including true positives, false positives, false negatives, and true negatives. The data were used to calculate the pooled diagnostic performance of the different models. Some studies included multiple models, and these models were considered independent of each other.

Study Quality Assessment

All included studies were independently assessed by 2 viewers using the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) tool [ 18 ]. QUADAS-2 assesses bias risk across 4 domains, which are patient selection, index test, reference standard, and flow and timing. The first 3 domains are also assessed for applicability concerns. In the systematic review, a specific extension of QUADAS-2, namely, QUADAS-AI, was used to specify the signaling questions [ 19 ].

Meta-Analysis

Meta-analyses were conducted between different AI models. Models with different algorithms or training data were considered different. To evaluate the performance differences between models, the contingency tables during model validation were extracted. Studies that did not report enough diagnostic accuracy data were excluded from meta-analysis.

Hierarchical summary receiver operating characteristic (SROC) curves were fitted to evaluate the diagnostic performance of AI models. These curves were plotted with 95% CIs and prediction regions around averaged sensitivity, specificity, and area under the curve estimates. Heterogeneity was assessed visually by forest plots. A funnel plot was constructed to evaluate the risk of bias.

Subgroup meta-analyses were conducted to evaluate the performance differences at both the model level and task level, and subgroups were created based on different tasks and the proportion of positive and negative samples.

All statistical analyses and plots were produced using RStudio (version 4.2.2; R Core Team) and the R package meta4diag (version 2.1.1; Guo J and Riebler A) [ 20 ].

Study Selection and Included Study Characteristics

A flow diagram representing the study selection process is shown in ( Figure 1 ). After removing 1039 duplicates, the titles and abstracts of a total of 5653 papers were screened, and the percentage agreement of title or abstract screening was 97%. After screening, 51 full-text reports were assessed for eligibility, among which 45 reports were included in the systematic review [ 21 - 65 ]. The percentage agreement of the full-text review was 87%. In 40 of the included studies, contingency tables could not be made. Meta-analyses were conducted based on 8 AI models extracted from 6 studies. Individual study characteristics included in the systematic review are provided in Tables 1 and 2 . The facial feature extraction method can be categorized into 2 classes: geometrical features (GFs) and deep features (DFs). One typical method of extracting GFs is to calculate the distance between facial landmarks. DFs are usually extracted by convolution operations. A total of 20 studies included temporal information, but most of them (18) extracted temporal information through the 3D convolution of video sequences. Feature transformation was also commonly applied to reduce the time for training or fuse features extracted by different methods before inputting them into the classifier. For classifiers, support vector machines (SVMs) and convolutional neural networks (CNNs) were mostly used. Table 1 presents the model designs of the included studies.

a No temporal features are shown by – symbol, time information extracted from 2 images at different time by +, and deep temporal features extracted through the convolution of video sequences by ++.

b SVM: support vector machine.

c GF: geometric feature.

d GMM: gaussian mixture model.

e TPS: thin plate spline.

f DML: distance metric learning.

g MDML: multiview distance metric learning.

h AAM: active appearance model.

i RVR: relevance vector regressor.

j PSPI: Prkachin and Solomon pain intensity.

k I-FES: individual facial expressiveness score.

l LSTM: long short-term memory.

m HCRF: hidden conditional random field.

n GLMM: generalized linear mixed model.

o VLAD: vector of locally aggregated descriptor.

p SVR: support vector regression.

q MDS: multidimensional scaling.

r ELM: extreme learning machine.

s Labeled to distinguish different architectures of ensembled deep learning models.

t DCNN: deep convolutional neural network.

u GSM: gaussian scale mixture.

v DOML: distance ordering metric learning.

w LIAN: locality and identity aware network.

x BiLSTM: bidirectional long short-term memory.

a UNBC: University of Northern British Columbia-McMaster shoulder pain expression archive database.

b LOSO: leave one subject out cross-validation.

c ICC: intraclass correlation coefficient.

d CT: contingency table.

e AUC: area under the curve.

f MSE: mean standard error.

g PCC: Pearson correlation coefficient.

h RMSE: root mean standard error.

i MAE: mean absolute error.

j ICC: intraclass coefficient.

k CCC: concordance correlation coefficient.

l Reported both external and internal validation results and summarized as intervals.

Table 2 summarizes the characteristics of model training and validation. Most studies used publicly available databases, for example, the University of Northern British Columbia-McMaster shoulder pain expression archive database [ 57 ]. Table S4 in Multimedia Appendix 1 summarizes the public databases. A total of 7 studies used self-prepared databases. Frames from video sequences were the most used test objects, as 37 studies output frame-level pain intensity, while few measure pain intensity from video sequences or photos. It was common that a study redefined pain levels to have fewer classes than ground-truth labels. For model validation, cross-validation and leave-one-subject-out validation were commonly used. Only 3 studies performed external validation. For reporting test accuracies, different evaluation metrics were used, including sensitivity, specificity, mean absolute error (MAE), mean standard error (MSE), Pearson correlation coefficient (PCC), and intraclass coefficient (ICC).

Methodological Quality of Included Studies

Table S2 in Multimedia Appendix 1 presents the study quality summary, as assessed by QUADAS-2. There was a risk of bias in all studies, specifically in terms of patient selection, caused by 2 issues. First, the training data are highly imbalanced, and any method to adjust the data distribution may introduce bias. Next, the QUADAS-AI correspondence letter [ 19 ] specifies that preprocessing of images that changes the image size or resolution may introduce bias. However, the applicability concern is low, as the images properly represent the feeling of pain. Studies that used cross-fold validation or leave-one-out cross-validation were considered to have a low risk of bias. Although the Prkachin and Solomon pain intensity (PSPI) score was used by most of the studies, its ability to represent individual pain levels was not clinically validated; as such, the risk of bias and applicability concerns were considered high when the PSPI score was used as the index test. As an advantage of computer vision techniques, the time interval between the index tests was short and was assessed as having a low risk of bias. Risk proportions are shown in Figure 2 . For all 315 entries, 39% (124) were assessed as high-risk. In total, 5 studies had the lowest risk of bias, with 6 domains assessed as low risk [ 26 , 27 , 31 , 32 , 59 ].

Pooled Performance of Included Models

In 6 studies included in the meta-analysis, there were 8 different models. The characteristics of these models are summarized in Table S1 in Multimedia Appendix 2 [ 23 , 24 , 26 , 32 , 41 , 57 ]. Classification of PSPI scores greater than 0, 2, 3, 6, and 9 was selected and considered as different tasks to create contingency tables. The test performance is shown in Figure 3 as hierarchical SROC curves; 27 contingency tables were extracted from 8 models. The sensitivity, specificity, and LDOR were calculated, and the combined sensitivity was 98% (95% CI 96%-99%), the specificity was 98% (95% CI 97%-99%), the LDOR was 7.99 (95% CI 6.73-9.31) and the AUC was 0.99 (95% CI 0.99-1).

Subgroup Analysis

In this study, subgroup analysis was conducted to investigate the performance differences within models. A total of 8 models were separated and summarized as a forest plot in Multimedia Appendix 3 [ 23 , 24 , 26 , 32 , 41 , 57 ]. For model 1, the pooled sensitivity, specificity, and LDOR were 95% (95% CI 86%-99%), 99% (95% CI 98%-100%), and 8.38 (95% CI 6.09-11.19), respectively. For model 2, the pooled sensitivity, specificity, and LDOR were 94% (95% CI 84%-99%), 95% (95% CI 88%-99%), and 6.23 (95% CI 3.52-9.04), respectively. For model 3, the pooled sensitivity, specificity, and LDOR were 100% (95% CI 99%-100%), 100% (95% CI 99%-100%), and 11.55% (95% CI 8.82-14.43), respectively. For model 4, the pooled sensitivity, specificity, and LDOR were 83% (95% CI 43%-99%), 94% (95% CI 79%-99%), and 5.14 (95% CI 0.93-9.31), respectively. For model 5, the pooled sensitivity, specificity, and LDOR were 92% (95% CI 68%-99%), 94% (95% CI 78%-99%), and 6.12 (95% CI 1.82-10.16), respectively. For model 6, the pooled sensitivity, specificity, and LDOR were 94% (95% CI 74%-100%), 94% (95% CI 78%-99%), and 6.59 (95% CI 2.21-11.13), respectively. For model 7, the pooled sensitivity, specificity, and LDOR were 98% (95% CI 90%-100%), 97% (95% CI 87%-100%), and 8.31 (95% CI 4.3-12.29), respectively. For model 8, the pooled sensitivity, specificity, and LDOR were 98% (95% CI 93%-100%), 97% (95% CI 88%-100%), and 8.65 (95% CI 4.84-12.67), respectively.

Heterogeneity Analysis

The meta-analysis results indicated that AI models are applicable for estimating pain intensity from facial images. However, extreme heterogeneity existed within the models except for models 3 and 5, which were proposed by Rathee and Ganotra [ 24 ] and Semwal and Londhe [ 32 ]. A funnel plot is presented in Figure 4 . A high risk of bias was observed.

Pain management has long been a critical problem in clinical practice, and the use of AI may be a solution. For acute pain management, automatic measurement of pain can reduce the burden on caregivers and provide timely warnings. For chronic pain management, as specified by Glare et al [ 2 ], further research is needed, and measurements of pain presence, intensity, and quality are one of the issues to be solved for chronic pain studies. Computer vision could improve pain monitoring through real-time detection for clinical use and data recording for prospective pain studies. To our knowledge, this is the first meta-analysis dedicated to AI performance in multilevel pain level classification.

In this study, one model’s performance at specific pain levels was described by stacking multiple classes into one to make each task a binary classification problem. After careful selection in both the medical and engineering databases, we observed promising results of AI in evaluating multilevel pain intensity through facial images, with high sensitivity (98%), specificity (98%), LDOR (7.99), and AUC (0.99). It is reasonable to believe that AI can accurately evaluate pain intensity from facial images. Moreover, the study quality and risk of bias were evaluated using an adapted QUADAS-2 assessment tool, which is a strength of this study.

To investigate the source of heterogeneity, it was assumed that a well-designed model should have familiar size effects regarding different levels, and a subgroup meta-analysis was conducted. The funnel and forest plots exhibited extreme heterogeneity. The model’s performance at specific pain levels was described and summarized by a forest plot. Within-model heterogeneity was observed in Multimedia Appendix 3 [ 23 , 24 , 26 , 32 , 41 , 57 ] except for 2 models. Models 3 and 5 were different in many aspects, including their algorithms and validation methods, but were both trained with a relatively small data set, and the proportion of positive and negative classes was relatively close to 1. Because training with imbalanced data is a critical problem in computer vision studies [ 66 ], for example, in the University of Northern British Columbia-McMaster pain data set, fewer than 10 frames out of 48,398 had a PSPI score greater than 13. Here, we emphasized that imbalanced data sets are one major cause of heterogeneity, resulting in the poorer performance of AI algorithms.

We tentatively propose a method to minimize the effect of training with imbalanced data by stacking multiple classes into one class, which is already presented in studies included in the systematic review [ 26 , 32 , 42 , 57 ]. Common methods to minimize bias include resampling and data augmentation [ 66 ]. This proposed method is used in the meta-analysis to compare the test results of different studies as well. The stacking method is available when classes are only different in intensity. A disadvantage of combined classes is that the model would be insufficient in clinical practice when the number of classes is low. Commonly used pain evaluation tools, such as VAS, have 10 discrete levels. It is recommended that future studies set the number of pain levels to be at least 10 for model training.

This study is limited for several reasons. First, insufficient data were included because different performance metrics (mean standard error and mean average error) were used in most studies, which could not be summarized into a contingency table. To create a contingency table that can be included in a meta-analysis, the study should report the following: the number of objects used in each pain class for model validation, and the accuracy, sensitivity, specificity, and F 1 -score for each pain class. This table cannot be created if a study reports the MAE, PCC, and other commonly used metrics in AI development. Second, a small study effect was observed in the funnel plot, and the heterogeneity could not be minimized. Another limitation is that the PSPI score is not clinically validated and is not the only tool that assesses pain from facial expressions. There are other clinically validated pain intensity assessment methods, such as the Faces Pain Scale-revised, Wong-Baker Faces Pain Rating Scale, and Oucher Scale [ 3 ]. More databases could be created based on the above-mentioned tools. Finally, AI-assisted pain assessments were supposed to cover larger populations, including incommunicable patients, for example, patients with dementia or patients with masked faces. However, only 1 study considered patients with dementia, which was also caused by limited databases [ 50 ].

AI is a promising tool that can help in pain research in the future. In this systematic review and meta-analysis, one approach using computer vision was investigated to measure pain intensity from facial images. Despite some risk of bias and applicability concerns, CV models can achieve excellent test accuracy. Finally, more CV studies in pain estimation, reporting accuracy in contingency tables, and more pain databases are encouraged for future studies. Specifically, the creation of a balanced public database that contains not only healthy but also nonhealthy participants should be prioritized. The recording process would be better in a clinical environment. Then, it is recommended that researchers report the validation results in terms of accuracy, sensitivity, specificity, or contingency tables, as well as the number of objects for each pain class, for the inclusion of a meta-analysis.

Acknowledgments

WL, AH, and CW contributed to the literature search and data extraction. JH and YY wrote the first draft of the manuscript. All authors contributed to the conception and design of the study, the risk of bias evaluation, data analysis and interpretation, and contributed to and approved the final version of the manuscript.

Data Availability

The data sets generated during and analyzed during this study are available in the Figshare repository [ 67 ].

Conflicts of Interest

None declared.

PRISMA checklist, risk of bias summary, search strategy, database summary and reported items and explanations.

Study performance summary.

Forest plot presenting pooled performance of subgroups in meta-analysis.

• Raja SN, Carr DB, Cohen M, Finnerup NB, Flor H, Gibson S, et al. The revised International Association for the Study of Pain definition of pain: concepts, challenges, and compromises. Pain. 2020;161(9):1976-1982. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Glare P, Aubrey KR, Myles PS. Transition from acute to chronic pain after surgery. Lancet. 2019;393(10180):1537-1546. [ CrossRef ] [ Medline ]
• Chou R, Gordon DB, de Leon-Casasola OA, Rosenberg JM, Bickler S, Brennan T, et al. Management of postoperative pain: a clinical practice guideline from the American Pain Society, the American Society of Regional Anesthesia and Pain Medicine, and the American Society of Anesthesiologists' Committee on Regional Anesthesia, Executive Committee, and Administrative Council. J Pain. 2016;17(2):131-157. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hassan T, Seus D, Wollenberg J, Weitz K, Kunz M, Lautenbacher S, et al. Automatic detection of pain from facial expressions: a survey. IEEE Trans Pattern Anal Mach Intell. 2021;43(6):1815-1831. [ CrossRef ] [ Medline ]
• Mussigmann T, Bardel B, Lefaucheur JP. Resting-State Electroencephalography (EEG) biomarkers of chronic neuropathic pain. A systematic review. Neuroimage. 2022;258:119351. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Moscato S, Cortelli P, Chiari L. Physiological responses to pain in cancer patients: a systematic review. Comput Methods Programs Biomed. 2022;217:106682. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Thiam P, Hihn H, Braun DA, Kestler HA, Schwenker F. Multi-modal pain intensity assessment based on physiological signals: a deep learning perspective. Front Physiol. 2021;12:720464. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rojas RF, Brown N, Waddington G, Goecke R. A systematic review of neurophysiological sensing for the assessment of acute pain. NPJ Digit Med. 2023;6(1):76. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mansutti I, Tomé-Pires C, Chiappinotto S, Palese A. Facilitating pain assessment and communication in people with deafness: a systematic review. BMC Public Health. 2023;23(1):1594. [ FREE Full text ] [ CrossRef ] [ Medline ]
• El-Tallawy SN, Ahmed RS, Nagiub MS. Pain management in the most vulnerable intellectual disability: a review. Pain Ther. 2023;12(4):939-961. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gkikas S, Tsiknakis M. Automatic assessment of pain based on deep learning methods: a systematic review. Comput Methods Programs Biomed. 2023;231:107365. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Borna S, Haider CR, Maita KC, Torres RA, Avila FR, Garcia JP, et al. A review of voice-based pain detection in adults using artificial intelligence. Bioengineering (Basel). 2023;10(4):500. [ FREE Full text ] [ CrossRef ] [ Medline ]
• De Sario GD, Haider CR, Maita KC, Torres-Guzman RA, Emam OS, Avila FR, et al. Using AI to detect pain through facial expressions: a review. Bioengineering (Basel). 2023;10(5):548. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zhang M, Zhu L, Lin SY, Herr K, Chi CL, Demir I, et al. Using artificial intelligence to improve pain assessment and pain management: a scoping review. J Am Med Inform Assoc. 2023;30(3):570-587. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hughes JD, Chivers P, Hoti K. The clinical suitability of an artificial intelligence-enabled pain assessment tool for use in infants: feasibility and usability evaluation study. J Med Internet Res. 2023;25:e41992. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fang J, Wu W, Liu J, Zhang S. Deep learning-guided postoperative pain assessment in children. Pain. 2023;164(9):2029-2035. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Whiting PF, Rutjes AWS, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529-536. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sounderajah V, Ashrafian H, Rose S, Shah NH, Ghassemi M, Golub R, et al. A quality assessment tool for artificial intelligence-centered diagnostic test accuracy studies: QUADAS-AI. Nat Med. 2021;27(10):1663-1665. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Guo J, Riebler A. meta4diag: Bayesian bivariate meta-analysis of diagnostic test studies for routine practice. J Stat Soft. 2018;83(1):1-31. [ CrossRef ]
• Hammal Z, Cohn JF. Automatic detection of pain intensity. Proc ACM Int Conf Multimodal Interact. 2012;2012:47-52. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Adibuzzaman M, Ostberg C, Ahamed S, Povinelli R, Sindhu B, Love R, et al. Assessment of pain using facial pictures taken with a smartphone. 2015. Presented at: 2015 IEEE 39th Annual Computer Software and Applications Conference; July 01-05, 2015;726-731; Taichung, Taiwan. [ CrossRef ]
• Majumder A, Dutta S, Behera L, Subramanian VK. Shoulder pain intensity recognition using Gaussian mixture models. 2015. Presented at: 2015 IEEE International WIE Conference on Electrical and Computer Engineering (WIECON-ECE); December 19-20, 2015;130-134; Dhaka, Bangladesh. [ CrossRef ]
• Rathee N, Ganotra D. A novel approach for pain intensity detection based on facial feature deformations. J Vis Commun Image Represent. 2015;33:247-254. [ CrossRef ]
• Sikka K, Ahmed AA, Diaz D, Goodwin MS, Craig KD, Bartlett MS, et al. Automated assessment of children's postoperative pain using computer vision. Pediatrics. 2015;136(1):e124-e131. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rathee N, Ganotra D. Multiview distance metric learning on facial feature descriptors for automatic pain intensity detection. Comput Vis Image Und. 2016;147:77-86. [ CrossRef ]
• Zhou J, Hong X, Su F, Zhao G. Recurrent convolutional neural network regression for continuous pain intensity estimation in video. 2016. Presented at: 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW); June 26-July 01, 2016; Las Vegas, NV. [ CrossRef ]
• Egede J, Valstar M, Martinez B. Fusing deep learned and hand-crafted features of appearance, shape, and dynamics for automatic pain estimation. 2017. Presented at: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017); May 30-June 03, 2017;689-696; Washington, DC. [ CrossRef ]
• Martinez DL, Rudovic O, Picard R. Personalized automatic estimation of self-reported pain intensity from facial expressions. 2017. Presented at: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW); July 21-26, 2017;2318-2327; Honolulu, HI. [ CrossRef ]
• Bourou D, Pampouchidou A, Tsiknakis M, Marias K, Simos P. Video-based pain level assessment: feature selection and inter-subject variability modeling. 2018. Presented at: 2018 41st International Conference on Telecommunications and Signal Processing (TSP); July 04-06, 2018;1-6; Athens, Greece. [ CrossRef ]
• Haque MA, Bautista RB, Noroozi F, Kulkarni K, Laursen C, Irani R. Deep multimodal pain recognition: a database and comparison of spatio-temporal visual modalities. 2018. Presented at: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018); May 15-19, 2018;250-257; Xi'an, China. [ CrossRef ]
• Semwal A, Londhe ND. Automated pain severity detection using convolutional neural network. 2018. Presented at: 2018 International Conference on Computational Techniques, Electronics and Mechanical Systems (CTEMS); December 21-22, 2018;66-70; Belgaum, India. [ CrossRef ]
• Tavakolian M, Hadid A. Deep binary representation of facial expressions: a novel framework for automatic pain intensity recognition. 2018. Presented at: 2018 25th IEEE International Conference on Image Processing (ICIP); October 07-10, 2018;1952-1956; Athens, Greece. [ CrossRef ]
• Tavakolian M, Hadid A. Deep spatiotemporal representation of the face for automatic pain intensity estimation. 2018. Presented at: 2018 24th International Conference on Pattern Recognition (ICPR); August 20-24, 2018;350-354; Beijing, China. [ CrossRef ]
• Wang J, Sun H. Pain intensity estimation using deep spatiotemporal and handcrafted features. IEICE Trans Inf & Syst. 2018;E101.D(6):1572-1580. [ CrossRef ]
• Bargshady G, Soar J, Zhou X, Deo RC, Whittaker F, Wang H. A joint deep neural network model for pain recognition from face. 2019. Presented at: 2019 IEEE 4th International Conference on Computer and Communication Systems (ICCCS); February 23-25, 2019;52-56; Singapore. [ CrossRef ]
• Casti P, Mencattini A, Comes MC, Callari G, Di Giuseppe D, Natoli S, et al. Calibration of vision-based measurement of pain intensity with multiple expert observers. IEEE Trans Instrum Meas. 2019;68(7):2442-2450. [ CrossRef ]
• Lee JS, Wang CW. Facial pain intensity estimation for ICU patient with partial occlusion coming from treatment. 2019. Presented at: BIBE 2019; The Third International Conference on Biological Information and Biomedical Engineering; June 20-22, 2019;1-4; Hangzhou, China.
• Saha AK, Ahsan GMT, Gani MO, Ahamed SI. Personalized pain study platform using evidence-based continuous learning tool. 2019. Presented at: 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC); July 15-19, 2019;490-495; Milwaukee, WI. [ CrossRef ]
• Tavakolian M, Hadid A. A spatiotemporal convolutional neural network for automatic pain intensity estimation from facial dynamics. Int J Comput Vis. 2019;127(10):1413-1425. [ FREE Full text ] [ CrossRef ]
• Bargshady G, Zhou X, Deo RC, Soar J, Whittaker F, Wang H. Ensemble neural network approach detecting pain intensity from facial expressions. Artif Intell Med. 2020;109:101954. [ CrossRef ] [ Medline ]
• Bargshady G, Zhou X, Deo RC, Soar J, Whittaker F, Wang H. Enhanced deep learning algorithm development to detect pain intensity from facial expression images. Expert Syst Appl. 2020;149:113305. [ CrossRef ]
• Dragomir MC, Florea C, Pupezescu V. Automatic subject independent pain intensity estimation using a deep learning approach. 2020. Presented at: 2020 International Conference on e-Health and Bioengineering (EHB); October 29-30, 2020;1-4; Iasi, Romania. [ CrossRef ]
• Huang D, Xia Z, Mwesigye J, Feng X. Pain-attentive network: a deep spatio-temporal attention model for pain estimation. Multimed Tools Appl. 2020;79(37-38):28329-28354. [ CrossRef ]
• Mallol-Ragolta A, Liu S, Cummins N, Schuller B. A curriculum learning approach for pain intensity recognition from facial expressions. 2020. Presented at: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020); November 16-20, 2020;829-833; Buenos Aires, Argentina. [ CrossRef ]
• Peng X, Huang D, Zhang H. Pain intensity recognition via multi‐scale deep network. IET Image Process. 2020;14(8):1645-1652. [ FREE Full text ] [ CrossRef ]
• Tavakolian M, Lopez MB, Liu L. Self-supervised pain intensity estimation from facial videos via statistical spatiotemporal distillation. Pattern Recognit Lett. 2020;140:26-33. [ CrossRef ]
• Xu X, de Sa VR. Exploring multidimensional measurements for pain evaluation using facial action units. 2020. Presented at: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020); November 16-20, 2020;786-792; Buenos Aires, Argentina. [ CrossRef ]
• Pikulkaew K, Boonchieng W, Boonchieng E, Chouvatut V. 2D facial expression and movement of motion for pain identification with deep learning methods. IEEE Access. 2021;9:109903-109914. [ CrossRef ]
• Rezaei S, Moturu A, Zhao S, Prkachin KM, Hadjistavropoulos T, Taati B. Unobtrusive pain monitoring in older adults with dementia using pairwise and contrastive training. IEEE J Biomed Health Inform. 2021;25(5):1450-1462. [ CrossRef ] [ Medline ]
• Semwal A, Londhe ND. S-PANET: a shallow convolutional neural network for pain severity assessment in uncontrolled environment. 2021. Presented at: 2021 IEEE 11th Annual Computing and Communication Workshop and Conference (CCWC); January 27-30, 2021;0800-0806; Las Vegas, NV. [ CrossRef ]
• Semwal A, Londhe ND. ECCNet: an ensemble of compact convolution neural network for pain severity assessment from face images. 2021. Presented at: 2021 11th International Conference on Cloud Computing, Data Science & Engineering (Confluence); January 28-29, 2021;761-766; Noida, India. [ CrossRef ]
• Szczapa B, Daoudi M, Berretti S, Pala P, Del Bimbo A, Hammal Z. Automatic estimation of self-reported pain by interpretable representations of motion dynamics. 2021. Presented at: 2020 25th International Conference on Pattern Recognition (ICPR); January 10-15, 2021;2544-2550; Milan, Italy. [ CrossRef ]
• Ting J, Yang YC, Fu LC, Tsai CL, Huang CH. Distance ordering: a deep supervised metric learning for pain intensity estimation. 2021. Presented at: 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA); December 13-16, 2021;1083-1088; Pasadena, CA. [ CrossRef ]
• Xin X, Li X, Yang S, Lin X, Zheng X. Pain expression assessment based on a locality and identity aware network. IET Image Process. 2021;15(12):2948-2958. [ FREE Full text ] [ CrossRef ]
• Alghamdi T, Alaghband G. Facial expressions based automatic pain assessment system. Appl Sci. 2022;12(13):6423. [ FREE Full text ] [ CrossRef ]
• Barua PD, Baygin N, Dogan S, Baygin M, Arunkumar N, Fujita H, et al. Automated detection of pain levels using deep feature extraction from shutter blinds-based dynamic-sized horizontal patches with facial images. Sci Rep. 2022;12(1):17297. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fontaine D, Vielzeuf V, Genestier P, Limeux P, Santucci-Sivilotto S, Mory E, et al. Artificial intelligence to evaluate postoperative pain based on facial expression recognition. Eur J Pain. 2022;26(6):1282-1291. [ CrossRef ] [ Medline ]
• Hosseini E, Fang R, Zhang R, Chuah CN, Orooji M, Rafatirad S, et al. Convolution neural network for pain intensity assessment from facial expression. 2022. Presented at: 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC); July 11-15, 2022;2697-2702; Glasgow, Scotland. [ CrossRef ]
• Huang Y, Qing L, Xu S, Wang L, Peng Y. HybNet: a hybrid network structure for pain intensity estimation. Vis Comput. 2021;38(3):871-882. [ CrossRef ]
• Islamadina R, Saddami K, Oktiana M, Abidin TF, Muharar R, Arnia F. Performance of deep learning benchmark models on thermal imagery of pain through facial expressions. 2022. Presented at: 2022 IEEE International Conference on Communication, Networks and Satellite (COMNETSAT); November 03-05, 2022;374-379; Solo, Indonesia. [ CrossRef ]
• Swetha L, Praiscia A, Juliet S. Pain assessment model using facial recognition. 2022. Presented at: 2022 6th International Conference on Intelligent Computing and Control Systems (ICICCS); May 25-27, 2022;1-5; Madurai, India. [ CrossRef ]
• Wu CL, Liu SF, Yu TL, Shih SJ, Chang CH, Mao SFY, et al. Deep learning-based pain classifier based on the facial expression in critically ill patients. Front Med (Lausanne). 2022;9:851690. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ismail L, Waseem MD. Towards a deep learning pain-level detection deployment at UAE for patient-centric-pain management and diagnosis support: framework and performance evaluation. Procedia Comput Sci. 2023;220:339-347. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vu MT, Beurton-Aimar M. Learning to focus on region-of-interests for pain intensity estimation. 2023. Presented at: 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG); January 05-08, 2023;1-6; Waikoloa Beach, HI. [ CrossRef ]
• Kaur H, Pannu HS, Malhi AK. A systematic review on imbalanced data challenges in machine learning: applications and solutions. ACM Comput Surv. 2019;52(4):1-36. [ CrossRef ]
• Data for meta-analysis of pain assessment from facial images. Figshare. 2023. URL: https:/​/figshare.​com/​articles/​dataset/​Data_for_Meta-Analysis_of_Pain_Assessment_from_Facial_Images/​24531466/​1 [accessed 2024-03-22]

Abbreviations

Edited by A Mavragani; submitted 26.07.23; peer-reviewed by M Arab-Zozani, M Zhang; comments to author 18.09.23; revised version received 08.10.23; accepted 28.02.24; published 12.04.24.

©Jian Huo, Yan Yu, Wei Lin, Anmin Hu, Chaoran Wu. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 12.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

• Mobile Site
• Staff Directory
• Advertise with Ars

Filter by topic

• Biz & IT
• Gaming & Culture

Front page layout

REMOTELY EXPLOITABLE VULNERABILITY —

Hackable intel and lenovo hardware that went undetected for 5 years won’t ever be fixed, multiple links in the supply chain failed for years to identify an unfixed vulnerability..

Dan Goodin - Apr 11, 2024 6:53 pm UTC

Hardware sold for years by the likes of Intel and Lenovo contains a remotely exploitable vulnerability that will never be fixed. The cause: a supply chain snafu involving an open source software package and hardware from multiple manufacturers that directly or indirectly incorporated it into their products.

Researchers from security firm Binarly have confirmed that the lapse has resulted in Intel, Lenovo, and Supermicro shipping server hardware that contains a vulnerability that can be exploited to reveal security-critical information. The researchers, however, went on to warn that any hardware that incorporates certain generations of baseboard management controllers made by Duluth, Georgia-based AMI or Taiwan-based AETN are also affected.

Chain of fools

BMCs are tiny computers soldered into the motherboard of servers that allow cloud centers, and sometimes their customers, to streamline the remote management of vast fleets of servers. They enable administrators to remotely reinstall OSes, install and uninstall apps, and control just about every other aspect of the system—even when it's turned off. BMCs provide what’s known in the industry as “lights-out” system management. AMI and AETN are two of several makers of BMCs.

For years, BMCs from multiple manufacturers have incorporated vulnerable versions of open source software known as lighttpd . Lighttpd is a fast, lightweight web server that’s compatible with various hardware and software platforms. It’s used in all kinds of wares, including in embedded devices like BMCs, to allow remote administrators to control servers remotely with HTTP requests.

In 2018, lighttpd developers released a new version that fixed “various use-after-free scenarios,” a vague reference to a class of vulnerability that can be remotely exploitable to tamper with security-sensitive memory functions of the affected software. Despite the description, the update didn’t use the word “vulnerability” and didn’t include a CVE vulnerability tracking number as is customary.

BMC makers including AMI and ATEN were using affected versions of lighttpd when the vulnerability was fixed and continued doing so for years, Binarly researchers said. Server manufacturers, in turn, continued putting the vulnerable BMCs into their hardware over the same multi-year time period. Binarly has identified three of those server makers as Intel, Lenovo, and Supermicro. Hardware sold by Intel as recently as last year is affected. Binarly said that both Intel and Lenovo have no plans to release fixes because they no longer support the affected hardware. Affected products from Supermicro are still supported.

“All these years, [the lighttpd vulnerability] was present inside the firmware and nobody cared to update one of the third-party components used to build this firmware image,” Binarly researchers wrote Thursday . “This is another perfect example of inconsistencies in the firmware supply chain. A very outdated third-party component present in the latest version of firmware, creating additional risk for end users. Are there more systems that use the vulnerable version of lighttpd across the industry?”

Defeating ASLR

The vulnerability makes it possible for hackers to identify memory addresses responsible for handling key functions. Operating systems take pains to randomize and conceal these locations so they can’t be used in software exploits. By chaining an exploit for the lighttpd vulnerability with a separate vulnerability, hackers could defeat this standard protection, which is known as address space layout randomization . The chaining of two or more exploits has become a common feature of hacking attacks these days as software makers continue to add anti-exploitation protections to their code.

Tracking the supply chain for multiple BMCs used in multiple server hardware is difficult. So far, Binarly has identified AMI’s MegaRAC BMC as one of the vulnerable BMCs. The security firm has confirmed that the AMI BMC is contained in the Intel Server System M70KLP hardware. Information about BMCs from ATEN or hardware from Lenovo and Supermicro aren’t available at the moment. The vulnerability is present in any hardware that uses lighttpd versions 1.4.35, 1.4.45, and 1.4.51.

In a statement, Lenovo officials wrote:

Lenovo is aware of the AMI MegaRAC concern identified by Binarly. We are working with our supplier to identify any potential impacts to Lenovo products. ThinkSystem servers with XClarity Controller (XCC) and System x servers with Integrated Management Module v2 (IMM2) do not use MegaRAC and are not affected.

An AMI representative declined to comment on the vulnerability but added the standard statements about security being an important priority. An Intel representative confirmed the accuracy of the Binarly report. Representatives from Supermicro didn't respond to an email seeking confirmation of the report.

The lighttpd flaw is what’s known as a heap out-of-bounds read vulnerability that's caused by bugs in HTTP request parsing logic. Hackers can exploit it using maliciously designed HTTP requests.

“A potential attacker can exploit this vulnerability in order to read memory of Lighttpd Web Server process,” Binarly researchers wrote in an advisory. “This may lead to sensitive data exfiltration, such as memory addresses, which can be used to bypass security mechanisms such as ASLR.” Advisories are available here , here , and here .

Further Reading

People or organizations using Supermicro gear should check with the manufacturer to find information on possible fixes. With no fixes available from Intel or Lenovo, there’s not much users of these affected hardware can do. It’s worth mentioning explicitly, however, that the severity of the lighttpd vulnerability is only moderate and is of no value unless an attacker has a working exploit for a much more severe vulnerability. In general, BMCs should be enabled only when needed and locked down carefully, as they allow for extraordinary control of entire fleets of servers with simple HTTP requests sent over the Internet.

reader comments

Promoted comments.

A while ago there were allegations that SuperMicro servers were shipped with backdoors. Is this maybe related to those?

Had the same thought. The Big Hack was published in late 2018, in the same timeframe that the security bug fix here was introduced (and ignored). It is not at all clear to me when the BMC bugs were introduced to the code (whether or not it was intentional). China Used a Tiny Chip in a Hack That Infiltrated U.S. Companies The attack by Chinese spies reached almost 30 U.S. companies by compromising America's technology supply chain. www.bloomberg.com They might not be the same thing – or Bloomberg may finally have been exonerated after years of nobody being able to reproduce their findings independently.

Channel Ars Technica