clinical research literature review

Literature Reviews: Types of Clinical Study Designs

• Library Basics
• 1. Choose Your Topic
• How to Find Books
• Types of Clinical Study Designs
• Types of Literature
• 3. Search the Literature
• 4. Read & Analyze the Literature
• 5. Write the Review
• Keeping Track of Information
• Style Guides
• Books, Tutorials & Examples

Types of Study Designs

Meta-Analysis A way of combining data from many different research studies. A meta-analysis is a statistical process that combines the findings from individual studies.  Example :  Anxiety outcomes after physical activity interventions: meta-analysis findings .  Conn V.  Nurs Res . 2010 May-Jun;59(3):224-31.

Systematic Review A summary of the clinical literature. A systematic review is a critical assessment and evaluation of all research studies that address a particular clinical issue. The researchers use an organized method of locating, assembling, and evaluating a body of literature on a particular topic using a set of specific criteria. A systematic review typically includes a description of the findings of the collection of research studies. The systematic review may also include a quantitative pooling of data, called a meta-analysis.  Example :  Complementary and alternative medicine use among women with breast cancer: a systematic review.   Wanchai A, Armer JM, Stewart BR. Clin J Oncol Nurs . 2010 Aug;14(4):E45-55.

Randomized Controlled Trial A controlled clinical trial that randomly (by chance) assigns participants to two or more groups. There are various methods to randomize study participants to their groups.  Example :  Meditation or exercise for preventing acute respiratory infection: a randomized controlled trial .  Barrett B, et al.  Ann Fam Med . 2012 Jul-Aug;10(4):337-46.

Cohort Study (Prospective Observational Study) A clinical research study in which people who presently have a certain condition or receive a particular treatment are followed over time and compared with another group of people who are not affected by the condition.  Example : Smokeless tobacco cessation in South Asian communities: a multi-centre prospective cohort study . Croucher R, et al. Addiction. 2012 Dec;107 Suppl 2:45-52.

Case-control Study Case-control studies begin with the outcomes and do not follow people over time. Researchers choose people with a particular result (the cases) and interview the groups or check their records to ascertain what different experiences they had. They compare the odds of having an experience with the outcome to the odds of having an experience without the outcome.  Example :  Non-use of bicycle helmets and risk of fatal head injury: a proportional mortality, case-control study .  Persaud N, et al.  CMAJ . 2012 Nov 20;184(17):E921-3.

Cross-sectional study The observation of a defined population at a single point in time or time interval. Exposure and outcome are determined simultaneously.  Example :  Fasting might not be necessary before lipid screening: a nationally representative cross-sectional study .  Steiner MJ, et al.  Pediatrics . 2011 Sep;128(3):463-70.

Case Reports and Series A report on a series of patients with an outcome of interest. No control group is involved.  Example :  Students mentoring students in a service-learning clinical supervision experience: an educational case report .  Lattanzi JB, et al.  Phys Ther . 2011 Oct;91(10):1513-24.

Ideas, Editorials, Opinions Put forth by experts in the field.  Example : Health and health care for the 21st century: for all the people . Koop CE.  Am J Public Health . 2006 Dec;96(12):2090-2.

Animal Research Studies Studies conducted using animal subjects.  Example : Intranasal leptin reduces appetite and induces weight loss in rats with diet-induced obesity (DIO) .  Schulz C, Paulus K, Jöhren O, Lehnert H.   Endocrinology . 2012 Jan;153(1):143-53.

Test-tube Lab Research "Test tube" experiments conducted in a controlled laboratory setting.

Adapted from Study Designs. In NICHSR Introduction to Health Services Research: a Self-Study Course.  http://www.nlm.nih.gov/nichsr/ihcm/06studies/studies03.html and Glossary of EBM Terms. http://www.cebm.utoronto.ca/glossary/index.htm#top  

Study Design Terminology

Bias - Any deviation of results or inferences from the truth, or processes leading to such deviation. Bias can result from several sources: one-sided or systematic variations in measurement from the true value (systematic error); flaws in study design; deviation of inferences, interpretations, or analyses based on flawed data or data collection; etc. There is no sense of prejudice or subjectivity implied in the assessment of bias under these conditions.

Case Control Studies - Studies which start with the identification of persons with a disease of interest and a control (comparison, referent) group without the disease. The relationship of an attribute to the disease is examined by comparing diseased and non-diseased persons with regard to the frequency or levels of the attribute in each group.

Causality - The relating of causes to the effects they produce. Causes are termed necessary when they must always precede an effect and sufficient when they initiate or produce an effect. Any of several factors may be associated with the potential disease causation or outcome, including predisposing factors, enabling factors, precipitating factors, reinforcing factors, and risk factors.

Control Groups - Groups that serve as a standard for comparison in experimental studies. They are similar in relevant characteristics to the experimental group but do not receive the experimental intervention.

Controlled Clinical Trials - Clinical trials involving one or more test treatments, at least one control treatment, specified outcome measures for evaluating the studied intervention, and a bias-free method for assigning patients to the test treatment. The treatment may be drugs, devices, or procedures studied for diagnostic, therapeutic, or prophylactic effectiveness. Control measures include placebos, active medicines, no-treatment, dosage forms and regimens, historical comparisons, etc. When randomization using mathematical techniques, such as the use of a random numbers table, is employed to assign patients to test or control treatments, the trials are characterized as Randomized Controlled Trials.

Cost-Benefit Analysis - A method of comparing the cost of a program with its expected benefits in dollars (or other currency). The benefit-to-cost ratio is a measure of total return expected per unit of money spent. This analysis generally excludes consideration of factors that are not measured ultimately in economic terms. Cost effectiveness compares alternative ways to achieve a specific set of results.

Cross-Over Studies - Studies comparing two or more treatments or interventions in which the subjects or patients, upon completion of the course of one treatment, are switched to another. In the case of two treatments, A and B, half the subjects are randomly allocated to receive these in the order A, B and half to receive them in the order B, A. A criticism of this design is that effects of the first treatment may carry over into the period when the second is given.

Cross-Sectional Studies - Studies in which the presence or absence of disease or other health-related variables are determined in each member of the study population or in a representative sample at one particular time. This contrasts with LONGITUDINAL STUDIES which are followed over a period of time.

Double-Blind Method - A method of studying a drug or procedure in which both the subjects and investigators are kept unaware of who is actually getting which specific treatment.

Empirical Research - The study, based on direct observation, use of statistical records, interviews, or experimental methods, of actual practices or the actual impact of practices or policies.

Evaluation Studies - Works consisting of studies determining the effectiveness or utility of processes, personnel, and equipment.

Genome-Wide Association Study - An analysis comparing the allele frequencies of all available (or a whole genome representative set of) polymorphic markers in unrelated patients with a specific symptom or disease condition, and those of healthy controls to identify markers associated with a specific disease or condition.

Intention to Treat Analysis - Strategy for the analysis of Randomized Controlled Trial that compares patients in the groups to which they were originally randomly assigned.

Logistic Models - Statistical models which describe the relationship between a qualitative dependent variable (that is, one which can take only certain discrete values, such as the presence or absence of a disease) and an independent variable. A common application is in epidemiology for estimating an individual's risk (probability of a disease) as a function of a given risk factor.

Longitudinal Studies - Studies in which variables relating to an individual or group of individuals are assessed over a period of time.

Lost to Follow-Up - Study subjects in cohort studies whose outcomes are unknown e.g., because they could not or did not wish to attend follow-up visits.

Matched-Pair Analysis - A type of analysis in which subjects in a study group and a comparison group are made comparable with respect to extraneous factors by individually pairing study subjects with the comparison group subjects (e.g., age-matched controls).

Meta-Analysis - Works consisting of studies using a quantitative method of combining the results of independent studies (usually drawn from the published literature) and synthesizing summaries and conclusions which may be used to evaluate therapeutic effectiveness, plan new studies, etc. It is often an overview of clinical trials. It is usually called a meta-analysis by the author or sponsoring body and should be differentiated from reviews of literature.

Numbers Needed To Treat - Number of patients who need to be treated in order to prevent one additional bad outcome. It is the inverse of Absolute Risk Reduction.

Odds Ratio - The ratio of two odds. The exposure-odds ratio for case control data is the ratio of the odds in favor of exposure among cases to the odds in favor of exposure among noncases. The disease-odds ratio for a cohort or cross section is the ratio of the odds in favor of disease among the exposed to the odds in favor of disease among the unexposed. The prevalence-odds ratio refers to an odds ratio derived cross-sectionally from studies of prevalent cases.

Patient Selection - Criteria and standards used for the determination of the appropriateness of the inclusion of patients with specific conditions in proposed treatment plans and the criteria used for the inclusion of subjects in various clinical trials and other research protocols.

Predictive Value of Tests - In screening and diagnostic tests, the probability that a person with a positive test is a true positive (i.e., has the disease), is referred to as the predictive value of a positive test; whereas, the predictive value of a negative test is the probability that the person with a negative test does not have the disease. Predictive value is related to the sensitivity and specificity of the test.

Prospective Studies - Observation of a population for a sufficient number of persons over a sufficient number of years to generate incidence or mortality rates subsequent to the selection of the study group.

Qualitative Studies - Research that derives data from observation, interviews, or verbal interactions and focuses on the meanings and interpretations of the participants.

Quantitative Studies - Quantitative research is research that uses numerical analysis.

Random Allocation - A process involving chance used in therapeutic trials or other research endeavor for allocating experimental subjects, human or animal, between treatment and control groups, or among treatment groups. It may also apply to experiments on inanimate objects.

Randomized Controlled Trial - Clinical trials that involve at least one test treatment and one control treatment, concurrent enrollment and follow-up of the test- and control-treated groups, and in which the treatments to be administered are selected by a random process, such as the use of a random-numbers table.

Reproducibility of Results - The statistical reproducibility of measurements (often in a clinical context), including the testing of instrumentation or techniques to obtain reproducible results. The concept includes reproducibility of physiological measurements, which may be used to develop rules to assess probability or prognosis, or response to a stimulus; reproducibility of occurrence of a condition; and reproducibility of experimental results.

Retrospective Studies - Studies used to test etiologic hypotheses in which inferences about an exposure to putative causal factors are derived from data relating to characteristics of persons under study or to events or experiences in their past. The essential feature is that some of the persons under study have the disease or outcome of interest and their characteristics are compared with those of unaffected persons.

Sample Size - The number of units (persons, animals, patients, specified circumstances, etc.) in a population to be studied. The sample size should be big enough to have a high likelihood of detecting a true difference between two groups.

Sensitivity and Specificity - Binary classification measures to assess test results. Sensitivity or recall rate is the proportion of true positives. Specificity is the probability of correctly determining the absence of a condition.

Single-Blind Method - A method in which either the observer(s) or the subject(s) is kept ignorant of the group to which the subjects are assigned.

Time Factors - Elements of limited time intervals, contributing to particular results or situations.

Source:  NLM MeSH Database

• << Previous: How to Find Books
• Next: Types of Literature >>
• Last Updated: Dec 29, 2023 11:41 AM
• URL: https://research.library.gsu.edu/litrev

• Search Menu
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Papyrology
• Greek and Roman Archaeology
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Acquisition
• Language Evolution
• Language Reference
• Language Variation
• Language Families
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Modernism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Religion
• Music and Media
• Music and Culture
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Science
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Lifestyle, Home, and Garden
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Clinical Neuroscience
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Medical Ethics
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Strategy
• Business Ethics
• Business History
• Business and Government
• Business and Technology
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic Systems
• Economic History
• Economic Methodology
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Behaviour
• Political Economy
• Political Institutions
• Political Theory
• Politics and Law
• Public Administration
• Public Policy
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Developmental and Physical Disabilities Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous
• Next chapter >

1Chapter 1 The literature review

• Published: November 2011
• Cite Icon Cite
• Permissions Icon Permissions

A major part of any study, the literature review has a variety of influences over a project which may not be immediately apparent to those new to the research field. In essence, pulling a number of citations out of a search engine to request from the library is only one small element of this important tool in addressing the research question. Before discussing the practicalities of the literature review we need to establish what it is, why it is necessary to do one in the first place, and how such a review fits into both the research process and the finished dissertation. Before being able to establish a research question or decide on a design to address a clinical problem, it is necessary to have an awareness of published material pertaining to that subject. In broad terms, what do we already know? Through an appraisal of the literature, one can identify if further investigation is merited, avoid duplication, and benefit from the experiences of those exploring similar issues.… It is generally unwise to define something as important as a dissertation topic without first obtaining a broad familiarity with the field. (Rudestam and Newton, 1992, p.9) (1)… A comprehensive review of the literature will ensure you have a grasp of up-to-date knowledge, which is key to putting any research project into context. Whilst in-depth critical appraisal of the relevant literature may not be necessary at this point, one can see that a literature review is required to turn an interesting idea into a research question. Having established its role in the conception of a research question, once the project is underway the review serves a variety of purposes. No research is unique in all its aspects—at least some of the features addressed in the project will have been previously described and may be used to justify why further investigation is required. To this end each individual research project forms one part of a bigger ‘evidence-based jigsaw puzzle’. Without an appreciation of all the other bits of the puzzle it is impossible to provide a background and identify a gap in the knowledge base which we wish to address.

Signed in as

Institutional accounts.

• GoogleCrawler [DO NOT DELETE]
• Google Scholar Indexing

Personal account

• Sign in with email/username & password
• Get email alerts
• Save searches
• Purchase content
• Activate your purchase/trial code
• Add your ORCID iD

Institutional access

Sign in with a library card.

• Sign in with username/password
• Recommend to your librarian
• Institutional account management
• Get help with access

Access to content on Oxford Academic is often provided through institutional subscriptions and purchases. If you are a member of an institution with an active account, you may be able to access content in one of the following ways:

IP based access

Typically, access is provided across an institutional network to a range of IP addresses. This authentication occurs automatically, and it is not possible to sign out of an IP authenticated account.

Sign in through your institution

Choose this option to get remote access when outside your institution. Shibboleth/Open Athens technology is used to provide single sign-on between your institution’s website and Oxford Academic.

• Click Sign in through your institution.
• Select your institution from the list provided, which will take you to your institution's website to sign in.
• When on the institution site, please use the credentials provided by your institution. Do not use an Oxford Academic personal account.
• Following successful sign in, you will be returned to Oxford Academic.

If your institution is not listed or you cannot sign in to your institution’s website, please contact your librarian or administrator.

Enter your library card number to sign in. If you cannot sign in, please contact your librarian.

Society Members

Society member access to a journal is achieved in one of the following ways:

Sign in through society site

Many societies offer single sign-on between the society website and Oxford Academic. If you see ‘Sign in through society site’ in the sign in pane within a journal:

• Click Sign in through society site.
• When on the society site, please use the credentials provided by that society. Do not use an Oxford Academic personal account.

If you do not have a society account or have forgotten your username or password, please contact your society.

Sign in using a personal account

Some societies use Oxford Academic personal accounts to provide access to their members. See below.

A personal account can be used to get email alerts, save searches, purchase content, and activate subscriptions.

Some societies use Oxford Academic personal accounts to provide access to their members.

Viewing your signed in accounts

Click the account icon in the top right to:

• View your signed in personal account and access account management features.
• View the institutional accounts that are providing access.

Signed in but can't access content

Oxford Academic is home to a wide variety of products. The institutional subscription may not cover the content that you are trying to access. If you believe you should have access to that content, please contact your librarian.

For librarians and administrators, your personal account also provides access to institutional account management. Here you will find options to view and activate subscriptions, manage institutional settings and access options, access usage statistics, and more.

Our books are available by subscription or purchase to libraries and institutions.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Rights and permissions
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Health (Nursing, Medicine, Allied Health)

• Find Articles/Databases
• Reference Resources
• Evidence Summaries & Clinical Guidelines
• Drug Information
• Health Data & Statistics
• Patient/Consumer Facing Materials
• Images and Streaming Video
• Grey Literature
• Mobile Apps & "Point of Care" Tools
• Tests & Measures This link opens in a new window
• Citing Sources
• Selecting Databases
• Framing Research Questions
• Crafting a Search
• Narrowing / Filtering a Search
• Expanding a Search
• Cited Reference Searching
• Saving Searches
• Term Glossary
• Critical Appraisal Resources
• What are Literature Reviews?
• Conducting & Reporting Systematic Reviews
• Finding Systematic Reviews
• Tutorials & Tools for Literature Reviews
• Finding Full Text

What are Systematic Reviews? (3 minutes, 24 second YouTube Video)

Systematic Literature Reviews: Steps & Resources

These steps for conducting a systematic literature review are listed below . 

Also see subpages for more information about:

• The different types of literature reviews, including systematic reviews and other evidence synthesis methods
• Tools & Tutorials

Literature Review & Systematic Review Steps

• Develop a Focused Question
• Scope the Literature  (Initial Search)
• Refine & Expand the Search
• Limit the Results
• Download Citations
• Abstract & Analyze
• Create Flow Diagram
• Synthesize & Report Results

1. Develop a Focused   Question 

Consider the PICO Format: Population/Problem, Intervention, Comparison, Outcome

Focus on defining the Population or Problem and Intervention (don't narrow by Comparison or Outcome just yet!)

"What are the effects of the Pilates method for patients with low back pain?"

Tools & Additional Resources:

• PICO Question Help
• Stillwell, Susan B., DNP, RN, CNE; Fineout-Overholt, Ellen, PhD, RN, FNAP, FAAN; Melnyk, Bernadette Mazurek, PhD, RN, CPNP/PMHNP, FNAP, FAAN; Williamson, Kathleen M., PhD, RN Evidence-Based Practice, Step by Step: Asking the Clinical Question, AJN The American Journal of Nursing : March 2010 - Volume 110 - Issue 3 - p 58-61 doi: 10.1097/01.NAJ.0000368959.11129.79

2. Scope the Literature

A "scoping search" investigates the breadth and/or depth of the initial question or may identify a gap in the literature. 

Eligible studies may be located by searching in:

• Background sources (books, point-of-care tools)
• Article databases
• Trial registries
• Grey literature
• Cited references
• Reference lists

When searching, if possible, translate terms to controlled vocabulary of the database. Use text word searching when necessary.

Use Boolean operators to connect search terms:

• Combine separate concepts with AND  (resulting in a narrower search)
• Connecting synonyms with OR  (resulting in an expanded search)

Search:  pilates AND ("low back pain"  OR  backache )

Video Tutorials - Translating PICO Questions into Search Queries

• Translate Your PICO Into a Search in PubMed (YouTube, Carrie Price, 5:11) 
• Translate Your PICO Into a Search in CINAHL (YouTube, Carrie Price, 4:56)

3. Refine & Expand Your Search

Expand your search strategy with synonymous search terms harvested from:

• database thesauri
• reference lists
• relevant studies

Example: 

(pilates OR exercise movement techniques) AND ("low back pain" OR backache* OR sciatica OR lumbago OR spondylosis)

As you develop a final, reproducible strategy for each database, save your strategies in a:

• a personal database account (e.g., MyNCBI for PubMed)
• Log in with your NYU credentials
• Open and "Make a Copy" to create your own tracker for your literature search strategies

4. Limit Your Results

Use database filters to limit your results based on your defined inclusion/exclusion criteria.  In addition to relying on the databases' categorical filters, you may also need to manually screen results.  

• Limit to Article type, e.g.,:  "randomized controlled trial" OR multicenter study
• Limit by publication years, age groups, language, etc.

NOTE: Many databases allow you to filter to "Full Text Only".  This filter is  not recommended . It excludes articles if their full text is not available in that particular database (CINAHL, PubMed, etc), but if the article is relevant, it is important that you are able to read its title and abstract, regardless of 'full text' status. The full text is likely to be accessible through another source (a different database, or Interlibrary Loan).  

• Filters in PubMed
• CINAHL Advanced Searching Tutorial

5. Download Citations

Selected citations and/or entire sets of search results can be downloaded from the database into a citation management tool. If you are conducting a systematic review that will require reporting according to PRISMA standards, a citation manager can help you keep track of the number of articles that came from each database, as well as the number of duplicate records.

In Zotero, you can create a Collection for the combined results set, and sub-collections for the results from each database you search.  You can then use Zotero's 'Duplicate Items" function to find and merge duplicate records.

• Citation Managers - General Guide

6. Abstract and Analyze

• Migrate citations to data collection/extraction tool
• Screen Title/Abstracts for inclusion/exclusion
• Screen and appraise full text for relevance, methods, 
• Resolve disagreements by consensus

Covidence is a web-based tool that enables you to work with a team to screen titles/abstracts and full text for inclusion in your review, as well as extract data from the included studies.

• Covidence Support
• Critical Appraisal Tools
• Data Extraction Tools

7. Create Flow Diagram

The PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) flow diagram is a visual representation of the flow of records through different phases of a systematic review.  It depicts the number of records identified, included and excluded.  It is best used in conjunction with the PRISMA checklist .

Example from: Stotz, S. A., McNealy, K., Begay, R. L., DeSanto, K., Manson, S. M., & Moore, K. R. (2021). Multi-level diabetes prevention and treatment interventions for Native people in the USA and Canada: A scoping review. Current Diabetes Reports, 2 (11), 46. https://doi.org/10.1007/s11892-021-01414-3

• PRISMA Flow Diagram Generator (ShinyApp.io, Haddaway et al. )
• PRISMA Diagram Templates  (Word and PDF)
• Make a copy of the file to fill out the template
• Image can be downloaded as PDF, PNG, JPG, or SVG
• Covidence generates a PRISMA diagram that is automatically updated as records move through the review phases

8. Synthesize & Report Results

There are a number of reporting guideline available to guide the synthesis and reporting of results in systematic literature reviews.

It is common to organize findings in a matrix, also known as a Table of Evidence (ToE).

• Reporting Guidelines for Systematic Reviews
• Download a sample template of a health sciences review matrix  (GoogleSheets)

Steps modified from: 

Cook, D. A., & West, C. P. (2012). Conducting systematic reviews in medical education: a stepwise approach.   Medical Education , 46 (10), 943–952.

• << Previous: Critical Appraisal Resources
• Next: What are Literature Reviews? >>
• Last Updated: May 16, 2024 9:53 AM
• URL: https://guides.nyu.edu/health

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Performing a...

Performing a literature review

• Related content
• Peer review
• Gulraj S Matharu , academic foundation doctor ,
• Christopher D Buckley , Arthritis Research UK professor of rheumatology
• 1 Institute of Biomedical Research, College of Medical and Dental Sciences, School of Immunity and Infection, University of Birmingham, UK

A necessary skill for any doctor

What causes disease, which drug is best, does this patient need surgery, and what is the prognosis? Although experience helps in answering these questions, ultimately they are best answered by evidence based medicine. But how do you assess the evidence? As a medical student, and throughout your career as a doctor, critical appraisal of published literature is an important skill to develop and refine. At medical school you will repeatedly appraise published literature and write literature reviews. These activities are commonly part of a special study module, research project for an intercalated degree, or another type of essay based assignment.

Formulating a question

Literature reviews are most commonly performed to help answer a particular question. While you are at medical school, there will usually be some choice regarding the area you are going to review.

Once you have identified a subject area for review, the next step is to formulate a specific research question. This is arguably the most important step because a clear question needs to be defined from the outset, which you aim to answer by doing the review. The clearer the question, the more likely it is that the answer will be clear too. It is important to have discussions with your supervisor when formulating a research question as his or her input will be invaluable. The research question must be objective and concise because it is easier to search through the evidence with a clear question. The question also needs to be feasible. What is the point in having a question for which no published evidence exists? Your supervisor’s input will ensure you are not trying to answer an unrealistic question. Finally, is the research question clinically important? There are many research questions that may be answered, but not all of them will be relevant to clinical practice. The research question we will use as an example to work through in this article is, “What is the evidence for using angiotensin converting enzyme (ACE) inhibitors in patients with hypertension?”

Collecting the evidence

After formulating a specific research question for your literature review, the next step is to collect the evidence. Your supervisor will initially point you in the right direction by highlighting some of the more relevant papers published. Before doing the literature search it is important to agree a list of keywords with your supervisor. A source of useful keywords can be obtained by reading Cochrane reviews or other systematic reviews, such as those published in the BMJ . 1 2 A relevant Cochrane review for our research question on ACE inhibitors in hypertension is that by Heran and colleagues. 3 Appropriate keywords to search for the evidence include the words used in your research question (“angiotensin converting enzyme inhibitor,” “hypertension,” “blood pressure”), details of the types of study you are looking for (“randomised controlled trial,” “case control,” “cohort”), and the specific drugs you are interested in (that is, the various ACE inhibitors such as “ramipril,” “perindopril,” and “lisinopril”).

Once keywords have been agreed it is time to search for the evidence using the various electronic medical databases (such as PubMed, Medline, and EMBASE). PubMed is the largest of these databases and contains online information and tutorials on how to do literature searches with worked examples. Searching the databases and obtaining the articles are usually free of charge through the subscription that your university pays. Early consultation with a medical librarian is important as it will help you perform your literature search in an impartial manner, and librarians can train you to do these searches for yourself.

Literature searches can be broad or tailored to be more specific. With our example, a broad search would entail searching all articles that contain the words “blood pressure” or “ACE inhibitor.” This provides a comprehensive list of all the literature, but there are likely to be thousands of articles to review subsequently (fig 1). ⇓ In contrast, various search restrictions can be applied on the electronic databases to filter out papers that may not be relevant to your review. Figure 2 gives an example of a specific search. ⇓ The search terms used in this case were “angiotensin converting enzyme inhibitor” and “hypertension.” The limits applied to this search were all randomised controlled trials carried out in humans, published in the English language over the last 10 years, with the search terms appearing in the title of the study only. Thus the more specific the search strategy, the more manageable the number of articles to review (fig 3), and this will save you time. ⇓ However, this method risks your not identifying all the evidence in the particular field. Striking a balance between a broad and a specific search strategy is therefore important. This will come with experience and consultation with your supervisor. It is important to note that evidence is continually becoming available on these electronic databases and therefore repeating the same search at a later date can provide new evidence relevant to your review.

Fig 1 Results from a broad literature search using the term “angiotensin converting enzyme inhibitor”

• Download figure
• Open in new tab
• Download powerpoint

Fig 2 Example of a specific literature search. The search terms used were “angiotensin converting enzyme inhibitor” and “hypertension.” The limits applied to this search were all randomised controlled trials carried out in humans, published in English over the past 10 years, with the search terms appearing in the title of the study only

Fig 3 Results from a specific literature search (using the search terms and limits from figure 2)

Reading the abstracts (study summary) of the articles identified in your search may help you decide whether the study is applicable for your review—for example, the work may have been carried out using an animal model rather than in humans. After excluding any inappropriate articles, you need to obtain the full articles of studies you have identified. Additional relevant articles that may not have come up in your original search can also be found by searching the reference lists of the articles you have already obtained. Once again, you may find that some articles are still not applicable for your review, and these can also be excluded at this stage. It is important to explain in your final review what criteria you used to exclude articles as well as those criteria used for inclusion.

The National Institute for Health and Clinical Excellence (NICE) publishes evidence based guidelines for the United Kingdom and therefore provides an additional resource for identifying the relevant literature in a particular field. 4 NICE critically appraises the published literature with recommendations for best clinical practice proposed and graded based on the quality of evidence available. Similarly, there are internationally published evidence based guidelines, such as those produced by the European Society of Cardiology and the American College of Chest Physicians, which can be useful when collecting the literature in a particular field. 5 6

Appraising the evidence

Once you have collected the evidence, you need to critically appraise the published material. Box 1 gives definitions of terms you will encounter when reading the literature. A brief guide of how to critically appraise a study is presented; however, it is advisable to consult the references cited for further details.

Box 1: Definitions of common terms in the literature 7

Prospective—collecting data in real time after the study is designed

Retrospective—analysis of data that have already been collected to determine associations between exposure and outcome

Hypothesis—proposed association between exposure and outcome. If presented in the negative it is called the null hypothesis

Variable—a quantity or quality that changes during the study and can be measured

Single blind—subjects are unaware of their treatment, but clinicians are aware

Double blind—both subjects and clinicians are unaware of treatment given

Placebo—a simulated medical intervention, with subjects not receiving the specific intervention or treatment being studied

Outcome measure/endpoint—clinical variable or variables measured in a study subsequently used to make conclusions about the original interventions or treatments administered

Bias—difference between reported results and true results. Many types exist (such as selection, allocation, and reporting biases)

Probability (P) value—number between 0 and 1 providing the likelihood the reported results occurred by chance. A P value of 0.05 means there is a 5% likelihood that the reported result occurred by chance

Confidence intervals—provides a range between two numbers within which one can be certain the results lie. A confidence interval of 95% means one can be 95% certain the actual results lie within the reported range

The study authors should clearly define their research question and ideally the hypothesis to be tested. If the hypothesis is presented in the negative, it is called the null hypothesis. An example of a null hypothesis is smoking does not cause lung cancer. The study is then performed to assess the significance of the exposure (smoking) on outcome (lung cancer).

A major part of the critical appraisal process is to focus on study methodology, with your key task being an assessment of the extent to which a study was susceptible to bias (the discrepancy between the reported results and the true results). It should be clear from the methods what type of study was performed (box 2).

Box 2: Different study types 7

Systematic review/meta-analysis—comprehensive review of published literature using predefined methodology. Meta-analyses combine results from various studies to give numerical data for the overall association between variables

Randomised controlled trial—random allocation of patients to one of two or more groups. Used to test a new drug or procedure

Cohort study—two or more groups followed up over a long period, with one group exposed to a certain agent (drug or environmental agent) and the other not exposed, with various outcomes compared. An example would be following up a group of smokers and a group of non-smokers with the outcome measure being the development of lung cancer

Case-control study—cases (those with a particular outcome) are matched as closely as possible (for age, sex, ethnicity) with controls (those without the particular outcome). Retrospective data analysis is performed to determine any factors associated with developing the particular outcomes

Cross sectional study—looks at a specific group of patients at a single point in time. Effectively a survey. An example is asking a group of people how many of them drink alcohol

Case report—detailed reports concerning single patients. Useful in highlighting adverse drug reactions

There are many different types of bias, which depend on the particular type of study performed, and it is important to look for these biases. Several published checklists are available that provide excellent resources to help you work through the various studies and identify sources of bias. The CONSORT statement (which stands for CONsolidated Standards Of Reporting Trials) provides a minimum set of recommendations for reporting randomised controlled trials and comprises a rigorous 25 item checklist, with variations available for other study types. 8 9 As would be expected, most (17 of 25) of the items focus on questions relating to the methods and results of the randomised trial. The remaining items relate to the title, abstract, introduction, and discussion of the study, in addition to questions on trial registration, protocol, and funding.

Jadad scoring provides a simple and validated system to assess the methodological quality of a randomised clinical trial using three questions. 10 The score ranges from zero to five, with one point given for a “yes” in each of the following questions. (1) Was the study described as randomised? (2) Was the study described as double blind? (3) Were there details of subject withdrawals, exclusions, and dropouts? A further point is given if (1) the method of randomisation was appropriate, and (2) the method of blinding was appropriate.

In addition, the Critical Appraisal Skills Programme provides excellent tools for assessing the evidence in all study types (box 2). 11 The Oxford Centre for Evidence-Based Medicine levels of evidence is yet another useful resource for assessing the methodological quality of all studies. 12

Ensure all patients have been accounted for and any exclusions, for whatever reason, are reported. Knowing the baseline demographic (age, sex, ethnicity) and clinical characteristics of the population is important. Results are usually reported as probability values or confidence intervals (box 1).

This should explain the major study findings, put the results in the context of the published literature, and attempt to account for any variations from previous work. Study limitations and sources of bias should be discussed. Authors’ conclusions should be supported by the study results and not unnecessarily extrapolated. For example, a treatment shown to be effective in animals does not necessarily mean it will work in humans.

The format for writing up the literature review usually consists of an abstract (short structured summary of the review), the introduction or background, methods, results, and discussion with conclusions. There are a number of good examples of how to structure a literature review and these can be used as an outline when writing your review. 13 14

The introduction should identify the specific research question you intend to address and briefly put this into the context of the published literature. As you have now probably realised, the methods used for the review must be clear to the reader and provide the necessary detail for someone to be able to reproduce the search. The search strategy needs to include a list of keywords used, which databases were searched, and the specific search limits or filters applied. Any grading of methodological quality, such as the CONSORT statement or Jadad scoring, must be explained in addition to any study inclusion or exclusion criteria. 6 7 8 The methods also need to include a section on the data collected from each of the studies, the specific outcomes of interest, and any statistical analysis used. The latter point is usually relevant only when performing meta-analyses.

The results section must clearly show the process of filtering down from the articles obtained from the original search to the final studies included in the review—that is, accounting for all excluded studies. A flowchart is usually best to illustrate this. Next should follow a brief description of what was done in the main studies, the number of participants, the relevant results, and any potential sources of bias. It is useful to group similar studies together as it allows comparisons to be made by the reader and saves repetition in your write-up. Boxes and figures should be used appropriately to illustrate important findings from the various studies.

Finally, in the discussion you need to consider the study findings in light of the methodological quality—that is, the extent of potential bias in each study that may have affected the study results. Using the evidence, you need to make conclusions in your review, and highlight any important gaps in the evidence base, which need to be dealt with in future studies. Working through drafts of the literature review with your supervisor will help refine your critical appraisal skills and the ability to present information concisely in a structured review article. Remember, if the work is good it may get published.

Originally published as: Student BMJ 2012;20:e404

Competing interests: None declared.

Provenance and peer review: Not commissioned; externally peer reviewed.

• ↵ The Cochrane Library. www3.interscience.wiley.com/cgibin/mrwhome/106568753/HOME?CRETRY=1&SRETRY=0 .
• ↵ British Medical Journal . www.bmj.com/ .
• ↵ Heran BS, Wong MMY, Heran IK, Wright JM. Blood pressure lowering efficacy of angiotensin converting enzyme (ACE) inhibitors for primary hypertension. Cochrane Database Syst Rev 2008 ; 4 : CD003823 , doi: 10.1002/14651858.CD003823.pub2. OpenUrl PubMed
• ↵ National Institute for Health and Clinical Excellence. www.nice.org.uk .
• ↵ European Society of Cardiology. www.escardio.org/guidelines .
• ↵ Geerts WH, Bergqvist D, Pineo GF, Heit JA, Samama CM, Lassen MR, et al. Prevention of venous thromboembolism: American College of Chest Physicians evidence-based clinical practice guidelines (8th ed). Chest 2008 ; 133 : 381 -453S. OpenUrl CrossRef
• ↵ Wikipedia. http://en.wikipedia.org/wiki .
• ↵ Moher D, Schulz KF, Altman DG, Egger M, Davidoff F, Elbourne D, et al. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001 ; 357 : 1191 -4. OpenUrl CrossRef PubMed Web of Science
• ↵ The CONSORT statement. www.consort-statement.org/ .
• ↵ Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996 ; 17 : 1 -12. OpenUrl CrossRef PubMed Web of Science
• ↵ Critical Appraisal Skills Programme (CASP). www.sph.nhs.uk/what-we-do/public-health-workforce/resources/critical-appraisals-skills-programme .
• ↵ Oxford Centre for Evidence-based Medicine—Levels of Evidence. www.cebm.net .
• ↵ Van den Bruel A, Thompson MJ, Haj-Hassan T, Stevens R, Moll H, Lakhanpaul M, et al . Diagnostic value of laboratory tests in identifying serious infections in febrile children: systematic review. BMJ 2011 ; 342 : d3082 . OpenUrl Abstract / FREE Full Text
• ↵ Awopetu AI, Moxey P, Hinchliffe RJ, Jones KG, Thompson MM, Holt PJ. Systematic review and meta-analysis of the relationship between hospital volume and outcome for lower limb arterial surgery. Br J Surg 2010 ; 97 : 797 -803. OpenUrl CrossRef PubMed

The “PP-ICONS” approach will help you separate the clinical wheat from the chaff in mere minutes .

ROBERT J. FLAHERTY, MD

Fam Pract Manag. 2004;11(5):47-52

Keeping up with the latest advances in diagnosis and treatment is a challenge we all face as phycians. We need information that is both valid (that is, accurate and correct) and relevant to our patients and practices. While we have many sources of clinical information, such as CME lectures, textbooks, pharmaceutical advertising, pharmaceutical representatives and colleagues, we often turn to journal articles for the most current clinical information.

Unfortunately, a great deal of research reported in journal articles is poorly done, poorly analyzed or both, and thus is not valid. A great deal of research is also irrelevant to our patients and practices. Separating the clinical wheat from the chaff can take skills that many of us never were taught.

Reading the abstract is often sufficient when evaluating an article using the PP-ICONS approach.

The most relevant studies will involve outcomes that matter to patients (e.g., morbidity, mortality and cost) versus outcomes that matter to physiologists (e.g., blood pressure, blood sugar or cholesterol levels).

Ignore the relative risk reduction, as it overstates research findings and will mislead you.

The article “Making Evidence-Based Medicine Doable in Everyday Practice” in the February 2004 issue of FPM describes several organizations that can help us. These organizations, such as the Cochrane Library, Bandolier and Clinical Evidence, develop clinical questions and then review one or more journal articles to identify the best available evidence that answers the question, with a focus on the quality of the study, the validity of the results and the relevance of the findings to everyday practice. These organizations provide a very valuable service, and the number of important clinical questions that they have studied has grown steadily over the past five years. (See “Four steps to an evidence-based answer.” )

FOUR STEPS TO AN EVIDENCE-BASED ANSWER

When faced with a clinical question, follow these steps to find an evidence-based answer:

Search the Web site of one of the evidence review organizations, such as Cochrane (http://www.cochrane.org/cochrane/revabstr/mainindex.htm), Bandolier ( http://www.jr2.ox.ac.uk/bandolier ) or Clinical Evidence ( http://www.clinicalevidence.com ), described in “Making Evidence-Based Medicine Doable in Everyday Practice,” FPM, February 2004, page 51 . You can also search the TRIP+ Web site ( http://www.tripdatabase.com ), which simultaneously searches the databases of many of the review organizations. If you find a systematic review or meta-analysis by one of these organizations, you can be confident that you’ve found the best evidence available.

If you don’t find the information you need through step 1, search for meta-analyses and systematic reviews using the PubMed Web site (see the tutorial at http://www.nlm.nih.gov/bsd/pubmed_tutorial/m1001.html ). Most of the recent abstracts found on PubMed provide enough information for you to determine the validity and relevance of the findings. If needed, you can get a copy of the full article through your hospital library or the journal’s Web site.

If you cannot find a systematic review or meta-analysis on PubMed, look for a randomized controlled trial (RCT). The RCT is the “gold standard” in medical research. Case reports, cohort studies and other research methods simply are not good enough to use for making patient care decisions.

Once you find the article you need, use the PP-ICONS approach to evaluate its usefulness to your patient.

If you find a systematic review or meta-analysis done by one of these organizations, you can feel confident that you have found the current best evidence. However, these organizations have not asked all of the common clinical questions yet, and you will frequently be faced with finding the pertinent articles and determining for yourself whether they are valuable. This is where the PP-ICONS approach can help.

What is PP-ICONS?

When you find a systematic review, meta-analysis or randomized controlled trial while reading your clinical journals or searching PubMed ( http://www.ncbi.nlm.nih.gov/entrez/query.fcgi ), you need to determine whether it is valid and relevant. There are many different ways to analyze an abstract or journal article, some more rigorous than others. 1 , 2 I have found a simple but effective way to identify a valid or relevant article within a couple of minutes, ensuring that I can use or discard the conclusions with confidence. This approach works well on articles regarding treatment and prevention, and can also be used with articles on diagnosis and screening.

The most important information to look for when reviewing an article can be summarized by the acronym “PP-ICONS,” which stands for the following:

Patient or population,

Intervention,

Comparison,

Number of subjects,

Statistics.

For example, imagine that you just saw a nine-year-old patient in the office with common warts on her hands, an ideal candidate for your usual cryotherapy. Her mother had heard about treating warts with duct tape and wondered if you would recommend this treatment. You promised to call Mom back after you had a chance to investigate this rather odd treatment.

When you get a free moment, you write down your clinical question: “Is duct tape an effective treatment for warts in children?” Writing down your clinical question is useful, as it can help you clarify exactly what you are looking for. Use the PPICO parts of the acronym to help you write your clinical question; this is actually how many researchers develop their research questions.

You search Cochrane and Bandolier without success, so now you search PubMed, which returns an abstract for the following article: “Focht DR 3rd, Spicer C, Fairchok MP. The efficacy of duct tape vs cryotherapy in the treatment of verruca vulgaris (the common wart). Arch Pediatr Adolesc Med . 2002 Oct;156(10):971-974.”

You decide to apply PP-ICONS to this abstract (see "Abstract from PubMed" ) to determine if the information is both valid and relevant.

ABSTRACT FROM PUBMED

Using the PP-ICONS approach, physicians can evaluate the validity and relevance of clinical articles in minutes using only the abstract, such as this one, obtained free online from PubMed, http://www.ncbi. nlm.nih.gov/entrez/query.fcgi. The author uses this abstract to evaluate the use of duct tape to treat common warts.

Problem. The first P in PP-ICONS is for “problem,” which refers to the clinical condition that was studied. From the abstract, it is clear that the researchers studied the same problem you are interested in, which is important since flat warts or genital warts may have responded differently. Obviously, if the problem studied were not sufficiently similar to your clinical problem, the results would not be relevant.

Patient or population. Next, consider the patient or population. Is the study group similar to your patient or practice? Are they primary care patients, for example, or are they patients who have been referred to a tertiary care center? Are they of a similar age and gender? In this case, the researchers studied children and young adults in outpatient clinics, which is similar to your patient population. If the patients in the study are not similar to your patient, for example if they are sicker, older, a different gender or more clinically complicated, the results might not be relevant.

Intervention. The intervention could be a diagnostic test or a treatment. Make sure the intervention is the same as what you are looking for. The patient’s mother was asking about duct tape for warts, so this is a relevant study.

Comparison. The comparison is what the intervention is tested against. It could be a different diagnostic test or another therapy, such as cryotherapy in this wart study. It could even be placebo or no treatment. Make sure the comparison fits your question. You usually use cryotherapy for common warts, so this is a relevant comparison.

Outcome. The outcome is particularly important. Many outcomes are “disease-oriented outcomes,” which are based on “disease-oriented evidence” (DOEs). DOEs usually reflect changes in physiologic parameters, such as blood pressure, blood sugar, cholesterol, etc. We have long assumed that improving the physiologic parameters of a disease will result in a better disease outcome, but that is not necessarily true. For instance, finasteride can improve urinary flow rate in prostatic hypertrophy, but it does not significantly change symptom scores. 3

DOEs look at the kinds of outcomes that physiologists care about. More relevant are outcomes that patients care about, often called “patient-oriented outcomes.” These are based on “patient-oriented evidence that matters” (POEMs) and look at outcomes such as morbidity, mortality and cost. Thus, when looking at a journal article, DOEs are interesting but of questionable relevance, whereas POEMs are very interesting and very relevant. In the study on the previous page, the outcome is complete resolution of the wart, which is something your patient is interested in.

Number. The number of subjects is crucial to whether accurate statistics can be generated from the data. Too few patients in a research study may not be enough to show that a difference actually exists between the intervention and comparison groups (known as the “power” of a study). Many studies are published with less than 100 subjects, which is usually inadequate to provide reliable statistics. A good rule of thumb is 400 subjects. 4 Fifty-one patients completed the wart study, which is a pretty small number to generate good statistics.

Statistics. The statistics you are interested in are few in number and easy to understand. Since statistics are frequently misused in journal articles, it is worth a few minutes to learn which to believe and which to ignore.

Relative risk reduction. It is not unusual to find a summary statement in a journal article similar to this one from an article titled “Long-Term Effects of Mammography Screening: Updated Overview of the Swedish Randomised Trials”: 5

“There were 511 breast cancer deaths in 1,864,770 women-years in the invited groups and 584 breast cancer deaths in 1,688,440 women-years in the control groups, a significant 21 percent reduction in breast cancer mortality.”

This 21-percent statistic is the relative risk reduction (RRR), which is the percent reduction in the measured outcome between the experimental and control groups. (See “Some important statistics” for more information on calculating the RRR and other statistics.) The RRR is not a good way to compare outcomes. It amplifies small differences and makes insignificant findings appear significant, and it doesn’t reflect the baseline risk of the outcome event. Nevertheless, the RRR is very popular and will be reported in nearly every journal article, perhaps because it makes weak results look good. Think of the RRR as the “reputation reviving ratio” or the “reporter’s reason for ‘riting.” Ignore the RRR. It will mislead you. In our wart treatment example, the RRR would be (85 percent - 60 percent)/60 percent x 100 = 42 percent. The RRR could thus be interpreted as showing that duct tape is 42 percent more effective than cryotherapy in treating warts.

SOME IMPORTANT STATISTICS

Absolute risk reduction (ARR): The difference between the control group’s event rate (CER) and the experimental group’s event rate (EER).

Control event rate (CER): The proportion of patients responding to placebo or other control treatment. For example, if 25 patients are in a control group and the event being studied is observed in 15 of those patients, the control event rate would be 15/25 = 0.60.

Experimental event rate (EER): The proportion of patients responding to the experimental treatment or intervention. For example, if 26 patients are in an experimental group and the event being studied is observed in 22 of those patients, the experimental event rate would be 22/26 = 0.85.

Number needed to treat (NNT): The number of patients that must be treated to prevent one adverse outcome or for one patient to benefit. The NNT is the inverse of the ARR; NNT = 1/ARR.

Relative risk reduction (RRR): The percent reduction in events in the treated group compared to the control group event rate.

Absolute risk reduction. A better statistic is the absolute risk reduction (ARR), which is the difference in the outcome event rate between the control group and the experimental treated group. Thus, in our wart treatment example, the ARR is the outcome event rate (complete resolution of warts) for duct tape (85 percent) minus the outcome event rate for cryotherapy (60 percent) = 25 percent. Unlike the RRR, the ARR does not amplify small differences but shows the true difference between the experimental and control interventions. Using the ARR, it would be accurate to say that duct tape is 25-percent more effective than cryotherapy in treating warts.

Number needed to treat. The single most clinically useful statistic is the number needed to treat (NNT). The NNT is the number of patients who must be treated to prevent one adverse outcome. To think about it another way, the NNT is the number of patients who must be treated for one patient to benefit. (The rest who were treated obtained no benefit, although they still suffered the risks and costs of treatment.) In our wart therapy article, the NNT would tell us how many patients must be treated with the experimental treatment for one to benefit more than if he or she had been treated with the standard treatment.

Now this is a statistic that physicians and their patients can really appreciate! Furthermore, the NNT is easy to calculate, as it is simply the inverse of the ARR. For our wart treatment study, the NNT is 1/25 percent =1/0.25 = 4, meaning that 4 patients need to be treated with duct tape for one to benefit more than if treated by cryotherapy.

Wrapped up in this simple little statistic are some very important concepts. The NNT provides you with the likelihood that the test or treatment will benefit any individual patient, an impression of the baseline risk of the adverse event, and a sense of the cost to society. Thus, it gives perspective and hints at the “reasonableness” of a treatment. The value of this statistic has become appreciated in the last five years, and more journal articles are reporting it.

What is a reasonable NNT? In a perfect world, a treatment would have an NNT of 1, meaning that every patient would benefit from the treatment. Real life is not so kind (see “Examples of NNTs” ). Clearly, an NNT of 1 is great and an NNT of 1,000 is terrible. Although it is hard to come up with firm guidelines, for primary therapies I am satisfied with an NNT of 10 or less and very pleased with an NNT less than 5. Our duct tape NNT of 4 is good, particularly since the treatment is cheap, easy and painless.

EXAMPLES OF NNTS

The number needed to treat (NNT) is one of the most useful statistics for physicians and patients. It calculates the number of patients that must be treated to prevent one adverse event or for one patient to benefit. Note that NNTs for preventive interventions will usually be higher than NNTs for treatment interventions. The lower the NNT, the better.

The following examples of NNTs are borrowed from an excellent list available through the Bandolier Web site at http://www.jr2.ox.ac.uk/bandolier/band50/b50-8.html .

Note that NNTs for preventive interventions (e.g., the use of aspirin to prevent cardiac problems) will usually be higher than NNTs for treatment interventions (e.g., the use of duct tape to cure warts). Prevention groups contain both higher-risk and lower-risk individuals, so they produce bigger denominators, whereas treatment groups only contain diseased patients. Thus, an NNT for prevention of less than 20 might be particularly good.

When discussing a particular therapy, I explain the NNT to my patient. Since this statistical concept is easy to understand, it can help the patient be a more informed partner in therapeutic decisions.

You will soon start to see a similar statistic, the number needed to screen (NNS), which is the number of patients needed to screen for a particular disease for a given duration for one patient to benefit. 6 Although few NNSs have been calculated, they are likely to involve higher numbers, since the screening population consists of patients with and without the disease. For example, in the article on mammography screening mentioned above, the NNS was 961 for 16 years. In other words, you would need to screen 961 women for 16 years to prevent one breast cancer death.

The good news and the bad

Using PP-ICONS to assess the wart study, the problem, the patient/population, the intervention, the comparison and the outcome are all relevant to your patient. The number of subjects is on the small side, making you a little wary, but the intervention is cheap and low-risk. The statistics, particularly the NNT, are reasonable. On balance, this looks like a fair approach, so you call the patient’s mother and discuss it with her.

The PP-ICONS approach is an easy way to screen an article for validity and relevance, and the abstract often contains all of the information you need. Even the statistics can be done quickly in your head. You can apply PP-ICONS when searching for a particular article, when you come across an article in your reading, when data are presented at lectures, when a pharmaceutical representative hands you an article to support his or her pitch, and even when reading news stories describing medical breakthroughs.

Don’t be discouraged if you find that high-quality articles are rare, even in the most prestigious journals. This seems to be changing for the better, although many careers are still being built on questionable research. Nevertheless, screening articles will help you find the truth that is out there and will help you practice the best medicine. And as we become more discerning end-users of research, we might just stimulate improvements in clinical research in the process.

Miser WF. Critical appraisal of the literature. J Amer Board Fam Pract . 1999;12(4):315-333.

Guyatt GH, et al. Users’ guides to the medical literature. How to use an article about therapy or prevention. Are the results of the study valid?. JAMA . 1993;270(21):2598-2601.

Lepor H, et al. The efficacy of terazosin, finasteride or both in benign prostatic hyperplasia. Veterans Affairs Cooperative Studies Benign Prostatic Hyperplasia Study Group. N Engl J Med . 1996;335(8):533-539.

Krejcie RV, Morgan DW. Determining sample size for research activities. Educational and Psychological Measurement . 1970;30:607-610.

Nystrom L, et al. Long-term effects of mammography screening: updated overview of the Swedish randomised trials. Lancet . 2002;359(9310):909-919.

Rembold CM. Number needed to screen: development of a statistic for disease screening. BMJ . 1998;317:307-312.

Continue Reading

More in fpm, more in pubmed.

Copyright © 2004 by the American Academy of Family Physicians.

This content is owned by the AAFP. A person viewing it online may make one printout of the material and may use that printout only for his or her personal, non-commercial reference. This material may not otherwise be downloaded, copied, printed, stored, transmitted or reproduced in any medium, whether now known or later invented, except as authorized in writing by the AAFP.  See permissions  for copyright questions and/or permission requests.

Copyright © 2024 American Academy of Family Physicians. All Rights Reserved.

We have a new app!

Take the Access library with you wherever you go—easy access to books, videos, images, podcasts, personalized features, and more.

Download the Access App here: iOS and Android . Learn more here!

• Remote Access
• Save figures into PowerPoint
• Download tables as PDFs

Chapter 17:  Evaluating Clinical Literature: An Overview

Jill T. Johnson

• Download Chapter PDF

Disclaimer: These citations have been automatically generated based on the information we have and it may not be 100% accurate. Please consult the latest official manual style if you have any questions regarding the format accuracy.

Download citation file:

• Search Book

Jump to a Section

Chapter objectives, key terminology, introduction, general format of an article.

• CRITICALLY APPRAISING A RESEARCH ARTICLE
• GENERAL CONSIDERATIONS
• SUMMARY AND CONCLUSION
• REVIEW QUESTIONS
• ONLINE RESOURCES
• Full Chapter
• Supplementary Content

Describe a stepwise approach to appraising published literature

Understand the clinical relevance of study objectives

Evaluate appropriateness of design and methods for study objectives

Evaluate methods of analysis and interpretation of results

Differentiate between clinical and statistical significance in the medical literature

Clinical endpoints

Clinical significance

Composite endpoints

Number needed to harm

Number needed to treat

Surrogate endpoints

Critical appraisal of medical literature by clinicians as a tool to improve patient care has evolved since the late 1970s. 1 It introduced a change in the practice paradigm based on knowledge and evolving medical literature rather than on traditional medical authority or anecdotal cases. Although early adoption of evidence-based medicine (EBM) was not readily accepted for various reasons, it has now become a mainstay of the way many clinicians practice. Sometimes the term “evidence-based” is used incorrectly by authors of research that is funded by for-profit agencies, including pharmaceutical manufacturers. Such research is often interpreted in favor of the industry product when compared to research funded by not-for-profit organizations. 2 , 3 Therefore, the ability to critically evaluate medical literature empowers clinicians with enlightened skepticism and the ability to identify biases and errors including inappropriate control interventions, surrogate outcomes, publication bias, and other types of biases, misleading conclusions, and other false interpretations.

A clinician’s ability to deliver the best possible patient care by applying evidence and providing treatment based on sound scientific principles reflects the importance of critical appraisal skills in clinical practice. This chapter will provide an overview on how to critically evaluate medical literature. Beginning with evaluating the clinical research question, the chapter will provide a stepwise approach in appraising research articles. This will include evaluating study design, methods, and statistical analysis. This chapter will particularly emphasize the interpretation of results in the context of patient care. It will also provide specific considerations in evaluating therapy and harm. Lastly, it will conclude with general considerations for the clinicians regarding other biases in publications and their implications.

Although there are several variations in presenting research evidence, most articles published in the medical literature today have four sections: Introduction, Methods, Results, and Discussion (IMRaD) to help guide the reader. In addition to the above sections, a research paper usually includes an abstract and references. 4

Sign in or create a free Access profile below to access even more exclusive content.

With an Access profile, you can save and manage favorites from your personal dashboard, complete case quizzes, review Q&A, and take these feature on the go with our Access app.

Pop-up div Successfully Displayed

This div only appears when the trigger link is hovered over. Otherwise it is hidden from view.

Please Wait

Writing in the Health Sciences: Research and Lit Reviews

• Research and Lit Reviews
• Tables and Figures
• Citation Management
• Further Reference

What Is a Literature Review?

In simple terms, a literature review investigates the available information on a certain topic. It may be only a knowledge survey with an intentional focus. However, it is often a well-organized examination of the existing research which evaluates each resource in a systematic way. Often a lit review will involve a series of inclusion/exclusion criteria or an assessment rubric which examines the research in-depth. Below are some interesting sources to consider.

The Writing Center's Literature Reviews - UNC-Chapel Hill's writing center explains some of the key criteria involved in doing a literature review.

Literature Review vs. Systematic Review - This recent article details the difference between a literature review and a systematic review. Though the two share similar attributes, key differences are identified here.

Literature Review Steps

1. Identify a research question. For example: "Does the use of warfarin in elderly patients recovering from myocardial infarction help prevent stroke?"

2. Consider which databases might provide information for your topic. Often PubMed or CINAHL will cover a wide spectrum of biomedical issues. However, other databases and grey literature sources may specialize in certain disciplines. Embase is generally comprehensive but also specializes in pharmacological interventions.

3. Select the major subjects or ideas from your question.  Focus in on the particular concepts involved in your research. Then brainstorm synonyms and related terminology for these topics.

4. Look for the  preferred indexing terms for each concept in your question. This is especially important with databases such as PubMed, CINAHL, or Scopus where headings within the MeSH database or under the Emtree umbrella are present.  For example, the above question's keywords such as " warfarin " or "myocardial infarction" can involve related terminology or subject headings such as "anti-coagulants" or "cardiovascular disease."

5. Build your search using boolean operators. Combine the synonyms in your database using boolean operators such as AND or OR. Sometimes it is necessary to research parts of a question rather than the whole. So you might link searches for things like the preventive effects of anti-coagulants with stroke or embolism, then AND these results with the therapy for patients with cardiovascular disease.

6. Filter and save your search results from the first database (do this for all databases). This may be a short list because of your topic's limitations, but it should be no longer than 15 articles for an initial search. Make sure your list is saved or archived and presents you with what's needed to access the full text.

7. Use the same process with the next databases on your list. But pay attention to how certain major headings may alter the terminology. "Stroke" may have a suggested term of "embolism" or even "cerebrovascular incident" depending on the database.

8. Read through the material for inclusion/exclusion . Based on your project's criteria and objective, consider which studies or reviews deserve to be included and which should be discarded. Make sure the information you have permits you to go forward. 

9. Write the literature review. Begin by summarizing why your research is important and explain why your approach will help fill gaps in current knowledge. Then incorporate how the information you've selected will help you to do this. You do not need to write about all of the included research you've chosen, only the most pe rtinent.

10. Select the most relevant literature for inclusion in the body of your report. Choose the articles and data sets that are most particularly relevant to your experimental approach. Consider how you might arrange these sources in the body of your draft. 

Library Books

Call #: WZ 345 G192h 2011

ISBN #: 9780763771867

This book details a practical, step-by-step method for conducting a literature review in the health sciences. Aiming to  synthesize the information while also analyzing it, the Matrix Indexing System enables users to establish a  structured process for tracking, organizing and integrating the knowledge within a collection.

Key Research Databases

PubMed -  The premier medical database for review articles in medicine, nursing, healthcare, other related biomedical disciplines. PubMed contains over 20 million citations and can be navigated through multiple database capabilities and searching strategies.

CINAHL Ultimate - Offers comprehensive coverage of health science literature. CINAHL is particularly useful for those researching the allied disciplines of nursing, medicine, and pharmaceutical sciences.

Scopus - Database with over 12 million abstracts and citations which include peer-reviewed titles from international and Open Access journals. Also includes interactive bibliometrics and researcher profiling.

Embase - Elsevier's fully interoperable database of both Medline and Emtree-indexed articles. Embase also specializes in pharmacologic interventions.

Cochrane - Selected evidence-based medicine resources from the Cochrane Collaboration that includes peer-reviewed systematic reviews and randomized controlled trials. Access this database through OVID with TTUHSC Libraries.

DARE - Literally the Datatase of Abstracts of Reviews of Effectiveness, this collection of systematic reviews and other evidence-based research contains critical assessments from a wide variety of medical journals.

TRIP - This TRIP database is structured according to the level of evidence for its EBM content. It allows users to quickly and easily locate high-quality, accredited medical literature for clinical and research purposes.

Web of Science - Contains bibliographic articles and data from a wide variety of publications in the life sciences and other fields. Also, see this link for conducting a lit review exclusively within Web of Science.

• << Previous: Welcome
• Next: Drafting >>
• Last Updated: Sep 29, 2023 10:07 AM
• URL: https://ttuhsc.libguides.com/Writing_HealthSciences

Systematic Review of the Literature: Best Practices

Affiliations.

• 1 Department of Radiology and Imaging, Rush University Medical Center, 3833, 1653 W Congress Pkwy, Chicago, IL 60612. Electronic address: [email protected].
• 2 Cardiothoracic Imaging, Department of Radiology, UT Southwestern Medical Center, 5323 Harry Hines Boulevard, Dallas, TX, USA.
• 3 Department of Radiology, Mayo Clinic, 4500 San Pablo Rd, Jacksonville, FL 32224.
• 4 Chief, Cardiothoracic and Emergency Radiology, Department of Radiology, Medical College of Wisconsin, Milwaukee, WI, USA.
• 5 Department of Diagnostic Radiology, The University of Texas MD Anderson Cancer Center, 1515 Holcombe Blvd. Unit 1478, Houston, TX, 77030.
• 6 Department of Medical Imaging, University of Toronto, 263 McCaul Street, 4th Floor, Toronto, ON, M5T 1W7.
• 7 Department of Radiology, Medical University of South Carolina, Charleston, South Carolina, Department of Radiology and Radiological Sciences, Medical University of South Carolina, 96 Jonathan Lucas St. MSC 323, Charleston, SC 29425.
• 8 Department of Radiology and Medical Imaging, University of Virginia Medical Center, 1215 Lee St., Charlottesville, VA 22903.
• PMID: 30442379
• DOI: 10.1016/j.acra.2018.04.025

Reviews of published scientific literature are a valuable resource that can underline best practices in medicine and clarify clinical controversies. Among the various types of reviews, the systematic review of the literature is ranked as the most rigorous since it is a high-level summary of existing evidence focused on answering a precise question. Systematic reviews employ a pre-defined protocol to identify relevant and trustworthy literature. Such reviews can accomplish several critical goals that are not easily achievable with typical empirical studies by allowing identification and discussion of best evidence, contradictory findings, and gaps in the literature. The Association of University Radiologists Radiology Research Alliance Systematic Review Task Force convened to explore the methodology and practical considerations involved in performing a systematic review. This article provides a detailed and practical guide for performing a systematic review and discusses its applications in radiology.

Keywords: effective systematic review; radiology review; research methodology; systematic review.

Copyright © 2018. Published by Elsevier Inc.

• Systematic Reviews as Topic*

University Libraries

• Ohio University Libraries
• Library Guides

Evidence-based Practice in Healthcare

• Performing a Literature Review
• EBP Tutorials
• Question- PICO
• Definitions
• Systematic Reviews
• Levels of Evidence
• Finding Evidence
• Filter by Study Type
• Too Much or Too Little?
• Critical Appraisal
• Quality Improvement (QI)
• Contact - Need Help?

Hanna's Performing a qualitity literature review presentation slides

• Link to the PPT slides via OneDrive anyone can view

Characteristics of a Good Literature Review in Health & Medicine

Clear Objectives and Research Questions : The review should start with clearly defined objectives and research questions that guide the scope and focus of the review.

Comprehensive Coverage : Include a wide range of relevant sources, such as research articles, review papers, clinical guidelines, and books. Aim for a broad understanding of the topic, covering historical developments and current advancements. To do this, an intentional and minimally biased search strategy.

• Link to relevant databases to consider for a comprehensive search (search 2+ databases)
• Link to the video "Searching your Topic: Strategies and Efficiencies" by Hanna Schmillen
• Link to the worksheet "From topic, to PICO, to search strategy" to help researchers work through their topic into an intentional search strategy by Hanna Schmillen

Transparency and Replicability : The review process, search strategy, should be transparent, with detailed documentation of all steps taken. This allows others to replicate the review or update it in the future.

Appraisal of Studies Included : Each included study should be critically appraised for methodological quality and relevance. Use standardized appraisal tools to assess the risk of bias and the quality of evidence.

• Link to the video " Evaluating Health Research" by Hanna Schmillen
• Link to evaluating and appraising studies tab, which includes a rubric and checklists

Clear Synthesis and Discussion of Findings : The review should provide a thorough discussion of the findings, including any patterns, relationships, or trends identified in the literature. Address the strengths and limitations of the reviewed studies and the review itself. Present findings in a balanced and unbiased manner, avoiding over interpretation or selective reporting of results.

Implications for Practice and Research : The review should highlight the practical implications of the findings for medical practice and policy. It should also identify gaps in the current literature and suggest areas for future research.

Referencing and Citation : Use proper citation practices to credit original sources. Provide a comprehensive reference list to guide readers to the original studies.

• Link to Citation Style Guide, includes tab about Zotero

Note: A literature review is not a systematic review. For more information about systematic reviews and different types of evidence synthesis projects, see the Evidence Synthesis guide .

• << Previous: Quality Improvement (QI)
• Next: Contact - Need Help? >>

Differences in quality of anticoagulation care delivery according to ethnoracial group in the United States: A scoping review

• Open access
• Published: 11 May 2024

Cite this article

You have full access to this open access article

• Sara R. Vazquez   ORCID: orcid.org/0000-0002-9267-8980 1 ,
• Naomi Y. Yates 2 ,
• Craig J. Beavers 3 , 4 ,
• Darren M. Triller 3 &
• Mary M. McFarland 5  

381 Accesses

6 Altmetric

Explore all metrics

Anticoagulation therapy is standard for conditions like atrial fibrillation, venous thromboembolism, and valvular heart disease, yet it is unclear if there are ethnoracial disparities in its quality and delivery in the United States. For this scoping review, electronic databases were searched for publications between January 1, 2011 – March 30, 2022. Eligible studies included all study designs, any setting within the United States, patients prescribed anticoagulation for any indication, outcomes reported for ≥ 2 distinct ethnoracial groups. The following four research questions were explored: Do ethnoracial differences exist in 1) access to guideline-based anticoagulation therapy, 2) quality of anticoagulation therapy management, 3) clinical outcomes related to anticoagulation care, 4) humanistic/educational outcomes related to anticoagulation therapy. A total of 5374 studies were screened, 570 studies received full-text review, and 96 studies were analyzed. The largest mapped focus was patients’ access to guideline-based anticoagulation therapy (88/96 articles, 91.7%). Seventy-eight articles made statistical outcomes comparisons among ethnoracial groups. Across all four research questions, 79 articles demonstrated favorable outcomes for White patients compared to non-White patients, 38 articles showed no difference between White and non-White groups, and 8 favored non-White groups (the total exceeds the 78 articles with statistical outcomes as many articles reported multiple outcomes). Disparities disadvantaging non-White patients were most pronounced in access to guideline-based anticoagulation therapy (43/66 articles analyzed) and quality of anticoagulation management (19/21 articles analyzed). Although treatment guidelines do not differentiate anticoagulant therapy by ethnoracial group, this scoping review found consistently favorable outcomes for White patients over non-White patients in the domains of access to anticoagulation therapy for guideline-based indications and quality of anticoagulation therapy management. No differences among groups were noted in clinical outcomes, and very few studies assessed humanistic or educational outcomes.

Graphical Abstract

Scoping Review: Differences in quality of United States anticoagulation care delivery by ethnoracial group. AF = atrial fibrillation; AMS = anticoagulation management service; DOACs = direct oral anticoagulants; INR = international normalized ratio; PSM = patient self-management; PST = patient self-testing

Similar content being viewed by others

Patients’ and clinicians’ perceptions of oral anticoagulants in atrial fibrillation: a systematic narrative review and meta-analysis

Factors influencing primary care physicians’ prescribing behavior of anticoagulant therapy for the management of patients with non-valvular atrial fibrillation in Singapore: a qualitative research study

Enrolling people of color to evaluate a practice intervention: lessons from the shared decision-making for atrial fibrillation (SDM4AFib) trial

Avoid common mistakes on your manuscript.

Introduction

It is well-established that in the United States (US) ethnoracial disparities exist in various aspects of health care. Specifically, persons identifying with an ethnoracial minority group may have more challenging access to health care, worse clinical outcomes, and higher dissatisfaction with care compared to White persons [ 1 , 2 , 3 , 4 , 5 ]. There are differences by ethnoracial group in the prevalence of the three most common indications for which anticoagulants are prescribed, stroke prevention in atrial fibrillation (AF), treatment of venous thromboembolism (VTE), and valvular heart disease [ 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 ]. Specifically, VTE is most prevalent in Black patients compared to White and Asian patients, whereas AF is most prevalent in White patients compared to Black, Asian, and Hispanic patients [ 9 , 10 , 15 ]. Calcific heart valve disease has the most relevance to the US population, and epidemiologic data has shown that aortic stenosis is more prevalent in White patients compared to Black, Asian, and Hispanic patients [ 17 ]. Despite these epidemiologic differences, there is no evidence to suggest there should be any difference in treatment strategies across ethnoracial patient groups.

While studies have demonstrated genotypic differences that may result in different warfarin dose requirements[ 18 ], and early studies may indicate genotypic differences in direct oral anticoagulant (DOAC) response [ 19 ], no US-based labeling or guidelines recommend a difference in prescription or delivery of anticoagulation care based on race or ethnicity. However, it is unclear if there are in fact differences in the type and quality of anticoagulation therapy, which is standard of care for each of these conditions [ 20 , 21 , 22 , 23 , 24 ]. Anticoagulants remain in the top three classes of drugs causing adverse drug events (primarily bleeding) in the United States, according to the 2014 National Action Plan for Adverse Drug Event Prevention. One of the goals of the National Action Plan was to identify patient populations at higher risk for these adverse drug events to inform the development of targeted harm reduction strategies [ 25 ]. If ethnoracial minority patients are receiving sub-optimal anticoagulation therapy in certain measurable areas of anticoagulation quality, it is vital to highlight the areas of disparity so that these can be explored and care optimized. Anticoagulation providers often have high frequency contact with their patients and can be a reliable connection between disproportionately affected patients and a system in need of change. Systematic reviews of ethnoracial disparities in AF and VTE have been conducted. The AF review assessed AF prevalence among racial groups as well as differences in symptoms and management, including stroke prevention with warfarin or DOACs [ 9 ]. The VTE review specifically assessed VTE prevalence and racial differences in COVID-19 and did report the use of any prophylactic anticoagulation, but this was not part of the analysis [ 26 ]. No review of racial disparities in quality of anticoagulation therapy was found in search results conducted prior to protocol.

In this study we aimed to identify any potential ethnoracial disparities in anticoagulation care quality in the US. The decision to limit the study to a US population was based on our observation that the US has a unique history of interactions between racial and ethnic groups that may not necessarily be reflected by studies conducted in other countries. Additionally, health care delivery systems vary widely across the world, and we wanted to include the data most relevant to the potential racial disparities existing in the US health care system. The term “race” was used to identify a group of people with shared physical characteristics believed to be of common ancestry whereas the term “ethnicity” refers to a group of people with shared cultural traditions [ 27 ]. We recognize these terms may be far more complex. In order to encompass both the physical and cultural aspects of a patient’s identity we have chosen to use the term “ethnoracial” for this study [ 27 ]. Highlighting existing differences will serve as a stimulus for institutions and clinicians to assess current services, implement quality improvement measures, and inform future research efforts to deliver optimal anticoagulation care for all patients. The scoping review protocol was registered December 22, 2021 to Open Science Framework, https://doi.org/10.17605/OSF.IO/9SE7H [ 28 ].

We conducted this scoping review with guidance from the 2020 version of the JBI Manual for Evidence Synthesis and organized to Arksey's five stages: 1) identifying the research question, 2) identifying relevant studies, 3) study selection, 4) charting the data and 5) collating, summarizing and reporting the results [ 29 , 30 ]. For transparency and reproducibility, we followed the PRISMA-ScR and PRISMA-S reporting guidelines in reporting our results [ 31 ]. We used Covidence (Veritas Health Innovation,) an online systematic reviewing platform to screen and select studies. Citation management and duplicate detection and removal was accomplished with EndNote, version 19 (Clarivate Analytics.) Data was charted from our selected studies using REDCap, an electronic data capture tool hosted at the University of Utah [ 32 ].

Literature searching

An information specialist developed and translated search strategies for the online databases using a combination of keywords and controlled subject headings unique to each database along with team feedback. Peer review of the strategies was conducted by library colleagues using the PRESS guidelines. [ 33 ] Electronic databases searched included Medline (Ovid) 2011–2022, Embase (embase.com) 2011–2022, CINAHL Complete (Ebscohost) 2011–2022, Sociological Abstracts (ProQuest) 2011–2022, International Pharmaceutical Abstracts (Ovid) 2011–2022, Scopus (scopus.org) 2011–2022 and Web of Science Core Collection (Clarivate Analytics) 2011–2022. Limits included a date range from January 1, 2011 to March 30—April 19, 2022, as not all database results were exported on the same day. See Supplemental File 1 for detailed search strategies. A search of grey literature was not conducted due to time and resource constraints.

Study Selection

For inclusion, each study required two votes by independent reviewers for screening of titles and abstracts followed by full-text review. A third reviewer provided the deciding vote. Data extraction was performed by two independent reviewers, and consensus on any discrepancies was reached via discussion between the reviewers. The data form was piloted by two team members using sentinel articles prior to data extraction.

Eligible studies included all types of study designs in any setting with a population of patients of any age or gender located within the US who were prescribed anticoagulant therapy for any indication, published between January 1, 2011 – March 30, 2022 in order to capture contemporary and clinically relevant practices.

We defined the following research questions for this scoping review as described in Table  1 .

Studies must have reported any of these anticoagulation care delivery outcomes for at least 2 distinct racial or ethnic groups. We excluded genotyping studies and non-English language articles at full text review, as we had no funding for translation services. In checking references of included studies, no additional studies met inclusion criteria. In accordance with scoping review methodology, no quality assessment of included studies was conducted as our goal was to rapidly map the literature. As this is a scoping review of the literature, no aggregate or pooled analysis was performed; however, for ease of interpretation, when assessing for the directionality of the outcomes in the various studies, we categorized studies into Favoring White Group, Favoring Non-White Group, and No Differences Among Ethnoracial Groups. If studies had mixed outcomes of favoring one group for one outcome and no difference for another, then the study was categorized with the favoring group.

A PRISMA flow diagram in Fig.  1 depicts search results, exclusions, and inclusions. The search strategies retrieved 6900 results with 1526 duplicates removed. Following title and abstract screening of 5374 references, 570 articles received full-text review. The most common reason for the exclusion of 474 studies was that outcomes were not reported for two distinct ethnoracial groups (171 studies). Ninety-six studies underwent data extraction.

PRISMA Flow Diagram

Study characteristics-overall

Fifty of the 96 studies were published between 2011 and 2018 (an average of 6.25 articles per year that compared outcomes between two ethnoracial groups) and 43 of 96 studies were published in the years 2019–2021 (average 14.3 articles per year; 2022 excluded here because only 4 months of data was captured) (Fig.  2 ). Most studies analyzed an outpatient population (65.6%) for an indication of stroke prevention in AF (67.7%) in patients taking warfarin (71.9%) or DOACs (49.0%). Study population size was heterogenous, ranging from a study size of 24 patients to over 1.3 million patients (median 5,238 patients) in the 69 studies that reported population size by racial group. When stratified by size, 60.9% of the articles in the scoping review (42 articles) represented < 10,000 patients (Table  2 ).

Number of Articles by Publication Year. *2022 excluded from this figure since the search period did not capture the entire year

Study characteristics-by ethnoracial group

There were 50 studies (52.1%) where race or ethnicity was either mentioned in the title or objective of the article, with 24 of these published over the 7-year period 2011–2018 and 26 published over the 3-year period 2019 to first quarter 2022. The method for reporting race or ethnicity was unclear or unspecified in most studies (77.1%) and 16 articles (16.7%) utilized self-reporting of race or ethnicity. Most studies analyzed White or Caucasian racial groups (94.8%), followed by Black or African-American (80.2%), and many studies grouped all other racial groups into an “Other” category (41.7%) (Fig.  3 ).

Number of Articles by Ethnoracial Groups. *For study inclusion, a study had to compare outcomes for least two distinct ethnoracial groups 

White patients accounted for a median 77% of study populations, Black patients 9.5%, Hispanic/Latino patients 6.2%, “Other” racial groups 5.3%, and Asian patients 2.5%.

Study outcomes-overall

Of the 4 research questions, most studies included in this review analyzed patients’ access to guideline-based anticoagulation therapy (88/96 articles, 91.7%), clinical outcomes (42/96 articles, 43.8%), or quality of anticoagulation management (24/96 articles, 25.0%), while very few addressed humanistic or educational outcomes (5/96 articles, 5.2%) (Fig.  4 ). Many studies addressed multiple outcomes within the single study.

Number of Articles Mapped by Research Question

Seventy-eight of the 96 included studies provided statistical comparisons between ethnoracial groups, and these data are presented below.

Outcomes for research question 1: Do ethnoracial differences exist in access to guideline-based anticoagulation therapy?

Anticoagulation for a guideline-based indication.

This question focused on patients who had an indication for anticoagulation actually receiving an anticoagulant, specifically AF and VTE prophylaxis (based on risk stratification) and acute VTE. The majority of the AF studies (25/34 studies) demonstrated White patients receiving anticoagulation at significantly higher rates compared to non-White patients [ 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 ], while the six VTE studies largely demonstrated no difference among ethnoracial groups [ 61 , 62 , 63 , 64 , 65 , 66 ].

DOACs as first-line therapy for AF or VTE

Eighteen individual studies statistically assessed the outcome of DOAC as first-line therapy (compared to warfarin) for AF (15 studies), VTE treatment (2 studies), or both indications (1 study). Twelve of the 15 AF studies showed a significantly higher proportion of White patients received DOACs as first-line therapy compared to non-White patients [ 36 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 54 , 55 , 67 , 68 ]. Of those 12, 9 specifically compared White patients to Black patients. Both VTE treatment studies and the study that assessed both AF and VTE indications showed significantly higher DOAC prescribing rates for White patients compared to Black patients [ 69 , 70 , 71 ].

Anticoagulant therapy adherence/persistence

The eight studies that addressed anticoagulation therapy adherence/persistence showed variability in outcome directionality by ethnoracial group: 5 no difference [ 41 , 72 , 73 , 74 , 75 ], 2 showed better treatment adherence/persistence for White patients compared to Black patients[ 76 ] or non-White patients [ 77 ], and one showed better treatment adherence/persistence for White patients compared to Hispanic patients, but no difference in White versus Black patients [ 78 ].

Figure  5 summarizes the outcome directionality for Research Question 1 regarding access to guideline-based anticoagulation therapy. Overall, the areas of disparity identified included anticoagulation for atrial fibrillation and preferential use of DOAC therapy for AF and VTE treatment.

Outcome Directionality for the 4 Research Questions and their Subcategories. AC = anticoagulant; AMS = anticoagulation management service; INR = international normalized ratio; PST = patient self-testing; PSM = patient self-management

Research question 2: Do ethnoracial differences exist in the quality of anticoagulation therapy management?

A total of 21 studies assessed quality of anticoagulation therapy management: Warfarin time in therapeutic range (TTR)/INR (International Normalized Ratio) control 12 studies, appropriate anticoagulant dosing 3 studies, enrollment in an anticoagulation management service 5 studies, and PST/PSM one study.

In statistical comparisons of INR control in warfarin patients, all 12 studies (7 assessed mean or median TTR, 5 assessed other measures of INR control such as days spent above/below range, gaps in INR monitoring) showed White patients had favorable INR control compared to non-White patients (most comparisons included Black patients) [ 41 , 75 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 , 87 ]. Enrollment in an anticoagulation management service was statistically compared among ethnoracial groups in 5 studies, and this opportunity favored White patients compared to other racial groups in four of the five [ 41 , 82 , 86 , 88 ]. Two of the three studies that statistically analyzed appropriate anticoagulant dosing showed a higher rate of appropriate DOAC dosing in White patients compared to non-White patients [ 41 , 89 ], and the third showed no difference among ethnoracial groups for enoxaparin dosing in the emergency department [ 90 ]. The one study assessing access to PST/PSM showed that more White patients used PST compared to Black or Hispanic patients[ 91 ] (Fig.  5 ).

Research question 3: Do ethnoracial differences exist in the clinical outcomes related to anticoagulation care?

Articles assessing clinical outcomes among ethnoracial groups primarily assessed bleeding (15 articles) or thrombosis (9 articles) outcomes, and 8 articles assessing anticoagulation related hospitalization or mortality. One article addressed a net clinical outcome including major bleeding, stroke or systemic embolism, and death from any cause. This was included in the bleeding outcomes category so that it was not double-counted in the other two outcome categories. Additional details about the 24 unique studies that statistically assessed clinical outcomes including the study design, population size, ethnoracial groups studied, anticoagulants used, and statistical outcomes measured can be found in Supplementary Tables 1 and 2 .

Sixteen studies statistically assessed bleeding outcomes of varying definitions (major bleeding 13 studies, clinically relevant non-major bleeding 3 studies, any bleeding 3 studies, bleeding otherwise defined 3 studies). Six studies demonstrated no difference in bleeding outcomes by ethnoracial group [ 55 , 92 , 93 , 94 , 95 , 96 ]9 reported that White patients had lower rates of bleeding compared to Black or Asian patients,[ 53 , 80 , 83 , 85 , 97 , 98 , 99 , 100 , 101 ]. In the remaining study, Asian patients had a more favorable net clinical outcome compared to non-Asian patients [ 102 ].

Nine studies statistically assessed thrombosis outcomes among ethnoracial groups, including stroke/systemic embolism (5 studies), recurrent VTE (3 studies), or any thrombosis (1 study). The stroke outcomes by racial group were heterogeneous, with 3 studies showing better outcomes for White patients compared to Black patients[ 103 , 104 , 105 ] and two studies showing no difference in outcomes when White patients were compared to Non-White patients [ 55 , 95 ]. In three of the four VTE studies there were no differences in outcomes by ethnoracial group [ 61 , 93 , 96 ], and in one study White patients had more favorable outcomes compared to Black patients [ 106 ].

Nine studies assessed anticoagulation-related hospitalizations or mortality by ethnoracial group. Outcomes were mixed, as four studies showed no difference in hospitalizations or mortality among ethnoracial groups,[ 89 , 95 , 96 , 107 ], three studies showed White patients had a lower rate of hospitalizations[ 85 , 105 ] or mortality[ 104 , 105 ] Another study showed lower rate of mortality or hospice after intracranial hemorrhage in Black and Other race patients [ 108 ].(Fig.  5 ).

Research question 4: Do ethnoracial differences exist in the humanistic/educational outcomes related to anticoagulation therapy?

The five studies reporting this category of outcomes were heterogeneous. Of the two studies assessing anticoagulation knowledge, one showed no difference by ethnoracial group [ 109 ], and the other favored the non-White group in appropriately estimating bleeding risk [ 110 ]. One study assessed an atrial fibrillation quality of life score at 2-year follow-up after AF diagnosis and found the outcomes favored White patients [ 79 ]. Another study assessed satisfaction with VTE care and found no difference among ethnoracial groups [ 111 ]. A third study found no difference in the percentage of racial groups having a cost conversation when initiating DOAC therapy (78% Whites, 72.2% non-Whites)[ 112 ] (Fig.  5 ).

Overall outcome directionality for all four research questions is shown in Fig.  6 . A total of 79 articles demonstrated favorable outcomes for White patients compared to non-White patients, 38 articles showed no difference between White and non-White groups, and 8 articles had outcomes favoring non-White groups (the total exceeds the 78 articles with statistical outcomes as many articles reported multiple outcomes). The biggest areas of disparity between White and non-White groups are access to guideline-based anticoagulation therapy and quality of anticoagulation therapy management. Clinical outcomes relating to anticoagulation care had the least difference among ethnoracial groups. Relatively few studies assessed potential ethnoracial disparities in humanistic and educational outcomes.

Outcome Directionality for All 4 Research Questions

This scoping review assessing ethnoracial differences in the quality of anticoagulation care and its delivery to patients in the United States encompassed eleven full years of literature and resulted in the inclusion of 96 studies, 78 of which contained statistical outcomes comparisons among ethnoracial groups. The most common reason for study exclusion was that outcomes were not reported for at least two distinct ethnoracial groups. We observed that beginning in 2019 and following the racial unrest of 2020, the density of articles addressing ethnoracial disparities in anticoagulation care more than doubled. During the entire study period, half of studies had race or ethnicity as the focus or objective of the paper, but this was largely driven by articles published after 2019.

Only 16% of included articles documented self-reporting of racial identity, with most of the remainder using an unspecified method for documenting racial identity. It is likely that many studies utilize demographic information extracted from an electronic medical record (EMR), but it is often unclear if that is truly self-reported race. A second element this scoping review identified was that many studies analyzed two or three ethnoracial groups and then categorized all others into a heterogenous “Other” category. For example, frequently studies would categorize patients as White, Black, and “Other.” It is unclear whether those in a racial category labeled as “Other” had an unknown or missing racial identity in the EMR, or intentionally chose not to disclose. It is also likely that study investigators decided to classify ethnoracial groups with lower population sizes into a miscellaneous category. There were few studies (15%) that specifically assessed patients identifying as Native American/Alaska Native, Native Hawaiian/Pacific Islander, and multiracial. While Hispanic/Latino is an ethnicity, most studies categorized it as a separate “race” category. Of the 37 studies that analyzed “Asian” patient populations, none specifically defined “Asian” beyond that. The US Census Bureau defines “Asian” race as a person having origins of the Far East, Southeast Asia, or the Indian subcontinent [ 113 ]. This broad definition encompasses many different ethnicities which could represent variability in health outcomes if better defined and more frequently analyzed. These may be opportunities for EMR systems to improve transparency for how race, ethnicity, and language preference are captured and for those designing research studies to be thoughtful and intentional about analyzing the ethnoracial identities of the study population, perhaps in alignment with the minimum 5 racial categories utilized by the US Census Bureau, the National Institutes of Health, and the Office of Management and Budget (White, Black, American Indian/Alaska Native, Asian, Native Hawaiian/Pacific Islander, with permission for a “some other race” category and the option to select multiple races) [ 113 ]. Since 2017 Clinicaltrials.gov has required the reporting of race/ethnicity if collected, and there is good compliance with this requirement, but less so in publication of the work [ 114 ].

We examined the proportion of ethnoracial groups represented for each of the disease states in the studies included in this scoping review, relative to disease state prevalence and found a discrepancy. For AF, prevalence in White patients was 11.3%, in Black patients 6.6%, and in Hispanic patients 7.8% [ 15 ]. However, the representation in AF studies in this review were 74% White, 13% Black, and 8% Hispanic. Assessing VTE incidence by race is more difficult, as studies have shown regional and time variation, with Black patients typically having a higher incidence compared to other ethnoracial groups [ 16 ]. In this review, however, of the studies assessing VTE treatment or prophylaxis, only 16% of the patient population identified as Black, whereas 70% identified as White. There were only 3 studies that assessed a valvular heart disease population, making ethnoracial group representation difficult to assess.

The majority of studies captured in this review analyzed patients in the outpatient setting, for the anticoagulation indication of stroke prevention in AF, taking either warfarin or DOAC. Few studies involved the acute care setting or injectable anticoagulants, representing an area for future study of potential ethnoracial disparities.

Overall, the majority of studies in this scoping review addressed ethnoracial disparities in patients’ access to guideline-based anticoagulation therapy, clinical outcomes related to anticoagulation care, and quality of anticoagulation management. A research gap identified was more study is needed to assess gaps in educational outcomes such as anticoagulation and disease state knowledge, shared decision-making willingness and capability, and humanistic outcomes such as quality of life or satisfaction with anticoagulation therapy.

In analyzing the first research question regarding ethnoracial differences in access to guideline-based anticoagulation therapy, the majority of studies addressed use of any anticoagulation for stroke prevention in AF in patients above a threshold risk score and the preferential use of DOACs as first-line therapy instead of warfarin for AF. In both categories, patients in a non-White ethnoracial group (particularly Black patients) received recommended therapy less often than patients identified as White. It is unclear why this is the case. It could be on the patient, provider, and/or system level. It is possible that some studies more successfully adjusted for covariates than others. Sites or settings with systematic processes like order sets or clinical decision support systems in place for standard prescribing may be more successful in equitably prescribing indicated therapies. In one large study in the Veterans Affairs population of AF patients, even after adjusting for numerous variables that included clinical, demographic, socioeconomic, prescriber, and geographic site factors, DOAC prescribing remained lower in Asian and Black patients when compared with White patients. The authors in that study postulate that non-White populations may be less receptive to novel therapies due to historical mistrust of the health care system or have reduced access to education about the latest treatments, and they give the example of direct-to-consumer advertising [ 42 ]. It has also previously been demonstrated that prescribing of oral anticoagulation and particularly DOACs is lower in non-White patients [ 41 ]. These are difficult to capture as standard covariates, which is why further study is needed. We examined the publication dates for both access categories to see if perhaps there was a lack of contemporary data skewing the outcomes. However, for both anticoagulation for a guideline-based indication and DOACs as first-line therapy, the majority of articles came from the time period 2019–2021 (24 of 40 articles, and 15 of 18 articles, respectively), well after guideline updates preferentially recommended DOACs [ 34 , 35 ]. Also, there were relatively few studies addressing guideline-based therapy for VTE treatment and prophylaxis, making assessment of disparities difficult. Regarding access, it is well established that race and ethnicity often determine a patient’s socioeconomic status and that low socioeconomic status and its correlates (e.g., reduced education, income, and healthcare access) are associated with poorer health outcomes [ 115 ]. However, at each level of income or education, Black patients experience worse health outcomes than Whites [ 116 ]. So, low socioeconomic status does not fully explain poorer health outcomes for non-White individuals.

After examining access to appropriate and preferred anticoagulation therapy, the second research question of this scoping review examined potential ethnoracial disparities in the quality of anticoagulation therapy management. INR control measures such as time in therapeutic INR range are a surrogate measure of both thrombotic and bleeding outcomes and frequently used as a way to assess quality of warfarin therapy. The studies identified in this review showed clear disparity between White and non-White patient groups (especially Black patients), however all twelve studies comparing TTR among ethnoracial groups were published prior to 2019. This could be due to the decline in warfarin prescribing relative to increases in DOAC prescribing [ 117 , 118 , 119 ], but there remain patient populations that require or choose warfarin, so this marker of anticoagulation control remains relevant and requires continued reassessment. There were relatively few studies assessing other markers of anticoagulation management quality such as anticoagulation management service enrollment, appropriate DOAC dosing, and access to quality improvement strategies like PST or PSM. Few studies assessed educational outcomes, yet this may have relevance to the above anticoagulation care quality question. For those patients who remain on warfarin, dietary Vitamin K consistency is an example of a key educational point that links directly to INR control. It is unclear if there are disparities in this type of education among ethnoracial groups that may have more far-reaching effects.

Of note, clinical outcomes related to anticoagulant therapy seemed to have the fewest areas of disparity, although the number of articles was small. This suggests that if patients have access to high quality anticoagulation therapy, there is a promising sign that optimal clinical outcomes can be achieved for all ethnoracial groups.

There are some limitations of this scoping review that warrant consideration. First, we chose fairly broad inclusion criteria (all anticoagulants, all study types) because a review of this type had never been performed before. This resulted in a relatively large number of included articles for a scoping review. Second, there is likely a high degree of heterogeneity among patient populations and outcomes definitions. However, as this is a scoping review with the goal to present an overview of the literature and not report on composite outcomes, a risk of bias assessment was not performed. Third is our decision to group patients into White and non-White groups for assessment of outcome directionality. In doing so, we may have missed subtle differences in outcomes between various non-White ethnoracial groups. Fourth, in our main search we included all studies that reported outcomes, but due to scope, we only reported outcome directionality for studies that statistically compared outcomes between ethnoracial groups. Finally, due to the large number of studies that required review and analysis, this was a lengthy undertaking and we are certain that additional studies have been published since the closure of our search period.

In line with the 2014 National Action Plan for Adverse Drug Event Prevention’s goal of identifying patient populations at higher risk of adverse drug events, this scoping review highlights several areas where quality of anticoagulation care can be optimized for all patients. Future research opportunities in ethnoracial differences in the quality of anticoagulation care are summarized in Table  3 . While the scoping review focused exclusively on the evaluation of peer-reviewed manuscripts, the heterogeneity of terminology and methodologies identified in the published papers may have implications for national health policy relating to the quality and safety of care (e.g.the Medicare Quality Payment Program) [ 120 ]. To accurately and reliably quantify important disparities in AC-related care and support effective improvement initiatives, attention and effort will need to be invested across the full continuum of quality measure development [ 121 ], measure endorsement [ 122 ], measure selection, and status assignment within value-based payment programs (e.g., required/optional, measure weighting) [ 123 ]. The findings of the scoping review may be of utility to such efforts, and the development and implementation of suitable quality measures will likely be of value to future research efforts in this important therapeutic area.

Conclusions

Treatment guidelines do not recommend differentiating anticoagulant therapy by ethnoracial group, yet this scoping review of the literature demonstrates consistent directionality in favor of White patients over non-White patients in the domains of access to anticoagulation therapy for guideline-based indications, prescription of preferred anticoagulation therapies, and quality of anticoagulation therapy management. These data should serve as a stimulus for an assessment of current services, implementation of quality improvement measures, and inform future research to make anticoagulation care quality more equitable.

Data Availability

Data are available on request from the corresponding author.

Wheeler SM, Bryant AS (2017) Racial and Ethnic Disparities in Health and Health Care. Obstet Gynecol Clin North Am 44(1):1–11

Article   PubMed   Google Scholar  

Manuel JI (2018) Racial/Ethnic and Gender Disparities in Health Care Use and Access. Health Serv Res 53(3):1407–1429

Nadeem MF, Kaiser LR (2022) Disparities in Health Care Delivery Systems. Thorac Surg Clin 32(1):13–21

Wallace J et al (2021) Changes in Racial and Ethnic Disparities in Access to Care and Health Among US Adults at Age 65 Years. JAMA Intern Med 181(9):1207–1215

Article   PubMed   PubMed Central   Google Scholar  

Churchwell K et al (2020) Call to Action: Structural Racism as a Fundamental Driver of Health Disparities: A Presidential Advisory From the American Heart Association. Circulation 142(24):e454–e468

Gomez SE et al (2023) Racial, ethnic, and sex disparities in atrial fibrillation management: rate and rhythm control. J Interv Card Electrophysiol 66(5):1279–1290

Tamirisa KP et al (2021) Racial and Ethnic Differences in the Management of Atrial Fibrillation. CJC Open 3(12 Suppl):S137–S148

Eberly LA et al (2021) Racial/Ethnic and Socioeconomic Disparities in Management of Incident Paroxysmal Atrial Fibrillation. JAMA Netw Open 4(2):e210247

Ugowe FE, Jackson LR, Thomas KL (2018) Racial and ethnic differences in the prevalence, management, and outcomes in patients with atrial fibrillation: A systematic review. Heart Rhythm 15(9):1337–1345

Xu Y, Siegal DM, Anand SS (2021) Ethnoracial variations in venous thrombosis: Implications for management, and a call to action. J Thromb Haemost 19(1):30–40

Erinne I et al (2021) Racial disparities in the treatment of aortic stenosis: Has transcatheter aortic valve replacement bridged the gap? Catheter Cardiovasc Interv 98(1):148–156

Ali A et al (2022) Racial and Ethnic Disparities in the Use of Transcatheter Aortic Valve Replacement in the State of Connecticut. Cardiovasc Revasc Med 37:7–12

Pienta MJ et al (2023) Racial disparities in mitral valve surgery: A statewide analysis. J Thorac Cardiovasc Surg 165(5):1815-1823.e8

Alkhouli M et al (2019) Racial Disparities in the Utilization and Outcomes of Structural Heart Disease Interventions in the United States. J Am Heart Assoc 8(15):e012125

Heckbert SR et al (2020) Differences by Race/Ethnicity in the Prevalence of Clinically Detected and Monitor-Detected Atrial Fibrillation: MESA. Circ Arrhythm Electrophysiol 13(1):e007698

Zakai NA et al (2014) Racial and regional differences in venous thromboembolism in the United States in 3 cohorts. Circulation 129(14):1502–1509

Lamprea-Montealegre JA et al (2021) Valvular Heart Disease in Relation to Race and Ethnicity: JACC Focus Seminar 4/9. J Am Coll Cardiol 78(24):2493–2504

Tamargo J et al (2022) Racial and ethnic differences in pharmacotherapy to prevent coronary artery disease and thrombotic events. Eur Heart J Cardiovasc Pharmacother 8(7):738–751

Kanuri SH, Kreutz RP (2019) Pharmacogenomics of Novel Direct Oral Anticoagulants: Newly Identified Genes and Genetic Variants. J Pers Med 9(1):7

Craig TJ, Wann LS, Calkins H, Chen LY, Cigarroa JE, Cleveland CJ Jr, Ellinor PT, Ezekowitz MD, Field ME, Furie LK, Heidenreich PA, Murray KT, Shea JB, Tracy CM, Yancy CW (2019) Circulation 140 (2):e125–e151. https://doi.org/10.1161/CIR.000000000000066

Ortel TL et al (2020) American Society of Hematology 2020 guidelines for management of venous thromboembolism: treatment of deep vein thrombosis and pulmonary embolism. Blood Adv 4(19):4693–4738

Article   CAS   PubMed   PubMed Central   Google Scholar  

Stevens SM et al (2021) Antithrombotic Therapy for VTE Disease: Second Update of the CHEST Guideline and Expert Panel Report. Chest 160(6):e545–e608

Article   CAS   PubMed   Google Scholar  

Schünemann HJ et al (2018) American Society of Hematology 2018 guidelines for management of venous thromboembolism: prophylaxis for hospitalized and nonhospitalized medical patients. Blood Adv 2(22):3198–3225

Otto CM et al (2021) 2020 ACC/AHA Guideline for the Management of Patients With Valvular Heart Disease: A Report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice Guidelines. Circulation 143(5):e72–e227

PubMed   Google Scholar  

National Action Plan for Adverse Drug Event Prevention (2014) Office of Disease Prevention and Health Promotion. US Department of Health and Human Services, Washington, DC

Google Scholar  

Bhakta S et al (2022) A systematic review and meta-analysis of racial disparities in deep vein thrombosis and pulmonary embolism events in patients hospitalized with coronavirus disease 2019. J Vasc Surg Venous Lymphat Disord 10(4):939-944.e3

Corbie-Smith G et al (2008) Conceptualizing race in research. J Natl Med Assoc 100(10):1235–1243

Vazquez SR, Yates NYY, McFarland MM (2022). Differences in anticoagulant care delivery according to ethnoracial group in the United States: a scoping review protocol. Open Sci Fram osf.io/eydsa. https://doi.org/10.17605/OSF.IO/9SE7H

Peters MDJ, Godfrey C, McInerney P, Munn Z, Tricco AC, Khalil H (2020) Joanna briggs institute reviewer's manual. Aromataris E, Munn Z (eds). Publisher: Joanna briggs institute. Chapter: chapter 11: scoping reviews

Arksey H, O’Malley L (2005) Scoping studies: towards a methodological framework. Int J Soc Res Methodol 8(1):19–32

Article   Google Scholar  

Rethlefsen ML et al (2021) PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev 10(1):39

Harris PA et al (2009) Research electronic data capture (REDCap)–a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 42(2):377–381

McGowan J et al (2016) PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol 75:40–46

Kearon C et al (2016) Antithrombotic Therapy for VTE Disease: CHEST Guideline and Expert Panel Report. Chest 149(2):315–352

Lip GYH et al (2018) Antithrombotic Therapy for Atrial Fibrillation: CHEST Guideline and Expert Panel Report. Chest 154(5):1121–1201

al Taii, H. and S. Al-Kindi, Racial disparities in prescriptions of oral anticoagulants among patients with atrial fibrillation., in 24th International Atrial Fibrillation Symposium (2019) Boston. MA, USA

Bhave PD et al (2015) Race- and sex-related differences in care for patients newly diagnosed with atrial fibrillation. Heart Rhythm 12(7):1406–1412

Chapman SA et al (2017) Adherence to treatment guidelines: the association between stroke risk stratified comparing CHADS. BMC Health Serv Res 17(1):127

Chen Q et al (2021) Prevalence and the factors associated with oral anticoagulant use among nursing home residents. J Clin Pharm Ther 46(6):1714–1728

Deitelzweig S et al (2020) Utilization of anticoagulants and predictors of treatment among hospitalized patients with atrial fibrillation in the USA. J Med Econ 23(12):1389–1400

Essien UR et al (2018) Association of Race/Ethnicity With Oral Anticoagulant Use in Patients With Atrial Fibrillation: Findings From the Outcomes Registry for Better Informed Treatment of Atrial Fibrillation II. JAMA Cardiol 3(12):1174–1182

Essien UR et al (2021) Disparities in Anticoagulant Therapy Initiation for Incident Atrial Fibrillation by Race/Ethnicity Among Patients in the Veterans Health Administration System. JAMA Netw Open 4(7):e2114234

Essien UR, Kim N, Hausmann LR, Mor MK, Good C, Gellad WF, Fine MJ (2020) Abstract: racial and ethnic disparities in anticoagulant choice for atrial fibrillation in the veterans health administration: results from the reach-af study. J Gen Int Med J 35(1):S250–251

Essien UR et al (2022) Association of Race and Ethnicity and Anticoagulation in Patients With Atrial Fibrillation Dually Enrolled in Veterans Health Administration and Medicare: Effects of Medicare Part D on Prescribing Disparities. Circ Cardiovasc Qual Outcomes 15(2):e008389

Essien UR et al (2020) Race/Ethnicity and Sex-Related Differences in Direct Oral Anticoagulant Initiation in Newly Diagnosed Atrial Fibrillation: A Retrospective Study of Medicare Data. J Natl Med Assoc 112(1):103–108

Fohtung RB, Novak E, Rich MW (2017) Effect of New Oral Anticoagulants on Prescribing Practices for Atrial Fibrillation in Older Adults. J Am Geriatr Soc 65(11):2405–2412

Lewis WR et al (2011) Improvement in use of anticoagulation therapy in patients with ischemic stroke: results from Get With The Guidelines-Stroke. Am Heart J 162(4):692-699.e2

Martin Diaz C et al (2021) Anticoagulation After Ischemic Stroke or Transient Ischemic Attack (TIA) in the Time of Direct Oral Anticoagulation (DOAC) and Thrombectomy. Cureus 13(8):e17392

PubMed   PubMed Central   Google Scholar  

Mentias A et al (2021) Racial and Sex Disparities in Anticoagulation After Electrical Cardioversion for Atrial Fibrillation and Flutter. J Am Heart Assoc 10(17):e021674

Obi CA et al (2022) Examination of anticoagulation prescription among elderly patients with atrial fibrillation after in-hospital fall. J Thromb Thrombolysis 53(3):683–689

Piccini JP et al (2019) Adherence to Guideline-Directed Stroke Prevention Therapy for Atrial Fibrillation Is Achievable. Circulation 139(12):1497–1506

Raji MA et al (2013) National utilization patterns of warfarin use in older patients with atrial fibrillation: a population-based study of Medicare Part D beneficiaries. Ann Pharmacother 47(1):35–42

Schwartz SM et al (2019) Discriminative Ability of CHA. Am J Cardiol 123(12):1949–1954

Shahid I et al (2021) Meta-Analysis of Racial Disparity in Utilization of Oral Anticoagulation for Stroke Prevention in Atrial Fibrillation. Am J Cardiol 153:147–149

Tedla YG et al (2020) Racial Disparity in the Prescription of Anticoagulants and Risk of Stroke and Bleeding in Atrial Fibrillation Patients. J Stroke Cerebrovasc Dis 29(5):104718

Thomas KL et al (2013) Racial differences in the prevalence and outcomes of atrial fibrillation among patients hospitalized with heart failure. J Am Heart Assoc 2(5):e000200

Waddy SP et al (2020) Racial/Ethnic Disparities in Atrial Fibrillation Treatment and Outcomes among Dialysis Patients in the United States. J Am Soc Nephrol 31(3):637–649

Wetmore JB et al (2019) Relation of Race, Apparent Disability, and Stroke Risk With Warfarin Prescribing for Atrial Fibrillation in Patients Receiving Maintenance Hemodialysis. Am J Cardiol 123(4):598–604

Winkelmayer WC et al (2012) Prevalence of atrial fibrillation and warfarin use in older patients receiving hemodialysis. J Nephrol 25(3):341–353

Zahuranec DB et al (2017) Stroke Quality Measures in Mexican Americans and Non-Hispanic Whites. J Health Dispar Res Pract 10(1):111–123

Addo-Tabiri NO et al (2020) Black Patients Experience Highest Rates of Cancer-associated Venous Thromboembolism. Am J Clin Oncol 43(2):94–100

Freeman AH et al (2016) Venous thromboembolism following minimally invasive surgery among women with endometrial cancer. Gynecol Oncol 142(2):267–272

Friedman AM et al (2013) Underuse of postcesarean thromboembolism prophylaxis. Obstet Gynecol 122(6):1197–1204

Lau BD et al (2015) Eliminating Health Care Disparities With Mandatory Clinical Decision Support: The Venous Thromboembolism (VTE) Example. Med Care 53(1):18–24

Owodunni OP et al (2020) Using electronic health record system triggers to target delivery of a patient-centered intervention to improve venous thromboembolism prevention for hospitalized patients: Is there a differential effect by race? PLoS ONE 15(1):e0227339

Shah S et al (2021) Implementation of an Anticoagulation Practice Guideline for COVID-19 via a Clinical Decision Support System in a Large Academic Health System and Its Evaluation: Observational Study. JMIR Med Inform 9(11):e30743

Steinberg BA et al (2013) Early adoption of dabigatran and its dosing in US patients with atrial fibrillation: results from the outcomes registry for better informed treatment of atrial fibrillation. J Am Heart Assoc 2(6):e000535

Vaughan Sarrazin MS et al (2014) Bleeding rates in Veterans Affairs patients with atrial fibrillation who switch from warfarin to dabigatran. Am J Med 127(12):1179–1185

Nathan AS et al (2019) Racial, Ethnic, and Socioeconomic Inequities in the Prescription of Direct Oral Anticoagulants in Patients With Venous Thromboembolism in the United States. Circ Cardiovasc Qual Outcomes 12(4):e005600

Singh BP, Sitlinger A, Saber I, Thames E, Beckman M, Reyes N, Schulteis RD, Ortel T (2016) Direct oral anticoagulant usage in the treatment of venous thromboembolism across racial groups in Durham County, NC. J Gen Int Med 31:S191–S192

Schaefer JK et al (2017) Sociodemographic factors in patients continuing warfarin vs those transitioning to direct oral anticoagulants. Blood Adv 1(26):2536–2540

Lank RJ et al (2019) Ethnic Differences in 90-Day Poststroke Medication Adherence. Stroke 50(6):1519–1524

O'Brien EC, Holmes ND, Thomas L, Fonarow GC, Kowey PR, Ansell JE, Mahaffey KW, Gersh BJ, Peterson ED, Piccini JP, Hylek EM (2018) J Am Heart Assoc 7(12):e006391

Singh RR et al (2016) Adherence to Anticoagulant Therapy in Pediatric Patients Hospitalized With Pulmonary Embolism or Deep Vein Thrombosis: A Retrospective Cohort Study. Clin Appl Thromb Hemost 22(3):260–264

Yong C, Xu XY, Than C, Ullal A, Schmitt S, Azarbal F, Heidenreich P, Turakhia M (2013) Abstract 14134: racial disparities in warfarin time in INR therapeutic range in patients with atrial fibrillation: Findings from the TREAT-AF Study. Circul 128(22).  https://www.ahajournals.org/doi/10.1161/circ.128.suppl_22.A14134

Chen N, Brooks MM, Hernandez I (2020) Latent Classes of Adherence to Oral Anticoagulation Therapy Among Patients With a New Diagnosis of Atrial Fibrillation. JAMA Netw Open 3(2):e1921357

Pham Nguyen TP et al (2003) Does hospitalization for thromboembolism improve oral anticoagulant adherence in patients with atrial fibrillation? J Am Pharm Assoc 60(6):986-992.e2

Patel AA et al (2013) Persistence of warfarin therapy for residents in long-term care who have atrial fibrillation. Clin Ther 35(11):1794–1804

Golwala H et al (2016) Racial/ethnic differences in atrial fibrillation symptoms, treatment patterns, and outcomes: Insights from Outcomes Registry for Better Informed Treatment for Atrial Fibrillation Registry. Am Heart J 174:29–36

Limdi NA et al (2017) Quality of anticoagulation control and hemorrhage risk among African American and European American warfarin users. Pharmacogenet Genomics 27(10):347–355

Lip GY et al (2016) Determinants of Time in Therapeutic Range in Patients Receiving Oral Anticoagulants (A Substudy of IMPACT). Am J Cardiol 118(11):1680–1684

Rao SR et al (2015) Explaining racial disparities in anticoagulation control: results from a study of patients at the Veterans Administration. Am J Med Qual 30(3):214–222

Thigpen JL, Yah Q, Beasley M, Limdi NA (2012) Abstract: racial differences in anticoagulation control and risk of hemorrhage among warfarin users. Pharmacoth 32(10):E189

Yong C et al (2016) Racial Differences in Quality of Anticoagulation Therapy for Atrial Fibrillation (from the TREAT-AF Study). Am J Cardiol 117(1):61–68

Moffett BS, Kim S, Bomgaars LR (2013) Readmissions for warfarin-related bleeding in pediatric patients after hospital discharge. Pediatr Blood Cancer 60(9):1503–1506

Rose AJ et al (2013) Gaps in monitoring during oral anticoagulation: insights into care transitions, monitoring barriers, and medication nonadherence. Chest 143(3):751–757

Mahle WT et al (2011) Management of warfarin in children with heart disease. Pediatr Cardiol 32(8):1115–1119

Meade M et al (2011) Impact of health disparities on staff workload in pharmacist-managed anticoagulation clinics. Am J Health Syst Pharm 68(15):1430–1435

Aguilar F et al (2021) Off-label direct oral anticoagulants dosing in atrial fibrillation and venous thromboembolism is associated with higher mortality. Expert Rev Cardiovasc Ther 19(12):1119–1126

Jellinek-Cohen SP, Li M, Husk G (2018) Enoxaparin dosing errors in the emergency department. World J Emerg Med 9(3):195–202

Triller DM et al (2015) Trends in Warfarin Monitoring Practices Among New York Medicare Beneficiaries, 2006–2011. J Community Health 40(5):845–854

Akhtar T et al (2020) Factors associated with bleeding events in patients on rivaroxaban for non-valvular atrial fibrillation: A real-world experience. Int J Cardiol 320:78–82

Cires-Drouet RS et al (2022) Prevalence and clinical outcomes of hospitalized patients with upper extremity deep vein thrombosis. J Vasc Surg Venous Lymphat Disord 10(1):102–110

Doucette K et al (2020) Efficacy and Safety of Direct-Acting Oral Anticoagulants (DOACs) in the Overweight and Obese. Adv Hematol 2020:3890706

Gu K et al (2021) Racial disparities among Asian Americans with atrial fibrillation: An analysis from the NCDR® PINNACLE Registry. Int J Cardiol 329:209–216

Yamashita Y et al (2018) Asian patients versus non-Asian patients in the efficacy and safety of direct oral anticoagulants relative to vitamin K antagonist for venous thromboembolism: A systemic review and meta-analysis. Thromb Res 166:37–42

Di Nisio M et al (2016) Risk of major bleeding in patients with venous thromboembolism treated with rivaroxaban or with heparin and vitamin K antagonists. Thromb Haemost 115(2):424–432

Hernandez I et al (2015) Risk of bleeding with dabigatran in atrial fibrillation. JAMA Intern Med 175(1):18–24

Majeed A et al (2016) Bleeding events with dabigatran or warfarin in patients with venous thromboembolism. Thromb Haemost 115(2):291–298

Hankey GJ et al (2014) Intracranial hemorrhage among patients with atrial fibrillation anticoagulated with warfarin or rivaroxaban: the rivaroxaban once daily, oral, direct factor Xa inhibition compared with vitamin K antagonism for prevention of stroke and embolism trial in atrial fibrillation. Stroke 45(5):1304–1312

Kobayashi L et al (2017) Novel oral anticoagulants and trauma: The results of a prospective American Association for the Surgery of Trauma Multi-Institutional Trial. J Trauma Acute Care Surg 82(5):827–835

Gencer B et al (2022) Edoxaban versus Warfarin in high-risk patients with atrial fibrillation: A comprehensive analysis of high-risk subgroups. Am Heart J 247:24–32

Chen N et al (2021) Joint Latent Class Analysis of Oral Anticoagulation Use and Risk of Stroke or Systemic Thromboembolism in Patients with Atrial Fibrillation. Am J Cardiovasc Drugs 21(5):573–580

Kabra R et al (2015) Effect of race on outcomes (stroke and death) in patients >65 years with atrial fibrillation. Am J Cardiol 116(2):230–235

Kim MH, Xu L, Puckrein G (2018) Patient Diversity and Population Health-Related Cardiovascular Outcomes Associated with Warfarin Use in Atrial Fibrillation: An Analysis Using Administrative Claims Data. Adv Ther 35(11):2069–2080

Abu-Zeinah G, Oromendia C, DeSancho MT (2019) Thrombotic risk factors in patients with antiphospholipid syndrome: a single center experience. J Thromb Thrombolysis 48(2):233–239

Banala SR et al (2017) Discharge or admit? Emergency department management of incidental pulmonary embolism in patients with cancer: a retrospective study. Int J Emerg Med 10(1):19

LaDuke ZJ et al (2019) Association of mortality among trauma patients taking preinjury direct oral anticoagulants versus vitamin K antagonists. Surgery 166(4):564–571

Moreland CJ et al (2013) Anticoagulation education: do patients understand potential medication-related emergencies? Jt Comm J Qual Patient Saf 39(1):22–31

Bamgbade BA et al (2021) Differences in Perceived and Predicted Bleeding Risk in Older Adults With Atrial Fibrillation: The SAGE-AF Study. J Am Heart Assoc 10(17):e019979

Webb D et al (2019) Patient Satisfaction With Venous Thromboembolism Treatment. Clin Appl Thromb Hemost 25:1076029619864663

Kamath CC et al (2021) Cost Conversations About Anticoagulation Between Patients With Atrial Fibrillation and Their Clinicians: A Secondary Analysis of a Randomized Clinical Trial. JAMA Netw Open 4(7):e2116009

U.S. Census Bureau . [cited 2023 10/20/23]; Available from: https://www.census.gov/quickfacts/fact/note/US/RHI625222#:~:text=Asian.,Japan%2C%20Korea%2C%20or%20Vietnam .

Fain KM et al (2021) Race and ethnicity reporting for clinical trials in ClinicalTrials.gov and publications. Contemp Clin Trials 101:106237

American Psychological Association Ethnic and racial minorities & socioeconomic status. 3/22/24]; Available from: https://www.apa.org/pi/ses/resources/publications/minorities#:~:text=The%20relationship%20between%20SES%2C%20race,SES%2C%20race%2C%20and%20ethnicity . Accessed 22 Mar 2024

Braveman PA et al (2010) Socioeconomic disparities in health in the United States: what the patterns tell us. Am J Public Health 100 Suppl 1(Suppl 1):186–96

Wheelock KM et al (2021) Clinician Trends in Prescribing Direct Oral Anticoagulants for US Medicare Beneficiaries. JAMA Netw Open 4(12):e2137288

Barnes GD et al (2015) National Trends in Ambulatory Oral Anticoagulant Use. Am J Med 128(12):1300–5.e2

Iyer GS et al (2023) Trends in the Use of Oral Anticoagulants for Adults With Venous Thromboembolism in the US, 2010–2020. JAMA Netw Open 6(3):e234059

MACRA Medicare Access and CHIP Reauthorization Act of 2015 (MACRA). 03/19/24]; Available from: https://www.cms.gov/medicare/quality/value-based-programs/chip-reauthorization-act . Accessed 19 Mar 2024

Blueprint Measure Lifecycle Overview. 03/19/2024]; Available from: https://mmshub.cms.gov/blueprint-measure-lifecycle-overview . Accessed 19 Mar 2024

Endorsement and Maintenance (E&M) Guidebook. 3/19/24]; Available from: https://p4qm.org/sites/default/files/2023-12/Del-3-6-Endorsement-and-Maintenance-Guidebook-Final_0_0.pdf . Accessed 19 Mar 2024

Measure Implementation. 3/19/24]; Available from: https://mmshub.cms.gov/measure-lifecycle/measure-implementation/selection . Accessed 19 Mar 2024

Download references

Acknowledgements

The authors wish to acknowledge the following individuals for their work in screening articles for this scoping review: April Allen, PharmD, CACP; Allison Burnett, PharmD, PhC, CACP; Stacy Ellsworth, RN, MSN, CCRC; Danielle Jenkins, MBA, RN, BSN, CRNI; Amanda Katz, MBA; Lea Kistenmacher, Julia Mulheman, PharmD; Surhabi Palkimas, PharmD, MBA; Terri Schnurr, RN, CCRC; Deborah Siegal, MD, MSc, FRCPC; Kimberly Terry, PharmD, BCPS, BCCCP; and Terri Wiggins, MS.

The authors wish to acknowledge the support of the Anticoagulation Forum in the development of this manuscript. The Anticoagulation Forum is a non-profit organization dedicated to improving the quality of care for patients taking antithrombotic medications.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and affiliations.

University of Utah Health Thrombosis Service, 6056 Fashion Square Drive, Suite 1200, Murray, UT, 84107, USA

Sara R. Vazquez

Kaiser Permanente Clinical Pharmacy Services, 200 Crescent Center Pkwy, Tucker, GA, 30084, USA

Naomi Y. Yates

Anticoagulation Forum, Inc, 17 Lincoln Street, Suite 2B, Newton, MA, 02461, USA

Craig J. Beavers & Darren M. Triller

University of Kentucky College of Pharmacy, 789 S Limestone, Lexington, KY, 40508, USA

Craig J. Beavers

University of Utah Spencer S. Eccles Health Sciences Library, 10 N 1900 E, Salt Lake City, UT, 84112, USA

Mary M. McFarland

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation was performed by Sara Vazquez, Naomi Yates, and Mary McFarland. Data collection and analysis were performed by Sara Vazquez, Naomi Yates, Craig Beavers, and Darren Triller. The first draft of the manuscript was written by Sara Vazquez and all authors edited subsequent drafts. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Sara R. Vazquez .

Ethics declarations

Competing interests.

Dr. Vazquez discloses that she is a member of the Anticoagulation Forum Advisory Council and an editorial consultant for UptoDate, Inc.

Dr. Yates has no conflicts of interest to disclose.

Dr. Beavers has no conflicts of interest to disclose.

Dr. Triller has no conflicts of interest to disclose.

Ms. McFarland has no conflicts of interest to disclose.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 22 KB)

Supplementary file2 (docx 14 kb), rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Vazquez, S.R., Yates, N.Y., Beavers, C.J. et al. Differences in quality of anticoagulation care delivery according to ethnoracial group in the United States: A scoping review. J Thromb Thrombolysis (2024). https://doi.org/10.1007/s11239-024-02991-2

Download citation

Accepted : 27 April 2024

Published : 11 May 2024

DOI : https://doi.org/10.1007/s11239-024-02991-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Ethnoracial
• Disparities
• Anticoagulation
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 09 May 2024

Designing an evaluation tool for evaluating training programs of medical students in clinical skill training center from consumers’ perspective

• Rezvan Azad 1 ,
• Mahsa Shakour 2 &
• Narjes Moharami 2  

BMC Medical Education volume  24 , Article number:  502 ( 2024 ) Cite this article

170 Accesses

Metrics details

Introduction

The Clinical Skill Training Center (CSTC) is the first environment where third year medical students learn clinical skills after passing basic science. Consumer- based evaluation is one of the ways to improve this center with the consumer. This study was conducted with the aim of preparing a consumer-oriented evaluation tool for CSTC among medical students.

The study was mixed method. The first phase was qualitative and for providing an evaluation tool. The second phase was for evaluating the tool. At the first phase, after literature review in the Divergent phase, a complete list of problems in the field of CSTC in medicine schools was prepared. In the convergent step, the prepared list was compared with the standards of clinical education and values of scriven. In the second phase it was evaluated by the scientific and authority committee. Validity has been measured by determining CVR and CVI: Index. The face and content validity of the tool was obtained through the approval of a group of specialists.

The findings of the research were in the form of 4 questionnaires: clinical instructors, pre-clinical medical students, and interns. All items were designed as a 5-point Likert. The main areas of evaluation included the objectives and content of training courses, implementation of operations, facilities and equipment, and the environment and indoor space. In order to examine the long-term effects, a special evaluation form was designed for intern.

The tool for consumer evaluation was designed with good reliability and trustworthiness and suitable for use in the CSTC, and its use can improve the effectiveness of clinical education activities.

Peer Review reports

Mastering clinical skills is one of the essential requirements for becoming a physician and pre-clinical courses play an important role in forming these clinical skills in medical students. The importance of these courses is such that a Clinical Skill Training Center (CSTC) has been formed especially for this purpose, which is nowadays used for training pre-clinical skills and some of the more advanced procedures such as operating room simulation [ 1 ]. The CSTC is an educational environment where students can use the available resources and the supervision of experienced faculty members to be introduced to clinical skills, train and gain experience in these skills and receive immediate feedback to resolve their mistakes and shortcomings [ 2 ]. The aim of the student’s participation in this center is the training of students who have sufficient theoretical knowledge but lack the necessary skills for working in the clinical setting. Therefore, this center supports students in the acquisition, maintenance and improvement of their clinical medical skills [ 3 ]. In this center, students can learn and repeat treatment procedures in a safe environment without severe consequences which reduces their stress and allows them to train and learn [ 4 ]. In this study, medical students attend this center for the first time after the end of theoretical course and before entering the hospital for the first time and Preliminary learn practical medical skills such as performing a variety of examinations and history taking. Then, in externship and internship, they can practice more advanced courses such as cardiopulmonary resuscitation, dressing and stitches etc. in small groups.

The importance of these centers like CSTCs is the fact that learning a large number of practical and communicational skills related to theoretical knowledge is one of the essential characteristics of medical education and can play an important role in the future careers of the students and training of specialized human resources in the field of medicine and healthcare [ 4 ]. However, one of the important matters in clinical training is the quality of education which can directly affect the quality of healthcare services provided to society. The quality of education is, in turn, affected by the details of the educational programs. Therefore, the evaluation of educational programs can play an important role in providing quality equations. In other words, using suitable evaluation mechanisms creates the requirements for performance transparency and accountability in the clinical education system in medical education [ 5 ]. Observing the principles of evaluation can also help determine the shortcomings and programs in educational programs [ 2 ]. However, the evaluation of educational programs is often faced with difficulties. Evaluations conducted to ensure the suitable quality of education for medical students must determine whether the students have achieved acceptable clinical standards which is only possible through careful evaluation of their training programs [ 2 ].

There are various problems concerning evaluation tools. The faculty members in medicine are still faced with challenges concerning the improvement of evaluation tools and the creation of tools for evaluating factors which are hard to quantify or qualify, such as professionalism, group work and expertise [ 6 ].

Despite various theories regarding evaluation, the lack of credible and valid evaluation tools for educational programs is still being felt [ 7 ]. Using suitable evaluation tools can create an overview of the current situation of the training programs based on the quality factors of the curriculum and can be used as a guideline for decision-making, planning, faculty development and improving the quality of education [ 8 ]. Perhaps the most important value of a suitable evaluation tool for training programs is providing a clear picture and operational and measurable measures regarding the implementation of educational programs. Furthermore, after completion, such a tool can be used as a constant interventional screening tool by academic groups, faculty members and authorities in practical training programs.

The consumer-oriented model advocated by evaluation expert and philosopher Michael Scriven. This model of evaluation like other models, is to make a value judgment about the quality of a program, product, or policy in order to determine its value, merit, or importance, but in this model, the value judgment is based on the level of satisfaction and usefulness of the curriculum for the consumers of the program. It is achieved and the evaluator considers himself to be responsive to their needs and demands. The models that are included in this approach have paid more attention to their responsibility towards the consumers of curriculum and educational programs.it is an exercise in value-free measurement of whether program goals were achieved [ 9 , 10 ].

The current study aims to design an evaluation tool for training programs in the CSTC based on consumers’ perspectives and assess its validity and reliability to facilitate the evaluation of educational programs and help improve the practical skills of medical students. Therefore, the prepared evaluation tool not only can be used for continuous improvement of educational equality but can also be used for validation of educational programs.

Subjects and methods

The study was mixed method with triangulation approach. This was a developmental study for developing an evaluation tool for educational programs of the CSTC in medicine schools from consumers’ perspective using data gathered through qualitative study, descriptive – survey study and from many resources. The study was done in 2020 until 2022 and in Arak University of Medical Sciences. Samples were students in different level, and clinical teachers who are consumers and main stakeholders. This study included two main phases.

The first phase was qualitative. Samples were literature and 10 experts. Sampling was purposeful. This phase was for decision-making regarding factors used for evaluating the educational programs of the CSTC. In this phase and to create a deep understanding of the topic, the literature related to the subject matter was reviewed. The reviewed literature related to evaluation was based on the consumers’ perspective evaluation and questionnaire preparation method. Then, using the Scriven consumer opinion questionnaire, standards for CSTC, and the available literature, interviews were conducted with experts and stakeholders in the CSTC. These interviews aimed to prepare a comprehensive list of problems, and concerns related to the educational programs at the clinical skill training center which the evaluation tool aimed to answer. This stage was known as the divergent stage where the topics discussed in the interviews included educational goals, content, equipment, educational processes, the environment and physical location. Some of the questions asked in this stage included “What is the level of achieving educational goals among students in the current program?”, “How effective is the practical program of the center in improving the clinical skills of the students?”, “Does the center has access to sufficient tools and equipment for completing its educational program?” and “what are the long-term effects of CSTC’s educational program?”

In the next step, known as the convergent step, the list prepared in the previous stage was combined with the educational standards for CSTCs provided by the deputy of education, ministry of health as well as Scriven criteria. The results were then carefully assessed by a scientific and authority committee consisting of the Educational Deputy of Clinical Education of the Faculty of Medicine, Director of Educational Affairs of the Faculty of Medicine, Director of Clinical Skills Training Center and Curriculum, Expert of Clinical Skills Center and Bachelor of Technical Affairs of Clinical Skills Training Center in the Faculty of Medicine of Arak University of Medical Sciences. The questionnaire items were selected based on the importance and evaluation criteria. The data gathering tool was prepared after determining the evaluation questions, data gathering sources and designing the evaluation method. Customers in this study were clinical training faculty members and medical students (externship, pre-clinical and internship students). Therefore, we designed four questionnaires with special questions. Every questionnaire is designed in 5 domains (Learning objectives and course content, Equipment and tools, Educational processes, Environment and physical location).

The second phase was quantitative and it was survey. Samples were professors and who were experts in subject and medical students (externship, pre-clinical and internship students). Sampling was conventional and purposeful. 10 faculty members and 71 students were selected. This phase was for measuring the questionnaire’s face and content validity. The validity was measured using Content Validity Ratio (CVR) and Content Validity Index (CVI) using Lawshe’s method. In this method, the opinion of experts in the field concerning the questionnaire content is used to calculate these factors [ 11 ]. A total of 10 faculty members participated in the validity survey and including faculty members from specialty fields of medical education, gynecology, infectious diseases, emergency medicine, pediatric medicine, nursing and midwifery. After explaining the research goals to the participants and providing them with the operational definitions related to the contents of the items, they were asked to mark each item in a table using a three-part Likert scale using “essential”, “useful not nonessential” and “nonessential” scores. Then, Content Validity Ratio was calculated using the following equations. CVR= \(\frac{\varvec{n}\varvec{e}-\varvec{n}/2}{\varvec{n}/2}\) . In this equation, n is the total number of experts, and n e is the number of experts who have selected the “essential” score. Using the CVR table, the minimum CVT value for accepting an item based on the participants’ opinions was set at 0.62.

After calculating CVR, the method proposed by Waltz & Bausell was used for determining the CVI. To this end, a CVI evaluation table was prepared for the items using a four-part scale including “unrelated”, “requiring major revision”, “requiring minor revision” and “relevant” scores and delivered to the 10 participating experts who were asked to provide their opinions regarding each item. Then, the CVI value was calculated for each item by dividing the total number of “requiring minor revision” and “relevant” answers by the total number of experts. The items with CVI values higher than 0.79 were accepted [ 11 , 12 ]. The reliability of the questionnaire was determined with emphasis on internal correlation with the help of SPSS software and was higher than 0.8, which confirmed the suitable reliability of the questionnaire. A panel of experts then conducted a qualitative review of the items, edited their grammar, and modified unclear statements based on the research goals. In general, the entire phrase should have been accepted by the majority of the panel based on simplicity, clarity and lack of ambiguity. The face validity was also calculated by scoring the effect of each item on the questionnaire. This score was then used to eliminate phrases with scores lower than 1.5. After evaluating the face validity, Content Validity Ratio (CVR) was calculated by the experts and items with CVR values less than the threshold value were selected and eliminated. After that, we used this tool by 71 students and 11 teachers to assess reliability according to Cronbach’s alpha.

The results of the current study indicate that according to the faculty members and experts participating in this study, the evaluation of educational programs of clinical skill training centers includes evaluation of programs in regards to goal and content, educational processes, equipment and tools, and environment and physical location. After interviews with clinical training experts and a review of relevant literature, 4 separate questionnaires were developed for clinical training faculty members, pre-clinical students, internship students, and externship students. All experts as samples answered all questions for validity and 71 students of 90 students completely answered the questionnaires.

The questionnaire for faculty members included 35 items (Table  1 ), the one for interns included 6 items (Table  2 ), the externship students’ questionnaire included 29 (Table  3 ) items and the questionnaire for pre-clinical students included 41 items (Table  4 ). All items were designed for scoring using a 5-point Likert system (very low, low, average, high, very high).

The face validity of questionnaires was evaluated using qualitative and quantitative approaches. Among 117 items in 4 questionnaires, 6 items didn’t have suitable content validity (CVR < 0.62) which were eliminated according to the following table (Table  5 ). 111 items had CVR ≥ 0.62 and the results of the CVI assessment indicated that all items were acceptable.

The reliability of the questionnaires was investigated using Cronbach’s Alpha with emphasis on internal correlation with the help of SPSS software as presented in the following table, which confirms the reliability of the questionnaires (Table  6 ). The reliability in all questionnaire was more than %83. Therefore, all items received acceptable reliability and validity scores.

In the current study, a comprehensive researcher-made questionnaire was prepared based on the opinions of experts and curriculum designers while considering all relevant resources and literature which is a unique tool in Iran regarding the expansiveness of the scope. The prepared tool was then used to evaluate the activities of the clinical skills training center in 5 domains (1) program goals and content, (2) tools and equipment, (3) educational processes, (4) environment and physical location and (5) long-term effects of the curriculum.

The first part of the evaluation tool prepared in the current study aims to assess the objective goals of program according to the consumer’s views. CSTC is suitable for training basic and practical skills which are often neglected due to time constraints during the students’ presence in clinical environments [ 6 ]. The factors investigated in this area using the current tool included basic skills such as patient interview, basic resuscitation, clinical examination, practical clinical activities, interpretation of essential clinical findings, prescription skills and patient management. Other studies have also investigated similar factors. For example, Imran et al. (2018) in their study evaluated the attitude of students towards this center and stated that participation in Skill Lab sessions in the pre-clinical years will assist students in their clinical year to achieve better overall performance, as well as better communication skills and self-esteem [ 1 ]. According to previous studies, the majority of students preferred participation in pre-clinical straining in these centers due to the advantages of skill labs for learning clinical skills [ 3 ]. Another study showed that the majority of students prefer participation in skill lab for learning essential clinical skills such as venous blood sampling, catheterization, endotracheal intubation, listening to respiratory sounds, genital examination, etc. compared to directly performing these procedures on patients [ 2 ]. The designed tools in current study evaluated some of these learning objectives. But because of evaluating 5 domains and many questions in every domain, we summarized them to be user friendly. Every questionnaire had some question for objectives that questionnaire respondents as customers (faculty members and medical students) could reply them.

The second part of this evaluation tool is for assessing educational tools such as educational mannequins and models, medical examination devices (Stethoscope, sphygmomanometer, otoscope and ophthalmoscope), medical consumables, audio-visual equipment and information technology facilities. According to the studies, a common factor in CSTCs is access to a wide range of tools in each university as well as using updated technologies for education. These innovations have even resulted in the improved academic ranking of some colleges and medical universities in the world [ 12 ]. The quality of these educational tools is the other important item in many studies [ 13 ]. The quality for mannequin is depended to fidelity. Brydges et al. in his study showed that higher fidelity causes more learning and less time for learning. They suggested that clinical curricula incorporate exposure to multiple simulations to maximize educational [ 14 ].

The third part of this tool is educational processes consisted of evaluating factors such as the length and number of workshops, the effect of CSTC on teaching in a clinical environment, the effect of the center on increasing the motivation and interest in clinical topics, use of volunteer patients and actors and use of modern teaching and assessment methods. This area evaluates the educational process as an important part of clinical training. The importance of this area is also confirmed in other studies. CSTC enables students, including interns and new students, to practice procedures without fearing the consequences. Furthermore, there is also no time of ethical constraints in these practices, enabling the students to be trained in treatment procedures and physical examinations which can be dangerous or painful for the patient [ 2 ]. In this regard, the standardized patient is one of the popular methods used in universities around the world. For example, the University of Massachusetts had been using standardized patients as an education and assessment tool and even as clinical trainers for more than 20 years [ 8 ]. Another example is the simulation center of Grand Valley State University, which provides significant tools for the management of standardized patients, including registration and deployment of standardized patients as needed. This center has designed a website for the registration of standardized patients, which allows individuals to register based on certain criteria, before being trained and deployed according to the protocols [ 8 ].

The effectiveness of clinical skill training centers on motivation was presented in a study by Hashim et al. (2016) on the effects of clinical skill training centers on medical education. According to the results of this study, 84 to 89 per cent of students believed that these centers increase the motivation for medical education as well as interest in learning clinical skills [ 3 ]. In regards to the use of modern methods, one of the most recent examples is the use of clinical simulations using multimedia tools and software which can be used for improving psychological and psychomotor skills. Studies have shown that these centers also lead to improved motivation and independent learning tendencies among students [ 13 ].

The forth part, is related to the evaluation of the environment and physical location in the current tool, accessibility, flexibility in application, similarity to a real environment, specialized training spaces, receiving feedback and use of multimedia technologies. These factors are extracted according to the opinions of experts and stakeholders and have been used in similar studies. According to the standard for clinical skill training centers presented by the Ministry of Health, Treatment and Medical Education, the preferred physical location for a clinical skill training center includes a large area with a flexible application as well as a wardroom, nursing station, ICU or smaller rooms with specialized applications such as operation room and resuscitation room. Furthermore, a clinical skill training center must have access to a suitable location for providing students with multimedia education [ 8 ].

James et al. in their study, have shown effectiveness of an experimental pharmacology skill lab to facilitate training of specific modules for development of core competencies of parenteral drug administration and intravenous drip settings using mannequins for development of skills in administering injections for undergraduate medical students [ 15 ]. These factors were included in the evaluation questionnaire prepared in the current study. In the study by Hashim et.al.(2016), 62 participants believed that the time constraints and pressure of the clinical environment were not present in CSTC during learning clinical skills. Therefore, these centers can help students improve their skills by making them feel secure and resolve their concerns about the consequences of their actions. According to the students participating in this study, approximately 70 to 75 per cent of students felt more secure regarding mistakes and less worried about harming patients during clinical procedures after training clinical skills on mannequins available at clinical skill training centers [ 3 ].

The fifth part includes evaluating the long-term effects of education and evaluating the conformity between the center’s curriculum and educational needs, the effect of the center on improving essential skills, the effect of curriculum on interest, stress and facilitating clinical procedures. Ji He Yu et al. observed that after training in a clinical skill training center and simulations, students show a significantly lower level of anxiety and a significantly higher level of self-esteem compared to before the training. Furthermore, after experiencing the simulation, students without previous simulation experiences showed lower anxiety and higher self-esteem [ 16 ]. In a systematic review by Alanazi et al., evidence showed that participation in CSTC and using simulation can significantly improve the knowledge, skill and self-esteem of medical students [ 17 ]. Furthermore, a study by Younes et al. showed that adding a simulation program to a normal psychology curriculum improves the quality of education and the self-esteem of medical students [ 18 ]. In another study, Hashim et.al.(2016) showed a positive attitude among the students regarding the effectiveness of clinical skill training centers for improving skills, self-esteem as well as learning new clinical skills [ 3 ]. Therefore, based on the role of clinical skill training centers in improving the motivation and self-esteem of students presented in previous studies, these factors can be important in the evaluation of clinical skill training centers and therefore included in the evaluation questionnaire.

Limitations:

We had some limitations in our study. 1)There wasn’t any evaluation tool for evaluating Training Programs of medical students in Clinical Skill Training Center according to Consumers’ Perspective. Therefore, comparison was difficult and we compared every domain with results of other studies. The study was triangulation and we used many resources to designing this tool and it reduced biases. 2) In convergent step we extracted many items, but because of the possibility of non-response all questions, we couldn’t use all of them and questionnaires are summarized. To assuring no important item is neglected, experts in medical education checked the items.

There are many items in an evaluation tool for evaluating the Clinical Skill Training Center from Consumers’ Perspective. Some of these items could be answered by some consumers not all of them. In this tool is defined in 4 tools for four type of consumers. In every tool respondent answer questions in 5 domains (Learning objectives and course content, Equipment and tools, Educational processes, Environment and physical location). The evaluation tool designed in the current study offers suitable reliability and validity and can be used for evaluating CSTC from consumers’ perspectives. The application of this tool can help improve the effectiveness of educational activities and the curriculum in clinical skill training centers.

Data availability

The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Abbreviations

Clinical Skill Training Center

Content Validity Ratio

Content Validity Index

- Imran M, Khan SA, Aftab T. Effect of preclinical skill lab training on clinical skills of students during clinical years. Pak J Phsyiol. 2018;12(3):30 – 2. https://pjp.pps.org.pk/index.php/PJP/article/view/580

- Upadhayay N. Clinical training in medical students during preclinical years in the skill lab. Adv Med Educ Pract. 2017;8:189–94.

Article   Google Scholar  

Hashim R, Qamar K, Khan MA, Rehman S. Role of Skill Laboratory Training in Medical Education - Students’ Perspective. J Coll Physicians Surg Pak. 2016;26(3):195-8. PMID: 26975950.

- Singh H, Kalani M, Acosta-Torres S, El Ahmadieh TY, Loya J, Ganju A. History of simulation in medicine: from Resusci Annie to the Ann Myers Medical Center. Neurosurgery. 2013;73(Suppl 1):9–14.

- Bazargan A. Educational evaluation. Tehran: Samt; 2020.

Google Scholar  

- Morgan J, Green V, Blair J. Using simulation to prepare for clinical practice. Clin Teach. 2018;15(1):57–61.

- Pazargadi M, Ashktorab T, Alavimajd H, Khosravi S. Developing an Assessment Tool for nursing Students` General Clinical performance. Iran J Med Educ. 2013;12(11):877–87.

- Denizon Arranz S, Blanco Canseco JM, Pouplana Malagarriga MM, Holgado Catalán MS, Gámez Cabero MI, Ruiz Sánchez A, et al. Multi-source evaluation of an educational program aimed at medical students for interviewing/taking the clinical history using standardized patients. GMS J Med Educ. 2021;38(2):Doc40.

- Lam CY. Consumer-oriented evaluation Approach. The SAGE Encyclopedia of Educational Research, Measurement, and evaluation. Thousand Oaks: SAGE; 2018. pp. 390–2.

- Fitzpatrick J, Sanders J, Worthen B. Program evaluation: alternative approaches and practical guidelines. 4th, editor: ed. Boston: Allyn Bacon; 2004.

- Waltz CF, Bausell RB. Nursing research: design, statistics, and computer analysis. Philadelphia: V. A. Davis; 1981.

- Zamanzadeh V, Rassouli M, Abbaszadeh A, Majd HA, Nikanfar A, Ghahramanian A, editors. Details of content validity and objectifying it in instrument development2014.

- O’Connor M, Rainford L. The impact of 3D virtual reality radiography practice on student performance in clinical practice. Radiography. 2023;29(1):159–64.

Brydges R, Carnahan H, Rose D, Rose L, Dubrowski A. Coordinating Progressive Levels of Simulation Fidelity to maximize Educational Benefit. Acad Med. 2010;85(5):806–12.

James J, Rani RJ. Novel strategy of skill lab training for parenteral injection techniques: a promising opportunity for medical students. Int J Basic Clin Pharmacol. 2022;11(4):315.

Yu JH, Chang HJ, Kim SS, Park JE, Chung WY, Lee SK, et al. Effects of high-fidelity simulation education on medical students’ anxiety and confidence. PLoS ONE. 2021;16(5):e0251078.

Alanazi A, Nicholson N, Thomas S. Use of simulation training to improve knowledge, skills, and confidence among healthcare students: a systematic review. Internet J Allied Health Sci Pract. 2017.

Younes N, Delaunay A, Roger M, et al. Evaluating the effectiveness of a single-day simulation-based program in psychiatry for medical students: a controlled study. BMC Med Educ. 2021;21(1):348.

Download references

Acknowledgements

Sincere thanks to the practice tutors who undertook these clinical assessments and also we are very thankful to professors of Arak University of Medical Sciences for helping us in successful designing the questionnaire.

Not applicable.

Author information

Authors and affiliations.

Medical Education development center, Arak University of Medical Sciences, Arak, Iran

Rezvan Azad

Medicine School, Arak University of Medical Sciences, Arak, Iran

Mahsa Shakour & Narjes Moharami

You can also search for this author in PubMed   Google Scholar

Contributions

The concept and framework were designed by MSH and RA. The questionnaires and data were collected by RA. Data analyzed by MSH and RA. The manuscript was prepared by NM and edited by MSH and NM. The technical editing was done by MSH.

Corresponding author

Correspondence to Mahsa Shakour .

Ethics declarations

Ethics approval and consent to participate.

This study received ethical approval from the Institutional Review Board (IRB) of University of Medical Sciences, Iran to which the researchers are affiliated. All study protocols were performed in accordance with the Declaration of Helsinki. This study considered ethical considerations such as the confidentiality of the participants’ names and the written consent of participants. survey was conducted in 2021. Informed consent from each participant was obtained after clearly explaining the objectives as well as the significance of the study for each study participant. We advised the study participants about the right to participate as well as refuse or discontinue participation at any time they want and the chance to ask anything about the study. The participants were also advised that all data collected would remain confidential.

Consent for publication

Not Applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Azad, R., Shakour, M. & Moharami, N. Designing an evaluation tool for evaluating training programs of medical students in clinical skill training center from consumers’ perspective. BMC Med Educ 24 , 502 (2024). https://doi.org/10.1186/s12909-024-05454-7

Download citation

Received : 22 November 2023

Accepted : 22 April 2024

Published : 09 May 2024

DOI : https://doi.org/10.1186/s12909-024-05454-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Program evaluation
• Consumer-oriented
• Clinical skills lab

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Int J Emerg Med

Clinical care review systems in healthcare: a systematic review

Laura e. walker.

1 Department of Emergency Medicine and Health Sciences Research, Mayo Clinic, 200 First St. SW, Rochester, MN 55905 USA

David M. Nestler

Torrey a. laack, casey m. clements, patricia j. erwin.

2 Mayo Clinic Libraries and Health Sciences Research, Mayo Clinic, Rochester, MN USA

Lori Scanlan-Hanson

M. fernanda bellolio, associated data.

Clinical care review is the process of retrospectively examining potential errors or gaps in medical care, aiming for future practice improvement. The objective of our systematic review is to identify the current state of care review reported in peer-reviewed publications and to identify domains that contribute to successful systems of care review.

A librarian designed and conducted a comprehensive literature search of eight electronic databases. We evaluated publications from January 1, 2000, through May 31, 2016, and identified common domains for care review. Sixteen domains were identified for further abstraction.

We found that there were few publications that described a comprehensive care review system and more focus on individual pathways within the overall systems. There is inconsistent inclusion of the identified domains of care review.

While guidelines for some aspects of care review exist and have gained traction, there is no comprehensive standardized process for care review with widespread implementation.

Electronic supplementary material

The online version of this article (10.1186/s12245-018-0166-y) contains supplementary material, which is available to authorized users.

Clinical care review is the process of retrospectively examining potential errors or gaps in medical care, with a goal of future practice improvement. This goes by many different names, sometimes with different audiences or case types, including peer review, adverse event review, sentinel event review, and root cause analysis. The concept of care review is widely accepted and encouraged among safety and quality healthcare leaders. However, a paucity of literature exists discussing and describing the current state of clinical care review.

The challenges and risks of contemporary medical care are well described. Medical error and its resulting outcomes have been defined and measured in many different ways, leading to varying quantifications of the effects [ 1 ]. The Institute of Medicine’s (IOM) 1999 report entitled “To Err is Human” [ 2 ] estimated that as many as 44,000 to 98,000 deaths annually in the USA occur as a result of medical error. Publication of “To Err is Human” was a landmark event in the recognition of the role of adverse events in medical care in the USA. This represents a shift in the focus on adverse events to look toward systems issues as a cause or error and a call to identify and act to prevent medical error. The National Quality Foundation estimates that, in 2010, medical errors affected 15.5% of Medicare beneficiaries, with nearly half of these errors considered preventable [ 3 ]. More recently, Makary and Daniel estimated that as many as 250,000 deaths per year in the USA are due to medical error, making it the third leading cause of death by their estimation [ 1 ]. Review of adverse events allows for investigation into, and classification of the causes of, the event and presents an opportunity to modify systems and behaviors to prevent future similar errors. As a part of the strategic approach for increasing safety, the IOM’s “To Err is Human” recommended “Identifying and learning from errors by ... encouraging health care organizations and practitioners to develop and participate in voluntary reporting systems.” They went on to say “Such systems can focus on a much broader set of errors, mainly those that do no or minimal harm, and help detect system weaknesses that can be fixed before the occurrence of serious harm, thereby providing rich information to health care organizations in support of their quality improvement efforts” [ 1 ].

Given the long standing call for clinical care review, with limited literature to inform care review systems, we conducted a qualitative systematic review to identify characteristics discussed in existing models for care review. The objectives are to (1) describe the current state of care review and (2) identify elements from published care review systems that contribute to their success. This systematic review will allow for a more complete evaluation of the current state of clinical care review and will identify areas for future scholarly activity.

This is a qualitative systematic review of studies describing and evaluating care review systems. This study was exempt from our IRB review. This report adheres to the recommendations made in the preferred reporting items for systematic reviews (PRISMA) statement [ 4 ]. A protocol was written before the beginning of the investigation.

We included original research studies with any methodological design including cohort studies, case controls, and randomized trials, as well as commentaries, narrative reviews, letters to the editor, and abstracts in peer-reviewed journals that reported models for care review. Search results were limited to publications after January 1, 2000, to focus on publications since the release of “To Err is Human” [ 1 ]—a turning point in the way adverse events are analyzed and regarded. In choosing relevant publications, some articles described their process as the main purpose of the article, while others incidentally described a care review process, while instead focusing on a specific intervention or aspect of their mechanism for review. Either was acceptable, as they both shed light on a review system for analysis.

All types of patients and hospital settings were included, as well as recommendations from professional organizations and companies. This study’s investigators are physicians with involvement in quality improvement, adverse event identification and management, patient safety, and leadership of committees for clinical care review.

A senior expert librarian (P.E.) designed and conducted a comprehensive search of eight electronic databases, including Ovid MEDLINE, Ovid EMBASE, EBSCO CINAHL, Ovid CENTRAL, Ovid Cochrane Database of Systematic Reviews, Web of Science, and Scopus. Our search was done on June 10, 2016, and includes publications from January 1, 2000, through May 31, 2016. We included published conference abstracts in our search. There was no language restriction to the search strategy. Bibliography and reference lists of the articles obtained through database search were reviewed to identify additional publications for inclusion. The search strategy can be found in the Additional file  1 .

Qualitative assessment and data abstraction process

Two investigators (L.W. and D.N.) identified common domains in the initial literature review to determine which data to abstract, and included additional variables determined to be clinically important based on their experience in the clinical care review process and practice improvement. Domains included were description of systems improvement, educational output and feedback, description of a standardized process and referral mechanism, consideration of the case outcome, deliverables of the review system including non-punitive process and recognition of excellence, multidisciplinary involvement, dedicated process leadership, reviewer training, case blinding/anonymity, and implementation of improvement recommendations by the investigating group. These are further described in Table  1 .

Descriptions of the 16 domains of care review

In phase I of the review, one investigator (L.W.) independently screened all titles yielded by the initial search strategy for possible inclusion. After identifying appropriateness for possible inclusion, phase II consisted of two reviewers (L.W. and D.N.) independently evaluating the abstracts of publications identified in phase I. The publications from phase II were then retrieved in full text and assessed for inclusion of domain abstraction in phase III by two independent reviewers. The agreed-upon articles were assessed by independent reviewers in duplicate, to abstract the identified domains of care review in phase III.

In phase II, disagreement between reviewers was reconciled by discussion and consensus. The investigators were not blinded to the authors, journals, or results of studies. In phase III, disagreements on the data abstraction were resolved by a third independent reviewer who assessed the article and determined if the theme was included in the care review process. Descriptions of the 16 domains were supplied to all reviewers prior to data abstraction for reference.

Critical appraisal is the process of systematically examining research evidence to assess its validity, results, and relevance before using it to inform a decision. Instruments developed to support quality appraisal usually share some basic criteria for the assessment of qualitative research. These include the need for research to have been conducted ethically, the consideration of relevance to inform practice or policy, the use of appropriate and rigorous methods, and the clarity, coherence of reporting, address of reliability, validity, and objectivity [ 5 ].

In considering the most appropriate instruments to use for critical appraisal, we considered using the Cochrane Collaboration Bias Appraisal Tool [ 6 ] and a modified Newcastle-Ottawa Scale tool [ 7 ]. The nature of our qualitative data abstraction precluded the use of these tools. While the studies we evaluated may have included randomized controlled trials and been at risk for bias, the results of the publications evaluated were not typically relevant to our goal of qualitative domain abstraction. Many publications we evaluated were narrative in nature—describing a process without presentation of data, either qualitative or quantitative. For those publications that did present data relevant to our domains, the effect of bias within the study was felt to be unlikely to impact our qualitative data collection because the abstracted domains—descriptions of processes—were not affected by the results of the studies. We reviewed the items described in the Standards for Reporting Qualitative Research (SRQR) [ 8 ] and the Enhancing Transparency in Reporting the Synthesis of Qualitative Research (ENTREQ) [ 9 ] statement. The SRQR and ENTREQ aim to improve the transparency of all aspects of qualitative research by providing clear standards for reporting qualitative research. When assessing the risk of bias, we decided not to exclude articles based on their quality assessment. All potentially valuable insights were included. From each study, we extracted the domains relevant to care review processes. We tabulated the results and created graphics based on frequencies. No quantitative data was appropriate for abstraction, so we did not perform a meta-analysis.

The initial library search strategy identified 1318 titles for review. In phase I, 440 abstracts were reviewed, 76 of which were selected for full-text review in phase II. Fifteen articles from outside sources and bibliography review were also identified and reviewed. In total, 91 full-text articles were assessed, and after reconciliation between two independent reviewers, 47 articles were initially found to be appropriate for inclusion in our analysis of the domains of care review. One article was removed in the abstraction process, as both reviewers independently determined that it did not meet inclusion criteria [ 10 ] leading to 46 unique articles reviewed.

Domains were abstracted by two independent reviewers for each of the 46 articles in phase III. Articles that described a care review process from the same institution were consolidated to reflect the most complete view of that process possible, as aspects may have been reported differently in multiple articles/abstracts. Figure  1 shows the study selection process. Ultimately, we evaluated the care review systems from 35 unique institutions.

Study selection process

Study characteristics

Among the 46 studies, 35 represented unique institutions and 11 were same authors/institutions describing different aspects of the process or domains. The types of articles identified included 14 descriptive [ 11 – 24 ], three editorials [ 25 – 27 ] 15 prospective [ 28 – 42 ], seven quality improvement projects [ 12 , 43 – 48 ], and ten retrospective [ 11 , 30 , 49 – 56 ]. The 16 domains of successful care review that were identified for abstraction are presented and defined in Table  1 .

The percentage of frequency of each component is shown in Fig.  2 . The most commonly identified component of a care review process was utilizing an analysis of systems issues contributing to the case (32 institutions, 91.4%), followed by utilizing a standardized process for case review (30 institutions, 85.7%) and use of a structured case classification system (28 institutions, 80.0%). The least common components identified were recognition of excellence and use of case blinding/anonymity in reviews (5 institutions, 14.3%). Some articles were consistent with more than one article type and were classified as both.

Frequency of domain identification

Table  2 shows the distribution of all components in the full-text articles reviewed, with consolidation of same-institutions. No article/institution identified all 16 items evaluated by reviewers. Two institutions identified 14 of 16 items: Lehigh Valley and Johns Hopkins.

Distribution of care review domains

Our systematic review shows that, in the first two decades since the IOM report calling for improved safety systems, there have been few articles outlining a comprehensive clinical care review process. Additionally, most articles discuss their care review systems in the context of describing an aspect of their process, or corresponding improvement initiative.

Systems analysis—defined as the assessment of the effects of external forces such as policies, workflows, and software such as the electronic medical record on the critical event—was the most commonly identified care review process characteristic. Many identified articles describe the importance of evaluating how a person works within a system, rather than in isolation, to identify improvement opportunities. Assuming individuals are properly motivated with benign intent, looking at the system surrounding, the care avoids an antagonistic approach and supports the IOM’s underlying reasons for calling for care review processes—to prevent future errors.

Similarly, standardized processes and structured case classification were frequently discussed in the literature. To meet the IOM’s recommendations for creating care review “systems,” having a standardized process that uses structured classification is likely necessary. Without standardization, reviews would likely be sporadic, inefficient, and challenging to implement and subsequently inform future practice. Without structured classification, one could assume that conclusions would also be difficult to interpret.

Although some of these characteristics were common among reported care review systems, others are only rarely reported. Recognition of excellence and blinding of cases were reported in just five (14%) of the reports. Institutions that recognize excellence while performing care reviews were supportive of the practice, and one can understand why this would support the culture needed to have an effective care review system, and perhaps designers of future care review systems may wish to consider implementing this component. Similarly, anonymous review, or blinding, is intended to reduce bias and may allow a more objective review of each case. However, its infrequent mention may be indicative of unpublished prior experiences that may have supported avoiding this practice. From our experience, these are controversial topics, and future work is needed to understand the effects of specific characteristics on the overall care review process.

One additional characteristic that review processes must be supported by a functioning organizational system should receive particular attention. Although this was specifically identified by only 19 organizations, the downstream benefit to reviewing an episode of care and making recommendations for change in a non-functional system is likely lost. Key stakeholders in the process (physicians, nurse practitioners, physician assistants, nursing staff, support staff, etc.) are seemingly necessary for the care review process, and the administrative and leadership structure must be supportive of recommendations for change after care review is completed. This combination is strongly conducive to a process that engenders trust from the care team, which in turn bolsters the system as cases are referred for review, and staff engage in further problem solving.

Limitations

The articles evaluated come from a variety of settings—from consulting firms to in situ care review systems. Some authors strove to describe a comprehensive local practice, while others focused primarily on a particular component of a larger system. This heterogeneity limits the generalizability as the variability from one system to another may indicate institution- or system-specific adaptations to facilitate the process. A solution for one setting may not represent a good solution for another. We included articles from institutions and consulting firms describing or self-reporting care review systems, and it is not possible to know the true effectiveness of the processes described when removed from clinical context. It may be that there is an over-emphasis on some areas of care review believed to be ideal that are not practiced as described, and also possible that not all aspects of a process are represented. Particularly in the articles that discuss the care review process as the context for a specific project, it is possible that not all the details on the over-arching system of care review in place are described resulting in abstraction of domains in what is an incomplete description.

The domains we used during abstraction were determined by screening the included articles and supplemented by expert opinion. It is possible that there are additional variables that are more important, but less common, and were not included in our analysis. It is possible that a care review process we reviewed may include some of the 16 characteristics but did not specifically mention them in the articles reviewed. Additionally, the qualitative nature of the abstraction and interpretation of each item definitions are complex and may lead to less reliable results.

In an effort to reduce the effects of bias and definition complexity, all articles were reviewed in duplicate—both for inclusion in the study as well as abstraction of data. Disagreements were resolved by discussion and consensus for article inclusion and adjudicated by a third reviewer for the domain abstraction.

Despite increased discussion among institutions such as IOM and the National Patient Safety Foundation, in the last 16 years, there have been relatively few publications describing clinical care review processes and no clear evidence of a cultural shift to embrace clinical care review in an organized fashion. We have identified 16 domains of focus in a care review process and found that the approach to care review is highly variable as represented in the literature.

Future research

The effects of different aspects of care review processes have not been well studied. This presents an opportunity to evaluate processes that are present in many hospitals and health systems and identify truly effective, rather than simply common, practices, as identified within.

Additional file

Search strategy. (DOCX 14 kb)

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Authors’ contributions

All the authors have contributed substantially to this manuscript. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Laura E. Walker, Email: [email protected] .

David M. Nestler, Email: [email protected] .

Torrey A. Laack, Email: [email protected] .

Casey M. Clements, Email: [email protected] .

Patricia J. Erwin, Email: [email protected] .

Lori Scanlan-Hanson, Email: [email protected] .

M. Fernanda Bellolio, Phone: 507-775-5607, Email: [email protected] .

REVIEW article

This article is part of the research topic.

Translational Research in Surgical Applications and Spinal Tumors

Role of Immunotherapy in Treatment Refractory Chordomas: Review of Current Evidence Provisionally Accepted

• 1 Department of Neurosurgery and Center for Skull Base and Pituitary Surgery, University of Minnesota, United States

The final, formatted version of the article will be published soon.

Chordomas are aggressive tumors that are thought to arise from remnants of the embryological notochord. They can arise along the ventromedial aspect of the sacrum, mobile spine, and clivus with most cases occurring in the sacrum or skull base. Despite surgery and radiation, chordomas often progress and become refractory to further treatment. The high recurrence rate has created an urgent need to develop new systemic treatment options. Recent case reports and clinical trials have highlighted the use of immunotherapy for refractory chordomas. In this review, we summarize the results of these studies and discuss the potential role of immunotherapy for chordomas.The PUBMED database was queried for studies mentioning both "Chordoma" and "Immunotherapy." All case series and case reports that involved administration of an immunotherapy for chordoma were included. Additional studies that were found during literature review were added. ClinicalTrials.Gov was queried for studies mentioning both "Chordoma" and "Immunotherapy." The final cohort consisted of all clinical trials that utilized immunotherapy for chordomas of any location.Eight case reports and series of immunotherapy for treatment refractory chordoma were identified.Most patients received immunotherapy targeting the PD-1/PD-L1 interaction, and two patient received therapy targeting this interaction along with the tyrosine kinase inhibitor pazopanib. One patient received a vaccine derived from autologous tumor cells, and one patient received a viral vector that downregulated the effect of TGF-beta. One clinical utilized a brachyury vaccine in conjunction with standard of care radiotherapy.Immunotherapy for chordoma is a promising area of investigation with increasing, but small, numbers of case series and clinical trials. Despite challenges in patient accrual, future directions in chordoma immunotherapy may lie in vaccine-based therapies and immune checkpoint inhibitors. Understanding chordoma heterogeneity and microenvironment will likely elucidate important chordoma features that will inform future clinical trial design.

Keywords: Chordoma, metastasis, Spine, Skull Base, Immunotherapy, oncology, tumor

Received: 24 Jan 2024; Accepted: 14 May 2024.

Copyright: © 2024 Alexander, Dhawan and Venteicher. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Dr. Andrew S. Venteicher, Department of Neurosurgery and Center for Skull Base and Pituitary Surgery, University of Minnesota, Minneapolis, United States

People also looked at