meta analysis vs quantitative research

• En español – ExME
• Em português – EME

Systematic reviews vs meta-analysis: what’s the difference?

Posted on 24th July 2023 by Verónica Tanco Tellechea

You may hear the terms ‘systematic review’ and ‘meta-analysis being used interchangeably’. Although they are related, they are distinctly different. Learn more in this blog for beginners.

What is a systematic review?

According to Cochrane (1), a systematic review attempts to identify, appraise and synthesize all the empirical evidence to answer a specific research question. Thus, a systematic review is where you might find the most relevant, adequate, and current information regarding a specific topic. In the levels of evidence pyramid , systematic reviews are only surpassed by meta-analyses. 

To conduct a systematic review, you will need, among other things: 

• A specific research question, usually in the form of a PICO question.
• Pre-specified eligibility criteria, to decide which articles will be included or discarded from the review. 
• To follow a systematic method that will minimize bias.

You can find protocols that will guide you from both Cochrane and the Equator Network , among other places, and if you are a beginner to the topic then have a read of an overview about systematic reviews.

What is a meta-analysis?

A meta-analysis is a quantitative, epidemiological study design used to systematically assess the results of previous research (2) . Usually, they are based on randomized controlled trials, though not always. This means that a meta-analysis is a mathematical tool that allows researchers to mathematically combine outcomes from multiple studies.

When can a meta-analysis be implemented?

There is always the possibility of conducting a meta-analysis, yet, for it to throw the best possible results it should be performed when the studies included in the systematic review are of good quality, similar designs, and have similar outcome measures.

Why are meta-analyses important?

Outcomes from a meta-analysis may provide more precise information regarding the estimate of the effect of what is being studied because it merges outcomes from multiple studies. In a meta-analysis, data from various trials are combined and generate an average result (1), which is portrayed in a forest plot diagram. Moreover, meta-analysis also include a funnel plot diagram to visually detect publication bias.

Conclusions

A systematic review is an article that synthesizes available evidence on a certain topic utilizing a specific research question, pre-specified eligibility criteria for including articles, and a systematic method for its production. Whereas a meta-analysis is a quantitative, epidemiological study design used to assess the results of articles included in a systematic-review. 

Remember: All meta-analyses involve a systematic review, but not all systematic reviews involve a meta-analysis.

If you would like some further reading on this topic, we suggest the following:

The systematic review – a S4BE blog article

Meta-analysis: what, why, and how – a S4BE blog article

The difference between a systematic review and a meta-analysis – a blog article via Covidence

Systematic review vs meta-analysis: what’s the difference? A 5-minute video from Research Masterminds:

• About Cochrane reviews [Internet]. Cochranelibrary.com. [cited 2023 Apr 30]. Available from: https://www.cochranelibrary.com/about/about-cochrane-reviews
• Haidich AB. Meta-analysis in medical research. Hippokratia. 2010;14(Suppl 1):29–37.

Verónica Tanco Tellechea

Leave a reply cancel reply.

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Subscribe to our newsletter

You will receive our monthly newsletter and free access to Trip Premium.

Related Articles

How to read a funnel plot

This blog introduces you to funnel plots, guiding you through how to read them and what may cause them to look asymmetrical.

Heterogeneity in meta-analysis

When you bring studies together in a meta-analysis, one of the things you need to consider is the variability in your studies – this is called heterogeneity. This blog presents the three types of heterogeneity, considers the different types of outcome data, and delves a little more into dealing with the variations.

Natural killer cells in glioblastoma therapy

As seen in a previous blog from Davide, modern neuroscience often interfaces with other medical specialities. In this blog, he provides a summary of new evidence about the potential of a therapeutic strategy born at the crossroad between neurology, immunology and oncology.

• Search Menu
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Papyrology
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Archaeology
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Evolution
• Language Reference
• Language Acquisition
• Language Variation
• Language Families
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Modernism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Media
• Music and Religion
• Music and Culture
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Science
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Lifestyle, Home, and Garden
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Clinical Neuroscience
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Ethics
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Ethics
• Business Strategy
• Business History
• Business and Technology
• Business and Government
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic History
• Economic Systems
• Economic Methodology
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Behaviour
• Political Economy
• Political Institutions
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Theory
• Politics and Law
• Public Policy
• Public Administration
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Developmental and Physical Disabilities Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

30 Meta-Analysis and Quantitative Research Synthesis

Noel A. Card, Family Studies and Human Development, University of Arizona, Tucson, AZ

Deborah M. Casper, Family Studies and Human Development, University of Arizona, Tucson, AZ

• Published: 01 October 2013
• Cite Icon Cite
• Permissions Icon Permissions

Meta-analysis is an increasingly common method of quantitatively synthesizing research results, with substantial advantages over traditional (i.e., qualitative or narrative) methods of literature review. This chapter is an overview of meta-analysis that provides the foundational knowledge necessary to understand the goals of meta-analysis and the process of conducting a meta-analysis, from the initial formulation of research questions through the interpretation of results. The chapter provides insights into the types of research questions that can and cannot be answered through meta-analysis as well as more practical information on the practices of meta-analysis. Finally, the chapter concludes with some advanced topics intended to alert readers to further possibilities available through meta-analysis.

Introduction to Meta-analysis

Meta-analysis, also referred to as quantitative research synthesis, is a systematic approach to quantitatively synthesizing empirical literature. By combining and comparing research results, metaanalysis is used to advance theory, resolve conflicts within a discipline, and identify directions for future research ( Cooper & Hedges, 2009 ). We begin by describing what meta-analysis is and what it is not.

Basic Terminology

It is important to provide a foundation of basic terminology on which to build a more technical and advanced understanding of meta-analysis. First, we draw the distinction between meta-analysis and primary and secondary analysis. The second distinction we draw is between quantitative research synthesis and qualitative literature review.

Glass (1976 ) defined primary-, secondary-, and meta-analysis as the analysis of data in an original study, the re-analysis of data previously explored in an effort to answer new questions or existing questions in a new way, and the quantitative analysis of results from multiple studies, respectively. A notable distinction between meta-analysis as compared to primary and secondary analysis involves the unit of analysis. In primary and secondary analyses, the units of analysis are most often the individual participants. In contrast, the units of analysis in a meta-analysis are the studies themselves or, more accurately, the effect sizes (defined below) of these studies.

A second foundational feature to consider is the distinction between quantitative research synthesis and qualitative literature review. Although both approaches are valuable to the advancement of knowledge, they differ with regard to focus and methodology. The focus of meta-analysis is on the integration of research outcomes, specifically in terms of effect sizes. In contrast, the focus of a qualitative literature review can be on research outcomes (although typically not focusing on effect sizes) but can also be on theoretical perspectives or typical practices in research. In terms of methods, scientists utilizing meta-analytic methodologies quantitatively synthesize findings to draw conclusions based on statistical principle. In contrast, scholars who conduct a qualitative literature review subjectively interpret and integrate research. Not considered in this chapter are other methodologies that fall between these two approaches on the taxonomy of literature review (for a more comprehensive review, see   Card, 2012 ; Cooper 1988 ).

As previously acknowledged, both quantitative research synthesis and qualitative literature review merit recognition for their respective contributions to the advancement of scientific knowledge. Quantitative literature reviews were developed to overcome many of the limitations of qualitative literature reviews, and we will highlight the advantages of quantitative literature reviews below. However, it is worth noting that quantitative research synthesis has also faced criticisms ( Chalmers, Hedges, & Cooper, 2002 ). Following are some highlights in the history of meta-analysis (for more thorough historical account, see   Chalmers, Hedges, & Cooper, 2002 ; Hedges, 1992 ; Hunt, 1997 ; Olkin, 1990 ).

A Brief History

Research synthesis methodology can be traced as far back as 1904 when Karl Pearson integrated five studies looking at the association between inoculation for typhoid fever and morality ( see   Olkin, 1990 ). By the 1970s, at least three independent groups had started to combine results from multiple studies ( Glass, 1976 ; Rosenthal & Rubin, 1978 ; Schmidt & Hunter, 1977 ), but the most influential work was Mary Smith and Gene Glass’ (1977) “meta-analysis” of psychotherapy, which was both ground-breaking and controversial. Smith and Glass’s (1977 ) meta-analysis sparked considerable controversy and debate as to the legitimacy of not only the findings but of the methodology itself ( Eysenck, 1978 ). It is worth noting, however, that some have suggested the controversy surrounding Smith and Glass’ (1977 ) meta-analysis had much more to do with the results than the methodology ( Card, 2012 ).

Following the somewhat turbulent introduction of meta-analysis into the social sciences, the 1980s offered significant contributions. These contributions came from both the advancement and dissemination of knowledge of meta-analytic techniques by way of published books describing the approach, as well as through the publication of research utilizing the methods ( Glass, McGaw, & Smith, 1981 ; Hedges & Olkin, 1985 ; Hunter, Schmidt, & Jackson, 1982 ; Rosenthal, 1984 ). Since its introduction into the social sciences in the 1970s, meta-analysis has become increasingly visible and has made considerable contributions to numerous bodies of scholarly research ( see   Cochran, 1937 ; Hunter, Schmidt, & Hunter, 1979 ; Pearsons, 1904 ; Rosenthal & Rubin, 1978 ; Glass & Smith, 1979 ; Smith & Glass, 1977 ).

Research Synthesis in the Social Sciences

Glass (1976 ) brought the need for meta-analysis to the forefront in a presidential address. It is not uncommon to observe conflicting findings across studies ( Cooper & Hedges, 2009 ). These inconsistencies lead to confusion and impede progress in social science (as well as in the so-called hard sciences; Hedges, 1987 ). Quantitative research synthesis is a powerful approach that addresses this problem through the systematic integration of results from multiple studies that often individually report conflicting results.

Chapter Overview

The following chapter is an overview of metaanalysis that provides the foundational knowledge necessary to understand the goals of meta-analysis and the process of conducting a meta-analysis, from the initial formulation of research questions through the interpretation of results. The chapter provides insights into the types of research questions that can and cannot be answered through meta-analysis as well as more practical information on the practices of meta-analysis. Finally, we conclude the chapter with some advanced topics intended to alert readers to further possibilities available through meta-analysis. To begin, we consider the types of questions that can and cannot be answered through meta-analysis.

Problem Formulation

Questions that can and cannot be answered through meta-analysis.

One of the first things to consider when conducting scientific research is the question for which you seek an answer; meta-analysis is no exception. A primary purpose for conducting a meta-analytic review is to integrate findings across multiple studies; however, not all questions are suitable for this type of synthesis. Hundreds, or sometimes thousands, of individual research reports potentially exist on any given topic; therefore, after an initial search of the literature, it is important to narrow the focus, identify goals, and articulate concise research questions that can be answered by conducting a tractable meta-analysis. A common misconception by those unfamiliar with meta-analysis is that an entire discipline or phenomenon can be “meta-analyzed” ( Card, 2012 ). Because of the infinite number of questions that could be asked—many of which could be answered using meta-analysis—this sort of goal is too as pecific. Rather, a more appropriate approach to quantitative research synthesis is to identify a narrowly focused goal or set of goals and corresponding research questions.

Identifying Goals and Research Questions

Cooper’s (1988 ) taxonomy of literature reviews identified multiple goals for meta-analysis. These include integration, theory development, and the identification of central issues within a discipline. We consider each of these goals in turn.

Integration . There are two general approaches to integrating research findings in meta-analysis: combining and comparing studies. The approach of combining studies is used to integrate effect sizes from multiple primary studies in an effort to estimate an overall, typical effect size. It would then be expectable to make inferences about this mean effect size by way of significance testing and/or confidence intervals. A second approach commonly used to integrate findings involves comparing studies. Also known as moderator analyses (addressed in more detail below), comparisons can be made across studies when a particular effect size is hypothesized to systematically vary on one or more of the coded study characteristics. Analyses to address each of these two approaches to integration will be described below.

Theory Development . A second goal of meta-analysis involves the development of theory. Meta-analysis can be used quite effectively and efficiently toward this end. If associations between variables that have been meta-analytically combined are weak, then this might indicate that a theory positing stronger relations of the constructs in question should be abandoned or modified ( Schmidt, 1992 ). If, on the other hand, associations are strong, then this may be an indication that the phenomenon under investigation is moving toward a more integrated theory. Ideally, meta-analyses can be used to evaluate competing theories that make different predictions about the associations studied. Either way, meta-analysis is a powerful tool that can be used toward the advancement of theory within the social sciences.

Integration of Central Issues . A final goal has to do with identifying central issues within a discipline or phenomenon. The exhaustive review of empirical findings can aid in the process of identifying key issues within a discipline, such as whether there is inadequate study of certain types of samples or methodologies. The statistical techniques of meta-analysis can address inconsistencies in the findings, attempting to predict these inconsistencies with coded study characteristics (i.e., moderator analyses). Both of these contributions are important to the process of identifying directions for future research and the advancement of knowledge.

Critiques of Meta-Analysis

Earlier, we described how the controversial nature of one of the earliest meta-analyses ( Smith & Glass, 1977 ) drew criticism not only of their findings but also of the technique of meta-analysis itself. Although these critiques have largely been rebuffed, they are still occasionally applied. Among the most common criticisms of meta-analysis are: (1) the “file drawer” problem; (2) the apples and oranges problem; (3) garbage in and garbage out; (4) the level of expertise required of the meta-analyst; and (5) the potential lack of qualitative finesse.

The “file drawer” problem . The “file drawer” problem, also known as the threat of publication bias, is based on the notion that significant results get published and nonsignificant findings get relegated to the “file drawer,” resulting in the potential for a publication bias in meta-analysis ( Rosenthal, 1979 ). To answer this criticism, however, meta-analysts typically employ both systematic and exhaustive search strategies to obtain published and unpublished reports in an effort to minimize this threat. In addition, there is an extensive collection of statistical procedures in meta-analysis that can be used to probe the existence, extent, and likely impact of publication bias ( Rothstein, Sutton, & Borenstein, 2005 ).

The apples and oranges problem . The apples and oranges problem describes the potential process of combing such a diverse range of studies that the aggregated results are meaningless. For example, if a meta-analyst attempted to investigate the predictors of childhood internalizing problems by including studies focusing on depression, anxiety, and social withdrawal, then it could be argued that the aggregation of results across this diverse range of problems is meaningless. This critique, in our opinion, is conceptual rather than methodological: Did the scientist using meta-analytic techniques define a sampling frame of studies within which it is useful to combine results? Fortunately, meta-analytic reviews can use both (1) combination to estimate mean results and (2) comparison to evaluate whether studies with certain features differ. Put differently, meta-analysis allows for both general and specific results. Returning to the example of a meta-analyst investigating the predictors of child psychopathology, it might be useful to present results of both (1) predictors of general internalizing problems, and (2) comparisons of the distinct predictors of depression, anxiety, and social withdrawal.

Garbage in and garbage out . Garbage in, garbage out describes the practice of including poor-quality research reports in a meta-analysis, which result in only poor-quality conclusions. Although this critique is valid in some situations, we believe a more nuanced consideration of “garbage” is needed before being used as a critique of a particular meta-analysis. In the next section , we will provide this consideration by discussing how the limits of primary research place limits on the conclusions that can be drawn from meta-analysis of that research.

The level of expertise required of the meta-analyst . A common misconception is that meta-analysis requires advanced statistical expertise. We would argue that with basic methodological and quantitative training, such as usually obtained in the first year of graduate school, many scientists could readily learn the basic techniques (through an introductory course or book on meta-analysis) to conduct a sound meta-analytic review.

The potential lack of qualitative finesse . A final criticism that has been raised is that meta-analysis lacks the “qualitative finesse” of a qualitative review. Perhaps tellingly, a definition of qualitative finesse is generally lacking when this critique is made, but it seems that this critique implies that a meta-analyst has not thought carefully and critically about the nuances of the studies and collection of studies. There certainly exist meta-analyses where this critique seems relevant—just as there exist primary quantitative studies in which careful thought seems lacking. The solution to this critique is not to abandon meta-analytic techniques, however, just as the solution to thoughtless primary studies is not to abandon statistical analyses of these data. Rather, this critique makes clear that meta-analysis—like any other methodological approach—is a tool to aid careful thinking, rather than a replacement for it.

Limits of Primary Research and Meta-Analysis

It is also important to recognize that the conclusions of a meta-analytic review must be tempered by the quality of the empirical research comprising this review. Many of the threats to drawing conclusions in primary research are likely to translate to meta-analysis as well. Perhaps the most salient threats involve flaws in the study design, sampling procedures, methodological artifacts, and statistical power.

Study design . The design of primary studies guides the types of conclusions that can be drawn from them; similarly, the design of studies included in a meta-analysis guides the types of conclusions that can be drawn. Experimental designs, although powerful in their ability to permit inferences of causality, often do not share the same ecological validity as correlational designs. Conversely, correlational designs cannot make inferences of causality. It would follow that any limitation existing within primary studies also exists within the meta-analyses that encompass these studies.

Sampling . Another limitation of primary studies is that it is difficult to support inferences generalizable beyond the sampling frame. When a sample is drawn from a homogeneous population, inferences can be made only for a limited set of individuals. Similarly, findings from a meta-analysis can only be generalized to populations within the sampling frame of the included studies; however, the collection of primary studies within a meta-analysis is likely to be more heterogeneous than one single primary study if it includes studies that are collectively diverse in their samples, even if each study sample is homogeneous.

Methodological artifacts . Both primary research and meta-analysis involve methodological shortcomings. Although it is difficult to describe all of the characteristics that make up a high-quality study, it is possible to identify those artifacts that likely lower the quality of the design. In primary studies, methodological issues need to be addressed prior to data collection. In contrast, meta-analysis can address these methodological artifacts in either one of two ways. The first way is to compare (through moderator analyses) whether studies with different methodological features actually yield different findings. Second, for some artifacts (e.g., measurement unreliability) described near the end of this chapter, corrections can be made that allow for the analysis of effect sizes free of these artifacts. Artifact correction is rarely performed in primary research (with the exception of latent variable modeling to correct for unreliability) but more commonly considered in meta-analyses.

Statistical power . Another limitation of much primary research is low statistical power ( Maxwell, 2004 ). Statistical power is the probability of detecting an effect that truly does exist but is often unacceptably low in many primary research studies. This low power results in incorrect conclusions in primary studies that an effect does not exist (despite cautions against “accepting” the null hypothesis). Fortunately, meta-analysis is usually less affected by inadequate power of primary studies because it combines a potentially large number of studies, thus resulting in greater statistical power.

Strengths of Meta-Analysis

As outlined above, there are limits to metaanalysis; however, meta-analysis should be recognized for its considerable strengths. We next briefly describe three of the most important of these: (1) a systematic and disciplined review process; (2) sophisticated reporting of findings; and (3) a way of combining and comparing large amounts of data ( Lipsey & Wilson, 2001 ).

Systematic and disciplined review process . First, systematic procedures must be followed to conduct a comprehensive literature search, consistently code comparable characteristics and effect sizes from studies, and to ensure the accuracy of combining results from multiple reports into one effect size. The processes of searching the literature, identifying studies, coding, and analyzing results have received tremendous attention in the literature on meta-analysis methodology, in contrast to most other forms of literature review. Although this work requires discipline, diligent attention to detail, and meticulous documentation on the part of the metaanalyst, when these procedures are followed, a large amount of data can be combined and compared and the outcome is likely to be a significant contribution to the field.

Combining and comparing large amounts of data . Perhaps one of the greatest strengths of meta-analytic techniques is the ability to combine and compare large amounts of data that would otherwise be impossible to integrate in a meaningful way. It would assuredly exceed the capacity of almost any scholar to combine the large amounts of data and draw meaningful conclusions without quantitative literature review techniques. Following the strength of combining and comparing large amounts of data is the strength in the way in which the findings are reported.

Sophisticated reporting of findings . Meta-analysis offers a level of sophistication in the way in which the findings are reported. Unlike qualitative literature reviews that derive and report conclusions and interpretations in a narrative format, meta-analysis uses statistical techniques to yield quantified conclusions. Meta-analysts commonly take advantage of visual tools such as stem-and-leaf plots, funnel plots, and tables of effect sizes to add a level of sophistication to the reporting of findings.

Searching the Literature

Defining a sampling frame.

Similarly to primary research, a sampling frame must be considered in meta-analysis. However, the unit of analysis in a meta-analysis is the study itself, as compared to the individuals in most primary studies. If we are to make inferences about the population of studies of interest, it is necessary to define the population a priori by articulating a set of criteria of the type of studies included versus excluded from this sampling frame.

Identifying Inclusion and Exclusion Criteria

As mentioned, the inclusion and exclusion criteria define the sampling frame of a meta-analysis. Establishing clear and explicit criteria will help guide the search process, a consideration particularly important if multiple individuals are working on the project. A second reason for identifying clear criteria is that it will help define the population of interest to which generalizations can be made. A final reason that clear criteria are necessary has to do with the ideas of transparency and replication. As with the sampling in well-conducted and well-reported primary studies, each decision and subsequent procedure utilized in the literature search of a meta-analysis must be transparent and replicable. Some of the more common search techniques and sources of information are described next.

Search Techniques and Identifying Resources

Many techniques have been used quite successfully toward the goal of searching the literature and identifying relevant resources. Two important concepts related to the literature search are recall and precision ( see   White, 2009 ). Recall is the percentage of studies retrieved that meet your inclusion criteria from all of those that actually exist. Precision is the percentage of studies retrieved that meet the inclusion criteria for the meta-analysis. The ideal literature search strategy provides both high recall and precision, although the reality is that decisions that affect efforts to improve recall often lower precision and vice versa.

By using multiple methods of searching for literature, meta-analysts strive to maximize recall without imposing impractical detriments on precision. The use of multiple search techniques helps this effort. The techniques most commonly used include searching: electronic databases using keywords, bibliographical reference volumes, unpublished works and other outlets (described below), conference presentations, funding agency lists, research registries, backward searches, forward searches, and personal communications with colleagues.

Electronic databases . Electronic databases are probably one of the most helpful tools for conducting literature searches developed in the past decades. Now, electronic database searches can identify as much of the relevant literature in a matter of hours or days, as would have taken weeks or months a few decades earlier (not to mention that these searches can be done from the comfort of one’s office rather than within the confines of a library). Most disciplines have electronic databases that serve primarily that particular discipline (e.g., PsychINFO for psychology, Medline for medicine, ERIC for education, etc.). With these and similar databases, the metaanalyst identifies the most relevant combination of keywords, wildcard marks (e.g., * ), and logical statements (e.g., and, or, not), and voluminous amounts of literature is quickly searched for matches. The electronic database is perhaps the most fruitful place to begin and is currently the primary tool used to search the literature.

Despite their advantages, it is worth mentioning a few cautions regarding electronic databases. First, an electronic search must not be used exclusively because of that which is not included in these databases. For example, many unpublished works might not be retrieved through electronic databases. Second, as mentioned previously, each discipline relies on one primary electronic database; therefore, multiple databases must be considered in your search. Third, electronic databases produce studies that match the keyword searches, but it is not possible to know what has been excluded. Using other search strategies and investigating why studies found by these strategies were not identified in the electronic database search is necessary to avoid unnecessary (and potentially embarrassing) omission of studies from a meta-analysis.

Bibliographical reference volumes . A method of locating relevant literature that was common as little as a decade ago is to search biographical reference volumes. These volumes are printed collections containing essentially the same information as electronic databases. Although these reference volumes are being phased out of circulation, you may find them useful if relevant literature was published some time ago (especially if the electronic databases have not yet incorporated this older literature).

Unpublished works . One of the challenges of meta-analysis has to do with publication bias ( see   Rothstein et al., 2005 ). If there is a tendency for significant findings to be more likely published than nonsignificant (presumably with smaller effect sizes) studies, then the exclusion of unpublished studies in a meta-analysis can be problematic. To balance this potential problem, the meta-analyst should make deliberate efforts to find and obtain unpublished studies. Some possible places to find such studies include conference program books, funding agency lists, and research registries.

Backward searches . Another technique commonly used in meta-analysis is the backward search. Once relevant reports are retrieved, it is recommended that the researcher thoroughly read each report and identify additional articles cited within these reports. This strategy is called a “backward” search because it proceeds backward in time from obtained studies toward previous studies.

Forward searches . A complimentary procedure, known as the forward search, involves searching for additional studies that have cited the relevant studies included in your meta-analysis (“forward” because the search proceeds from older studies to newer studies citing these previous works). To conduct this type of search, special databases (e.g., Social Science Citation Index) are used.

Personal communication with researchers in the field . A final search technique involves personal communication with the researchers in the field. It will be especially helpful to communicate with researchers in your field (those who will likely read your work) in an effort to locate resources that somehow escaped your comprehensive search efforts. An effective yet efficient way to do this is to simply email researchers in your field, let them know what type of meta-analysis you are conducting, and ask if they would be willing to peruse your reference list to see if there are any glaring oversights.

Coding Study Characteristics

In a meta-analysis, study characteristics are systematically coded for two reasons. First, this coded information is presented to describe the collective field being reviewed. For example, do studies primarily rely on White college students, or are the samples more diverse (either within or across studies)? Do studies rely on the same measures or types of measures, or has the phenomenon been studied using multiple measures?

A second reason for systematically coding study characteristics is for use as potential predictors of variation in effect sizes across studies (i.e., moderators, as described below in section titled Moderator Analyses). In other words, does variation across studies in the coded study characteristics co-occur with differences in results (i.e., effect sizes) from these studies? Ultimately, the decision of what study characteristics should be coded derives from the meta-analysts’ substantive understanding of the field. There are at least three general types of study features that are commonly considered: characteristics of the sample, the methodology, and the source.

Coding Sample Characteristics

Sample characteristics include any descriptions of the study samples that might systematically covary with study results (i.e., effect sizes). Some meta-analyses will include codes for the sampling procedures, such as whether the study used a representative sample or a convenience sample (e.g., college students), or whether the sample was selected from some specific setting, such as clinical treatment settings, schools, or prisons. Nearly all meta-analyses code various demographic features of the sample, such as the ethnic composition, proportion of the sample that is male or female, and the average age of participants in the sample.

Coding Methodological Characteristics

Potential methodological characteristics for coding include both design and measurement features. At a broad level, a meta-analyst might code broad types of designs, such as experimental, quasiexperimental, and single-subject ABAB studies. It might also be useful to code at more narrow levels, such as the type of control group used within experimental treatment studies (e.g., no contact, attention only, treatment as usual). Similarly, the types of measures used could be coded as either broad (e.g., parent vs. child reports) or narrow (CBCL vs. BASC parent reports). In practice, most meta-analysts will code methodological features at both broad and narrow levels, first considering broad-level features as predictors of variability in effect sizes, and then using more narrow-level feature if there exists unexplained variation in results within these broad features.

Coding Source Characteristics

Source characteristics include features of the report or author that might plausibly be related to study findings. The most commonly coded source characteristic is whether the study was published, which is often used to evaluate potential publication bias. The year of publication (or presentation, for unpublished works) is often used as a proxy for the historic time in which the study was conducted. If the year predicts differences in effect sizes, then this may be evidence for historic change in the phenomenon over time. Other source characteristics, such as characteristics of the researcher (e.g., gender, ethnicity, discipline), are less commonly coded but are possibilities. For example, some meta-analyses of gender differences have coded the gender of the first author to evaluate the possibility that the researchers’ presumed biases may somehow impact the results found (e.g., Card, Stucky, Sawalani, &Little, 2008 ).

Coding Effect Sizes

As mentioned, study results in meta-analysis are represented as effect sizes. To be useful in metaanalysis, a potential effect size needs to meet four criteria. First, it needs to quantify the direction and magnitude of a phenomenon of interest. Second, it needs to be comparable across studies that use different sample sizes and scales of measurement. Third, it needs to be either consistently reported in studies included in the meta-analysis or else it can be computed from commonly reported results. Fourth, it is necessary that the meta-analyst can compute its standard error, which is used for weighting of studies in subsequent meta-analytic combination and comparison.

The three effect sizes most commonly used in meta-analyses all index associations between two variables. The correlation coefficient (typically denoted as r ) quantifies associations between two continuous variables. The standardized mean differences are a family of effect sizes (we will focus on Hedges’ g ) that quantify associations between a dichotomous (group) variable and a continuous variable. The odds ratio (denoted as either o or OR) is a useful and commonly used index for associations between two dichotomous variables ( Fleiss, 1994 ). We next describe these three indexes of effect size, the correlation coefficient, the standardized mean difference, and the OR. After describing each of these effect sizes indexes, we will describe how these are computed from results commonly reported in empirical reports.

Correlation Coefficient

Correlation coefficients represent associations between two variables on a standardized scale from − 1 to +1. Correlations near 0 denote the absence of association between two variables, whereas positive values indicate that scores on one variable tend to be similar to scores on another (relatively high scores on one variable tend to occur with relatively high scores on the other, as do low scores tend to occur with low scores), whereas negative scores indicate the opposite (high scores with low scores). The correlation coefficient has the advantage of being widely recognized by scientists in diverse fields. A commonly applied suggestion is that r ≍ ±0.10 is considered small, r ≍ ±0.30 is considered medium, and r ≍ ±0.50 is considered large; however, disciplines and fields differ in their evaluations of what constitutes small or large correlations, and researchers should not be dogmatic in its application.

Although r has many advantages as an effect size, it has the undesirable property for meta-analysis of having sample estimates that are skewed around the population mean. For this reason, meta-analysts should transform r to Fisher’s Z r prior to analysis using the following equation:

Although Z r has desirable properties for meta-analytic combination and comparison, it is not very interpretable by most readers. Therefore, metaanalysts back-transform results in Z r metric (e.g., mean effect size) to r for reporting using the following equation:

As mentioned earlier, and will be described in greater detail below, it is necessary to compute the standard error of the estimation of the effect size ( Z r ) for use in weighting studies in meta-analysis. The equation for the standard error of Z r   ( S E Z r ) is a simple function of the study sample size:

Standardized Mean Differences

There exist several standardized mean differences, which index associations between a dichotomous “group” variable and a continuous variable. Each of these standardized mean differences indexes the direction and magnitude of differences between two groups in standard deviation units. We begin with one of the more common of these indices, Hedges’ g , which is defined as:

The numerator of this equation contains the difference between the means of two groups (groups 1 and 2) and will yield a positive value if group 1 has a higher mean than group 2 or a negative value if group 2 has a higher mean than group 1. Although it is arbitrary which group is designated 1 or 2, this designation must be consistent across all studies coded for a meta-analysis.

If all studies in a meta-analysis use the same measure, or else different measures with the same scale, then the numerator of this equation alone would suffice as an effect size for meta-analysis (this is the unstandardized mean difference). However, the more common situation is that different scales are used across different studies, and in this situation it would make no sense to attempt to combine these unstandardized mean differences across studies. To illustrate, if one study comparing treatment to control groups measured an outcome on a 1 to 100 scale and found a 10-point difference, whereas another study measured the outcome on a 0 to 5 scale and found a 2-point difference, then there would be no way of knowing which—if either—study had a larger effect. To make these differences comparable across studies, it is necessary to standardize them in some way, typically by dividing the mean difference by a standard deviation.

As seen in equation (4) above, this standard deviation in the divisor for g is the pooled (i.e., combined across the two groups) estimate of the population standard deviation. Other variants within the standardized mean difference family of effect sizes use different divisors. For example, the index d uses the pooled sample standard deviation and a less commonly used index, g Glass (also denoted as Glass’ Δ), uses the estimated population standard deviation for one group (the group that you believe is a more accurate estimate of population standard deviation, such as the control group if you believe that treatment impacts the standard deviation). The latter index ( g Glass ) is less preferred because it cannot be computed from some commonly reported statistics (e.g., t tests), and it is a poorer estimate if the standard deviations are, in fact, comparable across groups ( Hedges & Olkin, 1985 ).

In this chapter, we focus our attention primarily on g , and we will describe the computation of g from commonly reported results below. Like other standardized mean differences, g has a value of 0 when the groups do not differ (i.e., no association between the dichotomous group variable and the continuous variable), and positive or negative values depending on which group has a higher mean. Unlike r, g is not bounded at 1, but can have values greater than ±1.0 if the groups differ by more than one standard deviation.

Although g is a preferred index of standardized mean differences, it exhibits a slight bias when estimated from small samples (e.g., sample sizes less than 20). To correct for this bias, it is common to apply the following correction:

As with any effect size used in meta-analysis, it is necessary to compute the standard error of estimates of g for weighting during meta-analytic combination. The standard error of g is more precisely estimated using the sample sizes from both groups under consideration (i.e., n 1 and n 1 for groups 1 and 2, respectively) using the left portion of Equation 6 but can be reasonably estimated using overall sample size ( N Total ; right portion of Equation 6 ) when exact group sizes are unknown but approximately equal (no more than a 3-to-1 discrepancy in group sizes; Card, 2012 ; Rosenthal, 1991 ):

Odds Ratios

The odds ratio, denoted as either o or OR, is a useful index of associations between two dichotomous variables. Although readers might be familiar with other indices of two variable associations, such as the rate (also known as risk) ratio or the phi coefficient, the OR is advantageous because it is not affected by differences in the base rates of dichotomous variables across studies and is computed from a wider range of study designs ( see   Fleiss, 1994 ). The OR is estimated from 2 × 2 contingency tables by dividing the product of cell frequencies in the major diagonal (i.e., frequencies in the cells where values of the two variables are both 0 { n 00 }or both 1 { n 11 }) by the product of cell frequencies off the diagonal (i.e., frequencies in the cells where the two variables have different values, n 10 and n 01 ):

The OR has a rather different scale than either r or g . Values of 1.0 represent no association between the dichotomous variables, values from 1 down to 0 represent negative association, and values from 1 to infinity represent positive associations. Given this scale, o is obviously skewed; therefore, a log transformation is applied to o when included in a meta-analysis: ln ( o ). The standard error of this log-transformed odds ratio is a function of number of participants in each cell of the 2 × 2 contingency table:

Computing Effect Sizes From Commonly Reported Data

Ideally, all studies that you want to include in a meta-analysis will have effect sizes reported, and it is a fairly straightforward matter to simply record these. Unfortunately, many studies do not report effect sizes (despite many calls for this reporting; e.g., Wilkinson et al., 1999 ), and it is necessary to compute effect sizes from a wide variety of information reported in studies. Although it is not possible to consider all possibilities here, we next describe a few of the more common situations. Table 30.1 summarizes equations for computing r and g in these situations (note that it is typically necessary to reconstruct contingency tables from reported data to compute the odds ratio; see   Fleiss, 1994 ).

It is common for studies to report group comparisons in the form of either the (independent samples) t -test or as the results of a two-group (i.e., 1 df ) analysis of variance (ANOVA). This occurs either because the study focused on a truly dichotomous grouping variable (in which case, the desired effect size is a standardized mean difference such as g ) or because the study authors artificially dichotomized one of the continuous variables (in which case the desired effect size is r ). In these cases, either r or g can be computed from the t statistic of F ratio in Table 30.1 . For the F ratio, it is critical that the result is from a two-group (i.e., 1 df ) comparison (for discussion of computing effect sizes from > 1 df F ratios, see   Rosenthal, Rosnow, & Rubin, 2000 ). When computing g (but not r ), a more precise estimate can be made if the two group sizes are known; otherwise, it is necessary to use the approximations shown to the right of Table 30.1 (e.g., in the first row for g , the exact formula is on the left and the approximation is on the right).

An alternate situation is that the study has performed repeated-measures comparisons (e.g., pretreatment vs. posttreatment) and reported results of dependent, or repeated-measures, t -tests, or F ratios. The equations for computing r from these results are identical to those for computing from independent samples tests; however, for g , the equations differ for independent versus dependent sample statistics, as seen in Table 30.1 .

A third possibility is that the study authors represent both variables that constitute your effect size of interest as dichotomous variables. The study might report the 1 df χ 2 of this contingency or the data that can be used to construct the contingency table and the subsequent value. In this situation, r and g are computed from this χ 2 value and sample size ( N ). As with the F ratio, it is important to keep in mind that this equation only applied to 1 df χ 2 values (i.e., 2 × 2 contingency tables).

The last situation we will discuss is when the authors report none of the above statistics but do report a significance level (i.e., p ). Here, you can compute the one-tail standard normal deviate, Z , associated with this significance level (e.g., Z = 1.645 for p = 0.05) and then use the equations of Table 30.1 to compute r or g . These formulas are used when an exact significance level is reported (e.g., p = 0.027); if they are applied to ranges (e.g., p < 0.05), then they provide only a lower-bound estimate of the actual effect size.

Although we have certainly not covered all possible situations, these represent some of the most common situations you are likely to encounter when coding effect sizes for a meta-analysis. For details of these and other situations in which you might code effect sizes, see   Card (2012 ) or Lipsey and Wilson (2001 ).

Analysis of Mean Effect Sizes and Heterogeneity

After coding study characteristics and effect sizes from all studies included in a meta-analysis, it is possible to statistically combine and compare results across studies. In this section, we describe a method (fixed effects) of computing a mean effect size and making inferences about this mean. We then describe a test of heterogeneity that informs whether the between-study variability in effect sizes is greater than expectable by sampling fluctuation alone. Finally, we describe an alternative approach to computing mean effect sizes (random effects) that accounts for between-study variability.

Fixed-Effects Means

One of the primary goals of meta-analytic combination of effect sizes from multiple studies is to estimate an average effect size that exists in the literature and then to make inferences about this average effect size in the form of statistical significance and/or confidence intervals. Before describing how to estimate and make inferences about a mean effect size, we briefly describe the concept of weighting.

Weighting in Meta-Analysis . Nearly all (and all that we describe here) analyses of effect sizes in meta-analysis apply weights to studies. These weights are meant to in dex the degree of precision in each study’s estimate of the population effect size, such that studies with more precise estimates receive greater weight in the analyses than studies with less precise estimates. The most straightforward weight is the inverse of the variance of a study’s estimate of the population effect size. In other words, the weight of study i is the inverse of the squared standard error from that study:

As described above, the standard error of a study largely depends on the sample size (and for g , the effect size itself), such that studies with large samples have smaller standard errors than studies with small samples. Therefore, studies with large samples have larger weights than studies with smaller samples.

Fixed-Effects Mean Effect Sizes . After computing weights for each study using the equation above, estimating the mean effect size ( E ¯ S ¯ ) across studies is a relatively simple matter of computing the weighted mean of effect sizes across all studies:

This value represents the estimate of a single effect size in the population based on information combined from all studies included in the meta-analysis. Because it is often useful to draw inferential conclusions, the standard error of this estimate is computed using the equation:

This standard error can then be used to compute either statistical significance or confidence intervals. For determining statistical significance, the mean effect size is divided by the standard error, and the resulting ratio is evaluated as a standard normal deviate (i.e., Z -test, with, e.g., values larger than ±1.96 having p < 0.05). For computing confidence intervals, the standard error is multiplied by the standard normal deviate associated with the desired confidence interval (e.g., Z = 1.96 for a 95% confidence interval), and this product is then subtracted from and added to the mean effect size to identify the lower- and upper-bounds of the confidence interval.

If the effect size chosen for the meta-analysis (i.e., r, g , or o ) was transformed prior to analyses (e.g., r to Z r ), then the mean effect size and boundaries of its confidence interval will be in this transformed metric. It is usually more meaningful to back-transform these values to their original metrics for reporting.

Heterogeneity

In addition to estimating a mean effect size, meta-analysts evaluate the variability of effect sizes across studies. Some degree of variability in effect sizes across studies is always expectable; the fact that different studies relied on different samples results in somewhat different estimates of effect sizes because of sampling variability. In situations where effect sizes differ by an amount expectable due to sampling variability, the studies are considered homogeneous with respect to their population effect sizes. However, if effect sizes vary across studies more than expected by sampling fluctuation alone, then they are considered heterogeneous (or varying) with respect to their population effect sizes.

It is common to perform a statistical test to evaluate heterogeneity. In this test, the null hypothesis is of homogeneity, or no variability, in population effect sizes across studies (i.e., any variability in sample effect sizes is caused by sampling variability), whereas the alternative hypothesis is of heterogeneity, or variability, in population effect sizes across studies (i.e., variability in sample effect sizes that is not accounted for by sampling variability alone). The result of this test is denoted by Q :

The statistical significance of this Q is evaluated as a χ 2 distribution with df = number of studies – 1. You will note that this equation has two forms. The left portion of Equation 12 is the definitional equation, which makes clear that the squared deviation of the effect size from each study i from the overall mean effect size is being weighted and summed across studies. Therefore, small deviations from the mean will contribute to small values of Q (homogeneity), whereas large deviations from the mean will contribute to large values of Q (heterogeneity). The right portion of Equation 12 is an algebraic rearrangement that simplifies computation (i.e., a computational formula).

Results of this test have implications for subsequent analyses. Specifically, a conclusion of homogeneity (more properly, failure to conclude heterogeneity) suggests that the fixed-effects mean described above is an acceptable way to summarize effect sizes, and this conclusion may contraindicate moderator analyses (described below). In contrast, a conclusion of heterogeneity implies that the fixedeffects mean is not an appropriate way to summarize effect sizes, but, rather, a random-effects model (described in the next section ) should be used. Further, a conclusion of heterogeneity indicates that moderator analyses (described below) may help explain this between-study variance (i.e., heterogeneity). It is worth noting that the result of this heterogeneity test is not the sole basis of deciding to use random-effects models or to conduct moderator analyses, and meta-analysts often base these decisions on conceptual rather than empirical grounds ( see   Card, 2012 ; Hedges & Vevea, 1998 ).

Random-Effects Means

Estimation of means via a random-effects model relies on a different conceptual model and analytic approach than estimation via a fixed-effects model. We describe this conceptual model and estimation procedures next.

Conceptualization of Random-Effects Means . Previously, when we described estimation of a fixedeffects mean, we describe a single population effect size. In contrast, a random-effects model assumes that there is a normal distribution of population effect sizes. This distribution of population effect sizes has a mean, which we estimate as described next. However, it also has a degree of spread, which can be indexed by the standard deviation (or variance) of effect sizes at the population level. To explicate the assumptions in equation form, the fixed- and random-effects models assume that the effect sizes observed in study i ( ES i ) are a function of the following, respectively:

In both Equation 13 (fixed effects) and Equation 14 (random effects), effect sizes in a study partly result from the sampling fluctuation of that study (ε i ). In the fixed-effects model, this sampling fluctuation is around a single population effect size (θ). In contrast, the random-effects model specifies that the population effect size is a function of both a mean population effect size (μ) as well as the deviation of the population effect size of study i from this mean (ξ i ). Although it is impossible to know the sampling fluctuation and the population deviation from a single study, it is possible to estimate the respective variances of each across studies.

Estimating Between-Study Population Variance . We described above the heterogeneity test, indexed by Q , which is a statistical test of whether variability in observed effect sizes across studies could be accounted for by sampling variability alone (i.e., the null hypothesis of homogeneity) or was greater than expected by sampling variability (i.e., the alternate hypothesis of heterogeneity). To estimate betweenstudy population variance τ 2 in effect sizes, we evaluate how much greater Q is than that expected under the null hypothesis of homogeneity (i.e., sampling variance alone):

Note that this equation is used only if Q ≥ k − 1 to avoid negative variance estimates (if Q < k − 1, τ 2 = 0). Although this equation is not intuitively obvious, consideration of the numerator helps clarify. Recall that large values of Q result when studies have effect sizes with large deviations from the mean effect size and that under the null hypothesis of homogeneity, Q is expected to equal the number of studies ( k ) minus 1. To the extent that Q is much larger than this expected value, the numerator of this equation will be large, implying large population between-study variability. In contrast, if Q is not much higher than the expected value under homogeneity, then the population between-study variability will be near zero.

Estimating Random-Effects Means . If studies have a sizable amount of randomly distributed between-study variance in their population effect sizes, then this implies that each is a less precise estimate of mean population effect size. In other words, each contains more uncertainty as information for estimating this value. To capture this uncertainty, or lower precision, analyses under the random-effects model use a different weight than those of the fixed-effects model. Specifically, the random-effects weight, denoted as w ∗ (or sometimes w RE ), for study i is the inverse of the sum of this between-study variance (τ 2 ) and the sampling variance for that study (i.e., squared standard error, S E i 2 ):

This random-effects weight will be smaller than the comparable fixed-effects weight, with the discrepancy increasing with greater between-study variance. These random-effects weights are simply used in the equations above to estimate a random-effects mean effect size (Equation 10 ), as well as a standard error for this mean (Equation 9 ) for inferential tests.

Moderator Analyses

Moderator analyses are another approach to managing heterogeneity in effect sizes ( Hedges & Pigott, 2004 ), but here the focus is on explaining (versus simply modeling as random) this between-study variance. These analyses use coded study characteristics to predict effect sizes; the reason these analyses are called “moderator” analyses is because they evaluate whether the effect size—a two-variable association—differs depending on the level of the third, moderator, variable—the study characteristic. It is often of primary interest to understand whether the association between two variables differs based on the level of a third variable (the moderator). Therefore, moderator analyses identifying those characteristics of the study that lead to higher or lower effects sizes are very commonly performed in meta-analyses. In this section, we briefly consider two types of moderators (i.e., categorical and continuous) along with the procedures used to inves-tigate these two types of moderators in meta-analysis (i.e., an adapted ANOVA procedure and a multiple regression procedure, respectively).

Single Categorical Moderator

A categorical variable is any variable on which a participants, observations or, in the case of metaanalysis, studies can be distinctly classified. Testing categorical moderators in meta-analysis involves comparing the mean effects of groups of studies classified by their status on some categorical variable.

Evaluating the Significance of a Categorical Moderator . Categorical moderator analysis in meta-analysis is similar to ANOVA in primary research. In the context of primary research, ANOVA partitions variability between groups of individuals into variability between and within these groups. Similarly, in meta-analysis, the ANOVA procedure is used to partition between-study heterogeneity into heterogeneity that exists between and within groups of studies. Earlier (Equation 12 ), we provided equations for quantifying the heterogeneity as Q ; we now provide this equation again, but now specifying that this is the total heterogeneity among studies:

This Q Total refers to the heterogeneity that exists across all studies. It can be partitioned into between-group ( Q Between ) and within-group ( Q Within ) components by the fact that Q Total = Q Between + Q Within . It is simpler to compute Q Within than Q Between , so it is common to subtract this from the total heterogeneity to obtain the between group variance. Within each group of studies, g , the heterogeneity can be estimated among just the studies in this group:

Then, these estimates of heterogeneity within each group can be summed across groups to yield the within study heterogeneity:

As stated above, testing categorical moderators within an ANOVA framework is done by separating the total heterogeneity ( Q Total ) into between-group ( Q Between ) and within-group ( Q Within ) heterogeneity. Therefore, after computing the total heterogeneity ( Q Total ) and the within-group heterogeneity ( Q Within ), you simply subtract the within-group heterogeneity from the total heterogeneity to find Q Between . This value is evaluated as a χ 2 distribution with df = number of groups – 1. If this value is statistically significant, then this is evidence that the level of the categorical moderator predicts variability in effect sizes—moderation.

Single Continuous Moderator

A continuous study characteristic is one that is measured on a scale that can potentially take on an infinite, or at least large number of values. In metaanalysis, a continuous moderator is a coded study characteristic (e.g., sample age, SES) that varies along a continuum of values and is hypothesizes to predict effect sizes.

Similarly to the use of an adapted ANOVA procedure in the evaluation of categorical moderators in meta-analysis, we use an adapted multiple regression procedure for the evaluation of continuous moderators in meta-analysis ( Hedges & Pigott, 2004 ). This adaptation to the evaluation of a continuous moderator involves a weighted regression of the effect sizes (dependent variable) onto the continuous moderator.

To evaluate potential moderation of a continuous moderator within a multiple regression framework, we regress the effect sizes onto the hypothesized continuous moderator using a standard regression equation: Z ES = B O + B 1 (Study Characteistic) + e , using w as the (weighted least squares) weight. From the results, we are interested in the sum of squares of the regression model (which is the heterogeneity accounted for by the linear regression model, Q Regression , and evaluated on a chi-square distribution with df = number of predictors), and sometimes the residual sum of squares (which is Q Residual , or heterogeneity not explained by the study characteristic). The unstandardized regression coefficient indicates how the effect size changes per unit change in the continuous moderator. The standard error of this coefficient is inaccurate in the regression output and must be adjusted by dividing it by the square root of the MS Residual .

The statistical significance of the predictor can also be evaluated by dividing the regression coefficient ( B 1 ) by the adjusted standard error, evaluated on the standard two-tail Z distribution. Interpretation of moderation with continuous variables is not as straightforward as with categorical moderators; it is necessary to compute implied effect sizes at different levels of the continuous moderator.

Multiple Regression to Analyze Categorical Moderators

Thus far we have considered moderation by a single categorical variable within an ANOVA framework and by a continuous variable within a regression framework. Next, we address categorical moderators within a multiple regression framework. Before doing so, it is useful to consider how the analyses we have described to this point fit within this general multiple regression framework.

The Empty Model . By empty model, we are referring to a model that includes only an intercept (a constant value of 1 for all cases) as a predictor. A weighted regression of effect sizes predicted only by a constant is often useful for an initial analysis of the mean effect size and to evaluate heterogeneity of these effect sizes across studies. The following equation accomplishes this:

In this empty model, the intercept regression coefficient is the mean effect size, and the sum of squares of the residual is the heterogeneity.

Use of Dummy Variables to Analyze Categorical Moderators . To evaluate the categorical moderators in this meta-regression framework, dummy variables can be used to represent group membership. Here, we select a reference group to which we would assign the value 0 for all the dummy codes for studies using that particular reference group, and each dummy variable represents the difference of another group relative to this reference group. The effect size is regressed onto the dummy variables, weighted by the inverse variance weight w , with the following equation:

The results of this regression are interpreted as above. The Q Regression is equivalent to the Q Between of the ANOVA framework and is used to determine whether there is categorical moderation. To identify the particular groups that differ from the reference group, the regression coefficients for the dummy variables are considered. Again, the standard errors of these coefficients are inaccurate and need to be adjusted as described above.

Multiple Moderators . A multiple regression framework can also be used to evaluate multiple categorical and/or continuous predictors in meta-analysis. Here, it is likely of interest to consider both the overall regression model ( Q Regression ) as well as the results of particular predictors. The former is evaluated by interpreting the model sum of squares as a χ 2 distribution with df = number of predictors – 1. The latter is evaluated by dividing the regression coefficients by their adjusted standard errors.

Limitations to Interpretation of Moderators

Clearly, moderation analyses can enhance the conclusions drawn from meta-analysis, but there are some limitations that need also be considered. The first consideration is that of multicolinearity in meta-analytic moderator analyses. It is likely that some moderator variables will be correlated, but this can be assessed by regressing each moderator onto the set of other moderators using the weights you used in the moderator analyses. The second limitation is the possibility that uncoded variables are confounding the association and moderation of the variables that are coded. The best approach to avoid confounding variables is to code as many variables as possible. Finally, it will be important to feel confident that the literature included in your synthesis adequately covers the range of potential moderator values. This can best be analyzed by plotting the included studies at the various levels of the moderator.

Advanced Topics

Given the existence of meta-analytic techniques over several decades, and their widespread use during this time, it is not surprising that there exists a rich literature on meta-analytic techniques. Although space has precluded us from discussing all of these topics, we next briefly describe a few of the more advanced topics important in this field.

Alternative Effect Sizes

The three effect sizes described in this chapter (i.e., r, g , and o ) quantify two-variable associations and are the most commonly used effect sizes for meta-analysis. However, there exist many other possibilities that might be considered.

Single Variable Effect Sizes . In some cases, it may be valuable to meta-analytically combine and/or compare information about single variables rather than two-variable associations. Central tendency can be indexed by the mean for continuous data or by the proportion for dichotomous variables; both of these effect sizes can be used in meta-analyses ( see   Lipsey & Wilson, 2001 ). It is also possible to use standard deviations or variances as effect sizes for meta-analysis to draw conclusions about interindi-vidual differences. Meta-analytic combination of means and variances require that the same measure, or else different measures on the same scale, be used for all studies.

Meaningful Metric . The effect sizes we have described are all in some standardized metric. However, there may be instances when the scales of variables comprising the effect size are meaningful, and therefore it is useful to use unstandard-ized effect sizes. Meta-analysis of such effect sizes were described in a special section of the journal Psychological Methods ( see   Becker, 2003 ).

Multivariate Effect Sizes . Many research questions go beyond two-variable associations to consider multivariate effect sizes, such as whether X uniquely predicts Y above and beyond Z . It is statistically possible to meta-analytically combine and compare multivariate effects sizes, such as regression coefficients or partial/semipartial correlation to address the example associations among X, Y , and Z . However, it is typically not possible in practice to use multivariate effect sizes for meta-analyses. The primary reason is that their use would require that the same multivariate analyses are performed and reported across studies in the meta-analysis. For example, it would be necessary for all studies included to report the regression of Y on X controlling for Z ; studies that failed to control for Z , that instead controlled for W , or that controlled for both Z and W could not be included. A more tractable alternative to the meta-analysis of multivariate effect sizes is to perform multivari-ate meta-analysis of bivariate effect sizes, which we briefly describe below.

Artifact Corrections

Artifacts are study imperfections that lead to biases—typically underestimations—of effect sizes. For example, it is well known that unreliability of a measure attenuates (i.e., reduces) the magnitude of observed associations that this variable has with others relative to what would have been found with a perfectly measured variable. In addition to measurement unreliability, other artifacts include imperfect validity of measures, artificial dichotomization of continuous variables, and range restriction of the sample on a variable included in the effect size (direct range restriction) or another variable closely related to a variable in the effect size (indirect range restriction).

The general approach to correcting for artifacts is to compute a correction factor for each artifact using one of a variety of equations ( see   Hunter & Schmidt, 2004 ). For example, one of the more straightforward corrections is for unreliability of a measure of X (where r xx is the reliability of X ):

Each of the artifact corrections may yield a correction factor, which are then multiplied together to yield an overall artifact multiplier ( a ). This artifact multiplier is then used to estimate an adjusted effect size from the observed effect size to index what the effect size would likely have been if the artifacts (study imperfections) had not existed:

This estimation of artifact-free effect sizes from observed effect sizes is unbiased (i.e., it will not consistently over- or underestimate the true effect size), but it is also not entirely precise. In other words, the artifact correction introduces additional uncertainty in the effect size estimate that must be considered in the meta-analysis. Specifically, the standard error of the effect size, which can be thought of as representing imprecision in the estimate of the effect size, is also adjusted by this artifact multiplier to account for this additional uncertainty introduced by artifact correction:

Multivariate Meta-Analysis

Multivariate meta-analysis is a relatively new and underdeveloped approach, but one that has great potential for use. Because the approach is fairly complex, and there is not general agreement on what techniques are best in different situations, we describe this approach in fairly general terms, referring interested readers to Becker (2009 ) or Cheung and Chan (2005 ).

The key idea of multivariate meta-analysis is to meta-analytically combine multiple bivariate effect sizes, which are then used as sufficient statistics for multivariate analyses. For example, to fit a model in which variable X is regressed on variables Y and Z , you would perform three meta-analyses of the three correlations ( r XY , r XZ , and r YZ ), and this matrix of meta-analytically combined correlations would then be used to estimate the multiple regression parameters.

Although the logic of this approach is reasonably simple, the application is much more complex. Challenges include how one handles the likely possibility that different studies provide different effect sizes, what the effective sample size is for the multivariate model when different studies inform different correlations, how (or even whether) to test for and potentially model between-study heterogeneity, and how to perform moderator analyses. Answers to these challenges have not been entirely agreed upon even by quantitative experts, making it difficult for those wishing to apply these models to answer substantive research questions. However, these models offer an extremely valuable potential for extending meta-analytic techniques to answer richer research questions than two-variable associations that are the typical focus of meta-analyses.

Although we have been able to provide only a brief overview of meta-analysis in this chapter, we hope that the opportunities of this methodology are clear. Given the overwhelming and increasing quantity of empirical research in most fields, techniques for best synthesizing the existing research are a critical tool in advancing our understanding.

Becker, B. J. ( 2003 ). Introduction to the special section on metric in meta-analysis.   Psychological Methods, 8, 403–405.

Google Scholar

Becker, B. J. ( 2009 ). Model-based meta-analysis. In H. Cooper , L. V. Hedges , & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed.) (pp. 377–395). New York: Russell Sage Foundation

Google Preview

Card, N. A. ( 2012 ). Meta-analysis: Quantitative synthesis of social science research . New York: Guilford.

Card, N. A. , Stucky, B. D. , Sawalani, G. M. , & Little, T. D. ( 2008 ). Direct and indirect aggression during childhood and adolescence: A meta-analytic review of gender differences, intercorrelations, and relations to maladjustment.   Child Development, 79, 1185–1229.

Chalmers, I. , Hedges, L.V. , & Cooper, H. ( 2002 ). A brief history of research synthesis.   Evaluation and Health Professions, 25, 12–37.

Cheung, M. W. L. , & Chan, W. ( 2005 ). Meta-analytic structural equation modeling: A two-stage approach.   Psychological Methods, 10, 40–64.

Cochran, W. G. ( 1937 ). Problems arising in the analysis of a series of similar experiments.   Journal of the Royal Statistical Societ y, 4(Suppl.), 102–118.

Cooper, H. M. ( 1988 ). Organizing knowledge syntheses: A taxonomy of literature reviews.   Knowledge in Societ y, 1, 104–126.

Cooper, H. M. , & Hedges, L. V. ( 2009 ). Research synthesis as a scientificprocess. In H. Cooper , L. V. Hedges , & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed.) (pp. 3–16). New York: Russell Sage Foundation.

Eysenck, H. J. ( 1978 ). An exercise in mega-silliness.   American Psychologist, 33, 517.

Fleiss, J. H. ( 1994 ). Measures of effect size for categorical data. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 245–260). New York: Russell Sage Foundation.

Glass, G. V. ( 1976 ). The critical state of the critical review article.   Quarterly Review of Biology , 50th Anniversary Special Issue (1926–1976), 415–418.

Glass, G. V. , McGraw, B. , & Smith, M. L. ( 1981 ). Meta-analysis in social research . Thousand Oaks, CA: Sage.

Glass, G. V. & Smith, M. ( 1979 ). Meta-analysis of the relationship between class size and achievement.   Educational Evaluation and Policy Analysi s, 1, 2–16.

Hedges, L. V. ( 1987 ). How hard is hard science, how soft is soft science? The empirical cumulativeness of research.   American Psychologist, 42, 443–455.

Hedges, L. V. ( 1992 ). Meta-analysis.   Journal of Educational Statistics, 17, 279–296.

Hedges, L. V. , & Olkin, I. ( 1985 ). Statistical methods for meta-analysis . San Diego, CA: Academic Press.

Hedges, L. V. & Pigott, T. D. ( 2004 ). The power of statistical tests for moderators in meta analysis.   Psychological Method s, 9, 426–445.

Hedges, L. V. , & Vevea, J. L. ( 1998 ). Fixed- and random-effects models in meta-analysis.   Psychological Method s, 3, 486–504.

Hunt, M. ( 1997 ). How science takes stock: The story of meta-analysis . New York: Russell Sage Foundation.

Hunter, J. E. , & Schmidt, F. L. ( 2004 ). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). Thousand Oaks, CA: Sage.

Hunter, J. E. , Schmidt, F. L. , & Hunter, R. ( 1979 ). Differential validity of employment tests by race: A comprehensive review and analysis.   Psychological Bulletin, 86, 721–735.

Hunter, J. E. , Schmidt, F. L. , & Jackson, G. B. ( 1982 ). Meta-analysis: Cumulating research findings across studies . Beverly Hills, CA: Sage.

Lipsey, M. W. , & Wilson, D. B. ( 2001 ). Practical meta-analysis . Thousand Oaks, CA: Sage.

Maxwell, S. E. ( 2004 ). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies.   Psychological Method s, 9, 147–163.

Olkin , ( 1990 ). History and goals. In K. W. Wachter & M. L. Straf (Eds.), The future of meta analysis (pp. 3–10). New York: Russell Sage Foundation.

Pearsons, K. ( 1904 ). Report on certain enteric fever inoculation statistics.   British Medical Journa l, 3, 1243–1246.

Rosenthal, R. ( 1979 ). The “file drawer problem” and tolerance for null results.   Psychological Bulletin, 86, 638–641.

Rosenthal, R. ( 1984 ). Meta-analytic procedures for social research . Beverly Hills, CA: Sage.

Rosenthal, R. ( 1991 ). Meta-analytic procedures for social research (revised edition). Newbury Park, CA: Sage.

Rosenthal, R. , Rosnow, R. L. , & Rubin, D. B. ( 2000 ). Contrasts and effect sizes in behavioral research: A correlational approach . New York: Cambridge University Press.

Rosenthal, R. & Rubin, D. ( 1978 ). Interpersonal expectancy effects: The first 345 studies.   Behavioral and Brain Science s, 3, 377–415.

Rothstein, H. R. , Sutton, A. J. , & Borenstein, M. (Eds.) ( 2005 ). Publication bias in meta analysis: Prevention, assessment and adjustments . Hoboken, NJ: Wiley.

Schmidt, F. L. ( 1992 ). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology.   American Psychologist, 47, 1173–1181.

Schmidt, F. L. , & Hunter, J. E. ( 1977 ). Development of a general solution to the problem of validity generalization.   Journal of Applied Psychology, 62, 529–540.

Smith, M. L. & Glass, G.V. ( 1977 ). Meta-analysis of psychotherapy outcome studies.   American Psychologist, 32, 752–760.

White, H. D. ( 2009 ). Scientific communication and literature retrieval. In H. Cooper , L. V. Hedges , & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed.) (pp. 51–71). New York: Russell Sage Foundation.

Wilkinson, L. , & The Task Force on Statistical Significance ( 1999 ). Statistical methods in psychology journals: Guidelines and explanations.   American Psychologist, 54, 594–604.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Published: 08 March 2018

Meta-analysis and the science of research synthesis

• Jessica Gurevitch 1 ,
• Julia Koricheva 2 ,
• Shinichi Nakagawa 3 , 4 &
• Gavin Stewart 5  

Nature volume  555 ,  pages 175–182 ( 2018 ) Cite this article

54k Accesses

865 Citations

879 Altmetric

Metrics details

• Biodiversity
• Outcomes research

Meta-analysis is the quantitative, scientific synthesis of research results. Since the term and modern approaches to research synthesis were first introduced in the 1970s, meta-analysis has had a revolutionary effect in many scientific fields, helping to establish evidence-based practice and to resolve seemingly contradictory research outcomes. At the same time, its implementation has engendered criticism and controversy, in some cases general and others specific to particular disciplines. Here we take the opportunity provided by the recent fortieth anniversary of meta-analysis to reflect on the accomplishments, limitations, recent advances and directions for future developments in the field of research synthesis.

This is a preview of subscription content, access via your institution

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 51 print issues and online access

185,98 € per year

only 3,65 € per issue

Rent or buy this article

Prices vary by article type

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

Eight problems with literature reviews and how to fix them

Neal R. Haddaway, Alison Bethel, … Gavin B. Stewart

The past, present and future of Registered Reports

Christopher D. Chambers & Loukia Tzavella

Raiders of the lost HARK: a reproducible inference framework for big data science

Mattia Prosperi, Jiang Bian, … Mo Wang

Jennions, M. D ., Lortie, C. J. & Koricheva, J. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 23 , 364–380 (Princeton Univ. Press, 2013)

Article   Google Scholar  

Roberts, P. D ., Stewart, G. B. & Pullin, A. S. Are review articles a reliable source of evidence to support conservation and environmental management? A comparison with medicine. Biol. Conserv. 132 , 409–423 (2006)

Bastian, H ., Glasziou, P . & Chalmers, I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 7 , e1000326 (2010)

Article   PubMed   PubMed Central   Google Scholar  

Borman, G. D. & Grigg, J. A. in The Handbook of Research Synthesis and Meta-analysis 2nd edn (eds Cooper, H. M . et al.) 497–519 (Russell Sage Foundation, 2009)

Ioannidis, J. P. A. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 94 , 485–514 (2016)

Koricheva, J . & Gurevitch, J. Uses and misuses of meta-analysis in plant ecology. J. Ecol. 102 , 828–844 (2014)

Littell, J. H . & Shlonsky, A. Making sense of meta-analysis: a critique of “effectiveness of long-term psychodynamic psychotherapy”. Clin. Soc. Work J. 39 , 340–346 (2011)

Morrissey, M. B. Meta-analysis of magnitudes, differences and variation in evolutionary parameters. J. Evol. Biol. 29 , 1882–1904 (2016)

Article   CAS   PubMed   Google Scholar  

Whittaker, R. J. Meta-analyses and mega-mistakes: calling time on meta-analysis of the species richness-productivity relationship. Ecology 91 , 2522–2533 (2010)

Article   PubMed   Google Scholar  

Begley, C. G . & Ellis, L. M. Drug development: Raise standards for preclinical cancer research. Nature 483 , 531–533 (2012); clarification 485 , 41 (2012)

Article   CAS   ADS   PubMed   Google Scholar  

Hillebrand, H . & Cardinale, B. J. A critique for meta-analyses and the productivity-diversity relationship. Ecology 91 , 2545–2549 (2010)

Moher, D . et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 6 , e1000097 (2009). This paper provides a consensus regarding the reporting requirements for medical meta-analysis and has been highly influential in ensuring good reporting practice and standardizing language in evidence-based medicine, with further guidance for protocols, individual patient data meta-analyses and animal studies.

Moher, D . et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst. Rev. 4 , 1 (2015)

Nakagawa, S . & Santos, E. S. A. Methodological issues and advances in biological meta-analysis. Evol. Ecol. 26 , 1253–1274 (2012)

Nakagawa, S ., Noble, D. W. A ., Senior, A. M. & Lagisz, M. Meta-evaluation of meta-analysis: ten appraisal questions for biologists. BMC Biol. 15 , 18 (2017)

Hedges, L. & Olkin, I. Statistical Methods for Meta-analysis (Academic Press, 1985)

Viechtbauer, W. Conducting meta-analyses in R with the metafor package. J. Stat. Softw. 36 , 1–48 (2010)

Anzures-Cabrera, J . & Higgins, J. P. T. Graphical displays for meta-analysis: an overview with suggestions for practice. Res. Synth. Methods 1 , 66–80 (2010)

Egger, M ., Davey Smith, G ., Schneider, M. & Minder, C. Bias in meta-analysis detected by a simple, graphical test. Br. Med. J. 315 , 629–634 (1997)

Article   CAS   Google Scholar  

Duval, S . & Tweedie, R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56 , 455–463 (2000)

Article   CAS   MATH   PubMed   Google Scholar  

Leimu, R . & Koricheva, J. Cumulative meta-analysis: a new tool for detection of temporal trends and publication bias in ecology. Proc. R. Soc. Lond. B 271 , 1961–1966 (2004)

Higgins, J. P. T . & Green, S. (eds) Cochrane Handbook for Systematic Reviews of Interventions : Version 5.1.0 (Wiley, 2011). This large collaborative work provides definitive guidance for the production of systematic reviews in medicine and is of broad interest for methods development outside the medical field.

Lau, J ., Rothstein, H. R . & Stewart, G. B. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 25 , 407–419 (Princeton Univ. Press, 2013)

Lortie, C. J ., Stewart, G ., Rothstein, H. & Lau, J. How to critically read ecological meta-analyses. Res. Synth. Methods 6 , 124–133 (2015)

Murad, M. H . & Montori, V. M. Synthesizing evidence: shifting the focus from individual studies to the body of evidence. J. Am. Med. Assoc. 309 , 2217–2218 (2013)

Rasmussen, S. A ., Chu, S. Y ., Kim, S. Y ., Schmid, C. H . & Lau, J. Maternal obesity and risk of neural tube defects: a meta-analysis. Am. J. Obstet. Gynecol. 198 , 611–619 (2008)

Littell, J. H ., Campbell, M ., Green, S . & Toews, B. Multisystemic therapy for social, emotional, and behavioral problems in youth aged 10–17. Cochrane Database Syst. Rev. https://doi.org/10.1002/14651858.CD004797.pub4 (2005)

Schmidt, F. L. What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. Am. Psychol. 47 , 1173–1181 (1992)

Button, K. S . et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14 , 365–376 (2013); erratum 14 , 451 (2013)

Parker, T. H . et al. Transparency in ecology and evolution: real problems, real solutions. Trends Ecol. Evol. 31 , 711–719 (2016)

Stewart, G. Meta-analysis in applied ecology. Biol. Lett. 6 , 78–81 (2010)

Sutherland, W. J ., Pullin, A. S ., Dolman, P. M . & Knight, T. M. The need for evidence-based conservation. Trends Ecol. Evol. 19 , 305–308 (2004)

Lowry, E . et al. Biological invasions: a field synopsis, systematic review, and database of the literature. Ecol. Evol. 3 , 182–196 (2013)

Article   PubMed Central   Google Scholar  

Parmesan, C . & Yohe, G. A globally coherent fingerprint of climate change impacts across natural systems. Nature 421 , 37–42 (2003)

Jennions, M. D ., Lortie, C. J . & Koricheva, J. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 24 , 381–403 (Princeton Univ. Press, 2013)

Balvanera, P . et al. Quantifying the evidence for biodiversity effects on ecosystem functioning and services. Ecol. Lett. 9 , 1146–1156 (2006)

Cardinale, B. J . et al. Effects of biodiversity on the functioning of trophic groups and ecosystems. Nature 443 , 989–992 (2006)

Rey Benayas, J. M ., Newton, A. C ., Diaz, A. & Bullock, J. M. Enhancement of biodiversity and ecosystem services by ecological restoration: a meta-analysis. Science 325 , 1121–1124 (2009)

Article   ADS   PubMed   CAS   Google Scholar  

Leimu, R ., Mutikainen, P. I. A ., Koricheva, J. & Fischer, M. How general are positive relationships between plant population size, fitness and genetic variation? J. Ecol. 94 , 942–952 (2006)

Hillebrand, H. On the generality of the latitudinal diversity gradient. Am. Nat. 163 , 192–211 (2004)

Gurevitch, J. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 19 , 313–320 (Princeton Univ. Press, 2013)

Rustad, L . et al. A meta-analysis of the response of soil respiration, net nitrogen mineralization, and aboveground plant growth to experimental ecosystem warming. Oecologia 126 , 543–562 (2001)

Adams, D. C. Phylogenetic meta-analysis. Evolution 62 , 567–572 (2008)

Hadfield, J. D . & Nakagawa, S. General quantitative genetic methods for comparative biology: phylogenies, taxonomies and multi-trait models for continuous and categorical characters. J. Evol. Biol. 23 , 494–508 (2010)

Lajeunesse, M. J. Meta-analysis and the comparative phylogenetic method. Am. Nat. 174 , 369–381 (2009)

Rosenberg, M. S ., Adams, D. C . & Gurevitch, J. MetaWin: Statistical Software for Meta-Analysis with Resampling Tests Version 1 (Sinauer Associates, 1997)

Wallace, B. C . et al. OpenMEE: intuitive, open-source software for meta-analysis in ecology and evolutionary biology. Methods Ecol. Evol. 8 , 941–947 (2016)

Gurevitch, J ., Morrison, J. A . & Hedges, L. V. The interaction between competition and predation: a meta-analysis of field experiments. Am. Nat. 155 , 435–453 (2000)

Adams, D. C ., Gurevitch, J . & Rosenberg, M. S. Resampling tests for meta-analysis of ecological data. Ecology 78 , 1277–1283 (1997)

Gurevitch, J . & Hedges, L. V. Statistical issues in ecological meta-analyses. Ecology 80 , 1142–1149 (1999)

Schmid, C. H . & Mengersen, K. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 11 , 145–173 (Princeton Univ. Press, 2013)

Eysenck, H. J. Exercise in mega-silliness. Am. Psychol. 33 , 517 (1978)

Simberloff, D. Rejoinder to: Don’t calculate effect sizes; study ecological effects. Ecol. Lett. 9 , 921–922 (2006)

Cadotte, M. W ., Mehrkens, L. R . & Menge, D. N. L. Gauging the impact of meta-analysis on ecology. Evol. Ecol. 26 , 1153–1167 (2012)

Koricheva, J ., Jennions, M. D. & Lau, J. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 15 , 237–254 (Princeton Univ. Press, 2013)

Lau, J ., Ioannidis, J. P. A ., Terrin, N ., Schmid, C. H . & Olkin, I. The case of the misleading funnel plot. Br. Med. J. 333 , 597–600 (2006)

Vetter, D ., Rucker, G. & Storch, I. Meta-analysis: a need for well-defined usage in ecology and conservation biology. Ecosphere 4 , 1–24 (2013)

Mengersen, K ., Jennions, M. D. & Schmid, C. H. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J. et al.) Ch. 16 , 255–283 (Princeton Univ. Press, 2013)

Patsopoulos, N. A ., Analatos, A. A. & Ioannidis, J. P. A. Relative citation impact of various study designs in the health sciences. J. Am. Med. Assoc. 293 , 2362–2366 (2005)

Kueffer, C . et al. Fame, glory and neglect in meta-analyses. Trends Ecol. Evol. 26 , 493–494 (2011)

Cohnstaedt, L. W. & Poland, J. Review Articles: The black-market of scientific currency. Ann. Entomol. Soc. Am. 110 , 90 (2017)

Longo, D. L. & Drazen, J. M. Data sharing. N. Engl. J. Med. 374 , 276–277 (2016)

Gauch, H. G. Scientific Method in Practice (Cambridge Univ. Press, 2003)

Science Staff. Dealing with data: introduction. Challenges and opportunities. Science 331 , 692–693 (2011)

Nosek, B. A . et al. Promoting an open research culture. Science 348 , 1422–1425 (2015)

Article   CAS   ADS   PubMed   PubMed Central   Google Scholar  

Stewart, L. A . et al. Preferred reporting items for a systematic review and meta-analysis of individual participant data: the PRISMA-IPD statement. J. Am. Med. Assoc. 313 , 1657–1665 (2015)

Saldanha, I. J . et al. Evaluating Data Abstraction Assistant, a novel software application for data abstraction during systematic reviews: protocol for a randomized controlled trial. Syst. Rev. 5 , 196 (2016)

Tipton, E. & Pustejovsky, J. E. Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression. J. Educ. Behav. Stat. 40 , 604–634 (2015)

Mengersen, K ., MacNeil, M. A . & Caley, M. J. The potential for meta-analysis to support decision analysis in ecology. Res. Synth. Methods 6 , 111–121 (2015)

Ashby, D. Bayesian statistics in medicine: a 25 year review. Stat. Med. 25 , 3589–3631 (2006)

Article   MathSciNet   PubMed   Google Scholar  

Senior, A. M . et al. Heterogeneity in ecological and evolutionary meta-analyses: its magnitude and implications. Ecology 97 , 3293–3299 (2016)

McAuley, L ., Pham, B ., Tugwell, P . & Moher, D. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 356 , 1228–1231 (2000)

Koricheva, J ., Gurevitch, J . & Mengersen, K. (eds) The Handbook of Meta-Analysis in Ecology and Evolution (Princeton Univ. Press, 2013) This book provides the first comprehensive guide to undertaking meta-analyses in ecology and evolution and is also relevant to other fields where heterogeneity is expected, incorporating explicit consideration of the different approaches used in different domains.

Lumley, T. Network meta-analysis for indirect treatment comparisons. Stat. Med. 21 , 2313–2324 (2002)

Zarin, W . et al. Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review. BMC Med. 15 , 3 (2017)

Elliott, J. H . et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 11 , e1001603 (2014)

Vandvik, P. O ., Brignardello-Petersen, R . & Guyatt, G. H. Living cumulative network meta-analysis to reduce waste in research: a paradigmatic shift for systematic reviews? BMC Med. 14 , 59 (2016)

Jarvinen, A. A meta-analytic study of the effects of female age on laying date and clutch size in the Great Tit Parus major and the Pied Flycatcher Ficedula hypoleuca . Ibis 133 , 62–67 (1991)

Arnqvist, G. & Wooster, D. Meta-analysis: synthesizing research findings in ecology and evolution. Trends Ecol. Evol. 10 , 236–240 (1995)

Hedges, L. V ., Gurevitch, J . & Curtis, P. S. The meta-analysis of response ratios in experimental ecology. Ecology 80 , 1150–1156 (1999)

Gurevitch, J ., Curtis, P. S. & Jones, M. H. Meta-analysis in ecology. Adv. Ecol. Res 32 , 199–247 (2001)

Lajeunesse, M. J. phyloMeta: a program for phylogenetic comparative analyses with meta-analysis. Bioinformatics 27 , 2603–2604 (2011)

CAS   PubMed   Google Scholar  

Pearson, K. Report on certain enteric fever inoculation statistics. Br. Med. J. 2 , 1243–1246 (1904)

Fisher, R. A. Statistical Methods for Research Workers (Oliver and Boyd, 1925)

Yates, F. & Cochran, W. G. The analysis of groups of experiments. J. Agric. Sci. 28 , 556–580 (1938)

Cochran, W. G. The combination of estimates from different experiments. Biometrics 10 , 101–129 (1954)

Smith, M. L . & Glass, G. V. Meta-analysis of psychotherapy outcome studies. Am. Psychol. 32 , 752–760 (1977)

Glass, G. V. Meta-analysis at middle age: a personal history. Res. Synth. Methods 6 , 221–231 (2015)

Cooper, H. M ., Hedges, L. V . & Valentine, J. C. (eds) The Handbook of Research Synthesis and Meta-analysis 2nd edn (Russell Sage Foundation, 2009). This book is an important compilation that builds on the ground-breaking first edition to set the standard for best practice in meta-analysis, primarily in the social sciences but with applications to medicine and other fields.

Rosenthal, R. Meta-analytic Procedures for Social Research (Sage, 1991)

Hunter, J. E ., Schmidt, F. L. & Jackson, G. B. Meta-analysis: Cumulating Research Findings Across Studies (Sage, 1982)

Gurevitch, J ., Morrow, L. L ., Wallace, A . & Walsh, J. S. A meta-analysis of competition in field experiments. Am. Nat. 140 , 539–572 (1992). This influential early ecological meta-analysis reports multiple experimental outcomes on a longstanding and controversial topic that introduced a wide range of ecologists to research synthesis methods.

O’Rourke, K. An historical perspective on meta-analysis: dealing quantitatively with varying study results. J. R. Soc. Med. 100 , 579–582 (2007)

Shadish, W. R . & Lecy, J. D. The meta-analytic big bang. Res. Synth. Methods 6 , 246–264 (2015)

Glass, G. V. Primary, secondary, and meta-analysis of research. Educ. Res. 5 , 3–8 (1976)

DerSimonian, R . & Laird, N. Meta-analysis in clinical trials. Control. Clin. Trials 7 , 177–188 (1986)

Lipsey, M. W . & Wilson, D. B. The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis. Am. Psychol. 48 , 1181–1209 (1993)

Chalmers, I. & Altman, D. G. Systematic Reviews (BMJ Publishing Group, 1995)

Moher, D . et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of reporting of meta-analyses. Lancet 354 , 1896–1900 (1999)

Higgins, J. P. & Thompson, S. G. Quantifying heterogeneity in a meta-analysis. Stat. Med. 21 , 1539–1558 (2002)

Download references

Acknowledgements

We dedicate this Review to the memory of Ingram Olkin and William Shadish, founding members of the Society for Research Synthesis Methodology who made tremendous contributions to the development of meta-analysis and research synthesis and to the supervision of generations of students. We thank L. Lagisz for help in preparing the figures. We are grateful to the Center for Open Science and the Laura and John Arnold Foundation for hosting and funding a workshop, which was the origination of this article. S.N. is supported by Australian Research Council Future Fellowship (FT130100268). J.G. acknowledges funding from the US National Science Foundation (ABI 1262402).

Author information

Authors and affiliations.

Department of Ecology and Evolution, Stony Brook University, Stony Brook, 11794-5245, New York, USA

Jessica Gurevitch

School of Biological Sciences, Royal Holloway University of London, Egham, TW20 0EX, Surrey, UK

Julia Koricheva

Evolution and Ecology Research Centre and School of Biological, Earth and Environmental Sciences, University of New South Wales, Sydney, 2052, New South Wales, Australia

Shinichi Nakagawa

Diabetes and Metabolism Division, Garvan Institute of Medical Research, 384 Victoria Street, Darlinghurst, Sydney, 2010, New South Wales, Australia

School of Natural and Environmental Sciences, Newcastle University, Newcastle upon Tyne, NE1 7RU, UK

Gavin Stewart

You can also search for this author in PubMed   Google Scholar

Contributions

All authors contributed equally in designing the study and writing the manuscript, and so are listed alphabetically.

Corresponding authors

Correspondence to Jessica Gurevitch , Julia Koricheva , Shinichi Nakagawa or Gavin Stewart .

Ethics declarations

Competing interests.

The authors declare no competing financial interests.

Additional information

Reviewer Information Nature thanks D. Altman, M. Lajeunesse, D. Moher and G. Romero for their contribution to the peer review of this work.

Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

PowerPoint slides

Powerpoint slide for fig. 1, rights and permissions.

Reprints and permissions

About this article

Cite this article.

Gurevitch, J., Koricheva, J., Nakagawa, S. et al. Meta-analysis and the science of research synthesis. Nature 555 , 175–182 (2018). https://doi.org/10.1038/nature25753

Download citation

Received : 04 March 2017

Accepted : 12 January 2018

Published : 08 March 2018

Issue Date : 08 March 2018

DOI : https://doi.org/10.1038/nature25753

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Investigate the relationship between the retraction reasons and the quality of methodology in non-cochrane retracted systematic reviews: a systematic review.

• Azita Shahraki-Mohammadi
• Leila Keikha
• Razieh Zahedi

Systematic Reviews (2024)

Systematic review of the uncertainty of coral reef futures under climate change

• Shannon G. Klein
• Cassandra Roch
• Carlos M. Duarte

Nature Communications (2024)

Meta-analysis reveals weak associations between reef fishes and corals

• Pooventhran Muruga
• Alexandre C. Siqueira
• David R. Bellwood

Nature Ecology & Evolution (2024)

Farming practices to enhance biodiversity across biomes: a systematic review

• Felipe Cozim-Melges
• Raimon Ripoll-Bosch
• Hannah H. E. van Zanten

npj Biodiversity (2024)

Large language models reveal big disparities in current wildfire research

• Zhengyang Lin
• Anping Chen
• Shilong Piao

Communications Earth & Environment (2024)

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

• Meta-Analysis/Meta-Synthesis

Meta Analysis

Meta-analysis   is a set of statistical techniques for synthesizing data across studies. It is a statistical method for combining the findings from quantitative studies. It evaluates, synthesizes, and summarizes results. It may be conducted independently or as a specialized subset of a systematic review.  A systematic review attempts to collate empirical evidence that fits predefined eligibility criteria to answer a specific research question. Meta-analysis is a quantitative, formal, epidemiological study design used to systematically assess the results of previous research to derive conclusions about that body of research (Haidrich, 2010). Rigorously conducted meta-analyses are useful tools in evidence-based medicine . Outcomes from a meta-analysis may include a more precise estimate of the effect of a treatment or risk factor for disease or other outcomes. Not all systematic reviews include meta-analysis , but all meta-analyses are found in systematic reviews (Haidrich, 2010).

A Meta analysis is appropriate when a group of studies report quantitative results rather than qualitative findings or theory, if they examine the same or similar constructs or relationships, if they are derived from similar research designs and report the simple relationships between two variables rather than relationships that have been adjusted for the effect of additional variables (siddaway, et al., 2019).

Haidich A. B. (2010). Meta-analysis in medical research.  Hippokratia ,  14 (Suppl 1), 29–37.

Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019). How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses.  Annual Review of Psychology, 70 , 747–770.

Meta Synthesis

A meta synthesis is the systematic review and integration of findings from qualitative studies (Lachal et al., 2017). Reviews of qualitative information can be conducted and reported using the same replicable, rigorous, and transparent methodology and presentation. A meta-synthesis can be used when a review aims to integrate qualitative research.  A meta-synthesis attempts to synthesize qualitative studies on a topic to identify key themes, concepts, or theories that provide novel or more powerful explanations for the phenomenon under review (Siddaway et al., 2019).

Lachal, J., Revah-Levy, A., Orri, M., & Moro, M. R. (2017). Metasynthesis: An original method to synthesize qualitative literature in psychiatry.  Frontiers in Psychiatry, 8 , 269 . 

Siddaway, A. P., Wood, A. M., & Hedges, L. V. (2019). How to do a systematic review: A best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses.  Annual Review of Psychology, 70 , 747–770 .

• << Previous: Rapid Review
• Next: Selecting a Review Type >>
• Types of Questions
• Key Features and Limitations
• Is a Systematic Review Right for Your Research?
• Integrative Review
• Scoping Review
• Rapid Review
• Selecting a Review Type
• Reducing Bias
• Guidelines for Student Researchers
• Training Resources
• Register Your Protocol
• Handbooks & Manuals
• Reporting Guidelines
• PRESS 2015 Guidelines
• Search Strategies
• Selected Databases
• Grey Literature
• Handsearching
• Citation Searching
• Study Types & Terminology
• Quantitative vs. Qualitative Research
• Critical Appraisal of Studies
• Broad Functionality Programs & Tools
• Search Strategy Tools
• Deduplication Tools
• CItation Screening
• Critical Appraisal Tools
• Quality Assessment/Risk of Bias Tools
• Data Collection/Extraction
• Meta Analysis Tools
• Books on Systematic Reviews
• Finding Systematic Review Articles in the Databases
• Systematic Review Journals
• More Resources
• Evidence-Based Practice Research in Nursing
• Citation Management Programs
• Last Updated: Apr 5, 2024 2:54 PM
• URL: https://libguides.adelphi.edu/Systematic_Reviews

• Methodology
• Open access
• Published: 24 April 2023

Quantitative evidence synthesis: a practical guide on meta-analysis, meta-regression, and publication bias tests for environmental sciences

• Shinichi Nakagawa   ORCID: orcid.org/0000-0002-7765-5182 1 , 2 ,
• Yefeng Yang   ORCID: orcid.org/0000-0002-8610-4016 1 ,
• Erin L. Macartney   ORCID: orcid.org/0000-0003-3866-143X 1 ,
• Rebecca Spake   ORCID: orcid.org/0000-0003-4671-2225 3 &
• Malgorzata Lagisz   ORCID: orcid.org/0000-0002-3993-6127 1  

Environmental Evidence volume  12 , Article number:  8 ( 2023 ) Cite this article

6119 Accesses

14 Citations

25 Altmetric

Metrics details

Meta-analysis is a quantitative way of synthesizing results from multiple studies to obtain reliable evidence of an intervention or phenomenon. Indeed, an increasing number of meta-analyses are conducted in environmental sciences, and resulting meta-analytic evidence is often used in environmental policies and decision-making. We conducted a survey of recent meta-analyses in environmental sciences and found poor standards of current meta-analytic practice and reporting. For example, only ~ 40% of the 73 reviewed meta-analyses reported heterogeneity (variation among effect sizes beyond sampling error), and publication bias was assessed in fewer than half. Furthermore, although almost all the meta-analyses had multiple effect sizes originating from the same studies, non-independence among effect sizes was considered in only half of the meta-analyses. To improve the implementation of meta-analysis in environmental sciences, we here outline practical guidance for conducting a meta-analysis in environmental sciences. We describe the key concepts of effect size and meta-analysis and detail procedures for fitting multilevel meta-analysis and meta-regression models and performing associated publication bias tests. We demonstrate a clear need for environmental scientists to embrace multilevel meta-analytic models, which explicitly model dependence among effect sizes, rather than the commonly used random-effects models. Further, we discuss how reporting and visual presentations of meta-analytic results can be much improved by following reporting guidelines such as PRISMA-EcoEvo (Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Ecology and Evolutionary Biology). This paper, along with the accompanying online tutorial, serves as a practical guide on conducting a complete set of meta-analytic procedures (i.e., meta-analysis, heterogeneity quantification, meta-regression, publication bias tests and sensitivity analysis) and also as a gateway to more advanced, yet appropriate, methods.

Evidence synthesis is an essential part of science. The method of systematic review provides the most trusted and unbiased way to achieve the synthesis of evidence [ 1 , 2 , 3 ]. Systematic reviews often include a quantitative summary of studies on the topic of interest, referred to as a meta-analysis (for discussion on the definitions of ‘meta-analysis’, see [ 4 ]). The term meta-analysis can also mean a set of statistical techniques for quantitative data synthesis. The methodologies of the meta-analysis were initially developed and applied in medical and social sciences. However, meta-analytic methods are now used in many other fields, including environmental sciences [ 5 , 6 , 7 ]. In environmental sciences, the outcomes of meta-analyses (within systematic reviews) have been used to inform environmental and related policies (see [ 8 ]). Therefore, the reliability of meta-analytic results in environmental sciences is important beyond mere academic interests; indeed, incorrect results could lead to ineffective or sometimes harmful environmental policies [ 8 ].

As in medical and social sciences, environmental scientists frequently use traditional meta-analytic models, namely fixed-effect and random-effects models [ 9 , 10 ]. However, we contend that such models in their original formulation are no longer useful and are often incorrectly used, leading to unreliable estimates and errors. This is mainly because the traditional models assume independence among effect sizes, but almost all primary research papers include more than one effect size, and this non-independence is often not considered (e.g., [ 11 , 12 , 13 ]). Furthermore, previous reviews of published meta-analyses in environmental sciences (hereafter, ‘environmental meta-analyses’) have demonstrated that less than half report or investigate heterogeneity (inconsistency) among effect sizes [ 14 , 15 , 16 ]. Many environmental meta-analyses also do not present any sensitivity analysis, for example, for publication bias (i.e., statistically significant effects being more likely to be published, making collated data unreliable; [ 17 , 18 ]). These issues might have arisen for several reasons, for example, because of no clear conduct guideline for the statistical part of meta-analyses in environmental sciences and rapid developments in meta-analytic methods. Taken together, the field urgently requires a practical guide to implement correct meta-analyses and associated procedures (e.g., heterogeneity analysis, meta-regression, and publication bias tests; cf. [ 19 ]).

To assist environmental scientists in conducting meta-analyses, the aims of this paper are five-fold. First, we provide an overview of the processes involved in a meta-analysis while introducing some key concepts. Second, after introducing the main types of effect size measures, we mathematically describe the two commonly used traditional meta-analytic models, demonstrate their utility, and introduce a practical, multilevel meta-analytic model for environmental sciences that appropriately handles non-independence among effect sizes. Third, we show how to quantify heterogeneity (i.e., consistencies among effect sizes and/or studies) using this model, and then explain such heterogeneity using meta-regression. Fourth, we show how to test for publication bias in a meta-analysis and describe other common types of sensitivity analysis. Fifth, we cover other technical issues relevant to environmental sciences (e.g., scale and phylogenetic dependence) as well as some advanced meta-analytic techniques. In addition, these five aims (sections) are interspersed with two more sections, named ‘Notes’ on: (1) visualisation and interpretation; and (2) reporting and archiving. Some of these sections are accompanied by results from a survey of 73 environmental meta-analyses published between 2019 and 2021; survey results depict current practices and highlight associated problems (for the method of the survey, see Additional file 1 ). Importantly, we provide easy-to-follow implementations of much of what is described below, using the R package, metafor [ 20 ] and other R packages at the webpage ( https://itchyshin.github.io/Meta-analysis_tutorial/ ), which also connects the reader to the wealth of online information on meta-analysis (note that we also provide this tutorial as Additional file 2 ; see also [ 21 ]).

Overview with key concepts

Statistically speaking, we have three general objectives when conducting a meta-analysis [ 12 ]: (1) estimating an overall mean , (2) quantifying consistency ( heterogeneity ) between studies, and (3) explaining the heterogeneity (see Table 1 for the definitions of the terms in italic ). A notable feature of a meta-analysis is that an overall mean is estimated by taking the sampling variance of each effect size into account: a study (effect size) with a low sampling variance (usually based on a larger sample size) is assigned more weight in estimating an overall mean than one with a high sampling variance (usually based on a smaller sample size). However, an overall mean estimate itself is often not informative because one can get the same overall mean estimates in different ways. For example, we may get an overall estimate of zero if all studies have zero effects with no heterogeneity. In contrast, we might also obtain a zero mean across studies that have highly variable effects (e.g., ranging from strongly positive to strongly negative), signifying high heterogeneity. Therefore, quantifying indicators of heterogeneity is an essential part of a meta-analysis, necessary for interpreting the overall mean appropriately. Once we observe non-zero heterogeneity among effect sizes, then, our job is to explain this variation by running meta-regression models, and, at the same time, quantify how much variation is accounted for (often quantified as R 2 ). In addition, it is important to conduct an extra set of analyses, often referred to as publication bias tests , which are a type of sensitivity analysis [ 11 ], to check the robustness of meta-analytic results.

Choosing an effect size measure

In this section, we introduce different kinds of ‘effect size measures’ or ‘effect measures’. In the literature, the term ‘effect size’ is typically used to refer to the magnitude or strength of an effect of interest or its biological interpretation (e.g., environmental significance). Effect sizes can be quantified using a range of measures (for details, see [ 22 ]). In our survey of environmental meta-analyses (Additional file 1 ), the two most commonly used effect size measures are: the logarithm of response ratio, lnRR ([ 23 ]; also known as the ratio of means; [ 24 ]) and standardized mean difference, SMD (often referred to as Hedges’ g or Cohen’s d [ 25 , 26 ]). These are followed by proportion (%) and Fisher’s z -transformation of correlation, or Zr . These four effect measures nearly fit into the three categories, which are named: (1) single-group measures (a statistical summary from one group; e.g., proportion), (2) comparative measures (comparing between two groups e.g., SMD and lnRR), and (3) association measures (relationships between two variables; e.g., Zr ). Table 2 summarizes effect measures that are common or potentially useful for environmental scientists. It is important to note that any measures with sampling variance can become an ‘effect size’. The main reason why SMD, lnRR, Zr, or proportion are popular effect measures is that they are unitless, while a meta-analysis of mean, or mean difference, can only be conducted when all effect sizes have the same unit (e.g., cm, kg).

Table 2 also includes effect measures that are likely to be unfamiliar to environmental scientists; these are effect sizes that characterise differences in the observed variability between samples, (i.e., lnSD, lnCV, lnVR and lnCVR; [ 27 , 28 ]) rather than central tendencies (averages). These dispersion-based effect measures can provide us with extra insights along with average-based effect measures. Although the literature survey showed none of these were used in our sample, these effect sizes have been used in many fields, including agriculture (e.g., [ 29 ]), ecology (e.g., [ 30 ]), evolutionary biology (e.g., [ 31 ]), psychology (e.g., [ 32 ]), education (e.g., [ 33 ]), psychiatry (e.g., [ 34 ]), and neurosciences (e.g. [ 35 ],),. Perhaps, it is not difficult to think of an environmental intervention that can affect not only the mean but also the variance of measurements taken on a group of individuals or a set of plots. For example, environmental stressors such as pesticides and eutrophication are likely to increase variability in biological systems because stress accentuates individual differences in environmental responses (e.g. [ 36 , 37 ],). Such ideas are yet to be tested meta-analytically (cf. [ 38 , 39 ]).

Choosing a meta-analytic model

Fixed-effect and random-effects models.

Two traditional meta-analytic models are called the ‘fixed-effect’ model and the ‘random-effects’ model. The former assumes that all effect sizes (from different studies) come from one population (i.e., they have one true overall mean), while the latter does not have such an assumption (i.e., each study has different overall means or heterogeneity exists among studies; see below for more). The fixed-effect model, which should probably be more correctly referred to as the ‘common-effect’ model, can be written as [ 9 , 10 , 40 ]:

where the intercept, \({\beta }_{0}\) is the overall mean, z j (the response/dependent variable) is the effect size from the j th study ( j  = 1, 2,…, N study ; in this model, N study  = the number of studies = the number of effect sizes), m j is the sampling error, related to the j th sampling variance ( v j ), which is normally distributed with the mean of 0 and the ‘study-specific’ sampling variance, v j (see also Fig.  1 A).

Visualisation of the three statistical models of meta-analysis: A a fixed-effect model (1-level), B a random-effects model (2-level), and C a multilevel model (3-level; see the text for what symbols mean)

The overall mean needs to be estimated and often done so as the weighted average with the weights, \({w}_{j}=1/{v}_{j}\) (i.e., the inverse-variance approach). An important, but sometimes untenable, assumption of meta-analysis is that sampling variance is known. Indeed, we estimate sampling variance, using formulas, as in Table 2 , meaning that vj is submitted by sampling variance estimates (see also section ‘ Scale dependence ’). Of relevance, the use of the inverse-variance approach has been recently criticized, especially for SMD and lnRR [ 41 , 42 ] and we note that the inverse-variance approach using the formulas in Table 2 is one of several different weighting approaches used in meta-analysis (e.g., for adjusted sampling-variance weighing, see [ 43 , 44 ]; for sample-size-based weighting, see [ 41 , 42 , 45 , 46 ]). Importantly, the fixed-effect model assumes that the only source of variation in effect sizes ( z j ) is the effect due to sampling variance (which is inversely proportional to the sample size, n ; Table 2 ).

Similarly, the random-effects model can be expressed as:

where u j is the j th study effect, which is normally distributed with the mean of 0 and the between-study variance, \({\tau }^{2}\) (for different estimation methods, see [ 47 , 48 , 49 , 50 ]), and other notations are the same as in Eq.  1 (Fig.  1 B). Here, the overall mean can be estimated as the weighted average with weights \({w}_{j}=1/\left({\tau }^{2}+{v}_{j}^{2}\right)\) (note that different weighting approaches, mentioned above, are applicable to the random-effects model and some of them are to the multilevel model, introduced below). The model assumes each study has its specific mean, \({b}_{0}+{u}_{j}\) , and (in)consistencies among studies (effect sizes) are indicated by \({\tau }^{2}\) . When \({\tau }^{2}\) is 0 (or not statistically different from 0), the random-effects model simplifies to the fixed-effect model (cf. Equations  1 and 2 ). Given no studies in environmental sciences are conducted in the same manner or even at exactly the same place and time, we should expect different studies to have different means. Therefore, in almost all cases in the environmental sciences, the random-effects model is a more ‘realistic’ model [ 9 , 10 , 40 ]. Accordingly, most environmental meta-analyses (68.5%; 50 out of 73 studies) in our survey used the random-effects model, while only 2.7% (2 of 73 studies) used the fixed-effect model (Additional file 1 ).

Multilevel meta-analytic models

Although we have introduced the random-effects model as being more realistic than the fixed-effect model (Eq.  2 ), we argue that the random-effects model is rather limited and impractical for the environmental sciences. This is because random-effects models, like fixed-effect models, assume all effect sizes ( z j ) to be independent. However, when multiple effect sizes are obtained from a study, these effect sizes are dependent (for more details, see the next section on non-independence). Indeed, our survey showed that in almost all datasets used in environmental meta-analyses, this type of non-independence among effect sizes occurred (97.3%; 71 out of 73 studies, with two studies being unclear, so effectively 100%; Additional file 1 ). Therefore, we propose the simplest and most practical meta-analytic model for environmental sciences as [ 13 , 40 ] (see also [ 51 , 52 ]):

where we explicitly recognize that N effect ( i  = 1, 2,…, N effect ) >  N study ( j  = 1, 2,…, N study ) and, therefore, we now have the study effect (between-study effect), u j[i] (for the j th study and i th effect size) and effect-size level (within-study) effect, e i (for the i th effect size), with the between-study variance, \({\tau }^{2}\) , and with-study variance, \({\sigma }^{2}\) , respectively, and other notations are the same as above. We note that this model (Eq.  3 ) is an extension of the random-effects model (Eq.  2 ), and we refer to it as the multilevel/hierarchical model (used in 7 out of 73 studies: 9.6% [Additional file 1 ]; note that Eq.  3 is also known as a three-level meta-analytic model; Fig.  1 C). Also, environmental scientists who are familiar with (generalised) linear mixed-models may recognize u j (the study effect) as the effect of a random factor which is associated with a variance component, i.e., \({\tau }^{2}\) [ 53 ]; also, e i and m i can be seen as parts of random factors, associated with \({\sigma }^{2}\) and v i (the former is comparable to the residuals, while the latter is sampling variance, specific to a given effect size).

It seems that many researchers are aware of the issue of non-independence so that they often use average effect sizes per study or choose one effect size (at least 28.8%, 21 out of 73 environmental meta-analyses; Additional file 1 ). However, as we discussed elsewhere [ 13 , 40 ], such averaging or selection of one effect size per study dramatically reduces our ability to investigate environmental drivers of variation among effect sizes [ 13 ]. Therefore, we strongly support the use of the multilevel model. Nevertheless, this proposed multilevel model, formulated as Eq.  3 does not usually deal with the issue of non-independence completely, which we elaborate on in the next section.

Non-independence among effect sizes and among sampling errors

When you have multiple effect sizes from a study, there are two broad types and three cases of non-independence (cf. [ 11 , 12 ]): (1) effect sizes are calculated from different cohorts of individuals (or groups of plots) within a study (Fig.  2 A, referred to as ‘shared study identity’), and (2) effects sizes are calculated from the same cohort of individuals (or group of plots; Fig.  2 B, referred to as ‘shared measurements’) or partially from the same individuals and plots, more concretely, sharing individuals and plots from the control group (Fig.  2 C, referred to as ‘shared control group’). The first type of non-independence induces dependence among effect sizes, but not among sampling variances, and the second type leads to non-independence among sampling variances. Many datasets, if not almost all, will have a combination of these three cases (or even are more complex, see the section " Complex non-independence "). Failing to deal with these non-independences will inflate Type 1 error (note that the overall estimate, b 0 is unlikely to be biased, but standard error of b 0 , se( b 0 ), will be underestimated; note that this is also true for all other regression coefficients, e.g., b 1 ; see Table 1 ). The multilevel model (as in Eq.  3 ) only takes care of cases of non-independence that are due to the shared study identity but neither shared measurements nor shared control group.

Visualisation of the three types of non-independence among effect sizes: A due to shared study identities (effect sizes from the same study), B due to shared measurements (effect sizes come from the same group of individuals/plots but are based on different types of measurements), and C due to shared control (effect sizes are calculated using the same control group and multiple treatment groups; see the text for more details)

There are two practical ways to deal with non-independence among sampling variances. The first method is that we explicitly model such dependence using a variance–covariance (VCV) matrix (used in 6 out of 73 studies: 8.2%; Additional file 1 ). Imagine a simple scenario with a dataset of three effect sizes from two studies where two effects sizes from the first study are calculated (partially) using the same cohort of individuals (Fig.  2 B); in such a case, the sampling variance effect, \({m}_{i}\) , as in Eq.  3 , should be written as:

where M is the VCV matrix showing the sampling variances, \({v}_{1\left[1\right]}\) (study 1 and effect size 1), \({v}_{1\left[2\right]}\) (study 1 and effect size 2), and \({v}_{2\left[3\right]}\) (study 2 and effect size 3) in its diagonal, and sampling covariance, \(\rho \sqrt{{v}_{1\left[1\right]}{v}_{1\left[2\right]}}= \rho \sqrt{{v}_{1\left[2\right]}{v}_{1\left[1\right]}}\) in its off-diagonal elements, where \(\rho \) is a correlation between two sampling variances due to shared samples (individuals/plots). Once this VCV matrix is incorporated into the multilevel model (Eq.  3 ), all the types of non-independence, as in Fig.  2 , are taken care of. Table 3 shows formulas for the sampling variance and covariance of the four common effect sizes (SDM, lnRR, proportion and Zr ). For comparative effect measures (Table 2 ), exact covariances can be calculated under the case of ‘shared control group’ (see [ 54 , 55 ]). But this is not feasible for most circumstances because we usually do not know what \(\rho \) should be. Some have suggested fixing this value at 0.5 (e.g., [ 11 ]) or 0.8 (e.g., [ 56 ]); the latter is a more conservative assumption. Or one can run both and use one for the main analysis and the other for sensitivity analysis (for more, see the ‘ Conducting sensitivity analysis and critical appraisal " section).

The second method overcomes this very issue of unknown \(\rho \) by approximating average dependence among sampling variance (and effect sizes) from the data and incorporating such dependence to estimate standard errors (only used in 1 out of 73 studies; Additional file 1 ). This method is known as ‘robust variance estimation’, RVE, and the original estimator was proposed by Hedges and colleagues in 2010 [ 57 ]. Meta-analysis using RVE is relatively new, and this method has been applied to multilevel meta-analytic models only recently [ 58 ]. Note that the random-effects model (Eq.  2 ) and RVE could correctly model both types of non-independence. However, we do not recommend the use of RVE with Eq.  2 because, as we will later show, estimating \({\sigma }^{2}\) as well as \({\tau }^{2}\) will constitute an important part of understanding and gaining more insights from one’s data. We do not yet have a definite recommendation on which method to use to account for non-independence among sampling errors (using the VCV matrix or RVE). This is because no simulation work in the context of multilevel meta-analysis has been done so far, using multilevel meta-analyses [ 13 , 58 ]. For now, one could use both VCV matrices and RVE in the same model [ 58 ] (see also [ 21 ]).

Quantifying and explaining heterogeneity

Measuring consistencies with heterogeneity.

As mentioned earlier, quantifying heterogeneity among effect sizes is an essential component of any meta-analysis. Yet, our survey showed only 28 out of 73 environmental meta-analyses (38.4%; Additional file 1 ) report at least one index of heterogeneity (e.g., \({\tau }^{2}\) , Q , and I 2 ). Conventionally, the presence of heterogeneity is tested by Cochrane’s Q test. However, Q (often noted as Q T or Q total ), and its associated p value, are not particularly informative: the test does not tell us about the extent of heterogeneity (e.g. [ 10 ],), only whether heterogeneity is zero or not (when p  < 0.05). Therefore, for environmental scientists, we recommend two common ways of quantifying heterogeneity from a meta-analytic model: absolute heterogeneity measure (i.e., variance components, \({\tau }^{2}\) and \({\sigma }^{2}\) ) and relative heterogeneity measure (i.e., I 2 ; see also the " Notes on visualisation and interpretation " section for another way of quantifying and visualising heterogeneity at the same time, using prediction intervals; see also [ 59 ]). We have already covered the absolute measure (Eqs.  2 & 3 ), so here we explain I 2 , which ranges from 0 to 1 (for some caveats for I 2 , see [ 60 , 61 ]). The heterogeneity measure, I 2 , for the random-effect model (Eq.  2 ) can be written as:

Where \(\overline{v}\) is referred to as the typical sampling variance (originally this is called ‘within-study’ variance, as in Eq.  2 , and note that in this formulation, within-study effect and the effect of sampling error is confounded; see [ 62 , 63 ]; see also [ 64 ]) and the other notations are as above. As you can see from Eq.  5 , we can interpret I 2 as relative variation due to differences between studies (between-study variance) or relative variation not due to sampling variance.

By seeing I 2 as a type of interclass correlation (also known as repeatability [ 65 ],), we can generalize I 2 to multilevel models. In the case of Eq.  3 ([ 40 , 66 ]; see also [ 52 ]), we have:

Because we can have two more I 2 , Eq.  7 is written as \({I}_{total}^{2}\) ; these other two are \({I}_{study}^{2}\) and \({I}_{effect}^{2}\) , respectively:

\({I}_{total}^{2}\) represents relative variance due to differences both between and within studies (between- and within-study variance) or relative variation not due to sampling variance, while \({I}_{study}^{2}\) is relative variation due to differences between studies, and \({I}_{effect}^{2}\) is relative variation due to differences within studies (Fig.  3 A). Once heterogeneity is quantified (note almost all data will have non-zero heterogeneity and an earlier meta-meta-analysis suggests in ecology, we have on average, I 2 close to 90% [ 66 ]), it is time to fit a meta-regression model to explain the heterogeneity. Notably, the magnitude of \({I}_{study}^{2}\) (and \({\tau }^{2}\) ) and \({I}_{effect}^{2}\) (and \({\sigma }^{2}\) ) can already inform you which predictor variable (usually referred to as ‘moderator’) is likely to be important, which we explain in the next section.

Visualisation of variation (heterogeneity) partitioned into different variance components: A quantifying different types of I 2 from a multilevel model (3-level; see Fig.  1 C) and B variance explained, R 2 , by moderators. Note that different levels of variances would be explained, depending on which level a moderator belongs to (study level and effect-size level)

Explaining variance with meta-regression

We can extend the multilevel model (Eq.  3 ) to a meta-regression model with one moderator (also known as predictor, independent, explanatory variable, or fixed factor), as below:

where \({\beta }_{1}\) is a slope of the moderator ( x 1 ), \({x}_{1j\left[i\right]}\) denotes the value of x 1 , corresponding to the j th study (and the i th effect sizes). Equation ( 10 ) (meta-regression) is comparable to the simplest regression with the intercept ( \({\beta }_{0}\) ) and slope ( \({\beta }_{1}\) ). Notably, \({x}_{1j\left[i\right]}\) differs between studies and, therefore, it will mainly explain the variance component, \({\tau }^{2}\) (which relates to \({I}_{study}^{2}\) ). On the other hand, if noted like \({x}_{1i}\) , this moderator would vary within studies or at the level of effect sizes, therefore, explaining \({\sigma }^{2}\) (relating to \({I}_{effect}^{2}\) ). Therefore, when \({\tau }^{2}\) ( \({I}_{study}^{2}\) ), or \({\sigma }^{2}\) ( \({I}_{effect}^{2}\) ), is close to zero, there will be little point fitting a moderator(s) at the level of studies, or effect sizes, respectively.

As in multiple regression, we can have multiple (multi-moderator) meta-regression, which can be written as:

where \(\sum_{h=1}^{q}{\beta }_{h}{x}_{h\left[i\right]}\) denotes the sum of all the moderator effects, with q being the number of slopes (staring with h  = 1). We note that q is not necessarily the number of moderators. This is because when we have a categorical moderator, which is common, with more than two levels (e.g., method A, B & C), the fixed effect part of the formula is \({\beta }_{0}+{\beta }_{1}{x}_{1}+{\beta }_{2}{x}_{2}\) , where x 1 and x 2 are ‘dummy’ variables, which code whether the i th effect size belongs to, for example, method B or C, with \({\beta }_{1}\) and \({\beta }_{2}\) being contrasts between A and B and between A and C, respectively (for more explanations of dummy variables, see our tutorial page [ https://itchyshin.github.io/Meta-analysis_tutorial/ ]; also see [ 67 , 68 ]). Traditionally, researchers conduct separate meta-analyses per different groups (known as ‘sub-group analysis’), but we prefer a meta-regression approach with a categorical variable, which is statistically more powerful [ 40 ]. Also, importantly, what can be used as a moderator(s) is very flexible, including, for example, individual/plot characteristics (e.g., age, location), environmental factors (e.g., temperature), methodological differences between studies (e.g., randomization), and bibliometric information (e.g., publication year; see more in the section ‘Checking for publication bias and robustness’). Note that moderators should be decided and listed a priori in the meta-analysis plan (i.e., a review protocol or pre-registration).

As with meta-analysis, the Q -test ( Q m or Q moderator ) is often used to test the significance of the moderator(s). To complement this test, we can also quantify variance explained by the moderator(s) using R 2 . We can define R 2 using Eq. ( 11 ) as:

where R 2 is known as marginal R 2 (sensu [ 69 , 70 ]; cf. [ 71 ]), \({f}^{2}\) is the variance due to the moderator(s), and \({(f}^{2}+{\tau }^{2}+{\sigma }^{2})\) here equals to \(({\tau }^{2}+{\sigma }^{2})\) in Eq.  7 , as \({f}^{2}\) ‘absorbs’ variance from \({\tau }^{2}\) and/or \({\sigma }^{2}\) . We can compare the similarities and differences in Fig.  3 B where we denote a part of \({f}^{2}\) originating from \({\tau }^{2}\) as \({f}_{study}^{2}\) while \({\sigma }^{2}\) as \({f}_{effect}^{2}\) . In a multiple meta-regression model, we often want to find a model with the ‘best’ or an adequate set of predictors (i.e., moderators). R 2 can potentially help such a model selection process. Yet, methods based on information criteria (such as Akaike information criterion, AIC) may be preferable. Although model selection based on the information criteria is beyond the scope of the paper, we refer the reader to relevant articles (e.g., [ 72 , 73 ]), and we show an example of this procedure in our online tutorial ( https://itchyshin.github.io/Meta-analysis_tutorial/ ).

Notes on visualisation and interpretation

Visualization and interpretation of results is an essential part of a meta-analysis [ 74 , 75 ]. Traditionally, a forest plot is used to display the values and 95% of confidence intervals (CIs) for each effect size and the overall effect and its 95% CI (the diamond symbol is often used, as shown in Fig.  4 A). More recently, adding a 95% prediction interval (PI) to the overall estimate has been strongly recommended because 95% PIs show a predicted range of values in which an effect size from a new study would fall, assuming there is no sampling error [ 76 ]. Here, we think that examining the formulas for 95% CIs and PIs for the overall mean (from Eq.  3 ) is illuminating:

where \({t}_{df\left[\alpha =0.05\right]}\) denotes the t value with the degree of freedom, df , at 97.5 percentile (or \(\alpha =0.05\) ) and other notations are as above. In a meta-analysis, it has been conventional to use z value 1.96 instead of \({t}_{df\left[\alpha =0.05\right]}\) , but simulation studies have shown the use of t value over z value reduces Type 1 errors under many scenarios and, therefore, is recommended (e.g., [ 13 , 77 ]). Also, it is interesting to note that by plotting 95% PIs, we can visualize heterogeneity as Eq.  15 includes \({\tau }^{2}\) and \({\sigma }^{2}\) .

Different types of plots useful for a meta-analysis using data from Midolo et al. [ 133 ]: A a typical forest plot with the overall mean shown as a diamond at the bottom (20 effect sizes from 20 studies are used), B a caterpillar plot (100 effect sizes from 24 studies are used), C an orchard plot of categorical moderator with seven levels (all effect sizes are used), and D a bubble plot of a continuous moderator. Note that the first two only show confidence intervals, while the latter two also show prediction intervals (see the text for more details)

A ‘forest’ plot can become quickly illegible as the number of studies (effect sizes) becomes large, so other methods of visualizing the distribution of effect sizes have been suggested. Some suggested to present a ‘caterpillar’ plot, which is a version of the forest plot, instead (Fig.  4 B; e.g., [ 78 ]). We here recommend an ‘orchard’ plot, as it can present results across different groups (or a result of meta-regression with a categorical variable), as shown in Fig.  4 C [ 78 ]. For visualization of a continuous variable, we suggest what is called a ‘bubble’ plot, shown in Fig.  4 D. Visualization not only helps us interpret meta-analytic results, but can also help to identify something we may not see from statistical results, such as influential data points and outliers that could threaten the robustness of our results.

Checking for publication bias and robustness

Detecting and correcting for publication bias.

Checking for and adjusting for any publication bias is necessary to ensure the validity of meta-analytic inferences [ 79 ]. However, our survey showed almost half of the environmental meta-analyses (46.6%; 34 out of 73 studies; Additional file 1 ) neither tested for nor corrected for publication bias (cf. [ 14 , 15 , 16 ]). The most popular methods used were: (1) graphical tests using funnel plots (26 studies; 35.6%), (2) regression-based tests such as Egger regression (18 studies; 24.7%), (3) Fail-safe number tests (12 studies; 16.4%), and (4) trim-and-fill tests (10 studies; 13.7%). We recently showed that these methods are unsuitable for datasets with non-independent effect sizes, with the exception of funnel plots [ 80 ] (for an example of funnel plots, see Fig.  5 A). This is because these methods cannot deal with non-independence in the same way as the fixed-effect and random-effects models. Here, we only introduce a two-step method for multilevel models that can both detect and correct for publication bias [ 80 ] (originally proposed by [ 81 , 82 ]), more specifically, the “small study effect” where an effect size value from a small-sample-sized study can be much larger in magnitude than a ‘true’ effect [ 83 , 84 ]. This method is a simple extension of Egger’s regression [ 85 ], which can be easily implemented by using Eq.  10 :

where \({\widetilde{n}}_{i}\) is known as effective sample size; for Zr and proportion it is just n i , and for SMD and lnRR, it is \({n}_{iC}{n}_{iT}/\left({n}_{iC}+{n}_{iT}\right)\) , as in Table 2 . When \({\beta }_{1}\) is significant, we conclude there exists a small-study effect (in terms of a funnel plot, this is equivalent to significant funnel asymmetry). Then, we fit Eq.  17 and we look at the intercept \({\beta }_{0}\) , which will be a bias-corrected overall estimate [note that \({\beta }_{0}\) in Eq. ( 16 ) provides less accurate estimates when non-zero overall effects exist [ 81 , 82 ]; Fig.  5 B]. An intuitive explanation of why \({\beta }_{0}\) (Eq.  17 ) is the ‘bias-corrected’ estimate is that the intercept represents \(1/\widetilde{{n}_{i}}=0\) (or \(\widetilde{{n}_{i}}=\infty \) ); in other words, \({\beta }_{0}\) is the estimate of the overall effect when we have a very large (infinite) sample size. Of note, appropriate bias correction requires a selection-mode-based approach although such an approach is yet to be available for multilevel meta-analytic models [ 80 ].

Different types of plots for publication bias tests: A a funnel plot using model residuals, showing a funnel (white) that shows the region of statistical non-significance (30 effect sizes from 30 studies are used; note that we used the inverse of standard errors for the y -axis, but for some effect sizes, sample size or ‘effective’ sample size may be more appropriate), B a bubble plot visualising a multilevel meta-regression that tests for the small study effect (note that the slope was non-significant: b  = 0.120, 95% CI = [− 0.095, 0.334]; all effect sizes are used), and C a bubble plot visualising a multilevel meta-regression that tests for the decline effect (the slope was non-significant: b  = 0.003, 95%CI = [− 0.002, 0.008])

Conveniently, this proposed framework can be extended to test for another type of publication bias, known as time-lag bias, or the decline effect, where effect sizes tend to get closer to zero over time, as larger or statistically significant effects are published more quickly than smaller or non-statistically significant effects [ 86 , 87 ]. Again, a decline effect can be statistically tested by adding year to Eq. ( 3 ):

where \(c\left(yea{r}_{j\left[i\right]}\right)\) is the mean-centred publication year of a particular study (study j and effect size i ); this centring makes the intercept \({\beta }_{0}\) meaningful, representing the overall effect estimate at the mean value of publication years (see [ 68 ]). When the slope is significantly different from 0, we deem that we have a decline effect (or time-lag bias; Fig.  5 C).

However, there may be some confounding moderators, which need to be modelled together. Indeed, Egger’s regression (Eqs.  16 and 17 ) is known to detect the funnel asymmetry when there is little heterogeneity; this means that we need to model \(\sqrt{1/{\widetilde{n}}_{i}}\) with other moderators that account for heterogeneity. Given this, we probably should use a multiple meta-regression model, as below:

where \(\sum_{h=3}^{q}{\beta }_{h}{x}_{h\left[i\right]}\) is the sum of the other moderator effects apart from the small-study effect and decline effect, and other notations are as above (for more details see [ 80 ]). We need to carefully consider which moderators should go into Eq.  19 (e.g., fitting all moderators or using an AIC-based model selection method; see [ 72 , 73 ]). Of relevance, when running complex models, some model parameters cannot be estimated well, or they are not ‘identifiable’ [ 88 ]. This is especially so for variance components (random-effect part) rather than regression coeffects (fixed-effect part). Therefore, it is advisable to check whether model parameters are all identifiable, which can be checked using the profile function in metafor (for an example, see our tutorial webpage [ https://itchyshin.github.io/Meta-analysis_tutorial/ ]).

Conducting sensitivity analysis and critical appraisal

Sensitivity analysis explores the robustness of meta-analytic results by running a different set of analyses from the original analysis, and comparing the results (note that some consider publication bias tests a part of sensitivity analysis; [ 11 ]). For example, we might be interested in assessing how robust results are to the presence of influential studies, to the choice of method for addressing non-independence, or weighting effect sizes. Unfortunately, in our survey, only 37% of environmental meta-analyses (27 out of 73) conducted sensitivity analysis (Additional file 1 ). There are two general and interrelated ways to conduct sensitivity analyses [ 73 , 89 , 90 ]. The first one is to take out influential studies (e.g., outliers) and re-run meta-analytic and meta-regression models. We can also systematically take each effect size out and run a series of meta-analytic models to see whether any resulting overall effect estimates are different from others; this method is known as ‘leave-one-out’, which is considered less subjective and thus recommended.

The second way of approaching sensitivity analysis is known as subset analysis, where a certain group of effect sizes (studies) will be excluded to re-run the models without this group of effect sizes. For example, one may want to run an analysis without studies that did not randomize samples. Yet, as mentioned earlier, we recommend using meta-regression (Eq.  13 ) with a categorical variable of randomization status (‘randomized’ or ‘not randomized’), to statistically test for an influence of moderators. It is important to note that such tests for risk of bias (or study quality) can be considered as a way of quantitatively evaluating the importance of study features that were noted at the stage of critical appraisal, which is an essential part of any systematic review (see [ 11 , 91 ]). In other words, we can use meta-regression or subset analysis to quantitatively conduct critical appraisal using (study-level) moderators that code, for example, blinding, randomization, and selective reporting. Despite the importance of critical appraisal ([ 91 ]), only 4 of 73 environmental meta-analyses (5.6%) in our survey assessed the risk of bias in each study included in a meta-analysis (i.e., evaluating a primary study in terms of the internal validity of study design and reporting; Additional file 1 ). We emphasize that critically appraising each paper or checking them for risk of bias is an extremely important topic. Also, critical appraisal is not restricted to quantitative synthesis. Therefore, we do not cover any further in this paper for more, see [ 92 , 93 ]).

Notes on transparent reporting and open archiving

For environmental systematic reviews and maps, there are reporting guidelines called RepOrting standards for Systematic Evidence Syntheses in environmental research, ROSES [ 94 ] and synthesis assessment checklist, the Collaboration for Environmental Evidence Synthesis Appraisal Tool (CEESAT; [ 95 ]). However, these guidelines are somewhat limited in terms of reporting quantitative synthesis because they cover only a few core items. These two guidelines are complemented by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Ecology and Evolutionary Biology (PRISMA-EcoEvo; [ 96 ]; cf. [ 97 , 98 ]), which provides an extended set of reporting items covering what we have described above. Items 20–24 from PRISMA-EcoEvo are most relevant: these items outline what should be reported in the Methods section: (i) sample sizes and study characteristics, (ii) meta-analysis, (iii) heterogeneity, (iv) meta-regression and (v) outcomes of publication bias and sensitivity analysis (see Table 4 ). Our survey, as well as earlier surveys, suggest there is a large room for improvement in the current practice ([ 14 , 15 , 16 ]). Incidentally, the orchard plot is well aligned with Item 20, as this plot type shows both the number of effect sizes and studies for different groups (Fig.  4 C). Further, our survey of environmental meta-analyses highlighted the poor standards of data openness (with 24 studies sharing data: 32.9%) and code sharing (7 studies: 29.2%; Additional file 1 ). Environmental scientists must archive their data as well as their analysis code in accordance with the FAIR principles (Findable, Accessible, Interoperable, and Reusable [ 99 ]) using dedicated depositories such as Dryad, FigShare, Open Science Framework (OSF), Zenodo or others (cf. [ 100 , 101 ]), preferably not on publisher’s webpages (as paywall may block access). However, archiving itself is not enough; data requires metadata (detailed descriptions) and the code needs to also be FAIR [ 102 , 103 ].

Other relevant and advanced issues

Scale dependence.

The issue of scale dependence is a unique yet widespread problem in environmental sciences (see [ 7 , 104 ]); our literature survey indicated three quarters of the environmental meta-analyses (56 out of 73 studies) have inferences that are potentially vulnerable to scale-dependence [ 105 ]. For example, studies that set out to compare group means in biodiversity measures, such as species richness, can vary as a function of the scale (size) of the sampling unit. When the unit of replication is a plot (not an individual animal or plant), the aerial size of a plot (e.g., 100 cm 2 or 1 km 2 ) will affect both the precision and accuracy of effect size estimates (e.g., lnRR and SMD). In general, a study with larger plots might have more accurately estimated species richness differences, but less precisely than a study with smaller plots and greater replication. Lower replication means that our sampling variance estimates are likely to be misestimated, and the study with larger plots will generally have less weight than the study with smaller plots, due to higher sampling variance. Inaccurate variance estimates in little-replicated ecological studies are known to cause an accumulating bias in precision-weighted meta-analysis, requiring correction [ 43 ]. To assess the potential for scale-dependence, it is recommended that analysts test for possible covariation among plot size, replication, variances, and effect sizes [ 104 ]. If detected, analysts should use an effect size measure that is less sensitive to scale dependence (lnRR), and could use the size of a plot as a moderator in meta-regression, or alternatively, they consider running an unweighted model ([ 7 ]; note that only 12%, 9 out of 73 studies, accounted for sampling area in some way; Additional file 1 ).

• Missing data

In many fields, meta-analytic data almost always encompass missing values see [ 106 , 107 , 108 ]. Broadly, we have two types of missing data in meta-analyses [ 109 , 110 ]: (1) missing data in standard deviations or sample sizes, associated with means, preventing effect size calculations (Table 2 ), and (2) missing data in moderators. There are several solutions for both types. The best, and first to try, should be contacting the authors. If this fails, we can potentially ‘impute’ missing data. Single imputation methods using the strong correlation between standard deviation and mean values (known as mean–variance relationship) are available, although single imputation can lead to Type I error [ 106 , 107 ] (see also [ 43 ]) because we do not model the uncertainty of imputation itself. Contrastingly, multiple imputation, which creates multiple versions of imputed datasets, incorporates such uncertainty. Indeed, multiple imputation is a preferred and proven solution for missing data in effect sizes and moderators [ 109 , 110 ]. Yet, correct implementation can be challenging (see [ 110 ]). What we require now is an automated pipeline of merging meta-analysis and multiple imputation, which accounts for imputation uncertainty, although it may be challenging for complex meta-analytic models. Fortunately, however, for lnRR, there is a series of new methods that can perform better than the conventional method and which can deal with missing SDs [ 44 ]; note that these methods do not deal with missing moderators. Therefore, where applicable, we recommend these new methods, until an easy-to-implement multiple imputation workflow arrives.

Complex non-independence

Above, we have only dealt with the model that includes study identities as a clustering/grouping (random) factor. However, many datasets are more complex, with potentially more clustering variables in addition to the study identity. It is certainly possible that an environmental meta-analysis contains data from multiple species. Such a situation creates an interesting dependence among effect sizes from different species, known as phylogenetic relatedness, where closely related species are more likely to be similar in effect sizes compared to distantly related ones (e.g., mice vs. rats and mice vs. sparrows). Our multilevel model framework is flexible and can accommodate phylogenetic relatedness. A phylogenetic multilevel meta-analytic model can be written as [ 40 , 111 , 112 ]:

where \({a}_{k\left[i\right]}\) is the phylogenetic (species) effect for the k th species (effect size i ; N effect ( i  = 1, 2,…, N effect ) >  N study ( j  = 1, 2,…, N study ) >  N species ( k  = 1, 2,…, N species )), normally distributed with \({\omega }^{2}{\text{A}}\) where is the phylogenetic variance and A is a correlation matrix coding how close each species are to each other and \({\omega }^{2}\) is the phylogenetic variance, \({s}_{k\left[i\right]}\) is the non-phylogenetic (species) effect for the k th species (effect size i ), normally distributed with the variance of \({\gamma }^{2}\) (the non-phylogenetic variance), and other notations are as above. It is important to realize that A explicitly models relatedness among species, and we do need to provide this correlation matrix, using a distance relationship usually derived from a molecular-based phylogenetic tree (for more details, see [ 40 , 111 , 112 ]). Some may think that the non-phylogenetic term ( \({s}_{k\left[i\right]}\) ) is unnecessary or redundant because \({s}_{k\left[i\right]}\) and the phylogenetic term ( \({a}_{k\left[i\right]}\) ) are both modelling variance at the species level. However, a simulation recently demonstrated that failing to have the non-phylogenetic term ( \({s}_{k\left[i\right]}\) ) will often inflate the phylogenetic variance \({\omega }^{2}\) , leading to an incorrect conclusion that there is a strong phylogenetic signal (as shown in [ 112 ]). The non-phylogenetic variance ( \({\gamma }^{2}\) ) arises from, for example, ecological similarities among species (herbivores vs. carnivores or arboreal vs. ground-living) not phylogeny [ 40 ].

Like phylogenetic relatedness, effect sizes arising from closer geographical locations are likely to be more correlated [ 113 ]. Statistically, spatial correlation can be also modelled in a manner analogous to phylogenetic relatedness (i.e., rather than a phylogenetic correlation matrix, A , we fit a spatial correlation matrix). For example, Maire and colleagues [ 114 ] used a meta-analytic model with spatial autocorrelation to investigate the temporal trends of fish communities in the network of rivers in France. We note that a similar argument can be made for temporal correlation, but in many cases, temporal correlations could be dealt with, albeit less accurately, as a special case of ‘shared measurements’, as in Fig.  2 . An important idea to take away is that one can model different, if not all, types of non-independence as the random factor(s) in a multilevel model.

Advanced techniques

Here we touch upon five advanced meta-analytic techniques with potential utility for environmental sciences, providing relevant references so that interested readers can obtain more information on these advanced topics. The first one is the meta-analysis of magnitudes, or absolute values (effect sizes), where researchers may be interested in deviations from 0, rather than the directionality of the effect [ 115 ]. For example, Cohen and colleagues [ 116 ] investigated absolute values of phenological responses, as they were concerned with the magnitudes of changes in phenology rather than directionality.

The second method is the meta-analysis of interaction where our focus is on synthesizing the interaction effect of, usually, 2 × 2 factorial design (e.g., the effect of two simultaneous environmental stressors [ 54 , 117 , 118 ]; see also [ 119 ]). Recently, Siviter and colleagues [ 120 ] showed that agrochemicals interact synergistically (i.e., non-additively) to increase the mortality of bees; that is, two agrochemicals together caused more mortality than the sum of mortalities of each chemical.

Third, network meta-analysis has been heavily used in medical sciences; network meta-analysis usually compares different treatments in relation to placebo and ranks these treatments in terms of effectiveness [ 121 ]. The very first ‘environmental’ network meta-analysis, as far as we know, investigated the effectives of ecosystem services among different land types [ 122 ].

Fourth, a multivariate meta-analysis is where one can model two or more different types of effect sizes with the estimation of pair-wise correlations between different effect sizes. The benefit of such an approach is known as the ‘borrowing of strength’, where the error of fixed effects (moderators; e.g., b 0 and b 1 ) can be reduced when different types of effect sizes are correlated (i.e., se ( b 0 ) and se ( b 1 ) can be smaller [ 123 ]) For example, it is possible for lnRR (differences in mean) and lnVR (differences in SDs) to be modelled together (cf. [ 124 ]).

Fifth, as with network meta-analysis, there has been a surge in the use of ‘individual participants data’, called ‘IPD meta-analysis’, in medical sciences [ 125 , 126 ]. The idea of IPD meta-analysis is simple—rather than using summary statistics reported in papers (sample means and variances), we directly use raw data from all studies. We can either model raw data using one complex multilevel (hierarchical) model (one-step method) or calculate statistics for each study and use a meta-analysis (two-step method; note that both methods will usually give the same results). Study-level random effects can be incorporated to allow the response variable of interest to vary among studies, and overall effects correspond to fixed, population-level estimates. The use of IPD or ‘full-data analyses’ has also surged in ecology, aided by open-science policies that encourage the archival of raw data alongside articles, and initiatives that synthesise raw data (e.g., PREDICTS [ 127 ], BioTime [ 128 ]). In health disciplines, such meta-analyses are considered the ‘gold standard’ [ 129 ], owing to their potential for resolving issues regarding study-specific designs and confounding variation, and it is unclear whether and how they might resolve issues such as scale dependence in environmental meta-analyses [ 104 , 130 ].

Conclusions

In this article, we have attempted to describe the most practical ways to conduct quantitative synthesis, including meta-analysis, meta-regression, and publication bias tests. In addition, we have shown that there is much to be improved in terms of meta-analytic practice and reporting via a survey of 73 recent environmental meta-analyses. Such improvements are urgently required, especially given the potential influence that environmental meta-analyses can have on policies and decision-making [ 8 ]. So often, meta-analysts have called for better reporting of primary research (e.g. [ 131 , 132 ]), and now this is the time to raise the standards of reporting in meta-analyses. We hope our contribution will help to catalyse a turning point for better practice in quantitative synthesis in environmental sciences. We remind the reader most of what is described is implemented in the R environment on our tutorial webpage and researchers can readily use the proposed models and techniques ( https://itchyshin.github.io/Meta-analysis_tutorial/ ). Finally, meta-analytic techniques are always developing and improving. It is certainly possible that in the future, our proposed models and related methods will become dated, just as the traditional fixed-effect and random-effects models already are. Therefore, we must endeavour to be open-minded to new ways of doing quantitative research synthesis in environmental sciences.

Availability of data and materials

All data and material are provided as additional files.

Higgins JP, Thomas JE, Chandler JE, Cumpston ME, Li TE, Page MJ, Welch VA. Cochrane handbook for systematic reviews of interventions. 2nd ed. Chichester: Wikey; 2019.

Book   Google Scholar  

Cooper HM, Hedges LV, Valentine JC. The handbook of research synthesis and meta-analysis . 3rd ed. New York: Russell Sage Foundation; 2019.

Google Scholar  

Schmid CH, Stijnen TE, White IE. Handbook of meta-analysis. 1st ed. Boca Ranton: CRC; 2021.

Vetter D, Rucker G, Storch I. Meta-analysis: a need for well-defined usage in ecology and conservation biology. Ecosphere. 2013;4(6):1.

Article   Google Scholar  

Koricheva J, Gurevitch J, Mengersen K, editors. Handbook of meta-analysis in ecology and evolution. Princeton: Princeton Univesity Press; 2017.

Gurevitch J, Koricheva J, Nakagawa S, Stewart G. Meta-analysis and the science of research synthesis. Nature. 2018;555(7695):175–82.

Article   CAS   Google Scholar  

Spake R, Doncaster CP. Use of meta-analysis in forest biodiversity research: key challenges and considerations. Forest Ecol Manag. 2017;400:429–37.

Bilotta GS, Milner AM, Boyd I. On the use of systematic reviews to inform environmental policies. Environ Sci Policy. 2014;42:67–77.

Hedges LV, Vevea JL. Fixed- and random-effects models in meta-analysis. Psychol Methods. 1998;3(4):486–504.

Borenstein M, Hedges LV, Higgins JPT, Rothstein H. Introduction to meta-analysis. 2nd ed. Chichester: Wiley; 2021.

Noble DWA, Lagisz M, Odea RE, Nakagawa S. Nonindependence and sensitivity analyses in ecological and evolutionary meta-analyses. Mol Ecol. 2017;26(9):2410–25.

Nakagawa S, Noble DWA, Senior AM, Lagisz M. Meta-evaluation of meta-analysis: ten appraisal questions for biologists. Bmc Biol. 2017;15:1.

Nakagawa S, Senior AM, Viechtbauer W, Noble DWA. An assessment of statistical methods for nonindependent data in ecological meta-analyses: comment. Ecology. 2022;103(1): e03490.

Romanelli JP, Meli P, Naves RP, Alves MC, Rodrigues RR. Reliability of evidence-review methods in restoration ecology. Conserv Biol. 2021;35(1):142–54.

Koricheva J, Gurevitch J. Uses and misuses of meta-analysis in plant ecology. J Ecol. 2014;102(4):828–44.

O’Leary BC, Kvist K, Bayliss HR, Derroire G, Healey JR, Hughes K, Kleinschroth F, Sciberras M, Woodcock P, Pullin AS. The reliability of evidence review methodology in environmental science and conservation. Environ Sci Policy. 2016;64:75–82.

Rosenthal R. The “file drawer problem” and tolerance for null results. Psychol Bull. 1979;86(3):638–41.

Nakagawa S, Lagisz M, Jennions MD, Koricheva J, Noble DWA, Parker TH, Sánchez-Tójar A, Yang Y, O’Dea RE. Methods for testing publication bias in ecological and evolutionary meta-analyses. Methods Ecol Evol. 2022;13(1):4–21.

Cheung MWL. A guide to conducting a meta-analysis with non-independent effect sizes. Neuropsychol Rev. 2019;29(4):387–96.

Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010;36(3):1–48.

Yang Y, Macleod M, Pan J, Lagisz M, Nakagawa S. Advanced methods and implementations for the meta-analyses of animal models: current practices and future recommendations. Neurosci Biobehav Rev. 2022. https://doi.org/10.1016/j.neubiorev.2022.105016:105016 .

Nakagawa S, Cuthill IC. Effect size, confidence interval and statistical significance: a practical guide for biologists. Biol Rev. 2007;82(4):591–605.

Hedges LV, Gurevitch J, Curtis PS. The meta-analysis of response ratios in experimental ecology. Ecology. 1999;80(4):1150–6.

Friedrich JO, Adhikari NKJ, Beyene J. The ratio of means method as an alternative to mean differences for analyzing continuous outcome variables in meta-analysis: A simulation study. BMC Med Res Methodol. 2008;8:5.

Hedges L, Olkin I. Statistical methods for meta-analysis. New York: Academic Press; 1985.

Cohen J. Statistical power analysis for the beahvioral sciences. 2nd ed. Hillsdale: Lawrence Erlbaum; 1988.

Senior AM, Viechtbauer W, Nakagawa S. Revisiting and expanding the meta-analysis of variation: the log coefficient of variation ratio. Res Synth Methods. 2020;11(4):553–67.

Nakagawa S, Poulin R, Mengersen K, Reinhold K, Engqvist L, Lagisz M, Senior AM. Meta-analysis of variation: ecological and evolutionary applications and beyond. Methods Ecol Evol. 2015;6(2):143–52.

Knapp S, van der Heijden MGA. A global meta-analysis of yield stability in organic and conservation agriculture. Nat Commun. 2018;9:3632.

Porturas LD, Anneberg TJ, Cure AE, Wang SP, Althoff DM, Segraves KA. A meta-analysis of whole genome duplication and theeffects on flowering traits in plants. Am J Bot. 2019;106(3):469–76.

Janicke T, Morrow EH. Operational sex ratio predicts the opportunity and direction of sexual selection across animals. Ecol Lett. 2018;21(3):384–91.

Chamberlain R, Brunswick N, Siev J, McManus IC. Meta-analytic findings reveal lower means but higher variances in visuospatial ability in dyslexia. Brit J Psychol. 2018;109(4):897–916.

O’Dea RE, Lagisz M, Jennions MD, Nakagawa S. Gender differences in individual variation in academic grades fail to fit expected patterns for STEM. Nat Commun. 2018;9:3777.

Brugger SP, Angelescu I, Abi-Dargham A, Mizrahi R, Shahrezaei V, Howes OD. Heterogeneity of striatal dopamine function in schizophrenia: meta-analysis of variance. Biol Psychiat. 2020;87(3):215–24.

Usui T, Macleod MR, McCann SK, Senior AM, Nakagawa S. Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research. Plos Biol. 2021;19(5): e3001009.

Hoffmann AA, Merila J. Heritable variation and evolution under favourable and unfavourable conditions. Trends Ecol Evol. 1999;14(3):96–101.

Wood CW, Brodie ED 3rd. Environmental effects on the structure of the G-matrix. Evolution. 2015;69(11):2927–40.

Hillebrand H, Donohue I, Harpole WS, Hodapp D, Kucera M, Lewandowska AM, Merder J, Montoya JM, Freund JA. Thresholds for ecological responses to global change do not emerge from empirical data. Nat Ecol Evol. 2020;4(11):1502.

Yang YF, Hillebrand H, Lagisz M, Cleasby I, Nakagawa S. Low statistical power and overestimated anthropogenic impacts, exacerbated by publication bias, dominate field studies in global change biology. Global Change Biol. 2022;28(3):969–89.

Nakagawa S, Santos ESA. Methodological issues and advances in biological meta-analysis. Evol Ecol. 2012;26(5):1253–74.

Bakbergenuly I, Hoaglin DC, Kulinskaya E. Estimation in meta-analyses of response ratios. BMC Med Res Methodol. 2020;20(1):1.

Bakbergenuly I, Hoaglin DC, Kulinskaya E. Estimation in meta-analyses of mean difference and standardized mean difference. Stat Med. 2020;39(2):171–91.

Doncaster CP, Spake R. Correction for bias in meta-analysis of little-replicated studies. Methods Ecol Evol. 2018;9(3):634–44.

Nakagawa S, Noble DW, Lagisz M, Spake R, Viechtbauer W, Senior AM. A robust and readily implementable method for the meta-analysis of response ratios with and without missing standard deviations. Ecol Lett. 2023;26(2):232–44

Hamman EA, Pappalardo P, Bence JR, Peacor SD, Osenberg CW. Bias in meta-analyses using Hedges’ d. Ecosphere. 2018;9(9): e02419.

Bakbergenuly I, Hoaglin DC, Kulinskaya E. On the Q statistic with constant weights for standardized mean difference. Brit J Math Stat Psy. 2022;75(3):444–65.

DerSimonian R, Kacker R. Random-effects model for meta-analysis of clinical trials: an update. Contemp Clin Trials. 2007;28(2):105–14.

Veroniki AA, Jackson D, Viechtbauer W, Bender R, Bowden J, Knapp G, Kuss O, Higgins JPT, Langan D, Salanti G. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Res Synth Methods. 2016;7(1):55–79.

Langan D, Higgins JPT, Simmonds M. Comparative performance of heterogeneity variance estimators in meta-analysis: a review of simulation studies. Res Synth Methods. 2017;8(2):181–98.

Panityakul T, Bumrungsup C, Knapp G. On estimating residual heterogeneity in random-effects meta-regression: a comparative study. J Stat Theory Appl. 2013;12(3):253–65.

Bishop J, Nakagawa S. Quantifying crop pollinator dependence and its heterogeneity using multi-level meta-analysis. J Appl Ecol. 2021;58(5):1030–42.

Cheung MWL. Modeling dependent effect sizes with three-level meta-analyses: a structural equation modeling approach. Psychol Methods. 2014;19(2):211–29.

Bolker BM, Brooks ME, Clark CJ, Geange SW, Poulsen JR, Stevens MHH, White JSS. Generalized linear mixed models: a practical guide for ecology and evolution. Trends Ecol Evol. 2009;24(3):127–35.

Lajeunesse MJ. On the meta-analysis of response ratios for studies with correlated and multi-group designs. Ecology. 2011;92(11):2049–55.

Gleser LJ, Olkin I. Stochastically dependent effect sizes. In: Cooper H, Hedges LV, Valentine JC, editors. The handbook of research synthesis and meta-analysis. New York: Russell Sage Foundation; 2009.

Tipton E, Pustejovsky JE. Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression. J Educ Behav Stat. 2015;40(6):604–34.

Hedges LV, Tipton E, Johnson MC. Robust variance estimation in meta-regression with dependent effect size estimates (vol 1, pg 39, 2010). Res Synth Methods. 2010;1(2):164–5.

Pustejovsky JE, Tipton E. Meta-analysis with robust variance estimation: expanding the range of working models. Prev Sci. 2021. https://doi.org/10.1007/s11121-021-01246-3 .

Cairns M, Prendergast LA. On ratio measures of heterogeneity for meta-analyses. Res Synth Methods. 2022;13(1):28–47.

Borenstein M, Higgins JPT, Hedges LV, Rothstein HR. Basics of meta-analysis: I2 is not an absolute measure of heterogeneity. Res Synth Methods. 2017;8(1):5–18.

Hoaglin DC. Practical challenges of I-2 as a measure of heterogeneity. Res Synth Methods. 2017;8(3):254–254.

Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Stat Med. 2002;21(11):1539–58.

Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. Brit Med J. 2003;327(7414):557–60.

Xiong CJ, Miller JP, Morris JC. Measuring study-specific heterogeneity in meta-analysis: application to an antecedent biomarker study of Alzheimer’s disease. Stat Biopharm Res. 2010;2(3):300–9.

Nakagawa S, Schielzeth H. Repeatability for Gaussian and non-Gaussian data: a practical guide for biologists. Biol Rev. 2010;85(4):935–56.

Senior AM, Grueber CE, Kamiya T, Lagisz M, O’Dwyer K, Santos ESA, Nakagawa S. Heterogeneity in ecological and evolutionary meta-analyses: its magnitude and implications. Ecology. 2016;97(12):3293–9.

Gelman A, Hill J. Data analysis using regression and multilevel/hierarchical models. Cambridge: Cambridge University Press; 2007.

Schielzeth H. Simple means to improve the interpretability of regression coefficients. Methods Ecol Evol. 2010;1(2):103–13.

Nakagawa S, Schielzeth H. A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods Ecol Evol. 2013;4(2):133–42.

Nakagawa S, Johnson PCD, Schielzeth H. The coefficient of determination R-2 and intra-class correlation coefficient from generalized linear mixed-effects models revisited and expanded. J R Soc Interface. 2017;14(134):20170213.

Aloe AM, Becker BJ, Pigott TD. An alternative to R-2 for assessing linear models of effect size. Res Synth Methods. 2010;1(3–4):272–83.

Cinar O, Umbanhowar J, Hoeksema JD, Viechtbauer W. Using information-theoretic approaches for model selection in meta-analysis. Res Synth Methods. 2021. https://doi.org/10.1002/jrsm.1489 .

Viechtbauer W. Model checking in meta-analysis. In: Schmid CH, Stijnen T, White IR, editors. Handbook of meta-analysis. Boca Raton: CRC; 2021.

Anzures-Cabrera J, Higgins JPT. Graphical displays for meta-analysis: An overview with suggestions for practice. Res Synth Methods. 2010;1(1):66–80.

Kossmeier M, Tran US, Voracek M. Charting the landscape of graphical displays for meta-analysis and systematic reviews: a comprehensive review, taxonomy, and feature analysis. Bmc Med Res Methodol. 2020;20(1):1.

Intout J, Ioannidis JPA, Rovers MM, Goeman JJ. Plea for routinely presenting prediction intervals in meta-analysis. BMJ Open. 2016;6(7): e010247.

Moeyaert M, Ugille M, Beretvas SN, Ferron J, Bunuan R, Van den Noortgate W. Methods for dealing with multiple outcomes in meta-analysis a comparison between averaging effect sizes, robust variance estimation and multilevel meta-analysis. Int J Soc Res Methodol. 2017;20:559.

Nakagawa S, Lagisz M, O’Dea RE, Rutkowska J, Yang YF, Noble DWA, Senior AM. The orchard plot: cultivating a forest plot for use in ecology, evolution, and beyond. Res Synth Methods. 2021;12(1):4–12.

Rothstein H, Sutton AJ, Borenstein M. Publication bias in meta-analysis : prevention, assessment and adjustments. Hoboken: Wiley; 2005.

Nakagawa S, Lagisz M, Jennions MD, Koricheva J, Noble DWA, Parker TH, Sanchez-Tojar A, Yang YF, O’Dea RE. Methods for testing publication bias in ecological and evolutionary meta-analyses. Methods Ecol Evol. 2022;13(1):4–21.

Stanley TD, Doucouliagos H. Meta-regression analysis in economics and business. New York: Routledge; 2012.

Stanley TD, Doucouliagos H. Meta-regression approximations to reduce publication selection bias. Res Synth Methods. 2014;5(1):60–78.

Sterne JAC, Becker BJ, Egger M. The funnel plot. In: Rothstein H, Sutton AJ, Borenstein M, editors. Publication bias in meta-analysis: prevention, assessment and adjustments. Chichester: Wiley; 2005. p. 75–98.

Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones DR, Lau J, Carpenter J, Rucker G, Harbord RM, Schmid CH, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. Br Med J. 2011;343:4002.

Egger M, Smith GD, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. Brit Med J. 1997;315(7109):629–34.

Jennions MD, Moller AP. Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution. P Roy Soc B-Biol Sci. 2002;269(1486):43–8.

Koricheva J, Kulinskaya E. Temporal instability of evidence base: a threat to policy making? Trends Ecol Evol. 2019;34(10):895–902.

Raue A, Kreutz C, Maiwald T, Bachmann J, Schilling M, Klingmuller U, Timmer J. Structural and practical identifiability analysis of partially observed dynamical models by exploiting the profile likelihood. Bioinformatics. 2009;25(15):1923–9.

Matsushima Y, Noma H, Yamada T, Furukawa TA. Influence diagnostics and outlier detection for meta-analysis of diagnostic test accuracy. Res Synth Methods. 2020;11(2):237–47.

Viechtbauer W, Cheung MWL. Outlier and influence diagnostics for meta-analysis. Res Synth Methods. 2010;1(2):112–25.

Haddaway NR, Macura B. The role of reporting standards in producing robust literature reviews comment. Nat Clim Change. 2018;8(6):444–7.

Frampton G, Whaley P, Bennett M, Bilotta G, Dorne JLCM, Eales J, James K, Kohl C, Land M, Livoreil B, et al. Principles and framework for assessing the risk of bias for studies included in comparative quantitative environmental systematic reviews. Environ Evid. 2022;11(1):12.

Stanhope J, Weinstein P. Critical appraisal in ecology: what tools are available, and what is being used in systematic reviews? Res Synth Methods. 2022. https://doi.org/10.1002/jrsm.1609 .

Haddaway NR, Macura B, Whaley P, Pullin AS. ROSES RepOrting standards for systematic evidence syntheses: pro forma, flow-diagram and descriptive summary of the plan and conduct of environmental systematic reviews and systematic maps. Environ Evid. 2018;7(1):1.

Woodcock P, Pullin AS, Kaiser MJ. Evaluating and improving the reliability of evidence syntheses in conservation and environmental science: a methodology. Biol Conserv. 2014;176:54–62.

O’Dea RE, Lagisz M, Jennions MD, Koricheva J, Noble DWA, Parker TH, Gurevitch J, Page MJ, Stewart G, Moher D, et al. Preferred reporting items for systematic reviews and meta-analyses in ecology and evolutionary biology: a PRISMA extension. Biol Rev. 2021;96(5):1695–722.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Plos Med. 2009;6(7):e1000097.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Plos Med. 2021;18(3): e1003583.

Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, Blomberg N, Boiten JW, Santos LBD, Bourne PE, et al. Comment: the FAIR guiding principles for scientific data management and stewardship. Sci Data. 2016;3: 160018.

Culina A, Baglioni M, Crowther TW, Visser ME, Woutersen-Windhouwer S, Manghi P. Navigating the unfolding open data landscape in ecology and evolution. Nat Ecol Evol. 2018;2(3):420–6.

Roche DG, Lanfear R, Binning SA, Haff TM, Schwanz LE, Cain KE, Kokko H, Jennions MD, Kruuk LE. Troubleshooting public data archiving: suggestions to increase participation. Plos Biol. 2014;12(1): e1001779.

Roche DG, Kruuk LEB, Lanfear R, Binning SA. Public data archiving in ecology and evolution: how well are we doing? Plos Biol. 2015;13(11): e1002295.

Culina A, van den Berg I, Evans S, Sanchez-Tojar A. Low availability of code in ecology: a call for urgent action. Plos Biol. 2020;18(7): e3000763.

Spake R, Mori AS, Beckmann M, Martin PA, Christie AP, Duguid MC, Doncaster CP. Implications of scale dependence for cross-study syntheses of biodiversity differences. Ecol Lett. 2021;24(2):374–90.

Osenberg CW, Sarnelle O, Cooper SD. Effect size in ecological experiments: the application of biological models in meta-analysis. Am Nat. 1997;150(6):798–812.

Noble DWA, Nakagawa S. Planned missing data designs and methods: options for strengthening inference, increasing research efficiency and improving animal welfare in ecological and evolutionary research. Evol Appl. 2021;14(8):1958–68.

Nakagawa S, Freckleton RP. Missing inaction: the dangers of ignoring missing data. Trends Ecol Evol. 2008;23(11):592–6.

Mavridis D, Chaimani A, Efthimiou O, Leucht S, Salanti G. Addressing missing outcome data in meta-analysis. Evid-Based Ment Health. 2014;17(3):85.

Ellington EH, Bastille-Rousseau G, Austin C, Landolt KN, Pond BA, Rees EE, Robar N, Murray DL. Using multiple imputation to estimate missing data in meta-regression. Methods Ecol Evol. 2015;6(2):153–63.

Kambach S, Bruelheide H, Gerstner K, Gurevitch J, Beckmann M, Seppelt R. Consequences of multiple imputation of missing standard deviations and sample sizes in meta-analysis. Ecol Evol. 2020;10(20):11699–712.

Hadfield JD, Nakagawa S. General quantitative genetic methods for comparative biology: phylogenies, taxonomies and multi-trait models for continuous and categorical characters. J Evol Biol. 2010;23(3):494–508.

Cinar O, Nakagawa S, Viechtbauer W. Phylogenetic multilevel meta-analysis: a simulation study on the importance of modelling the phylogeny. Methods Ecol Evol. 2021. https://doi.org/10.1111/2041-210X.13760 .

Ives AR, Zhu J. Statistics for correlated data: phylogenies, space, and time. Ecol Appl. 2006;16(1):20–32.

Maire A, Thierry E, Viechtbauer W, Daufresne M. Poleward shift in large-river fish communities detected with a novel meta-analysis framework. Freshwater Biol. 2019;64(6):1143–56.

Morrissey MB. Meta-analysis of magnitudes, differences and variation in evolutionary parameters. J Evol Biol. 2016;29(10):1882–904.

Cohen JM, Lajeunesse MJ, Rohr JR. A global synthesis of animal phenological responses to climate change. Nat Clim Change. 2018;8(3):224.

Gurevitch J, Morrison JA, Hedges LV. The interaction between competition and predation: a meta-analysis of field experiments. Am Nat. 2000;155(4):435–53.

Macartney EL, Lagisz M, Nakagawa S. The relative benefits of environmental enrichment on learning and memory are greater when stressed: a meta-analysis of interactions in rodents. Neurosci Biobehav R. 2022. https://doi.org/10.1016/j.neubiorev.2022.104554 .

Spake R, Bowler DE, Callaghan CT, Blowes SA, Doncaster CP, Antão LH, Nakagawa S, McElreath R, Chase JM. Understanding ‘it depends’ in ecology: a guide to hypothesising, visualising and interpreting statistical interactions. Biol Rev. 2023. https://doi.org/10.1111/brv.12939 .

Siviter H, Bailes EJ, Martin CD, Oliver TR, Koricheva J, Leadbeater E, Brown MJF. Agrochemicals interact synergistically to increase bee mortality. Nature. 2021;596(7872):389.

Salanti G, Schmid CH. Research synthesis methods special issue on network meta-analysis: introduction from the editors. Res Synth Methods. 2012;3(2):69–70.

Gomez-Creutzberg C, Lagisz M, Nakagawa S, Brockerhoff EG, Tylianakis JM. Consistent trade-offs in ecosystem services between land covers with different production intensities. Biol Rev. 2021;96(5):1989–2008.

Jackson D, White IR, Price M, Copas J, Riley RD. Borrowing of strength and study weights in multivariate and network meta-analysis. Stat Methods Med Res. 2017;26(6):2853–68.

Sanchez-Tojar A, Moran NP, O’Dea RE, Reinhold K, Nakagawa S. Illustrating the importance of meta-analysing variances alongside means in ecology and evolution. J Evol Biol. 2020;33(9):1216–23.

Riley RD, Lambert PC, Abo-Zaid G. Meta-analysis of individual participant data: rationale, conduct, and reporting. BMJ. 2010;340:c221.

Riley RD, Tierney JF, Stewart LA. Individual participant data meta-analysis : a handbook for healthcare research. 1st ed. Hoboken: Wiley; 2021.

Hudson LN, Newbold T, Contu S, Hill SLL, Lysenko I, De Palma A, Phillips HRP, Alhusseini TI, Bedford FE, Bennett DJ, et al. The database of the PREDICTS (projecting responses of ecological diversity in changing terrestrial systems) project. Ecol Evol. 2017;7(1):145–88.

Dornelas M, Antao LH, Moyes F, Bates AE, Magurran AE, Adam D, Akhmetzhanova AA, Appeltans W, Arcos JM, Arnold H, et al. BioTIME: a database of biodiversity time series for the anthropocene. Glob Ecol Biogeogr. 2018;27(7):760–86.

Mengersen K, Gurevitch J, Schmid CH. Meta-analysis of primary data. In: Koricheva J, Gurevitch J, Mengersen K, editors. Handbook of meta-analysis in ecology and evolution. Priceton: Princeton university; 2013. p. 300–12.

Spake R, O’Dea RE, Nakagawa S, Doncaster CP, Ryo M, Callaghan CT, Bullock JM. Improving quantitative synthesis to achieve generality in ecology. Nat Ecol Evol. 2022;6(12):1818–28.

Gerstner K, Moreno-Mateos D, Gurevitch J, Beckmann M, Kambach S, Jones HP, Seppelt R. Will your paper be used in a meta-analysis? Make the reach of your research broader and longer lasting. Methods Ecol Evol. 2017;8(6):777–84.

Haddaway NR. A call for better reporting of conservation research data for use in meta-analyses. Conserv Biol. 2015;29(4):1242–5.

Midolo G, De Frenne P, Holzel N, Wellstein C. Global patterns of intraspecific leaf trait responses to elevation. Global Change Biol. 2019;25(7):2485–98.

White IR, Schmid CH, Stijnen T. Choice of effect measure and issues in extracting outcome data. In: Schmid CH, Stijnen T, White IR, editors. Handbook of meta-analysis. Boca Raton: CRC; 2021.

Lajeunesse MJ. Bias and correction for the log response ratio in ecological meta-analysis. Ecology. 2015;96(8):2056–63.

Download references

Acknowledgements

SN, ELM, and ML were supported by the ARC (Australian Research Council) Discovery grant (DP200100367), and SN, YY, and ML by the ARC Discovery grant (DP210100812). YY was also supported by the National Natural Science Foundation of China (32102597). A part of this research was conducted while visiting the Okinawa Institute of Science and Technology (OIST) through the Theoretical Sciences Visiting Program (TSVP) to SN.

Australian Research Council Discovery grant (DP200100367); Australian Research Council Discovery grant (DP210100812); The National Natural Science Foundation of China (32102597).

Author information

Authors and affiliations.

Evolution & Ecology Research Centre and School of Biological, Earth and Environmental Sciences, University of New South Wales, Sydney, NSW, 2052, Australia

Shinichi Nakagawa, Yefeng Yang, Erin L. Macartney & Malgorzata Lagisz

Theoretical Sciences Visiting Program, Okinawa Institute of Science and Technology Graduate University, Onna, 904-0495, Japan

Shinichi Nakagawa

School of Biological Sciences, Whiteknights Campus, University of Reading, Reading, RG6 6AS, UK

Rebecca Spake

You can also search for this author in PubMed   Google Scholar

Contributions

SN was commissioned to write this article so he assembled a team of co-authors. SN discussed the idea with YY, ELM, RS and ML, and all of them contributed to the design of this review. ML led the survey working with YY and ELM, while YY led the creation of the accompanying webpage working with RS. SN supervised all aspects of this work and wrote the first draft, which was commented on, edited, and therefore, significantly improved by the other co-authors. All authors read and approved the final manuscript.

Corresponding authors

Correspondence to Shinichi Nakagawa or Yefeng Yang .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Contest for publication

The authors provide consent for publication.

Competing interests

The authors report no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1:.

The survey of meta-analyses in environmnetal sciences.

Additional file 2:

The hands-on R tutorial.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Nakagawa, S., Yang, Y., Macartney, E.L. et al. Quantitative evidence synthesis: a practical guide on meta-analysis, meta-regression, and publication bias tests for environmental sciences. Environ Evid 12 , 8 (2023). https://doi.org/10.1186/s13750-023-00301-6

Download citation

Received : 13 January 2023

Accepted : 23 March 2023

Published : 24 April 2023

DOI : https://doi.org/10.1186/s13750-023-00301-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Hierarchical models
• Robust variance estimation
• Spatial dependency
• Variance–covariance matrix
• Meta-analysis of variance
• Network meta-analysis
• Multivariate meta-analysis

Environmental Evidence

ISSN: 2047-2382

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Advertisement

Meta-Analysis and Meta-Synthesis Methodologies: Rigorously Piecing Together Research

• Original Paper
• Published: 18 June 2018
• Volume 62 , pages 525–534, ( 2018 )

Cite this article

• Heather Leary   ORCID: orcid.org/0000-0002-2487-578X 1 &
• Andrew Walker 2  

5302 Accesses

31 Citations

7 Altmetric

Explore all metrics

For a variety of reasons, education research can be difficult to summarize. Varying contexts, designs, levels of quality, measurement challenges, definition of underlying constructs, and treatments as well as the complexity of research subjects themselves can result in variability. Education research is voluminous and draws on multiple methods including quantitative, as well as, qualitative approaches to answer key research questions. With increased numbers of empirical research in Instructional Design and Technology (IDT), using various synthesis methods can provide a means to more deeply understand trends and patterns in research findings across multiple studies. The purpose of this article is to illustrate structured review or meta-synthesis procedures for qualitative research, as well as, novel meta-analysis procedures for the kinds of multiple treatment designs common to IDT settings. Sample analyses are used to discuss key methodological ideas as a way to introduce researchers to these techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Theories of Motivation in Education: an Integrative Framework

Detlef Urhahne & Lisette Wijnia

How phenomenology can help us learn from the experiences of others

Brian E. Neubauer, Catherine T. Witkop & Lara Varpio

Ethical Considerations of Conducting Systematic Reviews in Educational Research

Attride-Stirling, J. (2001). Thematic networks: An analytic tool for qualitative research. Qualitative Research, 1 (3), 385–405.

Article   Google Scholar  

Barnett-Page, E., & Thomas, J. (2009). Methods for the synthesis of qualitative research: a critical review. ESRC National Centre for Research Methods, Social Science Research Unit, Institute of Education, University of London, London, 01/09.

Barton, E. E., Pustejovsky, J. E., Maggin, D. M., & Reichow, B. (2017). Technology-aided instruction and intervention for students with ASD: A meta-analysis using novel methods of estimating effect sizes for single-case research. Remedial and Special Education, 38 (6), 371–386.

Beal, C. R., Arroyo, I., Cohen, P. R., & Woolf, B. P. (2010). Evaluation of AnimalWatch: An intelligent tutoring system for arithmetic and fractions. Journal of Interactive Online Learning, 9 (1), 64–77.

Google Scholar  

Belland, B., Walker, A., Kim, N., & Lefler, M. (2017a). Synthesizing results from empirical research on computer-based scaffolding in STEM education: A meta-analysis. Review of Educational Research, 82 (2), 309–344.

Belland, B., Walker, A., & Kim, N. (2017b). A bayesian network meta-analysis to synthesize the influence of contexts of scaffolding use on cognitive outcomes in STEM education. Review of Educational Research, 87 (6), 1042–1081.

Bhatnagar, N., Lakshmi, P. V. M., & Jeyashree, K. (2014). Multiple treatment and indirect treatment comparisons: An overview of network meta-analysis. Perspectives in Clinical Research, 5 (4), 154–158. https://doi.org/10.4103/2229-3485.140550 .

Boote, D. N., & Beile, P. (2005). Scholars before researchers: On the centrality of the dissertation literature review in research preparation. Educational Researcher, 34 (6), 3–15. https://doi.org/10.3102/0013189X034006003 .

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis . Chichester: Wiley.

Book   Google Scholar  

Bulu, S., & Pedersen, S. (2010). Scaffolding middle school students’ content knowledge and illstructured problem solving in a problem-based hypermedia learning environment. Educational Technology Research & Development, 58 (5), 507–529.

Cooper, H. (2009). Research synthesis and meta-analysis: A step-by-step approach (applied social research methods) . Thousand Oaks: Sage Publications.

Cooper, H. (2016). Research synthesis and meta-analysis: A step-by-step approach (5th ed.). Thousand Oaks: Sage publications.

Cooper, H., & Hedges, L. V. (1994). The handbook of research synthesis . New York: Russell Sage Foundation.

Finfgeld, D. L. (2003). Metasynthesis: The state of the art - so far. Qualitative Health Research, 13 (7), 893–904.

Finfgeld-Connett, D. (2010). Generalizability and transferability of meta-synthesis research findings. Journal of Advanced Nursing, 66 (2), 246–254.

Finfgeld-Connett, D. (2014). Metasynthesis findings: Potential versus reality. Qualitative Health Research, 24 , 1581–1591. https://doi.org/10.1177/1049732314548878 .

Finfgeld-Connett, D. (2016). The future of theory-generating meta-synthesis research. Qualitative Health Research, 26 (3), 291–293.

Finlayson, K., & Dixon, A. (2008). Qualitative meta-synthesis: A guide for the novice. Nurse Researcher, 15 (2), 59–71.

Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5 (10), 3–8.

Glass, G. V. (2000). Meta-analysis at 25 . http://www.gvglass.info/papers/meta25.html

Gurevitch, J., Koricheva, J., Nakagawa, S., & Stewart, G. (2018). Meta-analysis and the science of research synthesis. Nature, 555 , 175–182.

Harden, A. (2010). Mixed-methods systematic reviews: Integrating quantitative and qualitative findings. Focus Technical Brief, 25 , 1–8.

Heller, R. (1990). The role of hypermedia in education: A look at the research issues. Journal of Research on Computing in Education, 22 , 431–441.

Higgins, S. (2016). Meta-synthesis and comparative meta-analysis of education research findings: Some risks and benefits. Review of Education, 4 (1), 31–53.

Higgins, J. P., & Green, S. (2008). Cochrane handbook for systematic reviews of interventions . Chichester: Wiley. https://doi.org/10.1002/9780470712184 .

Higgins, J. P. T., Altman, D. G., Gøtzsche, P. C., Jüni, P., Moher, D., Oxman, A. D., … Sterne, J. (2011). The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ, 343 . Retrieved from http://www.bmj.com/content/343/bmj.d5928

Karich, A. C., Burns, M. K., & Maki, K. E. (2014). Updated meta-analysis of learner control within educational technology. Review of Educational Research, 84 (3), 392–410.

Lachal, J., Revah-Levy, A., Orri, M., & Moro, M. R. (2017). Metasynthesis: An original method to synthesize qualitative literature. Frontiers in Psychiatry, 8 , 1–9.

Lam, S. K. H., & Owen, A. (2007). Combined resynchronisation and implantable defibrillator therapy in left ventricular dysfunction: Bayesian network meta-analysis of randomised controlled trials. BMJ: British Medical Journal, 335 (7626), 925–928.

Leary, H., Shelton, B. E., & Walker, A. (2010). Rich visual media meta-analyses for learning: An approach at meta-synthesis. Paper presented at the American Educational Research Association annual meeting, Denver, CO.

Lin, H. (2015). A meta-synthesis of empirical research on the effectiveness of computer-mediated communication (CMC) in SLA. Language Learning & technology, 19 (2), 85–117 Retrieved from http://llt.msu.edu/issues/june2015/lin.pdf .

Lipsey, M., & Wilson, D. (2001). Practical meta-analysis (Applied social research methods) . Thousand Oaks: Sage Publications.

Liu, Y., Cornish, A., & Clegg, J. (2007). ICT and special educational needs: Using meta-synthesis for bridging the multifaceted divide. In Y. Shi, G. D. van Albada, J. Dongarra, & P. M. A. Sloot (Eds.), Computational science–ICCS 2007. ICCS 2007. Lecture notes in computer science (Vol. 4490). Berlin, Heidelberg: Springer.

Ludvigsen, M. S., Hall, E. O. C., Meyer, G., Fegran, L., Aagaard, H., & Uhrenfeldt, L. (2016). Using Sandelowski and Barroso’s meta-synthesis method in advancing qualitative evidence. Qualitative Health Research, 26 (3), 320–329.

Lumley, T. (2002). Network meta-analysis for indirect treatment comparisons. Statistics in Medicine, 21 (16), 2313–2324. https://doi.org/10.1002/sim.1201 .

Lunn, D., Spiegelhalter, D., Thomas, A., & Best, N. (2009). The BUGS project: Evolution, critique and future directions. Statistics in Medicine, 28 (25), 3049–3067.

McKenney, S., & Reeves, T. C. (2012). Conducting educational design research . London: Routledge.

Mumtaz, S. (2000). Factors affecting teachers’ use of information and communications technology: A review of the literature. Journal of Information Technology for Teacher Education, 9 (3), 319–342.

Noblit, G. W., & Hare, R. D. (1988). Meta-ethnography: Synthesizing qualitative studies . London: Sage.

Paterson, B. L., Thorne, S. E., Canam, C., & Jillings, C. (2001). Meta-study of qualitative health research: A practical guide to meta-analysis and meta-synthesis . London: Sage.

Puhan, M. A., Schünemann, H. J., Murad, M. H., Li, T., Brignardello-Petersen, R., Singh, J. A., … Guyatt, G. H. (2014). A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ, 349, g5630.

Reeves, T., & Oh, E. (2017). The goals and methods of educational technology research over a quarter century (1989–2014). Educational Technology Research & Development, 65 (2), 325–339.

Salanti, G., Giovane, C. D., Chaimani, A., Caldwell, D. M., & Higgins, J. P. T. (2014). Evaluating the quality of evidence from a network meta-analysis. PLOS ONE, 9 (7), e99682.

Sandelowski, M., & Barroso, J. (2003). Toward a metasynthesis of qualitative findings on motherhood in HIV-positive women. Research in Nursing and Health, 26 (2), 153–170.

Sandelowski, M., Docherty, S., & Emden, C. (1997). Quality metasynthesis: Issues and techniques. Research in Nursing and Health, 20 , 365–371.

Taquero, J. M. (2011). A meta-ethnographic synthesis of support services in distance learning programs. Journal of Information Technology Education: Innovations in Practice, 10 , 157–179.

Thorne, S., Jensen, L., Kearney, M. H., Noblit, G., & Sandelowski, M. (2004). Qualitative metasynthesis: Reflection on methodological orientation and ideological agenda. Qualitative Health Research, 14 (10), 1342–1365.

Tondeur, J., van Braak, J., Sang, G., Voogt, J., Fisser, P., & Ottenbreit-Lewftwich, A. (2012). Preparing pre-service teachers to integrate technology in education: A synthesis of qualitative evidence. Computers & Education, 59 (1), 134–144.

Torraco, R. J. (2016). Writing integrative literature reviews: Using the past and present to explore the future. Human Resource Development Review, 15 (4), 404–428.

Walker, A., Belland, B., Kim, N., & Lefler, M. (2016). Searching for structure: A bayesian network meta-analysis of computer-based scaffolding research in STEM education. Paper presented at the American Educational Research Association, San Antonio, TX.

Walsh, D., & Downe, S. (2005). Meta-synthesis method for qualitative research: A literature review. Methodological Issues in Nursing Education, 50 (2), 204–211.

Young, J. (2017). Technology-enhanced mathematics instruction: A second-order meta-analysis of 30 years of research. Educational Research Review, 22 , 19–33.

Download references

Author information

Authors and affiliations.

Brigham Young University, 150-G MCKB, Provo, UT, 84602, USA

Heather Leary

Utah State University, 2830 Old Main Hill, Logan, UT, 84322, USA

Andrew Walker

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Heather Leary .

Ethics declarations

Conflict of interest.

The authors declare that they have no conflict of interest.

Human Studies

This article does not contain any studies with human participants performed by any of the authors.

Rights and permissions

Reprints and permissions

About this article

Leary, H., Walker, A. Meta-Analysis and Meta-Synthesis Methodologies: Rigorously Piecing Together Research. TechTrends 62 , 525–534 (2018). https://doi.org/10.1007/s11528-018-0312-7

Download citation

Published : 18 June 2018

Issue Date : September 2018

DOI : https://doi.org/10.1007/s11528-018-0312-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Meta-analysis
• Meta-synthesis
• Synthesis methods
• Education research
• Find a journal
• Publish with us
• Track your research

• How it works

Meta-Analysis – Guide with Definition, Steps & Examples

Published by Owen Ingram at April 26th, 2023 , Revised On April 26, 2023

“A meta-analysis is a formal, epidemiological, quantitative study design that uses statistical methods to generalise the findings of the selected independent studies. “

Meta-analysis and systematic review are the two most authentic strategies in research. When researchers start looking for the best available evidence concerning their research work, they are advised to begin from the top of the evidence pyramid. The evidence available in the form of meta-analysis or systematic reviews addressing important questions is significant in academics because it informs decision-making.

What is Meta-Analysis  

Meta-analysis estimates the absolute effect of individual independent research studies by systematically synthesising or merging the results. Meta-analysis isn’t only about achieving a wider population by combining several smaller studies. It involves systematic methods to evaluate the inconsistencies in participants, variability (also known as heterogeneity), and findings to check how sensitive their findings are to the selected systematic review protocol.   

When Should you Conduct a Meta-Analysis?

Meta-analysis has become a widely-used research method in medical sciences and other fields of work for several reasons. The technique involves summarising the results of independent systematic review studies. 

The Cochrane Handbook explains that “an important step in a systematic review is the thoughtful consideration of whether it is appropriate to combine the numerical results of all, or perhaps some, of the studies. Such a meta-analysis yields an overall statistic (together with its confidence interval) that summarizes the effectiveness of an experimental intervention compared with a comparator intervention” (section 10.2).

A researcher or a practitioner should choose meta-analysis when the following outcomes are desirable. 

For generating new hypotheses or ending controversies resulting from different research studies. Quantifying and evaluating the variable results and identifying the extent of conflict in literature through meta-analysis is possible. 

To find research gaps left unfilled and address questions not posed by individual studies. Primary research studies involve specific types of participants and interventions. A review of these studies with variable characteristics and methodologies can allow the researcher to gauge the consistency of findings across a wider range of participants and interventions. With the help of meta-analysis, the reasons for differences in the effect can also be explored. 

To provide convincing evidence. Estimating the effects with a larger sample size and interventions can provide convincing evidence. Many academic studies are based on a very small dataset, so the estimated intervention effects in isolation are not fully reliable.

Elements of a Meta-Analysis

Deeks et al. (2019), Haidilch (2010), and Grant & Booth (2009) explored the characteristics, strengths, and weaknesses of conducting the meta-analysis. They are briefly explained below. 

Characteristics: 

• A systematic review must be completed before conducting the meta-analysis because it provides a summary of the findings of the individual studies synthesised. 
• You can only conduct a meta-analysis by synthesising studies in a systematic review. 
• The studies selected for statistical analysis for the purpose of meta-analysis should be similar in terms of comparison, intervention, and population. 

Strengths: 

• A meta-analysis takes place after the systematic review. The end product is a comprehensive quantitative analysis that is complicated but reliable. 
• It gives more value and weightage to existing studies that do not hold practical value on their own. 
• Policy-makers and academicians cannot base their decisions on individual research studies. Meta-analysis provides them with a complex and solid analysis of evidence to make informed decisions. 

Criticisms: 

• The meta-analysis uses studies exploring similar topics. Finding similar studies for the meta-analysis can be challenging.
• When and if biases in the individual studies or those related to reporting and specific research methodologies are involved, the meta-analysis results could be misleading.

Steps of Conducting the Meta-Analysis 

The process of conducting the meta-analysis has remained a topic of debate among researchers and scientists. However, the following 5-step process is widely accepted. 

Step 1: Research Question

The first step in conducting clinical research involves identifying a research question and proposing a hypothesis . The potential clinical significance of the research question is then explained, and the study design and analytical plan are justified.

Step 2: Systematic Review 

The purpose of a systematic review (SR) is to address a research question by identifying all relevant studies that meet the required quality standards for inclusion. While established journals typically serve as the primary source for identified studies, it is important to also consider unpublished data to avoid publication bias or the exclusion of studies with negative results.

While some meta-analyses may limit their focus to randomized controlled trials (RCTs) for the sake of obtaining the highest quality evidence, other experimental and quasi-experimental studies may be included if they meet the specific inclusion/exclusion criteria established for the review.

Step 3: Data Extraction

After selecting studies for the meta-analysis, researchers extract summary data or outcomes, as well as sample sizes and measures of data variability for both intervention and control groups. The choice of outcome measures depends on the research question and the type of study, and may include numerical or categorical measures.

For instance, numerical means may be used to report differences in scores on a questionnaire or changes in a measurement, such as blood pressure. In contrast, risk measures like odds ratios (OR) or relative risks (RR) are typically used to report differences in the probability of belonging to one category or another, such as vaginal birth versus cesarean birth.

Step 4: Standardisation and Weighting Studies

After gathering all the required data, the fourth step involves computing suitable summary measures from each study for further examination. These measures are typically referred to as Effect Sizes and indicate the difference in average scores between the control and intervention groups. For instance, it could be the variation in blood pressure changes between study participants who used drug X and those who used a placebo.

Since the units of measurement often differ across the included studies, standardization is necessary to create comparable effect size estimates. Standardization is accomplished by determining, for each study, the average score for the intervention group, subtracting the average score for the control group, and dividing the result by the relevant measure of variability in that dataset.

In some cases, the results of certain studies must carry more significance than others. Larger studies, as measured by their sample sizes, are deemed to produce more precise estimates of effect size than smaller studies. Additionally, studies with less variability in data, such as smaller standard deviation or narrower confidence intervals, are typically regarded as higher quality in study design. A weighting statistic that aims to incorporate both of these factors, known as inverse variance, is commonly employed.

Step 5: Absolute Effect Estimation

The ultimate step in conducting a meta-analysis is to choose and utilize an appropriate model for comparing Effect Sizes among diverse studies. Two popular models for this purpose are the Fixed Effects and Random Effects models. The Fixed Effects model relies on the premise that each study is evaluating a common treatment effect, implying that all studies would have estimated the same Effect Size if sample variability were equal across all studies.

Conversely, the Random Effects model posits that the true treatment effects in individual studies may vary from each other, and endeavors to consider this additional source of interstudy variation in Effect Sizes. The existence and magnitude of this latter variability is usually evaluated within the meta-analysis through a test for ‘heterogeneity.’

Forest Plot

The results of a meta-analysis are often visually presented using a “Forest Plot”. This type of plot displays, for each study, included in the analysis, a horizontal line that indicates the standardized Effect Size estimate and 95% confidence interval for the risk ratio used. Figure A provides an example of a hypothetical Forest Plot in which drug X reduces the risk of death in all three studies.

However, the first study was larger than the other two, and as a result, the estimates for the smaller studies were not statistically significant. This is indicated by the lines emanating from their boxes, including the value of 1. The size of the boxes represents the relative weights assigned to each study by the meta-analysis. The combined estimate of the drug’s effect, represented by the diamond, provides a more precise estimate of the drug’s effect, with the diamond indicating both the combined risk ratio estimate and the 95% confidence interval limits.

Figure-A: Hypothetical Forest Plot

Relevance to Practice and Research 

  Evidence Based Nursing commentaries often include recently published systematic reviews and meta-analyses, as they can provide new insights and strengthen recommendations for effective healthcare practices. Additionally, they can identify gaps or limitations in current evidence and guide future research directions.

The quality of the data available for synthesis is a critical factor in the strength of conclusions drawn from meta-analyses, and this is influenced by the quality of individual studies and the systematic review itself. However, meta-analysis cannot overcome issues related to underpowered or poorly designed studies.

Therefore, clinicians may still encounter situations where the evidence is weak or uncertain, and where higher-quality research is required to improve clinical decision-making. While such findings can be frustrating, they remain important for informing practice and highlighting the need for further research to fill gaps in the evidence base.

Methods and Assumptions in Meta-Analysis 

Ensuring the credibility of findings is imperative in all types of research, including meta-analyses. To validate the outcomes of a meta-analysis, the researcher must confirm that the research techniques used were accurate in measuring the intended variables. Typically, researchers establish the validity of a meta-analysis by testing the outcomes for homogeneity or the degree of similarity between the results of the combined studies.

Homogeneity is preferred in meta-analyses as it allows the data to be combined without needing adjustments to suit the study’s requirements. To determine homogeneity, researchers assess heterogeneity, the opposite of homogeneity. Two widely used statistical methods for evaluating heterogeneity in research results are Cochran’s-Q and I-Square, also known as I-2 Index.

Difference Between Meta-Analysis and Systematic Reviews

Meta-analysis and systematic reviews are both research methods used to synthesise evidence from multiple studies on a particular topic. However, there are some key differences between the two.

Systematic reviews involve a comprehensive and structured approach to identifying, selecting, and critically appraising all available evidence relevant to a specific research question. This process involves searching multiple databases, screening the identified studies for relevance and quality, and summarizing the findings in a narrative report.

Meta-analysis, on the other hand, involves using statistical methods to combine and analyze the data from multiple studies, with the aim of producing a quantitative summary of the overall effect size. Meta-analysis requires the studies to be similar enough in terms of their design, methodology, and outcome measures to allow for meaningful comparison and analysis.

Therefore, systematic reviews are broader in scope and summarize the findings of all studies on a topic, while meta-analyses are more focused on producing a quantitative estimate of the effect size of an intervention across multiple studies that meet certain criteria. In some cases, a systematic review may be conducted without a meta-analysis if the studies are too diverse or the quality of the data is not sufficient to allow for statistical pooling.

Software Packages For Meta-Analysis

Meta-analysis can be done through software packages, including free and paid options. One of the most commonly used software packages for meta-analysis is RevMan by the Cochrane Collaboration.

Assessing the Quality of Meta-Analysis 

Assessing the quality of a meta-analysis involves evaluating the methods used to conduct the analysis and the quality of the studies included. Here are some key factors to consider:

• Study selection: The studies included in the meta-analysis should be relevant to the research question and meet predetermined criteria for quality.
• Search strategy: The search strategy should be comprehensive and transparent, including databases and search terms used to identify relevant studies.
• Study quality assessment: The quality of included studies should be assessed using appropriate tools, and this assessment should be reported in the meta-analysis.
• Data extraction: The data extraction process should be systematic and clearly reported, including any discrepancies that arose.
• Analysis methods: The meta-analysis should use appropriate statistical methods to combine the results of the included studies, and these methods should be transparently reported.
• Publication bias: The potential for publication bias should be assessed and reported in the meta-analysis, including any efforts to identify and include unpublished studies.
• Interpretation of results: The results should be interpreted in the context of the study limitations and the overall quality of the evidence.
• Sensitivity analysis: Sensitivity analysis should be conducted to evaluate the impact of study quality, inclusion criteria, and other factors on the overall results.

Overall, a high-quality meta-analysis should be transparent in its methods and clearly report the included studies’ limitations and the evidence’s overall quality.

Hire an Expert Writer

Orders completed by our expert writers are

• Formally drafted in an academic style
• Free Amendments and 100% Plagiarism Free – or your money back!
• 100% Confidential and Timely Delivery!
• Free anti-plagiarism report
• Appreciated by thousands of clients. Check client reviews

Examples of Meta-Analysis

• STANLEY T.D. et JARRELL S.B. (1989), « Meta-regression analysis : a quantitative method of literature surveys », Journal of Economics Surveys, vol. 3, n°2, pp. 161-170.
• DATTA D.K., PINCHES G.E. et NARAYANAN V.K. (1992), « Factors influencing wealth creation from mergers and acquisitions : a meta-analysis », Strategic Management Journal, Vol. 13, pp. 67-84.
• GLASS G. (1983), « Synthesising empirical research : Meta-analysis » in S.A. Ward and L.J. Reed (Eds), Knowledge structure and use : Implications for synthesis and interpretation, Philadelphia : Temple University Press.
• WOLF F.M. (1986), Meta-analysis : Quantitative methods for research synthesis, Sage University Paper n°59.
• HUNTER J.E., SCHMIDT F.L. et JACKSON G.B. (1982), « Meta-analysis : cumulating research findings across studies », Beverly Hills, CA : Sage.

Frequently Asked Questions

What is a meta-analysis in research.

Meta-analysis is a statistical method used to combine results from multiple studies on a specific topic. By pooling data from various sources, meta-analysis can provide a more precise estimate of the effect size of a treatment or intervention and identify areas for future research.

Why is meta-analysis important?

Meta-analysis is important because it combines and summarizes results from multiple studies to provide a more precise and reliable estimate of the effect of a treatment or intervention. This helps clinicians and policymakers make evidence-based decisions and identify areas for further research.

What is an example of a meta-analysis?

A meta-analysis of studies evaluating physical exercise’s effect on depression in adults is an example. Researchers gathered data from 49 studies involving a total of 2669 participants. The studies used different types of exercise and measures of depression, which made it difficult to compare the results.

Through meta-analysis, the researchers calculated an overall effect size and determined that exercise was associated with a statistically significant reduction in depression symptoms. The study also identified that moderate-intensity aerobic exercise, performed three to five times per week, was the most effective. The meta-analysis provided a more comprehensive understanding of the impact of exercise on depression than any single study could provide.

What is the definition of meta-analysis in clinical research?

Meta-analysis in clinical research is a statistical technique that combines data from multiple independent studies on a particular topic to generate a summary or “meta” estimate of the effect of a particular intervention or exposure.

This type of analysis allows researchers to synthesise the results of multiple studies, potentially increasing the statistical power and providing more precise estimates of treatment effects. Meta-analyses are commonly used in clinical research to evaluate the effectiveness and safety of medical interventions and to inform clinical practice guidelines.

Is meta-analysis qualitative or quantitative?

Meta-analysis is a quantitative method used to combine and analyze data from multiple studies. It involves the statistical synthesis of results from individual studies to obtain a pooled estimate of the effect size of a particular intervention or treatment. Therefore, meta-analysis is considered a quantitative approach to research synthesis.

You May Also Like

Struggling to figure out “whether I should choose primary research or secondary research in my dissertation?” Here are some tips to help you decide.

In correlational research, a researcher measures the relationship between two or more variables or sets of scores without having control over the variables.

This article provides the key advantages of primary research over secondary research so you can make an informed decision.

USEFUL LINKS

LEARNING RESOURCES

COMPANY DETAILS

• How It Works

Study Design 101: Meta-Analysis

• Case Report
• Case Control Study
• Cohort Study
• Randomized Controlled Trial
• Practice Guideline
• Systematic Review

Meta-Analysis

• Helpful Formulas
• Finding Specific Study Types

A subset of systematic reviews; a method for systematically combining pertinent qualitative and quantitative study data from several selected studies to develop a single conclusion that has greater statistical power. This conclusion is statistically stronger than the analysis of any single study, due to increased numbers of subjects, greater diversity among subjects, or accumulated effects and results.

Meta-analysis would be used for the following purposes:

• To establish statistical significance with studies that have conflicting results
• To develop a more correct estimate of effect magnitude
• To provide a more complex analysis of harms, safety data, and benefits
• To examine subgroups with individual numbers that are not statistically significant

If the individual studies utilized randomized controlled trials (RCT), combining several selected RCT results would be the highest-level of evidence on the evidence hierarchy, followed by systematic reviews, which analyze all available studies on a topic.

• Greater statistical power
• Confirmatory data analysis
• Greater ability to extrapolate to general population affected
• Considered an evidence-based resource

Disadvantages

• Difficult and time consuming to identify appropriate studies
• Not all studies provide adequate data for inclusion and analysis
• Requires advanced statistical techniques
• Heterogeneity of study populations

Design pitfalls to look out for

The studies pooled for review should be similar in type (i.e. all randomized controlled trials).

Are the studies being reviewed all the same type of study or are they a mixture of different types?

The analysis should include published and unpublished results to avoid publication bias.

Does the meta-analysis include any appropriate relevant studies that may have had negative outcomes?

Fictitious Example

Do individuals who wear sunscreen have fewer cases of melanoma than those who do not wear sunscreen? A MEDLINE search was conducted using the terms melanoma, sunscreening agents, and zinc oxide, resulting in 8 randomized controlled studies, each with between 100 and 120 subjects. All of the studies showed a positive effect between wearing sunscreen and reducing the likelihood of melanoma. The subjects from all eight studies (total: 860 subjects) were pooled and statistically analyzed to determine the effect of the relationship between wearing sunscreen and melanoma. This meta-analysis showed a 50% reduction in melanoma diagnosis among sunscreen-wearers.

Real-life Examples

Goyal, A., Elminawy, M., Kerezoudis, P., Lu, V., Yolcu, Y., Alvi, M., & Bydon, M. (2019). Impact of obesity on outcomes following lumbar spine surgery: A systematic review and meta-analysis. Clinical Neurology and Neurosurgery, 177 , 27-36. https://doi.org/10.1016/j.clineuro.2018.12.012

This meta-analysis was interested in determining whether obesity affects the outcome of spinal surgery. Some previous studies have shown higher perioperative morbidity in patients with obesity while other studies have not shown this effect. This study looked at surgical outcomes including "blood loss, operative time, length of stay, complication and reoperation rates and functional outcomes" between patients with and without obesity. A meta-analysis of 32 studies (23,415 patients) was conducted. There were no significant differences for patients undergoing minimally invasive surgery, but patients with obesity who had open surgery had experienced higher blood loss and longer operative times (not clinically meaningful) as well as higher complication and reoperation rates. Further research is needed to explore this issue in patients with morbid obesity.

Nakamura, A., van Der Waerden, J., Melchior, M., Bolze, C., El-Khoury, F., & Pryor, L. (2019). Physical activity during pregnancy and postpartum depression: Systematic review and meta-analysis. Journal of Affective Disorders, 246 , 29-41. https://doi.org/10.1016/j.jad.2018.12.009

This meta-analysis explored whether physical activity during pregnancy prevents postpartum depression. Seventeen studies were included (93,676 women) and analysis showed a "significant reduction in postpartum depression scores in women who were physically active during their pregnancies when compared with inactive women." Possible limitations or moderators of this effect include intensity and frequency of physical activity, type of physical activity, and timepoint in pregnancy (e.g. trimester).

Related Terms

A document often written by a panel that provides a comprehensive review of all relevant studies on a particular clinical or health-related topic/question.

Publication Bias

A phenomenon in which studies with positive results have a better chance of being published, are published earlier, and are published in journals with higher impact factors. Therefore, conclusions based exclusively on published studies can be misleading.

Now test yourself!

1. A Meta-Analysis pools together the sample populations from different studies, such as Randomized Controlled Trials, into one statistical analysis and treats them as one large sample population with one conclusion.

a) True b) False

2. One potential design pitfall of Meta-Analyses that is important to pay attention to is:

a) Whether it is evidence-based. b) If the authors combined studies with conflicting results. c) If the authors appropriately combined studies so they did not compare apples and oranges. d) If the authors used only quantitative data.

Evidence Pyramid - Navigation

• Meta- Analysis
• Case Reports
• << Previous: Systematic Review
• Next: Helpful Formulas >>

• Last Updated: Sep 25, 2023 10:59 AM
• URL: https://guides.himmelfarb.gwu.edu/studydesign101

• Himmelfarb Intranet
• Privacy Notice
• Terms of Use
• GW is committed to digital accessibility. If you experience a barrier that affects your ability to access content on this page, let us know via the Accessibility Feedback Form .
• Himmelfarb Health Sciences Library
• 2300 Eye St., NW, Washington, DC 20037
• Phone: (202) 994-2850
• [email protected]
• https://himmelfarb.gwu.edu

What Is Meta-Analysis? Definition, Research & Examples

Appinio Research · 01.02.2024 · 39min read

Are you looking to harness the power of data and uncover meaningful insights from a multitude of research studies? In a world overflowing with information, meta-analysis emerges as a guiding light, offering a systematic and quantitative approach to distilling knowledge from a sea of research.

This guide will demystify the art and science of meta-analysis, walking you through the process, from defining your research question to interpreting the results. Whether you're an academic researcher, a policymaker, or a curious mind eager to explore the depths of data, this guide will equip you with the tools and understanding needed to undertake robust and impactful meta-analyses.

What is a Meta Analysis?

Meta-analysis is a quantitative research method that involves the systematic synthesis and statistical analysis of data from multiple individual studies on a particular topic or research question. It aims to provide a comprehensive and robust summary of existing evidence by pooling the results of these studies, often leading to more precise and generalizable conclusions.

The primary purpose of meta-analysis is to:

• Quantify Effect Sizes:  Determine the magnitude and direction of an effect or relationship across studies.
• Evaluate Consistency:  Assess the consistency of findings among studies and identify sources of heterogeneity.
• Enhance Statistical Power:  Increase the statistical power to detect significant effects by combining data from multiple studies.
• Generalize Results:  Provide more generalizable results by analyzing a more extensive and diverse sample of participants or contexts.
• Examine Subgroup Effects:  Explore whether the effect varies across different subgroups or study characteristics.

Importance of Meta-Analysis

Meta-analysis plays a crucial role in scientific research and evidence-based decision-making. Here are key reasons why meta-analysis is highly valuable:

• Enhanced Precision:  By pooling data from multiple studies, meta-analysis provides a more precise estimate of the effect size, reducing the impact of random variation.
• Increased Statistical Power:  The combination of numerous studies enhances statistical power, making it easier to detect small but meaningful effects.
• Resolution of Inconsistencies:  Meta-analysis can help resolve conflicting findings in the literature by systematically analyzing and synthesizing evidence.
• Identification of Moderators:  It allows for the identification of factors that may moderate the effect, helping to understand when and for whom interventions or treatments are most effective.
• Evidence-Based Decision-Making:  Policymakers, clinicians, and researchers use meta-analysis to inform evidence-based decision-making, leading to more informed choices in healthcare , education, and other fields.
• Efficient Use of Resources:  Meta-analysis can guide future research by identifying gaps in knowledge, reducing duplication of efforts, and directing resources to areas with the most significant potential impact.

Types of Research Questions Addressed

Meta-analysis can address a wide range of research questions across various disciplines. Some common types of research questions that meta-analysis can tackle include:

• Treatment Efficacy:  Does a specific medical treatment, therapy, or intervention have a significant impact on patient outcomes or symptoms?
• Intervention Effectiveness:  How effective are educational programs, training methods, or interventions in improving learning outcomes or skills?
• Risk Factors and Associations:  What are the associations between specific risk factors, such as smoking or diet, and the likelihood of developing certain diseases or conditions?
• Impact of Policies:  What is the effect of government policies, regulations, or interventions on social, economic, or environmental outcomes?
• Psychological Constructs:  How do psychological constructs, such as self-esteem, anxiety, or motivation, influence behavior or mental health outcomes?
• Comparative Effectiveness:  Which of two or more competing interventions or treatments is more effective for a particular condition or population?
• Dose-Response Relationships:  Is there a dose-response relationship between exposure to a substance or treatment and the likelihood or severity of an outcome?

Meta-analysis is a versatile tool that can provide valuable insights into a wide array of research questions, making it an indispensable method in evidence synthesis and knowledge advancement.

Meta-Analysis vs. Systematic Review

In evidence synthesis and research aggregation, meta-analysis and systematic reviews are two commonly used methods, each serving distinct purposes while sharing some similarities. Let's explore the differences and similarities between these two approaches.

Meta-Analysis

• Purpose:  Meta-analysis is a statistical technique used to combine and analyze quantitative data from multiple individual studies that address the same research question. The primary aim of meta-analysis is to provide a single summary effect size that quantifies the magnitude and direction of an effect or relationship across studies.
• Data Synthesis:  In meta-analysis, researchers extract and analyze numerical data, such as means, standard deviations, correlation coefficients, or odds ratios, from each study. These effect size estimates are then combined using statistical methods to generate an overall effect size and associated confidence interval.
• Quantitative:  Meta-analysis is inherently quantitative, focusing on numerical data and statistical analyses to derive a single effect size estimate.
• Main Outcome:  The main outcome of a meta-analysis is the summary effect size, which provides a quantitative estimate of the research question's answer.

Systematic Review

• Purpose:  A systematic review is a comprehensive and structured overview of the available evidence on a specific research question. While systematic reviews may include meta-analysis, their primary goal is to provide a thorough and unbiased summary of the existing literature.
• Data Synthesis:  Systematic reviews involve a meticulous process of literature search, study selection, data extraction, and quality assessment. Researchers may narratively synthesize the findings, providing a qualitative summary of the evidence.
• Qualitative:  Systematic reviews are often qualitative in nature, summarizing and synthesizing findings in a narrative format. They do not always involve statistical analysis .
• Main Outcome:  The primary outcome of a systematic review is a comprehensive narrative summary of the existing evidence. While some systematic reviews include meta-analyses, not all do so.

Key Differences

• Nature of Data:  Meta-analysis primarily deals with quantitative data and statistical analysis , while systematic reviews encompass both quantitative and qualitative data, often presenting findings in a narrative format.
• Focus on Effect Size:  Meta-analysis focuses on deriving a single, quantitative effect size estimate, whereas systematic reviews emphasize providing a comprehensive overview of the literature, including study characteristics, methodologies, and key findings.
• Synthesis Approach:  Meta-analysis is a quantitative synthesis method, while systematic reviews may use both quantitative and qualitative synthesis approaches.

Commonalities

• Structured Process:  Both meta-analyses and systematic reviews follow a structured and systematic process for literature search, study selection, data extraction, and quality assessment.
• Evidence-Based:  Both approaches aim to provide evidence-based answers to specific research questions, offering valuable insights for decision-making in various fields.
• Transparency:  Both meta-analyses and systematic reviews prioritize transparency and rigor in their methodologies to minimize bias and enhance the reliability of their findings.

While meta-analysis and systematic reviews share the overarching goal of synthesizing research evidence, they differ in their approach and main outcomes. Meta-analysis is quantitative, focusing on effect sizes, while systematic reviews provide comprehensive overviews, utilizing both quantitative and qualitative data to summarize the literature. Depending on the research question and available data, one or both of these methods may be employed to provide valuable insights for evidence-based decision-making.

How to Conduct a Meta-Analysis?

Planning a meta-analysis is a critical phase that lays the groundwork for a successful and meaningful study. We will explore each component of the planning process in more detail, ensuring you have a solid foundation before diving into data analysis.

How to Formulate Research Questions?

Your research questions are the guiding compass of your meta-analysis. They should be precise and tailored to the topic you're investigating. To craft effective research questions:

• Clearly Define the Problem:  Start by identifying the specific problem or topic you want to address through meta-analysis.
• Specify Key Variables:  Determine the essential variables or factors you'll examine in the included studies.
• Frame Hypotheses:  If applicable, create clear hypotheses that your meta-analysis will test.

For example, if you're studying the impact of a specific intervention on patient outcomes, your research question might be: "What is the effect of Intervention X on Patient Outcome Y in published clinical trials?"

Eligibility Criteria

Eligibility criteria define the boundaries of your meta-analysis. By establishing clear criteria, you ensure that the studies you include are relevant and contribute to your research objectives. Key considerations for eligibility criteria include:

• Study Types:  Decide which types of studies will be considered (e.g., randomized controlled trials, cohort studies, case-control studies).
• Publication Time Frame:  Specify the publication date range for included studies.
• Language:  Determine whether studies in languages other than your primary language will be included.
• Geographic Region:  If relevant, define any geographic restrictions.

Your eligibility criteria should strike a balance between inclusivity and relevance. Excluding certain studies based on valid criteria ensures the quality and relevance of the data you analyze.

Search Strategy

A robust search strategy is fundamental to identifying all relevant studies. To create an effective search strategy:

• Select Databases:  Choose appropriate databases that cover your research area (e.g., PubMed, Scopus, Web of Science).
• Keywords and Search Terms:  Develop a comprehensive list of relevant keywords and search terms related to your research questions.
• Search Filters:  Utilize search filters and Boolean operators (AND, OR) to refine your search queries.
• Manual Searches:  Consider conducting hand-searches of key journals and reviewing the reference lists of relevant studies for additional sources.

Remember that the goal is to cast a wide net while maintaining precision to capture all relevant studies.

Data Extraction

Data extraction is the process of systematically collecting information from each selected study. It involves retrieving key data points, including:

• Study Characteristics:  Author(s), publication year, study design, sample size, duration, and location.
• Outcome Data:  Effect sizes, standard errors, confidence intervals, p-values, and any other relevant statistics.
• Methodological Details:  Information on study quality, risk of bias, and potential sources of heterogeneity.

Creating a standardized data extraction form is essential to ensure consistency and accuracy throughout this phase. Spreadsheet software, such as Microsoft Excel, is commonly used for data extraction.

Quality Assessment

Assessing the quality of included studies is crucial to determine their reliability and potential impact on your meta-analysis. Various quality assessment tools and checklists are available, depending on the study design. Some commonly used tools include:

• Newcastle-Ottawa Scale:  Used for assessing the quality of non-randomized studies (e.g., cohort, case-control studies).
• Cochrane Risk of Bias Tool:  Designed for evaluating randomized controlled trials.

Quality assessment typically involves evaluating aspects such as study design, sample size, data collection methods, and potential biases. This step helps you weigh the contribution of each study to the overall analysis.

How to Conduct a Literature Review?

Conducting a thorough literature review is a critical step in the meta-analysis process. We will explore the essential components of a literature review, from designing a comprehensive search strategy to establishing clear inclusion and exclusion criteria and, finally, the study selection process.

Comprehensive Search

To ensure the success of your meta-analysis, it's imperative to cast a wide net when searching for relevant studies. A comprehensive search strategy involves:

• Selecting Relevant Databases:  Identify databases that cover your research area comprehensively, such as PubMed, Scopus, Web of Science, or specialized databases specific to your field.
• Creating a Keyword List:  Develop a list of relevant keywords and search terms related to your research questions. Think broadly and consider synonyms, acronyms, and variations.
• Using Boolean Operators:  Utilize Boolean operators (AND, OR) to combine keywords effectively and refine your search.
• Applying Filters:  Employ search filters (e.g., publication date range, study type) to narrow down results based on your eligibility criteria.

Remember that the goal is to leave no relevant stone unturned, as missing key studies can introduce bias into your meta-analysis.

Inclusion and Exclusion Criteria

Clearly defined inclusion and exclusion criteria are the gatekeepers of your meta-analysis. These criteria ensure that the studies you include meet your research objectives and maintain the quality of your analysis. Consider the following factors when establishing criteria:

• Study Types:  Determine which types of studies are eligible for inclusion (e.g., randomized controlled trials, observational studies, case reports).
• Publication Time Frame:  Specify the time frame within which studies must have been published.
• Language:  Decide whether studies in languages other than your primary language will be included or excluded.
• Geographic Region:  If applicable, define any geographic restrictions.
• Relevance to Research Questions:  Ensure that selected studies align with your research questions and objectives.

Your inclusion and exclusion criteria should strike a balance between inclusivity and relevance. Rigorous criteria help maintain the quality and applicability of the studies included in your meta-analysis.

Study Selection Process

The study selection process involves systematically screening and evaluating each potential study to determine whether it meets your predefined inclusion criteria. Here's a step-by-step guide:

• Screen Titles and Abstracts:  Begin by reviewing the titles and abstracts of the retrieved studies. Exclude studies that clearly do not meet your inclusion criteria.
• Full-Text Assessment:  Assess the full text of potentially relevant studies to confirm their eligibility. Pay attention to study design, sample size, and other specific criteria.
• Data Extraction:  For studies that meet your criteria, extract the necessary data, including study characteristics, effect sizes, and other relevant information.
• Record Exclusions:  Keep a record of the reasons for excluding studies. This transparency is crucial for the reproducibility of your meta-analysis.
• Resolve Discrepancies:  If multiple reviewers are involved, resolve any disagreements through discussion or a third-party arbitrator.

Maintaining a clear and organized record of your study selection process is essential for transparency and reproducibility. Software tools like EndNote or Covidence can facilitate the screening and data extraction process.

By following these systematic steps in conducting a literature review, you ensure that your meta-analysis is built on a solid foundation of relevant and high-quality studies.

Data Extraction and Management

As you progress in your meta-analysis journey, the data extraction and management phase becomes paramount. We will delve deeper into the critical aspects of this phase, including the data collection process, data coding and transformation, and how to handle missing data effectively.

Data Collection Process

The data collection process is the heart of your meta-analysis, where you systematically extract essential information from each selected study. To ensure accuracy and consistency:

• Create a Data Extraction Form:  Develop a standardized data extraction form that includes all the necessary fields for collecting relevant data. This form should align with your research questions and inclusion criteria.
• Data Extractors:  Assign one or more reviewers to extract data from the selected studies. Ensure they are familiar with the form and the specific data points to collect.
• Double-Check Accuracy:  Implement a verification process where a second reviewer cross-checks a random sample of data extractions to identify discrepancies or errors.
• Extract All Relevant Information:  Collect data on study characteristics, participant demographics, outcome measures, effect sizes, confidence intervals, and any additional information required for your analysis.
• Maintain Consistency:  Use clear guidelines and definitions for data extraction to ensure uniformity across studies.

To optimize your data collection process and streamline the extraction and management of crucial information, consider leveraging innovative solutions like Appinio . With Appinio, you can effortlessly collect real-time consumer insights, ensuring your meta-analysis benefits from the latest data trends and user perspectives.   Ready to learn more? Book a demo today and unlock a world of data-driven possibilities!

Get a free demo and see the Appinio platform in action!

Data Coding and Transformation

After data collection, you may need to code and transform the extracted data to ensure uniformity and compatibility across studies. This process involves:

• Coding Categorical Variables:  If studies report data differently, code categorical variables consistently . For example, ensure that categories like "male" and "female" are coded consistently across studies.
• Standardizing Units of Measurement:  Convert all measurements to a common unit if studies use different measurement units. For instance, if one study reports height in inches and another in centimeters, standardize to one unit for comparability.
• Calculating Effect Sizes:  Calculate effect sizes and their standard errors or variances if they are not directly reported in the studies. Common effect size measures include Cohen's d, odds ratio (OR), and hazard ratio (HR).
• Data Transformation:  Transform data if necessary to meet assumptions of statistical tests. Common transformations include log transformation for skewed data or arcsine transformation for proportions.
• Heterogeneity Adjustment:  Consider using transformation methods to address heterogeneity among studies, such as applying the Freeman-Tukey double arcsine transformation for proportions.

The goal of data coding and transformation is to make sure that data from different studies are compatible and can be effectively synthesized during the analysis phase. Spreadsheet software like Excel or statistical software like R can be used for these tasks.

Handling Missing Data

Missing data is a common challenge in meta-analysis, and how you handle it can impact the validity and precision of your results. Strategies for handling missing data include:

• Contact Authors:  If feasible, contact the authors of the original studies to request missing data or clarifications.
• Imputation:  Consider using appropriate imputation methods to estimate missing values, but exercise caution and report the imputation methods used.
• Sensitivity Analysis:  Conduct sensitivity analyses to assess the impact of missing data on your results by comparing the main analysis to alternative scenarios.

Remember that transparency in reporting how you handled missing data is crucial for the credibility of your meta-analysis.

By following these steps in data extraction and management, you will ensure the integrity and reliability of your meta-analysis dataset.

Meta-Analysis Example

Meta-analysis is a versatile research method that can be applied to various fields and disciplines, providing valuable insights by synthesizing existing evidence.

Example 1: Analyzing the Impact of Advertising Campaigns on Sales

Background:  A market research agency is tasked with assessing the effectiveness of advertising campaigns on sales outcomes for a range of consumer products. They have access to multiple studies and reports conducted by different companies, each analyzing the impact of advertising on sales revenue.

Meta-Analysis Approach:

• Study Selection:  Identify relevant studies that meet specific inclusion criteria, such as the type of advertising campaign (e.g., TV commercials, social media ads), the products examined, and the sales metrics assessed.
• Data Extraction:  Collect data from each study, including details about the advertising campaign (e.g., budget, duration), sales data (e.g., revenue, units sold), and any reported effect sizes or correlations.
• Effect Size Calculation:  Calculate effect sizes (e.g., correlation coefficients) based on the data provided in each study, quantifying the strength and direction of the relationship between advertising and sales.
• Data Synthesis:  Employ meta-analysis techniques to combine the effect sizes from the selected studies. Compute a summary effect size and its confidence interval to estimate the overall impact of advertising on sales.
• Publication Bias Assessment:  Use funnel plots and statistical tests to assess the potential presence of publication bias, ensuring that the meta-analysis results are not unduly influenced by selective reporting.

Findings:  Through meta-analysis, the market research agency discovers that advertising campaigns have a statistically significant and positive impact on sales across various product categories. The findings provide evidence for the effectiveness of advertising efforts and assist companies in making data-driven decisions regarding their marketing strategies.

These examples illustrate how meta-analysis can be applied in diverse domains, from tech startups seeking to optimize user engagement to market research agencies evaluating the impact of advertising campaigns. By systematically synthesizing existing evidence, meta-analysis empowers decision-makers with valuable insights for informed choices and evidence-based strategies.

How to Assess Study Quality and Bias?

Ensuring the quality and reliability of the studies included in your meta-analysis is essential for drawing accurate conclusions. We'll show you how you can assess study quality using specific tools, evaluate potential bias, and address publication bias.

Quality Assessment Tools

Quality assessment tools provide structured frameworks for evaluating the methodological rigor of each included study. The choice of tool depends on the study design. Here are some commonly used quality assessment tools:

For Randomized Controlled Trials (RCTs):

• Cochrane Risk of Bias Tool:  This tool assesses the risk of bias in RCTs based on six domains: random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, and selective reporting.
• Jadad Scale:  A simpler tool specifically for RCTs, the Jadad Scale focuses on randomization, blinding, and the handling of withdrawals and dropouts.

For Observational Studies:

• Newcastle-Ottawa Scale (NOS):  The NOS assesses the quality of cohort and case-control studies based on three categories: selection, comparability, and outcome.
• ROBINS-I:  Designed for non-randomized studies of interventions, the Risk of Bias in Non-randomized Studies of Interventions tool evaluates bias in domains such as confounding, selection bias, and measurement bias.
• MINORS:  The Methodological Index for Non-Randomized Studies (MINORS) assesses non-comparative studies and includes items related to study design, reporting, and statistical analysis.

Bias Assessment

Evaluating potential sources of bias is crucial to understanding the limitations of the included studies. Common sources of bias include:

• Selection Bias:  Occurs when the selection of participants is not random or representative of the target population.
• Performance Bias:  Arises when participants or researchers are aware of the treatment or intervention status, potentially influencing outcomes.
• Detection Bias:  Occurs when outcome assessors are not blinded to the treatment groups.
• Attrition Bias:  Results from incomplete data or differential loss to follow-up between treatment groups.
• Reporting Bias:  Involves selective reporting of outcomes, where only positive or statistically significant results are published.

To assess bias, reviewers often use the quality assessment tools mentioned earlier, which include domains related to bias, or they may specifically address bias concerns in the narrative synthesis.

We'll move on to the core of meta-analysis: data synthesis. We'll explore different effect size measures, fixed-effect versus random-effects models, and techniques for assessing and addressing heterogeneity among studies.

Data Synthesis

Now that you've gathered data from multiple studies and assessed their quality, it's time to synthesize this information effectively.

Effect Size Measures

Effect size measures quantify the magnitude of the relationship or difference you're investigating in your meta-analysis. The choice of effect size measure depends on your research question and the type of data provided by the included studies. Here are some commonly used effect size measures:

Continuous Outcome Data:

• Cohen's d:  Measures the standardized mean difference between two groups. It's suitable for continuous outcome variables.
• Hedges' g:  Similar to Cohen's d but incorporates a correction factor for small sample sizes.

Binary Outcome Data:

• Odds Ratio (OR):  Used for dichotomous outcomes, such as success/failure or presence/absence.
• Risk Ratio (RR):  Similar to OR but used when the outcome is relatively common.

Time-to-Event Data:

• Hazard Ratio (HR):  Used in survival analysis to assess the risk of an event occurring over time.
• Risk Difference (RD):  Measures the absolute difference in event rates between two groups.

Selecting the appropriate effect size measure depends on the nature of your data and the research question. When effect sizes are not directly reported in the studies, you may need to calculate them using available data, such as means, standard deviations, and sample sizes.

Formula for Cohen's d:

d = (Mean of Group A - Mean of Group B) / Pooled Standard Deviation

Fixed-Effect vs. Random-Effects Models

In meta-analysis, you can choose between fixed-effect and random-effects models to combine the results of individual studies:

Fixed-Effect Model:

• Assumes that all included studies share a common true effect size.
• Accounts for only within-study variability (sampling error).
• Appropriate when studies are very similar or when there's minimal heterogeneity.

Random-Effects Model:

• Acknowledges that there may be variability in effect sizes across studies.
• Accounts for both within-study variability (sampling error) and between-study variability (real differences between studies).
• More conservative and applicable when there's substantial heterogeneity.

The choice between these models should be guided by the degree of heterogeneity observed among the included studies. If heterogeneity is significant, the random-effects model is often preferred, as it provides a more robust estimate of the overall effect.

Forest Plots

Forest plots are graphical representations commonly used in meta-analysis to display the results of individual studies along with the combined summary estimate. Key components of a forest plot include:

• Vertical Line:  Represents the null effect (e.g., no difference or no effect).
• Horizontal Lines:  Represent the confidence intervals for each study's effect size estimate.
• Diamond or Square:  Represents the summary effect size estimate, with its width indicating the confidence interval around the summary estimate.
• Study Names:  Listed on the left side of the plot, identifying each study.

Forest plots help visualize the distribution of effect sizes across studies and provide insights into the consistency and direction of the findings.

Heterogeneity Assessment

Heterogeneity refers to the variability in effect sizes among the included studies. It's important to assess and understand heterogeneity as it can impact the interpretation of your meta-analysis results. Standard methods for assessing heterogeneity include:

• Cochran's Q Test:  A statistical test that assesses whether there is significant heterogeneity among the effect sizes of the included studies.
• I² Statistic:  A measure that quantifies the proportion of total variation in effect sizes that is due to heterogeneity. I² values range from 0% to 100%, with higher values indicating greater heterogeneity.

Assessing heterogeneity is crucial because it informs your choice of meta-analysis model (fixed-effect vs. random-effects) and whether subgroup analyses or sensitivity analyses are warranted to explore potential sources of heterogeneity.

How to Interpret Meta-Analysis Results?

With the data synthesis complete, it's time to make sense of the results of your meta-analysis.

Meta-Analytic Summary

The meta-analytic summary is the culmination of your efforts in data synthesis. It provides a consolidated estimate of the effect size and its confidence interval, combining the results of all included studies. To interpret the meta-analytic summary effectively:

• Effect Size Estimate:  Understand the primary effect size estimate, such as Cohen's d, odds ratio, or hazard ratio, and its associated confidence interval.
• Significance:  Determine whether the summary effect size is statistically significant. This is indicated when the confidence interval does not include the null value (e.g., 0 for Cohen's d or 1 for odds ratio).
• Magnitude:  Assess the magnitude of the effect size. Is it large, moderate, or small, and what are the practical implications of this magnitude?
• Direction:  Consider the direction of the effect. Is it in the hypothesized direction, or does it contradict the expected outcome?
• Clinical or Practical Significance:  Reflect on the clinical or practical significance of the findings. Does the effect size have real-world implications?
• Consistency:  Evaluate the consistency of the findings across studies. Are most studies in agreement with the summary effect size estimate, or are there outliers?

Subgroup Analyses

Subgroup analyses allow you to explore whether the effect size varies across different subgroups of studies or participants. This can help identify potential sources of heterogeneity or assess whether the intervention's effect differs based on specific characteristics. Steps for conducting subgroup analyses:

• Define Subgroups:  Clearly define the subgroups you want to investigate based on relevant study characteristics (e.g., age groups, study design , intervention type).
• Analyze Subgroups:  Calculate separate summary effect sizes for each subgroup and compare them to the overall summary effect.
• Assess Heterogeneity:  Evaluate whether subgroup differences are statistically significant. If so, this suggests that the effect size varies significantly among subgroups.
• Interpretation:  Interpret the subgroup findings in the context of your research question. Are there meaningful differences in the effect across subgroups? What might explain these differences?

Subgroup analyses can provide valuable insights into the factors influencing the overall effect size and help tailor recommendations for specific populations or conditions.

Sensitivity Analyses

Sensitivity analyses are conducted to assess the robustness of your meta-analysis results by exploring how different choices or assumptions might affect the findings. Common sensitivity analyses include:

• Exclusion of Low-Quality Studies:  Repeating the meta-analysis after excluding studies with low quality or a high risk of bias.
• Changing Effect Size Measure:  Re-running the analysis using a different effect size measure to assess whether the choice of measure significantly impacts the results.
• Publication Bias Adjustment:  Applying methods like the trim-and-fill procedure to adjust for potential publication bias.
• Subsample Analysis:  Analyzing a subset of studies based on specific criteria or characteristics to investigate their impact on the summary effect.

Sensitivity analyses help assess the robustness and reliability of your meta-analysis results, providing a more comprehensive understanding of the potential influence of various factors.

Reporting and Publication

The final stages of your meta-analysis involve preparing your findings for publication.

Manuscript Preparation

When preparing your meta-analysis manuscript, consider the following:

• Structured Format:  Organize your manuscript following a structured format, including sections such as introduction, methods, results, discussion, and conclusions.
• Clarity and Conciseness:  Write your findings clearly and concisely, avoiding jargon or overly technical language. Use tables and figures to enhance clarity.
• Transparent Methods:  Provide detailed descriptions of your methods, including eligibility criteria, search strategy, data extraction, and statistical analysis.
• Incorporate Tables and Figures:  Present your meta-analysis results using tables and forest plots to visually convey key findings.
• Interpretation:  Interpret the implications of your findings, discussing the clinical or practical significance and limitations.

Transparent Reporting Guidelines

Adhering to transparent reporting guidelines ensures that your meta-analysis is transparent, reproducible, and credible. Some widely recognized guidelines include:

• PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses):  PRISMA provides a checklist and flow diagram for reporting systematic reviews and meta-analyses, enhancing transparency and rigor.
• MOOSE (Meta-analysis of Observational Studies in Epidemiology):  MOOSE guidelines are designed for meta-analyses of observational studies and provide a framework for transparent reporting.
• ROBINS-I:  If your meta-analysis involves non-randomized studies, follow the Risk Of Bias In Non-randomized Studies of Interventions guidelines for reporting.

Adhering to these guidelines ensures that your meta-analysis is transparent, reproducible, and credible. It enhances the quality of your research and aids readers and reviewers in assessing the rigor of your study.

PRISMA Statement

The PRISMA statement is a valuable resource for conducting and reporting systematic reviews and meta-analyses. Key elements of PRISMA include:

• Title:  Clearly indicate that your paper is a systematic review or meta-analysis.
• Structured Abstract:  Provide a structured summary of your study, including objectives, methods, results, and conclusions.
• Transparent Reporting:  Follow the PRISMA checklist, which covers items such as the rationale, eligibility criteria, search strategy, data extraction, and risk of bias assessment.
• Flow Diagram:  Include a flow diagram illustrating the study selection process.

By adhering to the PRISMA statement, you enhance the transparency and credibility of your meta-analysis, facilitating its acceptance for publication and aiding readers in evaluating the quality of your research.

Conclusion for Meta-Analysis

Meta-analysis is a powerful tool that allows you to combine and analyze data from multiple studies to find meaningful patterns and make informed decisions. It helps you see the bigger picture and draw more accurate conclusions than individual studies alone. Whether you're in healthcare, education, business, or any other field, the principles of meta-analysis can be applied to enhance your research and decision-making processes. Remember that conducting a successful meta-analysis requires careful planning, attention to detail, and transparency in reporting. By following the steps outlined in this guide, you can embark on your own meta-analysis journey with confidence, contributing to the advancement of knowledge and evidence-based practices in your area of interest.

How to Elevate Your Meta-Analysis With Real-Time Insights?

Introducing Appinio , the real-time market research platform that brings a new level of excitement to your meta-analysis journey. With Appinio, you can seamlessly collect your own market research data in minutes, empowering your meta-analysis with fresh, real-time consumer insights.

Here's why Appinio is your ideal partner for efficient data collection:

• From Questions to Insights in Minutes:  Appinio's lightning-fast platform ensures you get the answers you need when you need them, accelerating your meta-analysis process.
• No Research PhD Required:  Our intuitive platform is designed for everyone, eliminating the need for specialized research skills and putting the power of data collection in your hands.
• Global Reach, Minimal Time:  With an average field time of less than 23 minutes for 1,000 respondents and access to over 90 countries, you can define precise target groups and gather data swiftly.

Get free access to the platform!

Join the loop 💌

Be the first to hear about new updates, product news, and data insights. We'll send it all straight to your inbox.

Get the latest market research news straight to your inbox! 💌

Wait, there's more

05.04.2024 | 27min read

What is Field Research? Definition, Types, Methods, Examples

03.04.2024 | 29min read

What is Cluster Sampling? Definition, Methods, Examples

01.04.2024 | 26min read

Cross-Tabulation Analysis: A Full Guide (+ Examples)

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Family Med Prim Care
• v.2(1); Jan-Mar 2013

Systematic Reviews and Meta-analysis: Understanding the Best Evidence in Primary Healthcare

S. gopalakrishnan.

Department of Community Medicine, SRM Medical College, Hospital and Research Centre, Kattankulathur, Tamil Nadu, India

P. Ganeshkumar

Healthcare decisions for individual patients and for public health policies should be informed by the best available research evidence. The practice of evidence-based medicine is the integration of individual clinical expertise with the best available external clinical evidence from systematic research and patient's values and expectations. Primary care physicians need evidence for both clinical practice and for public health decision making. The evidence comes from good reviews which is a state-of-the-art synthesis of current evidence on a given research question. Given the explosion of medical literature, and the fact that time is always scarce, review articles play a vital role in decision making in evidence-based medical practice. Given that most clinicians and public health professionals do not have the time to track down all the original articles, critically read them, and obtain the evidence they need for their questions, systematic reviews and clinical practice guidelines may be their best source of evidence. Systematic reviews aim to identify, evaluate, and summarize the findings of all relevant individual studies over a health-related issue, thereby making the available evidence more accessible to decision makers. The objective of this article is to introduce the primary care physicians about the concept of systematic reviews and meta-analysis, outlining why they are important, describing their methods and terminologies used, and thereby helping them with the skills to recognize and understand a reliable review which will be helpful for their day-to-day clinical practice and research activities.

Introduction

Evidence-based healthcare is the integration of best research evidence with clinical expertise and patient values. Green denotes, “Using evidence from reliable research, to inform healthcare decisions, has the potential to ensure best practice and reduce variations in healthcare delivery.” However, incorporating research into practice is time consuming, and so we need methods of facilitating easy access to evidence for busy clinicians.[ 1 ] Ganeshkumar et al . mentioned that nearly half of the private practitioners in India were consulting more than 4 h per day in a locality,[ 2 ] which explains the difficulty of them in spending time in searching evidence during consultation. Ideally, clinical decision making ought to be based on the latest evidence available. However, to keep abreast with the continuously increasing number of publications in health research, a primary healthcare professional would need to read an insurmountable number of articles every day, covered in more than 13 million references and over 4800 biomedical and health journals in Medline alone. With the view to address this challenge, the systematic review method was developed. Systematic reviews aim to inform and facilitate this process through research synthesis of multiple studies, enabling increased and efficient access to evidence.[ 1 , 3 , 4 ]

Systematic reviews and meta-analyses have become increasingly important in healthcare settings. Clinicians read them to keep up-to-date with their field and they are often used as a starting point for developing clinical practice guidelines. Granting agencies may require a systematic review to ensure there is justification for further research and some healthcare journals are moving in this direction.[ 5 ]

This article is intended to provide an easy guide to understand the concept of systematic reviews and meta-analysis, which has been prepared with the aim of capacity building for general practitioners and other primary healthcare professionals in research methodology and day-to-day clinical practice.

The purpose of this article is to introduce readers to:

• The two approaches of evaluating all the available evidence on an issue i.e., systematic reviews and meta-analysis,
• Discuss the steps in doing a systematic review,
• Introduce the terms used in systematic reviews and meta-analysis,
• Interpret results of a meta-analysis, and
• The advantages and disadvantages of systematic review and meta-analysis.

Application

What is the effect of antiviral treatment in dengue fever? Most often a primary care physician needs to know convincing answers to questions like this in a primary care setting.

To find out the solutions or answers to a clinical question like this, one has to refer textbooks, ask a colleague, or search electronic database for reports of clinical trials. Doctors need reliable information on such problems and on the effectiveness of large number of therapeutic interventions, but the information sources are too many, i.e., nearly 20,000 journals publishing 2 million articles per year with unclear or confusing results. Because no study, regardless of its type, should be interpreted in isolation, a systematic review is generally the best form of evidence.[ 6 ] So, the preferred method is a good summary of research reports, i.e., systematic reviews and meta-analysis, which will give evidence-based answers to clinical situations.

There are two fundamental categories of research: Primary research and secondary research. Primary research is collecting data directly from patients or population, while secondary research is the analysis of data already collected through primary research. A review is an article that summarizes a number of primary studies and may draw conclusions on the topic of interest which can be traditional (unsystematic) or systematic.

Terminologies

Systematic review.

A systematic review is a summary of the medical literature that uses explicit and reproducible methods to systematically search, critically appraise, and synthesize on a specific issue. It synthesizes the results of multiple primary studies related to each other by using strategies that reduce biases and random errors.[ 7 ] To this end, systematic reviews may or may not include a statistical synthesis called meta-analysis, depending on whether the studies are similar enough so that combining their results is meaningful.[ 8 ] Systematic reviews are often called overviews.

The evidence-based practitioner, David Sackett, defines the following terminologies.[ 3 ]

• Review: The general term for all attempts to synthesize the results and conclusions of two or more publications on a given topic.
• Overview: When a review strives to comprehensively identify and track down all the literature on a given topic (also called “systematic literature review”).
• Meta-analysis: A specific statistical strategy for assembling the results of several studies into a single estimate.

Systematic reviews adhere to a strict scientific design based on explicit, pre-specified, and reproducible methods. Because of this, when carried out well, they provide reliable estimates about the effects of interventions so that conclusions are defensible. Systematic reviews can also demonstrate where knowledge is lacking. This can then be used to guide future research. Systematic reviews are usually carried out in the areas of clinical tests (diagnostic, screening, and prognostic), public health interventions, adverse (harm) effects, economic (cost) evaluations, and how and why interventions work.[ 9 ]

Cochrane reviews

Cochrane reviews are systematic reviews undertaken by members of the Cochrane Collaboration which is an international not-for-profit organization that aims to help people to make well-informed decisions about healthcare by preparing, maintaining, and promoting the accessibility of systematic reviews of the effects of healthcare interventions.

Cochrane Primary Health Care Field is a systematic review of primary healthcare research on prevention, treatment, rehabilitation, and diagnostic test accuracy. The overall aim and mission of the Primary Health Care Field is to promote the quality, quantity, dissemination, accessibility, applicability, and impact of Cochrane systematic reviews relevant to people who work in primary care and to ensure proper representation in the interests of primary care clinicians and consumers in Cochrane reviews and review groups, and in other entities. This field would serve to coordinate and promote the mission of the Cochrane Collaboration within the primary healthcare disciplines, as well as ensuring that primary care perspectives are adequately represented within the Collaboration.[ 10 ]

Meta-analysis

A meta-analysis is the combination of data from several independent primary studies that address the same question to produce a single estimate like the effect of treatment or risk factor. It is the statistical analysis of a large collection of analysis and results from individual studies for the purpose of integrating the findings.[ 11 ] The term meta-analysis has been used to denote the full range of quantitative methods for research reviews.[ 12 ] Meta-analyses are studies of studies.[ 13 ] Meta-analysis provides a logical framework to a research review where similar measures from comparable studies are listed systematically and the available effect measures are combined wherever possible.[ 14 ]

The fundamental rationale of meta-analysis is that it reduces the quantity of data by summarizing data from multiple resources and helps to plan research as well as to frame guidelines. It also helps to make efficient use of existing data, ensuring generalizability, helping to check consistency of relationships, explaining data inconsistency, and quantifies the data. It helps to improve the precision in estimating the risk by using explicit methods.

Therefore, “systematic review” will refer to the entire process of collecting, reviewing, and presenting all available evidence, while the term “meta-analysis” will refer to the statistical technique involved in extracting and combining data to produce a summary result.[ 15 ]

Steps in doing systematic reviews/meta-analysis

Following are the six fundamental essential steps while doing systematic review and meta-analysis.[ 16 ]

Define the question

This is the most important part of systematic reviews/meta-analysis. The research question for the systematic reviews may be related to a major public health problem or a controversial clinical situation which requires acceptable intervention as a possible solution to the present healthcare need of the community. This step is most important since the remaining steps will be based on this.

Reviewing the literature

This can be done by going through scientific resources such as electronic database, controlled clinical trials registers, other biomedical databases, non-English literatures, “gray literatures” (thesis, internal reports, non–peer-reviewed journals, pharmaceutical industry files), references listed in primary sources, raw data from published trials and other unpublished sources known to experts in the field. Among the available electronic scientific database, the popular ones are PUBMED, MEDLINE, and EMBASE.

Sift the studies to select relevant ones

To select the relevant studies from the searches, we need to sift through the studies thus identified. The first sift is pre-screening, i.e., to decide which studies to retrieve in full, and the second sift is selection which is to look again at these studies and decide which are to be included in the review. The next step is selecting the eligible studies based on similar study designs, year of publication, language, choice among multiple articles, sample size or follow-up issues, similarity of exposure, and or treatment and completeness of information.

It is necessary to ensure that the sifting includes all relevant studies like the unpublished studies (desk drawer problem), studies which came with negative conclusions or were published in non-English journals, and studies with small sample size.

Assess the quality of studies

The steps undertaken in evaluating the study quality are early definition of study quality and criteria, setting up a good scoring system, developing a standard form for assessment, calculating quality for each study, and finally using this for sensitivity analysis.

For example, the quality of a randomized controlled trial can be assessed by finding out the answers to the following questions:

• Was the assignment to the treatment groups really random?
• Was the treatment allocation concealed?
• Were the groups similar at baseline in terms of prognostic factors?
• Were the eligibility criteria specified?
• Were the assessors, the care provider, and the patient blinded?
• Were the point estimates and measure of variability presented for the primary outcome measure?
• Did the analyses include intention-to-treat analysis?

Calculate the outcome measures of each study and combine them

We need a standard measure of outcome which can be applied to each study on the basis of its effect size. Based on their type of outcome, following are the measures of outcome: Studies with binary outcomes (cured/not cured) have odds ratio, risk ratio; studies with continuous outcomes (blood pressure) have means, difference in means, standardized difference in means (effect sizes); and survival or time-to-event data have hazard ratios.

Combining studies

Homogeneity of different studies can be estimated at a glance from a forest plot (explained below). For example, if the lower confidence interval of every trial is below the upper of all the others, i.e., the lines all overlap to some extent, then the trials are homogeneous. If some lines do not overlap at all, these trials may be said to be heterogeneous.

The definitive test for assessing the heterogeneity of studies is a variant of Chi-square test (Mantel–Haenszel test). The final step is calculating the common estimate and its confidence interval with the original data or with the summary statistics from all the studies. The best estimate of treatment effect can be derived from the weighted summary statistics of all studies which will be based on weighting to sample size, standard errors, and other summary statistics. Log scale is used to combine the data to estimate the weighting.

Interpret results: Graph

The results of a meta-analysis are usually presented as a graph called forest plot because the typical forest plots appear as forest of lines. It provides a simple visual presentation of individual studies that went into the meta-analysis at a glance. It shows the variation between the studies and an estimate of the overall result of all the studies together.

Forest plot

Meta-analysis graphs can principally be divided into six columns [ Figure 1 ]. Individual study results are displayed in rows. The first column (“study”) lists the individual study IDs included in the meta-analysis; usually the first author and year are displayed. The second column relates to the intervention groups and the third column to the control groups. The fourth column visually displays the study results. The line in the middle is called “the line of no effect.” The weight (in %) in the fifth column indicates the weighting or influence of the study on the overall results of the meta-analysis of all included studies. The higher the percentage weight, the bigger the box, the more influence the study has on the overall results. The sixth column gives the numerical results for each study (e.g., odds ratio or relative risk and 95% confidence interval), which are identical to the graphical display in the fourth column. The diamond in the last row of the graph illustrates the overall result of the meta-analysis.[ 4 ]

Interpretation of meta-analysis[ 4 ]

Thus, the horizontal lines represent individual studies. Length of line is the confidence interval (usually 95%), squares on the line represent effect size (risk ratio) for the study, with area of the square being the study size (proportional to weight given) and position as point estimate (relative risk) of the study.[ 7 ]

For example, the forest plot of the effectiveness of dexamethasone compared with placebo in preventing the recurrence of acute severe migraine headache in adults is shown in Figure 2 .[ 17 ]

Forest plot of the effectiveness of dexamethasone compared with placebo in preventing the recurrence of acute severe migraine headache in adults[ 17 ]

The overall effect is shown as diamond where the position toward the center represents pooled point estimate, the width represents estimated 95% confidence interval for all studies, and the black plain line vertically in the middle of plot is the “line of no effect” (e.g., relative risk = 1).

Therefore, when examining the results of a systematic reviews/meta-analysis, the following questions should be kept in mind:

• Heterogeneity among studies may make any pooled estimate meaningless.
• The quality of a meta-analysis cannot be any better than the quality of the studies it is summarizing.
• An incomplete search of the literature can bias the findings of a meta-analysis.
• Make sure that the meta-analysis quantifies the size of the effect in units that you can understand.

Subgroup analysis and sensitivity analysis

Subgroup analysis looks at the results of different subgroups of trials, e.g., by considering trials on adults and children separately. This should be planned at the protocol stage itself which is based on good scientific reasoning and is to be kept to a minimum.

Sensitivity analysis is used to determine how results of a systematic review/meta-analysis change by fiddling with data, for example, what is the implication if the exclusion criteria or excluded unpublished studies or weightings are assigned differently. Thus, after the analysis, if changing makes little or no difference to the overall results, the reviewer's conclusions are robust. If the key findings disappear, then the conclusions need to be expressed more cautiously.

Advantages of Systematic Reviews

Systematic reviews have specific advantages because of using explicit methods which limit bias, draw reliable and accurate conclusions, easily deliver required information to healthcare providers, researchers, and policymakers, help to reduce the time delay in the research discoveries to implementation, improve the generalizability and consistency of results, generation of new hypotheses about subgroups of the study population, and overall they increase precision of the results.[ 18 ]

Limitations in Systematic Reviews/Meta-analysis

As with all research, the value of a systematic review depends on what was done, what was found, and the clarity of reporting. As with other publications, the reporting quality of systematic reviews varies, limiting readers’ ability to assess the strengths and weaknesses of those reviews.[ 5 ]

Even though systematic review and meta-analysis are considered the best evidence for getting a definitive answer to a research question, there are certain inherent flaws associated with it, such as the location and selection of studies, heterogeneity, loss of information on important outcomes, inappropriate subgroup analyses, conflict with new experimental data, and duplication of publication.

Publication Bias

Publication bias results in it being easier to find studies with a “positive” result.[ 19 ] This occurs particularly due to inappropriate sifting of the studies where there is always a tendency towards the studies with positive (significant) outcomes. This effect occurs more commonly in systematic reviews/meta-analysis which need to be eliminated.

The quality of reporting of systematic reviews is still not optimal. In a recent review of 300 systematic reviews, few authors reported assessing possible publication bias even though there is overwhelming evidence both for its existence and its impact on the results of systematic reviews. Even when the possibility of publication bias is assessed, there is no guarantee that systematic reviewers have assessed or interpreted it appropriately.[ 20 ]

To overcome certain limitations mentioned above, the Cochrane reviews are currently reported in a format where at the end of every review, findings are summarized in the author's point of view and also give an overall picture of the outcome by means of plain language summary. This is found to be much helpful to understand the existing evidence about the topic more easily by the reader.

A systematic review is an overview of primary studies which contains an explicit statement of objectives, materials, and methods, and has been conducted according to explicit and reproducible methodology. A meta-analysis is a mathematical synthesis of the results of two or more primary studies that addressed the same hypothesis in the same way. Although meta-analysis can increase the precision of a result, it is important to ensure that the methods used for the reviews were valid and reliable.

High-quality systematic reviews and meta-analyses take great care to find all relevant studies, critically assess each study, synthesize the findings from individual studies in an unbiased manner, and present balanced important summary of findings with due consideration of any flaws in the evidence. Systematic review and meta-analysis is a way of summarizing research evidence, which is generally the best form of evidence, and hence positioned at the top of the hierarchy of evidence.

Systematic reviews can be very useful decision-making tools for primary care/family physicians. They objectively summarize large amounts of information, identifying gaps in medical research, and identifying beneficial or harmful interventions which will be useful for clinicians, researchers, and even for public and policymakers.

Source of Support: Nil

Conflict of Interest: None declared.

• Manuscript Review

Systematic Review VS Meta-Analysis

• 3 minute read
• 55.9K views

Table of Contents

How you organize your research is incredibly important; whether you’re preparing a report, research review, thesis or an article to be published. What methodology you choose can make or break your work getting out into the world, so let’s take a look at two main types: systematic review and meta-analysis.

Let’s start with what they have in common – essentially, they are both based on high-quality filtered evidence related to a specific research topic. They’re both highly regarded as generally resulting in reliable findings, though there are differences, which we’ll discuss below. Additionally, they both support conclusions based on expert reviews, case-controlled studies, data analysis, etc., versus mere opinions and musings.

What is a Systematic Review?

A systematic review is a form of research done collecting, appraising and synthesizing evidence to answer a particular question, in a very transparent and systematic way. Data (or evidence) used in systematic reviews have their origin in scholarly literature – published or unpublished. So, findings are typically very reliable. In addition, they are normally collated and appraised by an independent panel of experts in the field. Unlike traditional reviews, systematic reviews are very comprehensive and don’t rely on a single author’s point of view, thus avoiding bias.

Systematic reviews are especially important in the medical field, where health practitioners need to be constantly up-to-date with new, high-quality information to lead their daily decisions. Since systematic reviews, by definition, collect information from previous research, the pitfalls of new primary studies is avoided. They often, in fact, identify lack of evidence or knowledge limitations, and consequently recommend further study, if needed.

Why are systematic reviews important?

• They combine and synthesize various studies and their findings.
• Systematic reviews appraise the validity of the results and findings of the collected studies in an impartial way.
• They define clear objectives and reproducible methodologies.

What is a Meta-analysis?

This form of research relies on combining statistical results from two or more existing studies. When multiple studies are addressing the same problem or question, it’s to be expected that there will be some potential for error. Most studies account for this within their results. A meta-analysis can help iron out any inconsistencies in data, as long as the studies are similar.

For instance, if your research is about the influence of the Mediterranean diet on diabetic people, between the ages of 30 and 45, but you only find a study about the Mediterranean diet in healthy people and another about the Mediterranean diet in diabetic teenagers. In this case, undertaking a meta-analysis would probably be a poor choice. You can either pursue the idea of comparing such different material, at the risk of findings that don’t really answer the review question. Or, you can decide to explore a different research method (perhaps more qualitative).

Why is meta-analysis important?

• They help improve precision about evidence since many studies are too small to provide convincing data.
• Meta-analyses can settle divergences between conflicting studies. By formally assessing the conflicting study results, it is possible to eventually reach new hypotheses and explore the reasons for controversy.
• They can also answer questions with a broader influence than individual studies. For example, the effect of a disease on several populations across the world, by comparing other modest research studies completed in specific countries or continents.

Undertaking research approaches, like systematic reviews and/or meta-analysis, involve great responsibility. They provide reliable information that has a real impact on society. Elsevier offers a number of services that aim to help researchers achieve excellence in written text, suggesting the necessary amendments to fit them into a targeted format. A perfectly written text, whether translated or edited from a manuscript, is the key to being respected within the scientific community, leading to more and more important positions like, let’s say…being part of an expert panel leading a systematic review or a widely acknowledged meta-analysis.

Check why it’s important to manage research data .

Language Editing Services by Elsevier Author Services:

• Publication Recognition

What is a Good H-index?

• Research Process

What is a Research Gap

You may also like.

Essential for High-Quality Paper Editing: Three Tips to Efficient Spellchecks

If You’re a Researcher, Remember These Before You Are Submitting Your Manuscript to Journals!

Navigating “Chinglish” Errors in Academic English Writing

Is The Use of AI in Manuscript Editing Feasible? Here’s Three Tips to Steer Clear of Potential Issues

A profound editing experience with English-speaking experts: Elsevier Language Services to learn more!

Research Fraud: Falsification and Fabrication in Research Data

Professor Anselmo Paiva: Using Computer Vision to Tackle Medical Issues with a Little Help from Elsevier Author Services

What is the main purpose of proofreading a paper?

Input your search keywords and press Enter.

Quantitative Research Methods

• Introduction
• Descriptive and Inferential Statistics
• Hypothesis Testing
• Regression and Correlation
• Time Series

Meta-Analysis

• Mixed Methods
• Additional Resources
• Get Research Help

A meta-analysis uses statistical methods to synthesize the results of multiple studies, often by calculating a weighted average of effect sizes.  Before embarking on a meta-analysis, make sure you are familiar with reviews, in particular systematic reviews.  

• Meta-analysis in medical research Hippokratia article by A. B. Haidich.
• Meta-Analysis: Recent Developments in Quantitative Methods for Literature Reviews Annual Review of Psychology article by R. Rosenthal and M. R. DiMatteo.
• Analyzing Data for Meta-analysis Chapter from the Cochrane Review Handbook.
• A typology of reviews Health Information & Libraries Journal article by M. J. Grant and A. Booth.

Forest Plots

A forest plot is a type of graph used in meta-analyses that displays the results of multiple studies next to each other.

Books and eBooks

• << Previous: Time Series
• Next: Mixed Methods >>
• Last Updated: Aug 18, 2023 11:55 AM
• URL: https://guides.library.duq.edu/quant-methods

Meta-analysis. A quantitative approach to research integration

Affiliation.

• 1 Center for Environmental Health and Injury Control, Atlanta, GA 30333.
• PMID: 3278147
• DOI: 10.1001/jama.259.11.1685

Meta-analysis is being used with increasing frequency in clinical medicine as an attempt to improve on traditional methods of narrative review by systematically aggregating information and quantifying its impact. Combining data from several studies using meta-analysis can increase statistical power, provide insight into the nature of relationships among variables, and increase generalizability of results more rigorously than less quantitative review methods. Like all review methods, meta-analysis can be limited by sampling bias, inadequate data, and biased outcome interpretation. Still, the advantages noted above make meta-analysis a methodology that warrants testing and empirical evaluation.

Publication types

• Data Collection / methods
• Data Interpretation, Statistical*
• Fetal Monitoring
• Research Design*
• United States

• West Coast University Libraries
• Research Guides
• Find Materials

Research Basics

• WCU Research Resources
• Selecting a Topic
• Evaluating Resources
• Search Tips
• Cite Your Resources
• Annotated Bibliographies
• Intro to Scientific Literature

Qualitative Research

Quantitative research, qualitative and quantitative research video, differences between qualitative and quantitative research, meta-analysis, meta-synthesis, and mixed method reviews, research consultations.

Need help?   Contact a library staff member to  schedule a research consultation . We’ll help you plan your research and ensure that you are using the best information that the library provides.

Have a Question?

Qualitative research contains data about quality and human behavior.  The data is usually gathered through interviews and observation.

Visit the  Qualitative Research Topic Page in Credo Reference for more information.

Quantitative research contains data about quantity or numbers.  The data can be measured and statistically processed.

Visit the  Quantitative Research Topic Page in Credo Reference  for more information.

Table adapted from page 57 of Nursing and Healthcare Research at a Glance .

Meta-analysis is a quantitative method that uses and synthesizes data from multiple individual studies to arrive at one or more conclusions.  Meta-synthesis is another method that analyzes and combines data from multiple qualitative studies.  Mixed method reviews include data from various qualitative and quantitative research studies. 

• << Previous: PICO(T)
• Last Updated: Nov 6, 2023 12:08 PM
• URL: https://guides.westcoastuniversity.edu/research

Terms of Use  |  Privacy Policy  |  Legal Notice  |  Federal Disclosures  |  State Disclosures  |  Title IX |  Accreditation

Student Consumer Information | BPPE Annual Report & Performance Fact Sheets | BPPE Website | Catalog | Careers With Us

West Coast University © 2024 All Rights Reserved