research methods policy analysis

A guide to policy analysis as a research method

Affiliations.

• 1 Department of Public Health, School of Psychology and Public Health, Latrobe University, Bundoora, Victoria, Australia.
• 2 Department of Global, Urban and Social Studies, RMIT University, 124 La Trobe Street, Melbourne, Victoria, Australia.
• 3 Department of Social Sciences, Faculty of Health, Arts and Design, Swinburne University, 24 Wakefield Street, Hawthorn, Victoria, Australia.
• 4 Department of Nutrition Dietetics and Food, Monash University, Level 1, 264 Ferntree Gully Road, Notting Hill, Victoria, Australia.
• PMID: 30101276
• DOI: 10.1093/heapro/day052

Policy analysis provides a way for understanding how and why governments enact certain policies, and their effects. Public health policy research is limited and lacks theoretical underpinnings. This article aims to describe and critique different approaches to policy analysis thus providing direction for undertaking policy analysis in the field of health promotion. Through the use of an illustrative example in nutrition it aims to illustrate the different approaches. Three broad orientations to policy analysis are outlined: (i) Traditional approaches aim to identify the 'best' solution, through undertaking objective analyses of possible solutions. (ii) Mainstream approaches focus on the interaction of policy actors in policymaking. (iii) Interpretive approaches examine the framing and representation of problems and how policies reflect the social construction of 'problems'. Policy analysis may assist understanding of how and why policies to improve nutrition are enacted (or rejected) and may inform practitioners in their advocacy. As such, policy analysis provides researchers with a powerful tool to understand the use of research evidence in policymaking and generate a heightened understanding of the values, interests and political contexts underpinning policy decisions. Such methods may enable more effective advocacy for policies that can lead to improvements in health.

Keywords: interpretive policy analysis; mainstream policy analysis; nutrition; public health; sugar sweetened beverage tax.

© The Author(s) 2018. Published by Oxford University Press. All rights reserved. For permissions, please email: [email protected].

• Health Policy*
• Health Promotion
• Policy Making*
• Public Health / legislation & jurisprudence
• Public Health Administration
• Public Policy
• Research Design

• Help & FAQ

A guide to policy analysis as a research method

• Nutrition Dietetics & Food

Research output : Contribution to journal › Article › Research › peer-review

Policy analysis provides a way for understanding how and why governments enact certain policies, and their effects. Public health policy research is limited and lacks theoretical underpinnings. This article aims to describe and critique different approaches to policy analysis thus providing direction for undertaking policy analysis in the field of health promotion. Through the use of an illustrative example in nutrition it aims to illustrate the different approaches. Three broad orientations to policy analysis are outlined: (i) Traditional approaches aim to identify the 'best' solution, through undertaking objective analyses of possible solutions. (ii) Mainstream approaches focus on the interaction of policy actors in policymaking. (iii) Interpretive approaches examine the framing and representation of problems and how policies reflect the social construction of 'problems'. Policy analysis may assist understanding of how and why policies to improve nutrition are enacted (or rejected) and may inform practitioners in their advocacy. As such, policy analysis provides researchers with a powerful tool to understand the use of research evidence in policymaking and generate a heightened understanding of the values, interests and political contexts underpinning policy decisions. Such methods may enable more effective advocacy for policies that can lead to improvements in health.

• interpretive policy analysis
• mainstream policy analysis
• public health
• sugar sweetened beverage tax

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

• 10.1093/heapro/day052

Other files and links

• Link to publication in Scopus

T1 - A guide to policy analysis as a research method

AU - Browne, Jennifer

AU - Coffey, Brian

AU - Cook, Kay

AU - Meiklejohn, Sarah

AU - Palermo, Claire

PY - 2019/10

Y1 - 2019/10

N2 - Policy analysis provides a way for understanding how and why governments enact certain policies, and their effects. Public health policy research is limited and lacks theoretical underpinnings. This article aims to describe and critique different approaches to policy analysis thus providing direction for undertaking policy analysis in the field of health promotion. Through the use of an illustrative example in nutrition it aims to illustrate the different approaches. Three broad orientations to policy analysis are outlined: (i) Traditional approaches aim to identify the 'best' solution, through undertaking objective analyses of possible solutions. (ii) Mainstream approaches focus on the interaction of policy actors in policymaking. (iii) Interpretive approaches examine the framing and representation of problems and how policies reflect the social construction of 'problems'. Policy analysis may assist understanding of how and why policies to improve nutrition are enacted (or rejected) and may inform practitioners in their advocacy. As such, policy analysis provides researchers with a powerful tool to understand the use of research evidence in policymaking and generate a heightened understanding of the values, interests and political contexts underpinning policy decisions. Such methods may enable more effective advocacy for policies that can lead to improvements in health.

AB - Policy analysis provides a way for understanding how and why governments enact certain policies, and their effects. Public health policy research is limited and lacks theoretical underpinnings. This article aims to describe and critique different approaches to policy analysis thus providing direction for undertaking policy analysis in the field of health promotion. Through the use of an illustrative example in nutrition it aims to illustrate the different approaches. Three broad orientations to policy analysis are outlined: (i) Traditional approaches aim to identify the 'best' solution, through undertaking objective analyses of possible solutions. (ii) Mainstream approaches focus on the interaction of policy actors in policymaking. (iii) Interpretive approaches examine the framing and representation of problems and how policies reflect the social construction of 'problems'. Policy analysis may assist understanding of how and why policies to improve nutrition are enacted (or rejected) and may inform practitioners in their advocacy. As such, policy analysis provides researchers with a powerful tool to understand the use of research evidence in policymaking and generate a heightened understanding of the values, interests and political contexts underpinning policy decisions. Such methods may enable more effective advocacy for policies that can lead to improvements in health.

KW - interpretive policy analysis

KW - mainstream policy analysis

KW - nutrition

KW - public health

KW - sugar sweetened beverage tax

UR - http://www.scopus.com/inward/record.url?scp=85074379871&partnerID=8YFLogxK

U2 - 10.1093/heapro/day052

DO - 10.1093/heapro/day052

M3 - Article

C2 - 30101276

AN - SCOPUS:85074379871

SN - 0957-4824

JO - Health Promotion International

JF - Health Promotion International

• Technical Support
• Find My Rep

You are here

Methods for Policy Research Taking Socially Responsible Action

• Ann Majchrzak - University of Southern California, USA
• M. Lynne Markus - Bentley University, USA
• Description

This book about responsible and evidence-based decision making is written for those interested in improving the decisions that affect people’s lives. It describes how to define policy research questions so that evidence can be applied to them, how to find and synthesize existing evidence, how to generate new evidence if needed, how to make acceptable recommendations that can solve policy problems without negative side effects, and how to describe evidence and recommendations in a manner that changes minds.

Policies are not just the decisions made by a country’s rulers or elected officials; policies are also set by corporate executives, managers of department stores, and project leaders in non-profit organizations pursuing environmental protection. The authors’ suggestion are based on the fundamental belief that evidence-based decision making is superior to decisions based purely on opinion, intuition, and emotion. Because much has happened since 1984 when the first edition was published, this is a substantially different book with a new co-author, new and updated examples, new chapters, and new frameworks for understanding.

See what’s new to this edition by selecting the Features tab on this page. Should you need additional information or have questions regarding the HEOA information provided for this title, including what is new to this edition, please email [email protected] . Please include your name, contact information, and the name of the title for which you would like more information. For information on the HEOA, please go to http://ed.gov/policy/highered/leg/hea08/index.html .

For assistance with your order: Please email us at [email protected] or connect with your SAGE representative.

SAGE 2455 Teller Road Thousand Oaks, CA 91320 www.sagepub.com

Loved the first edition as a graduate student when it came out in the mid-80s; so happy that a new edition was developed so that I can share it with my graduate students.

still under review for consideration.

NEW TO THIS EDITION:

• Each chapter’s phase in the policy research voyage (depicted by artwork with a nautical theme) includes clearly defined activities, deliverables, criteria for successful performance, and workflow diagrams.
• Policy Change Wheel and STORM Context Conditions frameworks make it easier for readers to remember what needs to be done.
• New chapters on synthesizing available evidence (Chapter 3) and reflecting on policy research experiences (Chapter 7) broaden the book’s coverage.
• Updated examples drawn from a variety of contexts, including international and business policy, as well as domestic issues, illustrate applications of evidence-based decision making in the real world.
• Chapter 1, Making a Difference with Policy Research , now reflects an action-orientation toward not just doing policy research, but also toward fostering change and doing policy research responsibly.

KEY FEATURES:

• A how-to orientation encourages readers to consider the evidence systematically and responsibly before making a decision and to communicate evidence and recommendations in a way that facilitates real change.
• Real world examples throughout the text show readers the everyday applications of policy decision making.
• Exercises at the end of each chapter give students an opportunity to apply what they’ve learned.

This is a substantially revised edition of Methods for Policy Research, originally published in 1984. This book reframes policy research as responsible and evidence-based decision making. It describes how to define policy research questions so that evidence can be applied to them, how to find and synthesize existing evidence, how to generate new evidence if needed, how to make acceptable recommendations that can solve policy problems without harmful side effects, how to describe evidence and recommendations in a manner that changes minds. This book is meant to help individuals who want to improve the policy decisions that affect people's lives.

Responsible and evidence-based decision making is needed not just in government and social service agencies. It is also needed in businesses and in nongovernmental organizations such as charities, foundations, and non-profits. In this book, we state our values clearly: We believe that evidence-based decision making is superior to decisions based purely on opinion, intuition, and emotion. We also believe that responsible decision-making requires taking into account the possibility of harmful consequences from policy change, no matter how well intentioned those changes may be.

Each chapter now has clearly defined activities and deliverables, supported by workflow diagrams, along with tracking indicators that policy researchers can use to assess how well they are performing the activities. New frameworks are presented such as the M2 test (meaningfulness and manageability), the Policy Change Wheel, and STORM (Social, Technical, Organizational, Regulatory, and Market) context conditions to make it easier for readers to remember what needs to be done. All examples are updated, they are drawn from a variety of contexts, including international and business policy, as well as domestic policy and social service.

Each chapter was substantially revised to make the activities and outcomes of policy research clear. We've introduced new content, including an entirely new chapter on synthesizing existing evidence. We've exposed the reader to useful websites, to new ways of involving stakeholders in the Case for Change, and to ways of ensuring that recommendations derived from evidence-gathering are meaningful and manageable. A nautical theme, a conversational style, and humor are used throughout to make the reading enjoyable. (Look out for puns!)

Sample Materials & Chapters

For instructors, select a purchasing option.

This title is also available on SAGE Research Methods , the ultimate digital methods library. If your library doesn’t have access, ask your librarian to start a trial .

No internet connection.

All search filters on the page have been cleared., your search has been saved..

• All content
• Dictionaries
• Encyclopedias
• Expert Insights
• Foundations
• How-to Guides
• Journal Articles
• Little Blue Books
• Little Green Books
• Project Planner
• Tools Directory
• Sign in to my profile My Profile

• Sign in Signed in
• My profile My Profile

Conducting Interpretive Policy Analysis

• By: Dvora Yanow
• Publisher: SAGE Publications, Inc.
• Series: Qualitative Research Methods
• Publication year: 2000
• Online pub date: January 01, 2011
• Discipline: Economics
• Methods: Policy evaluation , Policy research , Observational research
• DOI: https:// doi. org/10.4135/9781412983747
• Keywords: community centers , language Show all Show less
• Print ISBN: 9780761908272
• Online ISBN: 9781412983747
• Buy the book icon link

This book presents a much needed guide to interpretative techniques and methods for policy research. The author begins by describing what interpretative approaches are and what they can mean to policy analysis. The author shifts the frame of reference from thinking about values as costs and benefits to thinking about them more as a set of meanings. The book concludes with a chapter on how to move from fieldwork to deskwork to textwork"."

Front Matter

• Series Editors' Introduction
• Acknowledgments
• Underlying Assumptions of An Interpretive Approach: The Importance of Local Knowledge
• Accessing Local Knowledge: Identifying Interpretive Communities and Policy Artifacts
• Symbolic Language
• Symbolic Objects
• Symbolic Acts
• Moving From Fieldwork and Deskwork to Textwork and Beyond

Back Matter

• About the Author

Sign in to access this content

Get a 30 day free trial, more like this, sage recommends.

We found other relevant content for you on other Sage platforms.

Have you created a personal profile? Login or create a profile so that you can save clips, playlists and searches

• Sign in/register

Navigating away from this page will delete your results

Please save your results to "My Self-Assessments" in your profile before navigating away from this page.

Sign in to my profile

Sign up for a free trial and experience all Sage Learning Resources have to offer.

You must have a valid academic email address to sign up.

Get off-campus access

• View or download all content my institution has access to.

Sign up for a free trial and experience all Sage Research Methods has to offer.

• view my profile
• view my lists

• Search Menu
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Archaeology
• Greek and Roman Papyrology
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Agriculture
• History of Education
• History of Emotions
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Acquisition
• Language Variation
• Language Families
• Language Evolution
• Language Reference
• Lexicography
• Linguistic Theories
• Linguistic Typology
• Linguistic Anthropology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Modernism)
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Religion
• Music and Culture
• Music and Media
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Science
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Lifestyle, Home, and Garden
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Oncology
• Medical Toxicology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Clinical Neuroscience
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Ethics
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Neuroscience
• Cognitive Psychology
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Strategy
• Business History
• Business Ethics
• Business and Government
• Business and Technology
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic Systems
• Economic Methodology
• Economic History
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Theory
• Political Behaviour
• Political Economy
• Political Institutions
• Politics and Law
• Public Administration
• Public Policy
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Developmental and Physical Disabilities Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

40 The Unique Methodology of Policy Research

Amitai Etzioni is a university professor and Professor of International Relations at The George Washington University. He served as a Senior Advisor at the Carter White House; taught at Columbia University, Harvard University, and University of California, Berkeley; and served as president of the Society for the Advancement of Socio-Economics (SASE). A study by Richard Posner ranked him among the top 100 American intellectuals. Etzioni is the author of many books, including Security First (2007), Foreign Policy: Thinking Outside the Box (2016), and Avoiding War with China (2017). His most recent book, Happiness is the Wrong Metric: A Liberal Communitarian Response to Populism, was published by Springer in January 2018.

• Published: 02 September 2009
• Cite Icon Cite
• Permissions Icon Permissions

This article provides a unique methodology of policy research, focusing on the various factors that differentiate policy research from basic research. It identifies malleability as a key variable of policy research, and this is defined as the amount of resources that would have to be expended to cause change in a given variable or variables. The scope of analysis/factors of policy research is shown to encompass all the major facets of the social phenomenon it is trying to deal with. Basic research, on the other hand, fragments the world into abstract and analytical slices, which are then studied individually. The last two differentiating factors of policy research and basic research, which are privacy and communication, are studied in the last two sections of the article.

Policy research requires a profoundly different methodology from that on which basic research relies, because policy research is always dedicated to changing the world while basic research seeks to understand it as it is. 1 The notion that if one merely understands the world better, then one will in turn know how to better it, is not supported by the evidence.

Typical policy goals are the reduction of poverty, curbing crime, cutting pollution, or changing some other condition (Mitchell and Mitchell 1969, 393) . Even those policies whose purpose is to maintain the status quo are promoting change—they aim to slow down or even reverse processes of deterioration, for instance that of natural monuments or historical documents. When no change is sought, say, when no one is concerned with changing the face of the moon, then there is no need for policy research in that particular area.

Moreover, although understanding the causes of a phenomenon, which successful basic research allows, is helpful in formulating policy, often a large amount of other information that is structured in a different manner best serves policy makers. 2 Policy researchers draw on a large amount of information that has no particular analytical base or theoretical background (of the kind that basic research provides). 3 In this sense medical science, which deals with changing bodies and minds, is a protypical policy science. It is estimated that about half of the information physicians employ has no basis in biology, chemistry, or any other science; but rather it is based on an accumulation of experience. 4 This knowledge is passed on from one medical cohort to another, as “these are the way things are done” and “they work.”

The same holds true for other policy sciences. For instance, criminologists who inform a local government that studies show that rehabilitation works more effectively in minimum security prisons than in maximum security prisons (a fact that can be explained by sociological theoretical concepts based on basic research) 5 know from long experience that they had better also alert the local authorities that such a reduction in security could potentially lead some inmates to escape and commit crimes in surrounding areas. Without being willing to accept such a “side effect” of the changed security policy, those governments who introduced it may well lose the next election and security in the prison will be returned to its previously high level. There is no particular sociological theoretical reason for escapes to rise when security is lowered. It is an observation based on common sense and experience; however it is hardly an observation that policy makers, let alone policy researchers should ignore. (They may though explore ways of coping with this “side effect,” for instance by either preparing the public ahead of time, introducing an alert system when inmates escape, or some other such measure.)

The examples just given seek to illustrate the difference between the information that basic research generates versus information that plays a major role in policy research. That is, there are important parts of the knowledge on which policy research draws that are based on distilled practice and are not derivable from basic research. Much of what follows deals with major differences in the ways that information and analysis are structured in sound policy research in contrast to the ways basic research is carried out.

One clarification before I can proceed: Policy research should not be confused with applied research. Applied research presumes that a policy decision has already been made and those responsible are now looking for the most efficient ways to implement it. Policy research helps to determine what the policy decision ought to be.

1. Malleability

A major difference between basic and policy research is that malleability is a key variable for the latter though not the former (Weimer and Vining 1989; 4) . Indeed for policy researchers it is arguably the single most important variable. Malleability for the purposes at hand ought to be defined as the amount of resources (including time, energy, and political capital) that would have to be expended to cause change in a given variable or variables. For policy research, malleability is a cardinal consideration because resources always fall short of what is required to implement given policy goals. Hence, to employ resources effectively requires determining the relative results to be generated from different patterns of allocation (Dunn 1981, 334– 402) . In contrast, basic research has no principled reason to favor some factors (or variables) over others. For basic research, it matters little if at all whether a condition under study can be modified and if it can how much it would cost. To illustrate, many sociological studies compare people by gender and age and although these variables may seem relevant, they are of limited value to policy research. Other variables used, such as the levels of income of various populations, the extent of education of various racial and ethnic groups, and the average size of cities, are somewhat more malleable but still not highly so. In contrast, perceptions are much more malleable.

One may say that basic research should reveal a preference for variables that have been less studied; however, such a consideration concerns the economics and politics of science rather than methodology. Because all scientific findings are conditional and temporary and often subject to profound revision and recasting, for basic researchers, retesting old findings can be just as valuable as covering new variables. In short, although in principle for basic research the study of all variables is legitimate, in a given period of time or amongst a given group of scientists, some may consider certain variables as more “interesting” or “promising” than others. In contrast, to reiterate, for policy research, malleability is the most important variable as it is directly related to its core reason for being: Promoting change.

Given the dominance of basic research methodology in the ways policy research is taught, it is not surprising to find that the question of which variables are more malleable than others is rarely studied in any systematic way. Due to the importance of this issue for policy research, some elaboration and illustrations are called for. Economic feasibility is a good case in point. Many policy researchers' final reports do not include any, not even crude estimates of the costs involved in what they are recommending. 6 Even less common is any consideration of the question of whether such changes can be made acceptable to elected representatives and the public at large; that is, political feasibility (Weimer and Vining 1989, 292– 324) . For instance, over the last decades several groups favored advancing their policy goals through constitutional amendments, ignoring the fact that these are extremely difficult to get passed.

In other cases, feasibility is treated as a secondary “applied” question to be studied later, after policy makers adopt the recommended policy. However, the issue runs much deeper than the assessments of feasibility of one kind or another. The challenge to policy research is to determine the relative resistance to change according to the different variables that are to be tackled. And this question must be tackled not on an ad hoc basis, but rather as a major part of systematic policy research. Moreover, if the variables involved are studied from this viewpoint, they themselves may be changed; that is, feasibility is enhanced rather than treated as a given.

Another example of the cardinal need to take malleability into account when conducting policy research concerns changing public attitudes. Policy makers often favor a “public education' campaign when they desire to affect people's beliefs and conduct. Policy makers tend to assume that it is feasible to change such predispositions through a way that might be called the Madison Avenue approach, which entails running a series of commercials (or public service announcements), mounting billboards, obtaining celebrity endorsements, and so on.

For example, the United States engaged in such a campaign in 2003 and 2004 to change the hearts and minds of “the Arab street” through what has also been termed “public diplomacy.” 7 The way this was carried out provides a vivid example of lack of attention to feasibility issues. American public diplomacy, developed by the State Department, included commercials, websites, and speakers programs that sought to “reconnect the world's billion Muslims with the United States the way McDonald's highlights its billion customers served” (Satloff 2003, 18) . It was based on the premiss that “blitzing Arab and Muslim countries with Britney Spears videos and Arabic‐language sitcoms will earn Washington millions of new Muslim sympathizers” (Satloff 2003, 18) . A study found that the results were “disastrous” (Satloff 2003, 18) . Some countries declined to air the messages and many Muslims who did see the material viewed it as blatant propaganda and offensive rather than compelling.

Actually, policy researchers bent on studying feasibility report that the Madison Avenue approach works only when large amounts of money are spent to shift people from one product to another when there are next to no differences between them (e.g. two brands of toothpaste) and when there is an inclination to use the product in the first place. However, when these methods are applied to changing attitudes about matters as different as condom use, 8 the United Nations, 9 electoral reform, and so forth, they are much less successful. Changing people's behavior—say to conserve energy, drive slower, cease smoking—is many hundreds of times more difficult. This is a major reason why totalitarian regimes, despite intensive public education campaigns, usually fail. The question of what is most feasible is determined by fiat by policy makers and their staffs rather than by studies that are reported to the policy makers by policy researchers. Hence decisions are often based on a fly‐by‐the‐seat‐ of‐your‐pants sense of what can be changed rather than on empirical evidence. 10 One of the few exceptions is studies of nation building in which several key policy researchers presented the reasons why such endeavors can be carried out at best only slowly while at the same time many policy makers claimed that it could be achieved in short order and at low cost. 11

In a preliminary stab at outlining the relative malleability of various factors, one may note that as a rule the laws of nature are not malleable; social relations, including patterns of asset distribution and power, are of limited malleability; and symbolic relations are highly malleable. Thus any policy‐making body that would seek to modify the level of gravity, for example, not for a particular situation (for instance a space travel simulator) but in general, will find this task at best extremely difficult to advance. In contrast, those who seek to change a flag, a national motto, the ways people refer to one another (e.g. Ms Instead of girl or broad), have a relatively easy time of doing so. Changes in the distribution of wealth among the classes or races—by public policy—are easier than changes involving the laws of nature, but more difficult than changing hearts and minds.

When policy researchers or policy makers ignore these observations and enact laws that seek grand and quick changes in power relations and economic patterns, the laws are soon reversed. A case in point is the developments that ensued when a policy researcher inserted into legislation the phrase “maximum feasible participation of the poor.” This Act was used to try to circumvent prevailing local power structures by directing federal funds to voluntary groups that included the poor on their advisory boards, which thus helped “empower the poor.” The law was nullified shortly thereafter. Similarly, when a constitutional amendment was enacted that banned the consumption of alcohol in the United States, it had some severely distorted effects on the American justice and law enforcement systems and did little actually to reduce the consumption of alcohol. It was also the only constitutional amendment ever to be repealed.

Among social changes, often legal and political reduction in inequality is relatively easier to come by than are socioeconomic changes along similar lines. Thus, African‐Americans and women gained de jure and de facto voting rights long before the differences in their income and representation in the seats of power moved closer to those of whites (in the case of African‐Americans) and of men (in the case of women). Nor have socioeconomic differences been reduced nearly as much as legal and political differences, although in both realms considerable inequalities remain. The same is true not just for the United States, but for other free societies and those that have been recently liberated.

In short, there are important differences in which dedication of resources, commitment of political capital, and public education are needed in order to bring about change. Sound policy research best makes the determination of which factors are more malleable than others, which is a major subject of study.

2. Scope of Analysis

Another particularly important difference between basic research and policy research methodology concerns the scope of factors that are best encompassed. Policy research at its best encompasses all the major facets of the social phenomenon it is trying to deal with. 12 In contrast, basic research proceeds by fragmenting the world into abstract, analytical slices which are then studied individually.

A wit has suggested that in economics everything has a price; in sociology, nothing has a price. Policy makers and hence researchers are at a disadvantage when they formulate preferred policy alternatives without paying attention to the longer‐run economic and budgetary effects—or the effect of such policy on social relations including families (e.g. tax preferences for singles), socioeconomic classes (e.g. estate taxes), and so on.

To put it in elementary terms, a basic researcher may well study only the prices of flowers (together with other economic factors); a physiologist the wilting processes; a social psychologist the symbolic meaning of flowers; and so forth. But a community that plans to grow flowers in its public gardens must deal with most, if not all of these elements and the relations between them. Flowers that are quick to wilt will not be suitable for its public gardens; the community will be willing to pay more for flowers that have a longer life or those that command a positive symbolic meaning, and so on.

Medicine provides another model of a policy science. It cannot be based only on biology, chemistry, anatomy, or any one science that studies a subset of variables relating to the body. Instead physicians draw on all these sciences and add observations of interaction effects among the variables. This forms a medical knowledge base and drives “policy” recommendations (i.e. medical prescriptions). Indeed doctors have often been chastised when they do not take into account still other variables, such as those studied by psychologists and anthropologists. Similarly, international relations is a policy science that best combines variables studied by economists, political scientists, law professors, and many others.

In short, the scope of variables that basic research encompasses can be quite legitimate and effective but also rather narrow. Policy researchers must be more eclectic and include at least all the variables that account for a significant degree of variance in the phenomenon that the policy aims to change.

3. Private and Confidential

Basic research is a public endeavor. As a rule its results are published so that others can critically assess them and piece them together with their findings and those of still others in order to build ever more encompassing and robust bodies of knowledge. Unpublished work is often not considered when scientists are evaluated for hiring and promoting, for prizes, or for some other reason, especially not if the work is kept secret for commercial or public security reasons. Historically, scientific findings were published in monographs, books, and articles in suitable journals. These served as the main outlets for the findings of basic research both because only by making scientific findings public could they become part of the cumulative scientific knowledge base and also because publication indicates that they have already passed some measure of peer review. It is only through peer review that evidence can be critically scrutinized. In recent years findings are still made public but increasingly they are often posted on websites, most of which lack peer review foundations, which is one reason why they are less trusted and not treated as a full‐fledged publication. Publication is still considered an essential element of basic research.

In contrast, the findings of policy research are often not published—they are provided in private to one policy maker or another (Radin 1997, 204– 18) . The main purpose of policy research is not to contribute to the cumulative process of building knowledge but rather to put to service available knowledge. In that profound sense policy research is often not public but client oriented. 13 Although some policy research is conducted in think tanks and public policy schools that may treat it similarly to basic research, more often than not it is conducted in specialized units in government agencies, the White House, corporate associations, and labor unions. And often tools of policy research are memos and briefings, not publications.

Often the findings of policy researchers are considered confidential or are governed by state secret acts (which is the case in many nations that have a less strong view of civil liberties than does the United States). That is, the findings are merely aimed at a specific client or a group of clients, and sharing them with the public is considered an offense. 14

4. Communication

Basic researchers, as a rule, are much less concerned with communicating, especially with a larger, “secular” public than are policy researchers. This may at first seem a contradiction to the previously made point that science (in the basic research sense) is public while policy research is often “private” (even when conducted for public officials). The seeming contradiction vanishes once one notes that basic researchers are obligated to share their findings with their colleagues , often a small group, and that they seek feedback from this group for both scientific and psychological validation. However, as a rule basic researchers have little interest in the public at large. Indeed, they tend to be highly critical of those who seek to reach such an audience—as did scholars such as Jay Gould and Carl Sagan (Etzioni 2003, 57– 60) .

In contrast, policy researchers often recognize the need to mobilize public support for the policies that their findings favor and hence they tend to help policy makers to mobilize such support by communicating with the public. James Fishkin developed a policy idea he called “deliberative democracy,” which entailed bringing together a group of people who constitute a living sample of the population for a period of time during which they are exposed to public education and presentations by public figures, and they are given a chance to have a dialogue. By measuring the changes in the views of this living sample, Fishkin found that one is able to learn how to change the public's mind. Fishkin did not just develop the concept and publish his ideas, but conducted a long and intensive campaign through radio, TV, newspapers, visits with public leaders, and much more, until his living sample was implemented in several locations (Fishkin 1997) . Indeed, according to Eugene Bardach, policy researchers must prepare themselves for “a long campaign potentially involving many players, including the mass public” (Bardach 2002, 115– 17) .

Hence, basic researchers are more likely to use technical terms (which may sound like jargon to outsiders), mathematical notations, extensive footnotes, and other such scientific features. On the other hand, policy researchers are more likely to express themselves in the vernacular and avoid technical terms.

One can readily show numerous publications of professors at schools of public policy and even think tanks that are rather similar if not indistinguishable from those of basic researchers. 15 But this is the case because these schools conduct mostly basic, and surprisingly little policy research. For example, on 28 April 2004 Google search found only 210 entries for “policy research methodology,” the good part of which referred to university classes by that name. But on closer examination, most entries were referring to basic, not policy research methodology. For instance, a course titled “Cultural Policy Research Methodology” at Griffith University in Australia includes in its course description “basic research techniques, particularly survey methodologies, qualitative methods and a more in depth approach to statistics.” 16 Many other entries were for classes in policy or research methodology (usually basic). The main reasons for this are ( a ) because few places train people in the special methodologies that policy research requires and ( b ) the reward structure is closely tied to basic research. Typically, promotions (especially tenure) at public policy schools are determined by evaluations and votes by senior colleagues from the basic research departments at the same universities or at other ones. Thus the future of an economist at the Harvard Business School may depend on what her colleagues in the Harvard Economics department think of her work. More informally, being invited to become a member of a basic research department is considered a source of prestige and an opportunity to shore up one's training and research. Conversely, only being affiliated with a policy school (like other professional schools) indicates a lack of recognition, which may translate into objective disadvantages. This pecking order, which favors basic over policy (considered “applied”) research, is of considerable psychological importance to researchers in practically all universities. Even in think tanks dedicated to policy research, many respect basic research more than policy research and hope to conduct it one day or regret that they are not suited to carry it out. 17

People who work for think tanks, which are largely dedicated to policy research, often seek to move to universities, in which tenure is more common and there is a greater sense of prestige. Hence many such researchers are keen to keep their “basic” credentials, although often they are unaware of the special methodology that policy research requires or are untutored in carrying it out in the first place because they were trained in basic research modes instead.

At annual meetings of one's discipline, in which findings are presented and evaluated, jobs are negotiated and information about them shared, and prestige scoring is rearranged, policy researchers will typically attend those dominated by their basic research colleagues. And attendance at policy research associations (such as the Association for Public Policy Analysis and Management) is meager. Most prizes and other awards available to researchers go to those who conduct basic research.

In short, although the logic of policy research favors it to be more communicative than basic research, this is often not the case because the training and institutional formations in which policy research is largely conducted favor basic research.

Bardach, E. ( 2002 ). Educating the client: an introduction.   Journal of Policy Analysis and Management , 21 (1): 115–17. 10.1002/pam.1044

Google Scholar

Berelson, B. , and Steiner, G.   1964 . Human Behavior: An Inventory of Scientific Findings . New York: Harcourt Brace and World.

Google Preview

Carothers, T.   1999 . Aiding Democracy Abroad: The Learning Curve. Washington, DC: Carnegie Endowment for International Peace.

DeJong, W. , Wolf, R. C. , and Austin, S. B.   2001 . US federally funded television public service announcements (PSAs) to prevent HIV/AIDS: a content analysis.   Journal of Health Communication , 6: 249–63. 10.1080/108107301752384433

Dunn, W. N.   1981 . Public Policy Analysis: An Introduction . Englewood Cliffs, NJ: Prentice Hall.

Etzioni, A.   1968 . The Active Society: A Theory of Societal and Political Processes . New York: Free Press.

——  1971 a . A Comparative Analysis of Complex Organizations , rev. edn. New York: Free Press.

——  1971 b . Policy research.   American Sociologist , 6 (supplementary issue: June): 8–12.

——  2003 . My Brother's Keeper: A Memoir and a Message . Lanham, Md.: Rowman and Littlefield.

——  2004 . A self‐restrained approach to nation‐building by foreign powers.   International Affairs , 80: 1–17. 10.1111/j.1468-2346.2004.00362.x

Fishkin, J. S.   1997 . The Voice of the People: Public Opinion and Democracy . New Haven, Conn.: Yale University Press.

Free Expression Project   2003 . The Progress of Science and Useful Arts: Why Copyright Today Threatens Intellectual Freedom , 2nd edn. New York: Free Expression Project; available at: www.fepproject.org/policyreports/copyright2dconc.html (accessed 27 Apr. 2004).

Inglefinger, F. J. , Relman, A. S. , and Findland, M.   1966 . Controversy in Internal Medicine . Philadelphia: W. B. Saunder.

Lasswell, H. , and Lerner, D.   1951 . The Policy Sciences . Stanford, Calif.: Stanford University Press.

Miller, D. W. 2001. Poking holes in the theory of broken windows. Chronicle of Higher Education (Feb.): A14.

Miller, J. 2004. Censored study on bioterror doubts U.S. preparedness. New York Times (29 Mar.): A15.

Mitchell, J. , and Mitchell, W.   1969 . Policy‐Making and Human Welfare . Chicago: Rand McNally.

Nelson, B.   1999 . Diversity and public problem solving: ideas and practice in policy education.   Journal of Policy Analysis and Management , 18: 134–55. 10.1002/(SICI)1520-6688(199924)18:1<134::AID-PAM9>3.0.CO;2-6

Radin, B. A.   1997 . The evolution of the policy analysis field: from conversation to conversations.   Journal of Policy Analysis and Management , 16: 204–18. 10.1002/(SICI)1520-6688(199721)16:2<204::AID-PAM1>3.0.CO;2-M

Raver, C.   2002 . Emotions matter: making the case for the role of young children's emotional development for early school readiness.   Social Policy Report , 16 (3): 3–19.

Roe, E.   1998 . Taking Complexity Seriously: Policy Analysis, Triangulation, and Sustainable Development . Boston: Kluwer Academic.

Satloff, R.   2003 . How to win friends and influence Arabs.   Weekly Standard , 18 Aug: 18–19.

Schön, D.   1983 . The Reflective Practitioner . New York: Basic Books.

Scott, J. 1994. Condom ads get direct: use them and get sex. Atlanta Journal and Constitution (3 Oct.): B1.

Star, S. A. , and Hughes, H. M.   1950 . Report on an educational campaign: the Cincinnati plan for the United Nations. American   Journal of Sociology , 55: 389–400.

Weimer, D. L. , and Vining, A. R.   1989 . Policy Analysis: Concepts and Practice . Englewood Cliffs, NJ: Prentice Hall.

Weiss, C.   1983 . Ideology, interests and information: the basis of policy positions. Pp. 213–45 in Ethics, the Social Sciences and Policy Analysis , ed. D. Callahan and B. Jennings . New York: Plenum.

Wilson, J. Q. , and Kelling, G.   1982 . Broken windows: the police and neighborhood safety.   Atlantic Monthly , 249 (3: Mar.): 29–38.

The first book to deal with policy sciences and consequently often cited is Lasswell and Lerner's The Policy Sciences (1951) . However this book does not address the methodological issues at hand. For an early treatment of these issues, see Etzioni 1971 b , 1968 .

For an example of how to structure and present policy research and analysis, see Dunn 1981, 322 .

For example many policy makers subscribe to George L. Kelling and James Q. Wilson's criminology theories because they make sense, despite the fact that they are not grounded in academic research. See Wilson and Kelling 1982 . For criticisms of this approach to criminology, see Miller 2001 .

“Much” of medicine is not scientifically supported (Inglefinger, Relman, and Findland 1966) . “85 percent of the problems a doctor sees in his office are not in the book” (quoted from a physician in Schön 1983, 16) .

See Etzioni 1971 a , 246– 7 .

See for example Free Expression Project 2003 ; Raver 2002, 3– 19 .

See, for instance, The Advisory Group on Public Diplomacy in the Arab and Muslim World, “Changing minds, winning peace: a new strategic direction for U.S. public diplomacy in the Arab and Muslim world,” Oct, 2003, Edward P. Djerejian, chair.

For instance, the Centers for Disease Control conducted a ten‐year ad campaign to educate Americans about condoms and to encourage their use to prevent HIV transmission. After spending millions of dollars on these ads, a CDC study found that only 45 % of sexually active high school students used a condom the last time they had sex: see Scott 1994 . A recent evaluation of the program issued an unqualified “no” in answer to the question, “Has the U.S. federal government's HIV /AIDS television [public service announcement] campaign been designed not only to make the public aware of HIV /AIDS but also to provide appropriate messages to motivate and reinforce behavior change?” See DeJong, Wolf, and Austin 2001, 256 . Of the fifty‐six ads reviewed, fifty were created by the CDC, the other six were created by the National Institute on Drug Abuse.

Star and Hughes 1950 , quoted in Berelson and Steiner 1964, 530 .

Indeed unlike science, Carol Weiss has argued that in the policy field it may be impossible to separate objective knowledge from ideology or interests: see Weiss 1983 .

See Carothers 1999 ; Etzioni 2004 .

Roe 1998 . For an academic policy research perspective, see Nelson 1999 .

See “Professional practice symposium: educating the client,” Journal of Policy Analysis and Management , 21 (1: 2002): 115– 36.

For instance, the Defense Department has prohibited a Washington think tank from publishing a complete report about the lack of government preparedness for bioterror attacks: see Miller 2004 .

See for instance the reports of the family research division of the Heritage Foundation, available at www.heritage.org/research/family/issues2004.cfm (accessed 29 Apr. 2004). See also “The war on drugs: addicted to failure,” Recommendations of the Citizens' Commission on US Drug Policy, available at www.ips‐dc.org/projects /drugpolicy.htm (accessed 29 Apr. 2004).

See Griffith University course catalog. Available at: www22.gu.edu.au/STIP/servlet/STIP?s=7319AMC (accessed 28 Apr. 2004).

This section is based on my personal observations of organizations such as the John F. Kennedy School of Government, the American Enterprise Institute, RAND, CATO, the Heritage Foundation, and many others.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• Harvard Library
• Research Guides
• HKS Library & Research Services

Public Policy & Public Administration

• Research Methods
• HKS Research & Publications
• Overviews & Introductions
• In-Depth Research
• News & Current Affairs
• Data & Statistics
• Directories & Lists

Research Methods for Policy Analysis

• Recommended Books
• Related Research Guides
• Sage Research Methods (Harvard Login) This collection provides a range of tools to develop the right methods for your research project and links to relevant book, journal, and reference content as well as a collection of case studies of real social research.
• Policy Memo Resource Site with helpful suggestions on how to write persuasive policy memos, with many examples, from the HKS Communications Program.

In addition to these comprehensive sites, also check out some of the following books: 

• << Previous: Directories & Lists
• Next: Related Research Guides >>
• Last Updated: Aug 3, 2023 9:39 AM
• URL: https://guides.library.harvard.edu/hks/public_administration

Harvard University Digital Accessibility Policy

Advertisement

Comparative Policy Analysis and the Science of Conceptual Systems: A Candidate Pathway to a Common Variable

• Published: 20 March 2021
• Volume 27 , pages 287–304, ( 2022 )

Cite this article

• Guswin de Wee   ORCID: orcid.org/0000-0002-1249-1121 1  

8332 Accesses

3 Citations

Explore all metrics

In comparative policy analysis (CPA), a generally accepted historic problem that transcends time is that of identifying common variables. Coupled with this problem is the unanswered challenge of collaboration and interdisciplinary research. Additionally, there is the problem of the rare use of text-as-data in CPA and the fact it is rarely applied, despite the potential demonstrated in other subfields. CPA is multi-disciplinary in nature, and this article explores and proposes a common variable candidate that is found in almost (if not) all policies, using the science of conceptual systems (SOCS) as a pathway to investigate the structure found in policy as a lynchpin in CPA. Furthermore, the article proposes a new text-as-data approach that is less expensive, which could lead to a more accessible method for collaborative and interdisciplinary policy development. We find that the SOCS is uniquely positioned to serve in an alliance fashion in the larger qualitative comparative analysis that supports CPA. Because policies around the world are failing to reach their goals successfully, this article is expected to open a new path of inquiry in CPA, which could be used to support interdisciplinary research for knowledge of and knowledge in policy analysis.

Similar content being viewed by others

Tempest in a teapot? Toward new collaborations between mainstream policy process studies and interpretive policy studies

Anna P. Durnová & Christopher M. Weible

Conceptual and Methodological Challenges in Comparative Public Policy

Conceptualizing the transfer of knowledge across cases in transdisciplinary research.

Carolina Adler, Gertrude Hirsch Hadorn, … Christian Pohl

Avoid common mistakes on your manuscript.

1 Introduction

In the 1990s, comparative policy analysis (CPA) and comparative analytical studies, as a modern research tradition, joined comparative politics and comparative public administration, using the comparative method (Geva-May et al. 2020 : 368). Peters and Fontaine ( 2020 : 29) contend that the comparative method, as outlined by Lijphart, provides various opportunities for scholars of public policy to enhance their collective understanding of policy and policy processes. Peters et al. ( 2018 : 137) argue that CPA is in fact precisely designed to explain policy outcomes.

Cairney and Heikkila ( 2014 : 383) suggest that, by building on established literatures, new lenses on public policy seek to improve upon rather than compete with or replace existing perspectives. Thus, the posture taken in this article is that there are challenges in the field as well as opportunities, and this article suggests an opportunity to advance comparative public policy (Wong 2018 : 963).

Research seems to suggest that a common issue in comparative research is the great difficulties in comparing policies across a variety of situations/contexts, which is referred to as the problems of identifying common variables (Wong 2013 , 2016 ; Haque 1996 ; Welch and Wong 1998 ). In identifying future challenges and opportunities of the development of comparative public policy, Wong ( 2018 ) recognises the efforts made by scholars calling for collaboration and interdisciplinary research in comparative public policy, which is a problem that has for decades not been fully addressed.

A second issue in comparative public policy, and in particular CPA, is the fact that text-as-data methods have been rarely applied widely, despite their potential, which has been demonstrated in other subfields (Guy Peters and Fontaine 2020 : 203). Additionally, according to Guy Peters and Fontaine, “[t]hey have not created new text-as-data approaches as such.”

Peters ( 2020 : 21) argues that there is a clearly a need to utilise other methods and techniques to make comparisons. According to Peters and Fontaine ( 2020 : 14), CPA is in itself multi-disciplinary and as such more akin to multi-methods than any social sciences area. As a candidate to remedy this gap in literature, this article suggests the science of conceptual systems (SOCS) and in particular the use of integrative propositional analysis (IPA) methodology to examine the structure/structural logic of policy models as a way to provide a ‘common variable’ in CPA.

The SOCS is aimed at the pursuit of knowledge and understanding of conceptual systems using rigorous methodologies (Wallis 2016 ). Conceptual systems are defined as any collection of interrelated concepts found in theories, models, policy models, axioms, laws, strategic plans, and so on, which have a set of interrelated propositions. Generally, all conceptual systems from social sciences to hard/natural sciences have one aspect in common: they have some level of structure, which makes them amenable to an IPA-based analysis or evaluation. We will delve into the IPA in detail below.

Part of the motivation for this study is the critique and necessity for the systems-based approach of the SOCS. This is the problem of linear and simple policymaking (Sabatier 1999 ), which causes a mismatch between how real-world systems work and how we think of them (Cabrera and Colosi 2008 ). The simplistic models used to make decisions in complex systems lead to “worse outcomes than the previous status quo” (Beaulieu-B and Dufort 2017 : 1) and shortsighted practices (Sterman 2012 : 24). As such, implicitly the article also uses the SOCS and IPA as a vehicle to carry out the impact of systems thinking and its benefits in policy analysis.

The contribution of this article is twofold. Firstly, it explores how the structure that is found in each policy model or conceptual system can serve as a common variable when comparing policy. Secondly, the article provides a text-as-data approach with rigour and greater accessibility, which can be easily acquired and applied by policy makers, practitioners, and scholars. The article will specifically illustrate this by explaining how policy as a unit of analysis can be used to integrate theory, policy and research based on a ‘common language’ of structure , which will potentially allow for comparative analysis across systems.

To achieve this, the article is structured in three sections. Firstly, there is a review of comparative public policy and comparative analysis. Secondly, the article provides a background of the SOCS. Thirdly, the article illustrates how and why policies (policy models) are amenable to evaluation based on their structure, which will be shown to generate novel insights into the ability of IPA to improve CPA across different systems.

2 Comparative Policy Analysis

Lasswell ( 1971 ) made a distinction between knowledge of and knowledge in the policy process. However, both are very important for the purpose of this article. With regard to knowledge in the policy process, Radin and Weimar ( 2018 : 8) state that this perspective looks at “how can analysis improve the content of public policy?” This was premised on how the political process affects policy content application and so on. This current article looks at the structure of policy implementation, and in turn examines how the data that underlie the structure of a policy influence the structure of the policy.

Policy analysis is defined as the use of reason and evidence to choose the best policy among a number of alternatives (MacRae and Wilde 1979 : 14). Dror ( 1983 : 79) defines policy analysis as a “profession-craft clustering on providing systematic, rational, and science-based help with decision-making”. Brans et al. ( 2017 ) argue that what is central to policy analysis has always been the principle that decision-making should be systematic, evidence-based, verifiable and evaluative (transparent and accountable). Geva-May et al. ( 2020 ) maintain that evidence-based policymaking also implies, by definition, the search for evidence ‘elsewhere’ for historical, international, disciplinary, or other comparisons of data, facts, and events. Policy analysis also refers to analysis for public policymaking, such as the activities, methodology and tools used to assist and advise in the policymaking context (Parsons 1996 ; Hogwood and Gunn 1984 ; Mayer et al. 2004 ; Dunn 1994 ; Fishcer et al. 2007 ). This is also referred to as the interventionist branch of the policy science tree (Enserink et al. 2012 ). As such, policy analysis, as interventionism, likened to other disciplines, needs to bridge the gap between science and action (Latour 1987 ).

Reviewing the literature above, two aspects of policy analysis become very evident. Firstly, policy analysis is a means to improve decision-making supported by evidence and secondly, evidence can also be found elsewhere as argued by Geva-May et al. ( 2020 )—it does not always have to involve conventional policy analysis methods. It is fundamental to note here that the approach and methodology suggested in this article will support other existing forms of analysis, allowing for greater understanding of policy outcomes or the development of better policies.

Radin and Weimer ( 2018 : 8) support this idea by arguing that at the most general level, science in many disciplines produces policy-relevant research that can inform policy design. Guy-Peters et al. ( 2018 ) also concur with this idea, maintaining that in understanding the various alternatives for comparison, it provides interested researchers with flexibility in explaining policy outcomes, although this requires us to be thorough and to avoid bias. Additionally, Radin and Weimer ( 2018 : 2) note that researchers in many fields contribute to knowledge that is potentially relevant to public policy. Guy-Peters and Fontaine ( 2020 : 14) agree that because of the multidisciplinary nature of CPA, it is more akin to multi-methods than any other social sciences areas. It is clear that policy analysis, as an interdisciplinary field, always leaves the door open for alternative knowledge that would improve evidence supporting decision-making. In this article, we explore how the structure found in policies can add insight to our comparative studies.

Drawing on the reviewed literature for this current article, the perspective of ‘ other comparisons of data ’ or alternative knowledge will be taken, by using the emerging SOCS as a pathway to suggest the structure of policies as a common variable for CPA and by extension provide an alternative text-as-data method to policy analysis.

It is important to note that policy analysis methods are used to help analysts to design and assess policy alternatives systematically (Radin and Weimer 2018 : 8). Building on the idea that we can use other evidence to aid analysis, Guy-Peters et al. ( 2018 ) suggest that in comparative work, when focusing on the nature of policy itself, it is amenable to either quantitative or qualitative methods.

The next section conceptualises a public policy as a conceptual system and explains why and how it is amenable to the SOCS. Van de Ven ( 2007 : 278) holds that investigating and analysing the internal logics of policies can broadly be seen as a form of design science, policy science, or evaluation research.

2.1 Public Policy/Models/Theory as Conceptual Systems

Public policies can be viewed as policy design and the output of analysis (Kingdon 1997 ), which means that the documents representing the content of policies can be viewed as shared understanding. These generally include wordy or text-based design artefacts, such as legislation, guidelines, pronouncements, court rulings, programs, and constitutions (Ingram and Schneider 1997 ). These policy designs (their content) can be seen as abstract representations of physical world systems (Schwaninger 2015 , p. 572), which is intentional in approximating physical systems by building an artificial system (Simon 1969 )—this artificial system, can also be found in our conceptual systems.

Policy design as conceptual systems can be conceptualised in the following manner: A system can be seen as a set of elements or parts with interactions among the components of the pattern/structure (Meadows 2008 ). Policy designs as conceptual systems have a structural logic existing of elements (variables/concepts/boxes) and the patterns (causal relations/arrows in a diagram) in which the elements of the policy will occur (Mohr 1987 ). Now, as suggested by Schneider and Ingram ( 1988 ), because we can diagram a sentence linking together parts of speech, it is also possible to diagram the structural logic of a policy. A key assumption of this article is that more useful/effective policies will be more structured.

Wallis ( 2020b ), drawing on Warfield ( 2003 : 515), defines a system as “any portion of the material universe which we choose to separate in thought from the rest of the universe for the purpose of considering and discussing the various changes which may occur within it under various conditions”. He further makes the distinction between the ‘material’ universe as being separate from the ‘conceptual’ universe and emphasises the useful relationship between the two. The SOCS can thus be seen as an extension of systems thinking to describing, investigating, and understanding policy designs as conceptual systems (again extending systems thinking to policy analysis at the conceptual level).

This distinction and maybe more importantly the relationship between the two universes allow for a more holistic view to policies, thus not only studying the material universe “policy environment” systematically, but also making sure that we study our conceptual universe systematically. This is premised on Ashby’s law of Requisite Variety, building on the idea that the control mechanism (policy) must have greater or equal complexity with regard to the system it intends to control, or in this case the environment it intends to address (Ashby 1957 ). This premise suggests benefits of looking at policy models as conceptual systems.

One key benefit of viewing public policy models as conceptual systems is that, by building a policy map, it enables investigations into the likelihood of negative policy interactions (Siddiki 2018 ). Moreover, using the systemic mapping approach to facilitate more readily policies that are built on interdisciplinary theory as the study of policy design as conceptual systems allows us to investigate the policy (and perhaps why it failed) in its entirety, instead of a reductionist approach—as outlined in the introduction.

It was already briefly suggested earlier that the structure of our policies or our conceptual systems has an impact on the practical application or implementation of our policies. As such, drawing from the emerging SOCS, and remembering that policy analysis knowledge can be drawn from various types of evidence or scientific evidence, we now turn to the new science to explore how it can be a common variable in CPA.

2.2 The Importance of Causality for Structure

Causality in this sense can be understood as the relationship between the concepts in ae conceptual system. Additionally, it can be argued that these causal relationships create the structure by connecting different concepts. According to Sloman and Hagmayer ( 2006 : 408), who draw on Bayer’s nets theory, “studies of learning, attributes, explanation, reasoning, judgement and decision making suggest that people are highly sensitive to causal structure.” What is important here is the understanding that causality is essential, and Wallis ( 2016 ) suggests that we can improve the structure of policy by improving the causal relationships between concepts.

With regard to structure, we look at the relationships between the concepts (causal logic); even though in our map’s causal logic (concepts without any causal relation) forms part of the structure. However, the more relationships there are (arrows between boxes/circles) the more structured and the more useful diagram/conceptual system is. In this science, we look at usefulness. Wallis ( 2020c ) makes a good argument for this, drawing on Saltelli and Funtowicz ( 2014 ) for whom “all models are wrong, but some are useful”. This is evident because some policies/theories fail, and others are useful for their purpose. According to Wallis ( 2020c ), the structure of policies represents a kind of usefulness. This has been seen in over 30 years of science investigating structure.

2.3 The Science of Conceptual Systems: A Brief History

Briefly, Cabrera ( 2006 : 3) argues that concepts exist in a system made out of other concepts which has interconnected patterns and is a conceptual ecosystem. These concepts are bound by causal connections, which form propositions that are examples of “a declarative sentence expressing a relationship among some terms” (Van de Ven 2007 : 117) and that create a set of statements understandable to others by making predictions about empirical events (Baridam 2002 : 7). At the elemental level, policies like theories have concepts and causal relations with underlying data that create propositions as understandable statements, which implies that they are commutable and public (Baridam 2002 ). Like theories, we use policies to make predictions about empirical events, such as implementing COVID-19 regulations and measures (problems and solutions on paper) in the hope of stopping infections and eventually having a virus free country (empirical event), such as New Zealand and others.

According to Wallis ( 2016 ), the SOCS is aimed at the pursuit of knowledge and understanding of conceptual systems whilst using rigorous methodologies. At least three streams of research on structure suggest that structured knowledge is useful for changing the world positively and reaching desired goals. These streams are as follows:

In the field of education/human development, “Systematicity” is used to evaluate the structure of maps and evidence. This suggests that beginners create simplistic low structured maps compared with the more structured and complex maps of experts (Novak 2010 ).

In the field of political psychology, the measure of Integrative Complexity has been used for the past 30-years (Suedfeld et al. 1992 ; Wong et al. 2011 ).

The third stream of research on structure involves studies of formal theories and policy models within and between multiple disciplines (Wallis et al. 2016 ).

The third stream, which is of great significance for this current article is IPA.

2.4 The IPA Method

According to Wallis ( 2016 ), IPA is primarily used to analyse conceptual systems from text on paper to determine their structure (Wallis 2016 ). The IPA methodology uses the policy document’s text itself as data (Wallis 2016 : 585). This process includes the following six steps: (1) Identify propositions within one or more conceptual systems (models, etc.). (2) Diagram those propositions with one box for each concept and arrows indicating directions of causal effects. (3) Find linkages between causal concepts and resultant concepts between all propositions. (4) Identify the total number of concepts (to find the Complexity). (5) Identify concatenated concepts. (6) Divide the number of concatenated concepts by the total number of concepts in the model (to find the Systemicity).

For a very brief and abstract example, consider Fig.  1 . The figure has three variables/concepts (A, B, C), therefore, the Complexity is C = 3. There is one concatenated concept (C). So, the Systemicity is C = 0.33 (the result of one concatenated concept divided by three total concepts).

Abstract example of a model for demonstrating IPA

Concepts (relating to variables) are enumerated to show the Complexity or explanatory breadth of the conceptual system. The causal interconnectedness of those concepts is evaluated for their Systemicity (structure, or explanatory depth). Systemicity is measured on a scale of zero to one with one being the highest (Wallis and Valentinov 2017 : 109). Using IPA, Complexity, on its own, is seen as a weak indicator of success for a conceptual system, building on the idea of the SOCS. The main concern for this measurement of structure is that those theories, policy models and general conceptual systems with a higher level of structure are more useful for practical application and implementation.

It can be argued that the external validity of the methodology has been established. Firstly, the findings suggest that theories in the natural sciences have high levels of structure (Systemicity) and are proved to be effective and useful in application, such as Ohm’s Law (Wallis 2016 , b ; Wallis 2010a , b ). In contrast, theories in the social sciences have a Systemicity generally of less than 0,25, including theories of conflict, psychology, and very important for this article, policies (Wallis 2010a , b , 2011 ; 2013 ; Shackleford 2014 ; Parmentola et al. 2018; Wallis et al. 2016 ; de Wee 2020 ; de Wee & Asmah-Andoh, in press). Correlating with these findings, Light ( 2016 ) found that policies in the United States of America only succeed 20% of the time. This is parallel to the 0, 25 Systemicity generally found in social sciences, which essentially translates to a 75–80% chance that most of our policies could fail.

2.5 The Conceptual System: A Basis for Comparison

Briefly, in SOCS, using IPA, we quantify and diagram the structural logic of the policy, and find links between the measure (percentage providing a predictor) of the policy structure and its usefulness in the real world. As a brief example, one could consider Figs.  2 and 3 that are maps of different theories, one from physical science and the other from social science. These propositions are mapped out to indicate how we can use the structure found in any policy as the basis of comparison.

Georg Ohm developed the propositions: An increase in resistance and an increase in voltage would result in an increase of current, which will cause a decrease in resistance. An increase in current causes an increase in voltage that would result in an increase in resistance, which will cause a decrease in current.

The democratic peace proposition has many possible empirical and theoretical forms. One of these holds that “the more democracies there are in a region or the international system, the more peaceful the region or international system will be” (Reiter 2012 )

The structural logic of Ohm's law as a practical map (Wright and Wallis 2019 , p. 152)

Source : Authors own compilation

The structural logic of the democratic peace proposition as a practical map.

The two brief examples were created using IPA steps 1–3 (to diagram) and the person reading these propositions from any country who reads the English language can see the causal relations between the concepts. This is the benefit and perhaps the most important contribution of the SOCS and IPA to CPA. Firstly, the examples indicate how IPA takes a “common denominator” approach by presenting theories graphically in an easily understandable format of concepts (in boxes/circles) and causal arrows (indicating the direction and causality) (Wallis 2020a , b , c ). The key contribution here is that with IPA we have an objective and rigorous method we can use to evaluate, analyse, and develop policies based on their structural logic, which was first suggested by Schneider and Ingram ( 1988 ); however, without a clear measure. Furthermore, it also provides a “common language” for interdisciplinary work, which is important for policies if they are to be successful, remembering of course that no problem in policy is ever addressed using only ‘one’ policy. As Siddiki ( 2018 ) found, policies that are used to address a problem often led to inter policy conflict that causes problems with “policy coherence”, which May et al. ( 2006 ) refer to in terms of how well policies with similar objectives fit or go together.

Additionally, IPA and the SOCS can be of great benefit for policy design and CPA. Applying the IPA’s steps 4–6 to Figs.  2 and 3 , their level of structure differs significantly. Ohm’s law has a structure/Systemicity of 1 (100% usefulness in application) and the democratic peace proposition, has a structure/Systemicity of 0. In the real world, Ohms law has been very successful, and the democratic peace theory has numerous shortcomings.

For years, it has been accepted that policy designs are “copied, borrowed or pinched” from similar policies in other locales (Schneider and Ingram 1988 : 62) which makes policy design less a matter of invention than of selection (Simon 1981 ) involving large stores of information and making comparisons. However, this led to suboptimal situations where policy layering and patching is done haphazardly (Van der Heijden 2011 ) leading to a palimpsest-like mixture of incoherent or inconsistent policy (Howlett and Rayner 2007 ; Carter 2012 ). Again, perhaps using IPA and viewing policies as conceptual systems allows one to create coherent policies more easily, which are argued to be more effective in application (Siddiki 2018 ).

Based on the case made earlier, this stream of research seems to provide policy analysis with policy-relevant research that can inform policy analysis (Geva-May et al. 2020 ; Guy-Peters et al. 2018 ; Radin and Weimer 2018 ). Hence, it provides this current article the basis for suggesting the evaluation of the structure of policy as a common variable to be used in CPA, additionally because this “on paper” analysis provides a new text-as-data approach. Although other aspects of policymaking and analysis, such as empirical data and the implementation process, are important, we suggest focussing on the structure of policy in comparative analysis. This article suggests the different but allied perspective that this stream of research provides, which will be explained later.

Extending on the above, pragmatically, inter-rater evaluation can be done, where scores of the analysis can be compared and the consensus of the raters measured through agreement or concordance (Bless et al. 2016 : 226). Prior studies indicate how a repeated study using the same data would have stable findings over repeated observations, which confirms its external validity, or generalisability (de Wee 2020 : 135).

In the followings section, we turn to IPA and the application of the method on conceptual systems including policy. The section not only demonstrates the methodology but also the use of the method on policy in previous research. Additionally, it indicates how less “financially demanding” the method is than the computational and statistical models. Based on using textual analysis (IPA) on policy, the article suggests that this is an alternative or new pathway to compare, analyse and evaluate policies. The next section discusses text-as-data and the SOCS compatibility with it.

2.6 A New Text-as-Data Approach: The Basis for Collaboration and Interdisciplinary Policy Development

Current research (Fisher et al. 2013 ; Leifeld 2013 ; Guy-Peters and Fontaine 2020 ; Leifeld and Haunss 2012 ) indicates the prevalence of qualitative text analysis methods in policy analysis, including the discourse networks that rely on text analysis to measure discourse coalitions quantitatively through network analysis. Guy-Peters and Fontaine ( 2020 : 204) argue that text-as-data-methods for CPA can include new theories and methods relevant to public policy and policy analysis.

Guy-Peters and Fontaine ( 2020 ) identify the different directions text-as-data application is currently developing. Firstly, there is causality, with Egami et al.’s ( 2018 ) framework of estimating causal effects in sequential experiments. Owing to limited space, this process will not be explained in full (see Egami et al. 2018 ). Secondly, there is computer science research on word embedding and on artificial neutral networks, which is what they call ‘deep learning’ (LeCun, Bengio & Hinton 2015 ). From the findings of Guy-Peters and Fontaine ( 2020 ), it becomes apparent that the use of statistical and computational analysis is a challenge in the discipline, especially because there are no globally best methods for retrieving certain information from text. Secondly, with manual approaches, this is practically impossible because of the prohibitive costs (Guy-Peters and Fontaine 2020 : 213).

This article suggests a key benefit of viewing the IPA method as part of the SOCS when doing CPA as a new “text-as-data” technique. IPA objectively evaluates structure, and it provides a way for comparison. Additionally, the text-as-data method outlined in the most current literature tends to have two key problems: firstly, it is difficult to use and secondly, it is expensive and practically impossible. To the field of CPA, IPA provides a method that can be used easily and with rigor, which is not mentally taxing or too expensive. To the text-as-data method, this article provides accessibility. The IPA method is more accessible and can be applied with ease by scholars and practitioners from various countries and various disciplines to improve the policies they design. This method also provides a basis from which interdisciplinary work on policy can branch out.

As such, when analysing policy, one can do a within-case analysis and thereafter a cross-case comparison, to generalise findings further, whilst considering the context. However, we will explore this further in the next section.

2.7 Collaboration and Interdisciplinary Perspectives Based on Policy Structure

In fact, the IPA methodology and the resources to assist in the application are readily available online at https://projectfast.org/ , where there are additional tools to enhance the analysis, which can introduce more collaboration. Collaboration can be achieved, through analysing a policy individually and outcomes. Or you can build or analyse the policy together using the six-steps of IPA on the free online social network analysis software KUMU ( www.kumu.io ). This platform allows participants in policy analysis or policy making to organise complex information (concepts and causal relations of a policy based on IPA), to diagram graphically, to build and to analyse policies based on their structure. This platform is open for multiple participants globally who are given permission to participate. Potentially, all participants can have the same policy or empirical data/research and can build a policy or map on this platform; however, this article will present more on this topic in the section on future research areas and implications .

This article argues that when exploring integration studies and literature around interdisciplinary research or problem solving, it becomes apparent that we can use an interdisciplinary approach, which can clarify the observer’s standpoint, define and orient the observer to a problem. Moreover, this approach can be used to map the full social and decision-making context and apply multiple methods to generate, evaluate and implement solutions (Clark 2002 , 2011 ).

Ideas, concepts, and methods around interdisciplinary tools of the policy sciences to problem solving have been available for decades (Lasswell 1971 ; Brunner 1996 ); however, they have not been effectively used it seems. It has been recognised that the current complex problems cannot be resolved with just theories and policies but require contextual interdisciplinary understanding (Clark et al. 2011 ). One of the interdisciplinary questions to policy problems is “what is the common problem that underlies the increasingly apparent disconnect between real world problems and the knowledge and skills currently offered by policy makers?” For this article, the answer lies in the fact that we have policies that are developed but do not reflect our context enough, most of which are often imported from somewhere else and do not address the complexity of the environment.

For example, Wright and Wallis ( 2019 ) wrote a paper on poverty and the various explanations for poverty in the United States. These various “theories” on why there is poverty were then integrated using IPA, which led to a more coherent and interdisciplinary explanation as to why poverty exists. The benefit of the study suggests that by using IPA, one can synthesise various theoretical/data perspectives and create an interdisciplinary explanation to guide policy. A second and a more recent example is the study on responses to COVID-19 by Fink and Wallis ( 2020a , b , c ), where they looked at the models used to combat the disease. Two countries (Germany and New Zealand) handled the crisis the best, as their policies were built on a structured scientific model compared to that of the US and the UK, where the leadership generally rejected scientific results and responded with single values, which caused the US to have the largest number of infections. Here the simple single value approach coupled with a general denying of scientific data led to the US struggling in combating the virus. Again, this feeds back to the problem of having models that are not equal or at least the same as the complexity of the virus.

These two examples are used to indicate how interdisciplinary research can be used to improve CPA and policy development in general. Due to space, a complete case study is not possible. However, evidence seems to suggest that complex and structured plans/policies supported by good interdisciplinary data could lead to better action and decisions by our leaders in attempting to improve the human condition. Thus, the contribution of this article is to indicate that there is an accessible and rather easier method one can use to improve our society.

Echoing the words of Le´le´ and Norgaard ( 2005 ), this article suggests we identify the structure or lack thereof, of policies as a common variable. As demonstrated by Wallis ( 2019 ), “based on the structure of policies we can pull together what is known about a problem from academic and practical experience” (Bammer 2013 :6) and produce a more accurate and complete understanding for more effective action (Newell 2001 : 22). Theoretically this is sound, but pragmatically how can this be done in CPA?

2.8 A Few Practical Points for Collaboration and Interdisciplinary Work

The text-as-data approach of IPA, which is part of the larger SOCS, can be advanced as an answer to the call made by Wong ( 2018 ) for more interdisciplinary research.

Two policies can be comparatively analysed in two different (or multiple) states, for example, a drug policy in South Africa and another in the United States of America. The structure can be measured and compared. This allows us to trace how a design evolves over time.

Using IPA, you can design policies, map them out and see where there are concepts/variables (represented by circles/blocks) on your map that might have no arrows leading to any other box. Moreover, you can apply “gap analysis” by filling the gap with concepts (supported by data) and creating links with the other concepts.

Addressing “unknown unknowns”, Wallis ( 2020a ) argues that practically speaking, something that is unknowable for one person, or in one policy environment, system, or country, can be known by another. Thus, when we look at the systemic structure of policies comparatively, we can identify and determine when something is “missing” in our understanding (based on the diagram) and use different policies and policy empirical data to supplement our own map.

Finally, drawing from the discussion in this section, the idea of using IPA as a text-as-data methodology is supported by the assertion of Guy-Peters and Fontaine ( 2020 : 203) that like other inductive analysis procedures, IPA also makes it possible to discover new phenomena, concepts, and relations from the latent dimension of a text. This notion of new concepts and relationships is supported by Geva-may et al. ( 2020 :368) who state that the comparative perspective at micro- or macro-levels is paramount to effective and efficient policy development. Moreover, this novel idea based on the SOCS is one that can potentially advance effectiveness and efficiency. Using IPA in concert with other conventional CPA methods can improve policy analysis.

Next, the article places IPA as a potential method to be used in qualitative comparative analysis (QCA).

3 Qualitative Comparative Analysis and the CPA

QCA approaches focus on not only the independent variable and its necessity or sufficiency for an outcome of the dependent variable; it also focuses on how an outcome can be achieved by various configurations or combinations of independent variables (Schneider and Wagemann 2012; Guy-Peters and Fontaine 2020 : 43). Guy-Peters and Fontaine ( 2020 ) continue that based on this idea, QCA is similar to most-similar and most-different systems design, as they are also based on the concepts of necessity and sufficiency. QCA can produce empirically well-grounded, context-sensitive evidence about policy instruments (Pattyn et al. 2019 ), such as IPA when looking at policy models as conceptual systems. An advantage of QCA for public policy analysis is that it enhances analysists’ ability to capture the outcomes of decision-making, implementation and evaluation accurately.

Guy-Peters and Fontaine ( 2020 : 7) argue the following: “If we take political science, economics, and sociology as the “heartland” of CPA, then existing methodological discrepancies among scholars may not be too extreme”. However, these are not the only disciplines contributing to CPA. It is argued that many other new disciplines bring complementary approaches, hence new insights on causality and causation. Here they refer to various disciplines, such as anthropology, law and history with their techniques, which “…are other sources of evidence (that) contribute to the development of CPA” (Guy-Peters and Fontaine 2020 : 7). It is clear that CPA draws on various disciplines, and as such, it is sufficient to argue that with the insight of the emerging SOCS, IPA can also be used when analysing policy.

The suggestion is that this methodology of evaluating the structure of the policy against the implemented results can sit comfortably with the QCA method of which the “primary goal is to discover empirical relationships that can help theoretical proposals” (Anckar 2020 : 43). Of course, one has to remember that QCA is premised on the idea that an outcome (a certain value on the dependent variable) can be reached by several combinations of conditions (i.e., independent variables) (Anckar 2020 : 43). The contribution of this article is to propose a new dimension to the present methods, specifically focusing on analysing and evaluating policy documents (policy models or policy-like documents), based on their level of structure, which can be done across policy fields.

From the above, it is clear that for CPA and for the advancement of policy development itself, there is a need for the triangulation of methods and theory that can provide evidence to enhance effective decision-making. Thus, this article agrees with Guy-Peters and Fontaine (2020: 14, emphasis added) that future studies must ‘ think about triangulation and the use of multi-methods as a means of gaining a more complete picture of the policy issue under scrutiny”. Next, the article suggests potential future research areas in CPA.

4 Implications and Future Research

Although this is a preliminary exploration, conclusions can be drawn and taken from this article, as it is a starting point to a conversation about finding a common variable in CPA and introducing a new text-as-data approach.

The conceptual principles drawn and the introduction of a new potential field in CPA, using structure as a common variable, can be a starting point for future studies aiming to test the usefulness of applying IPA in CPA empirically. Scholars can analyse and evaluate policies from different environments, bearing in mind not to sample only a dependent variable based on successful (implementation) cases, which will take away the idea of an ‘inferential felony’ (Geddes 2003 ). IPA can be used in QCA, comparatively analysing policy outcomes with other analysis methods to test if the hypothesis of IPA holds, and potentially falsify it, which will advance the literature about IPA.

Two key deductions can be made. Firstly, comparing a policy based on its structure (coherence) can improve both the implementation success and understanding of a policy environment. Secondly, based on the ‘common denominator’ of IPA (applicable to all theories/policy models) we now have a ‘common language’ we can use for CPA when designing policies for policy relevant research, which in turn provides a good platform for interdisciplinary research.

Most studies of policy analysis compare concepts/elements (in policy) to the real world. However, such a reflective hermeneutic approach can only study selected chunks of policy situations comparatively. Using the text-as-data, or textual hermeneutic approach, analysists can use IPA to evaluate the policy, and identify potential gaps in the policy (where there are concepts with no causal relation to other concepts or where there is just a fuzzy implied causality). These gaps between concepts can be identified as potential reasons, which can pinpoint where analysis can be done. As such, future studies that aim to do comparative analysis can use the policy document (and its internal logic structure) itself as a lynchpin to determine or pinpoint potential areas that could have led to policy failures or success. A broader number of variables for comparative studies can be identified, which could require more interdisciplinary research to make for a more nuanced understanding of policy outcomes, which has for decades not been fully answered (Wong 2018 ). Thus, the article has interdisciplinary implications for scholars from various disciplines producing policy relevant evidence.

Because of limited time, space and resources, there are various aspects of CPA that were not covered in this attempt to introduce IPA to this relevant branch of research. However, because this is a preliminary exploration, future studies should cover more of the field. Additionally, the conversation through academic articles can be advanced to position the SOCS and IPA better in CPA.

5 Concluding Remarks and Limitations

Premised on the fact that it draws on the long history of the SOCS, attempts to answer longstanding calls for a common variable in CPA and suggests a new text-as-data method for CPA, this article is somewhat exploratory in nature. Moreover, this article is preliminary, in that it wants to locate the SOCS as a possible pathway to compare, analyse and evaluate policies based on their structure to improve both policy development and how we compare across systems. Depending on more extrapolation, this article presents CPA as a start to a new area for comparison.

The article explored avenues that could potentially contribute an answer to the calls for “a common variable” method that facilitates interdisciplinary and collaborative research and a new accessible text-as-data tool. In its exploration of the literature, the article established that the SOCS and the measurement of structure is a candidate for a common variable, as all policies in the world (dare we say) are written on paper. Additionally, the article provides an existing and developing method (IPA) as a tool to analyse policies and suggests the practical benefits of this tool to CPA (see future research areas and implications).

The use of the structure of the policy in CPA as a lynchpin in analysis or policymaking can bring various variables, concepts, and ideas into play. This approach enables policymakers, academics, practitioners, and other interested participants to find and manage potential “unknown unknowns” about a certain field of policy, such as health and the economy, because of the interdisciplinary, integrative and collaborative characteristic of the SOCS and IPA. Moreover, this could potentially improve our policy feedback, and policy learning in cross system comparative analyses.

What is important to take away here, which is in line with the findings of MacRae and Wilde ( 1979 ), Dror ( 1983 ), Brans et al., ( 2017 ) as well as Geva-May et al. ( 2020 ) is the fact that based on the structure of a policy as a variable, one can develop systemic and evidence-based policy. This is because IPA measures how structured a policy is and how well it reflects the real world. Furthermore, it is evidence based, as it identifies where possible shortcomings can be, where we can add information, collaborate and have interdisciplinary knowledge. Additionally, it is verifiable since interrater reliability can be used across systems, and it is evaluateable because it suggests measurable and actionable propositions in policies, which can be measured and as such keep government accountable. Moreover, using policy feedback, the evaluation of these propositions can improve policy through policy lessons that will advance its sustainability/longevity.

This article does not come without limitations. Firstly, IPA as a method does not consider the ‘off-the-page’ aspects of policy analysis such as data, the mood of analysts or if the implementation followed the policy to the letter. However, locating IPA within the larger QCA methods, the fact that CPA draws on various disciplines and the fact that the IPA provides CPA the means to use the policy structure as the lynchpin for policy analysis all mean that these conventional and existing methods can be used to fill that void. They will be building a stronger and more reliable alliance (to identify the gaps in policy structures and make sure they are measurable, as this will make evaluation and transparency easy).

This article also does not empirically test or compare policies; it is theoretical and grounded in a review of literature. Moreover, as proposed in the section above, an empirical approach would make findings more concrete and verify the strength of the ideas presented here.

Lastly, there is the problem of language, as it is clear that not all countries will have their policies written in English, which could serve as a limitation. However, the measuring of structure could be done using IPA in different languages. Additionally, most countries in the world use English in their academic institutions and publications, and thus translations could be done by language experts. This could help advance the IPA methodology and CPA.

In conclusion, echoing the words of Fuchs and Hofkirchner ( 2005 ), we have an ethical responsibility to develop new knowledge and new understandings to deal with issues. This exploration has opened a new pathway to improve how we develop policy and our ability to compare policies across systems, and essentially by extension improve the human condition.

Anckar, C. (2020). The most-similar and most-different systems design in comparative policy analysis. In Handbook of research methods and applications in comparative policy analysis . Edward Elgar Publishing.

Ashby, W. R. (1957). An Introduction to Cybernetics . London: Chapman and Hall.

Google Scholar  

Ball, S. (1993). What is policy? Texts, trajectories and toolboxes. Discourse, 13 (2), 10–17.

Bammer, G. (2013). Disciplining interdisciplinarity: Integration and implementation sciences for researching complex real-world problems . ANU E Press.

Baridam, D. M. (2002). Management and organisation theory (3rd ed.). Port-Harcourt: Sherbrooke Associates.

Beaulieu-B, P., & Dufort, P. (2017). Introduction: Revolution in Military Epistemology. Journal of Military and Strategic Studies, 17 (4), 1–20.

Bless, C., Higson-Smith, C., & Sithole, S. L. (2016). Fundamentals of Social Research Methods: An African Perspective . Cape Town: Juta & Company Ltd.

Brans, M., Geva-May, I., & Howlett, M. (2017). Policy analysis in comparative perspective: introduction. In M. Brans, I. Geva-May, & M. Howlett (Eds.), Routledge Handbook of Comparative Policy Analysis (pp. 1–24). New York: Routledge.

Chapter   Google Scholar  

Brunner, R. D. (1996). A milestone in the policy sciences. Policy Sciences, 29 , 45–68.

Article   Google Scholar  

Cabrera D. A. (2006) . Systems Thinking. A Dissertation Presented to the Faculty of the Graduate School of Cornell University Available. https://ecommons.cornell.edu/bitstream/handle/1813/2860/DerekCabreraDissertation.pdf

Cabrera, D., & Colosi, L. (2008). Distinctions, systems, relationships, and perspectives (DSRP): A theory of thinking and of things. Evaluation and Program Planning, 31 , 311–316.

Cairney, P., & Heikkila, T. (2014). “A Comparison of Theories of the Policy Process.” In Theories of the Policy Process , 3rd ed., ed. Paul A. Sabatier, and Christopher Weible. Boulder, CO: Westview, 363–90.

Cannavale, C., & Wallis, S. E. (2015). The entrepreneurial cultural approach: does culture impact on researchers’ perspective and on entrepreneurial success at a country level. Paper presented at the Syst Pract Action Res (2017) 30:103–116 113IACCM International association for cross-cultural competence and management 14th annual conference and 7th CEMPS/IACCM doctoral workshop,

Carter, P. (2012). “Policy as palimpsest”. Policy & Politics, 40 (3), 423–443.

Clark, T. W. (2002). The policy process: a practical guide for natural resource professionals . New Haven, CT: Yale University Press.

Clark, S. G., Rutherford, M. B., Auer, M. R., Cherney, D. N., Wallace, R. L., Mattson, D. J., & Wilshusen, P. (2011). College and university environmental programs as a policy problem (part 1): Integrating knowledge, education, and action for a better world? Environmental Management, 47 (5), 701–715.

de Wee, G. (2020). An Evaluation of the Integrative Propositional Analysis Model for Complex Policy Environments: The Case of the Drakenstein Housing Policy, 2010–2017. Masters dissertations submitted at Nelson Mandela University

de Wee, G., & Asmah-Andoh, K. (under submission). Model for Overcoming Policy Analysis Limitation and Implementation Challenges: Integrative Propositional Analysis of South African National Mental Health Policy Framework and Strategic Plan 2013–2020.  International Journal of Public Administration

Dror, Y. (1983). Public Policy Making Reexamined . Dunn, WN: Transaction Publishers.

Dubin, R. (1978). Theory building (Revised). New York: The Free Press.

Dunn, W. (1994). Public policy analysis: an introduction (2nd ed.). Englewood Cliffs: Prentice-Hall.

Egami, N., Fong, C. J., Grimmer, J., Roberts, M. E., & Stewart, B. M. (2018). How to make causal inferences using texts . Unpublished manuscript, Princeton University.

Enserink, B., Koppenjan, J. F. M., & Mayer, I. S. (2012). A Policy Sciences View on Policy Analysis. In W. A. H. Thissen & W. W. Walker (Eds.), Public Policy Analysis: New Developments (pp. 11–40). New York: Springer.

Fink, G., & Wallis, S. (2020). Understanding and avoiding negative consequences of value-based laws, policies, and programs. Systems research and behavioural science (Under submission)

Fishcer, F., Miller, G. J., & Sidney, M. S. (2007). Handbook for public policy analysis: Theory, politics and methods . New York: Taylor & Francis Group LLC.

Fisher, D. R., Leifeld, P., & Iwaki, Y. (2013). Mapping the ideological networks of American climate politics. Climatic Change, 116 (3–4), 523–545.

Fuchs, C., & Hofkirchner, W. (2005). Self-organization, knowledge, and responsibility. Kybernetes, 34 (1/2), 241–260. https://doi.org/10.1108/03684920510575825 .

Geddes, B. (2003). Paradigms and sand castles: Theory building and research design in comparative politics. Ann Arbor: University of Michigan Press.

Book   Google Scholar  

Geva-May, I., Hoffman, D. C., & Muhleisen, J. (2020). Trends in the development of comparative policy analysis. In B. Guy Peters & G. Fontaine (Eds.), Handbook of Research Methods and Applications in Comparative Policy Analysis . Northampton: Edward Elgar Publishing.

Gregor, S. (2006). The nature of theory in information systems. MIS Quarterly, 30 (3), 611–642.

Guy-Peters, B., & Fontaine, G. (2020). Handbook of Research Methods and Applications in Comparative Policy Analysis . Northampton: Edward Elgar Publishing.

Guy-Peters, B., Fontaine, G., & Mendez, J.-L. (2018). Substance and Methods in the Comparative Study of Policy Change. Journal of Comparative Policy Analysis: Research and Practice, 20 (2), 133–141.

Haque, M. S. (1996). The contextless nature of public administration in third world countries. International Review of Administration Science, 62, 315–329.

Hogwood, B., & Gunn, L. (1984). Policy Analysis in the Real World . Oxford: Oxford University Press.

Howell, S., & Couzyn, K. (2015). The South African National Drug Master Plan 2013–2017: A critical review. South African Journal of Criminal Justice, 28 (1), 22–23.

Howlett, M., & Rayner, J. (2007). Design principles for policy mixes: Cohesion and coherence in “new governance arrangements.” Policy and Society, 26 (4), 1–18.

Ingram, H., & Schneider, A. (1997). Policy design for democracy . Kansas: University Press of Kansas.

Kaplan, A. (1964). The conduct of inquiry: Methodology for behavioral science . San Francisco: Chandler Publishing Company.

Kerlinger, R. (1986). Foundations of behavioural research . New York: Holt, Rinehart, & Winston.

Kingdon, J. W. (1997). Agendas, alternatives, and public policies (2nd ed.). London: Pearson Education.

Lasswell, H. D. (1971). A pre-view of policy sciences . New York: Elsevier.

Latour, B. (1987). Science in action: How to follow scientists and engineers through society . Cambridge, Mass.: Harvard University Press.

Le´le´ S, Norgaard R. B. . (2005). Practicing interdisciplinarity. BioScience, 55 (11), 967–975.

LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning . Nature, 521 (7553), 436–444.

Leifeld, P. (2013). Reconceptualizing Major Policy Change in the Advocacy Coalition Framework: A Discourse Network Analysis of German Pension Politics. Policy Studies Journal, 41 (1), 169–198.

Leifeld, P., & Haunss, S. (2012). Political discourse networks and the conflict over software patents in Europe. European Journal of Political Research, 51 (3), 382–409.

Light, P. C. (2016). Vision plus Action = faithful execution: why government daydreams and how to stop the cascade of breakdowns that now haunts it. Politi Sci Polit, 49 (1), 5–26.

MacRae, D., & Wilde, J. A. (1979). Policy Analysis for Public Decisions . North Scituate, MA: Duxbury Press.

May, P. J., Sapotichne, J., & Workman, S. (2006). Policy coherence and policy domains. Policy Studies Journal, 34 (3), 381–403.

Mayer, I., van Daalen, C. E., & Bots, P. (2004). Perspectives on policy analysis: a framework for understanding and design. International Journal of Technology, Policy and Management, 4 (2), 169–191.

Meadows, D. H. (2008). Thinking in systems: A primer . Hartford: Chelsea Green Publishing.

Mohr, L. B. (1987). Impact Analysis for Program Evaluation . Chicago, IL: The Dorsey Press.

Newell, W. H. (2001). A theory of interdisciplinary studies. Issues in integrative studies, 19 (1), 1–25.

Novak, J. D. (2010). Learning, creating, and using knowledge: Concept maps as facilitative tools in schools and corporations. Journal of e-Learning and Knowledge Society, 6 (3), 21–30.

Parmentola, A., Simoni, M., Tutore, I., & Wallis, S. E. (2019). Boosting the spread of new technologies: An integrative propositional analysis of diffusion policies. Technology Analysis & Strategic Management, 32 (2), 133–145.

Parsons, W. (1996). Public policy: An introduction to the theory and practice of policy analysis . Chetterham/Lyme: Edward Elgar Publishing.

Pattyn, V., Molenveld, A., & Befani, B. (2019). Qualitative comparative analysis as an evaluation tool: Lessons from an application in development cooperation. American Journal of Evaluation, 40 (1), 55–74.

Raadschelders, J. C. N. (2018). Comparative Public Administration in a Globalizing World: A New Psychological Challenge next to the Traditional Scholarly Challenges Institute of Public Administration . University of Leiden Paper prepared for the annual conference of the American Society for Public Administration, Denver.

Radin, B. A., & Weimer, D. L. (2018). Compared to What? The Multiple Meanings of Comparative Policy Analysis. Journal of Comparative Policy Analysis, 1, 1. https://doi.org/10.1080/13876988.2017.1414475 .

Reiter, D. (2012). Democratic peace theory . Oxford: Oxford University Press.

Roux, N. L. (2002). Public policymaking and policy analysis in South Africa amidst transformation, change and globalisation: Views on participants and role players in the policy analytic procure. Journal of Public Administration, 37 (4), 418–437.

Russell, D., & Wallis, S. E. (2015). Designing a learning analytic system for assessing immersive virtual learning environments. In D. Russell & J. Laffey (Eds.), Handbook of research on gaming trends in P-12 education . Vienna: Hershey IGI Globalvienna University of Economics and Business.

Sabatier, P. A. (1999). Theories of the policy process . Boulder, CO: Westview.

Salmon, W. C. (1984). Scientific explanation and the causal structure of the world . New Jersey: Princeton University Press.

Saltelli, A., & Funtowicz, S. (2014). When all models are wrong. Issues in Science and Technology, 30 (2), 79–85.

Schneider, A. L., & Ingram, H. (1988). Systematically pinching ideas: a comparative approach to policy design. Journal of Public Policy, 8 (1), 61–80.

Schwaninger, M. (2015). Model-based management: A cybernetic concept. Systems Research and Behavioral Science, 32 (6), 564–578.

Shackleford, C. (2014). Proportional analysis, policy creation and complex environments in the United States’ 2009 Afghanistan-Pakistan Policy. Doctoral Dissertation (DPhil). Walden University.

Siddiki, S. (2018). Policy Design and Conflict. In Routledge Handbook of Policy Design  (pp. 212–223). Routledge.

Simon, H. (1969). The sciences of the artificial . Cambridge, Mass: The MIT Press.

Simon, H. A. (1981). The sciences of the artificial . Cambridge, Mass: The MIT Press.

Sloman, S. A., & Hagmayer, Y. (2006). The causal psycho-logic of choice [Opinion]. Trends in Cognitive Science, 10 (9), 407–412.

Steenhuisen, B. (2012). Policy design: who, what, how Paper for the panel ‘Policy as design’ at the Interpretive Policy Analysis Conference, Tilburg 2012 file:///C:/Users/nikki/Downloads/647–2225–1-PB%20(1).pdf

Sterman, J. D. (2012). Sustaining sustainability: Creating a systems science in a fragmented academy and polarized world. In M. P. Weinstein & R. E. Turner (Eds.), Sustainability science: The emerging paradigm and the urban environment (pp. 21–58). New York: Springer.

Stinchcombe, A. L. (1987). Constructing social theories . Chicago: University of Chicago Press.

Suedfeld, P., & Rank, A. D. (1976). Revolutionary leaders: Long-term success as a function of changes in conceptual complexity. Journal of Personality and Social Psychology, 34 (2), 169–178.

Suedfeld, P., & Tetlock, P. (1977). Integrative complexity of communications in international crises. The Journal of Conflict Resolution, 21 (1), 169–184.

Suedfeld, P., Tetlock, P. E., & Streufert, S. (1992). Conceptual/integrative complexity. In C. P. Smith (Ed.), Handbook of Thematic Content Analysis (pp. 393–400). New York: Cambridge University Press.

Van De Ven, A. H. (2007). Engaged scholarship: A guide for organizational and social research . Oxford: Oxford University Press.

Van der Heijden, J. (2011). Institutional Layering: A Review of the Use of the Concept. Politics, 31 (1), 9–18.

Wallis, S. E. (2010a). Toward the Development of More Effective Policy Models. Integral Review-Toward Development of Politics and the Political., 6 (1), 153–177.

Wallis, S. E. (2010b). The structure of theory and the structure of scientific revolutions: What constitutes an advance in theory? In S. E. Wallis (Ed.), Cybernetics and systems theory in management: Views, tools, and advancements (pp. 151–174). Hershey, PA: IGI Global.

Wallis, S. E. (2011). Avoiding Policy Failure: A Workable Approach . Litchfield Park: Emergent Publications.

Wallis, S. E. (2013). How to choose between policy proposals: A simple tool based on systems thinking and complexity theory. Emergence Complexity & Organization, 15 (3), 94–120.

Wallis, S. E. (2014). Structures of logic in policy and theory: Identifying sub-systemic bricks for investigating, building, and understanding conceptual systems. Foundations of Science., 20 (3), 213–231.

Wallis, S. E. (2015). Integrative propositional analysis: A new quantitative method for evaluating theories in psychology. Review of General Psychology, 19 (3), 365–380.

Wallis, S. E. (2016). The science of conceptual systems: A progress report. Foundations of Science, 21 (4), 579–602.

Wallis, S. E. (2018). Integrative propositional analysis for developing capacity in an academic research institution by improving strategic plans . Paper presented at the Unpublished Conference Paper ISSS 2018 (obtained from author via email).

Wallis, S. E. (2020a). Understanding and improving the usefulness of conceptual systems and integrative propositional analysis-based perspective on levels of structure and emergence. Syst Res Behav Sci., 1, 1–22. https://doi.org/10.1002/sres.2680 .

Wallis, S. E. (2020b). Evaluating and improving theory using conceptual loops: A science of conceptual systems (SOCS) approach. Cybernetics and Human Knowing, in press .

Wallis, S. E. (2020c). An Analysis of the Systemic Structure of the US Constitution Providing New Insight into the Fragmentation and Conflict Facing US Society and Providing Guidance Towards More Effective Governance.  Systems Research and Behavioral Science (forthcoming).

Wallis, S. E. (2019). The missing piece of the integrative studies puzzle. Interdisciplinary Science Reviews, 44 (3–4), 402–429.

Wallis, S., & Wright, B. (2019). Integrative Propositional Analysis for Understanding and Reducing Poverty. Kybernetes, 48 (6), 1264–1277.

Wallis, S. E., & Valentinov, V. (2017). A limit to our thinking and some unanticipated moral consequences: A science of conceptual systems perspective with some potential solutions. Systemic Practice and Action Research, 30 (2), 103–116.

Wallis, S. E., Wright, B., & Nash, F. D. (2016). Using integrative propositional analysis to evaluate and integrate economic policies of US presidential candidates. White Paper, 16, 1.

Warfield, J. N. (2003). A proposal for systems science. Systems Research and Behavioral Science: The Official Journal of the International Federation for Systems Research, 20 , 507–520.

Weick, K. E. (1989). Theory construction as disciplined imagination. Academy of Management Review, 14 (4), 516–531.

Welch, E., & Wong, W. (1998). Public administration in a global context: bridging the gaps of theory and practice between western and non-western countries. Public Adm Rev, 58 (1), 40–49.

Wong, E. M., Ormiston, M. E., & Tetlock, P. E. (2011). The Effects of Top Management Team Integrative Complexity and Decentralized Decision Making on Corporate Social Performance. Academy of Management Journal, 54 (6), 1207–1228.

Wong, W. (2013). The search for a model of public administration reform in Hong Kong: Weberian bureaucracy, new public management, or something else? Public Adm Dev, 33 (4), 297–310.

Wong, W. (2016). Comparative public policy. In A. Farazmand (Ed.), Global encyclopedia of public administration, public policy, and governance . Cham: Springer.

Wong, W. (2018). Comparative Public Policy. In: Farazmand A. (eds) Global Encyclopedia of Public Administration, Public Policy, and Governance. Springer, Cham

Wright, B., & Wallis, S. E. (2019). Practical mapping for applied research and program evaluation . Thousand Oaks, CA: SAGE.

Download references

Author information

Authors and affiliations.

Public Management and Leadership, Nelson Mandela University, Port Elizabeth, South Africa

Guswin de Wee

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Guswin de Wee .

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

de Wee, G. Comparative Policy Analysis and the Science of Conceptual Systems: A Candidate Pathway to a Common Variable. Found Sci 27 , 287–304 (2022). https://doi.org/10.1007/s10699-021-09782-5

Download citation

Accepted : 08 February 2021

Published : 20 March 2021

Issue Date : June 2022

DOI : https://doi.org/10.1007/s10699-021-09782-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Comparative policy analysis
• Policy analysis
• The science of conceptual systems
• Text-as-data
• Integrative propositional analysis
• Find a journal
• Publish with us
• Track your research

• Library Home
• Library Guides

Public Policy

• Research Methods
• Writing Policy Briefs & Memos
• Public Opinion
• Data & Statistics
• US Govt Documents

SAGE Research Methods

• SAGE Research Methods Core This link opens in a new window Reference guides to conducting social science research & more less... SAGE Research Methods provides material to guide users through every step of the research process. It contains more than 1000 books, reference works, journal articles, and instructional videos by academics from across the social sciences.

Books on Research Methods

Find More Books on Research Methods

• Policy sciences -- Methodology
• Policy sciences -- Handbooks, manuals, etc.
• Qualitative research -- Methodology
• << Previous: US Govt Documents
• Subjects: Public Policy , Social Sciences , Social Services
• Updated: Mar 26, 2024 12:13 PM
• URL: https://guides.lib.uchicago.edu/policystudies
• Report a problem
• Login to LibApps

Open sourcetools

What all policy analysts need to know about data science

Subscribe to techstream, alex engler alex engler former fellow - governance studies , center for technology innovation.

April 20, 2020

• 18 min read

Conversations around data science typically contain a lot of buzzwords and broad generalizations that make it difficult to understand its pertinence to governance and policy. Even when well-articulated, the private sector applications of data science can sound quite alien to public servants. This is understandable, as the problems that Netflix and Google strive to solve are very different than those government agencies, think tanks, and nonprofit service providers are focused on. This does not mean, however, that there is no public sector value in the modern field of data science. With qualifications, data science offers a powerful framework to expand our evidence-based understanding of policy choices, as well as directly improve service delivery.

To better understand its importance to public policy, it’s useful to distinguish between two broad (though highly interdependent) trends that define data science. The first is a gradual expansion of the types of data and statistical methods that can be used to glean insights into policy studies, such as predictive analytics, clustering, big data methods, and the analysis of networks, text, and images. The second trend is the emergence of a set of tools and the formalization of standards in the data analysis process. These tools include open-source programming languages, data visualization, cloud computing, reproducible research, as well as data collection and storage infrastructure.

Perhaps not coincidentally, these two trends align reasonably well with the commonly cited data science Venn diagram. In this diagram, data science is defined as the overlap of computer science (the new tools), statistics (the new data and methods), and critically, the pertinent domain knowledge (in our case, economics and public policy). While it is a simplification, it is still a useful and meaningful starting point. Moving beyond this high-level understanding, the goal of this paper is to explain in depth the first trend, illuminating why an expanded view of data and statistics has meaningful repercussions for both policy analysts and consumers of that analysis.

Traditional evidence-building for policy analysis

Using data to learn about public policy is not at all new. The origins of the social sciences using statistical analysis of observational data goes back at least to the 1950s , and experiments started even further back. Microsimulation models, less common but outsized in their influence, emerged as the third pillar of data-driven policy analysis in the 1970s . Beyond descriptive statistics, this trifecta—experiments, observational statistical analysis, and microsimulation—dominated the quantitative analysis of policy for around 40 years. To this day, they constitute the overwhelming majority of empirical knowledge about policy efficacy. While recent years have seen a substantial expansion in the set of pertinent methods (more on that below), it is still critical to have a strong grasp of experiments, observational causal inference, and microsimulation.

Experiments

Since public policy can’t be conducted in a laboratory, experiments are rare in policy studies. Experiments require random assignment, which for policy means a benefit or program is made available randomly to some people and not to others—hardly a politically popular strategy. Many would also say it is ethically questionable to do this, though randomized experiments have taken firm root in medicine, sacrificing fairness in the short term for progress in the long term. Regardless of the political and ethical barriers, they do happen. Experiments are most often supported by nonprofits or created by an accident of governance, and can produce relatively rigorous results, compared to the other methods discussed here.

Perhaps the most famous experiment in modern public policy is that of the Oregon Medicaid expansion. When Oregon moved to expand access to Medicaid in 2008 (before the Affordable Care Act), the state quickly realized that it could not afford to cover all the individuals eligible under the loosened criteria. Opting to randomly select which residents would be able to receive benefits, Oregon officials created the perfect circumstances for researchers to compare recipients of Medicaid with non-recipients who were otherwise very similar. Professors Katherine Baicker and Amy Finkelstein led the research efforts, resulting in extensive evidence that Medicaid improved some health outcomes and prevented catastrophic medical expenses , while also increasing health-care utilization and costs . Signaling a growing interest in this approach, the recent Nobel Prize in Economics recognized three scholars who have taken experiments (sometimes call randomized control trials, or RCTs) into the developing world to examine how to best tackle global poverty.

Statistical analysis of observational data

Due to the financial and political difficulties that experiments present, they remain rare, and much more research is based on the statistical analysis of observational data. Observational data refers to information collected without the presence of an explicit experiment—it comes from surveys, government administrative data, nonprofit service delivery, and other sources. Usually by obtaining and combining several datasets, researchers look for various opportunities to examine the causal effects of policy changes with statistical methods. These statistical methods, broadly called causal inference statistics (or quasi-experiments), take advantage of differences within populations, or policy changes over time and geography to estimate how effective a service or intervention is.

Individually, the strength of the evidence from a single study is limited. (This is true in any field, and it suggests prudence when changing your beliefs based on results from one study.) However, since observational data is far easier to gather and analyze than experimental data, it is possible to find many opportunities to re-examine the same policy questions. Eventually, it’s possible to examine many papers on the same subject, called a meta-analysis. Meta-analysis of observational studies have convincingly argued that increased school spending improves student outcomes , gun access leads to higher risk of suicide and homicide , and that taxes on sugary beverages are associated with lower demand for those beverages.

Although at times difficult to interpret, this slow accumulation of many observational analyses by different research groups often becomes the most informative and trustworthy source of information about potential policy changes.

Microsimulation

Although microsimulation is a lesser-known type of modeling, it remains a critical one. The news is frequently covered in estimates from microsimulation methods, such as how effective taxes would change under the Tax Cuts and Jobs Act and how many people would lose health insurance under the curtailing of the Affordable Care Act . Even a substantial part of the (in)famous Congressional Budget Office scoring of the cost of federal legislation employs microsimulation.

The Urban Institute-Brookings Institution Tax Policy Center model is perhaps the easiest to understand intuitively. The model starts with a sample of anonymized administrative data from the Internal Revenue Service, which contains lots of information about taxpayers that is specific to each person. (This puts the “micro” in microsimulation.) The model itself then does the same thing as online tax preparation software: It runs through the rules of the tax code and calculates how much this person should be paying in taxes. However, the model contains many different knobs that can be turned and switches that can be flicked, each one changing something about the way the tax code works. By altering some of these inputs, the model creates a simulation, that is, an alternative possible outcome from the real world of tax policy.

These models are highly complex, and usually take years to build. They also require a lot of information about how a set of public policies are currently affecting a population, so the data typically comes from government administration records. However, once they are built, they offer a quick and flexible lens into potential policy changes. In reality, the behavioral consequences—how people and firms react to new policy—are large enough that few experts are ever really convinced that estimates from these models are precisely correct. That said, microsimulation methods can ground policy discussions to reasonable predictions, make assumptions explicit, and give a reasonable sense of what complex and interacting policy changes might do. Compared to letting pundits invent numbers out of thin air, microsimulation offer a dramatically more rigorous approach to estimating policy outcomes.

The expanded methods of data science for policy analysis

Nothing discussed above falls outside the field of data science. These approaches all use data, programming, and statistics to infer meaningful conclusions about the world. Still, the term “data science” has some value, as it connotes a broader set of methods and data types than is traditional to the field of policy analysis. While many of these methods have existed for a long time, the proliferation of new and diverse data sources means this expanded toolkit should be more widely understood and applied by policy analysts. Many of the methods detailed below fall into the field of machine learning, but in this case, that terminology complicates the issue without adding much clarity.

Predictive analytics

There is a growing recognition that many government and nonprofit services can be improved with predictive analytics. In Chicago, predictive models are used to reduce the exposure of young children to lead paint, which has extensive and permanent health consequences. Before this effort, and still in most places across the United States, exposed lead paint is typically only discovered after children fall ill.

The Chicago model uses historical inspection data to find correlations between exposed lead paint and other information (like the age of the buildings, when they were last renovated, if they have been vacant, as well as demographic data). This model can then be used to evaluate the level of risk of lead paint in homes that are going to accommodate newborn children. Using those predictions, the Chicago Department of Public Health can more strategically prioritize lead paint inspections, saving many children from hazardous exposure.

This is a generalizable approach for service providers who have a valuable intervention, limited resources, and uncertainty in where their investments would be most beneficial. As another example, the Center for Data Insights at MDRC—a nonprofit, nonpartisan education and social policy research organization—is exploring how to use prediction modeling to better allocate employment services to former inmates. (Disclosure: I am a data science consultant on this project.) If there is trustworthy historical data and an opportunity to affect who gets an intervention, predictive analytics can be highly beneficial by getting services delivered to those who need it most.

In public policy, subgroups of a larger population can be very important. Some students score highly on tests, while other score poorly. Some people earn a lot of money from their jobs, while others earn very little. However, although it is tempting to think of groups as separated along a single variable, like the examples above, this is infrequently the case. Some people may earn little money from their jobs, but are in fact in graduate school, have highly educated parents with a strong social support system, suggesting that their income potential is quite high. In some cases, they may be more similar to people earning lots of money than those who earn little but do not have those other supports.

Clustering methods allow for the discovery of these underlying groups across many variables that might otherwise remain hidden or avoid our qualitative intuition. The Pew Research Center has demonstrated this by using clustering methods to examine our assumptions about the political spectrum. Pew researchers applied clustering methods to a survey with 23 questions about political opinions. They discovered that the traditional liberal-moderate-conservative spectrum does not effectively compass the many dimensions of political views that Americans hold. Instead, they argued for seven distinct political subgroups . As just one difference, this more nuanced political analysis notes two groups of conservatives: one more socially conservative, but also critical of Wall-Street; and another more socially moderate, pro-immigration, but also pro-Wall Street.

This is closer to how the world really works—subgroups are complex and nothing is unidimensional. It’s imperative to consider how many variables may be interacting to define the most meaningful differences in whatever populations are being analyzed.

Sometimes, though certainly not always, simply having more data enables better or different policy analysis. Over the past 12 years, a joint academic initiative at MIT Sloan and Harvard Business School has been using online prices to measure macroeconomic indicators. By scraping data from the internet, the Billion Prices Project has collected the prices of 15 million items from over a thousand retailers. This massive dataset has enabled them to create measures of inflation in 20 countries, updated on a daily basis. For the sake of comparison, the Bureau of Labor Statistics’ (BLS) Consumer Price Index is monthly. Although there are many challenges to this new approach, it’s worth keeping in mind that the traditional process used by the BLS (government employees surveying or physically examining prices) is far more expensive, complicated by its own problems (e.g., growing survey non-response), and painstakingly slow.

While big data can offer new insights, there are important statistical differences when analyzing big data. Most notably, it is generally harder to get data that accurately represents the whole of a population (like a country or a state). Cloud computing and modern software may easily enable analyzing multi-billion row datasets, but that makes it no easier to know who the data is relevant to. Phone records can illuminate strategies to improve traffic patterns, but does it overlook people without mobile phones? Credit card transactions can reveal lifestyle differences across socio-economic groups, but what could be missing without seeing cash transactions and cash-only consumers? This remains a large problem for effectively using big data that is not originally meant for social science (sometimes called “found data”). As a result, it’s a priority to continue the development of methods that can adjust these datasets to be representative and accurate, especially since this new data can offer so much.

Text analysis

Modern text analysis, or natural language processing, offers new ways to glean meaningful insights into the huge bodies of writing that societies produce. For example, consider the impressions that a community has of its law enforcement officers. Trust in the legitimacy of a police force can lead to more lawful behavior, as well as community collaboration to solve and reduce crimes. However, community impressions of police can be hard to measure. This is why the Urban Institute turned to Twitter. Researchers at Urban’s Justice Policy Center analyzed the sentiment of 65 million tweets, finding spikes in negative sentiment after violent police-citizen interactions. It’s worth nothing that this analysis is affected by the big data considerations detailed above.

In another instance, my colleagues in Brookings’s Metropolitan Policy Program looked for overlapping patterns in the text of job descriptions and AI patents. This allows them to create a quantitative estimate of how good AI might be at various job tasks, and thereby, how potentially automatable those jobs might be. This approach creates a new way to reason about the effects of automation that is less dependent on the qualitative judgement of experts.

Network analysis

In their book, “ Network Propaganda ,” three researchers of Harvard’s Berkman-Klein Center created networks of online media sources, like The New York Times and Fox News. They then measured the sources’ relationships to one another with the hyperlinks in their news content, as well as the social media sharing patterns of their audience. Critically, their research has shown how isolated and self-amplifying far-right media sources have become, leading them to grow more extreme and less tethered to the truth. This is the exact type of insight that network analysis can help deliver: Around whom is the network most dependent? Who is on the fringes and sidelines? How are relationships between actors changing?

While the internet and social media has created huge networks of people, any group of things with relationships to one another can be considered a network and analyzed as such. The states have long been considered laboratories of democracy, where experimental policies can be tested and successful ones shared. This also can be conceived of as a network, with states being connected to one another through the diffusion of similar legislation. Recent research of this kind has provided further evidence of California’s status as a leader in policy innovation. This might be unsurprising, but the same research also highlights Kentucky as the most influential state for the diffusion of public policy from the 1960s until California’s emergence in the mid-1990s.

Image analysis

Image data has proliferated in recent years, originating in everything from cell phones, traffic cameras, smart devices, and even constellations of new satellites. In many parts of the word, especially poor countries, it can be very hard to develop an accurate understanding of poverty—but analyzing image data can help. Unsurprisingly, knowing where impoverished people are is vital to targeting services and investments to help improve their socio-economic outcomes. This is why the World Bank has been developing methods to use high-definition satellite data to create geographically specific measures of poverty, especially relevant to the 57 countries with almost no survey data on poverty. Data science methods can look at satellite imagery and recognize cars, identify roofing materials, distinguish between paved and unpaved roads, and measure building height and density. In turn, these new variables can be used to estimate fairly accurate poverty measures that are substantial improvements over outdated (or nonexistent) estimates.

Satellite data has also been used to monitor international conflicts and as evidence of human rights abuses . Other efforts have proposed using social media photos of political protests to measure their size and degree of violence, though this is likely not ready for implementation. Currently, image analysis is largely limited to facial and object recognition; it is not close to genuine understanding of photos. Still, as imagery proliferates and related modeling techniques improve, this data will offer powerful new ways to examine the state of the world.

Why data science matters to public policy and governance

Evaluating data is becoming a core component of government oversight. The actions of private companies are more frequently in databases than file cabinets, and having that digital information obscured from regulators will undermine our societal safeguards. Government agencies should already be acting to evaluate problematic AI-hiring software and seeking to uncover biases in models that determine who gets health interventions . As algorithmic decision-making becomes more common, it will be necessary to have a core of talented civic data scientists to audit their use in regulated industries .

Even for public servants who never write code themselves, it will be critical to have enough data science literacy to meaningfully interpret the proliferation of empirical research. Despite recent setbacks—such as proposed cuts to evidence-building infrastructure in the Trump administration’s budget proposal—evidence-based policymaking is not going anywhere in the long term. There are already 125 federal statistical agencies, and the Foundations of Evidence Based Policymaking Act, passed early last year, expands the footprint and impact of evidence across government programs.

“Even for public servants who never write code themselves, it will be critical to have enough data science literacy to meaningfully interpret the proliferation of empirical research.”

Further, the mindset of a data scientist is tremendously valuable for public servants: It forces people to confront uncertainty, consider counterfactuals, reason about complex patterns, and wonder what information is missing. It makes people skeptical of anecdotes, which, while often emotionally powerful, are not sufficient sources of information on which to build expansive policies. The late and lauded Alice Rivlin knew all this in 1970, when she published “ Systemic Thinking for Social Action .” Arguing for more rigor and scientific processes in government decision-making, Rivlin wrote a pithy final line: “Put more simply, to do better, we must have a way of distinguishing better from worse.”

How to encourage data-scientific thinking and evidence-based policies

The tools and data to distinguish better from worse are more available than ever before, and more policymakers must know how to use and interpret them. A continued expansion of evidence-based decision-making relies on many individuals in many different roles, adopting practices that encourage data-scientific thinking. Managers in government agencies can hire analysts with a rigorous understanding of data in addition to a background in policy. They can also work to open up their datasets, contributing to Data.gov and the broader evidence-based infrastructure. Grant-making organizations have a critical role, too. They should be mandating an evaluation budget—at least 5% of a grant—to collect data and see if the programs they are funding actually work. When they fund research, it should require replicable research and open-data practices.

Related Content

Alex Engler

January 22, 2020

Darrell M. West

October 4, 2018

Rashawn Ray

February 20, 2020

For policy researchers looking to expand their sense of what is possible, keep an eye on the data science blogs at the Urban Institute and the Pew Research Center , which get into the weeds on how they are using emerging tools to build and disseminate new knowledge. And for current policy analysts who want to deepen their skills, they should consider applying to the Computational Social Science Summer Institute , a free two-week intensive to learn data skills in the context of social problems and policy data. Though much of it is not directly relevant to policy, there is a tremendous amount of online content for self-learners, too. I recommend looking into free online courses and learning to program in R . For those interested in a bigger investment, look to the joint data science and public policy graduate programs, like those at Georgetown University , the University of Chicago , and the University of Pennsylvania .

The Brookings Institution is a nonprofit organization devoted to independent research and policy solutions. Its mission is to conduct high-quality, independent research and, based on that research, to provide innovative, practical recommendations for policymakers and the public. The conclusions and recommendations of any Brookings publication are solely those of its author(s), and do not reflect the views of the Institution, its management, or its other scholars.

Google provides general, unrestricted support to the Institution. The findings, interpretations, and conclusions in this report are not influenced by any donation. Brookings recognizes that the value it provides is in its absolute commitment to quality, independence, and impact. Activities supported by its donors reflect this commitment.

Artificial Intelligence

Governance Studies

Center for Technology Innovation

Artificial Intelligence and Emerging Technology Initiative

Nicol Turner Lee

March 28, 2024

Joseph B. Keller

Cameron F. Kerry

March 27, 2024

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 01 May 2023

What are the core concerns of policy analysis? A multidisciplinary investigation based on in-depth bibliometric analysis

• Yuxue Yang   ORCID: orcid.org/0000-0002-8772-1024 1 , 2 ,
• Xuejiao Tan 1 ,
• Yafei Shi 1 &
• Jun Deng 1 , 2  

Humanities and Social Sciences Communications volume  10 , Article number:  190 ( 2023 ) Cite this article

6864 Accesses

2 Citations

9 Altmetric

Metrics details

• Environmental studies
• Medical humanities
• Social policy

Policy analysis provides multiple methods and tools for generating and transforming policy-relevant information and supporting policy evolution to address emerging social problems. In this study, a bibliometric analysis of a large number of studies on historical policy analysis was performed to provide a comprehensive understanding of the distribution and evolution of policy problems in different fields among countries. The analysis indicates that policy analysis has been a great concern for scholars in recent two decades, and is involved in multiple disciplines, among which the dominant ones are medicine, environment, energy and economy. The major concerns of policy analysts and scholars are human health needs, environmental pressures, energy consumption caused by economic growth and urbanization, and the resulting demand for sustainable development. The multidisciplinary dialog implies the complicated real-world social problems that calls for more endeavors to develop a harmonious society. A global profiling for policy analysis demonstrates that the central policy problems and the corresponding options align with national development, for example, developing countries represented by China are faced with greater environmental pressures after experiencing extensive economic growth, while developed countries such as the USA and the UK pay more attention to the social issues of health and economic transformation. Exploring the differences in policy priorities among countries can provide a new inspiration for further dialog and cooperation on the development of the international community in the future.

Similar content being viewed by others

Expert review of the science underlying nature-based climate solutions

B. Buma, D. R. Gordon, … S. P. Hamburg

Global supply chains amplify economic costs of future extreme heat risk

Yida Sun, Shupeng Zhu, … Dabo Guan

The role of artificial intelligence in achieving the Sustainable Development Goals

Ricardo Vinuesa, Hossein Azizpour, … Francesco Fuso Nerini

Introduction

Social problems are evolving with the rapid development of economy, and the problems mankind is facing and options they choose reflect the developmental demand. Policy is a political action with specific subjects, targets, and strategies in a certain period of time, which primarily aims to create a healthy environment for the development of society (Porter, 1998 ; Lasswell and Kaplan, 1950 ; Yang et al., 2020 ). As for policy analysis, the definition varies a lot. According to William Dunn ( 2015 ), policy analysis is ‘an applied social science discipline, which uses multiple methods of inquiry and argument to produce and transform policy-relevant information that may be utilized in political settings to resolve policy problems.’ Jabal et al. ( 2019 ) defined that policy analysis provides methods and tools for assessing whether a policy is ‘correct and fit for their use’ and supporting policy evolution. Manski ( 2019 ) regarded policy analysis as a shorthand term that describes the process of scientific evaluation for the impact of past public policies and prediction of the potential outcomes of future policies. More generically, policy analysis is aimed to understand who develops and implements certain policies, for whom, by what, with what effects, and what techniques and tools can be used, and so on (Blackmore and Lauder, 2005 ; Collins, 2005 ).

Accordingly, regarding the typology of policy analysis, three categories can be established based on ontology and epistemology (Fig. 1 ) (Bacchi, 1999 ; Colebatch, 2006 ; Jennifer et al., 2018 ): (1) Positivism paradigm. Focusing on policy facts, this orientation of policy analysis aims to identify policy problems and weighting the optimal solution guided by the theory of economic frameworks, basic scientific models, and behavioral psychology through objective analysis. Economic analysis, cost-benefit analysis, quantitative modeling and nudge politics are the most commonly used methods in this orientation (Althaus et al., 2013 ; Jennifer et al., 2018 ); (2) Constructivism paradigm. In this orientation, policy is conceptualized as ‘the interaction of values, interests and resources guided through institutions and mediated through politics’ (Davis et al., 1993 ) rather than a comprehensively rational and linear process in which analysis involves policy agenda setting, policy processes, policy networks and governance, mainly focusing on values, actors and political rationality of policy. Theoretical frameworks, such as multiple stream theory, behavioral psychology and advocacy coalition framework, etc. are typically used in such orientation (Kingdon, 1984 ; Browne et al., 2019 ; Sabatier and Weible, 2014 ); (3) Interpretivism paradigm. This orientation is focused on interpreting how policy problems can be defined or constructed and how the problem framing shapes the possible policy responses (Bardach, 2000 ). A substantial body of research has discussed the theory underlying the problem, framing and governmentality using narrative analysis, discourse analysis, ethnographic methods, etc. (Hajer, 1995 ; Hajer, 2006 ; Martson and Mcdonald, 2006 ). Therefore, a systematic review of policy analysis can present the past and present policy problems of concern and the relevant possible options from an evolutionary perspective.

The framework was organized according to Jennifer et al. ( 2018 ).

The profoundly complex and diversified realistic demands such as equity and sustainability (Akadiri et al., 2020 ), the changes of energy planning (Banerjee et al., 2000 ; Pandey et al., 2000 ; Pandey, 2002 ) and transition of modern markets (Blackman and Wu, 1999 ) have important implication on policy decisions (Munda, 2004 ). A multidisciplinary investigation on policy analysis can provide more reflections on how to develop a harmonious society. Studies have shown that the priority of policy agenda is determined by three key factors: the nature of the issue (Shiffman and Smith, 2007 ), the policy environment (Adams and Judd, 2016 ; Sweileh, 2021 ) and the capabilities of proponents (Shawar and Shiffman, 2017 ). Due to differences in geography, economics, politics and many other aspects, social concerns and policy priorities vary enormously in different countries. In the global context, how countries set policy priorities in different stages of development, and how policy priorities align with the national development remain unknown. So, developing a global profiling for policy analysis can present the differences in core concerns of polices among countries, thus promoting further dialog and cooperation on the development of the international community in the future.

Bibliometric analysis has long been used as a statistical tool to systematically review scientific literature (Hood and Concepcion, 2001 ). A rigorous bibliometric analysis can provide systematic insights into previous publications, which can not only delve into the academic research community of active and influential researchers, but also identify the current research topics, and further explore potential directions for future research (Fahimnia et al., 2015 ). Bibliometrics has been widely applied in a wide range of sectors and specific domains, for example, mapping and visualizing the knowledge progress avenues and research collaboration patterns of cultural heritage (Vlase and Lähdesmäki, 2023 ), analyzing the sub-areas and core aspects of disease (Baskaran et al., 2021 ), visualizing and graphing the evolution of research related to sustainable development goals (Belmonte-Ureña et al., 2021 ), and studying policies, such as agricultural policy (Fusco, 2021 ), medical information policy (Yuxi et al., 2018 ), and science, technology and innovation policy (Zhang et al., 2016 ). However, the research trajectory and focus of policy analysis around the world remain a black box. In the present paper, a bibliometric analysis was performed from three dimensions: time, intensity, and scope, which referred to hot point changes over time, the quantity of research and the core concerns of policy, respectively.

In the present paper, a bibliometric analysis of a large number of studies on historical policy analysis was performed to answer the questions: (1) What core concerns are reflected in the policy analysis and how does these core concerns reflect real-world social problems? (2) How do these core concerns change over time? (3) What are the differences in core concerns among countries and what drives those differences? From an evolutionary perspective, this paper aims to uncover the past and present policy problems of concern and the relevant possible options, thus providing a clue for future policy analysis. The analysis of the evolution and differences in policy problems among countries may provide a view of the development context of different countries and put forward new inspiration and hope for further dialog and cooperation on the development of the international community in the future. Furthermore, another possible key sustainability implication with respect to the core concerns of policy analysis is to provide a reference for exploring the gaps between academic research and policy agenda.

Literature research

In the present study, Web of Science (WOS) Core Collection database was used for data retrieval (Vlase and Lähdesmäki, 2023 ). This research was conducted in four steps. Firstly, articles related to policy analysis were searched to select the most cited ones, which reflect the most influential research and the cutting-edge knowledge over time. MerigÓ et al. ( 2016 ) and Markard et al. ( 2012 ) weighted the most citation in an absolute term that means the total citations of all time. According to Fusco ( 2021 ) and Essential Science Indicators, the most citation was weighted in a relative term, which means the citation number in the publication year. The top 1% papers, compared to other articles in the academic field published in the same publication year, were included in this study following the refining principle of Essential Science Indicators, ensuring that the impact of these articles does not fade with time. Secondly, the selected papers were further screened, and narrowed down to different collected datasets for in-depth analysis according to the results of screening. Thirdly, statistical analysis and network visualization of authorship, organization and geographical distribution, topics and their chronological trends in each dataset were performed using VOSviewer software, which is freely available to construct and visualize bibliometric network (see www.vosviewer.com ) (Van-Eck and Waltman, 2010 ). Lastly, the association between policy analysis and academic articles was explored in different fields.

Dataset construction

Originally, a total of 118,535 articles related to policy analysis were retrieved using the strategy “TS = (policy analysis)”. For further discipline analysis, the most cited articles were selected with the quick filtering toolbar of WOS. Consequently, 1287 most cited papers of policy analysis were included in dataset 1. Then co-citation analysis of journals was performed to provide clues for discipline research (Supplementary Table 2 ). Accordingly, policy analysis-related articles from journals in the medicine field were selected for dataset 2, and 7963 articles were finally included. Similarly, 15,705 articles from journals in the field of environment were included in dataset 3; 6253 articles from journals in the field of energy in dataset 4; 1268 articles from journals in the field of economy in dataset 5; and 2243 articles from multidisciplinary journals in dataset 6. According to Journal Citation Reports of WOS, multidisciplinary journals refer to those journals in which articles involve at least two disciplines, such as Ecological Economics that involves ecology and economics. The search strategy of each database is shown in Table 1 .

Network visualization

Publication information of policy analysis was presented, including publication number, countries and organizations of key players, which reflects the value of and actual needs for policy analysis. Then, VOSviewer was used for network visualization of co-authorship, co-occurrence and citation. Co-authorship analysis for organizations and countries, which met the thresholds identified more than 5 articles for further investigation of the key players’ geographical distributions and their collaboration patterns. Co-occurrence analysis for all keywords based on the frequency of keywords used in the same article was carried out for topic mining (Kern et al., 2019 ). Citation analysis was performed to investigate the citation attributes received by other items. Meaningless or common terms were removed (Zhang and Porter, 2021 ). The research framework is shown in Fig. 2 .

The research framework for multidisciplinary investigation in policy analysis.

Publication information of policy analysis

Firstly, the publication number of policy analysis was determined. A total of 118,535 policy analysis articles were published between 2003 and 2021 (Fig. 3 ), showing a surge in the development of policy analysis with an exponential growth rate of 53.98 and 84.03% in the last 5 years (2017–2021) and 10 years (2012–2021), respectively.

Source : Data was collected from Web of Science (WOS) Core Collection database on the topic (TS) “policy analysis”.

For network construction, 1287 most cited papers were screened. The collaboration network of countries was visualized and illustrated, showing that 112 countries have published the most cited policy analysis articles. As for the co-authorship of countries and organizations, 2286 universities were identified, and 193 of them from 59 countries met the criteria of network analysis, among which the universities from the USA (University of Washington, Harvard University), the UK (University of Oxford, University of Cambridge) and China (University of Chinese Academy of Sciences) had the largest number of links and the strongest willingness to cooperate with other organizations (Fig. 4A, B and Supplementary Table 1 ). The willingness of cooperation not only meets the needs of academic research, but also conforms to the general expectations of the international community. Citation analysis for sources identified 51 journals from five different fields (Fig. 4C and Supplementary Table 2 ), in which environment-related journals accounted for the largest number (e.g., Journal of Cleaner Production, Science of The Total Environment , Global Environmental Change-Human and Policy Dimensions , Transportation Research Part D: Transport and Environment and Environmental Modeling & Software) , followed by medicine-related journals ( The Lancet , JAMA , The Lancet Infectious Diseases , PLOS One and The Lancet Global Health) , the journals of energy science ( Sustainable Cities and Society , Energy Policy , Applied Energy , Renewable Energy and Energy ), the journals of economy ( International Journal of Production Economics and Transportation Research Part A: Policy and Practice ), and then several multidisciplinary journals ( Ecological Economics , Nature , PNAS, Nature Communications and European Journal of Operational Research ).

A Co-authorship analysis for countries; B Co-authorship analysis for organizations; C Citation network; D Co-occurrence network.

In the co-word network of policy analysis, four main clusters were displayed: the blue cluster concerned with environmental policy problems; the green cluster related to medicine (e.g., public health, prevalence and mortality of disease); the red cluster centering policy, such as policy framework, policy systems, and policy implementation; and the yellow cluster mainly concerned with energy (e.g., energy consumption, energy efficiency and electricity generation) (Fig. 4D and Table 2 ). Simultaneously, more details related to real-world social issues were also found, such as the common and core concerns about carbon emission, economic growth, prevalence and mortality of disease. Additionally, management is in the spotlight (e.g., system, framework, efficiency and challenge).

Publication information of policy analysis in different fields

Policy analysis-related articles mainly involved the fields of medicine, environment, energy, economy and multidiscipline. The publication information in different fields was investigated. First, the volume growth trend over time was traced. Generally, a growing number of articles were published annually. The most obvious growth was found in policy analysis in environment, followed by medicine and energy, and the growth in economy and multidiscipline was relatively stable (Fig. 5 ). Specifically, the first increase in the publication number of policy analysis in medicine was seen in 2009, and then a steady growth was maintained, followed by a second acceleration after 2019, which may relate to the pandemic of H1N1 influenza and COVID-19, respectively (WHO, 2012 ; Wouters et al., 2021 ). A great growth in environmental policy analysis was observed after 2015, and a linear growth after 2017. In energy policy analysis, the first increase occurred in 2009, reaching a peak in 2013, followed by a second increase in 2016, reaching another peak in 2020. Then the publication information about organizations and countries was explored. The top five countries and institutions with the largest number of policy analysis articles in different fields are presented in Supplementary Table 3 . The results showed that the USA, the UK and China attached great importance to policy analysis in all of these fields.

Publication dynamics of policy analysis-related articles in the fields of medicine, environment, energy, economy and multidiscipline between 2003 and 2021.

Policy analysis in the field of medicine

A total of 8381 organizations from 177 countries contributed to medical policy analysis. Further investigation showed that universities from the UK (e.g., University of London, London School of Hygiene & Tropical Medicine and University College London), the USA (e.g., Harvard University and University of California San Francisco), Canada (e.g., University of Toronto) and Australia (e.g., University of Melbourne, University of Sydney) contributed the most to medical policy analysis with the greatest willingness to collaborate both domestically and internationally. By contrast, Chinese universities, such as Peking University, University of Chinese Academy of Sciences and Zhejiang University, were more prone to domestic collaboration (Fig. 6A, B ).

A Co-authorship analysis for countries; B Co-authorship analysis for organizations; C Co-occurrence network; D Overlay network.

Co-occurrence analysis of keywords showed that of the 16,719 keywords identified from 7963 retrieved items, 1778 keywords met the threshold. In addition to the three core topics “medicine”, “policy” and “health” (e.g. health policy, public health), the mortality, prevalence, risk factors as well as prevention of diseases have been the key focus of medical policies. Additionally, the issues of children and adolescents, such as physical activity, overweight and childhood obesity, have also attracted medical scientists and policy analysts. Figure 6D shows the average annual overlay network of keywords. The most recent concerns are the prevalence of COVID-19 and relevant topics associated with SARS-CoV-2 and coronavirus. Moreover, sex-specific mortality, life satisfaction and affordable care act are also the hot topics in recent years (Fig. 6C, D ).

Policy analysis in the field of environment

Co-authorship analysis showed that 9060 organizations from 160 countries contributed to environmental policy analysis, among which universities from China played a key role, especially University of Chinese Academy of Sciences, Tsinghua University, Beijing Normal University, North China Electric Power University and Beijing Institute of Technology (Fig. 7A, B and Supplementary Table 3 ). Of the 44,213 keywords in retrieved 1 5705 articles related to environmental policy analysis, 3638 met the threshold of keyword co-occurrence analysis. The co-word network showed that apart from the words with vague meanings such as “policy”, “impact” and “management”, “carbon emission”, “climate change” and “sustainability” were the most visible in the network. Note that the terms like “energy”, “economic growth” and “urbanization” were also easy to notice (Fig. 7C ). The analysis for the average annual overlay showed that “kyoto protocol”, “acid deposition” and “policy development”, etc. were earlier terms, while “plastic pollution”, “Cross-Sectionally Augmented Autoregressive Distributed Lag” and “population structure”, though lightly weighted, were the most recent ones. The color of overlay network visualization of environmental policy analysis appeared to be yellow, indicating that environmental problems have attracted researchers all over the world in past decades (Fig. 7D ). The abovementioned results demonstrated the positive attitude of policy analysts and indicated a shift of their attention over time, possibly due to the evolution of environmental problems.

Policy analysis in the field of energy

The collaboration network showed that 3668 organizations from 117 countries performed policy analysis in energy. The top five organizations were Tsinghua University, University of Chinese Academy of Sciences, Xiamen University, North China Electric Power University and Beijing Institute of Technology, all of which showed strong willingness to collaborate both domestically and internationally. The network showed that there was complex knowledge interaction and flow in the citation of energy policy analysis (Fig. 8A, B ). Of the 15,027 keywords in retrieved 6253 articles, 1225 met the threshold. Co-occurrence network (Fig. 8C ) revealed that policy analysis in energy was primarily focused on the demand for renewable energy (such as “wind power”, “solar power”, “bioenergy”) due to emission (e.g. “carbon emission”, “greenhouse gas emission”) and energy consumption. The terms “restructuring”, “discount rates” and “kyoto protocol” were early noticed by researchers, and the analysis of kyoto protocol was performed earlier in energy than that in ecology. Then, “green power”, “green certificates” and “energy policy analysis” gradually came into the eyes of analysts. Similarly, the prevalence of COVID-19 was the greatest concern of energy policy analysts, followed by “energy communities” and “renewable energy consumption” (Fig. 8D ).

Policy analysis in the field of economy

1144 organizations from 67 countries were found to contribute almost the same to policy analysis in economy. Hong Kong Polytechnic University, Delft University of Technology, University of Leeds, Rensselaer Polytechnic Institute and University of Sydney had the largest number of publications. Hong Kong Polytechnic University, Delft University of Technology, University of British Columbia, University of Sydney and Rensselaer Polytechnic Institute had the highest collaboration (Fig. 9A, B ). Of the 5970 keywords in retrieved 1268 papers, 395 met the threshold. The co-word network showed that in addition to the general words frequently used in articles (e.g. “policy”, “impact”, “system”), the specific words reflecting the most common topics for policy problem of economy were “transport” (associated with vehicles, public transport, travel behavior, etc.), “supply chain” (related to supply chain management, supply chain coordination, green supply chain, etc.), and “inventory” (related to the model, control and system of inventory, etc.) (Fig. 9C ). The overlay network analysis showed that economic policy analysts had an early interest in inventory-related topics and the issue of supply chain management, but has been concerned with the sustainability of supply chain management only in recent years. Additionally, topics like “circular economy”, “life-cycle assessment”, “industry 4.0” and “automated vehicles” also attracted scholars’ attention. (Fig. 9D ).

Policy analysis in multidiscipline

In the co-authorship network, universities such as Stanford University, University of Chinese Academy of Sciences, University of Maryland, University of California, Berkeley and University of Cambridge had the most publications and a high collaboration. University of California Irvine had fewer publications but relatively higher link, showing that this university was strongly willing to cooperate with other organizations (Fig. 10A, B ). Of the 9467 keywords in retrieved 2243 articles, 648 met the threshold. This multidisciplinary research revealed the relationship between economy, environment and energy. However, there were obstacles to extend the relationship between them. Co-word network demonstrated that the policy analysis articles published on the multidisciplinary journals were mainly focused on the topics of “climate change”, “sustainability” and “inventory”. The term “climate change” is mainly related to issues of environmental resources (e.g., land use, deforestation, biodiversity), greenhouse gas emission (especially carbon emission) and energy consumption. The term “sustainability” is mainly connected with the relationship between environmental resources and economic growth. In addition to COVID-19, the terms “big data” and “circular economics” were on the cut edge (Fig. 10C, D ).

Policy analysis aims to understand what is the governments’ focal point, investigate why and how governments issue policies, evaluate the effects of certain policies (Browne et al., 2019 ), and reflect political agenda driven by social concerns or international trends (Kennedy et al., 2019 ). In this study, a bibliometric analysis of a large number of publications on historical policy analysis was carried out to explore the policy problems of concern and the relevant possible options from an evolutionary perspective, and provide a guide for future research. From 2003 to 2021, the number of publications on policy analysis grew exponentially. Before 2011, little attention was paid to policy analysis, but in recent decades, more importance has been attached to policy analysis around the world due to increasingly prominent social problems, especially the human health needs, degradation of environment, energy consumption and the relationship between economy, energy and environment.

From the perspective of global visibility, the policy analysis in medicine has received increasing attention from scholars from 8381 organizations of 177 countries, indicating that health problems, though not numerically dominant, have the widest coverage. Among these countries, the USA, the UK, Australia, Canada and China are the major contributors. The developed countries, such as the USA, the UK, Canada and Australia, have strongly supported addressing complex public health issues by developing effective policy responses (Moore et al., 2011 ; Atkinson et al., 2015 ). Typically, they spend the most on health, with 12318, 5387, 5905 and 5627 dollars per capital, respectively, while the developing countries spend relatively less, such as 894 dollars per capital in China and 231 dollars per capital in India (OECD, 2022 ). Great attempts have been made to analyze the burden of prevalence and mortality of diseases such as cancer, cardiovascular diseases and diabetes both globally and regionally (Yusuf et al., 2020 ; Rudd et al., 2020 ; Kearney et al., 2005 ). Other health issues of women, children and adolescents have been monitored and measured for years in many countries that respond to the Countdown to 2030 (Countdown to 2030 Collaboration, 2018 ). In addition, the worldwide outbreak of epidemics such as H1N1 influenza and COVID-19 pandemic has caused excess mortality and enormous social and economic costs all over the world, which greatly affect social policy and reveal the fragility of health systems to shocks (Wouters et al., 2021 ; Chu et al., 2020 ). By analyzing the global burden of disease, scholars have recommended policy-makers to give priority to the prevention and management of relevant diseases (Kearney et al., 2005 ).

Environmental policy analysis involving 15,705 articles has attracted largest attention from policy analysts and scientists. Greenhouse gas emission (mainly carbon emission) resulting in climate change and environmental degradation remains to be the most threatening and urgent issue, and has attracted attention of governments and the society (Tang et al., 2021 ; Ahmad et al., 2019 ). Different countries issued different climate policies aiming to reduce greenhouse gas emissions. The Kyoto protocol, ratified by 180 countries, committed to reduce the GHG emissions by 5% by 2012, compared with the 1990 emission levels (Kuosmanen et al., 2009 ). In the EU climate policy framework in 2014, the carbon emissions were projected to reduce by 40% by 2030, and by 80% by 2050 (European Council, 2014 ). The relationship between urbanization and environmental pressure was observed in the present research. During urbanization, the consumption of resources such as land, water and fuel has increased significantly, causing serious ecological pressure such as climate change, loss of biodiversity, land erosion and pollution. With the acceleration of economic growth and social commercialization, urbanization further increases the demands for housing, food, transportation, electricity and so on, which in turn aggravates the ecological pressure because of natural resource consumption, climate change, over-extraction and pollution (Ahmed et al., 2019 ; Wang et al., 2019 ). Hence, urbanization policies with restrictions on unplanned urban sprawl are under the way (Ahmed et al., 2020 ).

Energy is another big agenda for policy analysis. The close connection between energy and emission has been presented noticeably in this study. Governments have come to a consensus that there should be greater balance between ecological purity, energy supply and economic well-being if a country strives for healthy and sustainable economic development (Alola and Joshua, 2021 ). New environmental policies should be designed to control environmental pollution through reducing pollutant emissions and sustaining economic growth, and should be incorporated into governments’ macro policies (Halicioglu, 2009 ). Transformation of energy sector was on agenda to meet the ambitious goals (Cong, 2013 ). The UK, the USA and China are the global leaders in reducing actual emissions and increasing energy supply. In the USA, the shale revolution brought global attention to energy supply and remains to be a driving force for energy policies. Low-cost shale gas combined with the policy support for renewables have notably reduced CO 2 emissions over the past decades. Environmental deregulation is another central focus, which may affect the trajectory of greenhouse gas emission (International Energy Agency, IEA, 2019a , 2019b ). In the UK, the policy objectives of actual emission reduction, carbon budgets setting and investment in energy technology and innovation reflect the ambition for decarbonization (IEA, 2019a , 2019b ). As is known, China’s GDP grows rapidly, which has multiplied more than 170 times since the founding of the People’s Republic of China 73 years ago. However, the extensive economic growth mode depending on the primary and secondary industries has put high pressure on environment, such as large amounts of consumption and pollution (He et al., 2016 ; Yue et al., 2021 ; Yu and Liu, 2020 ). Data showed that the greenhouse gas emission (OECD, 2020 ) and air pollution exposure (OECD, 2022 ) in China have been far higher than those in other countries for a long time, posing great challenges to both the government and scholars. A specific policy package, such as the “Atmosphere Ten Articles”, “Soil Ten Plan” and “Water Ten Plan” from 2013 to 2016, and the “Regulation on the Implementation of the Environmental Protection Tax Law of the People’s Republic of China” in 2017, has been issued by Chinese government, aiming to improve the ecological environment. Furthermore, goals for renewable energy production were also set by scholars. Jacobson suggested that wind, water and sunlight energy should be produced by 2030, and then replace the existing energy by 2050 (Jacobson and Delucchi, 2011 ), while Lund proposed that renewable energy (the combination of biomass with wind, wave and solar) should account for 50% by 2030, and 100% by 2050 (Lund and Mathiesen, 2009 ). However, it remains unclear how many countries can achieve their stated goals. Numerous studies have shown the efforts of governments and scholars to transform the resource and energy usage-driven economic expansion to sustainable development.

From the economics perspective, the environmental Kuznets curve (EKC) hypothesis demonstrates the relationship between environmental quality and economic output, which has been proved by empirical studies (Fodha and Zaghdoud, 2010 ; Saboori et al., 2012 ). Additionally, the relationship between economic growth and energy consumption has also been confirmed (Shahbaz et al., 2015 ). In recent years, countries have been facing the challenge of economic structural transformation. The mode of economic growth that relies on the consumption of natural resource and waste disposal seems increasingly outdated (McDowall et al., 2017 ). Circular economy, a new mode for reconciling environmental and economic imperatives, has come into the public eye and appears to meet the common vision of sustainable development. With the increase of requirements of sustainable development and circular economy, greening of supply chain management also faces challenges, including inventory management, mode of transportation, life-cycle assessment and coordination with other areas (Ghosh and Shah, 2012 ; Ghosh and Shah, 2015 ). Thus, providing support for green supply chain supplier deserves the attention from policy-makers and practitioners.

Key findings

(1) Policy analysis has been a great concern of scholars for many years and has attracted increasing attention year by year, which reflects the value of and actual needs for policy analysis. (2) The world is facing common problems, which requires attention and efforts of the whole world, and a more harmonious social development such as the management of epidemics and complex disease, environmental-friendly development, green energy production and transformation from resource and energy usage-driven economic expansion to sustainable development is on the way. (3) Global profiling for policy analysis demonstrates that the central policy problems align with national development, which inspires further dialog and cooperation on the development of the international community in the future.

Limitations

This study has limitations. First, keywords cannot fully reflect the essential intent of an article although they are the key points of a study. Therefore, using keywords as an element for bibliometric analysis is far from enough. Second, this paper deals with academic research of policy analysis, but whether it is fully consistent with the policy agenda is unexplored. Moreover, we have shown the correlations between different phenomena, but the underlying mechanism remains indefinable.

Data availability

The datasets analyzed during the current study are available in the Dataverse repository ( https://doi.org/10.7910/DVN/XZMVMN ).

Adams B, Judd K (2016) 2030 agenda and the SDGs: indicator framework, monitoring and reporting. March 18. https://www.cadtm.org/2030-Agenda-and-the-SDGs-indicator

Ahmad M, Khan Z, Rahman ZU et al. (2019) Can innovation shocks determine CO 2 emissions (CO 2 e) in the OECD economies? a new perspective. Econ Innov New Technol 1:1–21

Google Scholar  

Ahmed Z, Wang Z, Ali S (2019) Investigating the non-linear relationship between urbanization and CO2 emissions: an empirical analysis. Air Qual Atmos Health 12(8):945–953

Article   CAS   Google Scholar  

Ahmed Z, Zafar MW, Ali S (2020) Linking urbanization, human capital, and the ecological footprint in G7 countries: an empirical analysis. Sustain Citi Soc 55:1–11

Akadiri SS, Alola AA, Bekun FV et al. (2020) Does electricity consumption and globalization increase pollutant emissions? Implications for environmental sustainability target for China. Environ Sci Pollut Res 27(20):25450–25460

Alola AA, Joshua U (2021) Carbon emission effect of energy transition and globalization: inference from the low-, lower middle-, upper middle-, and high-income economies. Environ Sci Poll Res 30:38276–38286

Althaus C, Bridgman P, David G (2013) The Australian policy handbook. Allen & Unwin, Sydney, p 137–153

Atkinson JA, Page A, Wells R et al. (2015) A modelling tool for policy analysis to support the design of efficient and effective policy responses for complex public health problems. Implement Sci 10(1):26

Article   PubMed   PubMed Central   Google Scholar  

Bacchi C (1999) Women, policy and politics: the construction of policy problems. Sage, London

Book   Google Scholar  

Bardach E (2000) A practical guide for policy analysis, 2nd ed. Chatham House, New York, pp. 2–46

Baskaran S, Agarwal A, Leisegang K et al. (2021) An in-depth bibliometric analysis and current perspective on male infertility research. World J Mens Health 39(2):302–314

Article   PubMed   Google Scholar  

Banerjee R, Pandey R, Bhure A et al. (2000) Electricity demand estimation for village electrification. National Renewable Energy Conference, Indian Institute of Technology, Powai, Mumbai, India

Belmonte-Ureña LJ, Plaza-Úbeda JA, Vazquez-Brust D et al. (2021) Circular economy, degrowth and green growth as pathways for research on sustainable development goals: a global analysis and future agenda. Ecol Econ 185(19):107050

Article   Google Scholar  

Blackman A, Wu X (1999) Foreign direct investment in China’s power sector: trends, benefits and barriers. Energy Policy 27(12):695–711

Blackmore J, Lauder H (2005) Researching policy. In: Somekh B, Lewin C (eds) Res methods in the Soc Sci. Sage, London

Browne J, Coffey B, Cook K et al. (2019) A guide to policy analysis as a research method. Health Promot Int 34(5):1032–1044

Chu DK, Akl EA, Duda S et al. (2020) Physical distancing, face masks, and eye protection to prevent person-to-person transmission of SARS-CoV-2 and COVID-19: a systematic review and meta-analysis. Lancet 395(10242):1973–1987

Article   CAS   PubMed   PubMed Central   Google Scholar  

Colebatch H (2006) Accounting for policy in Australia. Public Policy 1(1):37–51

Collins T (2005) Health policy analysis: a simple tool for policy makers. Public Health 119(3):192–196

Article   MathSciNet   CAS   PubMed   Google Scholar  

Cong RG (2013) An optimization model for renewable energy generation and its application in China: A perspective of maximum utilization. Renew Sustain Energy Rev 17:94–103

Countdown to 2030 Collaboration (2018) tracking progress towards universal coverage for reproductive, maternal, newborn, and child health. Lancet 389(10129):1538–1548

Davis G, Wanna J, Warhurst J et al. (1993) Public policy in Australia. Allen & Unwin, St Leonards, p 15

Dunn WN (2015) Public policy analysis. Routledge, Evanston, IL, USA

European Council (2014) European Council 23/24 October 2014—Conclusions (EUCO 169/14). European Council, Brussels

Fahimnia B, Sarkis J, Davarzani H (2015) Green supply chain management: A review and bibliometric analysis. Int J Prod Econ 162:101–114

Fodha M, Zaghdoud O (2010) Economic growth and pollutant emissions in Tunisia: An empirical analysis of the environmental Kuznets curve. Energy Policy 38(2):1150–1156

Fusco G (2021) Twenty years of common agricultural policy in Europe: a bibliometric analysis. Sustainability 13(19):1–15

Ghosh D, Shah J (2012) A comparative analysis of greening policies across supply chain structures. Int J Prod Econ 135(2):568–583

Ghosh D, Shah J (2015) Supply chain analysis under green sensitive consumer demand and cost sharing contract. Int J Prod Econ 164:319–329

Hajer M (1995) The politics of environmental discourse: ecological modernization and the policy process. Clarendon Press, Oxford

Hajer M (2006) Doing discourse analysis: Coalitions, Practices, Meaning. In: Brink van den, M. and Metze, T. (eds) Words matter in policy and planning: discourse theory and method in the social sciences.Koninklijk Nederlands Aardrijkskundig Genootschap, Utrecht

Halicioglu F (2009) An econometric study of CO 2 emissions, energy consumption, income and foreign trade in Turkey. Energy Policy 37(3):1156–1164

Article   MathSciNet   Google Scholar  

He C, Han L, Zhang RQ (2016) More than 500 million Chinese urban residents (14% of the global urban population) are imperiled by fine particulate hazard. Environ Pollut 218:558–562

Article   CAS   PubMed   Google Scholar  

Hood WW, Concepcion WS (2001) The literature of bibliometrics, cientometrics, and informetrics. Scientometrics 52(2):291–314

International Energy Agency (2019a) Energy policies of IEA countries, The United States

International Energy Agency (2019b) Energy policies of IEA countries, The United Kingdom

Jabal AA, Davari M, Bertino E et al. (2019) Methods and tools for policy analysis. ACM Comput Surv 51(6):1–35

Jacobson MZ, Delucchi MA (2011) Providing all global energy with wind, water, and solar power, Part I: Technologies, energy resources, quantities and areas of infrastructure, and materials. Energy Policy 39(3):1154–1169

Jennifer B, Brian C, Kay C et al. (2018) A guide to policy analysis as a research method. Health Promot Int5:5

Kearney PM, Whelton M, Reynolds K et al. (2005) Global burden of hypertension: analysis of worldwide data. Lancet 365(9455):217–223

Kennedy C, O’Reilly P, O’Connell R et al. (2019) Integrative review: identifying the evidence base for policy making and analysis in health care. J Adv Nurs 75(2):1–15

Kern F, Rogge KS, Howlett M (2019) Policy mixes for sustainability transitions: New approaches and insights through bridging innovation and policy studies. Res Policy 48(10):103832

Kingdon J (1984) Agendas, alternatives, and public policies. Little, Brown, Boston

Kuosmanen T, Bijsterbosch N, Dellink R (2009) Environmental cost–benefit analysis of alternative timing strategies in greenhouse gas abatement: a data envelopment analysis approach. Ecol Econ 68(6):1633–1642

Lasswell HD, Kaplan A (1950) Power and society: a framework for political inquiry. Yale University Press, New Haven, p 71

Lund H, Mathiesen BV (2009) Energy system analysis of 100% renewable energy systems-the case of Denmark in years 2030 and 2050. Energy 34(5):524–531

Manski CF (2019) Communicating uncertainty in policy analysis. Proc Natl Acad Sci USA 116(16):7634–7641

Article   ADS   CAS   PubMed   Google Scholar  

Markard J, Raven R, Truffer B (2012) Sustainability transitions: an emerging field of research and its prospects Research Policy 4:955–967

Martson G, Mcdonald C (2006) Analyzing social policy: a governmental approach. Edward Elgar, Cheltenham, pp. 205–222

McDowall W, Geng Y, Huang B et al. (2017) Circular economy policies in China and Europe. J Ind Ecol 21(3):651–661

MerigÓ JM, Rocafort A, Aznar-AlarcÓN JP (2016) Bibliometric overview of business & economics research. J Bus Econo Manag 17(3):397–413

Munda G (2004) Social multi-criteria evaluation: methodological foundations and operational consequences. Eur J Operat Res 158(3):662–677

Article   MathSciNet   MATH   Google Scholar  

Moore G, Redman S, Haines M, Todd A (2011) What works to increase the use of research in population health policy and programs: a review. Evid Policy 7(3):277–305

OECD (2020) Air and GHG emissions (indicator) (accessed on 24 Nov 2022)

OECD (2022) Air pollution exposure (indicator) (accessed on 24 Nov 2022)

OECD (2022) Health spending (indicator) (accessed on 30 Nov 2022)

Pandey R, Banerjee R, Phulluke S et al. (2000) Framework for design and evaluation of electric supply options for a rural area. National Renewable Energy Conference, Indian Institute of Technology, Powai, Mumbai, India

Pandey R (2002) Energy policy modelling: agenda for developing countries. Energy Policy 30(2):97–106

Porter ME (1998) The Competitive Advantage of Nations, Free Press

Rudd KE, Johnson SC, Agesa KM et al. (2020) Global, regional, and national sepsis incidence and mortality, 1990–2017: analysis for the Global Burden of Disease Study. Lancet 395(10219):200–211

Sabatier P, Weible C (2014) Theories of the policy process. Westview Press, Philadelphia

Saboori B, Sulaiman J, Mohd S (2012) Economic growth and CO 2 emissions in Malaysia: a cointegration analysis of the Environmental Kuznets Curve. Energy Policy 51:184–191

Shahbaz M, Loganathan N, Zeshan M et al. (2015) Does renewable energy consumption add in economic growth? An application of auto-regressive distributed lag model in Pakistan. Renew Sustain Energy Rev 44:576–585

Shawar YR, Shiffman J (2017) Generation of global political priority for early childhood development: the challenges of framing and governance. Lancet 389(10064):119–124

Shiffman J, Smith S (2007) Generation of political priority for global health initiatives: a framework and case study of maternal mortality. Lancet 370(9595):1370–1379

Sweileh WM (2021) Global research activity on elder abuse: a bibliometric analysis (1950-2017). J Immigr Minor Health 23(1):79–87

Tang K, Wang M, Zhou D (2021) Abatement potential and cost of agricultural greenhouse gases in australian dryland farming system. Environ Sci Pollut Res 28(17):21862–21873

Van-Eck NJ, Waltman L (2010) Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics 84(2):523–538

Vlase I, Lähdesmäki T (2023) A bibliometric analysis of cultural heritage research in the humanities: the web of science as a tool of knowledge management. Humanit Soc Sci Commun 10:84

Wang ZH, Ahmed Z, Bin Z (2019) The nexus between urbanization, road infrastructure, and transport energy demand: empirical evidence from Pakistan. Environ Sci Pollut Res Int 26(34):34884–34895

WHO (2012) Report of the WHO Pandemic Influenza A(H1N1) Vaccine Deployment initiative. https://www.who.int/influenza_vaccines_plan/resources/h1n1_deployment_report.pdf (accessed Dec 19, 2020)

Wouters OJ, Shadlen KC, Salcher-Konrad M et al. (2021) Challenges in ensuring global access to COVID-19 vaccines: production, affordability, allocation, and deployment. Lancet 397(10278):1023–1034

Yang C, Huang C, Su J (2020) A bibliometrics-based research framework for exploring policy evolution: a case study of China’s information technology policies. Technol Forecast Soc Change 157:120116

Yu Y, Liu HR (2020) Economic growth, industrial structure and nitrogen oxide emissions reduction and prediction in China. Atmos Pollut Res 11(7):1042–1050

Yue H, He C, Huang Q et al. (2021) Stronger policy required to substantially reduce deaths from PM2.5 pollution in China. Nat Commun 11(1):1462

Article   ADS   Google Scholar  

Yusuf S, Joseph P, Rangarajan S et al. (2020) Modifiable risk factors, cardiovascular disease, and mortality in 155 722 individuals from 21 high-income, middle-income, and low-income countries (PURE): a prospective cohort study. Lancet 395(10226):795–808

Yuxi J, Wei W, Jun L et al. (2018) Trends and characteristics of global medical informatics conferences from 2007 to 2017: a bibliometric comparison of conference publications from Chinese, American, European and the global conferences. Comput Method Prog Biomed 166:19–32

Zhang YPorter AL, Cunningham S, Chiavetta D, Newman N (2021) Parallel or intersecting lines? Intelligent bibliometrics for investigating the involvement of data science in policy analysis. IEEE Trans Eng Manag 68(5):1259–1271

Zhang Y, Zhang G, Chen H et al. (2016) Topic analysis and forecasting for science, technology and innovation: methodology with a case study focusing on big data research. Technol Forecast Soc Change 105(2016):179–191

Download references

Acknowledgements

This work was financially supported by Special Project on Innovation and Generation of Medical Support Capacity (NO. 20WQ008) and Chongqing Special Project on Technological Foresight and Institution Innovation (NO. cstc2019jsyj-zzysbAX0037). We are also deeply grateful to prof. Ying Li and prof. Xia Zhang for their constructive suggestions to improve the manuscript.

Author information

Authors and affiliations.

Army Medical University, Chongqing, China

Yuxue Yang, Xuejiao Tan, Yafei Shi & Jun Deng

General Hospital of Xinjiang Military Command, Urumqi, China

Yuxue Yang & Jun Deng

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Jun Deng .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Informed consent

Additional information.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Yang, Y., Tan, X., Shi, Y. et al. What are the core concerns of policy analysis? A multidisciplinary investigation based on in-depth bibliometric analysis. Humanit Soc Sci Commun 10 , 190 (2023). https://doi.org/10.1057/s41599-023-01703-0

Download citation

Received : 01 September 2022

Accepted : 18 April 2023

Published : 01 May 2023

DOI : https://doi.org/10.1057/s41599-023-01703-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Utility Menu

Harvard Ph.D. Program in Health Policy

• Methods for Policy Research

Concentration Chairs:   Mary Beth Landrum  and  Michael McWilliams  (on sabbatical for the 2023-2024 academic year)

Training in the MPR concentration ( formerly called Evaluative Science & Statistics ) will position students to conduct rigorous, policy-relevant research. The concentration includes intensive methodological training that draws from several quantitative fields, including statistics, econometrics, and epidemiology, and also offers opportunities to learn qualitative methods. Students in the MPR concentration will also develop proficiency in experimental and quasi-experimental research designs. The MPR concentration does not require training in a specific discipline of social science, but students are allowed to use two of their distributional requirements to acquire some disciplinary training (e.g., in economics or sociology), and many students pursue additional coursework in a substantive area of interest or advanced levels of statistics. By virtue of this training, MPR students develop a multi-disciplinary toolkit that borrows from the strengths of different quantitative sciences and supports novel applications to health-related questions. Graduates are able to communicate and collaborate effectively with statisticians, economists, epidemiologists, and clinician-investigators, as they lead their independent work.  The concentration’s broad methodological training flexibly supports scholarship on a wide range of topics, including common areas of interest in health services research related to health care access, quality, costs, and disparities, but also questions focused on clinical decision-making, behavior (e.g., of patients, physicians, or organizations), social determinants of health, and social programs with implications for health.

Methods for Policy Research Curriculum Guide

• Curriculum Overview
• Decision Sciences
• Political Analysis
• Core Course in Health Policy
• Policy Areas
• Joint Degrees

• Download PDF
• Share X Facebook Email LinkedIn
• Permissions

Same Old Challenges in Subgroup Analysis—Should We Do More About Methods Implementation?

• 1 Division of Clinical Epidemiology, Department of Clinical Research, University Hospital and University of Basel, Basel, Switzerland
• 2 MTA–PTE Lendület Momentum Evidence in Medicine Research Group, Medical School, University of Pécs, Pécs, Hungary
• 3 Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada
• 4 Department of Medicine, McMaster University, Hamilton, Ontario, Canada
• Original Investigation Differential Treatment Effects of Subgroup Analyses in Oncology Trials, 2004-2020 Alexander D. Sherry, MD; Andrew W. Hahn, MD; Zachary R. McCaw, PhD; Joseph Abi Jaoude, MD; Ramez Kouzy, MD; Timothy A. Lin, MD, MBA; Bruce Minsky, MD; C. David Fuller, MD, PhD; Tomer Meirson, MD; Pavlos Msaouel, MD, PhD; Ethan B. Ludmir, MD JAMA Network Open

Sherry et al 1 considered the methodological quality of subgroup analyses reported in 379 oncology trials. The authors explored a number of fundamental problems with these analyses: in forest plots, use of linear rather than logarithmic scales and failure to present overall pooled estimates; failure to conduct and highlight tests of interaction; and readiness to make unwarranted inferences regarding the credibility of postulated subgroup effects. They reported a distressingly high frequency of each of these problems. 1

The study thus adds to an already large body of subgroup literature finding that authors’ presentations of forest plots are often suboptimal. It also adds indirect evidence for the often-suspected abuse of subgroup analyses for post hoc data dredging in search of interesting findings 2 : trials that failed to show a significant effect reported more subgroup analyses. 1 The primary message of the study is, therefore, that the methodological limitations of subgroup analyses and misleading inferences in oncology trials remain the same as those that the methods community has been discussing for more than 3 decades. 3 , 4 We will briefly discuss these key limitations in subgroup analyses.

Failure to Design and Interpret Subgroup Analyses in the Light of Prior Knowledge

In only 10% of the subgroup claims did the primary study authors consider prior evidence or prespecify hypotheses. 1 In failing to do so, they fail to inform their audience whether the claim is consistent with prior knowledge. This information is critical for evaluating the credibility of the authors’ subgroup claim.

Not Reporting the Essential Statistic

The test of interaction is the single most crucial statistic for subgroup analysis: it tells us the extent to which chance can explain the apparent difference in effect sizes across subgroups. Sherry et al 1 found that only 17% of the trials reported a P value or estimate of interaction, leaving evidence users in the dark regarding the extent to which subgroup differences were compatible with random error (and they mostly are 5 ).

Ignoring Multiplicity Issues

The more hypotheses one tests, the more likely one will capitalize on a chance finding and then claim a spurious subgroup effect. With an average number of 9 subgroup analyses per oncology trial, 1 these subgroup analyses run a high risk of being misled—and misleading their audience—by the play of chance.

Poor Statistical Practice of Dichotomizing Continuous Effect Modifiers

For continuous variables such as age, trial authors set thresholds and reported effects on patients above and below their chosen threshold. A superior alternative would be to examine whether effects differ across the range of the continuous variable. The choice of a single threshold results in a high risk of further capitalizing on the play of chance, especially when choosing a threshold that maximizes apparent difference between groups, and weakening the analysis through discarding the extra information that the continuum provides.

And finally, as Sherry et al 1 confirmed, none of the included trials applied credibility criteria for subgroup effects that have been available since the 1990s. 3 , 4 , 6 We recently refined these criteria in the first formal rigorously developed instrument for judging the credibility of subgroup effects. 7

What can we do to improve the methodological quality of subgroup analyses? Learning from the past, publishing more commentaries, meta-studies, simulation studies, and guidance papers about the challenges and solutions in subgroup analysis, even if they are largely repetitive, seems unlikely to help.

One strategy, a systematic greater focus on methods implementation, has thus far failed to attract the attention it might deserve. A small number of pioneering studies have started to identify and better understand barriers to methods implementation 8 - 10 identify and test strategies for better methods implementation, 11 and raise the issue of whether the principles of implementation science could work in the methods context. 12

One simple application of implementation science would be to make it easier for investigators to identify methods guidance relevant to their studies. 13 Another would be to improve methods guidance by encouraging those providing such guidance to involve their target audience in ensuring the accessibility of the guidance they provide. 14 Reporting guidelines represent another approach that has demonstrated appreciable—although still perhaps somewhat disappointing—improvement in study design and methods implementation.

We might utilize the full potential of reporting guidelines by including more methodological details (eg, items addressing the typical limitations of subgroup analyses we have highlighted) in the Standard Protocol Items: Recommendations for Interventional Trials ( SPIRIT ) guideline and corresponding items in the Consolidated Standards of Reporting Trials ( CONSORT ) guideline. Some investigators might, however, find the complexity associated with these requirements daunting. Such concerns highlight the need for more systematic research to identify and understand the main barriers (lack of awareness, training, opportunity) to guide the development—and subsequent testing—of new strategies for methods implementation.

It remains speculative whether shifting some of the notoriously scarce resources for methodological research toward implementation science would eventually result in better research quality. The substantial potential for improvement, as Sherry et al 1 highlighted for subgroup analyses in oncology trials, and the obvious failure of our current approaches, suggest that now may be a good time to give it a try.

Published: March 28, 2024. doi:10.1001/jamanetworkopen.2024.3339

Open Access: This is an open access article distributed under the terms of the CC-BY License . © 2024 Schandelmaier S et al. JAMA Network Open .

Corresponding Author: Stefan Schandelmaier, MD, PhD, Division of Clinical Epidemiology, Department of Clinical Research, University Hospital and University of Basel, Totengässlein 3, 4051 Basel, Switzerland ( [email protected] ).

Conflict of Interest Disclosures: None reported.

See More About

Schandelmaier S , Guyatt G. Same Old Challenges in Subgroup Analysis—Should We Do More About Methods Implementation? JAMA Netw Open. 2024;7(3):e243339. doi:10.1001/jamanetworkopen.2024.3339

Manage citations:

© 2024

Select Your Interests

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Get the latest research based on your areas of interest.

Others also liked.

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts

Read our research on: Abortion | Podcasts | Election 2024

Regions & Countries

What the data says about abortion in the u.s..

Pew Research Center has conducted many surveys about abortion over the years, providing a lens into Americans’ views on whether the procedure should be legal, among a host of other questions.

In a  Center survey  conducted nearly a year after the Supreme Court’s June 2022 decision that  ended the constitutional right to abortion , 62% of U.S. adults said the practice should be legal in all or most cases, while 36% said it should be illegal in all or most cases. Another survey conducted a few months before the decision showed that relatively few Americans take an absolutist view on the issue .

Find answers to common questions about abortion in America, based on data from the Centers for Disease Control and Prevention (CDC) and the Guttmacher Institute, which have tracked these patterns for several decades:

How many abortions are there in the U.S. each year?

How has the number of abortions in the u.s. changed over time, what is the abortion rate among women in the u.s. how has it changed over time, what are the most common types of abortion, how many abortion providers are there in the u.s., and how has that number changed, what percentage of abortions are for women who live in a different state from the abortion provider, what are the demographics of women who have had abortions, when during pregnancy do most abortions occur, how often are there medical complications from abortion.

This compilation of data on abortion in the United States draws mainly from two sources: the Centers for Disease Control and Prevention (CDC) and the Guttmacher Institute, both of which have regularly compiled national abortion data for approximately half a century, and which collect their data in different ways.

The CDC data that is highlighted in this post comes from the agency’s “abortion surveillance” reports, which have been published annually since 1974 (and which have included data from 1969). Its figures from 1973 through 1996 include data from all 50 states, the District of Columbia and New York City – 52 “reporting areas” in all. Since 1997, the CDC’s totals have lacked data from some states (most notably California) for the years that those states did not report data to the agency. The four reporting areas that did not submit data to the CDC in 2021 – California, Maryland, New Hampshire and New Jersey – accounted for approximately 25% of all legal induced abortions in the U.S. in 2020, according to Guttmacher’s data. Most states, though,  do  have data in the reports, and the figures for the vast majority of them came from each state’s central health agency, while for some states, the figures came from hospitals and other medical facilities.

Discussion of CDC abortion data involving women’s state of residence, marital status, race, ethnicity, age, abortion history and the number of previous live births excludes the low share of abortions where that information was not supplied. Read the methodology for the CDC’s latest abortion surveillance report , which includes data from 2021, for more details. Previous reports can be found at  stacks.cdc.gov  by entering “abortion surveillance” into the search box.

For the numbers of deaths caused by induced abortions in 1963 and 1965, this analysis looks at reports by the then-U.S. Department of Health, Education and Welfare, a precursor to the Department of Health and Human Services. In computing those figures, we excluded abortions listed in the report under the categories “spontaneous or unspecified” or as “other.” (“Spontaneous abortion” is another way of referring to miscarriages.)

Guttmacher data in this post comes from national surveys of abortion providers that Guttmacher has conducted 19 times since 1973. Guttmacher compiles its figures after contacting every known provider of abortions – clinics, hospitals and physicians’ offices – in the country. It uses questionnaires and health department data, and it provides estimates for abortion providers that don’t respond to its inquiries. (In 2020, the last year for which it has released data on the number of abortions in the U.S., it used estimates for 12% of abortions.) For most of the 2000s, Guttmacher has conducted these national surveys every three years, each time getting abortion data for the prior two years. For each interim year, Guttmacher has calculated estimates based on trends from its own figures and from other data.

The latest full summary of Guttmacher data came in the institute’s report titled “Abortion Incidence and Service Availability in the United States, 2020.” It includes figures for 2020 and 2019 and estimates for 2018. The report includes a methods section.

In addition, this post uses data from StatPearls, an online health care resource, on complications from abortion.

An exact answer is hard to come by. The CDC and the Guttmacher Institute have each tried to measure this for around half a century, but they use different methods and publish different figures.

The last year for which the CDC reported a yearly national total for abortions is 2021. It found there were 625,978 abortions in the District of Columbia and the 46 states with available data that year, up from 597,355 in those states and D.C. in 2020. The corresponding figure for 2019 was 607,720.

The last year for which Guttmacher reported a yearly national total was 2020. It said there were 930,160 abortions that year in all 50 states and the District of Columbia, compared with 916,460 in 2019.

• How the CDC gets its data: It compiles figures that are voluntarily reported by states’ central health agencies, including separate figures for New York City and the District of Columbia. Its latest totals do not include figures from California, Maryland, New Hampshire or New Jersey, which did not report data to the CDC. ( Read the methodology from the latest CDC report .)
• How Guttmacher gets its data: It compiles its figures after contacting every known abortion provider – clinics, hospitals and physicians’ offices – in the country. It uses questionnaires and health department data, then provides estimates for abortion providers that don’t respond. Guttmacher’s figures are higher than the CDC’s in part because they include data (and in some instances, estimates) from all 50 states. ( Read the institute’s latest full report and methodology .)

While the Guttmacher Institute supports abortion rights, its empirical data on abortions in the U.S. has been widely cited by  groups  and  publications  across the political spectrum, including by a  number of those  that  disagree with its positions .

These estimates from Guttmacher and the CDC are results of multiyear efforts to collect data on abortion across the U.S. Last year, Guttmacher also began publishing less precise estimates every few months , based on a much smaller sample of providers.

The figures reported by these organizations include only legal induced abortions conducted by clinics, hospitals or physicians’ offices, or those that make use of abortion pills dispensed from certified facilities such as clinics or physicians’ offices. They do not account for the use of abortion pills that were obtained  outside of clinical settings .

(Back to top)

The annual number of U.S. abortions rose for years after Roe v. Wade legalized the procedure in 1973, reaching its highest levels around the late 1980s and early 1990s, according to both the CDC and Guttmacher. Since then, abortions have generally decreased at what a CDC analysis called  “a slow yet steady pace.”

Guttmacher says the number of abortions occurring in the U.S. in 2020 was 40% lower than it was in 1991. According to the CDC, the number was 36% lower in 2021 than in 1991, looking just at the District of Columbia and the 46 states that reported both of those years.

(The corresponding line graph shows the long-term trend in the number of legal abortions reported by both organizations. To allow for consistent comparisons over time, the CDC figures in the chart have been adjusted to ensure that the same states are counted from one year to the next. Using that approach, the CDC figure for 2021 is 622,108 legal abortions.)

There have been occasional breaks in this long-term pattern of decline – during the middle of the first decade of the 2000s, and then again in the late 2010s. The CDC reported modest 1% and 2% increases in abortions in 2018 and 2019, and then, after a 2% decrease in 2020, a 5% increase in 2021. Guttmacher reported an 8% increase over the three-year period from 2017 to 2020.

As noted above, these figures do not include abortions that use pills obtained outside of clinical settings.

Guttmacher says that in 2020 there were 14.4 abortions in the U.S. per 1,000 women ages 15 to 44. Its data shows that the rate of abortions among women has generally been declining in the U.S. since 1981, when it reported there were 29.3 abortions per 1,000 women in that age range.

The CDC says that in 2021, there were 11.6 abortions in the U.S. per 1,000 women ages 15 to 44. (That figure excludes data from California, the District of Columbia, Maryland, New Hampshire and New Jersey.) Like Guttmacher’s data, the CDC’s figures also suggest a general decline in the abortion rate over time. In 1980, when the CDC reported on all 50 states and D.C., it said there were 25 abortions per 1,000 women ages 15 to 44.

That said, both Guttmacher and the CDC say there were slight increases in the rate of abortions during the late 2010s and early 2020s. Guttmacher says the abortion rate per 1,000 women ages 15 to 44 rose from 13.5 in 2017 to 14.4 in 2020. The CDC says it rose from 11.2 per 1,000 in 2017 to 11.4 in 2019, before falling back to 11.1 in 2020 and then rising again to 11.6 in 2021. (The CDC’s figures for those years exclude data from California, D.C., Maryland, New Hampshire and New Jersey.)

The CDC broadly divides abortions into two categories: surgical abortions and medication abortions, which involve pills. Since the Food and Drug Administration first approved abortion pills in 2000, their use has increased over time as a share of abortions nationally, according to both the CDC and Guttmacher.

The majority of abortions in the U.S. now involve pills, according to both the CDC and Guttmacher. The CDC says 56% of U.S. abortions in 2021 involved pills, up from 53% in 2020 and 44% in 2019. Its figures for 2021 include the District of Columbia and 44 states that provided this data; its figures for 2020 include D.C. and 44 states (though not all of the same states as in 2021), and its figures for 2019 include D.C. and 45 states.

Guttmacher, which measures this every three years, says 53% of U.S. abortions involved pills in 2020, up from 39% in 2017.

Two pills commonly used together for medication abortions are mifepristone, which, taken first, blocks hormones that support a pregnancy, and misoprostol, which then causes the uterus to empty. According to the FDA, medication abortions are safe  until 10 weeks into pregnancy.

Surgical abortions conducted  during the first trimester  of pregnancy typically use a suction process, while the relatively few surgical abortions that occur  during the second trimester  of a pregnancy typically use a process called dilation and evacuation, according to the UCLA School of Medicine.

In 2020, there were 1,603 facilities in the U.S. that provided abortions,  according to Guttmacher . This included 807 clinics, 530 hospitals and 266 physicians’ offices.

While clinics make up half of the facilities that provide abortions, they are the sites where the vast majority (96%) of abortions are administered, either through procedures or the distribution of pills, according to Guttmacher’s 2020 data. (This includes 54% of abortions that are administered at specialized abortion clinics and 43% at nonspecialized clinics.) Hospitals made up 33% of the facilities that provided abortions in 2020 but accounted for only 3% of abortions that year, while just 1% of abortions were conducted by physicians’ offices.

Looking just at clinics – that is, the total number of specialized abortion clinics and nonspecialized clinics in the U.S. – Guttmacher found the total virtually unchanged between 2017 (808 clinics) and 2020 (807 clinics). However, there were regional differences. In the Midwest, the number of clinics that provide abortions increased by 11% during those years, and in the West by 6%. The number of clinics  decreased  during those years by 9% in the Northeast and 3% in the South.

The total number of abortion providers has declined dramatically since the 1980s. In 1982, according to Guttmacher, there were 2,908 facilities providing abortions in the U.S., including 789 clinics, 1,405 hospitals and 714 physicians’ offices.

The CDC does not track the number of abortion providers.

In the District of Columbia and the 46 states that provided abortion and residency information to the CDC in 2021, 10.9% of all abortions were performed on women known to live outside the state where the abortion occurred – slightly higher than the percentage in 2020 (9.7%). That year, D.C. and 46 states (though not the same ones as in 2021) reported abortion and residency data. (The total number of abortions used in these calculations included figures for women with both known and unknown residential status.)

The share of reported abortions performed on women outside their state of residence was much higher before the 1973 Roe decision that stopped states from banning abortion. In 1972, 41% of all abortions in D.C. and the 20 states that provided this information to the CDC that year were performed on women outside their state of residence. In 1973, the corresponding figure was 21% in the District of Columbia and the 41 states that provided this information, and in 1974 it was 11% in D.C. and the 43 states that provided data.

In the District of Columbia and the 46 states that reported age data to  the CDC in 2021, the majority of women who had abortions (57%) were in their 20s, while about three-in-ten (31%) were in their 30s. Teens ages 13 to 19 accounted for 8% of those who had abortions, while women ages 40 to 44 accounted for about 4%.

The vast majority of women who had abortions in 2021 were unmarried (87%), while married women accounted for 13%, according to  the CDC , which had data on this from 37 states.

In the District of Columbia, New York City (but not the rest of New York) and the 31 states that reported racial and ethnic data on abortion to  the CDC , 42% of all women who had abortions in 2021 were non-Hispanic Black, while 30% were non-Hispanic White, 22% were Hispanic and 6% were of other races.

Looking at abortion rates among those ages 15 to 44, there were 28.6 abortions per 1,000 non-Hispanic Black women in 2021; 12.3 abortions per 1,000 Hispanic women; 6.4 abortions per 1,000 non-Hispanic White women; and 9.2 abortions per 1,000 women of other races, the  CDC reported  from those same 31 states, D.C. and New York City.

For 57% of U.S. women who had induced abortions in 2021, it was the first time they had ever had one,  according to the CDC.  For nearly a quarter (24%), it was their second abortion. For 11% of women who had an abortion that year, it was their third, and for 8% it was their fourth or more. These CDC figures include data from 41 states and New York City, but not the rest of New York.

Nearly four-in-ten women who had abortions in 2021 (39%) had no previous live births at the time they had an abortion,  according to the CDC . Almost a quarter (24%) of women who had abortions in 2021 had one previous live birth, 20% had two previous live births, 10% had three, and 7% had four or more previous live births. These CDC figures include data from 41 states and New York City, but not the rest of New York.

The vast majority of abortions occur during the first trimester of a pregnancy. In 2021, 93% of abortions occurred during the first trimester – that is, at or before 13 weeks of gestation,  according to the CDC . An additional 6% occurred between 14 and 20 weeks of pregnancy, and about 1% were performed at 21 weeks or more of gestation. These CDC figures include data from 40 states and New York City, but not the rest of New York.

About 2% of all abortions in the U.S. involve some type of complication for the woman , according to an article in StatPearls, an online health care resource. “Most complications are considered minor such as pain, bleeding, infection and post-anesthesia complications,” according to the article.

The CDC calculates  case-fatality rates for women from induced abortions – that is, how many women die from abortion-related complications, for every 100,000 legal abortions that occur in the U.S .  The rate was lowest during the most recent period examined by the agency (2013 to 2020), when there were 0.45 deaths to women per 100,000 legal induced abortions. The case-fatality rate reported by the CDC was highest during the first period examined by the agency (1973 to 1977), when it was 2.09 deaths to women per 100,000 legal induced abortions. During the five-year periods in between, the figure ranged from 0.52 (from 1993 to 1997) to 0.78 (from 1978 to 1982).

The CDC calculates death rates by five-year and seven-year periods because of year-to-year fluctuation in the numbers and due to the relatively low number of women who die from legal induced abortions.

In 2020, the last year for which the CDC has information , six women in the U.S. died due to complications from induced abortions. Four women died in this way in 2019, two in 2018, and three in 2017. (These deaths all followed legal abortions.) Since 1990, the annual number of deaths among women due to legal induced abortion has ranged from two to 12.

The annual number of reported deaths from induced abortions (legal and illegal) tended to be higher in the 1980s, when it ranged from nine to 16, and from 1972 to 1979, when it ranged from 13 to 63. One driver of the decline was the drop in deaths from illegal abortions. There were 39 deaths from illegal abortions in 1972, the last full year before Roe v. Wade. The total fell to 19 in 1973 and to single digits or zero every year after that. (The number of deaths from legal abortions has also declined since then, though with some slight variation over time.)

The number of deaths from induced abortions was considerably higher in the 1960s than afterward. For instance, there were 119 deaths from induced abortions in  1963  and 99 in  1965 , according to reports by the then-U.S. Department of Health, Education and Welfare, a precursor to the Department of Health and Human Services. The CDC is a division of Health and Human Services.

Note: This is an update of a post originally published May 27, 2022, and first updated June 24, 2022.

Sign up for our weekly newsletter

Fresh data delivered Saturday mornings

Key facts about the abortion debate in America

Public opinion on abortion, three-in-ten or more democrats and republicans don’t agree with their party on abortion, partisanship a bigger factor than geography in views of abortion access locally, do state laws on abortion reflect public opinion, most popular.

About Pew Research Center Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. It conducts public opinion polling, demographic research, media content analysis and other empirical social science research. Pew Research Center does not take policy positions. It is a subsidiary of The Pew Charitable Trusts .

Help | Advanced Search

Computer Science > Computer Vision and Pattern Recognition

Title: mm1: methods, analysis & insights from multimodal llm pre-training.

Abstract: In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results. Further, we show that the image encoder together with image resolution and the image token count has substantial impact, while the vision-language connector design is of comparatively negligible importance. By scaling up the presented recipe, we build MM1, a family of multimodal models up to 30B parameters, including both dense models and mixture-of-experts (MoE) variants, that are SOTA in pre-training metrics and achieve competitive performance after supervised fine-tuning on a range of established multimodal benchmarks. Thanks to large-scale pre-training, MM1 enjoys appealing properties such as enhanced in-context learning, and multi-image reasoning, enabling few-shot chain-of-thought prompting.

Submission history

Access paper:.

• Download PDF
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

This paper is in the following e-collection/theme issue:

Published on 29.3.2024 in Vol 26 (2024)

Usability of Health Care Price Transparency Data in the United States: Mixed Methods Study

Authors of this article:

Original Paper

• Negar Maleki 1 , PhD   ; 
• Balaji Padmanabhan 2 , PhD   ; 
• Kaushik Dutta 1 , PhD  

1 School of Information Systems and Management, Muma College of Business, University of South Florida, Tampa, FL, United States

2 Decision, Operations & Information Technologies Department, Robert H. Smith School of Business, University of Maryland, College Park, MD, United States

Corresponding Author:

Negar Maleki, PhD

School of Information Systems and Management

Muma College of Business

University of South Florida

4202 E Fowler Avenue

Tampa, FL, 33620

United States

Phone: 1 8139742011

Email: [email protected]

Background: Increasing health care expenditure in the United States has put policy makers under enormous pressure to find ways to curtail costs. Starting January 1, 2021, hospitals operating in the United States were mandated to publish transparent, accessible pricing information online about the items and services in a consumer-friendly format within comprehensive machine-readable files on their websites.

Objective: The aims of this study are to analyze the available files on hospitals’ websites, answering the question—is price transparency (PT) information as provided usable for patients or for machines?—and to provide a solution.

Methods: We analyzed 39 main hospitals in Florida that have published machine-readable files on their website, including commercial carriers. We created an Excel (Microsoft) file that included those 39 hospitals along with the 4 most popular services—Current Procedural Terminology (CPT) 45380, 29827, and 70553 and Diagnosis-Related Group (DRG) 807—for the 4 most popular commercial carriers (Health Maintenance Organization [HMO] or Preferred Provider Organization [PPO] plans)—Aetna, Florida Blue, Cigna, and UnitedHealthcare. We conducted an A/B test using 67 MTurkers (randomly selected from US residents), investigating the level of awareness about PT legislation and the usability of available files. We also suggested format standardization, such as master field names using schema integration, to make machine-readable files consistent and usable for machines.

Results: The poor usability and inconsistent formats of the current PT information yielded no evidence of its usefulness for patients or its quality for machines. This indicates that the information does not meet the requirements for being consumer-friendly or machine readable as mandated by legislation. Based on the responses to the first part of the experiment (PT awareness), it was evident that participants need to be made aware of the PT legislation. However, they believe it is important to know the service price before receiving it. Based on the responses to the second part of the experiment (human usability of PT information), the average number of correct responses was not equal between the 2 groups, that is, the treatment group (mean 1.23, SD 1.30) found more correct answers than the control group (mean 2.76, SD 0.58; t 65 =6.46; P <.001; d =1.52).

Conclusions: Consistent machine-readable files across all health systems facilitate the development of tools for estimating customer out-of-pocket costs, aligning with the PT rule’s main objective—providing patients with valuable information and reducing health care expenditures.

Introduction

From 1970 to 2020, on a per capita basis, health care expenditures in the United States have increased sharply from US $353 per person to US $12,531 per person. In constant 2020 dollars, the increase was from US $1875 in 1970 to US $12,531 in 2020 [ 1 ]. The significant rise in health care expenses has put policy makers under enormous pressure to find ways to contain these expenditures. Price transparency (PT) in health care is 1 generally proposed strategy for addressing these problems [ 2 ] and has been debated for years [ 3 ]. Some economists believe that PT in health care will cut health care prices in the same way it has in other industries, while others argue that owing to the specific characteristics of the health care market, PT would not ameliorate rising health care costs. Price elasticity also does not typically apply in health care, since, if a problem gets severe, people will typically seek treatment regardless of cost, with the drawback that individuals learn of their health care costs after receiving treatment [ 4 ]. Complex billing processes, hidden insurer-provider contracts, the sheer quantity of third-party payers, and substantial quality differences in health care delivery are other unique aspects of health care that complicate the situation considerably.

The Centers for Medicare & Medicaid Services (CMS) mandated hospitals to post negotiated rates, including payer-specific negotiated costs, for 300 “shoppable services” beginning in January 2021. The list must include 70 CMS-specified services and an additional 230 services each hospital considers relevant to its patient population. Hospitals must include each third-party payer and their payer-specific fee when negotiating multiple rates for the same care. The data must be displayed simply, easily accessible (without requiring personal information from the patient), and saved in a machine-readable manner [ 5 ]. These efforts aim to facilitate informed patient decision-making, reduce out-of-pocket spending, and decrease health care expenditures. Former Secretary of Health and Human Services, Alex Azar, expressed a vision of hospital PT when declaring the new legislation “a patient-centered system that puts you in control and provides the affordability you need, the options and control you want, and the quality you deserve. Providing patients with clear, accessible information about the price of their care is a vital piece of delivering on that vision” [ 6 ].

Despite the legislation, it is not clear if people are actually engaging in using PT tools. For example, in 2007, New Hampshire’s HealthCost website was established, providing the negotiated price and out-of-pocket costs for 42 commonly used services by asking whether the patient is insured or their insurer and the zip code to post out-of-pocket costs in descending order. Mehrotra et al [ 7 ] examined this website over 3 years to understand how often and why these tools have mainly been used. Their analysis suggested that despite the growing interest in PT, approximately 1% of the state’s population used this tool. Low PT tool usage was also seen in other studies [ 8 - 10 ], suggesting that 3% to 12% of individuals who were offered the tool used it during the study period, and in all studies, the duration was at least 12 months. Thus, offering PT tools does not in itself lead to decreased total spending, since few people who have access to them use them to browse for lower-cost services [ 7 , 11 ].

In a recent paper, researchers addressed 1 possible reason for low engagement—lack of awareness. They implemented an extensive targeted online advertising campaign using Google Advertisements to increase awareness and assessed whether it increased New Hampshire’s PT website use. Their findings suggested that although lack of awareness is a possible reason for the low impact of PT tools in health care spending, structural factors might affect the use of health care information [ 12 ]. Individuals may not be able to exactly determine their out-of-pocket expenses from the information provided.

Surprisingly, there is little research on the awareness and usability of PT information after the current PT legislation went into effect. A recent study [ 13 ] highlighted the nonusability of existing machine-readable files for employers, policy makers, researchers, or consumers, and this paper adds to this literature by answering the question—is PT information as provided usable for patients or machines? Clearly, if it is of value to patients, it can be useful; the reason to take the perspective of machines was to examine whether this information as provided might also be useful for third-party programs that can extract information from the provided data (to subsequently help patients through other ways of presenting this information perhaps). We address this question through a combination of user experiments and data schema analysis. While there are recent papers that have also argued that PT data have deficiencies [ 13 , 14 ], ours is the first to combine user experiments with analysis of data schema from several hospitals in Florida to make a combined claim on value for patients and machines. We hope this can add to the discourse on PT and what needs to be done to extract value for patients and the health care system as a whole.

Impact of PT Tools

The impact of PT tools on consumers and health care facilities has been investigated in the literature. Some studies showed that consumers with access to PT tools are more likely to reduce forgone needed services over time. Moreover, consumers who use tools tend to find the lowest service prices [ 8 , 15 - 17 ]. A few studies investigated the impact of PT tools on the selection of health care facilities. They illustrated that some consumers tend to change health care facilities pursuing lower prices, while some others prefer to stay with expensive ones, although they are aware of some other facilities that offer lower prices [ 9 , 18 ]. Finally, some research studied the impact of PT tools on cost and showed that some consumers experienced no effect, while others experienced decreases in average consumer expenses [ 8 , 17 , 18 ]. However, the impact of PT tools on health care facilities is inconclusive, meaning different studies concluded different effects. Some stated that PT tools decrease the prices of imaging and laboratory services, while others said that although public charge disclosure lowers health care facility charges, the final prices remained unchanged [ 17 , 18 ].

Legislation Related Works

In a study, researchers considered 20 leading US hospitals to assess provided chargemasters to understand to what extent patients can obtain information from websites to determine the out-of-pocket costs [ 19 ]. Their findings showed that although all hospitals provided chargemasters on their websites, they rarely offered transparent information, making it hard for patients to determine out-of-pocket costs. Their analysis used advanced diagnostic imaging services to assess hospitals’ chargemasters since these are the most common services people look for. Mehrotra et al [ 7 ] also mentioned that the most common searches belonged to outpatient visits, magnetic resonance imaging (MRI), and emergency department visits. To this end, we used “MRI scan of the brain before and after contrast” as one of the shoppable services in our analysis. Another study examined imaging services in children’s hospitals (n=89), restricting the analysis to hospitals (n=35) that met PT requirements—published chargemaster rates, discounted cash prices, and payer-negotiated prices in a machine-readable file, and published costs for 300 common shoppable medical services in a consumer-friendly format. Their study revealed that, in addition to a broad range of imaging service charges, most hospitals lack the machine-readable file requirement [ 20 ].

Arvisais-Anhalt et al [ 21 ] identified 11 hospitals with available chargemasters in Dallas County to compare the prices of a wide range of available services. They observed significant variations for a laboratory test: partial thromboplastin time, a medication: 5 mg tablet of amlodipine, and a procedure: circumcision. Reddy et al [ 22 ] focus on New York State to assess the accessibility and usability of hospitals’ chargemasters from patients’ viewpoint. They found that 189 out of 202 hospitals had a locatable chargemaster on their home page. However, only 37 hospitals contain the Current Procedural Terminology (CPT) code, which makes those without the CPT code unusable due to the existence of many different descriptions for the same procedure; for example, an elective heart procedure had 34 entries. We add to this considerable literature by examining a subset of Florida hospitals.

In a competitive market, higher-quality goods and services require higher prices [ 23 ]. Based on this, Patel et al [ 24 ] examined the relationship between the Diagnosis-Related Group (DRG) chargemaster and quality measures. Although prior research found no convincing evidence that hospitals with greater costs also delivered better care [ 25 ], they discovered 2 important quality indicators that were linked to standard charges positively and substantially—mortality rate and readmission rates—which both are quality characteristics that are in line with economic theory. Moreover, Patel et al [ 24 ] studied the variety of one of the most commonly performed services (vaginal delivery) as a DRG code, which motivated us to select “Vaginal delivery without sterilization or D&C without CC/MCC” as another shoppable service in our analysis.

Ethical Considerations

All data used in this study, including the secondary data set obtained from hospitals’ websites and the data collected during the user experiment, underwent a thorough anonymization process. The study was conducted under protocols approved by the University of South Florida institutional review board (STUDY004145: “Effect of price transparency regulation (PTR) on the public decisions”) under HRP-502b(7) Social Behavioral Survey Consent. This approval encompassed the use of publicly available anonymized secondary data from hospitals’ websites, as well as a user experiment aimed at assessing awareness of the PT rule and the usability of hospitals’ files. No individual-specific data were collected during the experiment, which solely focused on capturing subjects’ awareness and opinions regarding the PT rule and associated files. At the onset of the experiment, participants were provided with a downloadable consent form and were allowed to withdraw their participation at any time. Survey participants were offered a US $2 reward, and their involvement was entirely anonymous.

Data Collection

According to CMS, “Starting January 1, 2021, each hospital operating in the United States will be required to provide clear, accessible pricing information online about the items and services they provide in two ways: 1- As a comprehensive machine-readable file with all items and services. 2- In a display of shoppable services in a consumer-friendly format.” As stated, files available on hospitals’ websites should be consumer-friendly, so the question of whether these files are for users arises. On the other hand, as stated, files should be machine-readable, so again the question of whether these files are for machines arises. Below we try to answer both questions in detail, respectively.

Value for Users: User Experiments

When a public announcement is disseminated, its efficacy relies on ensuring widespread awareness and facilitating practical use during times of necessity. Previous research on PT announcements has highlighted the challenges faced by patients in accurately estimating out-of-pocket expenses. However, a fundamental inquiry arises—are individuals adequately informed about the availability of tools that enable them to estimate their out-of-pocket costs for desired services? To address this, we conducted a survey to assess public awareness of PT legislation. The survey encompassed a range of yes or no and multiple-choice questions aimed at gauging participants’ familiarity with the PT rule in health care and their entitlement to obtain cost information prior to receiving a service. Additionally, we inquired about participants’ knowledge of resources for accessing pricing information and whether they were aware of the PT rule. Furthermore, we incorporated follow-up questions to ensure that the survey responses were not provided arbitrarily, thereby securing reliable and meaningful outcomes.

Moreover, considering the previously established evidence of subpar usability associated with the currently available files, we propose streamlining the existing files and developing a user-friendly and comprehensive document for conducting an A/B test. This test aims to evaluate which file better facilitates participants in accurately estimating their out-of-pocket costs. In collaboration with Florida Blue experts during biweekly meetings throughout the entire process outlined in this paper, the authors determined the optimal design for the summary table. This design, which presents prices in a more user-friendly format, enhancing overall participant comprehension, was used during the A/B testing. Participants were randomly assigned to either access the hospitals’ files or a meticulously constructed summary table, manually created in Excel, prominently displaying cost information (Please note that all files, including the hospitals’ files and our Excel file, are made available in the same format [Excel] on a cloud-based platform to eliminate any disparities in accessing the files. This ensures equitable ease of finding, downloading, and opening files, as accessing the hospitals’ files typically requires significant effort.). The experiment entailed presenting 3 distinct health-related scenarios and instructing participants to locate the price for the requested service. Subsequently, participants were asked to provide the hospital name, service price, insurer name, and insurance plan. Additionally, we sought feedback on the perceived difficulty of finding the requested service and their priority for selecting hospitals [ 26 ], followed by Likert scale questions to assess participants’ evaluation of the provided file’s efficacy in facilitating price retrieval.

The experiments were conducted to investigate the following questions: (1) Are the individuals aware of the PT legislation? and (2) Is the information provided usable for patients? To evaluate the usability of files found on websites, we selected 2 prevalent services based on existing literature and 2 other services recommended as high-demand ones by Florida Blue experts, Table 1 . Furthermore, meticulous efforts were made to ensure that both the control and treatment groups encountered identical circumstances, thus allowing for a systematic examination of the disparities solely attributable to variations in data representation.

a DRG: Diagnosis-Related Group.

b D&C: dilation and curettage.

c CC/MCC: complication or comorbidity/major complication or comorbidity.

d CPT: Current Procedural Terminology.

e MRI: magnetic resonance imaging.

Participants

A total of 67 adults (30 female individuals; mean 41.43, SD 12.39 years) were recruited on the Amazon Mechanical Turk platform, with no specific selection criteria other than being located in the United States.

We focused on 75 main hospitals (ie, the main hospital refers to distinguish a hospital from smaller clinics or specialized medical centers within the same health system) in the state of Florida. When we searched their websites for PT files (machine-readable files), only 89% (67/75) of hospitals included machine-readable files. According to the PT legislation, these files were supposed to contain information about 300 shoppable services. However, only 58% (39/67) of hospitals included information such as insurer prices in their files. Therefore, for the rest of the analysis, we only included the 39 hospitals that have the required information in their machine-readable files on their websites. We created an Excel file that included those 39 hospitals along with the 4 services—CPT 45380, 29827 and 70553 and DRG 807—mentioned in the literature ( Table 1 ) for 4 popular (suggested by Florida Blue experts) commercial carriers (Health Maintenance Organization [HMO] or Preferred Provider Organization [PPO] plans)—Aetna, Florida Blue, Cigna, and UnitedHealthcare.

Participants were recruited for the pilot and randomly assigned by the Qualtrics XM platform to answer multiple-choice questions and fill in blanks based on the given scenarios. First, participants responded to questions regarding the awareness of PT and then were divided into 2 groups randomly to answer questions regarding the usability of hospital-provided PT information. One group was assigned hospitals’ website links (control group), while the other group was given an Excel file with the same information provided in files on hospitals’ websites, but in a manner that was designed to allow easier comparison of prices across hospitals ( Multimedia Appendix 1 ). Participants were given 3 scenarios that asked them to find a procedure’s price based on their hospital and insurer selection to compare hospital-provided information with Excel. We provide some examples of hospitals’ files and our Excel file in Multimedia Appendix 1 and the survey experiment questions in Multimedia Appendix 2 .

Value for Machines: Schema Integration—Machine-Readable Files Representation

Through meticulous investigation of machine-readable files from 39 hospitals, we discovered that these files may vary in formats such as CSV or JSON, posing a challenge for machines to effectively manage the data within these files. Another significant obstacle arises from the lack of uniformity in data representation across these files, rendering them unsuitable for machine use without a cohesive system capable of processing them collectively. Our analysis revealed that hospitals within a single health system exhibit consistent data representation, although service prices may differ (we include both the same and different chargemaster prices in our study), while substantial disparities in data representation exist between hospitals affiliated with different health systems.

Moving forward, we will use the terms “data representation” and “schema” interchangeably, with “schema” denoting its database management context. In this context, a schema serves as a blueprint outlining the structure, organization, and relationships of data within a database system. It encompasses key details such as tables, fields, data types, and constraints that define the stored data. To systematically illustrate schema differences among hospitals associated with different health systems, we adopted the methodology outlined in reference [ 27 ] for schema integration, which offers a valid approach for comparing distinct data representations. The concept of schema integration encompasses four common categories: (1) identical: hospitals within the same health system adhere to this concept as their representations are identical; (2) equivalent: while hospitals in health system “A” may present different representations from those in health system “B,” they possess interchangeable columns; (3) compatible: in cases where hospitals across different health systems are neither identical nor equivalent, the modeling constructs, designer perception, and integrity constraints do not contradict one another; and (4) incompatible: in situations where hospitals within different health systems demonstrate contradictory representations, distinct columns exist for each health system due to specification incoherence.

Our analysis focused on health systems in Florida that encompassed a minimum of 4 main hospitals, using the most up-to-date data available on their respective websites. Within this scope, we identified 8 health systems with at least 4 main hospitals, of which 88% (7/8) of health systems had published machine-readable files on their websites. Consequently, our analysis included 65% (36/55) of hospitals that possessed machine-readable files available on their websites. To facilitate further investigation by interested researchers, we have made the analyzed data accessible on a cloud-based platform. During our analysis, we meticulously extracted the schema of each health system by closely scrutinizing the hospitals associated with each health system, capturing key details such as tables, fields, and data types. Subsequently, we compiled a comprehensive master field name table trying to have the same data type and field names that make it easier for machines to retrieve information. We elaborate on the master field names table in greater detail within the results section.

Value for Users

Question 1 (pt awareness).

Based on the responses, it is evident that participants need to be made aware of the PT legislation. Among the participants, 64% (49/76) reported that they had not heard about the legislation. However, they believe it is important to know the service price before receiving it—response charts are provided in Multimedia Appendix 3 .

Question 2 (Human Usability of PT Information)

Based on the responses to scenarios, the average number of correct responses is not equal between the 2 groups, that is, the treatment group (mean 1.23, SD 1.30) found more correct answers than the control group (mean 2.76, SD 0.58; t 65 =6.46; P <.049; d =1.52). The t tests (2-tailed) for the other questions in the experiment are in Multimedia Appendix 4 .

These suggest that current files on hospitals’ websites are not consumer-friendly, and participants find it challenging to estimate out-of-pocket costs for a desired service. For this reason, in addition to making the files easier to use, this information should also include thorough documentation that explains what each column represents, up to what amount an insurer covers for a specific service, or the stated price covers up to how many days of a particular service, that is, “contracting method.” For example, based on consulting with one of the senior network analysts of Florida Blue, some prices for a service like DRG 807 are presented as per diem costs, and based on the current information on these files, it cannot be recognizable without having comprehensive documentation for them.

Value for Machines

After carefully reviewing all machine-readable file schemas, we create a master field name table, including the available field names in machine-readable files ( Table 2 ). According to Table 2 , the first column represents master field names that we came up with, and the following columns each represent hospitals within a health system. The “✓” mark shows that hospitals within a health system have identical field names as we consider as master field names and the “written” cells show equivalent field names, meaning that hospitals within that health system use different field names—we write what they use in their representation—while the content is equivalent to what we select as the master field name. The “❋” mark means that although hospitals within health system #2 provide insurer names and plans in their field names, some codes make those columns unusable for machines to recognize them the same as master field names. We also include the type of field names for all representations in parentheses.

a As noted previously, since we focus on the health system level instead of the hospital level, our schema does not have hospital-level information; however, it would be beneficial to add hospital information to the table.

b ✓: it means the given master field name in that row appears on the given health system file in that column.

c str: shows “string” as the data type.

d int: shows “integer” as the data type.

e CPT: Current Procedural Terminology.

f HCPCS: Health care Common Procedure Coding System.

g Not applicable.

h Apr: all patients refined.

i DRG: Diagnosis-Related Group.

j Ms: Medicare severity.

k CDM: charge description master.

l UB: uniform billing.

m float: it shows “float” as the data type.

n ❋: it means that although hospitals within health system #2 provide insurer names and plans in their field names, some codes make those columns unusable for machines to recognize them the same as master field names.

We did reverse engineering and drew entity-relationship diagrams (ERDs) for each hospital based on their data representation. However, as hospitals within the same health system have the same ERDs, we only include 1 ERD for each health system ( Figure 1 ). According to Figure 1 , although hospitals have tried to follow an intuitive structure, we can still separate them into three groups: (1) group I: all hospitals within this group have several columns for different insurers. As shown in the ERDs, we decided to have a separate entity, called “Insurance” for this group; (2) group II: all hospitals within this group have many sheets, and each sheet belongs to a specific insurer with a specific plan. As shown in the ERDs, we decided to create an “Insurance_Name” entity for this group’s ERD to show the difference in data representation; and (3) group III: all hospitals within this system have a “payer” column which includes the names of insurers without their plans. As shown in the ERDs, we decided to put this column as an attribute in the “Service” entity, and do not have an “Insurance” entity for this group’s ERD.

In conclusion, although most hospitals have adopted group I logic for data representation, for full similarity, a standard representation with the same intuitive field names (like what we suggest as the master field name; Table 2 ) should be proposed so that it can cover all systems’ data representations and be used as machine-readable file, for at least machine benefits. Mainly, standardization in the format and semantics of the provided data can help substantially in making the data more machine friendly.

Comparison With New CMS Guidelines

Recently, CMS has published guidelines regarding the PT legislation [ 28 ]. The most recent CMS guideline is a step forward in ensuring standardization but is still only recommended and is not mandatory. These guidelines exhibit overlaps with our fields in Table 2 , with slight differences attributed to granularities. Our observation reveals that hospitals within the same health system adopt a uniform schema. Therefore, our suggested schema operates on the granularity of health systems rather than individual hospitals.

The recent CMS guidelines allocate 24% (6/25) of field names specifically to hospital information, encompassing details such as “Hospital Name,” “Hospital File Date,” “Version,” “Hospital Location,” “Hospital Financial Aid Policy,” and “Hospital Licensure Information.” These details, absent in current hospital files, are crucial for informed decision-making. As noted previously, since we focus on the health system level instead of the hospital level, our schema does not have hospital-level information; however, it would be beneficial to add hospital information to the tables.

Our analysis reveals that the 11 field names in Table 2 align with the field names in the new CMS guidelines, demonstrating a substantial overlap of 58% (11/19). The corresponding CMS field names (compatible with our schema) include “Item or Service Description (Description or CDM Service Description),” “Code (Code),” “Code Type (Code Type),” “Setting (Patient Class),” “Gross Charge (Gross Charge),” “Discounted Cash Price (Discounted Cash Price),” “Payer Name (Insurer Name),” “Plan Name (Insurer Plan),” “Payer Specific Negotiated Charge: Dollar Amount (Price),” “De-identified Minimum Negotiated Charge (Min Negotiated Rate),” and “De-identified Maximum Negotiated Charge (Max Negotiated Rate).” Additionally, both our schema and the new CMS guidelines propose data types for each field name.

In our schema, which represents current hospitals’ files, there are 5 field names absent in the new CMS guidelines “Revenue Description,” “Revenue Code,” “Package/Line Level,” “Procedure ID,” and “Self Pay.” Conversely, the new CMS guidelines introduce 8 additional field names “Billing Class,” “Drug Unit of Measurement,” “Drug Type of Measurement,” “Modifiers,” “Payer Specific Negotiated Charge: Percentage,” “Contracting Method,” “Additional Generic Notes,” and “Additional Payer-Specific Notes.” We regard these new field names as providing further detailed information and enhancing consumer decision-making. If hospitals within a health system adopt consistent formats and can map their formats to the new CMS guidelines clearly in a mapping document they also provide, this can be more useful than the current optional guideline that is suggested.

In summary, since our analysis is based on the current data schema that hospitals have in place, we believe the schema we put out is easier to implement with minimal change to what the hospitals are currently doing. However, given the recent CMS guidelines, we recommend adding 8 additional fields as well as hospital-specific information.

Implications

The PT legislation aims to enable informed decision-making, reduce out-of-pocket expenses, and decrease overall health care expenditures. This study investigates the usage of current files by individuals and machines. Our results, unfortunately, suggest that PT data—as currently reported—appear to be neither useful for patients nor machines, raising important questions as to what these appear to be achieving today. Moreover, the findings indicate that even individuals with basic computer knowledge struggle with the usability of these files, highlighting the need for significant revisions to make them consumer-friendly and accessible to individuals of all technical proficiency levels. Additionally, inconsistencies in data representation between hospitals affiliated with different health systems pose challenges for machines, necessitating schema design improvements and the implementation of a standardized data representation. By addressing these concerns, PT legislation can achieve consistency and enhance machine readability, thus improving its effectiveness in promoting informed decision-making and reducing health care costs.

Although the official announcement of PT legislation is recent, prior studies [ 15 - 17 ] have attempted to evaluate the usability of PT, while subsequent studies [ 19 - 22 ] have examined the effectiveness of PT tools following the announcement. However, despite the introduction of PT rules, it appears that the usability of these files has not undergone significant improvements, indicating the necessity for proactive measures from responsible executives to ensure the effectiveness of this legislation. Our analysis of this matter emphasizes 2 primary factors—a lack of awareness among stakeholders and the challenges associated with using files due to inconsistencies in their format and representation.

As of April 2023, the CMS has issued over 730 warning notices and 269 requests for Corrective Action Plans. A total of 4 hospitals have faced Civil Monetary Penalties for noncompliance, and these penalties are publicly disclosed on the CMS website. The remaining hospitals subjected to comprehensive compliance reviews have either rectified their deficiencies or are actively engaged in doing so. While we acknowledge these efforts to comply with PT rules, our research revealed a notable disparity in data representation among hospitals affiliated with different health systems. Consequently, we focused on schema design and proposed the implementation of a master field name that encompasses a comprehensive data representation derived from an analysis of 36 hospitals. Standardizing the data representation across all health systems’ machine-readable files will effectively address concerns about consistency. Therefore, significant modifications are required for the PT legislation to enhance machine readability and provide clearer guidance on the design and structure of the files’ schema. If the hospital-provided information is consistent and of high quality, PT tools provided by health insurers may be able to estimate an individual’s total expenses more accurately.

Limitations

Our objective was to have an equal number in both groups. However, in the case of the group tasked with obtaining information from the hospitals’ websites, most did not finish the task and dropped out without completing it. This occurred because the task of retrieving the cost from the hospitals’ websites in its current form is complex, as indicated by feedback from some participants. Only 19% (13/67) completed the task in that group (control group). Although this is a limitation of the study, it also highlights the complexity of obtaining cost information from hospitals’ websites in the current form. In the treatment group, 81% (54 out of 67) of participants completed the task of retrieving the data, and the completion percentage was much higher.

Conclusions

Due to the poor usability and inconsistency of the formats, we, unfortunately, did not find evidence that the PT rule as implemented currently is useful to consumers, researchers, or policy makers (despite the legislation’s goals that files are “consumer-friendly” and “machine-readable”). As 1 solution, we suggest a master field name for the data representation of machine-readable files to make them consistent, at least for the machines. Building tools that enable customers to estimate out-of-pocket costs is facilitated by having consistent machine-readable files across all health systems, which can be considered as future work for researchers and companies to help the PT rule reach its main goal, which is providing useful information for patients and reducing health care expenditures. In addition, another worthwhile approach to reducing some of the exorbitant health care costs in the United States would be to integrate clinical decision support tools into the providers’ workflow, triggered by orders for medications, diagnostic testing, and other billable services. In this regard, Bouayad et al [ 29 ] conducted experiments with physicians to demonstrate that PT, when included as part of the system they interact with, such as clinical decision support integrated into electronic health record systems, can significantly aid in cost reduction. This is a promising direction for practice but needs to be implemented carefully to avoid unanticipated consequences, such as scenarios where cost is incorrectly viewed as a proxy for quality, or where the use of this information introduces new biases for physicians and patients.

Conflicts of Interest

None declared.

Example of Excel format of hospitals’ files and our created Excel file.

Survey questions and experiment scenarios.

Participants’ responses chart regarding price transparency awareness.

The t test analysis regarding human usability of price transparency information based on participants’ responses.

• McGough M, Winger A, Rakshit S, Amin K. How has U.S. spending on healthcare changed over time? Health System Tracker. 2022. URL: https://www.healthsystemtracker.org/chart-collection/u-s-spending-healthcare-changed-time/ [accessed 2024-03-13]
• Christensen HB, Floyd E, Maffett M. The only prescription is transparency: the effect of charge-price-transparency regulation on healthcare prices. Manag Sci. 2020;66(7):2861-2882. [ CrossRef ]
• Muir MA, Alessi SA, King JS. Clarifying costs: can increased price transparency reduce healthcare spending? UC Hastings Research Paper No. 38 (SSRN). Feb 26, 2013.:319-367. [ FREE Full text ] [ CrossRef ]
• Reinhardt UE. Health care price transparency and economic theory. JAMA. 2014;312(16):1642-1643. [ CrossRef ] [ Medline ]
• CY 2020 hospital Outpatient Prospective Payment System (OPPS) policy changes: hospital price transparency requirements (CMS-1717-F2). CMS.gov. 2020. URL: https://tinyurl.com/mrafxtvd [accessed 2024-03-13]
• Secretary Azar statement on proposed rule for hospital price transparency. HHS.gov. 2020. URL: https://tinyurl.com/yc4dx2vx [accessed 2024-03-13]
• Mehrotra A, Brannen T, Sinaiko AD. Use patterns of a state health care price transparency web site: what do patients shop for? Inquiry. 2014;51:0046958014561496. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Desai S, Hatfield LA, Hicks AL, Sinaiko AD, Chernew ME, Cowling D, et al. Offering a price transparency tool did not reduce overall spending among California public employees and retirees. Health Aff (Millwood). 2017;36(8):1401-1407. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sinaiko AD, Joynt Maddox KE, Rosenthal MB. Association between viewing health care price information and choice of health care facility. JAMA Intern Med. 2016;176(12):1868-1870. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Desai S, Hatfield LA, Hicks AL, Chernew ME, Mehrotra A. Association between availability of a price transparency tool and outpatient spending. JAMA. 2016;315(17):1874-1881. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sinaiko AD, Rosenthal MB. Examining a health care price transparency tool: who uses it, and how they shop for care. Health Aff (Millwood). 2016;35(4):662-670. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Desai SM, Shambhu S, Mehrotra A. Online advertising increased New Hampshire residents' use of provider price tool but not use of lower-price providers. Health Aff (Millwood). 2021;40(3):521-528. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kona M, Corlette S. Hospital and insurer price transparency rules now in effect but compliance is still far away. Health Affairs Forefront. 2022. URL: hhttps://tinyurl.com/3x6ymxf2 [accessed 2024-03-13]
• Wheeler C, Taylor R. New year, new CMS price transparency rule for hospitals. Health Affairs Forefront. 2021. URL: https://www.healthaffairs.org/content/forefront/new-year-new-cms-price-transparency-rule-hospitals [accessed 2024-03-13]
• Chernew M, Cooper Z, Larsen-Hallock E, Morton FS. Are health care services shoppable? Evidence from the consumption of lower-limb MRI scans. National Bureau of Economic Research. 2021. URL: https://www.nber.org/papers/w24869 [accessed 2024-03-13]
• Mehrotra A, Dean KM, Sinaiko AD, Sood N. Americans support price shopping for health care, but few actually seek out price information. Health Aff (Millwood). 2017;36(8):1392-1400. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brown ZY. Equilibrium effects of health care price information. Rev Econ Stat. 2019;101(4):699-712. [ FREE Full text ] [ CrossRef ]
• Wu SJ, Sylwestrzak G, Shah C, DeVries A. Price transparency for MRIs increased use of less costly providers and triggered provider competition. Health Aff (Millwood). 2014;33(8):1391-1398. [ CrossRef ] [ Medline ]
• Glover M, Whorms DS, Singh R, Almeida RR, Prabhakar AM, Saini S, et al. A radiology-focused analysis of transparency and usability of top U.S. hospitals' chargemasters. Acad Radiol. 2020;27(11):1603-1607. [ CrossRef ] [ Medline ]
• Hayatghaibi SE, Alves VV, Ayyala RS, Dillman JR, Trout AT. Transparency and variability in pricing for pediatric outpatient imaging in US children's hospitals. JAMA Netw Open. 2022;5(3):e220736. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Arvisais-Anhalt S, McDonald S, Park JY, Kapinos K, Lehmann CU, Basit M. Survey of hospital chargemaster transparency. Appl Clin Inform. 2021;12(2):391-398. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Reddy S, Daly G, Baban S, Kadesh A, Block AE, Grimes CL. Accessibility and usability of hospital chargemasters in New York state. J Gen Intern Med. 2022;37(8):2130-2131. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Robinson JC. Hospital quality competition and the economics of imperfect information. Milbank Q. 1988;66(3):465-481. [ Medline ]
• Patel KN, Mazurenko O, Ford E. Analysis of hospital quality measures and web-based chargemasters, 2019: cross-sectional study. JMIR Form Res. 2021;5(8):e26887. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Batty M, Ippolito B. Mystery of the chargemaster: examining the role of hospital list prices in what patients actually pay. Health Aff (Millwood). 2017;36(4):689-696. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Muhlestein DB, Wilks CEA, Richter JP. Limited use of price and quality advertising among American hospitals. J Med Internet Res. 2013;15(8):e185. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Batini C, Lenzerini M, Navathe SB. A comparative analysis of methodologies for database schema integration. ACM Comput Surv. 1986;18(4):323-364. [ FREE Full text ] [ CrossRef ]
• Voluntary hospital price transparency machine-readable file sample format data dictionary (version 1.1). CMS.gov. URL: https:/​/www.​cms.gov/​files/​document/​hospital-price-transparency-machine-readable-data-dictionary-tall.​pdf [accessed 2024-03-13]
• Bouayad L, Padmanabhan B, Chari K. Can recommender systems reduce healthcare costs? the role of time pressure and cost transparency in prescription choice. MIS Q. 2020;44(4):1859-1903. [ CrossRef ]

Abbreviations

Edited by S He; submitted 07.07.23; peer-reviewed by KN Patel, R Marshall, G Deckard; comments to author 03.12.23; revised version received 21.01.24; accepted 26.02.24; published 29.03.24.

©Negar Maleki, Balaji Padmanabhan, Kaushik Dutta. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 29.03.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

• Study Protocol
• Open access
• Published: 26 March 2024

The effect of a midwifery continuity of care program on clinical competence of midwifery students and delivery outcomes: a mixed-methods protocol

• Fatemeh Razavinia   ORCID: orcid.org/0000-0002-6827-509X 1 , 2 ,
• Parvin Abedi   ORCID: orcid.org/0000-0002-6980-0693 3 ,
• Mina Iravani   ORCID: orcid.org/0000-0002-8854-1738 4 ,
• Eesa Mohammadi   ORCID: orcid.org/0000-0001-6169-9829 5 ,
• Bahman Cheraghian   ORCID: orcid.org/0000-0001-5446-6998 6 ,
• Shayesteh Jahanfar   ORCID: orcid.org/0000-0001-6149-1067 7 &
• Mahin Najafian   ORCID: orcid.org/0000-0002-6649-3931 8  

BMC Medical Education volume  24 , Article number:  338 ( 2024 ) Cite this article

65 Accesses

Metrics details

The midwifery continuity of care model is one of the care models that have not been evaluated well in some countries including Iran. We aimed to assess the effect of a program based on this model on the clinical competence of midwifery students and delivery outcomes in Ahvaz, Iran.

This sequential embedded mixed-methods study will include a quantitative and a qualitative phase. In the first stage, based on the Iranian midwifery curriculum and review of seminal midwifery texts, a questionnaire will be developed to assess midwifery students’ clinical competence. Then, in the second stage, the quantitative phase (randomized clinical trial) will be conducted to see the effect of continuity of care provided by students on maternal and neonatal outcomes. In the third stage, a qualitative study (conventional content analysis) will be carried out to investigate the students’ and mothers’ perception of continuity of care. Finally, the results of the quantitative and qualitative phases will be integrated.

According to the nature of the study, the findings of this research can be effectively used in providing conventional midwifery services in public centers and in midwifery education.

Trial registration

This study was approved by the Ethics Committee of Ahvaz Jundishapur University of Medical Sciences (IR.AJUMS.REC.1401.460). Also, the study protocol was registered in the Iranian Registry for Randomized Controlled Trials (IRCT20221227056938N1).

Peer Review reports

Providing quality services to pregnant women has been recommended to all countries to achieve the Millennium Development Goals (MDGs) (Goals 3, 4 and 5) [ 1 ]. There are different care methods to maintain maternal and neonatal health during pregnancy and postpartum [ 1 ]. One of these care models is continuity of care that can be provided by a midwife or an obstetrician.

Midwifery continuity of care is a relationship-based care provided by a midwife who can be supported by one to three more midwives. They provide planned care for a woman during pregnancy, labor, birth, and the early postpartum period up to 6 weeks after delivery [ 2 ].

Continuity of midwifery care has become a global effort to enable women to have access to high-quality maternity care and delivery services [ 3 ]. As a result, many service providers today are transitioning to a continuous care model [ 4 ], and they have considered continuous care to be necessary for realizing women's rights [ 5 ]. Also, continuous midwifery care is known as the gold standard in maternity care to achieve excellent results for women [ 5 , 6 ]. In order to strengthen midwifery services to achieve global health goals in 2015, the World Health Organization (WHO) proposed a midwife-led continuous care model [ 7 ].

Countries use different midwifery care models. In Iran, for example, primary health services that are specific to pregnant mothers are provided in public health centers by midwives working in the network system and in compliance with the level of services and the referral system [ 8 ].

In general, midwifery continuous care not only has an important impact on a wide range of health and clinical outcomes for mothers and neonates but also brings about economic consequences for the health system [ 2 , 9 ]. This care model is useful for healthcare professionals as well [ 10 ], and it has improved the job satisfaction of midwives [ 11 ]. The midwife is the main guide in planning, organizing and providing care to a woman from the beginning of pregnancy to the postpartum period [ 12 ]. In 2011, in order to increase job motivation and satisfaction, promote retention of the midwifery workforce [ 13 ], and alleviate the shortage of workforce at the international level [ 14 ], the Nursing and Midwifery Advisory Center recommended using midwifery students (at the bedside and to perform midwifery work) to overcome this problem.

Providing high quality care requires enhancing the clinical competence of the professionals [ 4 ]. There is a close relationship between the concept of patient care quality and clinical competence. Therefore, clinical competence is of unique importance in midwifery practice [ 15 ]. As a result, in order to achieve quality patient care, midwifery professionals need to train students to become workforce with clinical competence in order to provide quality care in the health system. WHO defined clinical competence as a level of performance that demonstrates the effective application of knowledge, skills, and judgment [ 16 ].

A previous study showed that clinical competence of midwives plays an important role in managing the process of providing care, achieving care goals, and improving the quality of midwifery services [ 17 ]. In other words, the graduates of this field must have an acceptable level of clinical and professional skills in performing midwifery duties so that the health of mothers, children, and ultimately the community can be improved.

In Iran, prenatal care and the care during labor, delivery and postpartum are not continuous, and a new health provider may take the responsibility of care at any stage. This fragmented care may negatively affect the pregnancy outcomes and increase the rate of cesarean section [ 18 ]. Furthermore, the results of some studies in Iran indicate that the clinical competence obtained by midwifery students is far from optimal and that they do not acquire the necessary skills and abilities at the end of their studies [ 19 ]. Farrokhi et al. showed that the performance quality of 70% of midwives is average, and only 18.5% of them have good quality performance [ 20 ]. Several factors play a role in acquiring, maintaining and improving clinical competence [ 21 ]. There are a number of solutions that can increase the clinical competence of midwifery students, and one is the use of different care models such as the continuity of care model. The continuity of care model allows students to develop their midwifery knowledge, skills, and values individually [ 22 ]. Despite the strong foundation of midwifery in Iran, midwifery care models have not yet been tested. Some studies have reported that the quality of services provided during pregnancy, delivery and after delivery in Iran is poor to moderate. Also, these studies emphasize the necessity of a paradigm shift for better quality care and greater satisfaction of mothers, and they consider lack of continuity of care as the reason for the increase in unnecessary cesarean sections [ 23 , 24 , 25 ]. Moreover, the lack of qualified and experienced workforce has led to low quality health services, including midwifery care, and an increase in the economic burden of health. In Iran, no study has yet been conducted to investigate the effect of the midwifery continuity of care model on the students’ clinical competence and pregnancy outcomes. Given the importance of this topic, using a mixed-methods study design, we aimed to assess the effect of a midwifery continuity of care program on the clinical competence of midwifery students and pregnancy outcomes in Ahvaz, Iran.

Specific objectives

To determine the effect of midwifery continuity of care program on the clinical competence of midwifery students.

To determine the effect of a midwifery continuity of care program provided by midwifery students on pregnancy outcomes.

To explain the perception of midwifery students and mothers about the use of the midwifery continuity of care program provided by midwifery students.

Methods/design

Study design.

This sequential embedded mixed-methods study will include a quantitative phase and a qualitative one. A mixed (embedded) experimental design involves the collection and analysis of quantitative and qualitative data by the researcher and the integration of the information into an experimental study or intervention trial. This design adds qualitative data to an experiment or intervention to integrate the personal experience of research participants. Therefore, the qualitative data are converted into a secondary source of data embedded before and after the test. Qualitative data is added to the experiment in differrent ways, including: before the experiment, during the experiment, or after the experiment [ 26 , 27 ]. Embedded mixed-methods studies that are qualitative followed by quantitative are used to understand the rationale for the results and receive feedback from participants (to confirm and support the findings of the quantitative studies) [ 27 ]. In the first stage of this study, a questionnaire for assessing midwifery students’ clinical competence will be created based on the midwifery curriculum of Iran and a review of seminal texts of midwifery. Then, the effect of continuity of care provided by midwifery students on maternal and neonatal outcomes will be assessed in a randomized clinical trial. In the third stage, a qualitative study will be carried out to investigate the perception of students and mothers. Finally, the results of the quantitative and qualitative phases will be integrated (Fig.  1 ).

Sequential and embedded mixed-methods design

First stage: questionnaire development

This questionnaire will be developed based on midwifery curriculum and a comprehensive and systematic search (with no time limit) in English and Persian databases (Web of Science, Embase, Scopus, ProQuest, Google scholar, Magiran, SID).

Tool design

There are four steps in tool development:

Choosing a conceptual model to show aspects of clinical competence in the measurement process

Explaining the purpose of the tool

Designing the route map

Developing the tool (use of methods, classification of objects, rules and procedures for scoring tools) [ 28 ].

Answer to the objects

A 1 to 4-point Likert scale will be used for scoring [ 29 ].

Content validity

To ensure the selection of the most important and correct content (necessity of the case), the content validity will be assessed. Also, to ensure that the instrument items are designed in the best way to measure the content, the content validity index will be calculated [ 30 ].

Reliability

Reliability will be evaluated using internal consistency (Cronbach's alpha coefficient ≥ 0.7) and stability (test-re-test ≥ 0.74) by piloting the questionnaire on 20 midwifery students [ 31 ].

Second stage: quantitative phase

A randomized controlled clinical trial will be conducted in this phase of research to examine the effect of the continuous care program of midwifery students on their clinical competence and pregnancy outcomes.

Sample size

According to the study objective and previous study results [ 32 ] with α = 0.01, β = 0.1, p 1  = 0.51 and p 2  = 0.021, the sample size will be n  = 23. Considering a 20% dropout rate, the final sample size will be 58 women (29 women in each group).

Data collection

This phase of the randomized clinical trial will be conducted with the participation of 58 undergraduate midwifery students at their 7th and 8th semesters. The students will be divided randomly to intervention (continuous care) and control (routine care) groups providing care to 58 pregnant women in six health centers and two hospitals (Sina and Razi) in Ahvaz city, southwest of Iran.

The study will begin after receiving the approval of the Ethics Committee of Ahvaz University of Medical Sciences and registering the study in the Iranian Registry for Randomized Clinical Trials. Inclusion criteria will be willingness to participate in the study.

Randomization

To implement the intervention, the students will be divided into two intervention (providing continuous care for pregnant women) and control (providing standard care for pregnant women) groups. Allocating students will be done using permuted block randomization technique with a block size of four and an allocation ratio of 1:1. Five blocks of 4 pieces and 3 blocks of 3 pieces will be extracted randomly using WIN PEPI software. In each block of 4, 2 students will be in control and 2 will be in intervention group. Also, in each block of 3 students, 1 student will be in control and 2 will be in intervention group, and the arrangement of each person is random. To prevent contamination, first the control group will provide routine care, and then the intervention group will conduct continuity of care for pregnant women. Mothers are randomly selected based on the hospital where they will give birth. As a result, Razi Hospital will be the control group and Sina Hospital will be the intervention group.

Intervention

Women who meet the inclusion criteria will be recruited in the study using a non-probability convenience sampling method. Women in the intervention group will be included in the study after their first pregnancy visit (6–10 weeks of gestation) and will receive continuous care by midwifery students. Women in the control group will receive the usual and routine care, and will be included in the study at the time of delivery. They will have a gestational age of more than 37 weeks based on the inclusion criteria of the study. Their delivery will be performed by midwifery students who will follow them up until six weeks after delivery.

At first, the necessary training will be given by the lead researcher (FR) to the students in orientation sessions held for both groups separately. In the intervention group, each midwifery student as the main midwife will be responsible for taking care of two or three pregnant women and will be the back-up midwife for two other pregnant women (under the supervision of other students). The lead researcher will create a group in WhatsApp with the participation of students in the intervention group, and they can communicate with each other and the researcher. Also, the midwifery students will be directly and indirectly under the supervision of a qualified person (lead researcher). Another WhatsApp group will be created for the women of the intervention and control groups (to facilitate communication between the researcher and the women). Two midwifery students will be introduced to each pregnant woman in the intervention group (as a main midwife and a backup midwife). If the main midwife is not available, the woman will be in contact with the backup midwife. The backup student will meet the woman at least once and will be introduced to her.

Instruments

All students and pregnant women participating in this study will complete a demographic questionnaire. A checklist will be provided for collecting data during prenatal care, labor, and delivery.

Also, the midwifery students will complete the clinical competency questionnaire at the beginning and end of the study.

Care will be provided and recorded by the main student according to the pregnancy care protocol. Also, danger signs will be taught to the students according to the national protocol, and emergencies will be handled by the midwifery student under the supervision of the lead researcher. Admission to hospital will be arranged by the student, and all information will be recorded. Pregnancy, labour and delivery, postpartum, and newborn checklist will be completed. Students will complete a demographic and obstetric questionnaire that includes questions about age, education, occupation, gravidity, parity, abortions, live and dead children, last contraceptive method, intended and unintended pregnancies, last menstrual period (LMP), gestational age, date of birth, body mass index (BMI), previous pregnancy and childbirth records, high-risk behavior of the mother and father, current history of special care, test and ultrasound results, and participation in childbirth preparation class. Also, the following data will be recorded in the labor and delivery and post-partum checklist: checking the conditions of labor according to the partograph, length of labor, need for induction and the method used type of delivery, examination of perineal trauma, postpartum bleeding, and examination of the condition of the mother up to 6 weeks after delivery. In addition, the amount of bleeding will be checked visually and by measuring the level of hemoglobin and hematocrit. Apgar score of the newborn will be recorded (in infant checklist) in minutes 1 and 5. Also, the newborn’s hospitalization status, breastfeeding and anthropometric indices will be recorded.

The students in the intervention group will start prenatal care < 20 weeks of gestation. At least five round of prenatal care will be provided by each student according to national guidelines for each pregnant woman. Pregnant women can communicate with their in-charge students in non-emergency cases from 8:00 a.m. to 23:00 p.m. and in emergency cases 24 h a day, all days a week. All reports will be recorded by the students. During labor and delivery, the student and the lead researcher will be present at the mother's bedside. In case of natural vaginal delivery (NVD), delivery will be done by a student midwife under the supervision of the researcher. In case of cesarean delivery (CS), a student will be present at the patient's bedside. Postpartum care will be provided by midwifery students in both groups (intervention and control). Each student will be at the mother's bedside for two hours after delivery. The conditions of labor, delivery, and the neonate will be recorded by the student in the relevant form. Also, the mother will be followed up by telephone for up to 6 weeks after delivery (postpartum). The clinical competency questionnaire will be completed by students before and after the intervention.

Inclusion criteria

Inclusion criteria for midwifery students will be: studying at the seventh and eighth semester and willingness to participate in the study.

Inclusion criteria for service recipients (pregnant women) will be: age 18 – 40 years, Iranian nationality, singleton pregnancy, low risk pregnancy, and gestational age < 20 wks.

Exclusion criteria

Exclusion criteria will be: history of psychiatric disorders, previous caesarean section, use of alcohol and tobacco, or having a disease that requires prenatal care by a specialist.

Primary outcome

Clinical competence of midwifery students.

Secondary outcome

Mode of delivery, length of labor stages, the need to induction, postpartum bleeding first and fifth minute Apgar score, admission of neonate to the neonatal intensive care unit, breastfeeding initiation, and exclusive breastfeeding up to 6 weeks postpartum.

Data analysis

Statistical analyses will be done using SPSS version 26.0 (SPSS, Inc., Chicago, IL, USA). The independent t-test and Chi-square tests will be used for continuous data and categorical data, respectively. ANCOVA test will be used to eliminate the influence of confounding variables. The effect size will be calculated. A 95% confidence interval (CI) and p values will be reported. P -values less than 0.5 will be considered statistically significant.

Third step of research: qualitative study

This phase will be a qualitative study using conventional content analysis.

Purposeful sampling will be used in this study [ 33 ]. Sampling will continue until data saturation [ 34 ], i.e., no new information or data about a class or relationships between classes is revealed.

This phase of the study is a conventional qualitative content analysis [ 35 ] aimed at examining the perceptions of midwifery students and mothers receiving continuous care. The researcher will conduct in-depth, semi-structured interviews with open-ended questions with students and mothers in the group of the continuous care program. All interviews will be done by the lead researcher who is qualified in qualitative research method. The interview will start with a general and open question such as: “Please tell me about your experiences or feelings about participating in the continuous midwifery care program. How did you feel about participating in this program?” Then, in-depth exploratory questions will be asked based on their answers (e.g., what do you mean? Why? Can you elaborate on that? Can you give me an example so I can understand what you mean?). All interviews will be recorded with the participants' consent. Paralinguistic features, such as mood and features of the participants, including tone of voice, facial expressions, and their posture, will be recorded by the researcher during the interview [ 35 ].

The data will be analyzed based on Granheim and Lundman's 2004 content analysis approach [ 36 ].

Interviews will be transcribed at the end of each interview. Data analysis begins with a careful study of all data so that the researcher can immerse herself in the data and gain an overview. Interviews will be transcribed verbatim. Key concepts will be highlighted and codes will be extracted. Then the first interpretations will be made and analyzed. Labels emerge for codes that represent more than one key concept and are usually taken directly from the text and become the initial coding map. Then the codes are placed in the category based on their similarity. Then, definitions will be created for each category, subcategory and code. When reporting findings, examples of each code and data category will be provided [ 35 ].

Inclusion criteria for midwifery students will be: studying at the seventh or eighth semester, willingness to participate in the study.

Inclusion criteria for service recipients (pregnant women) will be: receiving continuous care provided by the student, willingness to participate in the study, and being able to communicate.

The qualitative study and interview data will be analyzed based on the content analysis approach of Granheim and Lundman 2004 [ 36 ] as follows:

Reading and re-reading the interviews after completion of each interview

Selection of the unit of analysis

Determination of semantic units

Classification

Extraction of information content

In the first step, the data is converted into text format. As soon as possible after the interview, the interview will be typed verbatim. Then the whole text will be read several times to get a general understanding of the content of interview. Each meaning unit will be converted into condensed meaning units and then coded. The Codes will be classified into subcategories and categories based on their common characteristics. Finally, the content of the categories will be revealed, taking into account their hidden meaning [ 36 ].

Trustworthiness

Five criteria of will be used to increase data trustworthiness according to Lincoln & Guba [ 37 ]. These include: 1. Credibility, 2. Dependability, 3. Confirmability, 4. Transferability, 5. Authenticity.

Credibility of the data will be ensured by continuous engagement of the researchers with the subject, member checks, and external checks. Dependability will be ensured by relying on the insight of external observers. In order to increase the confirmability, data will be accurately recorded and reported. Also, transferability will be ensured by presenting the research process accurately, clearly and purposefully, which includes purposive sampling and presenting the research results to a number of people with the same profile of the participants who did not participate in the research. Finally, authenticity will be guaranteed by continuous reflection on information, long-term presence of the researcher, interview recording, writing, and reporting of findings.

Combining qualitative and quantitative phases

Data combination will be done using data integration strategies. The integration or combination of data starts from quantitative data analysis. Then qualitative data is collected by interview. In fact, the qualitative study is a secondary source of embedded data in the collection of experimental test data (continuous care) after the quantitative study. In this research, in order to understand the results of the RCT, the views of the participants will be unified in order to get a correct understanding of the intervention (implementation of the continuity of care model by the students) from the mothers' and students' point of view (Fig.  2 ).

Study diagram

Study status

The development of the evaluation tools was made. Also, sampling the quantitative phase of the study and the basic of the program are in process (Table 1 ).

This is the first mixed-methods study to be conducted in Iran investigating the effect of a midwifery continuity of care program on clinical competence of midwifery students and pregnancy outcomes. According to the recommendations of the WHO, midwifery continuity of care should be adopted in order to increase the quality of pregnancy care as well as the satisfaction of pregnant women and service providers [ 7 ]. Contrary to the recommendation of WHO, the continuous care program is neither implemented in Iran's health system nor included in the midwifery curriculum. The results of this study can help health planners and policy makers to implement high quality midwifery care program based on global recommendations.

The study has several strengths. The use of a mixed-methods study design (combination of quantitative and qualitative approaches) in contrast to the separate use of quantitative and qualitative studies provides a better understanding of the research questions [ 38 ]. In embedded design, one type of data collection (quantitative or qualitative) plays a supporting and essential role for another type. As a result, the embedded mixed-methods technique in the qualitative phase after designing the intervention will be used to receive feedback from the participants to confirm and support the findings of quantitative phase [ 39 ]. Also, interviews with mothers and midwifery students in the intervention group can reflect their positive and negative experiences of this program. Considering that Iran's healthcare system lacks continuous midwifery care, the findings of this research can be effectively used in providing conventional midwifery services in public centers and in midwifery education.

Considering that this care model will be implemented for the first time in Iran's midwifery education and healthcare system, there may be two possible limitations in this study: lack of infrastructure and interference with other educational programs.

Availability of data and materials

All the data that will be obtained will be published in the next article after the implementation of the study.

Abbreviations

Body mass index

Cesarean section

Last menstrual period

Millennium Development Goals

Natural vaginal delivery

World Health Organization

Bagheri A, Simbar M, Samimi M, Nahidi F, Majd HA. Exploring the concept of continuous midwifery-led care and its dimensions in the prenatal, perinatal, and postnatal periods in Iran (Kashan). Midwifery. 2017;51:44–52.

Article   Google Scholar  

Cummins A, Coddington R, Fox D, Symon A. Exploring the qualities of midwifery-led continuity of care in Australia (MiLCCA) using the quality maternal and newborn care framework. Women Birth. 2020;33(2):125–34.

Choudhary S, Jelly P, Mahala P. Models of maternity care: a continuity of midwifery care. Int J Reprod Contracept Obstet Gynecol. 2020;9(6):2666–70.

Bradford BF, Wilson AN, Portela A, McConville F, Fernandez Turienzo C, Homer CS. Midwifery continuity of care: a scoping review of where, how, by whom and for whom? PLOS Global Public Health. 2022;2(10):e0000935.

Lettink A, Chaibekava K, Smits L, Langenveld J, van de Laar R, Peeters B, et al. CCT: continuous care trial-a randomized controlled trial of the provision of continuous care during labor by maternity care assistants in the Netherlands. BMC Pregnancy Childbirth. 2020;20(1):1–6.

Chaibekava KV, Scheenen AJ, Lettink A, Smits LJ, Langenveld J, Van De Laar R, et al. Continuous care during labor by maternity care assistants in the Netherlands vs care-as-usual: a randomized controlled trial. Am J Obstet Gynecol MFM. 2023;5(11):101168.

Michel-Schuldt M, McFadden A, Renfrew M, Homer C. The provision of midwife-led care in low-and middle-income countries: an integrative review. Midwifery. 2020;84:102659.

Khosravi S, Babaey F, Abedi P, Kalahroodi ZM, Hajimirzaie SS. Strategies to improve the quality of midwifery care and developing midwife-centered care in Iran: analyzing the attitudes of midwifery experts. BMC Pregnancy Childbirth. 2022;22(1):40.

Donnellan Fernandez R, Scarf V, Devane D, Healey A. Is Midwifery Continuity of Care Cost Effective? Midwifery Continuity of Care. Elsevier; 2019.  p. 137–56.

Aune I, Tysland T, Amalie VS. Norwegian midwives’ experiences of relational continuity of midwifery care in the primary healthcare service: a qualitative descriptive study. Nordic J Nurs Res. 2021;41(1):5–13.

Hanley A, Davis D, Kurz E. Job satisfaction and sustainability of midwives working in caseload models of care: an integrative literature review. Women Birth. 2022;35(4):e397–407.

Grylka-Baeschlin S, Iglesias C, Erdin R, Pehlke-Milde J. Evaluation of a midwifery network to guarantee outpatient postpartum care: a mixed methods study. BMC Health Serv Res. 2020;20:1–12.

Evans J, Taylor J, Browne J, Ferguson S, Atchan M, Maher P, et al. The future in their hands: graduating student midwives’ plans, job satisfaction and the desire to work in midwifery continuity of care. Women Birth. 2020;33(1):e59–66.

Carter J, Sidebotham M, Dietsch E. Prepared and motivated to work in midwifery continuity of care? A descriptive analysis of midwifery students’ perspectives. Women Birth. 2022;35(2):160–71.

Inoue N, Nakao Y, Yoshidome A. Development and validity of an intrapartum self-assessment scale aimed at instilling midwife-led care competencies used at freestanding midwifery units. Int J Environ Res Public Health. 2023;20(3):1859.

Vázquez-Sánchez C, Gigirey L, editors. Design of a scoring rubric for the assessment of clinical competencies in the subject of optometry IV of the degree of optics and optometry-USC. INTED2023 Proceedings. Valencia: IATED; 2023. p. 2224–30. https://doi.org/10.21125/inted.2023.0613 .

Kubota S, Ando M, Murray J, Khambounheuang S, Theppanya K, Nanthavong P, et al. A regulatory gap analysis of midwifery to deliver essential reproductive, maternal, newborn, child and adolescent health services in Lao People’s Democratic Republic. Lancet Reg Health West Pac. 2024;43:1–10. https://doi.org/10.1016/j.lanwpc.2023.100959 .

Pazandeh F, Moridi M, Safari K. Labouring women perspectives on mistreatment during childbirth: a qualitative study. Nursing Ethics. 2023;30(4):513–25. https://doi.org/10.1177/09697330231158732 .

Abbasi A, Bazghaleh M, FadaeeAghdam N, Basirinezhad MH, Tanhan A, Montazeri R, et al. Efficacy of simulated video on test anxiety in objective structured clinical examination among nursing and midwifery students: a quasi-experimental study. Nurs Open. 2023;10(1):165–71.

Farokhi F. Quality assessment of midwives performance in prenatal cares in urban health centers in Mashhad, Iran. Payesh (Health Monitor). 2008;7(3):0.

Google Scholar  

Moradali MR, Hajian S, Majd HA, Rahbar M, Entezarmahdi R. Job Satisfaction and its Related Factors in Midwives Working in the Health Services System in Iran: A Systematic Review. J Midwifery Reprod Health. 2023;11(2):3650–63. https://doi.org/10.22038/jmrh.2023.64824.1890 .

Hainsworth N, Cummins A, Newnham E, Foureur M. Learning through relationships: the transformative learning experience of midwifery continuity of care for students: a qualitative study. Women Birth. 2023;36(4):385–92.

Lazar J. Exploring the experiences of midwives facilitating group antenatal care: City, University of London; 2023.

Adelson P, Fleet J-A, McKellar L. Evaluation of a regional midwifery caseload model of care integrated across five birthing sites in South Australia: women’s experiences and birth outcomes. Women Birth. 2023;36(1):80–8.

Prussing E, Browne G, Dowse E, Hartz D, Cummins A. Implementing midwifery continuity of care models in regional Australia: a constructivist grounded theory study. Women Birth. 2023;36(1):99–107.

Pintubatu J. keterampilan proses sains pada pembelajaran ipas berorientasi outdoor learning siswa SMK Negeri 1 Lolak. Charm Sains: Jurnal Pendidikan Fisika. 2023;4(1):7–12.

Creswell JW. Controversies in mixed methods research. Sage Handbook Qual Res. 2011;4(1):269–84.

Kristensen SB, Clausen A, Skjødt MK, Søndergaard J, Abrahamsen B, Möller S, et al. An enhanced version of FREM (Fracture Risk Evaluation Model) using national administrative health data: analysis protocol for development and validation of a multivariable prediction model. Diagn Progn Res. 2023;7(1):19.

Alabi AT, Jelili MO. Clarifying likert scale misconceptions for improved application in urban studies. Qual Quant. 2023;57(2):1337–50.

Roopashree MR. A pragmatic approach for the calculation content validity indices: a study on validation of training tool for pre and post-test questionnaire for the health care sector. QAI J Healthc Qual Patient Saf. 2023;4(1):17–23.

Martyushev NV, Malozyomov BV, Sorokova SN, Efremenkov EA, Valuev DV, Qi M. Review models and methods for determining and predicting the reliability of technical systems and transport. Mathematics. 2023;11(15):3317.

Hildingsson I, Karlström A, Haines H, Johansson M. Swedish women’s interest in models of midwifery care–Time to consider the system? A prospective longitudinal survey. Sex Reprod Healthc. 2016;7:27–32.

Creswell JW, Creswell JD. Research design: Qualitative, quantitative, and mixed methods approaches. 3rd ed. SAGE Publications Inc.; 2017.

Obilor EI. Convenience and purposive sampling techniques: are they the same. Int J Innov Soc Sci Educ Res. 2023;11(1):1–7.

Serafini F, Reid SF. Multimodal content analysis: expanding analytical approaches to content analysis. Vis Commun. 2023;22(4):623–49.

Graneheim UH, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today. 2004;24(2):105–12.

Lincoln YS, Guba EG. Naturalistic inquiry. SAGE Publications; 1985.

Matović N, Ovesni K. Interaction of quantitative and qualitative methodology in mixed methods research: integration and/or combination. Int J Social Res Methodol. 2023;26(1):51–65.

Creswell JW, Clark VLP. Designing and conducting mixed methods research. SAGE Publication, Inc; 2017.

Download references

The study was funded by Ahvaz Jundishapur University of Medical Sciences.

Author information

Authors and affiliations.

Midwifery Department, Reproductive Health Promotion Research Center, Ahvaz Jundishapur University of Medical Sciences, Ahvaz, Iran

Fatemeh Razavinia

Midwifery Department, Menopause Andropause Research Center, Ahvaz Jundishapur University of Medical Sciences, Ahvaz, Iran

Midwifery Department, Menopause Andropause Research Center, Ahvaz Jundisahpur University of Medical Sciences, Golestan BLvd, Ahvaz, Iran

Parvin Abedi

Reproductive Health Promotion Research Center, Midwifery Department, Nursing and Midwifery School, Ahvaz Jundishapur University of Medical Sciences, Ahvaz, Iran

Mina Iravani

Department of Nursing, Faculty of Medical Sciences, Tarbiat Modares University, Tehran, Iran

Eesa Mohammadi

Alimentary Tract Research Center, Clinical Sciences Research Institute, Department of Biostatistics and Epidemiology, School of Public Health, Ahvaz Jundishapur University of Medical Sciences, Ahvaz, Iran

Bahman Cheraghian

MPH Program, Department of Public Health and Community Medicine, Tufts University School of Medicine, Boston, USA

Shayesteh Jahanfar

Department of Obstetrics and Gynecology, School of Medicine, Ahvaz Jundishapur University of Medical Sciences, Ahvaz, Iran

Mahin Najafian

You can also search for this author in PubMed   Google Scholar

Contributions

FR, PA, MI, EM, BCh, ShJ and MN conceptualized the study. FR will collect the data. FR drafted the protocol. PA revised the manuscript. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Parvin Abedi .

Ethics declarations

Ethics approval and consent to participate.

This study was approved by the Ethics Committee of Ahvaz Jundishapur University of Medical Sciences (IR.AJUMS.REC.1401.460). Also, the study protocol was registered in the Iranian Registry for Randomized Controlled Trials (IRCT20221227056938N1). Informed consent will be obtained from all participants. The study’s findings will be shared via the publishing of peer-reviewed articles, talks at scientific conferences and meetings with related teams.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Razavinia, F., Abedi, P., Iravani, M. et al. The effect of a midwifery continuity of care program on clinical competence of midwifery students and delivery outcomes: a mixed-methods protocol. BMC Med Educ 24 , 338 (2024). https://doi.org/10.1186/s12909-024-05321-5

Download citation

Received : 28 October 2023

Accepted : 15 March 2024

Published : 26 March 2024

DOI : https://doi.org/10.1186/s12909-024-05321-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Continuity of care
• Clinical competence
• Mixed-methods
• Midwifery students
• Pregnancy outcomes

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]