research study using mixed methods

No internet connection.

All search filters on the page have been cleared., your search has been saved..

• All content
• Dictionaries
• Encyclopedias
• Expert Insights
• Foundations
• How-to Guides
• Journal Articles
• Little Blue Books
• Little Green Books
• Project Planner
• Tools Directory
• Sign in to my profile My Profile

• Sign in Signed in
• My profile My Profile

The Sage Handbook of Mixed Methods Research Design

• Edited by: Cheryl N. Poth
• Publisher: Sage Publications Ltd
• Publication year: 2023
• Online pub date: January 16, 2024
• Discipline: Sociology , Criminology and Criminal Justice , Business and Management , Communication and Media Studies , Education , Psychology , Health , Social Work , Political Science and International Relations
• Methods: Mixed methods , Research design , Quantitative data collection
• DOI: https:// doi. org/10.4135/9781529682663
• Keywords: handbooks , teams Show all Show less
• Print ISBN: 9781529723960
• Online ISBN: 9781529682663
• Buy the book icon link

Subject index

The Sage Handbook of Mixed Methods Research Design is a ground-breaking edited work that weaves together diverse perspectives and global examples of mixed-methods research to present a timely picture of this rapidly evolving field. With contributions from over 80 of the biggest names and rising stars of the field, this Handbook is an essential resource for anyone interested in the contemporary, emerging, and evolving practice of mixed methods research and scholarship. Exploring new and novel applications of existing mixed methods research design practices, the handbook provides comprehensive integration guidance while showcasing how design innovations inspire and contribute to investigating previously under-researched social issues and populations. Through its unique focus on design and the diverse contexts in which mixed methods research is being applied, this Handbook prepares researchers for the changing conditions in which they will conduct studies. Newcomers and seasoned mixed methods researchers alike will find this Handbook a go-to source for tools to think and act 'complexively' and creatively in research design. Using accessible language and illustrative examples, this Handbook is written for those with various roles and experience in mixed methods research design. The in-depth discussions led by the interdisciplinary group of 11 internationally renowned editorial section leads project our collective thinking of mixed methods research design into the future across the following six sections: Section 1: Inspiring Diversity and Innovation in Mixed Methods Design; Section 2: The Craft of Mixed Methods Research Design; Section 3: Expanding Mixed Methods Design Approaches; Section 4: Designing Innovative Integrations with Technology; Section 5: Navigating Research Cultures in Mixed Methods Design; and Section 6: Exploring Design Possibilities and Challenges for Mixed Methods Research

Front Matter

• Editorial Section Leads
• International Advisory Board
• List of Figures
• List of Tables
• List of Box
• Notes on the Editor, Section Leads, and Chapter Contributors
• Acknowledgements
• Chapter 1: Dilemmas and Opportunities for Mixed Methods Research Design: Handbook Introduction
• Evolving Tensions and Conversations in Mixed Methods Research Design Approaches: Section 1 Introduction
• Chapter 2: Revisiting Mixed Methods Research Designs Twenty Years Later
• Chapter 3: Mixed Methods Design in Historical Perspective: Implications for Researchers
• Chapter 4: Mixed Methods Designs to Further Social, Economic and Environmental Justice
• Chapter 5: Developments in Mixed Methods Designs: What Have Been the Dominant Pathways and Where Might They Take Us in the Future?
• Chapter 6: The Role of Methodological Paradigms for Dialogic Knowledge Production: Using a Conceptual Map of Discourse Development to Inform Mixed Methods Research Design
• Future Tensions and Design Conversations in the Mixed Methods Field: Section 1 Conclusions
• The Craft of Mixed Methods Research Design: Section 2 Introduction
• Chapter 7: Embracing Emergence in Mixed Methods Designs: Theoretical Foundations and Empirical Applications
• Chapter 8: The Methods-Inference Map: Visualizing the Interactions Between Methods and Inferences in Mixed Methods Research
• Chapter 9: Towards Sampling Designs that are Transparent, Rigorous, Ethical and Equitable (TREE): Using a Tree Metaphor as a Sampling Meta-Framework in Mixed Methods Research
• Chapter 10: Data Integration as a Form of Integrated Mixed Analysis in Mixed Methods Research Designs
• Chapter 11: Ethical Issues and Practices for Mixed Methods Research in an Era of Big Data
• Chapter 12: Building the Logic for an Integrated Methodology: Mixed Method Grounded Theory as an Example of Constructing a Methodology to Guide Design and Integration
• The Craft of Mixed Methods Research Design: Section 2 Conclusions
• Expanding Beyond Typology-Based Mixed Methods Designs: Section 3 Introduction
• Chapter 13: Exploring Interlocking Relationships of Race, Gender, and Class with an Intersectionality-Informed Mixed Methods Research Design Framework
• Chapter 14: Indigenous Cultural Values Instrument Development: Using Mixed Methods Research
• Chapter 15: What Can Mixed Methods Partnerships Learn from Kaupapa Māori Research Principles?
• Chapter 16: Prioritizing Cultural Responsiveness in Mixed Methods Research and Team Science with Underrepresented Communities
• Chapter 17: Using Participatory Methods in Randomised Controlled Trials of Complex Interventions
• Chapter 18: Illustrating the Mixed Methods Phenomenological Approach (MMPR)
• Chapter 19: Intersection of Mixed Methods and Case Study Research (MM+CSR): Two Design Options in Educational Research
• Chapter 20: Harnessing Mixed Methods for Research Instrument Development and Legitimation
• Chapter 21: Mixed Methods-Grounded Theory: Best Practices for Design and Implementation
• Moving Beyond Tradition: The Need for Expanded and Culturally Relevant Mixed Methods Design Typologies: Section 3 Conclusions
• Expanding Innovative Integrations with Technology: Section 4 Introduction
• Chapter 22: Using Software for Innovative Integration in Mixed Methods Research: Joint Displays, Insights and Inferences with MAXQDA
• Chapter 23: Grounded Text Mining Approach: An Integration Strategy of Grounded Theory and Textual Data Mining
• Chapter 24: A “Mixed Methods Way of Thinking” in Game-based Research Integrations
• Chapter 25: Integrating Secondary Data from Ethnically and Racially Minoritized Groups in Mixed Methods Research
• Chapter 26: Beyond the Joint Display in Mixed Methods Convergent Designs: A Case-Oriented Merged Analysis
• The Untapped Potential of Technology for Integration: Section 4 Conclusions
• From Margin to Center: The Design Implications of a Cultural Component in Mixed Methods Research: Section 5 Introduction
• Chapter 27: Culturally Responsive Mixed Methods Evaluation Design
• Chapter 28: Integrating a Four-Step Japanese Cultural Narrative Framework, Ki-Shou-Ten-Ketsu, into a Mixed Methods Study
• Chapter 29: Leveraging Mixed Methods Community-based Participatory Research (MMCBPR) in Diverse Social and Cultural Contexts to Advance Health Equity
• Chapter 30: Cultural Diversity in Intervention Designs: A Chinese Illustrative Example
• Chapter 31: Examining the Influences of Spanish Research Culture in Systematic Observation with Mixed Methods
• Future Direction for Navigating Research Cultures in Designs: Section 5 Conclusions
• Exploring Possibilities and Challenges for Mixed Methods Research for the Future: Section 6 Introduction
• Chapter 32: Visualizing the Process: Using Visuals to Teach and Learn Mixed Methods Research
• Chapter 33: Towards the Future Legitimacy of Mixed Methods Designs: Responsible Mixed Methods Research for Tackling Grand Challenges for the Betterment of Society
• Chapter 34: Realizing Methodological Potentials and Advantages of Mixed Methods Research Design for Knowledge Translation
• Chapter 35: Opportunities and Challenges for a Transdisciplinary Mixed Methods Research Future
• Chapter 36: Mapping Design Trends and Evolving Directions Using the Sage Handbook of Mixed Methods Research Design
• Where to Next in Exploring Possibilities and Challenges for Mixed Methods Research for the Future? Section 6 Conclusions
• Chapter 37: An Emerging and Exciting Future for Mixed Methods Research Design: Handbook Conclusions

Sign in to access this content

Get a 30 day free trial, more like this, sage recommends.

We found other relevant content for you on other Sage platforms.

Have you created a personal profile? Login or create a profile so that you can save clips, playlists and searches

• Sign in/register

Navigating away from this page will delete your results

Please save your results to "My Self-Assessments" in your profile before navigating away from this page.

Sign in to my profile

Sign up for a free trial and experience all Sage Learning Resources have to offer.

You must have a valid academic email address to sign up.

Get off-campus access

• View or download all content my institution has access to.

Sign up for a free trial and experience all Sage Research Methods has to offer.

• view my profile
• view my lists

Handbook of Research Methods in Health Social Sciences pp 695–713 Cite as

The Use of Mixed Methods in Research

• Kate A. McBride 2 ,
• Freya MacMillan 3 ,
• Emma S. George 4 &
• Genevieve Z. Steiner 5  
• Reference work entry
• First Online: 13 January 2019

2416 Accesses

9 Citations

Mixed methods research is becoming increasingly popular and is widely acknowledged as a means of achieving a more complex understanding of research problems. Combining both the in-depth, contextual views of qualitative research with the broader generalizations of larger population quantitative approaches, mixed methods research can be used to produce a rigorous and credible source of data. Using this methodology, the same core issue is investigated through the collection, analysis, and interpretation of both types of data within one study or a series of studies. Multiple designs are possible and can be guided by philosophical assumptions. Both qualitative and quantitative data can be collected simultaneously or sequentially (in any order) through a multiphase project. Integration of the two data sources then occurs with consideration is given to the weighting of both sources; these can either be equal or one can be prioritized over the other. Designed as a guide for novice mixed methods researchers, this chapter gives an overview of the historical and philosophical roots of mixed methods research. We also provide a practical overview of its application in health research as well as pragmatic considerations for those wishing to undertake mixed methods research.

• Mixed methods
• Concurrent triangulation
• Sequential exploratory
• Sequential explanatory
• Convergent parallel
• Embedded design
• Transformative design
• Multiphase design

This is a preview of subscription content, log in via an institution .

Buying options

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Appleton JV, King L. Journeying from the philosophical contemplation of constructivism to the methodological pragmatics of health services research. J Adv Nurs. 2002;40(6):641–8.

Article   Google Scholar  

Baba CT, Oliveira IM, Silva AEF, Vieira LM, Cerri NC, Florindo AA, de Oliveira Gomes GA. Evaluating the impact of a walking program in a disadvantaged area: using the RE-AIM framework by mixed methods. BMC Public Health. 2017;17(1):709. https://doi.org/10.1186/s12889-017-4698-5 .

Bryman A. The research question in social research: what is its role? Int J Soc Res Methodol. 2007;10(1):5–20. https://doi.org/10.1080/13645570600655282 .

Bryman A. The end of the paradigm wars. In: Alasuutari P, Bickman L, Brannen J, editors. The Sage handbook of social research methods. Thousand Oaks: Sage; 2008. p. 13–25.

Google Scholar  

Caracelli VJ, Greene JC. Data-analysis strategies for mixed-method evaluation designs. Educ Eval Policy Anal. 1993;15(2):195–207. https://doi.org/10.3102/01623737015002195 .

Castro FG, Kellison JG, Boyd SJ, Kopak A. A methodology for conducting integrative mixed methods research and data analyses. J Mixed Methods Res. 2010;4(4):342–60. https://doi.org/10.1177/1558689810382916 .

Center for Innovation in Teaching in Research. Choosing a mixed methods design. 2017. Retrieved 15 Nov 2017 from https://cirt.gcu.edu/research/developmentresources/research_ready/mixed_methods/choosing_design .

Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. 3rd ed. Thousand Oaks: Sage; 2018.

Curry LA, Krumholz HM, O’Cathain A, Clark VLP, Cherlin E, Bradley EH. Mixed methods in biomedical and health services research. Circ-Cardiovasc Qual Outcomes. 2013;6(1):119–23. https://doi.org/10.1161/circoutcomes.112.967885 .

Denscombe M. Communities of practice: a research paradigm for the mixed methods approach. J Mixed Methods Res. 2008;2(3):270–83.

Doyle L, Brady A-M, Byrne G. An overview of mixed methods research. J Res Nurs. 2009;14(2):175–85.

Foster NE, Bishop A, Bartlam B, Ogollah R, Barlas P, Holden M, … Young J. Evaluating acupuncture and standard carE for pregnant women with back pain (EASE back): a feasibility study and pilot randomised trial. Health Technol Assess. 2016;20(33):1–236. https://doi.org/10.3310/hta20330 .

Haider AH, Schneider EB, Kodadek LM, Adler RR, Ranjit A, Torain M, … Lau BD. Emergency department query for patient-centered approaches to sexual orientation and gender identity the EQUALITY study. JAMA Intern Med. 2017;177(6):819–828. https://doi.org/10.1001/jamainternmed.2017.0906 .

Harding KE, Taylor NF, Bowers B, Stafford M, Leggat SG. Clinician and patient perspectives of a new model of triage in a community rehabilitation program that reduced waiting time: a qualitative analysis. Aust Health Rev. 2013;37(3):324–30. https://doi.org/10.1071/ah13033 .

Hoddinott P, Britten J, Prescott GJ, Tappin D, Ludbrook A, Godden DJ. Effectiveness of policy to provide breastfeeding groups ( BIG) for pregnant and breastfeeding mothers in primary care: cluster randomised controlled trial. Br Med J. 2009;338:a3026, 10. https://doi.org/10.1136/bmj.a3026 .

Ivankova NV, Creswell JW, Stick SL. Using mixed-methods sequential explanatory design: from theory to practice. Field Methods. 2006;18(1):3–20. https://doi.org/10.1177/1525822x05282260 .

Johnson B, Gray R. A history of philosophical and theoretical issues for mixed methods research. In: Tashakkori A, Teddlie C, editors. Sage handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks: Sage; 2010. p. 69–94.

Chapter   Google Scholar  

Johnson BR, Onwuegbuzie AJ. Mixed methods research: a research paradigm whose time has come. Educ Res. 2004;33(7):14–26. https://doi.org/10.3102/0013189x033007014 .

Keeney S, McKenna H, Fleming P, McIlfatrick S. Attitudes to cancer and cancer prevention: what do people aged 35–54 years think? Eur J Cancer Care. 2010;19(6):769–77. https://doi.org/10.1111/j.1365-2354.2009.01137.x .

McBride KA, Ballinger ML, Schlub TE, Young MA, Tattersall MHN, Kirk J, et al. Psychosocial morbidity in TP53 mutation carriers: is whole-body cancer screening beneficial? Familial Cancer. 2017;16(3):423–32. https://doi.org/10.1007/s10689-016-9964-7 .

Mertens DM. Transformative mixed methods: addressing inequities. Am Behav Sci. 2012;56(6):802–13. https://doi.org/10.1177/0002764211433797 .

Moffatt S, White M, Mackintosh J, Howel D. Using quantitative and qualitative data in health services research – what happens when mixed method findings conflict? BMC Health Serv Res. 2006;6:28. https://doi.org/10.1186/1472-6963-6-28 .

Morse JM. Approaches to qualitative-quantitative methodological triangulation. Nurs Res. 1991a;40(2):120–3.

Morse JM. Principles of mixed methods and multimethod research design. In: Tashakkori A, Teddilie C, editors. SAGE handbook of mixed methods in social and behavioral research. Thousand Oaks: Sage; 1991b.

Mutrie N, Doolin O, Fitzsimons CF, Grant PM, Granat M, Grealy M, et al. Increasing older adults’ walking through primary care: results of a pilot randomized controlled trial. Fam Pract. 2012;29(6):633–42. https://doi.org/10.1093/fampra/cms038 .

Newman I, Ridenour C, Newman C, De Marco G. A typology of research purposes and its relationship to mixed methods in social and behavioral research. Thousand Oaks: Sage; 2003. p. 167–88.

O’Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. J Health Serv Res Policy. 2008;13(2):92–8. https://doi.org/10.1258/jhsrp.2007.007074 .

Onwuegbuzie AJ, Collins KMT. A typology of mixed methods sampling designs in social science research. Qual Rep. 2007;12(2):281–316.

Plano Clark VL, Badiee M. Research questions in mixed methods research. In: Tashakkori A, Teddlie C, editors. SAGE handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks: Sage; 2010. p. 275–304.

Prades, J., Algara, M., Espinas, J. A., Farrus, B., Arenas, M., Reyes, V., . . . Borras, J. M. (2017). Understanding variations in the use of hypofractionated radiotherapy and its specific indications for breast cancer: a mixed-methods study. Radiother Oncol, 123(1), 22–28. doi: https://doi.org/10.1016/j.radonc.2017.01.014 .

Tariq S, Woodman J. Using mixed methods in health research. JRSM Short Rep. 2013; 4 (6):1–8. https://doi.org/10.1177/2042533313479197 .

Tashakkori A, Creswell JW. Editorial: the new era of mixed methods. J Mixed Methods Res. 2007;1(1):3–7. https://doi.org/10.1177/2345678906293042 .

Tashakkori A, Teddlie C. The past and future of mixed methods research: from data triangulation to mixed model designs. In: Tashakkori A, Teddlie C, editors. Handbook of mixed methods in social & behavioral research. Thousand Oaks: Sage; 2003. p. 671–701.

Teddlie C, Tashakkori A. Foundations of mixed methods research: integrating quantitative and qualitative approaches in the social and behavioral sciences. Thousand Oaks: Sage; 2009.

Teddlie C, Yu F. Mixed methods sampling a typology with examples. J Mixed Methods Res. 2007;1(1):77–100. https://doi.org/10.1177/2345678906292430 .

Wellard SJ, Rasmussen B, Savage S, Dunning T. Exploring staff diabetes medication knowledge and practices in regional residential care: triangulation study. J Clin Nurs. 2013;22(13–14):1933–40. https://doi.org/10.1111/jocn.12043 .

Zhang W. Mixed methods application in health intervention research: a multiple case study. Int J Mult Res Approaches. 2014;8(1):24–35. https://doi.org/10.5172/mra.2014.8.1.24 .

Download references

Author information

Authors and affiliations.

School of Medicine and Translational Health Research Institute, Western Sydney University, Sydney, NSW, Australia

Kate A. McBride

School of Science and Health and Translational Health Research Institute (THRI), Western Sydney University, Penrith, NSW, Australia

Freya MacMillan

School of Science and Health, Western Sydney University, Sydney, NSW, Australia

Emma S. George

NICM and Translational Health Research Institute (THRI), Western Sydney University, Penrith, NSW, Australia

Genevieve Z. Steiner

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Kate A. McBride .

Editor information

Editors and affiliations.

School of Science and Health, Western Sydney University, Penrith, NSW, Australia

Pranee Liamputtong

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this entry

Cite this entry.

McBride, K.A., MacMillan, F., George, E.S., Steiner, G.Z. (2019). The Use of Mixed Methods in Research. In: Liamputtong, P. (eds) Handbook of Research Methods in Health Social Sciences. Springer, Singapore. https://doi.org/10.1007/978-981-10-5251-4_97

Download citation

DOI : https://doi.org/10.1007/978-981-10-5251-4_97

Published : 13 January 2019

Publisher Name : Springer, Singapore

Print ISBN : 978-981-10-5250-7

Online ISBN : 978-981-10-5251-4

eBook Packages : Social Sciences Reference Module Humanities and Social Sciences Reference Module Business, Economics and Social Sciences

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• News & Highlights

• Publications and Documents
• Postgraduate Education
• Browse Our Courses
• C/T Research Academy
• K12 Investigator Training
• Translational Innovator
• SMART IRB Reliance Request
• Biostatistics Consulting
• Regulatory Support
• Pilot Funding
• Informatics Program
• Community Engagement
• Diversity Inclusion
• Research Enrollment and Diversity
• Harvard Catalyst Profiles

Community Engagement Program

Innovation and improvement in public health via community engagement and research

• Getting Started
• Resources for Equity in Research
• Community-Engaged Student Practice Placement
• Maternal Health Equity
• Youth Mental Health
• Leadership and Membership
• Past Members
• Study Review Rubric
• Community Ambassador Initiative
• Implementation Science Working Group
• Past Webinars & Podcasts
• Policy Atlas
• Community Advisory Board

For more information:

Mixed methods research.

According to the National Institutes of Health , mixed methods strategically integrates or combines rigorous quantitative and qualitative research methods to draw on the strengths of each. Mixed method approaches allow researchers to use a diversity of methods, combining inductive and deductive thinking, and offsetting limitations of exclusively quantitative and qualitative research through a complementary approach that maximizes strengths of each data type and facilitates a more comprehensive understanding of health issues and potential resolutions.¹ Mixed methods may be employed to produce a robust description and interpretation of the data, make quantitative results more understandable, or understand broader applicability of small-sample qualitative findings.

Integration

This refers to the ways in which qualitative and quantitative research activities are brought together to achieve greater insight. Mixed methods is not simply having quantitative and qualitative data available or analyzing and presenting data findings separately. The integration process can occur during data collection, analysis, or in the presentation of results.

¹ NIH Office of Behavioral and Social Sciences Research: Best Practices for Mixed Methods Research in the Health Sciences

Basic Mixed Methods Research Designs 

View image description .

Five Key Questions for Getting Started

• What do you want to know?
• What will be the detailed quantitative, qualitative, and mixed methods research questions that you hope to address?
• What quantitative and qualitative data will you collect and analyze?
• Which rigorous methods will you use to collect data and/or engage stakeholders?
• How will you integrate the data in a way that allows you to address the first question?

Rationale for Using Mixed Methods

• Obtain different, multiple perspectives: validation
• Build comprehensive understanding
• Explain statistical results in more depth
• Have better contextualized measures
• Track the process of program or intervention
• Study patient-centered outcomes and stakeholder engagement

Sample Mixed Methods Research Study

The EQUALITY study used an exploratory sequential design to identify the optimal patient-centered approach to collect sexual orientation data in the emergency department.

Qualitative Data Collection and Analysis : Semi-structured interviews with patients of different sexual orientation, age, race/ethnicity, as well as healthcare professionals of different roles, age, and race/ethnicity.

Builds Into : Themes identified in the interviews were used to develop questions for the national survey.

Quantitative Data Collection and Analysis : Representative national survey of patients and healthcare professionals on the topic of reporting gender identity and sexual orientation in healthcare.

Other Resources:

  Introduction to Mixed Methods Research : Harvard Catalyst’s eight-week online course offers an opportunity for investigators who want to understand and apply a mixed methods approach to their research.

Best Practices for Mixed Methods Research in the Health Sciences [PDF] : This guide provides a detailed overview of mixed methods designs, best practices, and application to various types of grants and projects.

Mixed Methods Research Training Program for the Health Sciences (MMRTP ): Selected scholars for this summer training program, hosted by Johns Hopkins’ Bloomberg School of Public Health, have access to webinars, resources, a retreat to discuss their research project with expert faculty, and are matched with mixed methods consultants for ongoing support.

Michigan Mixed Methods : University of Michigan Mixed Methods program offers a variety of resources, including short web videos and recommended reading.

To use a mixed methods approach, you may want to first brush up on your qualitative skills. Below are a few helpful resources specific to qualitative research:

• Qualitative Research Guidelines Project : A comprehensive guide for designing, writing, reviewing and reporting qualitative research.
• Fundamentals of Qualitative Research Methods – What is Qualitative Research : A six-module web video series covering essential topics in qualitative research, including what is qualitative research and how to use the most common methods, in-depth interviews, and focus groups.

View PDF of the above information.

Join thousands of product people at Insight Out Conf on April 11. Register free.

Insights hub solutions

Analyze data

Uncover deep customer insights with fast, powerful features, store insights, curate and manage insights in one searchable platform, scale research, unlock the potential of customer insights at enterprise scale.

Featured reads

Inspiration

Three things to look forward to at Insight Out

Tips and tricks

Make magic with your customer data in Dovetail

Four ways Dovetail helps Product Managers master continuous product discovery

Events and videos

© Dovetail Research Pty. Ltd.

• What is mixed methods research?

Last updated

20 February 2023

Reviewed by

Miroslav Damyanov

By blending both quantitative and qualitative data, mixed methods research allows for a more thorough exploration of a research question. It can answer complex research queries that cannot be solved with either qualitative or quantitative research .

Analyze your mixed methods research

Dovetail streamlines analysis to help you uncover and share actionable insights

Mixed methods research combines the elements of two types of research: quantitative and qualitative.

Quantitative data is collected through the use of surveys and experiments, for example, containing numerical measures such as ages, scores, and percentages. 

Qualitative data involves non-numerical measures like beliefs, motivations, attitudes, and experiences, often derived through interviews and focus group research to gain a deeper understanding of a research question or phenomenon.

Mixed methods research is often used in the behavioral, health, and social sciences, as it allows for the collection of numerical and non-numerical data.

• When to use mixed methods research

Mixed methods research is a great choice when quantitative or qualitative data alone will not sufficiently answer a research question. By collecting and analyzing both quantitative and qualitative data in the same study, you can draw more meaningful conclusions. 

There are several reasons why mixed methods research can be beneficial, including generalizability, contextualization, and credibility. 

For example, let's say you are conducting a survey about consumer preferences for a certain product. You could collect only quantitative data, such as how many people prefer each product and their demographics. Or you could supplement your quantitative data with qualitative data, such as interviews and focus groups , to get a better sense of why people prefer one product over another.

It is important to note that mixed methods research does not only mean collecting both types of data. Rather, it also requires carefully considering the relationship between the two and method flexibility.

You may find differing or even conflicting results by combining quantitative and qualitative data . It is up to the researcher to then carefully analyze the results and consider them in the context of the research question to draw meaningful conclusions.

When designing a mixed methods study, it is important to consider your research approach, research questions, and available data. Think about how you can use different techniques to integrate the data to provide an answer to your research question.

• Mixed methods research design

A mixed methods research design  is   an approach to collecting and analyzing both qualitative and quantitative data in a single study.

Mixed methods designs allow for method flexibility and can provide differing and even conflicting results. Examples of mixed methods research designs include convergent parallel, explanatory sequential, and exploratory sequential.

By integrating data from both quantitative and qualitative sources, researchers can gain valuable insights into their research topic . For example, a study looking into the impact of technology on learning could use surveys to measure quantitative data on students' use of technology in the classroom. At the same time, interviews or focus groups can provide qualitative data on students' experiences and opinions.

• Types of mixed method research designs

Researchers often struggle to put mixed methods research into practice, as it is challenging and can lead to research bias. Although mixed methods research can reveal differences or conflicting results between studies, it can also offer method flexibility.

Designing a mixed methods study can be broken down into four types: convergent parallel, embedded, explanatory sequential, and exploratory sequential.

Convergent parallel

The convergent parallel design is when data collection and analysis of both quantitative and qualitative data occur simultaneously and are analyzed separately. This design aims to create mutually exclusive sets of data that inform each other. 

For example, you might interview people who live in a certain neighborhood while also conducting a survey of the same people to determine their satisfaction with the area.

Embedded design

The embedded design is when the quantitative and qualitative data are collected simultaneously, but the qualitative data is embedded within the quantitative data. This design is best used when you want to focus on the quantitative data but still need to understand how the qualitative data further explains it.

For instance, you may survey students about their opinions of an online learning platform and conduct individual interviews to gain further insight into their responses.

Explanatory sequential design

In an explanatory sequential design, quantitative data is collected first, followed by qualitative data. This design is used when you want to further explain a set of quantitative data with additional qualitative information.

An example of this would be if you surveyed employees at a company about their satisfaction with their job and then conducted interviews to gain more information about why they responded the way they did.

Exploratory sequential design

The exploratory sequential design collects qualitative data first, followed by quantitative data. This type of mixed methods research is used when the goal is to explore a topic before collecting any quantitative data.

An example of this could be studying how parents interact with their children by conducting interviews and then using a survey to further explore and measure these interactions.

Integrating data in mixed methods studies can be challenging, but it can be done successfully with careful planning.

No matter which type of design you choose, understanding and applying these principles can help you draw meaningful conclusions from your research.

• Strengths of mixed methods research

Mixed methods research designs combine the strengths of qualitative and quantitative data, deepening and enriching qualitative results with quantitative data and validating quantitative findings with qualitative data. This method offers more flexibility in designing research, combining theory generation and hypothesis testing, and being less tied to disciplines and established research paradigms.

Take the example of a study examining the impact of exercise on mental health. Mixed methods research would allow for a comprehensive look at the issue from different angles. 

Researchers could begin by collecting quantitative data through surveys to get an overall view of the participants' levels of physical activity and mental health. Qualitative interviews would follow this to explore the underlying dynamics of participants' experiences of exercise, physical activity, and mental health in greater detail.

Through a mixed methods approach, researchers could more easily compare and contrast their results to better understand the phenomenon as a whole.  

Additionally, mixed methods research is useful when there are conflicting or differing results in different studies. By combining both quantitative and qualitative data, mixed methods research can offer insights into why those differences exist.

For example, if a quantitative survey yields one result while a qualitative interview yields another, mixed methods research can help identify what factors influence these differences by integrating data from both sources.

Overall, mixed methods research designs offer a range of advantages for studying complex phenomena. They can provide insight into different elements of a phenomenon in ways that are not possible with either qualitative or quantitative data alone. Additionally, they allow researchers to integrate data from multiple sources to gain a deeper understanding of the phenomenon in question.  

• Challenges of mixed methods research

Mixed methods research is labor-intensive and often requires interdisciplinary teams of researchers to collaborate. It also has the potential to cost more than conducting a stand alone qualitative or quantitative study . 

Interpreting the results of mixed methods research can be tricky, as it can involve conflicting or differing results. Researchers must find ways to systematically compare the results from different sources and methods to avoid bias.

For example, imagine a situation where a team of researchers has employed an explanatory sequential design for their mixed methods study. After collecting data from both the quantitative and qualitative stages, the team finds that the two sets of data provide differing results. This could be challenging for the team, as they must now decide how to effectively integrate the two types of data in order to reach meaningful conclusions. The team would need to identify method flexibility and be strategic when integrating data in order to draw meaningful conclusions from the conflicting results.

• Advanced frameworks in mixed methods research

Mixed methods research offers powerful tools for investigating complex processes and systems, such as in health and healthcare.

Besides the three basic mixed method designs—exploratory sequential, explanatory sequential, and convergent parallel—you can use one of the four advanced frameworks to extend mixed methods research designs. These include multistage, intervention, case study , and participatory. 

This framework mixes qualitative and quantitative data collection methods in stages to gather a more nuanced view of the research question. An example of this is a study that first has an online survey to collect initial data and is followed by in-depth interviews to gain further insights.

Intervention

This design involves collecting quantitative data and then taking action, usually in the form of an intervention or intervention program. An example of this could be a research team who collects data from a group of participants, evaluates it, and then implements an intervention program based on their findings .

This utilizes both qualitative and quantitative research methods to analyze a single case. The researcher will examine the specific case in detail to understand the factors influencing it. An example of this could be a study of a specific business organization to understand the organizational dynamics and culture within the organization.

Participatory

This type of research focuses on the involvement of participants in the research process. It involves the active participation of participants in formulating and developing research questions, data collection, and analysis.

An example of this could be a study that involves forming focus groups with participants who actively develop the research questions and then provide feedback during the data collection and analysis stages.

The flexibility of mixed methods research designs means that researchers can choose any combination of the four frameworks outlined above and other methodologies , such as convergent parallel, explanatory sequential, and exploratory sequential, to suit their particular needs.

Through this method's flexibility, researchers can gain multiple perspectives and uncover differing or even conflicting results when integrating data.

When it comes to integration at the methods level, there are four approaches.

Connecting involves collecting both qualitative and quantitative data during different phases of the research.

Building involves the collection of both quantitative and qualitative data within a single phase.

Merging involves the concurrent collection of both qualitative and quantitative data.

Embedding involves including qualitative data within a quantitative study or vice versa.

• Techniques for integrating data in mixed method studies

Integrating data is an important step in mixed methods research designs. It allows researchers to gain further understanding from their research and gives credibility to the integration process. There are three main techniques for integrating data in mixed methods studies: triangulation protocol, following a thread, and the mixed methods matrix.

Triangulation protocol

This integration method combines different methods with differing or conflicting results to generate one unified answer.

For example, if a researcher wanted to know what type of music teenagers enjoy listening to, they might employ a survey of 1,000 teenagers as well as five focus group interviews to investigate this. The results might differ; the survey may find that rap is the most popular genre, whereas the focus groups may suggest rock music is more widely listened to. 

The researcher can then use the triangulation protocol to come up with a unified answer—such as that both rap and rock music are popular genres for teenage listeners. 

Following a thread

This is another method of integration where the researcher follows the same theme or idea from one method of data collection to the next. 

A research design that follows a thread starts by collecting quantitative data on a specific issue, followed by collecting qualitative data to explain the results. This allows whoever is conducting the research to detect any conflicting information and further look into the conflicting information to understand what is really going on.

For example, a researcher who used this research method might collect quantitative data about how satisfied employees are with their jobs at a certain company, followed by qualitative interviews to investigate why job satisfaction levels are low. They could then use the results to explore any conflicting or differing results, allowing them to gain a deeper understanding of job satisfaction at the company. 

By following a thread, the researcher can explore various research topics related to the original issue and gain a more comprehensive view of the issue.

Mixed methods matrix

This technique is a visual representation of the different types of mixed methods research designs and the order in which they should be implemented. It enables researchers to quickly assess their research design and adjust it as needed. 

The matrix consists of four boxes with four different types of mixed methods research designs: convergent parallel, explanatory sequential, exploratory sequential, and method flexibility. 

For example, imagine a researcher who wanted to understand why people don't exercise regularly. To answer this question, they could use a convergent parallel design, collecting both quantitative (e.g., survey responses) and qualitative (e.g., interviews) data simultaneously.

If the researcher found conflicting results, they could switch to an explanatory sequential design and collect quantitative data first, then follow up with qualitative data if needed. This way, the researcher can make adjustments based on their findings and integrate their data more effectively.

Mixed methods research is a powerful tool for understanding complex research topics. Using qualitative and quantitative data in one study allows researchers to understand their subject more deeply. 

Mixed methods research designs such as convergent parallel, explanatory sequential, and exploratory sequential provide method flexibility, enabling researchers to collect both types of data while avoiding the limitations of either approach alone.

However, it's important to remember that mixed methods research can produce differing or even conflicting results, so it's important to be aware of the potential pitfalls and take steps to ensure that data is being correctly integrated. If used effectively, mixed methods research can offer valuable insight into topics that would otherwise remain largely unexplored.

What is an example of mixed methods research?

An example of mixed methods research is a study that combines quantitative and qualitative data. This type of research uses surveys, interviews, and observations to collect data from multiple sources.

Which sampling method is best for mixed methods?

It depends on the research objectives, but a few methods are often used in mixed methods research designs. These include snowball sampling, convenience sampling, and purposive sampling. Each method has its own advantages and disadvantages.

What is the difference between mixed methods and multiple methods?

Mixed methods research combines quantitative and qualitative data in a single study. Multiple methods involve collecting data from different sources, such as surveys and interviews, but not necessarily combining them into one analysis. Mixed methods offer greater flexibility but can lead to differing or conflicting results when integrating data.

Get started today

Go from raw data to valuable insights with a flexible research platform

Editor’s picks

Last updated: 21 December 2023

Last updated: 16 December 2023

Last updated: 17 February 2024

Last updated: 19 November 2023

Last updated: 5 March 2024

Last updated: 15 February 2024

Last updated: 11 March 2024

Last updated: 12 December 2023

Last updated: 6 March 2024

Last updated: 10 April 2023

Last updated: 20 December 2023

Latest articles

Related topics, log in or sign up.

Get started for free

• Research article
• Open access
• Published: 24 September 2018

A mixed methods case study exploring the impact of membership of a multi-activity, multicentre community group on social wellbeing of older adults

• Gabrielle Lindsay-Smith   ORCID: orcid.org/0000-0003-3864-1412 1 ,
• Grant O’Sullivan 1 ,
• Rochelle Eime 1 , 2 ,
• Jack Harvey 1 , 2 &
• Jannique G. Z. van Uffelen 1 , 3  

BMC Geriatrics volume  18 , Article number:  226 ( 2018 ) Cite this article

69k Accesses

30 Citations

12 Altmetric

Metrics details

Social wellbeing factors such as loneliness and social support have a major impact on the health of older adults and can contribute to physical and mental wellbeing. However, with increasing age, social contacts and social support typically decrease and levels of loneliness increase. Group social engagement appears to have additional benefits for the health of older adults compared to socialising individually with friends and family, but further research is required to confirm whether group activities can be beneficial for the social wellbeing of older adults.

This one-year longitudinal mixed methods study investigated the effect of joining a community group, offering a range of social and physical activities, on social wellbeing of adults with a mean age of 70. The study combined a quantitative survey assessing loneliness and social support ( n  = 28; three time-points, analysed using linear mixed models) and a qualitative focus group study ( n  = 11, analysed using thematic analysis) of members from Life Activities Clubs Victoria, Australia.

There was a significant reduction in loneliness ( p  = 0.023) and a trend toward an increase in social support ( p  = 0.056) in the first year after joining. The focus group confirmed these observations and suggested that social support may take longer than 1 year to develop. Focus groups also identified that group membership provided important opportunities for developing new and diverse social connections through shared interest and experience. These connections were key in improving the social wellbeing of members, especially in their sense of feeling supported or connected and less lonely. Participants agreed that increasing connections was especially beneficial following significant life events such as retirement, moving to a new house or partners becoming unwell.

Conclusions

Becoming a member of a community group offering social and physical activities may improve social wellbeing in older adults, especially following significant life events such as retirement or moving-house, where social network changes. These results indicate that ageing policy and strategies would benefit from encouraging long-term participation in social groups to assist in adapting to changes that occur in later life and optimise healthy ageing.

Peer Review reports

Ageing population and the need to age well

Between 2015 and 2050 it is predicted that globally the number of adults over the age of 60 will more than double [ 1 ]. Increasing age is associated with a greater risk of chronic illnesses such as cardio vascular disease and cancer [ 2 ] and reduced functional capacity [ 3 , 4 ]. Consequently, an ageing population will continue to place considerable pressure on the health care systems.

However, it is also important to consider the individuals themselves and self-perceived good health is very important for the individual wellbeing and life-satisfaction of older adults [ 5 ]. The terms “successful ageing” [ 6 ] and “healthy ageing” [ 5 ] have been used to define a broader concept of ageing well, which not only includes factors relating to medically defined health but also wellbeing. Unfortunately, there is no agreed definition for what exactly constitutes healthy or successful ageing, with studies using a range of definitions. A review of 28 quantitative studies found that successful ageing was defined differently in each, with the majority only considering measures of disability or physical functioning. Social and wellbeing factors were included in only a few of the studies [ 7 ].

In contrast, qualitative studies of older adults’ opinions on successful ageing have found that while good physical and mental health and maintaining physical activity levels are agreed to assist successful ageing, being independent or doing something of value, acceptance of ageing, life satisfaction, social connectedness or keeping socially active were of greater importance [ 8 , 9 , 10 ].

In light of these findings, the definition that is most inclusive is “healthy ageing” defined by the World Health Organisation as “the process of developing and maintaining the functional ability (defined as a combination of intrinsic capacity and physical and social environmental characteristics), that enables well-being in older age” (p28) [ 5 ].This definition, and those provided in the research of older adults’ perceptions of successful ageing, highlight social engagement and social support as important factors contributing to successful ageing, in addition to being important social determinants of health [ 11 , 12 ].

Social determinants of health, including loneliness and social support, are important predictors of physical, cognitive and mental health and wellbeing in adults [ 12 ] and older adults [ 13 , 14 , 15 ]. Loneliness is defined as a perception of an inadequacy in the quality or quantity of one’s social relationships [ 16 ]. Social support, has various definitions but generally it relates to social relationships that are reciprocal, accessible and reliable and provide any or a combination of supportive resources (e.g. emotional, information, practical) and can be measured as perceived or received support [ 17 ]. These types of social determinants differ from those related to inequality (health gap social determinants) and are sometimes referred to as ‘social cure’ social determinants [ 11 ]. They will be referred to as ‘social wellbeing’ outcome measures in this study.

Unfortunately, with advancing age, there is often diminishing social support, leading to social isolation and loneliness [ 18 , 19 ]. Large nationally representative studies of adults and older adults reported that social activity predicted maintenance or improvement of life satisfaction as well as physical activity levels [ 20 ], however older adults spent less time in social activity than middle age adults.

Social wellbeing and health

A number of longitudinal studies have found that social isolation for older adults is a significant predictor of mortality and institutionalisation [ 21 , 22 , 23 ]. A meta-analysis by Holt-Lunstadt [ 12 ] reported that social determinants of health, including social integration and social support (including loneliness and lack of perceived social support) to be equal to, or a greater risk to mortality as common behavioural risk factors such as smoking, physical inactivity and obesity. Loneliness is independently associated with poor physical and mental health in the general population, and especially in older adults [ 13 , 14 , 15 ]. Adequate perceived social support has also been consistently associated with improved mental and physical health in both general and older adults [ 20 , 24 , 25 , 26 , 27 , 28 , 29 ]. The mechanism suggested for this association is that social support buffers the negative impacts of stressful situations and life events [ 30 ]. The above research demonstrates the benefit of social engagement for older adults; in turn this highlights the importance of strategies that reduce loneliness and improve social support and social connectedness for older adults.

Socialising in groups seems to be especially important for the health and wellbeing of older adults who may be adjusting to significant life events [ 26 , 31 , 32 , 33 ]. This is sometimes referred to as social engagement or social companionship [ 26 , 30 , 31 ]. It seems that the mechanism enabling such health benefits with group participation is through strengthening of social identification, which in turn increases social support [ 31 , 34 , 35 ]. Furthermore, involvement in community groups can be a sustainable strategy to reduce loneliness and increase social support in older adults, as they are generally low cost and run by volunteers [ 36 , 37 , 38 , 39 ].

Despite the demonstrated importance of social factors for successful ageing and the established risk associated with reduced social engagement as people age, few in-depth studies have longitudinally investigated the impact of community groups on social wellbeing. For example, a non-significant increase in social support and reduction in depression was found in a year-long randomised controlled trial conducted in senior centres in Norway with lonely older adults in poor physical and mental health [ 37 ]. Some qualitative studies have reported that community groups and senior centres can contribute to fun and socialisation for older adults, however social wellbeing was not the primary focus of the studies [ 38 , 40 , 41 ]. Given that social wellbeing is a broad and important area for the health and quality of life in older adults, an in-depth study is warranted to understand how it can be maximised in older adults. This mixed methods case study of an existing community aims to: i) examine whether loneliness and social support of new members of Life Activities Clubs (LACs) changes in the year after joining and ii) conduct an in-depth exploration of how social wellbeing changes in new and longer-term members of LACs.

A mixed methods study was chosen as the design for this research to enable an in-depth exploration of how loneliness and social support may change as a result of joining a community group. A case study was conducted using a concurrent mixed-methods design, with a qualitative component giving context to the quantitative results. Where the survey focused on the impact of group membership on social support and loneliness, the focus groups were an open discussion of the benefits in the lived context of LAC membership. The synthesis of the two sections of the study was undertaken at the time of interpretation of the results [ 42 ].

The two parts of our study were as follows:

a longitudinal survey (three time points over 1 year: baseline, 6 and 12 months). This part of the study formed the quantitative results;

a focus group study of members of the same organisation (qualitative).

Ethics approval to conduct this study was obtained from the Victoria University Human Research Ethics Committee (HRE14–071 [survey] and HRE15–291 [focus groups]) All participants provided informed consent to partake in the study prior to undertaking the first survey or focus group.

Setting and participants

Life activities clubs victoria.

Life Activities Clubs Victoria (LACVI) is a large not-for-profit group with 23 independently run Life Activities Clubs (LACs) based in both rural and metropolitan Victoria. It has approximately 4000 members. The organisation was established to assist in providing physical, social and recreational activities as well as education and motivational support to older adults managing significant change in their lives, especially retirement.

Eighteen out of 23 LAC clubs agreed to take part in the survey study. During the sampling period from May 2014 to December 2016, new members from the participating clubs were given information about the study and invited to take part. Invitations took place in the form of flyers distributed with new membership material.

Inclusion/ exclusion criteria

Community-dwelling older adults who self-reported that they could walk at least 100 m and who were new members to LACVI and able to complete a survey in English were eligible to participate. New members were defined as people who had never been members of LACVI or who had not been members in the last 2 years.

To ensure that the cohort of participants were of a similar functional level, people with significant health problems limiting them from being able to walk 100 m were excluded from participating in the study.

Once informed consent was received, the participants were invited to complete a self-report survey in either paper or online format (depending on preference). This first survey comprised the baseline data and the same survey was completed 6 months and 12 months after this initial time point. Participants were sent reminders if they had not completed each survey more than 2 weeks after each was delivered and then again 1 week later.

Focus groups

Two focus groups (FGs) were conducted with new and longer-term members of LACs. The first FG ( n  = 6) consisted of members who undertook physical activity in their LAC (e.g. walking groups, tennis, cycling). The second FG ( n  = 5) consisted of members who took part in activities with a non-physical activity (PA) focus (e.g. book groups, social groups, craft or cultural groups). LACs offer both social and physical activities and it was important to the study to capture both types of groups, but they were kept separate to assist participants in feeling a sense of commonality with other members and improving group dynamic and participation in the discussions [ 43 ]. Of the people who participated in the longitudinal survey study, seven also participated in the FGs.

The FG interviews were facilitated by one researcher (GLS) and notes around non-verbal communication, moments of divergence and convergence amongst group members, and other notable items were taken by a second researcher (GOS). Both researchers wrote additional notes after the focus groups and these were used in the analysis of themes. Focus groups were recorded and later transcribed verbatim by a professional transcriptionist, including identification of each participant speaking. One researcher (GLS) reviewed each transcription to check for any errors and made any required modifications before importing the transcriptions into NVivo for analysis. The transcriber identified each focus group participant so themes for individuals or other age or gender specific trends could be identified.

Dependent variables

• Social support

Social support was assessed using the Duke–UNC Functional Social support questionnaire [ 44 ]. This scale specifically measures participant perceived functional social support in two areas; i) confidant support (5 questions; e.g. chances to talk to others) and ii) affective support (3 questions; e.g. people who care about them). Participants rated each component of support on a 5-item likert scale between ‘much less than I would like’ (1 point) to ‘as much as I would like’ (5 points). The total score used for analysis was the mean of the eight scores (low social support = 1, maximum social support = 5). Construct validity, concurrent validity and discriminant validity are acceptable for confidant and affective support items in the survey in the general population [ 44 ].

Loneliness was measured using the de Jong Gierveld and UCLA-3 item loneliness scales developed for use in many populations including older adults [ 45 ]. The 11-item de Jong Gierveld loneliness scale (DJG loneliness) [ 46 ] is a multi-dimensional measure of loneliness and contains five positively worded and six negatively worded items. The items fall into four subscales; feelings of severe loneliness, feelings connected with specific problem situations, missing companionship, feelings of belongingness. The total score is the sum of the items scores (i.e. 11–55): 11 is low loneliness and 55 is severe loneliness. Self-administered versions of this scale have good internal consistency (> = 0.8) and inter-item homogeneity and person scalability that is as good or better than when conducted as face-to face interviews. The validity and reliability for the scale is adequate [ 47 ]. The UCLA 3-item loneliness scale consists of three questions about how often participants feel they lack companionship, feel left out and feel isolated. The responses are given on a three-point scale ranging from hardly ever (1) to often (3). The final score is the sum of these three items with the range being from lowest loneliness (3) to highest loneliness (9). Reliability of the scale is good, (alpha = 0.72) as are discriminant validity and internal consistency [ 48 ]. The scale is commonly used to measure loneliness with older adults ([ 49 ] – review), [ 50 , 51 ].

Sociodemographic variables

The following sociodemographic characteristics were collected in both the survey and the focus groups: age, sex, highest level of education, main life occupation [ 52 ], current employment, ability to manage on income available, present marital status, country of birth, area of residence [ 53 ]. They are categorised as indicated in Table  2 .

Health variables

The following health variables were collected: Self-rated general health (from SF-12) [ 54 ] and Functional health (ability to walk 100 m- formed part of the inclusion criteria) [ 55 ]. See Table 2 for details about the categories of these variables.

The effects of becoming a member on quantitative outcome variables (i.e. Social support, DJG loneliness and UCLA loneliness) were analysed using linear mixed models (LMM). LMM enabled testing for the presence of intra-subject random effects, or equivalently, correlation of subjects’ measures over time (baseline, 6-months and 12 months). Three correlation structures were examined: independence (no correlation), compound symmetry (constant correlation of each subjects’ measures over the three time points) and autoregressive (correlation diminishing with increase in spacing in time). The best fitting correlation structure was compound symmetry; this is equivalent to a random intercept component for each subject. The LMM incorporated longitudinal trends over time, with adjustment for age as a potential confounder. Statistical analyses were conducted using SPSS for windows (v24).

UCLA loneliness and social support residuals were not normally distributed and these scales were Log10 transformed for statistical analysis.

Analyses were all adjusted for age, group attendance (calculated as average attendance at 6 and 12 months) and employment status at baseline (Full-time, Part-time, not working).

Focus group transcripts were analysed using thematic analysis [ 56 , 57 ], a flexible qualitative methodology that can be used with a variety of epistemologies, approaches and analysis methods [ 56 ]. The transcribed data were analysed using a combination of theoretical and inductive thematic analysis [ 56 ]. It was theorised that membership in a LAC would assist with social factors relating to healthy ageing [ 5 ], possibly through a social identity pathway [ 58 ], although we wanted to explore this. Semantic themes were drawn from these codes in order to conduct a pragmatic evaluation of the LACVI programs [ 56 ]. Analytic rigour in the qualitative analysis was ensured through source and analyst triangulation. Transcriptions were compared to notes taken during the focus groups by the researchers (GOS and GLS). In addition, Initial coding and themes (by GLS) were checked by a second researcher (GOS) and any disagreements regarding coding and themes were discussed prior to finalisation of codes and themes [ 57 ].

Sociodemographic and health characteristics of the 28 participants who completed the survey study are reported in Table  1 . The mean age of the participants was 66.9 and 75% were female. These demographics are representative of the entire LACVI membership. Education levels varied, with 21% being university educated, and the remainder completing high school or technical certificates. Two thirds of participants were not married. Some sociodemographic characteristics changed slightly at 6 and 12 months, mainly employment (18% in paid employment at baseline and 11% at 12-months) and ability to manage on income (36% reporting trouble managing on their income at baseline and 46% at 12 months). Almost 90% of the participants described themselves as being in good-excellent health.

Types of activities

There were a variety of types of activities that participants took part in: physical activities such as walking groups ( n  = 7), table tennis ( n  = 5), dancing class ( n  = 2), exercise class ( n  = 1), bowls ( n  = 2), golf ( n  = 3), cycling groups ( n  = 1) and non-physical leisure activities such as art and literature groups ( n  = 5), craft groups ( n  = 5), entertainment groups ( n  = 12), food/dine out groups ( n  = 18) and other sedentary leisure activities (e.g. mah jong, cards),( n  = 4). A number of people took part in more than one activity.

Frequency of attendance at LACVI and changes in social wellbeing

At six and 12 months, participants indicated how many times in the last month they attended different types of activities at their LAC. Most participants maintained the same frequency of participation over both time points. Only four people participated more frequently at 12 than at 6 months and nine reduced participation levels. The latter group included predominantly those who reduced from more than two times per week at 6 months to 2×/week at 6 months to one to two times per week ( n  = 5) or less than one time per week ( n  = 2) at 12 months. Average weekly club attendance at six and 12 months was included as a covariate in the statistical model.

Outcome measures

Overall, participants reported moderate social support and loneliness levels at baseline (See Table 2 ). Loneliness, as measured by both scales, reduced significantly over time. There was a significant effect of time on the DJG loneliness scores (F (2, 52) = 3.83, p  = 0.028), with Post-Hoc analysis indicating a reduction in DJG loneliness between baseline and 12 months ( p  = 0.008). UCLA loneliness scores (transformed variable) also changed significantly over time (F (2, 52) = 4.08, p  = 0.023). Post hoc tests indicated a reduction in UCLA loneliness between baseline and 6 months ( p  = 0.007). There was a small non-significant increase in social support (F (2, 53) =2.88, p  = 0.065) during the first year of membership (see Table 2 and Figs. 1 and 2 ).

DJG loneliness for all participants over first year of membership at LAC club ( n  = 28).

*Represents significant difference compared to baseline ( p  < 0.01)

UCLA loneliness score for all participants over first year of membership at LAC club ( n  = 28).

*Indicates log values of the variable at 6-months were significantly different from baseline ( p  < 0.01)

In total, 11 participants attended the two focus groups, six people who participated in PA clubs (four women) and five who participated in social clubs (all women). All focus group participants were either retired ( n  = 9) or semi-retired ( n  = 2). The mean age of participants was 67 years (see Table 2 for further details). Most of the participants (82%) had been members of a LAC for less than 2 years and two females in the social group had been members of LAC clubs for 5 and 10 years respectively.

Analysis of the focus group transcripts identified two themes relating to social benefits of group participation; i) Social resources and ii) Social wellbeing (see Fig. 3 ). Group discussion suggested that membership of a LAC provides access to more social resources through greater and diverse social contact and opportunity. It is through this improvement in social resources that social wellbeing may improve.

Themes arising from focus group discussion around the benefits of LAC membership

Social resources

The social resources theme referred to an increase in the availability and variety of social connections that resulted from becoming a member of a LAC. The social nature of the groups enabled an expansion and diversification of members’ social network and improved their sense of social connectedness. There was widespread agreement in both the focus groups that significant life events, especially retirement, illness or death of spouse and moving house changes one’s social resources. Membership of the LAC had benefits especially at these times and these events were often motivators to join such a club. Most participants found that their social resources declined after retirement and even felt that they were grieving for the loss of their work.

“ I just saw work as a collection of, um, colleagues as opposed to friends. I had a few good friends there. Most were simply colleagues or acquaintances …. [interviewer- Mmm.] ..Okay, you’d talk to them every day. You’d chatter in the kitchen, oh, pass banter back and forth when things are busy or quiet, but... Um, in terms of a friendship with those people, like going to their home, getting to know them, doing other things with them, very few. But what I did miss was the interaction with other people. It had simply gone….. But, yeah, look, that, the, yeah, that intervening period was, oh, a couple of months. That was a bit tough…. But in that time the people in LAC and the people in U3A…. And the other dance group just drew me into more things. Got to know more people. So once again, yeah, reasonable group of acquaintances.” (Male, PAFG)

Group members indicated general agreement with these two responses, however one female found she had a greater social life following retirement due to the busy nature of her job.

Within the social resources theme, three subthemes were identified, i) Opportunity for social connectedness, ii) Opportunity for friendships, and iii) Opportunity for social responsibility/leadership . Interestingly, these subthemes were additional to the information gathered in the survey. This emphasises the power of the inductive nature of the qualitative exploration employed in the focus groups to broaden the knowledge in this area.

The most discussed and expanded subtheme in both focus groups was Opportunity for social connectedness , which arose through developing new connections, diversifying social connections, sharing interests and experiences with others and peer learning. Participants in both focus groups stated that being a member of LAC facilitated their socialising and connecting with others to share ideas, skills and to do activities with, which was especially important through times of significant life events. Furthermore, participants in each of the focus groups valued developing diverse connections:

“ Yeah, I think, as I said, I finished up work and I, and I had more time for wa-, walking. So I think a, in meeting, in going to this group which, I saw this group of women but then someone introduced me to them. They were just meeting, just meeting a new different set of people, you know? As I said, my work people and these were just a whole different group of women, mainly women. There’s not many men. [Interviewer: Yes.]….. Although our leader is a man, which is ironic and is about, this man out in front and there’s about 20 women behind him, but, um, so yeah, and people from different walks of life and different nationalities there which I never knew in my work life, so yeah. That’s been great. So from that goes on other things, you know, you might, uh, other activities and, yeah, people for coffee and go to the pictures or something, yeah. That’s great.” (Female, PAFG)

Simply making new connections was the most widely discussed aspect related to the opportunity for social connectedness subtheme, with all participants agreeing that this was an important benefit of participation in LAC groups.

“Well, my experience is very similar to everybody else’s…….: I, I went from having no social life to a social life once I joined a group.” (Female, PAFG)

There was agreement in both focus groups that these initial new connections made at a LAC are strengthened through development of deeper personal connections with others who have similar demographics and who are interested in the same activities. This concurs with the Social Identity Theory [ 58 ] discussed previously.

“and I was walking around the lake in Ballarat, like wandering on my own. I thought, This is ridiculous. I mean, you’ve met all those groups of women coming the opposite way, so I found out what it was all about, so I joined, yeah. So that’s how I got into that.[ Interviewer: Yeah.] Basically sick of walking round the lake on my own. [Interviewer: Yeah, yeah.] So that’s great. It’s very social and they have coffee afterwards which is good.” (female, PAFG)

The subtheme Opportunity for development of friendships describes how, for some people, a number of LAC members have progressed from being just initial social connections to an established friendship. This signifies the strength of the connections that may potentially develop through LAC membership. Some participants from each group mentioned friendships developing, with slightly more discussion of this seen in the social group.

“we all have a good old chat, you know, and, and it’s all about friendship as well.” (female, SocialFG)

The subtheme Opportunity for social responsibility or leadership was mentioned by two people in the active group, however it was not brought up in the social group. This opportunity for leadership is linked with the development of a group identity and desiring to contribute meaningfully to a valued group.

“with our riding group, um, you, a leader for probably two rides a year so you’ve gotta prepare for it, so some of them do reccie rides themselves, so, um, and also every, uh, so that’s something that’s, uh, a responsibility.” (male, PAFG)

Social wellbeing

The social resources described above seem to contribute to a number of social, wellbeing outcomes for participants. The sub themes identified for Social wellbeing were , i) Increased social support, ii) Reduced loneliness, iii) Improved home relationships and iv) Improved social skills.

Increased social support

Social support was measured quantitatively in the survey (no significant change over time for new members) and identified as a benefit of LAC membership during the focus group discussions. However, only one of the members of the active group mentioned social support directly.

‘it’s nice to be able to pick up the phone and share your problem with somebody else, and that’s come about through LAC. ……‘Cos before that it was through, with my family (female, PAFG)

There was some agreement amongst participants of the PA group that they felt this kind of support may develop in time but most of them had been members for less than 2 years.

“[Interviewer: Yeah. Does anyone else have that experience? (relating to above quote)]” There is one lady but she’s actually the one that I joined with anyway. [Interviewer: Okay.] But I, I feel there are others that are definitely getting towards that stage. It’s still going quite early days. (female1, PAFG) [Interviewer: I guess it’s quite early for some of you, yeah.] “yeah” (female 2, PAFG)

Social support through sharing of skills was mentioned by one participant in the social group also, with agreement indicated by most of the others in the social focus group.

Discussion in the focus groups also touched on the subthemes Reduced loneliness and Improved home relationships, which were each mentioned by one person. And focus groups also felt that group membership Improved social skills through opening up and becoming more approachable (male, PAFG) or enabling them to become more accepting of others’ who are different (general agreement in Social FG).

This case study integrated results from a one-year longitudinal survey study and focus group discussions to gather rich information regarding the potential changes in social wellbeing that older adults may experience when joining community organisations offering group activities. The findings from this study indicate that becoming a member of such a community organisation can be associated with a range of social benefits for older adults, particularly related to reducing loneliness and maintaining social connections.

Joining a LAC was associated with a reduction in loneliness over 1 year. This finding is in line with past group-intervention studies where social activity groups were found to assist in reducing loneliness and social isolation [ 49 ]. This systematic review highlighted that the majority of the literature explored the effectiveness of group activity interventions for reducing severe loneliness or loneliness in clinical populations [ 49 ]. The present study extends this research to the general older adult population who are not specifically lonely and reported to be of good general health, rather than a clinical focus. Our findings are in contrast to results from an evaluation of a community capacity-building program aimed at reducing social isolation in older adults in rural Australia [ 59 ]. That program did not successfully reduce loneliness or improve social support. The lack of change from pre- to post-program in that study was reasoned to be due to sampling error, unstandardised data collection, and changes in sample characteristics across the programs [ 59 ]. Qualitative assessment of the same program [ 59 ] did however suggest that participants felt it was successful in reducing social isolation, which does support our findings.

Changes in loneliness were not a main discussion point of the qualitative component of the current study, however some participants did express that they felt less lonely since joining LACVI and all felt they had become more connected with others. This is not so much of a contrast in results as a potential situational issue. The lack of discussion of loneliness may have been linked to the common social stigma around experiencing loneliness outside certain accepted circumstances (e.g. widowhood), which may lead to underreporting in front of others [ 45 ].

Overall, both components of the study suggest that becoming a member of an activity group may be associated with reductions in loneliness, or at least a greater sense of social connectedness. In addition to the social nature of the groups and increased opportunity for social connections, another possible link between group activity and reduced loneliness is an increased opportunity for time out of home. Previous research has found that more time away from home in an average day is associated with lower loneliness in older adults [ 60 ]. Given the significant health and social problems that are related to loneliness and social isolation [ 13 , 14 , 15 ], the importance of group involvement for newly retired adults to prevent loneliness should be advocated.

In line with a significant reduction in loneliness, there was also a trend ( p  = 0.056) toward an increase in social support from baseline to 12 months in the survey study. Whilst suggestive of a change, it is far less conclusive than the findings for loneliness. There are a number of possible explanations for the lack of statistically significant change in this variable over the course of the study. The first is the small sample size, which would reduce the statistical power of the study. It may be that larger studies are required to observe changes in social support, which are possibly only subtle over the course of 1 year. This idea is supported by a year-long randomised controlled trial with 90 mildly-depressed older adults who attended senior citizen’s club in Norway [ 37 ]. The study failed to see any change in general social support in the intervention group compared to the control over 1 year. Additional analysis in that study suggested that people who attended the intervention groups more often, tended to have greater increases in SS ( p  = 0.08). The researchers stated that the study suffered from significant drop-out rates and low power as a result. In this way, it was similar to our findings and suggests that social support studies require larger numbers than we were able to gain in this early exploratory study. Another possible reason for small changes in SS in the current study may be the type of SS measured. The scale used gathered information around functional support or support given to individuals in times of need. Maybe it is not this type of support that changes in such groups but more specific support such as task-specific support. It has been observed in other studies and reviews that task-specific support changes as a result of behavioural interventions (e.g. PA interventions) but general support does not seem to change in the time frames often studied [ 61 , 62 , 63 ].

There were many social wellbeing benefits such as increased social connectivity identified in focus group discussion, but the specific theme of social support was rarely mentioned. It may be that general social support through such community groups may take longer than 1 year to develop. There is evidence that strong group ties are sequentially positively associated between social identification and social support [ 34 ], suggesting that the connections formed through the groups may lead increased to social support from group members in the future. This is supported by results from the focus group discussions, where one new member felt she could call on colleagues she met in her new group. Other new members thought it was too soon for this support to be available, but they could see the bonds developing.

Other social wellbeing changes

In addition to social support and loneliness that were the focus of the quantitative study, the focus group discussions uncovered a number of other benefits of group membership that were related to social wellbeing (see Fig. 3 ). The social resources theme was of particular interest because it reflected some of the mechanisms that appeared enable social wellbeing changes as a result of being a member of a LAC but were not measured in the survey. The main social resources relating to group membership that were mentioned in the focus groups were social connectedness, development of friendships and opportunity for social responsibility or leadership. As mentioned above, there was wide-spread discussion within the focus groups of the development of social connections through the clubs. Social connectedness is defined as “the sense of belonging and subjective psychological bond that people feel in relation to individuals and groups of others.” ([ 25 ], pp1). As well as being an important predecessor of social support, greater social connectedness has been found to be highly important for the health of older adults, especially cognitive and mental health [ 26 , 32 , 34 , 35 , 64 ]. One suggested theory for this health benefit is that connections developed through groups that we strongly identify with are likely to be important for the development of social identity [ 34 ], defined by Taifel as: “knowledge that [we] belong to certain social groups together with some emotional and value significance to [us] of this group membership” (Tajfel, 1972, p. 31 in [ 58 ] p 2). These types of groups to which we identify may be a source of “personal security, social companionship, emotional bonding, intellectual stimulation, and collaborative learning and……allow us to achieve goals.” ([ 58 ] p2) and an overall sense of self-worth and wellbeing. There was a great deal of discussion relating to the opportunity for social connectedness derived through group membership being particularly pertinent following a significant life event such as moving to a new house or partners becoming unwell or dying and especially retirement. This change in their social circumstance is likely to have triggered the need to renew their social identity by joining a community group. Research with university students has shown that new group identification can assist in transition for university students who have lost their old groups of friends because of starting university [ 65 ]. In an example relevant to older adults, maintenance or increase in number of group memberships at the time of retirement reduced mortality risk 8 years later compared to people who reduce their number of group activities in a longitudinal cohort study [ 66 ]. This would fit with the original Activity Theory of ageing; whereby better ageing experience is achieved when levels of social participation are maintained, and role replacement occurs when old roles (such as working roles) must be relinquished [ 67 ]. These connections therefore appear to assist in maintaining resilience in older adults defined as “the ability to maintain or improve a level of functional ability (a combination of intrinsic physical and mental capacity and environment) in the face of adversity” (p29, [ 5 ]). Factors that were mentioned in the focus groups as assisting participants in forming connections with others were shared interest, learning from others, and a fun and accepting environment. It was not possible to assess all life events in the survey study. However, since the discussion from the focus groups suggested this to be an important motivator for joining clubs and potentially a beneficial time for joining them, it would be worth exploring in future studies.

Focus group discussion suggested that an especially valuable time for joining such clubs was around retirement, to assist with maintaining social connectivity. The social groups seem to provide social activity and new roles for these older adults at times of change. It is not necessarily important for all older adults but maybe these ones identify themselves as social beings and therefore this maintenance of social connection helps to continue their social role. Given the suggested importance of social connectivity gained through this organisation, especially at times of significant life events, it would valuable to investigate this further in future and consider encouragement of such through government policy and funding. The majority of these types of clubs exist for older adults in general, but this study emphasises the need for groups such as these to target newly retired individuals specifically and to ensure that they are not seen as ‘only for old people’.

Strengths and limitations

The use of mixed –methodologies, combining longitudinal survey study analysed quantitatively, with a qualitative exploration through focus group discussions and thematic analysis, was a strength of the current study. It allowed the researchers to not only examine the association between becoming a member of a community group on social support and loneliness over an extended period, but also obtain a deeper understanding of the underlying reasons behind any associations. Given the variability of social support definitions in research [ 17 ] and the broad area of social wellbeing, it allowed for open exploration of the topic, to understand associations that may exist but would have otherwise been missed. Embedding the research in an existing community organisation was a strength, although with this also came some difficulties with recruitment. Voluntary coordination of the community groups meant that informing new members about the study was not always feasible or a priority for the volunteers. In addition, calling for new members was innately challenging because they were not yet committed to the club fully. This meant that so some people did not want to commit to a year-long study if they were not sure how long they would be a member of the club. This resulted in slow recruitment and a resulting relatively low sample size and decreased power to show significant statistical differences, which is a limitation of the present study. However, the use of Linear Mixed Models for analysis of the survey data was a strength because it was able to include all data in the analyses and not remove participants if one time point of data was missing, as repeated measures ANOVAs would do. The length of the study (1 year) is another strength, especially compared to previous randomised controlled studies that are typically only 6–16 weeks in length. Drop-out rate in the current study is very low and probably attributable to the benefits of working with long-standing organisations.

The purpose of this study was to explore in detail whether there are any relationships between joining existing community groups for older adults and social wellbeing. The lack of existing evidence in the field meant that a small feasibility-type case study was a good sounding-board for future larger scale research on the topic, despite not being able to answer questions of causality. Owing to the particularistic nature of case studies, it can also be difficult to generalise to other types of organisations or groups unless there is a great deal of similarity between them [ 68 ]. There are however, other types of community organisations in existence that have a similar structure to LACVI (Seniors centres [ 36 , 40 ], Men’s Sheds [ 38 ], University of the Third Age [ 34 , 69 ], Japanese salons [ 70 , 71 ]) and it may be that the results from this study are transferable to these also. This study adds to the literature around the benefits of joining community organisations that offer social and physical activities for older adults and suggests that this engagement may assist with reducing loneliness and maintaining social connection, especially around the time of retirement.

Directions for future research

Given that social support trended toward a significant increase, it would be useful to repeat the study on a larger scale in future to confirm this. Either a case study on a similar but larger community group or combining a number of community organisations would enable recruitment of more participants. Such an approach would also assist in assessing the generalisability of our findings to other community groups. Given that discussions around social benefits of group membership in the focus groups was often raised in conjunction with the occurrence of significant life events, it would be beneficial to include a significant life event scale in any future studies in this area. The qualitative results also suggest that it would be useful to investigate whether people who join community groups in early years post retirement gain the same social benefits as those in later stages of retirement. Studies investigating additional health benefits of these community groups such as physical activity, depression and general wellbeing would also be warranted.

With an ageing population, it is important to investigate ways to enable older adults to age successfully to ensure optimal quality of life and minimisation of health care costs. Social determinants of health such as social support, loneliness and social contact are important contributors to successful ageing through improvements in cognitive health, quality of life, reduction in depression and reduction in mortality. Unfortunately, older adults are at risk of these social factors declining in older age and there is little research investigating how best to tackle this. Community groups offering a range of activities may assist by improving social connectedness and social support and reducing loneliness for older adults. Some factors that may assist with this are activities that encourage sharing interests, learning from others, and are conducted in a fun and accepting environment. Such groups may be particularly important in developing social contacts for newly retired individuals or around other significant life events such as moving or illness of loved ones. In conclusion, ageing policy and strategies should emphasise participation in community groups especially for those recently retired, as they may assist in reducing loneliness and increasing social connections for older adults.

Abbreviations

Focus group

Life Activities Club

Life Activities Clubs Victoria

Linear mixed model

Physical activity

World Health Organisation

United Nations. In: P.D. Department of Economic and Social Affairs, editor. World Population Ageing 2015. New York: United Nations; 2015.

Google Scholar  

World Health Organisation. Global Health and Ageing. 2011 [cited 2014 March 25]; Available from: http://www.who.int/ageing/publications/global_health/en/ .

Balogun JA, et al. Age-related changes in balance performance. Disabil Rehabil. 1994;16(2):58–62.

Article   CAS   Google Scholar  

Singh MAF. Exercise comes of age: rationale and recommendations for a geriatric exercise prescription. J Gerontol Ser A Biol Med Sci. 2002;57(5):M262–82.

Article   Google Scholar  

World Health Organisation. World report on ageing and health. Geneva: World Health Organisation; 2015.

Rowe JW, Kahn RL. Successful Aging1. The Gerontologist. 1997;37(4):433–40.

Depp CA, Jeste DV. Definitions and predictors of successful aging: a comprehensive review of larger quantitative studies. Am J Geriatr Psychiatry. 2006;14(1):6–20.

Song M, Kong E-H. Older adults’ definitions of health: a metasynthesis. Int J Nurs Stud. 2015;52(6):1097–106.

Tate RB, Lah L, Cuddy TE. Definition of successful aging by elderly Canadian males: the Manitoba follow-up study. The Gerontologist. 2003;43(5):735–44.

Phelan EA, et al. Older Adults’ views of “successful aging”—how do they compare with Researchers’ definitions? J Am Geriatr Soc. 2004;52(2):211–6.

Haslam SA, et al. Social cure, what social cure? The propensity to underestimate the importance of social factors for health. Soc Sci Med. 2018;198:14–21.

Holt-Lunstad J, Smith TB, Layton JB. Social relationships and mortality risk: a meta-analytic review. PLoS Med. 2010;7(7):e1000316.

Richard A, et al. Loneliness is adversely associated with physical and mental health and lifestyle factors: results from a Swiss national survey. PLoS One. 2017;12(7):1–18.

Luo Y, et al. Loneliness, health, and mortality in old age: a national longitudinal study. Soc Sci Med. 2012;74(6):907–14.

Luo Y, Waite LJ. Loneliness and mortality among older adults in China. J Gerontol B Psychol Sci Soc Sci. 2014;69(4):633–45.

de Jong Gierveld J, Van Tilburg T, Dykstra PA. Loneliness and social isolation. In: Vangelisti A, Perlman D, editors. Cambridge handbook of personal relationships. Cambridge: Cambridge University Press; 2006. p. 485-500.

Williams P, Barclay L, Schmied V. Defining social support in context: a necessary step in improving research, intervention, and practice. Qual Health Res. 2004;14(7):942–60.

Valtorta N, Hanratty B. Loneliness, isolation and the health of older adults: do we need a new research agenda? J R Soc Med. 2012;105(12):518–22.

Jylhä M. Old age and loneliness: cross-sectional and longitudinal analyses in the Tampere longitudinal study on aging. Can J Aging La Revue can du vieil. 2010;23(2):157–68.

Huxhold O, Miche M, Schüz B. Benefits of having friends in older ages: differential effects of informal social activities on well-being in middle-aged and older adults. J Gerontol B Psychol Sci Soc Sci. 2014;69(3):366–75.

Berkman L, Syme S. Social networks, host resistance and mortality: a nine year follow-up study of alameda county residents. Am J Epidemiol. 1979;185(11):1070–88.

House JS, Landis KR, Umberson D. Social relationships and health. Science. 1988;241(4865):540–5.

Pynnonen K, et al. Does social activity decrease risk for institutionalization and mortality in older people? J Gerontol B Psychol Sci Soc Sci. 2012;67(6):765–74.

Broadhead WE, et al. The epidemiologic evidence for a relationship between social support and health. Am J Epidemiol. 1983;117(5):521–37.

Haslam C, et al. Social connectedness and health. Encyclopedia of geropsychology. 2017:2174–82. https://doi.org/10.1007/978-981-287-080-3_46-1 .

Haslam C, Cruwys T, Haslam SA. “The we’s have it”: evidence for the distinctive benefits of group engagement in enhancing cognitive health in aging. Soc Sci Med. 2014;120:57–66.

Uebelacker LA, et al. Social support and physical activity as moderators of life stress in predicting baseline depression and change in depression over time in the Women’s Health Initiative. Soc Psychiatry Psychiatr Epidemiol. 2013;48(12):1971–82.

Tajvar M, et al. Social support and health of older people in middle eastern countries: a systematic review. Australas J Ageing. 2013;32(2):71–8.

Dalgard OS, Bjork S, Tambs K. Social support, negative life events and mental health. Br J Psychiatry. 1995;166(1):29–34.

Cohen S, Wills TA. Stress, social support, and the buffering hypothesis. Psychol Bull. 1985;98(2):310–57.

Gilmour H. Social participation and the health and well-being of Canadian seniors. Health Rep. 2012;23(4):1B.

Glei DA, et al. Participating in social activities helps preserve cognitive function: an analysis of a longitudinal, population-based study of the elderly. Int J Epidemiol. 2005;34(4):864–71.

Lee SH, Kim YB. Which type of social activities may reduce cognitive decline in the elderly?: a longitudinal population-based study. BMC Geriatr. 2016;16(1):165.

Haslam C, et al. Group ties protect cognitive health by promoting social identification and social support. J Aging Health. 2016;28(2):244–66.

Haslam C, et al. Groups 4 health: evidence that a social-identity intervention that builds and strengthens social group membership improves mental health. J Affect Disord. 2016;194:188–95.

Bøen H. Characteristics of senior Centre users -- and the impact of a group programme on social support and late-life depression. Norsk Epidemiologi. 2012;22(2):261–9.

Bøen H, et al. A randomized controlled trial of a senior Centre group programme for increasing social support and preventing depression in elderly people living at home in Norway. BMC Geriatr. 2012;12:20.

Golding BG. Social, local, and situated: recent findings about the effectiveness of older Men’s informal learning in community contexts. Adult Educ Q. 2011;61(2):103.

Life Activities Clubs. Life Activities Clubs. About Us. . 2014 [cited 2014 January 13, 2014]; Available from: http://www.life.org.au/aboutus .

Hutchinson SL, Gallant KA. Can senior Centres be contexts for aging in third places? J Leis Res. 2016;48(1):50–68.

Millard J. The health of older adults in community activities. Work Older People. 2017;21(2):90–9.

Creswell JW, Plano-Clark VL. Designing and conducting mixed methods research. Thousand Oaks, Calif: SAGE Publications; 2007. p. c2007.

Loeb S, Penrod J, Hupcey J. Focus groups and older adults: tactics for success. J Gerontol Nurs. 2006;32(3):32–8.

PubMed   Google Scholar  

Broadhead W, et al. The Duke-UNC functional social support questionnaire: measurement of social support in family medicine patients. Med Care. 1988;26(7):709–23.

De Jong Gierveld J, Van Tilburg T. Living arrangements of older adults in the Netherlands and Italy: Coresidence values and behaviour and their consequences for loneliness. J Cross Cult Gerontol. 1999;14(1):1–24.

de Jong-Gierveld J, Kamphuls F. The development of a Rasch-type loneliness scale. Appl Psychol Meas. 1985;9(3):289–99.

Tilburg Tv, Leeuw Ed. Stability of scale quality under various data collection procedures: a mode comparison on the ‘De Jong-Gierveld loneliness scale. Int J Public Opin Res. 1991;3(1):69–85.

Hughes ME, et al. A short scale for measuring loneliness in large surveys - results from two population-based studies. Res Aging. 2004;26(6):655–72.

Dickens AP, et al. Interventions targeting social isolation in older people: a systematic review. BMC Public Health. 2011;11:647.

Hawkley LC, et al. Loneliness predicts increased blood pressure: 5-year cross-lagged analyses in middle-aged and older adults. Psychol Aging. 2010;25(1):132.

Netz Y, et al. Loneliness is associated with an increased risk of sedentary life in older Israelis. Aging Ment Health. 2013;17(1):40–7.

Australian Bureau of Statistics. Australian and New Zealand Standard Classification of Occupations, 2013, Version 1.2. Canberra: Australian Bureau of Statistics; 2013.

Australian Bureau of Statistics, Australian Standard Geographical Classification (ASGC). Canberra: Australian Bureau of Statistics; 2011.

Ware JE, Kosinski M, Keller SD. A 12-item short-form health survey - construction of scales and preliminary tests of reliability and validity. Med Care. 1996;34(3):220–33.

Sanson-Fisher RW, Perkins JJ. Adaptation and validation of the SF-36 health survey for use in Australia. J Clin Epidemiol. 51(11):961–7.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

Patton MQ. Qualitative research & evaluation methods. 4 ed. Thousand Oaks, California: Sage Publications; 2015.

Haslam SA, et al. Social identity, health and well-being: an emerging agenda for applied psychology. Appl Psychol. 2009;58(1):1–23.

Bartlett H, et al. Preventing social isolation in later life: findings and insights from a pilot Queensland intervention study. Ageing Soc. 2013;33(07):1167–89.

Petersen J, et al. Time out-of-home and cognitive, physical, and emotional wellbeing of older adults: a longitudinal mixed effects model. PLoS One. 2015;10(10):e0139643.

Lindsay Smith G, et al. The association between social support and physical activity in older adults: a systematic review. Int J Behav Nutr Phys Act. 2017;14(1):56.

Sallis JF, et al. The development of scales to measure social support for diet and exercise behaviors. Prev Med. 1987;16(6):825–36.

Oka R, King A, Young DR. Sources of social support as predictors of exercise adherence in women and men ages 50 to 65 years. Womens Health Res Gender Behav Policy. 1995;1:161–75.

CAS   Google Scholar  

Greenaway KH, et al. From “we” to “me”: group identification enhances perceived personal control with consequences for health and well-being. J Pers Soc Psychol. 2015;109(1):53–74.

Iyer A, et al. The more (and the more compatible) the merrier: multiple group memberships and identity compatibility as predictors of adjustment after life transitions. Br J Soc Psychol. 2009;48(4):707–33.

Steffens NK, et al. Social group memberships in retirement are associated with reduced risk of premature death: evidence from a longitudinal cohort study. BMJ Open. 2016;6(2):e010164.

Havighurst RJ. Successful aging. Gerontol. 1961;1:8–13.

Yin R. Case study research: design and methods. Beverly Hills, CA: Sage publishing; 1994.

Merriam SB, Kee Y. Promoting community wellbeing: the case for lifelong learning for older adults. Adult Educ Q. 2014;64(2):128–44.

Hikichi H, et al. Social interaction and cognitive decline: results of a 7-year community intervention. Alzheimers Dement: Translat Res Clin Interv. 2017;3(1):23–32.

Hikichi H, et al. Effect of a community intervention programme promoting social interactions on functional disability prevention for older adults: propensity score matching and instrumental variable analyses, JAGES Taketoyo study. J Epidemiol Community Health. 2015;69(9):905–10.

Download references

The primary author contributing to this study (GLS) receives PhD scholarship funding from Victoria University. The other authors were funded through salaries at Victoria University.

Availability of data and materials

The datasets generated and/or analysed during the current study are not publicly available due the ethics approval for this study not allowing open access to the individual participant data but are available from the corresponding author on reasonable request.

Author information

Authors and affiliations.

Institute for Health and Sport, Victoria University, Melbourne, Australia

Gabrielle Lindsay-Smith, Grant O’Sullivan, Rochelle Eime, Jack Harvey & Jannique G. Z. van Uffelen

School of Health and Life Sciences, Federation University, Ballarat, Australia

Rochelle Eime & Jack Harvey

Department of Movement Sciences, Physical Activity, Sports and Health Research Group, KU Leuven - University of Leuven, Leuven, Belgium

Jannique G. Z. van Uffelen

You can also search for this author in PubMed   Google Scholar

Contributions

GLS, RE and JVU made substantial contributions to the conception and design of the study. GLS and GOS supervised data collection for the surveys (GLS) and focus groups (GOS and GLS). GLS, GOS, RE, JH and JVU were involved in data analysis and interpretation. All authors were involved in drafting, the manuscript and approved the final version.

Corresponding author

Correspondence to Gabrielle Lindsay-Smith .

Ethics declarations

Ethics approval and consent to participate.

Ethics approval to conduct this study was obtained from the Victoria University Human Research Ethics Committee (HRE14–071 [survey] and HRE15–291 [focus groups]). All survey participants provided informed consent to partake in the study prior to undertaking the first survey by clicking ‘agree’ in the survey software (online participants) or signing a written consent form (paper participants) after they had read all participant information relating to the study. Focus group participants also completed a written consent form after reading and understanding all participant information regarding the study.

Consent for publication

Focus group participants have given permission for their quotes to be used in de-identified form prior to undertaking the research.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Lindsay-Smith, G., O’Sullivan, G., Eime, R. et al. A mixed methods case study exploring the impact of membership of a multi-activity, multicentre community group on social wellbeing of older adults. BMC Geriatr 18 , 226 (2018). https://doi.org/10.1186/s12877-018-0913-1

Download citation

Received : 19 April 2018

Accepted : 10 September 2018

Published : 24 September 2018

DOI : https://doi.org/10.1186/s12877-018-0913-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Social engagement
• Group activity

BMC Geriatrics

ISSN: 1471-2318

• Submission enquiries: [email protected]
• General enquiries: [email protected]

ORIGINAL RESEARCH article

A mixed methods research study of parental perception of physical activity and quality of life of children under home lock down in the covid-19 pandemic.

• 1 Transdisciplinary Research Center in Psychology, Autonomous University of the State of Morelos, Cuernavaca, Mexico
• 2 Institute of Psychology and Special Education, Department of Applied Psychology, University Center for Health Sciences, University of Guadalajara, Guadalajara, Mexico
• 3 Facultad de Ciencias Sociales, Universidad Europea de Canarias, La Orotava, Spain
• 4 Faculty of Psychology and Speech Therapy, Department of Clinical Psychology, Psychobiology and Methodology, University of La Laguna, Santa Cruz de Tenerife, Spain

Household confinement due to the rapid spread of the pandemic caused by COVID-19 has brought very significant changes, such as the forced stay-at-home of children due to the closure of schools. This has meant drastic changes in the organization of daily life and restrictions on their activities, including exercise, which could affect the quality of life of the children due to its importance. In order to study the relationship between physical activity and psychological well-being of minors, a study has been carried out with Mixed Methods Research, combining survey methodology with transversal design with qualitative methodology using discourse analysis. A total of 234 parents of minors in Spain and several Spanish-speaking countries in America participated. The instrument was a questionnaire in Google Forms, which included the Kidscreen-27 quality of life scale. The results show significant differences in both the type of physical activity and its frequency due to age, and differences in parents’ perception of whether their children’s physical activity levels were sufficient or not, both on the health, mood and school subscales, and in the categorization of open responses referring to concerns due to the pandemic, analyzed with the ALCESTE technique. The relationship between physical activity of children and adolescents and quality of life is clearly concluded.

Introduction

At the end of December 2019, the first evidence appeared in Wuhan, China, that a new lethal viral disease had emerged, for which no vaccine or specific medication was available. In March the disease became a pandemic and a large majority of countries, either with specific regulations or through recommendations to the population, established confinement and social distance as the possible solution to prevent further spread of the disease, to avoid saturation of hospitals and curb the lethality of the virus. On March 10th, the global situation with regard to COVID-19 was 113,702 confirmed cases (4,125 new) and 4,012 deaths (203 new) ( World Health Organization [WHO], 2020a ). On December 29, the number of confirmed cases worldwide was to over 79 million, with a cumulative death toll of over 1.7 million ( World Health Organization [WHO], 2020b ).

In the field of Psychology, theoretical formulations have been made to explain the reasons why COVID-19 evolved so rapidly and was so widely spread. Urzúa et al. (2020) point out three factors: (a) illusory optimism; (b) inadequate perception of absence of contingencies produced by the population’s behavior; (c) optimistic risk perception. Vera-Villarroel (2020) has stated that physical and mental health are closely linked, and explains the expansion of the pandemic based on three psychological processes: cognitive, with the population having irrational beliefs about the disease and illusory optimism; emotional, with feelings such as fear, stress and anger; behavioral, with exposure and risk behaviors. The author points out that these factors must be considered in the intervention to save lives.

Several studies have shown the risk that social isolation caused by the pandemic implies not only for the most exposed groups (health workers), but also for the mental health of the general population. Problems of anxiety ( Chew et al., 2020 ; Holmes et al., 2020 ; Wang and Zhao, 2020 ), stress and psychological distress have been reported, both during and even after the biodisaster ( Liu D. et al., 2020 ). Along the same lines, the narrative review conducted by Huarcaya-Victoria (2020) points out three types of problems for the general population: health anxiety, depression and stress. Rajkumar (2020) groups the problems derived from the pandemic into four sections: general population, health workers, vulnerable people and therapeutic strategies and interventions. The author emphasizes the need to study the effect of the situation generated by the pandemic on children and adolescents.

A particularly vulnerable group in this whole situation is children and adolescents. Although results in children for Coronavirus-19 disease are still inconsistent. Changes produced in their environment since COVID-19, such as the restrictions that home isolation and not being able to access the main areas of socialization ( Socías et al., 2020 ), with risks such as stress from both them and their parents, since COVID-19 can cause psychological alterations in children such as those caused by other stressors ( Espada et al., 2020 ; Socías et al., 2020 ).

Certain factors can have effects not only during confinement but also afterward, such as the disappearance of healthy habits like attending classes, which have been replaced by unhealthy behaviors, such as sedentary lifestyles, inappropriate diets, excessive use of screens which can produce, in addition to weight gain, physical problems ( Brazendale et al., 2017 ; Wu et al., 2017 ). From this follows the importance of understanding the effects that a wide variety of personal and contextual factors ( Holgado-Tello et al., 2010 ) can have on children and adolescents and their interaction in the way they experience physical activity and sports during the pandemic situation. Other risks that have been highlighted, depending on age, include substance abuse, accommodation issues and overcrowding and change and disruption of social networks ( Holmes et al., 2020 ). It is expected that, after confinement, in most cases these problems will disappear ( Barlett et al., 2020 ), although some may persist after the situation generated by the pandemic has passed ( Espada et al., 2020 ). Space restrictions and not being able to go outside are especially important in childhood for the proper development of playing, which is essential for its maturation process ( García-Serrano and García-Fernández, 2015 ).

In view of the difficult situation experienced, the population has been provided with recommendations, some of which have been generated by institutions to support their citizens ( Socías et al., 2020 ). These guidelines have many points in common: maintaining routines, being active, supporting minors, carrying out social activities, in short, maintaining a normal life in safety ( Liu J. J. et al., 2020 ). The support of parents is important, who can strengthen family ties and meet the needs of children through appropriate parenting styles ( Wang et al., 2020 ). The need for physical exercise is also stressed ( Holmes et al., 2020 ; Mera-Mamián et al., 2020 ; Romero et al., 2020 ). Physical exercise plays a relevant role both on a physical level ( Vidarte Claros et al., 2011 ) and in mental processes ( Ramírez et al., 2004 ; Zhou et al., 2020 ) as well as on a psychological level ( Berger and Motl, 2000 ; Biddle and Mutrie, 2001 ; Tessier et al., 2007 ; Anderson and Brice, 2011 ). In particular, there is clear evidence of the contribution of physical activity to psychological well-being ( Molina-García et al., 2007 ; Jiménez et al., 2008 ; Romero et al., 2009 ; Fernández Ozcorta et al., 2015 ).

The relationship between physical activity and well-being linked to the quality of life has been the subject of multiple investigations in recent years, which have also emphasized its influence on the general health of the various sectors of the population ( Schwartzmann, 2003 ; Bize et al., 2007 ; Anokye et al., 2012 ). In particular, different studies have highlighted the association between high levels of physical activity, or the practice of sports, and the quality of life in children and adolescents ( Anokye et al., 2012 ; Marker et al., 2018 ; Luna et al., 2019 ).

Likewise, recent reviews of studies on interventions focused on the promotion of sports practices and their impact on issues such as mental health, self-esteem, anxiety levels, and perception of well-being in children and adolescents, underline the benefits of this kind of activities for the general health of this population in particular, showing that physical-sport education pilot programs might promoted significant improvements in specific indicators of subjective well-being and emotional intelligence of participating adolescents’ groups ( Bermejo-Cantarero et al., 2017 ; Luna et al., 2019 ).

The lack of physical activity is a widely reported public health problem ( Bermejo-Cantarero et al., 2017 ). For this reason, evaluation that focuses on the relationships between physical activity and health-related quality of life is an important focus of research in this field. On the other hand, there is little research aimed at exploring parents’ knowledge and perceptions of their children’s physical activity, their ideas about its importance and impact on the way they experience diverse dimensions of a stressful life ( Gallego-Méndez et al., 2020 ; Spinelli et al., 2020 ; Yarımkaya and Esentürk, 2020 ) particularly during the Coronavirus outbreak. Exploring these issues, including the different perspectives of persons involved in families’ life ( Izquierdo-Sotorrío et al., 2016 ), could help provide recommendations and support programs for parents to guide their children’s physical activity.

In the case of children and adolescents, physical activity has important benefits: it promotes growth and enhances both physical development ( Rosa et al., 2018 ) and psychomotor, cognitive and social development, and generally favors all body systems: metabolism of carbohydrates and lipids, control of blood pressure, decreases the risk of type 2 diabetes and improves body composition ( Camargo Lemos and Ortiz Dallos, 2010 ).

Physical activity also favors psychological factors: it helps to build a balanced self-concept and improves self-perception, mood, self-image, physical self-concept, perception of health and life satisfaction, and intellectual function ( Camargo Lemos and Ortiz Dallos, 2010 ; Reigal-Garrido et al., 2012 ; Rosa, 2015 ).

The home quarantine imposed by the COVID-19 may make physical activity more difficult, and as we have seen in the studies reviewed, this leads to a decline in the quality of life of children and adolescents. Quality of life (QoL) is understood as personal satisfaction (or dissatisfaction) with the cultural or intellectual conditions in which an individual lives. Health is one of the domains of quality of life, this domain comprises not only physical health but also psychological health, as well as the interaction that people have with others and with the community ( Ravens-Sieberer et al., 2005 ). For this research, we are interested in reviewing the quality of life, based on the assessment of the well-being perceived by parents.

Given that the collection of high quality data is a priority in order to understand the psychological effects that the quarantine may have produced in the population, and that there is an urgent need to discover, analyze and evaluate the psychological interventions that could alleviate the problems generated and minimize the risks that could occur in the mental health of society ( Holmes et al., 2020 ), the aim of this research is to analyze parents’ perceptions of their children’s quality of life in relation to observed physical activity in the conditions of staying in the housing due to the pandemic situation due to the COVID-19. It hypothesizes the existence of greater quality of life perceived by parents who consider their children to be sufficiently physically active.

In this sense, we try to find out if there is any difference in quality of life between children of different ages and sex in the conditions of staying in the housing due to the pandemic situation due to the COVID-19 as perceived by mothers and fathers. In addition, it is investigated whether the characteristics of the housing (the space) conditioned the perception of the parents about their children doing more or less physical activity, and whether there are differences between the age and the type of physical exercise done. It is also interesting to know the relationship between the level of physical activity and psychological well-being.

Materials and Methods

Methodology and design.

This is a non-experimental design. Mixed methodology was used (Mixed Methods Research, MMR; Johnson and Onwuegbuzie, 2004 ; Denscombe, 2008 ). The data was collected through a cross-sectional design with survey methodology, using an ex post facto design, and there are open questions that allow a qualitative analysis.

To determine the differences in physical activity, three independent variables were considered: age (children, adolescents), sex (male, female), as well as a third variable, grouping parents according to their opinion about the physical activity developed by their children in confinement (sufficient, insufficient). The dependent variables used has been the different scales that make up the KIDSCREEN test, which therefore requires multivariate analysis.

Participants

A total of 234 participants responded to the survey. The average age was 42.82 (SD = 7.10), with a range between 24 and 65. More mothers (203) than fathers (30) participated, and only one of the informants was guardian of the minors, relative in charge of the child. Table 1 presents the data regarding age (values corresponding to the percentiles 25, 50, 75, and over 75) and educational level. The procedure for selecting the sample was one of convenience.

Table 1. Parents’ age and educational level.

Parents and caregivers were asked to think of one of their children when answering the questionnaires. In this way, for the data analysis, they were grouped by the ages of the children, the largest group being children between 8 and 11 years old, 125 (52 female) and 109 adolescents between 12 and 17 (54 female).

The countries of origin of the participants were mainly Spain (134, 57.3%), and Mexico (86, 36.9%) and others American countries (Panamá, Colombia, Argentina, and Chile; 13, 5.8%).

Most families (230, 98.3%) reported not having been victims of the coronavirus. Only four families had a confirmed patient in the family unit, and in four other cases there was a suspicion that a family member had the disease.

In the questionnaire, a question was included about family and housing conditions. Most of the sample lived in the same dwelling with up to four family members (167, 71.4%), while it was less frequent for the family size to be greater than four (67, 28.6%). The average number of rooms, discounting common services, such as kitchen, living room and bathroom, was 3 (113 of the participants, 48.3%), with a range between 1 and 10 rooms. Most of the dwellings have at least one exterior space (177 of the participants, 75.6%).

Instruments

A questionnaire was designed to obtain data on parents’ perceptions of their children’s physical activity, some specific data on the type of housing during their child’s confinement. This questionnaire consists of 18 questions (15 closed, 3 open-ended) distributed in the following categories: (1) descriptive data of the participants (6 items); (2) family and housing conditions (5 items); (3) issues related to the situation produced by the COVID-19 pandemic (3 items); (4) complaints and needs caused by the situation produced by the COVID-19 pandemic (4 items) (see Supplementary Data Sheet 1 ). At the end of the questionnaire it was mentioned that if they wanted to ask for the results of the research they could leave their e-mail. All questions were marked as mandatory in the Google form, so there was no room for incomplete or missing data.

For the HRQoL measure, the Kidscreen-27 Parent Questionnaire ( Ravens-Sieberer et al., 2005 ). Spanish version was used, once the authorization for its use in this study was requested and obtained. This is a questionnaire that assesses health-related quality of life. This questionnaire was used because it provides a parameter to contrast the perception of psychological and health well-being in the child population with the physical activity observed by the parents. It consists of 27 items, which are answered in a Likert-type scale of five alternatives (from nothing to very much), structured in five scales: physical activity (4 items), mood (7 items), family life (7 items), friends (4 items), and school (4 items), and a single question about your child’s general state of health in the last week. The test is filled in by parents, for children and adolescents between the ages of 8 and 18. The original authors ( Ravens-Sieberer et al., 2005 ) offer evidence of the factorial validity of the test and its reliability in all the subscales of the test, in terms of internal consistency, with the total Cronbach’s Alpha value equal to 0.82. With our data, a similar Alpha of 0.831 has been obtained.

The questionnaires were assembled in electronic format with the Google Forms application. It was sent out by email and through social networks (Whatsapp, Facebook, and Twitter) to contacts in different educational associations, using the snowball technique. It was sent during the month of May 2020 (it can be defined as the first period of confinement). Only one of both parents was asked to answer the questionnaire with one of their children in mind (in case they have two or more), and who was in the age range of 8–17 years. The time required to fill in the questionnaire was 15 to 20 min.

At the time of data collection, all participants (regardless of country) were in the same conditions of confinement, leaving the home only for essential activities, with restrictions on going to school, physical activities or recreation outside the home.

As far as ethical aspects are concerned, the Commission on Ethics in Research and Animal Welfare of the University of La Laguna (CEIBA) was asked to authorize the study, which was granted (Registration Number: CEIBA2020-0396). In the questionnaire, the corresponding information for the participants was set out in the Organic Law 3/2018, of December 5th, on Personal Data Protection and guarantee of digital rights ( BOE, 2018 ), guaranteeing the anonymity and confidentiality of the data.

Data Analysis

The relationship between parental consideration of physical activity sufficiency and having or not having outdoor space in the home was calculated using the V of Cramer.

To check the absence of univariate outliers, we used Tukey’s test that takes as reference the difference in interquartile range, considering a slight outlier at 1.5 times this distance, and extreme when it is at three times that distance. To determine the existence of multivariate outliers, the Mahalanobis distance was calculated.

Regarding quality of life, it was analyzed in two ways taking three independent variables: age, sex and parents’ assessment. Since the quality of life variable, measured by Kidscreen, is split into several scales, it requires a multivariate approach, so three MANOVAs were carried out, one according to each independent variable studied. All quantitative analyses were conducted with the SPPS program, v.21.

For the qualitative analysis, the phenomenological discourse analysis method was used, which identifies the meanings of language, through lexical analysis using the ALCESTE software (in French: Analyse des Lexèmes Coocurrents dans les Enoncés Simples d’un Texte ) ( Reinert, 2003 ). This program facilitates the analysis of linguistic materials that generally arise in social research, such as answers to open-ended questions in questionnaires, in-depth interviews or answers based on projective techniques ( De Alba, 2004 ). The ALCESTE methodology consists of three stages: the construction of the data matrix, the classification of the context units (statements) and the description of the classes ( Gil et al., 1994 ). The methodology focuses on the statistical distribution of word succession, taking into account only the simultaneous presence of several words in the same statement. In this way, classes are identified as semantic fields, represented in trees or dendograms. In the ALCESTE method, the initial text is broken down into elementary contextual units (ECUs), which approximately match the size of a sentence.

The statistical analysis, although limited to explain in detail the meaning of a text, allows the elaboration of a “cartography” of the lexical worlds chosen by the speaker to express himself and, therefore, of the reference systems from which he constructs his way of seeing reality ( Gil et al., 1994 ; Reinert, 2003 ).

Quantitative Analysis

Physical activity.

In order to know if there is a relation between the participant’s perception of the sufficiency or not of the physical activity developed by his or her child and the space dedicated to exercises, these variables were analyzed, considering in the household conditions whether there was no outdoor space to carry out activities or if, on the contrary, there was. The results are shown in Table 2 . There is significant dependence between both variables (V of Cramer = 0.146; p = 0.026).

Table 2. Perception of adequacy of physical activity and space for it.

Elimination of Outsiders

Eleven extreme univariate cases were eliminated and none multivariate by Mahalanobis distance, with the criterion of probabilities less than 0.001.

Psychological Well-Being by Age and Sex

Most parents consider their child’s health to be excellent (88, 39.5%) or very good (114, 51.1%), while only 21 (9.4%) rate it as “fair.”

The group was divided into two ages: from 8 to 11 (children) and 12 and older (adolescents). Table 3 shows the descriptive statistics.

Table 3. Age and sex level descriptive statistics.

To know if there are differences by sex and age, a MANOVA was calculated, which was for sex (Wilk’s λ = 0.949, F 5 . 215 = 2.3, p = 0.046, Partial η 2 = 0.051), and for age (Wilk’s λ = 0.843, F 5 . 215 = 8.034, p = 0.001, Partial η 2 = 0.157) nor for interaction (Wilk’s λ = 0.982, F 5 . 215 = 0.796, p = 0.554, Partial η 2 = 0.018). Individual ANOVA results are only significant for the variable age in the health scale ( F 1 , 219 = 7.692, p = 0.006, Partial η 2 = 0.034), with a small effect size and in the friend one ( F 1 , 219 = 28.421, p < 0.001, Partial η 2 = 0.115), with a large effect size.

Physical Activity and Well-Being

In order to assess whether the children developed adequate physical activity, the parents were asked whether they considered it sufficient or insufficient. A total of 146 considered it to be insufficient and 77 sufficient. The informants were divided into two groups according to this variable and it was analyzed whether there were significant differences in their assessment of the psychological well-being of the children. Table 4 presents the mean values and standard deviations of each welfare scale.

Table 4. Descriptive statistics of physical activity and well-being.

The result of the MANOVA was significant (Wilk’s λ = 0.743, F 5 , 217 = 15.001, p < 0.001, Partial η 2 = 0.257). Individual ANOVA results are only significant for the health scale ( F 1 , 223 = 64.821, p < 0.001, Partial η 2 = 0.227), with a large effect size.

Qualitative Analysis

In order to find out the perceptions that families have regarding different aspects of stay-at-home confinement, both required by law and recommended, four open-ended questions were analyzed by ALCESTE, separating into two samples parents who considered that their children were getting enough exercise and those who thought it was insufficient: (a) Explain why you say you have sufficient or insufficient physical activity; (b) How did your child live it?; and (c) What or who does your child miss?

Analysis of the Question “Explain Why You Have Sufficient or Insufficient Physical Activity”

The analysis of ALCESTE, for the group of parents who consider that their children have sufficient physical activity (see Figure 1 ), the results are grouped into three classes, which explain 66% of textual units. The first class is linked to the link between the second and third classes. The most representative class is 1, as it groups the largest number of EUs. The details of the analysis, in terms of class name, UCEs grouped and percentage involved, most representative word and examples, are presented in Table 5 .

Figure 1. Dendogram corresponding to the question “Explain why you have sufficient physical activity.”

Table 5. Analysis of the question “Explain why you have sufficient or insufficient physical activity.”

The reasons given by parents for considering that their children could not get enough physical activity are more dispersed, as they have been grouped into six clases (see Figure 2 ). In this case, there are two groupings: on the one hand, class 2 connects with the union of classes 5 and 6, while class 1 connects with the link between classes 3 and 4. Classes 1, 5, and 6 are related to the impossibility of doing either exercise or sports that they did before the pandemic, while the difficulties of the other set of classes go in the direction of lack of space and the need to go outside.

Figure 2. Dendogram corresponding to the question “Explain why you have insufficient physical activity.”

Analysis of the Question “How Did Your Child Live Not Being Able to go Out on the Street?”

The analysis of the group that considers that their son or daughter has had enough activity explains 51% of the text corpus. The dendogram is shown in Figure 3 .

Figure 3. Dendogram of the question “How did your child live not being able to go outside?” Sufficient physical activity.

On the other hand, in the group of parents who consider the activity performed by their children insufficient, although it explains only 27% of the corpus, extracting only two classes ( Figure 4 ).

Figure 4. Dendogram of the question: “How did your child live not being able to go outside?” Insufficient physical activity.

Table 6 shows the detail of the classes, in terms of their name, number of UCEs they group, percentage of the corpus they explain and the most representative word, as well as representative examples of each class. In both groups, a distinction is made between positive aspects, of being at home, or pointing out some kind of problem.

Table 6. Information from the analysis to the question “How did your child live not being able to go out on the street?”

Analysis of the Question “What or Who Does Your Child Miss?”

The analysis of this question, for the group of parents who consider that the physical activity developed by their son or daughter is sufficient, gives two classes, which explain 65% of the textual units, that is, an average relevance of the treatment (see Figure 5 ). These are two antagonistic classes: the second is the one that groups the most textual units (71.70%), where it is clear that the child misses both the extended family and the people in his or her school environment. The first class includes those who responded that they have not missed anything or anyone and is quite homogeneous: they do not miss anyone (see Table 7 ).

Figure 5. Dendogram of the answers: What or who does your child miss? Sufficient physical activity.

Table 7. Information from the analysis to the question “What or who does your child miss?”

In the case of parents who feel that their son or daughter does not get enough physical activity, there are six classes, with a grouping of classes on a ladder: from class 1 to 4 are connected individually, linking class 5 with 6.

It explains 72% of the textual units, which means that the relevance of the treatment is high. They are presented in Figure 6 .

Figure 6. Dendogram of the answers: “What or who does your child miss?” Insufficient physical activity.

The first thing to note is that the data collection was done in the months of April and early May, at the time of the most severe confinement, as in Spain, Mexico, Panama, and Argentina ( BBC, 2020 ). It is true that the regulatory conditions regarding confinement have differed in the countries where the participants in this research live, in some cases, such as Spain, being obligatory by the State of Alarm, while in other countries governments strongly recommended avoiding going out and staying at home. This has meant 24-hour family life, with parents having to telework and children being taught online. The possibilities of exercising under these conditions have been very limited, which can have important effects on the psychological well-being of the minors.

As far as the health of their children is concerned, a large majority consider it to be good or very good. Furthermore, taking into account the five scales of quality of life in relation to age levels (children and adolescents), parents value the health of their children more the younger they are. In contrast, differences in contact with friends score higher for adolescents.

Parents’ perceptions of their children’s quality of life significant differences are observed with respect to sex at the global level, which is not maintained in the scales separately, but they do differ by age on two of the instrument’s scales: health, where younger children score higher, and friends, with the opposite result, as would be expected: adolescents score significantly higher on this scale.

Physical activity is conditioned by the type of housing. The results show that when there is no outdoor space to develop physical activity, parents find that exercise performed by their children is insufficient more often.

The objective of this study, to establish whether there is a relationship between physical activity and psychological quality of life in the conditions of confinement at home from the parents’ perspective, has been clearly corroborated, both in quantitative and qualitative analyses, finding differences between the two established groups of participants: those who considered that their children could develop sufficient physical exercise versus those who thought it was insufficient. Divergences are shown in both groups at the quantitative and qualitative levels.

With respect to the quality of life instrument, there are significant differences between the overall scores of the two groups; however, significant differences are only found in the health scale; when parents consider that the physical activity developed by their children is insufficient, lower scores are obtained in that scale. These results support the hypothesis of a positive relationship between quality of life and physical activity.

The differences found between the two groups of parents (those who consider their children’s physical activity sufficient and those who do not) in the quantitative analyses are also verified in the qualitative ones. The second group of parents shows more dispersion in the open responses given, as well as greater concern.

Thus, in the first open question analyzed qualitatively, “ Explain why you have sufficient or insufficient physical activity ,” the discourse of some parents differs significantly, as it is obvious, since the reasons they give for the physical activity done by their children being sufficient must be differentiated from those who consider it to be insufficient. In the latter, two perspectives are clearly distinguished in the two branches that appear in the dendogram: lack of space or impossibility of doing the exercise they would like to do. Moreover, it also confirms what has already been commented, that is, how there is a relationship between physical space and the facilities of households to exercise is related to the satisfaction or dissatisfaction with the physical activity performed by their children.

The second question, centered on their child’s experiences of not being able to go out, parents who feel their children have enough physical activity, report that their children experienced the lock down positively. On the other hand, in the other group there is a division of opinions: one part considers that their children lived the lock down without problems, but others think that their children lived it with stress, being this last one the most representative class. It confirms again a greater decline in the quality of life of their children for this group.

Finally, in the question relating to whether their child misses something or someone, there is greater variability among the children whose parents consider they do not have enough activity, since the answers are grouped together in one more class, where there is content where school life is missing.

The limitations of this work are about convenience samples, since there is no guarantee of absence of selection bias. However, having included several countries, all of them with a significant restriction on going out of the house, it gives indications of cross validity. This unusual development of the pandemic has evened out the differences between nations in a common struggle against an unprecedented biological crisis.

As far as the uncertainty of living under what has come to be called the new normality, together with the certainty that the threat of the pandemic is not over and that outbreaks, more or less virulent, may occur, it is particularly relevant to carry out research on mental health and psychological well-being, in order to be able to foresee more precisely the actions to be taken, knowing the dangers involved. Holmes et al. (2020) point out how important it is to accumulate experience based on the evidence that has provided the lessons learned so that those in power can coordinate measures that will damage the lives of citizens as little as possible, especially those who are most vulnerable. In this regard, since children are a vulnerable sector of the population, knowledge of their reactions and how they have been affected is particularly relevant. For future research, this could also include children’s self-report, comparing their perception with their mothers and fathers’s ( Izquierdo-Sotorrío et al., 2016 ). As a general recommendation in the light of the data collected, emphasizing the importance of exercise in guaranteeing the psychological well-being of minors is vital and must be conveyed to the population.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by The University of La Laguna’s Ethics Committee of Research and Animal Welfare has approved this research (Registration Number: CEIBA2020-0396). The patients/participants provided their written informed consent to participate in this study.

Author Contributions

ÁB, GL-A, MV, and DC-S had participated in theoretical review. ÁB, ER-N, GL-A, DC-S, and MV had participated in research design and instrument. ÁB had participated in the data analysis. ÁB, ER-N, GL-A, DC-S, and MV had participated in discussion. ÁB, ER-N, GL-A, DC-S, MV, and TA had participated in the study planning, writing, and revision of the article. All authors contributed to the article and approved the submitted version.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors thank all the families that have participated in this research.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2021.649481/full#supplementary-material

Anderson, R. J., and Brice, S. (2011). The mood-enhancing benefits of exercise: memory biases augment the effect. Psychol. Sport Exerc. 12, 79–82. doi: 10.1016/j.psychsport.2010.08.003

CrossRef Full Text | Google Scholar

Anokye, N. K., Trueman, P., Green, C., Pavey, T. G., and Taylor, R. S. (2012). Physical activity and health related quality of life. BMC Public Health 12:624. doi: 10.1186/1471-2458-12-624

PubMed Abstract | CrossRef Full Text | Google Scholar

Barlett, J. D., Griffin, J., and Thomson, D. (2020). Resources for Supporting Children’s Emotional Well-Being During the COVID-19 Pandemic. Available online at: https://www.childtrends.org/publications/resources-for-supporting-childrens-emotional-well-being-during-the-covid-19-pandemic [Accessed March 19, 2020]

Google Scholar

BBC (2020). Coronavirus: Qué tan Rápidamente se Expande el Covid-19 Por América Latina (y Cómo se Compara con Otras Regiones Del Mundo) [Coronavirus: How Fast Covid-19 is Spreading in Latin America (and How it Compares to Other Regions of the World)]. Available online at: https://www.bbc.com/mundo/noticias-america-latina-51802906 [Accessed June 20, 2020]

Berger, B. G., and Motl, R. W. (2000). Exercise and mood: a selective review and synthesis of research employing the Profile of mood states. J. Appl. Sport Psychol. 12, 69–92. doi: 10.1080/10413200008404214

Bermejo-Cantarero, A., Álvarez-Bueno, C., Martínez-Vizcaino, V., García-Hermoso, A., Torres-Costoso, A. I., and Sánchez-López, M. (2017). Association between physical activity, sedentary behavior, and fitness with health related quality of life in healthy children and adolescents: a protocol for a systematic review and meta-analysis. Medicine 96:e6407. doi: 10.1097/MD.0000000000006407

Biddle, S. J. H., and Mutrie, N. (2001). Psychology of Physical Activity: Determinants, Well-Being and Interventions. London: Routledge.

Bize, R., Johnson, J. A., and Plotnikoff, R. C. (2007). Physical activity level and health-related quality of life in the general adult population: a systematic review. Prev. Med. 45, 401–415. doi: 10.1016/j.ypmed.2007.07.017

BOE (2018). BOE (2018, n° 294 of December 6th) Boletín Oficial del Estado. Available online at: https://www.boe.es/boe/dias/2018/12/06/pdfs/BOE-S-2018-294.pdf [Accessed June 20, 2020]

Brazendale, K., Beets, M. W., Weaver, R. G., Pate, R. R., Turner-McGrievy, G. M., Kaczynski, A. T., et al. (2017). Understanding differences between summer vs. school obesogenic behaviors of children: the structured days hypothesis. Int. J. Behav. Nutr. Phys. Act. 14:100.

Camargo Lemos, D. M., and Ortiz Dallos, C. J. (2010). Actividad física en niños y adolescentes: determinantes y medición. Rev. Univ. Ind. Santander. Salud. 42, 153–165.

Chew, Q. H., Wei, K. C., Vasoo, S., Chua, H. C., and Sim, K. (2020). Narrative synthesis of psychological and coping responses towards emerging infectious disease outbreaks in the general population: practical considerations for the COVID-19 pandemic. Singapore Med. J. 61, 1–31. doi: 10.11622/smedj.2020046

De Alba, M. (2004). El método ALCESTE y su aplicación al estudio de las representaciones sociales del espacio urbano: el caso de la ciudad de México. Papers Soc. Representations 13, 1.1–1.20.

Denscombe, M. (2008). A Research Paradigm for the Mixed Methods Approach. J. Mix. Methods Res. 2, 270–283. doi: 10.1177/1558689808316807

Espada, J. P., Orgilés, M., Piqueras, J. A., and Morales, A. (2020). Las buenas prácticas en la atención psicológica infanto-juvenil ante el COVID-19. Clín. Salud. 31, 109–113. doi: 10.5093/clysa2020a14

Fernández Ozcorta, E. J., Almagro Torres, B. J., and Sáenz-López Buñuel, P. (2015). Inteligencia emocional percibida y el bienestar psicológico de estudiantes universitarios en función del nivel de actividad física. Cult. Cienc. Deporte 10, 31–39. doi: 10.12800/ccd.v10i28.513

Gallego-Méndez, J., Pérez-Gómez, J., Calzada-Rodríguez, J. I., Denche-Zamorano, ÁM., Mendoza-Muñoz, M., Carlos-Vivas, J., et al. (2020). Relationship between health-related quality of life and physical activity in children with hyperactivity. Int. J. Environ. Res. Public Health 17:2804. doi: 10.3390/ijerph17082804

García-Serrano, P., and García-Fernández, J. A. (2015). “La rueda del hámster. Crítica a la especialización y confinamiento infantil en el espacio público urbano,” in Proceedings of the 2nd Conference on Urban Processes. Facultad de CC Políticas y Sociología , (Madrid: Universidad Complutense de Madrid).

Gil, J., García, E., and Rodríguez, E. (1994). Una metodología para el análisis estadístico de datos textuales: el programa Alceste. Rev. Investig. Educ. 24, 107–124.

Holgado-Tello, F., Navas, L., López, M., and García, T. (2010). A structural model of goal orientation in sports: personal and contextual variables. Span. J. Psychol. 13, 257–266. doi: 10.1017/S1138741600003838

Holmes, E. A., O’Connor, R. C., Perry, V. H., Tracey, I., Wessely, S., Arseneault, L., et al. (2020). Multidisciplinary research priorities for the COVID-19 pandemic: a call for action for mental health science. Lancet Psychiatry 7, 547–560. doi: 10.1016/S2215-0366(20)30168-1

Huarcaya-Victoria, J. (2020). Consideraciones sobre la salud mental en la pandemia de COVID-19. Rev. Peru. Med. Exp. Salud Pública 37, 327–334. doi: 10.17843/rpmesp.2020.372.5419

Izquierdo-Sotorrío, E., Holgado-Tello, F. P., and Carrasco, M. Á (2016). Incremental validity and informant effect from a multi-method perspective: assessing relations between parental acceptance and children’s behavioral problems. Front. Psychol. 7:664. doi: 10.3389/fpsyg.2016.00664

Jiménez, M. G., Martínez, P., Miró, E., and Sánchez, A. I. (2008). Bienestar psicológico y hábitos saludables: ¿están asociados a la práctica de ejercicio físico? Inter. J. Clin. Health Psychol. 8, 185–202.

Johnson, R. B., and Onwuegbuzie, A. J. (2004). Mixed methods research: a research paradigm whose time has come. Educ. Res. 33, 14–26. doi: 10.3102/0013189X033007014

Liu, D., Ren, Y., Yan, F., Li, Y., Xu, X., Yu, X., et al. (2020). Psychological impact and predisposing factors of the coronavirus disease 2019 (COVID-19) pandemic on general public in China. Lancet Psychiatry doi: 10.2139/ssrn.3551415 [Epub ahead of print].

Liu, J. J., Bao, Y., Huang, X., Shi, J., and Lu, L. (2020). Mental health considerations for children quarantined because of COVID-19. Lancet Child Adolesc. Health 4, 347–349.

Luna, P., Guerrero, J., and Cejudo, J. (2019). Improving adolescents’ subjective well-being, trait emotional intelligence and social anxiety through a programme based on the sport education model. Int. J. Environ. Res. Public Health 16:1821. doi: 10.3390/ijerph16101821

Marker, A. M., Steele, R. G., and Noser, A. E. (2018). Physical activity and health-related quality of life in children and adolescents: a systematic review and meta-analysis. Health Psychol. 37, 893–903. doi: 10.1037/hea0000653

Mera-Mamián, A. Y., Tabares González, E., Montoya-González, S., Muñoz Rodríguez, D., and Monsalve, F. (2020). Recomendaciones prácticas para evitar el desacondicionamiento físico durante el confinamiento por la pandemia asociada a COVID-19. Universidad Salud. 22, 166–177. doi: 10.22267/rus.202202.188

Molina-García, J., Castillo, I., and Pablos, C. (2007). Bienestar psicológico y práctica deportiva en universitarios. Motricidad. Eur. J. Hum. Mov. 18, 79–91.

Rajkumar, R. P. (2020). COVID-19 and mental health: a review of the existing literature. Asian J. Psychiatry 52:102066. doi: 10.1016/j.ajp.2020.102066

Ramírez, R., Vinaccia, S., and Suárez, G. R. (2004). El impacto de la actividad física y el deporte sobre la salud, la cognición, la socialización y el rendimiento académico: una revisión teórica. Rev. Estudios Soc. 18, 67–75. doi: 10.7440/res18.2004.06

Ravens-Sieberer, U., Gosch, A., Rajmil, L., Erhart, M., Bruil, J., Dür, W., et al. (2005). KIDSCREEN-52 Medida de la calidad de vida de los niños y adolescentes. Rev. Exp. Farmacoeconomía Resultados Invest. 5, 353–364.

Reigal-Garrido, R., Videra García, A., Parra Flores, J. L., and Juárez Ruiz de Mier, R. (2012). Actividad Físico Deportiva, Autoconcepto Físico y Bienestar Psicológico en la Adolescencia. Retos: Nuevas tendencias en Educación Física, Deporte y Recreación. 22, 19–23.

Reinert, M. (2003). Alceste Users’ Manual.(English Version). Touluse: Image.

Romero, A., García-Mas, A., and Brustad, R. (2009). Estado del arte y perspectiva actual del concepto de bienestar psicológico en psicología del deporte. Rev. Latinoam. Psicol. 41, 335–347.

Romero, C. C., del Mazo Fuente, A., Besada, M. D., and Hernández, M. D. L. O. R. (2020). Algunas aportaciones de la orientación educativa en red durante el confinamiento por la pandemia de COVID-19. Rev. AOSMA, N° Extra 28, 33–49. Available online at: https://dialnet.unirioja.es/descarga/articulo/7381631.pdf

Rosa, A. (2015). Actividad Física y Autoconcepto: Una Revisión Teórica Aplicada al Ámbito Escolar. Available online at: https://www.efdeportes.com/efd202/actividad-fisica-y-autoconcepto.htm (accessed December 31, 2020).

Rosa, A., García-Cantó, E., and Carrillo, P. J. (2018). Percepción de salud, actividad física y condición física en escolares. Cuad. Psicol. Deport. 18, 179–189.

Schwartzmann, L. (2003). Calidad de vida relacionada con la salud: aspectos conceptuales. Cienc. Enferm. 9, 09–21. doi: 10.4067/S0717-95532003000200002

Socías, C. O., Ballester Brage, L. L., and Nevot-Caldentey, L. L. (2020). Apoyo Familiar Ante el Covid-19 en España [Family Support Against Covid-19 in Spain]. Available online at: https://doi.org/10.1590/SciELOPreprints.297 [Accessed March 3, 2020].

Spinelli, M., Lionetti, F., Pastore, M., and Fasolo, M. (2020). Parents’ stress and children’s psychological problems in families facing the COVID-19 outbreak in Italy. Front. Psychol. 11:1713. doi: 10.3389/fpsyg.2020.01713

Tessier, S., Vuillemin, A., Bertrais, S., Boini, S., Le Bihan, E., Oppert, J. M., et al. (2007). Association between leisure-time physical activity and health-related quality of life changes over time. Prev. Med. 44, 202–208. doi: 10.1016/j.ypmed.2006.11.012

Urzúa, A., Vera-Villarroel, P., Caqueo-Urízar, A., and Polanco-Carrasco, R. (2020). La Psicología en la prevención y manejo del COVID-19. Aportes desde la evidencia inicial. Terapia Psicol. 38, 103–118.

Vera-Villarroel, P. (2020). Psicología y covid-19: un análisis desde los procesos psicológicos básicos. Panam. J. Neuropsychol. 14, 10–18.

Vidarte Claros, J. A., Vélez, C., Sandoval, C., and Alfonso, M. L. (2011). Actividad física: estrategia de promoción de la salud. Rev. Hacia Promoción Salud. 16, 202–218.

Wang, C., and Zhao, H. (2020). The impact of COVID-19 on anxiety in Chinese university students. Front. Psychol. 11:1168. doi: 10.3389/fpsyg.2020.01168

Wang, G., Zhang, Y., Zhao, J., Zhang, F., and Jiang, F. (2020). Mitigate the effects of home confinement on children during the COVID-19 outbreak. Lancet 395, 945–947. doi: 10.1016/S0140-6736(20)30547-X

World Health Organization [WHO] (2020a). Coronavirus Disease 2019 (COVID-19) Situation Report – 50. Available online at: https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200310-sitrep-50-covid-19.pdf?sfvrsn=55e904fb_2 . [Accessed Mars 10, 2020]

World Health Organization [WHO] (2020b). Weekly Epidemiological Update - 29 December 2020. Available online at: https://www.who.int/publications/m/item/weekly-epidemiological-update—29-december-2020 [Accessed December 31, 2020]

Wu, X. Y., Han, L. H., Zhang, J. H., Luo, S., Hu, J. W., and Sun, K. (2017). The influence of physical activity, sedentary behavior on health-related quality of life among the general population of children and adolescents: a systematic review. PLoS One 12:e0187668. doi: 10.1371/journal.pone.0187668

Yarımkaya, E., and Esentürk, O. K. (2020). Promoting physical activity for children with autism spectrum disorders during Coronavirus outbreak: benefits, strategies, and examples. Int. J. Dev. Disabil. 1–6. doi: 10.1080/20473869.2020.1756115

Zhou, F., Xi, X., and Qin, C. H. (2020). Regular open-skill exercise generally enhances attentional resources related to perceptual processing in young males. Front. Psychol. 11:941. doi: 10.3389/fpsyg.2020.00941

Keywords : home lock down, physical activity, quality of life, pandemic, mixed methods research

Citation: López-Aymes G, Valadez MD, Rodríguez-Naveiras E, Castellanos-Simons D, Aguirre T and Borges Á (2021) A Mixed Methods Research Study of Parental Perception of Physical Activity and Quality of Life of Children Under Home Lock Down in the COVID-19 Pandemic. Front. Psychol. 12:649481. doi: 10.3389/fpsyg.2021.649481

Received: 04 January 2021; Accepted: 16 February 2021; Published: 15 March 2021.

Reviewed by:

Copyright © 2021 López-Aymes, Valadez, Rodríguez-Naveiras, Castellanos-Simons, Aguirre and Borges. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Elena Rodríguez-Naveiras, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

• Study Protocol
• Open access
• Published: 29 March 2024

Evaluation of heroin-assisted treatment in Norway: protocol for a mixed methods study

• Lars Henrik Myklebust 1 ,
• Desiree Eide 1 ,
• Espen A. Arnevik 4 ,
• Omid Dadras 2 ,
• Silvana De Pirro 1 , 6 ,
• Rune Ellefsen 4 ,
• Lars T. Fadnes 2 , 3 ,
• Morten Hesse 5 ,
• Timo L. Kvamme 5 ,
• Francesca Melis 1 ,
• Ann Oldervoll 1 ,
• Birgitte Thylstrup 5 ,
• Linda E.C. Wusthoff 1 , 4 &
• Thomas Clausen 1  

BMC Health Services Research volume  24 , Article number:  398 ( 2024 ) Cite this article

Metrics details

Opioid agonist treatment (OAT) for patients with opioid use disorder (OUD) has a convincing evidence base, although variable retention rates suggest that it may not be beneficial for all. One of the options to include more patients is the introduction of heroin-assisted treatment (HAT), which involves the prescribing of pharmaceutical heroin in a clinical supervised setting. Clinical trials suggest that HAT positively affects illicit drug use, criminal behavior, quality of life, and health. The results are less clear for longer-term outcomes such as mortality, level of function and social integration. This protocol describes a longitudinal evaluation of the introduction of HAT into the OAT services in Norway over a 5-year period. The main aim of the project is to study the individual, organizational and societal effects of implementing HAT in the specialized healthcare services for OUD.

The project adopts a multidisciplinary approach, where the primary cohort for analysis will consist of approximately 250 patients in Norway, observed during the period of 2022–2026. Cohorts for comparative analysis will include all HAT-patients in Denmark from 2010 to 2022 ( N  = 500) and all Norwegian patients in conventional OAT ( N  = 8300). Data comes from individual in-depth and semi-structured interviews, self-report questionnaires, clinical records, and national registries, collected at several time points throughout patients’ courses of treatment. Qualitative analyses will use a flexible inductive thematic approach. Quantitative analyses will employ a wide array of methods including bi-variate parametric and non-parametric tests, and various forms of multivariate modeling.

The project’s primary strength lies in its comprehensive and longitudinal approach. It has the potential to reveal new insights on whether pharmaceutical heroin should be an integral part of integrated conventional OAT services to individually tailor treatments for patients with OUD. This could affect considerations about drug treatment even beyond HAT-specific topics, where an expanded understanding of why some do not succeed with conventional OAT will strengthen the knowledge base for drug treatment in general. Results will be disseminated to the scientific community, clinicians, and policy makers.

Trial registration

The study was approved by the Norwegian Regional Committee for Medical and Health Research Ethics (REK), ref.nr.:195733.

Peer Review reports

Opioid use disorder (OUD) is a major global health concern with an estimated caseload of 31.5 million in 2022 [ 1 ]. It is frequently related to infectious diseases from injection-based drug use, psychiatric disorders, deterioration of social relations, reduced workforce participation, and a tenfold increase in crude all-cause rate of mortality [ 2 ]. The treatment and care for patients with OUD has gradually developed from an initial emphasis on abstinence and withdrawal management, to regular prescriptions of opioid agonists for maintenance treatment (OAT) [ 3 ].

Half a century after the first initiatives of prescribing methadone for OUD in a regular manner [ 4 , 5 ] OAT now has a strong evidence-base [ 6 ]. Overall, it contributes to a substantial reduction in mortality, general health benefits, and reduced use of illicit drugs and criminal activity [ 6 , 7 , 8 , 9 ]. Still, not all individuals find conventional OAT sufficiently attractive over time, and cycles of dropout and re-entering are ongoing challenges in these programs [ 10 , 11 , 12 ]. A variable retention rate of 20–84% has been observed [ 13 ]. Among the efforts to improve the inclusion of patients in OAT is the introduction of more diverse medication options, such as rapid-onset, short-acting injectable pharmaceutical opioids such as heroin [ 14 ].

The use of medical grade heroin (diacetylmorphine) in treating OUD has been applied in England since the 1920s, originally as hand-out prescriptions to take home [ 15 , 16 ]. Initiatives to incorporate it into more regular OAT started in Switzerland in 1994, with promising results [ 17 , 18 ]. Now, three decades later and after clinical trials from several European countries and Canada, the body of research suggests that heroin-assisted treatment (HAT) is beneficial for a sub-selection of patients in regard to health outcomes and reductions in use of illicit drugs and criminal behavior [ 19 , 20 , 21 ]. The results are less clear for longer-term outcomes such as mortality [ 6 , 19 ].

Still, HAT remains politically controversial [ 22 ], and reduced illicit heroin use and criminal behavior may not be compelling arguments for its efficacy. Rather, as for any other medical treatment its impact may better be assessed by patients’ improvement in quality of life, everyday level of function, and mortality [ 23 ].

Although newer studies suggests that take home doses are a feasible and safe alternative for patients deemed suitable [ 24 , 25 ], medical heroin is typically administered under rigorous and comprehensive medical supervision due to the risk of serious adverse events and diversion [ 26 ]. Studies on cost effectiveness suggest both excessive expenses and inconclusive results when compared with methadone treatment, which are possibly dependent on methodological issues and poor consideration of the mechanisms involved [ 20 , 27 , 28 ].

Additionally, most of the research on the effectiveness of HAT originates from randomized clinical trials which may have limitations concerning the understanding of long-term outcomes and the mechanisms behind [ 23 ]. Thus, the main contribution of HAT may lie in the engagement of a high-risk population in utilization of health- and social services over time, like the more conventional options of OAT [ 23 , 29 ]. A more comprehensive view of outcomes beyond the mere quantity and frequency of drug use and criminal behaviour can provide crucial information about the mechanisms responsible for treatment effectiveness, and its possible impact on other clinically and socially relevant parameters [ 30 ].

The current Norwegian HAT study is presented in this context. The study is part of a clinical project by the Norwegian Directorate of Health, with the aim to evaluate the implementation of HAT into the national OAT services. It is based on a model from Denmark where the use of medical heroin was introduced in 2010, following the British “RIOTT” line of test trials from 2005 [ 31 ]. Denmark currently has five clinics as permanent parts of the national healthcare system, although a limited amount of research has been published from this model [ 32 ].

The Norwegian HAT-project

OAT programs based on prescription of methadone and buprenorphine has in various forms been integrated into the Norwegian health and social services-system since 1997 [ 33 ]. In the spring of 2020, the Norwegian Directorate of Health introduced a time-limited, clinically based project on the use of pharmaceutical heroin in the specialist healthcare services. Based on a day-center model, treatment is offered at two designated clinics in the largest Norwegian cities of Oslo and Bergen. The clinics consist of injection sites and medical personnel for the administration of pharmaceutical heroin twice a day, in combination with a take-home oral overnight dose of slow-releasing opioid-agonist such as methadone or morphine. Take-home doses of heroin are not granted, and patients must attend daily all year around. Psychosocial services and support are also offered [ 34 ]. Patients are referred from other services of substance use disorder treatment, specialist healthcare services or general practitioners. Criteria for admission are ongoing OUD with at least one former attempt of conventional OAT, being over 18 years of age and with general competency of consent. Exclusion criteria are severe mental disorders with reduced competency of consent, pregnancy, or repeated violent behavior.

The Norwegian Centre for Addiction Research (SERAF) at the University of Oslo was granted the research-based evaluation of the HAT project in 2021. The study will be conducted together with Section for Clinical Addiction Research (RusForsk) at Oslo University Hospital, Bergen Addiction Research Group (BAR) at Haukeland University Hospital in Bergen, Centre for Alcohol and Drug Research (CRF) at Aarhus University in Denmark, and the Norwegian user organization proLARNett.

The primary aim of the research project is to examine the effects from implementing HAT in Norway for individual patients and for the health services organization. A secondary aim is to compare these findings with the Danish HAT program.

Based on the Norwegian Directorate of Health’s specifications in the project proposal, the study will cover the following thematic areas:

Explore the attitudes, experiences and challenges of HAT as perceived by patients, their relatives, and clinical staff.

Describe changes in mental and physical health among patients receiving HAT, and in what way it is associated with outcomes such as quality of life, utilization of health- and social services, social reintegration, criminal behavior and use of illicit drugs.

Report any serious adverse events and incidents at treatment initiation, during treatment, and after discharge from HAT.

Perform an economic evaluation of the program with associated clinical benefits and societal costs.

Evaluate the organizational processes involved in the implementation of HAT in Norwegian specialist healthcare services, and the eventual impact from HAT on OUD patients’ utilization of conventional OAT.

Additional research relevant to HAT that is not explicitly outlined in the proposal (may require additional approvals from the Norwegian Regional Committee for Medical and Health Research Ethics.)

The themes were operationalized into six work packages, with corresponding research questions and data sources (shown in Table  1 ).

Methods and design

The project is a multi-dimensional study, involving an array of methodological approaches and data sources. The main part is a prospective cohort study of all Norwegian HAT patients, compared with the cohorts of all Danish HAT patients and Norwegian patients in conventional OAT.

Study populations and size

The primary target group is all patients enrolled in the two HAT clinics in Oslo and Bergen during the period 2022–2026, with an expected total sample size of N  = 250. Based on earlier findings, the ratio of men to women is expected to be 4:1, with an age distribution of 27–60, presenting multiple substance use disorders. As the study is based on the total clinical population, representation will be determined by its demographics, with no exclusion of genders or ethnic minorities. The patients who have applied to but have not been accepted for HAT will be used for comparison, with an expected sample size of 100.

Comparative data from the Danish cohort will be drawn from the comprehensive dataset at Aarhus University from 2010 and onwards, with a sample size of approximately 500 [ 35 ]. Likewise, the comprehensive dataset at SERAF on the cohort of Norwegian patients in conventional OAT from 2003 has an approximate sample size of 8300.

Data sources

Data on the primary cohort of Norwegian HAT patients will be based on a prospective collection of both qualitative and quantitative variables from treatment inclusion and throughout the project period. For the cohorts of Danish patients, of Norwegian patients that have been referred to but not granted HAT, and of Norwegian patients in conventional OAT, data are mainly based on national registries.

In-depth and semi-structured interviews and observation

The qualitative part of the project includes individual in-depth and semi-structured interviews with patients and relatives on their views and experiences with HAT, and focus group interviews with staff concerning implementation, clinical and legal aspects of the project. Semi-structured interview protocols have been developed by the project group and user representatives. Interviews will include 25–35 patients and 10–20 family members, conducted by a team of researchers and user representatives at 1, 6, 18 months after patients enter treatment, and with relatives after 4 and 12 months. Focus group interviews with staff will be conducted at 3, 9 and 18 months. Further, the clinic managers are being interviewed at several timepoints from the planning of the clinics and throughout the duration of the project.

For insights into clinic aspects not identified through interviews, researchers will conduct participant observation in the clinics over several periods of 1–2 weeks throughout the study.

Questionnaires

The quantitative part of the project will use similar questionnaires to preceding projects involving patients in conventional OAT. These will evaluate changes in physical and mental health, personal economy, utilization of social services, criminal behavior and illegal drug use by repeated measures administered at inclusion, by 3, 6 and 12 months of treatment, and thereafter yearly (24, 36 and 48 months). Staff are asked to complete a separate questionnaire if a patient leaves treatment.

Clinical records

Information will also be obtained from the individual patient’s routine clinical records on variables such as main vital signs, nutritional status, cognitive function and mental health, medication, and comorbidities, as well as more HAT-specific variables such as adverse events, dosage, and administration routes of the pharmaceutical heroin.

Central register databases

Nordic national registers are an important and useful source for epidemiological and healthcare services research, including the study of substance use disorders [ 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 ]. The project will utilize databases from national registries in both Norway and Denmark to describe the cohorts and to monitor the changes and outcomes in a wider context. Currently, one study has explored the use of the Short Form (SF-36) Health Survey in patients enrolled in the Danish HAT database, finding support for the structural and external validity for its use in HAT [ 44 ].

Table  2 gives an overview of the relevant Norwegian and Danish register databases along with their relevant variables.

Additional studies

Currently, the only planned sub-study is on the pharmacokinetics of heroin and its metabolites, and its subjective effects on patients. Despite its widespread use, the pharmacology of heroin remains poorly understood [ 45 , 46 , 47 ]. A subsample of patients will therefore be invited to participate in this observational study with post-administration blood samples collected at different time points, with analysis of the concentration of heroin and its metabolites together with scales of subjective experience. The study has been granted separate approvals from Norwegian Regional Committee for Medical and Health Research Ethics.

Analysis strategy

Exploration and analysis of data will be both by qualitative and quantitative strategies, for individual patients and at the organizational level.

Qualitative

Treatment satisfaction of patients is particularly significant to the project and is often dependent on the context of factors such as staff, management, and clinical environment [ 48 , 49 ]. Qualitative analyses are widely considered valuable for description of phenomena and hypothesis generation, taking into consideration the natural context in which people and organizations function [ 50 ]. Transcribed interviews will be coded following the principles of a flexible inductive thematic analysis and multidimensional approach [ 51 ].

Quantitative

Given the large amount and comprehensive nature of the data, variables of interest will vary in levels of measurement and distribution, so parametric and non-parametric tests will be used accordingly.

Presentation of cohorts will include descriptive statistics by basic parameters such as mean or medians, standard deviations and ratios, and bivariate analyses by ANOVA and Chi-Square tests. Various advanced methods such as survival analysis and logistic and linear regression modeling will be applied based on the type and distribution of dependent variables and co-variates. To avoid ecological fallacy and nested dimensions, multi-level methods will be applied for analyses of patients in relation to services’ organization [ 52 ]. Given the longitudinal design and to address the repeated measurements and correlated data, linear mixed models (LMM) (random intercepts or random slope models) will be used for person-specific effects, and marginal models like Generalized Estimating Equations (GEE) for population effects.

A theoretical sample size for statistical power will not be calculated because the study is based on the total clinical population available. For analyses of discrete and possible repeated events such as the number of criminal acts or medical prescriptions, statistical power will most likely be sufficient even with a restricted number of individuals. For analyses where the proportion of patients to number of variables may imply low statistical power, stratification of the study-population and restrictions to the number of covariates in the multivariate models will be applied.

Economic evaluations

Health economics and methods of cost-effectiveness analysis can guide decision makers, but at the same time they intrinsically rely on sets of politically and administratively determined rules and contexts [ 53 ]. In general, the cost-effectiveness of a treatment is intended to reflect the difference between the recourse’s opportunity costs (medical heroin) and those of the foregone or conventional alternative, to capture a broader set of values beyond the scope of mere financial costs [ 54 ].

Initially, for operating costs a three-step, top-down methodology used and refined by a former healthcare services project will be applied, where total costs are distributed on service units and units of treatment for individual patients [ 55 ].

For cost-effectiveness analyses of healthcare interventions, outcome is often measured in quality-adjusted life years (QALYs) for individual patients, in number of accidents or fatal incidents, or as societal costs associated with patients’ level of functioning and societal (criminal) behavior [ 56 ]. This will readily apply to the project and is in line with the national Norwegian recommendations for evaluation of new health interventions [ 57 , 58 ]. The relationships between HAT and various forms of criminal behavior (both property crime and illegal drug offences), labor market attachment, income and drug expenditures are also unclear and possible subjects for investigation during the project [ 20 ].

The data for all analyses will come from key account figures and relevant variables already obtained in the project.

The main strength of the study comes from its clinical and longitudinal approaches. The use of patient-interviews combined with clinical records, self-report data and register-based information will enhance the analyses and may uncover important associations between the individual patients, treatment, and the organizational level of healthcare services. The results are therefore expected to address aspects of HAT that may contribute to the development of clinical services and individually tailored treatments for OUD.

Study limitations are mainly related to the designs’ limitation for isolation of the effects from HAT on the outcome variables. Although valuable associations often have been suggested by longitudinal ecological studies, this limited possibility of unbiased causal inference remains a major weakness of both epidemiological and cohort designs [ 59 ]. Consequently, analyses will be cautiously interpreted within the context of previous findings, as well as patient and staff experiences. The triangulation of different types of data sources and cohorts, with the use of multivariate analysis and modeling might nevertheless provide more nuanced insights than currently exist.

Also, socially desirable bias concerning self-report questionnaires may be inherent in all self-reported outcomes [ 60 ]. This will apply to the study, as patients in the Norwegian cohort are possibly aware that the prospects of HAT may depend on the results from the study.

The sample of patients in the main cohort might also not be representative of individuals with OUD who do not seek the HAT option for reasons related to the study outcomes, such as social deprivation and isolation, behavioral misconduct, and incarceration [ 61 , 62 ]. Comparison with patients not granted access to the HAT-treatment may partly address this, although not to a full extent.

Lastly, the results will emerge in the context of a Nordic cultural and political system with healthcare reimbursements, insurance models and legal aspects that may limit their generalizability to other countries and societies. Given a cautious interpretation, the project may nonetheless be considered relevant to populations where OAT is used, and a wide range of medications are potentially provided.

Results from this project have the potential to identify new insights of value to patients, healthcare personnel, service administrators and policy makers as to whether an option for pharmaceutical heroin could be implemented as a conventional part of OAT services. We believe that the results will suggest future themes for research within the field of HAT with a potential for individually tailored treatment and care for individuals with OUD. This could affect considerations about drug treatment even beyond HAT-specific topics, where an expanded understanding of why some patients do not succeed with conventional OAT or specific OAT medications will strengthen the knowledge base for drug treatment in general.

Data availability

Data sharing is not applicable to this article as no datasets are currently completed or analyzed. The data that support the eventual findings of this study are available from both national registries, individual health journals and the project-specific database, but restrictions apply to the availability which are under license for the current study. Data may be available from the authors upon reasonable request and dependent on permissions from the Norwegian Regional Committee for Medical and Health Research Ethics. All information on subjects will be stored in the University of Oslo's secure services for sensitive data (TSD). Files for analysis will not contain directly identifying information of patients. Data will be stored in a non-identifiable way for 15 years after the end of the project.

Abbreviations

Opioid agonist treatment

Opioid use disorder

• Heroin assisted treatment

Generalized Estimating Equations

Linear mixed models

Quality-Adjusted Life Years

Norwegian Regional Committee for Medical and Health Research Ethics

Norwegian Centre for Addiction Research, Oslo

Section for Clinical Addiction Research, Oslo

Bergen Addiction Research Group, Bergen

Centre for Alcohol and Drug Research, Aarhus

Norwegian User-union

United Nations Office on Drugs and Crime (UNODC). World Drug Report 2023. 2023.

Degenhardt L, Grebely J, Stone J, Hickman M, Vickerman P, Marshall BDL, et al. Global patterns of opioid use and dependence: harms to populations, interventions, and future action. Lancet. 2019;394(10208):1560–79.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Babor TF, Caulkins J, Fischer B, Foxcroft D, Medina-Mora ME, Obot I, et al. Drug Policy and the Public Good: a summary of the second edition. Addiction. 2019;114(11):1941–50.

Article   Google Scholar  

Dole VP, Nyswander M. A Medical treatment for Diacetylmorphine (Heroin) addiction. A clinical trial with Methadone Hydrochloride. JAMA. 1965;193:646–50.

Article   CAS   PubMed   Google Scholar  

Isbell H, Vogel VH. The addiction liability of methadon (amidone, dolophine, 10820) and its use in the treatment of the morphine abstinence syndrome. Am J Psychiatry. 1949;105(12):909–14.

Strang J, Volkow ND, Degenhardt L, Hickman M, Johnson K, Koob GF et al. Opioid use disorder. Nat Rev Dis Primers. 2020;6(1).

Chaillon A, Bharat C, Stone J, Jones N, Degenhardt L, Larney S, et al. Modeling the population-level impact of opioid agonist treatment on mortality among people accessing treatment between 2001 and 2020 in New South Wales, Australia. Addiction. 2022;117(5):1338–52.

Article   PubMed   Google Scholar  

Havnes I, Bukten A, Gossop M, Waal H, Stangeland P, Clausen T. Reductions in convictions for violent crime during opioid maintenance treatment: a longitudinal national cohort study. Drug Alcohol Depend. 2012;124(3):307–10.

Clausen T, Anchersen K, Waal H. Mortality prior to, during and after opioid maintenance treatment (OMT): a national prospective cross-registry study. Drug Alcohol Depend. 2008;94(1–3):151–7.

Sordo L, Barrio G, Bravo MJ, Indave BI, Degenhardt L, Wiessing L, et al. Mortality risk during and after opioid substitution treatment: systematic review and meta-analysis of cohort studies. BMJ. 2017;357:j1550.

Article   PubMed   PubMed Central   Google Scholar  

Bukten A, Skurtveit S, Waal H, Clausen T. Factors associated with dropout among patients in opioid maintenance treatment (OMT) and predictors of re-entry. A national registry-based study. Addict Behav. 2014;39(10):1504–9.

Bukten A, Roislien J, Skurtveit S, Waal H, Gossop M, Clausen T. A day-by-day investigation of changes in criminal convictions before and after entering and leaving opioid maintenance treatment: a national cohort study. BMC Psychiatry. 2013;13:262.

Klimas J, Hamilton MA, Gorfinkel L, Adam A, Cullen W, Wood E. Retention in opioid agonist treatment: a rapid review and meta-analysis comparing observational studies and randomized controlled trials. Syst Rev. 2021;10(1):216.

Timko C, Schultz NR, Cucciare MA, Vittorio L, Garrison-Diehn C. Retention in medication-assisted treatment for opiate dependence: a systematic review. J Addict Dis. 2016;35(1):22–35.

Strang J, Sheridan J. Heroin prescribing in the British system of the mid 1990s: data from the 1995 national survey of community pharmacies in England and Wales. Drug Alcohol Rev. 1997;16(1):7–16.

Berridge V. Heroin prescription and history. N Engl J Med. 2009;361(8):820–1.

van den Brink W, Hendriks VM, Blanken P, Koeter MW, van Zwieten BJ, van Ree JM. Medical prescription of heroin to treatment resistant heroin addicts: two randomised controlled trials. BMJ. 2003;327(7410):310.

Rehm J, Gschwend P, Steffen T, Gutzwiller F, Dobler-Mikola A, Uchtenhagen A. Feasibility, safety, and efficacy of injectable heroin prescription for refractory opioid addicts: a follow-up study. Lancet. 2001;358(9291):1417–23.

Strang J, Groshkova T, Uchtenhagen A, van den Brink W, Haasen C, Schechter MT, et al. Heroin on trial: systematic review and meta-analysis of randomised trials of diamorphine-prescribing as treatment for refractory heroin addictiondagger. Br J Psychiatry. 2015;207(1):5–14.

Smart R, Reuter P. Does heroin-assisted treatment reduce crime? A review of randomized-controlled trials. Addiction. 2022;117(3):518–31.

Oviedo-Joekes E, Brissette S, Marsh DC, Lauzon P, Guh D, Anis A, et al. Diacetylmorphine versus methadone for the treatment of opioid addiction. N Engl J Med. 2009;361(8):777–86.

Farrell M, Hall W. Heroin-assisted treatment: has a controversial treatment come of age? Br J Psychiatry. 2015;207(1):3–4.

Bell J, van der Waal R, Strang J. Supervised Injectable Heroin: a clinical perspective. Can J Psychiat. 2017;62(7):451–6.

Meyer M, Strasser J, Kock P, Walter M, Vogel M, Dursteler KM. Experiences with take-home dosing in heroin-assisted treatment in Switzerland during the COVID-19 pandemic-Is an update of legal restrictions warranted? Int J Drug Policy. 2022;101:103548.

Oviedo-Joekes E, Dobischok S, Carvajal J, MacDonald S, McDermid C, Klakowicz P, et al. Clients’ experiences on North America’s first take-home injectable opioid agonist treatment (iOAT) program: a qualitative study. BMC Health Serv Res. 2023;23(1):553.

Ferri M, Davoli M, Perucci CA. Heroin maintenance for chronic heroin-dependent individuals. Cochrane Database Syst Rev. 2011;2011(12):CD003410.

PubMed   PubMed Central   Google Scholar  

Dijkgraaf MG, van der Zanden BP, de Borgie CA, Blanken P, van Ree JM, van den Brink W. Cost utility analysis of co-prescribed heroin compared with methadone maintenance treatment in heroin addicts in two randomised trials. BMJ. 2005;330(7503):1297.

Byford S, Barrett B, Metrebian N, Groshkova T, Cary M, Charles V, et al. Cost-effectiveness of injectable opioid treatment v. oral methadone for chronic heroin addiction. Br J Psychiatry. 2013;203(5):341–9.

Poulter HL, Walker T, Ahmed D, Moore HJ, Riley F, Towl G, et al. More than just ‘free heroin’: caring whilst navigating constraint in the delivery of diamorphine assisted treatment. Int J Drug Policy. 2023;116:104025.

Tiffany ST, Friedman L, Greenfield SF, Hasin DS, Jackson R. Beyond drug use: a systematic consideration of other outcomes in evaluations of treatments for substance use disorders. Addiction. 2012;107(4):709–18.

Groshkova T, Metrebian N, Hallam C, Charles V, Martin A, Forzisi L, et al. Treatment expectations and satisfaction of treatment-refractory opioid-dependent patients in RIOTT, the Randomised Injectable Opiate Treatment Trial, the UK’s first supervised injectable maintenance clinics. Drug Alcohol Rev. 2013;32(6):566–73.

Sundhetsstyrelsen. Evaluering af ordningen med lægeordineret heroin til opioidafhængige patienter. Opgørelse over årene 2013–2020. København; 2021.

Waal H. Merits and problems in high-threshold methadone maintenance treatment. Evaluation of medication-assisted rehabilitation in Norway 1998–2004. Eur Addict Res. 2007;13(2):66–73.

Ellefsen R, Wusthoff LEC, Arnevik EA. Patients’ satisfaction with heroin-assisted treatment: a qualitative study. Harm Reduct J. 2023;20(1):73.

Gabrhelik R, Handal M, Mravcik V, Nechanska B, Tjagvad C, Thylstrup B et al. Opioid maintenance treatment in the Czech Republic, Norway and Denmark: a study protocol of a comparative registry linkage study. Bmj Open. 2021;11(5).

Arendt M, Munk-Jorgensen P, Sher L, Jensen SO. Mortality among individuals with cannabis, cocaine, amphetamine, MDMA, and opioid use disorders: a nationwide follow-up study of Danish substance users in treatment. Drug Alcohol Depend. 2011;114(2–3):134–9.

PubMed   Google Scholar  

Bukten A, Lokdam NT, Skjaervo I, Ugelvik T, Skurtveit S, Gabrhelik R et al. PriSUD-Nordic-diagnosing and treating Substance Use disorders in the Prison Population: protocol for a mixed methods study. Jmir Res Protoc. 2022;11(3).

Bukten A, Stavseth MR, Clasuen T. From restrictive to more liberal: variations in moratlity among patients in opioid maintenance treament over a 12-year period. BMC Health Serv Res. 2019;19(1):553.

Mortensen LH, Rehnberg J, Dahl E, Diderichsen F, Elstad JI, Martikainen P, et al. Shape of the association between income and mortality: a cohort study of Denmark, Finland, Norway and Sweden in 1995 and 2003. Bmj Open. 2016;6(12):e010974.

Munk-Jorgensen P, Ostergaard SD. Register-based studies of mental disorders. Scand J Public Health. 2011;39(7 Suppl):170–4.

Lassemo E, Myklebust LH. Changes in patterns of coercion during a nine-year period in a Norwegian psychiatric service area. Int J Methods Psychiatr Res. 2021;30(4):e1889.

Myklebust LH, Sorgaard KW, Bjorbekkmo S, Eisemann MR, Olstad R. Time-trends in the utilization of decentralized mental health services in Norway - A natural experiment: the VELO-project. Int J Ment Health Syst. 2010;4:5.

Myklebust LH, Lassemo E. The role of local inpatient psychiatric units and general practitioner on continuity of care in Northern Norway: a case-register study. Int J Methods Psychiatr Res. 2021;30(2):e1866.

Kvamme T, Thylstrup B, Hesse M. Quality of life assessment in Danish Heroin Assisted Treatment Patients: Validity of the SF-36 Survey. submitted.

De Pirro S, Galati G, Pizzamiglio L, Badiani A. The affective and neural correlates of Heroin versus Cocaine Use in Addiction are influenced by environmental setting but in Opposite directions. J Neurosci. 2018;38(22):5182–95.

De Pirro S, Lush P, Parkinson J, Duka T, Critchley HD, Badiani A. Effect of alcohol on the sense of agency in healthy humans. Addict Biol. 2020;25(4):e12796.

Milella MS, D’Ottavio G, De Pirro S, Barra M, Caprioli D, Badiani A. Heroin and its metabolites: relevance to heroin use disorder. Translational Psychiatry. 2023;13(1):120.

Kelly SM, O’Grady KE, Brown BS, Mitchell SG, Schwartz RP. The role of patient satisfaction in methadone treatment. Am J Drug Alcohol Abuse. 2010;36(3):150–4.

Barbosa CD, Balp MM, Kulich K, Germain N, Rofail D. A literature review to explore the link between treatment satisfaction and adherence, compliance, and persistence. Patient Prefer Adherence. 2012;6:39–48.

Korstjens I, Moser A, Series. Practical guidance to qualitative research. Part 2: context, research questions and designs. Eur J Gen Pract. 2017;23(1):274–9.

Braun V, Clarke V. One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qual Res Psychol. 2021;18(3):328–52.

Dyjack D. Ecological fallacies. J Environ Health. 2019;81(9):50–.

Google Scholar  

Gold MR, Russell JES, Weinstein LB. MC. Cost-Effectiveness in Health and Medicine: Oxford University Press; 1996 1996.

Turner HC, Sandmann FG, Downey LE, Orangi S, Teerawattananon Y, Vassall A, et al. What are economic costs and when should they be used in health economic studies? Cost Eff Resour Alloc. 2023;21(1):31.

Lassemo E, Myklebust L, Sorgaard K. Patient cost and treatment unit cost comparison in the VELO-Study. Psychiat Prax. 2011;38.

Yates BT. Cost-inclusive evaluation: a banquet of approaches for including costs, benefits, and cost-effectiveness and cost-benefit analyses in your next evaluation. Eval Program Plann. 2009;32(1):52–4.

The Directorate of Health. Recommendations for the economic evaluation of new interventions in the Norwegian health sector. Oslo2012.

Meld. St. 34. Principles for priority settings in health care. Summary of a white paper on priority setting in the Norwegian health care sector. In: Servces NMoHaC, editor. Oslo2017.

Jekel JF, Katz DL, Elmpre JG. Epideimology, biostatisticvcs and preventive medicine. Pennsylvania, USA: B.W.Saunders; 2001.

Latkin CA, Edwards C, Davey-Rothwell MA, Tobin KE. The relationship between social desirability bias and self-reports of health, substance use, and social network factors among urban substance users in Baltimore, Maryland. Addict Behav. 2017;73:133–6.

Hosseinbor M, Yassini Ardekani SM, Bakhshani S, Bakhshani S. Emotional and social loneliness in individuals with and without substance dependence disorder. Int J High Risk Behav Addict. 2014;3(3):e22688.

Chandler RK, Fletcher BW, Volkow ND. Treating drug abuse and addiction in the criminal justice system: improving public health and safety. JAMA. 2009;301(2):183–90.

Cho HL, Danis M, Grady C. Post-trial responsibilities beyond post-trial access. Lancet. 2018;391(10129):1478–9.

Cook K, Snyder J, Calvert J. Attitudes toward Post-trial Access to Medical interventions: a review of academic literature, legislation, and International guidelines. Dev World Bioeth. 2016;16(2):70–9.

Download references

Acknowledgements

We would like to thank the representatives from proLARNett for inputs on the design and aims of the study. Also thanks to associate professor Eva Lassemo at SINTEF-Helse, Norway for inputs on economical analysis.

The project is funded by the Norwegian Directorate of Health for the duration of 4.5 years with an annual limit of 5 million NOK (assignment No.20/00546). No remuneration is planned for the subjects’ participation in the project.

Open access funding provided by University of Oslo (incl Oslo University Hospital)

Author information

Authors and affiliations.

Norwegian Centre for Addiction Research, Faculty of Medicine, University of Oslo, Kirkevegen 166, Building 45, NO-0407, Oslo, Norway

Lars Henrik Myklebust, Desiree Eide, Silvana De Pirro, Francesca Melis, Ann Oldervoll, Linda E.C. Wusthoff & Thomas Clausen

Bergen Addiction Research Group, Department of Addiction Medicine, Haukeland University Hospital, P.b 1400, NO-5021, Bergen, Norway

Omid Dadras & Lars T. Fadnes

Department of Global Public Health and Primary Care, University of Bergen, P.b 7804, NO-5020, Bergen, Norway

Lars T. Fadnes

Section for Clinical Addiction Research, Oslo University Hospital, P.b 4950 Nydalen, NO-0424, Oslo, Norway

Espen A. Arnevik, Rune Ellefsen & Linda E.C. Wusthoff

Centre for Alcohol and Drug Research, Aarhus University, Bartholins Allé 10, DK-8000, Aarhus, Denmark

Morten Hesse, Timo L. Kvamme & Birgitte Thylstrup

Department of Physiology and Pharmacology “V. Erspamer,” La Sapienza, University of Rome, P. Ie Aldo Moro 5, 00185, Rome, Italy

Silvana De Pirro

You can also search for this author in PubMed   Google Scholar

Contributions

LHM wrote and drafted the manuscript with critical input from all the authors. The study was planned and designed by TC, DE, LTF and LECW. The statistical section had essential inputs from FM and LHM, the section on economic evaluation had substantial inputs from OD, FM and LHM. The litterature search was conducted LHM, with inputs from TC and LECW. Authors OD, SDP, RE, MH, BT, TLK, EAA and AO read the manuscript and had substantial contributions on data-aquisition and corresponding background material.

Corresponding author

Correspondence to Lars Henrik Myklebust .

Ethics declarations

Ethics approval and consent to participate.

The study was approved by the Norwegian Regional Committee for Medical and Health Research Ethics (REK 195733). Informed consent was obtained from all participants. A particular revision of the Helsinki declaration on eventual continued post-trial provisions of clinical care and treatment [ 63 , 64 ] does not apply as the project solely observes the outcomes from already provided treatment and does not initiate any research interventions. No specific insurances for subjects are taken out for the study. In case of injury or complications despite all precautions, patients have the right to apply for compensation through the Norwegian System of Patient Injury Compensation (NPE).

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Myklebust, L.H., Eide, D., Arnevik, E.A. et al. Evaluation of heroin-assisted treatment in Norway: protocol for a mixed methods study. BMC Health Serv Res 24 , 398 (2024). https://doi.org/10.1186/s12913-024-10767-w

Download citation

Received : 24 November 2023

Accepted : 21 February 2024

Published : 29 March 2024

DOI : https://doi.org/10.1186/s12913-024-10767-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Opioid maintenance treatment
• Health services
• Mixed methods
• Longitudinal

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

This paper is in the following e-collection/theme issue:

Published on 29.3.2024 in Vol 26 (2024)

Usability of Health Care Price Transparency Data in the United States: Mixed Methods Study

Authors of this article:

Original Paper

• Negar Maleki 1 , PhD   ; 
• Balaji Padmanabhan 2 , PhD   ; 
• Kaushik Dutta 1 , PhD  

1 School of Information Systems and Management, Muma College of Business, University of South Florida, Tampa, FL, United States

2 Decision, Operations & Information Technologies Department, Robert H. Smith School of Business, University of Maryland, College Park, MD, United States

Corresponding Author:

Negar Maleki, PhD

School of Information Systems and Management

Muma College of Business

University of South Florida

4202 E Fowler Avenue

Tampa, FL, 33620

United States

Phone: 1 8139742011

Email: [email protected]

Background: Increasing health care expenditure in the United States has put policy makers under enormous pressure to find ways to curtail costs. Starting January 1, 2021, hospitals operating in the United States were mandated to publish transparent, accessible pricing information online about the items and services in a consumer-friendly format within comprehensive machine-readable files on their websites.

Objective: The aims of this study are to analyze the available files on hospitals’ websites, answering the question—is price transparency (PT) information as provided usable for patients or for machines?—and to provide a solution.

Methods: We analyzed 39 main hospitals in Florida that have published machine-readable files on their website, including commercial carriers. We created an Excel (Microsoft) file that included those 39 hospitals along with the 4 most popular services—Current Procedural Terminology (CPT) 45380, 29827, and 70553 and Diagnosis-Related Group (DRG) 807—for the 4 most popular commercial carriers (Health Maintenance Organization [HMO] or Preferred Provider Organization [PPO] plans)—Aetna, Florida Blue, Cigna, and UnitedHealthcare. We conducted an A/B test using 67 MTurkers (randomly selected from US residents), investigating the level of awareness about PT legislation and the usability of available files. We also suggested format standardization, such as master field names using schema integration, to make machine-readable files consistent and usable for machines.

Results: The poor usability and inconsistent formats of the current PT information yielded no evidence of its usefulness for patients or its quality for machines. This indicates that the information does not meet the requirements for being consumer-friendly or machine readable as mandated by legislation. Based on the responses to the first part of the experiment (PT awareness), it was evident that participants need to be made aware of the PT legislation. However, they believe it is important to know the service price before receiving it. Based on the responses to the second part of the experiment (human usability of PT information), the average number of correct responses was not equal between the 2 groups, that is, the treatment group (mean 1.23, SD 1.30) found more correct answers than the control group (mean 2.76, SD 0.58; t 65 =6.46; P <.001; d =1.52).

Conclusions: Consistent machine-readable files across all health systems facilitate the development of tools for estimating customer out-of-pocket costs, aligning with the PT rule’s main objective—providing patients with valuable information and reducing health care expenditures.

Introduction

From 1970 to 2020, on a per capita basis, health care expenditures in the United States have increased sharply from US $353 per person to US $12,531 per person. In constant 2020 dollars, the increase was from US $1875 in 1970 to US $12,531 in 2020 [ 1 ]. The significant rise in health care expenses has put policy makers under enormous pressure to find ways to contain these expenditures. Price transparency (PT) in health care is 1 generally proposed strategy for addressing these problems [ 2 ] and has been debated for years [ 3 ]. Some economists believe that PT in health care will cut health care prices in the same way it has in other industries, while others argue that owing to the specific characteristics of the health care market, PT would not ameliorate rising health care costs. Price elasticity also does not typically apply in health care, since, if a problem gets severe, people will typically seek treatment regardless of cost, with the drawback that individuals learn of their health care costs after receiving treatment [ 4 ]. Complex billing processes, hidden insurer-provider contracts, the sheer quantity of third-party payers, and substantial quality differences in health care delivery are other unique aspects of health care that complicate the situation considerably.

The Centers for Medicare & Medicaid Services (CMS) mandated hospitals to post negotiated rates, including payer-specific negotiated costs, for 300 “shoppable services” beginning in January 2021. The list must include 70 CMS-specified services and an additional 230 services each hospital considers relevant to its patient population. Hospitals must include each third-party payer and their payer-specific fee when negotiating multiple rates for the same care. The data must be displayed simply, easily accessible (without requiring personal information from the patient), and saved in a machine-readable manner [ 5 ]. These efforts aim to facilitate informed patient decision-making, reduce out-of-pocket spending, and decrease health care expenditures. Former Secretary of Health and Human Services, Alex Azar, expressed a vision of hospital PT when declaring the new legislation “a patient-centered system that puts you in control and provides the affordability you need, the options and control you want, and the quality you deserve. Providing patients with clear, accessible information about the price of their care is a vital piece of delivering on that vision” [ 6 ].

Despite the legislation, it is not clear if people are actually engaging in using PT tools. For example, in 2007, New Hampshire’s HealthCost website was established, providing the negotiated price and out-of-pocket costs for 42 commonly used services by asking whether the patient is insured or their insurer and the zip code to post out-of-pocket costs in descending order. Mehrotra et al [ 7 ] examined this website over 3 years to understand how often and why these tools have mainly been used. Their analysis suggested that despite the growing interest in PT, approximately 1% of the state’s population used this tool. Low PT tool usage was also seen in other studies [ 8 - 10 ], suggesting that 3% to 12% of individuals who were offered the tool used it during the study period, and in all studies, the duration was at least 12 months. Thus, offering PT tools does not in itself lead to decreased total spending, since few people who have access to them use them to browse for lower-cost services [ 7 , 11 ].

In a recent paper, researchers addressed 1 possible reason for low engagement—lack of awareness. They implemented an extensive targeted online advertising campaign using Google Advertisements to increase awareness and assessed whether it increased New Hampshire’s PT website use. Their findings suggested that although lack of awareness is a possible reason for the low impact of PT tools in health care spending, structural factors might affect the use of health care information [ 12 ]. Individuals may not be able to exactly determine their out-of-pocket expenses from the information provided.

Surprisingly, there is little research on the awareness and usability of PT information after the current PT legislation went into effect. A recent study [ 13 ] highlighted the nonusability of existing machine-readable files for employers, policy makers, researchers, or consumers, and this paper adds to this literature by answering the question—is PT information as provided usable for patients or machines? Clearly, if it is of value to patients, it can be useful; the reason to take the perspective of machines was to examine whether this information as provided might also be useful for third-party programs that can extract information from the provided data (to subsequently help patients through other ways of presenting this information perhaps). We address this question through a combination of user experiments and data schema analysis. While there are recent papers that have also argued that PT data have deficiencies [ 13 , 14 ], ours is the first to combine user experiments with analysis of data schema from several hospitals in Florida to make a combined claim on value for patients and machines. We hope this can add to the discourse on PT and what needs to be done to extract value for patients and the health care system as a whole.

Impact of PT Tools

The impact of PT tools on consumers and health care facilities has been investigated in the literature. Some studies showed that consumers with access to PT tools are more likely to reduce forgone needed services over time. Moreover, consumers who use tools tend to find the lowest service prices [ 8 , 15 - 17 ]. A few studies investigated the impact of PT tools on the selection of health care facilities. They illustrated that some consumers tend to change health care facilities pursuing lower prices, while some others prefer to stay with expensive ones, although they are aware of some other facilities that offer lower prices [ 9 , 18 ]. Finally, some research studied the impact of PT tools on cost and showed that some consumers experienced no effect, while others experienced decreases in average consumer expenses [ 8 , 17 , 18 ]. However, the impact of PT tools on health care facilities is inconclusive, meaning different studies concluded different effects. Some stated that PT tools decrease the prices of imaging and laboratory services, while others said that although public charge disclosure lowers health care facility charges, the final prices remained unchanged [ 17 , 18 ].

Legislation Related Works

In a study, researchers considered 20 leading US hospitals to assess provided chargemasters to understand to what extent patients can obtain information from websites to determine the out-of-pocket costs [ 19 ]. Their findings showed that although all hospitals provided chargemasters on their websites, they rarely offered transparent information, making it hard for patients to determine out-of-pocket costs. Their analysis used advanced diagnostic imaging services to assess hospitals’ chargemasters since these are the most common services people look for. Mehrotra et al [ 7 ] also mentioned that the most common searches belonged to outpatient visits, magnetic resonance imaging (MRI), and emergency department visits. To this end, we used “MRI scan of the brain before and after contrast” as one of the shoppable services in our analysis. Another study examined imaging services in children’s hospitals (n=89), restricting the analysis to hospitals (n=35) that met PT requirements—published chargemaster rates, discounted cash prices, and payer-negotiated prices in a machine-readable file, and published costs for 300 common shoppable medical services in a consumer-friendly format. Their study revealed that, in addition to a broad range of imaging service charges, most hospitals lack the machine-readable file requirement [ 20 ].

Arvisais-Anhalt et al [ 21 ] identified 11 hospitals with available chargemasters in Dallas County to compare the prices of a wide range of available services. They observed significant variations for a laboratory test: partial thromboplastin time, a medication: 5 mg tablet of amlodipine, and a procedure: circumcision. Reddy et al [ 22 ] focus on New York State to assess the accessibility and usability of hospitals’ chargemasters from patients’ viewpoint. They found that 189 out of 202 hospitals had a locatable chargemaster on their home page. However, only 37 hospitals contain the Current Procedural Terminology (CPT) code, which makes those without the CPT code unusable due to the existence of many different descriptions for the same procedure; for example, an elective heart procedure had 34 entries. We add to this considerable literature by examining a subset of Florida hospitals.

In a competitive market, higher-quality goods and services require higher prices [ 23 ]. Based on this, Patel et al [ 24 ] examined the relationship between the Diagnosis-Related Group (DRG) chargemaster and quality measures. Although prior research found no convincing evidence that hospitals with greater costs also delivered better care [ 25 ], they discovered 2 important quality indicators that were linked to standard charges positively and substantially—mortality rate and readmission rates—which both are quality characteristics that are in line with economic theory. Moreover, Patel et al [ 24 ] studied the variety of one of the most commonly performed services (vaginal delivery) as a DRG code, which motivated us to select “Vaginal delivery without sterilization or D&C without CC/MCC” as another shoppable service in our analysis.

Ethical Considerations

All data used in this study, including the secondary data set obtained from hospitals’ websites and the data collected during the user experiment, underwent a thorough anonymization process. The study was conducted under protocols approved by the University of South Florida institutional review board (STUDY004145: “Effect of price transparency regulation (PTR) on the public decisions”) under HRP-502b(7) Social Behavioral Survey Consent. This approval encompassed the use of publicly available anonymized secondary data from hospitals’ websites, as well as a user experiment aimed at assessing awareness of the PT rule and the usability of hospitals’ files. No individual-specific data were collected during the experiment, which solely focused on capturing subjects’ awareness and opinions regarding the PT rule and associated files. At the onset of the experiment, participants were provided with a downloadable consent form and were allowed to withdraw their participation at any time. Survey participants were offered a US $2 reward, and their involvement was entirely anonymous.

Data Collection

According to CMS, “Starting January 1, 2021, each hospital operating in the United States will be required to provide clear, accessible pricing information online about the items and services they provide in two ways: 1- As a comprehensive machine-readable file with all items and services. 2- In a display of shoppable services in a consumer-friendly format.” As stated, files available on hospitals’ websites should be consumer-friendly, so the question of whether these files are for users arises. On the other hand, as stated, files should be machine-readable, so again the question of whether these files are for machines arises. Below we try to answer both questions in detail, respectively.

Value for Users: User Experiments

When a public announcement is disseminated, its efficacy relies on ensuring widespread awareness and facilitating practical use during times of necessity. Previous research on PT announcements has highlighted the challenges faced by patients in accurately estimating out-of-pocket expenses. However, a fundamental inquiry arises—are individuals adequately informed about the availability of tools that enable them to estimate their out-of-pocket costs for desired services? To address this, we conducted a survey to assess public awareness of PT legislation. The survey encompassed a range of yes or no and multiple-choice questions aimed at gauging participants’ familiarity with the PT rule in health care and their entitlement to obtain cost information prior to receiving a service. Additionally, we inquired about participants’ knowledge of resources for accessing pricing information and whether they were aware of the PT rule. Furthermore, we incorporated follow-up questions to ensure that the survey responses were not provided arbitrarily, thereby securing reliable and meaningful outcomes.

Moreover, considering the previously established evidence of subpar usability associated with the currently available files, we propose streamlining the existing files and developing a user-friendly and comprehensive document for conducting an A/B test. This test aims to evaluate which file better facilitates participants in accurately estimating their out-of-pocket costs. In collaboration with Florida Blue experts during biweekly meetings throughout the entire process outlined in this paper, the authors determined the optimal design for the summary table. This design, which presents prices in a more user-friendly format, enhancing overall participant comprehension, was used during the A/B testing. Participants were randomly assigned to either access the hospitals’ files or a meticulously constructed summary table, manually created in Excel, prominently displaying cost information (Please note that all files, including the hospitals’ files and our Excel file, are made available in the same format [Excel] on a cloud-based platform to eliminate any disparities in accessing the files. This ensures equitable ease of finding, downloading, and opening files, as accessing the hospitals’ files typically requires significant effort.). The experiment entailed presenting 3 distinct health-related scenarios and instructing participants to locate the price for the requested service. Subsequently, participants were asked to provide the hospital name, service price, insurer name, and insurance plan. Additionally, we sought feedback on the perceived difficulty of finding the requested service and their priority for selecting hospitals [ 26 ], followed by Likert scale questions to assess participants’ evaluation of the provided file’s efficacy in facilitating price retrieval.

The experiments were conducted to investigate the following questions: (1) Are the individuals aware of the PT legislation? and (2) Is the information provided usable for patients? To evaluate the usability of files found on websites, we selected 2 prevalent services based on existing literature and 2 other services recommended as high-demand ones by Florida Blue experts, Table 1 . Furthermore, meticulous efforts were made to ensure that both the control and treatment groups encountered identical circumstances, thus allowing for a systematic examination of the disparities solely attributable to variations in data representation.

a DRG: Diagnosis-Related Group.

b D&C: dilation and curettage.

c CC/MCC: complication or comorbidity/major complication or comorbidity.

d CPT: Current Procedural Terminology.

e MRI: magnetic resonance imaging.

Participants

A total of 67 adults (30 female individuals; mean 41.43, SD 12.39 years) were recruited on the Amazon Mechanical Turk platform, with no specific selection criteria other than being located in the United States.

We focused on 75 main hospitals (ie, the main hospital refers to distinguish a hospital from smaller clinics or specialized medical centers within the same health system) in the state of Florida. When we searched their websites for PT files (machine-readable files), only 89% (67/75) of hospitals included machine-readable files. According to the PT legislation, these files were supposed to contain information about 300 shoppable services. However, only 58% (39/67) of hospitals included information such as insurer prices in their files. Therefore, for the rest of the analysis, we only included the 39 hospitals that have the required information in their machine-readable files on their websites. We created an Excel file that included those 39 hospitals along with the 4 services—CPT 45380, 29827 and 70553 and DRG 807—mentioned in the literature ( Table 1 ) for 4 popular (suggested by Florida Blue experts) commercial carriers (Health Maintenance Organization [HMO] or Preferred Provider Organization [PPO] plans)—Aetna, Florida Blue, Cigna, and UnitedHealthcare.

Participants were recruited for the pilot and randomly assigned by the Qualtrics XM platform to answer multiple-choice questions and fill in blanks based on the given scenarios. First, participants responded to questions regarding the awareness of PT and then were divided into 2 groups randomly to answer questions regarding the usability of hospital-provided PT information. One group was assigned hospitals’ website links (control group), while the other group was given an Excel file with the same information provided in files on hospitals’ websites, but in a manner that was designed to allow easier comparison of prices across hospitals ( Multimedia Appendix 1 ). Participants were given 3 scenarios that asked them to find a procedure’s price based on their hospital and insurer selection to compare hospital-provided information with Excel. We provide some examples of hospitals’ files and our Excel file in Multimedia Appendix 1 and the survey experiment questions in Multimedia Appendix 2 .

Value for Machines: Schema Integration—Machine-Readable Files Representation

Through meticulous investigation of machine-readable files from 39 hospitals, we discovered that these files may vary in formats such as CSV or JSON, posing a challenge for machines to effectively manage the data within these files. Another significant obstacle arises from the lack of uniformity in data representation across these files, rendering them unsuitable for machine use without a cohesive system capable of processing them collectively. Our analysis revealed that hospitals within a single health system exhibit consistent data representation, although service prices may differ (we include both the same and different chargemaster prices in our study), while substantial disparities in data representation exist between hospitals affiliated with different health systems.

Moving forward, we will use the terms “data representation” and “schema” interchangeably, with “schema” denoting its database management context. In this context, a schema serves as a blueprint outlining the structure, organization, and relationships of data within a database system. It encompasses key details such as tables, fields, data types, and constraints that define the stored data. To systematically illustrate schema differences among hospitals associated with different health systems, we adopted the methodology outlined in reference [ 27 ] for schema integration, which offers a valid approach for comparing distinct data representations. The concept of schema integration encompasses four common categories: (1) identical: hospitals within the same health system adhere to this concept as their representations are identical; (2) equivalent: while hospitals in health system “A” may present different representations from those in health system “B,” they possess interchangeable columns; (3) compatible: in cases where hospitals across different health systems are neither identical nor equivalent, the modeling constructs, designer perception, and integrity constraints do not contradict one another; and (4) incompatible: in situations where hospitals within different health systems demonstrate contradictory representations, distinct columns exist for each health system due to specification incoherence.

Our analysis focused on health systems in Florida that encompassed a minimum of 4 main hospitals, using the most up-to-date data available on their respective websites. Within this scope, we identified 8 health systems with at least 4 main hospitals, of which 88% (7/8) of health systems had published machine-readable files on their websites. Consequently, our analysis included 65% (36/55) of hospitals that possessed machine-readable files available on their websites. To facilitate further investigation by interested researchers, we have made the analyzed data accessible on a cloud-based platform. During our analysis, we meticulously extracted the schema of each health system by closely scrutinizing the hospitals associated with each health system, capturing key details such as tables, fields, and data types. Subsequently, we compiled a comprehensive master field name table trying to have the same data type and field names that make it easier for machines to retrieve information. We elaborate on the master field names table in greater detail within the results section.

Value for Users

Question 1 (pt awareness).

Based on the responses, it is evident that participants need to be made aware of the PT legislation. Among the participants, 64% (49/76) reported that they had not heard about the legislation. However, they believe it is important to know the service price before receiving it—response charts are provided in Multimedia Appendix 3 .

Question 2 (Human Usability of PT Information)

Based on the responses to scenarios, the average number of correct responses is not equal between the 2 groups, that is, the treatment group (mean 1.23, SD 1.30) found more correct answers than the control group (mean 2.76, SD 0.58; t 65 =6.46; P <.049; d =1.52). The t tests (2-tailed) for the other questions in the experiment are in Multimedia Appendix 4 .

These suggest that current files on hospitals’ websites are not consumer-friendly, and participants find it challenging to estimate out-of-pocket costs for a desired service. For this reason, in addition to making the files easier to use, this information should also include thorough documentation that explains what each column represents, up to what amount an insurer covers for a specific service, or the stated price covers up to how many days of a particular service, that is, “contracting method.” For example, based on consulting with one of the senior network analysts of Florida Blue, some prices for a service like DRG 807 are presented as per diem costs, and based on the current information on these files, it cannot be recognizable without having comprehensive documentation for them.

Value for Machines

After carefully reviewing all machine-readable file schemas, we create a master field name table, including the available field names in machine-readable files ( Table 2 ). According to Table 2 , the first column represents master field names that we came up with, and the following columns each represent hospitals within a health system. The “✓” mark shows that hospitals within a health system have identical field names as we consider as master field names and the “written” cells show equivalent field names, meaning that hospitals within that health system use different field names—we write what they use in their representation—while the content is equivalent to what we select as the master field name. The “❋” mark means that although hospitals within health system #2 provide insurer names and plans in their field names, some codes make those columns unusable for machines to recognize them the same as master field names. We also include the type of field names for all representations in parentheses.

a As noted previously, since we focus on the health system level instead of the hospital level, our schema does not have hospital-level information; however, it would be beneficial to add hospital information to the table.

b ✓: it means the given master field name in that row appears on the given health system file in that column.

c str: shows “string” as the data type.

d int: shows “integer” as the data type.

e CPT: Current Procedural Terminology.

f HCPCS: Health care Common Procedure Coding System.

g Not applicable.

h Apr: all patients refined.

i DRG: Diagnosis-Related Group.

j Ms: Medicare severity.

k CDM: charge description master.

l UB: uniform billing.

m float: it shows “float” as the data type.

n ❋: it means that although hospitals within health system #2 provide insurer names and plans in their field names, some codes make those columns unusable for machines to recognize them the same as master field names.

We did reverse engineering and drew entity-relationship diagrams (ERDs) for each hospital based on their data representation. However, as hospitals within the same health system have the same ERDs, we only include 1 ERD for each health system ( Figure 1 ). According to Figure 1 , although hospitals have tried to follow an intuitive structure, we can still separate them into three groups: (1) group I: all hospitals within this group have several columns for different insurers. As shown in the ERDs, we decided to have a separate entity, called “Insurance” for this group; (2) group II: all hospitals within this group have many sheets, and each sheet belongs to a specific insurer with a specific plan. As shown in the ERDs, we decided to create an “Insurance_Name” entity for this group’s ERD to show the difference in data representation; and (3) group III: all hospitals within this system have a “payer” column which includes the names of insurers without their plans. As shown in the ERDs, we decided to put this column as an attribute in the “Service” entity, and do not have an “Insurance” entity for this group’s ERD.

In conclusion, although most hospitals have adopted group I logic for data representation, for full similarity, a standard representation with the same intuitive field names (like what we suggest as the master field name; Table 2 ) should be proposed so that it can cover all systems’ data representations and be used as machine-readable file, for at least machine benefits. Mainly, standardization in the format and semantics of the provided data can help substantially in making the data more machine friendly.

Comparison With New CMS Guidelines

Recently, CMS has published guidelines regarding the PT legislation [ 28 ]. The most recent CMS guideline is a step forward in ensuring standardization but is still only recommended and is not mandatory. These guidelines exhibit overlaps with our fields in Table 2 , with slight differences attributed to granularities. Our observation reveals that hospitals within the same health system adopt a uniform schema. Therefore, our suggested schema operates on the granularity of health systems rather than individual hospitals.

The recent CMS guidelines allocate 24% (6/25) of field names specifically to hospital information, encompassing details such as “Hospital Name,” “Hospital File Date,” “Version,” “Hospital Location,” “Hospital Financial Aid Policy,” and “Hospital Licensure Information.” These details, absent in current hospital files, are crucial for informed decision-making. As noted previously, since we focus on the health system level instead of the hospital level, our schema does not have hospital-level information; however, it would be beneficial to add hospital information to the tables.

Our analysis reveals that the 11 field names in Table 2 align with the field names in the new CMS guidelines, demonstrating a substantial overlap of 58% (11/19). The corresponding CMS field names (compatible with our schema) include “Item or Service Description (Description or CDM Service Description),” “Code (Code),” “Code Type (Code Type),” “Setting (Patient Class),” “Gross Charge (Gross Charge),” “Discounted Cash Price (Discounted Cash Price),” “Payer Name (Insurer Name),” “Plan Name (Insurer Plan),” “Payer Specific Negotiated Charge: Dollar Amount (Price),” “De-identified Minimum Negotiated Charge (Min Negotiated Rate),” and “De-identified Maximum Negotiated Charge (Max Negotiated Rate).” Additionally, both our schema and the new CMS guidelines propose data types for each field name.

In our schema, which represents current hospitals’ files, there are 5 field names absent in the new CMS guidelines “Revenue Description,” “Revenue Code,” “Package/Line Level,” “Procedure ID,” and “Self Pay.” Conversely, the new CMS guidelines introduce 8 additional field names “Billing Class,” “Drug Unit of Measurement,” “Drug Type of Measurement,” “Modifiers,” “Payer Specific Negotiated Charge: Percentage,” “Contracting Method,” “Additional Generic Notes,” and “Additional Payer-Specific Notes.” We regard these new field names as providing further detailed information and enhancing consumer decision-making. If hospitals within a health system adopt consistent formats and can map their formats to the new CMS guidelines clearly in a mapping document they also provide, this can be more useful than the current optional guideline that is suggested.

In summary, since our analysis is based on the current data schema that hospitals have in place, we believe the schema we put out is easier to implement with minimal change to what the hospitals are currently doing. However, given the recent CMS guidelines, we recommend adding 8 additional fields as well as hospital-specific information.

Implications

The PT legislation aims to enable informed decision-making, reduce out-of-pocket expenses, and decrease overall health care expenditures. This study investigates the usage of current files by individuals and machines. Our results, unfortunately, suggest that PT data—as currently reported—appear to be neither useful for patients nor machines, raising important questions as to what these appear to be achieving today. Moreover, the findings indicate that even individuals with basic computer knowledge struggle with the usability of these files, highlighting the need for significant revisions to make them consumer-friendly and accessible to individuals of all technical proficiency levels. Additionally, inconsistencies in data representation between hospitals affiliated with different health systems pose challenges for machines, necessitating schema design improvements and the implementation of a standardized data representation. By addressing these concerns, PT legislation can achieve consistency and enhance machine readability, thus improving its effectiveness in promoting informed decision-making and reducing health care costs.

Although the official announcement of PT legislation is recent, prior studies [ 15 - 17 ] have attempted to evaluate the usability of PT, while subsequent studies [ 19 - 22 ] have examined the effectiveness of PT tools following the announcement. However, despite the introduction of PT rules, it appears that the usability of these files has not undergone significant improvements, indicating the necessity for proactive measures from responsible executives to ensure the effectiveness of this legislation. Our analysis of this matter emphasizes 2 primary factors—a lack of awareness among stakeholders and the challenges associated with using files due to inconsistencies in their format and representation.

As of April 2023, the CMS has issued over 730 warning notices and 269 requests for Corrective Action Plans. A total of 4 hospitals have faced Civil Monetary Penalties for noncompliance, and these penalties are publicly disclosed on the CMS website. The remaining hospitals subjected to comprehensive compliance reviews have either rectified their deficiencies or are actively engaged in doing so. While we acknowledge these efforts to comply with PT rules, our research revealed a notable disparity in data representation among hospitals affiliated with different health systems. Consequently, we focused on schema design and proposed the implementation of a master field name that encompasses a comprehensive data representation derived from an analysis of 36 hospitals. Standardizing the data representation across all health systems’ machine-readable files will effectively address concerns about consistency. Therefore, significant modifications are required for the PT legislation to enhance machine readability and provide clearer guidance on the design and structure of the files’ schema. If the hospital-provided information is consistent and of high quality, PT tools provided by health insurers may be able to estimate an individual’s total expenses more accurately.

Limitations

Our objective was to have an equal number in both groups. However, in the case of the group tasked with obtaining information from the hospitals’ websites, most did not finish the task and dropped out without completing it. This occurred because the task of retrieving the cost from the hospitals’ websites in its current form is complex, as indicated by feedback from some participants. Only 19% (13/67) completed the task in that group (control group). Although this is a limitation of the study, it also highlights the complexity of obtaining cost information from hospitals’ websites in the current form. In the treatment group, 81% (54 out of 67) of participants completed the task of retrieving the data, and the completion percentage was much higher.

Conclusions

Due to the poor usability and inconsistency of the formats, we, unfortunately, did not find evidence that the PT rule as implemented currently is useful to consumers, researchers, or policy makers (despite the legislation’s goals that files are “consumer-friendly” and “machine-readable”). As 1 solution, we suggest a master field name for the data representation of machine-readable files to make them consistent, at least for the machines. Building tools that enable customers to estimate out-of-pocket costs is facilitated by having consistent machine-readable files across all health systems, which can be considered as future work for researchers and companies to help the PT rule reach its main goal, which is providing useful information for patients and reducing health care expenditures. In addition, another worthwhile approach to reducing some of the exorbitant health care costs in the United States would be to integrate clinical decision support tools into the providers’ workflow, triggered by orders for medications, diagnostic testing, and other billable services. In this regard, Bouayad et al [ 29 ] conducted experiments with physicians to demonstrate that PT, when included as part of the system they interact with, such as clinical decision support integrated into electronic health record systems, can significantly aid in cost reduction. This is a promising direction for practice but needs to be implemented carefully to avoid unanticipated consequences, such as scenarios where cost is incorrectly viewed as a proxy for quality, or where the use of this information introduces new biases for physicians and patients.

Conflicts of Interest

None declared.

Example of Excel format of hospitals’ files and our created Excel file.

Survey questions and experiment scenarios.

Participants’ responses chart regarding price transparency awareness.

The t test analysis regarding human usability of price transparency information based on participants’ responses.

• McGough M, Winger A, Rakshit S, Amin K. How has U.S. spending on healthcare changed over time? Health System Tracker. 2022. URL: https://www.healthsystemtracker.org/chart-collection/u-s-spending-healthcare-changed-time/ [accessed 2024-03-13]
• Christensen HB, Floyd E, Maffett M. The only prescription is transparency: the effect of charge-price-transparency regulation on healthcare prices. Manag Sci. 2020;66(7):2861-2882. [ CrossRef ]
• Muir MA, Alessi SA, King JS. Clarifying costs: can increased price transparency reduce healthcare spending? UC Hastings Research Paper No. 38 (SSRN). Feb 26, 2013.:319-367. [ FREE Full text ] [ CrossRef ]
• Reinhardt UE. Health care price transparency and economic theory. JAMA. 2014;312(16):1642-1643. [ CrossRef ] [ Medline ]
• CY 2020 hospital Outpatient Prospective Payment System (OPPS) policy changes: hospital price transparency requirements (CMS-1717-F2). CMS.gov. 2020. URL: https://tinyurl.com/mrafxtvd [accessed 2024-03-13]
• Secretary Azar statement on proposed rule for hospital price transparency. HHS.gov. 2020. URL: https://tinyurl.com/yc4dx2vx [accessed 2024-03-13]
• Mehrotra A, Brannen T, Sinaiko AD. Use patterns of a state health care price transparency web site: what do patients shop for? Inquiry. 2014;51:0046958014561496. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Desai S, Hatfield LA, Hicks AL, Sinaiko AD, Chernew ME, Cowling D, et al. Offering a price transparency tool did not reduce overall spending among California public employees and retirees. Health Aff (Millwood). 2017;36(8):1401-1407. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sinaiko AD, Joynt Maddox KE, Rosenthal MB. Association between viewing health care price information and choice of health care facility. JAMA Intern Med. 2016;176(12):1868-1870. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Desai S, Hatfield LA, Hicks AL, Chernew ME, Mehrotra A. Association between availability of a price transparency tool and outpatient spending. JAMA. 2016;315(17):1874-1881. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sinaiko AD, Rosenthal MB. Examining a health care price transparency tool: who uses it, and how they shop for care. Health Aff (Millwood). 2016;35(4):662-670. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Desai SM, Shambhu S, Mehrotra A. Online advertising increased New Hampshire residents' use of provider price tool but not use of lower-price providers. Health Aff (Millwood). 2021;40(3):521-528. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kona M, Corlette S. Hospital and insurer price transparency rules now in effect but compliance is still far away. Health Affairs Forefront. 2022. URL: hhttps://tinyurl.com/3x6ymxf2 [accessed 2024-03-13]
• Wheeler C, Taylor R. New year, new CMS price transparency rule for hospitals. Health Affairs Forefront. 2021. URL: https://www.healthaffairs.org/content/forefront/new-year-new-cms-price-transparency-rule-hospitals [accessed 2024-03-13]
• Chernew M, Cooper Z, Larsen-Hallock E, Morton FS. Are health care services shoppable? Evidence from the consumption of lower-limb MRI scans. National Bureau of Economic Research. 2021. URL: https://www.nber.org/papers/w24869 [accessed 2024-03-13]
• Mehrotra A, Dean KM, Sinaiko AD, Sood N. Americans support price shopping for health care, but few actually seek out price information. Health Aff (Millwood). 2017;36(8):1392-1400. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brown ZY. Equilibrium effects of health care price information. Rev Econ Stat. 2019;101(4):699-712. [ FREE Full text ] [ CrossRef ]
• Wu SJ, Sylwestrzak G, Shah C, DeVries A. Price transparency for MRIs increased use of less costly providers and triggered provider competition. Health Aff (Millwood). 2014;33(8):1391-1398. [ CrossRef ] [ Medline ]
• Glover M, Whorms DS, Singh R, Almeida RR, Prabhakar AM, Saini S, et al. A radiology-focused analysis of transparency and usability of top U.S. hospitals' chargemasters. Acad Radiol. 2020;27(11):1603-1607. [ CrossRef ] [ Medline ]
• Hayatghaibi SE, Alves VV, Ayyala RS, Dillman JR, Trout AT. Transparency and variability in pricing for pediatric outpatient imaging in US children's hospitals. JAMA Netw Open. 2022;5(3):e220736. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Arvisais-Anhalt S, McDonald S, Park JY, Kapinos K, Lehmann CU, Basit M. Survey of hospital chargemaster transparency. Appl Clin Inform. 2021;12(2):391-398. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Reddy S, Daly G, Baban S, Kadesh A, Block AE, Grimes CL. Accessibility and usability of hospital chargemasters in New York state. J Gen Intern Med. 2022;37(8):2130-2131. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Robinson JC. Hospital quality competition and the economics of imperfect information. Milbank Q. 1988;66(3):465-481. [ Medline ]
• Patel KN, Mazurenko O, Ford E. Analysis of hospital quality measures and web-based chargemasters, 2019: cross-sectional study. JMIR Form Res. 2021;5(8):e26887. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Batty M, Ippolito B. Mystery of the chargemaster: examining the role of hospital list prices in what patients actually pay. Health Aff (Millwood). 2017;36(4):689-696. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Muhlestein DB, Wilks CEA, Richter JP. Limited use of price and quality advertising among American hospitals. J Med Internet Res. 2013;15(8):e185. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Batini C, Lenzerini M, Navathe SB. A comparative analysis of methodologies for database schema integration. ACM Comput Surv. 1986;18(4):323-364. [ FREE Full text ] [ CrossRef ]
• Voluntary hospital price transparency machine-readable file sample format data dictionary (version 1.1). CMS.gov. URL: https:/​/www.​cms.gov/​files/​document/​hospital-price-transparency-machine-readable-data-dictionary-tall.​pdf [accessed 2024-03-13]
• Bouayad L, Padmanabhan B, Chari K. Can recommender systems reduce healthcare costs? the role of time pressure and cost transparency in prescription choice. MIS Q. 2020;44(4):1859-1903. [ CrossRef ]

Abbreviations

Edited by S He; submitted 07.07.23; peer-reviewed by KN Patel, R Marshall, G Deckard; comments to author 03.12.23; revised version received 21.01.24; accepted 26.02.24; published 29.03.24.

©Negar Maleki, Balaji Padmanabhan, Kaushik Dutta. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 29.03.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Elsevier - PMC COVID-19 Collection

Combining qualitative and quantitative research within mixed method research designs: A methodological review

Ulrika Östlund.

a Division of Nursing, Department of Neurobiology, Care Sciences and Society, Karolinska Institutet, Stockholm, Sweden

b Institute for Applied Health Research/School of Health, Glasgow Caledonian University, United Kingdom

Yvonne Wengström

c Division of Nursing, Department or Neurobiology, Care Sciences and Society, Karolinska Institutet, Stockholm, Sweden

Neneh Rowa-Dewar

d Public Health Sciences, University of Edinburgh, United Kingdom

It has been argued that mixed methods research can be useful in nursing and health science because of the complexity of the phenomena studied. However, the integration of qualitative and quantitative approaches continues to be one of much debate and there is a need for a rigorous framework for designing and interpreting mixed methods research. This paper explores the analytical approaches (i.e. parallel, concurrent or sequential) used in mixed methods studies within healthcare and exemplifies the use of triangulation as a methodological metaphor for drawing inferences from qualitative and quantitative findings originating from such analyses.

This review of the literature used systematic principles in searching CINAHL, Medline and PsycINFO for healthcare research studies which employed a mixed methods approach and were published in the English language between January 1999 and September 2009.

In total, 168 studies were included in the results. Most studies originated in the United States of America (USA), the United Kingdom (UK) and Canada. The analytic approach most widely used was parallel data analysis. A number of studies used sequential data analysis; far fewer studies employed concurrent data analysis. Very few of these studies clearly articulated the purpose for using a mixed methods design. The use of the methodological metaphor of triangulation on convergent, complementary, and divergent results from mixed methods studies is exemplified and an example of developing theory from such data is provided.

A trend for conducting parallel data analysis on quantitative and qualitative data in mixed methods healthcare research has been identified in the studies included in this review. Using triangulation as a methodological metaphor can facilitate the integration of qualitative and quantitative findings, help researchers to clarify their theoretical propositions and the basis of their results. This can offer a better understanding of the links between theory and empirical findings, challenge theoretical assumptions and develop new theory.

What is already known about the topic?

• • Mixed methods research, where quantitative and qualitative methods are combined, is increasingly recognized as valuable, because it can potentially capitalize on the respective strengths of quantitative and qualitative approaches.
• • There is a lack of pragmatic guidance in the research literature as how to combine qualitative and quantitative approaches and how to integrate qualitative and quantitative findings.
• • Analytical approaches used in mixed-methods studies differ on the basis of the sequence in which the components occur and the emphasis given to each, e.g. parallel, sequential or concurrent.

What this paper adds

• • A trend for conducting parallel analysis on quantitative and qualitative data in healthcare research is apparent within the literature.
• • Using triangulation as a methodological metaphor can facilitate the integration of qualitative and quantitative findings and help researchers to clearly present both their theoretical propositions and the basis of their results.
• • Using triangulation as a methodological metaphor may also support a better understanding of the links between theory and empirical findings, challenge theoretical assumptions and aid the development of new theory.

1. Introduction

Mixed methods research has been widely used within healthcare research for a variety of reasons. The integration of qualitative and quantitative approaches is an interesting issue and continues to be one of much debate ( Bryman, 2004 , Morgan, 2007 , Onwuegbuzie and Leech, 2005 ). In particular, the different epistemological and ontological assumptions and paradigms associated with qualitative and quantitative research have had a major influence on discussions on whether the integration of the two is feasible, let alone desirable ( Morgan, 2007 , Sale et al., 2002 ). Proponents of mixed methods research suggest that the purist view, that quantitative and qualitative approaches cannot be merged, poses a threat to the advancement of science ( Onwuegbuzie and Leech, 2005 ) and that while epistemological and ontological commitments may be associated with certain research methods, the connections are not necessary deterministic ( Bryman, 2004 ). Mixed methods research can be viewed as an approach which draws upon the strengths and perspectives of each method, recognising the existence and importance of the physical, natural world as well as the importance of reality and influence of human experience ( Johnson and Onquegbuzie, 2004 ). Rather than continue these debates in this paper, we aim to explore the approaches used to integrate qualitative and quantitative data within healthcare research to date. Accordingly, this paper focuses on the practical issues of conducting mixed methods studies and the need to develop a rigorous framework for designing and interpreting mixed methods studies to advance the field. In this paper, we will attempt to offer some guidance for those interested in mixed methods research on ways to combine qualitative and quantitative methods.

The concept of mixing methods was first introduced by Jick (1979) , as a means for seeking convergence across qualitative and quantitative methods within social science research ( Creswell, 2003 ). It has been argued that mixed methods research can be particularly useful in healthcare research as only a broader range of perspectives can do justice to the complexity of the phenomena studied ( Clarke and Yaros, 1988 , Foss and Ellefsen, 2002 , Steckler et al., 1992 ). By combining qualitative and quantitative findings, an overall or negotiated account of the findings can be forged, not possible by using a singular approach ( Bryman, 2007 ). Mixed methods can also help to highlight the similarities and differences between particular aspects of a phenomenon ( Bernardi et al., 2007 ). Interest in, and expansion of, the use of mixed methods designs have most recently been fuelled by pragmatic issues: the increasing demand for cost effective research and the move away from theoretically driven research to research which meets policymakers’ and practitioners’ needs and the growing competition for research funding ( Brannen, 2009 , O’Cathain et al., 2007 ).

Tashakkori and Creswell (2007) broadly define mixed methods research as “research in which the investigator collects and analyses data, integrates the findings and draws inferences using both qualitative and quantitative approaches” (2007:3). In any mixed methods study, the purpose of mixing qualitative and quantitative methods should be clear in order to determine how the analytic techniques relate to one another and how, if at all, the findings should be integrated ( O’Cathain et al., 2008 , Onwuegbuzie and Teddlie, 2003 ). It has been argued that a characteristic of truly mixed methods studies are those which involve integration of the qualitative and quantitative findings at some stage of the research process, be that during data collection, analysis or at the interpretative stage of the research ( Kroll and Neri, 2009 ). An example of this is found in mixed methods studies which use a concurrent data analysis approach, in which each data set is integrated during the analytic stage to provide a complete picture developed from both data sets after data has been qualitised or quantitised (i.e. where both forms of data have been converted into either qualitative or quantitative data so that it can be easily merged) ( Onwuegbuzie and Teddlie, 2003 ). Other analytic approaches have been identified including; parallel data analysis, in which collection and analysis of both data sets is carried out separately and the findings are not compared or consolidated until the interpretation stage, and finally sequential data analysis, in which data are analysed in a particular sequence with the purpose of informing, rather than being integrated with, the use of, or findings from, the other method ( Onwuegbuzie and Teddlie, 2003 ). An example of sequential data analysis might be where quantitative findings are intended to lead to theoretical sampling in an in depth qualitative investigation or where qualitative data is used to generate items for the development of quantitative measures.

When qualitative and quantitative methods are mixed in a single study, one method is usually given priority over the other. In such cases, the aim of the study, the rationale for employing mixed methods, and the weighting of each method determine whether, and how, the empirical findings will be integrated. This is less challenging in sequential mixed methods studies where one approach clearly informs the other, however, guidance on combining qualitative and quantitative data of equal weight, for example, in concurrent mixed methods studies, is rather less clear ( Foss and Ellefsen, 2002 ). This is made all the more challenging by a common flaw which is to insufficiently and inexplicitly identify the relationships between the epistemological and methodological concepts in a particular study and the theoretical propositions about the nature of the phenomena under investigation ( Kelle, 2001 ).

One approach to combining different data of equal weight and which facilitate clear identification of the links between the different levels of theory, epistemology, and methodology could be to frame triangulation as a ‘methodological metaphor’, as argued by Erzberger and Kelle (2003) . This can help to; describe the logical relations between the qualitative and quantitative findings and the theoretical concepts in a study; demonstrate the way in which both qualitative and quantitative data can be combined to facilitate an improved understanding of particular phenomena; and, can also be used to help generate new theory ( Erzberger and Kelle, 2003 ) (see Fig. 1 ). The points of the triangle represent theoretical propositions and empirical findings from qualitative and quantitative data while the sides of the triangle represent the logical relationships between these propositions and findings. The nature and use of the triangle depends upon the outcome from the analysis, whether that be convergent , where qualitative and quantitative findings lead to the same conclusion; complementary, where qualitative and quantitative results can be used to supplement each other or; divergent , where the combination of qualitative and quantitative results provides different (and at times contradictory) findings. Each of these outcomes requires a different way of using the triangulation metaphor to link theoretical propositions to empirical findings ( Erzberger and Kelle, 2003 ).

Illustrating the triangulation triangle ( Erzberger and Kelle, 2003 )

1.1. Purpose of this paper

In the following paper, we identify the analytical approaches used in mixed methods healthcare research and exemplify the use of triangulation ( Erzberger and Kelle, 2003 ) as a methodological metaphor for drawing inferences from qualitative and quantitative findings. Papers reporting on mixed methods studies within healthcare research were reviewed to (i) determine the type of analysis approach used, i.e. parallel, concurrent, or sequential data analysis and, (ii) identify studies which could be used to illustrate the use of the methodological metaphor of triangulation suggested by Erzberger and Kelle (2003) . Four papers were selected to illustrate the application of the triangulation metaphor on complementary, convergent and divergent outcomes and to develop theory.

This literature review has used systematic principles ( Cochrane, 2009 , Khan, 2001 ) to search for mixed methods studies within healthcare research. The first search was conducted in September 2009 in the data bases CINAHL, Medline and PsycINFO on papers published in English language between 1999 and 2009. To identify mixed methods studies, the search terms (used as keywords and where possible as MeSH terms) were: “mixed methods”, “mixed research methods”, “mixed research”, “triangulation”, “complementary methods”, “concurrent mixed analysis” and “multi-strategy research.” These terms were searched individually and then combined (with OR). This resulted in 1896 hits in CINAHL, 1177 in Medline and 1943 in PsycINFO.

To focus on studies within, or relevant to, a healthcare context the following search terms were used (as keywords or as MeSH terms and combined with OR): “health care research”; “health services research”; and “health”. These limits applied to the initial search (terms combined with AND) resulted in 205 hits in Medline and 100 hits in PsycINFO. Since this combination in CINAHL only limited the search results to 1017; a similar search was conducted but without using the search term triangulation to capture mixed methods papers; resulting in 237 hits. In CINAHL the search result on 1017 papers was further limited by using “interventions” as a keyword resulting in 160 papers also selected to be reviewed. Moreover; in Medline the mixed methods data set was limited by the MeSH term “research” resulting in 218 hits and in PsycINFO with “intervention” as keyword or MeSH term resulting in 178 hits.

When duplicates were removed the total numbers of papers identified were 843. The abstracts were then reviewed by each author and those identified as relevant to the review were selected to be retrieved and reviewed in full text. Papers were selected based on the following inclusion criteria: empirical studies; published in peer review journals; healthcare research (for the purpose of this paper defined as any study focussing on participants in receipt, or involved in the delivery, of healthcare or a study conducted within a healthcare setting, e.g. different kinds of care, health economics, decision making, and professionals’ role development); and using mixed methods (defined as a study in which both qualitative and quantitative data were collected and analysed ( Halcomb et al., 2009b ). To maintain rigour, a random sample (10%) of the full text papers was reviewed jointly by two authors. Any disagreements or uncertainties that arose between the reviewers regarding their inclusion or in determining the type of analytic approach used were resolved through discussion between the authors.

In addition to the criteria outlined above, papers were excluded if the qualitative element constituted a few open-ended questions in a questionnaire, as we would agree with previous authors who have argued such studies do not strictly constitute a mixed methods design ( Kroll and Neri, 2009 ). Papers were also excluded if they could not be retrieved in full text via the library services at the University of Edinburgh, Glasgow Caledonian University or the Karolinska Institutet, or did not adequately or clearly describe their analytic strategy, for example, failing to report how the qualitative and quantitative data sets were analysed individually and, where relevant, how these were integrated. See Table 1 for reasons for the exclusion of subsequent papers.

Reasons for exclusion.

A second search was conducted within the databases of Medline, PsychInfo and Cinahl to identify studies which have specifically used Erzberger and Kelle's (2003) triangulation metaphor to frame the description and interpretation of their findings. The term ‘triangulation metaphor’ (as keywords) and author searches on ‘Christian Erzberger’ and ‘Udo Kelle’ were conducted. Three papers, published by Christian Erzberger and Udo Kelle, were identified in the PsychInfo databases but none of these were relevant to the purpose of this review. There were no other relevant papers identified in the other two databases.

168 Papers were included in the final review and reviewed to determine the type of mixed analysis approach used, i.e. parallel, concurrent, or sequential mixed analysis. Four of these papers (identified from the first search on mixed methods studies and healthcare research) were also used to exemplify the use of the methodological metaphor of triangulation ( Erzberger and Kelle, 2003 ). Data was extracted from included papers accordingly in relation to these purposes.

In total, 168 papers were included in our review. The majority of these studies originated in the USA ( n  = 63), the UK ( n  = 39) and Canada ( n  = 19), perhaps reflecting the considerable interest and expertise in mixed methods research within these countries. The focus of the studies included in the review varied significantly and the populations studied included both patients and healthcare professionals.

3.1. Analytic approaches

Table 2 illustrates the types of analytic approaches adopted in each of the studies included in the review. The most widely used analytic approach ( n  = 98) was parallel analysis ( Creswell and Plano Clark, 2007 ). However, very few of the studies employing parallel analysis clearly articulate their purpose for mixing qualitative and quantitative data, the weighting (or priority) given to the qualitative and quantitative data or the expected outcomes from doing so, mirroring previous research findings ( O’Cathain et al., 2008 ). The weighting, or priority, of the qualitative and quantitative data in a mixed methods study is dependent upon various factors including; the aims of the study and whether the purpose is, for example, to contextualise quantitative data using qualitative data or to use qualitative data to inform a larger quantitative approach such as a survey. Nonetheless, the omission of this statement makes it difficult to determine which data set the conclusions have been drawn from and the role of, or emphasis on, each approach. Therefore, is of importance for authors to clearly state this in their papers ( Creswell and Plano Clark, 2007 ). A number of studies had also used sequential data analysis ( n  = 46), where qualitative approaches were visibly used to inform the development of both clinical tools (e.g. Canales and Rakowski, 2006 ) and research measures and surveys (e.g. Beatty et al., 2004 ) or where quantitative surveys were supplemented by and issues further explored using qualitative approaches (e.g. Abadia and Oviedo, 2009 , Cheng, 2004 , Halcomb et al., 2008 ).

Included papers illustrating their analytical approach and country of origin.

Most notably, with only 20 included studies using a concurrent approach to data analysis, this was the least common design employed. Compared to the studies using a parallel or sequential approach, the authors of concurrent studies more commonly provided an explanation for their purpose of using a mixed methods design in their study, e.g. how it addressed a gap or would facilitate and advance the state of knowledge (e.g. Bussing et al., 2005 , Kartalova-O’Doherty and Tedstone Doherty, 2009 ). Despite this, there remained a lack of clarity within these studies about the weighting given to, and priority of, each method. Consequently, the importance and relevance of the findings produced by each approach and how these have informed their conclusions and interpretation is lacking. In four of the included papers a combination of approaches to data analysis (i.e. sequential and concurrent, parallel and concurrent, or sequential and parallel) were used. In the next section, we have selected papers to illustrate the methodological metaphor of triangulation ( Erzberger and Kelle, 2003 ).

3.2. Using the methodological metaphor of triangulation

We have selected four papers from our review ( Lukkarinen, 2005 , Midtgaard et al., 2006 , Shipman et al., 2008 , Skilbeck et al., 2005 ) to illustrate how the methodological metaphor of triangulation ( Erzberger and Kelle, 2003 ) can be applied to mixed methods studies. Each of these studies has been used to illustrate how the metaphor of triangulation can be applied to studies producing: (i) complementary findings, (ii) convergent findings, and (iii) divergent findings. In the following section, we demonstrate how the application of the metaphor can be used as a framework both to develop theory and to facilitate the interpretation of the findings from mixed methods studies and their conclusions in each of these scenarios, and how using the metaphor can help to promote greater clarity of the study's purpose, its theoretical propositions, and the links between data sets.

3.2.1. Triangulating complementary results

To exemplify the use of the methodological metaphor of triangulation ( Erzberger and Kelle, 2003 ) for drawing inferences from complementary results, we have drawn on the results of a UK based study by Shipman et al. (2008) ( Fig. 2 ). In the UK, members of district nursing teams (DNs) provide most nursing care to people at home in the last year of life. Following concerns that inadequate education might limit the confidence of some DNs to support patients and their carers’ at home, and that low home death rates may in part be related to this, the Department of Health (DH) identified good examples of palliative care educational initiatives for DNs and invested in a 3-year national education and support programme in the principles and practice of palliative care. Shipman et al.’s study evaluates whether the programme had measurable effects on DN knowledge and confidence in competency in the principles and practice of palliative care. The study had two parts, a summative (concerned with outcomes) quantitative component which included ‘before and after’ postal questionnaires which measured effects on DNs’ ( n  = 1280) knowledge, confidence and perceived competence in the principles and practice of palliative care and a formative (concerned with process) qualitative component which included semi-structured focus groups and interviews with a sub-sample of DNs ( n  = 39).

Illustrating the use of triangulation ( Erzberger and Kelle, 2003 ) on complementary results in the study by Shipman et al. (2008) .

While their theoretical proposition may not be explicitly stated by the authors, there is clearly an implicit theoretical proposition that the educational intervention would improve DNs knowledge and confidence (theoretical proposition 1, Fig. 2 ). This was supported by the quantitative findings which showed significant improvement in the district nurses confidence in their professional competence post intervention. Qualitative results supported and complemented the quantitative findings as the district nurses described several benefits from the program including greater confidence in tackling complex problems and better communication with patient and carers’ because of greater understanding of the reasons for symptoms. Thus, a complementary theoretical proposition (theoretical proposition 2, Fig. 2 ) can be deduced from the qualitative findings: the DN's better understanding of factors contributing to complex problems and underlying reasons for symptoms led to improved confidence in competence raised from district nurses increased understanding.

Fig. 2 illustrates the theoretical propositions, the empirical findings from qualitative and quantitative data and the logical relationships between these. Theoretical proposition 1 is supported by the quantitative findings. From qualitative findings, a complementary theoretical proposition (theoretical proposition 2) can be stated explaining the process that led to the DNs improved confidence in competence.

3.2.2. Triangulating convergent results

To illustrate how the methodological metaphor of triangulation can be used to draw inferences from convergent findings, we have drawn on the example of a Danish study by Midtgaard et al. (2006) ( Fig. 3 ). This study was conducted to explore experiences of group cohesion and changes in quality of life (QoL) among people ( n  = 55) who participated in a weekly physical exercise intervention (for six weeks) during treatment for cancer. The study, conducted in a Danish hospital, involved the use of structured QoL questionnaires, administered at baseline and post intervention (at six weeks) to determine changes in QoL and health status, and qualitative focus groups, conducted post intervention (at six weeks), to explore aspects of cohesion within the group. With regards to the theoretical proposition of the study ( Fig. 3 ), group cohesion was seen as essential to understand the processes within the group that facilitated the achievement of desired outcomes and the satisfaction of affective needs as well as promoting a sense of belonging to the group itself.

Illustrating the use of triangulation ( Erzberger and Kelle, 2003 ) on convergent results in the study by Midtgaard et al. (2006) .

This proposition was deductively tested in an intervention where patients exercised in mixed gender groups of seven to nine members during a nine hour weekly session over a six week period and was supported by both the empirical quantitative and qualitative findings. The quantitative data showed significant improvements in peoples’ emotional functioning, social functioning and mental health. The qualitative data showed how the group setting motivated the individuals to pursue personal endeavors beyond physical limitations, that patients used each others as role models during ‘down periods’ and how exercising in a group made individuals feel a sense of obligation to train and to do their best. This subsequently helped to improve their social functioning which in turn satisfied their affective needs, improving their improved emotional functioning and mental health.

Fig. 3 illustrates the theoretical propositions, empirical findings from qualitative and quantitative data and the logical relationships between these. Both the quantitative and qualitative findings, demonstrating improvements in participants’ emotional and social functioning and their mental health, can be attributed to the nature of group cohesion within the programme as expected.

3.2.3. Triangulating divergent results

Qualitative and quantitative results that seem to contradict each other are often explained as resulting from methodological error. However, instead of a methodological flaw, a divergent result could be a consequence of the inadequacy of the theoretical concepts used. This may indicate the need for changing or developing the theoretical concepts involved ( Erzberger and Kelle, 2003 ). The following example of using the methodological metaphor of triangulation ( Erzberger and Kelle, 2003 ) for drawing inferences from divergent results is intended as an example rather than an attempt to change the theoretical concept involved. In a study by Skilbeck et al. (2005) ( Fig. 4 ), some results were found to be divergent which was explained as resulting from the use of inadequate questionnaires. We do not wish to critique their conclusion; rather we intend to simply offer an alternative interpretation for their findings.

Illustrating the use of triangulation ( Erzberger and Kelle, 2003 ) on divergent results using the study by Skilbeck et al. (2005) .

The study aimed to explore family carers’ expectations and experiences of respite services provided by one independent hospice in North England. This hospice provides inpatient respite beds specifically for planned respite admission for a two-week period. Referrals were predominated from general practitioners and patients and their carers were offered respite care twice a year, during the study this was reduced to once a year for each patient. Data was collected prior to respite admission and post respite care by semi-structured interviews and using the Relative Stress Scale inventory (RSSI), a validated scale to measure relative distress in relation to caring. Twenty-five carers were included but pre- and post-data were completed by 12 carers. Qualitative data was analysed by using a process of constant comparison and quantitative data by descriptive and comparative statistical analysis.

No clear theoretical proposition was stated by the authors, but from the definition of respite care it is possible to deduce that ‘respite care is expected to provide relief from care-giving to the primary care provider’ (theoretical proposition 1, Fig. 4 ). This proposition was tested quantitatively by pre- and post-test using the RSSI showing that the majority of carers experienced either a negative or no change in scores following the respite stay (no test of significance was stated). Accordingly, the theoretical proposition was not supported by the quantitative empirical data. The qualitative empirical results, however, were supportive in showing that most of the carers considered respite care to be important as it enabled them to have a break and a rest from ongoing care-responsibilities. From this divergent empirical data it could be suggested changing or developing the original theoretical proposition. It seems that respite care gave the carers relief from their care-responsibilities but not from the distress carers experienced in relation to caring (measured by the used scale). We would therefore suggest that in order to lessen distress related to caring, other types of support is also needed which would change the theoretical proposition as suggested (theoretical proposition 2).

Fig. 4 illustrates the theoretical propositions, empirical findings from qualitative and quantitative data and the logical relationships between these. Theoretical proposition 1 was not supported by the quantitative findings (indicated in Fig. 4 by the broken arrow), but the qualitative findings supported this proposition. From these divergent empirical findings, the theoretical proposition could accordingly be changed and developed. Respite care seemed to provide relief from carers’ on-going care-responsibilities, but other types of support need to be added to provide relief from distress experienced (theoretical proposition 2).

3.2.4. Triangulation to produce theoretical propositions

Methodological triangulation has also been applied to illustrate how theoretical propositions can be produced by drawing on the findings from a Finnish study by Lukkarinen (2005) ( Fig. 5 ). The purpose of this longitudinal study was to describe, explain and understand the subjective health related quality of life (QoL) and life course of people with coronary artery disease (CAD). A longitudinal quantitative study was undertaken during the year post treatment and 19 individuals also attended thematic interviews one year after treatment. This study is one of the few studies that clearly defines theoretical underpinnings for both the selected methods and their purpose, namely “to obtain quantitatively abundant average information about the QoL of CAD patients and the changes in it as well as the patients’ individual, unique experiences of their respective life situations” ( Lukkarinen, 2005 :622).

Illustrating the use of triangulation ( Erzberger and Kelle, 2003 ) to develop theory from the study by Lukkarinen (2005) .

The results of the quantitative analysis showed that the male and female CAD patients in the youngest age group had the poorest QoL. While patients’ QoL improved in the dimensions of pain, energy and mobility it deteriorated on dimensions of social isolation, sleep and emotional reactions. From the viewpoint of methodological triangulation used in the study the aim of the quantitative approach was to observe changes in QoL at the group level and also explore correlations of background factors to QoL. The qualitative approach generated information concerning both QoL in the individuals’ life situation and life course and the individuals’ rehabilitation. Both the quantitative and the qualitative analysis showed the youngest CAD patients to have the poorest psychosocial QoL. The results obtained using qualitative methods explained the quantitative findings and offered new insight into the factors related to poor psychosocial QoL, which could be used to help develop theoretical propositions around these. Patients at risk of poorer QoL were those with an acute onset of illness at a young age that led to an unexpected termination of career, resulting in financial problems, and worries about family. This group also experienced lack of emotional support (especially the females with CAD) and were concerned for the illness that was not alleviated by treatment. The interviews and the method of phenomenological psychology therefore helped to gain insight into the participants’ situational experience of QoL and life course, not detectable by the use of a questionnaire.

Fig. 5 illustrates the theoretical propositions, empirical findings from qualitative and quantitative data and the relationships between these. The use of the mixed methods approach enabled a clearer understanding to emerge in relation to the lived experience of CAD patients and the factors that were related to poor QoL. This understanding allows new theoretical propositions about these issues to be developed and further explored, as depicted at the theoretical level.

4. Discussion

As the need for, and use of, mixed methods research continues to grow, the issue of quality within mixed methods studies is becoming increasingly important ( O’Cathain et al., 2008 , O’Cathain et al., 2007 ). Similarly, the need for guidance on the analysis and integration of qualitative and quantitative data is a prominant issue ( Bazeley, 2009 ). This paper firstly intended to review the types of analytic approaches (parallel, concurrent or sequential data analysis) that have been used in mixed methods studies within healthcare research. As identified in previous research ( O’Cathain et al., 2008 ), we found that the majority of studies included in our review employed parallel data analysis in which the different analyses are not compared or consolidated until the full analysis of both data sets have been completed. A trend to conduct separate analysis on quantitative and qualitative data is apparent in mixed methods healthcare studies, despite the fact that if the data were correlated, a more complete picture of a particular phenomenon may be produced ( Onwuegbuzie and Teddlie, 2003 ). If qualitative and quantitative data are not integrated during data collection or analysis, the findings may be integrated at the stage of interpretation and conclusion.

Although little pragmatic guidance exists within the wider literature, Erzberger and Kelle (2003) have published some practical advice, on the integration of mixed methods findings. For mixed methodologists, the ‘triangulation metaphor’ offers a framework to facilitate a description of the relationships between data sets and theoretical concepts and can also assist in the integration of qualitative and quantitative data ( Erzberger and Kelle, 2003 ). Yet despite the fact that the framework was published in 2003 within Tashakkori and Teddlie's (2003) seminal work, the Handbook for Mixed Methods in Social and Behavioural Research, our search revealed that it has received little application within the published body of work around mixed methods studies since its publication. This is surprising since mixed methodologists are acutely aware of the lack of guidance with regards to the pragmatics and practicalities of conducting mixed methods research ( Bryman, 2006 , Leech et al., 2010 ). Furthermore, there have been frequent calls to move the field of mixed methods away from the “should we or shouldn’t we” debate towards the practical application, analysis and integration of mixed methods and its’ findings and what we can learn from each other's work and advice. Consequently, we have a state of ambiguity and instability in the field of mixed methods in which mixed methodologists find themselves lacking appropriate sources or evidence to draw upon with which to facilitate the future design, conduct and interpretation of mixed methods studies. It is for these reasons that we, in this paper, also intended to identify and select studies that could be used as examples for the application of Erzberger and Kelle's (2003) triangulation metaphor.

When reviewing the studies it was clear that the majority of theoretical assumptions were implicit, rather than explicitly stated by authors. Wu and Volker (2009) previously acknowledged that while studies undoubtedly have a theoretical basis in their literature reviews and the nature of their research questions, they often fail to clearly articulate a particular theoretical framework. This is unfortunate as theory can help researchers to clarify their ideas and also help data collection, analysis and to improve the study's rigour ( Wu and Volker, 2009 ). When using triangulation as a methodological metaphor ( Erzberger and Kelle, 2003 ), researchers are encouraged to articulate their theoretical propositions and to validate their conclusions in relation to the chosen theories. Theory can also guide researchers when defining outcome measures . Should the findings not support the chosen theory, as shown in our examples on complementary and divergent results, researchers can modify or expand their theory accordingly and new theory may be developed ( Wu and Volker, 2009 ). It is therefore our belief that using triangulation as a methodological metaphor in mixed methods research can also benefit the design of mixed method studies.

Like other researchers ( O’Cathain et al., 2008 ), we have also found that most of the papers reviewed lacked clarity in whether the reported results primarily stemmed from qualitative or quantitative findings. Many of the papers were even less clear when discussing their results and the basis of their conclusions. The reporting of mixed methods studies is notoriously challenging, but clarity and transparency are, at the very least, crucial in such reports ( O’Cathain, 2009 ). Using triangulation as a methodological metaphor ( Erzberger and Kelle, 2003 ) may be one way of addressing this lack of clarity by explicitly showing the types of data that researchers have based their interpretations on. It may even help address some of the issues raised in the debate on the feasibility of integrating research methods and results stemming from different epistemological and ontological assumptions and paradigms ( Morgan, 2007 , Sale et al., 2002 ). In order to carry out methodological triangulation researchers also need to identify and observe the consistency and adequacy of the two methods, positivistic and phenomenological regarding the research questions, data collection, methods of analysis and conclusions.

While we used systematic principles in our search for mixed methods studies in healthcare research, we cannot claim to have included all such studies. In many cases, reports of mixed methods studies are subjected to ‘salami slicing’ by researchers and hence the conduct of, and findings from, individual approaches are addressed in separate papers. Since these papers are often not indexed as a ‘mixed method’ study, they are undoubtedly more difficult to identify. Furthermore, different terminologies are used to describe and index mixed methods studies within the electronic databases ( Halcomb and Andrew, 2009a ), making it challenging to be certain that all relevant studies were captured in this review. However, the studies included in this review should give a sufficient overview of the use of mixed analysis in healthcare research and most importantly, they enable us to make suggestions for the future design, conduct, interpretation and reporting of mixed methods studies. It is also important to emphasise that we have based our triangulation examples on the data published but have no further knowledge of the analysis and findings undertaken by the authors. The examples should thus be taken as examples and not alternative explanations or interpretations.

Mixed methods research within healthcare remains an emerging field and its use is subject to much debate. It is therefore particularly important that researchers clearly describe their use of the approach and the conclusions made to improve transparency and quality within mixed methods research. The use of triangulation as a methodological metaphor ( Erzberger and Kelle, 2003 ) can help researchers not only to present their theoretical propositions but also the origin of their results in an explicit way and to understand the links between theory, epistemology and methodology in relation to their topic area. Furthermore it has the potential to make valid inferences, challenge existing theoretical assumptions and to develop or create new ones.

Conflict of interest

None declared.

Ethical approval

Not required.

• Abadia C.E., Oviedo D.G. Bureaucratic itineraries in Colombia. A theoretical and methodological tool to assess managed-care health care systems. Social Science & Medicine. 2009; 68 (6):1153–1160. [ PubMed ] [ Google Scholar ]
• Bazeley P. Analysing mixed methods data. In: Andrew S., Halcomb E.J., editors. Mixed Methods Research for Nursing and the Health Sciences. Wiley-Blackwell; Chichester: 2009. pp. 84–118. [ Google Scholar ]
• Beatty P.W., Neri M.T., Bell K., DeJong G. Use of outcomes information in acute inpatient rehabilitation. American Journal of Physical Medicine & Rehabilitation. 2004; 83 (6):468–478. [ PubMed ] [ Google Scholar ]
• Bernardi L., Kleim S., von der Lippe H. Social influences on fertility: a comparative mixed methods study in Eastern and Western Germany. Journal of Mixed Methods Research. 2007; 1 (1):23–47. [ Google Scholar ]
• Brannen J. Prologue: mixed methods for novice researchers: reflections and themes. International Journal of Multiple Research Approaches. 2009; 3 (1):8–12. [ Google Scholar ]
• Bryman A. University Press; Oxford: 2004. Social Research Methods. [ Google Scholar ]
• Bryman A. Integrating quantitative and qualitative research: how is it done? Qualitative Research. 2006; 6 (1):97–113. [ Google Scholar ]
• Bryman A. Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research. 2007; 1 (1):8–22. [ Google Scholar ]
• Bussing R., Koro-Ljungberg M.E., Gary F., Mason D.M., Garvan C.W. Exploring help-seeking for ADHD symptoms: a mixed-methods approach. Harvard Review of Psychiatry. 2005; 13 (2):85–101. [ PubMed ] [ Google Scholar ]
• Canales M.K., Rakowski W. Development of a culturally specific instrument for mammography screening: an example with American Indian women in Vermont. Journal of Nursing Measurement. 2006; 14 (2):99–115. [ PubMed ] [ Google Scholar ]
• Cheng G.Y. A study of clinical questions posed by hospital clinicians. Journal of the Medical Library Association. 2004; 92 (4):445–458. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Clarke P.N., Yaros P.S. Research blenders: commentary and response. Transitions to new methodologies in nursing sciences. Nursing Science Quarterly. 1988; 1 (4):147–151. [ PubMed ] [ Google Scholar ]
• Cochrane, 2009. Cochrane Handbook for Systematic Reviews of Interventions. http://www.cochrane-handbook.org/ . The Cochrane Collaboration.
• Creswell J.W. Sage; Thousand Oaks: 2003. Research Design: Qualitative, Quantitative and Mixed Methods Approaches. [ Google Scholar ]
• Creswell J.W., Plano Clark V.L. Sage; Thousand Oaks: 2007. Designing and Conducting Mixed Methods Research. [ Google Scholar ]
• Erzberger C., Kelle U. Making inferences in mixed methods: The rules of integration. In: Tashakkori A., Teddlie C., editors. Handbook of Mixed Methods in Social & Behavioural Research. Sage; Thousand Oaks: 2003. pp. 457–488. [ Google Scholar ]
• Foss C., Ellefsen B. The value of combining qualitative and quantitative approaches in nursing research by means of method triangulation. Journal of Advanced Nursing. 2002; 40 (2):242–248. [ PubMed ] [ Google Scholar ]
• Halcomb E.J., Andrew A. Managing mixed methods projects. In: Andrew S., Halcomb E.J., editors. Mixed Methods Research for Nursing and the Health Sciences. Wiley-Blackwell; Chichester: 2009. pp. 50–64. [ Google Scholar ]
• Halcomb E.J., Andrew A., Brannen J. Introduction to mixed methods research for nursing and the health sciences. In: Andrew S., Halcomb E.J., editors. Mixed Methods Research for Nursing and the Health Sciences. Wiley-Blackwell; Chichester: 2009. pp. 3–12. [ Google Scholar ]
• Halcomb E.J., Davidson P.M., Griffiths R., Daly J. Cardiovascular disease management: time to advance the practice nurse role? Australian Health Review. 2008; 32 (1):44–53. [ PubMed ] [ Google Scholar ]
• Jick T.D. Mixing qualitative and quantitative methods: triangulation in action. Administrative Science Quarterly. 1979; 24 :602–611. [ Google Scholar ]
• Johnson R.B., Onquegbuzie A.J. Mixed methods research: a paradigm whose time has come. Educational Researcher. 2004; 33 (7):14–26. [ Google Scholar ]
• Kartalova-O’Doherty Y., Tedstone Doherty D. Satisfied carers of persons with enduring mental illness: who and why? The International Journal of Social Psychiatry. 2009; 55 (3):257–271. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Kelle U. Sociological explanations between micro and macro and the integration of qualitative and quantitative methods. Forum: Qualitative Social Research. 2001; 2 (1):5. Art. [ Google Scholar ]
• Khan K.S. Centre for Review and Dissemination; York: 2001. Undertaking Systematic Reviews of Research on Effectiveness: CRD's Guidance for those Carrying Out or Commissioning Reviews. [ Google Scholar ]
• Kroll T., Neri M. Designs for mixed methods research. In: Andrew S., Halcomb E.J., editors. Mixed Methods Research for Nursing and the Health Sciences. Wiley-Blackwell; Chichester: 2009. pp. 31–49. [ Google Scholar ]
• Leech N.L., Dellinger A.M., Brannagan K.B., Tanaka H. Evaluating mixed research studies: a mixed methods approach. Journal of Mixed Methods Research. 2010; 4 (1):17–31. [ Google Scholar ]
• Lukkarinen H. Methodological triangulation showed the poorest quality of life in the youngest people following treatment of coronary artery disease: a longitudinal study. International Journal of Nursing Studies. 2005; 42 (6):619–627. [ PubMed ] [ Google Scholar ]
• Midtgaard J., Rorth M., Stelter R., Adamsen L. The group matters: an explorative study of group cohesion and quality of life in cancer patients participating in physical exercise intervention during treatment. European Journal of Cancer Care. 2006; 15 (1):25–33. [ PubMed ] [ Google Scholar ]
• Morgan D.L. Paradigms lost and pragmatism regained: methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research. 2007; 1 (1):48–76. [ Google Scholar ]
• O’Cathain A. Reporting mixed methods projects. In: Andrew S., Halcomb E.J., editors. Mixed Methods Research for Nursing and the Health Sciences. Wiley-Blackwell; Chichester: 2009. pp. 135–158. [ Google Scholar ]
• O’Cathain A., Murphy E., Nicholl J. Why, and how, mixed methods research is undertaken in health services research in England: a mixed methods study. BMC Health Services Research. 2007; 7 :85. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• O’Cathain A., Murphy E., Nicholl J. The quality of mixed methods studies in health services research. Journal of Health Services Research & Policy. 2008; 13 (2):92–98. [ PubMed ] [ Google Scholar ]
• Onwuegbuzie A., Teddlie C. A framework for analysing data in mixed methods research. In: Tashakkori A., Teddlie C., editors. Handbook of Mixed Methods in socIal & Behavioural Research. Sage; Thousands Oak: 2003. pp. 351–383. [ Google Scholar ]
• Onwuegbuzie A.J., Leech N.L. On becoming a pragmatic researcher: the importance of combining quantitative and qualitative methodologies. International Journal of Social Research Methodology. 2005; 8 (5):375–387. [ Google Scholar ]
• Sale J.E.M., Lohfeld L.H., Brazil K. Revisiting the quantitative-qualitative debate: implications for mixed methods research. Quality and Quantity. 2002; 36 :43–53. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Shipman C., Burt J., Ream E., Beynon T., Richardson A., Addington-Hall J. Improving district nurses’ confidence and knowledge in the principles and practice of palliative care. Journal of Advanced Nursing. 2008; 63 (5):494–505. [ PubMed ] [ Google Scholar ]
• Skilbeck J.K., Payne S.A., Ingleton M.C., Nolan M., Carey I., Hanson A. An exploration of family carers’ experience of respite services in one specialist palliative care unit. Palliative Medicine. 2005; 19 (8):610–618. [ PubMed ] [ Google Scholar ]
• Steckler A., McLeroy K.R., Goodman R.M., Bird S.T., McCormick L. Toward integrating qualitative and quantitative methods: an introduction. Health Education Quarterly. 1992; 19 (1):1–8. [ PubMed ] [ Google Scholar ]
• Tashakkori A., Creswell J.W. Editorial: the new era of mixed methods. Journal of Mixed Methods Research. 2007; 1 (1):3–7. [ Google Scholar ]
• Tashakkori A., Teddlie C. Sage; Thousands Oak: 2003. Handbook of Mixed Methods in Social & Behavioural Research. [ Google Scholar ]
• Wu H.L., Volker D.L. The use of theory in qualitative approaches to research: application in end-of-life studies. Journal of Advanced Nursing. 2009; 65 (12):2719–2732. [ PubMed ] [ Google Scholar ]