qualitative research using mixed methods

Chapter 15. Mixed Methods

Introduction.

Where deep ethnography (chapter 14) is a tradition that relies on naturalistic techniques of data collection, foregrounding the specificity of a particular culture and site, there are other times when researchers are looking for approaches that allow them to make use of some of the analytical techniques developed by statisticians and quantitative researchers to generalize the data they are collecting. Rather than push into a deeper understanding of a culture through thick interpretive descriptions, these researchers would rather abstract from a sufficiently large body of cases (or persons) to hazard predictions about a connection, relationship, or phenomenon. You may already have some experience learning basic statistical techniques for analyzing large data sets. In this chapter, we describe how some research harnesses those techniques to supplement or augment qualitative research, mixing methods for the purpose of building stronger claims and arguments. There are many ways this can be done, but perhaps the most common mixed methods research design involves the use of survey data (analyzed statistically via descriptive cross-tabs or fairly simple regression analyses of large number probability samples) plus semistructured interviews. This chapter will take a closer look at mixed methods approaches, explain why you might want to consider them (or not), and provide some guidance for successful mixed methods research designs.

What Is It? Triangulation, Multiple Methods, and Mixed Methods

First, a bit of nomenclature. Mixed methods can be understood as a path toward triangulation . Triangulation is a way of strengthening the validity of a study by employing multiple forms of data, multiple investigators, multiple theoretical perspectives, or multiple research methods. Let’s say that Anikit wants to know more about how first-year college students acclimate to college. He could talk to some college students (conduct interviews) and also observe their behavior (fieldwork). He is strengthening the validity of his study by including multiple forms of data. If both the interview and the observations indicate heavy reliance on peer networks, a reported finding about the importance of peers would be more credible than had he only interviewed students or only observed them. If he discovers that students say one thing but do another (which is pretty common, after all), then this, too, becomes an interesting finding (e.g., Why do they forget to talk about their peers when peers have so much observable influence?). In this case, we say that Anikit is employing multiple forms of data, or even that he relies on multiple methods. But he is not, strictly speaking, mixing data. Mixed methods refer specifically to the use of both quantitative and qualitative research methods. If Anikit were to supplement his interviews and/or observations with a random sample of one thousand college students, he would then be employing a mixed methods approach. Although he might not get the rich details of how friends matter in the survey, the large sample size allows statistical analyses of relationships among variables, perhaps showing which groups of students are more likely to benefit from strong peer networks. So to summarize, both multiple methods and mixed methods are forms of research triangulation, [1] but mixed methods include mixing both qualitative and quantitative research elements.

Mixed methods techniques, then, are pretty unique. Where many qualitative researchers have little interest in statistical generalizability, and many quantitative researchers undervalue the importance of rich descriptions of singular cases, the mixed methods researcher has an open mind about both approaches simultaneously. And they use the power of both approaches to build stronger results: [2]

Quantitative (mainly deductive) methods are ideal for measuring pervasiveness of “known” phenomena and central patterns of association, including inferences of causality. Qualitative (mainly inductive) methods allow for identification of previously unknown processes, explanations of why and how phenomena occur, and the range of their effects (Pasick et al. 2009). Mixed methods research, then, is more than simply collecting qualitative data from interviews, or collecting multiple forms of qualitative evidence (e.g., observations and interviews) or multiple types of quantitative evidence (e.g., surveys and diagnostic tests). It involves the intentional collection of both quantitative and qualitative data and the combination of the strengths of each to answer research questions . ( Creswell et al. 2011:5 ; emphases added)

Why Use Mixed Methods?

As with all methodological choices, the answer depends on your underlying research questions and goals. Some research questions are better answered by the strengths of the mixed methods approach. Small ( 2011 ) discusses the use of mixed methods as a confirmation or complement of one set of findings from one method by another. Creswell and Clark ( 2017:8ff .) note the following situations as being particularly aided by combining qualitative and quantitative data collection and analysis: (1) when you need to obtain both more complete (need for qualitative) and more corroborated (need for quantitative) information; (2) when you need to explain (need for qualitative) initial results (quantitative); (3) when you need to do an exploratory study (need for qualitative) before you can really create and administer a survey or other instrument (quantitative); (4) when you need to describe and compare different types of cases to get a more holistic understanding of what is going on; (5) when you need (or very much want!) to include participants in the study, adding in qualitative elements as you build a quantitative design; (6) when you need all the tools at your disposal to develop, implement, and evaluate a program.

Please note what is not included in this list: because you can . Mixed methods research is not always preferable, even if in general it makes your study “stronger.” Strength is not the only criterion for quality or value. I have met many students in my career who assume that the mixed methods approach is optimal because it includes both qualitative and quantitative research. That is the wrong way of looking at things. Mixed methods are optimal when and only when they fit the necessities of your research question (e.g., How can I corroborate this interesting finding from my interviews so that proper solutions can be fashioned?) or underlying goal (e.g., How can I make sure to include the people in this program as participants of the study?).

If you are just starting out and learning your way through designing your first study, mixed methods are not default requirements. As you will see in the next section on design, mixed methods studies often happen sequentially rather than consecutively, so I recommend you start with the study that has the most meaning to you, the one that is the most compelling. Later on, if you want to add (mix) another approach for the sake of strength or validity or “corroboration” (if you are adding quantitative) or “explanation” (if you are adding qualitative), you can always do that then, after the completion of your first study.

Segue: Historical Interlude

For those interested in a little history, one could make the case that mixed methods research in the social sciences actually predates the development of either quantitative or qualitative research methods. The very first social scientists (what we call “social science” in the West, which is itself a historical construct, as many other peoples have been exploring meaning and interpretation of the social world for centuries if not millennia) often employed a mélange of methods to address their research questions. For example, the first sociologists in the US operating out of the “Chicago School” of the early twentieth century surveyed neighborhoods, interviewing people, observing demographic subcultures, and making tallies of everything from the numbers of persons in households to what languages were being spoken. They learned many of these techniques from early statisticians and demographers in Europe—people like Charles Booth ( 1902 ), who surveyed neighborhoods in London, and Frédéric Le Play, who spent decades examining the material conditions of the working classes across Europe, famously including family “budgets” along with interviews and observations (see C. B. Silver 1982). The renowned American sociologist W. E. B. Du Bois, who was the first Black man to earn a PhD from Harvard University, also conducted one of the very first mixed methods studies in the US, The Philadelphia Negro ( 1899 ). This work mapped every Black residence, church, and business in Philadelphia’s Seventh Ward and included observations and details on family structure and occupation (similar to Booth’s earlier work on London). Continuing through the 1930s and 1940s, “community studies” were conducted by teams of researchers who basically tallied everything they could find about the particular town or city they chose to work in and performed countless interviews, months and years of fieldwork, and detailed mappings of community relationships and power relations. One of the most famous of these studies includes the “Middletown” studies conducted by Robert and Helen Lynd ( 1929 , 1937 ).

As statistical analysis progressed after World War II alongside the development of the technology that allowed for ever faster computations, quantitative research emerged as a separate field. There was a lot to learn about how to conduct statistical analyses, and there were more refinements in the creation of large survey instruments. Qualitative research—the observations and interviews at the heart of naturalistic inquiry—became a separate field for different kinds of researchers. One might even say qualitative research languished at the expense of new developments of quantitative analytical techniques until the 1970s, when feminist critiques of positivist social science emerged, casting doubt on the superiority of quantitative research methods. The rise of interdisciplinarity in recent decades combined with a lessening of the former harsh critique of quantitative research methods and the “paradigm wars” ( Small 2011 ) has allowed for an efflorescence of mixed methods research, which is where we are today.

Mixed-Methods Research Designs

Returning from our historical interlude to the list of possible uses of mixed methods, we now confront the question of research design. If we are using more than one method, how exactly do we do this, and when ? The how and the when will depend largely on why we are using mixed methods. For example, if we want to corroborate findings emerging from interviews, then we obviously begin with interviews and follow with, perhaps, a large survey. On the other hand, if we are seeking to explain findings generated from a survey, we begin with that survey and add interviews or observations or focus groups after its completion. And if we are seeking to include participants in the research design itself, we may want to work concurrently, interviewing and holding focus groups as surveys are administered. So it all depends on why we have chosen to use mixed methods.

We can think of our choices here in terms of three possibilities. The first, called sequential explanatory , begins with quantitative data (collection) and then follows with qualitative data (collection). After both are collected, interpretations are made. The second, called sequential exploratory , begins the other way around, with qualitative followed by quantitative. After both are collected, interpretations are made. The third, called concurrent triangulation , conceives of both quantitative and qualitative elements happening concurrently. In practice, one may still happen before the other, but one does not follow the other. The data then converge, and from that convergence, interpretations are made.

In sequential explanatory design (figure 15.1), we are asking ourselves, “In what ways do the qualitative findings explain the quantitative results?” ( Creswell et al. 2017 ). This design thus gives some priority to the quantitative data. The qualitative data, collected after the quantitative data, is used to provide a better understanding of the research problem and then the quantitative data alone.

Often, this means providing some context or explaining meanings and motivations behind the correlations found in the quantitative data. For example, in my research on college students ( Hurst 2019 ), I found a statistical correlation between upper-middle-class female students and study abroad. In other words, and stating this rather baldly, class*gender could be used to predict who studied abroad. But I couldn’t fully explain why, given the survey data I had collected. [3] To answer these (and other) questions that the survey results raised, I began interviewing students and holding focus groups. And it was through these qualitative forms of data collection that I found a partial answer: upper-middle-class female students had been taught to see study abroad as a final “finishing” component of their education in a way that other students simply had not. They often had mothers who had done the same. And they clearly saw connections here to the kinds of well-traveled cosmopolitan adults they wanted to become.

In sequential exploratory design (figure 15.2), we are asking ourselves, “In what ways do the quantitative findings generalize (or confirm) the qualitative results?” ( Creswell et al. 2018 ). This design thus gives some priority to the qualitative data. The quantitative data, collected after the qualitative data, is used to confirm the findings.

This approach is ideal for developing new instruments or when a researcher intends to generalize findings from a qualitative study to different groups or populations. The American Sociological Association (ASA) Task Force on First-Generation and Working-Class Persons wanted to understand how class background may have played a role in the success of sociology graduate students and faculty. Because this was a relatively new research question, the task force began by conducting several focus groups, asking general questions about how class might have affected careers in sociology. Based on several recurring findings (e.g., high debt burdens, mentorship, feelings of fit), the task force developed a survey instrument that it then administered to more than one thousand sociologists, thus generalizing the preliminary findings and providing corroboration of some of the key variables at play.

In concurrent triangulation design (figure 15.3), neither the quantitative nor the qualitative component takes precedence. Although in practice one might precede the other in time, neither is the tail that wags the dog, so to speak. They are both the dog. The general of this design is to better understand or deepen one’s understanding of the phenomenon under study. The goal is to obtain different but complementary data that strengthen (validate) the overall results.

These designs might be either nested or nonnested . In a nested design , a subsample of an original randomized sample is used for further interviews or observation. A common nested design form is where in-depth interviews are conducted with a subsample of those who filled out a survey. Nonnested designs occur when it is impractical or impossible to recruit the same individuals that took place in the survey. The research I conducted for my book Amplified Advantage ( Hurst 2019 ) is an example of this. I supplemented a large national survey of college students and recent college graduates with interviews and focus groups of similar college students and graduates who were not participants in the study (or who may have been randomly selected as participants but without my knowledge or linking their data). Nonnested designs are much more flexible than nested designs, but they eliminate the possibility of linking data across methods.

As with all research design, it is important to think about how best to address your particular research question. There are strengths and weaknesses of each design. Sequential design allows for the collection and analysis of different methods separately, which can make the process more manageable. Sequential designs are relatively easy to implement, design, and report. Sequential exploratory designs allow you to contextualize and generalize qualitative findings to larger samples, while sequential explanatory designs enable you to gain a deeper understanding of findings revealed by quantitative data analysis. All sequential design takes a lot of time, however. You are essentially doubling your research. This is why I do not recommend these approaches to undergraduate students or graduate students in master’s programs. In contrast, concurrent designs, whose dual methods may be conducted simultaneously, may be conducted more quickly. However, as a practical matter, you will probably end up focusing first on one data collection method and then the other, so the time saved might be minimal. [4] Concurrent design can also preclude following up on interesting findings that emerge from one side of the study, and the abbreviated form may prevent clarification of confusing issues that arise during analysis. If the results are contradictory or diverge, it may also be difficult to integrate the data. You might end up with more questions to pursue for further study and not much conclusive to say at the end of all your work.

Finally, there is what I will call here the recursive design model (figure 15.4), in which you combine both explanatory and exploratory sequential design.

This design is currently being used by the ASA task force mentioned above. The first stage of data collection involved several focus groups. From these focus groups, we constructed a survey that we administered to ASA members. The focus group survey could be viewed as an example of exploratory sequential design. As the surveys were being analyzed, we added a nested set of interviews with persons who had taken the survey and who indicated a willingness to participate in this later stage of data collection. These interviews then help explain some of the findings from the survey. The entire process takes several years, however, and involves multiple researchers!

Advanced: Crossover Design

Small’s ( 2011 ) review of the state of mixed methods research argues that mixed methods are being increasingly adopted in social science research. In addition to sequential and concurrent research designs, where quantitative and qualitative data work to either confirm or complement each other, he sets forth examples of innovative designs that go further toward truly blending the special techniques and strengths of both quantitative and qualitative methods. [5] Written in 2011, I have seen scant evidence so far that these blended techniques are becoming well established, but they are promising. As new software programs for data analysis emerge, along with increased computing power, there will be greater opportunities for crossover work. Perhaps you can take up the charge and attempt one of these more innovative approaches yourself.

Here is Small’s ( 2011:73ff .) list of innovative crossover research design:

• Network analyses of narrative textual data . Here, researchers use techniques of network analysis (typically quantitative) and apply them to narratives (qualitative), coding stories as separate “nodes” and then looking for connections between those nodes, as is done in network analysis.
• Sequence analyses of narrative textual data . Here, techniques of event structure analysis and optimal matching (designed for analysis of quantitative data) are applied to narratives (qualitative data). The narratives are reconceived as a series of events, and then causal pathways between these events are mapped. This allows for identification of crucial turning points as well as “nonsignificant” events that just happened.
• Quantitative analyses of semantic (meaning) elements of narrative textual data . The basic distinction between quantitative (data in the form of numbers) and qualitative (date in the form of words) gets blurred here, as words themselves and their meanings and contexts are coded numerically. I usually strongly advise beginning students to do this, as what often happens is that they begin to think quantitatively about the data, flattening it considerably. However, if done with full attention to meaning and context, the power of computing/analytical software may strengthen the coding process.
• Narrative analyses of large-n survey data. In contrast to the first three designs listed above, where quantitative techniques were applied to qualitative data, we now come to a situation where the reverse takes place. Here we have a large data set, either coded numerically or “raw” with various choice options for each question posed. Rather than read the data set as a series of factors (variables) whose relationship one explores through statistical analyses, the researcher creates a narrative from the survey responses, contextualizing the answers rather than abstracting them. [6]
• Regression-based analyses of small-n or narrative textual data. This is by far the most common crossover method and the reverse of the fourth example. Many qualitative software analysis programs now include basic quantitative analytical functions. The researcher can code interview transcripts and fieldnotes in such a way that allows for basic cross-tabulations, simple frequency statistics, or even basic regression analyses. Transcripts and fieldnotes can generate “variables” for such analyses.

Despite the promise of blending methods in this way, the possibility of doing damage to one’s study by discounting the particular values of either quantitative or qualitative approaches is a real one. Unlike mixed methods, where the two approaches work separately (even when designed to concur in time), crossover research blends or muddies the two. Small ( 2011 ) warns, “At a minimum, the application of techniques should not be fundamentally contrary to the epistemological principles from which they are derived or to the technical problems for which they were intended” ( 76 ). When employing any of these designs or blending approaches, it is very important to explain clearly and fully what one’s aims are and how the analysis has proceeded, as this allows others to evaluate the appropriateness of the design for the questions posed.

Further Readings

Cech, Erin. 2021. The Trouble with Passion: How Searching for Fulfillment at Work Fosters Inequality . Berkeley, CA: University of California Press.* Cech combines surveys with interviews to explore how people think about and talk about job searches and careers.

Cooper, Kristy S. 2014. “Eliciting Engagement in the High School Classroom: A Mixed-Methods Examination of Teaching Practices.” American Educational Research Journal 51(2):363–402. An example of using multilevel regression analyses with both interviews and observations to ascertain how best to engage students.

Creswell, John W., and J. David Creswell. 2018. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches . Thousand Oaks, CA: SAGE. Essential textbook for mixed-methods research.

Edin, Kathryn, and Maureen A. Pirog. 2014. “Special Symposium on Qualitative and Mixed-Methods for Policy Analysis.” Journal of Policy Analysis and Management 33(2):345–349. A good overview of the strengths of mixed-methods research, which, the authors argue, make it particularly well suited for public policy analysis.

Hurst, Allison L. 2019. Amplified Advantage: Going to a “Good” College in an Era of Inequality . Lanham, MD: Rowman & Littlefield: Lexington Books..* Employs a national survey of recent graduates of small liberal arts colleges combined with interviews, focus groups, and archival data to explore how class background affects college outcomes.

Johnson, R. Burke, and Anthony J. Onwuegbuzie. 2004. “Mixed Methods Research: A Research Paradigm Whose Time Has Come.” Educational Researcher 33(7):14–26. Takes a pragmatic approach and provides a framework for designing and conducting mixed-methods research.

Klinenberg, Eric. 2015. Heat Wave: A Social Autopsy of Disaster in Chicago . Chicago: University of Chicago Press.* A great read and could not be more timely. Klinenberg uses a combination of fieldwork, interviews, and archival research to investigate why some neighborhoods experience greater mortality than others.

Lynd, Robert, and Helen Lynd. 1929. Middletown: A Study in American Culture . New York: Harcourt, Brace.* This early mixed-methods study of a “typical” American city was a pioneering work in sociology. The husband-and-wife team seemingly explores every aspect of life in the city, mapping social networks, surveying attitudes and beliefs, talking to people about their expectations and lives, and observing people going about their everyday business. Although none of the techniques are very sophisticated, this remains a classic example of pragmatic research.

Lynd, Robert, and Helen Lynd. 1937. Middletown in Transition . New York: Harcourt, Brace. The follow-up to the Lynds’ original study of a small American city. More theoretical and critical than the first volume.

Markle, Gail. 2017. “Factors Influencing Achievement in Undergraduate Social Science Research Methods Courses: A Mixed Methods Analysis.” Teaching Sociology 45(2):105–115.* Examines the factors that influence student achievement using an initial survey with follow-up interviews.

Matthews, Wendy K. 2017. “‘Stand by Me’: A Mixed Methods Study of a Collegiate Marching Band Members’ Intragroup Beliefs throughout a Performance Season.” Journal of Research in Music Education 65(2):179–202.* The primary method here is focus groups, but the author also employed multivariate analysis of variance (MANOVA) to shore up the qualitative findings.

Monrad, Merete. 2013. “On a Scale of One to Five, Who Are You? Mixed Methods in Identity Research.” Acta Sociologica 56(4):347–360. A call to employ mixed methods in identity research.

Silver, Catherine Bodard. 1982. Frédéric Le Play on Family, Work and Social Change . Chicago: University of Chicago Press. For anyone interested in the historic roots of mixed-methods research, the work of Frédéric Le Play is essential. This biography is a good place to start.

Small, Mario Luis. 2011. “How to Conduct a Mixed Methods Study: Recent Trends in a Rapidly Growing Literature.” Annual Review of Sociology 37:57–86. A massive review of recent mixed-methods research, distinguishing between mixed-data-collection studies, which combine two or more kinds of data, and mixed-data-analysis studies, which combine two or more analytical strategies. Essential reading for graduate students wanting to use mixed methods.

• To extend this notion of triangulation a little further: if Anikit enlisted the help of Kanchan to interpret the observations and interview transcripts, he would be strengthening the validity of the study through multiple investigators, another form of triangulation having nothing at all to do with what methods are employed. He could also bring in multiple theoretical frameworks—say, Critical Race Theory and Bourdieusian field analysis—as a form of theoretical triangulation. ↵
• If stronger is your aim, that is. For many qualitative researchers, verisimilitude, or the truthfulness of a presentation, is a more desirable aim than strength in the sense of validity. ↵
• Actually, I could do a fair amount of testing on other variables’ relationships to this finding: students who had gone far away to college (more than five hundred miles) were significantly more likely to study abroad, for example, as were students who majored in arts and humanities courses. But I still missed any way of getting at personal motivations or how individuals explained these motivations. That is the part a survey is just never going to fully get at, no matter how well or numerous the questions asked. ↵
• The big exception here is when you are relying on data that has already been collected and is ready for analysis, as in the case of large survey data sets like the General Social Survey. In that case, it is not too time consuming to design a mixed methods study that uses (nonnested) interviews to supplement your analyses of survey data. ↵
• I refer to these as blended methods rather than mixed methods because the epistemological positions and science claims, usually rather distinct from quantitative (more positivistic) and qualitative (more naturalistic), blur considerably. ↵
• I admit that trained first as a qualitative researcher, this has always been my impulse when confronting a large survey data set. ↵

A research design that employs both quantitative and qualitative methods, as in the case of a survey supplemented by interviews.

The process of strengthening a study by employing multiple methods (most often, used in combining various qualitative methods of data collection and analysis).  This is sometimes referred to as data triangulation or methodological triangulation (in contrast to investigator triangulation or theory triangulation).  Contrast mixed methods .

A mixed-methods design that conceives of both quantitative and qualitative elements happening concurrently.  In practice, one may still happen before the other, but one does not follow the other.  The data then converge and from that convergence interpretations are made.  Compare sequential exploratory design and sequential explanatory design .

A mixed-methods design that begins with quantitative data collection followed by qualitative data collection, which helps “explain” the initial quantitative findings.  Compare sequential exploratory design and concurrent triangulation .

A mixed-methods design that begins with qualitative data collection followed by quantitative data collection.  In this case, the qualitative data suggests factors and variables to include in the quantitative design.  Compare sequential explanatory design and concurrent triangulation .

A form of mixed-methods design in which a subsample of an original randomized sample is used for further interviews or observation.

Introduction to Qualitative Research Methods Copyright © 2023 by Allison Hurst is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License , except where otherwise noted.

The Use of Mixed Methods in Research

• Reference work entry
• First Online: 13 January 2019
• Cite this reference work entry

• Kate A. McBride 2 ,
• Freya MacMillan 3 ,
• Emma S. George 4 &
• Genevieve Z. Steiner 5  

2511 Accesses

9 Citations

Mixed methods research is becoming increasingly popular and is widely acknowledged as a means of achieving a more complex understanding of research problems. Combining both the in-depth, contextual views of qualitative research with the broader generalizations of larger population quantitative approaches, mixed methods research can be used to produce a rigorous and credible source of data. Using this methodology, the same core issue is investigated through the collection, analysis, and interpretation of both types of data within one study or a series of studies. Multiple designs are possible and can be guided by philosophical assumptions. Both qualitative and quantitative data can be collected simultaneously or sequentially (in any order) through a multiphase project. Integration of the two data sources then occurs with consideration is given to the weighting of both sources; these can either be equal or one can be prioritized over the other. Designed as a guide for novice mixed methods researchers, this chapter gives an overview of the historical and philosophical roots of mixed methods research. We also provide a practical overview of its application in health research as well as pragmatic considerations for those wishing to undertake mixed methods research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Appleton JV, King L. Journeying from the philosophical contemplation of constructivism to the methodological pragmatics of health services research. J Adv Nurs. 2002;40(6):641–8.

Article   Google Scholar  

Baba CT, Oliveira IM, Silva AEF, Vieira LM, Cerri NC, Florindo AA, de Oliveira Gomes GA. Evaluating the impact of a walking program in a disadvantaged area: using the RE-AIM framework by mixed methods. BMC Public Health. 2017;17(1):709. https://doi.org/10.1186/s12889-017-4698-5 .

Bryman A. The research question in social research: what is its role? Int J Soc Res Methodol. 2007;10(1):5–20. https://doi.org/10.1080/13645570600655282 .

Bryman A. The end of the paradigm wars. In: Alasuutari P, Bickman L, Brannen J, editors. The Sage handbook of social research methods. Thousand Oaks: Sage; 2008. p. 13–25.

Google Scholar  

Caracelli VJ, Greene JC. Data-analysis strategies for mixed-method evaluation designs. Educ Eval Policy Anal. 1993;15(2):195–207. https://doi.org/10.3102/01623737015002195 .

Castro FG, Kellison JG, Boyd SJ, Kopak A. A methodology for conducting integrative mixed methods research and data analyses. J Mixed Methods Res. 2010;4(4):342–60. https://doi.org/10.1177/1558689810382916 .

Center for Innovation in Teaching in Research. Choosing a mixed methods design. 2017. Retrieved 15 Nov 2017 from https://cirt.gcu.edu/research/developmentresources/research_ready/mixed_methods/choosing_design .

Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. 3rd ed. Thousand Oaks: Sage; 2018.

Curry LA, Krumholz HM, O’Cathain A, Clark VLP, Cherlin E, Bradley EH. Mixed methods in biomedical and health services research. Circ-Cardiovasc Qual Outcomes. 2013;6(1):119–23. https://doi.org/10.1161/circoutcomes.112.967885 .

Denscombe M. Communities of practice: a research paradigm for the mixed methods approach. J Mixed Methods Res. 2008;2(3):270–83.

Doyle L, Brady A-M, Byrne G. An overview of mixed methods research. J Res Nurs. 2009;14(2):175–85.

Foster NE, Bishop A, Bartlam B, Ogollah R, Barlas P, Holden M, … Young J. Evaluating acupuncture and standard carE for pregnant women with back pain (EASE back): a feasibility study and pilot randomised trial. Health Technol Assess. 2016;20(33):1–236. https://doi.org/10.3310/hta20330 .

Haider AH, Schneider EB, Kodadek LM, Adler RR, Ranjit A, Torain M, … Lau BD. Emergency department query for patient-centered approaches to sexual orientation and gender identity the EQUALITY study. JAMA Intern Med. 2017;177(6):819–828. https://doi.org/10.1001/jamainternmed.2017.0906 .

Harding KE, Taylor NF, Bowers B, Stafford M, Leggat SG. Clinician and patient perspectives of a new model of triage in a community rehabilitation program that reduced waiting time: a qualitative analysis. Aust Health Rev. 2013;37(3):324–30. https://doi.org/10.1071/ah13033 .

Hoddinott P, Britten J, Prescott GJ, Tappin D, Ludbrook A, Godden DJ. Effectiveness of policy to provide breastfeeding groups ( BIG) for pregnant and breastfeeding mothers in primary care: cluster randomised controlled trial. Br Med J. 2009;338:a3026, 10. https://doi.org/10.1136/bmj.a3026 .

Ivankova NV, Creswell JW, Stick SL. Using mixed-methods sequential explanatory design: from theory to practice. Field Methods. 2006;18(1):3–20. https://doi.org/10.1177/1525822x05282260 .

Johnson B, Gray R. A history of philosophical and theoretical issues for mixed methods research. In: Tashakkori A, Teddlie C, editors. Sage handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks: Sage; 2010. p. 69–94.

Chapter   Google Scholar  

Johnson BR, Onwuegbuzie AJ. Mixed methods research: a research paradigm whose time has come. Educ Res. 2004;33(7):14–26. https://doi.org/10.3102/0013189x033007014 .

Keeney S, McKenna H, Fleming P, McIlfatrick S. Attitudes to cancer and cancer prevention: what do people aged 35–54 years think? Eur J Cancer Care. 2010;19(6):769–77. https://doi.org/10.1111/j.1365-2354.2009.01137.x .

McBride KA, Ballinger ML, Schlub TE, Young MA, Tattersall MHN, Kirk J, et al. Psychosocial morbidity in TP53 mutation carriers: is whole-body cancer screening beneficial? Familial Cancer. 2017;16(3):423–32. https://doi.org/10.1007/s10689-016-9964-7 .

Mertens DM. Transformative mixed methods: addressing inequities. Am Behav Sci. 2012;56(6):802–13. https://doi.org/10.1177/0002764211433797 .

Moffatt S, White M, Mackintosh J, Howel D. Using quantitative and qualitative data in health services research – what happens when mixed method findings conflict? BMC Health Serv Res. 2006;6:28. https://doi.org/10.1186/1472-6963-6-28 .

Morse JM. Approaches to qualitative-quantitative methodological triangulation. Nurs Res. 1991a;40(2):120–3.

Morse JM. Principles of mixed methods and multimethod research design. In: Tashakkori A, Teddilie C, editors. SAGE handbook of mixed methods in social and behavioral research. Thousand Oaks: Sage; 1991b.

Mutrie N, Doolin O, Fitzsimons CF, Grant PM, Granat M, Grealy M, et al. Increasing older adults’ walking through primary care: results of a pilot randomized controlled trial. Fam Pract. 2012;29(6):633–42. https://doi.org/10.1093/fampra/cms038 .

Newman I, Ridenour C, Newman C, De Marco G. A typology of research purposes and its relationship to mixed methods in social and behavioral research. Thousand Oaks: Sage; 2003. p. 167–88.

O’Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. J Health Serv Res Policy. 2008;13(2):92–8. https://doi.org/10.1258/jhsrp.2007.007074 .

Onwuegbuzie AJ, Collins KMT. A typology of mixed methods sampling designs in social science research. Qual Rep. 2007;12(2):281–316.

Plano Clark VL, Badiee M. Research questions in mixed methods research. In: Tashakkori A, Teddlie C, editors. SAGE handbook of mixed methods in social and behavioral research. 2nd ed. Thousand Oaks: Sage; 2010. p. 275–304.

Prades, J., Algara, M., Espinas, J. A., Farrus, B., Arenas, M., Reyes, V., . . . Borras, J. M. (2017). Understanding variations in the use of hypofractionated radiotherapy and its specific indications for breast cancer: a mixed-methods study. Radiother Oncol, 123(1), 22–28. doi: https://doi.org/10.1016/j.radonc.2017.01.014 .

Tariq S, Woodman J. Using mixed methods in health research. JRSM Short Rep. 2013; 4 (6):1–8. https://doi.org/10.1177/2042533313479197 .

Tashakkori A, Creswell JW. Editorial: the new era of mixed methods. J Mixed Methods Res. 2007;1(1):3–7. https://doi.org/10.1177/2345678906293042 .

Tashakkori A, Teddlie C. The past and future of mixed methods research: from data triangulation to mixed model designs. In: Tashakkori A, Teddlie C, editors. Handbook of mixed methods in social & behavioral research. Thousand Oaks: Sage; 2003. p. 671–701.

Teddlie C, Tashakkori A. Foundations of mixed methods research: integrating quantitative and qualitative approaches in the social and behavioral sciences. Thousand Oaks: Sage; 2009.

Teddlie C, Yu F. Mixed methods sampling a typology with examples. J Mixed Methods Res. 2007;1(1):77–100. https://doi.org/10.1177/2345678906292430 .

Wellard SJ, Rasmussen B, Savage S, Dunning T. Exploring staff diabetes medication knowledge and practices in regional residential care: triangulation study. J Clin Nurs. 2013;22(13–14):1933–40. https://doi.org/10.1111/jocn.12043 .

Zhang W. Mixed methods application in health intervention research: a multiple case study. Int J Mult Res Approaches. 2014;8(1):24–35. https://doi.org/10.5172/mra.2014.8.1.24 .

Download references

Author information

Authors and affiliations.

School of Medicine and Translational Health Research Institute, Western Sydney University, Sydney, NSW, Australia

Kate A. McBride

School of Science and Health and Translational Health Research Institute (THRI), Western Sydney University, Penrith, NSW, Australia

Freya MacMillan

School of Science and Health, Western Sydney University, Sydney, NSW, Australia

Emma S. George

NICM and Translational Health Research Institute (THRI), Western Sydney University, Penrith, NSW, Australia

Genevieve Z. Steiner

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Kate A. McBride .

Editor information

Editors and affiliations.

School of Science and Health, Western Sydney University, Penrith, NSW, Australia

Pranee Liamputtong

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this entry

Cite this entry.

McBride, K.A., MacMillan, F., George, E.S., Steiner, G.Z. (2019). The Use of Mixed Methods in Research. In: Liamputtong, P. (eds) Handbook of Research Methods in Health Social Sciences. Springer, Singapore. https://doi.org/10.1007/978-981-10-5251-4_97

Download citation

DOI : https://doi.org/10.1007/978-981-10-5251-4_97

Published : 13 January 2019

Publisher Name : Springer, Singapore

Print ISBN : 978-981-10-5250-7

Online ISBN : 978-981-10-5251-4

eBook Packages : Social Sciences Reference Module Humanities and Social Sciences Reference Module Business, Economics and Social Sciences

Share this entry

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

The Ultimate Guide to Qualitative Research - Part 1: The Basics

• Introduction and overview
• What is qualitative research?
• What is qualitative data?
• Examples of qualitative data
• Qualitative vs. quantitative research
• Introduction

What is a mixed methods design?

Triangulation in mixed methods research, types of mixed methods research designs, using atlas.ti for mixed methods research.

• Qualitative research preparation
• Theoretical perspective
• Theoretical framework
• Literature reviews
• Research question
• Conceptual framework
• Conceptual vs. theoretical framework
• Data collection
• Qualitative research methods
• Focus groups
• Observational research
• Case studies
• Ethnographical research
• Ethical considerations
• Confidentiality and privacy
• Power dynamics
• Reflexivity

What is mixed methods research?

When starting the research process, researchers sometimes think they have to decide whether qualitative research or quantitative research is more appropriate for their research design. However, the more important question is whether the methods they employ in data collection and analysis sufficiently capture the phenomenon they want to study. In some cases, answering this question requires using multiple methods of research.

Mixed methods research is a research paradigm that involves collecting qualitative data and quantitative data on the same object of inquiry. Researchers who employ mixed methods research synthesize qualitative findings with quantitative findings to achieve a better understanding.

Let's look at the established research paradigms, then mixed methods research, why it's useful, and which research methods complement each other. Then we'll examine how ATLAS.ti can help you execute a mixed methods design.

Mixed methods research is followed out of the need to understand concepts or phenomena at a deep level. A standalone quantitative study or qualitative study can provide great insight. Still, one method alone may not be able to capture all knowledge necessary to fully understand a topic or issue.

Those who conduct mixed methods research acknowledge the importance of pursuing both qualitative and quantitative research to achieve more complete results. However, this is not simply an issue of collecting more data just for its own sake. Mixed methods design is purposeful in carefully crafting research questions and employing appropriate research methods to essentially fill in the gaps of knowledge surrounding a particular research inquiry.

To determine which methods and data can address particular research needs, let's look at the capabilities of and differences between qualitative and quantitative data collection .

Qualitative and quantitative data

Researchers are often quick to make conclusions about whether qualitative research is better than quantitative research or vice versa. The reality is that quantitative and qualitative data can both look at the world in different ways that are useful at various points of a research inquiry. Qualitative and quantitative research are established research paradigms precisely because they provide relevant insights with the appropriate research design, data collection, and analysis.

One of the main goals of qualitative research is to generate a description of a social phenomenon. When something is difficult to quantify, it needs to be broken down into more constituent elements that are, by themselves, easier to perceive. In educational evaluation, for example, it is difficult to evaluate good academic writing with just a single score alone. Writing teachers employ a rubric to measure writing by a number of aspects which may include argumentation, organization, and cohesion.

Qualitative methods of research tend to collect data for an analysis that is capable of generating frameworks of constituent elements. Such a framework can then be used in subsequent research, evaluation, or decision-making processes. Researchers can collect qualitative data from observations , interviews , or records searches. Qualitative data analysis then aims to identify patterns and themes frequently appearing in the collected data.

The efficacy of experimental drugs in clinical trials, for example, is seldom easy to measure through quantitative methods alone. Qualitative research methods are often employed to determine a research participant's well-being, emotional state of mind, and other factors to help researchers decide the overall success of their clinical trials.

Quantitative research

If qualitative methods describe a concept or phenomenon, quantitative methods employ the resulting framework to measure that concept or phenomenon. Quantitative research methodology takes the theories generated from qualitative findings to collect quantitative data that can be used to measure a concept or phenomenon at scale.

Ultimately, numbers and values inform decision-making processes in many contexts. Quantitative results are useful in research areas where precision is valued or required. Still, they are also used in social and behavioral research to numerically describe phenomena that may not appear to be naturally quantifiable.

Mixing methods

Quantitative and qualitative strands of research are often pitted against each other for various reasons. Researchers might shun qualitative data collection as it is often time-consuming. In contrast, quantitative data collection is often critiqued for its reductive power (i.e., reducing ambiguous concepts into simplistic numerical values). Many scholarly disciplines, as a result, tend to prefer one research paradigm over the other (e.g., chemistry tends toward quantitative data collection, while anthropology tends toward qualitative data collection).

In the long run of any sufficiently complex research inquiry, however, it is seldom necessary to remain confined to one research approach. The main objective of scientific research is to organize knowledge through theories about the world around us. As a result, researchers employ mixed methods to combine theory generation in qualitative research with confirmatory testing in quantitative research to ultimately produce a robust theory and new knowledge.

However, research studies that combine qualitative and quantitative methods for the sake of having multiple methods of data collection and analysis are not as persuasive or impactful as true mixed methods studies where research methods are purposefully chosen to achieve a better understanding.

An example of mixed methods research

The objective of mixed methods research designs is to employ different inquiry components under one larger study. However, it might be easier to think of mixed methods research designs as having at least one qualitative study and one quantitative study, each with related but ultimately separate research questions . Examining a mixed methods research design in this way might make it easier to understand the need for pursuing multiple methods in certain cases.

• Consider the following example:

Remote work performance and job satisfaction

- RQ1: How have work outputs at XYZ Company changed since the shift to fully remote work?

- RQ2: What perceptions do remote workers at XYZ Company have about the shift to fully remote work?

In general terms, the goal of the study is to examine the efficacy of remote work in comparison to traditional, in-office work at one company. Actually determining this efficacy requires looking at the phenomenon of remote work through different methods.

As a result, one possible mixed methods study might look at the performance metrics of the company. Research question 1 (RQ1) is posed to conduct a quantitative research study that collects data on possibly quantifiable concepts related to work (e.g., amount of sales generated, number of new clients acquired). In this case, the researchers collect quantitative data to compare post-remote work performance to pre-remote work performance and determine if productivity has changed over time.

While this is a useful angle to examine remote work, it does not tell the whole story. After all, if people at Company XYZ are more or less productive than before, what are the reasons that explain this change? To address research question 2 (RQ2), researchers collect qualitative data on the level of satisfaction employees have with their jobs. Qualitative data from interviews with employees can be used to determine which aspects of their job they find satisfying or not.

With all the data collected, mixed methods researchers can combine the initial quantitative results and the initial qualitative results to form a deeper understanding of their topic of inquiry. In this case, if the quantitative data shows that worker productivity has suffered since the switch to remote work, the qualitative data might illuminate the aspects of remote work that employees don't like.

Other mixed methods research examples

While there are many different forms of mixed-methods research, the research approach is generally the same across mixed-methods research designs. A mixed methods research design is likely to require researchers to collect quantitative and qualitative data relevant to an overarching topic that necessitates examination from different methods. A couple of examples are:

Literacy development among children

RQ1: What is the rate of literacy development among children at ABC School based on scores from a standardized reading test?

RQ2: What are the instructional practices common in classrooms with high-performing students on standardized reading tests?

Market research for a new computer model

RQ1: How much time does it take to complete a series of tasks on an experimental computer model compared to a comparable computer model?

RQ2: What factors do potential customers take into consideration when buying a new computer?

Notice that qualitative and quantitative data pursue related but ultimately different aspects of the phenomena under study. As a result, the discrete inquiries in a mixed methods study will most likely employ different methods to collect data.

Conduct research from conception to analysis with ATLAS.ti.

Download a free trial of ATLAS.ti to see how our software can aid you in the research process.

Researchers do not employ mixed methods research just for the sake of having different methods in one research inquiry. The objective behind mixing methods is to generate new knowledge and strengthen understanding of that knowledge by examining it from different angles. This is a concept in research called triangulation, which refers to affirming a given location based on measures taken from different points. The equivalent notion in research is that viewing the same object of inquiry from multiple angles will provide a more reliable understanding of that object.

To further understand the utility of a mixed methods approach, imagine you and your friends are looking at a merry-go-round. You can only see one part of it at any one time, while other parts are obscured from your view. On the other hand, if your friends are positioned to see the merry-go-round from different angles, your combined observations can capture a more complete picture of the object you are studying.

Mixed methods research relies on multiple research methods, data sets, or theoretical approaches to assemble a more comprehensive picture of a concept or phenomenon. Especially in qualitative research or social science research, any set of findings can be considered more credible if they are supported with evidentiary data that comes from different perspectives.

Method triangulation

Method triangulation involves combining qualitative and quantitative methods together to study different but related aspects. In this respect, quantitative and qualitative research study the same phenomenon to lend support to each method's findings. Note that the goal of triangulated mixed methods research is not to simply use multiple methods to arrive at the same answer but to generate a better understanding of a phenomenon that one method alone cannot sufficiently capture.

In this case, method triangulation is a useful concept for a mixed methods researcher because it requires them to acknowledge the strengths and weaknesses of each particular research method. At scale, quantitative methods cannot capture concepts that are unquantifiable (e.g., beauty, convenience). In contrast, qualitative methods often do not conduct data collection at scales necessary to make generalizations about phenomena. Integrating quantitative and qualitative research components under the same mixed methods design ensures a comprehensive examination of a phenomenon that one method alone cannot accomplish.

Ethnography provides ample opportunities to pursue method triangulation. Data collection in ethnographic research often involves collecting qualitative data through observations and interviews . In contrast, data analysis can assess quantitative data by identifying patterns in behavior and perspectives and determining their frequencies.

Another example is a mixed methods study that examines patient outcomes at a hospital. Initial qualitative results might come from field notes from observations of doctors and nurses and interview data with patients. The quantitative findings might come from conducting a statistical analysis of the money and resources used for each patient observed or interviewed to determine whether the expenditure is commensurate with the patient outcomes achieved.

A standalone quantitative study might look only at the financial aspects of health care, while a qualitative study might do better at examining the social and emotional aspects. Conducting both of these studies in tandem can help researchers determine actionable insights for streamlining health care services while maintaining satisfactory standards of care.

Data triangulation

Mixed methods research usually depends on method triangulation, but it's important to identify other forms of triangulation that can strengthen the findings in any research. A study that relies on data triangulation looks at different sets of data. For example, an educational researcher might examine student outcomes at different schools or at the same school but at different times. Data triangulation is useful in affirming that the findings in one context are applicable across other contexts.

Theory triangulation

Another kind of triangulation less commonly associated with mixed methods research deals with analyzing data using different theories. A sequential research design, for example, may use the initial quantitative results from a survey study to generate a conceptual framework for the analysis of a subsequent qualitative study. At the same time, existing theories may also be employed in that analysis to compare and contrasts the kinds of insights and outcomes that each may produce.

Theory generation in mixed methods research

Many forms of research seek to generate or develop a theoretical framework to understand the object of inquiry. There are two common forms of theory generation, and both can manifest in the research questions that are posed in any study.

Research questions can either be exploratory, which try to define or gain a greater understanding of a phenomenon, or confirmatory, which try to test a theory or hypothesis regarding that phenomenon. With some exceptions, exploratory research questions call for collecting qualitative data , while confirmatory research questions require quantitative data .

In that respect, common mixed methods designs combine qualitative and quantitative components to generate a theory and either strengthen or challenge that theory, respectively. To understand what that theory generation looks like when employing mixed methods, we need to examine some of the different kinds of mixed methods research designs.

Look at your data from all angles with ATLAS.ti.

Analyze your data quickly and efficiently with ATLAS.ti. Click here for a free trial.

Data collection and analysis in mixed methods research depends on the research design you adopt. Ultimately, it might be easy to think about the different research designs in terms of the timing of the discrete inquiries within a mixed methods inquiry.

Concurrent triangulation design

A study that collects quantitative and qualitative data simultaneously is a common form of mixed methods design to achieve triangulation. The goal of a concurrent triangulation design is to observe the object of inquiry from multiple methods.

For example, imagine an educational researcher who wants to examine the efficacy of an after-school reading program. The researcher can then pursue two concurrent studies, one that qualitatively observes the reading program in action between educators and students and another that quantitatively tests students' reading comprehension. Over time, the researcher can draw correlations between improvements in test scores and any observations of the students in the program.

Exploratory sequential design

Another way to look at mixed methods research is with the idea that data collection and analysis are cyclical and evolve as new knowledge is generated. Researchers might undertake an exploratory sequential design if they don't yet know the aspects of a concept or phenomenon they want to test. In short, they need to conduct a qualitative study first in order to generate a conceptual framework to apply in a subsequent quantitative study.

Exploratory sequential design is useful in market research, for example, to identify the potential needs and preferences of prospective customers. Focus group research with a group of target customers can inquire about what they are looking for when choosing from a line of products. The researcher can take the initial qualitative findings to inform the design of a subsequent survey study that can confirm the extent to which the preferences of the focus group are reflected in the larger market.

Researchers can also conduct a quantitative study to preface observations in a qualitative study. Imagine that an educational researcher is adopting mixed methods approaches when examining learning outcomes among schools within a given geographical area. They might start by examining test scores published by these schools, using the initial quantitative results to determine where students are struggling and might need intervention. The resulting qualitative study might conduct observations in struggling schools to determine potential shortcomings in teaching and learning.

Concurrent nested design

This research design involves conducting multiple inquiries at the same time for the purpose of using one inquiry to strengthen the other. In a mixed methods approach, concurrent nested design places one research paradigm within another (e.g., a quantitative study within a qualitative study).

Sequential transformative design

This is a mixed methods research design with a critical or social justice orientation, meaning that the research is ultimately conducted to challenge the understanding of existing theory or produce meaningful social change, respectively. In either case, a sequential mixed methods research design can have a transformative effect by employing one study to create the rationale for a second critical or social justice research inquiry.

As you employ multiple research methods for a single mixed methods research design, you might find that your data collection will involve large sets of data, presenting a challenge in managing all that information in an orderly manner. Whether you are conducting research through qualitative data collection, quantitative data collection, or both, ATLAS.ti can help you organize and analyze your data. A robust mixed methods approach requires systematic organization of your data collection to ensure efficient and insightful analysis.

Document groups

Data in ATLAS.ti is stored in documents, which can be classified by the data type they contain. ATLAS.ti allows you to analyze text, images, video, audio , and more, and each document's data type is marked in the Document Manager for easy organization.

However, you may also need to divide your documents by type of study or method employed. In that case, you can use Document Groups in ATLAS.ti to label your documents so your project has categories for quantitative and qualitative data, interviews and focus groups, observations and test scores. Documents can belong to multiple document groups, allowing for easy organization of documents into multiple categories.

Once you have fully coded your data , it might be a challenge to narrow down your analysis to the relevant data you're looking for. If you have to sift through large numbers of documents, the Query Tool can help you look for the most relevant quotations based on the codes you have applied to your data.

Global filters

Studies that employ mixed methods research can accumulate such vast amounts of qualitative and quantitative data that it might become cumbersome for the human eye to keep track of it all manually. Even the most organized project in ATLAS.ti can have thousands of documents or hundreds of codes, making it a challenge to find the right data.

In ATLAS.ti, you can set a global filter using any of the elements of your project. For example, if you have a document group labeled " interviews ," you can set a global filter for that document group, which will lead ATLAS.ti to only show the documents in that group.

Working with both qualitative and quantitative software

ATLAS.ti has a number of tools that provide visualizations to help illustrate quantitative findings. However, you may find that other software, such as Microsoft Excel or SPSS, can help you further analyze and visualize the quantitative research components in your study. As a result, ATLAS.ti allows you to export your analysis into a Microsoft Excel spreadsheet. The Code Co-Occurrence Analysis and Code-Document Analysis tools, for example, can export their resulting tables into Microsoft Excel, which includes tools for deeper statistical analysis or for creating other kinds of data visualizations.

ATLAS.ti projects can also be exported as syntax files that can be imported into other statistical analysis software such as SPSS and R. These files convert qualitative data into quantitative data for further statistical analyses, regressions, and quantitative visualizations. Researchers can fully realize the convergence between qualitative and quantitative research when using multiple software platforms to conduct their analysis.

From data collection to data analysis, rely on ATLAS.ti.

Start with a free trial of our software to conduct your mixed methods research.

• News & Highlights

• Publications and Documents
• Postgraduate Education
• Browse Our Courses
• C/T Research Academy
• K12 Investigator Training
• Translational Innovator
• SMART IRB Reliance Request
• Biostatistics Consulting
• Regulatory Support
• Pilot Funding
• Informatics Program
• Community Engagement
• Diversity Inclusion
• Research Enrollment and Diversity
• Harvard Catalyst Profiles

Community Engagement Program

Supporting bi-directional community engagement to improve the relevance, quality, and impact of research.

• Getting Started
• Resources for Equity in Research
• Community-Engaged Student Practice Placement
• Maternal Health Equity
• Youth Mental Health
• Leadership and Membership
• Past Members
• Study Review Rubric
• Community Ambassador Initiative
• Implementation Science Working Group
• Past Webinars & Podcasts
• Policy Atlas
• Community Advisory Board

For more information:

Mixed methods research.

According to the National Institutes of Health , mixed methods strategically integrates or combines rigorous quantitative and qualitative research methods to draw on the strengths of each. Mixed method approaches allow researchers to use a diversity of methods, combining inductive and deductive thinking, and offsetting limitations of exclusively quantitative and qualitative research through a complementary approach that maximizes strengths of each data type and facilitates a more comprehensive understanding of health issues and potential resolutions.¹ Mixed methods may be employed to produce a robust description and interpretation of the data, make quantitative results more understandable, or understand broader applicability of small-sample qualitative findings.

Integration

This refers to the ways in which qualitative and quantitative research activities are brought together to achieve greater insight. Mixed methods is not simply having quantitative and qualitative data available or analyzing and presenting data findings separately. The integration process can occur during data collection, analysis, or in the presentation of results.

¹ NIH Office of Behavioral and Social Sciences Research: Best Practices for Mixed Methods Research in the Health Sciences

Basic Mixed Methods Research Designs 

View image description .

Five Key Questions for Getting Started

• What do you want to know?
• What will be the detailed quantitative, qualitative, and mixed methods research questions that you hope to address?
• What quantitative and qualitative data will you collect and analyze?
• Which rigorous methods will you use to collect data and/or engage stakeholders?
• How will you integrate the data in a way that allows you to address the first question?

Rationale for Using Mixed Methods

• Obtain different, multiple perspectives: validation
• Build comprehensive understanding
• Explain statistical results in more depth
• Have better contextualized measures
• Track the process of program or intervention
• Study patient-centered outcomes and stakeholder engagement

Sample Mixed Methods Research Study

The EQUALITY study used an exploratory sequential design to identify the optimal patient-centered approach to collect sexual orientation data in the emergency department.

Qualitative Data Collection and Analysis : Semi-structured interviews with patients of different sexual orientation, age, race/ethnicity, as well as healthcare professionals of different roles, age, and race/ethnicity.

Builds Into : Themes identified in the interviews were used to develop questions for the national survey.

Quantitative Data Collection and Analysis : Representative national survey of patients and healthcare professionals on the topic of reporting gender identity and sexual orientation in healthcare.

Other Resources:

  Introduction to Mixed Methods Research : Harvard Catalyst’s eight-week online course offers an opportunity for investigators who want to understand and apply a mixed methods approach to their research.

Best Practices for Mixed Methods Research in the Health Sciences [PDF] : This guide provides a detailed overview of mixed methods designs, best practices, and application to various types of grants and projects.

Mixed Methods Research Training Program for the Health Sciences (MMRTP ): Selected scholars for this summer training program, hosted by Johns Hopkins’ Bloomberg School of Public Health, have access to webinars, resources, a retreat to discuss their research project with expert faculty, and are matched with mixed methods consultants for ongoing support.

Michigan Mixed Methods : University of Michigan Mixed Methods program offers a variety of resources, including short web videos and recommended reading.

To use a mixed methods approach, you may want to first brush up on your qualitative skills. Below are a few helpful resources specific to qualitative research:

• Qualitative Research Guidelines Project : A comprehensive guide for designing, writing, reviewing and reporting qualitative research.
• Fundamentals of Qualitative Research Methods – What is Qualitative Research : A six-module web video series covering essential topics in qualitative research, including what is qualitative research and how to use the most common methods, in-depth interviews, and focus groups.

View PDF of the above information.

• What is mixed methods research?

Last updated

20 February 2023

Reviewed by

Miroslav Damyanov

By blending both quantitative and qualitative data, mixed methods research allows for a more thorough exploration of a research question. It can answer complex research queries that cannot be solved with either qualitative or quantitative research .

Analyze your mixed methods research

Dovetail streamlines analysis to help you uncover and share actionable insights

Mixed methods research combines the elements of two types of research: quantitative and qualitative.

Quantitative data is collected through the use of surveys and experiments, for example, containing numerical measures such as ages, scores, and percentages. 

Qualitative data involves non-numerical measures like beliefs, motivations, attitudes, and experiences, often derived through interviews and focus group research to gain a deeper understanding of a research question or phenomenon.

Mixed methods research is often used in the behavioral, health, and social sciences, as it allows for the collection of numerical and non-numerical data.

• When to use mixed methods research

Mixed methods research is a great choice when quantitative or qualitative data alone will not sufficiently answer a research question. By collecting and analyzing both quantitative and qualitative data in the same study, you can draw more meaningful conclusions. 

There are several reasons why mixed methods research can be beneficial, including generalizability, contextualization, and credibility. 

For example, let's say you are conducting a survey about consumer preferences for a certain product. You could collect only quantitative data, such as how many people prefer each product and their demographics. Or you could supplement your quantitative data with qualitative data, such as interviews and focus groups , to get a better sense of why people prefer one product over another.

It is important to note that mixed methods research does not only mean collecting both types of data. Rather, it also requires carefully considering the relationship between the two and method flexibility.

You may find differing or even conflicting results by combining quantitative and qualitative data . It is up to the researcher to then carefully analyze the results and consider them in the context of the research question to draw meaningful conclusions.

When designing a mixed methods study, it is important to consider your research approach, research questions, and available data. Think about how you can use different techniques to integrate the data to provide an answer to your research question.

• Mixed methods research design

A mixed methods research design  is   an approach to collecting and analyzing both qualitative and quantitative data in a single study.

Mixed methods designs allow for method flexibility and can provide differing and even conflicting results. Examples of mixed methods research designs include convergent parallel, explanatory sequential, and exploratory sequential.

By integrating data from both quantitative and qualitative sources, researchers can gain valuable insights into their research topic . For example, a study looking into the impact of technology on learning could use surveys to measure quantitative data on students' use of technology in the classroom. At the same time, interviews or focus groups can provide qualitative data on students' experiences and opinions.

• Types of mixed method research designs

Researchers often struggle to put mixed methods research into practice, as it is challenging and can lead to research bias. Although mixed methods research can reveal differences or conflicting results between studies, it can also offer method flexibility.

Designing a mixed methods study can be broken down into four types: convergent parallel, embedded, explanatory sequential, and exploratory sequential.

Convergent parallel

The convergent parallel design is when data collection and analysis of both quantitative and qualitative data occur simultaneously and are analyzed separately. This design aims to create mutually exclusive sets of data that inform each other. 

For example, you might interview people who live in a certain neighborhood while also conducting a survey of the same people to determine their satisfaction with the area.

Embedded design

The embedded design is when the quantitative and qualitative data are collected simultaneously, but the qualitative data is embedded within the quantitative data. This design is best used when you want to focus on the quantitative data but still need to understand how the qualitative data further explains it.

For instance, you may survey students about their opinions of an online learning platform and conduct individual interviews to gain further insight into their responses.

Explanatory sequential design

In an explanatory sequential design, quantitative data is collected first, followed by qualitative data. This design is used when you want to further explain a set of quantitative data with additional qualitative information.

An example of this would be if you surveyed employees at a company about their satisfaction with their job and then conducted interviews to gain more information about why they responded the way they did.

Exploratory sequential design

The exploratory sequential design collects qualitative data first, followed by quantitative data. This type of mixed methods research is used when the goal is to explore a topic before collecting any quantitative data.

An example of this could be studying how parents interact with their children by conducting interviews and then using a survey to further explore and measure these interactions.

Integrating data in mixed methods studies can be challenging, but it can be done successfully with careful planning.

No matter which type of design you choose, understanding and applying these principles can help you draw meaningful conclusions from your research.

• Strengths of mixed methods research

Mixed methods research designs combine the strengths of qualitative and quantitative data, deepening and enriching qualitative results with quantitative data and validating quantitative findings with qualitative data. This method offers more flexibility in designing research, combining theory generation and hypothesis testing, and being less tied to disciplines and established research paradigms.

Take the example of a study examining the impact of exercise on mental health. Mixed methods research would allow for a comprehensive look at the issue from different angles. 

Researchers could begin by collecting quantitative data through surveys to get an overall view of the participants' levels of physical activity and mental health. Qualitative interviews would follow this to explore the underlying dynamics of participants' experiences of exercise, physical activity, and mental health in greater detail.

Through a mixed methods approach, researchers could more easily compare and contrast their results to better understand the phenomenon as a whole.  

Additionally, mixed methods research is useful when there are conflicting or differing results in different studies. By combining both quantitative and qualitative data, mixed methods research can offer insights into why those differences exist.

For example, if a quantitative survey yields one result while a qualitative interview yields another, mixed methods research can help identify what factors influence these differences by integrating data from both sources.

Overall, mixed methods research designs offer a range of advantages for studying complex phenomena. They can provide insight into different elements of a phenomenon in ways that are not possible with either qualitative or quantitative data alone. Additionally, they allow researchers to integrate data from multiple sources to gain a deeper understanding of the phenomenon in question.  

• Challenges of mixed methods research

Mixed methods research is labor-intensive and often requires interdisciplinary teams of researchers to collaborate. It also has the potential to cost more than conducting a stand alone qualitative or quantitative study . 

Interpreting the results of mixed methods research can be tricky, as it can involve conflicting or differing results. Researchers must find ways to systematically compare the results from different sources and methods to avoid bias.

For example, imagine a situation where a team of researchers has employed an explanatory sequential design for their mixed methods study. After collecting data from both the quantitative and qualitative stages, the team finds that the two sets of data provide differing results. This could be challenging for the team, as they must now decide how to effectively integrate the two types of data in order to reach meaningful conclusions. The team would need to identify method flexibility and be strategic when integrating data in order to draw meaningful conclusions from the conflicting results.

• Advanced frameworks in mixed methods research

Mixed methods research offers powerful tools for investigating complex processes and systems, such as in health and healthcare.

Besides the three basic mixed method designs—exploratory sequential, explanatory sequential, and convergent parallel—you can use one of the four advanced frameworks to extend mixed methods research designs. These include multistage, intervention, case study , and participatory. 

This framework mixes qualitative and quantitative data collection methods in stages to gather a more nuanced view of the research question. An example of this is a study that first has an online survey to collect initial data and is followed by in-depth interviews to gain further insights.

Intervention

This design involves collecting quantitative data and then taking action, usually in the form of an intervention or intervention program. An example of this could be a research team who collects data from a group of participants, evaluates it, and then implements an intervention program based on their findings .

This utilizes both qualitative and quantitative research methods to analyze a single case. The researcher will examine the specific case in detail to understand the factors influencing it. An example of this could be a study of a specific business organization to understand the organizational dynamics and culture within the organization.

Participatory

This type of research focuses on the involvement of participants in the research process. It involves the active participation of participants in formulating and developing research questions, data collection, and analysis.

An example of this could be a study that involves forming focus groups with participants who actively develop the research questions and then provide feedback during the data collection and analysis stages.

The flexibility of mixed methods research designs means that researchers can choose any combination of the four frameworks outlined above and other methodologies , such as convergent parallel, explanatory sequential, and exploratory sequential, to suit their particular needs.

Through this method's flexibility, researchers can gain multiple perspectives and uncover differing or even conflicting results when integrating data.

When it comes to integration at the methods level, there are four approaches.

Connecting involves collecting both qualitative and quantitative data during different phases of the research.

Building involves the collection of both quantitative and qualitative data within a single phase.

Merging involves the concurrent collection of both qualitative and quantitative data.

Embedding involves including qualitative data within a quantitative study or vice versa.

• Techniques for integrating data in mixed method studies

Integrating data is an important step in mixed methods research designs. It allows researchers to gain further understanding from their research and gives credibility to the integration process. There are three main techniques for integrating data in mixed methods studies: triangulation protocol, following a thread, and the mixed methods matrix.

Triangulation protocol

This integration method combines different methods with differing or conflicting results to generate one unified answer.

For example, if a researcher wanted to know what type of music teenagers enjoy listening to, they might employ a survey of 1,000 teenagers as well as five focus group interviews to investigate this. The results might differ; the survey may find that rap is the most popular genre, whereas the focus groups may suggest rock music is more widely listened to. 

The researcher can then use the triangulation protocol to come up with a unified answer—such as that both rap and rock music are popular genres for teenage listeners. 

Following a thread

This is another method of integration where the researcher follows the same theme or idea from one method of data collection to the next. 

A research design that follows a thread starts by collecting quantitative data on a specific issue, followed by collecting qualitative data to explain the results. This allows whoever is conducting the research to detect any conflicting information and further look into the conflicting information to understand what is really going on.

For example, a researcher who used this research method might collect quantitative data about how satisfied employees are with their jobs at a certain company, followed by qualitative interviews to investigate why job satisfaction levels are low. They could then use the results to explore any conflicting or differing results, allowing them to gain a deeper understanding of job satisfaction at the company. 

By following a thread, the researcher can explore various research topics related to the original issue and gain a more comprehensive view of the issue.

Mixed methods matrix

This technique is a visual representation of the different types of mixed methods research designs and the order in which they should be implemented. It enables researchers to quickly assess their research design and adjust it as needed. 

The matrix consists of four boxes with four different types of mixed methods research designs: convergent parallel, explanatory sequential, exploratory sequential, and method flexibility. 

For example, imagine a researcher who wanted to understand why people don't exercise regularly. To answer this question, they could use a convergent parallel design, collecting both quantitative (e.g., survey responses) and qualitative (e.g., interviews) data simultaneously.

If the researcher found conflicting results, they could switch to an explanatory sequential design and collect quantitative data first, then follow up with qualitative data if needed. This way, the researcher can make adjustments based on their findings and integrate their data more effectively.

Mixed methods research is a powerful tool for understanding complex research topics. Using qualitative and quantitative data in one study allows researchers to understand their subject more deeply. 

Mixed methods research designs such as convergent parallel, explanatory sequential, and exploratory sequential provide method flexibility, enabling researchers to collect both types of data while avoiding the limitations of either approach alone.

However, it's important to remember that mixed methods research can produce differing or even conflicting results, so it's important to be aware of the potential pitfalls and take steps to ensure that data is being correctly integrated. If used effectively, mixed methods research can offer valuable insight into topics that would otherwise remain largely unexplored.

What is an example of mixed methods research?

An example of mixed methods research is a study that combines quantitative and qualitative data. This type of research uses surveys, interviews, and observations to collect data from multiple sources.

Which sampling method is best for mixed methods?

It depends on the research objectives, but a few methods are often used in mixed methods research designs. These include snowball sampling, convenience sampling, and purposive sampling. Each method has its own advantages and disadvantages.

What is the difference between mixed methods and multiple methods?

Mixed methods research combines quantitative and qualitative data in a single study. Multiple methods involve collecting data from different sources, such as surveys and interviews, but not necessarily combining them into one analysis. Mixed methods offer greater flexibility but can lead to differing or conflicting results when integrating data.

Get started today

Go from raw data to valuable insights with a flexible research platform

Editor’s picks

Last updated: 21 December 2023

Last updated: 16 December 2023

Last updated: 6 October 2023

Last updated: 25 November 2023

Last updated: 12 May 2023

Last updated: 15 February 2024

Last updated: 11 March 2024

Last updated: 12 December 2023

Last updated: 18 May 2023

Last updated: 6 March 2024

Last updated: 10 April 2023

Last updated: 20 December 2023

Latest articles

Related topics, log in or sign up.

Get started for free

• Policy and Social Care Move Fast: How…

Policy and Social Care Move Fast: How Rapid Qualitative Methods Can Help Researchers Match Their Pace

Date posted:.

The field of social care integration, which refers to the study and implementation of clinically based programs to address the social needs of patients and families, is advancing at an increasingly rapid pace. This acceleration, driven by heightened need post-pandemic as well as mandates at the state and federal levels for health systems to implement screening and referral programs, has increased the urgency for high-quality evidence to support policy decisions about the delivery of social care—in other words, how health systems identify and address social needs, like access to healthy food and safe housing.  

Qualitative research is particularly useful in guiding social care integration as it can shed light on the patient or caregiver experience of participating in social care interventions, barriers to getting help that should be addressed, and appropriate next steps from the perspective of those directly impacted.

However, traditional qualitative data analysis can be time consuming, and evidence-based solutions for addressing families’ social needs from the clinical setting are needed in the short term. In this post, I’ll share how we adapted and applied rapid qualitative methods to a social care-focused study as an example of how this approach can be used to inform social care integration in real time.

Integrating a Rapid Research Approach

The Socially Equitable Care by Understanding Resource Engagement ( SECURE ) study is a mixed method pragmatic trial aimed at understanding how best to increase family-level engagement with social resources from the pediatric health care setting. Caregivers in the study were randomized to complete one of three different social assessments (surveys asking about their social circumstances and/or desire for social resources) before receiving a resource map on their personal smartphone where, if interested, they could search for community resources in their neighborhood. Caregivers also had the option of talking to our study-specific resource navigator to receive additional support finding resources.

The overall goal of the qualitative component of the study is to capture caregivers’ preferences and experiences receiving social care through SECURE. Our traditional qualitative protocol involved transcribing caregiver interviews verbatim, coding transcripts and conducting thematic analysis. Recognizing the need for implementation-oriented results on a fast timeline, our team explored rapid qualitative methodologies to supplement the traditional approach. The rapid methods we chose were derived from existing literature on rapid qualitative approaches, which were then adapted to suit our study’s protocol and the social care field in general.

In our rapid approach, interviewers took notes using a structured template during or immediately after each caregiver interview. The template was designed to capture the data most salient to social care integration efforts such as caregiver’s likes, dislikes and preferences about receiving social care at their child’s doctor’s office. Then, content from the templates was transposed onto an analytic matrix, where we compared data across participants to identify themes. While we explored the full range of themes that emerged from our caregiver interviews in traditional qualitative analysis, we wanted to be sure that rapid analysis focused on findings that would be most applicable to social care integration efforts so the results could inform social care policy at Children’s Hospital of Philadelphia (CHOP) and elsewhere in real time. For example, what parts of participating in SECURE were helpful for caregivers? Did anything make them uncomfortable?

To ensure that our rapid approach produced results in line with those generated through traditional methods, we analyzed ten of our interviews using both traditional and rapid methods and compared the results. This analysis yielded a 92.8% theme match—meaning the two qualitative methods yielded largely the same themes. This builds upon previous literature, indicating that rapid analysis can be an effective tool in capturing implementation-oriented themes from qualitative data.

How the SECURE Study Can Inform Future Research Efforts

Our rapid qualitative methods allowed us to effectively adapt and respond to the quickly evolving landscape of social care integration, even before we had the full study results. I personally saw this first-hand while working with the SECURE team in 2023 conducting caregiver interviews. For example, we were able to inform hospital efforts in response to a recent insurance requirement of health systems to share caregivers' responses to social screening questions. We successfully gathered patients’ feedback on this new requirement and shared this information and suggestions for what CHOP could do to make caregivers feel more comfortable answering social assessment questions.

While not intended to replace traditional qualitative analysis, being able to produce actionable qualitative findings in a timely manner through rapid methods has allowed SECURE findings to help shape social care interventions at CHOP and beyond in real time.

Our hope is that other researchers in social care who face time pressures may find similar rapid qualitative methods as a useful and effective approach to adapt to the dynamic nature of the field and generate family-centered solutions faster than would otherwise be possible.

Latest Blog Posts

The Heat Is Turned Up: Focus on Climate Solutions for Kids

2023 was the hottest year on record and 2024 is predicted to rank among the top five warmest years. Extreme heat at this scale can be a danger to not…

A Sense of Purpose Can Support Teen Mental Health

Adolescence–the years between about 10 to about 25–is an essential time to help young people proactively build positive mental health and well-being…

Q&A: Engaging Communities to Alleviate Period Poverty with Lynette Medley

There are 16.9 million women, girls and all other people who experience a menstrual cycle in the United States living in poverty, many of whom are…

Check out our new Publications   View Publications

Adolescent Health & Well-Being

Behavioral health, population health sciences, health equity, family & community health.

• Student Notebook

Mixed Methods Research

• Experimental Psychology
• Quantitative
• Statistical Analysis

Traditionally, there are three branches of methodology: quantitative (numeric data), qualitative (observational or interview data), and mixed methods (using both types of data). Psychology relies heavily on quantitative-based data analyses but could benefit from incorporating the advantages of both quantitative and qualitative methodologies into one cohesive framework. Mixed Methods (MM) ideally includes the benefits of both methods (Johnson, Onwuegbuzie, & Turner, 2007): Quantitative analyses employ descriptive and inferential statistics, whereas qualitative analyses produce expressive data that provide descriptive details (often in narrative form) to examine the study’s research objectives. Whereas quantitative data may be collected via measures such as self-reports and physiological tests, qualitative data are collected via focus groups, structured or semistructured interviews, and other forms (Creswell, 2013).

MM hypotheses differ in comparison with solely quantitative or qualitative research questions. Not only must the quantitative and qualitative data be integrated, but the hypotheses also must be integrated. MM practitioners promote the development of a theory-based set of three hypotheses. Hypotheses should be conducted a priori and be both logical and sequential research questions (for more information, see Onwuegbuzie & Leech, 2006). Specialists encourage researchers to construct three separate types of hypotheses for an MM research project. There can be more than three hypotheses but there must be at least one of each type. The first hypothesis should be quantitative and the second should be qualitative. The third hypothesis will be an MM hypothesis.

Integration of these data is often complex, even when there is a strong theoretical rationale for doing so. Data integration occurs when quantitative and qualitative are combined in a data set. There are multiple ways for this to occur, including triangulation, following a thread, and the mixed methods matrix (see O’Cathain, Murphy, & Nicholl, 2010, for a brief review). Yet understanding the overall reasoning for using MM and how to best combine the approaches in practice can help lessen the challenge of MM data integration (Bryman, 2006).

Types of MM Research

• There are dozens of MM designs, but for the purpose of this article, six MM designs will be presented:
• The sequential explanatory method employs two different data-collection time points; the quantitative data are collected first and the qualitative collected last.
• The sequential exploratory design is best for testing emergent theory because both types of data are interpreted during the data integration phase.
• The sequential transformative approach has no preference for sequencing of data collection and emphasizes theory.
• Concurrent triangulation is the ideal method for cross-validation studies and has only one point of data collection.
• The concurrent nested design is best used to gain perspectives on understudied phenomena.
• The concurrent transformative approach is theory driven and allows the researcher to examine phenomena on several different levels.

Strengths and Challenges of MM Research

An MM approach is helpful in that one is able to conduct in-depth research and, when using complementary MM, provide for a more meaningful interpretation of the data and phenomenon being examined (Teddlie & Tashakkori, 2003).  Another strength of MM is the dynamic between the qualitative and quantitative portions of the study. If the design is planned appropriately, each type of data can mirror the other’s findings, so the methodology can benefit many types of research. However, interpreting data using the MM framework can be complicated and time intensive given that the data and interpretations are often abstract. Additionally, conducting MM research requires training and mastery of the methodology, so there can be a learning curve for researchers who traditionally use only quantitative or qualitative methods. Sticking to the theory-based and evidence-based designs will aid in your understanding and interpretation of the data.

Qualitative Data Analysis

Qualitative coding is a multistep process that includes different types of analyses depending on the nature of your data. Codebooks are important before, during, and after qualitative coding due to the detailed nature of the qualitative data. It is also important to know your expected codes and themes in order to promote interrater reliability (Hruschka et al., 2004). Expected codes are based on the theoretical foundation of your project. I suggest including the expected codes and themes in your codebooks. As previously mentioned, research designs involving this type of data can vary greatly, but in general, the following is a framework of how to conduct a thematic data analysis: Know your data inside and out, generate codes, search for themes, and review themes with a research team (Braun & Clarke, 2006). For more detailed instructions on conducting a qualitative analysis, please refer to last month’s Student Notebook article (Heydarian, 2016).

Lessons Learned

From the start, the researcher or research team must have a clear idea of their resources and the pros and cons of each method. Researchers also must be flexible. I am interested in examining the factors that compose seeking health information online. To investigate this topic, I developed an online, two-part study. Information obtained from qualitative prompts was used to inform the development of a scale measuring health-information-seeking behavior online. The first study used MM, and the data collection occurred on Amazon Mechanical Turk, a marketplace where researchers can post their available studies. Potential participants are paid a small fee, and data collection usually is completed in less than a week. I expected to conduct magnitude coding — a type of qualitative coding that evaluates the emphasis of content — but instead I had to choose a more appropriate type of coding because the participants provided extremely brief responses.

In closing, the design of your study (quantitative, qualitative, or MM) should align with your training and your research objectives. MM has the potential to bring your research to the next level by combining the strengths of quantitative and qualitative methodologies.

Suggestions for Conducting MM Research

Be proficient in MM research by keeping up to date with the latest techniques, software, textbooks, and manuals.

Think “outside the box” and consider other data-analytic approaches that are not used in your field.

Choose the research design that best fits the hypotheses, and know the assumptions and limitations of that design.

Incorporate figures and tables into your qualitative codebook to deepen the conceptualizations for the coders and provide a few examples of already coded data in order to provide thorough instructions.

Create and use summary statements for each participant to help with the abstract portion of the analyses. Summary statements should be a few sentences that describe the participant’s statement and provide an overall gist of the available qualitative information.

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3 , 77–101. doi:10.1191/1478088706qp063oa

Bryman, A. (2006). Integrating quantitative and qualitative research: How is it done? Qualitative Research, 6 , 97–113. doi:10.1177/1468794106058877

Creswell, J. W. (2013). Research design: Qualitative, quantitative, and mixed methods approaches . Thousand Oaks, CA: Sage Publications.

Heydarian, N. (2016). Developing theory with the grounded-theory approach and thematic analysis. Observer, 29(4) , 38–39.

Hruschka, D. J., Schwartz, D., John, D. C. S., Picone-Decaro, E., Jenkins, R. A., & Carey, J. W. (2004). Reliability in coding open-ended data: Lessons learned from HIV behavioral research. Field Methods, 16 , 307–331. doi:10.1177/1525822X04266540

Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1 , 112–133. doi:10.1177/1558689806298224

O’Cathain, A., Murphy, E., & Nicholl, J. (2010). Three techniques for integrating data in mixed methods studies. BMJ, 341 , c4587. doi:10.1136/bmj.c4587

Onwuegbuzie, A. J., & Leech, N. L. (2006). Linking research questions to mixed methods data analysis procedures 1. The Qualitative Report, 11 , 474–498.

Teddlie, C., & Tashakkori, A. (2003). Major issues and controversies in the use of mixed methods in the social and behavioral sciences. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 3–50). Thousand Oaks, CA: Sage Publications.

VERY RELEVANT AND COMPREHENSIVE TEXT ON MM ETHODS

The analysis of mixed methods is fairly comprehensive and educative especially for scholars and/researchers who are used to the traditional Qualitative and Quantitatve research as a stand alone methodologies. I feel like looking for a workshop sponsor so that I can share these ideas to our colleagues in African universities generally and Kenya in particular. Our postgraduate students have not yet embrased the use of mixed methods. Four of my own supervised doctoral students have successfully used th MMR.We should do much more!

I am currently pursuing my PhD and using mixed method. I am interested in this combination of research methods.

I have gained much from the source which clearly spells out the strengths of MM and its applicability in research.

Iam conducting a sequential explanatory mixed methods study in PhD Management and I have benefited a lot from combining quantitative and qualitative research approaches operating with what works best per given research probem.

APS regularly opens certain online articles for discussion on our website. Effective February 2021, you must be a logged-in APS member to post comments. By posting a comment, you agree to our Community Guidelines and the display of your profile information, including your name and affiliation. Any opinions, findings, conclusions, or recommendations present in article comments are those of the writers and do not necessarily reflect the views of APS or the article’s author. For more information, please see our Community Guidelines .

Please login with your APS account to comment.

About the Author

Allyson S. Hughes is a Health Psychology doctoral student at The University of Texas at El Paso. Her research examines judgment and decision-making concerning health decisions using Internet resources. She can be reached at [email protected] .

Research Briefs

Recent highlights from APS journals articles on learning about working memory, psychological measurement, patterns of implicit and explicit attitudes, and much more.

Careers Up Close: Jolynn Pek on Quantifying Uncertainty

Pek analyzes sources of uncertainty in statistical results and develops new approaches to practicing replicable psychological science.

Speaking Evidence to Policy

The Commission on Evidence-Based Policymaking has issued a report that points to significant new opportunities in research and leadership for psychological scientists with expertise in areas including judgment and decision making, analysis and research design, data privacy and ethics issues, and more.

Privacy Overview

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Methodology
• Mixed Methods Research | Definition, Guide, & Examples

Mixed Methods Research | Definition, Guide, & Examples

Published on 4 April 2022 by Tegan George . Revised on 25 October 2022.

Mixed methods research combines elements of quantitative research and qualitative research in order to answer your research question . Mixed methods can help you gain a more complete picture than a standalone quantitative or qualitative study, as it integrates benefits of both methods.

Mixed methods research is often used in the behavioral, health, and social sciences, especially in multidisciplinary settings and complex situational or societal research.

• To what extent does the frequency of traffic accidents ( quantitative ) reflect cyclist perceptions of road safety ( qualitative ) in Amsterdam?
• How do student perceptions of their school environment ( qualitative ) relate to differences in test scores ( quantitative ) ?
• How do interviews about job satisfaction at Company X ( qualitative ) help explain year-over-year sales performance and other KPIs ( quantitative ) ?
• How can voter and non-voter beliefs about democracy ( qualitative ) help explain election turnout patterns ( quantitative ) in Town X?
• How do average hospital salary measurements over time (quantitative) help to explain nurse testimonials about job satisfaction (qualitative) ?

Table of contents

When to use mixed methods research, mixed methods research designs, benefits of mixed methods research, disadvantages of mixed methods research, frequently asked questions about mixed methods research.

Mixed methods research may be the right choice if your research process suggests that quantitative or qualitative data alone will not sufficiently answer your research question. There are several common reasons for using mixed methods research:

• Generalisability : Qualitative research usually has a smaller sample size , and thus is not generalisable . In mixed methods research, this comparative weakness is mitigated by the comparative strength of ‘large N’, externally valid quantitative research.
• Contextualisation: Mixing methods allows you to put findings in context and add richer detail to your conclusions. Using qualitative data to illustrate quantitative findings can help ‘put meat on the bones’ of your analysis.
• Credibility: Using different methods to collect data on the same subject can make your results more credible. If the qualitative and quantitative data converge, this strengthens the validity of your conclusions. This process is called triangulation .

As you formulate your research question , try to directly address how qualitative and quantitative methods will be combined in your study. If your research question can be sufficiently answered via standalone quantitative or qualitative analysis, a mixed methods approach may not be the right fit.

Keep in mind that mixed methods research doesn’t just mean collecting both types of data; you need to carefully consider the relationship between the two and how you’ll integrate them into coherent conclusions. Mixed methods can be very challenging to put into practice, so it’s a less common choice than standalone qualitative or qualitative research.

Prevent plagiarism, run a free check.

There are different types of mixed methods research designs . The differences between them relate to the aim of the research, the timing of the data collection , and the importance given to each data type.

As you design your mixed methods study, also keep in mind:

• Your research approach ( inductive vs deductive )
• Your research questions
• What kind of data is already available for you to use
• What kind of data you’re able to collect yourself.

Here are a few of the most common mixed methods designs.

Convergent parallel

In a convergent parallel design, you collect quantitative and qualitative data at the same time and analyse them separately. After both analyses are complete, compare your results to draw overall conclusions.

• On the qualitative side, you analyse cyclist complaints via the city’s database and on social media to find out which areas are perceived as dangerous and why.
• On the quantitative side, you analyse accident reports in the city’s database to find out how frequently accidents occur in different areas of the city.

In an embedded design, you collect and analyse both types of data at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.

This is a good approach to take if you have limited time or resources. You can use an embedded design to strengthen or supplement your conclusions from the primary type of research design.

Explanatory sequential

In an explanatory sequential design, your quantitative data collection and analysis occurs first, followed by qualitative data collection and analysis.

You should use this design if you think your qualitative data will explain and contextualise your quantitative findings.

Exploratory sequential

In an exploratory sequential design, qualitative data collection and analysis occurs first, followed by quantitative data collection and analysis.

You can use this design to first explore initial questions and develop hypotheses. Then you can use the quantitative data to test or confirm your qualitative findings.

‘Best of both worlds’ analysis

Combining the two types of data means you benefit from both the detailed, contextualised insights of qualitative data and the generalisable, externally valid insights of quantitative data. The strengths of one type of data often mitigate the weaknesses of the other.

For example, solely quantitative studies often struggle to incorporate the lived experiences of your participants, so adding qualitative data deepens and enriches your quantitative results.

Solely qualitative studies are often not very generalisable, only reflecting the experiences of your participants, so adding quantitative data can validate your qualitative findings.

Method flexibility

Mixed methods are less tied to disciplines and established research paradigms. They offer more flexibility in designing your research, allowing you to combine aspects of different types of studies to distill the most informative results.

Mixed methods research can also combine theory generation and hypothesis testing within a single study, which is unusual for standalone qualitative or quantitative studies.

Mixed methods research is very labour-intensive. Collecting, analysing, and synthesising two types of data into one research product takes a lot of time and effort, and often involves interdisciplinary teams of researchers rather than individuals. For this reason, mixed methods research has the potential to cost much more than standalone studies.

Differing or conflicting results

If your analysis yields conflicting results, it can be very challenging to know how to interpret them in a mixed methods study. If the quantitative and qualitative results do not agree or you are concerned you may have confounding variables , it can be unclear how to proceed.

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to test a hypothesis by systematically collecting and analysing data, while qualitative methods allow you to explore ideas and experiences in depth.

In mixed methods research , you use both qualitative and quantitative data collection and analysis methods to answer your research question .

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organisations.

Triangulation in research means using multiple datasets, methods, theories and/or investigators to address a research question. It’s a research strategy that can help you enhance the validity and credibility of your findings.

Triangulation is mainly used in qualitative research , but it’s also commonly applied in quantitative research . Mixed methods research always uses triangulation.

These are four of the most common mixed methods designs :

• Convergent parallel: Quantitative and qualitative data are collected at the same time and analysed separately. After both analyses are complete, compare your results to draw overall conclusions. 
• Embedded: Quantitative and qualitative data are collected at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.
• Explanatory sequential: Quantitative data is collected and analysed first, followed by qualitative data. You can use this design if you think your qualitative data will explain and contextualise your quantitative findings.
• Exploratory sequential: Qualitative data is collected and analysed first, followed by quantitative data. You can use this design if you think the quantitative data will confirm or validate your qualitative findings.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

George, T. (2022, October 25). Mixed Methods Research | Definition, Guide, & Examples. Scribbr. Retrieved 22 April 2024, from https://www.scribbr.co.uk/research-methods/mixed-methods/

Is this article helpful?

Tegan George

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 22 April 2024

The design and evaluation of gamified online role-play as a telehealth training strategy in dental education: an explanatory sequential mixed-methods study

• Chayanid Teerawongpairoj 1 ,
• Chanita Tantipoj 1 &
• Kawin Sipiyaruk 2  

Scientific Reports volume  14 , Article number:  9216 ( 2024 ) Cite this article

103 Accesses

Metrics details

• Health care
• Health services
• Public health

To evaluate user perceptions and educational impact of gamified online role-play in teledentistry as well as to construct a conceptual framework highlighting how to design this interactive learning strategy, this research employed an explanatory sequential mixed-methods design. Participants were requested to complete self-perceived assessments toward confidence and awareness in teledentistry before and after participating in a gamified online role-play. They were also asked to complete a satisfaction questionnaire and participate in an in-depth interview to investigate their learning experience. The data were analyzed using descriptive statistics, paired sample t-test, one-way analysis of variance, and framework analysis. There were 18 participants who completed self-perceived assessments and satisfaction questionnaire, in which 12 of them participated in a semi-structured interview. There were statistically significant increases in self-perceived confidence and awareness after participating in the gamified online role-play ( P  < 0.001). In addition, the participants were likely to be satisfied with this learning strategy, where usefulness was perceived as the most positive aspect with a score of 4.44 out of 5, followed by ease of use (4.40) and enjoyment (4.03). The conceptual framework constructed from the qualitative findings has revealed five key elements in designing a gamified online role-play, including learner profile, learning settings, pedagogical components, interactive functions, and educational impact. The gamified online role-play has demonstrated its potential in improving self-perceived confidence and awareness in teledentistry. The conceptual framework developed in this research could be considered to design and implement a gamified online role-play in dental education. This research provides valuable evidence on the educational impact of gamified online role-play in teledentistry and how it could be designed and implemented in dental education. This information would be supportive for dental instructors or educators who are considering to implement teledentistry training in their practice.

Similar content being viewed by others

Games in dental education: playing to learn or learning to play?

A bibliometric analysis of the use of the Gamification Octalysis Framework in training: evidence from Web of Science

Helping general dental practitioners use the Index of Orthodontic Treatment Need: an assessment of available educational apps

Introduction.

Telehealth has gained significant attention from various organization due to its potential to improve healthcare quality and accessibility 1 . It can be supportive in several aspects in healthcare, including medical and nursing services, to enhance continuous monitoring and follow-up 2 . Its adoption has increased substantially during the COVID-19 pandemic, aiming to provide convenient healthcare services 3 . Even though the COVID-19 outbreak has passed, many patients still perceive telehealth as an effective tool in reducing a number of visits and enhancing access to health care services 4 , 5 . This supports the use of telehealth in the post-COVID-19 era.

Teledentistry, a form of telehealth specific to dentistry, has been employed to improve access to dental services 6 . This system offers benefits ranging from online history taking, oral diagnosis, treatment monitoring, and interdisciplinary communication among dental professionals, enabling comprehensive and holistic treatment planning for patients 7 . Teledentistry can also reduce travel time and costs associated with dental appointments 8 , 9 , 10 . There is evidence that teledentistry serves as a valuable tool to enhance access to dental care for patients 11 . Additionally, in the context of long-term management in patients, telehealth has contributed to patient-centered care, by enhancing their surrounding environments 12 . Therefore, teledentistry should be emphasized as one of digital dentistry to enhance treatment quality.

Albeit the benefits of teledentistry, available evidence demonstrates challenges and concerns in the implementation of telehealth. Lack of awareness and knowledge in the use of telehealth can hinder the adoption of telehealth 13 . Legal issues and privacy concerns also emerge as significant challenges in telehealth use 14 . Moreover, online communication skills and technology literacy, including competency in using technological tools and applications, have been frequently reported as challenges in teledentistry 15 , 16 . Concerns regarding limitations stemming from the lack of physical examination are also significant 17 . These challenges and complexities may impact the accuracy of diagnosis and the security and confidentiality of patient information. Therefore, telehealth training for dental professionals emerges as essential prerequisites to effectively navigate the use of teledentistry, fostering confidence and competence in remote oral healthcare delivery.

The feasibility and practicality of telehealth in dental education present ongoing challenges and concerns. Given the limitations of teledentistry compared to face-to-face appointments, areas of training should encompass the telehealth system, online communication, technical issues, confidentiality concerns, and legal compliance 18 . However, there is currently no educational strategy that effectively demonstrates the importance and application of teledentistry 19 . A role-play can be considered as a teaching strategy where learners play a role that closely resembles real-life scenarios. A well-organized storytelling allows learner to manage problematic situations, leading to the development of problem-solving skill 20 , 21 . When compared to traditional lecture-based learning, learners can also enhance their communication skills through conversations with simulated patients 22 , 23 . In addition, they could express their thoughts and emotions during a role-play through experiential learning 20 , 24 , 25 . Role-play through video teleconference would be considered as a distance learning tool for training dental professionals to effectively use teledentistry.

While there have been studies supporting online role-play as an effective learning tool due to its impact of flexibility, engagement, and anonymity 26 , 27 , no evidence has been yet reported whether or not this learning strategy could have potential for training teledentistry. Given the complicated issues in telehealth, role-play for training teledentistry should incorporate different learning aspects compared to face-to-face communication with patients. In addition, game components have proved to be supportive in dental education 28 , 29 . Consequently, this research aimed to evaluate user perceptions and educational impact of gamified online role-play to enhance learner competence and awareness in using teledentistry as well as to construct a conceptual framework highlighting how to design and implement this interactive learning strategy. This research would introduce and promote the design and implementation of gamified online role-play as a learning tool for training teledentistry. To achieve the aim, specific objectives were established as follows:

1. To design a gamified online role-play for teledentistry training.

2. To investigate learner perceptions regarding their confidence and awareness in the use of teledentistry after completing the gamified online role-play.

3. To explore user satisfactions toward the use of gamified online role-play.

4. To develop a conceptual framework for designing and implementing a gamified online role-play for teledentistry training.

Materials and methods

Research design.

This research employed an explanatory sequential mixed-methods design, where a quantitative phase was firstly performed followed by a qualitative phase 30 , 31 . The quantitative phase was conducted based on pre-experimental research using one-group pretest–posttest design. Participants were requested to complete self-perceived assessments toward confidence and awareness in the use of teledentistry before and after participating in a gamified online role-play. They were also asked to complete a satisfaction questionnaire in using a gamified online role-play for training teledentistry. The qualitative phase was afterwards conducted to explore in-depth information through semi-structured interviews, in order to enhance an understanding of the quantitative phase, and to develop a conceptual framework for designing and implementing an online role-play for training teledentistry.

A gamified online role-play for training teledentistry

A gamified online role-play was designed and developed by the author team. To ensure its educational impact was significant, the expected learning outcomes were formulated based on insights gathered from a survey with experienced instructors from the Department of Advanced General Dentistry, Faculty of Dentistry, Mahidol University. These learning outcomes covered areas of online communication skill, technical issues, technology literacy of patients, limitations of physical examination, and privacy concerns of personal information. Learning scenario and instructional content were subsequently designed to support learners in achieving the expected learning outcomes, with their alignments validated by three experts in dental education. A professional actress underwent training to role-play a patient with a dental problem, requesting a virtual consultation or teledentistry. Before conducting data collection, the simulated patient was required to undergo a training and adjusting process with a pilot group under supervision of two experts in advanced general dentistry and dental education who had experience with teledentistry to ensure realism and completeness of learning content.

According to the role-play scenario, an actress was assigned to portray a 34-year-old female with chief complaints of pain around both ears, accompanied by difficulties in chewing food due to tooth loss. She was instructed to express her anxiety and nervousness about addressing these issues. Additionally, it was specified that she could not take a day off from work during this period. Despite this constraint, she required a dental consultation to receive advice for initial self-care, as her symptoms significantly impacted her daily life. Furthermore, she was designated to encounter difficulties with the technological use of the teledentistry platform.

The game components were implemented into the online role-play to enhance motivation and engagement. As challenge and randomness appear to be game elements 32 , 33 , five challenge cards were designed and embedded into the online role-play, where a participant was asked to randomly select one of them before interacting with the simulated patient. The challenging situations were potential technical concerns which could occur frequently during video conferencing, including network problems (e.g., internet disconnection and poor connection) and audiovisual quality issues. The participants were blinded to the selected card, while it was revealed to only the simulated patient. The challenging conditions were mimicked by the organizers and simulated patient, allowing learners to deal with difficulties. Therefore, both challenges and randomness were implemented into this learning intervention not only to create learning situations but also to enhance engagement.

A feedback system was carefully considered and implemented into the gamified online role-play. Immediate feedback appears to be a key feature of interactive learning environments 29 . Formative feedback was instantly delivered to learners through verbal and non-verbal communication, including words (content), tone of voice, facial expressions, and gestures of the simulated patient. This type of feedback allowed participants to reflect on whether or not their inputs were appropriate, enabling them to learn from their mistakes, or so-called the role of failure 34 . Summative feedback was also provided at the end of the role-play through a reflection from a simulated patient and suggestions from an instructor.

Learners were able to interact with the simulated patient using an online meeting room by Cisco WebEx. According to the research setting (Fig.  1 ), a learner was asked to participate in the role-play activity using a computer laptop in a soundproof room, while a simulated patient was arranged in a prepared location showing her residential environment. The researcher and instructor also joined the online meeting room and observed the interaction between the simulated patient and learners during the role-play activity whether or not all necessary information was accurately obtained. The role-play activity took around 30 minutes.

A diagram demonstrating the setting of gamified online role-play.

Research participants

Quantitative phase.

The participants in this research were postgraduate students from the Residency Training Program in Advanced General Dentistry at Mahidol University Faculty of Dentistry in academic year 2022, using a volunteer sampling. This program was selected because its objective was to develop graduates capable of integrating competencies from various dental disciplines to provide comprehensive dental care for both normal patients and those with special needs. Therefore, teledentistry should be a supportive component of their service. The recruitment procedure involved posting a recruiting text in the group chat of the residents. Those interested in participating in the research were informed to directly contact us to request more information, and they were subsequently allowed to decide whether they would like to participate. This approach ensured that participation was voluntary. Although there could be a non-response bias within this non-probability sampling technique 35 , it was considered as appropriate for this study, as participants were willing to have contribution in the learning activity, and therefore accurate and reliable research findings with no dropout could be achieved 36 .

The inclusion and exclusion criteria were established to determine the eligibility of prospective participants for this research. This study included postgraduate students from Years 1 to 3 in the Residency Training Program in Advanced General Dentistry at Mahidol University Faculty of Dentistry, enrolled during the academic year 2022. They were also required to at least complete the first semester to be eligible for this research to ensure familiarity with comprehensive dental care. However, they were excluded if they had previous involvement in the pilot testing of the gamified online role-play or if they were not fluent in the Thai language. The sample size was determined using a formula for two dependent samples (comparing means) 37 . To detect a difference in self-perceived confidence and awareness between pre- and post-assessments at a power of 90% and a level of statistical significance of 1%, five participants were required. With an assumed dropout rate of 20%, the number of residents per year (Year 1–3) was set to be 6. Therefore, 18 residents were required for this research.

Qualitative phase

The participants from the quantitative phase were selected for semi-structured interviews using a purposive sampling. This sampling method involved the selection of information-rich participants based on specific criteria deemed relevant to the research objective and to ensure a diverse representation of perspectives and experiences within the sample group 38 . In this research, the information considered for the purposive sampling included demographic data (e.g., sex and year of study), along with self-perceived assessment scores. By incorporating perceptions from a variety of participants, a broad spectrum of insights from different experiences in comprehensive dental practice and diverse improvement levels in self-perceived confidence and awareness could inform the design and implementation of the training program effectively. The sample size for this phase was determined based on data saturation, wherein interviews continued until no new information or emerging themes were retrieved. This method ensured thorough exploration of the research topic and maximized the richness of the qualitative data obtained.

Outcome assessments

To evaluate the gamified online role-play, a triangular design approach was employed, enabling the researchers to compare the research outcomes from different assessment methods. In this research, self-perceived assessments (confidence and awareness) in teledentistry, satisfactions toward gamified online role-play, and learner experience were assessed to assure the quality and feasibility of the gamified online role-play.

Self-perceived confidence and awareness toward teledentistry

All participants were requested to rate their perceptions of teledentistry before and after participating in the gamified online role-play (Supplementary material 1 ). The self-perceived assessment was developed based on previous literature 39 , 40 , 41 , 42 . The assessment scores would inform whether or not the participants could improve their self-perceived confidence and awareness through a learning activity. The assessment consisted of two parts, which were (1) self-perceived confidence and (2) self-perceived awareness. Each part contained six items, which were similar between the pre- and post-assessments. All items were designed using a 5-point Likert scale, where 1 being ‘strongly disagree’ and 5 being ‘strongly agree’.

Satisfactions toward the gamified online role-play

All participants were asked to complete the satisfaction questionnaire after participating in the gamified online role-play, to investigate whether or not they felt satisfied with their learning (Supplementary material 2 ). The questionnaire was developed based on previous literature regarding gamification and role-play 41 , 42 , 43 , 44 . Most of the items were designed using a 5-point Likert scale, where 1 being ‘very dissatisfied’ and 5 being ‘very satisfied’. They were grouped into three aspects, which were (1) Perceived usefulness, (2) Perceived ease of use, and (3) Perceived enjoyment.

Learner experiences within the gamified online role-play

Semi-structured interviews were conducted with the purposively selected participants to gather in-depth information regarding their learning experiences within the gamified online role-play. This technique allowed researchers to ask additional interesting topics raised from the responses of participants. A topic guide for interviews were constructed based on the findings of previous literature 45 , 46 , 47 . The interview was conducted in a private room by a researcher who was trained in conducting qualitative research including interviews. The interview sessions took approximately 45–60 minutes, where all responses from participants were recorded using a digital audio recorder with their permission. The recorded audios were transcribed using a verbatim technique by a transcription service under a confidential agreement.

Validity and reliability of data collection tools

To enhance the quality of self-perceived assessment and satisfaction questionnaire, they were piloted and revised to assure their validity and reliability. According to the content validity, three experts in advanced general dentistry were asked to evaluate the questionnaire, where problematic items were iteratively revised until they achieved the index of item-objective congruence (IOC) higher than 0.5. To perform a test–retest reliability, the validated versions of both self-perceived assessment and satisfaction questionnaire were afterwards piloted in residents from other programs, and the data were analyzed using an intraclass correlation coefficient (ICC), where the values of all items were 0.7 or greater. The data from the first pilot completion of both data collection tools were analyzed using Cronbach’s alpha to ensure the internal consistency of all constructs. The problematic items were deleted to achieve the coefficient alpha of 0.7 or greater for all constructs, which was considered as acceptable internal consistency.

Data analysis

The quantitative data retrieved from self-perceived assessment and satisfaction questionnaire were analyzed with the Statistical Package for Social Sciences software (SPSS, version 29, IBM Corp.). Descriptive statistics were performed to present an overview of the data. The scores from pre- and post-assessments were analyzed using a paired sample t-test to evaluate whether or not the participants would better self-perceive their confidence and awareness in teledentistry after participating in the gamified online role-play. One-way analysis of variance (ANOVA) was conducted to compare whether or not there were statistically significant differences in self-perceived assessment and satisfaction scores among the three academic years.

The qualitative data retrieved from semi-structured interviews were analyzed using a framework analysis, where its procedure involved transcription, familiarization with the interview data, coding, developing an analytical framework, indexing, charting, and data interpreting qualitative findings 48 . In this research, the initial codes had been pre-defined from previous literature and subsequently adjusted following the analysis of each transcript to develop an analytical framework (themes and subthemes), requiring several iterations until no additional codes emerged. Subsequently, the established categories and codes were applied consistently across all transcripts (indexing). The data from each transcript were then charted to develop a matrix, facilitating the management and summarization of qualitative findings. This method enabled the researchers to compare and contrast differences within the data and to identify connections between categories, thereby exploring their relationships and informing data interpretation.

The procedure of framework analysis necessitated a transparent process for data management and interpretation of emerging themes to ensure the robustness of research 49 . The transparency of this analytic approach enabled two researchers (C.Te. and K.S.) to independently analyze the qualitative data, and the emerging themes afterwards were discussed to obtain consensus among the researchers. This technique can be considered as a triangular approach to assure the intercoder reliability and internal validity of this research. The transparent process also allowed an external expert in dental education to verify the accuracy of the analysis. All emerging themes and the decision on data saturation were based on a discussion of all researchers until an agreement was made. NVivo (version 14, QSR International) was used to performed the qualitative data analysis. Subsequently, a conceptual framework was constructed to demonstrate emerging themes and subthemes together with their relationships.

Ethical consideration

The ethical approval for the study was approved by the Institutional Review Board of Faculty of Dentistry and Faculty of Pharmacy, Mahidol University on 29 th September 2022, the ethical approval number: MU-DT/PY-IRB 2022/049.2909. All methods were performed in accordance with the relevant guidelines and regulations. Although the data were not anonymous in nature as they contained identifiable data, they were coded prior to the analysis to assure confidentiality of participants.

Informed consent

Informed consent was obtained from all participants.

There were 18 residents from Year 1 to 3 of the Residency Training Program in Advanced General Dentistry who participated in this research (six from each year). Of these, there were 14 females and 4 males. There was no participant dropout, as all of them completed all required tasks, including the pre- and post-perceived assessments, gamified online role-play, and satisfaction questionnaire. According to the purposive sampling, the participants from the quantitative phase were selected for semi-structured interviews by considering sex, year of study, and self-perceived assessment scores. Twelve students (ten females and two males) participated in semi-structured interviews, where their characteristics are presented in Table 1 .

Internal consistency of all constructs

The data collected from the research participants, in addition to the pilot samples, were analyzed with Cronbach’s alpha to confirm the internal consistency. The coefficient alpha of all constructs demonstrated high internal consistency, as demonstrated in Table 2 .

Self-perceived assessments toward confidence and awareness of teledentistry

There were statistically significant increases in the assessment scores of self-perceived confidence and awareness after participating in the gamified online role-play ( P  < 0.001). According to Table 3 , there was an increase in self-perceived confidence from 3.38 (SD = 0.68) for the pre-assessment to 4.22 (SD = 0.59) for the post-assessment ( P  < 0.001). The findings of self-perceived awareness also showed score improvement from 4.16 (SD = 0.48) to 4.55 (SD = 0.38) after interacting with the simulated patient ( P  < 0.001).

According to Fig.  2 , participants demonstrated a higher level of self-perceived assessments for both self-confidence and awareness in all aspects after participating in the gamified online role-play for teledentistry training.

Self-perceived assessments toward confidence and awareness of teledentistry.

When comparing the self-perceived assessment scores toward confidence and awareness in the use of teledentistry among the three years of study (Year 1–3), there were no statistically significant differences in the pre-assessment, post-assessment score, and score difference (Table 4 ).

Satisfactions toward the use of gamified online role-play

According to Fig.  3 , participants exhibited high levels of satisfaction with the use of gamified online role-play across all three aspects. The aspect of usefulness received the highest satisfaction rating with a score of 4.44 (SD = 0.23) out of 5, followed by ease of use and enjoyment, scoring 4.40 (SD = 0.23) and 4.03 (SD = 0.21), respectively. Particularly, participants expressed the highest satisfaction levels regarding the usefulness of gamified online role-play for identifying their role (Mean = 4.72, SD = 0.46) and developing problem-solving skills associated with teledentistry (Mean = 4.61, SD = 0.50). Additionally, they reported satisfaction with the learning sequence presented in the gamified online role-play (Mean = 4.61, SD = 0.50). However, participants did not strongly perceive that the format of the gamified online role-play could engage them with the learning task for an extended period (Mean = 3.72, SD = 0.83).

Satisfactions toward the use of gamified online role-play.

When comparing the satisfaction levels perceived by participants from different academic years (Table 5 ), no statistically significant differences were observed among the three groups for all three aspects ( P  > 0.05).

Following the framework analysis of qualitative data, there were five emerging themes, including: (1) learner profile, (2) learning settings of the gamified online role-play, (3) pedagogical components, (4) interactive functions, and (5) educational impact.

Theme 1: Learner profile

Learner experience and preferences appeared to have impact on how the participants perceived the use of gamified online role-play for teledentistry training. When learners preferred role-play or realized benefits of teledentistry, they were likely to support this learning intervention. In addition, they could have seen an overall picture of the assigned tasks before participating in this research.

“I had experience with a role-play activity when I was dental undergraduates, and I like this kind of learning where someone role-plays a patient with specific personalities in various contexts. This could be a reason why I felt interested to participate in this task (the gamified online role-play). I also believed that it would be supportive for my clinical practice.” Participant 12, Year 1, Female “Actually, I' have seen in several videos (about teledentistry), where dentists were teaching patients to perform self-examinations, such as checking their own mouth and taking pictures for consultations. Therefore, I could have thought about what I would experience during the activity (within the gamified online role-play).” Participant 8, Year 2, Female

Theme 2: Learning settings of the gamified online role-play

Subtheme 2.1: location.

Participants had agreed that the location for conducting a gamified online role-play should be in a private room without any disturbances, enabling learners to focus on the simulated patient. This could allow them to effectively communicate and understand of the needs of patient, leading to a better grasp of lesson content. In addition, the environments of both learners and simulated patient should be authentic to the learning quality.

“The room should be a private space without any disturbances. This will make us feel confident and engage in conversations with the simulated patient.” Participant 10, Year 1, Female “… simulating a realistic environment can engage me to interact with the simulated patient more effectively ...” Participant 8, Year 2, Female

Subtheme 2.2: Time allocated for the gamified online role-play

The time allocated for the gamified online role-play in this research was considered as appropriate, as participants believed that a 30-minutes period should be suitable to take information and afterwards give some advice to their patient. In addition, a 10-minutes discussion on how they interact with the patient could be supportive for participants to enhance their competencies in the use of teledentistry.

“… it would probably take about 20 minutes because we would need to gather a lot of information … it might need some time to request and gather various information … maybe another 10-15 minutes to provide some advice.” Participant 7, Year 1, Female “I think during the class … we could allocate around 30 minutes for role-play, … we may have discussion of learner performance for 10-15 minutes ... I think it should not be longer than 45 minutes in total.” Participant 6, Year 2, Female

Subtheme 2.3: Learning consequence within a postgraduate curriculum

Most participants suggested that the gamified online role-play in teledentistry should be arranged in the first year of their postgraduate program. This could maximize the effectiveness of online role-play, as they would be able to implement teledentistry for their clinical practice since the beginning of their training. However, some participants suggested that this learning approach could be rearranged in either second or third year of the program. As they already had experience in clinical practice, the gamified online role-play would reinforce their competence in teledentistry.

"Actually, it would be great if this session could be scheduled in the first year … I would feel more comfortable when dealing with my patients through an online platform." Participant 11, Year 2, Male "I believe this approach should be implemented in the first year because it allows students to be trained in teledentistry before being exposed to real patients. However, if this approach is implemented in either the second or third year when they have already had experience in patient care, they would be able to better learn from conversations with simulated patients." Participant 4, Year 3, Male

Theme 3: Pedagogical components

Subtheme 3.1: learning content.

Learning content appeared to be an important component of pedagogical aspect, as it would inform what participants should learn from the gamified online role-play. Based on the interview data, participants reported they could learn how to use a video teleconference platform for teledentistry. The conditions of simulated patient embedded in an online role-play also allowed them to realize the advantages of teledentistry. In addition, dental problems assigned to the simulated patient could reveal the limitations of teledentistry for participants.

“The learning tasks (within the gamified online role-play) let me know how to manage patients through the teleconference.” Participant 5, Year 2, Female “… there seemed to be limitations (of teledentistry) … there could be a risk of misdiagnosis … the poor quality of video may lead to diagnostic errors … it is difficult for patients to capture their oral lesions.” Participant 3, Year 2, Female

Subtheme 3.2: Feedback

During the use of online role-play, the simulated patient can provide formative feedback to participants through facial expressions and tones of voice, enabling participants to observe and learn to adjust their inquiries more accurately. In addition, at the completion of the gamified online role-play, summative feedback provided by instructors could summarize the performance of participants leading to further improvements in the implementation of teledentistry.

“I knew (whether or not I interacted correctly) from the gestures and emotions of the simulated patient between the conversation. I could have learnt from feedback provided during the role-play, especially from the facial expressions of the patient.” Participant 11, Year 2, Male “The feedback provided at the end let me know how well I performed within the learning tasks.” Participant 2, Year 1, Female

Theme 4: Interactive functions

Subtheme 4.1: the authenticity of the simulated patient.

Most participants believed that a simulated patient with high acting performance could enhance the flow of role-play, allowing learners to experience real consequences. The appropriate level of authenticity could engage learners with the learning activity, as they would have less awareness of time passing in the state of flow. Therefore, they could learn better from the gamified online role-play.

"It was so realistic. ... This allowed me to talk with the simulated patient naturally ... At first, when we were talking, I was not sure how I should perform … but afterwards I no longer had any doubts and felt like I wanted to explain things to her even more." Participant 3, Year 2, Female "At first, I believed that if there was a factor that could influence learning, it would probably be a simulated patient. I was impressed by how this simulated patient could perform very well. It made the conversation flow smoothly and gradually." Participant 9, Year 3, Female

Subtheme 4.2: Entertaining features

Participants were likely to be satisfied with the entertaining features embedded in the gamified online role-play. They felt excited when they were being exposed to the unrevealed challenge which they had randomly selected. In addition, participants suggested to have more learning scenarios or simulated patients where they could randomly select to enhance randomness and excitement.

“It was a playful experience while communicating with the simulated patient. There are elements of surprise from the challenge cards that make the conversation more engaging, and I did not feel bored during the role-play.” Participant 4, Year 3, Male “I like the challenge card we randomly selected, as we had no idea what we would encounter … more scenarios like eight choices and we can randomly choose to be more excited. I think we do not need additional challenge cards, as some of them have already been embedded in patient conditions.” Participant 5, Year 2, Female

Subtheme 4.3: Level of difficulty

Participants suggested the gamified online role-play to have various levels of difficulty, so learners could have a chance to select a suitable level for their competence. The difficulties could be represented through patient conditions (e.g., systemic diseases or socioeconomic status), personal health literacy, and emotional tendencies. They also recommended to design the gamified online role-play to have different levels where learners could select an option that is suitable for them.

“The patient had hidden their information, and I needed to bring them out from the conversation.” Participant 12, Year 1, Female “Patients' emotions could be more sensitive to increase level of challenges. This can provide us with more opportunities to enhance our management skills in handling patient emotions.” Participant 11, Year 2, Male “… we can gradually increase the difficult level, similar to playing a game. These challenges could be related to the simulated patient, such as limited knowledge or difficulties in communication, which is likely to occur in our profession.” Participant 6, Year 2, Female

Theme 5: Educational impact

Subtheme 5.1: self-perceived confidence in teledentistry, communication skills.

Participants were likely to perceive that they could learn from the gamified online role-play and felt more confident in the use of teledentistry. This educational impact was mostly achieved from the online conversation within the role-play activity, where the participants could improve their communication skills through a video teleconference platform.

“I feel like the online role-play was a unique form of learning. I believe that I gained confidence from the online communication the simulated patient. I could develop skills to communicate effectively with real patients.” Participant 11, Year 2, Male “I believe it support us to train communication skills ... It allowed us to practice both listening and speaking skills more comprehensively.” Participant 4, Year 3, Male

Critical thinking and problem-solving skills

In addition to communication skills, participants reported that challenges embedded in the role-play allowed them to enhance critical thinking and problem-solving skills, which were a set of skills required to deal with potential problems in the use of teledentistry.

"It was a way of training before experiencing real situations … It allowed us to think critically whether or not what we performed with the simulated patients was appropriate." Participant 7, Year 1, Female “It allowed us to learn how to effectively solve the arranged problems in simulated situation. We needed to solve problems in order to gather required information from the patient and think about how to deliver dental advice through teledentistry.” Participant 11, Year 2, Male

Subtheme 5.2: Self perceived awareness in teledentistry

Participants believed that they could realize the necessity of teledentistry from the gamified online role-play. The storytelling or patient conditions allowed learners to understand how teledentistry could have both physical and psychological support for dental patients.

“From the activity, I would consider teledentistry as a convenient tool for communicating with patients, especially if a patient cannot go to a dental office”. Participant 5, Year 2, Female “I learned about the benefits of teledentistry, particularly in terms of follow-up. The video conference platform could support information sharing, such as drawing images or presenting treatment plans, to patients.” Participant 8, Year 2, Female

A conceptual framework of learning experience within a gamified online role-play

Based on the qualitative findings, a conceptual framework was developed in which a gamified online role-play was conceptualized as a learning strategy in supporting learners to be able to implement teledentistry in their clinical practice (Fig.  4 ).

The conceptual framework of key elements in designing a gamified online role-play.

The conceptual framework has revealed key elements to be considered in designing a gamified online role-play. Learner profile, learning settings, pedagogical components, and interactive functions are considered as influential factors toward user experience within the gamified online role-play. The well-designed learning activity will support learners to achieve expected learning outcomes, considered as educational impact of the gamified online role-play. The contributions of these five key elements to the design of gamified online role-play were interpreted, as follows:

Learner profile: This element tailors the design of gamified online role-plays for teledentistry training involves considering the background knowledge, skills, and experiences of target learners to ensure relevance and engagement.

Learning settings: The element focuses the planning for gamified online role-plays in teledentistry training involves selecting appropriate contexts, such as location and timing, to enhance accessibility and achieve learning outcomes effectively.

Pedagogical components: This element emphasizes the alignment between learning components and learning outcomes within gamified online role-plays, to ensure that the content together with effective feedback design can support learners in improving their competencies from their mistakes.

Interactive functions: This element highlights interactivity features integrated into gamified online role-plays, such as the authenticity and entertaining components to enhance immersion and engagement, together with game difficulty for optimal flow. All these features should engage learners with the learning activities until the achievement of learner outcomes.

Educational impact: This element represents the expected learning outcomes, which will inform the design of learning content and activities within gamified online role-plays. In addition, this element could be considered to evaluate the efficacy of gamified online role-plays, reflecting how well learning designs align with the learning outcomes.

A gamified online role-play can be considered as a learning strategy for teledentistry according to its educational impact. This pedagogical approach could mimic real-life practice, where dental learners could gain experience in the use of teledentistry in simulated situations before interacting with actual patients. Role-play could provide learners opportunities to develop their required competencies, especially communication and real-time decision-making skills, in a predictable and safe learning environment 20 , 23 , 46 . Potential obstacles could also be arranged for learners to deal with, leading to the enhancement of problem-solving skill 50 . In addition, the recognition of teledentistry benefits can enhance awareness and encourage its adoption and implementation, which could be explained by the technology acceptance model 51 . Therefore, a gamified online role-play with a robust design and implementation appeared to have potential in enhancing self-perceived confidence and awareness in the use of teledentistry.

The pedagogical components comprised learning content, which was complemented by assessment and feedback. Learners could develop their competence with engagement through the learning content, gamified by storytelling of the online role-play 52 , 53 . Immediate feedback provided through facial expression and voice tone of simulated patients allowed participants to learn from their failure, considered as a key feature of game-based learning 29 , 45 . The discussion of summative feedback provided from an instructor at the end of role-play activity could support a debriefing process enabling participants to reflect their learning experience, considered as important of simulation-based game 54 . These key considerations should be initially considered in the design of gamified online role-play.

The interactive functions can be considered as another key component for designing and evaluating the gamified online role-play 45 . Several participants enjoyed with a learning process within the gamified online role-play and suggested it to have more learning scenarios. In other words, this tool could engage learners with an instructional process, leading to the achievement of learning outcomes 29 , 45 . As challenge and randomness appear to be game elements 32 , 33 , this learning intervention assigned a set of cards with obstacle tasks for learners to randomly pick up before interacting with simulated patients, which was perceived by participants as a feature to make the role-play more challenging and engaging. This is consistent with previous research, where challenging content for simulated patients could make learners more engaged with a learning process 55 . However, the balance between task challenges and learner competencies is certainly required for the design of learning activities 56 , 57 . The authenticity of simulated patient and immediate feedback could also affect the game flow, leading to the enhancement of learner engagement 45 . These elements could engage participants with a learning process, leading to the enhancement of educational impact.

The educational settings for implementing gamified online role-play into dental curriculum should be another concern. This aspect has been recognized as significant in existing evidence 45 . As this research found no significant differences in all aspects among the three groups of learners, this learning intervention demonstrated the potential for its implementation at any time of postgraduate dental curriculum. This argument can be supported by previous evidence where a role-play could be adaptable for learning at any time, as it requires a short learning period but provides learners with valuable experience prior to being exposed in real-life scenarios 58 . This strategy also provides opportunities for learners who have any question or concern to seek advice or guidance from their instructors 59 . Although the gamified online role-play can be arranged in the program at any time, the first academic year should be considered, as dental learners would be confidence in implementing teledentistry for their clinical practice.

While a gamified online role-play demonstrated its strengths as an interactive learning strategy specifically for teledentistry, there are a couple of potential drawbacks that need to be addressed. The requirement for synchronous participation could limit the flexibility of access time for learners (synchronous interactivity limitation). With only one learner able to engage with a simulated patient at a time (limited participants), more simulated patients would be required if there are a number of learners, otherwise they would need to wait for their turn. Time and resources are significantly required for preparing simulated patients 60 . Despite the use of trained and calibrated professional actors/actresses, inauthenticity may be perceived during role-plays, requiring a significant amount of effort to achieve both interactional and clinical authenticities 46 . Future research could investigate asynchronous learning approaches utilizing non-player character (NPC) controlled by an artificial intelligence system as a simulated patient. This setup would enable multiple learners to have the flexibility to engage with the material at their own pace and at times convenient to them 29 . While there are potential concerns about using gamified online role-plays, this interactive learning intervention offers opportunities for dental professionals to enhance their teledentistry competency in a safe and engaging environment.

Albeit the robust design and data collection tools to assure reliability and validity as well as transparency of this study, a few limitations were raised leading to a potential of further research. While this research recruited only postgraduate students to evaluate the feasibility of gamified online role-play in teledentistry training, further research should include not only experienced dental practitioners but also undergraduate students to confirm its potential use in participants with different learner profiles. More learning scenarios in other dental specialties should also be included to validate its effectiveness, as different specialties could have different limitations and variations. Additional learning scenarios from various dental disciplines should be considered to validate the effectiveness of gamified online role-plays, as different specialties may present unique limitations and variations. A randomized controlled trial with robust design should be required to compare the effectiveness of gamified online role-play with different approaches in training the use of teledentistry.

Conclusions

This research supports the design and implementation of a gamified online role-play in dental education, as dental learners could develop self-perceived confidence and awareness with satisfaction. A well-designed gamified online role-play is necessary to support learners to achieve expected learning outcomes, and the conceptual framework developed in this research can serve as a guidance to design and implement this interactive learning strategy in dental education. However, further research with robust design should be required to validate and ensure the educational impact of gamified online role-play in dental education. Additionally, efforts should be made to develop gamified online role-play in asynchronous learning approaches to enhance the flexibility of learning activities.

Data availability

The data that support the findings of this study are available from the corresponding author, up-on reasonable request. The data are not publicly available due to information that could compromise the privacy of research participants.

Van Dyk, L. A review of telehealth service implementation frameworks. Int. J. Environ. Res. Public Health 11 (2), 1279–1298 (2014).

Article   PubMed   PubMed Central   Google Scholar  

Bartz, C. C. Nursing care in telemedicine and telehealth across the world. Soins. 61 (810), 57–59 (2016).

Article   PubMed   Google Scholar  

Lin, G.S.S., Koh, S.H., Ter, K.Z., Lim, C.W., Sultana, S., Tan, W.W. Awareness, knowledge, attitude, and practice of teledentistry among dental practitioners during COVID-19: A systematic review and meta-analysis. Medicina (Kaunas). 58 (1), 130 (2022).

Wolf, T.G., Schulze, R.K.W., Ramos-Gomez, F., Campus, G. Effectiveness of telemedicine and teledentistry after the COVID-19 pandemic. Int. J. Environ. Res. Public Health. 19 (21), 13857 (2022).

Gajarawala, S. N. & Pelkowski, J. N. Telehealth benefits and barriers. J. Nurse Pract. 17 (2), 218–221 (2021).

Jampani, N. D., Nutalapati, R., Dontula, B. S. & Boyapati, R. Applications of teledentistry: A literature review and update. J. Int. Soc. Prev. Community Dent. 1 (2), 37–44 (2011).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Khan, S. A. & Omar, H. Teledentistry in practice: literature review. Telemed. J. E. Health. 19 (7), 565–567 (2013).

Baheti, M. J. B. S., Toshniwal, N. G. & Misal, A. Teledentistry: A need of the era. Int. J. Dent. Med. Res. 1 (2), 80–91 (2014).

Google Scholar  

Datta, N., Derenne, J., Sanders, M. & Lock, J. D. Telehealth transition in a comprehensive care unit for eating disorders: Challenges and long-term benefits. Int. J. Eat. Disord. 53 (11), 1774–1779 (2020).

Bursell, S. E., Brazionis, L. & Jenkins, A. Telemedicine and ocular health in diabetes mellitus. Clin. Exp. Optom. 95 (3), 311–327 (2012).

da Costa, C. B., Peralta, F. D. S. & Ferreira de Mello, A. L. S. How has teledentistry been applied in public dental health services? An integrative review. Telemed. J. E. Health. 26 (7), 945–954 (2020).

Heckemann, B., Wolf, A., Ali, L., Sonntag, S. M. & Ekman, I. Discovering untapped relationship potential with patients in telehealth: A qualitative interview study. BMJ Open. 6 (3), e009750 (2016).

Pérez-Noboa, B., Soledispa-Carrasco, A., Padilla, V. S. & Velasquez, W. Teleconsultation apps in the COVID-19 pandemic: The case of Guayaquil City, Ecuador. IEEE Eng. Manag. Rev. 49 (1), 27–37 (2021).

Article   Google Scholar  

Wamsley, C. E., Kramer, A., Kenkel, J. M. & Amirlak, B. Trends and challenges of telehealth in an academic institution: The unforeseen benefits of the COVID-19 global pandemic. Aesthetic Surg. J. 41 (1), 109–118 (2020).

Jonasdottir, S. K., Thordardottir, I. & Jonsdottir, T. Health professionals’ perspective towards challenges and opportunities of telehealth service provision: A scoping review. Int. J. Med. Inform. 167 , 104862 (2022).

Tan, S. H. X., Lee, C. K. J., Yong, C. W. & Ding, Y. Y. Scoping review: Facilitators and barriers in the adoption of teledentistry among older adults. Gerodontology. 38 (4), 351–365 (2021).

Minervini, G. et al. Teledentistry in the management of patients with dental and temporomandibular disorders. BioMed. Res. Int. 2022 , 7091153 (2022).

Edirippulige, S. & Armfield, N. Education and training to support the use of clinical telehealth: A review of the literature. J. Telemed. Telecare. 23 (2), 273–282 (2017).

Article   CAS   PubMed   Google Scholar  

Mariño, R. & Ghanim, A. Teledentistry: A systematic review of the literature. J. Telemed. Telecare. 19 (4), 179–183 (2013).

Article   ADS   PubMed   Google Scholar  

Armitage-Chan, E. & Whiting, M. Teaching professionalism: Using role-play simulations to generate professionalism learning outcomes. J. Vet. Med. Educ. 43 (4), 359–363 (2016).

Spyropoulos, F., Trichakis, I. & Vozinaki, A.-E. A narrative-driven role-playing game for raising flood awareness. Sustainability. 14 (1), 554 (2022).

Jiang, W. K. et al. Role-play in endodontic teaching: A case study. Chin. J. Dent. Res. 23 (4), 281–288 (2020).

PubMed   Google Scholar  

Vizeshfar, F., Zare, M. & Keshtkaran, Z. Role-play versus lecture methods in community health volunteers. Nurse Educ. Today. 79 , 175–179 (2019).

Nestel, D. & Tierney, T. Role-play for medical students learning about communication: Guidelines for maximising benefits. BMC Med. Educ. 7 , 3 (2007).

Gelis, A. et al. Peer role-play for training communication skills in medical students: A systematic review. Simulat. Health. 15 (2), 106–111 (2020).

Cornelius, S., Gordon, C. & Harris, M. Role engagement and anonymity in synchronous online role play. Int. Rev. Res. Open Distrib. Learn. 12 (5), 57–73 (2011).

Bell, M. Online role-play: Anonymity, engagement and risk. Educ. Med. Int. 38 (4), 251–260 (2001).

Sipiyaruk, K., Gallagher, J. E., Hatzipanagos, S. & Reynolds, P. A. A rapid review of serious games: From healthcare education to dental education. Eur. J. Dent. Educ. 22 (4), 243–257 (2018).

Sipiyaruk, K., Hatzipanagos, S., Reynolds, P. A. & Gallagher, J. E. Serious games and the COVID-19 pandemic in dental education: An integrative review of the literature. Computers. 10 (4), 42 (2021).

Morse, J.M., Niehaus, L. Mixed Method Design: Principles and Procedures. (2016).

Creswell, J. W. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches 3rd edn. (SAGE Publications, 2009).

Cheng, V. W. S., Davenport, T., Johnson, D., Vella, K. & Hickie, I. B. Gamification in apps and technologies for improving mental health and well-being: Systematic review. JMIR Ment. Health. 6 (6), e13717 (2019).

Gallego-Durán, F. J. et al. A guide for game-design-based gamification. Informatics. 6 (4), 49 (2019).

Gee, J. P. Learning and games. In The Ecology of Games: Connecting Youth, Games, and Learning (ed. Salen, K.) 21–40 (MIT Press, 2008).

Cheung, K. L., ten Klooster, P. M., Smit, C., de Vries, H. & Pieterse, M. E. The impact of non-response bias due to sampling in public health studies: A comparison of voluntary versus mandatory recruitment in a Dutch national survey on adolescent health. BMC Public Health. 17 (1), 276 (2017).

Murairwa, S. Voluntary sampling design. Int. J. Adv. Res. Manag. Social Sci. 4 (2), 185–200 (2015).

Chow, S.-C., Shao, J., Wang, H. & Lokhnygina, Y. Sample Size Calculations in Clinical Research (CRC Press, 2017).

Book   Google Scholar  

Palinkas, L. A. et al. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration Policy Mental Health Mental Health Services Res. 42 (5), 533–544 (2015).

McIlvried, D. E., Prucka, S. K., Herbst, M., Barger, C. & Robin, N. H. The use of role-play to enhance medical student understanding of genetic counseling. Genet. Med. 10 (10), 739–744 (2008).

Schlegel, C., Woermann, U., Shaha, M., Rethans, J.-J. & van der Vleuten, C. Effects of communication training on real practice performance: A role-play module versus a standardized patient module. J. Nursing Educ. 51 (1), 16–22 (2012).

Manzoor, I. M. F. & Hashmi, N. R. Medical students’ perspective about role-plays as a teaching strategy in community medicine. J. Coll. Physicians Surg. Pak. 22 (4), 222–225 (2012).

Cornes, S., Gelfand, J. M. & Calton, B. Foundational telemedicine workshop for first-year medical students developed during a pandemic. MedEdPORTAL. 17 , 11171 (2021).

King, J., Hill, K. & Gleason, A. All the world’sa stage: Evaluating psychiatry role-play based learning for medical students. Austral. Psychiatry. 23 (1), 76–79 (2015).

Arayapisit, T. et al. An educational board game for learning orofacial spaces: An experimental study comparing collaborative and competitive approaches. Anatomical Sci. Educ. 16 (4), 666–676 (2023).

Sipiyaruk, K., Hatzipanagos, S., Vichayanrat, T., Reynolds, P.A., Gallagher, J.E. Evaluating a dental public health game across two learning contexts. Educ. Sci. 12 (8), 517 (2022).

Pilnick, A. et al. Using conversation analysis to inform role play and simulated interaction in communications skills training for healthcare professionals: Identifying avenues for further development through a scoping review. BMC Med. Educ. 18 (1), 267 (2018).

Lane, C. & Rollnick, S. The use of simulated patients and role-play in communication skills training: A review of the literature to August 2005. Patient Educ. Counseling. 67 (1), 13–20 (2007).

Gale, N. K., Heath, G., Cameron, E., Rashid, S. & Redwood, S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med. Res. Methodol. 13 (1), 117 (2013).

Ritchie, J., Lewis, J., Nicholls, C. M. & Ormston, R. Qualitative Research Practice: A Guide for Social Science Students and Researchers (Sage, 2014).

Chen, J. C. & Martin, A. R. Role-play simulations as a transformative methodology in environmental education. J. Transform. Educ. 13 (1), 85–102 (2015).

Davis, F. D. Perceived usefulness, perceived ease of use, and user acceptance of information technology. Manag. Inform. Syst. Quart. 13 (3), 319–340 (1989).

Novak, E., Johnson, T. E., Tenenbaum, G. & Shute, V. J. Effects of an instructional gaming characteristic on learning effectiveness, efficiency, and engagement: Using a storyline for teaching basic statistical skills. Interact. Learn. Environ. 24 (3), 523–538 (2016).

Marchiori, E. J. et al. A narrative metaphor to facilitate educational game authoring. Comput. Educ. 58 (1), 590–599 (2012).

Luctkar-Flude, M. et al. Effectiveness of debriefing methods for virtual simulation: A systematic review. Clin. Simulat. Nursing. 57 , 18–30 (2021).

Joyner, B. & Young, L. Teaching medical students using role play: Twelve tips for successful role plays. Med. Teach. 28 (3), 225–229 (2006).

Csikszentmihalyi, M. Flow: The Psychology of Optimal Performance (HarperCollins Publishers, 1990).

Buajeeb, W., Chokpipatkun, J., Achalanan, N., Kriwattanawong, N. & Sipiyaruk, K. The development of an online serious game for oral diagnosis and treatment planning: Evaluation of knowledge acquisition and retention. BMC Med. Educ. 23 (1), 830 (2023).

Littlefield, J. H., Hahn, H. B. & Meyer, A. S. Evaluation of a role-play learning exercise in an ambulatory clinic setting. Adv. Health. Sci. Educ. Theory Pract. 4 (2), 167–173 (1999).

Alkin, M. C. & Christie, C. A. The use of role-play in teaching evaluation. Am. J. Evaluat. 23 (2), 209–218 (2002).

Lovell, K. L., Mavis, B. E., Turner, J. L., Ogle, K. S. & Griffith, M. Medical students as standardized patients in a second-year performance-based assessment experience. Med. Educ. Online. 3 (1), 4301 (1998).

Download references

Acknowledgements

The authors would like to express our sincere gratitude to participants for their contributions in this research. We would also like to thank the experts who provided their helpful suggestions in the validation process of the data collection tools.

This research project was funded by the Faculty of Dentistry, Mahidol University. The APC was funded by Mahidol University.

Author information

Authors and affiliations.

Department of Advanced General Dentistry, Faculty of Dentistry, Mahidol University, Bangkok, Thailand

Chayanid Teerawongpairoj & Chanita Tantipoj

Department of Orthodontics, Faculty of Dentistry, Mahidol University, Bangkok, Thailand

Kawin Sipiyaruk

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization, C.Te., C.Ta., and K.S.; methodology, C.Te., C.Ta., and K.S.; validation, C.Te., C.Ta., and K.S.; investigation, C.Te. and K.S.; formal analysis, C.Te., C.Ta., and K.S.; resources, C.Te., C.Ta., and K.S.; data curation, C.Ta. and K.S.; writing-original draft preparation, C.Te., C.Ta., and K.S.; writing-review and editing, C.Te., C.Ta., and K.S. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Kawin Sipiyaruk .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary information 1., supplementary information 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Teerawongpairoj, C., Tantipoj, C. & Sipiyaruk, K. The design and evaluation of gamified online role-play as a telehealth training strategy in dental education: an explanatory sequential mixed-methods study. Sci Rep 14 , 9216 (2024). https://doi.org/10.1038/s41598-024-58425-9

Download citation

Received : 30 September 2023

Accepted : 28 March 2024

Published : 22 April 2024

DOI : https://doi.org/10.1038/s41598-024-58425-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Dental education
• Distance learning
• Game-based learning
• Gamification

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

• Open access
• Published: 19 April 2024

A scoping review of continuous quality improvement in healthcare system: conceptualization, models and tools, barriers and facilitators, and impact

• Aklilu Endalamaw 1 , 2 ,
• Resham B Khatri 1 , 3 ,
• Tesfaye Setegn Mengistu 1 , 2 ,
• Daniel Erku 1 , 4 , 5 ,
• Eskinder Wolka 6 ,
• Anteneh Zewdie 6 &
• Yibeltal Assefa 1  

BMC Health Services Research volume  24 , Article number:  487 ( 2024 ) Cite this article

598 Accesses

Metrics details

The growing adoption of continuous quality improvement (CQI) initiatives in healthcare has generated a surge in research interest to gain a deeper understanding of CQI. However, comprehensive evidence regarding the diverse facets of CQI in healthcare has been limited. Our review sought to comprehensively grasp the conceptualization and principles of CQI, explore existing models and tools, analyze barriers and facilitators, and investigate its overall impacts.

This qualitative scoping review was conducted using Arksey and O’Malley’s methodological framework. We searched articles in PubMed, Web of Science, Scopus, and EMBASE databases. In addition, we accessed articles from Google Scholar. We used mixed-method analysis, including qualitative content analysis and quantitative descriptive for quantitative findings to summarize findings and PRISMA extension for scoping reviews (PRISMA-ScR) framework to report the overall works.

A total of 87 articles, which covered 14 CQI models, were included in the review. While 19 tools were used for CQI models and initiatives, Plan-Do-Study/Check-Act cycle was the commonly employed model to understand the CQI implementation process. The main reported purposes of using CQI, as its positive impact, are to improve the structure of the health system (e.g., leadership, health workforce, health technology use, supplies, and costs), enhance healthcare delivery processes and outputs (e.g., care coordination and linkages, satisfaction, accessibility, continuity of care, safety, and efficiency), and improve treatment outcome (reduce morbidity and mortality). The implementation of CQI is not without challenges. There are cultural (i.e., resistance/reluctance to quality-focused culture and fear of blame or punishment), technical, structural (related to organizational structure, processes, and systems), and strategic (inadequate planning and inappropriate goals) related barriers that were commonly reported during the implementation of CQI.

Conclusions

Implementing CQI initiatives necessitates thoroughly comprehending key principles such as teamwork and timeline. To effectively address challenges, it’s crucial to identify obstacles and implement optimal interventions proactively. Healthcare professionals and leaders need to be mentally equipped and cognizant of the significant role CQI initiatives play in achieving purposes for quality of care.

Peer Review reports

Continuous quality improvement (CQI) initiative is a crucial initiative aimed at enhancing quality in the health system that has gradually been adopted in the healthcare industry. In the early 20th century, Shewhart laid the foundation for quality improvement by describing three essential steps for process improvement: specification, production, and inspection [ 1 , 2 ]. Then, Deming expanded Shewhart’s three-step model into ‘plan, do, study/check, and act’ (PDSA or PDCA) cycle, which was applied to management practices in Japan in the 1950s [ 3 ] and was gradually translated into the health system. In 1991, Kuperman applied a CQI approach to healthcare, comprising selecting a process to be improved, assembling a team of expert clinicians that understands the process and the outcomes, determining key steps in the process and expected outcomes, collecting data that measure the key process steps and outcomes, and providing data feedback to the practitioners [ 4 ]. These philosophies have served as the baseline for the foundation of principles for continuous improvement [ 5 ].

Continuous quality improvement fosters a culture of continuous learning, innovation, and improvement. It encourages proactive identification and resolution of problems, promotes employee engagement and empowerment, encourages trust and respect, and aims for better quality of care [ 6 , 7 ]. These characteristics drive the interaction of CQI with other quality improvement projects, such as quality assurance and total quality management [ 8 ]. Quality assurance primarily focuses on identifying deviations or errors through inspections, audits, and formal reviews, often settling for what is considered ‘good enough’, rather than pursuing the highest possible standards [ 9 , 10 ], while total quality management is implemented as the management philosophy and system to improve all aspects of an organization continuously [ 11 ].

Continuous quality improvement has been implemented to provide quality care. However, providing effective healthcare is a complicated and complex task in achieving the desired health outcomes and the overall well-being of individuals and populations. It necessitates tackling issues, including access, patient safety, medical advances, care coordination, patient-centered care, and quality monitoring [ 12 , 13 ], rooted long ago. It is assumed that the history of quality improvement in healthcare started in 1854 when Florence Nightingale introduced quality improvement documentation [ 14 ]. Over the passing decades, Donabedian introduced structure, processes, and outcomes as quality of care components in 1966 [ 15 ]. More comprehensively, the Institute of Medicine in the United States of America (USA) has identified effectiveness, efficiency, equity, patient-centredness, safety, and timeliness as the components of quality of care [ 16 ]. Moreover, quality of care has recently been considered an integral part of universal health coverage (UHC) [ 17 ], which requires initiatives to mobilise essential inputs [ 18 ].

While the overall objective of CQI in health system is to enhance the quality of care, it is important to note that the purposes and principles of CQI can vary across different contexts [ 19 , 20 ]. This variation has sparked growing research interest. For instance, a review of CQI approaches for capacity building addressed its role in health workforce development [ 21 ]. Another systematic review, based on random-controlled design studies, assessed the effectiveness of CQI using training as an intervention and the PDSA model [ 22 ]. As a research gap, the former review was not directly related to the comprehensive elements of quality of care, while the latter focused solely on the impact of training using the PDSA model, among other potential models. Additionally, a review conducted in 2015 aimed to identify barriers and facilitators of CQI in Canadian contexts [ 23 ]. However, all these reviews presented different perspectives and investigated distinct outcomes. This suggests that there is still much to explore in terms of comprehensively understanding the various aspects of CQI initiatives in healthcare.

As a result, we conducted a scoping review to address several aspects of CQI. Scoping reviews serve as a valuable tool for systematically mapping the existing literature on a specific topic. They are instrumental when dealing with heterogeneous or complex bodies of research. Scoping reviews provide a comprehensive overview by summarizing and disseminating findings across multiple studies, even when evidence varies significantly [ 24 ]. In our specific scoping review, we included various types of literature, including systematic reviews, to enhance our understanding of CQI.

This scoping review examined how CQI is conceptualized and measured and investigated models and tools for its application while identifying implementation challenges and facilitators. It also analyzed the purposes and impact of CQI on the health systems, providing valuable insights for enhancing healthcare quality.

Protocol registration and results reporting

Protocol registration for this scoping review was not conducted. Arksey and O’Malley’s methodological framework was utilized to conduct this scoping review [ 25 ]. The scoping review procedures start by defining the research questions, identifying relevant literature, selecting articles, extracting data, and summarizing the results. The review findings are reported using the PRISMA extension for a scoping review (PRISMA-ScR) [ 26 ]. McGowan and colleagues also advised researchers to report findings from scoping reviews using PRISMA-ScR [ 27 ].

Defining the research problems

This review aims to comprehensively explore the conceptualization, models, tools, barriers, facilitators, and impacts of CQI within the healthcare system worldwide. Specifically, we address the following research questions: (1) How has CQI been defined across various contexts? (2) What are the diverse approaches to implementing CQI in healthcare settings? (3) Which tools are commonly employed for CQI implementation ? (4) What barriers hinder and facilitators support successful CQI initiatives? and (5) What effects CQI initiatives have on the overall care quality?

Information source and search strategy

We conducted the search in PubMed, Web of Science, Scopus, and EMBASE databases, and the Google Scholar search engine. The search terms were selected based on three main distinct concepts. One group was CQI-related terms. The second group included terms related to the purpose for which CQI has been implemented, and the third group included processes and impact. These terms were selected based on the Donabedian framework of structure, process, and outcome [ 28 ]. Additionally, the detailed keywords were recruited from the primary health framework, which has described lists of dimensions under process, output, outcome, and health system goals of any intervention for health [ 29 ]. The detailed search strategy is presented in the Supplementary file 1 (Search strategy). The search for articles was initiated on August 12, 2023, and the last search was conducted on September 01, 2023.

Eligibility criteria and article selection

Based on the scoping review’s population, concept, and context frameworks [ 30 ], the population included any patients or clients. Additionally, the concepts explored in the review encompassed definitions, implementation, models, tools, barriers, facilitators, and impacts of CQI. Furthermore, the review considered contexts at any level of health systems. We included articles if they reported results of qualitative or quantitative empirical study, case studies, analytic or descriptive synthesis, any review, and other written documents, were published in peer-reviewed journals, and were designed to address at least one of the identified research questions or one of the identified implementation outcomes or their synonymous taxonomy as described in the search strategy. Based on additional contexts, we included articles published in English without geographic and time limitations. We excluded articles with abstracts only, conference abstracts, letters to editors, commentators, and corrections.

We exported all citations to EndNote x20 to remove duplicates and screen relevant articles. The article selection process includes automatic duplicate removal by using EndNote x20, unmatched title and abstract removal, citation and abstract-only materials removal, and full-text assessment. The article selection process was mainly conducted by the first author (AE) and reported to the team during the weekly meetings. The first author encountered papers that caused confusion regarding whether to include or exclude them and discussed them with the last author (YA). Then, decisions were ultimately made. Whenever disagreements happened, they were resolved by discussion and reconsideration of the review questions in relation to the written documents of the article. Further statistical analysis, such as calculating Kappa, was not performed to determine article inclusion or exclusion.

Data extraction and data items

We extracted first author, publication year, country, settings, health problem, the purpose of the study, study design, types of intervention if applicable, CQI approaches/steps if applicable, CQI tools and procedures if applicable, and main findings using a customized Microsoft Excel form.

Summarizing and reporting the results

The main findings were summarized and described based on the main themes, including concepts under conceptualizing, principles, teams, timelines, models, tools, barriers, facilitators, and impacts of CQI. Results-based convergent synthesis, achieved through mixed-method analysis, involved content analysis to identify the thematic presentation of findings. Additionally, a narrative description was used for quantitative findings, aligning them with the appropriate theme. The authors meticulously reviewed the primary findings from each included material and contextualized these findings concerning the main themes1. This approach provides a comprehensive understanding of complex interventions and health systems, acknowledging quantitative and qualitative evidence.

Search results

A total of 11,251 documents were identified from various databases: SCOPUS ( n  = 4,339), PubMed ( n  = 2,893), Web of Science ( n  = 225), EMBASE ( n  = 3,651), and Google Scholar ( n  = 143). After removing duplicates ( n  = 5,061), 6,190 articles were evaluated by title and abstract. Subsequently, 208 articles were assessed for full-text eligibility. Following the eligibility criteria, 121 articles were excluded, leaving 87 included in the current review (Fig.  1 ).

Article selection process

Operationalizing continuous quality improvement

Continuous Quality Improvement (CQI) is operationalized as a cyclic process that requires commitment to implementation, teamwork, time allocation, and celebrating successes and failures.

CQI is a cyclic ongoing process that is followed reflexive, analytical and iterative steps, including identifying gaps, generating data, developing and implementing action plans, evaluating performance, providing feedback to implementers and leaders, and proposing necessary adjustments [ 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 ].

CQI requires committing to the philosophy, involving continuous improvement [ 19 , 38 ], establishing a mission statement [ 37 ], and understanding quality definition [ 19 ].

CQI involves a wide range of patient-oriented measures and performance indicators, specifically satisfying internal and external customers, developing quality assurance, adopting common quality measures, and selecting process measures [ 8 , 19 , 35 , 36 , 37 , 39 , 40 ].

CQI requires celebrating success and failure without personalization, leading each team member to develop error-free attitudes [ 19 ]. Success and failure are related to underlying organizational processes and systems as causes of failure rather than blaming individuals [ 8 ] because CQI is process-focused based on collaborative, data-driven, responsive, rigorous and problem-solving statistical analysis [ 8 , 19 , 38 ]. Furthermore, a gap or failure opens another opportunity for establishing a data-driven learning organization [ 41 ].

CQI cannot be implemented without a CQI team [ 8 , 19 , 37 , 39 , 42 , 43 , 44 , 45 , 46 ]. A CQI team comprises individuals from various disciplines, often comprising a team leader, a subject matter expert (physician or other healthcare provider), a data analyst, a facilitator, frontline staff, and stakeholders [ 39 , 43 , 47 , 48 , 49 ]. It is also important to note that inviting stakeholders or partners as part of the CQI support intervention is crucial [ 19 , 38 , 48 ].

The timeline is another distinct feature of CQI because the results of CQI vary based on the implementation duration of each cycle [ 35 ]. There is no specific time limit for CQI implementation, although there is a general consensus that a cycle of CQI should be relatively short [ 35 ]. For instance, a CQI implementation took 2 months [ 42 ], 4 months [ 50 ], 9 months [ 51 , 52 ], 12 months [ 53 , 54 , 55 ], and one year and 5 months [ 49 ] duration to achieve the desired positive outcome, while bi-weekly [ 47 ] and monthly data reviews and analyses [ 44 , 48 , 56 ], and activities over 3 months [ 57 ] have also resulted in a positive outcome.

Continuous quality improvement models and tools

There have been several models are utilized. The Plan-Do-Study/Check-Act cycle is a stepwise process involving project initiation, situation analysis, root cause identification, solution generation and selection, implementation, result evaluation, standardization, and future planning [ 7 , 36 , 37 , 45 , 47 , 48 , 49 , 50 , 51 , 53 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 ]. The FOCUS-PDCA cycle enhances the PDCA process by adding steps to find and improve a process (F), organize a knowledgeable team (O), clarify the process (C), understand variations (U), and select improvements (S) [ 55 , 71 , 72 , 73 ]. The FADE cycle involves identifying a problem (Focus), understanding it through data analysis (Analyze), devising solutions (Develop), and implementing the plan (Execute) [ 74 ]. The Logic Framework involves brainstorming to identify improvement areas, conducting root cause analysis to develop a problem tree, logically reasoning to create an objective tree, formulating the framework, and executing improvement projects [ 75 ]. Breakthrough series approach requires CQI teams to meet in quarterly collaborative learning sessions, share learning experiences, and continue discussion by telephone and cross-site visits to strengthen learning and idea exchange [ 47 ]. Another CQI model is the Lean approach, which has been conducted with Kaizen principles [ 52 ], 5 S principles, and the Six Sigma model. The 5 S (Sort, Set/Straighten, Shine, Standardize, Sustain) systematically organises and improves the workplace, focusing on sorting, setting order, shining, standardizing, and sustaining the improvement [ 54 , 76 ]. Kaizen principles guide CQI by advocating for continuous improvement, valuing all ideas, solving problems, focusing on practical, low-cost improvements, using data to drive change, acknowledging process defects, reducing variability and waste, recognizing every interaction as a customer-supplier relationship, empowering workers, responding to all ideas, and maintaining a disciplined workplace [ 77 ]. Lean Six Sigma, a CQI model, applies the DMAIC methodology, which involves defining (D) and measuring the problem (M), analyzing root causes (A), improving by finding solutions (I), and controlling by assessing process stability (C) [ 78 , 79 ]. The 5 C-cyclic model (consultation, collection, consideration, collaboration, and celebration), the first CQI framework for volunteer dental services in Aboriginal communities, ensures quality care based on community needs [ 80 ]. One study used meetings involving activities such as reviewing objectives, assigning roles, discussing the agenda, completing tasks, retaining key outputs, planning future steps, and evaluating the meeting’s effectiveness [ 81 ].

Various tools are involved in the implementation or evaluation of CQI initiatives: checklists [ 53 , 82 ], flowcharts [ 81 , 82 , 83 ], cause-and-effect diagrams (fishbone or Ishikawa diagrams) [ 60 , 62 , 79 , 81 , 82 ], fuzzy Pareto diagram [ 82 ], process maps [ 60 ], time series charts [ 48 ], why-why analysis [ 79 ], affinity diagrams and multivoting [ 81 ], and run chart [ 47 , 48 , 51 , 60 , 84 ], and others mentioned in the table (Table  1 ).

Barriers and facilitators of continuous quality improvement implementation

Implementing CQI initiatives is determined by various barriers and facilitators, which can be thematized into four dimensions. These dimensions are cultural, technical, structural, and strategic dimensions.

Continuous quality improvement initiatives face various cultural, strategic, technical, and structural barriers. Cultural dimension barriers involve resistance to change (e.g., not accepting online technology), lack of quality-focused culture, staff reporting apprehensiveness, and fear of blame or punishment [ 36 , 41 , 85 , 86 ]. The technical dimension barriers of CQI can include various factors that hinder the effective implementation and execution of CQI processes [ 36 , 86 , 87 , 88 , 89 ]. Structural dimension barriers of CQI arise from the organization structure, process, and systems that can impede the effective implementation and sustainability of CQI [ 36 , 85 , 86 , 87 , 88 ]. Strategic dimension barriers are, for example, the inability to select proper CQI goals and failure to integrate CQI into organizational planning and goals [ 36 , 85 , 86 , 87 , 88 , 90 ].

Facilitators are also grouped to cultural, structural, technical, and strategic dimensions to provide solutions to CQI barriers. Cultural challenges were addressed by developing a group culture to CQI and other rewards [ 39 , 41 , 80 , 85 , 86 , 87 , 90 , 91 , 92 ]. Technical facilitators are pivotal to improving technical barriers [ 39 , 42 , 53 , 69 , 86 , 90 , 91 ]. Structural-related facilitators are related to improving communication, infrastructure, and systems [ 86 , 92 , 93 ]. Strategic dimension facilitators include strengthening leadership and improving decision-making skills [ 43 , 53 , 67 , 86 , 87 , 92 , 94 , 95 ] (Table  2 ).

Impact of continuous quality improvement

Continuous quality improvement initiatives can significantly impact the quality of healthcare in a wide range of health areas, focusing on improving structure, the health service delivery process and improving client wellbeing and reducing mortality.

Structure components

These are health leadership, financing, workforce, technology, and equipment and supplies. CQI has improved planning, monitoring and evaluation [ 48 , 53 ], and leadership and planning [ 48 ], indicating improvement in leadership perspectives. Implementing CQI in primary health care (PHC) settings has shown potential for maintaining or reducing operation costs [ 67 ]. Findings from another study indicate that the costs associated with implementing CQI interventions per facility ranged from approximately $2,000 to $10,500 per year, with an average cost of approximately $10 to $60 per admitted client [ 57 ]. However, based on model predictions, the average cost savings after implementing CQI were estimated to be $5430 [ 31 ]. CQI can also be applied to health workforce development [ 32 ]. CQI in the institutional system improved medical education [ 66 , 96 , 97 ], human resources management [ 53 ], motivated staffs [ 76 ], and increased staff health awareness [ 69 ], while concerns raised about CQI impartiality, independence, and public accountability [ 96 ]. Regarding health technology, CQI also improved registration and documentation [ 48 , 53 , 98 ]. Furthermore, the CQI initiatives increased cleanliness [ 54 ] and improved logistics, supplies, and equipment [ 48 , 53 , 68 ].

Process and output components

The process component focuses on the activities and actions involved in delivering healthcare services.

Service delivery

CQI interventions improved service delivery [ 53 , 56 , 99 ], particularly a significant 18% increase in the overall quality of service performance [ 48 ], improved patient counselling, adherence to appropriate procedures, and infection prevention [ 48 , 68 ], and optimised workflow [ 52 ].

Coordination and collaboration

CQI initiatives improved coordination and collaboration through collecting and analysing data, onsite technical support, training, supportive supervision [ 53 ] and facilitating linkages between work processes and a quality control group [ 65 ].

Patient satisfaction

The CQI initiatives increased patient satisfaction and improved quality of life by optimizing care quality management, improving the quality of clinical nursing, reducing nursing defects and enhancing the wellbeing of clients [ 54 , 76 , 100 ], although CQI was not associated with changes in adolescent and young adults’ satisfaction [ 51 ].

CQI initiatives reduced medication error reports from 16 to 6 [ 101 ], and it significantly reduced the administration of inappropriate prophylactic antibiotics [ 44 ], decreased errors in inpatient care [ 52 ], decreased the overall episiotomy rate from 44.5 to 33.3% [ 83 ], reduced the overall incidence of unplanned endotracheal extubation [ 102 ], improving appropriate use of computed tomography angiography [ 103 ], and appropriate diagnosis and treatment selection [ 47 ].

Continuity of care

CQI initiatives effectively improve continuity of care by improving client and physician interaction. For instance, provider continuity levels showed a 64% increase [ 55 ]. Modifying electronic medical record templates, scheduling, staff and parental education, standardization of work processes, and birth to 1-year age-specific incentives in post-natal follow-up care increased continuity of care to 74% in 2018 compared to baseline 13% in 2012 [ 84 ].

The CQI initiative yielded enhanced efficiency in the cardiac catheterization laboratory, as evidenced by improved punctuality in procedure starts and increased efficiency in manual sheath-pulls inside [ 78 ].

Accessibility

CQI initiatives were effective in improving accessibility in terms of increasing service coverage and utilization rate. For instance, screening for cigarettes, nutrition counselling, folate prescription, maternal care, immunization coverage [ 53 , 81 , 104 , 105 ], reducing the percentage of non-attending patients to surgery to 0.9% from the baseline 3.9% [ 43 ], increasing Chlamydia screening rates from 29 to 60% [ 45 ], increasing HIV care continuum coverage [ 51 , 59 , 60 ], increasing in the uptake of postpartum long-acting reversible contraceptive use from 6.9% at the baseline to 25.4% [ 42 ], increasing post-caesarean section prophylaxis from 36 to 89% [ 62 ], a 31% increase of kangaroo care practice [ 50 ], and increased follow-up [ 65 ]. Similarly, the QI intervention increased the quality of antenatal care by 29.3%, correct partograph use by 51.7%, and correct active third-stage labour management, a 19.6% improvement from the baseline, but not significantly associated with improvement in contraceptive service uptake [ 61 ].

Timely access

CQI interventions improved the time care provision [ 52 ], and reduced waiting time [ 62 , 74 , 76 , 106 ]. For instance, the discharge process waiting time in the emergency department decreased from 76 min to 22 min [ 79 ]. It also reduced mean postprocedural length of stay from 2.8 days to 2.0 days [ 31 ].

Acceptability

Acceptability of CQI by healthcare providers was satisfactory. For instance, 88% of the faculty, 64% of the residents, and 82% of the staff believed CQI to be useful in the healthcare clinic [ 107 ].

Outcome components

Morbidity and mortality.

CQI efforts have demonstrated better management outcomes among diabetic patients [ 40 ], patients with oral mucositis [ 71 ], and anaemic patients [ 72 ]. It has also reduced infection rate in post-caesarean Sect. [ 62 ], reduced post-peritoneal dialysis peritonitis [ 49 , 108 ], and prevented pressure ulcers [ 70 ]. It is explained by peritonitis incidence from once every 40.1 patient months at baseline to once every 70.8 patient months after CQI [ 49 ] and a 63% reduction in pressure ulcer prevalence within 2 years from 2008 to 2010 [ 70 ]. Furthermore, CQI initiatives significantly reduced in-hospital deaths [ 31 ] and increased patient survival rates [ 108 ]. Figure  2 displays the overall process of the CQI implementations.

The overall mechanisms of continuous quality improvement implementation

In this review, we examined the fundamental concepts and principles underlying CQI, the factors that either hinder or assist in its successful application and implementation, and the purpose of CQI in enhancing quality of care across various health issues.

Our findings have brought attention to the application and implementation of CQI, emphasizing its underlying concepts and principles, as evident in the existing literature [ 31 , 32 , 33 , 34 , 35 , 36 , 39 , 40 , 43 , 45 , 46 ]. Continuous quality improvement has shared with the principles of continuous improvement, such as a customer-driven focus, effective leadership, active participation of individuals, a process-oriented approach, systematic implementation, emphasis on design improvement and prevention, evidence-based decision-making, and fostering partnership [ 5 ]. Moreover, Deming’s 14 principles laid the foundation for CQI principles [ 109 ]. These principles have been adapted and put into practice in various ways: ten [ 19 ] and five [ 38 ] principles in hospitals, five principles for capacity building [ 38 ], and two principles for medication error prevention [ 41 ]. As a principle, the application of CQI can be process-focused [ 8 , 19 ] or impact-focused [ 38 ]. Impact-focused CQI focuses on achieving specific outcomes or impacts, whereas process-focused CQI prioritizes and improves the underlying processes and systems. These principles complement each other and can be utilized based on the objectives of quality improvement initiatives in healthcare settings. Overall, CQI is an ongoing educational process that requires top management’s involvement, demands coordination across departments, encourages the incorporation of views beyond clinical area, and provides non-judgemental evidence based on objective data [ 110 ].

The current review recognized that it was not easy to implement CQI. It requires reasonable utilization of various models and tools. The application of each tool can be varied based on the studied health problem and the purpose of CQI initiative [ 111 ], varied in context, content, structure, and usability [ 112 ]. Additionally, overcoming the cultural, technical, structural, and strategic-related barriers. These barriers have emerged from clinical staff, managers, and health systems perspectives. Of the cultural obstacles, staff non-involvement, resistance to change, and reluctance to report error were staff-related. In contrast, others, such as the absence of celebration for success and hierarchical and rational culture, may require staff and manager involvement. Staff members may exhibit reluctance in reporting errors due to various cultural factors, including lack of trust, hierarchical structures, fear of retribution, and a blame-oriented culture. These challenges pose obstacles to implementing standardized CQI practices, as observed, for instance, in community pharmacy settings [ 85 ]. The hierarchical culture, characterized by clearly defined levels of power, authority, and decision-making, posed challenges to implementing CQI initiatives in public health [ 41 , 86 ]. Although rational culture, a type of organizational culture, emphasizes logical thinking and rational decision-making, it can also create challenges for CQI implementation [ 41 , 86 ] because hierarchical and rational cultures, which emphasize bureaucratic norms and narrow definitions of achievement, were found to act as barriers to the implementation of CQI [ 86 ]. These could be solved by developing a shared mindset and collective commitment, establishing a shared purpose, developing group norms, and cultivating psychological preparedness among staff, managers, and clients to implement and sustain CQI initiatives. Furthermore, reversing cultural-related barriers necessitates cultural-related solutions: development of a culture and group culture to CQI [ 41 , 86 ], positive comprehensive perception [ 91 ], commitment [ 85 ], involving patients, families, leaders, and staff [ 39 , 92 ], collaborating for a common goal [ 80 , 86 ], effective teamwork [ 86 , 87 ], and rewarding and celebrating successes [ 80 , 90 ].

The technical dimension barriers of CQI can include inadequate capitalization of a project and insufficient support for CQI facilitators and data entry managers [ 36 ], immature electronic medical records or poor information systems [ 36 , 86 ], and the lack of training and skills [ 86 , 87 , 88 ]. These challenges may cause the CQI team to rely on outdated information and technologies. The presence of barriers on the technical dimension may challenge the solid foundation of CQI expertise among staff, the ability to recognize opportunities for improvement, a comprehensive understanding of how services are produced and delivered, and routine use of expertise in daily work. Addressing these technical barriers requires knowledge creation activities (training, seminar, and education) [ 39 , 42 , 53 , 69 , 86 , 90 , 91 ], availability of quality data [ 86 ], reliable information [ 92 ], and a manual-online hybrid reporting system [ 85 ].

Structural dimension barriers of CQI include inadequate communication channels and lack of standardized process, specifically weak physician-to-physician synergies [ 36 ], lack of mechanisms for disseminating knowledge and limited use of communication mechanisms [ 86 ]. Lack of communication mechanism endangers sharing ideas and feedback among CQI teams, leading to misunderstandings, limited participation and misinterpretations, and a lack of learning [ 113 ]. Knowledge translation facilitates the co-production of research, subsequent diffusion of knowledge, and the developing stakeholder’s capacity and skills [ 114 ]. Thus, the absence of a knowledge translation mechanism may cause missed opportunities for learning, inefficient problem-solving, and limited creativity. To overcome these challenges, organizations should establish effective communication and information systems [ 86 , 93 ] and learning systems [ 92 ]. Though CQI and knowledge translation have interacted with each other, it is essential to recognize that they are distinct. CQI focuses on process improvement within health care systems, aiming to optimize existing processes, reduce errors, and enhance efficiency.

In contrast, knowledge translation bridges the gap between research evidence and clinical practice, translating research findings into actionable knowledge for practitioners. While both CQI and knowledge translation aim to enhance health care quality and patient outcomes, they employ different strategies: CQI utilizes tools like Plan-Do-Study-Act cycles and statistical process control, while knowledge translation involves knowledge synthesis and dissemination. Additionally, knowledge translation can also serve as a strategy to enhance CQI. Both concepts share the same principle: continuous improvement is essential for both. Therefore, effective strategies on the structural dimension may build efficient and effective steering councils, information systems, and structures to diffuse learning throughout the organization.

Strategic factors, such as goals, planning, funds, and resources, determine the overall purpose of CQI initiatives. Specific barriers were improper goals and poor planning [ 36 , 86 , 88 ], fragmentation of quality assurance policies [ 87 ], inadequate reinforcement to staff [ 36 , 90 ], time constraints [ 85 , 86 ], resource inadequacy [ 86 ], and work overload [ 86 ]. These barriers can be addressed through strengthening leadership [ 86 , 87 ], CQI-based mentoring [ 94 ], periodic monitoring, supportive supervision and coaching [ 43 , 53 , 87 , 92 , 95 ], participation, empowerment, and accountability [ 67 ], involving all stakeholders in decision-making [ 86 , 87 ], a provider-payer partnership [ 64 ], and compensating staff for after-hours meetings on CQI [ 85 ]. The strategic dimension, characterized by a strategic plan and integrated CQI efforts, is devoted to processes that are central to achieving strategic priorities. Roles and responsibilities are defined in terms of integrated strategic and quality-related goals [ 115 ].

The utmost goal of CQI has been to improve the quality of care, which is usually revealed by structure, process, and outcome. After resolving challenges and effectively using tools and running models, the goal of CQI reflects the ultimate reason and purpose of its implementation. First, effectively implemented CQI initiatives can improve leadership, health financing, health workforce development, health information technology, and availability of supplies as the building blocks of a health system [ 31 , 48 , 53 , 68 , 98 ]. Second, effectively implemented CQI initiatives improved care delivery process (counselling, adherence with standards, coordination, collaboration, and linkages) [ 48 , 53 , 65 , 68 ]. Third, the CQI can improve outputs of healthcare delivery, such as satisfaction, accessibility (timely access, utilization), continuity of care, safety, efficiency, and acceptability [ 52 , 54 , 55 , 76 , 78 ]. Finally, the effectiveness of the CQI initiatives has been tested in enhancing responses related to key aspects of the HIV response, maternal and child health, non-communicable disease control, and others (e.g., surgery and peritonitis). However, it is worth noting that CQI initiative has not always been effective. For instance, CQI using a two- to nine-times audit cycle model through systems assessment tools did not bring significant change to increase syphilis testing performance [ 116 ]. This study was conducted within the context of Aboriginal and Torres Strait Islander people’s primary health care settings. Notably, ‘the clinics may not have consistently prioritized syphilis testing performance in their improvement strategies, as facilitated by the CQI program’ [ 116 ]. Additionally, by applying CQI-based mentoring, uptake of facility-based interventions was not significantly improved, though it was effective in increasing community health worker visits during pregnancy and the postnatal period, knowledge about maternal and child health and exclusive breastfeeding practice, and HIV disclosure status [ 117 ]. The study conducted in South Africa revealed no significant association between the coverage of facility-based interventions and Continuous Quality Improvement (CQI) implementation. This lack of association was attributed to the already high antenatal and postnatal attendance rates in both control and intervention groups at baseline, leaving little room for improvement. Additionally, the coverage of HIV interventions remained consistently high throughout the study period [ 117 ].

Regarding health care and policy implications, CQI has played a vital role in advancing PHC and fostering the realization of UHC goals worldwide. The indicators found in Donabedian’s framework that are positively influenced by CQI efforts are comparable to those included in the PHC performance initiative’s conceptual framework [ 29 , 118 , 119 ]. It is clearly explained that PHC serves as the roadmap to realizing the vision of UHC [ 120 , 121 ]. Given these circumstances, implementing CQI can contribute to the achievement of PHC principles and the objectives of UHC. For instance, by implementing CQI methods, countries have enhanced the accessibility, affordability, and quality of PHC services, leading to better health outcomes for their populations. CQI has facilitated identifying and resolving healthcare gaps and inefficiencies, enabling countries to optimize resource allocation and deliver more effective and patient-centered care. However, it is crucial to recognize that the successful implementation of Continuous Quality Improvement (CQI) necessitates optimizing the duration of each cycle, understanding challenges and barriers that extend beyond the health system and settings, and acknowledging that its effectiveness may be compromised if these challenges are not adequately addressed.

Despite abundant literature, there are still gaps regarding the relationship between CQI and other dimensions within the healthcare system. No studies have examined the impact of CQI initiatives on catastrophic health expenditure, effective service coverage, patient-centredness, comprehensiveness, equity, health security, and responsiveness.

Limitations

In conducting this review, it has some limitations to consider. Firstly, only articles published in English were included, which may introduce the exclusion of relevant non-English articles. Additionally, as this review follows a scoping methodology, the focus is on synthesising available evidence rather than critically evaluating or scoring the quality of the included articles.

Continuous quality improvement is investigated as a continuous and ongoing intervention, where the implementation time can vary across different cycles. The CQI team and implementation timelines were critical elements of CQI in different models. Among the commonly used approaches, the PDSA or PDCA is frequently employed. In most CQI models, a wide range of tools, nineteen tools, are commonly utilized to support the improvement process. Cultural, technical, structural, and strategic barriers and facilitators are significant in implementing CQI initiatives. Implementing the CQI initiative aims to improve health system blocks, enhance health service delivery process and output, and ultimately prevent morbidity and reduce mortality. For future researchers, considering that CQI is context-dependent approach, conducting scale-up implementation research about catastrophic health expenditure, effective service coverage, patient-centredness, comprehensiveness, equity, health security, and responsiveness across various settings and health issues would be valuable.

Availability of data and materials

The data used and/or analyzed during the current study are available in this manuscript and/or the supplementary file.

Shewhart WA, Deming WE. Memoriam: Walter A. Shewhart, 1891–1967. Am Stat. 1967;21(2):39–40.

Article   Google Scholar  

Shewhart WA. Statistical method from the viewpoint of quality control. New York: Dover; 1986. ISBN 978-0486652320. OCLC 13822053. Reprint. Originally published: Washington, DC: Graduate School of the Department of Agriculture, 1939.

Moen R, editor Foundation and History of the PDSA Cycle. Asian network for quality conference Tokyo. https://www.deming.org/sites/default/files/pdf/2015/PDSA_History_Ron_MoenPdf . 2009.

Kuperman G, James B, Jacobsen J, Gardner RM. Continuous quality improvement applied to medical care: experiences at LDS hospital. Med Decis Making. 1991;11(4suppl):S60–65.

Article   CAS   PubMed   Google Scholar  

Singh J, Singh H. Continuous improvement philosophy–literature review and directions. Benchmarking: An International Journal. 2015;22(1):75–119.

Goldstone J. Presidential address: Sony, Porsche, and vascular surgery in the 21st century. J Vasc Surg. 1997;25(2):201–10.

Radawski D. Continuous quality improvement: origins, concepts, problems, and applications. J Physician Assistant Educ. 1999;10(1):12–6.

Shortell SM, O’Brien JL, Carman JM, Foster RW, Hughes E, Boerstler H, et al. Assessing the impact of continuous quality improvement/total quality management: concept versus implementation. Health Serv Res. 1995;30(2):377.

CAS   PubMed   PubMed Central   Google Scholar  

Lohr K. Quality of health care: an introduction to critical definitions, concepts, principles, and practicalities. Striving for quality in health care. 1991.

Berwick DM. The clinical process and the quality process. Qual Manage Healthc. 1992;1(1):1–8.

Article   CAS   Google Scholar  

Gift B. On the road to TQM. Food Manage. 1992;27(4):88–9.

CAS   PubMed   Google Scholar  

Greiner A, Knebel E. The core competencies needed for health care professionals. health professions education: A bridge to quality. 2003:45–73.

McCalman J, Bailie R, Bainbridge R, McPhail-Bell K, Percival N, Askew D et al. Continuous quality improvement and comprehensive primary health care: a systems framework to improve service quality and health outcomes. Front Public Health. 2018:6 (76):1–6.

Sheingold BH, Hahn JA. The history of healthcare quality: the first 100 years 1860–1960. Int J Afr Nurs Sci. 2014;1:18–22.

Google Scholar  

Donabedian A. Evaluating the quality of medical care. Milbank Q. 1966;44(3):166–206.

Institute of Medicine (US) Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington (DC): National Academies Press (US). 2001. 2, Improving the 21st-century Health Care System. Available from: https://www.ncbi.nlm.nih.gov/books/NBK222265/ .

Rubinstein A, Barani M, Lopez AS. Quality first for effective universal health coverage in low-income and middle-income countries. Lancet Global Health. 2018;6(11):e1142–1143.

Article   PubMed   Google Scholar  

Agency for Healthcare Reserach and Quality. Quality Improvement and monitoring at your fingertips USA,: Agency for Healthcare Reserach and Quality. 2022. Available from: https://qualityindicators.ahrq.gov/ .

Anderson CA, Cassidy B, Rivenburgh P. Implementing continuous quality improvement (CQI) in hospitals: lessons learned from the International Quality Study. Qual Assur Health Care. 1991;3(3):141–6.

Gardner K, Mazza D. Quality in general practice - definitions and frameworks. Aust Fam Physician. 2012;41(3):151–4.

PubMed   Google Scholar  

Loper AC, Jensen TM, Farley AB, Morgan JD, Metz AJ. A systematic review of approaches for continuous quality improvement capacity-building. J Public Health Manage Pract. 2022;28(2):E354.

Hill JE, Stephani A-M, Sapple P, Clegg AJ. The effectiveness of continuous quality improvement for developing professional practice and improving health care outcomes: a systematic review. Implement Sci. 2020;15(1):1–14.

Candas B, Jobin G, Dubé C, Tousignant M, Abdeljelil AB, Grenier S, et al. Barriers and facilitators to implementing continuous quality improvement programs in colonoscopy services: a mixed methods systematic review. Endoscopy Int Open. 2016;4(02):E118–133.

Peters MD, Marnie C, Colquhoun H, Garritty CM, Hempel S, Horsley T, et al. Scoping reviews: reinforcing and advancing the methodology and application. Syst Reviews. 2021;10(1):1–6.

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

McGowan J, Straus S, Moher D, Langlois EV, O’Brien KK, Horsley T, et al. Reporting scoping reviews—PRISMA ScR extension. J Clin Epidemiol. 2020;123:177–9.

Donabedian A. Explorations in quality assessment and monitoring: the definition of quality and approaches to its assessment. Health Administration Press, Ann Arbor. 1980;1.

World Health Organization. Operational framework for primary health care: transforming vision into action. Geneva: World Health Organization and the United Nations Children’s Fund (UNICEF); 2020 [updated 14 December 2020; cited 2023 Nov Oct 17]. Available from: https://www.who.int/publications/i/item/9789240017832 .

The Joanna Briggs Institute. The Joanna Briggs Institute Reviewers’ Manual :2014 edition. Australia: The Joanna Briggs Institute. 2014:88–91.

Rihal CS, Kamath CC, Holmes DR Jr, Reller MK, Anderson SS, McMurtry EK, et al. Economic and clinical outcomes of a physician-led continuous quality improvement intervention in the delivery of percutaneous coronary intervention. Am J Manag Care. 2006;12(8):445–52.

Ade-Oshifogun JB, Dufelmeier T. Prevention and Management of Do not return notices: a quality improvement process for Supplemental staffing nursing agencies. Nurs Forum. 2012;47(2):106–12.

Rubenstein L, Khodyakov D, Hempel S, Danz M, Salem-Schatz S, Foy R, et al. How can we recognize continuous quality improvement? Int J Qual Health Care. 2014;26(1):6–15.

O’Neill SM, Hempel S, Lim YW, Danz MS, Foy R, Suttorp MJ, et al. Identifying continuous quality improvement publications: what makes an improvement intervention ‘CQI’? BMJ Qual Saf. 2011;20(12):1011–9.

Article   PubMed   PubMed Central   Google Scholar  

Sibthorpe B, Gardner K, McAullay D. Furthering the quality agenda in Aboriginal community controlled health services: understanding the relationship between accreditation, continuous quality improvement and national key performance indicator reporting. Aust J Prim Health. 2016;22(4):270–5.

Bennett CL, Crane JM. Quality improvement efforts in oncology: are we ready to begin? Cancer Invest. 2001;19(1):86–95.

VanValkenburgh DA. Implementing continuous quality improvement at the facility level. Adv Ren Replace Ther. 2001;8(2):104–13.

Loper AC, Jensen TM, Farley AB, Morgan JD, Metz AJ. A systematic review of approaches for continuous quality improvement capacity-building. J Public Health Manage Practice. 2022;28(2):E354–361.

Ryan M. Achieving and sustaining quality in healthcare. Front Health Serv Manag. 2004;20(3):3–11.

Nicolucci A, Allotta G, Allegra G, Cordaro G, D’Agati F, Di Benedetto A, et al. Five-year impact of a continuous quality improvement effort implemented by a network of diabetes outpatient clinics. Diabetes Care. 2008;31(1):57–62.

Wakefield BJ, Blegen MA, Uden-Holman T, Vaughn T, Chrischilles E, Wakefield DS. Organizational culture, continuous quality improvement, and medication administration error reporting. Am J Med Qual. 2001;16(4):128–34.

Sori DA, Debelew GT, Degefa LS, Asefa Z. Continuous quality improvement strategy for increasing immediate postpartum long-acting reversible contraceptive use at Jimma University Medical Center, Jimma, Ethiopia. BMJ Open Qual. 2023;12(1):e002051.

Roche B, Robin C, Deleaval PJ, Marti MC. Continuous quality improvement in ambulatory surgery: the non-attending patient. Ambul Surg. 1998;6(2):97–100.

O’Connor JB, Sondhi SS, Mullen KD, McCullough AJ. A continuous quality improvement initiative reduces inappropriate prescribing of prophylactic antibiotics for endoscopic procedures. Am J Gastroenterol. 1999;94(8):2115–21.

Ursu A, Greenberg G, McKee M. Continuous quality improvement methodology: a case study on multidisciplinary collaboration to improve chlamydia screening. Fam Med Community Health. 2019;7(2):e000085.

Quick B, Nordstrom S, Johnson K. Using continuous quality improvement to implement evidence-based medicine. Lippincotts Case Manag. 2006;11(6):305–15 ( quiz 16 – 7 ).

Oyeledun B, Phillips A, Oronsaye F, Alo OD, Shaffer N, Osibo B, et al. The effect of a continuous quality improvement intervention on retention-in-care at 6 months postpartum in a PMTCT Program in Northern Nigeria: results of a cluster randomized controlled study. J Acquir Immune Defic Syndr. 2017;75(Suppl 2):S156–164.

Nyengerai T, Phohole M, Iqaba N, Kinge CW, Gori E, Moyo K, et al. Quality of service and continuous quality improvement in voluntary medical male circumcision programme across four provinces in South Africa: longitudinal and cross-sectional programme data. PLoS ONE. 2021;16(8):e0254850.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Wang J, Zhang H, Liu J, Zhang K, Yi B, Liu Y, et al. Implementation of a continuous quality improvement program reduces the occurrence of peritonitis in PD. Ren Fail. 2014;36(7):1029–32.

Stikes R, Barbier D. Applying the plan-do-study-act model to increase the use of kangaroo care. J Nurs Manag. 2013;21(1):70–8.

Wagner AD, Mugo C, Bluemer-Miroite S, Mutiti PM, Wamalwa DC, Bukusi D, et al. Continuous quality improvement intervention for adolescent and young adult HIV testing services in Kenya improves HIV knowledge. AIDS. 2017;31(Suppl 3):S243–252.

Le RD, Melanson SE, Santos KS, Paredes JD, Baum JM, Goonan EM, et al. Using lean principles to optimise inpatient phlebotomy services. J Clin Pathol. 2014;67(8):724–30.

Manyazewal T, Mekonnen A, Demelew T, Mengestu S, Abdu Y, Mammo D, et al. Improving immunization capacity in Ethiopia through continuous quality improvement interventions: a prospective quasi-experimental study. Infect Dis Poverty. 2018;7:7.

Kamiya Y, Ishijma H, Hagiwara A, Takahashi S, Ngonyani HAM, Samky E. Evaluating the impact of continuous quality improvement methods at hospitals in Tanzania: a cluster-randomized trial. Int J Qual Health Care. 2017;29(1):32–9.

Kibbe DC, Bentz E, McLaughlin CP. Continuous quality improvement for continuity of care. J Fam Pract. 1993;36(3):304–8.

Adrawa N, Ongiro S, Lotee K, Seret J, Adeke M, Izudi J. Use of a context-specific package to increase sputum smear monitoring among people with pulmonary tuberculosis in Uganda: a quality improvement study. BMJ Open Qual. 2023;12(3):1–6.

Hunt P, Hunter SB, Levan D. Continuous quality improvement in substance abuse treatment facilities: how much does it cost? J Subst Abuse Treat. 2017;77:133–40.

Azadeh A, Ameli M, Alisoltani N, Motevali Haghighi S. A unique fuzzy multi-control approach for continuous quality improvement in a radio therapy department. Qual Quantity. 2016;50(6):2469–93.

Memiah P, Tlale J, Shimabale M, Nzyoka S, Komba P, Sebeza J, et al. Continuous quality improvement (CQI) institutionalization to reach 95:95:95 HIV targets: a multicountry experience from the Global South. BMC Health Serv Res. 2021;21(1):711.

Yapa HM, De Neve JW, Chetty T, Herbst C, Post FA, Jiamsakul A, et al. The impact of continuous quality improvement on coverage of antenatal HIV care tests in rural South Africa: results of a stepped-wedge cluster-randomised controlled implementation trial. PLoS Med. 2020;17(10):e1003150.

Dadi TL, Abebo TA, Yeshitla A, Abera Y, Tadesse D, Tsegaye S, et al. Impact of quality improvement interventions on facility readiness, quality and uptake of maternal and child health services in developing regions of Ethiopia: a secondary analysis of programme data. BMJ Open Qual. 2023;12(4):e002140.

Weinberg M, Fuentes JM, Ruiz AI, Lozano FW, Angel E, Gaitan H, et al. Reducing infections among women undergoing cesarean section in Colombia by means of continuous quality improvement methods. Arch Intern Med. 2001;161(19):2357–65.

Andreoni V, Bilak Y, Bukumira M, Halfer D, Lynch-Stapleton P, Perez C. Project management: putting continuous quality improvement theory into practice. J Nurs Care Qual. 1995;9(3):29–37.

Balfour ME, Zinn TE, Cason K, Fox J, Morales M, Berdeja C, et al. Provider-payer partnerships as an engine for continuous quality improvement. Psychiatric Serv. 2018;69(6):623–5.

Agurto I, Sandoval J, De La Rosa M, Guardado ME. Improving cervical cancer prevention in a developing country. Int J Qual Health Care. 2006;18(2):81–6.

Anderson CI, Basson MD, Ali M, Davis AT, Osmer RL, McLeod MK, et al. Comprehensive multicenter graduate surgical education initiative incorporating entrustable professional activities, continuous quality improvement cycles, and a web-based platform to enhance teaching and learning. J Am Coll Surg. 2018;227(1):64–76.

Benjamin S, Seaman M. Applying continuous quality improvement and human performance technology to primary health care in Bahrain. Health Care Superv. 1998;17(1):62–71.

Byabagambi J, Marks P, Megere H, Karamagi E, Byakika S, Opio A, et al. Improving the quality of voluntary medical male circumcision through use of the continuous quality improvement approach: a pilot in 30 PEPFAR-Supported sites in Uganda. PLoS ONE. 2015;10(7):e0133369.

Hogg S, Roe Y, Mills R. Implementing evidence-based continuous quality improvement strategies in an urban Aboriginal Community Controlled Health Service in South East Queensland: a best practice implementation pilot. JBI Database Syst Rev Implement Rep. 2017;15(1):178–87.

Hopper MB, Morgan S. Continuous quality improvement initiative for pressure ulcer prevention. J Wound Ostomy Cont Nurs. 2014;41(2):178–80.

Ji J, Jiang DD, Xu Z, Yang YQ, Qian KY, Zhang MX. Continuous quality improvement of nutrition management during radiotherapy in patients with nasopharyngeal carcinoma. Nurs Open. 2021;8(6):3261–70.

Chen M, Deng JH, Zhou FD, Wang M, Wang HY. Improving the management of anemia in hemodialysis patients by implementing the continuous quality improvement program. Blood Purif. 2006;24(3):282–6.

Reeves S, Matney K, Crane V. Continuous quality improvement as an ideal in hospital practice. Health Care Superv. 1995;13(4):1–12.

Barton AJ, Danek G, Johns P, Coons M. Improving patient outcomes through CQI: vascular access planning. J Nurs Care Qual. 1998;13(2):77–85.

Buttigieg SC, Gauci D, Dey P. Continuous quality improvement in a Maltese hospital using logical framework analysis. J Health Organ Manag. 2016;30(7):1026–46.

Take N, Byakika S, Tasei H, Yoshikawa T. The effect of 5S-continuous quality improvement-total quality management approach on staff motivation, patients’ waiting time and patient satisfaction with services at hospitals in Uganda. J Public Health Afr. 2015;6(1):486.

PubMed   PubMed Central   Google Scholar  

Jacobson GH, McCoin NS, Lescallette R, Russ S, Slovis CM. Kaizen: a method of process improvement in the emergency department. Acad Emerg Med. 2009;16(12):1341–9.

Agarwal S, Gallo J, Parashar A, Agarwal K, Ellis S, Khot U, et al. Impact of lean six sigma process improvement methodology on cardiac catheterization laboratory efficiency. Catheter Cardiovasc Interv. 2015;85:S119.

Rahul G, Samanta AK, Varaprasad G A Lean Six Sigma approach to reduce overcrowding of patients and improving the discharge process in a super-specialty hospital. In 2020 International Conference on System, Computation, Automation and Networking (ICSCAN) 2020 July 3 (pp. 1-6). IEEE

Patel J, Nattabi B, Long R, Durey A, Naoum S, Kruger E, et al. The 5 C model: A proposed continuous quality improvement framework for volunteer dental services in remote Australian Aboriginal communities. Community Dent Oral Epidemiol. 2023;51(6):1150–8.

Van Acker B, McIntosh G, Gudes M. Continuous quality improvement techniques enhance HMO members’ immunization rates. J Healthc Qual. 1998;20(2):36–41.

Horine PD, Pohjala ED, Luecke RW. Healthcare financial managers and CQI. Healthc Financ Manage. 1993;47(9):34.

Reynolds JL. Reducing the frequency of episiotomies through a continuous quality improvement program. CMAJ. 1995;153(3):275–82.

Bunik M, Galloway K, Maughlin M, Hyman D. First five quality improvement program increases adherence and continuity with well-child care. Pediatr Qual Saf. 2021;6(6):e484.

Boyle TA, MacKinnon NJ, Mahaffey T, Duggan K, Dow N. Challenges of standardized continuous quality improvement programs in community pharmacies: the case of SafetyNET-Rx. Res Social Adm Pharm. 2012;8(6):499–508.

Price A, Schwartz R, Cohen J, Manson H, Scott F. Assessing continuous quality improvement in public health: adapting lessons from healthcare. Healthc Policy. 2017;12(3):34–49.

Gage AD, Gotsadze T, Seid E, Mutasa R, Friedman J. The influence of continuous quality improvement on healthcare quality: a mixed-methods study from Zimbabwe. Soc Sci Med. 2022;298:114831.

Chan YC, Ho SJ. Continuous quality improvement: a survey of American and Canadian healthcare executives. Hosp Health Serv Adm. 1997;42(4):525–44.

Balas EA, Puryear J, Mitchell JA, Barter B. How to structure clinical practice guidelines for continuous quality improvement? J Med Syst. 1994;18(5):289–97.

ElChamaa R, Seely AJE, Jeong D, Kitto S. Barriers and facilitators to the implementation and adoption of a continuous quality improvement program in surgery: a case study. J Contin Educ Health Prof. 2022;42(4):227–35.

Candas B, Jobin G, Dubé C, Tousignant M, Abdeljelil A, Grenier S, et al. Barriers and facilitators to implementing continuous quality improvement programs in colonoscopy services: a mixed methods systematic review. Endoscopy Int Open. 2016;4(2):E118–133.

Brandrud AS, Schreiner A, Hjortdahl P, Helljesen GS, Nyen B, Nelson EC. Three success factors for continual improvement in healthcare: an analysis of the reports of improvement team members. BMJ Qual Saf. 2011;20(3):251–9.

Lee S, Choi KS, Kang HY, Cho W, Chae YM. Assessing the factors influencing continuous quality improvement implementation: experience in Korean hospitals. Int J Qual Health Care. 2002;14(5):383–91.

Horwood C, Butler L, Barker P, Phakathi S, Haskins L, Grant M, et al. A continuous quality improvement intervention to improve the effectiveness of community health workers providing care to mothers and children: a cluster randomised controlled trial in South Africa. Hum Resour Health. 2017;15(1):39.

Hyrkäs K, Lehti K. Continuous quality improvement through team supervision supported by continuous self-monitoring of work and systematic patient feedback. J Nurs Manag. 2003;11(3):177–88.

Akdemir N, Peterson LN, Campbell CM, Scheele F. Evaluation of continuous quality improvement in accreditation for medical education. BMC Med Educ. 2020;20(Suppl 1):308.

Barzansky B, Hunt D, Moineau G, Ahn D, Lai CW, Humphrey H, et al. Continuous quality improvement in an accreditation system for undergraduate medical education: benefits and challenges. Med Teach. 2015;37(11):1032–8.

Gaylis F, Nasseri R, Salmasi A, Anderson C, Mohedin S, Prime R, et al. Implementing continuous quality improvement in an integrated community urology practice: lessons learned. Urology. 2021;153:139–46.

Gaga S, Mqoqi N, Chimatira R, Moko S, Igumbor JO. Continuous quality improvement in HIV and TB services at selected healthcare facilities in South Africa. South Afr J HIV Med. 2021;22(1):1202.

Wang F, Yao D. Application effect of continuous quality improvement measures on patient satisfaction and quality of life in gynecological nursing. Am J Transl Res. 2021;13(6):6391–8.

Lee SB, Lee LL, Yeung RS, Chan J. A continuous quality improvement project to reduce medication error in the emergency department. World J Emerg Med. 2013;4(3):179–82.

Chiang AA, Lee KC, Lee JC, Wei CH. Effectiveness of a continuous quality improvement program aiming to reduce unplanned extubation: a prospective study. Intensive Care Med. 1996;22(11):1269–71.

Chinnaiyan K, Al-Mallah M, Goraya T, Patel S, Kazerooni E, Poopat C, et al. Impact of a continuous quality improvement initiative on appropriate use of coronary CT angiography: results from a multicenter, statewide registry, the advanced cardiovascular imaging consortium (ACIC). J Cardiovasc Comput Tomogr. 2011;5(4):S29–30.

Gibson-Helm M, Rumbold A, Teede H, Ranasinha S, Bailie R, Boyle J. A continuous quality improvement initiative: improving the provision of pregnancy care for Aboriginal and Torres Strait Islander women. BJOG: Int J Obstet Gynecol. 2015;122:400–1.

Bennett IM, Coco A, Anderson J, Horst M, Gambler AS, Barr WB, et al. Improving maternal care with a continuous quality improvement strategy: a report from the interventions to minimize preterm and low birth weight infants through continuous improvement techniques (IMPLICIT) network. J Am Board Fam Med. 2009;22(4):380–6.

Krall SP, Iv CLR, Donahue L. Effect of continuous quality improvement methods on reducing triage to thrombolytic interval for Acute myocardial infarction. Acad Emerg Med. 1995;2(7):603–9.

Swanson TK, Eilers GM. Physician and staff acceptance of continuous quality improvement. Fam Med. 1994;26(9):583–6.

Yu Y, Zhou Y, Wang H, Zhou T, Li Q, Li T, et al. Impact of continuous quality improvement initiatives on clinical outcomes in peritoneal dialysis. Perit Dial Int. 2014;34(Suppl 2):S43–48.

Schiff GD, Goldfield NI. Deming meets Braverman: toward a progressive analysis of the continuous quality improvement paradigm. Int J Health Serv. 1994;24(4):655–73.

American Hospital Association Division of Quality Resources Chicago, IL: The role of hospital leadership in the continuous improvement of patient care quality. American Hospital Association. J Healthc Qual. 1992;14(5):8–14,22.

Scriven M. The Logic and Methodology of checklists [dissertation]. Western Michigan University; 2000.

Hales B, Terblanche M, Fowler R, Sibbald W. Development of medical checklists for improved quality of patient care. Int J Qual Health Care. 2008;20(1):22–30.

Vermeir P, Vandijck D, Degroote S, Peleman R, Verhaeghe R, Mortier E, et al. Communication in healthcare: a narrative review of the literature and practical recommendations. Int J Clin Pract. 2015;69(11):1257–67.

Eljiz K, Greenfield D, Hogden A, Taylor R, Siddiqui N, Agaliotis M, et al. Improving knowledge translation for increased engagement and impact in healthcare. BMJ open Qual. 2020;9(3):e000983.

O’Brien JL, Shortell SM, Hughes EF, Foster RW, Carman JM, Boerstler H, et al. An integrative model for organization-wide quality improvement: lessons from the field. Qual Manage Healthc. 1995;3(4):19–30.

Adily A, Girgis S, D’Este C, Matthews V, Ward JE. Syphilis testing performance in Aboriginal primary health care: exploring impact of continuous quality improvement over time. Aust J Prim Health. 2020;26(2):178–83.

Horwood C, Butler L, Barker P, Phakathi S, Haskins L, Grant M, et al. A continuous quality improvement intervention to improve the effectiveness of community health workers providing care to mothers and children: a cluster randomised controlled trial in South Africa. Hum Resour Health. 2017;15:1–11.

Veillard J, Cowling K, Bitton A, Ratcliffe H, Kimball M, Barkley S, et al. Better measurement for performance improvement in low- and middle-income countries: the primary Health Care Performance Initiative (PHCPI) experience of conceptual framework development and indicator selection. Milbank Q. 2017;95(4):836–83.

Barbazza E, Kringos D, Kruse I, Klazinga NS, Tello JE. Creating performance intelligence for primary health care strengthening in Europe. BMC Health Serv Res. 2019;19(1):1006.

Assefa Y, Hill PS, Gilks CF, Admassu M, Tesfaye D, Van Damme W. Primary health care contributions to universal health coverage. Ethiopia Bull World Health Organ. 2020;98(12):894.

Van Weel C, Kidd MR. Why strengthening primary health care is essential to achieving universal health coverage. CMAJ. 2018;190(15):E463–466.

Download references

Acknowledgements

Not applicable.

The authors received no fund.

Author information

Authors and affiliations.

School of Public Health, The University of Queensland, Brisbane, Australia

Aklilu Endalamaw, Resham B Khatri, Tesfaye Setegn Mengistu, Daniel Erku & Yibeltal Assefa

College of Medicine and Health Sciences, Bahir Dar University, Bahir Dar, Ethiopia

Aklilu Endalamaw & Tesfaye Setegn Mengistu

Health Social Science and Development Research Institute, Kathmandu, Nepal

Resham B Khatri

Centre for Applied Health Economics, School of Medicine, Grifth University, Brisbane, Australia

Daniel Erku

Menzies Health Institute Queensland, Grifth University, Brisbane, Australia

International Institute for Primary Health Care in Ethiopia, Addis Ababa, Ethiopia

Eskinder Wolka & Anteneh Zewdie

You can also search for this author in PubMed   Google Scholar

Contributions

AE conceptualized the study, developed the first draft of the manuscript, and managing feedbacks from co-authors. YA conceptualized the study, provided feedback, and supervised the whole processes. RBK provided feedback throughout. TSM provided feedback throughout. DE provided feedback throughout. EW provided feedback throughout. AZ provided feedback throughout. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Aklilu Endalamaw .

Ethics declarations

Ethics approval and consent to participate.

Not applicable because this research is based on publicly available articles.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Endalamaw, A., Khatri, R.B., Mengistu, T.S. et al. A scoping review of continuous quality improvement in healthcare system: conceptualization, models and tools, barriers and facilitators, and impact. BMC Health Serv Res 24 , 487 (2024). https://doi.org/10.1186/s12913-024-10828-0

Download citation

Received : 27 December 2023

Accepted : 05 March 2024

Published : 19 April 2024

DOI : https://doi.org/10.1186/s12913-024-10828-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Continuous quality improvement
• Quality of Care

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Open access
• Published: 24 April 2024

Access to family planning services and associated factors among young people in Lira city northern Uganda

• Eustes Kigongo 1 ,
• Raymond Tumwesigye 1 ,
• Maxson Kenneth Anyolitho 1 ,
• Marvin Musinguzi 1 ,
• Gad Kwizera 3 ,
• Everlyne Achan 1 ,
• Caroline Kambugu Nabasirye 2 ,
• Samson Udho 2 ,
• Amir Kabunga 4 &
• Bernard Omech 1  

BMC Public Health volume  24 , Article number:  1146 ( 2024 ) Cite this article

Metrics details

Access to family planning services among young people is crucial for reproductive health. This study explores the access and associated factors among young people in Lira City, Northern Uganda.

Methods and materials

A mixed-methods study was conducted in March to April 2022. Quantitative data were collected using a structured questionnaire from 553 participants aged 15–24 years. Qualitative data were obtained through in-depth interviews and focus group discussions. Data analysis included univariate, bivariate, and multivariate analyses for quantitative data, while interpretative phenomenological analysis was used for qualitative data.

Overall, 31.7% of the respondents had a good perceived access to family planning services, with 64.6% reporting perceived availability of FP methods. Challenges included lack of privacy (57.7%), fear of mistreatment (77.2%), and decision-making difficulties (66.2%). Among females, good perceived access to FP services was less likely among urban residents (AOR: 0.22, 95% CI: 0.09–0.53), Christian respondents (AOR: 0.51, 95% CI: 0.01–0.36), Muslim respondents (AOR: 0.07, 95% CI: 0.01–0.55) and respondents with poor attitude to FP services (AOR: 0.39, 95% CI: 0.24–0.64), but more likely among respondents with a sexual a partner (AOR: 4.48, 95% CI: 2.60–7.75). Among males, good perceived access to FP services was less likely among respondents living with parents (AOR: 0.19, 95% CI: 0.05–0.67) but more likely among respondents with good knowledge of FP services (AOR: 2.28, 95% CI: 1.02–5.32). Qualitative findings showed that three themes emerged; knowledge of family planning methods, beliefs about youth contraception and, friendliness of family planning services.

The study revealed a substantial gap in perceived access to family planning services among young people in Lira City. Barriers include privacy concerns, fear of mistreatment, and decision-making difficulties. Tailored interventions addressing urban access, religious beliefs for females, and knowledge enhancement for males are essential. Positive aspects like diverse FP methods and physical accessibility provide a foundation for targeted interventions. Youth-friendly services, comprehensive sexual education, and further research are emphasized for a nuanced understanding and effective interventions in Northern Uganda.

Peer Review reports

Globally, approximately 16 million girls aged 15–19 give birth each year, with 95% of these births occurring in developing countries [ 1 ]. Additionally, annually, 14 million unsafe abortions take place among adolescents, who face various sexual and reproductive health challenges, including early pregnancy, unsafe abortions, sexually transmitted infections (STIs), and sexual abuse, particularly in Sub-Saharan Africa (SSA) [ 2 ]. Family planning (FP) is a critical aspect of global public health, recognized for its impact on maternal and child health, gender equality, and socioeconomic development [ 3 ]. The international community, as reflected in various global health initiatives and sustainable development goals, acknowledges the importance of ensuring universal access to FP services for all individuals, including young people [ 4 ]. This global perspective emphasizes the interconnectedness of reproductive health and broader efforts to achieve sustainable development [ 4 ]. Access to these services is particularly pertinent among young people, who constitute a significant demographic in many Low-and Middle-income Countries (LMICs).

Uganda, boasting one of the world’s youngest and fastest-growing populations, has nearly half (48%) of its estimated 46 million people under the age of 15, significantly surpassing the averages for SSA (43%) and the world (26%) [ 5 ]. Uganda, as a signatory to global health agendas, has made significant strides in promoting FP services [ 6 ]. The National Population Policy, coupled with the National Reproductive Health Policy, reflects the government’s commitment to ensuring access to FP for all citizens [ 7 ]. However, challenges persist, especially in urban areas. An examination of the national context provides insights into the policy circumstances, healthcare infrastructure, and societal norms that shape family planning services’ availability and utilization among young people in Northern Uganda.

According to the Uganda Demographic and Health Survey (UDHS) 2016, 25% of women aged 15–19 and 1% below 15 had initiated childbearing, with the incidence of unplanned pregnancies significantly rising following the shutdown of schools during the COVID-19 pandemic [ 8 , 9 ]. Reported underlying causes of teenage pregnancy include gender inequality, restricted freedom for girls to voice their concerns, school dropout, and limited access to contraception and knowledge [ 10 ]. Unintended teenage pregnancies can have severe adverse effects on well-being, leading to maternal morbidity and mortality related to childbirth and unsafe abortion [ 11 ]. Moreover, these pregnancies contribute to social consequences such as stigma and discrimination, accounting for 59% of school dropouts in Uganda in 2012, potentially hindering education and future employment opportunities [ 12 , 13 ]. Reports by the World Bank and the World Health Organization (WHO) emphasize the association of adolescents-childbearing with social stigma, lifelong poverty, and health risks, necessitating a comprehensive approach to address these issues [ 14 , 15 ].

Uganda’s current health sector strategy aims to expand youth-friendly health services (YFHS) and promote adolescent sexual and reproductive health and rights information in schools, ensuring access to FP information and services irrespective of age, marital status, or school status [ 16 ]. The country plans to increase access to modern contraceptive use and reduce unmet need for contraception in the coming years [ 17 ]. According to WHO guidelines, addressing the underlying factors, including the timing of first sex and marriage, effective contraceptive use, and the socio-cultural and economic environment, is crucial for delaying childbearing and expanding FP access to adolescents [ 18 ].

Notably, northern Uganda bears one of the highest burdens of adolescent pregnancies, with reports indicating a significant percentage of unintended pregnancies in Lira district in 2019 (33.3%) and a notable number of teenage girls visiting antenatal clinics in Lango sub-region in 2021 [ 19 ]. A recent study in Oyam district reported a high percentage of unintended pregnancies among adolescent girls [ 8 ]. In the context of Lira City in northern Uganda, unintended pregnancies represent a significant challenge affecting various aspects of young people’s lives, including education and economic prospects [ 20 ]. Despite the recognized importance of FP, there is a need for a comprehensive understanding of the factors hindering or facilitating youth access to FP services in Lira City. Existing literature primarily focuses on the prevalence of unintended pregnancies and associated outcomes, emphasizing challenges in accessing reliable information, contraceptives, and quality reproductive health services [ 20 ]. However, there is limited research examining the factors contributing to these challenges, such as cultural norms, stigma, and structural barriers specific to Lira City. Moreover, the evolving circumstances of youth perspectives, preferences, and behaviors related to FP require an updated understanding, considering rapid socio-cultural changes and advancements in technology [ 21 ]. To develop evidence-based intervention strategies, our assessment focused on the knowledge, perceptions, and factors influencing access to contraceptive services among young people in the specific context of northern Uganda.

Study design

This was an explanatory-sequential mixed methods study [ 22 ] conducted in Lira city, northern Uganda between March and April 2023. The mixed-methods approach was adopted so as to generate a more holistic understanding and a stronger inference with two approaches complementing each other [ 23 ].

Study setting

Lira City is among the newly created cities, located approximately 375 km by road north of the capital city of Kampala via Karuma-Kamdini. Lira City is the central business hub for Northern Uganda and comprises the west and east divisions. According to projections by the Uganda Bureau of Statistics (UBOS) in 2014, the population of 2020 for the Lira district was 474,200 people, and it is traditionally inhabited by the Lango tribe, who are farmers and cattle keepers. The urban centers of the district also have people engaged in many small-scale businesses, such as produce businesses and trading.

Study population

The study was among young people aged 15 to 24 years, residing in Lira city. Inclusion into the study was based on being a young person of 15 to 24 years of age who has lived in Lira city for at least six months. Additionally, being present at the selected household during data collection, and those who consented to participate were included in the study. In households where more than one persons were eligible, simple random sampling by lottery method was employed to select one. Exclusion was based on being critically ill to participate, or refusing to participate in the interviews.

Sample size determination

The sample size of the study was estimated using Kish Leslie (1965) as follows:

In the equation above, n is the sample size for the study, Z is the Z score at the 95% confidence interval (1.96), p is the proportion of perceived access to FP services (50%), d is the desired precision of the study (5%), and deff is the design effect due to multistage random sampling. A factor of 1.5 has been used to adjust the sample size based on what previous studies have used [ 24 , 25 ]. A design effect of 1.5 was employed to increase the homogeneity of the participants following the use of a multistage random sampling procedure. Therefore, the final sample size obtained was 577.

Interpretative phenomenological analysis (IPA) was employed for qualitative research to delve into individual experiences, progressing towards an examination of shared and contrasting aspects within a limited sample [ 26 ]. This approach facilitated the identification of thematic connections. Adhering to IPA guidelines advocating for a compact and homogenous sample, purposive sampling was used to recruit 5 participants. This sample size aligns with the recommended number for an IPA study [ 27 ], and was considered sufficient to capture a distinct range of experiences related to the phenomenon under investigation.

Sampling technique

A multistage sampling procedure was employed to select the 577 study participants. The study was conducted in both divisions of Lira City, East and West. From each of the divisions, five wards were selected, making a total of ten wards. This was done by simple random sampling using the lottery method, where the names of wards were written on small papers, folded, mixed in a container, and shaken well, and then five were picked at random without replacement. From each of the wards, two cells were selected using the same procedure, which generated a total of 20 cells. From each of the cells, Village Health Teams (VHTs) were used to obtain lists of households with young people aged 15 to 24 years, and these were used as sampling frames per cell. The number of participants to be selected from each cell was determined by the sample size proportionate to the cell size. In each of the cells, participants were selected through simple random sampling using computer-generated random numbers. Purposive sampling was used to select participants for qualitative interviews [ 28 ]. While purposive sampling guided our selection process, we also sought to include a diverse range of perspectives by engaging with individuals from various backgrounds, including community health workers, educators, and youth leaders. Our rationale for selecting community peer educators stems from their unique position as trusted intermediaries within their communities, often serving as frontline advocates for reproductive health education and services. Similarly, the inclusion of university leaders was motivated by their influence and role in shaping policies and programs related to youth reproductive health within academic settings.

Study variables

Dependent variable.

The dependent variable for the study is perceived access to FP services. Access to healthcare means “the timely use of personal health services to achieve the best health outcomes” [ 29 ]. Many frameworks have been proposed to measure access to family planning services but have all proved not sufficient [ 30 ]. This study adopted one of the common frameworks, Penchansky and Thomas (1981) framework that reflects the fit between characteristics and expectations of the providers and the clients. These characteristics (5As of access) are availability, accessibility, acceptability, accommodation, and affordability [ 31 ]. This conceptualization of access has been adopted because it describes the broad dimensions and determinants that integrate demand and supply-side factors [ 32 ]. According to the model, the five As of access form a chain that is no stronger than its weakest link. For example, improving affordability by providing health insurance will not significantly improve access and utilization if the other four dimensions have not also been addressed. The perception of access to FP services index composed of five questions of yes or no response. For all the questions “yes” was coded 2 and “no” coded 1. The percentage of respondents that perceived access to be good on all five variables had good perceived access to FP services.

Availability: Are the family planning commodities available when you need them, and meet your FP needs?

Accessibility: Is the location of the facilities that provide family planning services convenient for you?

Acceptability: Are the characteristics of the FP service providers (including attitudes and attributes such as age, sex and religion) comfortable for you?

Accommodation: Do health providers organize FP services in ways (including appointment system, hours of operation and facility environment) that suit your needs and preferences?

Affordability: Do you have to pay for family planning services?

All the access questions were asked as yes and no questions and coded 1 and 2, respectively. To measure the index of perceived access, only participants who answered Yes to all the access questions were labeled as having good perceived access to FP services.

Independent variables

The independent variables included sociodemographic characteristics (age, sex, education, religion, marital status, living with parents), sexual-related characteristics (having a child, sexually active, sexual partners), knowledge, and attitudes. The knowledge of the participants was assessed based on a total of nine questions about family planning. Each of the questions was binary coded as 1(Yes) and 0(No). Overall knowledge was therefore measured as a composite score ranging from 0 to 9. The mean score was taken as a cut-off with individuals above the mean score categorized as having good knowledge and those below the mean as having poor knowledge. This measurement was adopted from a recent study [ 33 ]. The overall attitudes of the young people regarding actual use of family planning commodities, which includes the misconceptions, fears, cultural and religious beliefs about family planning commodities such as condoms were assessed based on a total of eight questions with a favorable response coded as Yes (0) and unfavorable response coded as No [ 1 ]. The responses were computed into an overall attitude to FP services score with a total of eight. Similarly, the cut-off was set as the mean with individuals above the mean classified as having a poor perception and those below the mean with a good perception, as from a recent related study [ 33 ]. The knowledge items had a scale reliability coefficient of 0.78 whereas the perception items had 0.70, all these are within the acceptable limits [ 34 ].

Participant recruitment and informed consent processes

After obtaining ethical approval and clearance, five research assistants from the city were recruited and trained on the study protocol and data collection procedures. A pretest of the questionnaire was carried out among 58 youths from Lira district to refine the questions for simplicity and comprehension and to assess validity and reliability using the Statistical Package for Social Sciences (SPSS) software. Lists of households with young people aged 15 to 24 years were obtained by Village Health Teams (VHTs). Sampling was then conducted, and eligible participants were approached for data collection after providing informed consent and, for minors, informed assent. During this process, the study objectives, procedures, benefits, risks, and voluntarism were explained. Interviews took place in a private space within the participants’ homes. In cases where the parent or guardian was absent during data collection, the household was skipped.

Data collection instruments

Quantitative data was collected using a pretested interviewer-administered questionnaire developed by the researcher (Supplementary file 1 ). The questionnaire consisted of four sections: sociodemographic characteristics (age, sex, education, religion, marital status, residence, and parent’s education), sexually related information (ever had a child, engaged in sexual relationships, number of sexual partners, sexual risks encountered), access questions (availability, accessibility, acceptability, accommodation, and affordability), knowledge of family planning services, and attitudes regarding family planning services. This was administered in approximately 15 min. Qualitative data was collected through in-depth interviews and focus group discussions using guides (Supplementary file 2 ). This was done after obtaining insights from quantitative data. Interviews with participants were done at proposed times and places deemed convenient to the participants themselves. During collection, audio recordings were made together with extended field notes to complement the audios. Data collection was done in Lango, verbatim transcribed, and then translated to English for analysis. Data collection was conducted through five in-depth interviews and four focus group discussions all from young people aged 15 to 24 years. A sample of 10 were from the University and 30 were from the community with equal proportions of males and females. These participants were community adolescent peer-educators and University reproductive health leaders. Some were picked after quantitative interviews while others based on their roles regarding reproductive health for the young people.

Statistical analysis

Quantitative data analysis.

The collected data was entered into SPSS software, where it was cleaned and coded, then exported to STATA version 17 software for final analysis. The analysis was conducted at three levels. At the univariate level, data was summarized as frequencies and proportions, means and standard deviations, or median with interquartile range, and presented in frequency tables. In bivariate analysis, perception of access to SRHR services was cross-tabulated with the independent variables one at a time to assess relationships. A crude odd ratio (COR) and a 95% confidence interval were reported. At this level, associations were considered at p  < 0.25 in order to consider all possible predictors [ 35 ], and all those associated factors were taken into multivariate analysis. In multivariate analysis, binary logistic regression was used to estimate the predictors of the primary outcome. The backward elimination method was used to build a predictive model. Results were reported as adjusted odds ratios and 95% confidence intervals. A p -value of < 0.05 was considered statistically significant for variables.

Qualitative data analysis

The data analysis adhered to the seven-stage IPA process outline, derived from Smith and colleagues, as outlined by Brown and colleagues [ 36 ]. Each interview underwent verbatim transcription and was entered into a customized IPA analysis framework. Multiple re-readings of the interviews were conducted, applying in-method triangulation by integrating field notes with observations and commentary from the fieldwork [ 37 ]. This triangulation process enhanced confidence in the outcomes post data analysis. Following the verbatim transcription of the audio data and thorough review of the text, initial notes were made, leading to the development of emerging themes. Connections across these emergent themes were sought to identify subordinate themes. Subsequently, a search for patterns across the cases was conducted to reveal the major themes.

We employed research assistants who are social scientists trained in qualitative study and interview techniques to assure the validity of our study. Data from diverse sources, including field notes and audio recordings, were independently analyzed by two researchers. The newly emerging themes were routinely compared to the original transcribed text, and the writers frequently convened for debriefings to make sure that the subjects were at the center of the data analysis and interpretation. The results of the data analysis were examined and discussed until a consensus was achieved in order to increase the dependability and accuracy of the results. To demonstrate confirmability (the degree to which the findings are shaped by participants and the context rather than the perspectives of the research), the researchers used participants’ narratives and words as noted in the transcripts. Additionally, the researchers dwelled on their previous experiences to reduce their influence on the findings. To ensure that the processes of data collecting and analysis could be traced back to the initial interviews, we have preserved all audit trails from data collection to analysis.

Quantitative findings

Sociodemographic characteristics.

Recruitment into the study was between March and April 2022. Out of a total of 577 participants, 553 were included generating a response rate of 95.8%. Table  1 shows that the majority of the respondents, 65.3% were female, with a mean age of 17 (± 2.1) years and 90.8% aged between 15 and 19 years. Most of the youth, 45.2% were in secondary school, 40.7% were Anglican, and 71.4% were living with their parents. The majority of the youths, 46.8% were sexually active and had had sex in the past 4 months.

Perceived access to family planning services

The mean score for perception of access to family planning services was 1.91 with a standard deviation of ± 0.29. Figure  1 shows that the percentage of respondents that perceived access to be good for all the five variables was 31.7% (95% CI: 28%, 36%). The majority of the young people, 64.6% reported that different FP methods were available at the health facilities. Most of the young people, 79.3%% also reported that the health facilities were within their reach, and 61.3% reported that attitudes and personal characteristics of FP service providers were comfortable for them. The majority of the young people, 66.7% also reported that the manner in which FP services are organized, including facility’s operating and environment, suited their needs and preferences. Additionally, females had overall favorable responses compared to their male counterparts.

Percentage of young people reporting a good response to variables on the perceived access index in Lira district, Northern Uganda

Knowledge and attitudes regarding perceived access to family planning services

Table  2 presents questions used to assess both knowledge and perceptions regarding family planning services. questions 1 to 9 were designed to measure knowledge and 10 to 17 were aimed at capturing perceptions regarding use of family planning commodities. The majority of the young people, 69.4% were aware that FP, 75.6% knew the facility that offers FP services, 89.3% knew how to prevent pregnancy and 75.8% knew about sexual rights. Table  3 also shows that the majority of the young people, 58.1% perceived that FP services were not for young people, 80.5% could access FP whenever they wanted and 90.4% knew that information at the health facility was always kept confidential. However, the majority of the young people, 66.2% cannot decide on using a FP method, 57.7% also reported that there is not enough privacy at the health facilities, and 77.2% fear being mistreated by the staff at the health facilities.

Factors associated with perceived access to family planning services among young people

The bivariate analysis was performed stratified by sex to prevent introduction of bias arising from differencing sample sizes because males were close to a third of the entire sample. Table  3 indicates that among the females, being aged 20–24 years, having a child, being sexually active and having a sexual partner was associated with a higher perceived access to FP services at p  value less than 0.25. On the other hand, having primary and secondary education, urban residence, Christians, Muslims, not in a marital relationship, secondary education of a mother, primary, secondary and tertiary education of father, and good attitude towards FP services were associated with a lower perceived access to FP services at p  value less than 0.25. Table  3 also shows that among males, living with parents, mother’s secondary education level, and good attitude towards FP services had a lower perceived access to FP, where as being sexually active and good overall knowledge of FP services had higher perceived access to FP services with p  value of less than 0.25.

Multivariate analysis was performed to assess the predictors of perceived access to family planning services for males and females, presented as two separate models for females and males. Table  4 show among females, residence, religion, sexual partners and perception regarding use of family planning methods had significant associations. Females respondents who were less likely to consider access to family planning services as good were urban residents (AOR: 0.22, 95% CI: 0.09–0.53, p  = 0.001), those who were Christian (AOR: 0.51, 95% CI: 0.01–0.36, p  = 0.003) and Muslim (AOR: 0.07, 95% CI: 0.01–0.55, p  = 0.012), and those who had a poor attitude towards family planning services (AOR: 0.39, 95% CI: 0.24–0.64, p  < 0.001). Female respondents who had a sexual partner were more likely to consider access to family planning services as good (AOR: 4.48, 95% CI: 2.60–7.75, p  < 0.001). Table  4 also shows that among males, living with parents and overall knowledge about family planning services were significantly associated with perceive access to family planning services. Male respondents that were less likely to consider access to family planning as good were those who lived with their parents (AOR: 0.19, 95% CI: 0.05–0.67, p  = 0.010), and those that were more likely to consider access to family planning as good were those that had good overall knowledge about family planning methods (AOR: 2.28, 95% CI: 1.02–5.32, p  = 0.050).

Qualitative findings

Characteristics of participants.

A total of 5 in-depth interviews and 4 FGDs were conducted with a total of 30 young people; 10 were university students, whereas 20 were from the community. The focus groups were homogeneous in nature, for males and females separated. Two focus groups of 10 participants for males and females were conducted in the community, and two groups of five males and females were conducted from the University youths. The participants were young people aged 15 to 24 years. Themes were obtained through finding similar texts, patterns, and insights. We generated 8 different codes, 7 subthemes, and 3 themes.

Theme 1: knowledge of family planning methods

Majority of the young people did not have adequate knowledge regarding access and use of FP services. This was evidenced as most participants from the community reported that many young people used off label benefits of paracetamol and traditional herbal medicines for contraception. Additionally, many reported their source of information to be friends who seemed not to have adequate knowledge as well. Here are some of the verbatim comments to support the results:

“After having sex today, you can take 4 Panadol tablets immediately after having sex or 6 tablets, though it depends, you can also take it a day after having the sexual intercourse, taking on the third day will be late for it to work well in preventing the pregnancy”. (Female, 22 years, Lira Town, Feb 2023) “When I was in Primary six class. I was living with my sister and she had maids who told me about Panadol use”. (Female, 21 years, Junior quarters, Feb 2023) “Some girls use paw paw leaves, others mixing diclofenac drug with herbs which can also cause abortion. But these procedures can also either lead to incomplete abortion, death or even over bleeding” (Male, 24 years, Barapwo, Feb 2023) . “There is no proper sexual information. In the past, parents called children to prepare for education but today nowhere it’s practiced. Now it is only in schools to ensure that people know that sex is good but has challenges” (Female, 23 years, LU, Feb 2023) .

Theme 2: beliefs about youth contraception

The majority of the participants also reported negative perceptions regarding family planning services. However, this appears to stem from the common narrative that frames sexual health for young people as taboo. To continue, many young people and the community reported distancing themselves from reproductive health programs, citing that their motives are not entirely transparent. Here are some of the quotes that were recorded to emphasize the narrative:

“I see no meaning in engaging in such because they are just avenues for disseminating homosexuality and encouraging the youths to abort. They come in the sense of advocating for rights but instead teach that abortion and homosexuality is okay and a human right” (Female, 24 years, LU, Feb 2023) . “Family planning services are for big people. But there is need for a comprehensive guidance in matters of Sexual health for adolescents and adults about hygiene and opposite sex interaction” (Male, 16 years, Lira town, Feb 2023) .

Theme 3: friendliness of family planning services

Most of the participants reported that reproductive health services for young people are not friendly. The services are provided in environments that do not guarantee privacy and confidentiality, as well as during inflexible hours. To emphasize the narrative, here are a few verbatim comments:

It’s very difficult to go and access family planning services like pills from the teaching hospital…, can you imagine being served by your own lecturer who discourages having sex before marriage……Hmmm it’s funny! (Female, 24 years, LU, Feb 2023) A friend can help buy contraceptives if the user is known to the health worker who is selling. The seller might inform the buyer’s parents when one goes to buy condoms. (Male, 18 years, Amuca, Feb 2023)

The study aimed to assess perceived access to FP services and associated factors among young people in Lira City, Northern Uganda. Though many models have been suggested to measure access, they have all showed deficiencies in measuring actual access to family planning methods [ 30 ]. This study adopted the Penchansky and Thomas (1981) framework that measures perception of access through a 5-item index to explore the level of perceived access in this study. Findings of the current study showed that good perceived access to FP services was among 31.7% of respondents, with 64.6% reporting availability, 76.5% accessibility, 61.3% acceptability, 66.7% accommodative and 87.9% affordability of FP methods at health facilities. Our study indicates a low perceived access to FP services. Among the various components, availability, acceptability and accommodation pose significant obstacles to contraceptive access. A similar study in South Africa also reported the accommodation component as the greatest obstacle for accessing FP services due to integrated care, long waiting hours, and limited operational hours [ 38 ]. Additionally, the study reported that community were less concerned about the availability of trained service providers and a variety of contraceptive methods [ 38 ]. These possibly explain the low perceived access in the current study. In line with the current study, a recent study on utilization of sexual and reproductive services including family planning among young people in Lira city also reported a low level of 42% [ 39 ].

The overall perceived access to FP services at 31.7% suggests a substantial gap in service availability, indicating the need for targeted interventions to enhance accessibility. The presence of different FP methods at health facilities (64.6%) is a positive aspect, but the study unveils underlying challenges that contribute to the overall low perceived access. One of the key positive findings is the proximity of health facilities for 79.3% of participants, emphasizing the importance of physical accessibility. Additionally, positive perceptions towards use of family planning commodities, such as acceptability of FP use by the young people (61.3%) and a conducive environment at health facilities (66.7%), indicate a foundation upon which interventions can build. However, challenges identified, particularly for females, including a lack of privacy (57.7%), fear of mistreatment by staff (77.2%), and difficulties in decision-making regarding FP use (66.2%), highlight the nature of barriers to access. These challenges align with existing literature on the importance of privacy [ 40 ], quality of service [ 41 ], and decision-making autonomy in shaping individuals’ willingness to utilize FP services [ 42 ].

Quantitative findings revealed significant associations between perceived access to FP services and various sociodemographic factors, emphasizing the complexity of the issue. For females, urban residence, religion, having sexual partners, and perception were identified as influencing factors, while for males, living with parents and overall knowledge played a significant role. These associations underline the necessity for tailored interventions that consider the specific challenges faced by each gender. Qualitative findings highlighted insufficient knowledge, negative perceptions, and unfriendly FP services. These findings provide a deeper understanding of the barriers, emphasizing inadequate knowledge of FP methods, negative cultural and societal perceptions about youth contraception, and unfriendly service environments. These findings are consistent with existing literature, highlighting the role of cultural perceptions, knowledge gaps, and service quality in shaping young people’s access to FP services [ 43 ]. In agreement with previous studies, the study underscores the importance of comprehensive sexual education programs and youth-friendly service initiatives [ 44 ]. Our study shows a notable link between Islam and Catholicism and perceived access to FP services, aligning with previous research on religious influences that notes that the use of contraception is not promoted by any of the two religions [ 45 ]. Further exploration and comparative analysis with other studies may help elucidate these discrepancies and provide a more nuanced understanding of the factors influencing access to FP services among young people in Northern Uganda.

Strength and limitations

The study benefits from a mixed-methods approach, which integrates both qualitative and quantitative data to offer a comprehensive understanding of the factors influencing young people’s perceived access to family planning services. However, the cross-sectional design presents a limitation as it hinders the establishment of causality, providing only a snapshot of the situation at a specific moment and limiting exploration of temporal relationships over time. Acknowledging the small sample size and the potential bias introduced by selecting individuals with extensive knowledge on the topic, we recognize the limitation on the generalizability of our findings only to Lira City. The selection of individuals for IDIs may have inadvertently limited the diversity of perspectives represented in our study. Furthermore, participants may exhibit social desirability bias, particularly in studies addressing sensitive topics like sexual and reproductive health. Recall bias among participants, particularly when recalling past experiences related to sensitive topics or events that occurred some time ago, is also a possibility. Lastly, the quantitative sample was skewed towards females and those aged 15–19 years, potentially affecting the representativeness of the findings.

Our study reveals a substantial gap in perceived access to family planning services among young people. Despite high awareness, barriers like privacy concerns and fear of mistreatment contribute to low access. Tailored interventions are needed, focusing on urban service access, religious beliefs for females, and knowledge enhancement for males. Positive aspects, such as diverse FP methods and physical accessibility, form a foundation for interventions. The study emphasizes the importance of youth-friendly services, comprehensive sexual education, and further research for a nuanced understanding and targeted interventions in Northern Uganda.

Data availability

The data for the study is not publicly available due to restrictions from the Research Ethics Committee (REC) for posting of public data. However, can be accessed from the principal investigator on a reasonable request ([email protected]).

Abbreviations

Sexual Reproductive Health and Rights

Sub-Saharan Africa

Village Health Team

World Health Organization

Stanburry POHCHR. 2021 [cited 2022 May 3]. Young people need their sexual and reproductive health and rights. https://www.ohchr.org/en/stories/2021/05/young-people-need-their-sexual-and-reproductive-health-and-rights

UNICEF. Adolescent development [Internet]. 2022 [cited 2022 May 3]. https://www.unicef.org/uganda/what-we-do/adolescent-development

Bongaarts J, Hardee K. The role of public-sector family planning programs in meeting the demand for contraception in Sub-saharan Africa. Int Perspect Sex Reprod Health. 2017;43(2):41–50.

Article   PubMed   Google Scholar  

Organization WH. A vision for primary health care in the 21st century: towards universal health coverage and the Sustainable Development Goals. World Health Organization; 2018.

09_2021Uganda-National-Survey-Report-2019-2020.pdf [Internet]. [cited 2023 Sep 9]. https://www.ubos.org/wp-content/uploads/publications/09_2021Uganda-National-Survey-Report-2019-2020.pdf

Nabwowe Kasule A, Mncwabe N. Reclaiming public health services in Uganda. 2021.

Boydell V, Neema S, Wright K, Hardee K. Closing the gap between people and programs: lessons from implementation of social accountability for family planning and reproductive health in Uganda. Afr J Reprod Health. 2018;22(1):73–84.

PubMed   Google Scholar  

Okalo P, Arach AA, Apili B, Oyat J, Halima N, Kabunga A. Predictors of Unintended Pregnancy Among Adolescent Girls During the Second Wave of COVID-19 Pandemic in Oyam District in Northern Uganda. Open Access J Contracept [Internet]. 2023 Dec 31 [cited 2023 Mar 27];14:15–21. https://www.tandfonline.com/doi/abs/ https://doi.org/10.2147/OAJC.S399973

Musinguzi M, Kumakech E, Auma AG, Akello RA, Kigongo E, Tumwesigye R et al. Prevalence and correlates of teenage pregnancy among in-school teenagers during the COVID-19 pandemic in Hoima district western Uganda–A cross sectional community-based study. PLOS ONE [Internet]. 2022 Dec 16 [cited 2023 Mar 27];17(12):e0278772. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0278772

Ochen AM, Chi PC, Lawoko S. Predictors of teenage pregnancy among girls aged 13–19 years in Uganda: a community based case-control study. BMC Pregnancy Childbirth [Internet]. 2019 Jun 24 [cited 2023 Mar 28];19(1):211. https://doi.org/10.1186/s12884-019-2347-y

Yaya S, Zegeye B, Ahinkorah BO, Oladimeji KE, Shibre G. Inequality in fertility rate among adolescents: evidence from Timor-Leste demographic and health surveys 2009–2016. Arch Public Health [Internet]. 2020 Oct 14 [cited 2023 Mar 27];78:98. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7557032/

Leerlooijer JN, Bos AE, Ruiter RA, van Reeuwijk MA, Rijsdijk LE, Nshakira N et al. Qualitative evaluation of the Teenage Mothers Project in Uganda: a community-based empowerment intervention for unmarried teenage mothers. BMC Public Health [Internet]. 2013 Sep 8 [cited 2023 Mar 27];13(1):816. https://doi.org/10.1186/1471-2458-13-816

MoES. Fact Booklet [Internet]. Ministry of Education And Sports. 2012 [cited 2023 Mar 27]. https://www.education.go.ug/fact-booklet/

World Health Organization. Adolescent pregnancy [Internet]. [cited 2023 Dec 18]. https://www.who.int/news-room/fact-sheets/detail/adolescent-pregnancy

World Bank. World Bank Gender Data Portal. 2022 [cited 2023 Dec 18]. The Social and Educational Consequences of Adolescent Childbearing. https://genderdata.worldbank.org/data-stories/adolescent-fertility/

The National. Adolescent Health-Strategy.

MoH. Uganda [Internet]. 2022 [cited 2023 May 8]. https://fp2030.org/uganda

Bhurtyal A. Policy brief Policy brief.

CDFU. Monitor. 2021 [cited 2023 Mar 28]. 23,000 girls impregnated in Lango in one year - NGO. https://www.monitor.co.ug/uganda/news/national/23-000-girls-impregnated-in-lango-in-one-year-ngo-3573252

Napyo A, Nankabirwa V, Mukunya D, Tumuhamye J, Ndeezi G, Arach AAO, et al. Prevalence and predictors for unintended pregnancy among HIV-infected pregnant women in Lira, Northern Uganda: a cross-sectional study. Sci Rep. 2020;10(1):1–8.

Article   Google Scholar  

Abdel-Tawab NG, Attia S, Bader N, Roushdy R, El-Nakib S, Oraby D. Fertility preferences and behaviors among younger cohorts in Egypt: Recent trends, correlates, and prospects for change. 2020.

Draucker CB, Rawl SM, Vode E, Carter-Harris L. Integration through connecting in explanatory sequential mixed method studies. West J Nurs Res. 2020;42(12):1137–47.

Article   PubMed   PubMed Central   Google Scholar  

Regnault A, Willgoss T, Barbic S, On behalf of the International Society for Quality of Life Research (ISOQOL) Mixed Methods Special Interest Group (SIG). Towards the use of mixed methods inquiry as best practice in health outcomes research. J Patient-Rep Outcomes [Internet]. 2018 Apr 11 [cited 2023 Mar 28];2(1):19. https://doi.org/10.1186/s41687-018-0043-8

Simegn A, Azale T, Addis A, Dile M, Ayalew Y, Minuye B. Youth friendly sexual and reproductive health service utilization among high and preparatory school students in Debre Tabor town, Northwest Ethiopia: a cross sectional study. PLoS ONE. 2020;15(9):e0240033.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Rade BK, Tamiru AT, Aynalem GL, Taye EB, Melkie M, Abera A, et al. Prevalence and factors associated with sexual and reproductive health services use among reproductive age women with disabilities: a community based cross-sectional study. BMC Womens Health. 2023;23(1):215.

Smith JA, Fieldsend M. Interpretative phenomenological analysis. American Psychological Association; 2021.

Eatough V, Smith JA. Interpretative phenomenological analysis. Sage Handb Qual Res Psychol. 2017;193–209.

Campbell S, Greenwood M, Prior S, Shearer T, Walkem K, Young S, et al. Purposive sampling: complex or simple? Research case examples. J Res Nurs JRN. 2020;25(8):652–61.

Institute of Medicine (US) Committee on Monitoring Access to Personal Health Care Services. Access to Health Care in America [Internet]. Millman M, editor. Washington (DC): National Academies Press (US). 1993 [cited 2023 Dec 18]. http://www.ncbi.nlm.nih.gov/books/NBK235882/

Choi Y, Fabic MS, Adetunji J. Measuring Access to Family Planning: conceptual frameworks and DHS Data. Stud Fam Plann. 2016;47(2):145–61.

Article   CAS   PubMed   Google Scholar  

Penchansky R, Thomas JW. The concept of access: definition and relationship to consumer satisfaction. Med Care. 1981;19(2):127–40.

Levesque JF, Harris MF, Russell G. Patient-centred access to health care: conceptualising access at the interface of health systems and populations. Int J Equity Health. 2013;12(1):18.

Bekele D, Surur F, Nigatu B, Teklu A, Getinet T, Kassa M, et al. Knowledge and attitude towards Family Planning among women of Reproductive Age in emerging regions of Ethiopia. J Multidiscip Healthc. 2020;13:1463–74.

Taber KS. The Use of Cronbach’s alpha when developing and Reporting Research Instruments in Science Education. Res Sci Educ. 2018;48(6):1273–96.

Collett D. Modelling Binary Data. 2nd ed. New York: Chapman and Hall/CRC; 2014. p. 408.

Google Scholar  

Brown HM, Stahmer AC, Dwyer P, Rivera S. Changing the story: How diagnosticians can support a neurodiversity perspective from the start. Vol. 25, Autism. SAGE Publications Sage UK: London, England; 2021. pp. 1171–4.

Flick U. Triangulation in data collection. SAGE Handb Qual Data Collect. 2018;527–44.

Kriel Y, Milford C, Cordero JP, Suleman F, Steyn PS, Smit JA. Access to public sector family planning services and modern contraceptive methods in South Africa: a qualitative evaluation from community and health care provider perspectives. PLoS ONE. 2023;18(3):e0282996.

Murungi T, Benyumiza D, Apio J, Nekesa C, Nalubuuka A, Misuk I, et al. Factors Associated with Utilization of Sexual and Reproductive Health Services among the Youth in Lira City West, Northern Uganda: a cross-sectional study. BioMed Res Int. 2023;2023:e9649792.

Brittain AW, Williams JR, Zapata LB, Pazol K, Romero LM, Weik TS. Youth-Friendly Family Planning services for Young people. Am J Prev Med. 2015;49(2 Suppl 1):S73–84.

Abera L, Ejigu E, Hailu M, Tadesse D, Omer A. Quality of family planning services and associated factors among reproductive age women attending family planning unit at public health facilities in dire Dawa, Eastern Ethiopia, 2021. Contracept Reprod Med. 2023;8(1):33.

Osamor PE, Grady C. Women’s autonomy in health care decision-making in developing countries: a synthesis of the literature. Int J Womens Health. 2016;8:191–202.

Namasivayam A, Schluter PJ, Namutamba S, Lovell S. Understanding the contextual and cultural influences on women’s modern contraceptive use in East Uganda: a qualitative study. PLOS Glob Public Health. 2022;2(8):e0000545.

Asingwire N, Muhangi D, Kyomuhendo S, Leight J. Impact evaluation of youth-friendly family planning services in Uganda. 2019.

Schenker JG, Rabenou V. Family planning: cultural and religious perspectives. Hum Reprod. 1993;8(6):969–76.

Download references

Acknowledgements

Pre-Publication Support Service (PREPSS) supported the development of this manuscript by providing author training, as well as pre-publication peer-review and copy editing.

The authors want to acknowledge all the young people who took part in the study. In a special way, we also want to thank Dr. Marc Sam Opollo of Faculty of Public Health Lira University for reviewing and guiding our results presentation for the annual sexual and reproductive symposium presentation in 2023.

This research work was supported by a seed grant from the Center for International Reproductive Health Training at the University of Michigan (CIRHT-UM). The content is solely the responsibility of the authors and does not necessarily represent the official views of CIRHT-UM. The funders had no role in study design, data collection and analysis, the decision to publish, or the preparation of the manuscript.

Author information

Authors and affiliations.

Faculty of Public Health, Lira University, Lira City, Northern, P. O Box 1035, Uganda

Eustes Kigongo, Raymond Tumwesigye, Maxson Kenneth Anyolitho, Marvin Musinguzi, Everlyne Achan & Bernard Omech

Faculty of Nursing , Mbarara University of Science and Technology, Mbarara, Uganda

Caroline Kambugu Nabasirye & Samson Udho

Faculty of Education, Lira University, Lira, Uganda

Gad Kwizera

Faculty of Medicine, Lira University, Lira, Uganda

Amir Kabunga

You can also search for this author in PubMed   Google Scholar

Contributions

All authors made significant contributions to the conceptualization, design, data collection, curation, manuscript writing, and editing. EK, MKA and RT conceptualized and designed the study. EA, MM, GK, CKN, and SU designed the data collection tools and conducted the study. AK and BO gave overall guidance for the study. All the authors gave final approval to the manuscript for journal submission and are responsible for the content of the manuscript.

Corresponding author

Correspondence to Eustes Kigongo .

Ethics declarations

Ethical approval and consent to participate.

Informed consent was obtained from all the participants and their legal guardians. All ethical approvals were obtained from the Gulu University Research Ethics Committee.

Consent for publication

Not applicable.

Competing interests

Authors declare no conflict of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Kigongo, E., Tumwesigye, R., Anyolitho, M.K. et al. Access to family planning services and associated factors among young people in Lira city northern Uganda. BMC Public Health 24 , 1146 (2024). https://doi.org/10.1186/s12889-024-18605-8

Download citation

Received : 11 September 2023

Accepted : 15 April 2024

Published : 24 April 2024

DOI : https://doi.org/10.1186/s12889-024-18605-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Contraception
• Family planning
• Young people

BMC Public Health

ISSN: 1471-2458

• Submission enquiries: [email protected]
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• HHS Author Manuscripts

Mixed Methods in Nursing Research : An Overview and Practical Examples

Ardith z. doorenbos.

School of Nursing, University of Washington, USA, Box 357266, Seattle, WA 98177

Mixed methods research methodologies are increasingly applied in nursing research to strengthen the depth and breadth of understanding of nursing phenomena. This article describes the background and benefits of using mixed methods research methodologies, and provides two examples of nursing research that used mixed methods. Mixed methods research produces several benefits. The examples provided demonstrate specific benefits in the creation of a culturally congruent picture of chronic pain management for American Indians, and the determination of a way to assess cost for providing chronic pain care.

Introduction

Mixed methods is one of the three major research paradigms: quantitative research, qualitative research, and mixed methods research. Mixed methods research combines elements of qualitative and quantitative research approaches for the broad purpose of increasing the breadth and depth of understanding. The definition of mixed methods, from the first issue of the Journal of Mixed Methods Research, is “research in which the investigator collects and analyzes data, integrates the findings, and draws inferences using both qualitative and quantitative approaches or methods in a single study or program of inquiry” ( Tashakkori & Creswell, 2007 , p.4).

Mixed methods research began among anthropologists and sociologists in the early 1960s. In the late 1970s, the term “triangulation” began to enter methodology conversations. Triangulation was identified as a combination of methodologies in the study of the same phenomenon to decrease the bias inherent in using one particular method ( Morse, 1991 ). Two types of sequencing for mixed methods design have been proposed: simultaneous and sequential. Type of sequencing is one of the key decisions in mixed methods study design. Simultaneous sequencing is postulated to be simultaneous use of qualitative and quantitative methods, where there is limited interaction between the two sources of data during data collection, but the data obtained is used in the data interpretation stage to support each method's findings and to reach a final understanding. Sequential sequencing is postulated to be the use of one method before the other, as when the results of one method are necessary for planning the next method.

Since the 1960s, the use of mixed methods has continued to grow in popularity ( O'Cathain, 2009 ). Currently, although there are numerous designs to consider for mixed methods research, the four major types of mixed methods designs are triangulation design, embedded design, explanatory design, and exploratory design ( Creswell & Plano Clark, 2007 ). The most common and well-known approach to mixed methods research continues to be triangulation design.

There are many benefits to using mixed methods. Quantitative data can support qualitative research components by identifying representative patients or outlying cases, while qualitative data can shed light on quantitative components by helping with development of the conceptual model or instrument. During data collection, quantitative data can provide baseline information to help researchers select patients to interview, while qualitative data can help researchers understand the barriers and facilitators to patient recruitment and retention. During data analysis, qualitative data can assist with interpreting, clarifying, describing, and validating quantitative results.

Four broad types of research situations have been reported as benefiting particularly from mixed methods research. The first situation is when concepts are new and not well understood. Thus, there is a need for qualitative exploration before quantitative methods can be used. The second situation is when findings from one approach can be better understood with a second source of data. The third situation is when neither a qualitative nor a quantitative approach, by itself, is adequate to understanding the concept being studied. Lastly, the fourth situation is when the quantitative results are difficult to interpret, and qualitative data can assist with understanding the results ( Creswell & Plano Clark, 2007 ).

The purpose of this article is to illustrate mixed methods methodology by using examples of research into the chronic pain management experience among American Indians. These examples demonstrate the methodology used to provide (a) a detailed multilevel understanding of the chronic pain care experience for American Indians using triangulation design (multilevel model), and (b) a comparison of cost for two different chronic pain care delivery models, also using triangulation design (data transformation model).

An Example : Understanding the Pain Management Experience Among American Indians

Chronic pain poses unique challenges to the American health care system, including ever-escalating costs, unintentional poisonings and deaths from overdoses of painkillers, and incalculable suffering for patients as well as their families. Approximately 100 million adults in the United States are affected by chronic pain, with treatment costs and losses in productivity totaling $635 billion annually ( Institute of Medicine, 2011 ). Symptoms of pain are the leading reason patients visit health care providers ( Hing, Cherry, & Woodwell, 2006 ).

At the level of the community-based primary care provider, especially in tribal areas of the United States, there is often not enough capacity to manage complex chronic pain cases, and this is often due to lack of access to specialty pain care ( Momper, Delva, Tauiliili, Mueller-Williams, & Goral, 2013 ). The American Indian population in particular is underserved by health care and the most vulnerable to the impact of chronic pain, with high rates of drug poisoning due to opioid analgesics ( Warner, Chen, Makuc, Anderson, & Minino, 2011 ). There are 2.9 million people who report exclusive and an additional 1.6 million who report partial American Indian ancestry in the United States. They are a diverse group, residing in 35 states and organized into 564 federally recognized tribes ( U.S. Census Bureau, 2010 ). However, there is a scarcity of published literature exploring the experience, epidemiology, and management of pain among American Indians ( Haozous, Knobf, & Brant, 2010 ; Haozous & Knobf, 2013 ; Jimenez, Garroutte, Kundu, Morales, & Buchwald, 2011 ).

Using Mixed Methods to Overcome Barriers to Research

Barriers to effective research into chronic pain management among American Indians include the relatively small number of American Indian patients in any circumscribed area or tribe, the limitations of individual databases, and widespread racial misclassification. A mixed methods research approach is needed to understand the complex experience, epidemiology, and management of chronic pain among American Indians and to address the strengths and weaknesses of quantitative methodologies (large sample size, trends, generalizable) with those of qualitative methodologies (small sample size, details, in-depth).

This first example is from an ongoing study that uses triangulation design to provide a better understanding of the phenomenon of chronic pain management among American Indians. The study uses a multilevel model in which quantitative data collected at the national and state levels will be analyzed in parallel with the collection and analysis of the qualitative data at the patient level (see Figure 1 ). This allows the weakness of one approach to be offset by the strengths of the other. The results of the separate level analyses will be compared, contrasted, and blended leading to an overall interpretation of results.

Role of quantitative data

Previous examination of U.S. national databases has reported a higher prevalence of lower back pain in American Indians than in the general population (35% compared to 26% ; Deyo, Mirza, & Martin, 2002 ). Thus, at level 1, quantitative administrative data sets representing health care received by American Indians, both across the United States and in broad regions, will be used to evaluate macro-level trends in utilization of health care and in basic outcomes, such as opioid-related deaths.

At level 2, more detailed quantitative Washington state tribal clinic data will be used to identify American Indian populations, evaluate breakdowns in the delivery of care, and identify processes that lead to unsuccessful outcomes. For example, in a study conducted with community health practitioners in Alaska, participants reported low levels of knowledge and comfort around discussing cancer pain ( Cueva, Lanier, Dignan, Kuhnley, & Jenkins, 2005 ).

Role of qualitative data

At level 3, qualitative research through focus groups and key informant interviews will provide even more refined information about perceptions of recommended and received care. These interviews will provide insight into selected immediate and proximal factors. These factors include patients' choice and use of services; attitudes, motivations, and perceptions that influence their decisions; interpersonal factors, such as social support; and perceived discrimination. This qualitative data will shed light on potential barriers to care that are not easily recognized in administrative or clinical records, and thereby will provide greater detail about patient views of chronic pain care.

Role of (qualitative) indigenous methodologies

Since the focus of this study is on the chronic pain experience among American Indian patients, it is important that the qualitative work in level 3 be guided by indigenous methodologies, in both data collection and analysis. The phrase “indigenous methodologies” refers to an evolving framework for creating research that places the epistemologies of indigenous participants and communities at the center of the work, while building an equitable and respectful setting for bidirectional learning ( Evans, Hole, Berg, Hutchinson, & Sookraj, 2009 ; Louis, 2007 .; Smith, 2004 ). Although the tenets of indigenous methodologies vary according to the source, there is agreement among sources that research with indigenous populations should be wellness-oriented, holistic, community-oriented, and focused on indigenous knowledge, and should incorporate bidirectional learning ( Louis, 2007 ; Smith, 2004 ).

The ongoing project aligns with these guidelines by building knowledge about the chronic pain experience from the perspective of American Indian patients. The data is being interpreted with the goal of designing a usable and relevant model that will resonate at the American Indian community level. The researchers have conducted focus groups with the needs and priorities of the participants placed at the forefront, to best achieve the goals of learning and building knowledge that reflects the participants' experiences. Specifically, the focus groups were scheduled within three tribes, ensuring high familiarity and social support among group members. These focus groups met either at a tribal community center or in a nearby tribally owned casino in the evening. Each focus group started with a dinner, followed by discussion.

The focus group facilitator was well-known to the community, and although not American Indian, had been an active participant in community events and had provided expert knowledge and consultation to the tribes. Additionally, each focus group was co-facilitated by a tribal elder. The high familiarity among the participants and the research team was an important component of the bidirectional learning: it helped reduce much of the mistrust that has historically prevented medical researchers from obtaining high-quality data in similarly vulnerable populations ( Guadagnolo, Cina, & Helbig, 2009 ).

Benefits of Triangulation Design: Multilevel Model

In summary, only a mixed methods study that included quantitative and qualitative methods could provide the data required for a comprehensive multilevel assessment of the chronic pain experience among American Indians. Although this study is ongoing, the plan is for a nationwide analysis of variations in chronic pain outcomes among American Indians to examine the structure of service delivery and organization. Analysis of the state tribal clinic data will address intermediate factors and will examine community-level variation in pain management and local access to pain specialists. Preliminary analysis of the focus group data has already demonstrated that there is insufficient pain management among American Indians, due in part to lack of knowledge about pain management among providers and lack of access to pain specialists.

An Example; Comparing the Costs of Two Models for Providing Chronic Pain Care to American Indians

Telehealth is one innovative approach to providing access to high-quality interdisciplinary pain care for American Indians. A telehealth model with a unique approach based on provider-to-provider videoconference consultations allows community-based providers to present complex chronic pain cases to a panel of pain specialists through a videoconferencing infrastructure that also incorporates longitudinal outcomes tracking to monitor patient progress. Telehealth is an innovative model of health care delivery, and its use among American Indians has been expanding over the past several years ( Doorenbos et al., 2010 ; Doorenbos et al., 2011a ; 2011b ). Although the use of telehealth for providing chronic pain consultation is still in early stages, the long-term effectiveness of this approach and its impact on increasing capacity for pain management among community providers is being investigated ( Haozous et al., 2012 ; Tauben, Towle, Gordon, Theodore, & Doorenbos, 2013 ). The mixed methods approach for this transaction cost analysis used a unique triangulation design with a data transformation model to build a body of evidence for telehealth pain management.

With ever increasing mandates to reduce the cost and increase the quality of pain management, health care institutions are faced with the challenge of demonstrating that new technologies provide value while maintaining or even improving the quality of care ( Harries & Yellowlees, 2013 ). Transaction cost analysis can provide this evidence by using mixed methods research methodologies to provide comparative evaluation of the costs and consequences of using alternative technologies and the accompanying organizational arrangements for delivering care ( Williamson, 2000 ).

The theory of transaction cost developed from the observation that our structures for governing transactions—the ways in which we organize, manage, support, and carry out exchange — have economic consequences ( Williamson, 1991 ). Though prices matter, this theory recognizes that prices can and do deviate from the cost of production and do not include the cost of transacting ( Coase, 1960 ). Setting aside neoclassical economic conceptions of price, output, demand, and supply, the transaction becomes the unit of analysis ( Williamson, 1985 ).

In transactions, there are typically two parties engaging in the exchange of goods or services, and both exert effort to carry out the transaction, incurring costs in the hope or with the expectation of realizing benefits. Some ways of structuring or supporting a given transaction, such as consultation or treatment for a patient from a health care provider, may be more efficient than others. The analysis examines the actual costs incurred and the related consequences experienced by the parties over time, with the hypothesis that efficiency results from the discriminating alignment of transactions with alternative, more efficient structures of governance ( Williamson, 2002 ).

Specialty health care services participating in the study described here included the University of Washington (UW) Center for Pain Relief and the UW TelePain program. The UW Center for Pain Relief is an outpatient multispecialty consultation and treatment clinic that uses the assembled expertise and skills of physicians and other medical team providers to assist in diagnosis and care for chronic pain, for example for people with painful disorders that have persisted beyond expected duration, or for people who have persistent uncontrolled pain despite appropriate treatment for the underlying medical condition. The clinic also offers pain consultation and treatment for a variety of new-onset or acute problems that may benefit from selective anesthetic procedures, such as nerve blocks or spinal nerve root compression.

The UW TelePain program serves tribal providers in the Washington, Wyoming, Alaska, Montana, and Idaho (WWAMI) region. These tribal providers include primary care physicians, physician assistants, and nurse practitioners. The tribal providers have access to weekly videoconferences both with other community providers and with university-based pain and symptom management experts. During videoconferences, providers manage cases, engage in evidence-based practice activities, and receive peer support. Throughout the process, these community providers are responsible for direct patient care, and they act on recommendations of the consulting pain specialists.

The two care delivery models discussed above — traditional in-clinic consultation at the Center for Pain Relief and telehealth case consultation through TelePain — provided this mixed methods study using triangulation design and a data transformation model with two comparative arrangements for delivering the same transaction: delivery of pain care to patients (see Figure 2 ).

Qualitative and Quantitative Data Collection Procedures

Participant observation and structured interviews were used to identify and describe two comparable completed transactions for patients with chronic pain. Members of the clinical care teams selected one transaction from each service for which the care could be said to represent the routines and norms of their health care organization. The chosen transactions were carried out with patients of the same gender, similar age, and similar health characteristics. For the study, clinical care teams from each service provided two qualitative on-site interviews documenting clinical work flow and processes (i.e., the steps in the transaction). For the in-clinic transaction, members of the clinical care team interviewed included a nurse care coordinator, pain specialist, medical assistant, patient outcomes assessment coordinator, nurse triage manager, patient support services supervisor, and financial authorization specialist. For the Tele-Pain transaction, team members interviewed included the TelePain nurse care coordinator, two pain specialists, an information technology specialist, and the clinic provider.

The following details the process of the mixed methods analysis. First, individual steps, or discrete tasks, within each transaction (in-clinic versus TelePain) were identified using qualitative interviews and itemized in detail. Details from the qualitative data included a description of each task, the person (s) engaged, the duration of engagement of each person in minutes, the information accrued to the patient's medical record, the technologies employed, and the locations where tasks were conducted and information was transmitted or stored.

The quantitative data collected included date and time, and therefore duration in business days, that accumulated with each step in the transaction. Finally, the costs of each step collected from the qualitative data were identified and transformed into quantitatively estimated data for each transaction. Analysis focused on the primary costs in health care: the value of people's time. These values were limited to labor costs for the in-clinic and telehealth personnel; proxies for the value of time were used with estimates of time for the patient. Costs were estimated as a function of time spent per task and per patient, and the actual wage, including benefits, of personnel engaged in the transaction.

Qualitative and Quantitative Data Analysis

Personal identifiable information was redacted from each patient's medical record, and the records were reviewed for comparability as well as for norms and routines of care for the in-clinic and telehealth organizations. The characteristics of the two patients were similar. Both were first-time patients to their respective organizations, and were referred by their primary care providers for specialized care. The reasons for seeking care and report of conditions potentially related to chronic pain were similar. Both transactions resulted in a consultation recommending referral for additional specialized care or treatment.

Two work flows, one in-clinic and one telehealth, were developed by documenting actual tasks undertaken during the transactions. In follow-up interviews, these work flows were presented to participants for review and comment. These interviews resulted in a complete itemized list of dates, personnel, and time spent per person on discrete steps or tasks. Tables and graphs expressing the steps, with cost accrual over time and in sum, were developed and compared for each transaction, to each other, and with respect to participants' rationales for the tasks in each transaction.

The equation expressing the cost per transaction is as follows, where the total cost of the transaction ( C T ) is the sum of the costs of each discrete task ( k i ) in the transaction, measured per participant ( x, y, z …) on the task, as the product of time ( t ) and wage rate ( w ), or in the case of the patient ( x, y, z …), a proxy for the value of time ( w ) and estimated time ( t ).

In total, 46 discrete steps were taken for the typical in-clinic transaction at the UW Center for Pain Relief (one patient case, reviewed by two pain specialists) versus 27 steps for the typical TelePain transaction (three patient cases, reviewed by six pain specialists). The greater number and types of administrative steps taken to schedule, execute, and follow up the in-clinic consultation resulted in greater duration of time between receipt of initial referral request and completion of the initial consultation with the pain specialists. A total of 153 business days (213 calendar days) elapsed between referral and the completion of the entire in-clinic transaction, versus 4 business days (4 calendar, days) for the TelePain transaction. Importantly, for the transaction at the UW Center for Pain Relief, 72 business days transpired before consultation concluded with a referral for the patient's record; the same conclusion was reached in 4 days in the TelePain transaction. These methods used to determine transaction costs provide an excellent example of mixed methods research, where both qualitative and quantitative data and analysis are needed to provide the transaction cost results.

Mixed methods are increasingly being used in nursing research. We have detailed two studies in which mixed methods research with triangulation design brought a richness to the examination of the phenomenon that a single methodology would not In the two examples described, a major advantage of the triangulation design is its efficiency, because both types of data are collected simultaneously. Each type of data can be collected and analyzed separately and independently, using the techniques traditionally associated with each data type. Both simultaneous and sequential data collection lend themselves to team research, in which the team includes researchers with both quantitative and qualitative expertise.

Challenges include the effort and expertise required due to the simultaneous data collection, and the fact that equal weight is usually given to each data type. Thus this research requires a team, or extensive training in both quantitative and qualitative methodologies, and careful adherence to the methodological rigor required for both methodologies. Nursing researchers may face the possibility of inconsistency in research findings arising from the objectivity of quantitative methods and the subjectivity of qualitative methods. In these cases, additional data collection may be required.

The first example, regarding the pain management experience among American Indians, used triangulation design in a multilevel model format. The multilevel model was useful in designing this study as different methods were needed at different levels to fully understand the complex health care system. In this example, quantitative data is being collected and analyzed at the national and state levels, and qualitative data is being collected at the patient level. Both qualitative and quantitative data are being collected simultaneously. The findings from each level will then be blended into one overall interpretation.

The second example, a transaction cost analysis, also used triangulation design, but the model used was that of data transformation. As in the multilevel model used in the first example, the data transformation model involved the separate but concurrent collection of qualitative and quantitative data. A novel step in this model involves transforming the qualitative data into quantitative data, and then comparing and interrelating the data sets. This required the development of procedures for transforming the qualitative data, related to, time spent on a step and salary of the provider, into quantitative cost data.

The two studies presented as examples demonstrate mixed methods research resulting in the creation of (a) a rich description of the American Indian chronic pain experience, and (b) a way to assess cost for providing chronic pain care via tribal clinics. In both examples, the quantitative data and their subsequent analysis provide a general understanding of the research problem. The qualitative data and their analysis refine and explain the results by exploring participants' views in more depth. Research using a single methodology would not have been able to achieve the same results.

Acknowledgments

Research reported in this paper was supported by the National Institute of Nursing Research of the National Institutes of Health under award number #R01NR012450 and the National Cancer Institute of the National Institutes of Health under award number #R42 CA141875. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

• Coase R. The problem of social cost. Journal of Law and Economics. 1960; 3 :1–44. [ Google Scholar ]
• Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. Thousand Oaks, CA: Sage; 2007. [ Google Scholar ]
• Cueva M, Lanier A, Dignan M, Kuhnley R, Jenkins C. Cancer education for community health aides/practitioners (CHA/Ps) in Alaska assessing comfort with cancer. Journal of Cancer Education. 2005; 20 :85–88. [ PubMed ] [ Google Scholar ]
• Deyo RA, Mirza SK, Martin BI. Back pain prevalence and visit rates : Estimates from U.S. national surveys. Spine. 2002; 31 :2724–2727. [ PubMed ] [ Google Scholar ]
• Doorenbos AZ, Demiris G, Towle C, Kundu A, Revels L, Colven R, Norris TE, Buchwald D. Developing the Native People for Cancer Control Telehealth Network. Telemedicine and E-Health. 2011; 17 :30–34. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Doorenbos AZ, Eaton LH, Haozous E, Towle C, Revels L, Buchwald D. Satisfaction with telehealth for cancer support groups in rural American Indian/Alaska Native communities. Clinical Journal of Oncology Nursing. 2010; 14 :765–770. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Doorenbos AZ, Kundu A, Eaton LH, Demiris G, Haozous EA, Towle C, Buchwald D. Enhancing access to cancer education for rural health care providers via Telehealth. Journal of Cancer Education. 2011; 26 :682–686. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Evans M, Hole R, Berg L, Hutchinson P, Sookraj D. Common insights, differing methodologies : Toward a fusion of indigenous methodologies, participatory action research, and White studies in an urban Aboriginal research agenda. Qualitative Inquiry. 2009; 15 (5):893–910. [ Google Scholar ]
• Guadagnolo BA, Cina K, Helbig P. Medical mistrust and less satisfaction with health care among Native Americans presenting for cancer treatment. Journal of Health Care for the Poor and Underserved. 2009; 20 (1):210–226. doi: 10.1353/hpu.0.0108. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Haozous E, Doorenbos AZ, Demiris G, Eaton LH, Towle C, Kundu A, Buchwald D. Role of Telehealth/videoconferencing in managing cancer pain in rural American Indian communities. Psycho-Oncology. 2012; 21 :219–223. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Haozous EA, Knobf MT. ‘All My Tears Were Gone’: Suffering and cancer pain in Southwest American Indians. Journal of Pain and Symptom Management. 2013; 45 :1050–1060. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Haozous EA, Knobf MT, Brant J. Understanding the cancer pain experience in American Indians of the Northern Plains. Psycho Oncology. 2010; 20 (4):404–410. doi: 10.1002/pon.l741. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Harries D, Yellowlees PM. Cyberterrorism: Is the U.S. healthcare system safe? Telemedicine Journal and e-Health. 2013; 19 (1):61–66. [ PubMed ] [ Google Scholar ]
• Hing E, Cherry DK, Woodwell DA. National Ambulatory Medical Care Survey : 2004 Summary. Hyattsville, MD: National Center of Health Statistics; 2006. [ PubMed ] [ Google Scholar ]
• Institute of Medicine, Committee on Advancing Pain Research Care, and Education. Relieving pain in America : A blueprint for transforming prevention, care, education, and research. Washington, DC: Institute of Medicine; 2011. [ PubMed ] [ Google Scholar ]
• Jimenez N, Garroutte E, Kundu A, Morales L, Buchwald D. A review of the experience, epidemiology, and management of pain among American Indian, Alaska Native, and Aboriginal Canadian peoples. Journal of Pain. 2011; 12 :511–522. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Louis R. Can you hear us now? Voices from the margin: Using indigenous methodologies in geographic research. Geographic Research. 2007; 45 (2):130–139. [ Google Scholar ]
• Momper SL, Delva J, Tauiliili D, Mueller-Williams AC, Goral P. OxyContin use on a rural Midwest American Indian reservation : Demographic correlates and reasons for using. American Journal of Public Health. 2013; 103 :1997–1999. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Morse JM. Approaches to qualitative-quantitative methodological triangulation. Nursing Research. 1991; 40 :120–123. [ PubMed ] [ Google Scholar ]
• O'Cathain A. Mixed methods research in the health sciences : A quiet revolution. Journal of Mixed Methods Research. 2009; 3 :3–6. [ Google Scholar ]
• Smith L. Decolonizing methodologies: Research and indigenous peoples. London: Zed Books; 2004. [ Google Scholar ]
• Tashakkori A, Creswell J. The new era of mixed methods. Journal of Mixed Methods Research. 2007; 1 :3–7. [ Google Scholar ]
• Tauben D, Towle C, Gordon DB, Theodore BR, Doorenbos AZ. TelePain : A platform for concurrent education, clinical care, and research. Pain Medicine. 2013; 14 (4):560. [ Google Scholar ]
• U.S. Census Bureau. Statistical Abstract of the United States. Washington, DC, U.S: Bureau of the Census; 2010. [ Google Scholar ]
• Warner M, Chen LH, Makuc DM, Anderson RN, Minino AM. Drug poisoning deaths in the United States, 1980-2008. NCHS Data Brief. 2011; 81 :1–8. [ PubMed ] [ Google Scholar ]
• Williamson OE. The Economic Institutions of Capitalism. New York: The Free Press; 1985. [ Google Scholar ]
• Williamson OE. Comparative economic organization : The analysis of discrete structural alternatives. Administrative Science Quarterly. 1991; 36 (2):269–296. [ Google Scholar ]
• Williamson OE. The new institutional economics : Taking stock, looking ahead. Journal of Economic Literature. 2000; 38 (3):595–613. [ Google Scholar ]
• Williamson OE. The theory of the firm as governance structure : From choice to contract. Journal of Economic Perspectives. 2002; 16 (3):171–195. [ Google Scholar ]

This paper is in the following e-collection/theme issue:

Published on 23.4.2024 in Vol 26 (2024)

This is a member publication of University of Oxford (Jisc)

Empowering School Staff to Support Pupil Mental Health Through a Brief, Interactive Web-Based Training Program: Mixed Methods Study

Authors of this article:

Original Paper

• Emma Soneson 1, 2 , PhD   ; 
• Emma Howarth 3 , PhD   ; 
• Alison Weir 4, 5 , MA, MSc   ; 
• Peter B Jones 2 * , PhD   ; 
• Mina Fazel 1 * , DM  

1 Department of Psychiatry, University of Oxford, Oxford, United Kingdom

2 Department of Psychiatry, University of Cambridge, Cambridge, United Kingdom

3 School of Psychology, University of East London, London, United Kingdom

4 Faculty of Education, University of Cambridge, Cambridge, United Kingdom

5 Howard Community Academy, Anglian Learning multi-academy trust, Bury St Edmunds, United Kingdom

*these authors contributed equally

Corresponding Author:

Emma Soneson, PhD

Department of Psychiatry

University of Oxford

Warneford Lane

Oxford, OX3 7JX

United Kingdom

Phone: 44 1865 613127

Email: [email protected]

Background: Schools in the United Kingdom and elsewhere are expected to protect and promote pupil mental health. However, many school staff members do not feel confident in identifying and responding to pupil mental health difficulties and report wanting additional training in this area.

Objective: We aimed to explore the feasibility of Kognito’s At-Risk for Elementary School Educators , a brief, interactive web-based training program that uses a simulation-based approach to improve school staff’s knowledge and skills in supporting pupil mental health.

Methods: We conducted a mixed methods, nonrandomized feasibility study of At-Risk for Elementary School Educators in 6 UK primary schools. Our outcomes were (1) school staff’s self-efficacy and preparedness to identify and respond to pupil mental health difficulties, (2) school staff’s identification of mental health difficulties and increased risk of mental health difficulties, (3) mental health support for identified pupils (including conversations about concerns, documentation of concerns, in-class and in-school support, and referral and access to specialist mental health services), and (4) the acceptability and practicality of the training. We assessed these outcomes using a series of questionnaires completed at baseline (T1), 1 week after the training (T2), and 3 months after the training (T3), as well as semistructured qualitative interviews. Following guidance for feasibility studies, we assessed quantitative outcomes across time points by comparing medians and IQRs and analyzed qualitative data using reflexive thematic analysis.

Results: A total of 108 teachers and teaching assistants (TAs) completed T1 questionnaires, 89 (82.4%) completed T2 questionnaires, and 70 (64.8%) completed T3 questionnaires; 54 (50%) completed all 3. Eight school staff members, including teachers, TAs, mental health leads, and senior leaders, participated in the interviews. School staff reported greater confidence and preparedness in identifying and responding to mental health difficulties after completing the training. The proportion of pupils whom they identified as having mental health difficulties or increased risk declined slightly over time (median T1 =10%; median T2 =10%; median T3 =7.4%), but findings suggested a slight increase in accuracy compared with a validated screening measure (the Strengths and Difficulties Questionnaire). In-school mental health support outcomes for identified pupils improved after the training, with increases in formal documentation and communication of concerns as well as provision of in-class and in-school support. Referrals and access to external mental health services remained constant. The qualitative findings indicated that school staff perceived the training as useful, practical, and acceptable.

Conclusions: The findings suggest that brief, interactive web-based training programs such as At-Risk for Elementary School Educators are a feasible means to improve the identification of and response to mental health difficulties in UK primary schools. Such training may help address the high prevalence of mental health difficulties in this age group by helping facilitate access to care and support.

Introduction

In recent years, there has been an increased emphasis on the role of schools in supporting children’s mental health [ 1 - 3 ]. This enhanced focus has been driven in large part by an apparent increase in mental health difficulties (including behavioral, social, and emotional difficulties) present in school-aged populations [ 4 - 6 ]—a concern that became increasingly prominent in the context of the COVID-19 pandemic and the associated school closures and social distancing measures [ 7 , 8 ]. There is also a growing recognition of the many unique advantages of using the school setting to promote and protect pupil mental health [ 9 ]. First, most lifetime disorders begin during the schooling years [ 10 ], which suggests that schools may be an ideal setting for early identification and intervention. Second, schools have access to most children, meaning that they are an important component of any public health approach to address child mental health difficulties [ 11 - 14 ]. Third, schools benefit from prolonged engagement with pupils, which can facilitate the implementation of mental health promotion and prevention strategies as well as support and interventions for pupils with identified mental health needs [ 12 ]. Finally, mental health support in schools is often more accessible to families than other types of support [ 15 ].

However, while school staff are increasingly expected to support children’s mental health [ 1 ], many do not feel prepared to do so [ 16 - 19 ] due in part to receiving limited training and supervision in this area [ 20 ]. Therefore, improving school staff’s confidence and preparedness are important considerations for supporting them in taking an expanded role in pupil mental health [ 21 ]. Most schools offer some form of mental health training [ 22 , 23 ], but many staff members believe that they could benefit from additional training [ 18 - 20 , 24 - 26 ]. One area where staff training may be particularly beneficial is the identification of and first response to pupils who have mental health difficulties or who are believed to be at increased risk of developing them. However, although there is evidence suggesting that school staff, parents, and practitioners see such training as an acceptable, feasible, and potentially useful way to support pupil mental health [ 20 , 27 - 29 ], empirical evidence for the effectiveness of such training is limited and focuses primarily on intermediate outcomes (eg, staff knowledge and confidence) rather than downstream outcomes (eg, accurate identification, access to support, and mental health outcomes) [ 30 , 31 ]. Furthermore, there are several potential barriers to implementing training programs in schools, including time, cost, and resource requirements [ 28 ].

At-Risk for Elementary School Educators : A Brief, Interactive Web-Based Training

Training programs that address these barriers may be beneficial for supporting schools in identifying and responding to pupil mental health difficulties. Brief, interactive web-based training programs are a particularly promising avenue as they have the potential to be more affordable, flexible, and scalable than other training formats. One such training is At-Risk for Elementary School Educators (hereinafter, At-Risk ), a virtual simulation-based program developed by the American company Kognito [ 32 ]. The program, which has been completed by >125,000 teachers in the United States, aims to improve pupil mental health by “[building] awareness, knowledge, and skills about mental health, and [preparing] users to lead real-life conversations with pupils, parents, and caregivers about their concerns and available support” [ 33 ].

The program addresses many common implementation barriers to school-based mental health training. For example, At-Risk only requires approximately 1 hour to complete, which is much shorter than many other available training programs [ 31 , 34 ]. This comparatively low time commitment may address the concern that training programs are overly time intensive and, thus, make the training more feasible for busy schools [ 28 , 34 , 35 ]. The web-based format of At-Risk may also address concerns about school-based mental health programs being resource intensive [ 28 ]. Nearly all school mental health training programs documented in the literature are face-to-face sessions led by external facilitators [ 34 , 36 ], with only a few examples of web-based training [ 37 - 39 ]. For schools with limited budgets, programs requiring external facilitators can prove unsustainable and have limited scalability. In terms of financial resources, the costs of At-Risk vary depending on the number of licenses purchased, but the maximum cost is approximately £22 (US $30) per user, a price point that is feasible for many UK schools. In the United States, there have been many examples of bulk purchases at the district or state level that have made the training even more affordable per teacher. In many areas, the training is even free at point of use due to state- or district-wide licensing agreements [ 40 ].

To date, 3 randomized studies have examined the effectiveness and acceptability of At-Risk among samples of American teachers [ 17 , 41 ] and teachers in training [ 42 ] across school years. Each study found high satisfaction ratings, with between 75% and 85% of participants rating the training as useful, well constructed, relevant, and easy to use, and nearly all (88%-95%) reporting that they would recommend it to colleagues. The training also improved teachers’ self-rated preparedness, self-efficacy, and likelihood of identifying and discussing concerns about pupils’ mental health and referring them to appropriate support when needed. These improvements were reflected in the teachers’ behaviors—compared with teachers in the control group , those who completed At-Risk self-reported significantly more helping behaviors (eg, identifying psychological distress, discussing concerns with pupils and parents, and consulting with parents about options for care and support) and gatekeeping behaviors (ie, connecting pupils with care and support) after the training and at 3 months after the training. The findings of these studies indicate that At-Risk may help improve teachers’ ability to identify and respond to pupil mental health needs and lead to positive behavior change in terms of discussing concerns and facilitating access to care and support.

At-Risk in a UK Context: Considerations for Transportability

These 3 studies suggest that At-Risk may be a promising intervention for improving children’s mental health; however, there is still much to be learned about the training’s effectiveness, feasibility, and acceptability. Furthermore, to date, no evaluation of the training has been conducted outside the United States. There is increasing focus on the influence of context on the effectiveness of complex interventions [ 43 - 48 ], and while some interventions have shown success in terms of transportability [ 48 ], other interventions that have evidence of effectiveness in one context have demonstrated null or even negative effects in another [ 46 ]. Furthermore, information that could inform “transportability” is often not collected as part of evaluations [ 44 ], making it difficult to determine the likelihood of success in a new setting.

There are many contextual differences between the United States and the United Kingdom that could mean that school-based interventions developed in one country may not translate well to the other. Cross-country differences in education systems and (mental) health services are particularly relevant to this study. Differences in the education system include the length and content of initial teacher training, the number and roles of teaching assistants (TAs), and school funding structures. There are also key differences in the structure and availability of school-based mental health provision. In the United States, schools often have staff whose sole or at least main responsibility is mental health, such as school psychologists. While these roles are becoming more common in the United Kingdom with the implementation of the Green Paper recommendations [ 1 ], in most UK primary schools, mental health is included within the broader roles of the special educational needs coordinator (SENCo) and pastoral team. Finally, differences in the wider health care systems across the countries also mean that the process and outcomes of external referrals to specialist mental health services vary across settings, another fact that may influence the transportability of school-based interventions such as At-Risk.

Given these uncertainties regarding intervention transportability, additional evaluation of At-Risk is needed to understand whether it is a potentially useful and feasible tool to improve the identification of and response to mental health difficulties in UK primary schools. To explore the potential value of the training in this new context, we conducted a mixed methods feasibility study of At-Risk in 6 UK primary schools covering pupils aged 4 to 11 years . We aimed to examine the influence of At-Risk on staff confidence and preparedness, identification of pupils with mental health difficulties or increased risk of developing mental health difficulties, mental health support outcomes for identified children, and intervention acceptability and practicality.

Study Design

We used a mixed methods, nonrandomized, pretest-posttest study design to explore the feasibility of At-Risk in UK primary schools. While feasibility studies are acknowledged as a key stage of intervention design and evaluation [ 49 , 50 ], there is no universally agreed-upon definition of a feasibility study [ 50 , 51 ]. Therefore, we focused on 3 criteria from the guidance by Bowen et al [ 52 ]: acceptability, practicality, and limited effectiveness testing.

Intervention: At-Risk for Elementary School Educators

At-Risk is a web-based training that is delivered individually and requires only a log-in and internet connection. Using a simulation-based teaching model, the training aims to (1) improve mental health awareness and knowledge, (2) empower users to approach pupils about what they have noticed, (3) impart skills to have meaningful conversations with pupils and parents, and (4) train users to refer pupils to further support. The diagram in Figure 1 illustrates how the training might lead to improved mental health outcomes for pupils.

The simulation begins with an introduction by a virtual coach, who defines and explains how to recognize the warning signs of psychological distress and specific mental health difficulties and provides guidance and practical advice for discussing and acting upon concerns. Users then practice 2 virtual scenarios. The first scenario involves a fifth-grade (UK Year 6; ages of 10-11 years) teacher speaking with the parent of a pupil showing signs of behavioral difficulties. The second involves a third-grade (UK Year 4; ages of 8-9 years) teacher speaking with a pupil showing signs of emotional difficulties. During the conversations, users choose what to say via drop-down menus organized into categories (eg, “bring up concerns” or “ask a question”) and phrases (eg, “Mia sometimes seems a little agitated in class”). Throughout the conversation, users receive feedback through a “comfort bar” (based on how the pupil or parent perceives the conversation), opportunities to “see” the thoughts of the pupil or parent, and suggestions from the virtual coach.

Importantly, there is no one “right” way to conduct the conversations, and several approaches can lead to a positive outcome. Throughout the conversation, users can “undo” actions to backtrack after receiving an undesirable response or to explore what the response would have been had they chosen another option. At the end of each conversation, the pupil or parent provides feedback on the conversation. The training finishes with a short segment on connecting pupils with further support.

For this feasibility study, we used an unmodified version of the training (ie, the standard training designed for American schools, not tailored to the UK context) provided free of cost by Kognito. The potential need for adaptation and tailoring was an important consideration that we explicitly examined as part of our exploration of the acceptability and practicality of At-Risk in this new setting.

Recruitment

We originally sought to purposively sample 5 primary schools from Cambridgeshire or Norfolk that (1) had a higher-than-average proportion of pupils eligible for free school meals or (2) were located in an area in the top tertile of deprivation as measured using the Index of Multiple Deprivation [ 53 ], which we calculated with the publicly available Schools, pupils and their characteristics data [ 54 ]. We emailed headteachers, SENCos, and mental health leads from 131 candidate schools in September and October 2019 about participating in the study. To increase recruitment, we contacted additional schools in January 2020 for a study start date of March 2020. However, the study was suspended in March 2020 due to the in-person school closures associated with the onset of the COVID-19 pandemic. As some of the participating schools dropped out due to the pandemic, we reopened recruitment for a January 2021 study start date. In this round, we did not restrict participation by the 2 deprivation criteria described previously (ie, free school meal eligibility and Index of Multiple Deprivation), so any UK-based mainstream primary school was eligible to participate. The January 2021 start date was again delayed by the pandemic, but there was no subsequent recruitment.

Teachers and TAs

Schools were responsible for recruiting individual teachers and TAs to participate in the training. We encouraged schools to invite all teachers and TAs to participate, but schools made a variety of decisions in this regard. Three schools (schools D, E, and F) had all staff complete the training during inset (in-service training) days or other designated times, 2 schools (schools A and C) had staff volunteer to participate, and 1 school (school B) selected 2 to 3 staff members in each year group to participate.

Measures and Materials

School characteristics.

The characteristics of the participating schools, including school type, school sex (ie, whether they were single or mixed sex), urbanicity, head count, area-level deprivation, level of free school meal eligibility, ethnic composition, and proportion of pupils with special educational needs, were obtained from publicly available data from the Department for Education [ 54 , 55 ].

Teacher and TA Identification Form

The purpose of the Teacher and TA Identification Form ( Multimedia Appendix 1 ) was to understand which pupils participants would identify as having mental health difficulties or an increased risk of developing mental health difficulties. As systematic reviews in this area have identified no suitable questionnaires [ 28 , 30 ], we developed a bespoke questionnaire, which was reviewed by a school staff advisory group to ensure accuracy and relevance. The questionnaire begins with instructions, including explanations and examples of what is meant by “mental health difficulties or risk for mental health difficulties.” Full definitions are provided in Multimedia Appendix 1 , but in brief, “mental health difficulties” are described as “behavioural and social-emotional problems” regardless of formal diagnosis, and “risk for mental health difficulties” is described as experiences that increase the chance of a child developing mental health difficulties in the future.

For all pupils in their class, participants first indicated whether they thought a pupil had mental health difficulties or increased risk. If yes, they answered 9 subsequent questions about mental health support outcomes. The first four outcomes were about communication of concerns, namely whether they had (1) formally documented their concerns with the school, (2) communicated concerns to the SENCo, pastoral care lead, or mental health lead, (3) communicated concerns to another member of the school staff, or (4) communicated concerns to the child or their parents. The next five outcomes pertained to the provision of mental health support, namely whether the pupil (5) received in-class support; (6) received in-school support or had an in-house support plan; (7) had documented social, emotional, and mental health (SEMH) status (a type of special educational need focused on mental health difficulties); (8) had been referred to external mental health services; or (9) had access to external mental health services.

Strengths and Difficulties Questionnaire

The teacher-report Strengths and Difficulties Questionnaire (SDQ) [ 56 - 59 ] served as the comparator for findings about teachers’ and TAs’ identification of pupils. The SDQ includes 25 positive and negative psychological attributes across 5 scales: emotional symptoms, conduct problems, hyperactivity/inattention, peer relationship problems, and prosocial behavior. The first 4 scales add up to a Total Difficulties Score (0-40, with higher scores representing greater difficulties). The SDQ has demonstrated acceptable psychometric properties in primary school samples [ 60 ]. It is important to note that the SDQ is not an exact comparator as it measures a narrower concept than the Teacher and TA Identification Form (which also includes increased risk ). However, this comparison could potentially yield valuable information regarding feasibility .

Pre- and Posttraining Surveys

Kognito uses pre- and posttraining surveys to assess their training. These surveys (based on the validated Gatekeeper Behavior Scale [ 61 ]) explore teachers’ self-efficacy in identifying and responding to mental health difficulties and whether their attitudes, self-efficacy, or practice have changed since completing the At-Risk training. The posttraining survey also includes questions on perceptions of the training’s impact. We independently (ie, with no input from Kognito) reviewed the merits of these questionnaires and decided to use them in this study because (1) they covered relevant and useful concepts related to our aims and (2) using them increased comparability to the other 3 US-based studies of At-Risk . We slightly adapted the surveys to make them more relevant to the UK context ( Multimedia Appendix 2 ).

Interview Schedules

For the pretraining interviews with SENCos and mental health leads, we developed a topic guide about current practice ( Multimedia Appendix 3 ) with the specific purpose of creating Mental Health Resource Maps for each school (refer to the Procedures section). The main topics pertained to formal and informal procedures for when staff members suspect that a child might have mental health difficulties or increased risk, as well as the types of support available.

For the posttraining interviews with teachers, TAs, and strategic stakeholders (ie, those with key leadership roles, including senior leadership teams [SLTs], school governors, and SENCos and mental health leads), we developed 3 separate topic guides ( Multimedia Appendix 3 ), which were informed by our research questions, systematic reviews [ 28 , 30 ], and the Consolidated Framework for Implementation Research [ 62 , 63 ]. For teachers and TAs who completed At-Risk and strategic stakeholders, interview topics included the acceptability of the training, the practicality of implementing it in schools, the utility of further refinement and testing, possible harms associated with the training (if any), and suggestions for adaptations. For teachers and TAs who did not complete the training, topics included reasons for not completing it, barriers to acceptability and practicality, and suggestions for adaptations.

Interviews With SENCos and Mental Health Leads

We conducted a pretraining interview with each school’s SENCo or mental health lead to develop a “Mental Health Resource Map” with information on referral processes and available support. These maps served an ethical purpose by ensuring that pupils identified as potentially having mental health difficulties would have the best possible chance of being linked to care and support.

Completing At-Risk

Schools’ timelines for the study varied due to the pandemic and other commitments. School D completed the training in December 2020; school E completed the training in March 2021; schools B, C, and F completed the training in May 2021; and school A completed the training in June 2021. At baseline (T1), participants completed a Teacher and TA Identification Form and the pretraining survey. They then completed the At-Risk training. We encouraged schools to designate specific time for the training, which 3 schools (schools D, E, and F) did. One week after training (T2), participants were asked to complete a second Teacher and TA Identification Form and the posttraining survey. Three months after the training or at the end of the school year (whichever came first; T3), participants completed a third Teacher and TA Identification Form as well as SDQs for all pupils. All questionnaires were completed on the University of Cambridge Qualtrics platform (Qualtrics International Inc).

Feedback Provision

After T2, we provided all SENCos and mental health leads but not teachers or TAs with feedback regarding which children had been identified as having mental health difficulties or increased risk. After T3, we provided SDQ scores for each child as well as whole-class distributions (where available). This feedback was provided to ensure the ethical conduct of the study.

Interviews With Teachers, TAs, and Strategic Stakeholders

We aimed to recruit at least 3 teachers or TAs who completed the training per school, 3 to 5 teachers or TAs who had not completed the training across all schools, and up to 3 strategic stakeholders per school for posttraining semistructured interviews. Schools contacted staff members directly with an invitation to complete a virtual interview.

Quantitative Outcomes

Analytical samples.

For the main analysis, participants were included if they (1) completed at least the pretraining (T1) questionnaires and the training itself and (2) had what we judged to be a typical number of children they regularly worked with. For the latter criterion, given that the average UK primary school class size is approximately 27 to 28 pupils [ 64 ], we excluded teachers and TAs who worked with <10 children (as we suspected this would not be a random selection of pupils and would therefore influence aggregate identification rates) and those who worked with >60 children (as we believed that it would be difficult for a teacher or TA to know >2 classes’ worth of children well enough to make accurate judgments about their mental health).

Teacher and TA Self-Efficacy and Preparedness

To assess teachers’ and TAs’ preparedness, self-efficacy, and perceptions of training impact, we calculated the absolute and relative frequencies of responses to the pre- and posttraining surveys. Participants were eligible for inclusion in this analysis only if they had pretraining (T1) data.

Identification Outcomes

On the basis of the Teacher and TA Identification Forms, we calculated the number and percentage of pupils in each class whom teachers and TAs perceived as having mental health difficulties or increased risk at each time point. We summarized these across all participants using medians and IQRs.

We then calculated SDQ scores, which we compared with responses from the Teacher and TA Identification Form by calculating (1) the median and IQR for the percentage of children identified by participants who did not have elevated SDQ scores and (2) the median and IQR for the percentage of children with elevated SDQ scores who were not identified by participants. To be included in these analyses, participants had to have completed all 3 time points. For the first outcome, they had to have completed an SDQ for all children they identified in the Teacher and TA Identification Form . For the second outcome, they had to have completed SDQs for at least 80% of their class. Where it was possible to match pupil IDs between teachers and TAs, we pooled SDQ data such that, if one participant did not meet the inclusion criteria themselves, they could still be included if the SDQ data were available from another staff member working with the same children.

Mental Health Support Outcomes

Finally, for each time point, we calculated medians and IQRs for the proportion of identified children with each of the 9 mental health support outcomes (refer to the Teacher and TA Identification Form section for the outcomes) .

Sensitivity Analyses

We also conducted 2 post hoc sensitivity analyses. The first sensitivity analysis excluded all participants from school D. When we prepared feedback for school D (the first school to complete the training), we learned that most participants at the school had misinterpreted the Teacher and TA Identification Form. We edited the form and instructions accordingly to address this issue, but therefore, school D participants completed a slightly different form than the other schools. The second sensitivity analysis was a complete case analysis intended to explore observed differences in outcomes according to whether participants had completed all 3 time points. For the analysis of outcomes pertaining to preparedness, self-efficacy, and perceptions of training impact, we included all participants who completed the surveys at least at T1 and T2.

Statistical Analysis

For all quantitative outcomes, we focused on preliminary, descriptive comparisons across the 3 time points and did not perform any formal hypothesis testing. This aligns with established recommendations for feasibility studies, which generally lack the statistical power necessary for a clear interpretation of hypothesis-testing results [ 65 - 68 ]. We conducted all quantitative analyses in R (version 4.0.3; R Foundation for Statistical Computing) [ 69 ] except for the comparison of Teacher and TA Identification Forms and SDQ scores, for which we used Microsoft Excel (Microsoft Corp). We created all plots using the ggplot2 [ 70 ] and likert packages [ 71 ]. To score the SDQs, we used the freely available R code on the Youthinmind website [ 72 ].

Qualitative Outcomes

We considered 3 analysis approaches for the interview and qualitative questionnaire data: content analysis [ 73 ], framework analysis [ 74 ], and reflexive thematic analysis [ 75 , 76 ]. We initially decided to use content analysis for the survey comments and reflexive thematic analysis for the interviews; however, as we familiarized ourselves with the data, we realized that there was significant overlap between the survey comments and interviews and decided that analyzing them separately was not a useful distinction. As our main aim was to generate insights into the program and its future potential, we decided to use the 6-phase reflexive thematic analysis by Braun and Clarke [ 76 ] for all qualitative data due to its flexibility and ability to generate themes both inductively and deductively. ES developed the initial themes, and MF and EH helped clarify and enrich them. ES and MF worked together to name and refine the themes before the final write-up. We managed and coded all qualitative data in ATLAS.ti (version 9.1.3; ATLAS.ti Scientific Software Development GmbH) and additionally created manual thematic maps to better visualize and understand patterns between our data.

Ethical Considerations

This study was approved by the University of Cambridge Psychology Research Ethics Committee (PRE 2019.076). We obtained active informed consent from all teachers and TAs who took part in the study. We used an opt-out model for parental consent whereby parents received (directly from the schools via their preferred communication routes) an information sheet detailing study aims, procedures, how data would be used, and the right to opt their child out of participation. Parents had 2 weeks to opt their child out of the study by returning a hard copy of the opt-out form or emailing or calling the school. Schools kept track of all opt-outs and instructed teachers and TAs not to include these children in their forms. All quantitative data were collected using anonymous pupil and staff identifiers generated by the participating schools, and all qualitative data were deidentified before analysis, with identifiable information stored on secure servers at the University of Cambridge. Teachers and TAs received £20 (approximately US $28) vouchers for completing the training and questionnaires for at least 2 of the 3 time points and an additional £10 (approximately US $14) for taking part in an interview. School staff members who created the anonymous identifiers received £10 (approximately US $14) vouchers to thank them for their time.

Participants

A total of 6 schools participated in this study (Table S1 in Multimedia Appendix 4 [ 40 ]). Among these 6 schools, there were 4 (67%) from Cambridgeshire and 1 (17%) each from Greater London and Merseyside; 5 (83%) were located in urban areas and 1 (17%) was located in a rural area. All but 1 school (5/6, 83%) were situated in areas of above-average deprivation, and 50% (3/6) of the schools had a higher-than-average proportion of pupils eligible for free school meals. In total, 67% (4/6) of the schools had a high proportion of White pupils (>80%), and 33% (2/6) of the schools were more diverse, with approximately 20% of pupils from Black, Black British, Caribbean, or African backgrounds (school B) or Asian or Asian British backgrounds (school E).

A total of 108 teachers and TAs completed the T1 questionnaires and the training itself, 89 (82.4%) completed the T2 questionnaires, and 70 (64.8%) completed the T3 questionnaires ( Table 1 ), with 54 (50%) having completed all 3. After excluding those teachers and TAs who did not meet the inclusion criteria for the analyses, the final analytical samples were as follows:

• Main analysis of identification and mental health support outcomes: n=97 at T1, n=75 at T2, and n=57 at T3.
• Main analysis of preparedness, self-efficacy, and training impact outcomes: n=107 at T1 and n=83 at T2.
• Main analysis comparing identification outcomes with SDQ scores: n=28 and n=25 (refer to the following section).
• Complete case sensitivity analysis: n=51 at T1, T2, and T3.
• Sensitivity analysis excluding all teachers and TAs from school D: n=70 at T1, n=54 at T2, and n=41 at T3.

Compared with the 2019-2020 national workforce statistics for teachers and TAs working in state-funded nursery and primary schools [ 77 ], our sample had a similar proportion of women (81/89, 91% in our sample vs 90.9% nationally) and a slightly higher proportion of White staff members (82/89, 92% in our sample vs 90.5% nationally).

A total of 7.4% (8/108) of school staff members from 67% (4/6) of the schools completed an interview ( Table 2 ).

a N=89 because this information was collected only at T2.

b N NA =4 (number with missing data for this question).

c Percentages add up to >100 because some participants had multiple roles.

d SENCo: special educational needs coordinator.

e N NA =7 (number with missing data for this question).

a PSHE: Personal, Social, Health and Economic.

b SENCo: special educational needs coordinator.

c TA: teaching assistant.

d HLTA: higher-level teaching assistant.

Pretest-posttest changes suggested that participating in the training was beneficial for the staff and that they had positive perceptions of the training. Findings regarding preparedness ( Figure 2 ) suggest improvements across all domains of recognizing and acting upon concerns about pupils’ mental health, particularly in terms of using key communication strategies and working with parents. Findings regarding self-efficacy ( Figure 3 ) suggest that participants were more confident in their abilities to discuss their concerns about pupils’ mental health after the training than before. Again, the largest changes were observed in discussing concerns with parents and applying key communication strategies. Finally, findings regarding teachers’ and TAs’ perceptions of the impact of applying the skills of the training ( Figure 4 ) suggest that they were generally positive about the possible effects of the training on pupil outcomes (ie, attendance and academic success), teacher-pupil rapport, and the classroom environment. The results from the complete case analysis ( Multimedia Appendix 5 ) were nearly identical to those of the main analysis (all differences were ≤3 percentage points in magnitude).

In terms of how many pupils were identified as having mental health difficulties or increased risk, participants identified similar proportions of their pupils before and immediately after the training and then fewer over time. The median percentage of pupils whom participants believed had mental health difficulties or increased risk was 10% (IQR 6.7%-18.2%) at T1, 10% (IQR 4.5%-16.7%) at T2, and 7.4% (IQR 5.0%-16.7%) at T3. The directions of change were similar for both sensitivity analyses (whereby teachers and TAs identified fewer children over time), with slight differences. For the sensitivity analysis excluding school D ( Multimedia Appendix 6 ), the percentages were slightly (approximately 2 percentage points) higher. For the complete case analysis, the decrease was also notable 1 week after the training, decreasing from 10% (IQR 6.7%-17.3%) at T1 to 8% (IQR 3.9%-16.7%) at T2 and 7.4% (IQR 5.7%-16.7%) at T3.

In terms of the accuracy of identification, it seems that teachers and TAs became slightly more accurate over time in comparison to pupils’ SDQ scores (although it is important to acknowledge the limitations described in the Methods section regarding questionnaire comparability). The median percentage of children identified by participants who did not have elevated SDQ scores was 40% (IQR 0%-50%) at T1, 27.2% (IQR 0%-50%) at T2, and 25% (IQR 0%-50%) at T3. The median percentage of children with elevated SDQ scores who were not identified by participants was 68.8% (IQR 42.9%-87.5%) at T1, 66.7% (IQR 50%-88.2%) at T2, and 57.1% (IQR 33.3%-87.5%) at T3. In the sensitivity analysis excluding school D, the results were similar (typically within 5 percentage points); one small difference was that the median percentage of children identified by teachers and TAs who did not have elevated SDQ scores was 0% (IQR 0%-50%) at T2. The results of the complete case analysis were identical to those of the main analysis.

Overall, the findings suggest that the training may be beneficial for facilitating conversations and access to school-based support (but not external support) for pupils with identified mental health difficulties or increased risk. Figure 5 presents the findings for the 9 mental health support outcomes among identified children across the 3 study time points. As with before the training, there was typically a wide variation in outcomes.

A comparison across time points suggests that participants formally documented their concerns and spoke with the SENCo, pastoral lead, or mental health lead for a greater proportion of identified pupils after the training than before. For example, at T1, teachers and TAs documented concerns for a median of 50% (IQR 0%-100%) of identified pupils; this increased to 56.3% (IQR 4.2%-100%) at T2 and 75.7% (IQR 0%-100%) at T3. The equivalent statistics for speaking with the SENCo, pastoral lead, or mental health lead were a median of 66.7% (IQR 16.7%-100%) at T1, 75% (IQR 50.0%-100%) at T2, and 95.5% (IQR 50.0%-100%) at T3. There was no change in speaking with another staff member, but this was because nearly all participants did so across all time points. Finally, the percentage of pupils whom teachers and TAs spoke with (or whose parents they spoke with) also increased after the training, with a median of 33.3% (IQR 0%-87.5%) at T1, 61.9% (IQR 0%-100%) at T2, and 50% (IQR 0%-100%) at T3.

A comparison across time points also suggests increases in school-based support for identified children after the training compared with before. The median percentage of pupils identified by teachers and TAs who received in-class support increased from 75% (IQR 35.4%-100%) at T1 to 100% at T2 and T3 (IQR 50%-100% and 66.7%-100%, respectively). There was a more modest increase in the receipt of in-school support or in-house support plans, with a median of 40% (IQR 0%-71.4%) of identified pupils receiving them at T1 compared with 50% at T2 and T3 (IQR 3.6%-100% and 8.3%-81.4%, respectively). There was very little change in documented SEMH status or referral or access to specialist mental health services. For each of these outcomes, the median percentage of identified pupils was 0% across time points.

The findings from the sensitivity analyses were similar to those of the main analysis in terms of direction, although improvements across time in the complete case analysis ( Multimedia Appendix 5 ) tended to be more modest than for the main analysis.

Acceptability and Practicality

Quantitative findings.

Quantitative data from the posttraining survey showed that participants were generally positive about the training. Of the 83 participants who completed the survey, 53 (64%) rated it as “good” and 13 (16%) rated it as “very good.” An additional 17% (14/83) rated it as “fair,” 2% (2/83) rated it as “poor”, and 1% (1/83) as “very poor.” A total of 84% (70/83) of the teachers and TAs said that the scenarios in the training were relevant to them. Finally, most participants (74/83, 89%) would recommend the training to other educators.

Qualitative Findings

Qualitative data also suggested that school staff generally found the training practical and acceptable. We generated three themes from our survey and interview data:

• Individual fit: positive perceptions, self-efficacy, and change.
• Institutional fit: alignment with school values and context.
• Taking it forward: improvements and implementation.

Additional findings on possible harms are presented in Multimedia Appendix 7 .

Individual Fit: Positive Perceptions, Self-Efficacy, and Change

In general, participants perceived the program to be a “good fit” with their personal philosophies and practice. Regarding the training itself, many appreciated the included scenarios, particularly in terms of their relevance to their practice. The format of the training—primarily that it was web-based and required active role-play—was also viewed as useful, engaging, and novel and might have contributed to its perceived usefulness. For example, one teacher commented:

The interactive elements of the training were brilliant and something which I have never encountered before! [Survey respondent (SR) 56; school E]

One teacher and well-being lead described:

I think it definitely made you think. [...] you had to really think about what was being said and the response that you would give, reflecting back on sort of the knowledge that they’d given you beforehand, so I thought that was good. [Interviewee 1]

Other participants suggested that opportunities to practice skills during the training improved the likelihood of using those skills in day-to-day practice.

Participants also believed that they had learned a lot from the training, especially in terms of skills and strategies. These included but were not limited to the skills within the At-Risk “EASING” strategy (check your Emotions, Ask for permission, be Specific, use I statements, keep it Neutral, and show Genuine curiosity). Importantly, there was evidence that participants had also applied new skills. Several participants described having new conversations with pupils or parents facilitated by the skills and strategies from the training. For example, one teacher described:

It was that permission thing [...] I wanted to ask [a child] about his home life [...] and kind of he just cried and didn’t want to speak about it anymore, and then when I asked him if we were OK to talk about it, he said, “Actually no, because I think I’m going to cry again,” so then we left it. And then he came to me the following week, and [...] said, “Can we talk about it now?” [...] so actually me asking that, it was the wrong time for him to talk about it, he wasn’t ready, he would have just been emotional, and wouldn’t have been able to get his words out, and actually the week after, him coming to me and saying, “Can we have a little chat,” works perfectly [...] And now we’re more aware of his situation. [Interviewee 2]

This skill seems to have enabled this pupil to have this conversation with the teacher in a manner (in terms of time, place, and identified person) that suited him. Other participants provided similar examples, referencing how skills from the training had facilitated better outcomes.

However, it is important to note that the perceived usefulness of the training varied. Most notably, some participants indicated that their previous training or role made the training less impactful. Illustratively, when asked how the training had impacted their practice, one TA responded:

Having previously received similar training, due to my role, I do not have any recent cases where the training would have changed the way I carried out discussions. [SR 60; school E]

Institutional Fit: Alignment With School Values and Context

Sustainable school-based programs should also align with the values of the school more broadly. Participants often referenced the importance of schools’ prioritization of pupil mental health. For example, one teacher described:

[Mental health is] a conversation which is constantly ongoing and trying to constantly better our practices and make sure we’re looking after them as best as we can and spotting things as best we can as well. [Interviewee 3]

This description demonstrates how prioritizing mental health can promote the critical evaluation of related school practices as well as the additional provision of training opportunities. In many cases, support from the SLT led to formal recognition of pupil mental health within school policies or plans. One strategic stakeholder explained:

I think because our school have well-being and mental health as such a focus, SLT are very supportive of doing things like this and they’re very accommodating. So when I said we had the training and people were going to have to take part in the training, it was very flexible, although they had other ones lined up, they were quite happy to move things around to make things work. And I think, the fact it is such a priority in our school definitely makes that easier. [Interviewee 1]

In this school, mental health and well-being were one of three main school priorities. As indicated previously, direction setting from the SLT is key to ensuring momentum and impetus. However, as others noted, it is important that support from the SLT is genuine rather than being “just another tick box” (Interviewee 4) exercise.

Another facet of institutional fit pertained to the practical aspects of the training. Schools are time- and resource-limited settings, so mental health training needs to fit within this context. The format of At-Risk, especially its flexibility and relatively low time requirements, was viewed as beneficial, with comments such as “For the amount of time [...] I got a huge amount from it” (Interviewee 4). Others made direct comparisons with other training courses. For example, one higher-level TA had previously completed a 1-day, in-person training course with a similar purpose to that of At-Risk. While she preferred the in-person training, she listed the benefits of both types:

[In the in-person training] you can then query and question to your trainer, so you’ve got that interaction, so that obviously isn’t there, is it, on the computer one. [...] if I was looking from a management point of view, I would say, budgetary, I’m sure it’s cheaper [...] to use [At-Risk], not just cheaper as in [...] money, [...] but also cheaper in time [...] So probably if I was looking [...] with my management hat on, I would say the computer-based [training] would get the same message, or similar message, across for a wider audience for probably a cheaper cost. [Interviewee 7]

In terms of efficiency, this participant highlighted the favorable input-to-output ratio of At-Risk , which could allow more staff members to participate in training. This quote also highlights that schools could use At-Risk flexibly. For example, schools might assign staff members to different training programs based on their roles and previous experience, with more intensive, in-person training for staff members with more significant mental health roles and At-Risk for those with fewer responsibilities or less experience.

Taking It Forward: Improvements and Implementation

Participants offered key insights into how to take the training forward in terms of both changes to the training itself and how best to implement it, primarily by tailoring it to the UK context. In terms of language, there was some reference to the American accent, but more so, participants highlighted the need to adapt some of the terminology and signposting resources to reflect UK support systems. They also made suggestions about additional training that could be useful with different topics (such as bullying) and age groups (particularly for younger children).

In addition to improvements and adaptations to the training itself, participants illustrated the importance of implementation. A common theme was that, to maximize impact, the training should include follow-up discussions or live workshopping. One teacher suggested:

I think some kind of “live” element to conclude the training—to have a “real” person to ask questions to as part of a group video chat could have been useful. Also, maybe to ask advice about particular scenarios that we may have found ourselves in in the past. [SR 56; school E]

By facilitating greater engagement and critical thinking, a live element could enhance the impact of the training and potentially make At-Risk more acceptable to those who generally prefer face-to-face training. Participants indicated that someone internal, for example, the SENCo, would be best placed to lead a live element and would enable staff to practice role-playing based on situations and scenarios specific to each school.

There was also wide acknowledgment that any training had to lie within a strong support system. This began with having a clear referral pathway for identified concerns, which was viewed as important for facilitating access to support. In some cases, teachers and TAs were able to find new ways to support children after completing the training. However, in many cases, participants—and strategic stakeholders in particular—explained that support had not always been readily available. For example, one strategic stakeholder recounted what happened after the training:

A lot of them are people saying to me, “What are you going to do about it?” about different children. And I, because some of our support staff don’t know the sort of route for getting extra support, or they’re really shocked to find actually there’s nothing out there for these children...it’s about what we can do in school, and I think people have been really quite shocked about that. You know, they just presume I can make a phone call and these children will get face-to-face counselling. [Interviewee 5]

This shows the importance of embedding the training within a wider support system, including collaboration with external agencies. However, many interviewees referenced the systemic issues that schools face in helping pupils access specialist support, particularly in terms of the high thresholds and long waiting lists that exist for many external services. While schools may be able to provide beneficial support for children, particularly for those with lower-level difficulties, this indicates an ongoing area of need for schools and their pupils.

Summary of Findings

This study offers the first UK evidence for Kognito’s At-Risk for Elementary School Educators , extending findings from 3 US-based trials and providing needed evidence regarding the potential utility, acceptability, and practicality of brief, interactive web-based mental health training for school staff. Overall, the findings showed that At-Risk is a feasible means of improving the identification of and response to pupil mental health difficulties in UK primary schools. Quantitative findings showed that staff preparedness and self-efficacy in identifying and responding to mental health difficulties increased after the training. Identification rates did not increase (and, in fact, decreased at the 3-month follow-up), but there was some suggestion that teachers’ and TAs’ identification became slightly more accurate in comparison with SDQ scores. Crucially, for those pupils identified as having mental health difficulties or increased risk, in-school mental health support outcomes (ie, documentation or discussion of concerns, conversations with pupils and parents, and in-class and in-school support) increased after the training, but more “downstream” outcomes (ie, documented SEMH status and referral and access to external mental health services) did not. Qualitative findings indicated that participants generally found the training acceptable and practical, with many explaining how they intended to use or had already used the skills they learned to improve their practice. Participants also suggested several useful improvements for the training and its implementation, including making it more relevant to the United Kingdom, adding more scenarios, and including a live element in the implementation of the training.

Findings regarding confidence and preparedness reflect those of the 3 US-based studies of At-Risk [ 17 , 41 , 42 ] and the wider literature surrounding teacher mental health training [ 31 ]. In general, mental health training seems to be effective in improving staff confidence. For example, 2 Australian-based studies [ 37 , 78 ] found that secondary school teachers who completed training felt more confident discussing their concerns and helping pupils with their mental health. Another UK-based study of a psychoeducational training program to improve recognition of depression in secondary schools [ 79 ] found significant pretest-posttest improvements in teacher confidence in their knowledge of symptoms, ability to recognize symptoms, and knowledge about how to speak with pupils about their mental health. However, not all studies have shown an impact, with a prominent UK-based study of mental health first aid training finding no effect on educators’ confidence in helping pupils with their mental health [ 80 ].

The general decrease in the proportion of pupils identified as having mental health difficulties or increased risk stands in contrast to previous studies of At-Risk , which found that school staff identified significantly more pupils of concern after completing the training [ 17 , 41 ]. Evidence of the effect of other training programs on identification is extremely limited [ 30 , 31 , 36 ], and differences in context, training content and delivery, baseline knowledge, and outcome measurement make it difficult to compare findings across studies. Two vignette-based studies showed little effect of either mental health first aid [ 78 ] or psychoeducational [ 81 ] training on identification (although each study also reported high recognition of difficulties before the training), whereas studies focused on real-world identification have shown mixed results [ 79 , 82 ]. However, changes in the proportion of identified pupils must be contextualized within the accuracy of identification. There are consequences of both over- and underidentification [ 83 , 84 ], most notably in terms of inefficient allocation of limited mental health support resources. While comparison with the SDQ suggested that there was some improvement in terms of the accuracy of identification following the training, underidentification remained a substantial challenge, with between one-half and two-thirds of pupils with elevated SDQ scores remaining unidentified by teachers and TAs. The underidentification of children with mental health difficulties in educational settings, particularly for children with internalizing as opposed to externalizing problems [ 85 ], has been well documented in the literature [ 30 ], and it is likely that a combination of identification models is required to address this challenge [ 27 , 29 ].

Promisingly, the training appeared to be useful in terms of connecting pupils with care and support, an outcome not frequently measured in other studies [ 30 , 31 , 34 ]. First, the findings suggested that participants had conversations about or documented concerns for a greater proportion of identified pupils following the training, which reflects findings from previous studies of At-Risk [ 17 , 41 ]. This is a rather unique outcome in the literature as other training evaluations have found no difference between training and control groups in terms of conversations with pupils and colleagues [ 78 ]. Importantly, this study went beyond conversations to include outcomes pertaining to in-school and external support. The increases in in-class and in-school support for identified pupils reflect findings of the UK-based study by Kidger et al [ 80 ] of mental health first aid training and the Australian-based pilot study by Parker et al [ 37 ] of a web-based training program, each of which found a positive effect of the training on helping behaviors. Although in-class and in-school support seemed to increase following the training, it is notable that referrals and access to specialist services did not. There are several plausible explanations for this finding. For example, it is likely that school staff were already aware of children with the most severe mental health difficulties and were confident and able to support newly identified pupils—who might have had lower-level mental health difficulties—within the school setting. However, if the training did lead to the identification of children who might benefit from specialist care, there are many barriers to accessing such support (eg, availability and long waiting lists) that might have influenced these outcomes, as reflected in both the qualitative interviews and the wider literature [ 23 , 86 ].

In addition, quantitative and qualitative findings suggested that the program was a good fit for individuals and schools, which aligns with previous research on the acceptability and perceived need for mental health training for school staff [ 18 , 20 , 27 - 29 , 87 , 88 ]. The training’s format seemed to be a key contributor to its feasibility. With a few exceptions [ 37 , 39 , 89 ], the web-based simulation-based format of At-Risk is unique among training programs and is well aligned with teachers’ preferences. For example, in their focus group study of UK secondary school teachers, Shelemy et al [ 20 ] found that participants wanted engaging, interactive, and concise training that included practical strategies and illustrative case studies, all of which are central to At-Risk . While the authors found that teachers disagreed over the usefulness of web-based training, it is possible that these concerns would have decreased during the COVID-19 pandemic as staff became more accepting of web-based opportunities to learn.

Qualitative findings also demonstrated the importance of school context and culture, which have been highlighted in previous research [ 27 ]. In particular, participants noted the importance of school culture in adopting mental health interventions into regular practice. In their systematic review, Moore et al [ 90 ] identified school culture, values, and policies as key facilitators of sustaining mental health interventions. A related area of focus was support from the SLT. This support is a well-recognized factor contributing to intervention success and sustainability for several reasons, including these leaders’ practical role in communicating about interventions and allocating specific time and resources to them [ 43 , 90 , 91 ]. However, it is important to recognize that mental health training for school staff may be even more needed and impactful in schools where mental health is not as much of a priority.

Limitations

Our mixed methods approach, wide range of outcomes, and diverse sample of participating schools offer rich information regarding the feasibility of At-Risk in the United Kingdom. These strengths notwithstanding, there are also several limitations to consider when interpreting the findings. The nonrandomized design, while common for feasibility studies, prevents any conclusions regarding causality and also limits the exploration of other factors that may have influenced outcomes (eg, providing teachers and TAs with the Mental Health Resource Maps or SENCos and mental health leads with feedback on identified pupils). In terms of recruitment and retention, the study had 50% (54/108) attrition. Several factors may have influenced this, including the increased pressure on school staff due to the COVID-19 pandemic, the timing of the study within the school year, and the requirement to communicate with participants only via the study link person. While we tried to explore the effect of attrition through a complete case sensitivity analysis, we lacked important information on the characteristics of those who dropped out as this information was collected only at T2. Furthermore, we were only able to recruit 8 staff members for the posttraining interviews, which was far below our recruitment target. Low participation rates could again be due to several factors, including the impact of the COVID-19 pandemic or competing priorities. Of note, we were not able to recruit anyone who did not complete the training, any headteachers, or any staff from 2 of the schools (schools A and D). This could mean that we lack viewpoints that may be important for understanding the feasibility and utility of the training.

There were also limitations associated with the study measures. While the Teacher and TA Identification Form was informed by the literature and reviewed by our primary school staff advisory group, its validity and reliability are unknown. In addition, the questionnaire only measured mental health support outcomes for those pupils identified as having mental health difficulties or increased risk. Therefore, we do not have information on those who were not identified. The measure is also based on teacher and TA reports and so may not have complete information about all types of support that pupils receive. Another important limitation pertains to the comparator used to assess the identification outcomes. To understand the potential utility of the training program, it is important to have a robust comparator. While we chose to use the teacher-report SDQ, it would also have been interesting to compare identification outcomes with parent-rated mental health difficulties, particularly in light of the low interrater agreement of common measures of child mental health difficulties [ 92 ]. An even stronger comparator would be to assess the teacher and TA identification outcomes against a clinical interview; however, this was not feasible in this study.

Finally, at the time of writing, At-Risk is currently not available for use as Kognito restructures its offerings. This demonstrates a trend that unfortunately is a common occurrence in the field of mental health, whereby many evidence-based digital tools are not available to potential end users [ 93 ]. Nonetheless, the learnings from this feasibility study offer rich information on what type of content and format may be useful for training programs in this area and, as such, can support further development and evaluation in the field.

Implications for Practice

Studies have consistently demonstrated that school staff would appreciate additional training on how best to support pupil mental health [ 18 , 20 , 87 , 88 ]. However, to be scalable, such programs must be realistic in terms of time, cost, and resource requirements [ 28 , 90 , 91 ]. Contextualized within the wider literature on school-based mental health interventions, the findings from this study suggest that mental health training is a feasible option for upskilling school staff to identify and respond to pupil mental health difficulties. They further highlight several specific factors that might positively contribute to feasibility and scalability, many of which are reflected in the broader literature on mental health training [ 20 , 28 ]. For example, teachers and TAs appreciated that the training actively engaged them in learning and applying new skills and that it used realistic examples to demonstrate the real-world applicability of the training, whereas school leaders identified the relatively low time and cost requirements and flexibility as key factors that could make the training feasible for their school context.

However, this is not to say that there are no implementation barriers associated with At-Risk or similar training programs. While the resources required to implement At-Risk are relatively low compared with other training programs, they must still be considered within the context of other school priorities. As demonstrated in the interviews and the wider literature [ 3 , 43 , 90 , 91 ], support from school leadership is essential for securing the time and budget required to implement a training such as At-Risk , and in schools where mental health is not a priority, there are likely to be many barriers to implementation . Even in schools with strong support from the leadership team, it may be difficult to find the requisite budget, time, and human resources to devote to the training. Finally, as is the case with any school-based mental health intervention, it is important that schools do not take sole responsibility for pupils’ mental health. Active partnership between schools and mental health services is key to ensuring that schools feel empowered and supported in this role [ 21 , 90 , 94 ]. While the schools in this study worked hard to support pupils as best they could, interviewees expressed frustration about the difficulty of accessing external support for children who could benefit from it. This is not an uncommon theme in the wider literature surrounding school-based interventions [ 20 , 23 , 91 ] and is a key consideration for scaling up training programs.

Implications for Future Research

The promising findings of this study suggest that additional research is needed to explore the role of scalable mental health training in supporting schools to protect and promote children’s mental health. On the basis of gaps in the literature, particular areas of interest include training for primary school staff (as most are focused on secondary school staff), web-based training (as opposed to traditional time- and resource-intensive in-person training), and training that takes a “whole school approach” by including all school staff members (rather than only teachers). This final area is especially interesting as findings from this study and others [ 27 ] have highlighted stakeholders’ preference that training programs include all school staff members. While our study jointly analyzed findings for teachers and TAs, future research would do well to consider how the unique roles and perspectives of these professionals—as well as other staff members within the school setting—might influence outcomes. Furthermore, future research should be more inclusive about their choice of outcomes, as too often evaluations of school staff training programs have focused on intermediate outcomes such as knowledge or confidence [ 31 ] without considering more “downstream” outcomes such as access to support. Finally, as demonstrated in our study, there is great value in using mixed methods approaches and including information about wider issues of feasibility and implementation, and studies that take this broader lens can help identify programs that are scalable, sustainable, and effective.

Conclusions

School staff would welcome additional mental health training to enable them to respond to pupil mental health difficulties, but there are many barriers to implementing such training at scale. Therefore, training programs that have relatively low time and resource requirements have great potential to fulfill an unmet need in schools. This mixed methods feasibility study showed that At-Risk for Elementary School Educators —an example of a brief, interactive web-based training program—is a feasible means of empowering school staff to accurately identify and respond to pupil mental health difficulties and increased risk.

Acknowledgments

The authors would like to thank Professor Paul Ramchandani for his input in the design of this study, the Cambridge Mental Health in Schools Advisory Group for sharing their views and advice throughout the study, and the staff at the 6 schools that took part in this study for their time and effort. This study was funded by the UK Research and Innovation Emerging Minds network (grant ES/S004726/2), and the training was provided to the schools free of cost by Kognito. ES was funded by a Gates Cambridge Scholarship (grant OPP1144) for the duration of the study. MF is funded by the National Institute for Health and Care Research (NIHR) Oxford and Thames Valley Applied Research Collaboration at the Oxford Health National Health Service (NHS) Foundation Trust. PBJ is funded by the NIHR (grant 0616-20003). All research in the Cambridge Department of Psychiatry is supported by the NIHR Applied Research Collaboration East of England and the Cambridge Biomedical Research Centre at the Cambridgeshire and Peterborough NHS Foundation Trust. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, the Department of Health and Social Care, or the Bill & Melinda Gates Foundation.

Data Availability

The data sets generated during and analyzed during this study are not publicly available due to restrictions associated with our ethics approvals but are available from the corresponding author on reasonable request.

Conflicts of Interest

Kognito is a for-profit company. After reviewing their questionnaires on preparedness and self-efficacy and having found them rigorous and unbiased, we independently decided to include them as outcomes in our study. Kognito had no role in the study design, analysis, or publication. PBJ was a scientific advisory board member for MSD. All other authors declare no other conflicts of interest.

Teacher and Teaching Assistant Identification Form.

Kognito pre- and posttraining surveys.

Interview topic guides.

School characteristics.

Results from complete case sensitivity analysis.

Results from the sensitivity analysis excluding school D.

Quantitative and qualitative findings pertaining to the potential harms of At-Risk.

• Transforming children and young people's mental health provision: a green paper. Department of Health and Department for Education, United Kingdom Government. Dec 2017. URL: https:/​/assets.​publishing.service.gov.uk/​media/​5a823518e5 274a2e87dc1b56/​Transforming_children_and_young_people_s_mental_health_provision.​pdf [accessed 2024-03-25]
• Future in mind: promoting, protecting and improving our children and young people’s mental health and wellbeing. National Health Service England. URL: https://assets.publishing.service.gov.uk/media/5a80b26bed915d74e33fbe3c/Childrens_Mental _Health.pdf [accessed 2024-03-25]
• Promoting children and young people's mental health and wellbeing. Office for Health Improvement and Disparities and Department for Education, United Kingdom Government. Mar 20, 2015. URL: https:/​/www.​gov.uk/​government/​publications/​promoting-children-and-young-peoples-emotional-health-and-wellbeing [accessed 2024-03-25]
• Newlove-Delgado T, Williams T, Robertson K, McManus S, Sadler K, Vizard T, et al. Mental health of children and young people in England 2021 - wave 2 follow up to the 2017 survey. National Health Service Digital. 2021. URL: https://openac cess.city.ac.uk/id/eprint/28718/ [accessed 2024-03-25]
• Vizard T, Sadler K, Ford T, Newlove-Delgado T, McManus S, Marcheselli F, et al. Mental health of children and young people in England, 2020. National Health Service Digital. Oct 22, 2020. URL: https://www.infocoponline.es/pdf/mhcyp _2020_rep.pdf [accessed 2024-03-25]
• Pitchforth J, Fahy K, Ford T, Wolpert M, Viner RM, Hargreaves DS. Mental health and well-being trends among children and young people in the UK, 1995-2014: analysis of repeated cross-sectional national health surveys. Psychol Med. Jun 2019;49(8):1275-1285. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Newlove-Delgado T, McManus S, Sadler K, Thandi S, Vizard T, Cartwright C, et al. Child mental health in England before and during the COVID-19 lockdown. Lancet Psychiatry. May 2021;8(5):353-354. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Creswell C, Shum A, Pearcey S, Skripkauskaite S, Patalay P, Waite P. Young people's mental health during the COVID-19 pandemic. Lancet Child Adolesc Health. Aug 2021;5(8):535-537. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fazel M, Hoagwood K, Stephan S, Ford T. Mental health interventions in schools 1: mental health interventions in schools in high-income countries. Lancet Psychiatry. Oct 2014;1(5):377-387. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. Jun 2005;62(6):593-602. [ CrossRef ] [ Medline ]
• Greenberg MT, Domitrovich CE, Weissberg RP, Durlak JA. Social and emotional learning as a public health approach to education. Future Child. 2017;27(1):13-32. [ CrossRef ]
• Humphrey N, Wigelsworth M. Making the case for universal school-based mental health screening. Emot Behav Difficulties. 2016;21(1):22-42. [ CrossRef ]
• Kutcher S, Wei Y. School mental health: a necessary component of youth mental health policy and plans. World Psychiatry. Jun 2020;19(2):174-175. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fazel M, Soneson E. Current evidence and opportunities in child and adolescent public mental health: a research review. J Child Psychol Psychiatry. Dec 2023;64(12):1699-1719. [ CrossRef ] [ Medline ]
• Stephan SH, Weist M, Kataoka S, Adelsheim S, Mills C. Transformation of children's mental health services: the role of school mental health. Psychiatr Res. Oct 2007;58(10):1330-1338. [ CrossRef ]
• Day L, Blades R, Spence C, Ronicle J. Mental health services and schools link pilots: evaluation brief. Department for Education, United Kingdom Government. Feb 2017. URL: https:/​/assets.​publishing.service.gov.uk/​media/​5a817dfaed915d74 e33fe812/​Evaluation_of_the_MH_services_and_schools_link_pilots-RB.​pdf [accessed 2024-03-25]
• Long MW, Albright G, McMillan J, Shockley KM, Price OA. Enhancing educator engagement in school mental health care through digital simulation professional development. J Sch Health. Sep 2018;88(9):651-659. [ CrossRef ] [ Medline ]
• Rothì DM, Leavey G, Best R. On the front-line: teachers as active observers of pupils’ mental health. Teach Teach Educ. Jul 2008;24(5):1217-1231. [ CrossRef ]
• Mazzer KR, Rickwood DJ. Teachers' role breadth and perceived efficacy in supporting student mental health. Adv Sch Ment Health Promot. 2015;8(1):29-41. [ CrossRef ]
• Shelemy L, Harvey K, Waite P. Supporting students’ mental health in schools: what do teachers want and need? Emot Behav Difficulties. 2019;24(1):100-116. [ CrossRef ]
• Fazel M, Soneson E, Sellars E, Butler G, Stein A. Partnerships at the interface of education and mental health services: the utilisation and acceptability of the provision of specialist liaison and teacher skills training. Int J Environ Res Public Health. Feb 24, 2023;20(5):4066. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Marshall L, Wishart R, Dunatchik A, Smith N. Supporting mental health in schools and colleges: quantitative survey. Department for Education, United Kingdom Government. Aug 2017. URL: https:/​/assets.​publishing.service.gov.uk/​media/​5a82e5bb40f0b6230269d46c/​Supporting_Mental-Health_survey_report.​pdf [accessed 2024-03-25]
• Sharpe H, Ford T, Lereya ST, Owen C, Viner RM, Wolpert M. Survey of schools' work with child and adolescent mental health across England: a system in need of support. Child Adolesc Ment Health. Sep 2016;21(3):148-153. [ CrossRef ] [ Medline ]
• Danby G, Hamilton P. Addressing the ‘elephant in the room’. The role of the primary school practitioner in supporting children’s mental well-being. Pastor Care Educ. 2016;34(2):90-103. [ CrossRef ]
• Reinke WM, Stormont M, Herman KC, Puri R, Goel N. Supporting children's mental health in schools: teacher perceptions of needs, roles, and barriers. Sch Psychol Q. 2011;26(1):1-13. [ CrossRef ]
• Kidger J, Gunnell D, Biddle L, Campbell R, Donovan J. Part and parcel of teaching? Secondary school staff's views on supporting student emotional health and well-being. Br Educ Res J. Jan 02, 2013;36(6):919-935. [ CrossRef ]
• Soneson E, Burn AM, Anderson JK, Humphrey A, Jones PB, Fazel M, et al. Determining stakeholder priorities and core components for school-based identification of mental health difficulties: a Delphi study. J Sch Psychol. Apr 2022;91:209-227. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Soneson E, Howarth E, Ford T, Humphrey A, Jones PB, Thompson Coon J, et al. Feasibility of school-based identification of children and adolescents experiencing, or at-risk of developing, mental health difficulties: a systematic review. Prev Sci. Jul 2020;21(5):581-603. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Childs-Fegredo J, Burn AM, Duschinsky R, Humphrey A, Ford T, Jones PB, et al. Acceptability and feasibility of early identification of mental health difficulties in primary schools: a qualitative exploration of UK school staff and parents’ perceptions. Sch Ment Health. Nov 18, 2020;13:143-159. [ CrossRef ]
• Anderson JK, Ford T, Soneson E, Coon JT, Humphrey A, Rogers M, et al. A systematic review of effectiveness and cost-effectiveness of school-based identification of children and young people at risk of, or currently experiencing mental health difficulties. Psychol Med. Sep 13, 2018;49(1):9-19. [ CrossRef ]
• Yamaguchi S, Foo JC, Nishida A, Ogawa S, Togo F, Sasaki T. Mental health literacy programs for school teachers: a systematic review and narrative synthesis. Early Interv Psychiatry. Feb 2020;14(1):14-25. [ CrossRef ] [ Medline ]
• Kognito homepage. Kognito. URL: https://kognito.com/ [accessed 2024-03-26]
• At-risk for elementary school educators. Kognito. URL: https://kognito.com/products/at-risk-for-elementary-schools [accessed 2024-03-25]
• Anderson M, Werner-Seidler A, King C, Gayed A, Harvey SB, O’Dea B. Mental health training programs for secondary school teachers: a systematic review. Sch Ment Health. Oct 3, 2018;11:489-508. [ CrossRef ]
• Ekornes S. Teacher perspectives on their role and the challenges of inter-professional collaboration in mental health promotion. Sch Ment Health. Mar 17, 2015;7:193-211. [ CrossRef ]
• Ohrt JH, Deaton JD, Linich K, Guest JD, Wymer B, Sandonato B. Teacher training in K–12 student mental health: a systematic review. Psychol Sch. Mar 03, 2020;57(5):833-846. [ CrossRef ]
• Parker BL, Anderson M, Batterham PJ, Gayed A, Subotic-Kerry M, Achilles MR, et al. Examining the preliminary effectiveness and acceptability of a web-based training program for Australian secondary school teachers: pilot study of the BEAM (building educators' skills in adolescent mental health) program. JMIR Ment Health. Oct 22, 2021;8(10):e29989. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McVey G, Gusella J, Tweed S, Ferrari M. A controlled evaluation of web-based training for teachers and public health practitioners on the prevention of eating disorders. Eat Disord. 2009;17(1):1-26. [ CrossRef ] [ Medline ]
• Pereira CA, Wen CL, Miguel EC, Polanczyk GV. A randomised controlled trial of a web-based educational program in child mental health for schoolteachers. Eur Child Adolesc Psychiatry. Aug 2015;24(8):931-940. [ CrossRef ] [ Medline ]
• Kognito At-Risk in PK-12. Oklahoma Department of Mental Health and Substance Abuse Services. URL: https://oklahoma.gov/odmhsas/trainings/training-institute/at-risk-pk-12.html [accessed 2024-03-14]
• Albright G, Fazel M, Khalid N, McMillan J, Hilty D, Shockley K, et al. High school educator training by simulation to address emotional and behavioral concerns in school settings: a randomized study. J Technol Behav Sci. 2022;7(3):277-289. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Greif Green J, Levine RS, Oblath R, Corriveau KH, Holt MK, Albright G. Pilot evaluation of preservice teacher training to improve preparedness and confidence to address student mental health. Evid Based Pract Child Adolesc Ment Health. 2020;5(1):42-52. [ CrossRef ]
• Domitrovich CE, Bradshaw CP, Poduska JM, Hoagwood K, Buckley JA, Olin S, et al. Maximizing the implementation quality of evidence-based preventive interventions in schools: a conceptual framework. Adv Sch Ment Health Promot. Jul 2008;1(3):6-28. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bonell C, Prost A, Melendez-Torres GJ, Davey C, Hargreaves JR. Will it work here? A realist approach to local decisions about implementing interventions evaluated as effective elsewhere. J Epidemiol Community Health. Jan 2021;75(1):46-50. [ CrossRef ] [ Medline ]
• Domitrovich CE, Greenberg MT. The study of implementation: current findings from effective programs that prevent mental disorders in school-aged children. J Educ Psychol Consult. 2000;11(2):193-221. [ CrossRef ]
• Evans RE, Craig P, Hoddinott P, Littlecott H, Moore L, Murphy S, et al. When and how do 'effective' interventions need to be adapted and/or re-evaluated in new contexts? The need for guidance. J Epidemiol Community Health. Jun 2019;73(6):481-482. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gardner F, Montgomery P, Knerr W. Transporting evidence-based parenting programs for child problem behavior (age 3-10) between countries: systematic review and meta-analysis. J Clin Child Adolesc Psychol. 2016;45(6):749-762. [ CrossRef ] [ Medline ]
• Leijten P, Melendez-Torres GJ, Knerr W, Gardner F. Transported versus homegrown parenting interventions for reducing disruptive child behavior: a multilevel meta-regression study. J Am Acad Child Adolesc Psychiatry. Jul 2016;55(7):610-617. [ CrossRef ] [ Medline ]
• Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new medical research Council guidance. BMJ. Sep 29, 2008;337:a1655. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Moore L, Hallingberg B, Wight D, Turley R, Segrott J, Craig P, et al. Exploratory studies to inform full-scale evaluations of complex public health interventions: the need for guidance. J Epidemiol Community Health. Oct 2018;72(10):865-866. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Eldridge SM, Lancaster GA, Campbell MJ, Thabane L, Hopewell S, Coleman CL, et al. Defining feasibility and pilot studies in preparation for randomised controlled trials: development of a conceptual framework. PLoS One. 2016;11(3):e0150205. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D, et al. How we design feasibility studies. Am J Prev Med. May 2009;36(5):452-457. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McLennan D, Noble S, Noble M, Plunkett E, Wright G, Gutacker N. The English indices of deprivation 2019: technical report. Ministry of Housing, Communities and Local Government, United Kingdom Government. Sep 2019. URL: https://dera.ioe.ac.uk/34259/1/IoD2019_Technical_Report.pdf [accessed 2024-03-25]
• Schools, pupils and their characteristics: January 2019. Department for Education, United Kingdom Government. Jun 2019. URL: https://www.gov.uk/government/statistics/schools-pupils-and-their-characteristics-january-2019 [accessed 2024-03-25]
• Special educational needs in England: January 2019. Department for Education, United Kingdom Government. Jul 4, 2019. URL: https://assets.publishing.service.gov.uk/media/5d1c91cce5274a08df3d35bd/SEN_2019_Text.docx.pdf [accessed 2024-03-25]
• Goodman R. The Strengths and Difficulties Questionnaire: a research note. J Child Psychol Psychiatry. Jul 07, 1997;38(5):581-586. [ CrossRef ] [ Medline ]
• Goodman R. The extended version of the Strengths and Difficulties Questionnaire as a guide to child psychiatric caseness and consequent burden. J Child Psychol Psychiatry. Jul 1999;40(5):791-799. [ CrossRef ]
• Goodman R, Ford T, Simmons H, Gatward R, Meltzer H. Using the Strengths and Difficulties Questionnaire (SDQ) to screen for child psychiatric disorders in a community sample. Br J Psychiatry. Dec 02, 2000;177(6):534-539. [ CrossRef ] [ Medline ]
• Goodman R, Meltzer H, Bailey V. The Strengths and Difficulties Questionnaire: a pilot study on the validity of the self-report version. Eur Child Adolesc Psychiatry. Sep 12, 1998;7(3):125-130. [ CrossRef ] [ Medline ]
• Stone LL, Otten R, Engels RC, Vermulst AA, Janssens JM. Psychometric properties of the parent and teacher versions of the Strengths and Difficulties Questionnaire for 4- to 12-year-olds: a review. Clin Child Fam Psychol Rev. Sep 2010;13(3):254-274. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Albright GL, Davidson J, Goldman R, Shockley KM, Timmons-Mitchell J. Development and validation of the Gatekeeper Behavior Scale. Crisis. Jul 2016;37(4):271-280. [ CrossRef ] [ Medline ]
• Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. Aug 07, 2009;4:50. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lyon AR, Bruns EJ. From evidence to impact: joining our best school mental health practices with our best implementation strategies. School Ment Health. Mar 1, 2019;11(1):106-114. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Schools, pupils and their characteristics: January 2020. Department for Education, United Kingdom Government. Jun 25, 2020. URL: https://www.gov.uk/government/statistics/schools-pupils-and-their-characteristics-january-2020 [accessed 2024-03-25]
• Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, et al. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. BMJ. Oct 24, 2016;355:i5239. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wilson DT, Walwyn RE, Brown J, Farrin AJ, Brown SR. Statistical challenges in assessing potential efficacy of complex interventions in pilot or feasibility studies. Stat Methods Med Res. Jun 2016;25(3):997-1009. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract. May 2004;10(2):307-312. [ CrossRef ] [ Medline ]
• Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. Jan 06, 2010;10:1. [ FREE Full text ] [ CrossRef ] [ Medline ]
• R: a language and environment for statistical computing. The Comprehensive R Archive Network. Feb 10, 2015. URL: https://www.gbif.org/tool/81287/r-a-language-and-environment-for-statistical-computing [accessed 2024-03-25]
• Wickham H. Ggplot2: Elegant Graphics for Data Analysis. New York, NY. Springer; 2009.
• Bryer J, Speerschneider K. Analysis and visualization of Likert based items. GitHub. URL: http://github.com/jbryer/likert
• YouthinMind homepage. YouthinMind. URL: https://youthinmind.com/ [accessed 2024-03-26]
• Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. Nov 2005;15(9):1277-1288. [ CrossRef ] [ Medline ]
• Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. Sep 18, 2013;13:117. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77-101. [ CrossRef ]
• Braun V, Clarke V. Thematic Analysis: A Practical Guide. Thousand Oaks, CA. SAGE Publications; 2021.
• School workforce in England. United Kingdom Government. Jun 8, 2023. URL: https://explore-education-statistics.service .gov.uk/find-statistics/school-workforce-in-england [accessed 2024-03-14]
• Jorm AF, Kitchener BA, Sawyer MG, Scales H, Cvetkovski S. Mental health first aid training for high school teachers: a cluster randomized trial. BMC Psychiatry. Jun 24, 2010;10:51. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Moor S, Maguire A, McQueen H, Wells EJ, Elton R, Wrate R, et al. Improving the recognition of depression in adolescence: can we teach the teachers? J Adolesc. Feb 2007;30(1):81-95. [ CrossRef ] [ Medline ]
• Kidger J, Stone T, Tilling K, Brockman R, Campbell R, Ford T, et al. A pilot cluster randomised controlled trial of a support and training intervention to improve the mental health of secondary school teachers and students - the WISE (Wellbeing in Secondary Education) study. BMC Public Health. Oct 06, 2016;16(1):1060. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vieira MA, Gadelha AA, Moriyama TS, Bressan RA, Bordin IA. Evaluating the effectiveness of a training program that builds teachers' capability to identify and appropriately refer middle and high school students with mental health problems in Brazil: an exploratory study. BMC Public Health. Feb 28, 2014;14:210. [ FREE Full text ] [ CrossRef ] [ Medline ]
• McLaughlin RJ, Holcomb JD, Jibaja-Rusth ML, Webb J. Teacher ratings of student risk for substance use as a function of specialized training. J Drug Educ. 1993;23(1):83-95. [ CrossRef ]
• Soneson E, Childs-Fegredo J, Anderson JK, Stochl J, Fazel M, Ford T, et al. Acceptability of screening for mental health difficulties in primary schools: a survey of UK parents. BMC Public Health. Dec 22, 2018;18(1):1404. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Levitt JM, Saka N, Hunter Romanelli L, Hoagwood K. Early identification of mental health problems in schools: the status of instrumentation. J Sch Psychol. Apr 2007;45(2):163-191. [ CrossRef ]
• Loades ME, Mastroyannopoulou K. Teachers' recognition of children's mental health problems. Child Adolesc Ment Health. Sep 2010;15(3):150-156. [ CrossRef ] [ Medline ]
• Anderson JK, Howarth E, Vainre M, Jones PB, Humphrey A. A scoping literature review of service-level barriers for access and engagement with mental health services for children and young people. Child Youth Serv Rev. Jun 2017;77:164-176. [ CrossRef ]
• Moon J, Williford A, Mendenhall A. Educators' perceptions of youth mental health: implications for training and the promotion of mental health services in schools. Child Youth Serv Rev. Feb 2017;73:384-391. [ CrossRef ]
• Graham A, Phelps R, Maddison C, Fitzgerald R. Supporting children’s mental health in schools: teacher views. Teach Teach. 2011;17(4):479-496. [ CrossRef ]
• Atkins MS, Graczyk PA, Frazier SL, Abdul-Adil J. Toward a new model for promoting urban children's mental health: accessible, effective, and sustainable school-based mental health services. Sch Psychol Rev. 2003;32(4):503-514. [ CrossRef ]
• Moore A, Stapley E, Hayes D, Town R, Deighton J. Barriers and facilitators to sustaining school-based mental health and wellbeing interventions: a systematic review. Int J Environ Res Public Health. Mar 17, 2022;19(6):3587. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gee B, Wilson J, Clarke T, Farthing S, Carroll B, Jackson C, et al. Review: delivering mental health support within schools and colleges - a thematic synthesis of barriers and facilitators to implementation of indicated psychological interventions for adolescents. Child Adolesc Ment Health. Feb 2021;26(1):34-46. [ CrossRef ] [ Medline ]
• Fält E, Wallby T, Sarkadi A, Salari R, Fabian H. Agreement between mothers', fathers', and teachers' ratings of behavioural and emotional problems in 3-5-year-old children. PLoS One. Nov 01, 2018;13(11):e0206752. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bear HA, Ayala Nunes L, DeJesus J, Liverpool S, Moltrecht B, Neelakantan L, et al. Determination of markers of successful implementation of mental health apps for young people: systematic review. J Med Internet Res. Nov 09, 2022;24(11):e40347. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Durbin K, Harmon S. We are on the same team: child psychiatry and the school system. JAACAP Connect. 2020;7(1):32-36. [ CrossRef ]

Abbreviations

Edited by T Leung; submitted 24.02.23; peer-reviewed by E Widnall, B Fernandes, J Burns, K Cohen; comments to author 27.08.23; accepted 01.03.24; published 23.04.24.

©Emma Soneson, Emma Howarth, Alison Weir, Peter B Jones, Mina Fazel. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 23.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.