mixed methods research articles

Mixed methods research: what it is and what it could be

• Open access
• Published: 29 March 2019
• Volume 48 , pages 193–216, ( 2019 )

Cite this article

You have full access to this open access article

• Rob Timans 1 ,
• Paul Wouters 2 &
• Johan Heilbron 3  

109k Accesses

88 Citations

13 Altmetric

Explore all metrics

A Correction to this article was published on 06 May 2019

This article has been updated

Combining methods in social scientific research has recently gained momentum through a research strand called Mixed Methods Research (MMR). This approach, which explicitly aims to offer a framework for combining methods, has rapidly spread through the social and behavioural sciences, and this article offers an analysis of the approach from a field theoretical perspective. After a brief outline of the MMR program, we ask how its recent rise can be understood. We then delve deeper into some of the specific elements that constitute the MMR approach, and we engage critically with the assumptions that underlay this particular conception of using multiple methods. We conclude by offering an alternative view regarding methods and method use.

Similar content being viewed by others

What is Qualitative in Qualitative Research

Patrik Aspers & Ugo Corte

Criteria for Good Qualitative Research: A Comprehensive Review

Drishti Yadav

Different uses of Bronfenbrenner’s ecological theory in public mental health research: what is their value for guiding public mental health policy and practice?

Malin Eriksson, Mehdi Ghazinour & Anne Hammarström

Avoid common mistakes on your manuscript.

The interest in combining methods in social scientific research has a long history. Terms such as “triangulation,” “combining methods,” and “multiple methods” have been around for quite a while to designate using different methods of data analysis in empirical studies. However, this practice has gained new momentum through a research strand that has recently emerged and that explicitly aims to offer a framework for combining methods. This approach, which goes by the name of Mixed Methods Research (MMR), has rapidly become popular in the social and behavioural sciences. This can be seen, for instance, in Fig.  1 , where the number of publications mentioning “mixed methods” in the title or abstract in the Thomson Reuters Web of Science is depicted. The number increased rapidly over the past ten years, especially after 2006. Footnote 1

Fraction of the total of articles mentioning Mixed Method Research appearing in a given year, 1990–2017 (yearly values sum to 1). See footnote 1

The subject of mixed methods thus seems to have gained recognition among social scientists. The rapid rise of the number of articles mentioning the term raises various sociological questions. In this article, we address three of these questions. The first question concerns the degree to which the approach of MMR has become institutionalized within the field of the social sciences. Has MMR become a recognizable realm of knowledge production? Has its ascendance been accompanied by the production of textbooks, the founding of journals, and other indicators of institutionalization? The answer to this question provides an assessment of the current state of MMR. Once that is determined, the second question is how MMR’s rise can be understood. Where does the approach come from and how can its emergence and spread be understood? To answer this question, we use Pierre Bourdieu’s field analytical approach to science and academic institutions (Bourdieu 1975 , 1988 , 2004 , 2007 ; Bourdieu et al. 1991 ). We flesh out this approach in the next section. The third question concerns the substance of the MMR corpus seen in the light of the answers to the previous questions: how can we interpret the specific content of this approach in the context of its socio-historical genesis and institutionalization, and how can we understand its proposal for “mixing methods” in practice?

We proceed as follows. In the next section, we give an account of our theoretical approach. Then, in the third, we assess the degree of institutionalization of MMR, drawing on the indicators of academic institutionalization developed by Fleck et al. ( 2016 ). In the fourth section, we address the second question by examining the position of the academic entrepreneurs behind the rise of MMR. The aim is to understand these agents’ engagement in MMR, as well as its distinctive content as being informed by their position in this field. Viewing MMR as a position-taking of academic entrepreneurs, linked to their objective position in this field, allows us to reflect sociologically on the substance of the approach. We offer this reflection in the fifth section, where we indicate some problems with MMR. To get ahead of the discussion, these problems have to do with the framing of MMR as a distinct methodology and its specific conceptualization of data and methods of data analysis. We argue that these problems hinder fruitfully combining methods in a practical understanding of social scientific research. Finally, we conclude with some tentative proposals for an alternative view on combining methods.

A field approach

Our investigation of the rise and institutionalization of MMR relies on Bourdieu’s field approach. In general, field theory provides a model for the structural dimensions of practices. In fields, agents occupy a position relative to each other based on the differences in the volume and structure of their capital holdings. Capital can be seen as a resource that agents employ to exert power in the field. The distribution of the form of capital that is specific to the field serves as a principle of hierarchization in the field, differentiating those that hold more capital from those that hold less. This principle allows us to make a distinction between, respectively, the dominant and dominated factions in a field. However, in mature fields all agents—dominant and dominated—share an understanding of what is at stake in the field and tend to accept its principle of hierarchization. They are invested in the game, have an interest in it, and share the field’s illusio .

In the present case, we can interpret the various disciplines in the social sciences as more or less autonomous spaces that revolve around the shared stake in producing legitimate scientific knowledge by the standards of the field. What constitutes legitimate knowledge in these disciplinary fields, the production of which bestows scholars with prestige and an aura of competence, is in large part determined by the dominant agents in the field, who occupy positions in which most of the consecration of scientific work takes place. Scholars operating in a field are endowed with initial and accumulated field-specific capital, and are engaged in the struggle to gain additional capital (mainly scientific and intellectual prestige) in order to advance their position in the field. The main focus of these agents will generally be the disciplinary field in which they built their careers and invested their capital. These various disciplinary spaces are in turn part of a broader field of the social sciences in which the social status and prestige of the various disciplines is at stake. The ensuing disciplinary hierarchy is an important factor to take into account when analysing the circulation of new scientific products such as MMR. Furthermore, a distinction needs to be made between the academic and the scientific field. While the academic field revolves around universities and other degree-granting institutions, the stakes in the scientific field entail the production and valuation of knowledge. Of course, in modern science these fields are closely related, but they do not coincide (Gingras and Gemme 2006 ). For instance, part of the production of legitimate knowledge takes place outside of universities.

This framework makes it possible to contextualize the emergence of MMR in a socio-historical way. It also enables an assessment of some of the characteristics of MMR as a scientific product, since Bourdieu insists on the homology between the objective positions in a field and the position-takings of the agents who occupy these positions. As a new methodological approach, MMR is the result of the position-takings of its producers. The position-takings of the entrepreneurs at the core of MMR can therefore be seen as expressions in the struggles over the authority to define the proper methodology that underlies good scientific work regarding combining methods, and the potential rewards that come with being seen, by other agents, as authoritative on these matters. Possible rewards include a strengthened autonomy of the subfield of MMR and an improved position in the social-scientific field.

The role of these entrepreneurs or ‘intellectual leaders’ who can channel intellectual energy and can take the lead in institution building has been emphasised by sociologists of science as an important aspect of the production of knowledge that is visible and recognized as distinct in the larger scientific field (e.g., Mullins 1973 ; Collins 1998 ). According to Bourdieu, their position can, to a certain degree, explain the strategy they pursue and the options they perceive to be viable in the trade-off regarding the risks and potential rewards for their work.

We do not provide a full-fledged field analysis of MMR here. Rather, we use the concept as a heuristic device to account for the phenomenon of MMR in the social context in which it emerged and diffused. But first, we take stock of the current situation of MMR by focusing on the degree of institutionalization of MMR in the scientific field.

The institutionalization of mixed methods research

When discussing institutionalization, we have to be careful about what we mean by this term. More precisely, we need to be specific about the context and distinguish between institutionalization in the academic field and institutionalization within the scientific field (see Gingras and Gemme 2006 ; Sapiro et al. 2018 ). The first process refers to the establishment of degrees, curricula, faculties, etc., or to institutions tied to the academic bureaucracy and academic politics. The latter refers to the emergence of institutions that support the autonomization of scholarship such as scholarly associations and scientific journals. Since MMR is still a relatively young phenomenon and academic institutionalization tends to lag scientific institutionalization (e.g., for the case of sociology and psychology, see Sapiro et al. 2018 , p. 26), we mainly focus here on the latter dimension.

Drawing on criteria proposed by Fleck et al. ( 2016 ) for the institutionalization of academic disciplines, MMR seems to have achieved a significant degree of institutionalization within the scientific field. MMR quickly gained popularity in the first decade of the twenty-first century (e.g., Tashakkori and Teddlie 2010c , pp. 803–804). A distinct corpus of publications has been produced that aims to educate those interested in MMR and to function as a source of reference for researchers: there are a number of textbooks (e.g., Plowright 2010 ; Creswell and Plano Clark 2011 ; Teddlie and Tashakkori 2008 ); a handbook that is now in its second edition (Tashakkori and Teddlie 2003 , 2010a ); as well as a reader (Plano Clark and Creswell 2007 ). Furthermore, a journal (the Journal of Mixed Methods Research [ JMMR] ) was established in 2007. The JMMR was founded by the editors John Creswell and Abbas Tashakkori with the primary aim of “building an international and multidisciplinary community of mixed methods researchers.” Footnote 2 Contributions to the journal must “fit the definition of mixed methods research” Footnote 3 and explicitly integrate qualitative and quantitative aspects of research, either in an empirical study or in a more theoretical-methodologically oriented piece.

In addition, general textbooks on social research methods and methodology now increasingly devote sections to the issue of combining methods (e.g., Creswell 2008 ; Nagy Hesse-Biber and Leavy 2008 ; Bryman 2012 ), and MMR has been described as a “third paradigm” (Denscombe 2008 ), a “movement” (Bryman 2009 ), a “third methodology” (Tashakkori and Teddlie 2010b ), a “distinct approach” (Greene 2008 ) and an “emerging field” (Tashakkori and Teddlie 2011 ), defined by a common name (that sets it apart from other approaches to combining methods) and shared terminology (Tashakkori and Teddlie 2010b , p. 19). As a further indication of institutionalization, a research association (the Mixed Methods International Research Association—MMIRA) was founded in 2013 and its inaugural conference was held in 2014. Prior to this, there have been a number of conferences on MMR or occasions on which MMR was presented and discussed in other contexts. An example of the first is the conference on mixed method research design held in Basel in 2005. Starting also in 2005, the British Homerton School of Health Studies has organised a series of international conferences on mixed methods. Moreover, MMR was on the list of sessions in a number of conferences on qualitative research (see, e.g., Creswell 2012 ).

Another sign of institutionalization can be found in efforts to forge a common disciplinary identity by providing a narrative about its history. This involves the identification of precursors and pioneers as well as an interpretation of the process that gave rise to a distinctive set of ideas and practices. An explicit attempt to chart the early history of MMR is provided by Johnson and Gray ( 2010 ). They frame MMR as rooted in the philosophy of science, particularly as a way of thinking about science that has transcended some of the most salient historical oppositions in philosophy. Philosophers like Aristotle and Kant are portrayed as thinkers who sought to integrate opposing stances, forwarding “proto-mixed methods ideas” that exhibited the spirit of MMR (Johnson and Gray 2010 , p. 72, p. 86). In this capacity, they (as well as other philosophers like Vico and Montesquieu) are presented as part of MMR providing a philosophical validation of the project by presenting it as a continuation of ideas that have already been voiced by great thinkers in the past.

In the second edition of their textbook, Creswell and Plano Clark ( 2011 ) provide an overview of the history of MMR by identifying five historical stages: the first one being a precursor to the MMR approach, consisting of rather atomised attempts by different authors to combine methods in their research. For Creswell and Plano Clark, one of the earliest examples is Campbell and Fiske’s ( 1959 ) combination of quantitative methods to improve the validity of psychological scales that gave rise to the triangulation approach to research. However, they regard this and other studies that combined methods around that time, as “antecedents to (…) more systematic attempts to forge mixed methods into a complete research design” (Creswell and Plano Clark 2011 , p. 21), and hence label this stage as the “formative period” (ibid., p. 25). Their second stage consists of the emergence of MMR as an identifiable research strand, accompanied by a “paradigm debate” about the possibility of combining qualitative and quantitative data. They locate its beginnings in the late 1980s when researchers in various fields began to combine qualitative and quantitative methods (ibid., pp. 20–21). This provoked a discussion about the feasibility of combining data that were viewed as coming from very different philosophical points of view. The third stage, the “procedural development period,” saw an emphasis on developing more hands-on procedures for designing a mixed methods study, while stage four is identified as consisting of “advocacy and expansion” of MMR as a separate methodology, involving conferences, the establishment of a journal and the first edition of the aforementioned handbook (Tashakkori and Teddlie 2003 ). Finally, the fifth stage is seen as a “reflective period,” in which discussions about the unique philosophical underpinnings and the scientific position of MMR emerge.

Creswell and Plano Clark thus locate the emergence of “MMR proper” at the second stage, when researchers started to use both qualitative and quantitative methods within a single research effort. As reasons for the emergence of MMR at this stage they identify the growing complexity of research problems, the perception of qualitative research as a legitimate form of inquiry (also by quantitative researchers) and the increasing need qualitative researchers felt for generalising their findings. They therefore perceive the emergence of the practice of combining methods as a bottom up process that grew out of research practices, and at some point in time converged towards a more structural approach. Footnote 4 Historical accounts such as these add a cognitive dimension to the efforts to institutionalize MMR. They lay the groundwork for MMR as a separate subfield with its own identity, topics, problems and intellectual history. The use of terms such as “third paradigm” and “third methodology” also suggests that there is a tendency to perceive and promote MMR as a distinct and coherent way to do research.

In view of the brief exploration of the indicators of institutionalisation of MMR, it seems reasonable to conclude that MMR has become a recognizable and fairly institutionalized strand of research with its own identity and profile within the social scientific field. This can be seen both from the establishment of formal institutions (like associations and journals) and more informal ones that rely more on the tacit agreement between agents about “what MMR is” (an example of this, which we address later in the article, is the search for a common definition of MMR in order to fix the meaning of the term). The establishment of these institutions supports the autonomization of MMR and its emancipation from the field in which it originated, but in which it continues to be embedded. This way, it can be viewed as a semi-autonomous subfield within the larger field of the social sciences and as the result of a differentiation internal to this field (Steinmetz 2016 , p. 109). It is a space that is clearly embedded within this higher level field; for example, members of the subfield of MMR also qualify as members of the overarching field, and the allocation of the most valuable and current form of capital is determined there as well. Nevertheless, as a distinct subfield, it also has specific principles that govern the production of knowledge and the rewards of domination.

We return to the content and form of this specific knowledge later in the article. The next section addresses the question of the socio-genesis of MMR.

Where does mixed methods research come from?

The origins of the subfield of MMR lay in the broader field of social scientific disciplines. We interpret the positions of the scholars most involved in MMR (the “pioneers” or “scientific entrepreneurs”) as occupying particular positions within the larger academic and scientific field. Who, then, are the researchers at the heart of MMR? Leech ( 2010 ) interviewed 4 scholars (out of 6) that she identified as early developers of the field: Alan Bryman (UK; sociology), John Creswell (USA; educational psychology), Jennifer Greene (USA; educational psychology) and Janice Morse (USA; nursing and anthropology). Educated in the 1970s and early 1980s, all four of them indicated that they were initially trained in “quantitative methods” and later acquired skills in “qualitative methods.” For two of them (Bryman and Creswell) the impetus to learn qualitative methods was their involvement in writing on, and teaching of, research methods; for Greene and Morse the initial motivation was more instrumental and related to their concrete research activity at the time. Creswell describes himself as “a postpositivist in the 1970s, self-education as a constructivist through teaching qualitative courses in the 1980s, and advocacy for mixed methods (…) from the 1990s to the present” (Creswell 2011 , p. 269). Of this group, only Morse had the benefit of learning about qualitative methods as part of her educational training (in nursing and anthropology; Leech 2010 , p. 267). Independently, Creswell ( 2012 ) identified (in addition to Bryman, Greene and Morse) John Hunter, Allen Brewer (USA; Northwestern and Boston College) and Nigel Fielding (University of Surrey, UK) as important early movers in MMR.

The selections that Leech and Creswell make regarding the key actors are based on their close involvement with the “MMR movement.” It is corroborated by a simple analysis of the articles that appeared in the Journal of Mixed Methods Research ( JMMR ), founded in 2007 as an outlet for MMR.

Table 1 lists all the authors that have published in the issues of the journal since its first publication in 2007 and that have either received more than 14 (4%) of the citations allocated between the group of 343 authors (the TLCS score in Table 1 ), or have written more than 2 articles for the Journal (1.2% of all the articles that have appeared from 2007 until October 2013) together with their educational background (i.e., the discipline in which they completed their PhD).

All the members of Leech’s selection, except for Morse, and the members of Creswell’s selection (except Hunter, Brewer, and Fielding) are represented in the selection based on the entries in the JMMR . Footnote 5 The same holds for two of the three additional authors identified by Creswell. Hunter and Brewer have developed a somewhat different approach to combining methods that explicitly targets data gathering techniques and largely avoids epistemological discussions. In Brewer and Hunter ( 2006 ) they discuss the MMR approach very briefly and only include two references in their bibliography to the handbook of Tashakkori and Teddlie ( 2003 ), and at the end of 2013 they had not published in the JMMR . Fielding, meanwhile, has written two articles for the JMMR (Fielding and Cisneros-Puebla 2009 ; Fielding 2012 ). In general, it seems reasonable to assume that a publication in a journal that positions itself as part of a systematic attempt to build a research tradition, and can be viewed as part of a strategic effort to advance MMR as a distinct alternative to more “traditional” academic research—particularly in methods—at least signals a degree of adherence to the effort and acceptance of the rules of the game it lays out. This would locate Fielding closer to the MMR movement than the others.

The majority of the researchers listed in Table 1 have a background in psychology or social psychology (35%), and sociology (25%). Most of them work in the United States or are UK citizens, and the positions they occupied at the beginning of 2013 indicates that most of these are in applied research: educational research and educational psychology account for 50% of all the disciplinary occupations of the group that were still employed in academia. This is consistent with the view that MMR originated in applied disciplines and thematic studies like education and nursing, rather than “pure disciplines” like psychology and sociology (Tashakkori and Teddlie ( 2010b ), p. 32). Although most of the 20 individuals mentioned in Table 1 have taught methods courses in academic curricula (for 15 of them, we could determine that they were involved in the teaching of qualitative, quantitative, or mixed methods), there are few individuals with a background in statistics or a neighbouring discipline: only Amy Dellinger did her PhD in “research methodology.” In addition, as far as we could determine, only three individuals held a position in a methodological department at some time: Dellinger, Tony Onwuegbuzie, and Nancy Leech.

The pre-eminence of applied fields in MMR is supported when we turn our attention to the circulation of MMR. To assess this we proceeded as follows. We selected 10 categories in the Web of Science that form a rough representation of the space of social science disciplines, taking care to include the most important so-called “studies.” These thematically orientated, interdisciplinary research areas have progressively expanded since they emerged at the end of the 1960s as a critique of the traditional disciplines (Heilbron et al. 2017 ). For each category, we selected the 10 journals with the highest 5-year impact factor in their category in the period 2007–2015. The lists were compiled bi-annually over this period, resulting in 5 top ten lists for the following Web of Science categories: Economics, Psychology, Sociology, Anthropology, Political Science, Nursing, Education & Educational Research, Business, Cultural Studies, and Family Studies. After removing multiple occurring journals, we obtained a list of 164 journals.

We searched the titles and abstracts of the articles appearing in these journals over the period 1992–2016 for occurrences of the terms “mixed method” or “multiple methods” and variants thereof. We chose this particular period and combination of search terms to see if a shift from a more general use of the term “multiple methods” to “mixed methods” occurred following the institutionalization of MMR. In total, we found 797 articles (out of a total of 241,521 articles that appeared in these journals during that time), published in 95 different journals. Table 2 lists the 20 journals that contain at least 1% (8 articles) of the total amount of articles.

As is clear from Table 2 , the largest number of articles in the sample were published in journals in the field of nursing: 332 articles (42%) appeared in journals that can be assigned to this category. The next largest category is Education & Educational Research, to which 224 (28 percentage) of the articles can be allocated. By contrast, classical social science disciples are barely represented. In Table 2 only the journal Field Methods (Anthropology) and the Journal of Child Psychology and Psychiatry (Psychology) are related to classical disciplines. In Table 3 , the articles in the sample are categorized according to the disciplinary category of the journal in which they appeared. Overall, the traditional disciplines are clearly underrepresented: for the Economics category, for example, only the Journal of Economic Geography contains three articles that make a reference to mixed methods.

Focusing on the core MMR group, the top ten authors of the group together collect 458 citations from the 797 articles in the sample, locating them at the center of the citation network. Creswell is the most cited author (210 citations) and his work too receives most citations from journals in nursing and education studies.

The question whether a terminological shift has occurred from “multiple methods” to “mixed methods” must be answered affirmative for this sample. Prior to 2001 most articles (23 out of 31) refer to “multiple methods” or “multi-method” in their title or abstract, while the term “mixed methods” gains traction after 2001. This shift occurs first in journals in nursing studies, with journals in education studies following somewhat later. The same fields are also the first to cite the first textbooks and handbooks of MMR.

Taken together, these results corroborate the notion that MMR circulates mainly in nursing and education studies. How can this be understood from a field theoretical perspective? MMR can be seen as an innovation in the social scientific field, introducing a new methodology for combining existing methods in research. In general, innovation is a relatively risky strategy. Coming up with a truly rule-breaking innovation often involves a small probability of great success and a large probability of failure. However, it is important to add some nuance to this general observation. First, the risk an innovator faces depends on her position in the field. Agents occupying positions at the top of their field’s hierarchy are rich in specific capital and can more easily afford to undertake risky projects. In the scientific field, these are the agents richest in scientific capital. They have the knowledge, authority, and reputation (derived from recognition by their peers; Bourdieu 2004 , p. 34) that tends to decrease the risk they face and increase the chances of success. Moreover, the positions richest in scientific capital will, by definition, be the most consecrated ones. This consecration involves scientific rather than academic capital (cf. Wacquant 2013 , p. 20) and within disciplines these consecrated positions often are related to orthodox position-takings. This presents a paradox: although they have the capital to take more risks, they have also invested heavily in the orthodoxy of the field and will thus be reluctant to upset the status quo and risk destroying the value of their investment. This results in a tendency to take a more conservative stance, aimed at preserving the status quo in the field and defending their position. Footnote 6

For agents in dominated positions this logic is reversed. Possessing less scientific capital, they hold less consecrated positions and their chances of introducing successful innovations are much lower. This leaves them too with two possible strategies. One is to revert to a strategy of adaptation, accepting the established hierarchy in the field and embarking on a slow advancement to gain the necessary capital to make their mark from within the established order. However, Bourdieu notes that sometimes agents with a relatively marginal position in the field will engage in a “flight forward” and pursue higher risk strategies. Strategies promoting a heterodox approach challenge the orthodoxy and the principles of hierarchization of the field, and, if successful (which will be the case only with a small probability), can rake in significant profits by laying claim to a new orthodoxy (Bourdieu 1975 , p. 104; Bourdieu 1993 , pp. 116–117).

Thus, the coupling of innovative strategies to specific field positions based on the amount of scientific capital alone is not straightforward. It is therefore helpful to introduce a second differentiation in the field that, following Bourdieu ( 1975 , p. 103), is based on the differences between the expected profits from these strategies. Here a distinction can be made between an autonomous and a heteronomous pole of the field, i.e., between the purest, most “disinterested” positions and the most “temporal” positions that are more pervious to the heteronomous logic of social hierarchies outside the scientific field. Of course, this difference is a matter of degree, as even the works produced at the most heteronomous positions still have to adhere to the standards of the scientific field to be seen as legitimate. But within each discipline this dimension captures the difference between agents predominantly engaged in fundamental, scholarly work—“production solely for the producers”—and agents more involved in applied lines of research. The main component of the expected profit from innovation in the first case is scientific, whereas in the second case the balance tends to shift towards more temporal profits. This two-fold structuring of the field allows for a more nuanced conception of innovation than the dichotomy “conservative” versus “radical.” Holders of large amounts of scientific capital at the autonomous pole of the field are the producers and conservators of orthodoxy, producing and diffusing what can be called “orthodox innovations” through their control of relatively powerful networks of consecration and circulation. Innovations can be radical or revolutionary in a rational sense, but they tend to originate from questions raised by the orthodoxy of the field. Likewise, the strategy to innovate in this sense can be very risky in that success is in no way guaranteed, but the risk is mitigated by the assurance of peers that these are legitimate questions, tackled in a way that is consistent with orthodoxy and that does not threaten control of the consecration and circulation networks.

These producers are seen as intellectual leaders by most agents in the field, especially by those aspiring to become part of the specific networks of production and circulation they maintain. The exception are the agents located at the autonomous end of the field who possess less scientific capital and outright reject this orthodoxy produced by the field’s elite. Being strictly focused on the most autonomous principles of legitimacy, they are unable to accommodate and have no choice but to reject the orthodoxy. Their only hope is to engage in heterodox innovations that may one day become the new orthodoxy.

The issue is less antagonistic at the heteronomous side of the field, at least as far as the irreconcilable position-takings at the autonomous pole are concerned. The main battle here is also for scientific capital, but is complemented by the legitimacy it brings to gain access to those who are in power outside of the scientific field. At the dominant side, those with more scientific capital tend to have access to the field of power, agents who hold the most economic and cultural capital, for example by holding positions in policy advisory committees or company boards. The dominated groups at this side of the field will cater more to practitioners or professionals outside of the field of science.

Overall, there will be fewer innovations on this side. Moreover, innovative strategies will be less concerned with the intricacies of the pure discussions that prevail at the autonomous pole and be of a more practical nature, but pursued from different degrees of legitimacy according to the differences in scientific capital. This affects the form these more practical, process-orientated innovations take. At the dominant side of this pole, agents tend to accept the outcome of the struggles at the autonomous pole: they will accept the orthodoxy because mastery of this provides them with scientific capital and the legitimacy they need to gain access to those in power. In contrast, agents at the dominated side will be more interested in doing “what works,” neutralizing the points of conflict at the autonomous pole and deriving less value from strictly following the orthodoxy. This way, a four-fold classification of innovative strategies in the scientific field emerges (see Fig.  2 ) that helps to understand the context in which MMR was developed.

Scientific field and scientific innovation

In summary, the small group of researchers who have been identified as the core of MMR consist predominantly of users of methods, who were educated and have worked exclusively at US and British universities. The specific approach to combining methods that is proposed by MMR has been successful from an institutional point of view, achieving visibility through the foundation of a journal and association and a considerable output of core MMR scholars in terms of books, conference proceedings, and journal articles. Its origins and circulation in vocational studies rather than classical academic disciplines can be understood from the position these studies occupy in the scientific field and the kinds of position-taking and innovations these positions give rise to. This context allows a reflexive understanding of the content of MMR and the issues that are dominant in the approach. We turn to this in the next section.

Mixed methods research: Position-taking

The position of the subfield of MMR in the scientific field is related to the position-takings of agents that form the core of this subfield (Bourdieu 1993 , p. 35). The space of position takings, in turn, provides the framework to study the most salient issues that are debated within the subfield. Since we can consider MMR to be an emerging subfield, where positions and position takings are not as clearly defined as in more mature and settled fields, it comes as no surprise that there is a lively discussion of fundamental matters. Out of the various topics that are actively discussed, we have distilled three themes that are important for the way the subfield of MMR conveys its autonomy as a field and as a distinct approach to research. Footnote 7 In our view, these also represent the main problems with the way MMR approaches the issue of combining methods.

Methodology making and standardization

The first topic is that the approach is moving towards defining a unified MMR methodology. There are differences in opinion as to how this is best achieved, but there is widespread agreement that some kind of common methodological and conceptual foundation of MMR is needed. To this end, some propose a broad methodology that can serve as distinct marker of MMR research. For instance, in their introduction to the handbook, Tashakkori and Teddlie ( 2010b ) propose a definition of the methodology of mixed methods research as “the broad inquiry logic that guides the selection of specific methods and that is informed by conceptual positions common to mixed methods practitioners” (Tashakkori and Teddlie 2010b , p. 5). When they (later on in the text) provide two methodological principles that differentiate MMR from other communities of scholars, they state that they regard it as a “crucial mission” for the MMR community to generate distinct methodological principles (Tashakkori and Teddlie 2010b , pp. 16–17). They envision an MMR methodology that can function as a “guide” for selecting specific methods. Others are more in favour of finding a philosophical foundation that underlies MMR. For instance, Morgan ( 2007 ) and Hesse-Biber ( 2010 ) consider pragmatism as a philosophy that distinguishes MMR from qualitative (constructivism) and quantitative (positivist) research and that can provide a rationale for the paradigmatic pluralism typical of MMR.

Furthermore, there is wide agreement that some unified definition of MMR would be beneficial, but it is precisely here that there is a large variation in interpretations regarding the essentials of MMR. This can be seen in the plethora of definitions that have been proposed. Johnson et al. ( 2007 ) identified 19 alternative definitions of MMR at the time, out of which they condensed their own:

[MMR] is the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purpose of breath and depth of understanding and corroboration. Footnote 8

Four years later, the issue is not settled yet. Creswell and Plano Clark ( 2011 ) list a number of authors who have proposed a different definition of MMR, and conclude that there is a common trend in the content of these definitions over time. They take the view that earlier texts on mixing methods stressed a “disentanglement of methods and philosophy,” while later texts locate the practice of mixing methods in “all phases of the research process” (Creswell and Plano Clark 2011 , p. 2). It would seem, then, that according to these authors the definitions of MMR have become more abstract, further away from the practicality of “merely” combining methods. Specifically, researchers now seem to speak of mixing higher order concepts: some speak of mixing methodologies, others refer to mixing “research approaches,” or combining “types of research,” or engage in “multiple ways of seeing the social world” (Creswell and Plano Clark 2011 ).

This shift is in line with the direction in which MMR has developed and that emphasises practical ‘manuals’ and schemas for conducting research. A relatively large portion of the MMR literature is devoted to classifications of mixed methods designs. These classifications provide the basis for typologies that, in turn, provide guidelines to conduct MMR in a concrete research project. Tashakkori and Teddlie ( 2003 ) view these typologies as important elements of the organizational structure and legitimacy of the field. In addition, Leech and Onwuegbuzie ( 2009 ) see typologies as helpful guides for researchers and of pedagogical value (Leech and Onwuegbuzie 2009 , p. 272). Proposals for typologies can be found in textbooks, articles, and contributions to the handbook(s). For example, Creswell et al. ( 2003 , pp. 169-170) reviewed a number of studies and identified 8 different ways to classify MMR studies. This list was updated and extended by Creswell and Plano Clark ( 2011 , pp. 56-59) to 15 typologies. Leech and Onwuegbuzie ( 2009 ) identified 35 different research designs in the contributions to Teddlie and Tashakkori (2003) alone, and proposed their own three-dimensional typology that resulted in 8 different types of mixed methods studies. As another example of the ubiquity of these typologies, Nastasi et al. ( 2010 ) classified a large number of existing typologies in MMR into 7”meta-typologies” that each emphasize different aspects of the research process as important markers for MMR. According to the authors, these typologies have the same function in MMR as the more familiar names of “qualitative” or “quantitative” methods (e.g., “content analysis” or “structural equation modelling”) have: to signal readers of research what is going on, what procedures have been followed, how to interpret results, etc. (see also Creswell et al. 2003 , pp. 162–163). The criteria underlying these typologies mainly have to do with the degree of mixing (e.g., are methods mixed throughout the research project or not?), the timing (e.g., sequential or concurrent mixing of methods) and the emphasis (e.g., is one approach dominant, or do they have equal status?).

We find this strong drive to develop methodologies, definitions, and typologies of MMR as guides to valid mixed methods research problematic. What it amounts to in practice is a methodology that lays out the basic guidelines for doing MMR in a “proper way.” This entails the danger of straight-jacketing reflection about the use of methods, decoupling it from theoretical and empirical considerations, thus favouring the unreflexive use of a standard methodology. Researchers are asked to make a choice for a particular MMR design and adhere to the guidelines for a “proper” MMR study. Such methodological prescription diametrically opposes the initial critique of the mechanical and unreflexive use of methods. The insight offered by Bourdieu’s notion of reflexivity is, on the contrary, that the actual research practice is fundamentally open in terms of being guided by a logic of practice that cannot be captured by a preconceived and all-encompassing logic independent of that practice. Reflexivity in this view cannot be achieved by hiding behind the construct of a standardized methodology—of whatever signature—it can only be achieved by objectifying the process of objectification that goes on within the context of the field in which the researcher is embedded. This reflexivity, then, requires an analysis of the position of the researcher as a critical component of the research process, both as the embodiment of past choices that have consequences for the strategic position in the scientific field, and as predispositions regarding the choice for the subject and content of a research project. By adding the insight of STS researchers that the point of deconstructing science and technology is not so much to offer a new best way of doing science or technology, but to provide insights into the critical moments in research (for a take on such a debate, see, for example, Edge 1995 , pp. 16–20), this calls for a sociology of science that takes methods much more seriously as objects of study. Such a programme should be based on studying the process of codification and standardization of methods in their historical context of production, circulation, and use. It would provide a basis for a sociological understanding of methods that can illuminate the critical moments in research alluded to above, enabling a systematic reflection on the process of objectification. This, in turn, allows a more sophisticated validation of using—and combining—methods than relying on prescribed methodologies.

The role of epistemology

The second theme discussed in a large number of contributions is the role epistemology plays in MMR. In a sense, epistemology provides the lifeblood for MMR in that methods in MMR are mainly seen in epistemological terms. This interpretation of methods is at the core of the knowledge claim of MMR practitioners, i.e., that the mixing of methods means mixing broad, different ways of knowing, which leads to better knowledge of the research object. It is also part of the identity that MMR consciously assumes, and that serves to set it apart from previous, more practical attempts to combine methods. This can be seen in the historical overview that Creswell and Plano Clark ( 2011 ) presented and that was discussed above. This reading, in which combining methods has evolved from the rather unproblematic level (one could alternatively say “naïve” or “unaware”) of instrumental use of various tools and techniques into an act that requires deeper thinking on a methodological and epistemological level, provides the legitimacy of MMR.

At the core of the MMR approach we thus find that methods are seen as unproblematic representations of different epistemologies. But this leads to a paradox, since the epistemological frameworks need to be held flexible enough to allow researchers to integrate elements of each of them (in the shape of methods) into one MMR design. As a consequence, the issue becomes the following: methods need to be disengaged from too strict an interpretation of the epistemological context in which they were developed in order for them to be “mixable,”’, but, at the same time, they must keep the epistemology attributed to them firmly intact.

In the MMR discourse two epistemological positions are identified that matter most: a positivist approach that gives rise to quantitative methods and a constructivist approach that is home to qualitative methods. For MMR to be a feasible endeavour, the differences between both forms of research must be defined as reconcilable. This position necessitates an engagement with those who hold that the quantitative/qualitative dichotomy is unbridgeable. Within MMR an interesting way of doing so has emerged. In the first issue of the Journal of Mixed Methods Research, Morgan ( 2007 ) frames the debate about research methodology in the social sciences in terms of Kuhnian paradigms, and he argues that the pioneers of the emancipation of qualitative research methods used a particular interpretation of the paradigm-concept to state their case against the then dominant paradigm in the social sciences. According to Morgan, they interpreted a paradigm mainly in metaphysical terms, stressing the connections among the trinity of ontology, epistemology, and methodology as used in the philosophy of knowledge (Morgan 2007 , p. 57). This allowed these scholars to depict the line between research traditions in stark, contrasting terms, using Kuhn’s idea of “incommensurability” in the sense of its “early Kuhn” interpretation. This strategy fixed the contrast between the proposed alternative approach (a “constructivist paradigm”), and the traditional approach (constructed as “the positivist paradigm”) to research as a whole, and offered the alternative approach as a valid option rooted in the philosophy of knowledge. Morgan focuses especially on the work of Egon Guba and Yvonne Lincoln who developed what they initially termed a “naturalistic paradigm” as an alternative to their perception of positivism in the social sciences (e.g., Guba and Lincoln 1985 ). Footnote 9 MMR requires a more flexible or “a-paradigmatic stance” towards research, which would entail that “in real-world practice, methods can be separated from the epistemology out of which they emerged” (Patton 2002 , quoted in Tashakkori and Teddlie 2010b , p. 14).

This proposal of an ‘interpretative flexibility’ (Bijker 1987 , 1997 ) regarding paradigms is an interesting proposition. But it immediately raises the question: why stop there? Why not take a deeper look into the epistemological technology of methods themselves, to let the muted components speak up in order to look for alternative “mixing interfaces” that could potentially provide equally valid benefits in terms of the understanding of a research object? The answer, of course, was already seen above. It is that the MMR approach requires situating methods epistemologically in order to keep them intact as unproblematic mediators of specific epistemologies and, thus, make the methodological prescriptions work. There are several problems with this. First, seeing methods solely through an epistemological lens is problematic, but it would be less consequential if it were applied to multiple elements of methods separately. This would at least allow a look under the hood of a method, and new ways of mixing methods could be opened up that go beyond the crude “qualitative” versus “quantitative” dichotomy. Second, there is also the issue of the ontological dimension of methods that is disregarded in an exclusively epistemological framing of methods (e.g., Law 2004 ). Taking this ontological dimension seriously has at least two important facets. First, it draws attention to the ontological assumptions that are woven into methods in their respective fields of production and that are imported into fields of users. Second, it entails the ontological consequences of practising methods: using, applying, and referring to methods and the realities this produces. This latter facet brings the world-making and boundary-drawing capacities of methods to the fore. Both facets are ignored in MMR. We say more about the first facet in the next section. With regard to the second facet, a crucial element concerns the data that are generated, collected, and analysed in a research project. But rather than problematizing the link between the performativity of methods and the data that are enacted within the frame of a method, here too MMR relies on a dichotomy: that between quantitative and qualitative data. Methods are primarily viewed as ways of gathering data or as analytic techniques dealing with a specific kind of data. Methods and data are conceptualised intertwiningly: methods too are seen as either quantitative or qualitative (often written as QUANT and QUAL in the literature), and perform the role of linking epistemology and data. In the final analysis, the MMR approach is based on the epistemological legitimization of the dichotomy between qualitative and quantitative data in order to define and combine methods: data obtain epistemological currency through the supposed in-severable link to certain methods, and methods are reduced to the role of acting as neutral mediators between them.

In this way, methods are effectively reduced to, on the one hand, placeholders for epistemological paradigms and, on the other hand, mediators between one kind of data and the appropriate epistemology. To put it bluntly, the name “mixed methods research” is actually a misnomer, because what is mixed are paradigms or “approaches,” not methods. Thus, the act of mixing methods à la MMR has the paradoxical effect of encouraging a crude black box approach to methods. This is a third problematic characteristic of MMR, because it hinders a detailed study of methods that can lead to a much richer perspective on mixing methods.

Black boxed methods and how to open them

The third problem that we identified with the MMR approach, then, is that with the impetus to standardize the MMR methodology by fixing methods epistemologically, complemented by a dichotomous view of data, they are, in the words of philosopher Bruno Latour, “blackboxed.” This is a peculiar result of the prescription for mixing methods as proposed by MMR that thus not only denies practice and the ontological dimensions of methods and data, but also casts methods in the role of unyielding black boxes. Footnote 10 With this in mind, it will come as no surprise that most foundational contributions to the MMR literature do not explicitly define what a method is, nor that they do not provide an elaborative historical account of individual methods. The particular framing of methods in MMR results in a blind spot for the historical and social context of the production and circulation of methods as intellectual products. Instead it chooses to reify the boundaries that are drawn between “qualitative” and “quantitative” methods and reproduce them in the methodology it proposes. Footnote 11 This is an example of “circulation without context” (Bourdieu 2002 , p. 4): classifications that are constructed in the field of use or reception without taking the constellation within the field of production seriously.

Of course, this does not mean that the reality of the differences between quantitative and qualitative research must be denied. These labels are sticky and symbolically laden. They have come, in many ways, to represent “two cultures” (Goertz and Mahony 2012 ) of research, institutionalised in academia, and the effects of nominally “belonging” to (or being assigned to) one particular category have very real consequences in terms of, for instance, access to research grants and specific journals. However, if the goal of an approach such as MMR is to open up new pathways in social science research, (and why should that not be the case?) it is hard to see how that is accomplished by defining the act of combining methods solely in terms of reified differences between research using qualitative and quantitative data. In our view, methods are far richer and more interesting constructs than that, and a practice of combining methods in research should reflect that. Footnote 12

Addressing these problems entices a reflection on methods and using (multiple) methods that is missing in the MMR perspective. A fruitful way to open up the black boxes and take into account the epistemological and ontological facets of methods is to make them, and their use, the object of sociological-historical investigation. Methods are constituted through particular practices. In Bourdieusian terms, they are objectifications of the subjectively understood practices of scientists “in other fields.” Rather than basing a practice of combining methods on an uncritical acceptance of the historically grown classification of types of social research (and using these as the building stones of a methodology of mixing methods), we propose the development of a multifaceted approach that is based on a study of the different socio-historical contexts and practices in which methods developed and circulated.

A sociological understanding of methods based on these premises provides the tools to break with the dichotomously designed interface for combining methods in MMR. Instead, focusing on the historical and social contexts of production and use can reveal the traces that these contexts leave, both in the internal structure of methods, how they are perceived, how they are put into practice, and how this practice informs the ontological effects of methods. Seeing methods as complex technologies, with a history that entails the struggles among the different agents involved in their production, and use opens the way to identify multiple interfaces for combining them: the one-sided boxes become polyhedra. The critical study of methods as “objects of objectification” also entices analyses of the way in which methods intervene between subject (researcher) and object and the way in which different methods are employed in practice to draw this boundary differently. The reflexive position generated by such a systematic juxtaposition of methods is a fruitful basis to come to a richer perspective on combining methods.

We critically reviewed the emerging practice of combining methods under the label of MMR. MMR challenges the mono-method approaches that are still dominant in the social sciences, and this is both refreshing and important. Combining methods should indeed be taken much more seriously in the social sciences.

However, the direction that the practice of combining methods is taking under the MMR approach seems problematic to us. We identified three main concerns. First, MMR scholars seem to be committed to designing a standardized methodological framework for combining methods. This is unfortunate, since it amounts to enforcing an unnecessary codification of aspects of research practices that should not be formally standardized. Second, MMR constructs methods as unproblematic representations of an epistemology. Although methods must be separable from their native epistemology for MMR to work, at the same time they have to be nested within a qualitative or a quantitative research approach, which are characterized by the data they use. By this logic, combining quantitative methods with other quantitative methods, or qualitative methods with other qualitative methods, cannot offer the same benefits: they originate from the same way of viewing and knowing the world, so it would have the same effect as blending two gradations of the same colour paint. The importance attached to the epistemological grounding of methods and data in MMR also disregards the ontological aspects of methods. In this article, we are arguing that this one-sided perspective is problematic. Seeing combining methods as equivalent to combining epistemologies that are somehow pure and internally homogeneous because they can be placed in a qualitative or quantitative framework essentially amounts to reifying these categories.

It also leads to the third problem: the black boxing of methods as neutral mediators between these epistemologies and data. This not only constitutes a problem for trying to understand methods as intellectual products, but also for regarding the practice of combining methods, because it ignores the social-historical context of the development of individual methods and hinders a sociologically grounded notion of combining methods.

We proceed from a different perspective on methods. In our view, methods are complex constructions. They are world-making technologies that encapsulate different assumptions on causality, rely on different conceptual relations and categorizations, allow for different degrees of emergence, and employ different theories of the data that they internalise as objects of analysis. Even more importantly, their current form as intellectual products cannot be separated from the historical context of their production, circulation, and use.

A fully developed exposition of such an approach will have to await further work. Footnote 13 So far, the sociological study of methods has not (yet) developed into a consistent research programme, but important elements can be derived from existing contributions such as MacKenzie ( 1981 ), Chapoulie ( 1984 ), Platt ( 1996 ), Freeman ( 2004 ), and Desrosières ( 2008a , b ). The work on the “social life of methods” (e.g., Savage 2013 ) also contains important leads for the development of a systematic sociological approach to method production and circulation. Based on the discussion in this article and the contributions listed above, some tantalizing questions can be formulated. How are methods and their elements objectified? How are epistemology and ontology defined in different fields and how do those definitions feed into methods? How do they circulate and how are they translated and used in different contexts? What are the main controversies in fields of users and how are these related to the field of production? What are the homologies between these fields?

Setting out to answer these questions opens up the possibility of exploring other interesting combinations of methods that emerge from the combination of different practices, situated in different historical and epistemological contexts, and with their unique set of interpretations regarding their constituent elements. One of these must surely be the data-theoretical elements that different methods incorporate. The problematization of data has become all the more pressing now that the debate about the consequences of “big data” for social scientific practices has become prominent (Savage and Burrows 2007 ; Levallois et al. 2013 ; Burrows and Savage 2014 ). Whereas MMR emphasizes the dichotomy between qualitative and quantitative data, a historical analysis of the production and use of methods can explore the more subtle, different interpretations and enactments of the “same” data. These differences inform method construction, controversies surrounding methods and, hence, opportunities for combining methods. These could then be constructed based on alternative conceptualisations of data. Again, while in some contexts it might be enlightening to rely on the distinction between data as qualitative or quantitative, and to combine methods based on this categorization, it is an exciting possibility that in other research contexts other conceptualisations of data might be of more value to enhance a specific (contextual) form of knowledge.

Change history

06 may 2019.

Unfortunately, figure 2 was incorrectly published.

The search term used was “mixed method*” in the “topic” search field of SSCI, A&HCI, and CPCI-SSH as contained in the Web of Science. A Google NGram search (not shown) confirmed this pattern. The results of a search for “mixed methods” and “mixed methods research” showed a very steep increase after 1994: in the first case, the normalized share in the total corpus increased by 855% from 1994 till 2008. Also, Creswell ( 2012 ) reports an almost hundred-fold increase in the number of theses and dissertations with mixed methods’ in the citation and abstract (from 26 in 1990–1994 to 2524 in 2005–2009).

Retrieved from https://uk.sagepub.com/en-gb/eur/journal-of-mixed-methods-research/journal201775#aims-and-scope on 1/17/2019.

In terms of antecedents of mixed methods research, it is interesting to note that Bourdieu, whose sociology of science we draw on, was, from his earliest studies in Algeria onwards, a strong advocate of combining research methods. He made it into a central characteristic of his approach to social science in Bourdieu et al. ( 1991 [1968]). His approach, as we see below, was very different from the one now proposed under the banner of MMR. Significantly, there is no mention of Bourdieu’s take on combining methods in any of the sources we studied.

Morse’s example in particular warns us that restricting the analysis to the authors that have published in the JMMR runs the risk of missing some important contributors to the spread of MMR through the social sciences. On her website, Morse lists 11 publications (journal articles, book chapters, and books) that explicitly make reference to mixed methods (and a substantial number of other publications are about methodological aspects of research), so the fact that she has not (yet) published in the JMMR cannot, by itself, be taken as an indication of a lesser involvement with the practice of combining methods. See the website of Janice Morse at https://faculty.utah.edu/u0556920-Janice_Morse_RN,_PhD,_FAAN/hm/index.hml accessed 1/17/2019.

Bourdieu ( 1999 , p. 26) mentions that one has to be a scientific capitalist to be able to start a scientific revolution. But here he refers explicitly to the autonomy of the scientific field, making it virtually impossible for amateurs to stand up against the historically accumulated capital in the field and incite a revolution.

The themes summarize the key issues through which MMR as a group comes “into difference” (Bourdieu 1993 , p. 32). Of course, as in any (sub)field, the agents identified above often differ in their opinions on some of these key issues or disagree on the answer to the question if there should be a high degree of convergence of opinions at all. For instance, Bryman ( 2009 ) worried that MMR could become “a ghetto.” For him, the institutional landmarks of having a journal, conferences, and a handbook increase the risk of “not considering the whole range of possibilities.” He added: “I don’t regard it as a field, I kind of think of it as a way of thinking about how you go about research.” (Bryman, cited in Leech 2010 , p. 261). It is interesting to note that Bryman, like fellow sociologists Morgan and Denscombe, had published only one paper in the JMMR by the end of 2016 (Bryman passed away in June of 2017). Although these papers are among the most cited papers in the journal (see Table 1 ), this low number is consistent with the more eclectic approach that Bryman proposed.

Johnson, Onwuegbuzie, and Turner ( 2007 , p. 123).

Guba and Lincoln ( 1985 ) discuss the features of their version of a positivistic approach mainly in ontological and epistemological terms, but they are also careful to distinguish the opposition between naturalistic and positivist approaches from the difference between what they call the quantitative and the qualitative paradigms. Since they go on to state that, in principle, quantitative methods can be used within a naturalistic approach (although in practice, qualitative methods would be preferred by researchers embracing this paradigm), they seem to locate methods on a somewhat “lower,” i.e., less incommensurable level. However, in their later work (both together as well as with others or individually) and that of others in their wake, there seems to have been a shift towards a stricter interpretation of the qualitative/quantitative divide in metaphysical terms, enabling Teddlie and Tashakkori (2010b) to label this group “purists” (Tashakkori and Teddlie 2010b , p. 13).

See, for instance, Onwuegbuzie et al.’s ( 2011 ) classification of 58 qualitative data analysis techniques and 18 quantitative data analysis techniques.

This can also be seen in Morgan’s ( 2018 ) response to Sandelowski’s ( 2014 ) critique of the binary distinctions in MMR between qualitative and quantitative research approaches and methods. Morgan denounces the essentialist approach to categorizing qualitative and quantitative research in favor of a categorization based on “family resemblances,” in which he draws on Wittgenstein. However, this denies the fact that the essentialist way of categorizing is very common in the MMR corpus, particularly in textbooks and manuals (e.g., Plano Clark and Ivankova 2016 ). Moreover, and more importantly, he still does not extend this non-essentialist model of categorization to the level of methods, referring, for instance, to the different strengths of qualitative and quantitative methods in mixed methods studies (Morgan 2018 , p. 276).

While it goes beyond the scope of this article to delve into the history of the qualitative-quantitative divide in the social sciences, some broad observations can be made here. The history of method use in the social sciences can briefly be summarized as first, a rather fluid use of what can retrospectively be called different methods in large scale research projects—such as the Yankee City study of Lloyd Warner and his associates (see Platt 1996 , p. 102), the study on union democracy of Lipset et al. ( 1956 ), and the Marienthal study by Lazarsfeld and his associates (Jahoda et al. 1933 ); see Brewer and Hunter ( 2006 , p. xvi)—followed by an increasing emphasis on quantitative data and the objectification and standardization of methods. The rise of research using qualitative data can be understood as a reaction against this use and interpretation of method in the social sciences. However, out of the ensuing clash a new, still dominant classification of methods emerged, one that relies on the framing of methods as either “qualitative” or “quantitative.” Moreover, these labels have become synonymous with epistemological positions that are reproduced in MMR.

A proposal to come to such an approach can be found in Timans ( 2015 ).

Bijker, W. (1987). The social construction of bakelite: Toward a theory of invention. In W. Bijker, T. Hughes, T. Pinch, & D. Douglas (Eds.), The social construction of technological systems: New directions in the sociology and history of technology . Cambridge, MA: MIT press.

Google Scholar  

Bijker, W. (1997). Of bicycles, bakelites, and bulbs: Toward a theory of sociotechnical change . Cambridge, MA: MIT Press.

Bourdieu, P. (1975). La spécifité du champ scientifique et les conditions sociales du progrès de la raison. Sociologie et Sociétés, 7 (1), 91–118.

Article   Google Scholar  

Bourdieu, P. (1988). Homo academicus . Stanford, CA: Stanford University Press.

Bourdieu, P. (1993). The field of cultural production . Cambridge, UK: Polity Press.

Bourdieu, P. (1999). Les historiens et la sociologie de Pierre Bourdieu. Le Bulletin de la Société d'Histoire Moderne et Contemporaine/SHMC, 1999 (3&4), 4–27.

Bourdieu, P. (2002). Les conditions sociales de la circulation internationale des idées. Actes de la Recherche en Sciences Sociales, 145 (5), 3–8.

Bourdieu, P. (2004). Science of science and reflexivity . Cambridge, UK: Polity.

Bourdieu, P. (2007). Sketch for a self-analysis . Cambridge, UK: Polity.

Bourdieu, P., Chamboredon, J., & Passeron, J. (1991). The craft of sociology: Epistemological preliminaries . Berlin, Germany: De Gruyter.

Book   Google Scholar  

Brewer, J., & Hunter, A. (2006). Multimethod research: A synthesis of styles . London, UK: Sage.

Bryman, A. (2009). Sage Methodspace: Alan Bryman on research methods. Retrieved from http://www.youtube.com/watch?v=bHzM9RlO6j0 . Accessed 3/7/2019.

Bryman, A. (2012). Social research methods . Oxford, UK: Oxford University Press.

Burrows, R., & Savage, M. (2014). After the crisis? Big data and the methodological challenges of empirical sociology. Big Data & Society, 1 (1), 1–6.

Campbell, D., & Fiske, D. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56 (2), 81–105.

Chapoulie, J. (1984). Everett C. Hughes et le développement du travail de terrain en sociologie. Revue Française de Sociologie, 25 (4), 582–608.

Collins, R. (1998). The sociology of philosophies: A global theory of intellectual change . Cambridge, MA: Harvard University Press.

Creswell, J. (2008). Research design: Qualitative, quantitative, and mixed methods approaches . Thousand Oaks, CA: Sage.

Creswell, J. (2011). Controversies in mixed methods research. In N. Denzin & Y. Lincoln (Eds.), The Sage handbook of qualitative research . Thousand Oaks, CA: Sage.

Creswell, J. (2012). Qualitative inquiry and research design: Choosing among five approaches . Thousand Oaks, CA: Sage.

Creswell, J., & Plano Clark, V. (2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.

Creswell, J., Plano Clark, V., Gutmann, M., & Hanson, W. (2003). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research . Thousand Oaks, CA: Sage.

Denscombe, M. (2008). Communities of practice a research paradigm for the mixed methods approach. Journal of Mixed Methods Research, 2 (3), 270–283.

Desrosières, A. (2008a). Pour une sociologie historique de la quantification - L’Argument statistique I . Paris, France: Presses des Mines.

Desrosières, A. (2008b). Gouverner par les nombres - L’Argument statistique II . Paris, France: Presses des Mines.

Edge, D. (1995). Reinventing the wheel. In D. Edge, S. Jasanof, G. Markle, J. Petersen, & T. Pinch (Eds.), Handbook of science and technology studies . Thousand Oaks, CA: Sage.

Fielding, N. (2012). Triangulation and mixed methods designs data integration with new research technologies. Journal of Mixed Methods Research, 6 (2), 124–136.

Fielding, N., & Cisneros-Puebla, C. (2009). CAQDAS-GIS convergence: Toward a new integrated mixed method research practice? Journal of Mixed Methods Research, 3 (4), 349–370.

Fleck, C., Heilbron, J., Karady, V., & Sapiro, G. (2016). Handbook of indicators of institutionalization of academic disciplines in SSH. Serendipities, Journal for the Sociology and History of the Social Sciences, 1 (1) Retrieved from http://serendipities.uni-graz.at/index.php/serendipities/issue/view/1 . Accessed 10/10/2018.

Freeman, L. (2004). The development of social network analysis: A study in the sociology of science . Vancouver, Canada: Empirical Press.

Gingras, Y., & Gemme, B. (2006). L’Emprise du champ scientifique sur le champ universitaire et ses effets. Actes de la Recherche en Sciences Sociales, 164 , 51–60.

Goertz, G., & Mahony, J. (2012). A tale of two cultures: Qualitative and quantitative research in the social sciences . Princeton: Princeton University Press.

Greene, J. (2008). Is mixed methods social inquiry a distinctive methodology? Journal of Mixed Methods Research, 2 (1), 7–22.

Guba, E., & Lincoln, Y. (1985). Naturalistic inquiry . Thousand Oaks, CA: Sage.

Heilbron, J., Bedecarré, M., & Timans, R. (2017). European journals in the social sciences and humanities. Serendipities, Journal for the Sociology and History of the Social Sciences, 2 (1), 33–49 Retrieved from http://serendipities.uni-graz.at/index.php/serendipities/issue/view/5 . Accessed 10/10/2018.

Hesse-Biber, S. (2010). Mixed methods research: Merging theory with practice . New York, NY: Guilford Press.

Jahoda, M., Lazarsfeld, P., & Zeisel, H. (1933). Die Arbeitslosen von Marienthal. Psychologische Monographen, 5 .

Johnson, R., & Gray, R. (2010). A history of philosophical and theoretical issues for mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Johnson, R., Onwuegbuzie, A., & Turner, L. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1 (2), 112–133.

Law, J. (2004). After method: Mess in social science research . London, UK: Routledge.

Leech, N. (2010). Interviews with the early developers of mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Leech, N., & Onwuegbuzie, T. (2009). A typology of mixed methods research designs. Quality & Quantity, 43 (2), 265–275.

Levallois, C., Steinmetz, S., & Wouters, P. (2013). Sloppy data floods or precise social science methodologies? In P. Wouters, A. Beaulieu, A. Scharnhorst, & S. Wyatt (Eds.), Virtual knowledge . Cambridge, MA: MIT Press.

Lipset, S., Trow, M., & Coleman, J. (1956). Union democracy: The internal politics of the international typographical union . Glencoe, UK: Free Press.

MacKenzie, D. (1981). Statistics in Britain: 1865–1930: The social construction of scientific knowledge . Edinburgh, UK: Edinburgh University Press.

Morgan, D. (2007). Paradigms lost and pragmatism regained: Methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1 (1), 48–76.

Morgan, D. (2018). Living with blurry boundaries: The values of distinguishing between qualitative and quantitative research. Journal of Mixed Methods Research, 12 (3), 268–276.

Mullins, N. (1973). Theories and theory groups in contemporary American sociology . New York, NY: Harper & Row.

Nagy Hesse-Biber, S., & Leavy, P. (Eds.). (2008). Handbook of emergent methods . New York, NY and London, UK: Guilford Press.

Nastasi, B., Hitchcock, J., & Brown, L. (2010). An inclusive framework for conceptualizing mixed method design typologies. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Onwuegbuzie, A., Leech, N., & Collins, K. (2011). Toward a new era for conducting mixed analyses: The role of quantitative dominant and qualitative dominant crossover mixed analyses. In M. Williams & P. Vogt (Eds.), Handbook of innovation in social research methods . London, UK: Sage.

Patton, M. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.

Plano Clark, V., & Creswell, J. (Eds.). (2007). The mixed methods reader . Thousand Oaks, CA: Sage.

Plano Clark, V., & Ivankova, N. (2016). Mixed methods research: A guide to the field . Thousand Oaks, CA: Sage.

Platt, J. (1996). A history of sociological research methods in America: 1920–1960 . Cambridge, UK: Cambridge University Press.

Plowright, D. (2010). Using mixed methods – Frameworks for an integrated methodology . Thousand Oaks, CA: Sage.

Sandelowski, M. (2014). Unmixing mixed methods research. Research in Nursing & Health, 37 (1), 3–8.

Sapiro, G., Brun, E., & Fordant, C. (2018). The rise of the social sciences and humanities in France: Iinstitutionalization, professionalization and autonomization. In C. Fleck, M. Duller, & V. Karady (Eds.), Shaping human science disciplines: Institutional developments in Europe and beyond . Basingstoke, UK: Palgrave.

Savage, M. (2013). The ‘social life of methods’: A critical introduction. Theory, Culture and Society, 30 (4), 3–21.

Savage, M., & Burrows, R. (2007). The coming crisis of empirical sociology. Sociology, 41 (5), 885–899.

Steinmetz, G. (2016). Social fields, subfields and social spaces at the scale of empires: Explaining the colonial state and colonial sociology. The Sociological Review, 64(2_suppl). 98-123.

Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social and behavioral research . Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (Eds.). (2010a). Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (2010b). Overview of contemporary issues in mixed methods. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Chapter   Google Scholar  

Tashakkori, A., & Teddlie, C. (2010c). Epilogue. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (2011). Mixed methods research: Contemporary issues in an emerging field. In N. Denzin & Y. Lincoln (Eds.), The SAGE handbook of qualitative research (4th ed.). Thousand Oaks, CA: Sage.

Teddlie, C., & Tashakkori, A. (2008). Foundations of mixed methods research – Integrating quantitative and qualitative approaches in the social and behavioral sciences . Thousand Oaks, CA: Sage.

Timans, R. (2015). Studying the Dutch business elite: Relational concepts and methods . Doctoral dissertation, Erasmus University Rotterdam, the Netherlands.

Wacquant, L. (2013). Bourdieu 1993: A case study in scientific consecration. Sociology, 47 (1), 15–29.

Download references

Acknowledgments

This research is part of the Interco-SSH project, funded by the European Union under the 7th Research Framework Programme (grant agreement no. 319974). Johan Heilbron would like to thank Louise and John Steffens, members of the Friends Founders’ Circle, who assisted his stay at the Princeton Institute for Advanced Study in 2017-18 during which he completed his part of the present article.

Author information

Authors and affiliations.

Erasmus Centre for Economic Sociology (ECES), Erasmus University Rotterdam, Rotterdam, Netherlands

Centre for Science and Technology Studies (CWTS), Leiden University, Leiden, Netherlands

Paul Wouters

Erasmus Centre for Economic Sociology (ECES), Rotterdam and Centre européen de sociologie et de science politique de la Sorbonne (CESSP-CNRS-EHESS), Paris, France

Johan Heilbron

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rob Timans .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Timans, R., Wouters, P. & Heilbron, J. Mixed methods research: what it is and what it could be. Theor Soc 48 , 193–216 (2019). https://doi.org/10.1007/s11186-019-09345-5

Download citation

Published : 29 March 2019

Issue Date : 01 April 2019

DOI : https://doi.org/10.1007/s11186-019-09345-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Data, field analysis
• Mixed methods research
• Multiple methods
• Reflexivity
• Sociology of science
• Find a journal
• Publish with us
• Track your research

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 20, Issue 3
• Mixed methods research: expanding the evidence base
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Allison Shorten 1 ,
• Joanna Smith 2
• 1 School of Nursing , University of Alabama at Birmingham , USA
• 2 Children's Nursing, School of Healthcare , University of Leeds , UK
• Correspondence to Dr Allison Shorten, School of Nursing, University of Alabama at Birmingham, 1720 2nd Ave South, Birmingham, AL, 35294, USA; [email protected]; ashorten{at}uab.edu

https://doi.org/10.1136/eb-2017-102699

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

‘Mixed methods’ is a research approach whereby researchers collect and analyse both quantitative and qualitative data within the same study. 1 2 Growth of mixed methods research in nursing and healthcare has occurred at a time of internationally increasing complexity in healthcare delivery. Mixed methods research draws on potential strengths of both qualitative and quantitative methods, 3 allowing researchers to explore diverse perspectives and uncover relationships that exist between the intricate layers of our multifaceted research questions. As providers and policy makers strive to ensure quality and safety for patients and families, researchers can use mixed methods to explore contemporary healthcare trends and practices across increasingly diverse practice settings.

What is mixed methods research?

Mixed methods research requires a purposeful mixing of methods in data collection, data analysis and interpretation of the evidence. The key word is ‘mixed’, as an essential step in the mixed methods approach is data linkage, or integration at an appropriate stage in the research process. 4 Purposeful data integration enables researchers to seek a more panoramic view of their research landscape, viewing phenomena from different viewpoints and through diverse research lenses. For example, in a randomised controlled trial (RCT) evaluating a decision aid for women making choices about birth after caesarean, quantitative data were collected to assess knowledge change, levels of decisional conflict, birth choices and outcomes. 5 Qualitative narrative data were collected to gain insight into women’s decision-making experiences and factors that influenced their choices for mode of birth. 5

In contrast, multimethod research uses a single research paradigm, either quantitative or qualitative. Data are collected and analysed using different methods within the same paradigm. 6 7 For example, in a multimethods qualitative study investigating parent–professional shared decision-making regarding diagnosis of suspected shunt malfunction in children, data collection included audio recordings of admission consultations and interviews 1 week post consultation, with interactions analysed using conversational analysis and the framework approach for the interview data. 8

What are the strengths and challenges in using mixed methods?

Selecting the right research method starts with identifying the research question and study aims. A mixed methods design is appropriate for answering research questions that neither quantitative nor qualitative methods could answer alone. 4 9–11 Mixed methods can be used to gain a better understanding of connections or contradictions between qualitative and quantitative data; they can provide opportunities for participants to have a strong voice and share their experiences across the research process, and they can facilitate different avenues of exploration that enrich the evidence and enable questions to be answered more deeply. 11 Mixed methods can facilitate greater scholarly interaction and enrich the experiences of researchers as different perspectives illuminate the issues being studied. 11

The process of mixing methods within one study, however, can add to the complexity of conducting research. It often requires more resources (time and personnel) and additional research training, as multidisciplinary research teams need to become conversant with alternative research paradigms and different approaches to sample selection, data collection, data analysis and data synthesis or integration. 11

What are the different types of mixed methods designs?

Mixed methods research comprises different types of design categories, including explanatory, exploratory, parallel and nested (embedded) designs. 2   Table 1 summarises the characteristics of each design, the process used and models of connecting or integrating data. For each type of research, an example was created to illustrate how each study design might be applied to address similar but different nursing research aims within the same general nursing research area.

• View inline

Types of mixed methods designs*

What should be considered when evaluating mixed methods research?

When reading mixed methods research or writing a proposal using mixed methods to answer a research question, the six questions below are a useful guide 12 :

Does the research question justify the use of mixed methods?

Is the method sequence clearly described, logical in flow and well aligned with study aims?

Is data collection and analysis clearly described and well aligned with study aims?

Does one method dominate the other or are they equally important?

Did the use of one method limit or confound the other method?

When, how and by whom is data integration (mixing) achieved?

For more detail of the evaluation guide, refer to the McMaster University Mixed Methods Appraisal Tool. 12 The quality checklist for appraising published mixed methods research could also be used as a design checklist when planning mixed methods studies.

• Elliot AE , et al
• Creswell JW ,
• Plano ClarkV L
• Greene JC ,
• Caracelli VJ ,
• Ivankova NV
• Shorten A ,
• Shorten B ,
• Halcomb E ,
• Cheater F ,
• Bekker H , et al
• Tashakkori A ,
• Creswell JW
• 12. ↵ National Collaborating Centre for Methods and Tools . Appraising qualitative, quantitative, and mixed methods studies included in mixed studies reviews: the MMAT . Hamilton, ON : BMJ Publishing Group , 2015 . http://www.nccmt.ca/resources/search/232 (accessed May 2017) .

Competing interests None declared.

Provenance and peer review Commissioned; internally peer reviewed.

Read the full text or download the PDF:

• Technical Support
• Find My Rep

You are here

Journal of Mixed Methods Research

Preview this book.

• Description
• Aims and Scope
• Editorial Board
• Abstracting / Indexing
• Submission Guidelines

NOW INDEXED IN THE SSCI © ! Visit http://mc.manuscriptcentral.com/jmmr For details about developing and including critical elements of publishable manuscripts in the Journal of Mixed Methods Research (JMMR) , please read this editorial. Submitting authors are encouraged also to review and include with their submission a checklist of elements found in successful articles. **************************************************************** Mixed methods research is defined as research in which the investigator collects and analyzes data, integrates the findings, and draws inferences using both qualitative and quantitative approaches or methods in a single study or program of inquiry.

The JMMR is an innovative, quarterly, international publication that focuses on empirical methodological articles, methodological/theoretical articles, research notes and commentaries about mixed methods research across the social, behavioral, health, and human sciences.

Article types published include:

• Original mixed methods research that fits the definition of mixed methods research; explicitly integrates quantitative and qualitative dimensions of the study; makes a novel methodological contribution to mixed methods research; and adds to a substantive area in the scholar’s field of inquiry
• Methodological/theoretical topics that advance knowledge about mixed methods research and integration, such as mixed methods research/evaluation questions, types of designs, sampling and/or measurement procedures, approaches to data analysis, drawing metainferences, theory, validity/research integrity, software applications, paradigm stances, writing approaches, and value and use of mixed methods research
• Prevalence studies that are a type of methodological manuscript which systematically examine the adoption and use of mixed methods research
• Research Notes that are brief articles with a focus on a specific or narrowly framed concept, methodological innovation or application
• Commentaries that provide an alternative interpretation, application, or expansion of ideas raised previously in a JMMR original article and will entail a response from the index author(s)
• Media Reviews that critically evaluate the content, quality, and value of textbooks, software, or other media, such as websites, companion material, and video series
• Special Issues that are a group of articles focused on a specific topic about mixed methods research and are led and managed by one or more guest editors
• Virtual Special Issues that have assembled previously published JMMR articles and editorials are listed online together about a particular theme of relevance to mixed methods research such as the inaugural virtual special issue on paradigms.

The Journal of Mixed Methods Research's scope includes

• Exploring terminology and nomenclature used in mixed methods research
• Providing a forum for discussion of parameters effecting mixed methods research
• Creating and interpreting paradigmatic and philosophical foundations for mixed methods research
• Illuminating theory, design, procedural, and analysis issues
• Determining the logistics of conducting mixed methods research
• Investigating approaches for integration in multiple dimensions of mixed methods research

EDITORIAL BOARD The editorial board consists of distinguished international researchers in the social, behavioral, health, and human sciences with expertise in quantitative, qualitative, and mixed methods research. Editorial board members' names and affiliations are found under the Editorial Board link.

The JMMR is a premiere outlet for ground-breaking and seminal work in the field of mixed methods research, as well as a primary forum for the growing community of international and multidisciplinary scholars of mixed methods research.

The Journal of Mixed Methods Research ( JMMR ) serves as a premiere outlet for ground-breaking and seminal work in the field of mixed methods research. Of primary importance is building an international and multidisciplinary community of mixed methods researchers. The journal's scope includes developing and exploring the dimensions of integration in mixed methods research, generating a global terminology and nomenclature for mixed methods research, delineating where mixed methods research may be used most effectively, creating the paradigmatic and philosophical foundations for mixed methods research, illuminating design and procedure issues, and determining the logistics of conducting mixed methods research. The journal features articles from a wide variety of international perspectives, including academics and practitioners from psychology, sociology, education, evaluation, health sciences, geography, communication, management, family studies, marketing, social work, and other related disciplines across the social, behavioral, health, arts and human sciences.

Submission types:

• Original mixed methods research manuscripts in the social, behavioral, health, arts and human sciences that:
• identify and support with a substantive literature review a unique contribution to mixed methods research;
• fit the definition of mixed methods research by collecting and analyzing data, integrating the findings, and drawing inferences using both qualitative and quantitative approaches or methods;
• provide a design figure of the mixed methods procedures;
• explicitly integrate the quantitative and qualitative aspects of the study;
• delineate how the study makes a contribution to a substantive area in the scholar’s field of inquiry; and
• discuss explicitly how the research makes a novel contribution to literature on mixed methods research.

Original research manuscripts that do not articulate a specific methodological purpose, or fail to show integration and discuss a novel contribution to the mixed methods literature will be returned to the author(s).

• Methodological / theoretical research  manuscripts that advance knowledge about mixed methods research, such as:
• Types of research/evaluation questions
• Types of designs
• Sampling and/or measurement procedures
• Approaches to data analysis and drawing metainferences
• Validity and research integrity
• Software applications
• Paradigm stance
• Writing approaches
• The value and use of mixed methods research
• Theoretical lenses
• Cultural issues
• Ethical issues
• Quality in mixed methods research
• Prevalence studies (see below)

Methodological/theoretical manuscripts that do not articulate a specific methodological purpose, or fail to review relevant literature and discuss a novel contribution to the mixed methods literature will be returned to the author(s).

• Mixed methods research prevalence manuscripts that are a type of methodological manuscript which systematically examines the adoption and use of mixed methods research and:
• specify a methodological aim and define how the chosen disciplinary focus addresses that aim;
• conduct a rigorous literature review to identify the MMR literature relevant to the MMR prevalence study
• present findings in a format that illustrates the methodological issues;
• interpret the value of the prevalence study in the field of the authors and reflect about how it advances understanding about MMR methodology.
• Research notes manuscripts developed with a focus on a specific concept of methodological innovation relevant to mixed methods research that:
• follow the structure of an original mixed methods research or methodological/theoretical paper;
• articulate a methodological gap, need or challenge faced in using mixed methods research;
• review succinctly literature relevant to the methodological gap, need or challenge;
• provide a brief review of the substantive topic content sufficiently to illustrate the methodological context; and
• illustrate the innovation through examples and rhetoric.

Research notes manuscripts that do not articulate a specific methodological purpose, or fail to review relevant literature and discuss a novel contribution to the mixed methods literature will be returned to the author(s).

• Commentaries authored in response to recently published articles in the JMMR that provide an alternative interpretation, application, or expansion of ideas raised previously in an original article
• Media reviews that critically evaluate the content, quality, and value of textbooks, software, or other media, such as websites, companion material, and video series that are of interest to mixed methods researchers
• Special issues that comprise a group of articles focused on a specific topic about mixed methods research that are typically led and managed by one or more guest editors in consultation with the editors
• Virtual Special Issues ­­­­ that are a group of related and previously published JMMR articles and editorials that have been assembled and listed online together about a particular theme of relevance to mixed methods research, e.g., the inaugural virtual special issue on paradigms.

This journal is a member of the Committee on Publication ­­Ethics (COPE).

• Clarivate Analytics: Current Contents - Physical, Chemical & Earth Sciences
• ERIC (Education Resources Information Center)
• ProQuest: CSA Sociological Abstracts
• Royal College of Nursing Journals
• Social SciSearch
• Social Sciences Citation Index (Web of Science)
• Social Services Abstracts
• Social Work Abstracts
• Worldwide Political Science Abstracts

The scope of the Journal of Mixed Methods Research (JMMR) includes developing and exploring the dimensions of integration in mixed methods research, generating a global terminology and nomenclature for mixed methods research, delineating where mixed methods research may be used most effectively, creating the paradigmatic and philosophical foundations for mixed methods research, illuminating design and procedure issues, and determining the logistics of conducting mixed methods research. JMMR uses a double-anonymize review process and manuscripts are generally reviewed by three scholars, at least one of which serves on the JMMR editorial board. The journal aims to have each manuscript go through its initial review within 4 months of receipt.

Submitting the Manuscript

Electronic Submissions

JMMR uses an online submission and review platform. Manuscripts should be submitted electronically to http://mc.manuscriptcentral.com/jmmr . Authors will be required to set up an online account on the SageTrack system powered by ScholarOne. From their account, a new submission can be initiated. Authors will be asked to provide the required information (author names and contact information, abstract, keywords, etc.). All submitting authors should include their ORCID identification number. The submitting author will upload the "title page" and "main document" separately to ensure that the manuscript is ready for an anonymize review. The site contains links to an online user's guide (Get Help Now) for help navigating the site.

If you have any questions about the submission process please contact:

Satoko Motohara Managing Editor Email: [email protected]

Authors who need to publish their articles via open access due to institutional or funding agency requirements are welcome to submit to JMMR . The journal participates in Sage Choice, Sage’s open access option for authors. For more about Sage Choice, please click here http://www.sagepub.com/sagechoice/faq.sp

Manuscript Expectations

JMMR primarily publishes two types of manuscripts: original research, and methodological/theoretical discussions. All manuscripts submitted for consideration must make a novel methodological contribution.

Original research manuscripts identify a methodological contribution and report empirical mixed methods research in the social, behavioral, health, arts, and human sciences to illustrate the methodological contribution. These manuscripts must:

• fit the definition of mixed methods research by collecting and analyzing data, integrating the findings, and drawing overall interpretations or metainferences using both qualitative and quantitative approaches or methods;
• integrate explicitly the quantitative and qualitative aspects of the study; and
• discuss how the manuscript makes a novel contribution both to the literature on mixed methods research, and to a substantive area in the scholar’s field of inquiry.

Original research manuscripts that do not show integration or discuss how they make a novel contribution to the mixed methods literature will be returned to the author(s).

Manuscripts that discuss methodological/theoretical issues advance knowledge about mixed methods research. These manuscripts all must:

• address an important topic in the field of mixed methods;
• incorporate relevant literature about the mixed methods topic; and
• make a novel contribution to the understanding of mixed methods research.

You can learn more about the expectations for authoring these two main type of manuscripts by reading several JMMR editorials:  

• the July 2015 editorial should be used to guide the structuring of your manuscript;
• the January 2017 editorial provides principles for consideration when developing and defining novel terminology for the field of mixed methods research
• the April 2019 editorial details three required elements in manuscripts with the emphasis on articulating and illustrating the novel methodological contribution.
• the October 2019 editorial provides a checklist of characteristics found in successful methodological articles to guide authors.  

In addition, JMMR also publishes:

MMR prevalence studies that are a type of methodological manuscript which systematically examines and reviews the adoption and use of mixed methods research. You can learn about the expectations of prevalence studies by reading the April 2016 editorial .

Research Notes developed as brief articles that focus on a specific concept of methodological innovation relevant to mixed methods research. These may contain a maximum of 3 tables and/or figures.

Commentaries authored in response to published articles in the JMMR that provide an alternative interpretation, application, or expansion of ideas raised in a previously published JMMR original article. You can learn about publishing commentaries in the April 2018 editorial .

Media Reviews that critically evaluate the content, quality, and value of textbooks, software, or other media, such as websites, companion material, and video series that are of interest to mixed methods researchers. Media reviews should include  complete bibliographic detail (author, year, title, and publisher information) of the reviewed media.  You can learn about publishing media reviews in the October 2016 editorial .

Special Issues that are a group of articles focused on a specific topic about mixed methods research that are typically led and managed by one or more guest editors in consultation with the editors. You can learn more about publishing a special issue in the October 2018 editorial .

Manuscript Requirements

Title Page: Prepare a title page that includes the manuscript title, authors and affiliations, contact information for the corresponding author, and author acknowledgements, if relevant. The author names should only appear in the title page document. Manuscripts should be prepared with author names anonymized throughout to facilitate the review process.

Style: All manuscripts should conform to the Publication Manual of the American Psychological Association, 7th Edition (APA, 2019). The entire document should be double spaced and left justified, use a 12-point font, and have pages numbered consecutively. Authors should prepare an abstract and 3-5 key words. The manuscript should be prepared in MS Word (.doc) or rich text format (.rtf) and include: the text, references, endnotes, appendix(ces), tables, and figures. References, endnotes, tables, and figures must be on separate pages. Figures may also be placed into separate files. Indicate the approximate placement of each table and figure within the text with a comment such as "Insert Figure 1 about here." Please use endnotes instead of footnotes. Authors should use U.S. spellings and all references to authors within the manuscript should be anonymized (such as by replacing author names in citations with XXXXXX).

Word Limits: The full manuscript (including references, appendices, and tables) should not exceed 10,000 words in length for original empirical research manuscripts, 8,000 words for methodological/theoretical manuscripts. The word count should range from 1500 to 3500 words for research notes, and 1000 to 2000 words for commentaries. Manuscripts that exceed the stated word limitations will be returned to the author(s).

Abstract: Authors should prepare an abstract limited to 120 words. Authors must include an explicit statement about the contribution of the manuscript to mixed methods research. Please, read the April 2019 editorial for examples.

Main Text: Authors should recognize that JMMR seeks to advance understanding of mixed methods research and thus position their manuscript within the mixed methods literature.  For all types of manuscripts (empirical, methodological/theoretical, research notes, commentaries), authors must provide an explicit methodological purpose/aim statement in the introduction/background section of the body of the manuscript. To support the methodological aim, the authors need to provide compelling arguments for a unique methodological contribution to mixed methods research based on relevant mixed methods literature. Therefore, the methodological aim should represent a novel contribution to mixed methods as shown by an extensive review of the mixed methods literature. The methodological aim should elaborate on the contribution stated in the abstract, and build on existing methodological literature in the field. Moreover, authors must also incorporate into the discussion a section titled ‘‘Contribution to the Field of Mixed Methods”. Please, read the editorials indicated above in the manuscript expectations section (especially the April 2019 editorial) and the review criteria noted below.

Supplemental materials: Authors may submit supplemental materials for publication only online that will NOT appear in the print version should the paper be accepted. In the manuscript submission, the materials should be clearly marked, and should be denoted in the manuscript as supplemental material for an online Appendix.

Originality of the Manuscript: Authors submitting manuscripts to JMMR should not simultaneously submit them to another journal, nor should manuscripts have been published elsewhere in substantially similar form or with substantially similar content. Authors who have submitted manuscripts to other journals based on the same data are asked to state this in their cover letter and to submit copies of these other papers/publications for review by the Editors at the time of submitting a manuscript to JMMR . Authors in doubt about what constitutes prior publication should consult the Editors.

Obtaining Permissions: Obtaining permission for any quoted or reprinted material that requires permission is the responsibility of the author.

Preprints:  JMMR may accept submissions of papers that have been posted on pre-print servers; please alert the Editorial Office when submitting (contact details above) and include the DOI for the preprint in the designated field in the manuscript submission system. Authors should not post an updated version of their paper on the preprint server while it is being peer reviewed for possible publication in JMMR. If the article is accepted for publication, the author may re-use their work according to Administration & Society's author archiving policy. If your paper is accepted, you must include a link on your preprint to the final version of your paper.

Submission of a manuscript implies commitment to publish in JMMR .

REVIEW CRITERIA

JMMR requires all original research manuscripts to include a contribution to mixed methods research statement and a section in the discussion about the contribution to the mixed methods literature. You can learn more about writing a mixed methods contribution statement in the April 2019 editorial . All authors, especially those new to JMMR , are encouraged to review and submit the checklist of characteristics found in publishable manuscripts in the October 2019 editorial .

The review criteria for empirical mixed methods research will include:

• Methodological contribution
• Noteworthiness of the problem
• Theoretical framework
• Fit of questions to mixed methods design
• Mixed methods design
• Mixed methods sampling
• Mixed methods analysis and integration
• Insightfulness of discussion
• Writing quality
• Quality of conclusions
• Contribution to mixed methods literature
• Interest to JMMR readership

For further information on review of the quality of the empirical component of manuscripts can be found in Good Reporting of A Mixed Methods Study (GRAMMS) criteria as published in the Journal of Health Services Research & Policy .

The review criteria for methodological/theoretical manuscripts include:

• Addresses an important topic
• Adequacy of the literature
• Soundness of the argument
• Originality of the suggestions

Mixed methods prevalence manuscripts and research notes manuscripts will reviewed in accordance to the relevant criteria for empirical or methodological/theoretical manuscripts. Commentaries will be reviewed by the editors. The editors reserve the right to edit for format and style commentary manuscripts submitted to JMMR.

Special Issue proposals must be reviewed and approved by the editors. Manuscripts proposed for a JMMR special issue will go through the same peer review process and judged by the same criteria as other manuscripts. You can find more about submitting a Special Issue proposal in the October 2018 editorial .

Obtaining English-Language help for your Submission:

Authors who would like to refine the use of English in their manuscripts might consider using the services of a professional English-language editing company. We highlight some of these companies at http://www.sagepub.com/journalgateway/engLang.htm .

Please be aware that Sage has no affiliation with these companies and makes no endorsement of them. An author's use of these services in no way guarantees that his or her submission will ultimately be accepted. Any arrangement an author enters into will be exclusively between the author and the particular company, and any costs incurred are the sole responsibility of the author.

Data Sharing:

At Sage, we are committed to facilitating openness, transparency and reproducibility of research. Where relevant, The Journal encourages authors to share their research data in a suitable public repository subject to ethical considerations and where data is included, to add a data accessibility statement in their manuscript file. Authors should also follow data citation principles. For more information please visit the Sage Author Gateway , which includes information about Sage’s partnership with the data repository Figshare.

PubMed Depositing

JMMR does not automatically deposit articles to PubMed, but JMMR and Sage do absolutely allow authors the right to post their final accepted, pre-published version of their manuscript (not the final PDF) to PubMed Central. Our policy permits authors to make their papers publicly available in this manner immediately after publication—no embargo.

• Read Online
• Sample Issues
• Current Issue
• Email Alert
• Permissions
• Foreign rights
• Reprints and sponsorship
• Advertising

Individual Subscription, E-access

Individual Subscription, Print Only

Institutional Subscription, E-access

Institutional Subscription, Print Only

Institutional Subscription, Combined (Print & E-access)

Individual, Single Print Issue

Institutional, Single Print Issue

To order single issues of this journal, please contact SAGE Customer Services at 1-800-818-7243 / 1-805-583-9774 with details of the volume and issue you would like to purchase.

• Research article
• Open access
• Published: 24 September 2018

A mixed methods case study exploring the impact of membership of a multi-activity, multicentre community group on social wellbeing of older adults

• Gabrielle Lindsay-Smith   ORCID: orcid.org/0000-0003-3864-1412 1 ,
• Grant O’Sullivan 1 ,
• Rochelle Eime 1 , 2 ,
• Jack Harvey 1 , 2 &
• Jannique G. Z. van Uffelen 1 , 3  

BMC Geriatrics volume  18 , Article number:  226 ( 2018 ) Cite this article

69k Accesses

27 Citations

12 Altmetric

Metrics details

Social wellbeing factors such as loneliness and social support have a major impact on the health of older adults and can contribute to physical and mental wellbeing. However, with increasing age, social contacts and social support typically decrease and levels of loneliness increase. Group social engagement appears to have additional benefits for the health of older adults compared to socialising individually with friends and family, but further research is required to confirm whether group activities can be beneficial for the social wellbeing of older adults.

This one-year longitudinal mixed methods study investigated the effect of joining a community group, offering a range of social and physical activities, on social wellbeing of adults with a mean age of 70. The study combined a quantitative survey assessing loneliness and social support ( n  = 28; three time-points, analysed using linear mixed models) and a qualitative focus group study ( n  = 11, analysed using thematic analysis) of members from Life Activities Clubs Victoria, Australia.

There was a significant reduction in loneliness ( p  = 0.023) and a trend toward an increase in social support ( p  = 0.056) in the first year after joining. The focus group confirmed these observations and suggested that social support may take longer than 1 year to develop. Focus groups also identified that group membership provided important opportunities for developing new and diverse social connections through shared interest and experience. These connections were key in improving the social wellbeing of members, especially in their sense of feeling supported or connected and less lonely. Participants agreed that increasing connections was especially beneficial following significant life events such as retirement, moving to a new house or partners becoming unwell.

Conclusions

Becoming a member of a community group offering social and physical activities may improve social wellbeing in older adults, especially following significant life events such as retirement or moving-house, where social network changes. These results indicate that ageing policy and strategies would benefit from encouraging long-term participation in social groups to assist in adapting to changes that occur in later life and optimise healthy ageing.

Peer Review reports

Ageing population and the need to age well

Between 2015 and 2050 it is predicted that globally the number of adults over the age of 60 will more than double [ 1 ]. Increasing age is associated with a greater risk of chronic illnesses such as cardio vascular disease and cancer [ 2 ] and reduced functional capacity [ 3 , 4 ]. Consequently, an ageing population will continue to place considerable pressure on the health care systems.

However, it is also important to consider the individuals themselves and self-perceived good health is very important for the individual wellbeing and life-satisfaction of older adults [ 5 ]. The terms “successful ageing” [ 6 ] and “healthy ageing” [ 5 ] have been used to define a broader concept of ageing well, which not only includes factors relating to medically defined health but also wellbeing. Unfortunately, there is no agreed definition for what exactly constitutes healthy or successful ageing, with studies using a range of definitions. A review of 28 quantitative studies found that successful ageing was defined differently in each, with the majority only considering measures of disability or physical functioning. Social and wellbeing factors were included in only a few of the studies [ 7 ].

In contrast, qualitative studies of older adults’ opinions on successful ageing have found that while good physical and mental health and maintaining physical activity levels are agreed to assist successful ageing, being independent or doing something of value, acceptance of ageing, life satisfaction, social connectedness or keeping socially active were of greater importance [ 8 , 9 , 10 ].

In light of these findings, the definition that is most inclusive is “healthy ageing” defined by the World Health Organisation as “the process of developing and maintaining the functional ability (defined as a combination of intrinsic capacity and physical and social environmental characteristics), that enables well-being in older age” (p28) [ 5 ].This definition, and those provided in the research of older adults’ perceptions of successful ageing, highlight social engagement and social support as important factors contributing to successful ageing, in addition to being important social determinants of health [ 11 , 12 ].

Social determinants of health, including loneliness and social support, are important predictors of physical, cognitive and mental health and wellbeing in adults [ 12 ] and older adults [ 13 , 14 , 15 ]. Loneliness is defined as a perception of an inadequacy in the quality or quantity of one’s social relationships [ 16 ]. Social support, has various definitions but generally it relates to social relationships that are reciprocal, accessible and reliable and provide any or a combination of supportive resources (e.g. emotional, information, practical) and can be measured as perceived or received support [ 17 ]. These types of social determinants differ from those related to inequality (health gap social determinants) and are sometimes referred to as ‘social cure’ social determinants [ 11 ]. They will be referred to as ‘social wellbeing’ outcome measures in this study.

Unfortunately, with advancing age, there is often diminishing social support, leading to social isolation and loneliness [ 18 , 19 ]. Large nationally representative studies of adults and older adults reported that social activity predicted maintenance or improvement of life satisfaction as well as physical activity levels [ 20 ], however older adults spent less time in social activity than middle age adults.

Social wellbeing and health

A number of longitudinal studies have found that social isolation for older adults is a significant predictor of mortality and institutionalisation [ 21 , 22 , 23 ]. A meta-analysis by Holt-Lunstadt [ 12 ] reported that social determinants of health, including social integration and social support (including loneliness and lack of perceived social support) to be equal to, or a greater risk to mortality as common behavioural risk factors such as smoking, physical inactivity and obesity. Loneliness is independently associated with poor physical and mental health in the general population, and especially in older adults [ 13 , 14 , 15 ]. Adequate perceived social support has also been consistently associated with improved mental and physical health in both general and older adults [ 20 , 24 , 25 , 26 , 27 , 28 , 29 ]. The mechanism suggested for this association is that social support buffers the negative impacts of stressful situations and life events [ 30 ]. The above research demonstrates the benefit of social engagement for older adults; in turn this highlights the importance of strategies that reduce loneliness and improve social support and social connectedness for older adults.

Socialising in groups seems to be especially important for the health and wellbeing of older adults who may be adjusting to significant life events [ 26 , 31 , 32 , 33 ]. This is sometimes referred to as social engagement or social companionship [ 26 , 30 , 31 ]. It seems that the mechanism enabling such health benefits with group participation is through strengthening of social identification, which in turn increases social support [ 31 , 34 , 35 ]. Furthermore, involvement in community groups can be a sustainable strategy to reduce loneliness and increase social support in older adults, as they are generally low cost and run by volunteers [ 36 , 37 , 38 , 39 ].

Despite the demonstrated importance of social factors for successful ageing and the established risk associated with reduced social engagement as people age, few in-depth studies have longitudinally investigated the impact of community groups on social wellbeing. For example, a non-significant increase in social support and reduction in depression was found in a year-long randomised controlled trial conducted in senior centres in Norway with lonely older adults in poor physical and mental health [ 37 ]. Some qualitative studies have reported that community groups and senior centres can contribute to fun and socialisation for older adults, however social wellbeing was not the primary focus of the studies [ 38 , 40 , 41 ]. Given that social wellbeing is a broad and important area for the health and quality of life in older adults, an in-depth study is warranted to understand how it can be maximised in older adults. This mixed methods case study of an existing community aims to: i) examine whether loneliness and social support of new members of Life Activities Clubs (LACs) changes in the year after joining and ii) conduct an in-depth exploration of how social wellbeing changes in new and longer-term members of LACs.

A mixed methods study was chosen as the design for this research to enable an in-depth exploration of how loneliness and social support may change as a result of joining a community group. A case study was conducted using a concurrent mixed-methods design, with a qualitative component giving context to the quantitative results. Where the survey focused on the impact of group membership on social support and loneliness, the focus groups were an open discussion of the benefits in the lived context of LAC membership. The synthesis of the two sections of the study was undertaken at the time of interpretation of the results [ 42 ].

The two parts of our study were as follows:

a longitudinal survey (three time points over 1 year: baseline, 6 and 12 months). This part of the study formed the quantitative results;

a focus group study of members of the same organisation (qualitative).

Ethics approval to conduct this study was obtained from the Victoria University Human Research Ethics Committee (HRE14–071 [survey] and HRE15–291 [focus groups]) All participants provided informed consent to partake in the study prior to undertaking the first survey or focus group.

Setting and participants

Life activities clubs victoria.

Life Activities Clubs Victoria (LACVI) is a large not-for-profit group with 23 independently run Life Activities Clubs (LACs) based in both rural and metropolitan Victoria. It has approximately 4000 members. The organisation was established to assist in providing physical, social and recreational activities as well as education and motivational support to older adults managing significant change in their lives, especially retirement.

Eighteen out of 23 LAC clubs agreed to take part in the survey study. During the sampling period from May 2014 to December 2016, new members from the participating clubs were given information about the study and invited to take part. Invitations took place in the form of flyers distributed with new membership material.

Inclusion/ exclusion criteria

Community-dwelling older adults who self-reported that they could walk at least 100 m and who were new members to LACVI and able to complete a survey in English were eligible to participate. New members were defined as people who had never been members of LACVI or who had not been members in the last 2 years.

To ensure that the cohort of participants were of a similar functional level, people with significant health problems limiting them from being able to walk 100 m were excluded from participating in the study.

Once informed consent was received, the participants were invited to complete a self-report survey in either paper or online format (depending on preference). This first survey comprised the baseline data and the same survey was completed 6 months and 12 months after this initial time point. Participants were sent reminders if they had not completed each survey more than 2 weeks after each was delivered and then again 1 week later.

Focus groups

Two focus groups (FGs) were conducted with new and longer-term members of LACs. The first FG ( n  = 6) consisted of members who undertook physical activity in their LAC (e.g. walking groups, tennis, cycling). The second FG ( n  = 5) consisted of members who took part in activities with a non-physical activity (PA) focus (e.g. book groups, social groups, craft or cultural groups). LACs offer both social and physical activities and it was important to the study to capture both types of groups, but they were kept separate to assist participants in feeling a sense of commonality with other members and improving group dynamic and participation in the discussions [ 43 ]. Of the people who participated in the longitudinal survey study, seven also participated in the FGs.

The FG interviews were facilitated by one researcher (GLS) and notes around non-verbal communication, moments of divergence and convergence amongst group members, and other notable items were taken by a second researcher (GOS). Both researchers wrote additional notes after the focus groups and these were used in the analysis of themes. Focus groups were recorded and later transcribed verbatim by a professional transcriptionist, including identification of each participant speaking. One researcher (GLS) reviewed each transcription to check for any errors and made any required modifications before importing the transcriptions into NVivo for analysis. The transcriber identified each focus group participant so themes for individuals or other age or gender specific trends could be identified.

Dependent variables

• Social support

Social support was assessed using the Duke–UNC Functional Social support questionnaire [ 44 ]. This scale specifically measures participant perceived functional social support in two areas; i) confidant support (5 questions; e.g. chances to talk to others) and ii) affective support (3 questions; e.g. people who care about them). Participants rated each component of support on a 5-item likert scale between ‘much less than I would like’ (1 point) to ‘as much as I would like’ (5 points). The total score used for analysis was the mean of the eight scores (low social support = 1, maximum social support = 5). Construct validity, concurrent validity and discriminant validity are acceptable for confidant and affective support items in the survey in the general population [ 44 ].

Loneliness was measured using the de Jong Gierveld and UCLA-3 item loneliness scales developed for use in many populations including older adults [ 45 ]. The 11-item de Jong Gierveld loneliness scale (DJG loneliness) [ 46 ] is a multi-dimensional measure of loneliness and contains five positively worded and six negatively worded items. The items fall into four subscales; feelings of severe loneliness, feelings connected with specific problem situations, missing companionship, feelings of belongingness. The total score is the sum of the items scores (i.e. 11–55): 11 is low loneliness and 55 is severe loneliness. Self-administered versions of this scale have good internal consistency (> = 0.8) and inter-item homogeneity and person scalability that is as good or better than when conducted as face-to face interviews. The validity and reliability for the scale is adequate [ 47 ]. The UCLA 3-item loneliness scale consists of three questions about how often participants feel they lack companionship, feel left out and feel isolated. The responses are given on a three-point scale ranging from hardly ever (1) to often (3). The final score is the sum of these three items with the range being from lowest loneliness (3) to highest loneliness (9). Reliability of the scale is good, (alpha = 0.72) as are discriminant validity and internal consistency [ 48 ]. The scale is commonly used to measure loneliness with older adults ([ 49 ] – review), [ 50 , 51 ].

Sociodemographic variables

The following sociodemographic characteristics were collected in both the survey and the focus groups: age, sex, highest level of education, main life occupation [ 52 ], current employment, ability to manage on income available, present marital status, country of birth, area of residence [ 53 ]. They are categorised as indicated in Table  2 .

Health variables

The following health variables were collected: Self-rated general health (from SF-12) [ 54 ] and Functional health (ability to walk 100 m- formed part of the inclusion criteria) [ 55 ]. See Table 2 for details about the categories of these variables.

The effects of becoming a member on quantitative outcome variables (i.e. Social support, DJG loneliness and UCLA loneliness) were analysed using linear mixed models (LMM). LMM enabled testing for the presence of intra-subject random effects, or equivalently, correlation of subjects’ measures over time (baseline, 6-months and 12 months). Three correlation structures were examined: independence (no correlation), compound symmetry (constant correlation of each subjects’ measures over the three time points) and autoregressive (correlation diminishing with increase in spacing in time). The best fitting correlation structure was compound symmetry; this is equivalent to a random intercept component for each subject. The LMM incorporated longitudinal trends over time, with adjustment for age as a potential confounder. Statistical analyses were conducted using SPSS for windows (v24).

UCLA loneliness and social support residuals were not normally distributed and these scales were Log10 transformed for statistical analysis.

Analyses were all adjusted for age, group attendance (calculated as average attendance at 6 and 12 months) and employment status at baseline (Full-time, Part-time, not working).

Focus group transcripts were analysed using thematic analysis [ 56 , 57 ], a flexible qualitative methodology that can be used with a variety of epistemologies, approaches and analysis methods [ 56 ]. The transcribed data were analysed using a combination of theoretical and inductive thematic analysis [ 56 ]. It was theorised that membership in a LAC would assist with social factors relating to healthy ageing [ 5 ], possibly through a social identity pathway [ 58 ], although we wanted to explore this. Semantic themes were drawn from these codes in order to conduct a pragmatic evaluation of the LACVI programs [ 56 ]. Analytic rigour in the qualitative analysis was ensured through source and analyst triangulation. Transcriptions were compared to notes taken during the focus groups by the researchers (GOS and GLS). In addition, Initial coding and themes (by GLS) were checked by a second researcher (GOS) and any disagreements regarding coding and themes were discussed prior to finalisation of codes and themes [ 57 ].

Sociodemographic and health characteristics of the 28 participants who completed the survey study are reported in Table  1 . The mean age of the participants was 66.9 and 75% were female. These demographics are representative of the entire LACVI membership. Education levels varied, with 21% being university educated, and the remainder completing high school or technical certificates. Two thirds of participants were not married. Some sociodemographic characteristics changed slightly at 6 and 12 months, mainly employment (18% in paid employment at baseline and 11% at 12-months) and ability to manage on income (36% reporting trouble managing on their income at baseline and 46% at 12 months). Almost 90% of the participants described themselves as being in good-excellent health.

Types of activities

There were a variety of types of activities that participants took part in: physical activities such as walking groups ( n  = 7), table tennis ( n  = 5), dancing class ( n  = 2), exercise class ( n  = 1), bowls ( n  = 2), golf ( n  = 3), cycling groups ( n  = 1) and non-physical leisure activities such as art and literature groups ( n  = 5), craft groups ( n  = 5), entertainment groups ( n  = 12), food/dine out groups ( n  = 18) and other sedentary leisure activities (e.g. mah jong, cards),( n  = 4). A number of people took part in more than one activity.

Frequency of attendance at LACVI and changes in social wellbeing

At six and 12 months, participants indicated how many times in the last month they attended different types of activities at their LAC. Most participants maintained the same frequency of participation over both time points. Only four people participated more frequently at 12 than at 6 months and nine reduced participation levels. The latter group included predominantly those who reduced from more than two times per week at 6 months to 2×/week at 6 months to one to two times per week ( n  = 5) or less than one time per week ( n  = 2) at 12 months. Average weekly club attendance at six and 12 months was included as a covariate in the statistical model.

Outcome measures

Overall, participants reported moderate social support and loneliness levels at baseline (See Table 2 ). Loneliness, as measured by both scales, reduced significantly over time. There was a significant effect of time on the DJG loneliness scores (F (2, 52) = 3.83, p  = 0.028), with Post-Hoc analysis indicating a reduction in DJG loneliness between baseline and 12 months ( p  = 0.008). UCLA loneliness scores (transformed variable) also changed significantly over time (F (2, 52) = 4.08, p  = 0.023). Post hoc tests indicated a reduction in UCLA loneliness between baseline and 6 months ( p  = 0.007). There was a small non-significant increase in social support (F (2, 53) =2.88, p  = 0.065) during the first year of membership (see Table 2 and Figs. 1 and 2 ).

DJG loneliness for all participants over first year of membership at LAC club ( n  = 28).

*Represents significant difference compared to baseline ( p  < 0.01)

UCLA loneliness score for all participants over first year of membership at LAC club ( n  = 28).

*Indicates log values of the variable at 6-months were significantly different from baseline ( p  < 0.01)

In total, 11 participants attended the two focus groups, six people who participated in PA clubs (four women) and five who participated in social clubs (all women). All focus group participants were either retired ( n  = 9) or semi-retired ( n  = 2). The mean age of participants was 67 years (see Table 2 for further details). Most of the participants (82%) had been members of a LAC for less than 2 years and two females in the social group had been members of LAC clubs for 5 and 10 years respectively.

Analysis of the focus group transcripts identified two themes relating to social benefits of group participation; i) Social resources and ii) Social wellbeing (see Fig. 3 ). Group discussion suggested that membership of a LAC provides access to more social resources through greater and diverse social contact and opportunity. It is through this improvement in social resources that social wellbeing may improve.

Themes arising from focus group discussion around the benefits of LAC membership

Social resources

The social resources theme referred to an increase in the availability and variety of social connections that resulted from becoming a member of a LAC. The social nature of the groups enabled an expansion and diversification of members’ social network and improved their sense of social connectedness. There was widespread agreement in both the focus groups that significant life events, especially retirement, illness or death of spouse and moving house changes one’s social resources. Membership of the LAC had benefits especially at these times and these events were often motivators to join such a club. Most participants found that their social resources declined after retirement and even felt that they were grieving for the loss of their work.

“ I just saw work as a collection of, um, colleagues as opposed to friends. I had a few good friends there. Most were simply colleagues or acquaintances …. [interviewer- Mmm.] ..Okay, you’d talk to them every day. You’d chatter in the kitchen, oh, pass banter back and forth when things are busy or quiet, but... Um, in terms of a friendship with those people, like going to their home, getting to know them, doing other things with them, very few. But what I did miss was the interaction with other people. It had simply gone….. But, yeah, look, that, the, yeah, that intervening period was, oh, a couple of months. That was a bit tough…. But in that time the people in LAC and the people in U3A…. And the other dance group just drew me into more things. Got to know more people. So once again, yeah, reasonable group of acquaintances.” (Male, PAFG)

Group members indicated general agreement with these two responses, however one female found she had a greater social life following retirement due to the busy nature of her job.

Within the social resources theme, three subthemes were identified, i) Opportunity for social connectedness, ii) Opportunity for friendships, and iii) Opportunity for social responsibility/leadership . Interestingly, these subthemes were additional to the information gathered in the survey. This emphasises the power of the inductive nature of the qualitative exploration employed in the focus groups to broaden the knowledge in this area.

The most discussed and expanded subtheme in both focus groups was Opportunity for social connectedness , which arose through developing new connections, diversifying social connections, sharing interests and experiences with others and peer learning. Participants in both focus groups stated that being a member of LAC facilitated their socialising and connecting with others to share ideas, skills and to do activities with, which was especially important through times of significant life events. Furthermore, participants in each of the focus groups valued developing diverse connections:

“ Yeah, I think, as I said, I finished up work and I, and I had more time for wa-, walking. So I think a, in meeting, in going to this group which, I saw this group of women but then someone introduced me to them. They were just meeting, just meeting a new different set of people, you know? As I said, my work people and these were just a whole different group of women, mainly women. There’s not many men. [Interviewer: Yes.]….. Although our leader is a man, which is ironic and is about, this man out in front and there’s about 20 women behind him, but, um, so yeah, and people from different walks of life and different nationalities there which I never knew in my work life, so yeah. That’s been great. So from that goes on other things, you know, you might, uh, other activities and, yeah, people for coffee and go to the pictures or something, yeah. That’s great.” (Female, PAFG)

Simply making new connections was the most widely discussed aspect related to the opportunity for social connectedness subtheme, with all participants agreeing that this was an important benefit of participation in LAC groups.

“Well, my experience is very similar to everybody else’s…….: I, I went from having no social life to a social life once I joined a group.” (Female, PAFG)

There was agreement in both focus groups that these initial new connections made at a LAC are strengthened through development of deeper personal connections with others who have similar demographics and who are interested in the same activities. This concurs with the Social Identity Theory [ 58 ] discussed previously.

“and I was walking around the lake in Ballarat, like wandering on my own. I thought, This is ridiculous. I mean, you’ve met all those groups of women coming the opposite way, so I found out what it was all about, so I joined, yeah. So that’s how I got into that.[ Interviewer: Yeah.] Basically sick of walking round the lake on my own. [Interviewer: Yeah, yeah.] So that’s great. It’s very social and they have coffee afterwards which is good.” (female, PAFG)

The subtheme Opportunity for development of friendships describes how, for some people, a number of LAC members have progressed from being just initial social connections to an established friendship. This signifies the strength of the connections that may potentially develop through LAC membership. Some participants from each group mentioned friendships developing, with slightly more discussion of this seen in the social group.

“we all have a good old chat, you know, and, and it’s all about friendship as well.” (female, SocialFG)

The subtheme Opportunity for social responsibility or leadership was mentioned by two people in the active group, however it was not brought up in the social group. This opportunity for leadership is linked with the development of a group identity and desiring to contribute meaningfully to a valued group.

“with our riding group, um, you, a leader for probably two rides a year so you’ve gotta prepare for it, so some of them do reccie rides themselves, so, um, and also every, uh, so that’s something that’s, uh, a responsibility.” (male, PAFG)

Social wellbeing

The social resources described above seem to contribute to a number of social, wellbeing outcomes for participants. The sub themes identified for Social wellbeing were , i) Increased social support, ii) Reduced loneliness, iii) Improved home relationships and iv) Improved social skills.

Increased social support

Social support was measured quantitatively in the survey (no significant change over time for new members) and identified as a benefit of LAC membership during the focus group discussions. However, only one of the members of the active group mentioned social support directly.

‘it’s nice to be able to pick up the phone and share your problem with somebody else, and that’s come about through LAC. ……‘Cos before that it was through, with my family (female, PAFG)

There was some agreement amongst participants of the PA group that they felt this kind of support may develop in time but most of them had been members for less than 2 years.

“[Interviewer: Yeah. Does anyone else have that experience? (relating to above quote)]” There is one lady but she’s actually the one that I joined with anyway. [Interviewer: Okay.] But I, I feel there are others that are definitely getting towards that stage. It’s still going quite early days. (female1, PAFG) [Interviewer: I guess it’s quite early for some of you, yeah.] “yeah” (female 2, PAFG)

Social support through sharing of skills was mentioned by one participant in the social group also, with agreement indicated by most of the others in the social focus group.

Discussion in the focus groups also touched on the subthemes Reduced loneliness and Improved home relationships, which were each mentioned by one person. And focus groups also felt that group membership Improved social skills through opening up and becoming more approachable (male, PAFG) or enabling them to become more accepting of others’ who are different (general agreement in Social FG).

This case study integrated results from a one-year longitudinal survey study and focus group discussions to gather rich information regarding the potential changes in social wellbeing that older adults may experience when joining community organisations offering group activities. The findings from this study indicate that becoming a member of such a community organisation can be associated with a range of social benefits for older adults, particularly related to reducing loneliness and maintaining social connections.

Joining a LAC was associated with a reduction in loneliness over 1 year. This finding is in line with past group-intervention studies where social activity groups were found to assist in reducing loneliness and social isolation [ 49 ]. This systematic review highlighted that the majority of the literature explored the effectiveness of group activity interventions for reducing severe loneliness or loneliness in clinical populations [ 49 ]. The present study extends this research to the general older adult population who are not specifically lonely and reported to be of good general health, rather than a clinical focus. Our findings are in contrast to results from an evaluation of a community capacity-building program aimed at reducing social isolation in older adults in rural Australia [ 59 ]. That program did not successfully reduce loneliness or improve social support. The lack of change from pre- to post-program in that study was reasoned to be due to sampling error, unstandardised data collection, and changes in sample characteristics across the programs [ 59 ]. Qualitative assessment of the same program [ 59 ] did however suggest that participants felt it was successful in reducing social isolation, which does support our findings.

Changes in loneliness were not a main discussion point of the qualitative component of the current study, however some participants did express that they felt less lonely since joining LACVI and all felt they had become more connected with others. This is not so much of a contrast in results as a potential situational issue. The lack of discussion of loneliness may have been linked to the common social stigma around experiencing loneliness outside certain accepted circumstances (e.g. widowhood), which may lead to underreporting in front of others [ 45 ].

Overall, both components of the study suggest that becoming a member of an activity group may be associated with reductions in loneliness, or at least a greater sense of social connectedness. In addition to the social nature of the groups and increased opportunity for social connections, another possible link between group activity and reduced loneliness is an increased opportunity for time out of home. Previous research has found that more time away from home in an average day is associated with lower loneliness in older adults [ 60 ]. Given the significant health and social problems that are related to loneliness and social isolation [ 13 , 14 , 15 ], the importance of group involvement for newly retired adults to prevent loneliness should be advocated.

In line with a significant reduction in loneliness, there was also a trend ( p  = 0.056) toward an increase in social support from baseline to 12 months in the survey study. Whilst suggestive of a change, it is far less conclusive than the findings for loneliness. There are a number of possible explanations for the lack of statistically significant change in this variable over the course of the study. The first is the small sample size, which would reduce the statistical power of the study. It may be that larger studies are required to observe changes in social support, which are possibly only subtle over the course of 1 year. This idea is supported by a year-long randomised controlled trial with 90 mildly-depressed older adults who attended senior citizen’s club in Norway [ 37 ]. The study failed to see any change in general social support in the intervention group compared to the control over 1 year. Additional analysis in that study suggested that people who attended the intervention groups more often, tended to have greater increases in SS ( p  = 0.08). The researchers stated that the study suffered from significant drop-out rates and low power as a result. In this way, it was similar to our findings and suggests that social support studies require larger numbers than we were able to gain in this early exploratory study. Another possible reason for small changes in SS in the current study may be the type of SS measured. The scale used gathered information around functional support or support given to individuals in times of need. Maybe it is not this type of support that changes in such groups but more specific support such as task-specific support. It has been observed in other studies and reviews that task-specific support changes as a result of behavioural interventions (e.g. PA interventions) but general support does not seem to change in the time frames often studied [ 61 , 62 , 63 ].

There were many social wellbeing benefits such as increased social connectivity identified in focus group discussion, but the specific theme of social support was rarely mentioned. It may be that general social support through such community groups may take longer than 1 year to develop. There is evidence that strong group ties are sequentially positively associated between social identification and social support [ 34 ], suggesting that the connections formed through the groups may lead increased to social support from group members in the future. This is supported by results from the focus group discussions, where one new member felt she could call on colleagues she met in her new group. Other new members thought it was too soon for this support to be available, but they could see the bonds developing.

Other social wellbeing changes

In addition to social support and loneliness that were the focus of the quantitative study, the focus group discussions uncovered a number of other benefits of group membership that were related to social wellbeing (see Fig. 3 ). The social resources theme was of particular interest because it reflected some of the mechanisms that appeared enable social wellbeing changes as a result of being a member of a LAC but were not measured in the survey. The main social resources relating to group membership that were mentioned in the focus groups were social connectedness, development of friendships and opportunity for social responsibility or leadership. As mentioned above, there was wide-spread discussion within the focus groups of the development of social connections through the clubs. Social connectedness is defined as “the sense of belonging and subjective psychological bond that people feel in relation to individuals and groups of others.” ([ 25 ], pp1). As well as being an important predecessor of social support, greater social connectedness has been found to be highly important for the health of older adults, especially cognitive and mental health [ 26 , 32 , 34 , 35 , 64 ]. One suggested theory for this health benefit is that connections developed through groups that we strongly identify with are likely to be important for the development of social identity [ 34 ], defined by Taifel as: “knowledge that [we] belong to certain social groups together with some emotional and value significance to [us] of this group membership” (Tajfel, 1972, p. 31 in [ 58 ] p 2). These types of groups to which we identify may be a source of “personal security, social companionship, emotional bonding, intellectual stimulation, and collaborative learning and……allow us to achieve goals.” ([ 58 ] p2) and an overall sense of self-worth and wellbeing. There was a great deal of discussion relating to the opportunity for social connectedness derived through group membership being particularly pertinent following a significant life event such as moving to a new house or partners becoming unwell or dying and especially retirement. This change in their social circumstance is likely to have triggered the need to renew their social identity by joining a community group. Research with university students has shown that new group identification can assist in transition for university students who have lost their old groups of friends because of starting university [ 65 ]. In an example relevant to older adults, maintenance or increase in number of group memberships at the time of retirement reduced mortality risk 8 years later compared to people who reduce their number of group activities in a longitudinal cohort study [ 66 ]. This would fit with the original Activity Theory of ageing; whereby better ageing experience is achieved when levels of social participation are maintained, and role replacement occurs when old roles (such as working roles) must be relinquished [ 67 ]. These connections therefore appear to assist in maintaining resilience in older adults defined as “the ability to maintain or improve a level of functional ability (a combination of intrinsic physical and mental capacity and environment) in the face of adversity” (p29, [ 5 ]). Factors that were mentioned in the focus groups as assisting participants in forming connections with others were shared interest, learning from others, and a fun and accepting environment. It was not possible to assess all life events in the survey study. However, since the discussion from the focus groups suggested this to be an important motivator for joining clubs and potentially a beneficial time for joining them, it would be worth exploring in future studies.

Focus group discussion suggested that an especially valuable time for joining such clubs was around retirement, to assist with maintaining social connectivity. The social groups seem to provide social activity and new roles for these older adults at times of change. It is not necessarily important for all older adults but maybe these ones identify themselves as social beings and therefore this maintenance of social connection helps to continue their social role. Given the suggested importance of social connectivity gained through this organisation, especially at times of significant life events, it would valuable to investigate this further in future and consider encouragement of such through government policy and funding. The majority of these types of clubs exist for older adults in general, but this study emphasises the need for groups such as these to target newly retired individuals specifically and to ensure that they are not seen as ‘only for old people’.

Strengths and limitations

The use of mixed –methodologies, combining longitudinal survey study analysed quantitatively, with a qualitative exploration through focus group discussions and thematic analysis, was a strength of the current study. It allowed the researchers to not only examine the association between becoming a member of a community group on social support and loneliness over an extended period, but also obtain a deeper understanding of the underlying reasons behind any associations. Given the variability of social support definitions in research [ 17 ] and the broad area of social wellbeing, it allowed for open exploration of the topic, to understand associations that may exist but would have otherwise been missed. Embedding the research in an existing community organisation was a strength, although with this also came some difficulties with recruitment. Voluntary coordination of the community groups meant that informing new members about the study was not always feasible or a priority for the volunteers. In addition, calling for new members was innately challenging because they were not yet committed to the club fully. This meant that so some people did not want to commit to a year-long study if they were not sure how long they would be a member of the club. This resulted in slow recruitment and a resulting relatively low sample size and decreased power to show significant statistical differences, which is a limitation of the present study. However, the use of Linear Mixed Models for analysis of the survey data was a strength because it was able to include all data in the analyses and not remove participants if one time point of data was missing, as repeated measures ANOVAs would do. The length of the study (1 year) is another strength, especially compared to previous randomised controlled studies that are typically only 6–16 weeks in length. Drop-out rate in the current study is very low and probably attributable to the benefits of working with long-standing organisations.

The purpose of this study was to explore in detail whether there are any relationships between joining existing community groups for older adults and social wellbeing. The lack of existing evidence in the field meant that a small feasibility-type case study was a good sounding-board for future larger scale research on the topic, despite not being able to answer questions of causality. Owing to the particularistic nature of case studies, it can also be difficult to generalise to other types of organisations or groups unless there is a great deal of similarity between them [ 68 ]. There are however, other types of community organisations in existence that have a similar structure to LACVI (Seniors centres [ 36 , 40 ], Men’s Sheds [ 38 ], University of the Third Age [ 34 , 69 ], Japanese salons [ 70 , 71 ]) and it may be that the results from this study are transferable to these also. This study adds to the literature around the benefits of joining community organisations that offer social and physical activities for older adults and suggests that this engagement may assist with reducing loneliness and maintaining social connection, especially around the time of retirement.

Directions for future research

Given that social support trended toward a significant increase, it would be useful to repeat the study on a larger scale in future to confirm this. Either a case study on a similar but larger community group or combining a number of community organisations would enable recruitment of more participants. Such an approach would also assist in assessing the generalisability of our findings to other community groups. Given that discussions around social benefits of group membership in the focus groups was often raised in conjunction with the occurrence of significant life events, it would be beneficial to include a significant life event scale in any future studies in this area. The qualitative results also suggest that it would be useful to investigate whether people who join community groups in early years post retirement gain the same social benefits as those in later stages of retirement. Studies investigating additional health benefits of these community groups such as physical activity, depression and general wellbeing would also be warranted.

With an ageing population, it is important to investigate ways to enable older adults to age successfully to ensure optimal quality of life and minimisation of health care costs. Social determinants of health such as social support, loneliness and social contact are important contributors to successful ageing through improvements in cognitive health, quality of life, reduction in depression and reduction in mortality. Unfortunately, older adults are at risk of these social factors declining in older age and there is little research investigating how best to tackle this. Community groups offering a range of activities may assist by improving social connectedness and social support and reducing loneliness for older adults. Some factors that may assist with this are activities that encourage sharing interests, learning from others, and are conducted in a fun and accepting environment. Such groups may be particularly important in developing social contacts for newly retired individuals or around other significant life events such as moving or illness of loved ones. In conclusion, ageing policy and strategies should emphasise participation in community groups especially for those recently retired, as they may assist in reducing loneliness and increasing social connections for older adults.

Abbreviations

Focus group

Life Activities Club

Life Activities Clubs Victoria

Linear mixed model

Physical activity

World Health Organisation

United Nations. In: P.D. Department of Economic and Social Affairs, editor. World Population Ageing 2015. New York: United Nations; 2015.

Google Scholar  

World Health Organisation. Global Health and Ageing. 2011 [cited 2014 March 25]; Available from: http://www.who.int/ageing/publications/global_health/en/ .

Balogun JA, et al. Age-related changes in balance performance. Disabil Rehabil. 1994;16(2):58–62.

Article   CAS   Google Scholar  

Singh MAF. Exercise comes of age: rationale and recommendations for a geriatric exercise prescription. J Gerontol Ser A Biol Med Sci. 2002;57(5):M262–82.

Article   Google Scholar  

World Health Organisation. World report on ageing and health. Geneva: World Health Organisation; 2015.

Rowe JW, Kahn RL. Successful Aging1. The Gerontologist. 1997;37(4):433–40.

Depp CA, Jeste DV. Definitions and predictors of successful aging: a comprehensive review of larger quantitative studies. Am J Geriatr Psychiatry. 2006;14(1):6–20.

Song M, Kong E-H. Older adults’ definitions of health: a metasynthesis. Int J Nurs Stud. 2015;52(6):1097–106.

Tate RB, Lah L, Cuddy TE. Definition of successful aging by elderly Canadian males: the Manitoba follow-up study. The Gerontologist. 2003;43(5):735–44.

Phelan EA, et al. Older Adults’ views of “successful aging”—how do they compare with Researchers’ definitions? J Am Geriatr Soc. 2004;52(2):211–6.

Haslam SA, et al. Social cure, what social cure? The propensity to underestimate the importance of social factors for health. Soc Sci Med. 2018;198:14–21.

Holt-Lunstad J, Smith TB, Layton JB. Social relationships and mortality risk: a meta-analytic review. PLoS Med. 2010;7(7):e1000316.

Richard A, et al. Loneliness is adversely associated with physical and mental health and lifestyle factors: results from a Swiss national survey. PLoS One. 2017;12(7):1–18.

Luo Y, et al. Loneliness, health, and mortality in old age: a national longitudinal study. Soc Sci Med. 2012;74(6):907–14.

Luo Y, Waite LJ. Loneliness and mortality among older adults in China. J Gerontol B Psychol Sci Soc Sci. 2014;69(4):633–45.

de Jong Gierveld J, Van Tilburg T, Dykstra PA. Loneliness and social isolation. In: Vangelisti A, Perlman D, editors. Cambridge handbook of personal relationships. Cambridge: Cambridge University Press; 2006. p. 485-500.

Williams P, Barclay L, Schmied V. Defining social support in context: a necessary step in improving research, intervention, and practice. Qual Health Res. 2004;14(7):942–60.

Valtorta N, Hanratty B. Loneliness, isolation and the health of older adults: do we need a new research agenda? J R Soc Med. 2012;105(12):518–22.

Jylhä M. Old age and loneliness: cross-sectional and longitudinal analyses in the Tampere longitudinal study on aging. Can J Aging La Revue can du vieil. 2010;23(2):157–68.

Huxhold O, Miche M, Schüz B. Benefits of having friends in older ages: differential effects of informal social activities on well-being in middle-aged and older adults. J Gerontol B Psychol Sci Soc Sci. 2014;69(3):366–75.

Berkman L, Syme S. Social networks, host resistance and mortality: a nine year follow-up study of alameda county residents. Am J Epidemiol. 1979;185(11):1070–88.

House JS, Landis KR, Umberson D. Social relationships and health. Science. 1988;241(4865):540–5.

Pynnonen K, et al. Does social activity decrease risk for institutionalization and mortality in older people? J Gerontol B Psychol Sci Soc Sci. 2012;67(6):765–74.

Broadhead WE, et al. The epidemiologic evidence for a relationship between social support and health. Am J Epidemiol. 1983;117(5):521–37.

Haslam C, et al. Social connectedness and health. Encyclopedia of geropsychology. 2017:2174–82. https://doi.org/10.1007/978-981-287-080-3_46-1 .

Haslam C, Cruwys T, Haslam SA. “The we’s have it”: evidence for the distinctive benefits of group engagement in enhancing cognitive health in aging. Soc Sci Med. 2014;120:57–66.

Uebelacker LA, et al. Social support and physical activity as moderators of life stress in predicting baseline depression and change in depression over time in the Women’s Health Initiative. Soc Psychiatry Psychiatr Epidemiol. 2013;48(12):1971–82.

Tajvar M, et al. Social support and health of older people in middle eastern countries: a systematic review. Australas J Ageing. 2013;32(2):71–8.

Dalgard OS, Bjork S, Tambs K. Social support, negative life events and mental health. Br J Psychiatry. 1995;166(1):29–34.

Cohen S, Wills TA. Stress, social support, and the buffering hypothesis. Psychol Bull. 1985;98(2):310–57.

Gilmour H. Social participation and the health and well-being of Canadian seniors. Health Rep. 2012;23(4):1B.

Glei DA, et al. Participating in social activities helps preserve cognitive function: an analysis of a longitudinal, population-based study of the elderly. Int J Epidemiol. 2005;34(4):864–71.

Lee SH, Kim YB. Which type of social activities may reduce cognitive decline in the elderly?: a longitudinal population-based study. BMC Geriatr. 2016;16(1):165.

Haslam C, et al. Group ties protect cognitive health by promoting social identification and social support. J Aging Health. 2016;28(2):244–66.

Haslam C, et al. Groups 4 health: evidence that a social-identity intervention that builds and strengthens social group membership improves mental health. J Affect Disord. 2016;194:188–95.

Bøen H. Characteristics of senior Centre users -- and the impact of a group programme on social support and late-life depression. Norsk Epidemiologi. 2012;22(2):261–9.

Bøen H, et al. A randomized controlled trial of a senior Centre group programme for increasing social support and preventing depression in elderly people living at home in Norway. BMC Geriatr. 2012;12:20.

Golding BG. Social, local, and situated: recent findings about the effectiveness of older Men’s informal learning in community contexts. Adult Educ Q. 2011;61(2):103.

Life Activities Clubs. Life Activities Clubs. About Us. . 2014 [cited 2014 January 13, 2014]; Available from: http://www.life.org.au/aboutus .

Hutchinson SL, Gallant KA. Can senior Centres be contexts for aging in third places? J Leis Res. 2016;48(1):50–68.

Millard J. The health of older adults in community activities. Work Older People. 2017;21(2):90–9.

Creswell JW, Plano-Clark VL. Designing and conducting mixed methods research. Thousand Oaks, Calif: SAGE Publications; 2007. p. c2007.

Loeb S, Penrod J, Hupcey J. Focus groups and older adults: tactics for success. J Gerontol Nurs. 2006;32(3):32–8.

PubMed   Google Scholar  

Broadhead W, et al. The Duke-UNC functional social support questionnaire: measurement of social support in family medicine patients. Med Care. 1988;26(7):709–23.

De Jong Gierveld J, Van Tilburg T. Living arrangements of older adults in the Netherlands and Italy: Coresidence values and behaviour and their consequences for loneliness. J Cross Cult Gerontol. 1999;14(1):1–24.

de Jong-Gierveld J, Kamphuls F. The development of a Rasch-type loneliness scale. Appl Psychol Meas. 1985;9(3):289–99.

Tilburg Tv, Leeuw Ed. Stability of scale quality under various data collection procedures: a mode comparison on the ‘De Jong-Gierveld loneliness scale. Int J Public Opin Res. 1991;3(1):69–85.

Hughes ME, et al. A short scale for measuring loneliness in large surveys - results from two population-based studies. Res Aging. 2004;26(6):655–72.

Dickens AP, et al. Interventions targeting social isolation in older people: a systematic review. BMC Public Health. 2011;11:647.

Hawkley LC, et al. Loneliness predicts increased blood pressure: 5-year cross-lagged analyses in middle-aged and older adults. Psychol Aging. 2010;25(1):132.

Netz Y, et al. Loneliness is associated with an increased risk of sedentary life in older Israelis. Aging Ment Health. 2013;17(1):40–7.

Australian Bureau of Statistics. Australian and New Zealand Standard Classification of Occupations, 2013, Version 1.2. Canberra: Australian Bureau of Statistics; 2013.

Australian Bureau of Statistics, Australian Standard Geographical Classification (ASGC). Canberra: Australian Bureau of Statistics; 2011.

Ware JE, Kosinski M, Keller SD. A 12-item short-form health survey - construction of scales and preliminary tests of reliability and validity. Med Care. 1996;34(3):220–33.

Sanson-Fisher RW, Perkins JJ. Adaptation and validation of the SF-36 health survey for use in Australia. J Clin Epidemiol. 51(11):961–7.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

Patton MQ. Qualitative research & evaluation methods. 4 ed. Thousand Oaks, California: Sage Publications; 2015.

Haslam SA, et al. Social identity, health and well-being: an emerging agenda for applied psychology. Appl Psychol. 2009;58(1):1–23.

Bartlett H, et al. Preventing social isolation in later life: findings and insights from a pilot Queensland intervention study. Ageing Soc. 2013;33(07):1167–89.

Petersen J, et al. Time out-of-home and cognitive, physical, and emotional wellbeing of older adults: a longitudinal mixed effects model. PLoS One. 2015;10(10):e0139643.

Lindsay Smith G, et al. The association between social support and physical activity in older adults: a systematic review. Int J Behav Nutr Phys Act. 2017;14(1):56.

Sallis JF, et al. The development of scales to measure social support for diet and exercise behaviors. Prev Med. 1987;16(6):825–36.

Oka R, King A, Young DR. Sources of social support as predictors of exercise adherence in women and men ages 50 to 65 years. Womens Health Res Gender Behav Policy. 1995;1:161–75.

CAS   Google Scholar  

Greenaway KH, et al. From “we” to “me”: group identification enhances perceived personal control with consequences for health and well-being. J Pers Soc Psychol. 2015;109(1):53–74.

Iyer A, et al. The more (and the more compatible) the merrier: multiple group memberships and identity compatibility as predictors of adjustment after life transitions. Br J Soc Psychol. 2009;48(4):707–33.

Steffens NK, et al. Social group memberships in retirement are associated with reduced risk of premature death: evidence from a longitudinal cohort study. BMJ Open. 2016;6(2):e010164.

Havighurst RJ. Successful aging. Gerontol. 1961;1:8–13.

Yin R. Case study research: design and methods. Beverly Hills, CA: Sage publishing; 1994.

Merriam SB, Kee Y. Promoting community wellbeing: the case for lifelong learning for older adults. Adult Educ Q. 2014;64(2):128–44.

Hikichi H, et al. Social interaction and cognitive decline: results of a 7-year community intervention. Alzheimers Dement: Translat Res Clin Interv. 2017;3(1):23–32.

Hikichi H, et al. Effect of a community intervention programme promoting social interactions on functional disability prevention for older adults: propensity score matching and instrumental variable analyses, JAGES Taketoyo study. J Epidemiol Community Health. 2015;69(9):905–10.

Download references

The primary author contributing to this study (GLS) receives PhD scholarship funding from Victoria University. The other authors were funded through salaries at Victoria University.

Availability of data and materials

The datasets generated and/or analysed during the current study are not publicly available due the ethics approval for this study not allowing open access to the individual participant data but are available from the corresponding author on reasonable request.

Author information

Authors and affiliations.

Institute for Health and Sport, Victoria University, Melbourne, Australia

Gabrielle Lindsay-Smith, Grant O’Sullivan, Rochelle Eime, Jack Harvey & Jannique G. Z. van Uffelen

School of Health and Life Sciences, Federation University, Ballarat, Australia

Rochelle Eime & Jack Harvey

Department of Movement Sciences, Physical Activity, Sports and Health Research Group, KU Leuven - University of Leuven, Leuven, Belgium

Jannique G. Z. van Uffelen

You can also search for this author in PubMed   Google Scholar

Contributions

GLS, RE and JVU made substantial contributions to the conception and design of the study. GLS and GOS supervised data collection for the surveys (GLS) and focus groups (GOS and GLS). GLS, GOS, RE, JH and JVU were involved in data analysis and interpretation. All authors were involved in drafting, the manuscript and approved the final version.

Corresponding author

Correspondence to Gabrielle Lindsay-Smith .

Ethics declarations

Ethics approval and consent to participate.

Ethics approval to conduct this study was obtained from the Victoria University Human Research Ethics Committee (HRE14–071 [survey] and HRE15–291 [focus groups]). All survey participants provided informed consent to partake in the study prior to undertaking the first survey by clicking ‘agree’ in the survey software (online participants) or signing a written consent form (paper participants) after they had read all participant information relating to the study. Focus group participants also completed a written consent form after reading and understanding all participant information regarding the study.

Consent for publication

Focus group participants have given permission for their quotes to be used in de-identified form prior to undertaking the research.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Lindsay-Smith, G., O’Sullivan, G., Eime, R. et al. A mixed methods case study exploring the impact of membership of a multi-activity, multicentre community group on social wellbeing of older adults. BMC Geriatr 18 , 226 (2018). https://doi.org/10.1186/s12877-018-0913-1

Download citation

Received : 19 April 2018

Accepted : 10 September 2018

Published : 24 September 2018

DOI : https://doi.org/10.1186/s12877-018-0913-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Social engagement
• Group activity

BMC Geriatrics

ISSN: 1471-2318

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Mixed Methods Research | Definition, Guide & Examples

Mixed Methods Research | Definition, Guide & Examples

Published on August 13, 2021 by Tegan George . Revised on June 22, 2023.

Mixed methods research combines elements of quantitative research and qualitative research in order to answer your research question . Mixed methods can help you gain a more complete picture than a standalone quantitative or qualitative study, as it integrates benefits of both methods.

Mixed methods research is often used in the behavioral, health, and social sciences, especially in multidisciplinary settings and complex situational or societal research.

• To what extent does the frequency of traffic accidents ( quantitative ) reflect cyclist perceptions of road safety ( qualitative ) in Amsterdam?
• How do student perceptions of their school environment ( qualitative ) relate to differences in test scores ( quantitative ) ?
• How do interviews about job satisfaction at Company X ( qualitative ) help explain year-over-year sales performance and other KPIs ( quantitative ) ?
• How can voter and non-voter beliefs about democracy ( qualitative ) help explain election turnout patterns ( quantitative ) in Town X?
• How do average hospital salary measurements over time (quantitative) help to explain nurse testimonials about job satisfaction (qualitative) ?

Table of contents

When to use mixed methods research, mixed methods research designs, advantages of mixed methods research, disadvantages of mixed methods research, other interesting articles, frequently asked questions.

Mixed methods research may be the right choice if your research process suggests that quantitative or qualitative data alone will not sufficiently answer your research question. There are several common reasons for using mixed methods research:

• Generalizability : Qualitative research usually has a smaller sample size , and thus is not generalizable. In mixed methods research, this comparative weakness is mitigated by the comparative strength of “large N,” externally valid quantitative research.
• Contextualization: Mixing methods allows you to put findings in context and add richer detail to your conclusions. Using qualitative data to illustrate quantitative findings can help “put meat on the bones” of your analysis.
• Credibility: Using different methods to collect data on the same subject can make your results more credible. If the qualitative and quantitative data converge, this strengthens the validity of your conclusions. This process is called triangulation .

As you formulate your research question , try to directly address how qualitative and quantitative methods will be combined in your study. If your research question can be sufficiently answered via standalone quantitative or qualitative analysis, a mixed methods approach may not be the right fit.

But mixed methods might be a good choice if you want to meaningfully integrate both of these questions in one research study.

Keep in mind that mixed methods research doesn’t just mean collecting both types of data; you need to carefully consider the relationship between the two and how you’ll integrate them into coherent conclusions.

Mixed methods can be very challenging to put into practice, and comes with the same risk of research biases as standalone studies, so it’s a less common choice than standalone qualitative or qualitative research.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

There are different types of mixed methods research designs . The differences between them relate to the aim of the research, the timing of the data collection , and the importance given to each data type.

As you design your mixed methods study, also keep in mind:

• Your research approach ( inductive vs deductive )
• Your research questions
• What kind of data is already available for you to use
• What kind of data you’re able to collect yourself.

Here are a few of the most common mixed methods designs.

Convergent parallel

In a convergent parallel design, you collect quantitative and qualitative data at the same time and analyze them separately. After both analyses are complete, compare your results to draw overall conclusions.

• On the qualitative side, you analyze cyclist complaints via the city’s database and on social media to find out which areas are perceived as dangerous and why.
• On the quantitative side, you analyze accident reports in the city’s database to find out how frequently accidents occur in different areas of the city.

In an embedded design, you collect and analyze both types of data at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.

This is a good approach to take if you have limited time or resources. You can use an embedded design to strengthen or supplement your conclusions from the primary type of research design.

Explanatory sequential

In an explanatory sequential design, your quantitative data collection and analysis occurs first, followed by qualitative data collection and analysis.

You should use this design if you think your qualitative data will explain and contextualize your quantitative findings.

Exploratory sequential

In an exploratory sequential design, qualitative data collection and analysis occurs first, followed by quantitative data collection and analysis.

You can use this design to first explore initial questions and develop hypotheses . Then you can use the quantitative data to test or confirm your qualitative findings.

“Best of both worlds” analysis

Combining the two types of data means you benefit from both the detailed, contextualized insights of qualitative data and the generalizable , externally valid insights of quantitative data. The strengths of one type of data often mitigate the weaknesses of the other.

For example, solely quantitative studies often struggle to incorporate the lived experiences of your participants, so adding qualitative data deepens and enriches your quantitative results.

Solely qualitative studies are often not very generalizable, only reflecting the experiences of your participants, so adding quantitative data can validate your qualitative findings.

Method flexibility

Mixed methods are less tied to disciplines and established research paradigms. They offer more flexibility in designing your research, allowing you to combine aspects of different types of studies to distill the most informative results.

Mixed methods research can also combine theory generation and hypothesis testing within a single study, which is unusual for standalone qualitative or quantitative studies.

Mixed methods research is very labor-intensive. Collecting, analyzing, and synthesizing two types of data into one research product takes a lot of time and effort, and often involves interdisciplinary teams of researchers rather than individuals. For this reason, mixed methods research has the potential to cost much more than standalone studies.

Differing or conflicting results

If your analysis yields conflicting results, it can be very challenging to know how to interpret them in a mixed methods study. If the quantitative and qualitative results do not agree or you are concerned you may have confounding variables , it can be unclear how to proceed.

Due to the fact that quantitative and qualitative data take two vastly different forms, it can also be difficult to find ways to systematically compare the results, putting your data at risk for bias in the interpretation stage.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Non-probability sampling
• Quantitative research
• Inclusion and exclusion criteria

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to systematically measure variables and test hypotheses . Qualitative methods allow you to explore concepts and experiences in more detail.

In mixed methods research , you use both qualitative and quantitative data collection and analysis methods to answer your research question .

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organizations.

Triangulation in research means using multiple datasets, methods, theories and/or investigators to address a research question. It’s a research strategy that can help you enhance the validity and credibility of your findings.

Triangulation is mainly used in qualitative research , but it’s also commonly applied in quantitative research . Mixed methods research always uses triangulation.

These are four of the most common mixed methods designs :

• Convergent parallel: Quantitative and qualitative data are collected at the same time and analyzed separately. After both analyses are complete, compare your results to draw overall conclusions. 
• Embedded: Quantitative and qualitative data are collected at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.
• Explanatory sequential: Quantitative data is collected and analyzed first, followed by qualitative data. You can use this design if you think your qualitative data will explain and contextualize your quantitative findings.
• Exploratory sequential: Qualitative data is collected and analyzed first, followed by quantitative data. You can use this design if you think the quantitative data will confirm or validate your qualitative findings.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

George, T. (2023, June 22). Mixed Methods Research | Definition, Guide & Examples. Scribbr. Retrieved March 20, 2024, from https://www.scribbr.com/methodology/mixed-methods-research/

Is this article helpful?

Tegan George

Other students also liked, writing strong research questions | criteria & examples, what is quantitative research | definition, uses & methods, what is qualitative research | methods & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

• Study Protocol
• Open access
• Published: 18 March 2024

Integrating perspectives of transgender and gender-diverse youth, family members, and professionals to support their health and wellbeing – a mixed-method study protocol

• Julie Servais 1 ,
• Bram Vanhoutte 1 ,
• Sara Aguirre-Sánchez-Beato 2 ,
• Isabelle Aujoulat 3 ,
• Cynthia Kraus 4 ,
• Guy T’Sjoen 5 ,
• Sandra Tricas-Sauras 1 &
• Isabelle Godin 1  

Archives of Public Health volume  82 , Article number:  40 ( 2024 ) Cite this article

27 Accesses

1 Altmetric

Metrics details

The current literature highlights a strong link between the poor health outcomes of transgender and gender diverse (TGD) individuals and their negative experiences in various areas of life. Most of these publications rely on adults’ memories, lacking a focus on the current experiences and needs of young transgender and gender-diverse individuals. Furthermore, previous studies on support for these young people often solely consider the perspectives of TGD adults or professionals and rarely involve parents’ viewpoints.

This study will use a mixed sequential method with a participatory approach. Firstly, the qualitative phase will explore the difficulties and needs of TGD (15–20 years old) and of the families and professionals who support them. Results from this part will be used to develop the questionnaire for the quantitative phase, with the help of a community board. Secondly, based on participatory epidemiological research, the quantitative phase will use an intersectional perspective to measure the impact of individual and structural factors on the quality of life and well-being of transgender and gender-diverse young people. Finally, a co-creation phase will be undertaken to formulate recommendations based on the results of the first two phases.

This research aims at better understanding the influence of gender identity on the quality of life and health of TGD young people and their families and to identify protective and risk factors that affect their vulnerabilities.

Ethics and dissemination

This study has been approved by the Ethics Committee of the Erasme Faculty Hospital (CCB B4062023000140). As this research is participatory and part of a PhD dissertation, we aim to disseminate the results through our partners’ networks and structures locally, and internationally through conferences and peer-reviewed journals.

Peer Review reports

The lack of inclusion of lesbian, gay, bisexual, transgender, queer, intersex, asexual (LGBTQIA +) communities is a significant issue in schools, sports, work, and healthcare [ 1 , 2 , 3 , 4 , 5 ]. Discrimination, integration issues, greater risk of harassment are all challenges faced by young LGBTQIA + people [ 6 , 7 , 8 ]. Previous research revealed the many aspects of their oppression, all of which have deleterious effects on the health and well-being of this population [ 9 , 10 , 11 , 12 ]. Similarly, limited access to health care and quality care [ 13 , 14 ], substance use [ 15 ], greater risk of harassment [ 11 , 12 ], homelessness and poverty [ 16 , 17 , 18 ] all contribute to the potentially poor health of transgender and gender diverse (TGD) people.

The period between 15 and 20 years of age is widely recognised as a period of life characterised by both physical and psychological profound changes as well as changes in the social and family sphere [ 19 , 20 , 21 ]. It is a period marked by the transition from a vertical socialisation provided by the family home to a horizontal socialisation characterised by a detachment from the home and increasing closeness to peers [ 22 , 23 ]. During this period, young people may be confronted with specific vulnerabilities that can endanger their future prospects such as discontent, self-esteem and self-image disorders, family breakdowns, etc. [ 21 , 24 , 25 ]. These vulnerabilities can be amplified in a situation of gender non-conformity because the youth has to deal with gendered aspects of bodily changes, identity issues, the gaze and opinions of his or her family and the judgement of others [ 26 , 27 ]. Although there are few qualitative studies on the experiences of young TGD people [ 28 ], the frequency of negative experiences related to the gaze or opinions of others seems to be linked to the age of the young person at the time of the occurrence. They are more prevalent in primary and secondary schools than in tertiary education, according to the testimony of TGD adults [ 29 ].

Gender non-conforming can cause difficulties in family life, social life, school life and other important areas, especially in the 15–20 age group [ 15 , 30 , 31 ]. Yet parent- or youth-initiated counselling often lags, partly because the topic is still taboo in many families, regions and countries [ 32 , 33 ]. The literature indicates that most parents accept that there is variation in their child’s gender expression on a transitory basis [ 34 , 35 , 36 ]. However, when this persists, they become concerned about their child’s psychosocial well-being. Unsure of the appropriate way to deal with the issue, most of them seek help and support by taking the step to consult a mental health professional together [ 34 , 35 , 36 , 37 , 38 ]. In other cases, parents find it more difficult to accept a young person that does not correspond to their gender expectations, notably because of cultural or community pressures and/or their own beliefs [ 39 ].

Regarding ‘generic’ health care, several studies have shown that the main barriers to quality care are related to the lack of training of health care professionals as well as to their representations regarding the TGD youth [ 40 , 41 ]. Indeed, family doctors and psychologists that are not specifically active or trained in the field of TGD care seem to lack the relevant and useful information to meet the needs of this population and to appropriately refer them and/or answer their questions [ 29 , 42 , 43 , 44 , 45 ].

To identify the age-specific needs of young people, awareness raising in schools is necessary [ 46 , 47 , 48 , 49 ]. Educational professionals can also play a key role in the acceptance and affirmation of a young person’s gender identity by having a supportive and caring attitude [ 29 , 49 , 50 ]. But when faced with difficult situations at school, such as access to locker rooms or gymnastics classes that are still too cisnormative, or when subjected to bullying based on gender identity, professionals such as psycho-medical-social workers generally don’t have much knowledge or answers to offer on TGD-related issues [ 49 , 50 ].

While TGD people are becoming more visible, with media coverage of some artists coming out and several films or series following the journeys of TGD people, and while Belgian policy emphasises the promotion of sexual, emotional and relational health (especially in schools) to reduce gender-related social and health inequalities, current literature shows that negative health outcomes for TGD people are still strongly correlated with negative experiences in their own environment, such as—family, school, health care, etc. [ 15 , 49 , 51 , 52 , 53 ]. However, these findings are often the result of retrospective accounts from adults and there are very few studies on the current experiences and needs of TGD youth. According to Költö, this is one of the current research gaps, at least in Europe, for this community [ 28 ]. Furthermore, previous studies on the issue of support for young TGD people have approached it either from the perspective of TGD people or from the perspective of professionals, but rarely from a combination of the two and fail to consider parents’ perceptions. The aim of the project is to co-construct recommendations and proposals for relevant support strategies in order to improve the well-being of TGD young people in their different life settings.

Theorical underpinnings

This research project will use the framework of the bio-ecological model [ 54 , 55 , 56 ] and intersectionality theory [ 57 ].

Bronfenbrenner’s bio-ecological systems theory was developed to understand human development within various interconnected systems [ 54 , 55 , 56 ]. The subsystems identified by Bronfenbrenner include: 1/ the microsystem (directly interacting groups such homes, schools, or religious communities); 2/ the mesosystem (relationships between two or more microsystems, such as school and parents); 3/ the exosystem (environments influencing development without direct influence, such as the media); and 4/ the macrosystem (broader systems encompassing community, culture, and politics) [ 54 , 55 , 56 ]. Finally, chronosystem includes the experiences and life changes of youth over time, both personally and socio-culturally, as well as their individual developmental trajectories. This theory has been applied to a variety of contexts, including to transidentity [ 58 , 59 , 60 ].

While recognising individuals are situated at the intersection of various systems, ecological system theory places less emphasis on how social group membership impacts their experiences within these contexts. This is exactly what intersectionality theory emphasizes: individuals’ experiences and functioning are strongly influenced by the interplay between social categories (e.g., ethnicity, social class, gender, sexual orientation) in multiple systems of oppression and privilege [ 57 , 61 ].

Nevertheless, while intersectionality highlights the multiple and interconnecting systems that perpetuate inequality and opportunity [ 57 ], it lacks an explicitly developmental dimension [ 62 ]. By merging intersectional and ecological perspectives, as demonstrated by some authors [ 62 , 63 , 64 ], it becomes possible to achieve a comprehensive integration of each identity, their intersections, and their interactions with different subsystems. In essence, this approach aims to establish connections between different systems of oppression (e.g., racism, transphobia, classism) and contexts (e.g., family, school, neighbourhood) that are intricately intertwined.

This research project seeks to gain a better understanding of the influence of gender identity on the quality of life and health of TGD adolescents and young adults (AYAs) (15 to 20 years old) and their families, and to identify the risk factors that increase their vulnerability. The ultimate aim of this research will be to co-construct intervention and support approaches based on the preferences and needs of TGD AYAs and their families, as well as on the needs of the people who support them. In this sense, participatory research methods will be favoured as they enable the sources of marginalisation to be identified, understood, and addressed in close collaboration with communities such as young TGD people. Moreover, this participatory approach has the potential to cultivate collaborative relationships among individuals engaged in the lives of these youths with various gender identities. This promotes a collective comprehension of the most efficient methods for shaping systems and instigating profound change [ 65 , 66 , 67 ].

This main objective and methodology give rise to three areas of research linked to interdependent secondary objectives:

Qualitative study to address the difficulties and needs of TGD young people by integrating their perspective, knowledge, and experience as well as those around them namely parents, siblings, extended family, and professionals.

To understand and characterise the representations of gender diversity, the lived meaning, and perceptions of TGD AYAs currently living in French-speaking Belgium.

To identify the resources (social, professional and/or other) that young TGD AYAs use and the types of additional support they would need.

To understand and characterise the representations and perceptions of the experiences of young TGD AYAs through the eyes of parents (and extended family) and professionals who accompany them.

Quantitative study to be designed on the basis of the results of the qualitative component and designed in collaboration with the expert group. This part aims principally to assess whether there is a difference in quality of life, self-confidence, and satisfaction with the support between the different TGD identities through the lens of intersectionality, and if so, to describe the nature of this difference.

To describe and clarify the average scores on outcomes listed above, considering gender diversity beyond binarity.

To describe and clarify the average scores on outcomes above while considering the intersection of different social locations, power relations and experiences.

Co-construction to produce recommendations on the basis of the results of the qualitative and quantitative data collection. The aim of this section is to help build a more inclusive society that takes into account the diversity of genders and identities, and to improve support for young TGD people by taking into account their perspectives, their needs and those who support them. (parents, extended family, professionals).

Study design

This project will use mixed methods by conducting both qualitative and quantitative research in a sequential manner (see Fig.  1 ). As some authors point out, the combination of qualitative and quantitative approaches allows for a greater in-depth understanding of the results for the study of complex social, behavioural and health phenomena [ 68 , 69 , 70 ]. An additional reason for using a mixed method approach is the possibility to triangulate the results, which assumes that ‘the use of different sources of information will confirm and improve the clarity of a research result’ and in the case of this project, the relevance of the recommendations [ 71 ]. The qualitative study aims to understand the representations, perceptions and lived meanings of gender identity of the different stakeholders. Due to our research priorities, the qualitative component will be strongly developed due to the lack of research in this area in French-speaking Belgium, particularly in our study population. From a sequential exploratory perspective, identifying the living environments that are important for young people with TGD and their needs in terms of guidance and support will enable us to design the questionnaire for the quantitative part of this research [ 72 ]. This part will enable us to reach a wider population and see whether the living environments and support needs identified are the same according to the internal diversity of gender identities within the TGD population.

Research design—sequential mixed methods

Resonance group

Co-construction methods allow us to bring together different types of knowledge (experiential, professional) without taking into account a hierarchy between them [ 73 ]. Therefore, from the start of the project, our methodology will include the creation of a resonance group composed of TGD adults, family members (experiential knowledge), and field professionals (professional knowledge). Throughout the project and beyond, the members of this resonance group commit to accompanying and questioning the researcher individually and as a group, based on their expertise and availability. This group will guide the researcher through the different parts of the project (see Fig.  1 ), such as 1/ language and terminology, 2/ development of the interview guide, 3/ recruitment in both qualitative and quantitative parts, 4/ interpretation of qualitative results, 5/ co-designing the survey tool, 6/ interpretation of quantitative results, 7/ support for the dissemination of the results, 8/ participation in the co-construction step and 9/ clarification and reflection on our position throughout the research process.

Qualitative phase

Participants.

Participants in the qualitative part of the study will be from three stakeholder groups, transgender youth, parents (and/or extended family) and professionals. The study will include participants from French-speaking Belgium and Brussels.

Transgender youth

For the semi-structured interviews with young TGD people, we aim to include French speaking young people aged 15–20. They will be recruited through different networks: 1/ via trans-specific care consultations; 2/ via partner associations in the research project; 3/ via social networks. Depending on the location, professionals will either be asked to inform young people and their parents about the research project, or they will select young people capable of participating in this type of project beforehand. At first, the Schedule for the Evaluation of Individual Quality of Life (SEIQoL) tool will be used to encourage young people to narrate what they consider essential to their quality of life [ 74 ]. The semi-structured interview, which is central to the approach, invites the participants to express themselves initially on the dimensions that are most important to them in terms of quality of life, then on their current level of satisfaction in each of the dimensions. Finally, they are asked to rank the dimensions that make up their quality of life according to level of importance [ 75 , 76 ]. Following this assessment, we will ask the young people to take one or two photos per dimension (those mentioned by the young person), and we will schedule a second interview with them. Photovoice often enables comprehensive data to be obtained by facilitating a relationship between the participant and the interviewer and by encouraging participants to provide an in-depth understanding of their experiences in terms of emotions, feelings and ideas [ 77 ]. This method is a form of participatory method and is particularly suited to research with adolescents and young people [ 78 , 79 , 80 ].

Parents and professionals

Parents (and extended family) and professionals’ recruitment will be undertaken via 1/ trans-specific care consultations; 2/ via partner associations in the research project; 3/ via social networks; 4/ via email or phone call campaigns to professionals working in the areas highlighted by the interviews with young people. Data will be collected through semi-structured interviews supported by a thematic interview guide as the content helps to sustain the discussion but leaves it open to flexibility and creativity [ 81 ]. The interview guide will be developed with the resonance group and based on the literature. Given the sensitive nature of the topic, we will consider the use of tools such as vignettes, photos, or any other relevant instrument to facilitate dialogue and reduce participant reluctance [ 82 , 83 , 84 ]. Through these interviews we would like to understand how the different stakeholders perceive the current organisation of the comprehensive care system in relation to the needs highlighted by the young people. We would also like to understand how they perceive their own needs and the specific needs of young TGD people are being currently considered and what potential improvements they would like to see introduced into the Belgian system to improve the inclusion of this community.

Data collection

With the consent of participants, the sessions will be recorded. This will allow us to fully concentrate on the interview. The recordings will be transcribed ad verbatim and analysed thematically, based on the content of the answers, and iteratively. Based on the first interviews, an initial list of emerging codes will be identified and organised in a tree structure. The theme is identified inductively: once identified, this theme is compared with other data to confirm its presence in other interviews (deductive approach) [ 81 , 85 , 86 ]. The data will be compared between the interviews of the TGD young people, the parents, and the professionals to make an initial theoretical comparison. In a later stage, we will bring them into dialogue with theoretical constructs from the literature. Upon completion of the analysis, the recordings will be destroyed.

Quantitative phase

When collecting qualitative data, we chose not to select our participants based on their gender identity. Nevertheless, several articles highlight the importance of considering the internal diversity of the TGD population in research [ 15 , 52 , 87 , 88 , 89 ]. Therefore, we plan to undertake a quantitative data collection on gender plurality in French-speaking Belgium with an online survey tool.

Survey participants will be recruited using two methods: either in person (through events, community gatherings, trans-specific care clinics) or online (through mailing lists, social networks). The eligibility criteria will be: 1/ to identify as a TGD person; 2/ to be at least 15 years old and less than 20; 3/ to live in Belgium; 4/ to be able to understand French, or English. A convenience sampling approach within the TGD population will be used. Although there is no precise data on the size of the TGD population worldwide and figures depend on the definition of transidentity used, estimates suggest a prevalence of 0.7% to 2.5% among young people aged 15 to 20 [ 15 , 90 , 91 , 92 , 93 ]. In French speaking Belgium, based on the age pyramid, transidentity would therefore concern between 2,472 and 8,827 young people between 15 and 20 years of age [ 94 ]. For this reason, it would be ideal to have a sample size of at least 300 people, i.e. an estimate of 5% of the population of young people targeted by the study, in order to have sufficient statistical power to carry out the analyses.

Type of study

This is a cross-sectional observational study. Like the other parts of this research project, this study will be based on four of the principles of participatory research as defined by Israel et al. [ 95 ] and adapted by Bach et al. [ 96 , 97 ] to fit participatory epidemiology: 1/ joint definition of objectives and research questions, 2/ joint definition of the populations studied and their health-related contexts, 3/ selection or development of appropriate survey instruments, and 4/ dialogical forms of interpretation of results. We therefore cannot add supplementary data as the questionnaire will be based on findings from the qualitative data and constructed with the help of the resonance group and some of the people interviewed in the first part of the project and who are willing to participate [ 98 , 99 ]. Nevertheless, some parts of the Health Behaviour in School-aged Children (HBSC) study will be used as a baseline. HBSC is an international cross-sectional school survey conducted every four years in around fifty countries [ 100 ], including Belgium. Data on health status, health behaviours and well-being are collected from children and adolescents between 10 and 21 years of age.

Data analysis

Data analysis will be undertaken using Stata 17.0 software. All variables included the study questionnaire will be analysed. Missing data will be reported. For the description of the sample, the qualitative variables will be described and compared using the Pearson’s Chi-2 test or Fisher’s exact test if the former is not applicable.

Then, the data analysis will be divided into two stages: bivariate analysis and multivariate analysis. The bivariate analysis will consist of measuring separately the strength of the associations between gender identity and each of the exposure factors, without adjusting for potential confounders. The multivariate analysis will be performed using adjusted multivariate regression models with binary and non-binary identities as the main statistical predictor of social and health outcomes, adjusted for the following control variables: age, education, ethnicity, region of residence [ 101 ].

Co-creation phase

This part of the project will focus on comparing and contrasting the perceptions of young people, parents and professionals regarding gender identity and will allow to co-construct with them recommendations for good practice around non-medical support for young TGD people. We will encourage group discussions with an emphasis on participatory methods to empower the different actors [ 82 , 102 , 103 , 104 , 105 ].

The resonance group and the participants interviewed in the first part of the project will be invited to participate in this stage of the project. Each person interested in participating in this group, whether they were interviewed in the previous phase or came through another channel, will be asked to fill in a description form and explain why they wish to participate in this group. This information will allow for the composition of the groups to be balanced, ensuring a good flow of discussion between participants, as recommended in co-construction methods [ 73 , 103 ].

To prepare TGD young people to participate in larger groups, we will meet them first through focus groups. In a second phase, more diverse groups will be organised with 8–12 participants, mixing equally TGD youth, parents (and family members) and professionals. They will be invited to regular participation like communities of practice to promote the co-construction of knowledge through an iterative process [ 103 , 106 ].

Course of the activities

The members of each group will be asked to respect the confidentiality of exchanges and any other provision that the group members consider necessary for their proper functioning [ 107 , 108 ]. Prior to the meetings (4 sessions will be planned), preparatory material will be communicated to the participants, either for individual appropriation or during a collective session between participants. Indeed, the co-construction of knowledge is an iterative process, which proceeds in sequences: ideally, the participants alternate times of individual and collective work [ 106 ].

Each participant will also be asked to keep a virtual or paper log of their experience [ 109 , 110 ]. If participants wish to do so, they may share all or part of this with the researcher to illustrate this in the final manuscript. An external support person will be identified and will participate in the different groups to ensure the psychological safety of the participants: [ 111 ].

These group discussions will have two purposes: 1/ to discuss the emerging themes from the interviews and the results of the quantitative data collection; 2/ to formulate recommendations and good practices based on these results.

Data management and analysis

Thanks to the recording of the discussions (with the consent of the participants), a report will be written for each group session, summarising the discussions, the elements to be included in the analysis of the results of the interviews and the elements that the participants wish to use in the next session. The data collected will be used to inform the analysis of the interviews and potentially to undertake additional analysis if new data is collected during the discussions. The minutes will be shared with the participants and will only be accessible to members of the working group.

This will therefore be used to support the integrated findings of the qualitative and quantitative parts and to help formulate recommendations.

Ethical considerations

When considering research, particularly when involving sensitive issues and adolescents and young adults, ethical aspects are essential to consider [ 112 ], which is why the research protocol was submitted to the Ethics Committee of the Erasme Faculty Hospital and received its approval on the 27 th of July 2023 (CCB B4062023000140).

Before submitting the research protocol to the ethics committee, a literature review was carried out to identify the methodological and ethical challenges associated with participatory research with TGD young people. A secondary objective of this literature review was to highlight the considerations when parental consent is required, as obtaining parental consent may compromise their safety, well-being, or privacy, if they live in case their family is not supportive [ 113 , 114 , 115 ]. This issue puts young people’s rights to autonomy, privacy, and freedom in tension with parents’ rights to protect their children and is perhaps one of the reasons why findings about the TGD community are often the product of adult retrospective accounts [ 112 , 114 , 116 ].

For the qualitative study, we will pay particular attention to the participation of minors in this project: consent will be adapted, and they may be accompanied by a trusted person. In addition to a description of the project, a consent form will be provided to each participant in order to collect their consent and to provide them with information on confidentiality, their right to access the data, their right to rectify it if considered to be incorrect, their right to object to its the use, and the right to be forgotten. The participants’ identity will be protected by using pseudonyms, and any data that may identify an individual will not be transcribed or coded. Pseudonyms will be used on all files and transcripts. Each participant will be given a copy of the transcript of their interview on request. All the data used are solely for research purposes in the context of this project. Finally, in. regards to the risks of participating in research in a health field that includes vulnerable populations, we will provide a contact person to liaise with participants [ 112 ]. For the quantitative study, we will not include any identification information as it will be an anonymous survey. An information and consent section will be provided before participation in the survey.

Dissemination

Participative research emphasises on prioritising the experiences, perspectives, and actions of participants (e.g., storytellers, data producers) over academic researchers. The goal is to safeguard participants’ narratives, stories, and their roles as ‘tellers’ or authors from being dismissed or silenced [ 117 ]. Therefore, an emphasis on the principles of dissemination as described by Israel [ 95 ] will be made. One key principle in this regard indicates that the researcher must consult participants before submitting documents for publication, acknowledge the contributions of participants and, where appropriate, develop collaborative publications [ 95 ].

As this research is part of a doctoral thesis, dissemination will also be carried out through its publication as a dissertation, as well as through several scientific articles and presenting of results at conferences. The members of the resonance group will be able to ensure their own dissemination plan with or without the help of the researcher for dissemination of results. Particular attention will be paid to translating research findings into understandable language so that they can be disseminated as widely as possible [ 118 ].

This research aims to better understand the influence of gender identity on the quality of life and health of TGD young people and their families and to identify protective and risk factors that decrease or increase their vulnerabilities. However, the approach chosen for this study presents several challenges. Ethically, like many other countries, Belgium requires parental consent prior to participation in medical or non-medical research [ 119 , 120 ]. This is known as the statutory approach to consent, in other words, the legal approach to consent. Although seen as an important safety barrier and good practice, the requirement for parental consent is open to criticism as it may discourage young people, particularly those who are marginalised, from participating in research [ 112 , 115 , 121 , 122 , 123 , 124 ]. For this reason, this project will favour a maturity or skills-based approach , which values young people’s agency. This approach highlights the fact that young people’s ability to understand is dynamic and developing, and is most certainly influenced by their life experiences and socio-cultural contexts [ 125 ].

Beyond the question of consent, and from a methodological perspective, the specific participation of young people in this type of research also highlights some important issues. Firstly, like other social groups, young people are not a homogenous group. Secondly, factors such as socio-economic class, ethnicity, culture, and environment play a very important role in their life experiences. Although participatory research does not necessarily prioritise representativeness, it is often more beneficial to involve some young people while being aware of and acknowledging the voices that are included or overlooked, rather than conducting research without any input from young people. A combination of data collection methods can then enable us to hear the voices of those who were not reached in the qualitative part. In addition, participatory methods pose challenges in terms of negotiating boundaries and power dynamics, especially when involving young people. Firstly, researchers need to move from their traditional role of producing results and recommendations (problem definition) to the role of facilitators, working with communities to find solutions that meet their needs [ 126 ]. Secondly, in order to avoid adultism (where young people are marginalised because of their age and experience), they need to re-conceptualise the relationship between adults and young people as an equal one [ 127 ].

This research project also has strengths. The creation of a resonance group to overcome the challenges mentioned above is considered to be a strength. Indeed, this group will help us ground our research locally, including recruiting these young people. It will also ensure that the study and its methods are acceptable to the community [ 105 ]. The use of mixed methods is also a strong point of this research project, as these methods allow a better understanding of complex social phenomena [ 68 , 70 , 72 ].

At the macro-level, the study may help to address important policy and research questions. The knowledge co-construction approach suits to influence policy by generating evidence and supporting citizen participation—understood as the involvement of actors in society in the broadest sense [ 128 ]. Therefore, political authorities are likely to be interested in the results of our project, especially given the development of various policies that consider LGBT + communities.

Availability of data and materials

No datasets were generated or analysed during the current study.

Beemyn G. The experiences and needs of transgender community college students. Community Coll J Res Pract. 2012;36:504–10.

Article   Google Scholar  

McKinney JS. On the margins: a study of the experiences of transgender college students. J Gay Lesbian Issues Educ. 2005;3(1):63–76.

Patchett E, Foster J. Inclusive recreation: the state of campus policies, facilities, trainings programs for transgender participants. Recreat Sports J. 2018;39(2):83–91.

Safer JD, Coleman E, Feldman J, Garofalo R, Hembree W, Radix A, Sevelius J. Barriers to health care for transgender individuals HHS public access. Curr Opin Endocrinol Diabetes Obes. 2016;23(2):168–71.

Article   PubMed   PubMed Central   Google Scholar  

Seelman KL. Recommendations of transgender students, staff, and faculty in the USA for improving college campuses. Gend Educ. 2014;26(6):618–35.

Newhook JT, Winters K, Pyne J, Jamieson A, Holmes C, Feder S, Pickett S, Sinnott M-L. Teach your parents and providers well. Can Fam Physician. 2018;64(5):332–5.

PubMed   PubMed Central   Google Scholar  

Smalley KB, Warren JC, Barefoot KN. Differences in health risk behaviors across understudied LGBT subgroups. Health Psychol. 2016;35(2):103–14.

Article   PubMed   Google Scholar  

Veale JF, Watson RJ, Peter T, Saewyc E. Mental health disparities among Canadian transgender youth. J Adolesc Health. 2017;60(1):44–9.

Albuquerque GA, de Lima Garcia C, da Silva Quirino G, Alves MJH, Belém JM, dos Santos Figueiredo FW, da Silva Paiva L, do Nascimento VB, da Silva Maciel É, Valenti VE, et al. Access to health services by lesbian, gay, bisexual, and transgender persons: systematic literature review. BMC Int Health Hum Rights. 2016;16:2.

Brandes A. The negative effect of stigma, discrimination, and the health care system on the health of gender and sexual minorities. Law Sex. 2014;23:155–78.

Google Scholar  

McCann E, Brown M. Discrimination and resilience and the needs of people who identify as transgender: a narrative review of quantitative research studies. J Clin Nurs. 2017;26(23–24):4080–93.

Winter S, Diamond M, Green J, Karasic D, Reed T, Whittle S, Wylie K. Transgender people: health at the margins of society. Lancet. 2016;388(10042):390–400.

Noonan EJ, Sawning S, Combs R, Weingartner LA, Martin LJ, Jones VF, Holthouser A. Engaging the transgender community to improve medical education and prioritize healthcare initiatives. Teach Learn Med. 2018;30(2):119–32.

Rider GN, McMorris BJ, Gower AL, Coleman E, Eisenberg ME. Health and care utilization of transgender and gender nonconforming youth: a population-based study. Pediatrics. 2018;141(3):e20171683.

Eisenberg ME, Gower AL, McMorris BJ, Rider GN, Shea G, Coleman E. Risk and protective factors in the lives of transgender/gender nonconforming adolescents. J Adolesc Health. 2017;61(4):521–6.

Keuroghlian AS, Shtasel D, Bassuk EL. Out on the street: a public health and policy agenda for lesbian, gay, bisexual, and transgender youth who are homeless. Am J Orthopsychiatry. 2014;84(1):66–72.

McCann E, Brown M. Homelessness among youth who identify as LGBTQ+: a systematic review. J Clin Nurs. 2019;28:2061–72.

McCann E, Brown M. Homeless experiences and support needs of transgender people: a systematic review of the international evidence. J Nurs Manag. 2021;29(1):85–94.

Tanti C, Stukas AA, Halloran MJ, Foddy M. Social identity change: shifts in social identity during adolescence. J Adolesc. 2011;34:535–67.

Albarello F, Crocetti E, Rubini M. I and us: a longitudinal study on the interplay of personal and social identity in adolescence. J Youth Adolesc. 2018;47(4):689–702.

Potterton R, Austin A, Robinson L, Webb H, Allen KL, Schmidt U. Identity development and social-emotional disorders during adolescence and emerging adulthood: a systematic review and meta-analysis. J Youth Adolesc. 2022;51:16–29.

Erikson E. Adolescence et crise : La quête de l’identité. Paris: Flammarion; 1968.

Schachter A, Galliher R. Fifty years since “Identity: Youth and Crisis”: a renewed Loof at Erikson’s writings on identity. Identity. 2018;18(4):247–50.

Délégation interministérielle à la prévention et à la lutte contre la pauvreté des enfants et des jeunes (DIPLPEJ). Prévenir la vulnérabilité des jeunes et favoriser leur insertion. Stratégie de prévention et de lutte contre la pauvreté des enfants et des jeunes. DIPLPEJ. 2018.

Verot C, Dulin A. Arrêtons de les mettre dans des cases ! Pour un choc de simplification en faveur de la jeunesse. France: Gouvernement Français; 2017. p. 182.

Cohen-Kettenis PT, Pfäfflin F. Transgenderism and intersexuality in childhood and adolescence: making choices. Thousand Oaks: SAGE Publications, Inc; 2003.

Wallien MSC, Cohen-Kettenis PT. Psychosexual outcome of gender-dysphoric children. J Am Acad Child Adolesc Psychiatry. 2008;47(12):1413–23.

Költö A, Vaughan E, O’Sullivan L, Kelly C, Saewyc EM, NicGabhainn S. LGBTI+ youth in Ireland and across Europe: a two-phased landscape and research gap analysis. Dublin: Departement of Children Equality Disability Integration and Youth; 2021.

Motmans J, Wyverkens E, Defreyne J. Être une personne transgenre en Belgique Dix ans plus tard. Brussels: IGVM-IEFH; 2017.

Hale AE, Chertow SY, Weng Y, Tabuenca A, Aye T. Perceptions of support among transgender and gender-expansive adolescents and their parents. J Adolesc Health. 2021;68(6):1075–81.

Martinerie L, Le Heuzey MF, Delorme R, Carel JC, Bargiacchi A. Évaluation Et Prise En Charge D’Une Dysphorie De Genre Chez L’Enfant Et L’Adolescent. Arch Pediatr. 2016;23(6):668–73.

Article   CAS   PubMed   Google Scholar  

Greenberg KB, Handelman M, Alongi D. “And then I knew that trans was a thing”; gender expansive youth and the role of primary care screening. J Adolesc Health. 2017;60(2):S89.

Khatchadourian K, Amed S, Metzger DL. Clinical management of youth with Gendre dysphoria in Vancouver. J Pediatr. 2014;164(4):906–11.

Ehrensaft D. Gender born, gender made. Raising healthy gender-nonconforming children. New-York: The Experiment; 2011.

Grossman AH, D’Augelli AR, Howell TJ, Hubbard S. Parents’ reactions to transgender youths’ gender nonconforming expression and identity. J Gay Lesbian Soc Serv. 2006;18(1):3–16.

Harper A, Singh A. Supporting ally development with families of Trans and Gender Nonconforming (TGNC) Youth. J LGBT Issues Couns. 2014;8(4):376–88.

Bernal AT, Coolhart D. Treatment and ethical considerations with transgender children and youth in family therapy. J Psychother Fam. 2012;23(4):287–303.

Menvielle EJ. Transgender children: clinical and ethical issues in prepubertal presentations. J Gay Lesbian Ment Health. 2009;13(4):292–7.

Malpas J. L’enfant en non-conformité de genre et sa famille: une approche systémique. Cahiers Critiq Thér Fam Pratiq Réseaux. 2014;52(1):139–65.

Snelgrove JW, Jasudavisius AM, Rowe BW, Head EM, Bauer GR. “Completely out-at-sea” with “two-gender medicine”: a qualitative analysis of physician-side barriers to providing healthcare for transgender patients. BMC Health Serv Res. 2012;12(101):1–13.

Stroumsa D, Shires DA, Richardson CR, Jaffee KD, Woodford MR. Transphobia rather than education predicts provider knowledge of transgender health care. Med Educ. 2019;53(4):398–407.

Allen BJ, Coles MS, Montano GT. A call to improve guidelines for transgender health and well-being: promoting youth-centered and gender-inclusive care. J Adolesc Health. 2019;65(4):443–5.

Transidentités dans l’enfance et professionel.le.s: des clés pour accompagner. https://www.noahgottlob.com/cles-pour-professionnel-le-s . Accessed Nov 2022.

Radix AE. Addressing needs of transgender patients: the role of family physicians. J Am Board Fam Med. 2020;33:314–21.

Coleman E, Bockting W, Botzer M, Cohen-Kettenis P, DeCuypere G, Feldman J, Fraser L, Green J, Knudson G, Meyer WJ, Monstrey S, Adler RK, Brown GR, Devor AH, Ehrbar R, Ettner R, Eyler E, Garofalo R, Karasic DH, Lev AI, Mayer G, Meyer-Bahlburg H, Hall BP, Pfaefflin F, Rachlin K, Robinson B, Schechter LS, Tangpricha V, van Trotsenburg M, Vitale A, Winter S, Whittle S, Wylie KR, Zucker K. Standards of care for the health of transsexual, transgender, and gender-nonconforming people, version 7. Int J Transgend. 2012;13(4):165–232. https://doi.org/10.1080/15532739.2011.700873 .

Abreu RL, Black WW, Mosley DV, Fedewa AL. LGBTQ youth bullying experiences in schools: the role of school counselors within a system of oppression. J Creat Ment Health. 2016;11(3–4):325–42.

Blackburn MV, McCready LT. Voices of queer youth in urban schools: Possibilities and limitations. Theory Pract. 2009;48(3):222–30.

Bradlow J, Bartram F, Guasp A, Jadva V. The experiences of lesbian, gay, bi and trans young people in Britain’s schools in 2017. London: Stonewall; 2017.

Saewyc EM, Thawer Z, O’Dwyer C, Sinclair J, Smith A. Gender-Diverse: a spotlight on th health of trans and non-binary young people in BC. Vancouver: University of British Columbia and McCreary Center Society; 2021.

Vie quotidienne en pratique. https://infotransgenre.be/m/vie-quotidienne/ecole/ . Accessed Nov 2022.

Andrzejewski J, Pampati S, Steiner RJ, Boyce L, Johns MM. Perspectives of transgender youth on parental support: qualitative findings from the resilience and transgender youth study. Health Educ Behav. 2021;48(1):74–81.

Lefevor GT, Boyd-Rogers CC, Sprague BM, Janis RA. Health disparities between genderqueer, transgender, and cisgender individuals: an extension of minority stress theory. J Couns Psychol. 2019;66(4):385–95.

OCDE. Le défi LGBT: Comment améliorer l’intégration des minorités sexuelles et de genre? Paris: Organisation for Economic Co-operation and Development (OECD); 2019.

Bronfenbrenner U, Evans GW. Developmental science in the 21st century: emerging questions, theoretical models, research designs and empirical findings. Soc Dev. 2000;9(1):115–25.

Rosa EM, Tudge J. Urie Bronfenbrenner’s theory of human development: its evolution from ecology to bioecology. J Fam Theory Rev. 2013;5(4):243–58.

Bronfenbrenner U, Morris PA. The ecology of developmental processes. In: Lerner RM, editor. Handbook of child psychology. 5th ed. New-York: Wiley; 1998. p. 993–1028.

Crenshaw K. Demarginalizing the intersection of race and sex: a black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. Univ Chic Leg Forum. 1989;140:139–67.

Alessi EJ, Kahn S, Chatterji S. ‘The darkest times of my life’: recollections of child abuse among forced migrants persecuted because of their sexual orientation and gender identity. Child Abuse Negl. 2016;51:93–105.

Bounds DT, Otwell CH, Melendez A, Karnik NS, Julion WA. Adapting a family intervention to reduce risk factors for sexual exploitation. Child Adolesc Psychiatry Ment Health. 2020;14:8.

Paceley MS, Sattler P, Goffnett J, Jen S. “It feels like home”: transgender youth in the Midwest and conceptualizations of community climate. J Community Psycholol. 2020;48(6):1863–81.

Hankivsky O. Intersectionality 101. Canada: The Institute for Intersectionality Research & Policy, SFU; 2014.

Roy AL. Intersectional ecologies: Positioning intersectionality in settings-level research. New Dir Child Adolesc Dev. 2018(161):57–74. https://doi.org/10.1002/cad.20248 .

Carter MKA, McGill LS, Aaron RV, Hosey MM, Keatley E, Sanchez Gonzalez ML. We still cannot breathe: applying intersectional ecological model to COVID-19 survivorship. Rehabil Psychol. 2023;68(2):112–20.

Granski M, Javdani S, Sichel CE, Rentko M. Gender differences in the relationship between self- silencing, trauma, and mental health among juvenile legal system-involved youth. Fem Criminol. 2020;15(5):545–66.

Katz-Wise SL, Pullen Sansfaçon A, Bogart LM, Ehrensaft D, Goldman RE, Bryn Austin S. Lessons from a community-based participatory research study with transgender and gender nonconforming youth and their families. Act Res. 2019;17(2):186–207.

Pullen Sansfaçon A, Hébert W, OuJin Lee E, Faddoul M, Tourki D, Bellot C. Digging beneath the surface: results from stage one of a qualitative analysis of factors influencing the well-being of trans youth in Quebec. Int J Transgender. 2018;19(2):184–202.

Singh AA, Richmond K, Burnes TR. Feminist participatory action research with transgender communities: fostering the practice of ethical and empowering research designs. Int J Transgender. 2013;14:93–104.

Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. 3rd ed. Thousand Oaks: Sage Publications, Inc.; 2017.

Johnson RB, Onwuegbuzie AJ. Toward a definition of mixed methods research. J Mixed Methods Res. 2007;1(2):112–33.

Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health. 2014;35(1):29–45.

Lewis J, Ritchie J. Generalising from qualitative research. In: Ritchie J, Lewis J, editors. Qualitative research practice. London: Sage Publications; 2003. p. 263–86.

Guest G, Fleming PJ. Mixed methods research. In: Guest G, Namey E, editors. Public health research methods. California: SAGE Publications, Inc.; 2015. p. 581–610.

Aujoulat I, Dauvrin M, Lenoble T, Schmitz O, Servais J. Participate Brussels : méthodologie de l’enquête de terrain, des activités de croisement des savoirs et de la co-construction des pistes pour la personnalisation des soins. Cahier 2. Bruxelles: UCLouvain-IRSS & HE Vinci; 2021.

O’Boyle C, McGee H, Hickey A, Malley K, Joyce C. Individual quality of life on patients undergoing hip replacement. Lancet. 1992;339(8801):1088–91.

Farrand P, Woodford J. Measurement of individualised quality of life amongst young people with indicated personality disorder during emerging adulthood using the SEIQoL-DW. Qual Life Res. 2013;22(4):829–38.

Wagner J. Acceptability of the Schedule for the Evaluation of Individual Quality of Life-Direct Weight (SEIQoL-DW) in youth with type 1 diabetes. Qual Life Res. 2004;13(7):1279–85.

Smith EF, Gidlow B, Steel G. Engaging adolescent participants in academic research: the use of photo-elicitation interviews to evaluate school-based outdoor education programmes. Qual Res. 2012;12(4):367–87.

Aldridge J. Participation, ‘vulnerability’ and voice. In: Aldridge J, editor. Participatory research: working with vulnerable groups in research and practice. Bristol: Policy Press; 2015. p. 7–30.

Chapter   Google Scholar  

Leonard M, McKnight M. Look and tell: using photo-elicitation methods with teenagers. Child Geogr. 2015;13(6):629–42.

Holtby A, Klein K, Cook K, Travers R. To be seen or not to be seen: photovoice, queer and trans youth, and the dilemma of representation. Act Res. 2015;13(4):317–35.

Patton MQ. Qualitative research & evaluation methods. Integrating theory and practice. 4th ed. Thousand Oaks: SAGE Publications, Inc; 2014.

Brouwer H, Brouwers J. The MSP tool guide: sixty tools to facilitate multi-stakeholder partnerships. Companion to the MSP guide. Wageningen: Wageningen University & Research, CDI; 2017.

Douiller A. 25 techniques d’animation pour promouvoir la santé. Paris: Le Coudrier Editions; 2012.

Fondation Roi Baudouin. Méthodes participatives. Un guide pour l’utilisateur. Bruxelles: Fondation Roi Baudouin; 2006.

Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.

Silverman D. Interpreting qualitative data. A guide to the principle of qualitative research. London: SAGE Publications, Ltd; 2011.

Aparicio-García ME, Díaz-Ramiro EM, Rubio-Valdehita S, López-Núñez MI, García-Nieto I. Health and well-being of cisgender, transgender and non-binary young people. Int J Environ Res Public Health. 2018;15(10):2133.

Poquiz JL, Coyne CA, Garofalo R, Chen D. Comparison of gender minority stress and resilience among transmasculine, transfeminine, and nonbinary adolescents and young adults. J Adolesc Health. 2021;68(3):615–8.

Reisner SL, Hughto JMW. Comparing the health of non-binary and binary transgender adults in a statewide non-probability sample. PLoS One. 2019;14(8):e0221583.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Challa M, Scott C, Turban JL. Epidemiology of pediatric gender identity. In: Forcier M, Van Schalkwyk G, Turban JL, editors. Pediatric gender identity - gender-affirminf care for transgender & gender diverse youth. Cham: Springer Nature Switzerland AG; 2020. p. 15–32.

Rafferty J. Ensuring comprehensive care and support for transgender and gender-diverse children and adolescents. Pediatrics. 2018;142(4):ee20182162.

Reisner SL, Poteat T, Keatley JA, Cabral M, Mothopeng T, Dunham E, Holland CE, Max R, Baral SD. Global health burden and needs of transgender populations: a review. Lancet. 2016;388(10042):412–36.

Zhang Q, Goodman M, Adams N, Corneil T, Hashemi L, Kreukels B, Motmans J, Snyder R, Coleman E. Epidemiological considerations in transgender health: a systematic review with focus on higher quality data. International Journal of Transgender Health. 2020;21(2):125–37.

Structure of the population. https://statbel.fgov.be/en/themes/population/structure-population . Accessed Oct 2023.

Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: assessing partnership approaches to improve public health. Annu Rev Public Health. 1998;19:173–202.

Bach M, Jordan S, Hartung S, Santos-Hövener C, Wright MT. Participatory epidemiology: the contribution of participatory research to epidemiology. Emerg Themes Epidemiol. 2017;14(2):1–15.

Bach M, Jordan S, Santos-Hövener C. What is participatory epidemiology? A definition. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2019;62(8):1020–3.

Leung MW, Yen IH, Minkler M. Community-based participatory research: a promising approach for increasing epidemiology’s relevance in the 21st century. Int J Epidemiol. 2004;33:499–506.

Reisner SL, Hughto JMW, Dunham EE, Heflin KJ, Begenyi JBG, Coffey-Esquivel J, Cahill S. Legal protections in public accommodations settings: a critical public health issue for transgender and gender-nonconforming people. Milbank Q. 2015;93:484–515.

Inchley J, Currie D, Samdal O, Jåstad A, A C, NicGabhainn S. Health Behaviour in School-aged Children (HBSC) study protocol: background, methodology and mandatory items for the 2021/22 survey. Glasgow: MRC/CSO Social and Public Health Sciences Unit, University of Glasgow; 2023.

Public Health Agency of Canada. How to integrate intersectionality theory in quantitative health equity analysis? A rapid review and checklist of promising practices. Ottawa: Public Health Agency of Canada; 2022.

Jourian TJ, Nicolazzo Z. Bringing our communities to the research table: the liberatory potential of collaborative methodological practices alongside LGBTQ participants. Educ Act Res. 2017;25(4):594–609.

Leask CF, Sandlund M, Skelton DA, Altenburg TM, Cardon G, Chinapaw MJM, De Bourdeaudhuij I, Verloigne M, Chastin SFM. Framework, principles and recommendations for utilising participatory methodologies in the co-creation and evaluation of public health interventions. Res Involv Engagem. 2019;5(1):1–6.

Verloigne M, Altenburg TM, Chinapaw MJM, Chastin S, Cardon G, De Bourdeaudhuij I. Using a co-creational approach to develop, implement and evaluate an intervention to promote physical activity in adolescent girls from vocational and technical schools: a case control study. Int J Environ Res Public Health. 2017;14(8):862.

Vincent B. Studying trans: recommendations for ethical recruitment and collaboration with transgender participants in academic research. Psychol Sex. 2018;9(2):102–16.

De Jaegher H, Peräkylä A, Stevanovic M. The co-creation of meaningful action: bridging enaction and interactional sociology. Philos Trans R Soc Lond B Biol Sci. 2016;371(1693):20150378.

Abboud S, Kim SK, Jacoby S, Mooney-Doyle K, Waite T, Froh E, Sefcik JS, Kim H, Sowicz TJ, Kelly T-A, et al. Co-creation of a pedagogical space to support qualitative inquiry: an advanced qualitative collective. Nurse Educ Today. 2017;50:8–11.

Benito Sánchez JC, Biotteau M, Boulard M, Demol N, Duval F, Guichart H, Greiss J, Ioannidis V, Mukana M, Peuch J, et al. L’expérience de l’aide alimentaire. Quelle(s) alternatives(s)? Bruxelles: ATD Quart Monde; 2019.

Marshall J. First person action research: living life as inquiry. London: SAGE Publications, Ltd; 2016.

Book   Google Scholar  

McDonnell L, Scott S, Dawson M. A multidimensional view? Evaluating the different and combined contributions of diaries and interviews in an exploration of asexual identities and intimacies. Qual Res. 2017;17(5):520–36.

ErichsenAndersson A, Frödin M, Dellenborg L, Wallin L, Hök J, Gillespie BM, Wikström E. Iterative co-creation for improved hand hygiene and aseptic techniques in the operating room: experiences from the safe hands study. BMC Health Serv Res. 2018;18(1):2.

Mustanski B. Ethical and regulatory issues with conducting sexuality research with LGBT adolescents: a call to action for a scientifically informed approach. Arch Sex Behav. 2011;40(4):673–86.

Martin JI, Meezan W. Applying ethical standards to research and evaluations involving lesbian, gay, bisexual, and transgender populations. J Gay Lesbian Soc Serv. 2003;15(1–2):181–201.

Miller RL, Forte D, Wilson BDM, Greene GJ. Protecting sexual minority youth from research risks: conflicting perspectives. Am J Community Psychol. 2006;37(3–4):341–8.

PubMed   Google Scholar  

Sims JP, Nolen C. “I wouldn’t trust the parents to ‘do no harm’ to a queer kid”: rethinking parental permission requirements for youth participation in social science research. J Empir Res Hum Res Ethics. 2021;16(1–2):35–45.

UN General Assembly. Convention on the rights of the child. Geneve: UN General Assembly; 1989.

Aldridge J. Participatory research: interpretation, representation and transformation. In: Participatory research: working with vulnerable groups in research and practice. Bristol: Policy Press; 2015. p. 121–60.

Chen PG, Diaz N, Lucas G, Rosenthal MS. Dissemination of results in community-based participatory research. Am J Prev Med. 2010;39(4):372–8.

Kennan D. Understanding the ethical requirement for parental consent when engaging youth in research. In: Bastien S, Holmarsdottir HB, editors. Youth ‘At the Margins’ critical perspectives and experiences of engaging youth in research worldwide. Volume 4. Rotterdam: Sense Publishers; 2015. p. 85–101.

Skelton T. Research with children and young people: exploring the tensions between ethics, competence and participation. Child Geogr. 2008;6(1):21–36.

Panfil VR, Miller J, Greathouse M. Utilizing youth advocates and community agencies in research with LGBTQ young people: ethical and practical considerations. Sociol Stud Child Youth. 2017;22:35.

Schrager SM, Steiner RJ, Bouris AM, Macapagal K, Brown CH. Methodological considerations for advancing research on the health and wellbeing of sexual and gender minority youth. LGBT Health. 2019;6(4):156–65.

Smith AU, Schwartz SJ. Waivers of parental consent for sexual minority youth. Account Res. 2019;26(6):379–90.

Taylor CG. Counterproductive effects of parental consent in research involving LGBTTIQ youth: international research ethics and a study of a transgender and two-spirit community in Canada. J LGBT Youth. 2008;5(3):34–56.

Powell MA, Taylor NJ, Fitzgerald R, Graham A, Anderson D. Ethical research involving children. Florence: Innocenti, Florence UNICEF Office of Research; 2013.

James T, Platzer H. Ethical consideration in qualitative research with vulnerable groups: exploring lesbians’ and gay men’s experiences of health care - a personal perspective. Nurs Ethics. 1999;6(1):73–81.

Bettencourt GM. Embracing problems, processes, and contact zones: using youth participatory action research to challenge adultism. Act Res. 2020;18(2):153–70.

Greenhalgh T, Jackson C, Shaw S, Janamian T. Achieving research impact through co-creation in community-based health services : literature review and case study. Milbank Q. 2016;94(2):392–392.

Download references

The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Author information

Authors and affiliations.

School of Public Health, Université Libre de Bruxelles, CP 596, Route de Lennik, 808, Brussels, 1070, Belgium

Julie Servais, Bram Vanhoutte, Sandra Tricas-Sauras & Isabelle Godin

Faculty of Psychology and Education, Université Libre de Bruxelles, CP 122, Avenue F.D. Roosevelt, 50, Brussels, 1050, Belgium

Sara Aguirre-Sánchez-Beato

Health and Society Research Institute – UCLouvain, Clos Chapelle-Aux-Champs 30/B1.30.15, Woluwe-Saint-Lambert, 1200, Belgium

Isabelle Aujoulat

Faculty of Social and Political Sciences – UNIL, CH-1015, Lausanne, Switzerland

Cynthia Kraus

Faculty of Medicine and Health Sciences, Ghent University Hospital, C. Heymanslaan 10, Ghent, 9000, Belgium

Guy T’Sjoen

You can also search for this author in PubMed   Google Scholar

Contributions

Writing – original draft preparation: JS. Writing – review and editing: BV, IG, CK, SASB, IA, GTS, STS. Final review prior to submission: JS, BV, IG.

Corresponding author

Correspondence to Julie Servais .

Ethics declarations

Ethics approval and consent to participate.

This protocol was approved by the Ethics Committee of the Erasme Faculty Hospital (CCB B4062023000140).

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Servais, J., Vanhoutte, B., Aguirre-Sánchez-Beato, S. et al. Integrating perspectives of transgender and gender-diverse youth, family members, and professionals to support their health and wellbeing – a mixed-method study protocol. Arch Public Health 82 , 40 (2024). https://doi.org/10.1186/s13690-024-01270-z

Download citation

Received : 17 December 2023

Accepted : 11 March 2024

Published : 18 March 2024

DOI : https://doi.org/10.1186/s13690-024-01270-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Mixed-methods
• Transgender
• Gender-diverse
• Co-construction

Archives of Public Health

ISSN: 2049-3258

• General enquiries: [email protected]

This paper is in the following e-collection/theme issue:

Published on 18.3.2024 in Vol 26 (2024)

Outcomes and Costs of the Transition From a Paper-Based Immunization System to a Digital Immunization System in Vietnam: Mixed Methods Study

Authors of this article:

Original Paper

• Thi Thanh Huyen Dang 1 , MD, PhD   ; 
• Emily Carnahan 2 , MPH   ; 
• Linh Nguyen 3 , MPH   ; 
• Mercy Mvundura 2 , PhD   ; 
• Sang Dao 3 , MPH   ; 
• Thi Hong Duong 1 , MD, PhD   ; 
• Trung Nguyen 1 , MPH   ; 
• Doan Nguyen 1 , MD   ; 
• Tu Nguyen 3 , MSc   ; 
• Laurie Werner 2 , MPA   ; 
• Tove K Ryman 4 , MPH, PhD   ; 
• Nga Nguyen 3 , MD, PhD  

1 National Expanded Program on Immunization, National Institute of Hygiene and Epidemiology, Hanoi, Vietnam

2 PATH, Seattle, WA, United States

3 PATH, Hanoi, Vietnam

4 Bill & Melinda Gates Foundation, Seattle, WA, United States

Corresponding Author:

Nga Nguyen, MD, PhD

1101, 11th floor, Hanoi Towers

49 Hai Ba Trung Street

Hanoi, 100000

Phone: 84 243936221 ext 130

Email: [email protected]

Background: The electronic National Immunization Information System (NIIS) was introduced nationwide in Vietnam in 2017. Health workers were expected to use the NIIS alongside the legacy paper-based system. Starting in 2018, Hanoi and Son La provinces transitioned to paperless reporting. Interventions to support this transition included data guidelines and training, internet-based data review meetings, and additional supportive supervision visits.

Objective: This study aims to assess (1) changes in NIIS data quality and use, (2) changes in immunization program outcomes, and (3) the economic costs of using the NIIS versus the traditional paper system.

Methods: This mixed methods study took place in Hanoi and Son La provinces. It aimed to analyses pre- and postintervention data from various sources including the NIIS; household and health facility surveys; and interviews to measure NIIS data quality, data use, and immunization program outcomes. Financial data were collected at the national, provincial, district, and health facility levels through record review and interviews. An activity-based costing approach was conducted from a health system perspective.

Results: NIIS data timeliness significantly improved from pre- to postintervention in both provinces. For example, the mean number of days from birth date to NIIS registration before and after intervention dropped from 18.6 (SD 65.5) to 5.7 (SD 31.4) days in Hanoi ( P <.001) and from 36.1 (SD 94.2) to 11.7 (40.1) days in Son La ( P <.001). Data from Son La showed that the completeness and accuracy improved, while Hanoi exhibited mixed results, possibly influenced by the COVID-19 pandemic. Data use improved; at postintervention, 100% (667/667) of facilities in both provinces used NIIS data for activities beyond monthly reporting compared with 34.8% (202/580) in Hanoi and 29.4% (55/187) in Son La at preintervention. Across nearly all antigens, the percentage of children who received the vaccine on time was higher in the postintervention cohort compared with the preintervention cohort. Up-front costs associated with developing and deploying the NIIS were estimated at US $0.48 per child in the study provinces. The commune health center level showed cost savings from changing from the paper system to the NIIS, mainly driven by human resource time savings. At the administrative level, incremental costs resulted from changing from the paper system to the NIIS, as some costs increased, such as labor costs for supportive supervision and additional capital costs for equipment associated with the NIIS.

Conclusions: The Hanoi and Son La provinces successfully transitioned to paperless reporting while maintaining or improving NIIS data quality and data use. However, improvements in data quality were not associated with improvements in the immunization program outcomes in both provinces. The COVID-19 pandemic likely had a negative influence on immunization program outcomes, particularly in Hanoi. These improvements entail up-front financial costs.

Introduction

Since 2017, the National Immunization Information System (NIIS) in Vietnam has been used nationwide by immunization facilities from the national, provincial, district, and commune levels to capture, store, and access immunization data [ 1 ]. The NIIS is a digital system that includes an immunization registry comprising individual-level, longitudinal information on vaccine doses administered and a logistics management system for vaccines and related supplies. As of June 2022, the NIIS has recorded data from approximately 31 million clients and has been used across 15,000 immunization facilities.

Although there is a growing body of evidence on the outcomes associated with digital systems to support vaccine service delivery in low- and middle-income countries [ 2 - 9 ], there are still many questions regarding how they affect data quality, use, and vaccination outcomes in practice. A challenge that has emerged in multiple country contexts is the parallel or dual reporting required when digital systems are introduced and health workers are expected to continue to use the legacy paper-based forms in addition to the new digital system. For example, Tanzanian facilities that had transitioned entirely to using an electronic immunization registry had higher odds of system use compared with those maintaining parallel electronic and paper-based systems [ 10 ].

Moreover, there is limited evidence on the costs of development and implementation of the system, recurrent costs of the system, and cost implications of eliminating a parallel paper system at the service delivery level [ 11 , 12 ]. The Vietnamese experience of introducing and scaling digital tools for immunization presents an opportunity to help fill these evidence gaps.

History of the NIIS

Before the introduction of the NIIS, a paper-based immunization registry and vaccine stock management system were used. Health workers captured data on paper forms, which were compiled in a monthly report. The paper-based system created a significant workload for health workers, and the immunization data were often delayed or incomplete, limiting the availability of reliable data at the higher levels of the health system [ 13 ].

From 2009 to 2012, the National Expanded Program on Immunization (NEPI), in collaboration with PATH and the World Health Organization, developed and piloted a logistics management tracking system for vaccines and related supplies (VaxTrak) and an immunization registry (ImmReg) at the commune and district levels. After the pilot phase, ImmReg expanded to all districts in the pilot province, and VaxTrak was scaled nationwide. An evaluation of ImmReg in 2015 showed that the system was highly accepted by health workers and improved vaccine coverage and timeliness [ 6 ]. In 2014 and 2015, NEPI and PATH integrated ImmReg and VaxTrak into a single comprehensive software system. From 2016 to 2018, the Vietnam General Department of Preventive Medicine and PATH developed the NIIS based on the pilot software and scaled it nationwide [ 1 ].

Transition to a Paperless System

Beginning in 2018, the government of Vietnam collaborated with PATH to provide technical support to strengthen NIIS implementation and transition to a paperless system in 2 provinces: Hanoi and Son La. This work was funded by the Bill & Melinda Gates Foundation and was implemented from 2018 to 2022.

We hypothesized that transitioning to a paperless system would improve data quality and data use in the intervention areas. We hypothesized that if data quality and data use improved, these changes could lead to improvements in immunization program outcomes. The objective of this study was to examine the short-term outcomes and costs associated with the NIIS and the transition to paperless reporting, focusing on three main categories:

• What are the changes in data quality and data use because to the paperless transition interventions?
• What are the changes in immunization program outcomes (timeliness, dropout rates, and coverage) because of the paperless transition interventions?
• What are the incremental financial costs associated with developing, deploying, and maintaining the NIIS, including the economic cost implications of transitioning to paperless reporting?

This mixed methods study aimed to evaluate the short-term outcomes and costs associated with the transition to a paperless system. Mixed methods were used to quantify the observed changes in outcomes and costs and to qualitatively describe why and how changes occurred. Pre-post analyses were conducted to understand the short-term changes in data quality, data use, and immunization program outcomes. Financial data were extracted from the project and partners’ records, and interviews were conducted at various levels of the health system to inform the cost analysis. This study was conducted from July 2019 to November 2021.

This study was conducted in 2 intervention provinces, Hanoi and Son La. These provinces were selected for the transition to a paperless system because they have a variety of geographic, demographic, and health system characteristics that may influence digital readiness. Hanoi, the capital city, is primarily urban with a high population density, high immigration rate, good infrastructure, and many private-sector and fee-based facilities. In contrast, Son La is a mountainous border province primarily composed of rural districts and has low population density, limited resources, and fee-based facilities.

Each province provides immunizations in public district health centers, hospitals, and commune health centers (CHCs) as well as private fee-based immunization facilities (FIFs). Primary data collection for this study also occurred at the national level, for example, to capture financial costs from the project and partners related to the development of the NIIS.

The transition to a paperless system and overall immunization activities in Vietnam were impacted by the COVID-19 pandemic, starting in 2020. The government mandated social distancing lockdowns multiple times in each province, which meant that individuals were not allowed to leave their home without special authorization, and nonessential businesses were closed. Immunization services were disrupted and, in some cases, unavailable, as health care workers were occupied with the COVID-19 response.

Multimedia Appendix 1 includes an overview of the characteristics of the study provinces (Table S1 in Multimedia Appendix 1 ) and dates when COVID-19 social distancing was applied in each province (Table S2 in Multimedia Appendix 1 ).

Interventions

A technical working group composed of the Ministry of Health, NEPI, PATH, and Viettel (the NIIS developer) oversees the implementation of the NIIS. A readiness assessment was conducted from June to July 2019 to provide the technical working group with information about the progress, needs, and challenges of transitioning to a fully paperless immunization system [ 14 ]. NEPI and PATH designed interventions (summarized in Textbox 1 ) to support the transition to paperless reporting based on the readiness assessment results. Interventions included detailed implementation guidelines and standard operating procedures for the transition to paperless reporting, internet-based data review meetings, additional supportive supervision visits, and Zalo (Vietnam’s popular social media and chat app) groups for end users to exchange knowledge and experiences in NIIS use.

Key interventions

• Guidelines and training on the shift to paperless reporting : implementation guidelines and standard operating procedures for the transition to paperless were implemented through a training of trainers and cascaded training approach for health workers.
• Data quality and data use guidelines and training for health workers at the province, district, and commune health center levels.
• Internet-based data review meetings at the district level where all communes share progress on paperless transition. Challenges identified through these meetings were used to prioritize areas for support. Initially, these meetings were held monthly but later shifted to quarterly.
• Additional supportive supervision visits from the government and PATH at district and commune facilities to support data quality, data use, and the overall transition to paperless. During the COVID-19 pandemic, these shifted to internet-based supportive supervision visits. Internet-based supportive supervision guidelines and trainings were developed for districts and provinces.
• Zalo groups for end users (at least 1 National Immunization Information System [NIIS] user per facility) in each district to exchange knowledge and experiences in NIIS use. Zalo is a popular social media and chat app in Vietnam.

Facilities began transitioning to paperless reporting in November 2019 in Son La and in January 2020 in Hanoi. All facilities in both provinces have retired from the paper-based immunization management logbook and have completely transitioned to paperless reporting using the NIIS as of January 2020. Although all sites are officially reporting using the NIIS, some paper-based systems are still used to comply with various inspections, payment procedures, or requirements from other ministries.

In addition to the key interventions ( Textbox 1 ), an e-learning system and e-immunization cards were piloted at a smaller scale. The e-learning system was developed and piloted in 6 districts (in the 2 provinces) to train managers at the national, regional, provincial, and district levels and facility health workers (CHCs, FIFs, and hospitals) on using the NIIS. The e-immunization card, a mobile phone app that allows parents or clients to access their individual demographic information and vaccination data, was also developed and launched in the 2 provinces.

Study Design and Data Collection

Various methods were used to collect data related to each of the study aims (data quality and use, immunization performance, and costing) in the 2 provinces. This section describes the study design, data sources, and data collection approach for each study objective. Table S3 in Multimedia Appendix 1 summarizes the data collection methods across the 3 study aims.

Data Quality and Use

A pre- or postintervention study design was used to assess changes in data quality and use. Data collection included self-administered facility assessments, household surveys, and facility surveys conducted at a sample of CHCs, FIFs, and hospitals; details of the methodology have been published elsewhere [ 14 ]. The same methods were used for pre- and postintervention data collection, with the addition of in-depth interviews with Expanded Program on Immunization (EPI) officers at the district and commune levels at postintervention.

The self-administered facility assessment was sent via email to all immunization facilities to collect basic information regarding infrastructure, capacity, and NIIS data use.

Facility surveys were conducted on a purposively selected sample of districts, communes, hospitals, and FIFs in each province. Purposive sampling was performed in consultation with NEPI and provincial Centers for Disease Control and Prevention (CDC) to select a mix of facility types (fee based, private, and public), geographies (urban, semiurban, rural, and mountainous), and experiences with NIIS. The study was designed to survey the same facilities at pre- and postintervention. At preintervention, 8 FIFs and 7 hospitals were included; at postintervention, 5 FIFs and 7 hospitals were included (the change was owing to 3 FIFs that had closed by the time of the postintervention survey). In each facility, 20 clients in the paper logbook were randomly selected. The facility surveys captured structured information about the facility, the use of the NIIS, and demographic and immunization information for the 20 sampled clients.

A household survey was conducted in a sample of households with children aged <2 years to capture demographic and immunization information about the children from their home-based immunization cards. The household survey was conducted in the same purposively selected communes as the facility surveys. Within each commune, villages or living quarters were selected for convenience, and all households with children in the defined age range were included.

The NEPI and PATH staff trained data collectors who used KoboToolbox platform for data entry for all data collection forms. More details on the sampling approach and a full list of facilities included in the different evaluations conducted as part of this study are included in Table S4 in Multimedia Appendix 1 .

Structured interviews were conducted to understand the factors related to data quality and data use. CDC staff led the interviews with immunization managers at the district level and health workers at the commune level. Interviews involving NEPI and CDC personnel were performed by the PATH staff. Interviews were conducted over the phone and in person at the interviewees’ workplaces, lasting around 35 minutes. The interviews were recorded with consent and transcribed.

Immunization Program Outcomes

A pre- or postintervention study design was used to assess changes in immunization program outcomes in the 2 provinces using NIIS data. NIIS data were exported for a pre- and postintervention cohort of children in each province. Due to the differing commencement dates of paperless reporting in 2 project provinces was different, Son La began in November 2019, while Hanoi followed in Jan 2020. As a result, the preintervention cohort group comprises children born between July 1 and September 30, 2018, while the postintervention group comprises those born between July 1 and September 30, 2020. Each child’s immunization information was analyzed for their first 12 months of life.

The costing study used a mixed methods approach including primary data collection using a microcosting approach and secondary data collection from financial record reviews. The costs of all activities were estimated from the perspectives of the implementing organizations (Hanoi and Son La provinces for NIIS systems use and Viettel, PATH, and NEPI) and hence take the health system perspective. No client costs were included.

We captured the costs for the different activities of implementing the NIIS from the software design and development activities to the deployment, accounting for the costs of the different partners engaged in this process. For the partner leading the software design, development, and deployment, we include the costs of the infrastructure, server, bandwidth, technical support, help desk, training, and maintenance and operations of the system. For the NEPI and the subnational levels, the costs included those for development of training materials, conducting staff training, data entry, meetings, and internet setup, at facilities where it was needed.

Data on the incremental financial costs for designing, developing, and deploying the NIIS software were obtained through the NEPI and partner organization expenditure records review. We also obtained information from each EPI administration and health facility in the sample on the expenditures for NIIS-related training and meetings and other deployment costs, including costs for data back entry and internet or phone setup at facilities.

Data on the recurrent financial costs of the NIIS were obtained via interviews with the head of each facility or person in charge of the facility finances in each study facility in the 2 provinces. The NEPI provided records on hardware inventory and repair requests, which were used to estimate replacement rates and maintenance costs for equipment.

Data Analysis

This section describes the variables and data analysis approach for each study objective.

For qualitative data analysis, we first transcribed all the interviews. Then, a member of the research team, who was trained on qualitative data methods, was assigned to code the transcripts using Microsoft Excel. This coding process followed a content analysis approach and involved a 3-level coding process: initially, open coding was applied to 5 transcripts to identify major themes; subsequently, the research team held discussions to reach a consensus on the major themes and any emergent themes; and finally, a final codebook was created before coding the remaining transcripts. This approach helped to gain a comprehensive understanding of health workers’ perspectives on improving data quality and data use from the NIIS as well as to identify the barriers and facilitators of immunization coverage.

In the analysis of quantitative data, we used Stata (version 14; StataCorp) as our statistical tool. For categorical variables, we used the chi-square test, and in cases where the expected cell counts were <5, Fisher exact test was used. For continuous variables, we first checked the variable distribution, and given the absence of a normal distribution, the Wilcoxon-Mann-Whitney U test was used to compare group differences. In addition, to investigate the relationship between immunization outcomes and its determinants, we conducted a multivariable logistic regression analysis. The threshold for statistical significance was set at P <.05.

The data use outcome of interest was the percentage of facilities using NIIS data to inform specified routine activities (eg, making monthly vaccination plans).

The data quality outcomes of interest were the quantitative measures of timeliness, completeness, and accuracy. Information from the household and health facility surveys was compared with the NIIS data to assess data quality. Refer to Table S5 in Multimedia Appendix 1 for the definitions of the data quality indicators.

Data exported from the NIIS were cleaned. Duplicate records, unreliable data (eg, vaccination date before birth date), and records with “lost to follow-up” status were excluded. The primary outcome of interest was on-time vaccination, determined by the recommended age for vaccine delivery according to the NEPI vaccination schedule [ 15 ]. The secondary immunization program outcomes of interest were dropout rates and full vaccination coverage. Multimedia Appendix 1 provides details on NIIS data cleaning and definitions for the primary and secondary outcomes of interest (Table S6 in Multimedia Appendix 1 ).

Quantitative data were analyzed using Stata (version 14). We computed the total incremental costs per health system level associated with the NIIS implementation. We also estimated the cost per child for the NIIS implementation activities. For this analysis, we allocated a proportion of the costs for NIIS implementation to the 2 provinces according to their annual birth cohort size relative to the national birth cohort. The annual birth cohort for Vietnam in 2019 was approximately 1.5 million [ 16 ], whereas the 2 study provinces (Hanoi and Son La) had an annual birth cohort of 162,000 and 25,000, respectively, based on data from the NIIS, representing approximately 12% (187,000/1,500,000) of the annual birth cohort of Vietnam. The NIIS implementation costs were spread over 5 birth cohorts in these cost-per-child calculations, as the system implementation was done over the 5-year period.

To estimate the economic costs associated with service delivery and reporting using either the paper-based system, electronic system, or both, we used an activity-based costing approach. Ingredients or components of the activities were quantified for each resource type, including human resource time use for different immunization activities where there would be a change when using the NIIS versus the paper system, capital costs for equipment and supplies, and recurrent costs for internet connectivity and equipment maintenance attributable to using the NIIS or the paper system. The quantification was done by conducting interviews at the study facilities using structured costing questionnaires. The unit cost of each resource was obtained from secondary data sources, and the total costs for each activity by resource type were estimated. To obtain the total costs, we aggregated the costs for each activity by resource type. We estimated the recurrent costs per facility and per child, with the latter based on only 1 birth cohort as these are annual recurrent costs.

Ethical Considerations

This study served as the end-line evaluation activity within the project’s work plan. The project was a collaborative effort between the NEPI and PATH. This evaluation constitutes 1 of the project activities outlined in the project documents submitted to the Vietnam Ministry of Health. As per the regulations set forth by the Vietnamese government, the project documents were reviewed and certified by units within the Ministry of Health and other relevant ministries before receiving approval from the Ministry of Health. The study was reviewed, considered as project evaluation, and approved by the Vietnam Ministry of Health (decision 1996/QĐ-BYT), and this study does not require ethics review in accordance with the circular 04/TT-BYT issued by the Vietnam Ministry of Health [ 17 ]. This circular [ 17 ] regulates the establishment, functions, tasks, and rights of research ethics committees, and it is specified that only research involving human subjects necessitates research ethics committee approval before implementation and supervision during the research process. In addition, the study was reviewed by PATH’s US-based Research Determination Committee, which concluded that the activity did not involve “human subjects” as defined in the US Government 45 Code of Federal Regulations 46.102(e) [ 18 ] and did not require US ethics review.

In addition, before conducting the interviews, comprehensive information was provided to all participants, encompassing the study’s objectives, participant rights, and strict confidentiality measures applied to protect their personal information. Informed consent was diligently obtained from each participant before starting the interviews. Their consent to participate in the study was obtained before proceeding with the interviews. Each qualitative interviewee received VN ₫150.000 (US $6.50) as payment for their time. In facilities selected for the costing study, health facility staff received VN ₫400.000 (US $17) for their time participating in structured costing interviews. Interviews were conducted in private settings to ensure confidentiality. All information was coded and only accessible to the study team, and data privacy was emphasized during training of the data collection team. The NEPI provided official permission for the use of the data extracted from the NIIS. All identifying data were coded, and names were eliminated before data analysis.

Data quality and use were measured through the NIIS data export, household surveys, and facility surveys and further explained through qualitative interviews.

Data Quality

The NIIS data quality evaluation considered the attributes of timeliness, completeness, and accuracy.

On the basis of the data exported from the NIIS, timeliness significantly improved from pre- to postintervention for all indicators across all health system levels in both provinces ( Table 1 ). Between pre- and postintervention, there was a significant decrease in the mean number of days from birth date to NIIS registration (Hanoi: 18.6, SD 65.5 to 5.7, SD: 31.4 days; P <.001 and Son La: 36.1, SD 94.2 to 11.7, SD 40.1; P <.001). Across all health system levels (CHCs, FIFs, and hospitals), there were significant decreases in the mean number of days from the injection date to when the injection was updated in the NIIS. Stock transactions (only assessed at the CHC level) also showed a significant decrease in the mean number of days from stock arrival date to NIIS voucher date (Hanoi: 10.5, SD 36.1 to 5.2, SD: 19.8 days; P <.001 and Son La 13.4, SD 38.1 to 6.5, SD 23.5; P <.001).

a N/A: not applicable.

b NIIS: National Immunization Information System.

c Dependent variables were not normally distributed; therefore, the Wilcoxon-Mann-Whitney U test was used.

Completeness

Completeness was assessed by comparing information from the household and facility surveys with the information from the NIIS ( Table 2 ). At the CHC level, completeness of registration, client information, and injection information captured in the NIIS significantly improved between pre- and postintervention in Hanoi and Son La. At the FIF level in Hanoi, there was a decline in the percentage of clients registered in the NIIS; however, among those registered, there was an increase in the completeness of client information. At the FIF level in Son La, there was increased completeness of registration, client information, and immunization information. At the hospital level, there was an improvement in the percentage of clients registered in the NIIS and a decline in the completeness of client information in both provinces, and the completeness of immunization information remained unchanged at 100% (Hanoi: preintervention n=63, postintervention n=153; Son La: preintervention n=41, postintervention n=74) in both provinces at pre- and postintervention.

c HH: household.

d FIF: fee-based immunization facility.

The accuracy of demographic and immunization information was assessed by comparing information on clients’ personal immunization cards (captured through the household survey) with their information entered in the NIIS. The percentage of clients with demographic information matched between the 2 sources significantly increased in Hanoi (199/217, 91.7% to 340/353, 96.3%; P =.02) and Son La (101/119, 84.9% to 107/107, 100%; P <.001) from pre- to postintervention ( Table 3 ). The percentage of injections with immunization information matched between the 2 sources also significantly increased in Son La (1037/1216, 85.3% to 1097/1097, 100%; P <.001) but significantly decreased in Hanoi (2147/2188, 98.1% to 3786/3981, 95.1%; P =.01).

The interviews indicated that respondents at all levels (national, provincial, district, and facility levels) and across both provinces had a strong understanding of data quality, defined as timeliness, completeness, and accuracy. Most respondents (13/16, 81%) participated in the intervention training on data quality and data use and indicated that it was useful for their work and that they had applied what they learned:

I have participated in the training course on data quality and data use last year. After the training, my knowledge and skill in data quality and data use are better, so I applied to my daily work, I usually check the input data to make them complete and accurate before entering into the system. [CHC staff]

All respondents rated their facility’s data quality in the NIIS as “good” or “very good” in terms of timeliness, completeness, and accuracy. All respondents mentioned human resources as the most important factor associated with data quality, including health workers’ knowledge and skills (in data entry, analysis, quality assessment, and use), understanding the importance of data quality, and bandwidth to support the immunization program when working across health areas.

Data use was measured through facility assessments asking health workers about the activities that the NIIS data were used to inform. In the preintervention survey, 34.8% (202/580) of the facilities in Hanoi and 29.4% (55/187) of the facilities in Son La indicated that they had used the NIIS data for additional activities beyond monthly reporting. In the postintervention survey, 100% (Hanoi: 468/468 and Son La: 199/199) of the facilities in both provinces indicated using the NIIS data for activities beyond monthly reporting. Table S7 in Multimedia Appendix 1 shows the frequency by activity. At postintervention, the most common uses of data among facilities were to inform monthly vaccination plans, campaign plans, or annual immunization plans. At the management level, the most common use of the NIIS data was to evaluate the performance of health facilities. From the qualitative interviews, the most frequently mentioned obstacle to using the data was the lack of health workers’ capacity for data analysis and use.

On-time vaccination was the primary immunization performance outcome of interest. Secondary outcomes were dropout rates and vaccination coverage (refer to the Tables S8-12 in Multimedia Appendix 1 ).

Study Population Characteristics

Immunization outcomes were assessed by comparing the NIIS data for a cohort of children pre- and postintervention in the 2 provinces. After data cleaning, 81,485 children were included in the sample. Their population characteristics are summarized in Table S8 in Multimedia Appendix 1 . In Hanoi, there were small but significant differences in the ethnicity and rural and urban location of children in the pre- and postintervention cohorts. In Son La, there was also a significant difference in the ethnicity of children in the pre- and postintervention cohorts. In both provinces, there were significant differences in the percentage of children vaccinated primarily from FIFs versus CHCs at pre- and postintervention.

On-Time Vaccination by Antigen

Across all antigens, apart from the measles-containing vaccine first dose in Hanoi, the percentage of children who received the vaccine on time increased from pre- to postintervention, and nearly all increases were statistically significant ( Table 4 ).

a BCG: bacillus Calmette-Guérin.

b Penta: pentavalent.

c MCV1: measles-containing vaccine first dose.

In the multiple logistic regression analysis, children at postintervention were approximately 1.18 times as likely in Hanoi and 1.69 times as likely in Son La to receive timely administration of pentavalent (Penta) 3 compared with those in the preintervention cohort ( P <.001; Table 5 ). In Hanoi, there was no significant difference in timely Penta 3 vaccination by gender or ethnicity, but children in urban areas were 1.6 times as likely to receive Penta 3 on time compared with those in rural areas. In Son La, there was also no difference by gender, but Thai children were 1.3 times as likely to receive Penta 3 on time compared with other ethnicities. In both provinces, children who were mostly vaccinated from FIFs were more likely to receive Penta 3 on time compared with children mostly vaccinated from CHCs.

A separate logistic regression analysis for on-time full vaccination is included in Table S12 in Multimedia Appendix 1

On-Time Full Immunization Coverage

Figure 1 shows the full immunization coverage over time for each birth cohort. In Hanoi, 87.8% (29,649/33,752) of the children in the preintervention cohort and 81.9% (29,167/35,611) of the children in the postintervention cohort ( P <.001) had reached full immunization before their first birthday. In contrast, there was an increase in the percentage of children reaching full immunization before their first birthday in Son La, from 63.3% (3947/6233) preintervention to 88.3% (5040/5705) postintervention ( P <.001). The results for immunization coverage by antigen for the pre- and postintervention birth cohorts in each province are included Table S9 in Multimedia Appendix 1 .

Up-Front Costs of NIIS Implementation

The up-front NIIS software design, development, and deployment took approximately 2300 person-months of labor from 2015 to 2020, and the partner costs for this labor were estimated at approximately US $1.75 million ( Table 6 ). Most of the software developer costs (US $1,233,152/US $1,745,712, 71%) pertained to the deployment of the system. In addition, there were up-front costs per facility for implementing the NIIS, including costs for back entry of data, internet setup (where needed), training of users, and meeting costs. At the national and provincial levels, the bulk of the up-front costs were spent on training and meetings. As trainings were paid for by the higher administrative levels, training costs are low or 0 at district levels and health facilities. At these levels, the larger cost share was for deploying the NIIS.

a NIIS: National Immunization Information System.

The up-front costs for the NIIS implementation allocated to the 2 study provinces were approximately US $419,000 ( Table 7 ). When these costs are allocated over 5 birth cohorts, the estimated cost per child for the NIIS implementation was estimated at US $0.48.

b NEPI: National Expanded Program on Immunization.

Recurrent Costs for the NIIS

The software developer estimated the annual recurrent costs for the system operation to be US $85,000. At health facilities, the average monthly economic cost for health worker labor for immunization-related activities done using the paper system were estimated to be US $146 ( Table 8 ). In comparison, the monthly cost for labor with the NIIS was US $67, which is less than half of the labor cost when using the paper system. The most significant savings in labor time costs, resulting from the transition from the paper system to the NIIS, occurred through reduced time spent on organizing immunization sessions, data management, and reporting. However, the NIIS also resulted in additional activities for staff, including checking of duplicates and introduction of e-immunization cards. The NIIS also added new facility costs, including recurrent costs for internet, printing, and SMS text messaging reminders and the capital costs of equipment, which amounts to an average cost of US $28 per facility. However, with the NIIS, there were savings in printing costs as registers and ledgers would not be printed, and the average costs of these were US $4.51 per month or US $58 per year. Overall, the total monthly costs per facility with NIIS (US $95) are lower than with the paper system (US $151).

Table 9 presents similar costs for the administrative levels. At most levels, except at provincial CDC, labor costs are lower with the NIIS than with the paper system. At the CDC, the labor costs for supportive supervision are the largest share of costs, and these costs increase when using the NIIS compared with when using the paper system. At the district health centers, there is a decline in labor costs for activities such as management and reporting when using the NIIS, and hence, labor costs are lower with the NIIS. As mentioned above, implementing the NIIS incurs additional costs, including recurrent costs for internet, printing, and equipment maintenance and capital costs for equipment, which makes the total monthly costs for NIIS more than the total monthly costs for the paper system. There are incremental costs resulting from the change from the paper system to the NIIS at the administrative levels.

b N/A: not applicable.

Although labor costs decrease at most health system levels with the transition to the NIIS, in practice, this may not translate into budget line savings as staff are retained and their time is reallocated. The financial costs associated with the paper system primarily entail printing registers, but these expenses are relatively minor in comparison with the those incurred with the electronic system. The estimated annual financial recurrent costs per child for the NIIS are US $3.17, and these account for the annual costs of the server (with the relevant portion allocated to the study provinces) and the costs for capital equipment and internet. There are incremental financial costs to the health system when moving from the legacy paper system to the electronic system.

Principal Findings

Health workers at the province, district, and CHC levels successfully transitioned from the legacy paper-based system to the NIIS for paperless reporting, proving that the transition was possible in 2 very different provincial contexts. Although the transition to paperless reporting results in lower labor costs at the facility, district, and national levels, it requires incremental financial recurrent costs (estimated at US $3.17 per child per year) to maintain the NIIS.

This study found improvements in data quality in Hanoi and Son La between pre- and postintervention. There were significant improvements in data timeliness at all levels of the health system and in both provinces. It is likely that the interventions contributed to this improvement because the guidelines, trainings, data review meetings, and supportive supervision visits emphasized the importance of timely data. Timeliness indicators performed better in Hanoi than in Son La, which may be because of the different service delivery practices; in Hanoi, vaccines are delivered on a weekly basic, whereas in Son La, they are delivered monthly. In addition, in Hanoi, more clients visit FIFs that deliver immunization daily.

In Son La, there were also improvements across nearly all the completeness and accuracy indicators. However, there were mixed results in terms of data completeness and accuracy in Hanoi, with some statistically significant declines in quality. This may be partly because Hanoi was more affected by the COVID-19 pandemic than Son La; there were 6 rounds of mandated social distancing across all 30 districts in Hanoi. When immunization services were available, facilities may have been understaffed if human resources were reassigned to the COVID-19 response and they may have faced higher demand from a backlog of children who could not be vaccinated during social distancing. Overburdened health workers may prioritize service delivery over data quality.

The observed differences in provincial results may also have been influenced by the way interventions were delivered in each province. The facilities in Son La received timely support through internet-based supervisory assistance. In Hanoi, they maintained in-person supportive supervision, which was less timely, and limited the number of facilities that could be visited. Facilities in Son La also received refresher trainings, which Hanoi did not, as Hanoi had an ongoing focus on COVID-19 vaccination.

There were large increases from pre- to postintervention in the percentage of facilities using the NIIS data to inform their activities. Results from the costing analysis showed that at the CHCs, health workers’ time spent on immunization program data entry and reporting declined as the transition to paperless resulted in efficiencies and reduced workload. It is possible that this saved time was dedicated to data use.

For most activities, a higher percentage of facilities in Son La reported using NIIS data compared with Hanoi. These results may have also been influenced by the COVID-19 pandemic or the differences in project interventions. In addition, Hanoi has a more mobile population, with clients frequently moving between facilities. This implies that vaccination plans informed by NIIS data may not be as accurate or comprehensive compared with a location such as Son La, where there is less population movement.

Although only 44.7% (84/188) of facilities in Son La and 27.7% (125/451) of facilities in Hanoi reported using the NIIS to evaluate data quality, previous studies have shown that data use can drive data quality improvements [ 19 ]. Through the qualitative interviews, health workers recommended building data quality indicators directly into the NIIS to help users understand their data quality easily.

The study also found that the majority of on-time vaccination rates for various antigens in the 2 project provinces improved compared to those in the preintervention group. With more timely data captured in the NIIS, health workers know which vaccines a child is due for and can send reminders or follow-up to deliver the vaccines on time. For the clients using the e-immunization card, it may have also contributed to timely vaccination by reminding clients of their vaccination schedule and due dates. However, the on-time vaccination rate for the first dose of measles in Hanoi decreased after the intervention compared to before, which may have been owing to the COVID-19 lockdowns from July to September 2021, when the postintervention cohort was aged 9 to 12 months. In Hanoi, many parents prefer waiting to receive the measles-rubella vaccine at 12 months, which is delivered at FIFs, versus the monovalent measles vaccine, which is part of the standard EPI schedule at 9 months.

In Son La, improvements in data quality and on-time vaccination were also associated with improvements in vaccination coverage. Full immunization at 12 months was 88.3% (5040/5705) in the postintervention cohort compared with 63.3% (3947/6233) in the preintervention cohort. A previous evaluation of ImmReg (the earlier version of the NIIS) in another province in Vietnam, Ben Tre, also found improvements in on-time vaccination and vaccination coverage when comparing pre- and postintroduction of the digital immunization registry, and these improvements were sustained or increased 1 year later [ 6 ]. However, in Hanoi, despite improvements in on-time vaccination, there were declines in vaccination coverage, again likely because of the COVID-19 pandemic. According to the 2021 annual EPI report for Vietnam, full immunization coverage decreased nationwide from 96.8% in 2020 to 87.1% in 2021 [ 20 ].

Introducing these new electronic systems, which have improved data quality and timeliness and could potentially improve immunization outcomes, comes at a cost, as up-front financial investments have to be made for software design, development, and deployment. The cost per child for NIIS development and deployment was estimated at US $0.48 in the 2 study provinces in Vietnam. This is much lower than that estimated in Tanzania and Zambia, the only other low- and middle-income countries with comparable cost estimates for the development of electronic immunization registries [ 11 , 12 ]. In our study, costs were annualized over 5 birth cohorts, whereas in Tanzania and Zambia, costs were annualized over 3 birth cohorts. Even if we were to use 3 birth cohorts for this analysis, our estimated costs for Vietnam would still be lower. The estimated lower costs per child for Vietnam could be in part because most CHCs had existing computers, and hence, there was no mass procurement of equipment required, which is not the case in many other countries. In Vietnam, the equipment and connectivity costs were shared across multiple health programs, further reducing the costs borne by the immunization program, unlike in Tanzania and Zambia, where immunization was the first health area to be digitalized at the health facility level. The availability of electricity at the health facility level in Vietnam also reduced costs, as there was no need for the procurement of alternative power sources such as solar chargers. In addition, software development in Vietnam was conducted by an in-country telecommunications partner that provided in-kind and pro bono services and support, some of which could not be quantified, resulting in lower costs than if done through an external organization.

Our study found that at the CHC level, there are cost savings for health workers’ time use and other resources such as printing with the NIIS compared with the paper system, and at the administrative level, there are incremental costs. The NIIS results in some savings in labor time, although some new activities are added for the staff. There are also recurring costs, such as ongoing technical support and maintenance provided by Viettel, the need for refresher training at the local level for new staff, and expenses associated with equipment maintenance and replacement as well as connectivity provided by local government. These recurrent costs are important to consider for sustainability.

Limitations

Our study has several limitations. First, we conducted the study in 2 provinces and had small sample sizes for the costing results, facility survey results for data quality, and other results based on provider interviews. The small sample size and purposive sampling limit the generalizability of our findings; however, 2 distinct provinces with a mix of facility types were intentionally sampled so that the results can be used to understand the costs and outcomes of the transition to paperless reporting in a range of settings. Second, the time frame for the evaluation was short, which limited the magnitude of the change that could be observed. Third, there were minor differences in the data collection approaches and responses between pre- and postintervention. For example, the urban sampling approach was adjusted for the feasibility of data collection, and the response rate for facility assessment was lower at postintervention. Fourth, for data collected through provider interviews, such as for the costing study, potential recall bias arises due to retrospective nature of the inquires, as interviewees were asked to recall cost from past periods. Future studies should consider including costing data collection at different phases of the project to reduce recall bias. This would also facilitate the availability of costing data available to inform decision-making at various implementation phases. Our study may also have underestimated the costs associated with the paper system, as we did not incorporate expenses related to printing paper home-based records and invitation letters. These materials are printed to assist the caregivers in monitoring their children’s vaccination history or reminding them when their child is due for vaccination. We also have not accounted for the saved labor costs for health care workers who would deliver the invitation letters to households. Another limitation is that the COVID-19 pandemic affected the originally planned timelines for data collection and the immunization service delivery. We did not try to account for the impact of the pandemic on findings regarding the data quality and data use presented in this study.

Conclusions

Health workers in the 2 provinces successfully transitioned to paperless reporting while maintaining or improving data quality. We recommend that other provinces in Vietnam transition to paperless reporting by introducing the guidelines and standard operating procedures used in Hanoi and Son La and providing ongoing support through trainings, data review meetings, supportive supervision, and peer networks. Future studies should monitor data quality and immunization outcomes in other provinces as well as the sustainability of the observed changes in Hanoi and Son La.

Introducing these new electronic systems comes with costs—both up-front and recurrent—but there are advantages, as seen in the improvements in data quality and on-time vaccination. In Vietnam, stakeholders should plan and budget for the sustainability of the system at each level of the health system, given the recurrent costs including repairing and replacement of equipment, connectivity, refresher training, software system, and supportive supervision. Other countries planning to implement similar interventions should plan to collect costing data throughout to inform decision-making and budgeting.

Acknowledgments

The authors thank the Bill & Melinda Gates Foundation for providing support for this study and the National Immunization Information System (NIIS) Technical Working Group members for their leadership in designing, developing, and deploying the NIIS. The authors thank the data collectors who participated in this study. Finally, the authors acknowledge the invaluable collaboration of the health workers, managers, and leaders in Hanoi and Son La provinces.

Data Availability

The data sets generated during and analyzed during this study are available from the corresponding author on reasonable request.

Conflicts of Interest

None declared.

Supplementary tables.

• Dang H, Dao S, Carnahan E, Kawakyu N, Duong H, Nguyen T, et al. Determinants of scale-up from a small pilot to a national electronic immunization registry in Vietnam: qualitative evaluation. J Med Internet Res. Sep 22, 2020;22(9):e19923. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yohana E, Mitiku S, Kayumba PC, Swalehe O. Electronic immunization registry in improving vaccine supply chain availability in Tanga City Council, Tanzania. Rwanda J Med Health Sci. Sep 10, 2021;4(2):223-236. [ CrossRef ]
• Siddiqi DA, Abdullah S, Dharma VK, Shah MT, Akhter MA, Habib A, et al. Using a low-cost, real-time electronic immunization registry in Pakistan to demonstrate utility of data for immunization programs and evidence-based decision making to achieve SDG-3: insights from analysis of big data on vaccines. Int J Med Inform. May 2021;149:104413. [ CrossRef ] [ Medline ]
• Uddin MJ, Shamsuzzaman M, Horng L, Labrique A, Vasudevan L, Zeller K, et al. Use of mobile phones for improving vaccination coverage among children living in rural hard-to-reach areas and urban streets of Bangladesh. Vaccine. Jan 04, 2016;34(2):276-283. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chen L, Du X, Zhang L, van Velthoven MH, Wu Q, Yang R, et al. Effectiveness of a smartphone app on improving immunization of children in rural Sichuan Province, China: a cluster randomized controlled trial. BMC Public Health. Aug 31, 2016;16(1):909. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nguyen NT, Vu HM, Dao SD, Tran HT, Nguyen TX. Digital immunization registry: evidence for the impact of mHealth on enhancing the immunization system and improving immunization coverage for children under one year old in Vietnam. Mhealth. 2017;3:26. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gilbert SS, Thakare N, Ramanujapuram A, Akkihal A. Assessing stability and performance of a digitally enabled supply chain: retrospective of a pilot in Uttar Pradesh, India. Vaccine. Apr 19, 2017;35(17):2203-2208. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gilbert SS, Bulula N, Yohana E, Thompson J, Beylerian E, Werner L, et al. The impact of an integrated electronic immunization registry and logistics management information system (EIR-eLMIS) on vaccine availability in three regions in Tanzania: a pre-post and time-series analysis. Vaccine. Jan 16, 2020;38(3):562-569. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fritz J, Herrick T, Gilbert SS. Estimation of health impact from digitalizing last-mile logistics management information systems (LMIS) in Ethiopia, Tanzania, and Mozambique: a lives saved tool (LiST) model analysis. PLoS One. 2021;16(10):e0258354. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Carnahan E, Ferriss E, Beylerian E, Mwansa FD, Bulula N, Lyimo D, et al. Determinants of facility-level use of electronic immunization registries in Tanzania and Zambia: an observational analysis. Glob Health Sci Pract. Sep 30, 2020;8(3):488-504. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mvundura M, Di Giorgio L, Lymo D, Mwansa FD, Ngwegwe B, Werner L. The costs of developing, deploying and maintaining electronic immunisation registries in Tanzania and Zambia. BMJ Glob Health. Nov 25, 2019;4(6):e001904. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mvundura M, Di Giorgio L, Vodicka E, Kindoli R, Zulu C. Assessing the incremental costs and savings of introducing electronic immunization registries and stock management systems: evidence from the better immunization data initiative in Tanzania and Zambia. Pan Afr Med J. Feb 12, 2020;35(Suppl 1):11. [ FREE Full text ] [ CrossRef ] [ Medline ]
• World Health Organization, PATH. Optimize: Vietnam report. PATH. 2013. URL: https://media.path.org/documents/TS_opt_vietnam_rpt.pdf [accessed 2024-01-29]
• Duong H, Dao S, Dang H, Nguyen L, Ngo T, Nguyen T, et al. The transition to an entirely digital immunization registry in Ha Noi province and Son La Province, Vietnam: readiness assessment study. JMIR Form Res. Oct 25, 2021;5(10):e28096. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Minh An DT, Lee JK, Van Minh HV, Trang NT, Thu Huong NT, Nam YS, et al. Timely immunization completion among children in Vietnam from 2000 to 2011: a multilevel analysis of individual and contextual factors. Glob Health Action. Mar 01, 2016;9(1):29189. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vietnam: country information. Gavi, The Vaccine Alliance. URL: https://www.gavi.org/programmes-impact/country-hub/west-pacific/vietnam [accessed 2022-09-28]
• 4/TT-BYT on establishment, functions, tasks and rights of research ethics committees, 05 March 2020. Vietnam Ministry of Health. URL: https:/​/thuvienphapluat.​vn/​van-ban/​Bo-may-hanh-chinh/​Thong-tu-4-TT-BYT-2020-thanh-lap-chuc-nang-nhiem-vu-Hoi-dong-dao-duc-trong-nghien-cuu-y-sinh-hoc-440446.​aspx [accessed 2024-01-25]
• Home page. Code of Federal Regulations. URL: https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-A/part-46/subpart-A/section-46.102 [accessed 2024-01-25]
• Werner L, Seymour D, Puta C, Gilbert S. Three waves of data use among health workers: the experience of the better immunization data initiative in Tanzania and Zambia. Glob Health Sci Pract. Sep 2019;7(3):447-456. [ FREE Full text ] [ CrossRef ] [ Medline ]
• National immunization program and vaccine delivery the national expanded program on immunization. The World Bank Document. URL: https:/​/openknowledge.​worldbank.org/​server/​api/​core/​bitstreams/​022f4737-59de-4f41-960f-e82f05093402/​content [accessed 2024-02-27]

Abbreviations

Edited by T Leung, T de Azevedo Cardoso; submitted 14.12.22; peer-reviewed by V Horner, N Mejia, M Muhonde; comments to author 07.06.23; revised version received 28.07.23; accepted 26.01.24; published 18.03.24.

©Thi Thanh Huyen Dang, Emily Carnahan, Linh Nguyen, Mercy Mvundura, Sang Dao, Thi Hong Duong, Trung Nguyen, Doan Nguyen, Tu Nguyen, Laurie Werner, Tove K Ryman, Nga Nguyen. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 18.03.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

• Open access
• Published: 15 March 2024

Can the Reboot coaching programme support critical care nurses in coping with stressful clinical events? A mixed-methods evaluation assessing resilience, burnout, depression and turnover intentions

• K. S. Vogt 1 , 2 , 8 ,
• J. Johnson 1 , 2 , 3 ,
• R. Coleman 1 , 7 ,
• R. Simms-Ellis 1 , 2 ,
• R. Harrison 3 , 4 ,
• N. Shearman 5 , 11 ,
• J. Marran 1 ,
• L. Budworth 1 , 6 , 10 ,
• C. Horsfield 9 ,
• R. Lawton 1 , 2 &
• A. Grange 1  

BMC Health Services Research volume  24 , Article number:  343 ( 2024 ) Cite this article

143 Accesses

6 Altmetric

Metrics details

Critical care nurses (CCNs) are routinely exposed to highly stressful situations, and at high-risk of suffering from work-related stress and developing burnout. Thus, supporting CCN wellbeing is crucial. One approach for delivering this support is by preparing CCNs for situations they may encounter, drawing on evidence-based techniques to strengthen psychological coping strategies. The current study tailored a Resilience-boosting psychological coaching programme [Reboot] to CCNs. Other healthcare staff receiving Reboot have reported improvements in confidence in coping with stressful clinical events and increased psychological resilience. The current study tailored Reboot for online, remote delivery to CCNs (as it had not previously been delivered to nurses, or in remote format), to (1) assess the feasibility of delivering Reboot remotely, and to (2) provide a preliminary assessment of whether Reboot could increase resilience, confidence in coping with adverse events and burnout.

A single-arm mixed-methods (questionnaires, interviews) before-after feasibility study design was used. Feasibility was measured via demand, recruitment, and retention (recruitment goal: 80 CCNs, retention goal: 70% of recruited CCNs). Potential efficacy was measured via questionnaires at five timepoints; measures included confidence in coping with adverse events (Confidence scale), Resilience (Brief Resilience Scale), depression (PHQ-9) and burnout (Oldenburg-Burnout-Inventory). Intention to leave (current role, nursing more generally) was measured post-intervention. Interviews were analysed using Reflexive Thematic Analysis.

Results suggest that delivering Reboot remotely is feasible and acceptable. Seventy-seven nurses were recruited, 81% of whom completed the 8-week intervention. Thus, the retention rate was over 10% higher than the target. Regarding preliminary efficacy, follow-up measures showed significant increases in resilience, confidence in coping with adverse events and reductions in depression, burnout, and intention to leave. Qualitative analysis suggested that CCNs found the psychological techniques helpful and particularly valued practical exercises that could be translated into everyday practice.

This study demonstrates the feasibility of remote delivery of Reboot and potential efficacy for CCNs. Results are limited due to the single-arm feasibility design; thus, a larger trial with a control group is needed.

Peer Review reports

The healthcare professions are seen as some of the most stressful occupations, due to the close human contact, involvement with illness, death and dying, quick decision-making, risk of making errors and the involvement in adverse events they entail [ 1 , 2 , 3 , 4 , 5 , 6 ]. This stress and the demands on health care professionals (HCPs) have been exacerbated by the Covid-19 pandemic. Over the past 3 years, HCPs have had to cope with extreme emotional and physical stress, which has included redeployment, insufficient provision of medical supplies and personal protective equipment (PPE) and witnessing a record number of deaths among patients and colleagues. They have also been under pressure to adhere to ever-evolving infection control measures and have experienced anxiety about their personal health (as well as that of their families) [ 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ]. Out of all areas of healthcare, Critical Care has been the most significantly affected clinical area by Covid-19 [ 17 , 18 , 19 ]. This has had detrimental effects to the psychological wellbeing of staff working in critical care units and is especially true for critical care nurses (CCNs) [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 ].

The international literature has consistently identified CCNs as having the worst outcomes on psychological wellbeing measures, such as depression, burnout, and post- traumatic stress disorder (PTSD) both during, and since the pandemic, and both compared to other critical care HCPs, such as physicians, and compared to non-critical care HCPs [ 21 ]. Two studies that illustrate the impact of working as a critical care nurse during the pandemic were conducted by Greenberg et al. [ 22 ] and Moll et al. [ 21 ]. In the United Kingdom (UK), Greenberg et al. surveyed 709 HCPs working in Critical Care on nine intensive care units (ICUs). Out of the three groups (doctors, nurses and ‘other’ ), CCNs ( n  = 344; 49% of the sample) were significantly more likely to screen positive for depression (moderate, and severe), PTSD and anxiety (moderate, and severe). Further,19% of these nurses reported suicidal ideation [ 21 ]. In the US, Moll et al. compared the burnout scores of healthcare professionals working on critical care units between 2017 ( n  = 572, nurses n  = 323) and 2020 ( n  = 710, nurses n  = 372). Nurses were found to have the sharpest increase in burnout, despite increases in burnout across all professions surveyed. Taken together, these findings show that CCN wellbeing has been significantly impacted by the pandemic. Therefore, it essential that HCPs are supported in their wellbeing, and that they can draw on evidence-based techniques to recover from stressful events, without suffering negative psychological consequences.

Furthermore, poor CCNs wellbeing, the development of burnout and PTSD have been linked with intention to leave critical care nursing, and nursing altogether [ 15 , 23 , 27 , 28 , 29 ]. Thus, supporting CCNs’ wellbeing is not only a priority at an individual level (for individual CCNs), but must also be a priority at organizational level, to avoid further staff shortages [ 20 ]. One of the protective factors against the development of PTSD and burnout is psychological resilience [ 30 , 31 , 32 , 33 ]. Resilience refers to someone’s ability to maintain an emotional equilibrium during difficult experiences [ 34 ]. There is now increasing evidence that resilience can be increased with the help of psychological interventions [ 35 , 36 ]. The Recovery-boosting [“Reboot”] coaching programme evaluated in the current study seeks to enhance HCP resilience by providing them with evidence-based psychological tools to prepare and recover from stressful clinical events. Reboot aims to develop psychological constructs known to confer resilience, including higher self-esteem, greater mental flexibility, and a more positive explanatory style for negative events [ 19 , 37 , 38 ]. Reboot was first developed and piloted in 66 HCPs and healthcare students; groups included paediatric doctors, midwives, and physician associate students [ 37 ]. It consisted of one 4-hour workshop and one 1-hour coaching phone call. At follow-up, participants showed significantly higher levels of psychological resilience and confidence in coping with adverse events; suggesting the intervention was feasible and acceptable to participants and potentially effective for increasing resilience. Although these results were promising, nurses were not included in the study [ 37 ]. Therefore, the feasibility of Reboot for nurses remains to be established through further research [ 19 ].

The COVID-19 pandemic generated an increased need for psychological support for nurses, particularly CCNs, given their significant distress and worse psychological outcomes than other critical care professionals [ 22 , 25 ]. However, the pandemic also drastically reduced the feasibility of delivering in-person psychological interventions. Thus, the current study aimed to adapt the pre-existing Reboot programme for remote delivery for CCNs [ 19 ].

The primary objective was to assess the feasibility of delivering Reboot via online, remote delivery to CCNs. This was measured via demand, recruitment, and programme retention statistics.

The secondary objective was to provide a preliminary assessment of whether Reboot was associated with increases in self-reported psychological resilience and confidence in coping with adverse events, and decreases in depression and burnout, via analysis of questionnaires and interviews.

A more detailed report of the methods can be found in the open-access study protocol paper [ 19 ].

Study design & settings

A single-arm before-after feasibility study design was used; with a mixed-methods evaluation. Participants were invited to complete online questionnaires at five time points, which were Baseline (Time 1), following completion of two group workshops (Time 2), following completion of two individual coaching calls (Time 3), at 2-week follow-up post the final coaching call (Time 4). A fifth timepoint (Time 5) was added in May 2022 to investigate participants’ intention to leave nursing.

Online interviews were conducted with 25% of participants [who were randomly selected], after completion of the intervention. The Kirkpatrick model for assessing training interventions [ 39 ] was used, and four levels of outcome data were collected (Reaction, Learning, Behaviour, Results), as per Johnson et al. [ 37 ].

Ethics approval

Ethics approval was granted by the School of Psychology, University of Leeds Ethics committee (approved on 25-08-2021, PSYC-302; an ethics amendment was approved on 09-05-2022, PSYC-535). The study adheres to both the British Psychological Society’s Code of Ethics and Conduct, as well as Declaration of Helsinki.

Adaptation to online, remote delivery

Reboot was adapted for online delivery from a previous in-person group delivery method. This is reported in-depth in the study protocol [ 19 ]. The original intervention consisted of one 4-hour in-person group workshop and one 1-hour individual coaching phone call; and was delivered by a Clinical Psychologist (JJ) and an Occupational Health Psychologist (RSE) [ 37 ]. Adaptation for online, remote delivery involved changing this to two 2-hour online group workshops hosted via Zoom (each pair of workshops was termed a ‘cycle’), and two 1-hour individual coaching calls and was delivered by a Cognitive-behavioural (CBT) therapist (RC).

Participants

The recruitment target was 80 CCNs working in the National Health Service (NHS) in the UK. Full inclusion/exclusion criteria, and sample size justification can be found in the protocol paper [ 19 ].

Primary feasibility outcomes

As per protocol, feasibility outcomes were measured via demand [how many CCNs signed up], recruitment [how many CCNs consented and attended the first workshop], and retention [a) how many participants completed both workshops, b) how many participants completed both workshops and coaching calls, and c) how many completed the final follow-up questionnaires]. Using results of the in-person version of Reboot delivered by, feasibility success for the current study was met, if the following criteria were met:

at least 80 CCNs signed up to the study (demand)

at least 80 CCNs consented to taking part in the study and attend the first workshop (recruitment).

at least 90% of recruited CCNs complete both workshops

at least 70% completed both workshops and coaching calls, and

at least 50% of recruited CCNs complete all follow-up measures, up to Time 4.

Secondary outcomes

Secondary outcomes were resilience [measured via the Brief Resilience Scale (BRS)] [ 40 ]], confidence in coping with adverse events [measured via Confidence in Coping with Adverse Events Questionnaire [ 37 ]] , knowledge of resilience [measured via Knowledge Assessment [ 37 ] , Burnout [ measured via an abbreviated version of the Oldenburg Burnout Inventory (OLBI) [ 41 ]], and depression [measured via the Patient Health Questionnaire (PHQ-9) [ 42 ]]. Feedback and reactions to the Reboot workshops [assessed via “Feedback” questionnaire [ 37 ] were also assessed. Internal reliability coefficients for the measures are reported in the ‘ Results ’ section of this paper.

• Intention to leave

In addition to the above outcomes, an amendment was made to the original protocol to include a measure of intention to leave. All participants who completed the programme (both workshops, both coaching calls) were asked to answer an extra questionnaire as part of an additional follow-up survey. Participants were firstly asked to answer a set of questions measuring their turnover intentions as they recalled them prior to participating in Reboot (“ Think back to two weeks before you attended your first Reboot workshop, how were you feeling …?” and then a set of questions about their current turnover intentions (“How are you feeling now …?”). More specifically, intention to leave was measured via three items for the two time points (“I was/am planning to leave critical care nursing for another type of nursing”, “ I was/am planning to leave nursing altogether” and “ I was/am planning to continue working as a critical care nurse ” [reverse coded]) and answered on a scale from Strongly agree (1) to strongly disagree (5), with lower scores indicating lower intention to leave in critical care nursing.

Study information was circulated to the CCNs via Critical Care Networks and social media, via flyers, tweets, websites, and emails. A QR code could be used to access a website containing study information and a sign-up link. During sign-up, participants provided their details, and selected dates for workshops 1 and 2 from a list of cycles. Confirmation of dates was confirmed by email. Seven days prior to workshop 1, participants received an email with a questionnaire link, containing 1) consent form, 2) baseline survey, 3) a video to watch prior to attending the first workshop and 4) online links to access their workshops. Around the same time, participants also received a booklet in the post to use in the workshops, as well as a welcome phone call from the therapist facilitating the workshops. Both workshops took place via Zoom. At the end of the second workshop, the therapist asked participants to complete the Time-2 questionnaire and booked participants in for their two coaching calls. The coaching calls took place via phone or video call, depending on preference. After coaching call 2, participants completed Time-3 questionnaires, which they were sent by the therapist. Two to three weeks after the second coaching call, participants were emailed Time-4 questionnaires, and were invited to take part in an interview if selected (see Appendix 1 for Interview Guide). Interviewees were selected via random number generation from 0 to 100, numbers were assigned to participants in order of sign-up.

In May 2022, participants received a further questionnaire, assessing intention to leave critical care nursing. This questionnaire was an amendment to the protocol. This was added due to several stakeholder groups and the research literature indicating that measures of intention to leave are paramount to evaluation and implementation of interventions, and especially salient considering the current international healthcare workforce crisis. The questionnaire was distributed via email to all 62 nurses who completed the whole programme (thus, both workshops and both coaching calls). A £5 voucher was offered to all as an incentive to participate.

Analysis plan

Quantitative analyses.

A more detailed report of the analysis can be found in the study protocol paper [ 19 ]. Data were analysed with both R and SPSS. Multilevel (random intercepts for participants) regression models for each outcome included a timepoint coefficient, and were unadjusted, or sequentially adjusted for gender, age, and experience (years in profession). Holm-corrected t- tests further assessed between timepoint differences in outcomes.

Qualitative analyses

As per protocol, reflexive thematic analysis (RTA) [ 43 ] was used to analyse the interviews. RTA does not require a pre-determined ontological or epistemological framework; and is therefore commonly used in applied health research. KSV coded all interviews; and RSE coded a subset of these [ n  = 3, 20%]. Similarities and differences in coding was discussed between the researchers; however, the researchers generally agreed on the use of codes and salient aspects to code.

Participant characteristics

A total of 84 participants consented to participate in the study. Most participants were female [86%], and their mean age was 39.7 [SD = 9.2; range: 22-60; missing n  = 3]. Participants’ years of experience as registered nurses ranged from 0 [i.e., less than 1 year’s experience] to 39, with a mean of 13.9 [SD = 9.0, missing n  = 3]; while their years of experience as registered nurses in critical care ranged from 0 to 35, with a mean of 10.7 [SD = 8.8, missing n  = 3]. Three CCNs indicated that they were off work with stress when they completed the baseline questionnaire; however, when/if those nurses return to work was not followed up. At baseline, two CCNs were also taking part in other workplace wellbeing initiatives, and four others indicated that they had taken part in workplace wellbeing initiatives in the past. Fifteen interviews were conducted by KV (24% of participants), one of which was a pilot interview to trial the interview schedule, so is not included in the analysis.

Intervention delivery

Twenty-five workshop cycles were offered to participants. Nineteen cycles were chosen by participants, 6 cycles were cancelled and participant numbers in each ranged from two to six participants.

A total of 102 UK CCNs signed-up to the study by booking a place; thus, the target of recruiting at least 80 CCNs was met.

Recruitment

A total of 84 CCNs consented to participate in the study. Out of the 84, 77 attended the first workshop [91.7%], thus the objective of recruiting at least 80 CCNs was not met, but this was within 5% of the goal figure.

Programme retention: online, remote delivery of reboot

Of the 102 sign-ups, there were 62 completions, 15 dropped-out during the programme and 25 signed up but did not attend or cancelled their first workshop. Out of the 77 who attended the first workshop, 62 completed both workshops and both coaching calls; thus, 80.5% of those who attended the first workshop completed the programme – this means that the objective of achieving a (participation) retention rate of ≥70% was met.

Retention: feasibility of evaluation of online, remote delivery of reboot

Out of the 77 who completed the first workshop, 58.4% completed final, time-4 measures. Thus, the objective of ≥50% completion rates for the final follow-up questionnaire was met.

The secondary objective was to provide a preliminary assessment of whether Reboot could potentially significantly increase both self-reported psychological resilience and confidence in coping with adverse events, via analysis of questionnaires and interviews.

Quantitative results

Descriptive statistics are presented in Tables  1 , 2 and 3 ; and model-fit and results are presented in Tables  4 and 5 (4 for unadjusted models, 5 for adjusted models). All analyses indicated considerable clustering, supporting the use of random intercepts. The proportion of variance explained by all indicator variables was sizeable across measures, however, a higher proportion of variance was explained by fixed time points explained plus random effects. Adjusting the model for gender, age and experience did not alter model fit, thus are not reported here but results can be viewed in Table  4 .

Confidence scores increased significantly, compared to pre-intervention (Time 1) [Time 2: unadjusted β = 0.80, CI: 0.66 - 0.94, p  < .001, d = .81; Time 3: unadjusted R 2  = 0.75, CI: 0.59 - 0.91, p  < .001, d = 0.78; Time 4: unadjusted R 2  = 0.85, CI: 0.68 - 1.01, p  < .001, d = 0.80]. Post-hoc t-tests comparing timepoint means showed no further increase in confidence when comparing between T2, T3 and T4, indicating that initial increases were maintained and remained stable [range p holm .977; adjusted for multiple comparisons]. Cronbach’s α for the confidence measure ranged from 0.64 - 0.87 across timepoints.

Knowledge scores increased significantly between Time 1 and Time 2 [unadjusted β = 0.48, CI: 0.16-0.30, p  < .001, d = 0.79].

Descriptive statistic present sums of items, whereas models used the mean of items.

Resilience scores increased significantly between Time 1 and Time 3 [unadjusted β = 0.39, CI: 0.23 - 0.56, p  < .001, d = 0.43] as well as between Time 1 and Time 4 [unadjusted β = 0.42, CI: 0.26 - 0.59, p  < .001, d = 0.49]. Post-hoc tests comparing timepoint means indicated that there was no further increase in resilience between Time 3 and Time 4 ( p holm  = .75), suggesting that initial gains remained stable. Cronbach’s alpha for the three time points it was used at, was between .80-.83; thus indicating good reliability.

Burnout scores decreased significantly between Time 1 and Time 3 [unadjusted β = −.037, CI: − 0.50- (− 0.25), p  < .001, d = − 0.51 and between Time 1 and Time 4 [unadjusted β = −.039, CI-0.52 – (− 0.26), p  < .001, d = − 0.56]. Post-hoc tests showed no significant difference was found on the BRS between T2 and T4; p holm  = .82, indicating that decreases were maintained and remained stable. Reliability for the questionnaire was good, with Cronbach’s alpha ranging from .76-.84 for the three time points.

Scores on the PHQ-9 indicated a significant decrease in depression from both Time 1 to Time 3 [unadjusted β = −.045, CI: − 0.59 – - 0.32, p  < .001 as well as from Time 1 to Time 4 [unadjusted β = −.048, CI: − 0.6 – (− 0.35); p  < .001].

Post-hoc tests showed no significant difference on PHQ-9 scores between T3 and T4; p holm  = .73, indicating that reductions were maintained and remained stable. Cronbach’s alpha for the PHQ-9 for the three time points it was used ranged from .83-.88, thus indicating good reliability. Out of the 39 participants who completed the PHQ-9 at both baseline and Time 4, almost 80% of participants screened for the presence of mild or severe depression at baseline (PHQ-9 score of 4 or above), whereas at Time 4, only 31.8% did.

Feedback/reactions

Feedback and reactions to the Reboot workshop were overwhelmingly positive (Tables  2 and 3 ). Most participants agreed or strongly agreed that it was relevant for their professional group; they learned useful skills and felt the workshops were adequate in length and were engaging. The majority also indicated that they would react differently if they were involved in a stressful workplace event after attending the workshop. Only a minority ( n  = 5) indicated that there were aspects of the workshops that they did not find useful. Seventy-two CCNs answered the question as to whether they would recommend the workshops to other HCPs; 71 indicated that yes, they would, whereas one participant said they would not.

Thirty-two out of the 62 nurses who completed the full programme responded to the invitation to complete an additional questionnaire in May 2022 (response rate: 51.6%). Participants were asked to answer a set of questions measuring their turnover intentions as they recalled them prior to participating in Reboot (pre-Reboot), and as they are now (post-Reboot). Higher scores indicate lower intention to leave. Using a paired-samples t-test, a significant difference in intention to leave between pre-Reboot (mean = 11.50, SD =2.64) to post-Reboot (mean = 13.56, SD = 1.63) was found [t (31) = 4.93, p  < .001, d = 0.94], showing that nurses reported significantly lower intention to leave critical care nursing after completing the programme than before. Cronbach’s α for post-Reboot was 0.73 and 0.78. for pre-Reboot.

Qualitative results

From the 15 interviews, two themes were developed. These were: “The value and impact of Reboot for participants and beyond ” and “Online delivery and content”. Both themes had subthemes, illustrated in Fig.  1 .

Graphic representation of qualitative findings

Theme 1: the value and impact of reboot for participants and beyond

The value and impact of Reboot was described as “priceless ” (Interview 14) for participants themselves, for their peers with whom they were able to share the psychological tools and knowledge with, and for organisations.

The value and impact of reboot for participants

Specific benefits that participants identified for themselves as a result of attending Reboot were better understanding of their own thought processes and emotions, better understanding of why errors happen at work, having a “ tool kit ” (Interview 15) of simple, psychological tools that they can draw on in times of stress (both at work and outside of work), being able to better manage mental and physical stress, and being able to better compartmentalise work and life outside of work.

Participants also specifically identified increases in wellbeing, confidence and knowledge about resilience, and decreases in burnout and intention to leave (Table  6 ). In addition, CCNs expressed that the group workshops made them feel validated in their feelings of stress, and that it was helpful to meet other professionals outside of their organisation who had similar experiences:

“… but actually speaking to other people who'd been through a similar experience, who they wish they'd done some things differently as well you know… made you kind of realize we are human, we tried our best and hindsight is a wonderful thing, and experience is a wonderful thing…” (Interview 4)

The value and impact of reboot for peers of participants

Five participants, who were predominantly senior CCNs, also expressed that after Reboot, they were “ also able now to help other people bounce back “(Interview 14) by sharing knowledge and tools learnt during Reboot.

“I was sat with them one of my nurses who, who thought they'd made an error. I'm not sure they did but they were being really hard on themselves, they were ruminating and going over and over and over, to a really unhealthy extent. So, I brought in some of what we've done at the workshop, and I said you know this is what you're doing and it's not healthy and these are ways that you can you know, you need to try and break the cycle.” (Interview 4)

The value and impact of reboot for organisations

Being offered training around psychological tools to cope with stress and how to boost resilience was described as essential by CCNs, as otherwise they would not be able to do their jobs. Thus, CCNs drew links between accessing programs like Reboot and the sustainability of the workforce in critical care.

“I'm sure that it will give me the staying power because things are going - always going to come up at work I think that are challenging” (Interview 13)

“you've got to have a lot of resilience to be able to even want to turn up ” (Interview 2)

All participants said that Reboot should be offered to nurses early on in their nursing career, especially within the first year of working in critical care.

“everybody else had ought to be going” (Interview 7)

“I think really early on in their career, to be able… to know how to approach negative thinking habits… to stop the rumination;… the amount of time I have ruminated on situations and blamed myself for things… they really do play on your mind for weeks sometimes. I think having these tools, so just to have them really early on in your career, so you know how to, how to approach those situations…” (Interview 15)

One participant suggested that while the value of Reboot lay in its focus on the acute, stressful situations that occur in intensive care settings, it does not address more long-term problems, such as issues with turnover and short staffing which are also affecting staff wellbeing.

“I think it's a bit more difficult with everything that's happening due to Covid and staffing at the moment with us, because we've got a lot of turnover of staff because, I guess, people just aren’t happy, but in acute situations, definitely.” (Interview 2)

Theme 2: online delivery and content

This theme incorporates narratives around the delivery and content of the workshops, coaching calls and suggestions for improvement.

Delivery and content of the workshops

Participants spoke positively about the online delivery of Reboot. The workshops were perceived as “ delivered at the right pitch” (Interview 14) and “comfortable ” (Interview 4), with no problems with internet connectivity. Participants liked the presented background to the interventions, the opportunity to share their experiences with other CCNs and the practical content of the workshops. Some participants commented on the fact that the ‘ homework’ set between the first and second workshop was helpful as it made them more conscious of what they were doing and feeling. Participants liked having the workbook to accompany the sessions, and to have as a point of reference for the future.

‘I liked the activities that you did and it was quite personal to you, so you could bring your own experiences, and use them and we went through them, shared with each person how you could use strategies to help you. That was good as well.” -Interview 2

“Really enjoyed and it was very active, not like one speaker is speaking and someone else just listening in - no, really, everything was really practical, in a realistic way… It was really a natural, realistic knowledgeable feeling.” - Interview 11

Delivery and content of the coaching calls

The coaching calls following the workshops were described as deepening understanding, empowering, helpful, professional, and relaxed. Participants spoke very highly of the CBT therapist delivering the intervention, praising their kindness and helpfulness. CCNs felt understood, and appreciated the individualised support, which often included specific materials being sent to them by email following the coaching calls.

“they [the coaching calls] were probably the most helpful” -Interview 2

“Touched in every corner… whenever I got a doubt, I was suddenly sharing with [therapist name] and she was listening and giving some kind of tools and… it was really touching it.” – Interview 11

“I thought, I thought she was brilliant… and really kind, and she listened… I was really thankful.” – Interview 13

One of the participants described the coaching calls as “ invaluable ” (Interview 14) and liked the fact that the coaching calls gave her opportunity to discuss what she was struggling with and tools to solve the problems for herself, with support of the therapist, rather than being given a solution to a problem.

“Quite invaluable and as a supported tool… because it wasn't like, …“this this is the problem… okay well, here's the answer”, it wasn’t that.. it was a “right, well that is the problem, let’s look at some tools you can use to help and support you to find a way through that yourself”, which was really empowering…. It's not sort of… “this the problem I had…this is how you fix it… this is what you've got to do. It wasn't that - it's “here's some tools, work through those tools, see what you think when we come back on the next coaching call” … very empowering, so I had to sit there and do that myself, which was great.” – Interview 14

Suggestions and recommendations

Participants made a small number of suggestions to improve Reboot, which included a slightly longer workbook with more content, to ensure that content is not forgotten about, more coaching calls and delivering some ‘refreshers’ on content later on. One participant also suggested including an example about long-term stressors, such as sustained short staffing.

Unlike with the workshops, participants did not receive reminders about their coaching calls from the research team or therapist, which meant that some participants forgot they had booked their coaching calls. Some participants suggested reminders about upcoming coaching calls would be beneficial, to ensure they remember and attended.

The current study sought to assess the feasibility of delivering Reboot via online, remote delivery to CCNs, and to provide a preliminary assessment of whether Reboot could potentially increase resilience and confidence in coping with adverse events and decrease burnout, depression, and intention to leave. The results suggested that it is feasible to deliver Reboot via online, remote delivery to CCNs, and found significant increases in resilience and confidence in coping with adverse events and decreases in burnout and depression. Retrospective recall also indicated that nurses believed they had reduced intention to leave after participating in the programme. The qualitative findings echoed the quantitative findings, with CCNs particularly valuing the practical exercises that could be translated into everyday practice.

These findings support those of previous studies indicating that Reboot may be a valuable intervention for HCPs [ 37 , 38 , 44 ], but also extend this in four main ways.

First, the current results were the first to indicate that Reboot may have value in a post-pandemic context. Pre-pandemic, there were already around a third of doctors and nurses suffering from burnout and significant increases reported for work-related stress among healthcare staff [ 37 , 45 ]. However, rates have increased internationally following the onset of the pandemic [ 46 , 47 ]. In the UK, the General Medical Council (GMC) has been running its annual workforce burnout survey since 2018, making it the largest and most comprehensive annual workforce survey in the UK. In 2022, the burnout risk for doctors was at its highest since 2018. In 2021, 46,793 UK medical trainees completed the survey; 43% said that they found their work emotionally exhausting to a high or very high degree, and 33% indicated that they were feeling burnt out from work to either a high or very high degree [ 48 ]. A year later, in 2022, the numbers worsened as 39% (a 6% increase) of trainees indicated that they were feeling burnt out to either a high or very high degree, and 51% of trainees (8% increase) indicated that they found their work emotionally exhausting to a high or very high degree [ 49 ]. While, unfortunately, there is no equivalent study of this scale and magnitude for nurses, this survey, alongside reports from the Nursing and Midwifery Council, show just how extreme the situation has become in healthcare in the UK [ 50 ], and that HCPs desperately need support. While it was possible that these increases in burnout across healthcare professions may have rendered Reboot unworkable or irrelevant, this study shows that Reboot is still feasible and potentially effective, even in the context of psychological changes within the healthcare workforce.

Second, the current results extend the existing literature by showing that Reboot is feasible and potentially effective for CCNs in particular. To date, there are no systematic reviews or meta-analyses that assess the efficacy of intervention to increase resilience or decrease burnout in CCNs. There are, however, reviews that either assess the efficacy of interventions on reducing burnout, or increasing resilience, in physicians and nurses concomitantly [ 35 , 51 , 52 ], or the efficacy of resilience or interventions more generally [ 36 , 53 ]. Overall, these reviews conclude that online programmes and internet-based interventions, as well as psychosocial training interventions, are among the interventions that have a positive effect on burnout and resilience, and that CBT-based resilience interventions and mixed-methods most effective at increasing resilience [ 36 , 51 , 53 ]. However, one major criticism of existing interventions is that they are generic and lack relevance for the work stresses different types of HCPs, or even nurses, are facing. Reboot overcomes this by the fact that it can be tailored to each disciplinary group, including critical care nurses or other specialist areas of nursing, ensuring relevance and saliency of the material for specific discipline groups, rather than for HCPs more generally. For example, CCNs will require different content to be included in a resilience and burnout intervention that is salient and acceptable to them, compared to trainee doctors, surgeons or midwives [ 37 ] but also compared to other nurses. CCNs tend to have different psychological profiles, compared to non-CCNs. For example, nurses working on either orthopaedic or dialysis wards have been found to have much lower burnout scores, compared to nurses working on critical care units [ 54 ] – a difference that has likely been further exacerbated by the pandemic, and effective resilience interventions must take this into account. This will also be a challenge for implementation into practice, as delivery of Reboot would need to be planned and tailored in advance for each HCP group.

Third, the current results also add to the wealth of evidence for the efficacy of person-directed interventions [ 36 , 51 ]. Person-directed interventions can be defined as those which aim to improve an individual’s capacity to cope with the demands of their job, which is often achieved via mindfulness or CBT programmes. While the quantitative findings highlight that Reboot is feasible and potentially effective for CCNs, the qualitative findings add important knowledge to the aspects of person-directed interventions which CCNs found valuable. For example, participants especially valued the practical applications of the programme which helped them, and by proxy, their peers, cope with the demands of their critical care nursing. In this context, it is not possible to suggest that Reboot can be considered superior to other existing interventions for nurses but as one of several candidate interventions which should be tested using more rigorous research designs. However, it should be noted that Reboot has some unique features not shared with other existing interventions: for example, it involves a mixed-modality format, ensuring the benefit of both peer support and one-to-one confidential space with a therapist.

It is clear that interventions, such as Reboot, cannot compensate for organisational failings; and should be used alongside, rather than in place of, organisational interventions [ 19 , 37 , 55 ]. However, organisational changes are often decided at a regional or national level and influenced by political and economic factors. As such they can be challenging to implement. In this context, person-directed interventions are often appealing to organisations as they are within their decision-making latitude/capability to select and deliver. At the same time though, staff often do not currently have the time to attend, and engage with, wellbeing programmes on offer, leading to lack of uptake and furthering intention to leave among NHS employees [ 56 ]. Thus, organisational changes that allow the attendance of, and engagement with, wellbeing programs are desperately needed, alongside changes that have been associated with reduced burnout in nursing, such as higher pay, more work flexibility, higher autonomy and fewer/better working hours [ 57 , 58 , 59 , 60 ].

Fourth, the present study also contributes to a growing literature which is focused on the prevention rather than amelioration of work-related mental distress. Research is starting to highlight the importance of higher levels of resilience as protective factors against burnout and the development of post-traumatic-stress disorder (PTSD) for CCNs, and beyond [ 30 , 36 , 61 , 62 , 63 ]. For example, a 2021 study conducted in Poland [ 63 ] found that higher levels of resilience were associated with lower levels of burnout and secondary traumatic stress, while exposure to secondary traumatic stress was positively related to burnout. This supports the development, and implementation, of prophylactic resilience interventions for healthcare staff, rather than ameliorative burnout or PTSD interventions.

Limitations

While strengths of the current study include its mixed-methods design, which can elucidate not just the potential impact of the intervention but also the mechanisms underlying this, there are a number of limitations.

Firstly, the current uncontrolled study design means that causal associations between Reboot and the outcomes measured cannot be assumed. Higher quality evidence, perhaps in the form of a wait-list control design or randomised controlled trial, is now needed.

Secondly, intention to leave scores were collected retrospectively for pre- and post-Reboot, thus future work should include intention to leave measures from baseline.

Thirdly, the majority of workshop and coaching sessions were delivered by the same therapist, a future trial should ensure the inclusion of multiple therapists. The therapist also encouraged the completion of outcome measures (especially at Time 2, post completion of second workshop), which means the therapist was not entirely independent of the evaluation.

Fourth, non-completers were not further followed up or invited to interview. Future research should consider their perspectives too.

Fifth, due to the study focus on CCNs, generalization to nurses working outside of critical care is not possible.

The current results suggest that it is feasible to deliver Reboot via online delivery to CCNs, and that it is associated with self-reported increases in resilience and confidence in coping with adverse events and decreases in burnout and depression. Participants also reported that their intention to leave reduced following the programme. The qualitative findings echoed the quantitative findings, with CCNs particularly valuing the practical exercises that could be translated into everyday practice. These findings, alongside those of the previously investigated in-person (rather than remote) version support the evidence-base and efficacy of Reboot. However, a randomised controlled trial design is now needed to more fully and robustly ascertain the efficacy of Reboot.

Availability of data and materials

Anonymised behavioural data and statistical analysis may be requested via email from Dr. KS Vogt, after data collection and publication of results.

Abbreviations

Brief Resilience Scale

Cognitive Behavioural Therapy

Critical Care Nurse(s)

Healthcare professional(s)

UK National Health Service

Patient Health Questionnaire

Post-traumatic stress disorder

Recovery boosting coaching programme

Senior Research Fellow

United Kingdom

World Health Organization. Patient safety: fact sheet. World Health Organization; 2019. Internet, Cited 2023 May 10. Available from: https://www.who.int/news-room/fact-sheets/detail/patient-safety

Google Scholar  

Elliott RA, Camacho E, Jankovic D, Sculpher MJ, Faria R. Economic analysis of the prevalence and clinical and economic burden of medication error in England. BMJ Qual Saf. 2021;30(2):96–105.

Article   PubMed   Google Scholar  

Nibbelink C, Brewer B. Decision-making in nursing practice: an integrative literature review Christine. J Clin Nurs. 2018;27(5–6):917–28.

Article   PubMed   PubMed Central   Google Scholar  

Ma RH, Zhao XP, Ni ZH, Xue XL. Paediatric oncology ward nurses’ experiences of patients’ deaths in China: a qualitative study. BMC Nurs. 2021;20(1):1–8.

Article   Google Scholar  

Page P, Simpson A, Reynolds L. Bearing witness and being bounded: the experiences of nurses in adult critical care in relation to the survivorship needs of patients and families. J Clin Nurs. 2019;28(17–18):3210–21.

Wu A, Shapiro J, Harrison R, Scott S, Conners C, Kenney L, et al. The impact of adverse events on clinicians: What’s in a name? J Patient Saf. 2020;16(1):65–72.

Article   CAS   PubMed   Google Scholar  

Bohlken J, Schömig F, Lemke MR, Pumberger M, Riedel-Heller SG. COVID-19-Pandemie: Belastungen des medizinischen Personals. Psychiatr Prax. 2020;47(04):190–7.

Britt TW, Shuffler ML, Pegram RL, Xoxakos P, Rosopa PJ, Hirsh E, et al. Job demands and resources among healthcare professionals during virus pandemics: a review and examination of fluctuations in mental health strain during COVID-19. Appl Psychol. 2021;70(1):120–49.

Mehta S, Machado F, Kwizera A, Papazian L, Moss M, Azoulay É, et al. COVID-19: a heavy toll on health-care workers. Lancet Respir Med. 2021;9(3):226–8.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Browne D, Roy S, Phillips M, Shamon S, Stephenson M. Supporting patient and clinician mental health during COVID-19. Can Fam Physician. 2020;66(7):E190–2.

PubMed   PubMed Central   Google Scholar  

Montgomery CM, Humphreys S, McCulloch C, Docherty AB, Sturdy S, Pattison N. Critical care work during COVID-19: a qualitative study of staff experiences in the UK. BMJ Open. 2021;11(5).  https://bmjopen.bmj.com/content/11/5/e048124.long .

Seah KM. Redeployment in COVID-19: old dogs and new tricks. Emerg Med J. 2020;37(7):456.

Sindhu KK. Schrödinger’s resident: redeployment in the age of COVID-19. Acad Med. 2020;95(9):1353.

Mosheva M, Gross R, Hertz-Palmor N, Hasson-Ohayon I, Kaplan R, Cleper R, et al. The association between witnessing patient death and mental health outcomes in frontline COVID-19 healthcare workers. Depress Anxiety. 2021;38(4):468–79.

Shah SHA, Haider A, Jindong J, Mumtaz A, Rafiq N. The impact of job stress and state anger on turnover intention among nurses during COVID-19: the mediating role of emotional exhaustion. Front Psychol. 2022;12:810378.

Harris ML, McLeod A, Titler MG. Health experiences of nurses during the COVID-19 pandemic: a mixed methods study. West J Nurs Res. 2023;45(5):443–54.

Alharbi J, Jackson D, Usher K. The potential for COVID-19 to contribute to compassion fatigue in critical care nurses. J Clin Nurs. 2020;29(15–16):2762–4.

Dennis JM, McGovern AP, Vollmer SJ, Mateen BA. Improving Survival of Critical Care Patients with Coronavirus Disease 2019 in England: A National Cohort Study, March to June 2020. Crit Care Med. 2021;49:209–14.

Vogt KS, Grange A, Johnson J, Marran J, Budworth L, Coleman R, et al. Study protocol for the online adaptation and evaluation of the ‘reboot’ (recovery-boosting) coaching programme, to prepare critical care nurses for, and aid recovery after, stressful clinical events. Pilot Feasibility Stud. 2022;8(1):1–10.

Vogt KS, Simms-Ellis R, Grange A, Griffiths ME, Coleman R, Harrison R, et al. Critical care nursing workforce in crisis: a discussion paper examining contributing factors, the impact of the COVID-19 pandemic and potential solutions. J Clin Nurs. 2023;32:7125–34.

Moll V, Meissen H, Pappas S, Xu K, Rimawi R, Buchman TG, et al. The coronavirus disease 2019 pandemic impacts burnout syndrome differently among multiprofessional critical care clinicians - a longitudinal survey study. Crit Care Med. 2022;50(3):440–8.

Greenberg N, Weston D, Hall C, Caulfield T, Williamson V, Fong K. Mental health of staff working in intensive care during Covid-19. Occup Med. 2021;71(2):62–7.

Article   CAS   Google Scholar  

Said RM, El-Shafei DA. Occupational stress, job satisfaction, and intent to leave: nurses working on front lines during COVID-19 pandemic in Zagazig City, Egypt. Environ Sci Pollut Res. 2021;28(7):8791–801.

Pan L, Xu Q, Kuang X, Zhang X, Fang F, Gui L, et al. Prevalence and factors associated with post-traumatic stress disorder in healthcare workers exposed to COVID-19 in Wuhan, China: a cross-sectional survey. BMC Psychiatry. 2021;21(1):1–9.

Crowe S, Howard AF, Vanderspank-wright B, Gillis P, Mcleod F. The effect of COVID-19 pandemic on the mental health of Canadian critical care nurses providing patient care during the early phase pandemic: A mixed method study Sarah. Intensive Crit Care Nurs. 2021;63:102999.

Heesakkers H, Zegers M, van Mol MM, van den Boogaard M. The impact of the first COVID-19 surge on the mental well-being of ICU nurses: a nationwide survey study. Intensive Crit Care Nurs. 2021;65:103034.

Cortese CG. Predictors of critical care nurses’ intention to leave the unit, the hospital, and the nursing profession. Open J Nurs. 2012;02(03):311–26.

Labrague LJ, de Los Santos JA. Fear of COVID-19, psychological distress, work satisfaction and turnover intention among frontline nurses. J Nurs Manag. 2021;29(3):395–403.

Tolksdorf K, Tischler U, Heinrichs K. Correlates of turnover intention among nursing staff in the COVID-19 pandemic: a systematic review. BMC Nurs. 2022;21(1):174.

Montgomery A, Panagopoulou E, Esmail A, Richards T, Maslach C. Burnout in healthcare: the case for organisational change. BMJ. 2019;366:12777.

Mealer BM, Jones J, Meek P. Factors affecting Resilience and development of posttraumatic stress disorder in critical care nurses. Am J Crit Care Nurs. 2017;26(3):184–92.

Rushton CH, Batcheller J, Schroeder K, Donohue P. Burnout and Resilience among nurses practicing in high- intensity settings. Am J Crit Care Nurs. 2015;24(5):412–20.

Johnson J, Panagioti M, Bass J, Ramsey L, Harrison R. Resilience to emotional distress in response to failure, error or mistakes: a systematic review. Clin Psychol Rev. 2017;52:19–42.

Resilience JJ. The bi-dimensional framework. In: Wood A, Johnson J, editors. The Wiley handbook of positive clinical psychology. Wiley-Blackwell; 2016.

Kunzler AM, Helmreich I, Chmitorz A, König J, Binder H, Wessa M, et al. Psychological interventions to foster resilience in healthcare professionals. Cochrane Database Syst Rev. 2020;7(7).  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8121081/ .

Joyce S, Shand F, Tighe J, Laurent SJ, Bryant RA, Harvey SB. Road to resilience: a systematic review and meta-analysis of resilience training programmes and interventions. BMJ Open. 2018;8(6):1–9.

Johnson J, Simms-Ellis R, Janes G, Mills T, Budworth L, Atkinson L, et al. Can we prepare healthcare professionals and students for involvement in stressful healthcare events? A mixed-methods evaluation of a resilience training intervention. BMC Health Serv Res. 2020;20(1):1–14.

Al-Ghunaim T, Johnson J, Biyani CS, Coleman R, Simms-Ellis R, O’Connor DB. Evaluation of the reboot coaching workshops among urology trainees: a mixed method approach. BJUI Compass. 2023;4(5):533–42.

Kirkpatrick DL. Evaluating training programs: the four levels. Emeryville: Berrett-Koehler Publishers; 1994.

Smith BW, Dalen J, Wiggins K, Tooley E, Christopher P, Bernard J. The brief resilience scale: assessing the ability to bounce back. Int J Behav Med. 2008;15(3):194–200.

Demerouti E, Bakker AB. The Oldenburg burnout inventory: a good alternative to measure burnout (and engagement). 2007. Internet, Available from: https://www.isonderhouden.nl/doc/pdf/arnoldbakker/articles/articles_arnold_bakker_173.pdf

Kroenke K, Spitzer RL, Williams JBW. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606–13.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

Johnson J, Pointon L, Talbot R, Coleman R, Budworth L, Simms-Ellis R, Johnson, J., Pointon, L., Talbot, R., Coleman, R., Budworth, L., Simms-Ellis, R., Vogt, K., Tsimpida, D., Biyani, C. S., Harrison, R., Cheung, G., Melville, C., Jayagopal, V. & Lea, W. (2023). Reboot coaching programme: a mixed-methods evaluation assessing resilience, confidence, burnout and depression in medical students. Scott Med J, in press.

Hall LH, Johnson J, Watt I, Tsipa A, O’Connor DB. Healthcare staff wellbeing, burnout, and patient safety: a systematic review. PLoS One. 2016;11(7):1–12.

Wise J. Covid-19: experts divide into two camps of action-shielding versus blanket policies. BMJ. 2020;370:m3702.

Wei H, Aucoin J, Kuntapay GR, Justice A, Jones A, Zhang C, et al. The prevalence of nurse burnout and its association with telomere length pre and during the COVID-19 pandemic. PLoS One. 2022;17(3 March):1–14.

General Medical Council. National training survey 2021 - results. 2021. Internet, Available from: https://www.gmc-uk.org/-/media/documents/national-training-survey-results-2021%2D%2D-summary-report_pdf-87050829.pdf

General Medical Council. National training survey 2022 results. 2022. Internet, Available from: https://www.gmc-uk.org/-/media/documents/national-training-survey-summary-report-2022-final_pdf-91826501.pdf

Nursing & Midwifery Council. Nursing and midwifery register grows but so does number of people leaving. 2022, Cited 2023 Nov 15]. Internet, Available from: https://www.nmc.org.uk/news/news-and-updates/nursing-and-midwifery-register-grows-but-so-does-number-of-people-leaving/

Aryankhesal A, Mohammadibakhsh R, Hamidi Y, Alidoost S, Behzadifar M, Sohrabi R, et al. Interventions on reducing burnout in physician and nurses: a systematic review. Med J Islam Repub Iran. 2019;33:77.

Zhang YY, Han WL, Qin W, Yin HX, Zhang CF, Kong C, et al. Extent of compassion satisfaction, compassion fatigue and burnout in nursing: a meta-analysis. J Nurs Manag. 2018;26(7):810–9.

Awa WL, Plaumann M, Walter U. Burnout prevention: a review of intervention programs. Patient Educ Couns. 2010;78(2):184–90.

Ahmadi Q, Azizkhani R, Basravi M. Correlation between workplace and occupational burnout syndrome in nurses. Adv Biomed Res. 2014;3(44).  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3949345/ .

Public Health England. Interventions to prevent burnout in high risk individuals: evidence review. 2016. Available from: https://assets.publishing.service.gov.uk/media/5a81a3aee5274a2e8ab55122/25022016_Burnout_Rapid_Review_2015709.pdf .

Body NPR. NHS pay review Body thirty-fifth report 2022. Office of Manpower Economics; 2022. Internet, Available from: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1092270/NHSPRB_2022_Accessible.pdf

Alzailai N, Barriball L, Xyrichis A. Burnout and job satisfaction among critical care nurses in Saudi Arabia and their contributing factors: a scoping review. Nurs Open. 2021;8(5):2331–44.

Cañadas-De la Fuente GA, Vargas C, San Luis C, García I, Cañadas GR, Emilia I. Risk factors and prevalence of burnout syndrome in the nursing profession. Int J Nurs Stud. 2015;52(1):240–9.

Moustaka Ε. Sources and effects of work-related stress in nursing. Health Sci J. 2011;4:210–6.

Shah MK, Gandrakota N, Cimiotti JP, Ghose N, Moore M, Ali MK. Prevalence of and factors associated with nurse burnout in the US. JAMA Netw Open. 2021;4(2):1–11.

Cacciatori I, Grossi C, D’Auria C, Bruneri A, Casella C. Resilience skills as a protective factor against burnout for health professionals: a cross-sectional study on new hires from the hospital of Lodi. G Ital Med Lav Ergon. 2021;43(2):131–6.

PubMed   Google Scholar  

Vagni M, Maiorano T, Giostra V, Pajardi D. Protective factors against emergency stress and burnout in healthcare and emergency workers during second wave of COVID-19. Soc Sci. 2021;10(5):178.

Ogi’nska-Bulik N, Michalska P. Psychological Resilience and secondary traumatic stress in nurses working with terminally ill patients—the mediating role of job burnout. Psychol Serv. 2021;18(3):398–405.

Download references

Acknowledgements

The authors would like to thank the following individuals for their contributions to this study: Sobia Bibi and Lucy Chapman, our Industrial placement students Ryan Carter and Rameen Haq, our Steering Group and the CC3N Network, for their support of the study.

This work is funded by the Burdett Trust for Nursing [Grant code SB/ZA/101010662/632762, Funding Stream: Covid-19: Supporting Resilience in the Nursing Workforce] and supported by the NIHR Yorkshire and Humber Patient Safety Translational Research Centre [under grant PSTRC-2016-006]. The sponsor is Bradford Teaching Hospitals [contact details for sponsor: [email protected] ]. Neither study sponsor nor funder have no role in the study design, the collection of data, the management of data, the analysis thereof, or its interpretation.

This report is independent research supported by National Institute for Health and Care Research Yorkshire and Humber ARC [under grant NIHR200166]. The views expressed in this publication are those of the authors and not necessarily those of the National Institute for Health Research or the Department of Health and Social Care.

Author information

Authors and affiliations.

Bradford Institute for Health Research, Bradford Royal Infirmary, Temple Bank House, Duckworth Lane, Bradford, BD9 6RJ, UK

K. S. Vogt, J. Johnson, R. Coleman, R. Simms-Ellis, J. Marran, L. Budworth, R. Lawton & A. Grange

Department of Psychology, University of Leeds, Leeds, LS2 9JT, UK

K. S. Vogt, J. Johnson, R. Simms-Ellis & R. Lawton

School of Population Health, University of New South Wales, Sydney, 2052, Australia

J. Johnson & R. Harrison

Centre for Health Systems and Safety Research: Australian Institute of Health Innovation, Macquarie University, Sydney, Australia

R. Harrison

Leeds Teaching Hospitals NHS Trust, Great George Street, Leeds, LS1 3EX, UK

N. Shearman

Leeds Institute of Rheumatic and Musculoskeletal Medicine, University of Leeds, Leeds, UK

L. Budworth

School of Health and Wellbeing: College of Medical, Veterinary and Life Sciences, University of Glasgow, Clarice Pears Building, Glasgow, G12 8TB, UK

Department of Primary Care & Mental Health, Institute of Population Health, University of Liverpool, Eleanor Rathbone Building, Liverpool, L69 7ZA, UK

West Yorkshire Adult Critical Care Network, Leeds Teaching Hospitals, Leeds, UK

C. Horsfield

NIHR Yorkshire & Humber Patient Safety Research Collaboration, Bradford Teaching Hospitals Foundation Trust, Bradford, UK

Mid Yorkshire Teaching NHS Trust, Wakefield, UK

You can also search for this author in PubMed   Google Scholar

Contributions

This work is based on previous work conducted by JJ and RSE. For this project, RSE, JJ, JM, RL, AG, RL, HR and LB conceived the study and initiated the adaptation. NS and CH contributed to the design of the study. Adaptation to online, remote delivery was led by JJ and RSE. The Principal Investigator [and grant holder] is AG. JJ, RSE, AG and KSV worked on the ethical approvals. JJ and RC delivered the workshops, RC delivered the coaching calls. JJ provided regular supervision to RC. KSV led on recruitment, supported by CH, and conducted all data collection [quantitative via online questionnaire and qualitative interviews]. LB and KSV conducted statistical analyses; KSV the qualitative analysis. JJ, AG, RSE and JM assisted with data analysis, when needed. KSV wrote the first version of this paper, all authors have had opportunity to read and contribute to the manuscript prior to submission.

Corresponding author

Correspondence to K. S. Vogt .

Ethics declarations

Ethics approval and consent to participate.

Ethics approval was granted by the University of Leeds Ethics committee for this study. Participants gave full informed consent before participation. The study adheres to both the British Psychological Society’s Code of Ethics and Conduct, as well as Declaration of Helsinki.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Vogt, K.S., Johnson, J., Coleman, R. et al. Can the Reboot coaching programme support critical care nurses in coping with stressful clinical events? A mixed-methods evaluation assessing resilience, burnout, depression and turnover intentions. BMC Health Serv Res 24 , 343 (2024). https://doi.org/10.1186/s12913-023-10468-w

Download citation

Received : 05 April 2023

Accepted : 12 December 2023

Published : 15 March 2024

DOI : https://doi.org/10.1186/s12913-023-10468-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Critical care
• Healthcare staff

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

ORIGINAL RESEARCH article

This article is part of the research topic.

Research on Teaching Strategies and Skills in Different Educational Stages

Physical education (PE) students' reflections about the learning outcomes of different teaching methods: a mixed methods study Provisionally Accepted

• 1 Nord University, Norway

The final, formatted version of the article will be published soon.

Teaching in higher education is still mainly executed as lectures, even though research about student-active instruction methods points to more motivated students, higher enjoyment, and more optimal learning outcomes. The purpose of this study was to obtain better insight into how physical education (PE) students assessed their learning outcomes in relation to the use of different pedagogical approaches. A master's course in PE was planned and implemented using the following eight different learning approaches: lectures; practical exercises about themes in lectures; discussions during lectures; discussions outside of lectures; planning and exercises for peer students; individual work preparing to write an academic text; individual work writing the academic text; and reading for an exam. The study constituted a mixed methods study, which used quantitative data from students' evaluation of eight different learning approaches on a Likert-type scale, and in-depth qualitative data from follow-up interviews with some of the same students, with the aim of explaining the main findings.Quantitative data about the students' reflections on the learning outcomes of the different learning approaches were collected among 59 different students at three different times (2021, 2022, and 2023), after finishing a course in the fifth semester in a master's program in PE.The findings showed that the students reported achieving the highest learning outcomes from practical exercises and attaining the lowest learning outcomes from lectures. In depth interviews among seven randomly selected students were also used to obtain reflections from the students about the different learning approaches. Quantitative analyses again revealed that practical exercises produced the highest learning outcomes, while lectures resulted in the lowest learning outcomes. Qualitative analyses of the in-depth interviews indicated that practical activities enabled students to relate theory to practice, make them active, and are associated with future work, while the quality of lectures depended on characteristics of the teacher and were often experienced as long and unstimulating. According to the results, we recommend that student teachers in higher education acquire the ability to plan and execute practical lessons in relation to themes focused upon in lectures and involve students more in discussions during lectures.

Keywords: student teachers, learning outcomes, Lectures, practical activities, Pedagogical methods

Received: 05 Jan 2024; Accepted: 19 Mar 2024.

Copyright: © 2024 Sørensen and Lagestad. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: PhD. Arne Sørensen, Nord University, Bodø, Norway

People also looked at

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Does the Wim Hof Method have a beneficial impact on physiological and psychological outcomes in healthy and non-healthy participants? A systematic review

Roles Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation Warwick Medical School, University of Warwick, Coventry, United Kingdom

Roles Writing – review & editing

• Omar Almahayni, 
• Lucy Hammond

• Published: March 13, 2024
• https://doi.org/10.1371/journal.pone.0286933
• Reader Comments

Introduction

Wim Hof, also known as the iceman, developed a method called Wim Hof Method (WHM) which he claims to have several benefits on physical and mental health. The aim of this systematic review is to identify and synthesise the results of the studies conducted on WHM on physiological and psychological health-related outcomes.

Materials and methods

This systematic review followed the PRISMA guidelines for systematic reviews. The protocol was registered in the International Prospective Register of Systematic Reviews (PROSPERO), registration number CRD42022333209. Medline and Web of Science were searched and included studies from January 1, 2014, to July 4, 2022. Studies were included if they met the predetermined inclusion/exclusion criteria. Inclusion criteria included RCTs and cohort studies published in peer-reviewed journals, studies conducted on healthy individuals and people with pre-existing medical conditions (adolescents and adults over the age of 14), studies that included all three pillars (breathing, cold exposure, and commitment) of the WHM, and studies that only focused on Wim Hof breathing method (WHBM). Exclusion criteria included studies that discussed WHM but are not original experimental research or are not peer-reviewed, studies that included children under the age of 14, and studies that used methods similar to WHM, but not actually WHM, such as tummo meditation. The articles were assessed for risk of bias using RoB 2.0 and Scottish Intercollegiate Guidelines Network (SIGN) tools. The effects of WHM were categorised into physiological or psychological related outcomes and narrative synthesis was conducted.

Nine papers were included in this review which consisted of eight individual trials. The findings suggest that the WHM may reduce inflammation in healthy and non-healthy participants as it increases epinephrine levels, causing an increase in interleukin-10 and a decrease in pro-inflammatory cytokines. Additionally, effect of WHBM on exercise performance showed mixed findings. Effects on respiratory parameters of minute ventilation, tidal volume, and breathing frequency were mixed following bouts of exercise.

Taken together, the findings of this review show promising use of WHM in the inflammatory response category. The focus of future studies should further investigate the benefits of WHM in non-healthy participants with inflammatory disorders and explore the use of Wim Hof breathing method to enhance exercise performance.

Citation: Almahayni O, Hammond L (2024) Does the Wim Hof Method have a beneficial impact on physiological and psychological outcomes in healthy and non-healthy participants? A systematic review. PLoS ONE 19(3): e0286933. https://doi.org/10.1371/journal.pone.0286933

Editor: Hidetaka Hamasaki, Hamasaki Clinic, JAPAN

Received: May 25, 2023; Accepted: December 26, 2023; Published: March 13, 2024

Copyright: © 2024 Almahayni, Hammond. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the paper and its Supporting Information files.

Funding: The authors received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

The Wim Hof Method (WHM) is a lifestyle change that claims to enhance a person’s quality of life which consists of three pillars: Wim Hof breathing method (WHBM), cold therapy, and commitment. According to Wim Hof, the creator of this method, a combination of these three pillars enhances a person’s autonomic nervous and immune systems, thus strengthening a person’s health and mentality.

Wim Hof is recognized for his ability to resist extreme cold temperatures. He holds 21 Guinness World Records for some of his remarkable human achievements, including climbing Mount Kilimanjaro while wearing shorts, swimming 66 metres beneath ice, standing for two hours in a container filled with ice cubes, and running a half marathon over the Arctic Circle, only wearing shorts and barefoot [ 1 ]. Wim Hof credits his achievements to the WHM.

The first pillar of the WHM is WHBM which consists of hyperventilation 30–40 times then holding the breath voluntarily at low lung volume [ 2 ]. Secondly, cold therapy involves taking daily cold showers or sitting in an ice bath [ 3 ]. The final pillar, which is commitment, is the foundation of the other two pillars; mastering both mindful breathing and cold exposure takes patience and persistence [ 4 ]. To support his claims, Wim Hof has submitted his method and himself for research. Many studies, including case studies, randomised control trials (RCTs), and observational trials have been conducted on him and his method [ 5 – 16 ].

To identify some of the benefits people are experiencing, Wim Hof collaborated with RMIT University by conducting a survey inquiring about the impact WHM had on an individual’s health and well-being [ 17 ]. Over 3,200 people answered the survey, and the majority of the answers were positive. Respondents reported a good mix of physical and mental benefits such as an increase in their tolerance to resist cold, as well as an increase in energy, mood, mental focus, and general health. The survey findings also claimed that the WHM had benefits for specific conditions such as stress, tiredness and fatigue, anxiety, depression, back pain, insomnia, arthritis, and chronic pain [ 17 ].

Collectively, these findings suggest that the WHM may be of importance to healthy and non-healthy individuals and that the WHM approach may be given consideration in the public health and lifestyle medicine fields. The notion of lifestyle medicine refers to the study of how actions and habits affect illness prevention and treatment [ 18 ]. However, caution should be applied insofar as the aforementioned survey [ 17 ] has not been peer-reviewed or published, and Wim Hof’s own involvement in this study and others [ 5 , 9 ] may give rise to conflicts of interest in the pursuit of both an evidence base and a commercial opportunity. Wim Hof has attained a level of celebrity as a result of his achievements, method and associated media career. Due to all his remarkable achievements, his charisma, and authenticity, Wim Hof might be considered to be both achieved and micro celebrity [ 19 ], where achieved celebrity refers to the attainment of fame based on recognized talent, achievement, or ability and micro celebrity describes an individual that achieves status and social media presence by self-broadcasting about niche subjects to a small community of followers. Wim Hof is an example of a celebrity that embodies the para-social relationship, exchanging the allure of intimacy, integrity, and authenticity. The growing number of celebrities giving lifestyle and health advice and scientific knowledge to the public indicates the need to critically examine such advice [ 20 ]. Nunan et al . urge caution for the potential of “the unintended consequences of uncritical endorsement and application of lifestyle medicine including the infiltration of pseudoscience, profiteering, and the potential for widening health inequalities by a continued focus on the ‘individual’” (p229) [ 21 ]. Therefore, an independent, systematic synthesis of this evidence is warranted to evaluate the positive health claims of the WHM.

A systematic review has not yet been conducted on the WHM body of evidence. Therefore, an independent, systematic synthesis of this evidence is warranted to evaluate the positive health claims of the WHM. Hence, this review aims to identify and synthesise the results of the studies conducted on WHM on physiological and psychological health-related outcomes.

This systematic review followed the PRISMA guidelines for systematic reviews [ 22 ]. The protocol was registered in the International Prospective Register of Systematic Reviews (PROSPERO), registration number CRD42022333209.

Literature search

MEDLINE and Web of Science databases were used to search for published studies for potential inclusion in this systematic review. The following search phrases were used in combination: Wim Hof; breathing exercise OR breathing technique; meditation OR commitment; cold temperature OR cold exposure OR ice bath OR cold shower. The filters English language, year = "2014—July 4, 2022", and journal articles or observational studies or randomised controlled trials (RCTs) were applied. A filter for the year was applied because the first study performed on WHM was in 2014 [ 10 ].

Study selection

Duplicates were deleted before abstracts were screened. The screening of titles, abstracts and full texts was conducted by two independent reviewers, with any disagreements resolved by consensus. Screening was performed in Covidence (title and abstract screening, full text review, and extraction of the chosen articles). Studies were included according to the predetermined inclusion/exclusion criteria. Inclusion criteria included RCTs and cohort studies published in peer-reviewed journals, studies conducted on healthy individuals and people with pre-existing medical conditions (adolescents and adults over the age of 14), studies that included all three pillars (breathing, cold exposure, and commitment) of the WHM as defined above, and studies that only focused on WHBM. Articles just covering WHBM were permitted as it has been deemed more important and studied more frequently than other pillars. Furthermore, exclusion criteria included studies that discussed WHM but are not original experimental research or are not peer-reviewed, studies that included children under the age of 14, and studies that used methods similar to WHM, but not actually WHM, such as Tummo meditation. Both WHM and Tummo methods allow for the controlment of body temperature, and both have similar breathing techniques, however Tummo meditation consists of breathing and visualisation techniques which are intended to summon spiritual insight, whereas WHM is not religious [ 23 ]. After deciding on the included studies, the reference lists were hand-searched for other relevant articles.

Data extraction comprised the name of the study, place where the study was conducted, research design, participant demographics, study context, description of intervention and control, and key findings. One reviewer completed the extraction while a second reviewer checked it for accuracy. Furthermore, study quality was assessed using the RoB 2.0 risk bias tool [ 24 ] for RCTs and the Scottish Intercollegiate Guidelines Network (SIGN) risk bias tool [ 25 ] for cohort studies. Two reviewers independently evaluated the possibility of bias. The answers were then compared, and any disagreement was discussed between the reviewers.

Data synthesis

A narrative synthesis of the findings from the studies was included by providing a description and comparison between the different studies. This comprised a discussion of the contexts in which the studies have been undertaken, and the effects of WHM thematically grouped into physiological or psychological related outcomes. In each thematic grouping, different sets of outcomes were included. The data were not suitable for meta-analysis as there were various unrelated outcome measures resulting in heterogeneity across studies.

Study characteristics

Nine papers, seen in Fig 1 , published between 2014 and July 4, 2022, were identified that examined the WHM. One of these articles was identified from hand-searching the included studies’ reference lists [ 6 ]. Two of these articles [ 13 , 16 ] were further analyses of a study [ 10 ]. Table 1 summarises the characteristics of the listed studies. Of the nine papers included, there were eight individual trials. Four of the trials performed the full WHM [ 7 , 10 , 12 , 15 ] and four did only WHBM [ 6 , 8 , 11 , 15 ]. The application of WHBM differed between studies as some studies had one breathing exercise [ 6 , 8 , 11 , 12 ] while others had two breathing exercises [ 7 , 10 , 13 , 15 , 16 ]. The first breathing exercise was common in all studies. Cold exposure was conducted according to what was accessible to each study. However, all studies agreed on daily exposure to the cold through having cold showers, but the duration of exposure under the shower differed across studies. The meditation technique was the least focused component of the WHM. All studies had the same description of the meditation technique except for Petraskova Touskova et al . [ 12 ]. All the authors described it as visualisations meant to promote complete relaxation, while Petraskova Touskova et al . described it as a form of focusing to obtain self-awareness and will-power. Out of the nine papers, five conducted a randomised controlled trial (RCT) design [ 10 , 11 , 13 , 15 , 16 ], three conducted a crossover RCT design [ 6 – 8 ], and one conducted a prospective cohort design [ 12 ].

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0286933.g001

https://doi.org/10.1371/journal.pone.0286933.t001

Study quality

For the RCT studies, the RoB 2.0 overall risk of bias judgement was ‘high concerns’ for all studies except one study which was judged as ‘some concerns’ [ 8 ] ( Table 2 ). Most of the articles were judged as high concern due to the difficulty in blinding the participants and researchers to the intervention. For the cohort study [ 12 ], the SIGN tool was used. The cohort study was scored ‘unacceptable’ as it was not able to reduce the possibility of bias or confounding. An explanation of the results of the SIGN tool can be found in the S2 Table .

https://doi.org/10.1371/journal.pone.0286933.t002

Table 3 summarises the WHM description, key findings, and quality assessment. The outcomes were separated into physiological or psychological outcomes. The physiological outcome has six categories: stress response, pro-inflammatory/anti-inflammatory response, metabolites response, respiratory parameters, blood gas measurements, and reporting of symptoms, while psychological outcome has one category: psychological response. The stress response included outcomes such as epinephrine, norepinephrine, dopamine, cortisol, and heart rate (HR) whereas pro-inflammatory/anti-inflammatory response included tumour necrosis factor (TNF-α), interleukin-6 (IL-6), interleukin-8 (IL-8), interleukin-10 (IL-10), erythrocyte sedimentation rate (ESR), Ankylosing Spondylitis Disease Activity Score C-reactive protein (ASDAS-CRP), calprotectin, high-sensitivity C-reactive protein (hs-CRP), and Bath Ankylosing Spondylitis Disease Activity Index (BASDAI). Additionally, metabolites response contained lactate and pyruvate while respiratory parameters looked at minute ventilation (V E ), tidal volume (V T ), and breathing frequency (BF). The blood gas measurements category included carbon dioxide partial pressure (pCO 2 ), pH, and oxygen saturation outcomes and the psychological response category contained expectancy, optimism, neuroticism, short-form 36 (SF-36), EuroQol-5D (EQ-5D), Borg Rating of Perceived Exertion (RPE) scale, and customised questionnaire. Finally, the reporting of symptoms investigated flu-like, self-reported, and depressive symptoms, and Trauma Symptom Checklist-40 (TSC-40).

https://doi.org/10.1371/journal.pone.0286933.t003

Stress response

Two trials investigated epinephrine levels during the endotoxemia experiment [ 10 , 15 ] and one trial studied epinephrine levels during WHBM [ 15 ]. Baseline epinephrine levels in WHM-trained participants were significantly higher compared to non-trained participants (p = 0.007). After starting WHBM, epinephrine levels increased further in this group and peaked just before endotoxemia administration and remained elevated until WHBM was stopped [ 10 ]. Additionally, in Zwaag et al . endotoxemia study [ 15 ], participants had no variations in epinephrine levels throughout the experiment (p = 0.48) in both participants exposed to cold (CEX) and participants not performing the WHM (CON). However, an increase in epinephrine levels began much earlier and was significantly greater in both the cold exposure with breathing exercise without retention (CBR) (p = 0.01) and breathing exercise without retention (BRT) (p = 0.04) compared to the CEX and CON groups [ 15 ]. Furthermore, in Zwaag et al . breathing exercises study [ 15 ], both breathing exercises (WHBM with/without retention) raised epinephrine levels, which were unaffected by training duration (short vs extensive training: p = 0.71) or the trainer (independent trainer vs Wim Hof: p = 0.46). However, epinephrine was slightly more sustained in participants performing WHBM with retention compared to without retention (p = 0.003) [ 15 ]. Moreover, Kox et al . [ 10 ] explored norepinephrine and dopamine levels during the endotoxemia trial and found them to be within the reference range throughout the experiment [ 10 ].

The associations of cortisol with WHM was also explored. In Petraskova Touskova et al . study [ 12 ], the group not practising WHM had higher, but statistically non-significant cortisol levels at the end of the expedition (p = 0.327) compared to WHM-trained participants [ 12 ]. Whereas Kox et al . [ 10 ] found no variations in serum cortisol levels between the WHM-trained and non-trained groups. The serum cortisol levels were drawn before and throughout the period of WHM training. The only variation between the two groups was that serum cortisol levels normalised faster in WHM-trained participants [ 10 ]. The difference in the results could be due to the different collection procedures for cortisol. Kox et al . [ 10 ] collected cortisol from blood while Petraskova Touskova et al . [ 12 ] collected hair cortisol.

Four trials examined HR recordings [ 6 , 8 , 10 , 15 ]. Kox et al . [ 10 ] showed an increase in HR after starting WHBM which normalised faster in WHM-trained group compared to non-trained group [ 10 ]. Furthermore, Bahenský et al . [ 6 ] reported that HR tended to be lower after WHBM compared to no WHBM [ 6 ]. Citherlet et al . [ 8 ] also found HR significantly increased during hyperventilation (HV) and significantly decreased during breath hold in participants practising WHBM when compared to resting values [ 8 ]. Similarly, Zwaag et al . [ 15 ] observed a significant increase in HR in both CBR (p<0.001) and BRT (p<0.001) groups, during the first hyperventilation cycle, compared to the CON group. No changes in HR responses between CEX and CON groups (p = 0.89) were found during the endotoxemia experiment. The HR measurements of BRT and CBR groups were comparable to those of CEX and CON groups when WHBM was stopped [ 15 ]. In conclusion, HR was increased when WHBM was initiated and decreased when WHBM was stopped compared to control group.

Pro-inflammatory/anti-inflammatory responses

Two trials investigated cytokines levels during endotoxemia experiment [ 10 , 15 ]. In Kox et al . study [ 10 ], TNF-α, interleukin-6, and interleukin-8 levels were significantly lower in WHM-trained participants, whilst interleukin-10 levels were significantly higher (TNF-α, interleukin-6 and interleukin-8 levels were 53%, 57%, and 51% lower respectively; interleukin-10 levels were 194% higher) compared to non-trained group [ 10 ]. Similarly, Zwaag et al . endotoxemia study [ 15 ], found TNF-α (p = 0.03), interleukin-6 (p = 0.03), and interleukin-8 (p<0.001) levels significantly lower, whereas interleukin-10 levels significantly higher (p = 0.02) in CBR group than CON group. However, similar but less pronounced effects on cytokine levels were found when comparing BRT group to CON group for interleukin-6 (p = 0.04) and interleukin-8 (p = 0.02), but not for interleukin-10 (p = 0.17). Additionally, CEX group did not show any significant changes in TNF-α (p = 0.93), interleukin-6 (p = 0.73), interleukin-8 (p = 0.99), and interleukin-10 (p = 0.44) levels compared to CON [ 15 ]. Both studies found similar results although there was a difference in WHBM. Kox et al . [ 10 ] performed WHBM with retention, while Zwaag et al . endotoxemia study [ 15 ] performed WHBM without retention.

Inflammation outcomes were also measured in participants with axial spondyloarthritis. During the 8-week WHM training, ESR, median BASDAI, ASDAS-CRP, and serum calprotectin decreased significantly (p = 0.040, p = 0.012, p = 0.044, p = 0.064 respectively) compared to participants not receiving WHM training. However, no statistical significance was found in hs-CRP between both groups (p = 0.103) [ 7 ].

Metabolites response

Zwaag et al . [ 16 ] showed that lactate and pyruvate play an important role in the anti-inflammatory response in WHM-trained participants. High pyruvate concentration, but not lactate concentration, increased endotoxemia-induced interleukin-10 production, and the combination of both metabolites resulted in an even more prominent and statistically robust increase. Although these two metabolites were highly intercorrelated (p<0.0001), they were not associated with the highly elevated epinephrine levels at any of the examined time points (p-values>0.15). Additionally, pyruvate reduced the production of interleukin-6. Lactate, pyruvate, and the two metabolites combined also tended to reduce endotoxemia-induced TNF-α production; however, significance was not achieved [ 16 ]. In Kox et al . study [ 10 ], lactate levels were significantly higher but not to a clinically relevant level in WHM-trained compared to non-trained participants [ 10 ].

Respiratory parameters

Three papers measured V E [ 6 , 8 , 11 ] and two of them also measured V T and BF [ 6 , 11 ]. All three papers were investigating whether performing WHBM before an exercise will improve performance. Bahenský et al . [ 6 ] found the difference in mean values at each load to be significant for V E [ 6 ] and Citherlet et al . [ 8 ] found a significant difference (p = 0.039) between participants performing spontaneous breathing (SB) and WHBM at rest, but the mean difference was not statistically significant after performing an exercise (p>0.05) [ 8 ]. Additionally, Marko et al . [ 11 ] showed that at each load stage, V E depicted statistical insignificance between WHBM and normal breathing (p = 0.138, p = 0.825, p = 0.479, p = 0.489) [ 11 ]. Furthermore, for V T and BF measurements, Bahenský et al . [ 6 ] found the difference in mean values at each load to be significant (p<0.001, p<0.001) [ 6 ]; however, Marko et al . [ 11 ] showed no statistical significance between WHBM participants and normal breathing at any load stage (V T : p = 0.630, p = 0.377, p = 0.688, p = 0.087; BF: p = 0.794, p = 0.917, p = 0.956, p = 0.296) [ 11 ].

Blood gas measurements

Two trials examined blood gas measurements [ 10 , 15 ]. In Kox et al . study [ 10 ], starting the WHBM resulted in an immediate and profound decrease in pCO 2 and an increase in pH, but it was not stated if these changes were statistically significant. Furthermore, during WHBM training, the trained group’s oxygen saturation decreased significantly [ 10 ]. In contrast, Zwaag et al . endotoxemia study [ 15 ], found pH and oxygen saturation levels significantly higher in the groups performing WHBM (BRT: p<0.01, p<0.001; CBR: p<0.01, p<0.001 respectively) when compared to CON. Whereas, pCO 2 was significantly lower in the same groups (BRT: p<0.001; CBR: p<0.001) when compared to CON [ 15 ]. In Zwaag et al . breathing exercises study [ 15 ], oxygen saturation significantly decreased at the end of each retention phase, only when WHBM is performed with retention (p<0.001) compared to without retention. While pH and pCO 2 were similar in both WHBM exercises, except for a small but statistically significant difference at the final measurement time point (pH: p<0.001; pCO 2 : p<0.001) [ 15 ].

Psychological response

One paper [ 13 ] investigated the effect of optimism and neuroticism on WHM. It found that a higher level of expectancy and optimism helped significantly potentiate the effect of WHM while a decrease in neuroticism level was not found to significantly help. Participants in the training group were overall relatively optimistic and low in neuroticism. A higher level of optimism was associated with higher interleukin-10 levels (p<0.05) and epinephrine levels (p<0.01). Neuroticism was not found to be a significant predictor of endotoxin response. Participants’ expectations to overcome the endotoxemia experiment significantly increased from before WHM training to after the endotoxemia experiment (p = 0.003) [ 13 ].

Another paper [ 7 ] explored the quality of life of participants and found that SF-36 physical and mental component scores significantly increased over the WHM-training period (p = 0.004, p = 0.004 respectively) compared to the non-trained group. While EQ-5D and EQ-5D visual analogue scale did not experience a significant change between the WHM-trained and non-trained groups (EQ-5D: p = 0.933, p = 0.102; EQ-5D visual analogue scale: p = 0.090, p = 0.674) [ 7 ].

The last paper [ 8 ] reported the results of a customised questionnaire, used for the subjective examination of the three sessions, and Borg RPE scale. Positive effects of increased energy, less fatigue, and improved breathing were reported by 87% of the participants. Almost half of the participants (47%) also said they would think about incorporating the WHBM into their own practices in the future. Additionally, when the experiment was performed using the WHBM, 66.7% of participants rated it the best method in terms of perceived performance compared to 13.3% for SB and 20% for HV. Likewise, 73.3% of participants judged the WHBM to be the best method to perform, while 53.3% judged SB as the worst. However, 33% of the participants using WHBM reported negative effects of deafness and heaviness [ 8 ]. Moreover, Borg RPE score was found significantly lower in Bahenský et al . (p<0.001) [ 6 ] and Citherlet et al . (WHBM compared to HV (p = 0.008) and SB (p = 0.017)) [ 8 ], meaning that the perceived intensity of a training session is less when WHM is being used [ 6 , 8 ].

Reporting of symptoms

Three papers reported experience of flu-like symptoms following endotoxemia administration [ 10 , 13 , 15 ]. Kox et al . [ 10 ] reported lower flu-like and self-reported symptoms and faster recovery from fever in participants performing WHM [ 10 ]. Similarly, Middendorp et al . [ 13 ] showed that a higher expectation of the training’s effects was associated with lower flu-like clinical symptoms (p<0.01) compared to non-trained group [ 13 ]. Likewise, Zwaag et al . endotoxemia study [ 15 ] results showed that only CEX group had significantly lower flu-like symptoms (p = 0.017) compared to CON group. All other groups had comparable peak symptom scores to CON group (BRT: (p = 0.70); CBR: (p = 0.21)). Additionally, when compared to the CON group, symptoms disappeared considerably faster in all three intervention groups (BRT: p<0.001; CEX: p = 0.01; CBR: p = 0.002) [ 15 ]. Overall all participants reported flu-like symptoms but it was significantly lower in participants practising WHM or any one of its pillars such as CEX. The flu-like symptoms also disappeared faster in these participants.

Two papers also reported depressive symptoms [ 7 , 12 ]. Petraskova Touskova et al . [ 12 ] found that at the end of the expedition, depressive symptoms were significantly lower in the WHM-trained group compared to non-trained group (p = 0.03), while TSC-40 scores, which measures stess-related symptoms, were higher in the non-trained group, but not significantly [ 12 ]. Additionally, Buijze et al . [ 7 ] found no significant effect on depressive symptoms between WHM-trained and non-trained groups [ 7 ].

This is the first systematic review conducted on WHM. The findings of this review suggest that WHM may affect the reduction of inflammation in healthy and non-healthy participants. In addition, there are currently mixed findings on the effect of WHBM and exercise performance. Despite the statistical significance observed in some studies, it must be noted that the quality of the studies is very low, meaning that all the results must be interpreted with caution. Additionally, the low sample size (15–48 individuals per study) and large proportion of males in the studies (86.4%) make the results non-generalizable to the public. Consideration should also be given that participants might experience the placebo effect, where improvements in patients’ symptoms are due to their participation in the therapeutic encounter, with its rituals, symbols, and interactions [ 27 ]. For example, Middendorp et al . [ 13 ] found that a higher level of expectancy and optimism helped significantly potentiate the effect of WHM [ 13 ]. However, Zwaag et al . [ 15 ] who queried the so-called ‘guru effect’ [ 28 ] and whether it is necessary to be trained by the creator of the intervention to influence symptomatology [ 15 ], found that observed physiological and immunological effects are independent from the individual who provides the WHM treatment intervention.

Out of all the categories, WHM appears to have the most benefit in the stress and pro-inflammatory/anti-inflammatory response categories. Inflammation, especially when chronic, can cause severe complications such as cardiovascular diseases, cancer, diabetes, rheumatoid arthritis, asthma, chronic obstructive pulmonary disease, alzheimer, chronic kidney disease, and inflammatory bowel disease, therefore, reducing inflammation is beneficial [ 29 ]. The WHM reduces inflammation using a different mechanism of action (MOA) than other anti-inflammatory interventions. The closest MOA to that of WHM is corticosteroids’ MOA as they both increase interleukin-10 production. WHM increases epinephrine, causing an increase in interleukin-10 which leads to a reduction in pro-inflammatory cytokines [ 10 , 15 ]. Whereas corticosteroids activate anti-inflammatory genes, including interleukin-10 gene, and inactivate multiple inflammatory genes by inhibiting histone acetyltransferase and recruiting histone deacetylase-2 activity to the inflammatory gene transcriptional complex [ 30 ]. The findings of this review suggest that WHM may provide some benefits in healthy and non-healthy people as it was suggested to be safe and might decrease inflammation, unlike corticosteroids, which are only given to non-healthy people due to having many side effects.

WHBM effect on exercise performance category showed mixed results. Although it is too early to decide whether WHBM enhances the performance or not, WHBM could have other potential applications in the physical exertion category. For example, a published letter suggests that WHBM may be helpful in preventing and reversing symptoms of acute mountain sickness (AMS) [ 31 ]. Wim Hof and a combination of healthy and non-healthy participants climbed a mountain using the WHBM in 2 days instead of the usual 4–7 days and none of the participants suffered from severe AMS. These findings suggest that a trial on this topic may be of importance, especially for the rescue teams that must ascend quickly [ 31 ]. However, although this letter was published in a journal, it was not peer-reviewed and there was no control group; hence the results should be interpreted with caution. Further studies on the effect of WHBM and physical exertion must be conducted before drawing a conclusion.

There are three pillars to WHM: cold exposure, WHBM, and meditation. The latter is a foundation of the other two pillars, thus there were no studies testing it by itself. In this review, cold exposure alone was suggested to have an insignificant effect on epinephrine and cytokine levels [ 15 ]. Similarly, the wider literature agrees with this finding. For example, cold water immersion (CWI) did activate the immune system and alter systematic inflammation but not significantly [ 32 – 35 ]. Only one study found CWI helpful in reducing inflammation in rock climbers by reducing vessel permeability toward the site of inflammation [ 36 ]. Furthermore, in this review, WHBM alone was claimed to significantly increase epinephrine levels, regardless of the duration of training or trainer, and that increase was more prolonged in participants performing WHBM with retention [ 15 ]. On the other hand, cytokine levels were only measured in participants performing WHBM without retention. There were no trials comparing WHBM with retention alone on cytokine levels. When WHBM was performed without retention, pro-inflammatory cytokines significantly decreased, but interleukin-10 levels were not increased significantly [ 15 ]. This might be because hypoxia improves interleukin-10 release and reduces the pro-inflammatory response via improved adenosine release [ 37 ]. Thus, it would be interesting for future trials to investigate the effects of WHBM with retention alone on cytokine levels. Nevertheless, when cold exposure was combined with WHBM, regardless of whether it was with retention or without, statistical significance was achieved on epinephrine and cytokine levels [ 10 , 15 ]. Therefore, it is probably best to perform all the pillars of WHM to achieve a significant improvement in immunomodulatory effects. Future trials should investigate whether there is a significant difference between performing WHBM with retention and cold exposure compared to WHBM without retention and cold exposure on epinephrine and cytokine levels.

This review suffers from several limitations. Firstly, only two bibiographic databases were searched to identify suitable papers for this review. Including the database EMBASE, as recommended for inclusion in systematic reviews by Bramer et al . [ 38 ] may have increased the reach of the search and identified further relevant papers on this topic. Secondly, the search strategy keywords used such as breathing exercise, breathing technique, cold temperature, cold exposure, and cold shower may have excluded studies that used words combined with breathing and cold other than these words. Using breathing and cold as search words alone may have identified further relevant studies. Thirdly, the outcomes were very heterogeneous as many unrelated outcomes are included. This review could have focused on synthesising specific outcomes such as only focusing on outcomes related to stress and inflammation. Fourthly, all the trials had a very high risk of bias. This was due to the lack of a prior published protocol, small sample size, and complexity of blinding the participants and outcome assessors to the intervention. Furthermore, some outcomes such as psychological outcomes were difficult to measure as they are subjective measures usually assessed using a questionnaire. Since the participants were not blinded, it was very difficult to ensure that the answers were honest and valid to the experience. Lastly, the sample size was very small, sometimes affecting the results as any error in measurement or a loss of follow-up can significantly shift the trial results. More evidence needs to be synthesised about WHM before being recommended to the public. An intervention like WHM presents a challenge for rigorous experimental research as participants cannot be blinded to a WHM intervention, therefore the gold standard approach of a double-blinded RCT is not possible. It does appear to offer benefits in attenuating inflammation with minimal serious adverse events and good positive effects of increased energy, less fatigue, and improved breathing [ 7 , 8 , 10 , 15 ]. Thus, WHM can probably be used within lifestyle medicine to decrease inflammation in people suffering from inflammatory disorders. However, to be able to recommend WHM in an unwell population, future trials should publish a protocol outlining their experiment before starting the trial, increase the sample size, and make sure to account for loss to follow-up. Additionally, researchers should make sure to blind outcome assessors, and ensure that the outcomes assessed are objective and not subjective.

Considering all the studies, the WHM may produce promising immunomodulatory effects but more research of higher quality is needed to substantiate this finding. The combination of cold exposure and WHBM appeared to most effectively reduce the inflammatory response. Hence, all the pillars of the WHM are important to extract the benefits. The focus of future studies should further investigate the benefits of WHM in preventing or treating diseases, such as inflammatory disorders, in non-healthy participants and explore the use of WHBM in enhancing exercise performance. Studies about WHM have not yet investigated all the beneficial claims the WHM states to have. Future studies may provide valuable insights about WHM as there is still much to explore.

Supporting information

S1 checklist. prisma 2009 checklist..

https://doi.org/10.1371/journal.pone.0286933.s001

S1 Table. SIGN tool assessment.

https://doi.org/10.1371/journal.pone.0286933.s002

• View Article
• Google Scholar
• PubMed/NCBI

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Elsevier - PMC COVID-19 Collection

Combining qualitative and quantitative research within mixed method research designs: A methodological review

Ulrika Östlund.

a Division of Nursing, Department of Neurobiology, Care Sciences and Society, Karolinska Institutet, Stockholm, Sweden

b Institute for Applied Health Research/School of Health, Glasgow Caledonian University, United Kingdom

Yvonne Wengström

c Division of Nursing, Department or Neurobiology, Care Sciences and Society, Karolinska Institutet, Stockholm, Sweden

Neneh Rowa-Dewar

d Public Health Sciences, University of Edinburgh, United Kingdom

It has been argued that mixed methods research can be useful in nursing and health science because of the complexity of the phenomena studied. However, the integration of qualitative and quantitative approaches continues to be one of much debate and there is a need for a rigorous framework for designing and interpreting mixed methods research. This paper explores the analytical approaches (i.e. parallel, concurrent or sequential) used in mixed methods studies within healthcare and exemplifies the use of triangulation as a methodological metaphor for drawing inferences from qualitative and quantitative findings originating from such analyses.

This review of the literature used systematic principles in searching CINAHL, Medline and PsycINFO for healthcare research studies which employed a mixed methods approach and were published in the English language between January 1999 and September 2009.

In total, 168 studies were included in the results. Most studies originated in the United States of America (USA), the United Kingdom (UK) and Canada. The analytic approach most widely used was parallel data analysis. A number of studies used sequential data analysis; far fewer studies employed concurrent data analysis. Very few of these studies clearly articulated the purpose for using a mixed methods design. The use of the methodological metaphor of triangulation on convergent, complementary, and divergent results from mixed methods studies is exemplified and an example of developing theory from such data is provided.

A trend for conducting parallel data analysis on quantitative and qualitative data in mixed methods healthcare research has been identified in the studies included in this review. Using triangulation as a methodological metaphor can facilitate the integration of qualitative and quantitative findings, help researchers to clarify their theoretical propositions and the basis of their results. This can offer a better understanding of the links between theory and empirical findings, challenge theoretical assumptions and develop new theory.

What is already known about the topic?

• • Mixed methods research, where quantitative and qualitative methods are combined, is increasingly recognized as valuable, because it can potentially capitalize on the respective strengths of quantitative and qualitative approaches.
• • There is a lack of pragmatic guidance in the research literature as how to combine qualitative and quantitative approaches and how to integrate qualitative and quantitative findings.
• • Analytical approaches used in mixed-methods studies differ on the basis of the sequence in which the components occur and the emphasis given to each, e.g. parallel, sequential or concurrent.

What this paper adds

• • A trend for conducting parallel analysis on quantitative and qualitative data in healthcare research is apparent within the literature.
• • Using triangulation as a methodological metaphor can facilitate the integration of qualitative and quantitative findings and help researchers to clearly present both their theoretical propositions and the basis of their results.
• • Using triangulation as a methodological metaphor may also support a better understanding of the links between theory and empirical findings, challenge theoretical assumptions and aid the development of new theory.

1. Introduction

Mixed methods research has been widely used within healthcare research for a variety of reasons. The integration of qualitative and quantitative approaches is an interesting issue and continues to be one of much debate ( Bryman, 2004 , Morgan, 2007 , Onwuegbuzie and Leech, 2005 ). In particular, the different epistemological and ontological assumptions and paradigms associated with qualitative and quantitative research have had a major influence on discussions on whether the integration of the two is feasible, let alone desirable ( Morgan, 2007 , Sale et al., 2002 ). Proponents of mixed methods research suggest that the purist view, that quantitative and qualitative approaches cannot be merged, poses a threat to the advancement of science ( Onwuegbuzie and Leech, 2005 ) and that while epistemological and ontological commitments may be associated with certain research methods, the connections are not necessary deterministic ( Bryman, 2004 ). Mixed methods research can be viewed as an approach which draws upon the strengths and perspectives of each method, recognising the existence and importance of the physical, natural world as well as the importance of reality and influence of human experience ( Johnson and Onquegbuzie, 2004 ). Rather than continue these debates in this paper, we aim to explore the approaches used to integrate qualitative and quantitative data within healthcare research to date. Accordingly, this paper focuses on the practical issues of conducting mixed methods studies and the need to develop a rigorous framework for designing and interpreting mixed methods studies to advance the field. In this paper, we will attempt to offer some guidance for those interested in mixed methods research on ways to combine qualitative and quantitative methods.

The concept of mixing methods was first introduced by Jick (1979) , as a means for seeking convergence across qualitative and quantitative methods within social science research ( Creswell, 2003 ). It has been argued that mixed methods research can be particularly useful in healthcare research as only a broader range of perspectives can do justice to the complexity of the phenomena studied ( Clarke and Yaros, 1988 , Foss and Ellefsen, 2002 , Steckler et al., 1992 ). By combining qualitative and quantitative findings, an overall or negotiated account of the findings can be forged, not possible by using a singular approach ( Bryman, 2007 ). Mixed methods can also help to highlight the similarities and differences between particular aspects of a phenomenon ( Bernardi et al., 2007 ). Interest in, and expansion of, the use of mixed methods designs have most recently been fuelled by pragmatic issues: the increasing demand for cost effective research and the move away from theoretically driven research to research which meets policymakers’ and practitioners’ needs and the growing competition for research funding ( Brannen, 2009 , O’Cathain et al., 2007 ).

Tashakkori and Creswell (2007) broadly define mixed methods research as “research in which the investigator collects and analyses data, integrates the findings and draws inferences using both qualitative and quantitative approaches” (2007:3). In any mixed methods study, the purpose of mixing qualitative and quantitative methods should be clear in order to determine how the analytic techniques relate to one another and how, if at all, the findings should be integrated ( O’Cathain et al., 2008 , Onwuegbuzie and Teddlie, 2003 ). It has been argued that a characteristic of truly mixed methods studies are those which involve integration of the qualitative and quantitative findings at some stage of the research process, be that during data collection, analysis or at the interpretative stage of the research ( Kroll and Neri, 2009 ). An example of this is found in mixed methods studies which use a concurrent data analysis approach, in which each data set is integrated during the analytic stage to provide a complete picture developed from both data sets after data has been qualitised or quantitised (i.e. where both forms of data have been converted into either qualitative or quantitative data so that it can be easily merged) ( Onwuegbuzie and Teddlie, 2003 ). Other analytic approaches have been identified including; parallel data analysis, in which collection and analysis of both data sets is carried out separately and the findings are not compared or consolidated until the interpretation stage, and finally sequential data analysis, in which data are analysed in a particular sequence with the purpose of informing, rather than being integrated with, the use of, or findings from, the other method ( Onwuegbuzie and Teddlie, 2003 ). An example of sequential data analysis might be where quantitative findings are intended to lead to theoretical sampling in an in depth qualitative investigation or where qualitative data is used to generate items for the development of quantitative measures.

When qualitative and quantitative methods are mixed in a single study, one method is usually given priority over the other. In such cases, the aim of the study, the rationale for employing mixed methods, and the weighting of each method determine whether, and how, the empirical findings will be integrated. This is less challenging in sequential mixed methods studies where one approach clearly informs the other, however, guidance on combining qualitative and quantitative data of equal weight, for example, in concurrent mixed methods studies, is rather less clear ( Foss and Ellefsen, 2002 ). This is made all the more challenging by a common flaw which is to insufficiently and inexplicitly identify the relationships between the epistemological and methodological concepts in a particular study and the theoretical propositions about the nature of the phenomena under investigation ( Kelle, 2001 ).

One approach to combining different data of equal weight and which facilitate clear identification of the links between the different levels of theory, epistemology, and methodology could be to frame triangulation as a ‘methodological metaphor’, as argued by Erzberger and Kelle (2003) . This can help to; describe the logical relations between the qualitative and quantitative findings and the theoretical concepts in a study; demonstrate the way in which both qualitative and quantitative data can be combined to facilitate an improved understanding of particular phenomena; and, can also be used to help generate new theory ( Erzberger and Kelle, 2003 ) (see Fig. 1 ). The points of the triangle represent theoretical propositions and empirical findings from qualitative and quantitative data while the sides of the triangle represent the logical relationships between these propositions and findings. The nature and use of the triangle depends upon the outcome from the analysis, whether that be convergent , where qualitative and quantitative findings lead to the same conclusion; complementary, where qualitative and quantitative results can be used to supplement each other or; divergent , where the combination of qualitative and quantitative results provides different (and at times contradictory) findings. Each of these outcomes requires a different way of using the triangulation metaphor to link theoretical propositions to empirical findings ( Erzberger and Kelle, 2003 ).

Illustrating the triangulation triangle ( Erzberger and Kelle, 2003 )

1.1. Purpose of this paper

In the following paper, we identify the analytical approaches used in mixed methods healthcare research and exemplify the use of triangulation ( Erzberger and Kelle, 2003 ) as a methodological metaphor for drawing inferences from qualitative and quantitative findings. Papers reporting on mixed methods studies within healthcare research were reviewed to (i) determine the type of analysis approach used, i.e. parallel, concurrent, or sequential data analysis and, (ii) identify studies which could be used to illustrate the use of the methodological metaphor of triangulation suggested by Erzberger and Kelle (2003) . Four papers were selected to illustrate the application of the triangulation metaphor on complementary, convergent and divergent outcomes and to develop theory.

This literature review has used systematic principles ( Cochrane, 2009 , Khan, 2001 ) to search for mixed methods studies within healthcare research. The first search was conducted in September 2009 in the data bases CINAHL, Medline and PsycINFO on papers published in English language between 1999 and 2009. To identify mixed methods studies, the search terms (used as keywords and where possible as MeSH terms) were: “mixed methods”, “mixed research methods”, “mixed research”, “triangulation”, “complementary methods”, “concurrent mixed analysis” and “multi-strategy research.” These terms were searched individually and then combined (with OR). This resulted in 1896 hits in CINAHL, 1177 in Medline and 1943 in PsycINFO.

To focus on studies within, or relevant to, a healthcare context the following search terms were used (as keywords or as MeSH terms and combined with OR): “health care research”; “health services research”; and “health”. These limits applied to the initial search (terms combined with AND) resulted in 205 hits in Medline and 100 hits in PsycINFO. Since this combination in CINAHL only limited the search results to 1017; a similar search was conducted but without using the search term triangulation to capture mixed methods papers; resulting in 237 hits. In CINAHL the search result on 1017 papers was further limited by using “interventions” as a keyword resulting in 160 papers also selected to be reviewed. Moreover; in Medline the mixed methods data set was limited by the MeSH term “research” resulting in 218 hits and in PsycINFO with “intervention” as keyword or MeSH term resulting in 178 hits.

When duplicates were removed the total numbers of papers identified were 843. The abstracts were then reviewed by each author and those identified as relevant to the review were selected to be retrieved and reviewed in full text. Papers were selected based on the following inclusion criteria: empirical studies; published in peer review journals; healthcare research (for the purpose of this paper defined as any study focussing on participants in receipt, or involved in the delivery, of healthcare or a study conducted within a healthcare setting, e.g. different kinds of care, health economics, decision making, and professionals’ role development); and using mixed methods (defined as a study in which both qualitative and quantitative data were collected and analysed ( Halcomb et al., 2009b ). To maintain rigour, a random sample (10%) of the full text papers was reviewed jointly by two authors. Any disagreements or uncertainties that arose between the reviewers regarding their inclusion or in determining the type of analytic approach used were resolved through discussion between the authors.

In addition to the criteria outlined above, papers were excluded if the qualitative element constituted a few open-ended questions in a questionnaire, as we would agree with previous authors who have argued such studies do not strictly constitute a mixed methods design ( Kroll and Neri, 2009 ). Papers were also excluded if they could not be retrieved in full text via the library services at the University of Edinburgh, Glasgow Caledonian University or the Karolinska Institutet, or did not adequately or clearly describe their analytic strategy, for example, failing to report how the qualitative and quantitative data sets were analysed individually and, where relevant, how these were integrated. See Table 1 for reasons for the exclusion of subsequent papers.

Reasons for exclusion.

A second search was conducted within the databases of Medline, PsychInfo and Cinahl to identify studies which have specifically used Erzberger and Kelle's (2003) triangulation metaphor to frame the description and interpretation of their findings. The term ‘triangulation metaphor’ (as keywords) and author searches on ‘Christian Erzberger’ and ‘Udo Kelle’ were conducted. Three papers, published by Christian Erzberger and Udo Kelle, were identified in the PsychInfo databases but none of these were relevant to the purpose of this review. There were no other relevant papers identified in the other two databases.

168 Papers were included in the final review and reviewed to determine the type of mixed analysis approach used, i.e. parallel, concurrent, or sequential mixed analysis. Four of these papers (identified from the first search on mixed methods studies and healthcare research) were also used to exemplify the use of the methodological metaphor of triangulation ( Erzberger and Kelle, 2003 ). Data was extracted from included papers accordingly in relation to these purposes.

In total, 168 papers were included in our review. The majority of these studies originated in the USA ( n  = 63), the UK ( n  = 39) and Canada ( n  = 19), perhaps reflecting the considerable interest and expertise in mixed methods research within these countries. The focus of the studies included in the review varied significantly and the populations studied included both patients and healthcare professionals.

3.1. Analytic approaches

Table 2 illustrates the types of analytic approaches adopted in each of the studies included in the review. The most widely used analytic approach ( n  = 98) was parallel analysis ( Creswell and Plano Clark, 2007 ). However, very few of the studies employing parallel analysis clearly articulate their purpose for mixing qualitative and quantitative data, the weighting (or priority) given to the qualitative and quantitative data or the expected outcomes from doing so, mirroring previous research findings ( O’Cathain et al., 2008 ). The weighting, or priority, of the qualitative and quantitative data in a mixed methods study is dependent upon various factors including; the aims of the study and whether the purpose is, for example, to contextualise quantitative data using qualitative data or to use qualitative data to inform a larger quantitative approach such as a survey. Nonetheless, the omission of this statement makes it difficult to determine which data set the conclusions have been drawn from and the role of, or emphasis on, each approach. Therefore, is of importance for authors to clearly state this in their papers ( Creswell and Plano Clark, 2007 ). A number of studies had also used sequential data analysis ( n  = 46), where qualitative approaches were visibly used to inform the development of both clinical tools (e.g. Canales and Rakowski, 2006 ) and research measures and surveys (e.g. Beatty et al., 2004 ) or where quantitative surveys were supplemented by and issues further explored using qualitative approaches (e.g. Abadia and Oviedo, 2009 , Cheng, 2004 , Halcomb et al., 2008 ).

Included papers illustrating their analytical approach and country of origin.

Most notably, with only 20 included studies using a concurrent approach to data analysis, this was the least common design employed. Compared to the studies using a parallel or sequential approach, the authors of concurrent studies more commonly provided an explanation for their purpose of using a mixed methods design in their study, e.g. how it addressed a gap or would facilitate and advance the state of knowledge (e.g. Bussing et al., 2005 , Kartalova-O’Doherty and Tedstone Doherty, 2009 ). Despite this, there remained a lack of clarity within these studies about the weighting given to, and priority of, each method. Consequently, the importance and relevance of the findings produced by each approach and how these have informed their conclusions and interpretation is lacking. In four of the included papers a combination of approaches to data analysis (i.e. sequential and concurrent, parallel and concurrent, or sequential and parallel) were used. In the next section, we have selected papers to illustrate the methodological metaphor of triangulation ( Erzberger and Kelle, 2003 ).

3.2. Using the methodological metaphor of triangulation

We have selected four papers from our review ( Lukkarinen, 2005 , Midtgaard et al., 2006 , Shipman et al., 2008 , Skilbeck et al., 2005 ) to illustrate how the methodological metaphor of triangulation ( Erzberger and Kelle, 2003 ) can be applied to mixed methods studies. Each of these studies has been used to illustrate how the metaphor of triangulation can be applied to studies producing: (i) complementary findings, (ii) convergent findings, and (iii) divergent findings. In the following section, we demonstrate how the application of the metaphor can be used as a framework both to develop theory and to facilitate the interpretation of the findings from mixed methods studies and their conclusions in each of these scenarios, and how using the metaphor can help to promote greater clarity of the study's purpose, its theoretical propositions, and the links between data sets.

3.2.1. Triangulating complementary results

To exemplify the use of the methodological metaphor of triangulation ( Erzberger and Kelle, 2003 ) for drawing inferences from complementary results, we have drawn on the results of a UK based study by Shipman et al. (2008) ( Fig. 2 ). In the UK, members of district nursing teams (DNs) provide most nursing care to people at home in the last year of life. Following concerns that inadequate education might limit the confidence of some DNs to support patients and their carers’ at home, and that low home death rates may in part be related to this, the Department of Health (DH) identified good examples of palliative care educational initiatives for DNs and invested in a 3-year national education and support programme in the principles and practice of palliative care. Shipman et al.’s study evaluates whether the programme had measurable effects on DN knowledge and confidence in competency in the principles and practice of palliative care. The study had two parts, a summative (concerned with outcomes) quantitative component which included ‘before and after’ postal questionnaires which measured effects on DNs’ ( n  = 1280) knowledge, confidence and perceived competence in the principles and practice of palliative care and a formative (concerned with process) qualitative component which included semi-structured focus groups and interviews with a sub-sample of DNs ( n  = 39).

Illustrating the use of triangulation ( Erzberger and Kelle, 2003 ) on complementary results in the study by Shipman et al. (2008) .

While their theoretical proposition may not be explicitly stated by the authors, there is clearly an implicit theoretical proposition that the educational intervention would improve DNs knowledge and confidence (theoretical proposition 1, Fig. 2 ). This was supported by the quantitative findings which showed significant improvement in the district nurses confidence in their professional competence post intervention. Qualitative results supported and complemented the quantitative findings as the district nurses described several benefits from the program including greater confidence in tackling complex problems and better communication with patient and carers’ because of greater understanding of the reasons for symptoms. Thus, a complementary theoretical proposition (theoretical proposition 2, Fig. 2 ) can be deduced from the qualitative findings: the DN's better understanding of factors contributing to complex problems and underlying reasons for symptoms led to improved confidence in competence raised from district nurses increased understanding.

Fig. 2 illustrates the theoretical propositions, the empirical findings from qualitative and quantitative data and the logical relationships between these. Theoretical proposition 1 is supported by the quantitative findings. From qualitative findings, a complementary theoretical proposition (theoretical proposition 2) can be stated explaining the process that led to the DNs improved confidence in competence.

3.2.2. Triangulating convergent results

To illustrate how the methodological metaphor of triangulation can be used to draw inferences from convergent findings, we have drawn on the example of a Danish study by Midtgaard et al. (2006) ( Fig. 3 ). This study was conducted to explore experiences of group cohesion and changes in quality of life (QoL) among people ( n  = 55) who participated in a weekly physical exercise intervention (for six weeks) during treatment for cancer. The study, conducted in a Danish hospital, involved the use of structured QoL questionnaires, administered at baseline and post intervention (at six weeks) to determine changes in QoL and health status, and qualitative focus groups, conducted post intervention (at six weeks), to explore aspects of cohesion within the group. With regards to the theoretical proposition of the study ( Fig. 3 ), group cohesion was seen as essential to understand the processes within the group that facilitated the achievement of desired outcomes and the satisfaction of affective needs as well as promoting a sense of belonging to the group itself.

Illustrating the use of triangulation ( Erzberger and Kelle, 2003 ) on convergent results in the study by Midtgaard et al. (2006) .

This proposition was deductively tested in an intervention where patients exercised in mixed gender groups of seven to nine members during a nine hour weekly session over a six week period and was supported by both the empirical quantitative and qualitative findings. The quantitative data showed significant improvements in peoples’ emotional functioning, social functioning and mental health. The qualitative data showed how the group setting motivated the individuals to pursue personal endeavors beyond physical limitations, that patients used each others as role models during ‘down periods’ and how exercising in a group made individuals feel a sense of obligation to train and to do their best. This subsequently helped to improve their social functioning which in turn satisfied their affective needs, improving their improved emotional functioning and mental health.

Fig. 3 illustrates the theoretical propositions, empirical findings from qualitative and quantitative data and the logical relationships between these. Both the quantitative and qualitative findings, demonstrating improvements in participants’ emotional and social functioning and their mental health, can be attributed to the nature of group cohesion within the programme as expected.

3.2.3. Triangulating divergent results

Qualitative and quantitative results that seem to contradict each other are often explained as resulting from methodological error. However, instead of a methodological flaw, a divergent result could be a consequence of the inadequacy of the theoretical concepts used. This may indicate the need for changing or developing the theoretical concepts involved ( Erzberger and Kelle, 2003 ). The following example of using the methodological metaphor of triangulation ( Erzberger and Kelle, 2003 ) for drawing inferences from divergent results is intended as an example rather than an attempt to change the theoretical concept involved. In a study by Skilbeck et al. (2005) ( Fig. 4 ), some results were found to be divergent which was explained as resulting from the use of inadequate questionnaires. We do not wish to critique their conclusion; rather we intend to simply offer an alternative interpretation for their findings.

Illustrating the use of triangulation ( Erzberger and Kelle, 2003 ) on divergent results using the study by Skilbeck et al. (2005) .

The study aimed to explore family carers’ expectations and experiences of respite services provided by one independent hospice in North England. This hospice provides inpatient respite beds specifically for planned respite admission for a two-week period. Referrals were predominated from general practitioners and patients and their carers were offered respite care twice a year, during the study this was reduced to once a year for each patient. Data was collected prior to respite admission and post respite care by semi-structured interviews and using the Relative Stress Scale inventory (RSSI), a validated scale to measure relative distress in relation to caring. Twenty-five carers were included but pre- and post-data were completed by 12 carers. Qualitative data was analysed by using a process of constant comparison and quantitative data by descriptive and comparative statistical analysis.

No clear theoretical proposition was stated by the authors, but from the definition of respite care it is possible to deduce that ‘respite care is expected to provide relief from care-giving to the primary care provider’ (theoretical proposition 1, Fig. 4 ). This proposition was tested quantitatively by pre- and post-test using the RSSI showing that the majority of carers experienced either a negative or no change in scores following the respite stay (no test of significance was stated). Accordingly, the theoretical proposition was not supported by the quantitative empirical data. The qualitative empirical results, however, were supportive in showing that most of the carers considered respite care to be important as it enabled them to have a break and a rest from ongoing care-responsibilities. From this divergent empirical data it could be suggested changing or developing the original theoretical proposition. It seems that respite care gave the carers relief from their care-responsibilities but not from the distress carers experienced in relation to caring (measured by the used scale). We would therefore suggest that in order to lessen distress related to caring, other types of support is also needed which would change the theoretical proposition as suggested (theoretical proposition 2).

Fig. 4 illustrates the theoretical propositions, empirical findings from qualitative and quantitative data and the logical relationships between these. Theoretical proposition 1 was not supported by the quantitative findings (indicated in Fig. 4 by the broken arrow), but the qualitative findings supported this proposition. From these divergent empirical findings, the theoretical proposition could accordingly be changed and developed. Respite care seemed to provide relief from carers’ on-going care-responsibilities, but other types of support need to be added to provide relief from distress experienced (theoretical proposition 2).

3.2.4. Triangulation to produce theoretical propositions

Methodological triangulation has also been applied to illustrate how theoretical propositions can be produced by drawing on the findings from a Finnish study by Lukkarinen (2005) ( Fig. 5 ). The purpose of this longitudinal study was to describe, explain and understand the subjective health related quality of life (QoL) and life course of people with coronary artery disease (CAD). A longitudinal quantitative study was undertaken during the year post treatment and 19 individuals also attended thematic interviews one year after treatment. This study is one of the few studies that clearly defines theoretical underpinnings for both the selected methods and their purpose, namely “to obtain quantitatively abundant average information about the QoL of CAD patients and the changes in it as well as the patients’ individual, unique experiences of their respective life situations” ( Lukkarinen, 2005 :622).

Illustrating the use of triangulation ( Erzberger and Kelle, 2003 ) to develop theory from the study by Lukkarinen (2005) .

The results of the quantitative analysis showed that the male and female CAD patients in the youngest age group had the poorest QoL. While patients’ QoL improved in the dimensions of pain, energy and mobility it deteriorated on dimensions of social isolation, sleep and emotional reactions. From the viewpoint of methodological triangulation used in the study the aim of the quantitative approach was to observe changes in QoL at the group level and also explore correlations of background factors to QoL. The qualitative approach generated information concerning both QoL in the individuals’ life situation and life course and the individuals’ rehabilitation. Both the quantitative and the qualitative analysis showed the youngest CAD patients to have the poorest psychosocial QoL. The results obtained using qualitative methods explained the quantitative findings and offered new insight into the factors related to poor psychosocial QoL, which could be used to help develop theoretical propositions around these. Patients at risk of poorer QoL were those with an acute onset of illness at a young age that led to an unexpected termination of career, resulting in financial problems, and worries about family. This group also experienced lack of emotional support (especially the females with CAD) and were concerned for the illness that was not alleviated by treatment. The interviews and the method of phenomenological psychology therefore helped to gain insight into the participants’ situational experience of QoL and life course, not detectable by the use of a questionnaire.

Fig. 5 illustrates the theoretical propositions, empirical findings from qualitative and quantitative data and the relationships between these. The use of the mixed methods approach enabled a clearer understanding to emerge in relation to the lived experience of CAD patients and the factors that were related to poor QoL. This understanding allows new theoretical propositions about these issues to be developed and further explored, as depicted at the theoretical level.

4. Discussion

As the need for, and use of, mixed methods research continues to grow, the issue of quality within mixed methods studies is becoming increasingly important ( O’Cathain et al., 2008 , O’Cathain et al., 2007 ). Similarly, the need for guidance on the analysis and integration of qualitative and quantitative data is a prominant issue ( Bazeley, 2009 ). This paper firstly intended to review the types of analytic approaches (parallel, concurrent or sequential data analysis) that have been used in mixed methods studies within healthcare research. As identified in previous research ( O’Cathain et al., 2008 ), we found that the majority of studies included in our review employed parallel data analysis in which the different analyses are not compared or consolidated until the full analysis of both data sets have been completed. A trend to conduct separate analysis on quantitative and qualitative data is apparent in mixed methods healthcare studies, despite the fact that if the data were correlated, a more complete picture of a particular phenomenon may be produced ( Onwuegbuzie and Teddlie, 2003 ). If qualitative and quantitative data are not integrated during data collection or analysis, the findings may be integrated at the stage of interpretation and conclusion.

Although little pragmatic guidance exists within the wider literature, Erzberger and Kelle (2003) have published some practical advice, on the integration of mixed methods findings. For mixed methodologists, the ‘triangulation metaphor’ offers a framework to facilitate a description of the relationships between data sets and theoretical concepts and can also assist in the integration of qualitative and quantitative data ( Erzberger and Kelle, 2003 ). Yet despite the fact that the framework was published in 2003 within Tashakkori and Teddlie's (2003) seminal work, the Handbook for Mixed Methods in Social and Behavioural Research, our search revealed that it has received little application within the published body of work around mixed methods studies since its publication. This is surprising since mixed methodologists are acutely aware of the lack of guidance with regards to the pragmatics and practicalities of conducting mixed methods research ( Bryman, 2006 , Leech et al., 2010 ). Furthermore, there have been frequent calls to move the field of mixed methods away from the “should we or shouldn’t we” debate towards the practical application, analysis and integration of mixed methods and its’ findings and what we can learn from each other's work and advice. Consequently, we have a state of ambiguity and instability in the field of mixed methods in which mixed methodologists find themselves lacking appropriate sources or evidence to draw upon with which to facilitate the future design, conduct and interpretation of mixed methods studies. It is for these reasons that we, in this paper, also intended to identify and select studies that could be used as examples for the application of Erzberger and Kelle's (2003) triangulation metaphor.

When reviewing the studies it was clear that the majority of theoretical assumptions were implicit, rather than explicitly stated by authors. Wu and Volker (2009) previously acknowledged that while studies undoubtedly have a theoretical basis in their literature reviews and the nature of their research questions, they often fail to clearly articulate a particular theoretical framework. This is unfortunate as theory can help researchers to clarify their ideas and also help data collection, analysis and to improve the study's rigour ( Wu and Volker, 2009 ). When using triangulation as a methodological metaphor ( Erzberger and Kelle, 2003 ), researchers are encouraged to articulate their theoretical propositions and to validate their conclusions in relation to the chosen theories. Theory can also guide researchers when defining outcome measures . Should the findings not support the chosen theory, as shown in our examples on complementary and divergent results, researchers can modify or expand their theory accordingly and new theory may be developed ( Wu and Volker, 2009 ). It is therefore our belief that using triangulation as a methodological metaphor in mixed methods research can also benefit the design of mixed method studies.

Like other researchers ( O’Cathain et al., 2008 ), we have also found that most of the papers reviewed lacked clarity in whether the reported results primarily stemmed from qualitative or quantitative findings. Many of the papers were even less clear when discussing their results and the basis of their conclusions. The reporting of mixed methods studies is notoriously challenging, but clarity and transparency are, at the very least, crucial in such reports ( O’Cathain, 2009 ). Using triangulation as a methodological metaphor ( Erzberger and Kelle, 2003 ) may be one way of addressing this lack of clarity by explicitly showing the types of data that researchers have based their interpretations on. It may even help address some of the issues raised in the debate on the feasibility of integrating research methods and results stemming from different epistemological and ontological assumptions and paradigms ( Morgan, 2007 , Sale et al., 2002 ). In order to carry out methodological triangulation researchers also need to identify and observe the consistency and adequacy of the two methods, positivistic and phenomenological regarding the research questions, data collection, methods of analysis and conclusions.

While we used systematic principles in our search for mixed methods studies in healthcare research, we cannot claim to have included all such studies. In many cases, reports of mixed methods studies are subjected to ‘salami slicing’ by researchers and hence the conduct of, and findings from, individual approaches are addressed in separate papers. Since these papers are often not indexed as a ‘mixed method’ study, they are undoubtedly more difficult to identify. Furthermore, different terminologies are used to describe and index mixed methods studies within the electronic databases ( Halcomb and Andrew, 2009a ), making it challenging to be certain that all relevant studies were captured in this review. However, the studies included in this review should give a sufficient overview of the use of mixed analysis in healthcare research and most importantly, they enable us to make suggestions for the future design, conduct, interpretation and reporting of mixed methods studies. It is also important to emphasise that we have based our triangulation examples on the data published but have no further knowledge of the analysis and findings undertaken by the authors. The examples should thus be taken as examples and not alternative explanations or interpretations.

Mixed methods research within healthcare remains an emerging field and its use is subject to much debate. It is therefore particularly important that researchers clearly describe their use of the approach and the conclusions made to improve transparency and quality within mixed methods research. The use of triangulation as a methodological metaphor ( Erzberger and Kelle, 2003 ) can help researchers not only to present their theoretical propositions but also the origin of their results in an explicit way and to understand the links between theory, epistemology and methodology in relation to their topic area. Furthermore it has the potential to make valid inferences, challenge existing theoretical assumptions and to develop or create new ones.

Conflict of interest

None declared.

Ethical approval

Not required.

• Abadia C.E., Oviedo D.G. Bureaucratic itineraries in Colombia. A theoretical and methodological tool to assess managed-care health care systems. Social Science & Medicine. 2009; 68 (6):1153–1160. [ PubMed ] [ Google Scholar ]
• Bazeley P. Analysing mixed methods data. In: Andrew S., Halcomb E.J., editors. Mixed Methods Research for Nursing and the Health Sciences. Wiley-Blackwell; Chichester: 2009. pp. 84–118. [ Google Scholar ]
• Beatty P.W., Neri M.T., Bell K., DeJong G. Use of outcomes information in acute inpatient rehabilitation. American Journal of Physical Medicine & Rehabilitation. 2004; 83 (6):468–478. [ PubMed ] [ Google Scholar ]
• Bernardi L., Kleim S., von der Lippe H. Social influences on fertility: a comparative mixed methods study in Eastern and Western Germany. Journal of Mixed Methods Research. 2007; 1 (1):23–47. [ Google Scholar ]
• Brannen J. Prologue: mixed methods for novice researchers: reflections and themes. International Journal of Multiple Research Approaches. 2009; 3 (1):8–12. [ Google Scholar ]
• Bryman A. University Press; Oxford: 2004. Social Research Methods. [ Google Scholar ]
• Bryman A. Integrating quantitative and qualitative research: how is it done? Qualitative Research. 2006; 6 (1):97–113. [ Google Scholar ]
• Bryman A. Barriers to integrating quantitative and qualitative research. Journal of Mixed Methods Research. 2007; 1 (1):8–22. [ Google Scholar ]
• Bussing R., Koro-Ljungberg M.E., Gary F., Mason D.M., Garvan C.W. Exploring help-seeking for ADHD symptoms: a mixed-methods approach. Harvard Review of Psychiatry. 2005; 13 (2):85–101. [ PubMed ] [ Google Scholar ]
• Canales M.K., Rakowski W. Development of a culturally specific instrument for mammography screening: an example with American Indian women in Vermont. Journal of Nursing Measurement. 2006; 14 (2):99–115. [ PubMed ] [ Google Scholar ]
• Cheng G.Y. A study of clinical questions posed by hospital clinicians. Journal of the Medical Library Association. 2004; 92 (4):445–458. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Clarke P.N., Yaros P.S. Research blenders: commentary and response. Transitions to new methodologies in nursing sciences. Nursing Science Quarterly. 1988; 1 (4):147–151. [ PubMed ] [ Google Scholar ]
• Cochrane, 2009. Cochrane Handbook for Systematic Reviews of Interventions. http://www.cochrane-handbook.org/ . The Cochrane Collaboration.
• Creswell J.W. Sage; Thousand Oaks: 2003. Research Design: Qualitative, Quantitative and Mixed Methods Approaches. [ Google Scholar ]
• Creswell J.W., Plano Clark V.L. Sage; Thousand Oaks: 2007. Designing and Conducting Mixed Methods Research. [ Google Scholar ]
• Erzberger C., Kelle U. Making inferences in mixed methods: The rules of integration. In: Tashakkori A., Teddlie C., editors. Handbook of Mixed Methods in Social & Behavioural Research. Sage; Thousand Oaks: 2003. pp. 457–488. [ Google Scholar ]
• Foss C., Ellefsen B. The value of combining qualitative and quantitative approaches in nursing research by means of method triangulation. Journal of Advanced Nursing. 2002; 40 (2):242–248. [ PubMed ] [ Google Scholar ]
• Halcomb E.J., Andrew A. Managing mixed methods projects. In: Andrew S., Halcomb E.J., editors. Mixed Methods Research for Nursing and the Health Sciences. Wiley-Blackwell; Chichester: 2009. pp. 50–64. [ Google Scholar ]
• Halcomb E.J., Andrew A., Brannen J. Introduction to mixed methods research for nursing and the health sciences. In: Andrew S., Halcomb E.J., editors. Mixed Methods Research for Nursing and the Health Sciences. Wiley-Blackwell; Chichester: 2009. pp. 3–12. [ Google Scholar ]
• Halcomb E.J., Davidson P.M., Griffiths R., Daly J. Cardiovascular disease management: time to advance the practice nurse role? Australian Health Review. 2008; 32 (1):44–53. [ PubMed ] [ Google Scholar ]
• Jick T.D. Mixing qualitative and quantitative methods: triangulation in action. Administrative Science Quarterly. 1979; 24 :602–611. [ Google Scholar ]
• Johnson R.B., Onquegbuzie A.J. Mixed methods research: a paradigm whose time has come. Educational Researcher. 2004; 33 (7):14–26. [ Google Scholar ]
• Kartalova-O’Doherty Y., Tedstone Doherty D. Satisfied carers of persons with enduring mental illness: who and why? The International Journal of Social Psychiatry. 2009; 55 (3):257–271. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Kelle U. Sociological explanations between micro and macro and the integration of qualitative and quantitative methods. Forum: Qualitative Social Research. 2001; 2 (1):5. Art. [ Google Scholar ]
• Khan K.S. Centre for Review and Dissemination; York: 2001. Undertaking Systematic Reviews of Research on Effectiveness: CRD's Guidance for those Carrying Out or Commissioning Reviews. [ Google Scholar ]
• Kroll T., Neri M. Designs for mixed methods research. In: Andrew S., Halcomb E.J., editors. Mixed Methods Research for Nursing and the Health Sciences. Wiley-Blackwell; Chichester: 2009. pp. 31–49. [ Google Scholar ]
• Leech N.L., Dellinger A.M., Brannagan K.B., Tanaka H. Evaluating mixed research studies: a mixed methods approach. Journal of Mixed Methods Research. 2010; 4 (1):17–31. [ Google Scholar ]
• Lukkarinen H. Methodological triangulation showed the poorest quality of life in the youngest people following treatment of coronary artery disease: a longitudinal study. International Journal of Nursing Studies. 2005; 42 (6):619–627. [ PubMed ] [ Google Scholar ]
• Midtgaard J., Rorth M., Stelter R., Adamsen L. The group matters: an explorative study of group cohesion and quality of life in cancer patients participating in physical exercise intervention during treatment. European Journal of Cancer Care. 2006; 15 (1):25–33. [ PubMed ] [ Google Scholar ]
• Morgan D.L. Paradigms lost and pragmatism regained: methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research. 2007; 1 (1):48–76. [ Google Scholar ]
• O’Cathain A. Reporting mixed methods projects. In: Andrew S., Halcomb E.J., editors. Mixed Methods Research for Nursing and the Health Sciences. Wiley-Blackwell; Chichester: 2009. pp. 135–158. [ Google Scholar ]
• O’Cathain A., Murphy E., Nicholl J. Why, and how, mixed methods research is undertaken in health services research in England: a mixed methods study. BMC Health Services Research. 2007; 7 :85. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• O’Cathain A., Murphy E., Nicholl J. The quality of mixed methods studies in health services research. Journal of Health Services Research & Policy. 2008; 13 (2):92–98. [ PubMed ] [ Google Scholar ]
• Onwuegbuzie A., Teddlie C. A framework for analysing data in mixed methods research. In: Tashakkori A., Teddlie C., editors. Handbook of Mixed Methods in socIal & Behavioural Research. Sage; Thousands Oak: 2003. pp. 351–383. [ Google Scholar ]
• Onwuegbuzie A.J., Leech N.L. On becoming a pragmatic researcher: the importance of combining quantitative and qualitative methodologies. International Journal of Social Research Methodology. 2005; 8 (5):375–387. [ Google Scholar ]
• Sale J.E.M., Lohfeld L.H., Brazil K. Revisiting the quantitative-qualitative debate: implications for mixed methods research. Quality and Quantity. 2002; 36 :43–53. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Shipman C., Burt J., Ream E., Beynon T., Richardson A., Addington-Hall J. Improving district nurses’ confidence and knowledge in the principles and practice of palliative care. Journal of Advanced Nursing. 2008; 63 (5):494–505. [ PubMed ] [ Google Scholar ]
• Skilbeck J.K., Payne S.A., Ingleton M.C., Nolan M., Carey I., Hanson A. An exploration of family carers’ experience of respite services in one specialist palliative care unit. Palliative Medicine. 2005; 19 (8):610–618. [ PubMed ] [ Google Scholar ]
• Steckler A., McLeroy K.R., Goodman R.M., Bird S.T., McCormick L. Toward integrating qualitative and quantitative methods: an introduction. Health Education Quarterly. 1992; 19 (1):1–8. [ PubMed ] [ Google Scholar ]
• Tashakkori A., Creswell J.W. Editorial: the new era of mixed methods. Journal of Mixed Methods Research. 2007; 1 (1):3–7. [ Google Scholar ]
• Tashakkori A., Teddlie C. Sage; Thousands Oak: 2003. Handbook of Mixed Methods in Social & Behavioural Research. [ Google Scholar ]
• Wu H.L., Volker D.L. The use of theory in qualitative approaches to research: application in end-of-life studies. Journal of Advanced Nursing. 2009; 65 (12):2719–2732. [ PubMed ] [ Google Scholar ]