Logo for Pressbooks

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Research Guides

Multiple Case Studies

Nadia Alqahtani and Pengtong Qu

Description

The case study approach is popular across disciplines in education, anthropology, sociology, psychology, medicine, law, and political science (Creswell, 2013). It is both a research method and a strategy (Creswell, 2013; Yin, 2017). In this type of research design, a case can be an individual, an event, or an entity, as determined by the research questions. There are two variants of the case study: the single-case study and the multiple-case study. The former design can be used to study and understand an unusual case, a critical case, a longitudinal case, or a revelatory case. On the other hand, a multiple-case study includes two or more cases or replications across the cases to investigate the same phenomena (Lewis-Beck, Bryman & Liao, 2003; Yin, 2017). …a multiple-case study includes two or more cases or replications across the cases to investigate the same phenomena

The difference between the single- and multiple-case study is the research design; however, they are within the same methodological framework (Yin, 2017). Multiple cases are selected so that “individual case studies either (a) predict similar results (a literal replication) or (b) predict contrasting results but for anticipatable reasons (a theoretical replication)” (p. 55). When the purpose of the study is to compare and replicate the findings, the multiple-case study produces more compelling evidence so that the study is considered more robust than the single-case study (Yin, 2017).

To write a multiple-case study, a summary of individual cases should be reported, and researchers need to draw cross-case conclusions and form a cross-case report (Yin, 2017). With evidence from multiple cases, researchers may have generalizable findings and develop theories (Lewis-Beck, Bryman & Liao, 2003).

Creswell, J. W. (2013). Qualitative inquiry and research design: Choosing among five approaches (3rd ed.). Los Angeles, CA: Sage.

Lewis-Beck, M., Bryman, A. E., & Liao, T. F. (2003). The Sage encyclopedia of social science research methods . Los Angeles, CA: Sage.

Yin, R. K. (2017). Case study research and applications: Design and methods . Los Angeles, CA: Sage.

Key Research Books and Articles on Multiple Case Study Methodology

Yin discusses how to decide if a case study should be used in research. Novice researchers can learn about research design, data collection, and data analysis of different types of case studies, as well as writing a case study report.

Chapter 2 introduces four major types of research design in case studies: holistic single-case design, embedded single-case design, holistic multiple-case design, and embedded multiple-case design. Novice researchers will learn about the definitions and characteristics of different designs. This chapter also teaches researchers how to examine and discuss the reliability and validity of the designs.

Creswell, J. W., & Poth, C. N. (2017). Qualitative inquiry and research design: Choosing among five approaches . Los Angeles, CA: Sage.

This book compares five different qualitative research designs: narrative research, phenomenology, grounded theory, ethnography, and case study. It compares the characteristics, data collection, data analysis and representation, validity, and writing-up procedures among five inquiry approaches using texts with tables. For each approach, the author introduced the definition, features, types, and procedures and contextualized these components in a study, which was conducted through the same method. Each chapter ends with a list of relevant readings of each inquiry approach.

This book invites readers to compare these five qualitative methods and see the value of each approach. Readers can consider which approach would serve for their research contexts and questions, as well as how to design their research and conduct the data analysis based on their choice of research method.

Günes, E., & Bahçivan, E. (2016). A multiple case study of preservice science teachers’ TPACK: Embedded in a comprehensive belief system. International Journal of Environmental and Science Education, 11 (15), 8040-8054.

In this article, the researchers showed the importance of using technological opportunities in improving the education process and how they enhanced the students’ learning in science education. The study examined the connection between “Technological Pedagogical Content Knowledge” (TPACK) and belief system in a science teaching context. The researchers used the multiple-case study to explore the effect of TPACK on the preservice science teachers’ (PST) beliefs on their TPACK level. The participants were three teachers with the low, medium, and high level of TPACK confidence. Content analysis was utilized to analyze the data, which were collected by individual semi-structured interviews with the participants about their lesson plans. The study first discussed each case, then compared features and relations across cases. The researchers found that there was a positive relationship between PST’s TPACK confidence and TPACK level; when PST had higher TPACK confidence, the participant had a higher competent TPACK level and vice versa.

Recent Dissertations Using Multiple Case Study Methodology

Milholland, E. S. (2015). A multiple case study of instructors utilizing Classroom Response Systems (CRS) to achieve pedagogical goals . Retrieved from ProQuest Dissertations & Theses Global. (Order Number 3706380)

The researcher of this study critiques the use of Classroom Responses Systems by five instructors who employed this program five years ago in their classrooms. The researcher conducted the multiple-case study methodology and categorized themes. He interviewed each instructor with questions about their initial pedagogical goals, the changes in pedagogy during teaching, and the teaching techniques individuals used while practicing the CRS. The researcher used the multiple-case study with five instructors. He found that all instructors changed their goals during employing CRS; they decided to reduce the time of lecturing and to spend more time engaging students in interactive activities. This study also demonstrated that CRS was useful for the instructors to achieve multiple learning goals; all the instructors provided examples of the positive aspect of implementing CRS in their classrooms.

Li, C. L. (2010). The emergence of fairy tale literacy: A multiple case study on promoting critical literacy of children through a juxtaposed reading of classic fairy tales and their contemporary disruptive variants . Retrieved from ProQuest Dissertations & Theses Global. (Order Number 3572104)

To explore how children’s development of critical literacy can be impacted by their reactions to fairy tales, the author conducted a multiple-case study with 4 cases, in which each child was a unit of analysis. Two Chinese immigrant children (a boy and a girl) and two American children (a boy and a girl) at the second or third grade were recruited in the study. The data were collected through interviews, discussions on fairy tales, and drawing pictures. The analysis was conducted within both individual cases and cross cases. Across four cases, the researcher found that the young children’s’ knowledge of traditional fairy tales was built upon mass-media based adaptations. The children believed that the representations on mass-media were the original stories, even though fairy tales are included in the elementary school curriculum. The author also found that introducing classic versions of fairy tales increased children’s knowledge in the genre’s origin, which would benefit their understanding of the genre. She argued that introducing fairy tales can be the first step to promote children’s development of critical literacy.

Asher, K. C. (2014). Mediating occupational socialization and occupational individuation in teacher education: A multiple case study of five elementary pre-service student teachers . Retrieved from ProQuest Dissertations & Theses Global. (Order Number 3671989)

This study portrayed five pre-service teachers’ teaching experience in their student teaching phase and explored how pre-service teachers mediate their occupational socialization with occupational individuation. The study used the multiple-case study design and recruited five pre-service teachers from a Midwestern university as five cases. Qualitative data were collected through interviews, classroom observations, and field notes. The author implemented the case study analysis and found five strategies that the participants used to mediate occupational socialization with occupational individuation. These strategies were: 1) hindering from practicing their beliefs, 2) mimicking the styles of supervising teachers, 3) teaching in the ways in alignment with school’s existing practice, 4) enacting their own ideas, and 5) integrating and balancing occupational socialization and occupational individuation. The study also provided recommendations and implications to policymakers and educators in teacher education so that pre-service teachers can be better supported.

Multiple Case Studies Copyright © 2019 by Nadia Alqahtani and Pengtong Qu is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

  • Knowledge Base

Methodology

  • What Is a Case Study? | Definition, Examples & Methods

What Is a Case Study? | Definition, Examples & Methods

Published on May 8, 2019 by Shona McCombes . Revised on November 20, 2023.

A case study is a detailed study of a specific subject, such as a person, group, place, event, organization, or phenomenon. Case studies are commonly used in social, educational, clinical, and business research.

A case study research design usually involves qualitative methods , but quantitative methods are sometimes also used. Case studies are good for describing , comparing, evaluating and understanding different aspects of a research problem .

Table of contents

When to do a case study, step 1: select a case, step 2: build a theoretical framework, step 3: collect your data, step 4: describe and analyze the case, other interesting articles.

A case study is an appropriate research design when you want to gain concrete, contextual, in-depth knowledge about a specific real-world subject. It allows you to explore the key characteristics, meanings, and implications of the case.

Case studies are often a good choice in a thesis or dissertation . They keep your project focused and manageable when you don’t have the time or resources to do large-scale research.

You might use just one complex case study where you explore a single subject in depth, or conduct multiple case studies to compare and illuminate different aspects of your research problem.

Prevent plagiarism. Run a free check.

Once you have developed your problem statement and research questions , you should be ready to choose the specific case that you want to focus on. A good case study should have the potential to:

  • Provide new or unexpected insights into the subject
  • Challenge or complicate existing assumptions and theories
  • Propose practical courses of action to resolve a problem
  • Open up new directions for future research

TipIf your research is more practical in nature and aims to simultaneously investigate an issue as you solve it, consider conducting action research instead.

Unlike quantitative or experimental research , a strong case study does not require a random or representative sample. In fact, case studies often deliberately focus on unusual, neglected, or outlying cases which may shed new light on the research problem.

Example of an outlying case studyIn the 1960s the town of Roseto, Pennsylvania was discovered to have extremely low rates of heart disease compared to the US average. It became an important case study for understanding previously neglected causes of heart disease.

However, you can also choose a more common or representative case to exemplify a particular category, experience or phenomenon.

Example of a representative case studyIn the 1920s, two sociologists used Muncie, Indiana as a case study of a typical American city that supposedly exemplified the changing culture of the US at the time.

While case studies focus more on concrete details than general theories, they should usually have some connection with theory in the field. This way the case study is not just an isolated description, but is integrated into existing knowledge about the topic. It might aim to:

  • Exemplify a theory by showing how it explains the case under investigation
  • Expand on a theory by uncovering new concepts and ideas that need to be incorporated
  • Challenge a theory by exploring an outlier case that doesn’t fit with established assumptions

To ensure that your analysis of the case has a solid academic grounding, you should conduct a literature review of sources related to the topic and develop a theoretical framework . This means identifying key concepts and theories to guide your analysis and interpretation.

There are many different research methods you can use to collect data on your subject. Case studies tend to focus on qualitative data using methods such as interviews , observations , and analysis of primary and secondary sources (e.g., newspaper articles, photographs, official records). Sometimes a case study will also collect quantitative data.

Example of a mixed methods case studyFor a case study of a wind farm development in a rural area, you could collect quantitative data on employment rates and business revenue, collect qualitative data on local people’s perceptions and experiences, and analyze local and national media coverage of the development.

The aim is to gain as thorough an understanding as possible of the case and its context.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

  • Academic style
  • Vague sentences
  • Style consistency

See an example

multiple case study individual

In writing up the case study, you need to bring together all the relevant aspects to give as complete a picture as possible of the subject.

How you report your findings depends on the type of research you are doing. Some case studies are structured like a standard scientific paper or thesis , with separate sections or chapters for the methods , results and discussion .

Others are written in a more narrative style, aiming to explore the case from various angles and analyze its meanings and implications (for example, by using textual analysis or discourse analysis ).

In all cases, though, make sure to give contextual details about the case, connect it back to the literature and theory, and discuss how it fits into wider patterns or debates.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

  • Normal distribution
  • Degrees of freedom
  • Null hypothesis
  • Discourse analysis
  • Control groups
  • Mixed methods research
  • Non-probability sampling
  • Quantitative research
  • Ecological validity

Research bias

  • Rosenthal effect
  • Implicit bias
  • Cognitive bias
  • Selection bias
  • Negativity bias
  • Status quo bias

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, November 20). What Is a Case Study? | Definition, Examples & Methods. Scribbr. Retrieved February 22, 2024, from https://www.scribbr.com/methodology/case-study/

Is this article helpful?

Shona McCombes

Shona McCombes

Other students also liked, primary vs. secondary sources | difference & examples, what is a theoretical framework | guide to organizing, what is action research | definition & examples, what is your plagiarism score.

  • Privacy Policy
  • SignUp/Login

Research Method

Home » Case Study – Methods, Examples and Guide

Case Study – Methods, Examples and Guide

Table of Contents

Case Study Research

A case study is a research method that involves an in-depth examination and analysis of a particular phenomenon or case, such as an individual, organization, community, event, or situation.

It is a qualitative research approach that aims to provide a detailed and comprehensive understanding of the case being studied. Case studies typically involve multiple sources of data, including interviews, observations, documents, and artifacts, which are analyzed using various techniques, such as content analysis, thematic analysis, and grounded theory. The findings of a case study are often used to develop theories, inform policy or practice, or generate new research questions.

Types of Case Study

Types and Methods of Case Study are as follows:

Single-Case Study

A single-case study is an in-depth analysis of a single case. This type of case study is useful when the researcher wants to understand a specific phenomenon in detail.

For Example , A researcher might conduct a single-case study on a particular individual to understand their experiences with a particular health condition or a specific organization to explore their management practices. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as content analysis or thematic analysis. The findings of a single-case study are often used to generate new research questions, develop theories, or inform policy or practice.

Multiple-Case Study

A multiple-case study involves the analysis of several cases that are similar in nature. This type of case study is useful when the researcher wants to identify similarities and differences between the cases.

For Example, a researcher might conduct a multiple-case study on several companies to explore the factors that contribute to their success or failure. The researcher collects data from each case, compares and contrasts the findings, and uses various techniques to analyze the data, such as comparative analysis or pattern-matching. The findings of a multiple-case study can be used to develop theories, inform policy or practice, or generate new research questions.

Exploratory Case Study

An exploratory case study is used to explore a new or understudied phenomenon. This type of case study is useful when the researcher wants to generate hypotheses or theories about the phenomenon.

For Example, a researcher might conduct an exploratory case study on a new technology to understand its potential impact on society. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as grounded theory or content analysis. The findings of an exploratory case study can be used to generate new research questions, develop theories, or inform policy or practice.

Descriptive Case Study

A descriptive case study is used to describe a particular phenomenon in detail. This type of case study is useful when the researcher wants to provide a comprehensive account of the phenomenon.

For Example, a researcher might conduct a descriptive case study on a particular community to understand its social and economic characteristics. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as content analysis or thematic analysis. The findings of a descriptive case study can be used to inform policy or practice or generate new research questions.

Instrumental Case Study

An instrumental case study is used to understand a particular phenomenon that is instrumental in achieving a particular goal. This type of case study is useful when the researcher wants to understand the role of the phenomenon in achieving the goal.

For Example, a researcher might conduct an instrumental case study on a particular policy to understand its impact on achieving a particular goal, such as reducing poverty. The researcher collects data from multiple sources, such as interviews, observations, and documents, and uses various techniques to analyze the data, such as content analysis or thematic analysis. The findings of an instrumental case study can be used to inform policy or practice or generate new research questions.

Case Study Data Collection Methods

Here are some common data collection methods for case studies:

Interviews involve asking questions to individuals who have knowledge or experience relevant to the case study. Interviews can be structured (where the same questions are asked to all participants) or unstructured (where the interviewer follows up on the responses with further questions). Interviews can be conducted in person, over the phone, or through video conferencing.

Observations

Observations involve watching and recording the behavior and activities of individuals or groups relevant to the case study. Observations can be participant (where the researcher actively participates in the activities) or non-participant (where the researcher observes from a distance). Observations can be recorded using notes, audio or video recordings, or photographs.

Documents can be used as a source of information for case studies. Documents can include reports, memos, emails, letters, and other written materials related to the case study. Documents can be collected from the case study participants or from public sources.

Surveys involve asking a set of questions to a sample of individuals relevant to the case study. Surveys can be administered in person, over the phone, through mail or email, or online. Surveys can be used to gather information on attitudes, opinions, or behaviors related to the case study.

Artifacts are physical objects relevant to the case study. Artifacts can include tools, equipment, products, or other objects that provide insights into the case study phenomenon.

How to conduct Case Study Research

Conducting a case study research involves several steps that need to be followed to ensure the quality and rigor of the study. Here are the steps to conduct case study research:

  • Define the research questions: The first step in conducting a case study research is to define the research questions. The research questions should be specific, measurable, and relevant to the case study phenomenon under investigation.
  • Select the case: The next step is to select the case or cases to be studied. The case should be relevant to the research questions and should provide rich and diverse data that can be used to answer the research questions.
  • Collect data: Data can be collected using various methods, such as interviews, observations, documents, surveys, and artifacts. The data collection method should be selected based on the research questions and the nature of the case study phenomenon.
  • Analyze the data: The data collected from the case study should be analyzed using various techniques, such as content analysis, thematic analysis, or grounded theory. The analysis should be guided by the research questions and should aim to provide insights and conclusions relevant to the research questions.
  • Draw conclusions: The conclusions drawn from the case study should be based on the data analysis and should be relevant to the research questions. The conclusions should be supported by evidence and should be clearly stated.
  • Validate the findings: The findings of the case study should be validated by reviewing the data and the analysis with participants or other experts in the field. This helps to ensure the validity and reliability of the findings.
  • Write the report: The final step is to write the report of the case study research. The report should provide a clear description of the case study phenomenon, the research questions, the data collection methods, the data analysis, the findings, and the conclusions. The report should be written in a clear and concise manner and should follow the guidelines for academic writing.

Examples of Case Study

Here are some examples of case study research:

  • The Hawthorne Studies : Conducted between 1924 and 1932, the Hawthorne Studies were a series of case studies conducted by Elton Mayo and his colleagues to examine the impact of work environment on employee productivity. The studies were conducted at the Hawthorne Works plant of the Western Electric Company in Chicago and included interviews, observations, and experiments.
  • The Stanford Prison Experiment: Conducted in 1971, the Stanford Prison Experiment was a case study conducted by Philip Zimbardo to examine the psychological effects of power and authority. The study involved simulating a prison environment and assigning participants to the role of guards or prisoners. The study was controversial due to the ethical issues it raised.
  • The Challenger Disaster: The Challenger Disaster was a case study conducted to examine the causes of the Space Shuttle Challenger explosion in 1986. The study included interviews, observations, and analysis of data to identify the technical, organizational, and cultural factors that contributed to the disaster.
  • The Enron Scandal: The Enron Scandal was a case study conducted to examine the causes of the Enron Corporation’s bankruptcy in 2001. The study included interviews, analysis of financial data, and review of documents to identify the accounting practices, corporate culture, and ethical issues that led to the company’s downfall.
  • The Fukushima Nuclear Disaster : The Fukushima Nuclear Disaster was a case study conducted to examine the causes of the nuclear accident that occurred at the Fukushima Daiichi Nuclear Power Plant in Japan in 2011. The study included interviews, analysis of data, and review of documents to identify the technical, organizational, and cultural factors that contributed to the disaster.

Application of Case Study

Case studies have a wide range of applications across various fields and industries. Here are some examples:

Business and Management

Case studies are widely used in business and management to examine real-life situations and develop problem-solving skills. Case studies can help students and professionals to develop a deep understanding of business concepts, theories, and best practices.

Case studies are used in healthcare to examine patient care, treatment options, and outcomes. Case studies can help healthcare professionals to develop critical thinking skills, diagnose complex medical conditions, and develop effective treatment plans.

Case studies are used in education to examine teaching and learning practices. Case studies can help educators to develop effective teaching strategies, evaluate student progress, and identify areas for improvement.

Social Sciences

Case studies are widely used in social sciences to examine human behavior, social phenomena, and cultural practices. Case studies can help researchers to develop theories, test hypotheses, and gain insights into complex social issues.

Law and Ethics

Case studies are used in law and ethics to examine legal and ethical dilemmas. Case studies can help lawyers, policymakers, and ethical professionals to develop critical thinking skills, analyze complex cases, and make informed decisions.

Purpose of Case Study

The purpose of a case study is to provide a detailed analysis of a specific phenomenon, issue, or problem in its real-life context. A case study is a qualitative research method that involves the in-depth exploration and analysis of a particular case, which can be an individual, group, organization, event, or community.

The primary purpose of a case study is to generate a comprehensive and nuanced understanding of the case, including its history, context, and dynamics. Case studies can help researchers to identify and examine the underlying factors, processes, and mechanisms that contribute to the case and its outcomes. This can help to develop a more accurate and detailed understanding of the case, which can inform future research, practice, or policy.

Case studies can also serve other purposes, including:

  • Illustrating a theory or concept: Case studies can be used to illustrate and explain theoretical concepts and frameworks, providing concrete examples of how they can be applied in real-life situations.
  • Developing hypotheses: Case studies can help to generate hypotheses about the causal relationships between different factors and outcomes, which can be tested through further research.
  • Providing insight into complex issues: Case studies can provide insights into complex and multifaceted issues, which may be difficult to understand through other research methods.
  • Informing practice or policy: Case studies can be used to inform practice or policy by identifying best practices, lessons learned, or areas for improvement.

Advantages of Case Study Research

There are several advantages of case study research, including:

  • In-depth exploration: Case study research allows for a detailed exploration and analysis of a specific phenomenon, issue, or problem in its real-life context. This can provide a comprehensive understanding of the case and its dynamics, which may not be possible through other research methods.
  • Rich data: Case study research can generate rich and detailed data, including qualitative data such as interviews, observations, and documents. This can provide a nuanced understanding of the case and its complexity.
  • Holistic perspective: Case study research allows for a holistic perspective of the case, taking into account the various factors, processes, and mechanisms that contribute to the case and its outcomes. This can help to develop a more accurate and comprehensive understanding of the case.
  • Theory development: Case study research can help to develop and refine theories and concepts by providing empirical evidence and concrete examples of how they can be applied in real-life situations.
  • Practical application: Case study research can inform practice or policy by identifying best practices, lessons learned, or areas for improvement.
  • Contextualization: Case study research takes into account the specific context in which the case is situated, which can help to understand how the case is influenced by the social, cultural, and historical factors of its environment.

Limitations of Case Study Research

There are several limitations of case study research, including:

  • Limited generalizability : Case studies are typically focused on a single case or a small number of cases, which limits the generalizability of the findings. The unique characteristics of the case may not be applicable to other contexts or populations, which may limit the external validity of the research.
  • Biased sampling: Case studies may rely on purposive or convenience sampling, which can introduce bias into the sample selection process. This may limit the representativeness of the sample and the generalizability of the findings.
  • Subjectivity: Case studies rely on the interpretation of the researcher, which can introduce subjectivity into the analysis. The researcher’s own biases, assumptions, and perspectives may influence the findings, which may limit the objectivity of the research.
  • Limited control: Case studies are typically conducted in naturalistic settings, which limits the control that the researcher has over the environment and the variables being studied. This may limit the ability to establish causal relationships between variables.
  • Time-consuming: Case studies can be time-consuming to conduct, as they typically involve a detailed exploration and analysis of a specific case. This may limit the feasibility of conducting multiple case studies or conducting case studies in a timely manner.
  • Resource-intensive: Case studies may require significant resources, including time, funding, and expertise. This may limit the ability of researchers to conduct case studies in resource-constrained settings.

About the author

' src=

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Qualitative Research

Qualitative Research – Methods, Analysis Types...

Descriptive Research Design

Descriptive Research Design – Types, Methods and...

Qualitative Research Methods

Qualitative Research Methods

Basic Research

Basic Research – Types, Methods and Examples

Exploratory Research

Exploratory Research – Types, Methods and...

One-to-One Interview in Research

One-to-One Interview – Methods and Guide

  • Bipolar Disorder
  • Therapy Center
  • When To See a Therapist
  • Types of Therapy
  • Best Online Therapy
  • Best Couples Therapy
  • Best Family Therapy
  • Managing Stress
  • Sleep and Dreaming
  • Understanding Emotions
  • Self-Improvement
  • Healthy Relationships
  • Student Resources
  • Personality Types
  • Verywell Mind Insights
  • 2023 Verywell Mind 25
  • Mental Health in the Classroom
  • Editorial Process
  • Meet Our Review Board
  • Crisis Support

What Is a Case Study?

Weighing the pros and cons of this method of research

Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

multiple case study individual

Cara Lustik is a fact-checker and copywriter.

multiple case study individual

Verywell / Colleen Tighe

  • Pros and Cons

What Types of Case Studies Are Out There?

Where do you find data for a case study, how do i write a psychology case study.

A case study is an in-depth study of one person, group, or event. In a case study, nearly every aspect of the subject's life and history is analyzed to seek patterns and causes of behavior. Case studies can be used in many different fields, including psychology, medicine, education, anthropology, political science, and social work.

The point of a case study is to learn as much as possible about an individual or group so that the information can be generalized to many others. Unfortunately, case studies tend to be highly subjective, and it is sometimes difficult to generalize results to a larger population.

While case studies focus on a single individual or group, they follow a format similar to other types of psychology writing. If you are writing a case study, we got you—here are some rules of APA format to reference.  

At a Glance

A case study, or an in-depth study of a person, group, or event, can be a useful research tool when used wisely. In many cases, case studies are best used in situations where it would be difficult or impossible for you to conduct an experiment. They are helpful for looking at unique situations and allow researchers to gather a lot of˜ information about a specific individual or group of people. However, it's important to be cautious of any bias we draw from them as they are highly subjective.

What Are the Benefits and Limitations of Case Studies?

A case study can have its strengths and weaknesses. Researchers must consider these pros and cons before deciding if this type of study is appropriate for their needs.

One of the greatest advantages of a case study is that it allows researchers to investigate things that are often difficult or impossible to replicate in a lab. Some other benefits of a case study:

  • Allows researchers to capture information on the 'how,' 'what,' and 'why,' of something that's implemented
  • Gives researchers the chance to collect information on why one strategy might be chosen over another
  • Permits researchers to develop hypotheses that can be explored in experimental research

On the other hand, a case study can have some drawbacks:

  • It cannot necessarily be generalized to the larger population
  • Cannot demonstrate cause and effect
  • It may not be scientifically rigorous
  • It can lead to bias

Researchers may choose to perform a case study if they want to explore a unique or recently discovered phenomenon. Through their insights, researchers develop additional ideas and study questions that might be explored in future studies.

It's important to remember that the insights from case studies cannot be used to determine cause-and-effect relationships between variables. However, case studies may be used to develop hypotheses that can then be addressed in experimental research.

Case Study Examples

There have been a number of notable case studies in the history of psychology. Much of  Freud's work and theories were developed through individual case studies. Some great examples of case studies in psychology include:

  • Anna O : Anna O. was a pseudonym of a woman named Bertha Pappenheim, a patient of a physician named Josef Breuer. While she was never a patient of Freud's, Freud and Breuer discussed her case extensively. The woman was experiencing symptoms of a condition that was then known as hysteria and found that talking about her problems helped relieve her symptoms. Her case played an important part in the development of talk therapy as an approach to mental health treatment.
  • Phineas Gage : Phineas Gage was a railroad employee who experienced a terrible accident in which an explosion sent a metal rod through his skull, damaging important portions of his brain. Gage recovered from his accident but was left with serious changes in both personality and behavior.
  • Genie : Genie was a young girl subjected to horrific abuse and isolation. The case study of Genie allowed researchers to study whether language learning was possible, even after missing critical periods for language development. Her case also served as an example of how scientific research may interfere with treatment and lead to further abuse of vulnerable individuals.

Such cases demonstrate how case research can be used to study things that researchers could not replicate in experimental settings. In Genie's case, her horrific abuse denied her the opportunity to learn a language at critical points in her development.

This is clearly not something researchers could ethically replicate, but conducting a case study on Genie allowed researchers to study phenomena that are otherwise impossible to reproduce.

There are a few different types of case studies that psychologists and other researchers might use:

  • Collective case studies : These involve studying a group of individuals. Researchers might study a group of people in a certain setting or look at an entire community. For example, psychologists might explore how access to resources in a community has affected the collective mental well-being of those who live there.
  • Descriptive case studies : These involve starting with a descriptive theory. The subjects are then observed, and the information gathered is compared to the pre-existing theory.
  • Explanatory case studies : These   are often used to do causal investigations. In other words, researchers are interested in looking at factors that may have caused certain things to occur.
  • Exploratory case studies : These are sometimes used as a prelude to further, more in-depth research. This allows researchers to gather more information before developing their research questions and hypotheses .
  • Instrumental case studies : These occur when the individual or group allows researchers to understand more than what is initially obvious to observers.
  • Intrinsic case studies : This type of case study is when the researcher has a personal interest in the case. Jean Piaget's observations of his own children are good examples of how an intrinsic case study can contribute to the development of a psychological theory.

The three main case study types often used are intrinsic, instrumental, and collective. Intrinsic case studies are useful for learning about unique cases. Instrumental case studies help look at an individual to learn more about a broader issue. A collective case study can be useful for looking at several cases simultaneously.

The type of case study that psychology researchers use depends on the unique characteristics of the situation and the case itself.

There are a number of different sources and methods that researchers can use to gather information about an individual or group. Six major sources that have been identified by researchers are:

  • Archival records : Census records, survey records, and name lists are examples of archival records.
  • Direct observation : This strategy involves observing the subject, often in a natural setting . While an individual observer is sometimes used, it is more common to utilize a group of observers.
  • Documents : Letters, newspaper articles, administrative records, etc., are the types of documents often used as sources.
  • Interviews : Interviews are one of the most important methods for gathering information in case studies. An interview can involve structured survey questions or more open-ended questions.
  • Participant observation : When the researcher serves as a participant in events and observes the actions and outcomes, it is called participant observation.
  • Physical artifacts : Tools, objects, instruments, and other artifacts are often observed during a direct observation of the subject.

If you have been directed to write a case study for a psychology course, be sure to check with your instructor for any specific guidelines you need to follow. If you are writing your case study for a professional publication, check with the publisher for their specific guidelines for submitting a case study.

Here is a general outline of what should be included in a case study.

Section 1: A Case History

This section will have the following structure and content:

Background information : The first section of your paper will present your client's background. Include factors such as age, gender, work, health status, family mental health history, family and social relationships, drug and alcohol history, life difficulties, goals, and coping skills and weaknesses.

Description of the presenting problem : In the next section of your case study, you will describe the problem or symptoms that the client presented with.

Describe any physical, emotional, or sensory symptoms reported by the client. Thoughts, feelings, and perceptions related to the symptoms should also be noted. Any screening or diagnostic assessments that are used should also be described in detail and all scores reported.

Your diagnosis : Provide your diagnosis and give the appropriate Diagnostic and Statistical Manual code. Explain how you reached your diagnosis, how the client's symptoms fit the diagnostic criteria for the disorder(s), or any possible difficulties in reaching a diagnosis.

Section 2: Treatment Plan

This portion of the paper will address the chosen treatment for the condition. This might also include the theoretical basis for the chosen treatment or any other evidence that might exist to support why this approach was chosen.

  • Cognitive behavioral approach : Explain how a cognitive behavioral therapist would approach treatment. Offer background information on cognitive behavioral therapy and describe the treatment sessions, client response, and outcome of this type of treatment. Make note of any difficulties or successes encountered by your client during treatment.
  • Humanistic approach : Describe a humanistic approach that could be used to treat your client, such as client-centered therapy . Provide information on the type of treatment you chose, the client's reaction to the treatment, and the end result of this approach. Explain why the treatment was successful or unsuccessful.
  • Psychoanalytic approach : Describe how a psychoanalytic therapist would view the client's problem. Provide some background on the psychoanalytic approach and cite relevant references. Explain how psychoanalytic therapy would be used to treat the client, how the client would respond to therapy, and the effectiveness of this treatment approach.
  • Pharmacological approach : If treatment primarily involves the use of medications, explain which medications were used and why. Provide background on the effectiveness of these medications and how monotherapy may compare with an approach that combines medications with therapy or other treatments.

This section of a case study should also include information about the treatment goals, process, and outcomes.

When you are writing a case study, you should also include a section where you discuss the case study itself, including the strengths and limitiations of the study. You should note how the findings of your case study might support previous research. 

In your discussion section, you should also describe some of the implications of your case study. What ideas or findings might require further exploration? How might researchers go about exploring some of these questions in additional studies?

Need More Tips?

Here are a few additional pointers to keep in mind when formatting your case study:

  • Never refer to the subject of your case study as "the client." Instead, use their name or a pseudonym.
  • Read examples of case studies to gain an idea about the style and format.
  • Remember to use APA format when citing references .

Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach .  BMC Med Res Methodol . 2011;11:100.

Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach . BMC Med Res Methodol . 2011 Jun 27;11:100. doi:10.1186/1471-2288-11-100

Gagnon, Yves-Chantal.  The Case Study as Research Method: A Practical Handbook . Canada, Chicago Review Press Incorporated DBA Independent Pub Group, 2010.

Yin, Robert K. Case Study Research and Applications: Design and Methods . United States, SAGE Publications, 2017.

By Kendra Cherry, MSEd Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

  • AI Content Shield
  • AI KW Research
  • AI Assistant
  • SEO Optimizer
  • AI KW Clustering
  • Customer reviews
  • The NLO Revolution
  • Press Center
  • Help Center
  • Content Resources
  • Facebook Group

How to Write a Multiple Case Study Effectively

Table of Contents

Have you ever been assigned to write a multiple case study but don’t know where to begin? Are you intimidated by the complexity and challenge it brings? Don’t worry! This article will help you learn how to write a multiple case study effectively that will make an impactful impression. So, let’s begin by defining a multiple case study.

What Is a Multiple Case Study?

A multiple case study is a research method examining several different entities. It helps researchers gain an understanding of the entities’ individual characteristics and disclose any shared patterns or insights. This type of investigation often uses both qualitative and quantitative data. These are usually collected from interviews, surveys, field observations, archival records, and other sources. This is done to analyze the relationships between each entity and its environment. The results can provide valuable insights for policymakers and decision-makers.

Why Is a Multiple Case Study Important?

A multiple case study is invaluable in providing a comprehensive view of a particular issue or phenomenon. Analyzing a range of cases allows for comparisons and contrasts to be drawn. And this can help identify broader trends, implications, and causes that might otherwise remain undetected. This method is particularly useful in developing theories and testing hypotheses. This is because the range of data collected provides more robust evidence than what could be achieved from one single case alone.

A person writing on a notebook with a laptop next to them

How to Write a Multiple Case Study

Below are the key steps on how to write a multiple case study :

1. Brainstorm Potential Case Studies

Before beginning your multiple case study, you should brainstorm potential cases suitable for the research project. Consider both theoretical and practical implications when deciding which cases are most appropriate. Think about how these cases can best illustrate the issue or question at hand. Make sure to consider all relevant information before making any decisions.

2. Conduct Background Research on Each Case

After selecting the individual cases for your multiple case study, the next step is to do background research for each case. Conducting extensive background research on each case will help you better understand the context of the study. This research will allow you to form an educated opinion and provide insight into the problems and challenges that each case may present.

3. Establish a Research Methodology

A successful multiple-case study requires a sound research methodology. This includes deciding on the methods of data collection and analysis and setting objectives. It also involves developing criteria for evaluating the results and determining what kind of data needs to be collected from each case. All of this must be done carefully, considering the purpose of the study and its outcomes.

4. Collect Data

Once a research method has been established, it is time to collect data from each case included in the study. Depending on the nature of the research project, this could involve interviewing participants, gathering statistics, or observing behaviors in certain settings. It is crucial to ensure that all data collected is accurate and reliable.

5. Analyze & Interpret Data

After the data has been collected, it must be analyzed to draw meaningful conclusions from it. This process involves examining patterns and trends within the data, identifying relationships between variables, and looking for commonalities among different cases. These findings must then be interpreted in light of the initial questions posed by the study.

6. Write the Report

After completing the analysis and interpretation of the data, it is finally time to write up the results of the multiple case study. This should include a summary of the key findings and an explanation of why these findings are significant. In addition, the limitations of the study should be acknowledged, along with recommendations for future research in this area.

Writing a multiple case study requires careful planning and execution. But the process becomes easier when you know the proper steps to conduct and create a multiple case study. It requires you to focus on the design of the study, including the sample chosen and the research methodology established. Conducting background research on each case and collecting data are also crucial steps in the process. To guide you through the process, this article outlines the key steps to help you easily write a well-structured multiple-case study .

How to Write a Multiple Case Study Effectively

Abir Ghenaiet

Abir is a data analyst and researcher. Among her interests are artificial intelligence, machine learning, and natural language processing. As a humanitarian and educator, she actively supports women in tech and promotes diversity.

Explore All Write A Case Study Articles

How to write a leadership case study (sample) .

Writing a case study isn’t as straightforward as writing essays. But it has proven to be an effective way of…

  • Write A Case Study

Top 5 Online Expert Case Study Writing Services 

It’s a few hours to your deadline — and your case study college assignment is still a mystery to you.…

Examples Of Business Case Study In Research

A business case study can prevent an imminent mistake in business. How? It’s an effective teaching technique that teaches students…

Have you ever been assigned to write a multiple case study but don’t know where to begin? Are you intimidated…

How to Write a Case Study Presentation: 6 Key Steps

Case studies are an essential element of the business world. Understanding how to write a case study presentation will give…

How to Write a Case Study for Your Portfolio

Are you ready to showcase your design skills and move your career to the next level? Crafting a compelling case…

Support and process in individual placement and support: a multiple case study

Affiliation.

  • 1 Division of Occupational Therapy and Gerontology, Department of Health Sciences, Lund University, Lund, Sweden. [email protected]
  • PMID: 22927583
  • DOI: 10.3233/WOR-2012-1360

Objective: This multiple case study investigated support and process in the Individual Placement and Support (IPS) approach from individual client, longitudinal, and Person-Environment-Occupation (PEO) model perspectives.

Participants: Five IPS-participants, or cases, with severe mental illness (SMI) who worked a minimum of 4 hours a week entered the study.

Methods: A multiple data collection method was used over a period of 12 months and included IPS-vocational profiles and plans as well as various instruments and questionnaires concerning socio-demographics, work performance, limitations, and accommodations. Both within- and across-case analyses were performed.

Results: The IPS-process concerned job search support, job-matches (PEO-match), and adjustment of the PEO-match by providing accommodations by on- and off-worksite support. All participants had limitations concerning social interactions and handling symptoms/tolerating stress. Several accommodations were made for the same limitations, mostly directed towards the social environment. Prior work experience, disclosure, and not being in an acute phase of illness seemed important to the support provided.

Conclusions: This study has visualised the support and process in IPS and provided a theoretical framework, the PEO-model, to detect limitations and provide IPS-support. The organization of IPS-support and methods of providing it to individuals may be important for job tenure and employment success.

Publication types

  • Randomized Controlled Trial
  • Research Support, Non-U.S. Gov't
  • Employment, Supported / psychology*
  • Longitudinal Studies
  • Mental Disorders / diagnosis
  • Mental Disorders / rehabilitation*
  • Occupational Health Services / standards
  • Organizational Case Studies
  • Professional Competence
  • Rehabilitation, Vocational / methods*
  • Self Disclosure
  • Social Class
  • Social Environment
  • Surveys and Questionnaires
  • Time Factors
  • Work Capacity Evaluation*

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • HHS Author Manuscripts

Logo of nihpa

Integrative analysis of multiple case-control studies

1 Division of Cancer Epidemiology and Genetics, National Cancer Institute, Bethesda, Maryland, USA

William Wheeler

2 Information Management Services, Silver Spring, Maryland, USA

3 National Institute of Allergy and Infectious Diseases, National Institute of Health, Bethesda, Maryland, USA

Associated Data

The data that support the findings in this paper are openly available in the database of Genotypes and Phenotypes (dbGaP) at https://www.ncbi.nlm.nih.gov/gap/ , with dbGaP Study Accession # phs000206.v5.p3.

It is often challenging to share detailed individual-level data among studies due to various informatics and privacy constraints. However, it is relatively easy to pool together aggregated summary level data, such as the ones required for standard meta-analyses. Focusing on data generated from case-control studies, we present a flexible inference procedure that integrates individual-level data collected from an “internal” study with summary data borrowed from “external” studies. This procedure is built on a retrospective empirical likelihood framework to account for the sampling bias in case-control studies. It can incorporate summary statistics extracted from various working models adopted by multiple independent or overlapping external studies. It also allows for external studies to be conducted in a population that is different from the internal study population. We show both theoretically and numerically its efficiency advantage over several competing alternatives.

1 |. INTRODUCTION

In the era of big data, collaborative multicenter studies are often carried out to study a disease outcome, with detailed individual-level data being collected by participating centers. If individual-level data from all studies is available, the most efficient way to draw inference is to conduct a pooled analysis by applying a unified statistical model to all data. However, sharing of individual-level data can be challenging due to various informatic and privacy constraints. Also, meta-analysis of summary data (i.e., estimated coefficients) generated from participating studies can be challenging when summary data are derived from different working models (e.g., varying sets of covariates, or inconsistent covariate definitions).

We consider a setting where researchers have collected individual-level data in their own study (the internal study), and in the meantime can acquire summary data from published literature or other studies (external studies). Since the case-control sampling design is most commonly used for studying a binary disease outcome ( Breslow and Day, 1980 ), we focus on integrating data from case-control studies. The goal is to develop a flexible statistical inference framework that can effectively synthesize all information from individual-level and aggregated summary data.

A number of procedures based on the empirical likelihood have been proposed to achieve the goal of integrative analysis ( Chen and Sitter, 1999 ; Qin, 2000 ; Chaudhuri et al., 2008 ; Chatterjee et al., 2016 ; Han and Lawless, 2019 ; Zhang et al., 2020 ). The summary data can be quite general, as long as they satisfy a set of constraint equations defined by certain population moment conditions. For example, the summary data can be the population mean, disease prevalence in a given strata, or estimates of coefficients in a working regression model chosen by an external study ( Chatterjee et al., 2016 ). Those procedures obtain their estimates by maximizing the empirical likelihood of observed individual-level data, under moment condition constraints imposed by summary information. Other procedures based on the generalized method of moment ( Imbens and Lancaster, 1994 ; Kundu et al., 2019 ; Huang and Qin, 2020 ) and Bayesian approaches ( Cheng et al., 2018 , 2019 ) have also been proposed.

Most existing procedures are built on the prospective likelihood approach, which focuses on modeling the probability of the disease outcome given covariates under the assumption that both individual-level and summary data come from prospective studies. They cannot be directly applied to data from case-control studies without further investigation. Qin et al. (2015) developed a procedure to improve the efficiency of case-control studies by utilizing knowledge on stratum-specific disease prevalence. Chatterjee et al. (2016) incorporated more general summary information into the analysis of an internal case-control study but assumed that summary data were derived from prospective studies. Both Qin et al. (2015) and Chatterjee et al. (2016) treated summary data as known parameters without any uncertainty. However, as shown by Zhang et al. (2020) , this strategy is not optimal for integrating summary data with unignorable variability.

We present a retrospective likelihood approach for the integrative analysis of data from multiple case-control studies. Following Zhang et al. (2020) , we treat both individual-level and summary data as observed random variables and derive their joint likelihood function. In order to account for the sampling bias in the case-control study design, the individual-level data are modeled with a retrospective empirical likelihood, which specifies the distribution of covariates given the disease status. Moment conditions satisfied by the summary data are used as constraints on the space of the parameters under investigation. As those constraints narrow the search region for the unknown parameters, they can help to reduce the uncertainty of parameter estimates. We show in theory and through simulation studies that estimates derived from the joint likelihood subject to those constraints are more efficient than existing approaches. A real example is used to illustrate the application of the proposed procedure.

2 |. METHOD

2.1 |. setup and notations.

We assume that we have a case-control study (called the internal study) of a binary disease outcome D and a set of covariates X . The study consists of n 1 subjects, with n 1,0 controls ( D = 0) and n 1,1 cases ( D = 1), n 1 = n 1,0 + n 1,1 . We represent the individual-level data for this internal study as ( X i , D i ), i = 1, … , n 1 , with the first n 1,0 subjects being controls, and the remaining n 1,1 subjects being cases. We further assume that the following logistic regression model is correctly specified as the underlying risk model:

where H ( X ; θ 2 ) is a given function, such as, H ( X ; θ 2 ) = X T θ 2 , with θ 2 being the set of parameters of interest. By Bayes’ formula, model ( 1 ) specifies the following connection between covariate distributions in cases and in controls ( Qin and Zhang, 1997 ),

where Δ ( X ; θ ) = exp { θ 1 + H ( X ; θ 2 ) } with θ = ( θ 1 , θ 2 t ) t , and θ 1 = θ 1 ∗ − log it { P ( D = 1 ) } . Since we consider data from case-control studies, we here after adopt this retrospective form ( 2 ) to represent the underlying model.

To draw inference on θ 2 based on the internal study, the standard prospective likelihood procedure can be used ( Prentice and Pyke, 1979 ). Here we briefly review the equivalent retrospective empirical likelihood approach, as our proposed procedure is built on this framework. Denote P = { p i ≜ P ( X i | D = 0 ) , i = 1 , … , n 1 } as the empirical version of P ( X | D = 0) supported on samples from the internal study. Following Qin and Zhang (1997) , θ can inferred by maximizing the following empirical log-likelihood function:

subject to constraint equations ∑ i = 1 n 1 p i = 1 , and ∑ i = 1 n 1 p i { Δ ( X i ; θ ) − 1 } = 0 . After profiling out p i , the empirical likelihood estimate of θ is the stationary point of the profile log-likelihood,

with ρ 1 = n 1 , 1 / n 1 , 0 . It is evident that the estimate of θ 2 (not θ 1 ) based on this function is exactly the same as the one based on the prospective likelihood specified by ( 1 ). Furthermore, similar to the result of Prentice and Pyke (1979) , Qin and Zhang (1997) showed that the asymptotic distribution of the empirical likelihood estimate is the same as the one based on the prospective likelihood if H ( X ; θ 2 ) = X t θ 2 .

Our goal is to estimate θ 2 by integrating individual-level data from the internal study and summary data (i.e., coefficient estimates) extracted from other case-control studies (called the external studies). We first present the method for incorporating summary data from one external study that is conducted within the same source population as the internal study. Later, we will expand the procedure to more complicated scenarios where multiple external studies are conducted in the same or a different source population.

For an external study consisting of n 2 subjects, with n 2,0 controls and n 2,1 cases, we represent its (unobserved) individual-level data as ( X i , D i ), i = n 1 + 1, … , n 1 + n 2 , with the first n 2,0 subjects being controls, and the remainder being cases. We assume that the external study was analyzed with a working model logit { P ( D = 1 | X ) } = α 1 ∗ + M ( X ; α 2 , β ) , with M ( X ; α 2 , β ) being a chosen function. This working model might be different from ( 1 ). Equivalently, we can represent this working model as

where W ( X | D = 1) and W ( X | D = 0) represent distributions of X in cases and in controls, with their connection being misspecified. Here we separate all unknown parameters into two parts, with β being the set of parameters whose estimates are presented as summary data for the integrative analysis, and α = ( α 1 , α 2 t ) t being the set of parameters whose estimates are not given. We allow the existence of α (called nuisance parameters) to accommodate the situations where not all estimates from the working model are available. The procedure also needs to know the definition of the working model ( 4 ) in order to use the summary data properly. Here we provide two examples to illustrate setups for some applications.

Example 1. Suppose X = ( X 1 , X 2 , X 3 ) t , and the underlying model assumed by the internal study is P ( X | D = 1 ) = P ( X | D = 0 ) exp ( θ 1 + θ 21 X 1 + θ 22 X 2 + θ 23 X 3 ) . The external study can choose a reduced model nested within the assumed model, such as W ( X | D = 1) = W ( X | D = 0) exp( α 1 + α 2 X 1 + βX 2 ), or a nonnested working model as W ( X | D = 1) = W ( X | D = 0) exp{ α 1 + α 2 X 1 + β log( X 2 )}. If only the estimate of β is provided as summary data, then ( α 1 , α 2 ) are considered as nuisance parameters. Note that these two working models are misspecified due to the noncollapsibility property of the logistic regression model.

Example 2. The true model is given by P ( X | D = 1 ) = P ( X | D = 0 ) exp { θ 1 + θ 2 ε ( X ) } , with ε ( X ) a known function of X = ( X 1 , … , X m ) t . The external study fits several marginal models, W k ( X | D = 1 ) = W k ( X | D = 0 ) exp { α k + β k X k } , k = 1, …, m . The summary data consist of estimates of β k , k = 1, … , m . This is an example of using summary data from multiple working models fitted with the same external study, a similar setting considered in the real example.

2.2 |. Asymptotic distribution of summary data

If summary data come from a misspecified working model, its variance–covariance matrix estimated by standard statistical packages is not correct, even the robust sandwich estimate derived from the prospective logistic model is not valid. Here we present the proper asymptotic distribution of the summary data.

Based on ( 4 ), the quasi-log-likelihood function of the external study can be expressed as

where ρ 2 = n 2,1 / n 2,0 , δ ( X ; α , β ) ≜ exp { α 1 + M ( X ; α 2 , β ) } . Estimates ( α ~ , β ~ ) of ( α, β ) are obtained from the estimating equation:

Let ϕ 0 ( X ; α , β ) = − ρ 2 δ ( X ; α , β ) 1 + ρ 2 δ ( X ; α , β ) ∂ log δ ( X ; α , β ) ∂ ( α , β ) , and ϕ 1 ( X ; α , β ) = 1 1 + ρ 2 δ ( X ; α , β ) ∂ log δ ( X ; α , β ) ∂ ( α , β ) . With the reparameterization of the intercept term, ℓ 2 ( α, β ) is equivalent to the prospective log-likelihood formation. Thus, β ~ is same as the one obtained by the standard prospective model.

Based on the estimating equation theory ( White, 1982 ), we know ( α ~ t , β ~ t ) t is a consistent estimate of ( α ∗ t , β ∗ t ) t , which is the solution of the following stochastic constraint equation:

where E 0 is the expectation over P ( X | D = 0), the true conditional distribution in controls, and E 1 is the expectation over P ( X | D = 1 ) . Here we assume n 2,1 / n 2,0 = ρ 2 as n 2 → ∞. Let μ ∗ = ( θ ∗ t , α ∗ t , β ∗ t ) t be the true value for μ = ( θ t , α t , β t ) t . Based on ( 2 ), we know μ * satisfies the following constraint equation:

with g ( X ; μ ) = ϕ 0 ( X ; α , β ) + ρ 2 Δ ( X ; θ ) ϕ 1 ( X ; α , β ) .

The asymptotic distribution of ( α ~ t , β ~ t ) t is given by

Let Σ 0 be the submatrix of A −1 B ( A −1 ) t corresponding to β . We know Cov ( β ~ ) = n 2 − 1 Σ 0 . Here we choose to represent A and B in terms of the expectation defined by P ( X | D = 0), as we can obtain the estimate of P ( X | D = 0) within the empirical likelihood framework described below.

Because of the specific form of B , following Carroll et al. (1995) , it can be seen that the asymptotic covariance matrix of β ~ is not theoretically equivalent to the one given by the corresponding prospective formula derived under the assumption that the external data are collected from a prospective study.

2.3 |. The integrative procedure

Here we extend the framework given by Zhang et al. (2020) to combine individual-level data with summary data β ~ . The main difference is that all considered data come from case-control studies. Rather than using the empirical distribution of X observed in the source population, we build our likelihood function using the empirical distribution of X observed among controls. We take a joint likelihood approach for the inference of μ = ( θ t , α t , β t ) t by treating both individual-level and summary data as observed random variables. The log-likelihood for the internal case-control study is given by ( 3 ). For the summary data β ~ , because of ( 6 ), its log-likelihood function can be written as − n 2 / 2 ( β − β ~ ) t Σ 0 − 1 ( β − β ~ ) . Since Σ 0 is unknown, we propose to estimate μ by solving the following optimization problem over ( P, μ ):

with p i ≥ 0, and V being any given positive definite matrix. The last constraint equation in ( 7 ) is due to ( 5 ). A simple choice for V is the identity matrix. We will show that Σ 0 is the most optimal choice of V under our framework. Since Σ 0 is unknown, we can use an iterative algorithm discussed later to obtain the most efficient estimate of θ based on a consistent estimate of Σ 0 .

For any given V , we use the Lagrange multiplier approach to solve the constrained optimization problem ( 7 ). Define the corresponding Lagrange function as

with κ , λ , and ξ being the Lagrange multipliers. It is easy to show that κ = 1, and the maximizer should satisfy

Therefore, let η = ( λ , ξ t , μ t ) t , we can express the profile log-likelihood as

We use the Newton-Raphson algorithm to find the stationary point η ^ V = ( λ ^ V , ξ ^ V t , μ ^ V t ) t of ( 8 ), with μ ^ V = ( θ ^ V t , α ^ V t , β ^ V t ) t . A good initial point can be ( n 1 , 1 / n 1 , 0 , 0 t , μ ^ 0 t ) t , with μ ^ 0 obtained by using the data from the internal study to fit models ( 2 ) and ( 4 ). We adjust the case-control sample size ratio when fitting ( 4 ).

We show in the Web Appendix A (Lemma 1) that under some regularity conditions, η ^ V is a consistent estimate of η ∗ = ( λ ∗ , ξ ∗ t , μ ∗ t ) t , with λ ∗ = ρ 1 1 + ρ 1 , ξ ∗ t = 0 , and μ ∗ = ( θ ∗ t , α ∗ t , β ∗ t ) t as defined before. After we obtain η ^ V , we can estimate P ( X i | D = 0), i = 1, …, n 1 , as

We want to point out that the nuisance parameter α is identifiable as α can influence ℓ V ( η ) through g ( X ; μ ).

The asymptotic distribution of η ^ V is given by the following result, with its proof shown in Web Appendix A .

Theorem 1. Assuming n 2 / n 1 → τ, n 1,1 / n 1,0 = ρ 1 , and n 2,1 / n 2,0 = ρ 2 remain constant as n 1 → ∞, we have

where J V and I V are defined in the Appendix .

Definitions of J V and I V rely on η *, Σ 0 , and P ( X | D = 0). To estimate the covariance of η ^ V , we can replace η ∗ with η ^ V , and use the estimated empirical distribution ( 9 ) for P ( X | D = 0). Similarly, we can replace Σ 0 with its consistent estimate Σ ^ 0 , which is a submatrix of A ^ − 1 B ^ ( A ^ − 1 ) t corresponding to β . A ^ and B ^ are estimates of A and B. They can be obtained by replacing μ * with μ ^ V , and by calculating the expectation in controls using estimates given by ( 9 ).

Although η ^ V is consistent given any V in ( 8 ), its level of efficiency depends on the choice of V. In fact, we can obtain the most efficient estimate of η by using V = Σ 0 , or its consistent estimate. We need to introduce some notations before presenting the result. Let s = ( λ , ξ t ) t be the vector of Lagrange multipliers. We can represent J v as the following block matrix:

Note that only the lower right corner submatrix depends on V. We can represent J Σ 0 similarly by replacing J V , μμ with J Σ 0 , μμ . By letting V = Σ 0 in I V , we can see that I Σ 0 can be written as

with J. λ being the first column of J Σ 0 . We have the following result (see Web Appendix A for the proof).

Corollary 1. The asymptotic variance–covariance matrix of n 1 ( η ^ V − η ∗ ) attains its minimum at V = Σ 0 . At V = Σ 0 , the asymptotic variance–covariance matrix of n 1 η ^ Σ 0 has the following form:

where e 1 is a vector with its first element being 1, all others being zero. This asymptotic variance–covariance matrix remains the same if V is a consistent estimate of Σ 0 .

Because of Corollary 1, we can use an iterative algorithm to find the optimal estimate. First, we let V in ( 8 ) be the identity matrix to obtain an initial estimate η 1 ). Second, we use η (1) to obtain Σ ^ 0 , which is a consistent estimate of Σ 0 . Third, by letting V = Σ ^ 0 , we obtain an updated estimate of η. We can iterate the second and third steps several times until the estimate converges. We define the final estimate of θ as θ ^ Σ ^ 0 and refer to the procedure as the retrospective generalized integration method (rGIM). In contrast, we call the original integration method that was developed for integrating data from prospective studies ( Zhang et al. 2020 ) the prospective generalized integration method (pGIM).

Although rGIM is designed for case-control studies, it can also be applied to the setting when one or both studies are conducted under a simple random sampling design. This is true because P ( X | D ) can be inferred properly with a set of random samples of ( X, D ). Simulation studies presented later also confirm this.

We can extend the aforementioned results to a more general situation where summary data come from multiple independent or partially overlapping external studies. Summary data from multiple models fitted within a given study are also allowed (see Example 2 as an illustration). All these can be achieved by specifying the correct variance–covariance matrix of ( α ~ t , β ~ t ) t . (See Web Appendix B for more details.)

2.4 |. Alternative approaches

Instead of treating the summary statistics β ~ as observed random variables, we can set β = β ~ as known and only estimate the other parameters (i.e., θ and α ). Qin (2000) and Chatterjee et al. (2016) used this strategy to incorporate summary information from a prospective study, although they did not consider the existence of nuisance parameter α. Using this strategy in case-control studies, we can estimate θ and α by solving the following constrained optimization problem:

where μ β ^ = ( θ t , α t , β ~ t ) t since β is fixed at β ~ . We denote the resultant constraint maximum likelihood (CML) estimate as υ ^ β ^ = ( θ ^ β ~ t , α ^ β ~ t ) t . Another alternative estimate of θ is the standard maximum likelihood estimate based on the internal study (MLE-Int), denoted as θ ^ m l e .

In Web Appendix A5 , we derive the proper variance-covariance matrix of the CML estimate by accounting for the variability of β ~ and prove the following result.

Corollary 2. The rGIM estimate θ ^ Σ ^ 0 is asymptotically more efficient than the internal data based MLE θ ^ m l e and the CML estimate θ ^ β ^ .

2.5 |. Different study populations

So far, we present procedures assuming that the internal and external studies are conducted within a common underlying population. In some applications, the two studies might be conducted in two source populations (called internal study population and external study population), which have different distributions of X. We can expand the proposed procedure to this more general setting.

We assume that the disease risk in the two populations can be specified by the following model:

where s = 1, or 2 indicates the internal or external study population. Thus, we in fact assume that regression coefficients except the intercept term are the same between the two risk models. Since we allow the two study populations to have different covariate distributions, each study population has its own covariate distribution in cases and in controls, denoted as P ( X | D = 1, s ), and P ( X | D = 0, s ), respectively. Following ( 2 ), we know

where θ ( s ) = θ ∗ ( s ) − log it { P ( D = 1 | s ) } . In addition to the individual level data and summary statistics, we also require observations on a set of controls { X i ∗ , i = 1 , … , n ∗ } from the external study population and use them as a reference set to estimate P ( X | D = 0, s = 2), which is different from P ( X | D = 0, s = 1). In Web Appendix C , we extend the rGIM procedure to the setting of different study populations and refer to it as GIM REF .

3 |. SIMULATION STUDIES

3.1 |. same internal and external study population.

We first considered the scenario when the internal and external studies were conducted within the same study population. We assumed that X = ( X 1 , X 2 ) presented genotypes on two genetic markers, called single nucleotide polymorphisms (SNPs), and the true disease risk model was log it { P ( D = 1 | X ) } = θ 0 ∗ + θ 1 X 1 + θ 2 X 2 . We further assumed that the two SNPs’ locations were in close proximity and let G = ( G 1 , G 2 ) represent allele type (0 or 1) of the two SNPs on a given chromosome (i.e., haplotype). In the study population, the distribution probability of G = (0, 0) , (0,1), (1,0), and (1,1) is 0.28, 0.12, 0.18, and 0.42, respectively. For each subject, its joint genotype X was given by the sum of two randomly selected haplotypes. We let θ 1 = log(2), θ 2 = log(1.5), and chose θ 1 * such that the disease prevalence was around 10%. Each pair of internal and external studies was generated from this population, with the internal study consisting of 500 cases and 500 controls, and the external study consisting of 500 cases and 2000 (or 5000) controls. Based on the external study, the summary data β ~ = ( β ~ 1 , β ~ 2 ) were coefficient estimates from the following two working models: W k ( X | D = 1 ) = W k ( X | D = 0 ) exp { α k + β k X k } , k = 1, 2.

We analyzed each simulated dataset with rGIM, pGIM, CML, and MLE-Int. Except MLE-Int, the other three methods used summary data consisting of either ( β ~ 1 , β ~ 2 ) , or β ~ 1 . Table 1 summarizes performances of the four methods over 5000 simulated datasets under each scenario.

Simulation results in situations when internal and external studies are conducted under a case-control sampling design in a common source population

The internal study has 500 cases, and 500 controls. The external study has 500 cases and 2000 (or 5000) controls. The true model is logit { P ( D = 1| G )} = θ 0 * + θ 1 G 1 + θ 2 G 2 . Summary data ( β ~ 1 , β ~ 2 ) are coefficient estimates based on two working models, logit { P ( D = 1| G k )} = α k + β k G k , k = 1, 2. Empirical bias (bias), empirical standard error (SE-Emp), mean of estimated asymptotic standard error (SE-Est), and coverage probability (CP) of a 95% confidence interval are summarized over 5000 simulated datasets. All numbers are multiplied by 100. Note : MLE-Int: standard MLE based on the internal study; CML: constraint MLE developed for case-control studies; pGIM: GIM procedure developed under the prospective likelihood framework; rGIM: GIM procedure developed under the retrospective likelihood framework.

First, we can notice from Table 1 that the asymptotic distribution of the rGIM estimate presented in Corollary 1 is quite accurate as the estimate is unbiased, with its estimated asymptotic standard error matching well with the empirical standard error in all considered settings. Its 95% confidence interval also has the correct coverage probability (CP). Second, as predicted by Corollary 2 , rGIM is at least as efficient as MLE-Int and CML. The magnitude of efficiency gain depends on the available summary data. Specifically, when summary data consist of only β ~ 1 , the estimated marginal effect of X 1 , rGIM is more efficient than MLE-Int and CML for estimating θ 1 (the true effect of X 1 ), but all three methods have the same level of efficiency for estimating θ 2 . If using ( β ~ 1 , β ~ 2 ) as summary data, the rGIM procedure shows a clear advantage over MLE-Int and CML for estimating both θ 1 and θ 2 . Third, the pGIM procedure that ignores the case-control sampling can generate incorrect standard error estimates. For example, when using ( β ~ 1 , β ~ 2 ) estimated from an external study with 500 cases and 2000 controls, the pGIM estimate of θ 1 has an empirical standard error of 0.071, compared to the mean estimated asymptotic standard error of 0.059. As a result, its 95% confidence interval has only 89% coverage probability. When we increase the number of controls in the external study to 5000, its coverage probability decreases further to 76%.

On a 2.30 GHz Linux machine, the running time of MLE-Int on an internal study of 500 cases and 500 controls is about 0.004 s. rGIM and CML procedures integrating summary data ( β ~ 1 , β ~ 2 ) with the same internal study take about 6.1 and 3.3 s, respectively. rGIM is slower than CML as it needs to reestimate ( β 1 , β 2 ), instead of assuming β i = β ~ i , i = 1, 2.

3.2 |. Different internal and external study populations

We considered the scenario when the internal and external studies were conducted in two different study populations. The internal study population was the same as the one given in Section 3.1 . For the external study population, the distribution of haplotype G was specified as G = (0,0) , (0,1), (1,0), and (1,1) with probability of 0.14, 0.06, 0.56, and 0.24, respectively. We assumed that the same aforementioned disease risk model applies to both populations. For each pair of simulated internal and external studies, we also generated an additional set of 300 controls from the external population and used them as reference samples in the GIM REF procedure. Simulation results are summarized in Table 2 . We did not present results from the pGIM procedure as we have shown in Section 3.1 that pGIM was not appropriate for case-control studies.

Simulation results in situations when internal and external studies are conducted under a case-control sampling design in two different source populations

The internal study has 500 cases and 500 controls. The external study has 500 cases, and 2000 (or 5000) controls. The reference set consists of 300 controls from the external population. The same risk model logit { P ( D = 1| G } = θ 0 * + θ 1 G 1 + θ 2 G 2 is assumed for both populations. Summary data ( β ~ 1 , β ~ 2 ) are coefficient estimates based on two working models, logit { P ( D = 1| G k )} = α k + β k G k , k = 1, 2. Empirical bias (bias), empirical standard error (SE-Emp), mean of estimated asymptotic standard error (SE-Est), and coverage probability (CP) of a 95% confidence interval are summarized over 5000 simulated datasets. All numbers are multiplied by 100. Note : MLE-Int: standard MLE based on the internal study; GIM REF : retrospective GIM procedure with a reference set developed for the setting in which external and internal studies are conducted in two different populations; rGIM: retrospective GIM procedure with the assumption that two studies are conducted in a common population.

Results in Table 2 demonstrate that GIM REF has the desired performance when dealing with studies from different populations. GIM REF is also more efficient than MLE-Int. On the other hand, the rGIM procedure, which is designed for data collected from a common study population, can generate inconsistent estimate and erroneously estimated standard error ( Table 2 ).

3.3 |. Studies under different designs

We conducted additional simulations to evaluate the performance of the retrospective likelihood approach in situations when one or both studies were conducted under a simple random sampling design.

First, we considered the scenario in which the internal study was a case-control study with 500 cases and 500 controls from the source population defined in Section 3.1 , while the external study consisted of 5500 subjects randomly sampled from the same source population. For the purpose of illustration, we only present results from analyses using { β ~ 1 , β ~ 2 } as summary data. Results shown in Table 3 indicate that both rGIM and CML estimates are consistent, with proper estimated standard errors. But rGIM is more efficient than CML and MLE-int. On the other hand, the prospective likelihood approach pGIM does not estimate the standard error correctly. This is expected as pGIM ignores the sampling bias in the internal data. We further considered the situation in which the external study had their 5500 subjects randomly sampled from a different source population (as defined in Section 3.2 ). In this setup, we also generated a reference set of 300 controls from the external study population and used it as part of the input for GIM REF . Results summarized in Table 4 indicate that GIM REF has the expected performance in this setting.

Simulation results in situations when the internal study is conducted under a case-control sampling design in a source population, and the external study is conducted under a simple random sampling design in the same population

The internal study has 500 cases and 500 controls. The external study consists of 5500 random sampled subjects. The true risk model is logit { P ( D = 1| G )} = θ 0 * + θ 1 G 1 + θ 2 G 2 Summary data ( β ~ 1 , β ~ 2 ) are coefficient estimates from two working models, logit { P ( D = 1| G k )} = α k + β k G k , k = 1, 2. Empirical bias (bias), empirical standard error (SE-Emp), mean of estimated asymptotic standard error (SE-Est), and coverage probability (CP) of a 95% confidence interval are summarized over 5000 simulated datasets. All numbers are multiplied by 100. Note : MLE-Int: standard MLE based on the internal study; CML: constraint MLE developed for case-control studies; pGIM: GIM procedure developed under the prospective likelihood framework; rGIM: GIM procedure developed under the retrospective likelihood framework.

Simulation results in situations when the internal study is conducted under a case-control sampling design in one population, the external study is conducted under a simple random sampling design in a different population

The internal study has 500 cases and 500 controls. The external study consists of 5500 random sampled subjects. The same risk model logit { P ( D = 1| G )} θ 0 * + θ 1 G 1 + θ 2 G 2 is assumed for both populations. Summary data ( β ~ 1 , β ~ 2 ) are coefficient estimates from two working models, logit {( P ( D = 1| G k )} = α k + β k G k , k = 1, 2. Empirical bias (bias), empirical standard error (SE-Emp), mean of estimated asymptotic standard error (SE-Est), and coverage probability (CP) of a 95% confidence interval are summarized over 5000 simulated datasets. All numbers are multiplied by 100. Note : MLE-Int: standard MLE based on the internal study; GIM REF : retrospective GIM procedure with a reference set developed for the setting in which external and internal studies are conducted in two different populations; rGIM: retrospective GIM procedure with the assumption that two studies are conducted in a common population.

Next, we considered situations when the internal study was generated under a simple random sampling design with 5500 subjects, and the external study of 500 cases and 500 controls was compiled under a case-control sampling design in the same or a different source population (see Web Appendix Tables 1 and 2 ). From those two tables, we can draw similar conclusions as those from Tables 3 and ​ and4, 4 , respectively. Finally, we considered the setting in which both internal and external studies were carried out under a simple random sampling design, with each consisting of 5500 subjects. Web Appendix Table 3 shows simulation results in the scenario when both studies are conducted in the same source population. Web Appendix Table 4 shows results when the two studies are sampled from different populations. Again, we can see from both tables that the retrospective likelihood approach works fine. As expected, the prospective approach (pGIM) also performs well in this setting as data are collected from prospective studies.

4 |. REAL APPLICATION

We illustrated the application of our method by applying it to two genome-wide association studies (GWAS) of pancreatic cancer ( Amundadottir et al., 2009 ; Petersen et al., 2010 ). Both GWAS had genotypes measured on over half a million SNPs. Genotypes on additional SNPs were obtained by imputation. We focused on subjects with predominantly European ancestry from the two studies. The first GWAS (called PanScan I) consisted of 1761 cases and 1804 controls. The second GWAS (called PanScan II) had 1768 cases and 1851 controls ( PanScan, 2015 ).

In this application, we concentrated on genes from the PredictDB Data Repository defined by gene expression in pancreatic tissues ( Gamazon et al., 2015 ; Barbeira et al., 2018 ). For a given gene, PredictDB provided a prediction model that used genotypes on a set of SNPs selected around the neighborhood of the gene to predict its gene expression level. The prediction model can be represented as ∑ k = 1 m w k X k , with X = (X 1 , …, X m ) being genotypes on the set of m selected SNPs, and w = ( w 1 , … , w m ) being their corresponding weights. A transcriptome-wide association study (TWAS) searches for genes associated with the outcome D by assessing the correlation between ε ( X ) and D assuming w is known.

We considered PanScan I GWAS as the internal study. From the independent PanScan II GWAS, we randomly selected a set of 500 controls as reference samples and treated the remaining data as the external study, based on which summary data were derived. To assess each gene’s association with pancreatic cancer, we used the model log it { P ( D = 1 | X , Z ) } = Z t ϕ + θ ε ( X ) , where θ is the parameter of interest, Z is the set of adjusted covariates, including the intercept, gender, and top five eigenvectors identified in the two combined GWAS. We assumed the summary data β ~ = ( β ~ 1 , … , β ~ m ) were coefficient estimates from the following working models fitted with data from the external study, W k ( X | D = 1 ) = W k ( X | D = 0 ) exp { α k + β k X k } , k = 1 , …, m .

In the integrative analysis, we only used summary information on SNPs whose pairwise correlation was less than 0.95 in order to avoid the problem of collinearity, although we still included all SNPs in the definition of ε( X ). For some genes, the number of summary statistics can be more than 60.

We analyzed a total of 5832 genes from PredictDB Data Repository with MLE-Int and GIM REF . In Figure 1 , we showed Q–Q plots for the two analyses. Although there is no globally significant gene (based on the Bonferroni threshold) detected by either method, there is more suggestive evidence of association indicated at the tail end of the Q–Q plot produced by GIM REF , than that by MLE-Int. This is expected as the GIM method utilizes more information than MLE-Int.

An external file that holds a picture, illustration, etc.
Object name is nihms-1747007-f0001.jpg

Q–Q plots for TWAS results based on pancreatic cancer GWAS. Each of 5832 considered genes is analyzed with the standard MLE based on the internal study (MLE-Int), as well as the retrospective GIM procedure with an additional reference set (GIM REF )

Zhong et al. (2020) recently conducted TWAS analysis of pancreatic cancer using a different strategy for defining the gene expression prediction model. Based on a much larger set of GWAS data, including those from PanScan I and II, they identified 12 pancreatic cancer associated genes that passed the Bonferroni threshold. Seven of those genes are among the set of 5832 genes we analyzed. Results on those seven genes are summarized in Table 5 . Again, it appears GIM ref can detect more signals than MLE-Int.

Gene-level association testing results based on pancreatic cancer GWAS

Results are given on seven genes that have been established to be associated with pancreatic cancer by Zhong et al. (2020) . Note : MLE-Int: standard MLE based on the internal study (PanScan I); GIM REF : retrospective GIM procedure with a reference set developed for the setting in which external (PanScan II) and internal (PanScan I) studies are conducted in two different populations.

5 |. DISCUSSION

We have developed an integrative procedure for effectively combining aggregated summary data with detailed individual-level data. We adopt a retrospective likelihood framework to account for the sampling bias resulting from the case-control study design. The procedure is very flexible to incorporate summary data generated from distinct working models from multiple external studies. It also allows for external studies to be conducted in a population different from the internal study population, provided that individual-level data on a set of reference control samples is available from the external study population. We establish asymptotic properties for the procedure and prove that its estimate is more efficient than the MLE based on the internal study and the constraint MLE procedure, which derives its estimate under the restriction imposed by the summary data.

We demonstrate that it is important to adjust for the sampling bias in integrative analyses. The prospective likelihood approach that ignores the case-control sampling design can generate inaccurate standard error estimates. Although the proposed procedure is developed specifically for integrating data from multiple case-control studies, it can be used for studies conducted under a simple random sampling design. We show through simulation studies that the proposed retrospective likelihood approach has the desired performance for internal and external studies conducted under either a case-control or a simple random sampling design. Another advantage of our procedure is that it can combine individual-level and summary data taken from different underlying populations, as long as a set of reference control samples are available from each external study population. The reference set is critical to ensure a correct estimate of the covariate distribution in the underlying population. Since the reference set only requires a random sample of controls, it is appealing in practices when case samples are not easily accessible. This feature is especially useful for genetic association studies, as ample reference genomes with different ethnic backgrounds are available from public resources. In situations where such reference samples are not available, our simulation studies suggest that applying the procedure under the assumption of a common source population can generate biased estimates. Further investigations are needed to develop procedures that can provide more robust estimates in this setting.

Supplementary Material

Supporting information, acknowledgements.

The study utilized the computational resource of the NIH Biowulf cluster ( https://hpc.nih.gov/ ).

SUPPORTING INFORMATION

Web Appendices and Tables referenced in Sections 2.3 , 2.4 , 2.5 and 3.3 are available with this article at the Biometrics website on Wiley Online Library. We have implemented the proposed procedure in the R package “gim” that can be obtained from https://CRAN.R-project.org/package=gim . Example codes on how to use the gim package are posted at the Biometrics website.

DATA AVAILABILITY STATEMENT

  • Amundadottir L, Kraft P, Stolzenberg-Solomon RZ, Fuchs CS, Petersen GM, Arslan AA, et al. (2009) Genome-wide association study identifies variants in the ABO locus associated with susceptibility to pancreatic cancer . Nature Genetics , 41 , 986–990. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Barbeira AN, Dickinson SP, Bonazzola R, Zheng J, Wheeler HE, Torres JM, et al. (2018) Exploring the phenotypic consequences of tissue specific gene expression variation inferred from GWAS summary statistics . Nature Communications , 9 , 1825. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Breslow NE, and Day NE (1980 ) Statistical methods in cancer research. Volume I – The analysis of case-control studies . IARC Scientific Publications , 32 , 5–338. [ PubMed ] [ Google Scholar ]
  • Carroll RJ, Wang S, and Wang CY (1995) Prospective analysis of logistic case-control studies . Journal of the American Statistical Association , 90 ,157–169. [ Google Scholar ]
  • Chatterjee N, Chen YH, Maas P, and Carroll RJ (2016) Constrained maximum likelihood estimation for model calibration using summary-level information from external big data sources . Journal of the American Statistical Association , 111 , 107–117. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Chaudhuri S, Handcock MS, and Rendall MS (2008) Generalised linear models incorporating population level information: An empirical likelihood based approach . Journal of the Royal Statistical Society Series B (Statistical Methodology) , 70 , 311–328. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Chen J, and Sitter RR (1999) A pseudo empirical likelihood approach to the effective use of auxiliary information in complex surveys . Statistica Sinica , 9 , 385–406. [ Google Scholar ]
  • Cheng W, Taylor JMG, Gu T, Tomlins SA, and Mukheqee B (2019) Informing a risk prediction model for binary outcomes with external coefficient information . Journal of the Royal Statistical Society, Series C (Applied Statistics) , 68 ,121–139. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Cheng W, Taylor JMG, Vokonas PS, Park SK, and Mukheqee B (2018) Improving estimation and prediction in linear regression incorporating external information from an established reduced model . Statistics in Medicine , 37 ,1515–1530. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Gamazon ER, Wheeler HE, Shah KP, Mozaffari SV, Aquino-Michaels K, Carroll RJ, et al. (2015) A gene-based association method for mapping traits using reference transcriptome data . Nature Genetics , 47 ,1091–1098. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Han P, and Lawless JF (2019) Empirical likelihood estimation using auxiliary summary information with different covariate distributions . Statistica Sinica , 29 , 1321–1342. [ Google Scholar ]
  • Huang CY, and Qin J (2020) A unified approach for synthesizing population-level covariate effect information in semiparametric estimation with survival data . Statistics in Medicine , 39 , 1573–1590. [ PubMed ] [ Google Scholar ]
  • Imbens GW, and Lancaster T (1994) Combining micro and macro data in microeconometric models . The Review of Economic Studies , 61 , 655–680. [ Google Scholar ]
  • Kundu P, Tang R, and Chatterjee N (2019) Generalized meta-analysis for multiple regression models across studies with disparate covariate information . Biometrika , 106 , 567–585. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • PanScan (2015) Whole genome scan for pancreatic cancer risk in the Pancreatic Cancer Cohort Consortium and Pancreatic Cancer Case-Control Consortium (PanScan) . dbGaP Study Accession # phs000206.v5.p3. [ Google Scholar ]
  • Petersen GM, Amundadottir L, Fuchs CS, Kraft P, Stolzenberg-Solomon RZ, Jacobs ZB, et al. (2010) A genome-wide association study identifies pancreatic cancer susceptibility loci on chromosomes 13q22.1, 1q32.1 and 5p15.33 . Nature Genetics , 42 , 224–228. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Prentice RL, and Pyke R (1979) Logistic disease incidence models and case-control studies . Biometrika , 66 , 403–411. [ Google Scholar ]
  • Qin J (2000) Combining parametric and empirical likelihoods . Biometrika , 87 , 484–490. [ Google Scholar ]
  • Qin J, and Zhang B (1997) A goodness-of-fit test for logistic regression models based on case-control data . Biometrika , 84 , 609–618. [ Google Scholar ]
  • Qin J, Zhang H, Li P, Albanes D, and Yu K (2015) Using covariate-specific disease prevalence information to increase the power of case-control studies . Biometrika , 102 ,169–180. [ Google Scholar ]
  • White H (1982) Maximum likelihood estimation of misspecified models . Econometrica , 50 ,1–25. [ Google Scholar ]
  • Zhang H, Deng L, Schiffman M, Qin J, and Yu K (2020) Generalized integration model for improved statistical inference by leveraging external summary data . Biometrika , 107 , 689–703. [ Google Scholar ]
  • Zhong J, Jermusyk A, Wu L, Hoskins JW, Collins I, Mocci E, et al. (2020) A transcriptome-wide association study identifies novel candidate susceptibility genes for pancreatic cancer . Journal of National Cancer Institute , 112 , 1003–1012. [ PMC free article ] [ PubMed ] [ Google Scholar ]
  • Open access
  • Published: 19 February 2024

Sustaining the collaborative chronic care model in outpatient mental health: a matrixed multiple case study

  • Bo Kim 1 , 2 ,
  • Jennifer L. Sullivan 3 , 4 ,
  • Madisen E. Brown 1 ,
  • Samantha L. Connolly 1 , 2 ,
  • Elizabeth G. Spitzer 1 , 5 ,
  • Hannah M. Bailey 1 ,
  • Lauren M. Sippel 6 , 7 ,
  • Kendra Weaver 8 &
  • Christopher J. Miller 1 , 2  

Implementation Science volume  19 , Article number:  16 ( 2024 ) Cite this article

469 Accesses

1 Altmetric

Metrics details

Sustaining evidence-based practices (EBPs) is crucial to ensuring care quality and addressing health disparities. Approaches to identifying factors related to sustainability are critically needed. One such approach is Matrixed Multiple Case Study (MMCS), which identifies factors and their combinations that influence implementation. We applied MMCS to identify factors related to the sustainability of the evidence-based Collaborative Chronic Care Model (CCM) at nine Department of Veterans Affairs (VA) outpatient mental health clinics, 3–4 years after implementation support had concluded.

We conducted a directed content analysis of 30 provider interviews, using 6 CCM elements and 4 Integrated Promoting Action on Research Implementation in Health Services (i-PARIHS) domains as codes. Based on CCM code summaries, we designated each site as high/medium/low sustainability. We used i-PARIHS code summaries to identify relevant factors for each site, the extent of their presence, and the type of influence they had on sustainability (enabling/neutral/hindering/unclear). We organized these data into a sortable matrix and assessed sustainability-related cross-site trends.

CCM sustainability status was distributed among the sites, with three sites each being high, medium, and low. Twenty-five factors were identified from the i-PARIHS code summaries, of which 3 exhibited strong trends by sustainability status (relevant i-PARIHS domain in square brackets): “Collaborativeness/Teamwork [Recipients],” “Staff/Leadership turnover [Recipients],” and “Having a consistent/strong internal facilitator [Facilitation]” during and after active implementation. At most high-sustainability sites only, (i) “Having a knowledgeable/helpful external facilitator [Facilitation]” was variably present and enabled sustainability when present, while (ii) “Clarity about what CCM comprises [Innovation],” “Interdisciplinary coordination [Recipients],” and “Adequate clinic space for CCM team members [Context]” were somewhat or less present with mixed influences on sustainability.

Conclusions

MMCS revealed that CCM sustainability in VA outpatient mental health clinics may be related most strongly to provider collaboration, knowledge retention during staff/leadership transitions, and availability of skilled internal facilitators. These findings have informed a subsequent CCM implementation trial that prospectively examines whether enhancing the above-mentioned factors within implementation facilitation improves sustainability. MMCS is a systematic approach to multi-site examination that can be used to investigate sustainability-related factors applicable to other EBPs and across multiple contexts.

Peer Review reports

Contributions to the literature

We examined the ways in which the sustainability of the evidence-based Collaborative Chronic Care Model differed across nine outpatient mental health clinics where it was implemented.

This work demonstrates a unique application of the Matrixed Multiple Case Study (MMCS) method, originally developed to identify factors and their combinations that influence implementation, to investigate the long-term sustainability of a previously implemented evidence-based practice.

Contextual influences on sustainability identified through this work, as well as the systematic approach to multi-site examination offered by MMCS, can inform future efforts to sustainably implement and methodically evaluate an evidence-based practice’s uptake and continued use in routine care.

The sustainability of evidence-based practices (EBPs) over time is crucial to maximize the public health impact of EBPs implemented into routine care. Implementation evaluators focus on sustainability as a central implementation outcome, and funders of implementation efforts seek sustained long-term returns on their investment. Furthermore, practitioners and leadership at implementation sites face the task of sustaining an EBP’s usage even after implementation funding, support, and associated evaluation efforts conclude. The circumstances and influences contributing to EBP sustainability are therefore of high interest to the field of implementation science.

Sustainability depends on the specific EBP being implemented, the individuals undergoing the implementation, the contexts in which the implementation takes place, and the facilitation of (i.e., support for) the implementation. Hence, universal conditions that invariably lead to sustainability are challenging to establish. Even if a set of conditions could be identified as being associated with high sustainability “on average,” its usefulness is questionable when most real-world implementation contexts may deviate from “average” on key implementation-relevant metrics.

Thus, when seeking a better understanding of EBP sustainability, there is a critical need for methods that examine the ways in which sustainability varies in diverse contexts. One such method is Matrixed Multiple Case Study (MMCS) [ 1 ], which is beginning to be applied in implementation research to identify factors related to implementation [ 2 , 3 , 4 , 5 ]. MMCS capitalizes on the many contextual variations and heterogeneous outcomes that are expected when an EBP is implemented across multiple sites. Specifically, MMCS provides a formalized sequence of steps for cross-site analysis by arranging data into an array of matrices, which are sorted and filtered to test for expected factors and identify less expected factors influencing an implementation outcome of interest.

Although the MMCS represents a promising method for systematically exploring the “black box” of the ways in which implementation is more or less successful, it has not yet been applied to investigate the long-term sustainability of implemented EBPs. Therefore, we applied MMCS to identify factors related to the sustainability of the evidence-based Collaborative Chronic Care Model (CCM), previously implemented using implementation facilitation [ 6 , 7 , 8 ], at nine VA medical centers’ outpatient general mental health clinics. An earlier interview-based investigation of CCM provider perspectives had identified key determinants of CCM sustainability at the sites, yet characteristics related to the ways in which CCM sustainability differed at the sites are still not well understood. For this reason, our objective was to apply MMCS to examine the interview data to determine factors associated with CCM sustainability at each site.

Clinical and implementation contexts

CCM-based care aims to ensure that patients are treated in a coordinated, patient-centered, and anticipatory manner. This project’s nine outpatient general mental health clinics had participated in a hybrid CCM effectiveness-implementation trial 3 to 4 years prior, which had resulted in improved clinical outcomes that were not universally maintained post-implementation (i.e., after implementation funding and associated evaluation efforts concluded) [ 7 , 9 ]. This lack of aggregate sustainability across the nine clinics is what prompted the earlier interview-based investigation of CCM provider perspectives that identified key determinants of CCM sustainability at the trial sites [ 10 ].

These prior works were conducted in VA outpatient mental health teams, known as Behavioral Health Interdisciplinary Program (BHIP) teams. While there was variability in the exact composition of each BHIP team, all teams consisted of a multidisciplinary set of frontline clinicians (e.g., psychiatrists, psychologists, social workers, nurses) and support staff, serving a panel of about 1000 patients each.

This current project applied MMCS to examine the data from the earlier interviews [ 10 ] for the ways in which CCM sustainability differed at the sites and the factors related to sustainability. The project was determined to be non-research by the VA Boston Research and Development Service, and therefore did not require oversight by the Institutional Review Board (IRB). Details regarding the procedures undertaken for the completed hybrid CCM effectiveness-implementation trial, which serves as the context for this project, have been previously published [ 6 , 7 ]. Similarly, details regarding data collection for the follow-up provider interviews have also been previously published [ 10 ]. We provide a brief overview of the steps that we took for data collection and describe the steps that we took for applying MMCS to analyze the interview data. Additional file  1 outlines our use of the Consolidated Criteria for Reporting Qualitative Research (COREQ) Checklist [ 11 ].

Data collection

We recruited 30 outpatient mental health providers across the nine sites that had participated in the CCM implementation trial, including a multidisciplinary mix of mental health leaders and frontline staff. We recruited participants via email, and we obtained verbal informed consent from all participants. Each interview lasted between 30 and 60 min and focused on the degree to which the participant perceived care processes to have remained aligned to the CCM’s six core elements: work role redesign, patient self-management support, provider decision support, clinical information systems, linkages to community resources, and organizational/leadership support [ 12 , 13 , 14 ]. Interview questions also inquired about the participant’s perceived barriers and enablers influencing CCM sustainability, as well as about the latest status of CCM-based care practices. Interviews were digitally recorded and professionally transcribed. Additional details regarding data collection have been previously published [ 10 ].

Data analysis

We applied MMCS’ nine analytical steps [ 1 ] to the interview data. Each step described below was led by one designated member of the project team, with subsequent review by all project team members to reach a consensus on the examination conducted for each step.

We established the evaluation goal (step 1) to identify the ways in which sustainability differed across the sites and the factors related to sustainability, defining sustainability (step 2) as the continued existence of CCM-aligned care practices—namely, that care processes remained aligned with the six core CCM elements. Table  1 shows examples of care processes that align with each CCM element. As our prior works directly leading up to this project (i.e., design and evaluation of the CCM implementation trial that involved the very sites included in this project [ 6 , 15 , 16 ]) were guided by the Integrated Promoting Action on Research Implementation in Health Services (i-PARIHS) framework [ 17 ] and i-PARIHS positions facilitation (the implementation strategy that our trial was testing) as the core ingredient that drives implementation [ 17 ], we selected i-PARIHS’ four domains—innovation, recipients, context, and facilitation—as relevant domains under which to examine factors influencing sustainability (step 3). i-PARIHS posits that the successful implementation of an innovation and its sustained use by recipients in a context is enabled by facilitation (both the individuals doing the facilitation and the process used for facilitation). We examined the data on both sustainability and potentially relevant i-PARIHS domains (step 4) by conducting directed content analysis [ 18 ] of the recorded and professionally transcribed interview data. We used the six CCM elements and the four i-PARIHS domains as a priori codes.

Additional file  2 provides an overview of data input, tasks performed, and analysis output for MMCS steps 5 through 9 described below. We assessed sustainability per site (step 5) by generating CCM code summaries per site, and reached a consensus on whether each site exhibited high, medium, or low sustainability relative to other sites based on the summary data. We assigned a higher sustainability level for sites that exhibited more CCM-aligned care processes, had more participants consistently mention those processes, and considered those processes more as “just the way things are done” at the site. Namely, (i) high sustainability sites had concrete examples of CCM-aligned care processes (such as the ones shown in Table  1 ) for many of the six CCM elements, which multiple participants mentioned as central to how they deliver care, (ii) low sustainability sites had only a few concrete examples of CCM-aligned care processes, mentioned by only a small subset of participants and/or inconsistently practiced, and (iii) medium sustainability sites matched neither of the high nor low sustainability cases, having several concrete examples of CCM-aligned care process for some of the CCM elements, varying in whether they are mentioned by multiple participants or how consistently they are a part of delivering care. For the CCM code summaries per site, one project team member initially reviewed the coded data to draft the summaries including exemplar quotes. Each summary and relevant exemplar quotes were then reviewed by and refined with input from all six project team members during recurring team meetings to finalize the high, medium, or low sustainability designation to use in the subsequent MMCS steps. Reviewing and refining the summaries for the nine sites took approximately four 60-min meetings of the six project team members, with each site’s CCM code summary taking approximately 20–35 min to discuss and reach consensus on. We referred to lists of specific examples of how the six core CCM elements were operationalized in our CCM implementation trial [ 19 , 20 ]. Refinements occurred mostly around familiarizing the newer members of the project team (i.e., those who had not participated in our prior CCM-related work) with the examples and definitions. We aligned to established qualitative analysis methods for consensus-reaching discussions [ 18 , 21 ]. Recognizing the common challenge faced by such discussions in adequately accounting for everyone’s interpretations of the data [ 22 ], we drew on Bens’ meeting facilitation techniques [ 23 ] that include setting ground rules, ensuring balanced participation from all project team members, and accurately recording decisions and action items.

We then identified influencing factors per site (step 6), by generating i-PARIHS code summaries per site and identifying distinct factors under each domain of i-PARIHS (e.g., Collaborativeness and teamwork as a factor under the Recipients domain). For the i-PARIHS code summaries per site, one project team member initially reviewed the coded data to draft the summaries including exemplar quotes. They elaborated on each i-PARIHS domain-specific summary by noting distinct factors that they deemed relevant to the summary, proposing descriptive wording to refer to each factor (e.g., “team members share a commitment to their patients” under the Recipients domain). Each summary, associated factor descriptions, and relevant exemplar quotes were then reviewed and refined with input from all six project team members during recurring team meetings to finalize the relevant factors to use in the subsequent MMCS steps. Finalizing the factors included deciding which similar proposed factor descriptions from different sites to consolidate into one factor and which wording to use to refer to the consolidated factor (e.g., “team members share a commitment to their patients,” “team members collaborate well,” and “team members know each other’s styles and what to expect” were consolidated into the Collaborativeness and teamwork factor under the Recipients domain). It took approximately four 60-min meetings of the six project team members to review and refine the summaries and factors for the nine sites, with each site’s i-PARIHS code summary and factors taking approximately 20–35 min to discuss and reach consensus on. We referred to lists of explicit definitions of i-PARIHS constructs that our team members had previously developed and published [ 16 , 24 ]. We once again aligned to established qualitative analysis methods for consensus-reaching discussions [ 18 , 21 ], drawing on Bens’ meeting facilitation techniques [ 23 ] to adequately account for everyone’s interpretations of the data [ 22 ].

We organized the examined data (i.e., the assessed sustainability and identified factors per site) into a sortable matrix (step 7) using Microsoft Excel [ 25 ], laid out by influencing factor (row), sustainability (column), and site (sheet). We conducted within-site analysis of the matrixed data (step 8), examining the data on each influencing factor and designating whether the factor (i) was present, somewhat present, or minimally present [based on aggregate reports from the site’s participants; used “minimally present” when, considering all available data from a site regarding a factor, the factor was predominantly weak (e.g., predominantly weak Ability to continue patient care during COVID at a medium sustainability site); used “somewhat present” when, considering all available data from a site regarding a factor, the factor was neither predominantly strong nor predominantly weak (e.g., neither predominantly strong nor predominantly weak Collaborativeness and teamwork at a low sustainability site)], and (ii) had an enabling, hindering, or neutral/unclear influence on sustainability (designated as “neutral” when, considering all available data from a site regarding a factor, the factor had neither a predominantly enabling nor a predominantly hindering influence on sustainability). These designations of factors’ presence and influence are conceptually representative of what is commonly referred to as magnitude and valence, respectively, by other efforts that construct scoring for qualitative data (e.g., [ 26 , 27 ]). Like the team-based consensus approach of earlier MMCS steps, factors’ presence and type of influence per site were initially proposed by one project team member after reviewing the matrix’s site-specific data, then refined with input from all project team members during recurring team meetings that reviewed the matrix. Accordingly, similar to the earlier MMCS steps, we aligned to established qualitative methods [ 18 , 21 ] and meeting facilitation techniques [ 23 ] for these consensus-reaching discussions.

We then conducted a cross-site analysis of the matrixed data (step 9), assessing whether factors and their combinations were (i) present across multiple sites, (ii) consistently associated with higher or lower sustainability, and (iii) emphasized at some sites more than others. We noted that any factor may have not come up during interviews with a site because either it is not pertinent or it is pertinent but still did not come up, although we asked an open-ended question at the end of each interview about whether there was anything else that the participant wanted to share regarding sustainability. To adequately account for these possibilities, we decided as a team to regard a factor or a combination of factors as being associated with high/medium/low sustainability if it was identified at a majority (i.e., even if not all) of the sites designated as high/medium/low sustainability (e.g., if the Collaborativeness and teamwork factor is identified at a majority, even if not all, of the high sustainability sites, we would find it to be associated with high sustainability). Like the team-based consensus approach of earlier MMCS steps, cross-site patterns were initially proposed by one project team member after reviewing the matrix’s cross-site data, then refined with input from all project team members during recurring team meetings that reviewed the matrix. Accordingly, similar to the earlier MMCS steps, we aligned to established qualitative methods [ 18 , 21 ] and meeting facilitation techniques [ 23 ] for these consensus-reaching discussions. We acknowledged the potential existence of additional factors influencing sustainability that may not have emerged during our interviews and also may vary substantially between sites. For example, adaptation of the CCM, characteristics of the patient population, and availability of continued funding, which are factors that extant literature reports as being relevant to sustainability [ 28 , 29 ], were not seen in our interview data. To maintain our analytic focus on the factors seen in our data, we did not add these factors to our analysis.

For the nine sites included in this project, we found the degree of CCM sustainability to be split evenly across the sites—three high-, three medium-, and three low-sustainability. Twenty-five total influencing factors were identified under the i-PARIHS domains of Innovation (6), Recipients (6), Context (8), and Facilitation (5). Table  2 shows these identified influencing factors by domain. Figure  1 shows 11 influencing factors that were identified for at least two sites within a group of high/medium/low sustainability sites—e.g., the factor “consistent and strong internal facilitator” is shown as being present at high sustainability sites with an enabling influence on sustainability, because it was identified as such at two or more of the high sustainability sites. Of these 11 influencing factors, four were identified only for sites with high CCM sustainability and two were identified only for sites with medium or low CCM sustainability.

figure 1

Influencing factors that were identified for at least two sites within a group of high/medium/low sustainability sites

Key trends in influencing factors associated with high, medium, and/or low CCM sustainability

Three factors across two i-PARIHS domains exhibited strong trends by sustainability status. They were the Collaborativeness and teamwork and Turnover of clinic staff and leadership factors under the Recipients domain, and the Having a consistent and strong internal facilitator factor under the Facilitation domain.

Recipients-related factors

Collaborativeness and teamwork was present with an enabling influence on CCM sustainability at most high and medium sustainability sites, while it was only somewhat present with a neutral influence on CCM sustainability at most low sustainability sites. When asked what had made their BHIP team work well, a participant from a high sustainability site said,

“Just a collaborative spirit.” (Participant 604)

A participant from a medium sustainability site said,

“We joke that [the BHIP teams] are even family, that the teams really do function pretty tightly and they each have their own personality.” (Participant 201)

At the low sustainability sites, willingness to work as a team varied across team members; a participant from a low sustainability site said,

“… I think it has to be the commitment of the people who are on the team. So those that are regularly attending, we get a lot more out of it than those that probably don't ever come [to team meetings].” (Participant 904)

Collaborativeness and teamwork of BHIP team members were often perceived as the highlight of pursuing interdisciplinary care.

Turnover of clinic staff and leadership was present with a hindering influence on CCM sustainability at most high, medium, and low sustainability sites.

“We’ve lost a lot of really, really good providers here in the time I’ve been here …,” (Participant 102)

said a participant from a low-sustainability site that had to reconfigure its BHIP teams due to clinic staff shortages. Turnover of mental health clinic leadership made it difficult to maintain CCM practices, especially beyond the teams that participated in the original CCM implementation trial. A participant from a medium sustainability site said,

“Probably about 90 percent of the things that we came up with have fallen by the wayside. Within our team, many of those remain but again, that hand off towards the other teams that I think partly is due to the turnover rate with program managers, supervisors, didn’t get fully implemented.” (Participant 703)

Although turnover was an issue for high sustainability sites as well, there was also indication of the situation improving in recent years; a participant from a high sustainability site said,

“… our attrition rollover rate has dropped quite a bit and I would really attribute that to [the CCM being] more functional and more sustainable and tolerable for the providers.” (Participant 502)

As such, staff and leadership turnover was deemed a major challenge for CCM sustainability for all sites regardless of the overall level of sustainability.

Facilitation-related factor

Having a consistent and strong internal facilitator was present with an enabling influence on CCM sustainability at high sustainability sites, not identified as an influencing factor at most of the medium sustainability sites, and variably present with a hindering, neutral, or unclear influence on CCM sustainability at low sustainability sites. Participants from a high sustainability site perceived that it was important for the internal facilitator to understand different BHIP team members’ personalities and know the clinic’s history. A participant from another high sustainability site shared that, as an internal facilitator themselves, they focused on recognizing and reinforcing the progress of team members:

“… I'm often the person who kind of [starts] off with, ‘Hey, look at what we've done in this location,’ ‘Hey look at what the team's done this month.’” (Participant 402)

A participant from a low sustainability site had also served as an internal facilitator and recounted the difficulty and importance of readying the BHIP team to function in the long run without their assistance:

“I should have been able to get out sooner, I think, to get it to have them running this themselves. And that was just a really difficult process.” (Participant 301)

Participants, especially from the high and low sustainability sites, attributed their BHIP teams’ successes and challenges to the skills of the internal facilitator.

Influencing factors identified only for sites with high CCM sustainability

Four factors across four i-PARIHS domains were identified for high sustainability sites and not for medium or low sustainability sites. They were the factors Details about the CCM being well understood (Innovation domain), Interdisciplinary coordination (Recipients domain), Having adequate clinic space for CCM team members (Context domain), and Having a knowledgeable and helpful external facilitator (Facilitation domain).

Innovation-related factor

Details about the CCM being well understood was minimal to somewhat present with an unclear influence on CCM sustainability.

“We’ve … been trying to help our providers see the benefit of team-based care and the episodes-of-care idea, and I would say that is something our folks really have continued to struggle with as well,” (Participant 401)

said a participant from a high sustainability site. “What is considered CCM-based care?” continued to be a question on providers’ minds. A participant from a high sustainability site asked during the interview,

“Is there kind of a clearing house of some of the best practices for [CCM] that you guys have … or some other collection of resources that we could draw from?” (Participant 601)

Although such references are indeed accessible online organization-wide, participants were not always aware of those resources or what exactly CCM entails.

Recipients-related factor

Interdisciplinary coordination was somewhat present with a hindering, neutral, or unclear influence on CCM sustainability. Coordination between psychotherapy and psychiatry providers was deemed difficult by participants from high-sustainability sites. A participant said,

“We were initially kind of top heavy on the psychiatry so just making sure we have … therapy staff balancing that out [has been important].” (Participant 501)

Another participant perceived that BHIP teams were helpful in managing.

… ‘sibling rivalry’ between different disciplines … because [CCM] puts us all in one team and we communicate.” (Participant 505)

Interdisciplinary coordination was understood by the participants as being necessary for effective CCM-based care yet difficult to achieve.

Context-related factor

Having adequate clinic space for CCM team members was minimal to somewhat present with a hindering, neutral, or unclear influence on CCM sustainability. COVID-19 led to changes in how clinic space was used/assigned. A participant from a high sustainability site remarked,

“Pre-COVID everything was in a room instead of online. And now all our meetings are online and so it's actually really easy for the supervisors to be able to rotate through them and then, you know, they can answer programmatic questions ….” (Participant 402)

Participants from another high sustainability site found that issues regarding limited clinic space were both exacerbated and alleviated by COVID, with the mental health service losing space to vaccine clinics but more mental health clinicians teleworking and in less need of clinic space. Virtual connections were seen to alleviate some physical workspace-related concerns.

Having a knowledgeable and helpful external facilitator was variably present; when present, it had an enabling influence on CCM sustainability. Participants from a high sustainability site noted how many of the external facilitator’s efforts to change the BHIP team’s work processes very much remained over time. An example of a change was to have team meetings be structured to meet evolving patient needs. Team members came to meetings with the shared knowledge and expectation that,

“… we need to touch on folks who are coming out of the hospital, we need to touch on folks with higher acuity needs.” (Participant 402)

Implementation support that sites received from their external facilitator mostly occurred during the time period of the original CCM implementation trial; correspondence with the external facilitator after that trial time period was not common for sites. Participants still largely found the external facilitator to provide helpful guidance and advice on delivering CCM-based care.

Influencing factors identified only for sites with medium or low CCM sustainability

Two factors were identified for medium or low sustainability sites and not for high sustainability sites. They were the factors Ability to continue patient care during COVID and Adequate resources/capacity for care delivery . These factors were both under i-PARIHS’ Context domain, unlike the influencing factors above that were identified only for high sustainability sites, which spanned all four i-PARIHS domains.

Context-related factors

Ability to continue patient care during COVID had a hindering influence on CCM sustainability when minimally present. Participants felt that their CCM work was challenged when delivering care through telehealth was made difficult—e.g., at a medium sustainability site, site policies during the pandemic required a higher number of in-person services than the BHIP team providers expected or desired to deliver. On the other hand, this factor had an enabling influence on CCM sustainability when present. A participant at a low sustainability site mentioned the effect of telehealth on being able to follow up more easily with patients who did not show up for their appointments:

“… my no-show rate has dropped dramatically because if people don’t log on after a couple minutes, I call them. They're like ‘oh, I forgot, let me pop right on,’ whereas, you know, in the face-to-face space, you know, you wait 15 minutes, you call them, it’s too late for them to come in so then they're no shows.” (Participant 102)

The advantages of virtual care delivery, as well as the challenges of getting approvals to pursue it to varying extents, were well recognized by the participants.

Adequate resources/capacity for care delivery was minimally present at medium sustainability sites with a hindering influence on CCM sustainability. At a medium sustainability site, although leadership was supportive of CCM, resources were being used to keep clinics operational (especially during COVID) rather than investing in building new CCM-based care delivery processes.

“I think that if my boss came to me, [and asked] what could I do for [the clinics] … I would say even more staff,” (Participant 202)

said a participant from a medium sustainability site. At the same time, the participant, as many others we interviewed, understood and emphasized the need for BHIP teams to proceed with care delivery even when resources were limited:

“… when you’re already dealing with a very busy clinic, short staff and then you’re hit with a pandemic you handle it the best that you can.” (Participant 202)

Participants felt the need for basic resource requirements to be met in order for CCM-based care to be feasible.

In this project, we examined factors influencing the sustainability of CCM-aligned care practices at general mental health clinics within nine VA medical centers that previously participated in a CCM implementation trial. Guided by the core CCM elements and i-PARIHS domains, we conducted and analyzed CCM provider interviews. Using MMCS, we found CCM sustainability to be split evenly across the nine sites (three high, three medium, and three low), and that sustainability may be related most strongly to provider collaboration, knowledge retention during staff/leadership transitions, and availability of skilled internal facilitators.

In comparison to most high sustainability sites, participants from most medium or low sustainability sites did not mention a knowledgeable and helpful external facilitator who enabled sustainability. Participants at the high sustainability sites also emphasized the need for clarity about what CCM-based care comprises, interdisciplinary coordination in delivering CCM-aligned care, and adequate clinic space for BHIP team members to connect and collaborate. In contrast, in comparison to participants at most high sustainability sites, participants at most medium or low sustainability sites emphasized the need for better continuity of patient-facing activities during the COVID-19 pandemic and more resources/capacity for care delivery. A notable difference between these two groups of influencing factors is that the ones emphasized at most high sustainability sites are more CCM-specific (e.g., external facilitator with CCM expertise, knowledge, and structures to support delivery of CCM-aligned care), while the ones emphasized at most medium or low sustainability sites are factors that certainly relate to CCM sustainability but are focused on care delivery operations beyond CCM-aligned care (e.g., COVID’s widespread impacts, limited staff availability). In short, an emphasis on immediate, short-term clinical needs in the face of the COVID-19 pandemic and staffing challenges appeared to sap sites’ enthusiasm for sustaining more collaborative, CCM-consistent care processes.

Our previous qualitative analysis of these interview data suggested that in order to achieve sustainability, it is important to establish appropriate infrastructure, organizational readiness, and mental health service- or department-wide coordination for CCM implementation [ 10 ]. The findings from the current project augment these previous findings by highlighting the specific factors associated with higher and lower CCM sustainability across the project sites. This additional knowledge provides two important insights into what CCM implementation efforts should prioritize with regard to the previously recommended appropriate infrastructure, readiness, and coordination. First, for knowledge retention and coordination during personnel changes (including any changes in internal facilitators through and following implementation), care processes and their specific procedures should be established and documented in order to bring new personnel up to speed on those care processes. Management sciences, as applied to health care and other fields, suggest that such organizational knowledge retention can be maximized when there are (i) structures set up to formally recognize/praise staff when they share key knowledge, (ii) succession plans to be applied in the event of staff turnover, (iii) opportunities for mentoring and shadowing, and (iv) after action reviews of conducted care processes, which allow staff to learn about and shape the processes themselves [ 30 , 31 , 32 , 33 ]. Future CCM implementation efforts may thus benefit from enacting these suggestions alongside establishing and documenting CCM-based care processes and associated procedures.

Second, efforts to implement CCM-aligned practices into routine care should account for the extent to which sites’ more fundamental operational needs are met or being addressed. That information can be used to appropriately scope the plan, expectations, and timeline for implementation. For instance, ongoing critical staffing shortages or high turnover [ 34 ] at a site are unlikely to be resolved through a few months of CCM implementation. In fact, in that situation, it is possible that CCM implementation efforts could lead to reduced team effectiveness in the short term, given the effort required to establish more collaborative and coordinated care processes [ 35 ]. Should CCM implementation move forward at a given site, implementation goals ought to be set on making progress in realms that are within the implementation effort’s control (e.g., designing CCM-aligned practices that take staffing challenges into consideration) [ 36 , 37 ] rather than on factors outside of the effort’s control (e.g., staffing shortages). As healthcare systems determine how to deploy support (e.g., facilitators) to sites for CCM implementation, they would benefit from considering whether it is primarily CCM expertise that the site needs at the moment, or more foundational organizational resources (e.g., mental health staffing, clinical space, leadership enhancement) [ 38 ] to first reach an operational state that can most benefit from CCM implementation efforts at a later point in time. There is growing consensus across the field that the readiness of a healthcare organization to innovate is a prerequisite to successful innovation (e.g., CCM implementation) regardless of the specific innovation [ 39 , 40 ]. Several promising strategies specifically target these organizational considerations for implementing evidence-based practices (e.g., [ 41 , 42 ]). Further, recent works have begun to more clearly delineate leadership-related, climate-related, and other contextual factors that contribute to organizations’ innovation readiness [ 43 ], which can inform healthcare systems’ future decisions regarding preparatory work leading to, and timing of, CCM implementation at their sites.

These considerations informed by MMCS may have useful implications for implementation strategy selection and tailoring for future CCM implementation efforts, especially in delineating the target level (e.g., system, organizational, clinic, individual) and timeline of implementation strategies to be deployed. For instance, of the three factors found to most notably trend with CCM sustainability, Collaborativeness and teamwork may be strengthened through shorter-term team-building interventions at the organizational and/or clinic levels [ 38 ], Turnover of clinic staff and leadership may be mitigated by aiming for longer-term culture/climate change at the system and/or organizational levels [ 44 , 45 , 46 ], and Having a consistent and strong internal facilitator may be ensured more immediately by selecting an individual with fitting expertise/characteristics to serve in the role [ 15 ] and imparting innovation/facilitation knowledge to them [ 47 ]. Which of these factors to focus on, and through what specific strategies, can be decided in partnership with an implementation site—for instance, candidate strategies can be identified based on ones that literature points to for addressing these factors [ 48 ], systematic selection of the strategies to move forward can happen with close input from site personnel [ 49 ], and explicit further specification of those strategies [ 50 ] can also happen in collaboration with site personnel to amply account for site-specific contexts [ 51 ].

As is common for implementation projects, the findings of this project are highly context-dependent. It involves the implementation of a specific evidence-based practice (the CCM) using a specific implementation strategy (implementation facilitation) at specific sites (BHIP teams within general mental health clinics at nine VA medical centers). For such context-dependent findings to be transferable [ 52 , 53 ] to meaningfully inform future implementation efforts, sources of variation in the findings and how the findings were reached must be documented and traceable. This means being explicit about each step and decision that led up to cross-site analysis, as MMCS encourages, so that future implementation efforts can accurately view and consider why and how findings might be transferable to their own work. For instance, beyond the finding that Turnover of clinic staff and leadership was a factor present at most of the examined sites, MMCS’ traceable documentation of qualitative data associated with this factor at high sustainability sites also allowed highlighting the perception that CCM implementation is contributing to mitigating turnover of providers in the clinic over time, which may be a crucial piece of information that fuels future CCM implementation efforts.

Furthermore, to compare findings and interpretations across projects, consistent procedures for setting up and conducting these multi-site investigations are indispensable [ 54 , 55 , 56 ]. Although many projects involve multiple sites and assess variations across the sites, it is less common to have clearly delineated protocols for conducting such assessments. MMCS is meant to target this very gap, by offering a formalized sequence of steps that prompt specification of analytical procedures and decisions that are often interpretive and left less specified. MMCS uses a concrete data structure (the matrix) to traceably organize information and knowledge gained from a project, and the matrix can accommodate various data sources and conceptual groundings (e.g., guiding theories, models, and frameworks) that may differ from project to project – for instance, although our application of MMCS aligned to i-PARIHS, other projects applying MMCS [ 2 , 5 ] use different conceptual guides (e.g., Consolidated Framework for Implementation Research [ 57 ], Theoretical Domains Framework [ 58 ]). Therefore, as more projects align to the MMCS steps [ 1 ] to identify factors related to implementation and sustainability, better comparisons, consolidations, and transfers of knowledge between projects may become possible.

This project has several limitations. First, the high, medium, and low sustainability assigned to the sites were based on the sites’ CCM sustainability relative to one another, rather than based on an external metric of sustainability. As measures of sustainability such as the Program Sustainability Assessment Tool [ 59 , 60 ] and the Sustainment Measurement System Scale [ 61 ] become increasingly developed and tested, future projects may consider the feasibility of incorporating such measures to assess each site’s sustainability. In our case, we worked on addressing this limitation by using a consensus approach within our project team to assign sustainability levels to sites, as well as by confirming that the sites that we designated as high sustainability exhibited CCM elements that we had previously observed at the end of their participation in the original CCM implementation trial [ 19 ]. Second, we did not assign strict thresholds above/below which the counts or proportions of data regarding a factor would automatically indicate whether the factor (i) was present, somewhat present, or minimally present and (ii) had an enabling, hindering, or neutral/unclear influence on sustainability. This follows widely accepted qualitative analytical guidance that discourages characterizing findings solely based on the frequency with which a notion is mentioned by participants [ 62 , 63 , 64 ], in order to prevent unsubstantiated inferences or conclusions. We sought to address this limitation in two ways: We carefully documented the project team’s rationale for each consensus reached, and we reviewed all consensuses reached in their entirety to ensure that any two factors with the same designation (e.g., “minimally present”) do not have associated rationale that conflict across those factors. These endeavors we undertook closely adhere to established case study research methods [ 65 ], which MMCS builds on, that emphasize strengthening the validity and reliability of findings through documenting a detailed analytic protocol, as well as reviewing data to ensure that patterns match across analytic units (e.g., factors, interviewees, sites). Third, our findings are based on three sites each for high/medium/low sustainability, and although we identified single factors associated with sustainability, we found no specific combinations of factors’ presence and influence that were repeatedly existent at a majority of the sites designated as high/medium/low sustainability. Examining additional sites on the factors identified through this work (as we will for our subsequent CCM implementation trial described below) will allow more opportunities for repeated combinations and other factors to emerge, making possible firmer conclusions regarding the extent to which the currently identified factors and absence of identified combinations are applicable beyond the sites included in this study. Fourth, the identified influencing factor “leadership support for CCM” (under the Context domain of the i-PARIHS framework) substantially overlaps in concept with the core “organizational/leadership support” element of the CCM. To avoid circular reasoning, we used leadership support-related data to inform our assignment of sites’ high, medium, or low CCM sustainability, rather than as a reason for the sites’ CCM sustainability. In reality, strong leadership support may both result from and contribute to implementation and sustainability [ 16 , 66 ], and thus causal relationships between the i-PARIHS-aligned influencing factors and the CCM elements (possibly with feedback loops) warrant further examination to most appropriately use leadership support-related data in future analyses of CCM sustainability. Fifth, findings may be subject to both social desirability bias in participants providing more positive than negative evidence of sustainability (especially participants who are responsible for implementing and sustaining CCM-aligned care at their site) and the project team members’ bias in interpreting the findings to align to their expectations of further effort being necessary to sustainably implement the CCM. To help mitigate this challenge, the project interviewers strove to elicit from participants both positive and negative perceptions and experiences related to CCM-based care delivery, both of which were present in the examined interview data.

Future work stemming from this project is twofold. Regarding CCM implementation, we will conduct a subsequent CCM implementation trial involving eight new sites to prospectively examine how implementation facilitation with an enhanced focus on these findings affects CCM sustainability. We started planning for sustainability prior to implementation, looking to this work for indicators of specific modifications needed to the previous way in which we used implementation facilitation to promote the uptake of CCM-based care [ 67 ]. Findings from this work suggest that sustainability may be related most strongly to (i) provider collaboration, (ii) knowledge retention during staff/leadership transitions, and (iii) availability of skilled internal facilitators. Hence, we will accordingly prioritize developing procedures for (i) regular CCM-related information exchange amongst BHIP team members, as well as between the BHIP team and clinic leadership, (ii) both translating knowledge to and keeping knowledge documented at the site, and (iii) supporting the sites’ own personnel to take the lead in driving CCM implementation.

Regarding MMCS, we will continuously refine and improve the method by learning from other projects applying, testing, and critiquing MMCS. Outside of our CCM-related projects, examinations of implementation data using MMCS are actively underway for various implementation efforts including that of a data dashboard for decision support on transitioning psychiatrically stable patients from specialty mental health to primary care [ 2 ], a peer-led healthy lifestyle intervention for individuals with serious mental illness [ 3 ], screening programs for intimate partner violence [ 4 ], and a policy- and organization-based health system strengthening intervention to improve health systems in sub-Saharan Africa [ 5 ]. As MMCS is used by more projects that differ from one another in their specific outcome of interest, and especially in light of our MMCS application that examines factors related to sustainability, we are curious whether certain proximal to distal outcomes are more subject to heterogeneity in influencing factors than other outcomes. For instance, sustainability outcomes, which are tracked following a longer passage of time than some other outcomes, may be subject to more contextual variations that occur over time and thus could particularly benefit from being examined using MMCS. We will also explore MMCS’ complementarity with coincidence analysis and other configurational analytical approaches [ 68 ] for examining implementation phenomena. We are excited about both the step-by-step traceability that MMCS can bring to such methods and those methods’ computational algorithms that can be beneficial to incorporate into MMCS for projects with larger numbers of sites. For example, Salvati and colleagues [ 69 ] described both the inspiration that MMCS provided in structuring their data as well as how they addressed MMCS’ visualization shortcomings through their innovative data matrix heat mapping, which led to their selection of specific factors to include in their subsequent coincidence analysis. Coincidence analysis is an enhancement to qualitative comparative analysis and other configurational analytical methods, in that it is formulated specifically for causal inference [ 70 ]. Thus, in considering improved reformulations of MMCS’ steps to better characterize examined factors as explicit causes to the outcomes of interest, we are inspired by and can draw on coincidence analysis’ approach to building and evaluating causal chains that link factors to outcomes. Relatedly, we have begun to actively consider the potential contribution that MMCS can make to hypothesis generation and theory development for implementation science. As efforts to understand the mechanisms through which implementation strategies work are gaining momentum [ 71 , 72 , 73 ], there is an increased need for methods that help decompose our understanding of factors that influence the mechanistic pathways from strategies to outcomes [ 74 ]. Implementation science is facing the need to develop theories, beyond frameworks, which delineate hypotheses for observed implementation phenomena that can be subsequently tested [ 75 ]. The methodical approach that MMCS offers can aid this important endeavor, by enabling data curation and examination of pertinent factors in a consistent way that allows meaningful synthesis of findings across sites and studies. We see these future directions as concrete steps toward elucidating the factors related to sustainable implementation of EBPs, especially leveraging data from projects where the number of sites is much smaller than the number of factors that may matter—which is indeed the case for most implementation projects.

Using MMCS, we found that provider collaboration, knowledge retention during staff/leadership transitions, and availability of skilled internal facilitators may be most strongly related to CCM sustainability in VA outpatient mental health clinics. Informed by these findings, we have a subsequent CCM implementation trial underway to prospectively test whether increasing the aforementioned factors within implementation facilitation enhances sustainability. The MMCS steps used here for systematic multi-site examination can also be applied to determining sustainability-related factors relevant to various other EBPs and implementation contexts.

Availability of data and materials

The data analyzed during the current project are not publicly available because participant privacy could be compromised.

Abbreviations

Behavioral Health Interdisciplinary Program

Collaborative Chronic Care Model

Consolidated Criteria for Reporting Qualitative Research

coronavirus disease

evidence-based practice

Institutional Review Board

Integrated Promoting Action on Research Implementation in Health Services

Matrixed Multiple Case Study

United States Department of Veterans Affairs

Kim B, Sullivan JL, Ritchie MJ, Connolly SL, Drummond KL, Miller CJ, et al. Comparing variations in implementation processes and influences across multiple sites: What works, for whom, and how? Psychiatry Res. 2020;283:112520.

Article   PubMed   Google Scholar  

Hundt NE, Yusuf ZI, Amspoker AB, Nagamoto HT, Kim B, Boykin DM, et al. Improving the transition of patients with mental health disorders back to primary care: A protocol for a partnered, mixed-methods, stepped-wedge implementation trial. Contemp Clin Trials. 2021;105:106398.

Tuda D, Bochicchio L, Stefancic A, Hawes M, Chen J-H, Powell BJ, et al. Using the matrixed multiple case study methodology to understand site differences in the outcomes of a Hybrid Type 1 trial of a peer-led healthy lifestyle intervention for people with serious mental illness. Transl Behav Med. 2023;13(12):919–27.

Adjognon OL, Brady JE, Iverson KM, Stolzmann K, Dichter ME, Lew RA, et al. Using the Matrixed Multiple Case Study approach to identify factors affecting the uptake of IPV screening programs following the use of implementation facilitation. Implement Sci Commun. 2023;4(1):145.

Article   PubMed   PubMed Central   Google Scholar  

Seward N, Murdoch J, Hanlon C, Araya R, Gao W, Harding R, et al. Implementation science protocol for a participatory, theory-informed implementation research programme in the context of health system strengthening in sub-Saharan Africa (ASSET-ImplementER). BMJ Open. 2021;11(7):e048742.

Bauer MS, Miller C, Kim B, Lew R, Weaver K, Coldwell C, et al. Partnering with health system operations leadership to develop a controlled implementation trial. Implement Sci. 2016;11:22.

Bauer MS, Miller CJ, Kim B, Lew R, Stolzmann K, Sullivan J, et al. Effectiveness of implementing a Collaborative Chronic Care Model for clinician teams on patient outcomes and health status in mental health: a randomized clinical trial. JAMA Netw Open. 2019;2(3):e190230.

Ritchie MJ, Dollar KM, Miller CJ, Smith JL, Oliver KA, Kim B, et al. Using Implementation Facilitation to Improve Healthcare (Version 3): Veterans Health Administration, Behavioral Health Quality Enhancement Research Initiative (QUERI). 2020.

Google Scholar  

Bauer MS, Stolzmann K, Miller CJ, Kim B, Connolly SL, Lew R. Implementing the Collaborative Chronic Care Model in mental health clinics: achieving and sustaining clinical effects. Psychiatr Serv. 2021;72(5):586–9.

Miller CJ, Kim B, Connolly SL, Spitzer EG, Brown M, Bailey HM, et al. Sustainability of the Collaborative Chronic Care Model in outpatient mental health teams three years post-implementation: a qualitative analysis. Adm Policy Ment Health. 2023;50(1):151–9.

Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

Von Korff M, Gruman J, Schaefer J, Curry SJ, Wagner EH. Collaborative management of chronic illness. Ann Intern Med. 1997;127(12):1097–102.

Article   Google Scholar  

Wagner EH, Austin BT, Von Korff M. Organizing care for patients with chronic illness. Milbank Q. 1996;74(4):511–44.

Article   CAS   PubMed   Google Scholar  

Coleman K, Austin BT, Brach C, Wagner EH. Evidence on the chronic care model in the new millennium. Health Aff (Millwood). 2009;28(1):75–85.

Connolly SL, Sullivan JL, Ritchie MJ, Kim B, Miller CJ, Bauer MS. External facilitators’ perceptions of internal facilitation skills during implementation of collaborative care for mental health teams: a qualitative analysis informed by the i-PARIHS framework. BMC Health Serv Res. 2020;20(1):165.

Kim B, Sullivan JL, Drummond KL, Connolly SL, Miller CJ, Weaver K, et al. Interdisciplinary behavioral health provider perceptions of implementing the Collaborative Chronic Care Model: an i-PARIHS-guided qualitative study. Implement Sci Commun. 2023;4(1):35.

Harvey G, Kitson A. PARIHS revisited: from heuristic to integrated framework for the successful implementation of knowledge into practice. Implement Sci. 2016;11:33.

Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.

Sullivan JL, Kim B, Miller CJ, Elwy AR, Drummond KL, Connolly SL, et al. Collaborative Chronic Care Model implementation within outpatient behavioral health care teams: qualitative results from a multisite trial using implementation facilitation. Implement Sci Commun. 2021;2(1):33.

Miller CJ, Sullivan JL, Kim B, Elwy AR, Drummond KL, Connolly S, et al. Assessing collaborative care in mental health teams: qualitative analysis to guide future implementation. Adm Policy Ment Health. 2019;46(2):154–66.

Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook: sage. 1994.

Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311(7001):376–80.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Bens I. Facilitating with Ease!: core skills for facilitators, team leaders and members, managers, consultants, and trainers. Hoboken: John Wiley & Sons; 2017.

Ritchie MJ, Drummond KL, Smith BN, Sullivan JL, Landes SJ. Development of a qualitative data analysis codebook informed by the i-PARIHS framework. Implement Sci Commun. 2022;3(1):98.

Excel: Microsoft. Available from: https://www.microsoft.com/en-us/microsoft-365/excel . Accessed 15 Feb 2024.

Madrigal L, Manders OC, Kegler M, Haardörfer R, Piper S, Blais LM, et al. Inner and outer setting factors that influence the implementation of the National Diabetes Prevention Program (National DPP) using the Consolidated Framework for Implementation Research (CFIR): a qualitative study. Implement Sci Commun. 2022;3(1):104.

Wilson HK, Wieler C, Bell DL, Bhattarai AP, Castillo-Hernandez IM, Williams ER, et al. Implementation of the Diabetes Prevention Program in Georgia Cooperative Extension According to RE-AIM and the Consolidated Framework for Implementation Research. Prev Sci. 2023;Epub ahead of print.

Proctor E, Luke D, Calhoun A, McMillen C, Brownson R, McCrary S, et al. Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support. Implement Sci. 2015;10:88.

Fathi LI, Walker J, Dix CF, Cartwright JR, Joubert S, Carmichael KA, et al. Applying the Integrated Sustainability Framework to explore the long-term sustainability of nutrition education programmes in schools: a systematic review. Public Health Nutr. 2023;26(10):2165–79.

Guptill J. Knowledge management in health care. J Health Care Finance. 2005;31(3):10–4.

PubMed   Google Scholar  

Gammelgaard J. Why not use incentives to encourage knowledge sharing. J Knowledge Manage Pract. 2007;8(1):115–23.

Liebowitz J. Knowledge retention: strategies and solutions. Boca Raton: CRC Press; 2008.

Ensslin L, CarneiroMussi C, RolimEnsslin S, Dutra A, Pereira Bez Fontana L. Organizational knowledge retention management using a constructivist multi-criteria model. J Knowledge Manage. 2020;24(5):985–1004.

Peterson AE, Bond GR, Drake RE, McHugo GJ, Jones AM, Williams JR. Predicting the long-term sustainability of evidence-based practices in mental health care: an 8-year longitudinal analysis. J Behav Health Serv Res. 2014;41(3):337–46.

Miller CJ, Griffith KN, Stolzmann K, Kim B, Connolly SL, Bauer MS. An economic analysis of the implementation of team-based collaborative care in outpatient general mental health clinics. Med Care. 2020;58(10):874–80.

Silver SA, Harel Z, McQuillan R, Weizman AV, Thomas A, Chertow GM, et al. How to begin a quality improvement project. Clin J Am Soc Nephrol. 2016;11(5):893–900.

Dixon-Woods M. How to improve healthcare improvement-an essay by Mary Dixon-Woods. BMJ. 2019;367:l5514.

Miller CJ, Kim B, Silverman A, Bauer MS. A systematic review of team-building interventions in non-acute healthcare settings. BMC Health Serv Res. 2018;18(1):146.

Robert G, Greenhalgh T, MacFarlane F, Peacock R. Organisational factors influencing technology adoption and assimilation in the NHS: a systematic literature review. Report for the National Institute for Health Research Service Delivery and Organisation programme. London; 2009.

Kelly CJ, Young AJ. Promoting innovation in healthcare. Future Healthc J. 2017;4(2):121–5.

PubMed   PubMed Central   Google Scholar  

Aarons GA, Ehrhart MG, Farahnak LR, Hurlburt MS. Leadership and organizational change for implementation (LOCI): a randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implement Sci. 2015;10:11.

Ritchie MJ, Parker LE, Kirchner JE. Facilitating implementation of primary care mental health over time and across organizational contexts: a qualitative study of role and process. BMC Health Serv Res. 2023;23(1):565.

van den Hoed MW, Backhaus R, de Vries E, Hamers JPH, Daniëls R. Factors contributing to innovation readiness in health care organizations: a scoping review. BMC Health Serv Res. 2022;22(1):997.

Melnyk BM, Hsieh AP, Messinger J, Thomas B, Connor L, Gallagher-Ford L. Budgetary investment in evidence-based practice by chief nurses and stronger EBP cultures are associated with less turnover and better patient outcomes. Worldviews Evid Based Nurs. 2023;20(2):162–71.

Jacob RR, Parks RG, Allen P, Mazzucca S, Yan Y, Kang S, et al. How to “start small and just keep moving forward”: mixed methods results from a stepped-wedge trial to support evidence-based processes in local health departments. Front Public Health. 2022;10:853791.

Aarons GA, Conover KL, Ehrhart MG, Torres EM, Reeder K. Leader-member exchange and organizational climate effects on clinician turnover intentions. J Health Organ Manag. 2020;35(1):68–87.

Kirchner JE, Ritchie MJ, Pitcock JA, Parker LE, Curran GM, Fortney JC. Outcomes of a partnered facilitation strategy to implement primary care-mental health. J Gen Intern Med. 2014;29 Suppl 4(Suppl 4):904–12.

Strategy Design: CFIR research team-center for clinical management research. Available from: https://cfirguide.org/choosing-strategies/ . Accessed 15 Feb 2024.

Kim B, Wilson SM, Mosher TM, Breland JY. Systematic decision-making for using technological strategies to implement evidence-based interventions: an illustrated case study. Front Psychiatry. 2021;12:640240.

Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139.

Lewis CC, Scott K, Marriott BR. A methodology for generating a tailored implementation blueprint: an exemplar from a youth residential setting. Implement Sci. 2018;13(1):68.

Maher C, Hadfield M, Hutchings M, de Eyto A. Ensuring rigor in qualitative data analysis: a design research approach to coding combining NVivo with traditional material methods. Int J Qual Methods. 2018;17(1):1609406918786362.

Holloway I. A-Z of qualitative research in healthcare. 2nd ed. Oxford: Wiley-Blackwell; 2008.

Reproducibility and Replicability in Research: National Academies. Available from: https://www.nationalacademies.org/news/2019/09/reproducibility-and-replicability-in-research . Accessed 15 Feb 2024.

Chinman M, Acosta J, Ebener P, Shearer A. “What we have here, is a failure to [Replicate]”: ways to solve a replication crisis in implementation science. Prev Sci. 2022;23(5):739–50.

Vicente-Saez R, Martinez-Fuentes C. Open Science now: a systematic literature review for an integrated definition. J Bus Res. 2018;88:428–36.

Consolidated Framework for Implementation Research: CFIR Research Team-Center for Clinical Management Research. Available from: https://cfirguide.org/ . Accessed 15 Feb 2024.

Atkins L, Francis J, Islam R, O’Connor D, Patey A, Ivers N, et al. A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems. Implement Sci. 2017;12(1):77.

Luke DA, Calhoun A, Robichaux CB, Elliott MB, Moreland-Russell S. The Program Sustainability Assessment Tool: a new instrument for public health programs. Prev Chronic Dis. 2014;11:130184.

Calhoun A, Mainor A, Moreland-Russell S, Maier RC, Brossart L, Luke DA. Using the Program Sustainability Assessment Tool to assess and plan for sustainability. Prev Chronic Dis. 2014;11:130185.

Palinkas LA, Chou CP, Spear SE, Mendon SJ, Villamar J, Brown CH. Measurement of sustainment of prevention programs and initiatives: the sustainment measurement system scale. Implement Sci. 2020;15(1):71.

Sandelowski M. Real qualitative researchers do not count: the use of numbers in qualitative research. Res Nurs Health. 2001;24(3):230–40.

Wood M, Christy R. Sampling for Possibilities. Qual Quant. 1999;33(2):185–202.

Chang Y, Voils CI, Sandelowski M, Hasselblad V, Crandell JL. Transforming verbal counts in reports of qualitative descriptive studies into numbers. West J Nurs Res. 2009;31(7):837–52.

Yin RK. Case study research and applications. Los Angeles: Sage; 2018.

Bauer MS, Weaver K, Kim B, Miller C, Lew R, Stolzmann K, et al. The Collaborative Chronic Care Model for mental health conditions: from evidence synthesis to policy impact to scale-up and spread. Med Care. 2019;57 Suppl 10 Suppl 3(10 Suppl 3):S221-s7.

Miller CJ, Sullivan JL, Connolly SL, Richardson EJ, Stolzmann K, Brown ME, et al. Adaptation for sustainability in an implementation trial of team-based collaborative care. Implement Res Pract. 2024;5:26334895231226197.

Curran GM, Smith JD, Landsverk J, Vermeer W, Miech EJ, Kim B, et al. Design and analysis in dissemination and implementation research. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and Implementation Research in Health: Translating Science to Practice. 3 ed. New York: Oxford University Press; In press.

Salvati ZM, Rahm AK, Williams MS, Ladd I, Schlieder V, Atondo J, et al. A picture is worth a thousand words: advancing the use of visualization tools in implementation science through process mapping and matrix heat mapping. Implement Sci Commun. 2023;4(1):43.

Whitaker RG, Sperber N, Baumgartner M, Thiem A, Cragun D, Damschroder L, et al. Coincidence analysis: a new method for causal inference in implementation science. Implement Sci. 2020;15(1):108.

Lewis CC, Powell BJ, Brewer SK, Nguyen AM, Schriger SH, Vejnoska SF, et al. Advancing mechanisms of implementation to accelerate sustainable evidence-based practice integration: protocol for generating a research agenda. BMJ Open. 2021;11(10):e053474.

Kilbourne AM, Geng E, Eshun-Wilson I, Sweeney S, Shelley D, Cohen DJ, et al. How does facilitation in healthcare work? Using mechanism mapping to illuminate the black box of a meta-implementation strategy. Implement Sci Commun. 2023;4(1):53.

Kim B, Cruden G, Crable EL, Quanbeck A, Mittman BS, Wagner AD. A structured approach to applying systems analysis methods for examining implementation mechanisms. Implementation Sci Commun. 2023;4(1):127.

Geng EH, Baumann AA, Powell BJ. Mechanism mapping to advance research on implementation strategies. PLoS Med. 2022;19(2):e1003918.

Luke DA, Powell BJ, Paniagua-Avila A. Bridges and mechanisms: integrating systems science thinking into implementation research. Annu Rev Public Health. In press.

Download references

Acknowledgements

The authors sincerely thank the project participants for their time, as well as the project team members for their guidance and support. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.

This project was funded by VA grant QUE 20–026 and was designed and conducted in partnership with the VA Office of Mental Health and Suicide Prevention.

Author information

Authors and affiliations.

Center for Healthcare Organization and Implementation Research (CHOIR), VA Boston Healthcare System, 150 South Huntington Avenue, Boston, MA, 02130, USA

Bo Kim, Madisen E. Brown, Samantha L. Connolly, Elizabeth G. Spitzer, Hannah M. Bailey & Christopher J. Miller

Harvard Medical School, 25 Shattuck Street, Boston, MA, 02115, USA

Bo Kim, Samantha L. Connolly & Christopher J. Miller

Center of Innovation in Long Term Services and Supports (LTSS COIN), VA Providence Healthcare System, 385 Niagara Street, Providence, RI, 02907, USA

Jennifer L. Sullivan

Brown University School of Public Health, 121 South Main Street, Providence, RI, 02903, USA

VA Rocky Mountain Mental Illness Research, Education and Clinical Center (MIRECC), 1700 N Wheeling Street, Aurora, CO, 80045, USA

Elizabeth G. Spitzer

VA Northeast Program Evaluation Center, 950 Campbell Avenue, West Haven, CT, 06516, USA

Lauren M. Sippel

Geisel School of Medicine at Dartmouth, 1 Rope Ferry Road, Hanover, NH, 03755, USA

VA Office of Mental Health and Suicide Prevention, 810 Vermont Avenue NW, Washington, DC, 20420, USA

Kendra Weaver

You can also search for this author in PubMed   Google Scholar

Contributions

Concept and design: BK, JS, and CM. Acquisition, analysis, and/or interpretation of data: BK, JS, MB, SC, ES, and CM. Initial drafting of the manuscript: BK. Critical revisions of the manuscript for important intellectual content: JS, MB, SC, ES, HB, LS, KW, and CM. All the authors read and approved the final manuscript.

Corresponding author

Correspondence to Bo Kim .

Ethics declarations

Ethics approval and consent to participate.

This project was determined to be non-research by the VA Boston Research and Development Service, and therefore did not require oversight by the Institutional Review Board (IRB).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

COREQ (COnsolidated criteria for REporting Qualitative research) Checklist.

Additional file 2.

Data input, tasks performed, and analysis output for MMCS Steps 5 through 9.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Kim, B., Sullivan, J.L., Brown, M.E. et al. Sustaining the collaborative chronic care model in outpatient mental health: a matrixed multiple case study. Implementation Sci 19 , 16 (2024). https://doi.org/10.1186/s13012-024-01342-2

Download citation

Received : 14 June 2023

Accepted : 21 January 2024

Published : 19 February 2024

DOI : https://doi.org/10.1186/s13012-024-01342-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Collaborative care
  • Implementation
  • Interdisciplinary care
  • Mental health
  • Sustainability

Implementation Science

ISSN: 1748-5908

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

multiple case study individual

The HBCU Center

Hbcu students’ basic needs and capacity building: a multiple case study exploration.

Funded by the ECMC Foundation, the project “ HBCU Students’ Basic Needs and Capacity Building: A Multiple Case Study Exploration ” aids the HBCU Center’s mission to support a thriving HBCU ecosystem by developing research-informed strategies to better serve the holistic needs of students. According to Dahl and colleagues (2022), nearly two thirds of HBCU students experienced basic needs insecurity during the pandemic, and nearly half of these students faced food and housing insecurity within that year.

Guided by ECMC’s Foundation's mission to “improve higher education for career success among underserved populations through evidence-based innovation,” the purpose of this study is to conduct a multi-campus exploration focused on HBCU undergraduate students’ basic needs. This study explores campus policies, interventions, and practices that promote HBCU students’ holistic wellbeing.

The following are the the expected outcomes of this study:

- To identify today’s most pressing HBCU students’ basic needs

- To evaluate the infrastructure of HBCUs for serving their students’ basic needs.

- To build capacity across HBCUs through research-informed practices

- To ignite coalition among HBCUs committed to serving their students’ needs holistically

Other key aspects about this study include:

- The HBCUs targeted for this study are mostly residential, research-intensive, and bachelor’s degree granting. This study is seeking HBCUs with undergraduate enrollment of 25% or more of students who are Pell-grant recipients.

- Through individual interviews and focus groups, data will be collected from the following:

- Staff , particularly frontline staff (e.g., student affairs professionals)

- Faculty , particularly those who teach core/signature/general education courses

- Select senior administrations , including but not limited to vice president of student affairs and chief academic officer.

Efforts will rely on campus liaisons who will provide assistance on identifying participants who fit the criteria for this research. Campus liaisons will receive a one-time honorarium of $1,500 for their participation. If you wish to enlist your institution and/or serve as a campus liaison for this study, please contact the PI, Dr. Jorge Burmicky at [email protected]

IRB protocol #2024-1257: HBCU Students' Basic Needs and Capacity Building

The Cases Against Trump: A Guide

Fraud. Hush money. Election subversion. Mar-a-Lago documents. One place to keep track of the presidential candidate’s legal troubles.

Arrows pointing at Donald Trump

Not long ago, the idea that a former president—or major-party presidential nominee—would face serious legal jeopardy was nearly unthinkable. Today, merely keeping track of the many cases against Donald Trump requires a law degree, a great deal of attention, or both.

In all, Trump faces 91 felony counts across two state courts and two different federal districts, any of which could potentially produce a prison sentence. He’s also dealing with a civil suit in New York that could force drastic changes to his business empire, including closing down its operations in his home state. Meanwhile, he is the leading Republican candidate in the race to become the next president—though the Supreme Court has now heard a case seeking to disqualify him. If the criminal and civil cases unfold with any reasonable timeliness, he could be in the heat of the campaign at the same time that his legal fate is being decided.

David A. Graham: The end of Trump Inc.

Here’s a summary of the major legal cases against Trump, including key dates, an assessment of the gravity of the charges, and expectations about how they could turn out. This guide will be updated regularly as the cases proceed.

New York State: Fraud

In the fall of 2022, New York Attorney General Letitia James filed a civil suit against Trump, his adult sons, and his former aide Allen Weisselberg, alleging a years-long scheme in which Trump fraudulently reported the value of properties in order to either lower his tax bill or improve the terms of his loans, all with an eye toward inflating his net worth.

When? Justice Arthur Engoron ruled against Trump and his co-defendants in late September 2023, concluding that many of the defendants’ claims were “clearly” fraudulent—so clearly that he didn’t need a trial to hear them. (He also sanctioned Trump’s lawyers for making repeated frivolous arguments.) Engoron has also fined Trump a total of $15,000 for violating a gag order in the case. The trial ended in January, and a ruling is currently expected in mid-February .

How grave is the allegation? Fraud is fraud, and in this case, the sum of the fraud stretched into the millions—but compared with some of the other legal matters in which Trump is embroiled, this is pretty pedestrian. The case is also civil rather than criminal. But although the stakes are lower for the nation, they remain high for Trump: Engoron could bar Trump’s famed company from business in New York, strip it of several key properties, and fine Trump hundreds of millions of dollars.

How plausible is a guilty verdict? Engoron has already ruled that Trump committed fraud. The outstanding questions are what damages he might have to pay and what exactly Engoron’s ruling means for Trump’s business and properties in New York.

Manhattan: Defamation and Sexual Assault

Although these other cases are all brought by government entities, Trump also faced a pair of defamation suits from the writer E. Jean Carroll, who said that Trump sexually assaulted her in a department-store dressing room in the 1990s. When he denied it, she sued him for defamation and later added a battery claim.

When? In May 2023, a jury concluded that Trump had sexually assaulted and defamed Carroll, and awarded her $5 million. A second defamation case produced an $83.3 million judgment in January 2024.

How grave was the allegation? Although these cases don’t directly connect to the same fundamental issues of rule of law and democratic governance that some of the criminal cases do, they were a serious matter, and a federal judge’s blunt statement that Trump raped Carroll has gone underappreciated.

What happens now? Trump has appealed both cases. During the second trial, he also continued to insult Carroll, which may have courted additional defamation suits.

Manhattan: Hush Money

In March 2023, Manhattan District Attorney Alvin Bragg became the first prosecutor to bring felony charges against Trump, alleging that the former president falsified business records as part of a scheme to pay hush money to women who said they had had sexual relationships with Trump.

When? The case is set to go to trial on March 25, Judge Juan Merchant said on February 15.

How grave is the allegation? Falsifying records is a crime, and crime is bad. But many people have analogized this case to Al Capone’s conviction on tax evasion: It’s not that he didn’t deserve it, but it wasn’t really why he was an infamous villain. That this case alleges behavior that didn’t directly attack elections or put national secrets at risk makes it feel more minor—in part because other cases have set a grossly high standard for what constitutes gravity.

How plausible is a guilty verdict? Bragg’s case faces hurdles including arguments over the statute of limitations, a questionable key witness in the former Trump fixer Michael Cohen, and some fresh legal theories. In short, the Manhattan case seems like perhaps the least significant and most tenuous criminal case. Some Trump critics were dismayed that Bragg was the first to bring criminal charges against the former president.

Department of Justice: Mar-a-Lago Documents

Jack Smith, a special counsel in the U.S. Justice Department, has charged Trump with 37 felonies in connection with his removal of documents from the White House when he left office. The charges include willful retention of national-security information, obstruction of justice, withholding of documents, and false statements. Trump took boxes of documents to properties where they were stored haphazardly, but the indictment centers on his refusal to give them back to the government despite repeated requests.

David A. Graham: This indictment is different

When? Smith filed charges in June 2023. Judge Aileen Cannon has set a trial date of May 20, 2024. In November, she rejected Trump’s request to push that back but said she would reconsider timing in March . Smith faces a de facto deadline of January 20, 2025, at which point Trump or any Republican president would likely shut down a case.

How grave is the allegation? These are, I have written, the stupidest crimes imaginable , but they are nevertheless very serious. Protecting the nation’s secrets is one of the greatest responsibilities of any public official with classified clearance, and not only did Trump put these documents at risk, but he also (allegedly) refused to comply with a subpoena, tried to hide them, and lied to the government through his attorneys.

How plausible is a guilty verdict? This may be the most open-and-shut case, and the facts and legal theory here are pretty straightforward. But Smith seems to have drawn a short straw when he was randomly assigned Cannon, a Trump appointee who has sometimes ruled favorably for Trump on procedural matters. Some legal commentators have even accused her of “ sabotaging ” the case.

Fulton County: Election Subversion

In Fulton County, Georgia, which includes most of Atlanta, District Attorney Fani Willis brought a huge racketeering case against Trump and 18 others, alleging a conspiracy that spread across weeks and states with the aim of stealing the 2020 election.

When? Willis obtained the indictment in August 2023. The number of people charged makes the case unwieldy and difficult to track. Several of them, including Kenneth Chesebro , Sidney Powell , and Jenna Ellis, struck plea deals in the fall. Willis has proposed a trial date of August 5, 2024, for the remaining defendants.

How grave is the allegation? More than any other case, this one attempts to reckon with the full breadth of the assault on democracy following the 2020 election.

How plausible is a guilty verdict? Expert views differ. This is a huge case for a local prosecutor, even in a county as large as Fulton, to bring. The racketeering law allows Willis to sweep in a great deal of material, and she has some strong evidence—such as a call in which Trump asked Georgia Secretary of State Brad Raffensperger to “find” some 11,000 votes. Three major plea deals from co-defendants may also ease Willis’s path, but getting a jury to convict Trump will still be a challenge. Complicating matters, Willis is now under fire for a romantic relationship with an attorney she hired as a special prosecutor.

Department of Justice: Election Subversion

Special Counsel Smith has also charged Trump with four federal felonies in connection with his attempt to remain in power after losing the 2020 election. This case is in court in Washington, D.C.

When? A grand jury indicted Trump on August 1, 2023. The trial was originally schedule for March 4, but Judge Tanya Chutkan said in early February that the date would change, as an appeals court deliberated on Trump’s claim of absolute immunity. A three-judge panel roundly rejected that claim on February 6, but no new trial date has been announced yet. As with the other DOJ case, Smith will need to move quickly, before Trump or any other Republican president could shut down a case upon taking office in January 2025. Other tangential legal skirmishes continue: In October, after verbal attacks by Trump on witnesses and Smith’s wife, Chutkan issued an order limiting what Trump can say about the case.

David A. Graham: Trump attempted a brazen, dead-serious attack on American democracy

How grave is the allegation? This case rivals the Fulton County one in importance. It is narrower, focusing just on Trump and a few key elements of the paperwork coup , but the symbolic weight of the U.S. Justice Department prosecuting an attempt to subvert the American election system is heavy.

How plausible is a guilty verdict? It’s very hard to say. Smith avoided some of the more unconventional potential charges, including aiding insurrection, and everyone watched much of the alleged crime unfold in public in real time, but no precedent exists for a case like this, with a defendant like this.

Additionally …

In more than 30 states , cases have been filed over whether Trump should be thrown off the 2024 ballot under a novel legal theory about the Fourteenth Amendment. Proponents, including J. Michael Luttig and Laurence H. Tribe in The Atlantic , argued that the former president is ineligible to serve again under a clause that disqualifies anyone who took an oath defending the Constitution and then subsequently participated in a rebellion or an insurrection. They said that Trump’s attempt to steal the 2020 election and his incitement of the January 6 riot meet the criteria.

Cases were brought in many states, and state authorities issued conflicting opinions. Several states ruled against removing Trump from the ballot, but the Colorado Supreme Court and the Maine secretary of state both disqualified him, ruling that he had engaged in an insurrection—a remarkable legal finding. Trump then appealed to the U.S. Supreme Court.

When? The U.S. Supreme Court heard arguments in the case on February 8. The timing for a decision is not clear.

How grave is the allegation? In a sense, the claim made here is even graver than the criminal election-subversion cases filed against Trump by the U.S. Department of Justice and in Fulton County, Georgia, because neither of those cases alleges insurrection or rebellion. But the stakes are also much different—rather than criminal conviction, they concern the ability to serve as president.

How plausible is a disqualification? Though there is a robust debate among legal scholars on this question, the nine who matter are the ones on the Supreme Court, and they appeared very skeptical of arguments in favor of disqualification during the February 8 hearing.

Book cover

First Year in a Multilingual University pp 31–53 Cite as

Methodology: Multiple-Case Qualitative Study

  • Feng Ding 2  
  • First Online: 01 July 2021

228 Accesses

This longitudinal multiple-case study is qualitative in nature, with data collection lasting for more than a year. As this study set out to understand the experiences, feelings, emotions, conceptions, and understanding of students about their experiences, a qualitative research approach was adopted to “identify issues from the perspective of” the participants, and to study them in a natural setting (Hennink et al., 2011, p. 9).

This is a preview of subscription content, log in via an institution .

Buying options

  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
  • Available as EPUB and PDF
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
  • Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Apple Newspaper, 28/04/2013, http://hk.apple.nextmedia.com/news/art/20130428/18242744

Source: http://life.dayoo.com/edu/113887/201301/22/113887_28581159.htm

University of Hong Kong Public Poll, 2012. http://hkupop.hku.hk/english/release/release937.html

Buddy writer, 2010-1011 http://web02.carmelss.edu.hk/buddingwriters/index.php?option=com_content&view=article&id=971:importing-students-from-the-mainland-brings-more-harm-than-good-to-hong-kong&catid=109&Itemid=66

Bazeley, P., & Richards, L. (2000). The NVivo qualitative project book . Sage.

Book   Google Scholar  

Bogdan, R. C., & Biklen, S. K. (2007). Qualitative research for education. An introduction to theory and methods . Allyn and Bacon.

Google Scholar  

Chee, W. C. (2012). Envisioned belonging: Cultural differences and ethnicities in Hong Kong schooling. Asian Anthropology (1683478X), 11 (1), 89–105.

Article   Google Scholar  

Creswell, J. W. (2007). Qualitative inquiry and research design: Choosing among five approaches . Sage.

Denzin, N. K., & Lincoln, Y. S. (2000). Handbook of qualitative research . Sage Publications.

Ding, F., & Stapleton, P. (2015). Self-emergent peer support using online social networking during cross-border transition. Australasian Journal of Educational Technology, 31 (6), 671–684.

Dörnyei, Z. (2007). Research methods in applied linguistics: Quantitative, qualitative, and mixed methodologies . Oxford University Press.

Evans, S. (2011). Hong Kong English and the professional world. World Englishes, 30 (3), 293–316.

Gao, X. (2010). Strategic language learning: The roles of agency and context . Multilingual Matters.

Hennink, M., Hutter, I., & Bailey, A. (2011). Qualitative research methods . Sage.

Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Sage.

Ministry of Education. (2007). College English curriculum requirements . Foreign Language Teaching and Research Press.

Patton, M. (1990). Qualitative evaluation and research methods . Sage Publications.

Patton, M. Q. (2002). Qualitative research and evaluation methods . Sage Publications.

Reason, R., Terenzini, P., & Domingo, R. (2006). First things first: Developing academic competence in the first year of college. Research in Higher Education, 47 (2), 149–175.

Stake, R. E. (2006). Multiple case study analysis . Guilford Press.

Strauss, A. L., & Corbin, J. M. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory (2nd ed.). Sage Publications.

Thaler, R. H., & Sunstein, C. R. (2009). Nudge: Improving decisions about health, wealth, and happiness . Penguin Books.

Willis, J. (2007). Qualitative research methods in education and educational technology . Sage Publications.

Xie, C. X. (2010). Mainland Chinese students’ adjustment to studying and living in Hong Kong . Unpublished doctoral dissertation of the University of Leicester.

Yin, R. (1994). Case study research: Design and methods . Sage.

Yin, R. (2003). Case study research: Design and methods . Sage.

Download references

Author information

Authors and affiliations.

Guangdong University of Foreign Studies, Guangzhou, Guangdong, China

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Singapore Pte Ltd.

About this chapter

Cite this chapter.

Ding, F. (2021). Methodology: Multiple-Case Qualitative Study. In: First Year in a Multilingual University. Springer, Singapore. https://doi.org/10.1007/978-981-16-0796-7_3

Download citation

DOI : https://doi.org/10.1007/978-981-16-0796-7_3

Published : 01 July 2021

Publisher Name : Springer, Singapore

Print ISBN : 978-981-16-0795-0

Online ISBN : 978-981-16-0796-7

eBook Packages : Education Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Publish with us

Policies and ethics

  • Find a journal
  • Track your research

IMAGES

  1. Multiple Case Study Method

    multiple case study individual

  2. Overview of multiple case study (Adapted from [Yin, 2014])

    multiple case study individual

  3. PPT

    multiple case study individual

  4. Stages in multi-case study analysis

    multiple case study individual

  5. Multiple Case Study Analysis

    multiple case study individual

  6. FREE 9+ Case Study Analysis Samples in PDF

    multiple case study individual

VIDEO

  1. Case Studies

  2. Case Studies

  3. Day-2 Topic: Case Study Method for better Teaching

  4. Case Studies

  5. CASE STUDY COM165 VIDEO PRESENTATION

  6. Case study practical

COMMENTS

  1. Multiple Case Studies

    To write a multiple-case study, a summary of individual cases should be reported, and researchers need to draw cross-case conclusions and form a cross-case report (Yin, 2017). With evidence from multiple cases, researchers may have generalizable findings and develop theories (Lewis-Beck, Bryman & Liao, 2003).

  2. Multiple Case Research Design

    A multiple case research design shifts the focus from understanding a single case to the differences and similarities between cases. Thus, it is not just conducting more (second, third, etc.) case studies. Rather, it is the next step in developing a theory about factors driving differences and similarities.

  3. Case Study Methodology of Qualitative Research: Key Attributes and

    In a case study research, multiple methods of data collection are used, as it involves an in-depth study of a phenomenon. It must be noted, as highlighted by Yin ( 2009 ), a case study is not a method of data collection, rather is a research strategy or design to study a social unit.

  4. PDF 9 Multiple Case Research Design

    With a multiple case study, researchers conduct an in-depth analysis of several cases. First, an investigation of individual cases is con-ducted. Later, these individual results are combined. Researchers try to find similarities and differences.

  5. Planning Qualitative Research: Design and Decision Making for New

    In a multiple case study, there is a commonality among the cases, enabling researchers to analyze and find specific differences that are replicable or transferable to similar contexts. ... The narrative approach focuses on the meaning an individual attaches to an experience through the process of storytelling (Esin et al., 2014) and exposes a ...

  6. Case Study Method: A Step-by-Step Guide for Business Researchers

    The multiple case studies used in this article as an application of step-by-step guideline are specifically designed to facilitate these business and management researchers. This article presents an easy to read, practical, experience-based, step-by-step guided path to select, conduct, and complete the qualitative case study successfully.

  7. What Is a Case Study?

    Step 1: Select a case Once you have developed your problem statement and research questions, you should be ready to choose the specific case that you want to focus on. A good case study should have the potential to: Provide new or unexpected insights into the subject Challenge or complicate existing assumptions and theories

  8. The Multiple Case Study Design

    The multiple case study design is a valuable qualitative research tool in studying the links between the personal, social, behavioral, psychological, organizational, cultural, and environmental factors that guide organizational and leadership development.

  9. Case Study

    A multiple-case study involves the analysis of several cases that are similar in nature. This type of case study is useful when the researcher wants to identify similarities and differences between the cases.

  10. PDF Kurt Schoch I

    A case study also gives you an opportunity to collect different kinds of data, such as interviews, documents, observations, surveys, and others, about the case and provides you with the chance to get an in-depth look at an organiza-tion or individual and the inner workings and interactions of that organization or individual.

  11. Single case studies vs. multiple case studies: A comparative study

    Because of different reasons the case studies can be either single or multiple. This study attempts to answer when to write a single case study and when to write a multiple case study. It will further answer the benefits and disadvantages with the different types. The literature review, which is based on secondary sources, is about case studies.

  12. Multiple Case Research Design

    A multiple-case research design shifts the focus from understanding a single case to the differences and similarities between cases. Thus, it is more than conducting more case studies (second, third, etc.). Instead, it is the next step in developing a theory about factors driving differences and similarities.

  13. Individual Growth Through Forgiveness: A Multiple Case Study on the

    multiple case study was to examine an individual's process of forgiveness and how it was achieved following the experience of a significant transgression. Nine participants agreed to ... individual's positive features that function as a pathway toward a more positive life (Park et al., 2004). Forgiveness is also recognized as a coping ...

  14. Case Study: Definition, Examples, Types, and How to Write

    The three main case study types often used are intrinsic, instrumental, and collective. Intrinsic case studies are useful for learning about unique cases. Instrumental case studies help look at an individual to learn more about a broader issue. A collective case study can be useful for looking at several cases simultaneously.

  15. Methodology: Multiple-Case Qualitative Study

    The multiple-case approach was used to understand the complexity of the experience of nine mainland Chinese students in their school-university and cross-border transitions, their problems, and reasons behind the complexity.

  16. (PDF) Developing the PhD thesis project in relation to individual

    The results of this small-scale longitudinal multiple case study of five first-year UK PhD social science researchers suggest that there are at least three approaches PhD researchers may adopt in ...

  17. Constructivist Stakian Multicase Study: Methodological Issues

    The multiple definitions of case study research tend to have two major components in common: a real-life particular bounded context, and a level of complexity requiring holistic and in-depth exploration. ... Multi-case assertions and the quantification of an individual case's utility appear relatively positivist and reductionist in nature ...

  18. PDF Mixed Methods Case Study Research

    The Mixed Methods Case Study Purpose: •a multiple case study design. Each case was selected as a tool to illuminating a particular issue"(p.101). The case study was instrumental. Bound •"Each case study was bounded by one individual and by the time he or she matriculated in the ELHE program" (p.101). Case Selection

  19. How to Write a Multiple Case Study Effectively

    1. Brainstorm Potential Case Studies 2. Conduct Background Research on Each Case 3. Establish a Research Methodology 4. Collect Data 5. Analyze & Interpret Data 6. Write the Report Conclusion Have you ever been assigned to write a multiple case study but don't know where to begin? Are you intimidated by the complexity and challenge it brings?

  20. Support and process in individual placement and support: a multiple

    Objective: This multiple case study investigated support and process in the Individual Placement and Support (IPS) approach from individual client, longitudinal, and Person-Environment-Occupation (PEO) model perspectives. Participants: Five IPS-participants, or cases, with severe mental illness (SMI) who worked a minimum of 4 hours a week entered the study.

  21. Integrative analysis of multiple case-control studies

    We consider a setting where researchers have collected individual-level data in their own study (the internal study), and in the meantime can acquire summary data from published literature or other studies (external studies). Since the case-control sampling design is most commonly used for studying a binary disease outcome (Breslow and Day ...

  22. Sustaining the collaborative chronic care model in outpatient mental

    Sustaining evidence-based practices (EBPs) is crucial to ensuring care quality and addressing health disparities. Approaches to identifying factors related to sustainability are critically needed. One such approach is Matrixed Multiple Case Study (MMCS), which identifies factors and their combinations that influence implementation. We applied MMCS to identify factors related to the ...

  23. Research Approach: Multiple-Case Study

    Multiple-case studies provide an even more powerful base for theory building than single-case studies, as they allow for replication and extension between cases, and thus comparison. The replication of case studies enables the researcher to affirm certain propositions by examining several individual cases.

  24. HBCU Students' Basic Needs and Capacity Building: A Multiple Case Study

    - The HBCUs targeted for this study are mostly residential, research-intensive, and bachelor's degree granting. This study is seeking HBCUs with undergraduate enrollment of 25% or more of students who are Pell-grant recipients. - Through individual interviews and focus groups, data will be collected from the following: - Students

  25. Remote Sensing

    The GeoDetector was employed to ascertain individual or interactive effects of natural and anthropogenic factors on these ESs and their trade-offs/synergies. ... and Huan Deng. 2024. "Mapping and Analyzing the Spatiotemporal Patterns and Drivers of Multiple Ecosystem Services: A Case Study in the Yangtze and Yellow River Basins" Remote Sensing ...

  26. The Cases Against Trump: A Guide

    February 15, 2024, 1:28 PM ET. Not long ago, the idea that a former president—or major-party presidential nominee—would face serious legal jeopardy was nearly unthinkable. Today, merely ...

  27. PDF Chapter 3 Methodology: Multiple-Case Qualitative Study

    According to Stake (2006), multiple cases are studied to investigate and understand a more general matter. In this study, each participant is a case. The multiple-case approach was used to understand the complexity of the experience of nine mainland Chinese students in their school-university and cross-border transitions, their prob-