research paper on system analysis

• Conference proceedings
• © 2011

Research in Systems Analysis and Design: Models and Methods

4th SIGSAND/PLAIS EuroSymposium 2011, Gdańsk, Poland, September 29, 2011, Revised Selected Papers

• Stanisław Wrycza 0

Department of Business Informatics, University of Gdansk, Sopot, Poland

You can also search for this editor in PubMed   Google Scholar

• Fast track proceedings

Part of the book series: Lecture Notes in Business Information Processing (LNBIP, volume 93)

Conference series link(s): PLAIS EuroSymposium: PLAIS EuroSymposium on Digital Transformation

34k Accesses

59 Citations

23 Altmetric

Conference proceedings info: SIGSAND/PLAIS 2011.

• Table of contents
• Other volumes

About this book

Editors and affiliations, bibliographic information.

• Publish with us

Buying options

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Other ways to access

This is a preview of subscription content, log in via an institution to check for access.

Table of contents (9 papers)

Front matter, business processes modeling, reengineering university: modeling business processes to achieve strategic goals.

• Aleksey Shutov

SysML Requirement Diagrams: Banking Transactional Platform Case Study

• Stanislaw Wrycza, Bartosz Marcinkowski

Integrated Systems Development

Customer knowledge management models: assessment and proposal.

• Dorota Buchnowska

ERP in the Cloud – Benefits and Challenges

• Anna Lenart

Building Project Teams in Enterprise System Adoption: The Need for the Incorporation of the Project Type

Model of information systems’ selection for the company management.

• Iryna Zolotaryova, Anna Khodyrevska

Software Development

A method to discover trend reversal patterns using behavioral data.

• Jerzy Korczak, Aleksander Fafuła

Toward a Theory of Debiasing Software Development

Obcas - an ontology-based cluster analysis system.

• Janusz Tuchowski, Katarzyna Wójcik, Paweł Lula, Grażyna Paliwoda-Pękosz

Back Matter

Other volumes.

This book constitutes the proceedings of the 4th EuroSymposium on Systems Analysis and Design, SIGSAND/PLAIS 2011, held in Gdańsk, Poland, in September 2011.

The objective of this symposium is to promote and develop high-quality research on all issues related to systems analysis and design (SAND). It provides a forum for SAND researchers and practitioners in Europe and beyond to interact, collaborate, and develop their field.

The 9 papers were carefully reviewed and selected from 20 submissions. An additional revision took place after the conference to incorporate discussion results from the presentation. The contributions are organized into topical sections on business process modeling, integrated systems development, and software development.

• business process modeling
• enterprise systems
• requirements engineering
• software development
• systems development

Stanisław Wrycza

Book Title : Research in Systems Analysis and Design: Models and Methods

Book Subtitle : 4th SIGSAND/PLAIS EuroSymposium 2011, Gdańsk, Poland, September 29, 2011, Revised Selected Papers

Editors : Stanisław Wrycza

Series Title : Lecture Notes in Business Information Processing

DOI : https://doi.org/10.1007/978-3-642-25676-9

Publisher : Springer Berlin, Heidelberg

eBook Packages : Computer Science , Computer Science (R0)

Copyright Information : Springer-Verlag GmbH Berlin Heidelberg 2011

Softcover ISBN : 978-3-642-25675-2 Published: 02 December 2011

eBook ISBN : 978-3-642-25676-9 Published: 29 November 2011

Series ISSN : 1865-1348

Series E-ISSN : 1865-1356

Edition Number : 1

Number of Pages : IX, 113

Topics : IT in Business , Computer Appl. in Administrative Data Processing , Information Systems Applications (incl. Internet) , Software Engineering

Policies and ethics

• Find a journal
• Track your research

• Methodology
• Open access
• Published: 19 October 2023

A structured approach to applying systems analysis methods for examining implementation mechanisms

• Bo Kim   ORCID: orcid.org/0000-0001-7730-1627 1 , 2 ,
• Gracelyn Cruden 3 ,
• Erika L. Crable 4 , 5 , 6 ,
• Andrew Quanbeck 7 ,
• Brian S. Mittman 8 , 9 , 10 &
• Anjuli D. Wagner 11  

Implementation Science Communications volume  4 , Article number:  127 ( 2023 ) Cite this article

1850 Accesses

1 Citations

10 Altmetric

Metrics details

It is challenging to identify and understand the specific mechanisms through which an implementation strategy affects implementation outcomes, as implementation happens in the context of complex, multi-level systems. These systems and the mechanisms within each level have their own dynamic environments that change frequently. For instance, sequencing may matter in that a mechanism may only be activated indirectly by a strategy through another mechanism. The dosage or strength of a mechanism may vary over time or across different health care system levels. To elucidate the mechanisms relevant to successful implementation amidst this complexity, systems analysis methods are needed to model and manage complexity.

The fields of systems engineering and systems science offer methods—which we refer to as systems analysis methods—to help explain the interdependent relationships between and within systems, as well as dynamic changes to systems over time. When applied to studying implementation mechanisms, systems analysis methods can help (i) better identify and manage unknown conditions that may or may not activate mechanisms (both expected mechanisms targeted by a strategy and unexpected mechanisms that the methods help detect) and (ii) flexibly guide strategy adaptations to address contextual influences that emerge after the strategy is selected and used.

In this paper, we delineate a structured approach to applying systems analysis methods for examining implementation mechanisms. The approach includes explicit steps for selecting, tailoring, and evaluating an implementation strategy regarding the mechanisms that the strategy is initially hypothesized to activate, as well as additional mechanisms that are identified through the steps. We illustrate the approach using a case example. We then discuss the strengths and limitations of this approach, as well as when these steps might be most appropriate, and suggest work to further the contributions of systems analysis methods to implementation mechanisms research.

Conclusions

Our approach to applying systems analysis methods can encourage more mechanisms research efforts to consider these methods and in turn fuel both (i) rigorous comparisons of these methods to alternative mechanisms research approaches and (ii) an active discourse across the field to better delineate when these methods are appropriate for advancing mechanisms-related knowledge.

Peer Review reports

Contributions to the literature

We offer a four-step approach to applying systems analysis methods for identifying, specifying, testing, and refining the understanding of implementation mechanisms that need to be activated for implementation strategies to lead to desirable implementation, service, and clinical outcomes.

Systems analysis methods can capture and reflect synergistic, antagonistic, or other non-additive patterns of co-occurrence, especially for multiple strategies that target distinct mechanisms and are thus bundled to implement interventions into multi-level systems (which are common in health care and community settings).

Knowledge of such patterns is particularly crucial for implementation that must carefully plan how to allocate resources across different strategies.

The field of implementation science pursues knowledge regarding methods that successfully promote the uptake of evidence-based interventions into routine practice [ 1 ]. Such methods tested by the field are implementation strategies, which are “techniques used to enhance the adoption, implementation, and sustainability” of an intervention [ 2 ]. An implementation strategy’s success often varies across different implementation contexts [ 3 ]—i.e., the success of an implementation strategy is influenced by who uses the strategy, for what purpose it is employed, and also when, where, and how the strategy is used. There is a growing interest in understanding the specific mechanisms through which an implementation strategy functions to achieve optimal implementation outcomes [ 4 ]. In other words, enhancing our knowledge of mechanisms is critical to learning why a specific implementation strategy works or not in moving evidence into practice within a given context. Failure to learn why implementation strategies work may result in the use of sub-optimal strategies that do not activate the desired process or inadvertently activate unplanned processes, leading to unintended and undesirable implementation and health outcomes.

Lewis and colleagues [ 4 ] define an implementation mechanism as a “process or event through which an implementation strategy operates to affect desired implementation outcomes.” Mechanisms include moderators (factors that affect the strength and direction of a relationship) and mediators (factors that sit between the strategy and the outcome and can account for an observed relationship between the two). Notably, not all mediators will serve as mechanisms. Table 1 summarizes the implementation mechanism-related terms as defined by Lewis and colleagues.

In Lewis and colleagues’ seminal article, they describe how a “training” implementation strategy can increase clinicians’ use of an evidence-based intervention because it works through the “skill building” mechanism. In this example (which we will refer to and further explain under the “ Steps to apply systems analysis methods for studying implementation mechanisms ” section), this mechanism is meant to increase the clinicians’ use of the focal evidence-based practice. When a chosen implementation strategy does not achieve the desired implementation outcomes (i.e., clinicians’ use does not increase), it could be that (i) appropriate preconditions were not met for the mechanism to take place (or “be activated”; e.g., clinicians’ work schedules did not allow them to attend training sessions), (ii) preconditions were met but other factors attenuated the strategy’s impact (e.g., clinicians’ low desire to learn), or (iii) there were additional variables along the causal pathway that hindered the strategy’s impact (e.g., skills were built, but not the confidence, to use the intervention).

Understanding relevant implementation mechanisms and their associated preconditions, mediators, and moderators is challenged by the complexity of health care systems [ 5 ]. Complex systems have numerous components that dynamically change over time, exhibiting behaviors that influence an implementation strategy’s success. For instance, in the training implementation strategy example above, group training sessions instead of one-on-one sessions may increase or decrease clinicians’ desire to learn through a mechanism such as social learning. If these changes are prevalent among the clinicians, then the strategy’s impact may be increasingly enhanced or diminished. The strength of this impact may also attenuate or non-linearly vary over time. Systems analysis methods that specialize in characterizing, modeling, and managing complexity are needed to grow knowledge regarding implementation phenomena within these complex systems (i.e., knowledge that under certain conditions, a strategy will be operated through one or more specific mechanisms to affect one or more implementation outcomes).

Specifically, whether an implementation strategy activates a mechanism within a system may depend heavily on dynamically changing system components and their interconnections. The ability to make these complexities explicit is indispensable to identifying both the mechanism and the conditions that activate it, such that future implementation efforts seeking to target the mechanism can appropriately devise strategies that enable those conditions across different contexts. Systems analysis methods offer the very tools for unpacking such complexities of real-world systems in which mechanisms operate [ 6 ], as they are uniquely capable of monitoring interconnections within systems that dynamically change over time. These dynamics occur from non-linear changes and interconnected elements that lead to emergent phenomena such as policy resistance [ 7 ].

Systems analysis methods also allow for simulating observed and anticipated trends given a system’s dynamic complexity (e.g., emergent phenomena, structural changes resulting from implementation, feedback loops represented by changes in variables that result from endogenous changes within the system [ 8 ]). Furthermore, simulations can be calibrated with historical data to increase confidence in the model’s simulated outcomes for unobserved time periods (i.e., the future). These simulations can then be used to conduct experiments to explore questions such as anticipated system effects from an implementation strategy given contextual determinants (e.g., organizational size, structure of social networks), time points upon which changes of a given magnitude are expected to be observed (for multiple variables along the hypothesized causal pathway, including those within key feedback loops), and trade-offs such as who will benefit most from a given implementation approach [ 6 , 7 , 9 , 10 ]. Such experiments are particularly valuable since it is infeasible to directly test (e.g., through a randomized controlled trial) each of the many potentially relevant conditions' (and their combinations') influences on mechanisms. Especially in support of implementation science’s mission to accelerate real-world impact, systems analysis methods can complement existing interventional and observational methods to more comprehensively model and iteratively refine our understanding of implementation mechanisms.

Systems analysis methods are approaches offered by the fields of systems engineering and systems science that apply qualitative or quantitative modeling techniques to reflect complexity within a system and identify optimal solutions given the system’s context. These methods focus on identifying and evaluating properties of complex systems (such as interactions between heterogeneous system components, feedback loops, dynamic relationships, and emergent behaviors resulting from heterogeneous, adaptive actors), thereby demystifying the relationships between a system’s components and changes to the system over time. By making system boundaries and goals explicit, systems analysis methods may help minimize implementation resistance. Many aspects of systems analysis methods reside under the umbrellas of systems science and systems engineering, which aim to grow knowledge regarding systems-related phenomena and to develop specific solutions to problems faced by complex systems, respectively [ 11 ]. Applied to studying implementation mechanisms, systems analysis methods can help (i) better identify and manage conditions that may or may not activate mechanisms (both expected mechanisms targeted by a strategy and unexpected mechanisms that the methods help detect) and (ii) flexibly guide strategy adaptations to address emergent influences of context (e.g., individuals’ motivations, norms, organizational policies and structures, financial resources) on the mechanisms that were not foreseen when the strategy was initially selected and used.

Wagner and colleagues [ 12 ] define the systems engineering approach to studying health systems as “an approach that uses data to improve decision-making … by (a) diagnosing problems and identifying needs, (b) evaluating decision options to address a selected problem or need through modeling or optimization, and (c) translating optimized decision options into practical recommendations or actions.” Building on this definition, we outline four steps to apply systems analysis methods for studying implementation mechanisms. To illustrate the steps, we use Lewis and colleagues’ depression screening implementation [ 4 ] as a case example and point the reader to other published literature relevant to the steps. We conclude by discussing the steps’ strengths, limitations, and implications for future implementation mechanisms research. Additional file 1 provides the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines [ 13 ] that we consulted in reporting our work (since we drew on published works to demonstrate the application of systems analysis methods to studying implementation mechanisms, without interventional attempts to impact the methods’ applications).

Steps to apply systems analysis methods for studying implementation mechanisms

Figure  1 shows how we extended Wagner and colleagues’ work [ 12 ] to arrive at our steps for applying systems analysis methods to study implementation mechanisms. For our steps, we start from the point at which an active implementation effort has yet to be launched. During this pre-implementation phase, we assume that:

There is a shared understanding between implementers and their implementation partners about the intended distal outcome.

Implementers are aware of at least some of the key potential barriers, enablers, and relevant mechanisms.

At least one potential implementation strategy that accounts for these barriers, enablers, and mechanisms is planned for use in achieving the intended distal outcome.

Evolution of the systems engineering approach [ 12 ] into four steps for applying systems analysis methods to study implementation mechanisms

The “ Discussion ” section elaborates on ways in which systems analysis methods can guide implementation endeavors prior to this starting point—e.g., deciding whether to launch an implementation effort at all, selecting an innovation to implement, or identifying potential barriers, enablers, and mechanisms to accordingly inform new strategies.

We chose the Lewis et al. [ 4 ] case example to help advance the field’s understanding of mechanisms by promoting cohesiveness across mechanisms-related papers. In this example, a community mental health center aims to implement a brief depression symptom severity screening measure (Patient Health Questionnaire). The distal outcome targeted is enhanced depression screening fidelity. We refer to two of the implementation strategies considered by the implementers: (i) Patient Health Questionnaire administration training and (ii) financial disincentive for each missed screening opportunity. See Fig.  2 for an adapted visualization of the example, which we further explain under step 1.

Example causal pathway diagrams, adapted from [ 4 ]

Note: individual-level moderators and mechanisms can have group- and organizational-level implications; similarly, organizational-level moderators and preconditions can have individual-level implications

Step 1: Specify the implementation strategies and articulate the mechanistic pathway of each implementation strategy, including target mechanisms, preconditions, mediators, moderators, and proximal outcomes relevant to mechanism activation

Step 1 consists of four sub-steps that focus on specifying the (i) implementation strategy or strategies to apply, (ii) mechanisms expected to be at play through each strategy, (iii) preconditions/mediators/moderators that are expected to be relevant for mechanism activation, and (iv) relevant proximal outcomes that are expected to indicate mechanism activation. It is also important to specify these for multiple levels of the health care system [ 14 ]—e.g., clinician- versus organization-level moderators. For these specifications, one possible approach outlined by Lewis and colleagues [ 4 ] is to use causal pathway diagrams to specify the hypothesized relationships between strategies, mechanisms, preconditions, mediators, moderators, and outcomes.

Potentially useful systems analysis methods for step 1 include techniques that are widely used for quality improvement and patient safety, such as 5 whys, fishbone diagrams, and other tools used for root cause analysis [ 15 , 16 ] that can be adapted to identify the causes for a mechanism being or not being activated. End users’ (i.e., implementation actors’) input is often used when conducting root cause analysis or failure modes and effects analysis [ 15 , 16 , 17 ]. For example, possible causes of the skill-building mechanism not being activated can be explored by using these structured root cause analysis tools to seek input from and consensus among implementation actors most knowledgeable about or experienced in skill-building for depression screening, as well as individuals that the mechanism involves (in this case, clinicians). Similarly, tools used for failure modes and effects analysis [ 17 ] can also be adapted to identify possible influences on the path from strategy through mechanism to outcome.

Causal loop diagrams (CLDs) are another useful systems analytic tool for identifying how mechanisms relate to the selected implementation strategies and targeted implementation outcomes [ 7 ]. Although CLDs, like causal pathway diagrams, focus on how specific factors (e.g., preconditions) interconnect to cause changes in outcomes, CLDs emphasize dynamic change. CLDs characterize how factors or behaviors emerge and perpetuate over time through consistently escalating or de-escalating trajectories (reinforcing feedback loops). CLDs also illustrate how the introduction of a particular variable helps a system reach or maintain equilibrium (balancing feedback loops). {Note: Feedback loops have unique definitions across related, but distinct, fields such as psychology [ 18 , 19 , 20 , 21 ]. For the purposes of this paper, we follow the definitions used in systems engineering and system dynamics, in which a “balancing feedback loop” is made up of interrelated factors that together lead to a system returning to the status quo or reaching a type of equilibrium due to goal-seeking behaviors or system constraints that prevent perpetual improvement or perpetual decline (i.e., reinforcing feedback loops) [ 7 , 9 , 10 , 22 , 23 ]. For example, even with the most successful implementation strategy to reduce clinic wait times through structural efficiencies (e.g., administration), the rate at which individuals leave the wait list will always be restricted (or “balanced”) by the number of clinicians and the average client’s time in care [ 24 , 25 ]. The following description of Fig.  3 provides additional examples of balancing feedback loops.} Figure  3 depicts a CLD of the example causal pathway diagram components, from which several insights emerge:

The CLD makes it clearer how central organizational leadership supports the use of the two implementation strategies and their preconditions. Furthermore, the strategies’ success is hypothesized to increase leadership support over time, increasing the likelihood of implementation sustainment. Thus, CLDs can help identify measurement targets—in this case, the CLD suggests that measuring organizational leadership over time can help monitor the likelihood of implementation success.

The CLD identifies two critical balancing loops, B1 and B2, that were unidentified in the causal pathway diagram.

B1 suggests that the level of incentive required to change motivation influences the value of a disincentive on motivation—a value which will likely vary by clinician based on factors such as their salary, time in position, and perceived burden of conducting the Patient Health Questionnaire-based screening.

B2 highlights how the hypothesized pathway by which a financial incentive increases screening might vary over time, namely, as fidelity increases and is easier to achieve (requiring a lower threshold to maintain clinician’s motivation to increase their screening rate), the financial incentive may be decreased.

Such insights from CLDs have important implications for planning the allocation of resources associated with an implementation strategy over time.

The CLD highlights a shared mechanism—skill building—that was previously only associated with one of the two strategies. Thus, CLDs are also a promising way to identify efficient bundles of implementation strategies that synergistically benefit from the activation of shared mechanisms.

A causal loop diagram of the example causal pathway diagram components in Fig.  2 ; color scheme follows that of Fig. 2

Note: underlined are new variables that are not among the causal pathway diagram components in Fig. 2 ; black arrows indicate new pathways not depicted in Fig. 2

Causal pathway diagrams or similar diagrams generated in this step not only point to hypothesized relationships, but importantly also imply which relationships should not exist. For example, as diagrammed in Fig.  2 , the “ability to attend training without penalty” precondition and the “clinician desire to learn” moderator are not related. If available data (e.g., from previous studies) show a relationship between preconditions and moderators that should not exist, or if the relationship identified does not reflect reality, it is an indication that the diagram, and in turn the understanding of the causal pathway, is incomplete and warrants an update. Comprehensive identification of such testable implications of a hypothesized causal pathway can rely on existing causal inference tools. For instance, the pathway can be expressed using a CLD as above or a directed acyclic graph (systematic visual representations of causal relationships) [ 26 ] to understand confounders and potential sources of bias in the pathway between strategy and outcome.

Scenario simulation can be helpful for updating the mechanistic pathway to be consistent with existing data. For implementation efforts more generally (not specific to step 1), simulations allow implementers to computationally “try out” different scenarios of implementation strategies and observe their potential impact on outcomes, prior to deploying the strategies. Multiple scenarios can be simulated to estimate different means and ranges of outcomes arising from incomplete data/knowledge or to identify implementation strategies that seem most robust to the uncertainty even under limited resources (i.e., a mathematical optimization problem of finding the best solution given constraints—e.g., multiphase optimization strategy [ 27 ]). Because simulation models do not rely only on already available or easily measurable data, they can be used to identify what data are needed for more precise outcome estimates or strategy design, and how much of a difference having additional data would make. As explained by Sheldrick and colleagues [ 24 ], simulation models can thus be used to conduct virtual experiments that help decision-makers consider the trade-offs in using a given implementation strategy over another. (Sheldrick and colleagues recommend using simple models, such as Monte Carlo models, that are transparent and easily understood by decision-makers with limited time to engage with the models. Alternatively, more complex system dynamics, agent-based, and microsimulation models can also support decision-makers’ learning with appropriate, acceptable guidance [ 25 , 28 , 29 ]).

A range of potential causal pathways describing the mechanisms can be simulated to identify ones that best match available knowledge and data. As per Fig.  2 , if available data show a relationship between the “ability to attend training without penalty” precondition and the “clinician desire to learn” moderator, thereby contradicting the currently conceptualized causal pathway, then alternative relationships can be “tried out” using the simulation model. For instance, a direct relationship between the two components, or a relationship that connects through a third (either specified or unspecified) component, can be built into alternative versions of the model. After enumerating all such feasible causal pathways, it is possible to choose one that most closely matches the available data—i.e., select a model version that minimizes the difference between available data and model-simulated data. If the selected model has one or more unspecified components, it means that mechanisms may be at play that were not originally hypothesized; this warrants an update to the conceptualized causal pathway before proceeding. The unspecified components’ placement in the model can help researchers speculate about missing mechanisms and/or what to measure and when to identify those mechanisms.

Step 2: Estimate and test the potential impact of mutable strategies, preconditions, mediators, and moderators on their related mechanisms and observable outcomes, given the specific context(s) in which the implementation is to happen

Using the simulation model of the causal relationships between strategies, mechanisms, preconditions, mediators, moderators, and outcomes, we can virtually test the impact of changing different combinations of the mutable strategies, preconditions, mediators, and moderators in the model. Mutable components are ones that we consider to be realistically changeable as a part of the implementation effort. For instance, the “Patient Health Questionnaire form is available/accessible” precondition may be mutable for implementation settings in which the form can be made available to clinicians through an existing electronic health record system. We can use simulation to estimate the relative impact of these changes on whether and how hypothesized mechanisms are activated and how outcomes are subsequently affected.

Simulations can also help set realistic expectations for mechanism activation and/or the strength of a mechanism’s influence on implementation outcomes. For instance, we can simulate whether there are threshold levels or tipping points of the precondition “Patient Health Questionnaire form is available/accessible” and the moderator “organizational support for training” at which changes in their values do not meaningfully change their impact on the increased screening rate. Such an insight would help make valuable decisions about resource efficiency (i.e., not devoting more time and resources beyond these thresholds). If such thresholds are simulated, the implementation effort can accordingly monitor when threshold levels are reached and resources can be reallocated toward other mutable strategies, preconditions, mediators, and moderators to further activate targeted mechanisms and drive desirable outcomes.

Importantly, simulations can help identify measurement targets along the modeled causal pathway that are more or less sensitive at detecting mechanism activation. For instance, if simulated scenarios for the depression screening example indicate very small changes in the increased screening rate despite large changes in the level of skill building, then a more immediate indicator than the screening rate is needed to specifically gauge the extent to which the skill-building mechanism is activated (e.g., the number of clinicians whose skills are built through training). If feasible, the implementation effort can accordingly plan to monitor this more immediate indicator to measure mechanism activation more accurately given the expected timeline and magnitude of change.

Step 3: Tailor and update the implementation strategy or strategies (including the timing, dosage, and sequencing of actions to be taken) to incorporate making changes to mutable preconditions/mediators/moderators, to enable mechanism activation and subsequent implementation outcomes

Model simulations as described under step 2 can be used to select and shape the strategy or a combination of strategies for implementation. For instance, the relative impact of using one or both of the “Patient Health Questionnaire administration training” and “financial disincentive for each missed screening opportunity” strategies can be simulated under varying preconditions/mediators/moderators to decide whether one or both strategies should be used simultaneously or sequentially, and with the same or different types of implementation actors (e.g., executive director, clinician, front-line staff, patients across heterogeneous contexts), to reach desired implementation outcomes. Similarly, if moving forward with both strategies, simulation can help determine which and how many resources to allocate toward enhancing the “organizational support for training” precondition versus the “organizational communication infrastructure” precondition for optimal impact.

As described in step 1, enumerating and selecting a strategy or a combination of strategies to use among all realistic options can be posed as an optimization problem. Namely, using a simulation model, all potential scenarios of utilizing one or more strategies that trigger changes in one or more mutable preconditions/mediators/moderators can be simulated to identify the scenario(s) expected to maximize desirable outcomes while minimizing the effort (or some other optimization factors, such as resources) needed to reach those outcomes. The implementation strategy or strategies can subsequently be tailored and updated to mirror the identified optimal scenario(s). Especially as multi-strategy implementation efforts have become the norm for promoting the uptake of evidence-based practices, simulations can inform how multiple potential strategies can be combined to enable mechanism activation given an implementation setting’s key contextual factors that include relevant preconditions, mediators, and moderators. Importantly, simulations can help predict multiple strategies’ combined effects on mechanisms that, given system complexities, may not be simply additive in nature and thus difficult to predict using other methods.

Step 4: Apply the updated implementation strategy or strategies, then use newly available data from the implementation effort to further refine or revise the previously specified causal relationships between strategies, mechanisms, preconditions, mediators, moderators, and outcomes

We can now apply the implementation strategy or strategies devised in step 3 and assess implementation outcomes. During this step, it is necessary to collect data on components in the causal pathway that were identified in step 2 as most indicative of mechanism activation (e.g., the number of clinicians whose skills are built through training, as mentioned above). To grow our understanding of implementation mechanisms, it is critical to then reconduct step 1 with the newly available data; in other words, we must use the new data to test whether the implications of the hypothesized causal relationships still hold (e.g., whether the “ability to attend training without penalty” precondition and the “clinician desire to learn” moderator remain unrelated, as mentioned above). If the new data contradict the currently conceptualized causal pathway, then tasks outlined under step 1 can be followed to explore alternative relationships that better explain both previous and new data (examples of this are outlined in the “ Examples of systems analysis methods for studying implementation mechanisms ” section) and to accordingly update the pathway.

Examples of systems analysis methods for studying implementation mechanisms

Various systems analysis methods can be used for studying implementation mechanisms, not limited to the ones that are mentioned in the steps’ descriptions in the “ Steps to apply systems analysis methods for studying implementation mechanisms ” section above. Table 2 shares examples from the literature of systems analysis methods that are beginning to be used for mechanisms research, curated using the approach described below. Recognizing that the different methods may also vary widely in the level of systems analytic expertise needed for their use, the table also refers the reader to tutorials and other resources that can help them decide whether they need to engage collaborators with additional expertise to pursue using a listed method.

We conducted a targeted search of the literature and ongoing studies to identify key examples of systems analysis methods applied to elucidating and testing mechanisms. We searched PubMed and NIH Reporter and used forward searching for a select set of seminal articles. Within PubMed, we crafted the following search string, based on a modified version of the search string utilized by Lewis and colleagues [ 59 ] for implementation science and mechanisms research, and the specific systems analysis methods identified by Wagner, Crocker, and colleagues [ 12 ]:

((Implement*[tiab] OR disseminate*[tiab] OR “knowledge translation”[tiab]) AND (Mediate*[tiab] OR moderator[tiab] OR mechanism*[tiab]) AND (“empirically supported treatment”[tiab] OR “evidence-based practice”[tiab] OR “evidence-based treatment”[tiab] OR innovation[tiab] OR guideline[tiab]) AND (“structural equation model*”[tiab] OR “directed acyclic graph”[tiab] OR “DAG”[tiab] OR “causal loop diagram”[tiab] OR “process evaluation”[tiab] OR “process analysis”[tiab] OR “optimiz*”[tiab] OR “simulat*”[tiab] OR “agent”[tiab] OR “quality improvement”[tiab] OR “fish bone”[tiab] OR “failure modes and effects analysis”[tiab] OR “FMEA”[tiab] OR “Ishikawa”[tiab] OR “flow map*”[tiab] OR “process map*”[tiab] OR “value stream map*”[tiab] OR “root cause analysis”[tiab] OR “PDSA”[tiab] OR “system dynamic*”[tiab])) NOT (Biology OR microbiology)

The search string that we used in PubMed (shown above) covered synonyms and variations of implementation science, evidence-based interventions, and implementation mechanisms, as well as various systems analysis methods, while attempting to exclude works that discuss biological mechanisms. Then, within NIH Reporter, we crafted the following two search strings, which were limited by the search capabilities of the database: (1) “mechanism” AND “implementation science” and (2) “mechanism” AND “implementation research.” Finally, we forward searched two seminal papers ([ 4 ] and [ 59 ]) within Google Scholar.

The search was conducted during March and April 2022. Each entry identified by the search strategies was reviewed by one member of the authorship team for relevance. After the first ten entries identified by each search strategy were screened, the authorship team met and discussed the relevance of the articles being returned, agreeing by consensus to proceed with the search strategy and complete the full screening. Those entries deemed to be relevant examples of systems analysis methods applied to elucidating and testing mechanisms were then included; relevant fields were extracted by one member of the authorship team using a pre-set template, including the fields in Table 2 , then reviewed and refined by all members.

The examples that we found of systems analysis methods applied to studying mechanisms ranged from primarily quantitative (e.g., structural equation modeling [ 45 ]) to primarily qualitative (e.g., realist evaluation [ 36 ]) approaches. Visual representations of mechanisms and additional interrelated factors that influence implementation were central to several of the examples (e.g., causal pathway diagramming [ 25 ], ripple effects mapping [ 40 ]), while other examples focused on the computational modeling of those interrelationships (e.g., agent-based modeling [ 22 ], simulation modeling [ 16 ]). Emphasized throughout the examples (particularly process evaluation [ 30 ] and participatory system dynamics [ 28 ]) was the intentional, close incorporation of the perspectives of individuals involved in the modeled system or systems in which implementation was occurring. These individuals were involved in identifying which system components to model, defining model boundaries, proposing relevant mechanisms and their connections to contextual factors, and interpreting the relevance of model outputs for implementation efforts.

It is worth noting that the examples shown in Table 2 use systems analysis methods predominantly for steps 1 and 2—i.e., for identifying potential mechanisms and for assessing the expected impact of strategies and contextual factors on mechanisms. Even as implementation research generally may be embracing the use of systems analysis methods more, our search had difficulty finding studies that use systems analysis methods for steps 3 and 4—i.e., for refining strategies to explicitly enable better mechanism activation and for examining the resulting changes on mechanism activation.

We provide a four-step approach to applying systems analysis methods for examining implementation mechanisms. The steps integrate Wagner and colleagues’ systems engineering approach for decision-making in global health systems [ 12 ] with Lewis and colleagues’ approach to identifying and studying mechanisms of implementation strategies [ 4 ], to guide the practical use of systems analysis methods for mechanisms research in implementation science. To demonstrate the steps, we use as a case example Lewis and colleagues’ implementation of the Patient Health Questionnaire for depression screening in a community mental health center [ 4 ]. We also point the reader to additional examples of systems analysis methods and resources that can guide their usage in future implementation research.

Systems analysis methods’ roles in making mechanism-related assumptions explicit

The steps encourage rigorous specification and methodical refinement of assumptions surrounding the mechanisms that are expected to be at play when implementation strategies lead to the uptake of evidence-based interventions into routine practice. These assumptions relate to which implementation strategies target specific mechanisms, and the preconditions, mediators, and moderators specific to implementation contexts that influence a strategy’s ability to activate mechanism(s) necessary for implementation success. Unless these assumptions can be specified, the reason why an implementation strategy is or is not successful cannot be fully understood. Without this understanding, it cannot be made explicit how the strategy can be improved for continued use or tailored to fit new implementation contexts.

Considerations for multi-level systems

Systems analysis methods are used in a variety of disciplines to understand how components within and across systems change and interact with one another to affect system properties. Especially for multi-level systems common in health care and community settings—for which system properties (e.g., implementation outcomes, moderators) are affected by components at multiple levels (e.g., individuals, clinics, organizations, community)—making changes to the system requires strategies (e.g., implementation strategies) that target multiple levels (e.g., training for clinicians, update to organizational policy) and key leverage points (i.e., factors in a system that drive change) [ 10 ]. As strategies become multi-level and complex, so do the causal pathways that link strategies to mechanisms to outcomes. Techniques involving structured inquiries (e.g., root cause analysis), visualization (e.g., causal pathway diagrams), computational modeling (e.g., scenario simulation), and other systems analysis methods can help accurately characterize and manage the complex knowledge regarding multi-level system components and their interrelationships. In the setting of multiple strategies, systems analysis methods can capture and reflect synergistic, antagonistic, or other non-additive patterns of co-occurrence.

Incorporating implementation actors’ input and conceptual guidance

Hypothesized causal pathways, from implementation strategies through mechanisms to outcomes, should stem not only from theories and frameworks, but also from the experiences, values, and beliefs of implementation actors [ 60 ]—individuals who impact and/or are impacted by an implementation effort. It is thus important to leverage actor-engaged approaches that capture multiple perspectives to complement, or officially be a part of, efforts to apply systems analysis methods to mechanisms research. For instance, actors’ mental models of how different factors are linked to outcomes are critical to accurately characterizing and building the system structure underlying a computer simulation model to be used in examining potential scenarios [ 9 , 61 , 62 ]. Hypothesized strategy-mechanism-outcome links may also be based on one or more theories, models, and frameworks that categorize or provide explanations for implementation-related phenomena. Reviewing the domains and their relationships per an implementation-relevant theory/model/framework {e.g., Exploration, Preparation, Implementation, Sustainment (EPIS) framework [ 63 ]} can prompt consideration of mechanisms and causal pathways that have not previously been considered (e.g., EPIS’ bridging factors that span multi-level outer and inner contexts [ 64 , 65 ]), which can be specified and examined using systems analysis methods. Such examinations may also have opportunities to reciprocally inform implementation theories/models/frameworks of the relative prevalence, strengths, and further specifications of their domains and relationships as observed through implementation mechanisms research.

Leveraging concurrent advances in other aspects of mechanisms research

For such actor-engaged and theory/model/framework-aligned examination of implementation mechanisms to successfully apply systems analysis methods (i.e., carry out the four steps outlined in this article), concurrent advances in other aspects of mechanisms research are indispensable. The steps, and particularly whether an iteration back to an earlier step is warranted, depend on comparing the systems analysis method-based observations (e.g., simulation results) to available empirical implementation knowledge and data. Especially as the observations suggest measures that implementation efforts can focus on collecting (e.g., for better indication of mechanism activation), the suggested measurements can only yield useful data if measurement approaches are and continue to be practical, accurate, timely, and replicable. Relatedly, to iteratively refine understandings of causal pathways from implementation strategies to outcomes across one or more implementation efforts with shared target mechanisms, it is important to have methods for clearly documenting when and how specific mechanisms are tracked and examined, as well as methods for tracking and evaluating resulting implementation and clinical outcomes or other observations. Visualizations of causal pathways (e.g., CLDs), simulation records (e.g., simulated/computed model outputs), and other documentations (e.g., fishbone diagrams) generated from applying systems analysis methods to implementation mechanisms research can offer some approaches to documentation, while a wider consensus across the field is necessary for shared terminologies and other conventions for documentation [ 66 ].

Limitations

Using systems analysis methods, such as those in the four steps described above, can help study mechanisms as they relate to multi-level strategies and contextual factors, require mechanism-related assumptions to be specified and tested, and identify targets along the causal pathway to inform mechanisms research. This work also has limitations. First, the starting point for the steps assumes that the decision to implement, the distal outcome, and at least one implementation strategy are set. Although this starting point allows the iterative nature of the steps to be underscored, it leads to excluding explicit discussion within this article of the potential utility of systems analysis methods for pre-implementation efforts to decide whether to implement, agree with implementation actors (particularly those who might be end users of a simulation model or with implementation decision-making authority) on the distal outcome, and inform the selection of an initial strategy. We encourage readers to refer to implementation mapping [ 67 ], group model building [ 68 ], and other established implementation research methods that focus on problem identification, implementation needs assessments, outcomes selection, and strategy design. Second, we use a single case example of depression screening implementation to outline the steps to apply systems analysis methods for studying implementation mechanisms. While implementation efforts concerning different evidence-based practices and settings may call for considerations distinct from that of our case example, we aligned to established case study research practices of focusing on a single case when the case is atypical and noteworthy [ 53 , 54 ]. We therefore chose to anchor the illustration of our ideas on this example that is unique in its association with a seminal paper that both conceptualizes mechanisms of implementation strategies and establishes a visual representation of foundational mechanisms-related definitions [ 4 ] on which we build in this manuscript. Third, given that this article is the first in articulating explicit steps for applying systems analysis methods for implementation mechanisms research, neither the main case example that we use to illustrate the steps nor the examples that we point to in the literature for each step were pursued by their implementation team with these specific steps in mind. Although this work is grounded in both a review of systems-based health research methods [ 12 ] and foundational definitions and examples of mechanisms-related concepts [ 4 ], further work is needed to prospectively apply and test these now articulated steps for mechanisms research.

Especially as implementations of evidence-based interventions increasingly target underserved populations and are pursued in low-resource settings that contextually differ from the high-resource settings in which the interventions were originally developed and implemented, elucidating the mechanisms that explain the “why” and “how” of implementation is more essential now than ever before [ 69 , 70 ]. Systems analysis methods, widely used in multiple disciplines to investigate causal relationships and behaviors of complex systems, offer opportunities to identify, specify, test, and refine our understanding of implementation mechanisms that need to be activated for implementation strategies to lead to desirable outcomes. We hope that the four steps to applying systems analysis methods we introduced here can encourage more mechanisms research efforts to consider these methods and in turn fuel both (i) rigorous comparisons of these methods to alternative mechanisms research approaches and (ii) an active discourse across the field to better delineate when these methods are more or less appropriate to use for advancing the knowledge regarding implementation mechanisms.

Availability of data and materials

No new data were used in this work. Information supporting the work’s claims is available within the article.

Abbreviations

Causal loop diagram

Exploration, Preparation, Implementation, Sustainment

National Institutes of Health

Patient Health Questionnaire-9

Strengthening the Reporting of Observational Studies in Epidemiology

Eccles MP, Mittman BS. Welcome to Implementation Science. Implement Sci. 2006;1(1):1.

PubMed Central   Google Scholar  

Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139.

PubMed   PubMed Central   Google Scholar  

Wensing M, Oxman A, Baker R, Godycki-Cwirko M, Flottorp S, Szecsenyi J, et al. Tailored Implementation For Chronic Diseases (TICD): a project protocol. Implement Sci. 2011;6:103.

Lewis CC, Klasnja P, Powell BJ, Lyon AR, Tuzzio L, Jones S, et al. From classification to causality: advancing understanding of mechanisms of change in implementation science. Front Public Health. 2018;6:136.

Lipsitz LA. Understanding health care as a complex system: the foundation for unintended consequences. JAMA. 2012;308(3):243–4.

PubMed   PubMed Central   CAS   Google Scholar  

Luke DA, Powell BJ, Paniagua-Avila A. Bridges and mechanisms: integrating systems science thinking into implementation research. Annu Rev Public Health. In press.

Sterman J. Business dynamics, system thinking and modeling for a complex world. Homewood: Irwin/McGraw-Hill; 2000. p. 19.

Richardson GP. Reflections on the foundations of system dynamics. Syst Dyn Rev. 2011;27(3):219–43.

Google Scholar  

Meadows DH. Thinking in systems: a primer. White River Junction: Chelsea Green Publishing; 2008.

Meadows D. Places to intervene in a system (in increasing order of effectiveness). Whole Earth. 1997;91(1):78.

Luke D, Morshed A, McKay V, Combs T. Systems science methods in dissemination and implementation research. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and implementation research in health: translating science to practice. 2nd ed. New York: Oxford University Press; 2018.

Wagner AD, Crocker J, Liu S, Cherutich P, Gimbel S, Fernandes Q, et al. Making smarter decisions faster: systems engineering to improve the global public health response to HIV. Curr HIV/AIDS Rep. 2019;16(4):279–91.

von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. 2007;335(7624):806–8.

Ferlie EB, Shortell SM. Improving the quality of health care in the United Kingdom and the United States: a framework for change. Milbank Q. 2001;79(2):281–315.

Whys: finding the root cause: Institute for Healthcare Improvement. Available from: https://www.ihi.org/resources/Pages/Tools/5-Whys-Finding-the-Root-Cause.aspx . Accessed 5 Oct 2023.

Cause and effect diagram: Institute for Healthcare Improvement. Available from: https://www.ihi.org/resources/Pages/Tools/CauseandEffectDiagram.aspx . Accessed 5 Oct 2023.

Failure modes and effects analysis (FMEA) tool: Institute for Healthcare Improvement. Available from: https://www.ihi.org/resources/Pages/Tools/FailureModesandEffectsAnalysisTool.aspx . Accessed 5 Oct 2023.

Frank H. On the notion of balance of a signed graph. Mich Math J. 1953;2(2):143–6.

Cartwright D, Harary F. Structural balance: a generalization of Heider’s theory. Psychol Rev. 1956;63(5):277–93.

PubMed   CAS   Google Scholar  

Abelson R. Symbolic psycho-logic: a model of attitudinal cognition. Behav Sci. 2007;3:1–13.

Harary F, Kabell JA. A simple algorithm to detect balance in signed graphs. Math Soc Sci. 1980;1(1):131–6.

Ford DN. A system dynamics glossary. Syst Dyn Rev. 2019;35(4):369–79.

Richardson GP. Core of system dynamics. System dynamics: theory and applications. New York: Springer Science+Business Media; 2020. pp. 11–20.

Sheldrick RC, Cruden G, Schaefer AJ, Mackie TI. Rapid-cycle systems modeling to support evidence-informed decision-making during system-wide implementation. Implement Sci Commun. 2021;2(1):116.

Zimmerman L, Lounsbury DW, Rosen CS, Kimerling R, Trafton JA, Lindley SE. Participatory system dynamics modeling: increasing stakeholder engagement and precision to improve implementation planning in systems. Adm Policy Ment Health. 2016;43(6):834–49.

Shrier I, Platt RW. Reducing bias through directed acyclic graphs. BMC Med Res Methodol. 2008;8(1):70.

Guastaferro K, Collins LM. Optimization methods and implementation science: an opportunity for behavioral and biobehavioral interventions. Implementation Res Pract. 2021;2:26334895211054364.

Vermeer WH, Smith JD, Wilensky U, Brown CH. High-fidelity agent-based modeling to support prevention decision-making: an open science approach. Prev Sci. 2022;23(5):832–43.

PubMed   Google Scholar  

Curran GM, Smith JD, Landsverk J, Vermeer W, Miech EJ, Kim B, et al. Design and analysis in dissemination and implementation research. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and implementation research in health: translating science to practice. 3 ed. New York: Oxford University Press; 2023.

Tracy M, Cerdá M, Keyes KM. Agent-based modeling in public health: current applications and future directions. Annu Rev Public Health. 2018;39:77–94.

Frank KA, Xu R, Penuel WR. Implementation of evidence-based practice in human service organizations: implications from agent-based models. J Policy Anal Manage. 2018;37(4):867–95.

Agent-based modeling: Columbia University. Available from: https://www.publichealth.columbia.edu/research/population-health-methods/agent-based-modeling . Accessed 5 Oct 2023.

Huang W, Chang CH, Stuart EA, Daumit GL, Wang NY, McGinty EE, et al. Agent-based modeling for implementation research: an application to tobacco smoking cessation for persons with serious mental illness. Implement Res Pract. 2021;2.

Springer MV, Sales AE, Islam N, McBride AC, Landis-Lewis Z, Tupper M, et al. A step toward understanding the mechanism of action of audit and feedback: a qualitative study of implementation strategies. Implement Sci. 2021;16(1):35.

Optimizing implementation: optimizing implementation in cancer control. Available from: https://www.opticc.org/our-methods/ . Accessed 5 Oct 2023.

Reese TJ, Schlechter CR, Kramer H, Kukhareva P, Weir CR, Del Fiol G, et al. Implementing lung cancer screening in primary care: needs assessment and implementation strategy design. Transl Behav Med. 2022;12(2):187–97.

Király G, Miskolczi P. Dynamics of participation: system dynamics and participation—an empirical review. Syst Res Behav Sci. 2019;36(2):199–210.

Best A, Berland A, Herbert C, Bitz J, van Dijk MW, Krause C, et al. Using systems thinking to support clinical system transformation. J Health Organ Manag. 2016;30(3):302–23.

Freebairn L, Occhipinti JA, Song YJC, Skinner A, Lawson K, Lee GY, et al. Participatory methods for systems modeling of youth mental health: implementation protocol. JMIR Res Protoc. 2022;11(2):e32988.

Hulscher M, Wensing M. Process evaluation of implementation strategies. Improving Patient Care. Hoboken; 2020. p. 369–87.

Huis A, Holleman G, van Achterberg T, Grol R, Schoonhoven L, Hulscher M. Explaining the effects of two different strategies for promoting hand hygiene in hospital nurses: a process evaluation alongside a cluster randomised controlled trial. Implement Sci. 2013;8:41.

Carpenter JG, Scott WJ, Kononowech J, Foglia MB, Haverhals LM, Hogikyan R, et al. Evaluating implementation strategies to support documentation of veterans’ care preferences. Health Serv Res. 2022;57(4):734–43.

Hatch MR, Carandang K, Moullin JC, Ehrhart MG, Aarons GA. Barriers to implementing motivational interviewing in addiction treatment: a nominal group technique process evaluation. Implement Res Pract. 2021;2.

Lyon AR, Pullmann MD, Dorsey S, Levin C, Gaias LM, Brewer SK, et al. Protocol for a hybrid type 2 cluster randomized trial of trauma-focused cognitive behavioral therapy and a pragmatic individual-level implementation strategy. Implement Sci. 2021;16(1):3.

Ogrinc G, Batalden P. Realist evaluation as a framework for the assessment of teaching about the improvement of care. J Nurs Educ. 2009;48(12):661–7.

Flynn R, Mrklas K, Campbell A, Wasylak T, Scott SD. Contextual factors and mechanisms that influence sustainability: a realist evaluation of two scaled, multi-component interventions. BMC Health Serv Res. 2021;21(1):1194.

Pawson R, Tilley N. Realistic evaluation. London: Sage; 1997.

Sheaff R, Doran N, Harris M, Lang I, Medina-Lara A, Fornasiero M, et al. Categories of context in realist evaluation. Evaluation. 2021;27(2):184–209.

Ripple effects mapping: Washington State University. Available from: https://ppe.cw.wsu.edu/ripple-effects-mapping/ . Accessed 5 Oct 2023.

Hall J, Bingham DD, Seims A, Dogra SA, Burkhardt J, Nobles J, et al. A whole system approach to increasing children’s physical activity in a multi-ethnic UK city: a process evaluation protocol. BMC Public Health. 2021;21(1):2296.

Rubio MA, Guevara-Aladino P, Urbano M, Cabas S, Mejia-Arbelaez C, Rodriguez Espinosa P, et al. Innovative participatory evaluation methodologies to assess and sustain multilevel impacts of two community-based physical activity programs for women in Colombia. BMC Public Health. 2022;22(1):771.

Ripple effect mapping makes waves in the world of evaluation: University of Minnesota. Available from: https://extension.umn.edu/community-development/ripple-effect-mapping .

Salleh S, Thokala P, Brennan A, Hughes R, Booth A. Simulation modelling in healthcare: an umbrella review of systematic literature reviews. Pharmacoeconomics. 2017;35(9):937–49.

Bagozzi RP, Yi Y. Specification, evaluation, and interpretation of structural equation models. J Acad Mark Sci. 2012;40(1):8–34.

Lambooij MS, Koster F. How organizational escalation prevention potential affects success of implementation of innovations: electronic medical records in hospitals. Implement Sci. 2016;11(1):75.

Brumbach BH, Birmingham WC, Boonyasiriwat W, Walters S, Kinney AY. Intervention mediators in a randomized controlled trial to increase colonoscopy uptake among individuals at increased risk of familial colorectal cancer. Ann Behav Med. 2017;51(5):694–706.

Jacobs SR, Weiner BJ, Reeve BB, Hofmann DA, Christian M, Weinberger M. Determining the predictors of innovation implementation in healthcare: a quantitative analysis of implementation effectiveness. BMC Health Serv Res. 2015;15(1):6.

Lo SB, Conley CC, Brothers BM, Ryba MM, Frierson GF, Shelby RA, et al. Replicating dissemination and identifying mechanisms of implementation of an empirically supported treatment. Health Psychol. 2021;40:450–8.

Lewis CC, Boyd MR, Walsh-Bailey C, Lyon AR, Beidas R, Mittman B, et al. A systematic review of empirical studies examining mechanisms of implementation in health. Implement Sci. 2020;15(1):21.

Sheldrick RC, Hyde J, Leslie LK, Mackie T. The debate over rational decision making in evidence-based medicine: implications for evidence-informed policy. Evidence & Policy. 2021;17(1):147–59.

Holtrop JS, Scherer LD, Matlock DD, Glasgow RE, Green LA. The importance of mental models in implementation science. Front Public Health. 2021;9:680316.

Hysong SJ, Smitham K, SoRelle R, Amspoker A, Hughes AM, Haidet P. Mental models of audit and feedback in primary care settings. Implement Sci. 2018;13(1):73.

Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38(1):4–23.

Moullin JC, Dickson KS, Stadnick NA, Rabin B, Aarons GA. Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implement Sci. 2019;14(1):1.

Crable EL, Lengnick-Hall R, Stadnick NA, Moullin JC, Aarons GA. Where is “policy” in dissemination and implementation science? Recommendations to advance theories, models, and frameworks: EPIS as a case example. Implement Sci. 2022;17(1):80.

Martinez-Moyano IJ. Documentation for model transparency. Syst Dyn Rev. 2012;28(2):199–208.

Fernandez ME, Ten Hoor GA, van Lieshout S, Rodriguez SA, Beidas RS, Parcel G, et al. Implementation mapping: using intervention mapping to develop implementation strategies. Front Public Health. 2019;7:158.

Vennix J. Group model building: facilitating team learning using system dynamics. Chichester, UK: Jonh Wiley & Sons Ltd.; 1996.

Baumann AA, Mejia A, Lachman JM, Parra-Cardona R, López-Zerón G, Amador Buenabad NG, et al. Parenting programs for underserved populations in low- and middle-income countries: issues of scientific integrity and social justice. Global Social Welfare. 2019;6(3):199–207.

Dickson KS, Holt T, Arredondo E. Applying implementation mapping to expand a care coordination program at a federally qualified health center. Front Public Health. 2022;10.

Download references

Acknowledgements

The authors sincerely thank the other members of the Mechanisms Network of Expertise for facilitating the discourse that formed the basis of this work. This work began while GC was employed at the Oregon Social Learning Center and concluded at her current affiliation.

The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of their affiliated institutions.

The authors are members of the Mechanisms Network of Expertise supported by AHRQ R13 HS025632. Dr. Cruden was supported by NIMH K01 MH128761. Dr. Crable was supported by NIDA K01 DA056838-01 and is also a fellow with the Lifespan/Brown University Criminal Justice Research Training Program (NIDA R25 DA037190). Dr. Quanbeck was supported by NIH grants R01DA055527-01A1, R01DA047279-01, and R01AA024150-01A1. Dr. Wagner was supported by NIMH K01 MH121124. Drs. Crable, Cruden, Kim, and Quanbeck are alumni, and Dr. Mittman is core faculty with the Implementation Research Institute (IRI) at the George Warren Brown School of Social Work, Washington University in St. Louis, supported by NIMH (R25 MH080916), NIDA, and VA.

Author information

Authors and affiliations.

VA Boston Healthcare System, 150 South Huntington Avenue, Boston, MA, 02130, USA

Harvard Medical School, 25 Shattuck Street, Boston, MA, 02115, USA

Chestnut Health Systems, Lighthouse Institute-Oregon Group, 1255 Pearl Street, Eugene, OR, 97401, USA

Gracelyn Cruden

UC San Diego, 9500 Gilman Drive, La Jolla, CA, 92093, USA

Erika L. Crable

Child and Adolescent Services Research Center, 3665 Kearny Villa Road, San Diego, CA, 92123, USA

UC San Diego ACTRI Dissemination and Implementation Science Center, 9500 Gilman Drive, La Jolla, CA, 92093, USA

University of Wisconsin-Madison, 610 North Whitney Way, Madison, WI, 53705, USA

Andrew Quanbeck

Kaiser Permanente Southern California, 200 North Lewis Street, Orange, CA, 92868, USA

Brian S. Mittman

University of Southern California, 2025 Zonal Avenue, Los Angeles, CA, 90089, USA

UCLA, 405 Hilgard Avenue, Los Angeles, CA, 90095, USA

University of Washington, 3980 15Th Avenue NE, Seattle, WA, 98195, USA

Anjuli D. Wagner

You can also search for this author in PubMed   Google Scholar

Contributions

Concept and design: BK, GC, EC, AQ, BM, and AW. Literature search and summary: BK, GC, EC, and AW. Initial drafting of the manuscript: BK. Critical revisions of the manuscript for important intellectual content: GC, EC, AQ, BM, and AW. All the authors read and approved the final manuscript.

Corresponding author

Correspondence to Bo Kim .

Ethics declarations

Ethics approval and consent to participate.

Not applicable. This work did not involve any humans as research participants.

Consent for publication

Not applicable. This article does not contain data from any individual person.

Competing interests

Dr. Quanbeck has a shareholder interest in CHESS Health, a small business that markets a digital health application for patients with substance use disorders to the addiction treatment field. The authors declare that they have no other competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

STROBE Statement—checklist of items that should be included in reports of observational studies.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Kim, B., Cruden, G., Crable, E.L. et al. A structured approach to applying systems analysis methods for examining implementation mechanisms. Implement Sci Commun 4 , 127 (2023). https://doi.org/10.1186/s43058-023-00504-5

Download citation

Received : 08 April 2023

Accepted : 23 September 2023

Published : 19 October 2023

DOI : https://doi.org/10.1186/s43058-023-00504-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Implementation mechanisms
• Implementation strategies
• Health care systems
• Systems engineering
• Systems science

Implementation Science Communications

ISSN: 2662-2211

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Henriksen K, Battles JB, Marks ES, et al., editors. Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology). Rockville (MD): Agency for Healthcare Research and Quality (US); 2005 Feb.

Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology).

Work system analysis: the key to understanding health care systems.

Ben-Tzion Karsh and Samuel J. Alper .

Affiliations

Many articles in the medical literature state that medical errors are the result of systems problems, require systems analyses, and can only be addressed with systems solutions. Within that same body of literature is a growing recognition that human factors engineering methods and design principles are needed to reduce medical errors and, hence, increase patient safety. Work system analysis methods, which are based on industrial and human factors engineering tools, have much to contribute toward patient safety, specifically because of their focus on systems. They offer principles and methods for analyzing systems, which, if followed, should help health care administrators and clinicians properly analyze their units or facilities, and should lead to more robust patient safety interventions. In this paper, steps for executing a work system analysis are provided. To facilitate comprehension of the steps, the medication administration system is used as an example.

• Introduction

Many articles in the medical literature state that medical errors are the result of systems problems, require systems analyses, and can only be addressed with systems solutions. 1 – 5 However, few articles have been written that specifically explain how to analyze a system so that systemwide problems can be uncovered and solutions implemented; this paper does just that.

System analysis can help manage and reduce risks by identifying hazards so they can be controlled through good design. That is, in order to improve safety, quality, performance, and comfort, a good place to start is by analyzing the involved systems so they can be improved. The key to improving safety and reducing risk is through good system design, which can only be achieved though a complete understanding of the system. To understand the system, it is essential to know how to analyze it. Industrial and human factors engineering work system analysis methods provide a set of tools that can be used to analyze health care systems.

Chapanis states that human factors engineering “discovers and applies information about human behavior, abilities, limitations, and other characteristics to the design of tools, machines, systems, tasks, jobs, and environments for productive, safe, comfortable, and effective human use.” 6, 7 Following many years of successful implementation in a diverse array of fields, human factors tools and methodologies are now gaining acceptance in health care. This is most notable in the two recent Institute of Medicine (IOM) reports, To Err Is Human 8 and Crossing the Quality Chasm, 9 which claim that human factors methods should be used to address patient safety and quality-of-care issues. Since then, a growing number of examples in the literature show that human factors tools and methods are being accepted and used in health care settings. 10 – 16

The increase in the use and popularity of human factors in health care can be traced to successes of human factors in fields such as mining, 17 nuclear power, 18 manufacturing, 19, 20 and aviation. 21, 22 Although these industries have successfully used human factors for decades, adoption of human factors tools and methodologies in health care has been slow. Human factors engineering methods and design guidelines have been slowly adopted in health care for some of the same reasons that health care is thought to have safety problems. Most prominent is the expectation that health care professionals will perform perfectly, which has resulted in a heavy focus on addressing individual performance versus system design. Cook and others 12 wrote, “The conventional view is that the [health care] system is safe by design but can be degraded by the failure of its human components.” From this perspective, all medical errors and adverse events are somebody's fault. In other words, a health care professional is more likely to get “redesigned” (retrained) than the system. Also working against a systems view is that “for physicians, the ever-present threat of malpractice litigation provides an additional incentive to keep silent about mistakes.” 4 These are system problems, and they contribute to the slow diffusion of human factors engineering methods and design guidelines throughout health care delivery systems precisely because they are antithetical to system analysis and design. 23, 24

Although this medical culture has existed for many years, health care has come to a proverbial fork in the road regarding patient safety and errors. This fork was inspired by research that uncovered a disturbing numbers of errors in health care. 10, 14, 25, 26 Currently, health care professionals are beginning to recognize the need to find tools to address the current patient safety problems—as well as patient safety problems that are yet unknown. This search has opened the door for the use of human factors engineering tools and methods in health care. For example, Weinger et al. 16 noted that “human factors research techniques, such as task analysis and workload assessment, may provide useful objective data on the structure and characteristics of the anesthesiologist's job and the impact of design innovations on task performance.” In the same domain, Cook and Woods 27 evaluated how anesthesiologists reacted to a new patient-monitoring technology. Anesthesiologists adapted the new technology to suit their needs using two methods: making the system compatible with their current cognitive strategies, and adapting their strategies for completing tasks to accommodate the constraints of the new system. That study showed how studying human-technology interactions can yield important information about the design of health care systems. Surgical errors have also been studied using human factors engineering techniques, 11 though the most common reason for using human factors methods in health care has been to study the possible impacts of new technologies, especially information technology, on patient safety. 13, 28 – 32

Importantly, system analysis techniques have also begun to uncover and solve health care safety problems. In one example, Patterson et al. 33 analyzed the work of nurses using barcoded medication administration technology. The system analysis uncovered new side effects of the technology, including unanticipated routes to adverse drug events. Using a structured work system analysis, Hallock et al. 34 used a sociotechnical system analysis to analyze and improve the safety of outpatient diagnostic testing systems at a large urban outpatient health care facility. Using many work system analytical tools, including variance matrices and variance control tables, hazards were discovered and solutions were proposed to reduce or eliminate the problems affecting the safety and quality of the preanalytic, analytic, and postanalytic phases of diagnostic testing. Clearly, many different human factors tools and methodologies, including system analysis, can be used to improve health care delivery. As Nolan noted, “The physician who is willing to learn about the nature of systems, how to control them, and how to improve them can significantly influence medical systems.” 3

• Work system analysis

System analysis has much to contribute to patient safety, specifically through its study of organizational and work systems. In general, a system analysis yields an understanding of how a system works and how different elements in a system interact. This facilitates system design and system redesign, and aims to improve the interface between components of a system in order to enhance the functioning of each individual component in the overall system. Adopting a systems approach to error reduction requires a shift from blaming individuals for errors to analyzing systems to uncover design flaws, thus moving from addressing problems reactively (i.e., after problems occur) to proactively preventing accidents through system analysis and design.

Although many different methods have been used to conduct system analysis in industry, few methods have been widely used in health care. System analysis methodologies include, among others, the macroergonomic analysis and design (MEAD), 35, 36 fault tree analysis, 37, 38 failure modes and effects analysis (FMEA), 39, 40 health care failure modes and effects analysis (HFMEA), 41 and probabilistic risk assessment (PRA). 42, 43 Each of these methods uses similar principles to analyze and determine the weaknesses of the system and facilitate its redesign. In the remainder of this paper, the main steps that these methods share are identified and explained in detail.

Key systems terms

Before presenting the main steps in a system analysis, an understanding of system terms must be developed. To facilitate reader comprehension of the terms and steps in the system analysis, the medication administration system will be used as an example throughout the remainder of the paper.

• System element: A system element is anything that is part of a particular system. Elements can include people, technologies, policies, lighting, furniture, and jobs. In the case of the medication administration system, elements include the administering nurses, patients, medications, medication administration record (MAR), medication stock room, patient rooms, and identification bands.
• System attribute: System attributes are the perceived characteristics of the system. The medication administration system attributes could include “error-free,” “time consuming,” “chaotic,” and “high quality.”
• Temporal boundary: A temporal boundary separates systems in time. For the medication administration system, a temporal boundary could be drawn between the first and second shift.
• Spatial boundary: A spatial boundary separates systems in space. An example could be the medication administration system for one particular unit versus that of another unit.
• Process boundary: A process boundary separates systems into adjacent component processes, also known as subprocesses. The medication use system contains component processes of ordering, transcribing, verifying, dispensing, administering, and documenting. An example of a process boundary might then be the boundary between the process of dispensing and delivering medications to the unit and the process of administering the medication.
• Hierarchical boundary: A hierarchical boundary separates systems by their location in a hierarchy of systems. For example, the medication administration system exists within a larger system known as a unit. The unit exists within a larger system of a hospital. A hospital exists within a larger community health system.
• System input: A system input is anything necessary to energize the system. For medication administration, inputs include nurses who administer drugs, drugs, MARs, physician orders, and pharmacy dispensing. These elements are inputs because they are necessary for medication administration to take place.
• Transformation: Transformations are processes that turn inputs into outputs; they are actions in the system. The action of administering a medication to a patient is a transformation of an input (i.e., a medication) into an output (i.e., a medicated patient). However, many other transformations take place in the medication administration system. These include patient manipulations, patient monitoring, retrieving drugs from medication carts or cabinets, and reading MARs.
• Outputs: Outputs are the results of transformations. For example, the output of administering a medication is a medicated patient.
• Unit operation: A unit operation is a simple input-transformation-output process that does not contain any other input-transformation-output processes. It is the most basic component process of interest. For example, within the larger process of administering medications, which might have a process boundary that starts with a nurse examining a MAR and ends when the administration is documented, there are a number of unit operations. Each of the following actions is an example of a unit operation: check MAR, locate medication, compare medication to MAR, locate patient, identify patient, administer medication, and document administration.

Conducting a system analysis

What follows are detailed explanations of 10 steps common to the various system analysis methodologies. These steps, if followed, should help to ensure an effective analysis and thus effective system redesign.

Decide what system will be the subject of the analysis. This determination dictates the direction of the remaining steps. For example, this might be the entire medication system, from physician order to documented administration, or just the ordering system; it might be the nurse shift transition, or just the communication system between unit nurses and the charge nurse.

Produce a preliminary work system map. This often-overlooked step is critical to the success of the entire analysis. The purpose of this map is to identify inputs and outputs relevant to the system, which facilitates the identification of people who should be represented on the analysis team (step 3). Furthermore, by understanding the inputs and outputs of the system being studied, the system boundaries can be determined (step 5).

A work system map is similar to a workflow diagram, but it contains additional detail. A traditional workflow diagram presents steps in a process, including decision points. A work system map also provides further detail for each step in the process, by identifying at least six additional pieces of information:

• What technology is used
• What policies and rules (internal to the unit and external to the organization) are involved in determining how, when, why, or where the step is executed
• What supervision is involved in the step
• What environmental factors (e.g., lighting, noise, vibration) might affect the step or how it is executed
• What other people might influence the execution of this step or determine whether the step takes place
• What information is needed for the execution of this step

This preliminary work system map can be assembled through brainstorming or short observations. Because it is preliminary and designed for identifying team members and system boundaries, it does not have to be exhaustive at this point. It will be modified throughout the work system analysis.

Use the preliminary work system map to determine who should be represented on the team that will carry out the analysis. The importance of good representation cannot be overestimated. Without representation from all involved stakeholders, it is likely that the team will lack the expertise necessary to correctly analyze the system, identify hazards, and control hazards. Stakeholders will likely include sharp-end employees as well as blunt-end employees, like supervisors and managers. One way to identify who should be represented is to consider groups and individuals who (a) communicate with frontline employees, (b) provide inputs, (c) receive outputs, and (d) assume responsibility for various parts of the process. The team must also contain at least one expert in system analysis to guide the process. Once the team is assembled, its first task is to examine the preliminary work system map to make sure all stakeholders are represented.

The assembled team should conduct an initial scan of the system. An initial scan has two scanning components. First, the team studies the preliminary work system map and gauges its accuracy. Several questions can guide this process:

• What other people take part in this system? Where do they come from: other units or temporary agencies? Are there other people who provide information relevant to the system?
• What other technologies, tools, or equipment are used in this process? Where do they come from? Who is responsible for maintaining them?
• What other policies or regulations are relevant to the steps in this process? Who develops them? Who enforces them?
• What other organizational variables, such as reward systems or communication systems, affect the performance of this process? Who is in charge of these?
• What other jobs or tasks affect the performance of the steps in the process? Who carries out these jobs? Who is responsible for them?
• What other environmental factors impact the system?
• What people, technologies, tools, equipment, processes, and/or jobs are impacted by the process under study?

The team members can also informally interview stakeholders and observe steps to help complete this part of the scan. If the process of agreeing on the work system map leads to the identification of additional stakeholders, representatives of those stakeholders should be invited to join the team.

The second component of the initial scan is to “scan the horizons.” The team will investigate any pending decisions, policies, or regulations, internal and external to the organization, that could impact the system. Similarly, the team will want to determine if any internal process changes are planned and if any new technologies are scheduled for implementation. This information must be incorporated into subsequent redesign plans. If this “horizon scan” is not completed, the entire system redesign plan could be rendered ineffective because it may not integrate with pending changes.

Put boundaries on the system under study. The team needs to determine process, hierarchical, temporal, and spatial boundaries. If these boundaries are too narrow, the team may miss important variables that contribute to the performance of the system; if they are too broad, the analysis may take up unnecessary resources. Setting the boundaries too broad is less dangerous than setting them too narrow. Setting broad boundaries usually results in increased analysis time. Setting boundaries too narrow can result in misrepresentation of the system, which can lead to poor or dangerous redesign ideas. For example, bounding the medication administration process to include only the day shift may not capture differences between shifts (e.g., level of supervision or number of medications administered), which would limit the effectiveness of redesign proposals. Similarly, if the medication administration system is bounded to include only the steps from reviewing the MAR to documenting the medication, important inputs from pharmacy processes would likely be overlooked. In that case, hazardous processes in the pharmacy that lead to administration errors might be left unchanged.

Performance expectations for each step in the system should now be determined. Performance expectations are quantitative or qualitative statements that describe what outcomes should come from each unit operation, component process, or overall process studied in the system. For example, certain processes should be done safely, some should be done quickly and safely, and others may have production and quality goals. The team should do its best to determine specific, measurable performance goals. For example, instead of stating that a step should be done “safely,” the performance expectation might be “zero medication administration errors.” Similarly, instead of stating that a process should be completed “quickly,” the team could determine that the process should be completed “within 15 minutes of receiving the order.” A process can have one or more performance expectations. Performance expectations are important because they provide the team with measures by which to evaluate the performance of the current and redesigned systems. They also provide criteria on which to evaluate the impact of failures, weaknesses, or hazards identified in the data analysis.

The team should begin formal data collection to revise and update the work system map, gauge the current performance of the system, and determine baseline measures that will be used to evaluate the effectiveness of the redesign. Data can be collected through time studies, administrative databases, maintenance records, structured observations of the process, and interviews of the involved stakeholders. Interviews should be used to collect details about the system elements and attributes, and to reconcile and/or clarify data collected from observations. This data collection step is typically the lengthiest part of the system analysis, because extensive data collection is required for an accurate map. Depending on the system being studied, this could take days, weeks, or months. Questions and guidelines presented in steps 2 and 4 should be used to develop the data collection tools for observers and interviewers. Once the team puts together the work system map, stakeholders outside of the analysis team should be invited to examine the map, to make necessary changes, and to validate its accuracy.

The team can begin analyzing the collected data. The purposes of the analysis are to (a) identify weaknesses, variances, and any series of events that could cause the system to fail; and (b) prioritize the identified problems for redesign. This can be executed qualitatively, quantitatively, or using mixed methods, depending on the system analysis method used (e.g., failure modes and effects analysis, sociotechnical system analysis, probabilistic risk assessment). Typically, a combination of analytical methods will provide better analysis than any one method, because one method may compensate for the weaknesses of another. Readers are advised to consult the references provided, as well as other sources, for details on using specific methods. It is important to understand that using quantitative analytical methods does not increase the objectivity of the analysis. Objectivity only increases to the extent that the analysis is based on valid and reliable data, which is true for both qualitative and quantitative analysis.

Once hazards (i.e., causes of failure modes or variances) have been identified, control strategies should be developed. These strategies should be based on the hierarchy of hazard control, which states that the best hazard controls are those that completely eliminate the hazard. The next best option is to guard against the hazard, and the two least effective controls are training and warning. To help facilitate the development of effective control strategies, a hazard-control matrix should be developed. A hazard-control matrix, which is based on the variance control table, 35 has 10 headings in the top row, as shown in the example in Table 1 .

Hazard-control matrix: example of one hypothetical hazard in the diagnostic testing process.

All of the hazards identified for control are then listed underneath the first heading, and the rest of the cells are filled in. This matrix displays the identified problems, current system status, and proposed problem solutions. The matrix also facilitates progress tracking by displaying solution status and the names of individuals responsible for the solutions.

The final step in a work system analysis is to conduct a system analysis on the redesign hazard-control ideas that the team develops. This should be done before redesigns are implemented so the team is confident its proposed redesign ideas do not unintentionally reduce the effectiveness of the system nor create new safety problems. Each redesign idea or hazard-control idea should subsequently be subjected to a full system analysis (i.e., develop a new process map with the redesign idea, compare it with the original system design, identify hazards, see if they need to be controlled). Once completed, pilot testing and implementation can begin.

• Conclusions

The Joint Commission on Accreditation of Healthcare Organizations (JCAHO) currently requires all hospitals to conduct at least one annual risk assessment for high-risk processes and redesign the process to minimize the likelihood of failure. 44 To meet this requirement, a health care organization must identify a target system and conduct a system analysis on it, precisely what has been discussed in this paper. However, the JCAHO standard is only a minimum requirement, and requires only one such analysis to be conducted each year. But a work system analysis is warranted anytime that a health care organization intends to change a system, implement a new technology, update a process, or modify job requirements. Such an analysis should provide details regarding the current state of the system, which is necessary to accurately identify problems and their causes before patient or employee safety is affected. Once problems and causes are identified and prioritized, system analysis techniques can be used to determine effective approaches to redesign the current system into a safer one. This paper provides detailed steps as a guide through the process of conducting a system analysis. With this information, health care administrators and clinicians should be able to more confidently proceed with transforming their organizations into safe systems.

• Acknowledgments

This work was funded in part by two grants from the Agency for Healthcare Research and Quality: R01 HS013610-01 (Principal Investigator—B. Karsh) and P20 HS11561-01 (Principal Investigator—P. Carayon).

• Cite this Page Karsh BT, Alper SJ. Work System Analysis: The Key to Understanding Health Care Systems. In: Henriksen K, Battles JB, Marks ES, et al., editors. Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology). Rockville (MD): Agency for Healthcare Research and Quality (US); 2005 Feb.
• PDF version of this page (184K)

In this Page

Other titles in this collection.

• Advances in Patient Safety

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Similar articles in PubMed

• Making inpatient medication reconciliation patient centered, clinically relevant and implementable: a consensus statement on key principles and necessary first steps. [J Hosp Med. 2010] Making inpatient medication reconciliation patient centered, clinically relevant and implementable: a consensus statement on key principles and necessary first steps. Greenwald JL, Halasyamani L, Greene J, LaCivita C, Stucky E, Benjamin B, Reid W, Griffin FA, Vaida AJ, Williams MV. J Hosp Med. 2010 Oct; 5(8):477-85.
• Review Translational Metabolomics of Head Injury: Exploring Dysfunctional Cerebral Metabolism with Ex Vivo NMR Spectroscopy-Based Metabolite Quantification. [Brain Neurotrauma: Molecular, ...] Review Translational Metabolomics of Head Injury: Exploring Dysfunctional Cerebral Metabolism with Ex Vivo NMR Spectroscopy-Based Metabolite Quantification. Wolahan SM, Hirt D, Glenn TC. Brain Neurotrauma: Molecular, Neuropsychological, and Rehabilitation Aspects. 2015
• Review Medication errors in the intensive care unit: literature review using the SEIPS model. [AACN Adv Crit Care. 2013] Review Medication errors in the intensive care unit: literature review using the SEIPS model. Frith KH. AACN Adv Crit Care. 2013 Oct-Dec; 24(4):389-404.
• Example of a Human Factors Engineering approach to a medication administration work system: potential impact on patient safety. [Int J Med Inform. 2010] Example of a Human Factors Engineering approach to a medication administration work system: potential impact on patient safety. Beuscart-Zéphir MC, Pelayo S, Bernonville S. Int J Med Inform. 2010 Apr; 79(4):e43-57. Epub 2009 Sep 8.
• Review Human factors and safe patient care. [J Nurs Manag. 2009] Review Human factors and safe patient care. Norris B. J Nurs Manag. 2009 Mar; 17(2):203-11.

Recent Activity

• Work System Analysis: The Key to Understanding Health Care Systems - Advances in... Work System Analysis: The Key to Understanding Health Care Systems - Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology)

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

• Survey paper
• Open access
• Published: 03 May 2022

A systematic review and research perspective on recommender systems

• Deepjyoti Roy   ORCID: orcid.org/0000-0002-8020-7145 1 &
• Mala Dutta 1  

Journal of Big Data volume  9 , Article number:  59 ( 2022 ) Cite this article

56k Accesses

85 Citations

6 Altmetric

Metrics details

Recommender systems are efficient tools for filtering online information, which is widespread owing to the changing habits of computer users, personalization trends, and emerging access to the internet. Even though the recent recommender systems are eminent in giving precise recommendations, they suffer from various limitations and challenges like scalability, cold-start, sparsity, etc. Due to the existence of various techniques, the selection of techniques becomes a complex work while building application-focused recommender systems. In addition, each technique comes with its own set of features, advantages and disadvantages which raises even more questions, which should be addressed. This paper aims to undergo a systematic review on various recent contributions in the domain of recommender systems, focusing on diverse applications like books, movies, products, etc. Initially, the various applications of each recommender system are analysed. Then, the algorithmic analysis on various recommender systems is performed and a taxonomy is framed that accounts for various components required for developing an effective recommender system. In addition, the datasets gathered, simulation platform, and performance metrics focused on each contribution are evaluated and noted. Finally, this review provides a much-needed overview of the current state of research in this field and points out the existing gaps and challenges to help posterity in developing an efficient recommender system.

Introduction

The recent advancements in technology along with the prevalence of online services has offered more abilities for accessing a huge amount of online information in a faster manner. Users can post reviews, comments, and ratings for various types of services and products available online. However, the recent advancements in pervasive computing have resulted in an online data overload problem. This data overload complicates the process of finding relevant and useful content over the internet. The recent establishment of several procedures having lower computational requirements can however guide users to the relevant content in a much easy and fast manner. Because of this, the development of recommender systems has recently gained significant attention. In general, recommender systems act as information filtering tools, offering users suitable and personalized content or information. Recommender systems primarily aim to reduce the user’s effort and time required for searching relevant information over the internet.

Nowadays, recommender systems are being increasingly used for a large number of applications such as web [ 1 , 67 , 70 ], books [ 2 ], e-learning [ 4 , 16 , 61 ], tourism [ 5 , 8 , 78 ], movies [ 66 ], music [ 79 ], e-commerce, news, specialized research resources [ 65 ], television programs [ 72 , 81 ], etc. It is therefore important to build high-quality and exclusive recommender systems for providing personalized recommendations to the users in various applications. Despite the various advances in recommender systems, the present generation of recommender systems requires further improvements to provide more efficient recommendations applicable to a broader range of applications. More investigation of the existing latest works on recommender systems is required which focus on diverse applications.

There is hardly any review paper that has categorically synthesized and reviewed the literature of all the classification fields and application domains of recommender systems. The few existing literature reviews in the field cover just a fraction of the articles or focus only on selected aspects such as system evaluation. Thus, they do not provide an overview of the application field, algorithmic categorization, or identify the most promising approaches. Also, review papers often neglect to analyze the dataset description and the simulation platforms used. This paper aims to fulfil this significant gap by reviewing and comparing existing articles on recommender systems based on a defined classification framework, their algorithmic categorization, simulation platforms used, applications focused, their features and challenges, dataset description and system performance. Finally, we provide researchers and practitioners with insight into the most promising directions for further investigation in the field of recommender systems under various applications.

In essence, recommender systems deal with two entities—users and items, where each user gives a rating (or preference value) to an item (or product). User ratings are generally collected by using implicit or explicit methods. Implicit ratings are collected indirectly from the user through the user’s interaction with the items. Explicit ratings, on the other hand, are given directly by the user by picking a value on some finite scale of points or labelled interval values. For example, a website may obtain implicit ratings for different items based on clickstream data or from the amount of time a user spends on a webpage and so on. Most recommender systems gather user ratings through both explicit and implicit methods. These feedbacks or ratings provided by the user are arranged in a user-item matrix called the utility matrix as presented in Table 1 .

The utility matrix often contains many missing values. The problem of recommender systems is mainly focused on finding the values which are missing in the utility matrix. This task is often difficult as the initial matrix is usually very sparse because users generally tend to rate only a small number of items. It may also be noted that we are interested in only the high user ratings because only such items would be suggested back to the users. The efficiency of a recommender system greatly depends on the type of algorithm used and the nature of the data source—which may be contextual, textual, visual etc.

Types of recommender systems

Recommender systems are broadly categorized into three different types viz. content-based recommender systems, collaborative recommender systems and hybrid recommender systems. A diagrammatic representation of the different types of recommender systems is given in Fig.  1 .

Content-based recommender system

In content-based recommender systems, all the data items are collected into different item profiles based on their description or features. For example, in the case of a book, the features will be author, publisher, etc. In the case of a movie, the features will be the movie director, actor, etc. When a user gives a positive rating to an item, then the other items present in that item profile are aggregated together to build a user profile. This user profile combines all the item profiles, whose items are rated positively by the user. Items present in this user profile are then recommended to the user, as shown in Fig.  2 .

One drawback of this approach is that it demands in-depth knowledge of the item features for an accurate recommendation. This knowledge or information may not be always available for all items. Also, this approach has limited capacity to expand on the users' existing choices or interests. However, this approach has many advantages. As user preferences tend to change with time, this approach has the quick capability of dynamically adapting itself to the changing user preferences. Since one user profile is specific only to that user, this algorithm does not require the profile details of any other users because they provide no influence in the recommendation process. This ensures the security and privacy of user data. If new items have sufficient description, content-based techniques can overcome the cold-start problem i.e., this technique can recommend an item even when that item has not been previously rated by any user. Content-based filtering approaches are more common in systems like personalized news recommender systems, publications, web pages recommender systems, etc.

Collaborative filtering-based recommender system

Collaborative approaches make use of the measure of similarity between users. This technique starts with finding a group or collection of user X whose preferences, likes, and dislikes are similar to that of user A. X is called the neighbourhood of A. The new items which are liked by most of the users in X are then recommended to user A. The efficiency of a collaborative algorithm depends on how accurately the algorithm can find the neighbourhood of the target user. Traditionally collaborative filtering-based systems suffer from the cold-start problem and privacy concerns as there is a need to share user data. However, collaborative filtering approaches do not require any knowledge of item features for generating a recommendation. Also, this approach can help to expand on the user’s existing interests by discovering new items. Collaborative approaches are again divided into two types: memory-based approaches and model-based approaches.

Memory-based collaborative approaches recommend new items by taking into consideration the preferences of its neighbourhood. They make use of the utility matrix directly for prediction. In this approach, the first step is to build a model. The model is equal to a function that takes the utility matrix as input.

Model = f (utility matrix)

Then recommendations are made based on a function that takes the model and user profile as input. Here we can make recommendations only to users whose user profile belongs to the utility matrix. Therefore, to make recommendations for a new user, the user profile must be added to the utility matrix, and the similarity matrix should be recomputed, which makes this technique computation heavy.

Recommendation = f (defined model, user profile) where user profile  ∈  utility matrix

Memory-based collaborative approaches are again sub-divided into two types: user-based collaborative filtering and item-based collaborative filtering. In the user-based approach, the user rating of a new item is calculated by finding other users from the user neighbourhood who has previously rated that same item. If a new item receives positive ratings from the user neighbourhood, the new item is recommended to the user. Figure  3 depicts the user-based filtering approach.

User-based collaborative filtering

In the item-based approach, an item-neighbourhood is built consisting of all similar items which the user has rated previously. Then that user’s rating for a different new item is predicted by calculating the weighted average of all ratings present in a similar item-neighbourhood as shown in Fig.  4 .

Item-based collaborative filtering

Model-based systems use various data mining and machine learning algorithms to develop a model for predicting the user’s rating for an unrated item. They do not rely on the complete dataset when recommendations are computed but extract features from the dataset to compute a model. Hence the name, model-based technique. These techniques also need two steps for prediction—the first step is to build the model, and the second step is to predict ratings using a function (f) which takes the model defined in the first step and the user profile as input.

Recommendation = f (defined model, user profile) where user profile  ∉  utility matrix

Model-based techniques do not require adding the user profile of a new user into the utility matrix before making predictions. We can make recommendations even to users that are not present in the model. Model-based systems are more efficient for group recommendations. They can quickly recommend a group of items by using the pre-trained model. The accuracy of this technique largely relies on the efficiency of the underlying learning algorithm used to create the model. Model-based techniques are capable of solving some traditional problems of recommender systems such as sparsity and scalability by employing dimensionality reduction techniques [ 86 ] and model learning techniques.

Hybrid filtering

A hybrid technique is an aggregation of two or more techniques employed together for addressing the limitations of individual recommender techniques. The incorporation of different techniques can be performed in various ways. A hybrid algorithm may incorporate the results achieved from separate techniques, or it can use content-based filtering in a collaborative method or use a collaborative filtering technique in a content-based method. This hybrid incorporation of different techniques generally results in increased performance and increased accuracy in many recommender applications. Some of the hybridization approaches are meta-level, feature-augmentation, feature-combination, mixed hybridization, cascade hybridization, switching hybridization and weighted hybridization [ 86 ]. Table 2 describes these approaches.

Recommender system challenges

This section briefly describes the various challenges present in current recommender systems and offers different solutions to overcome these challenges.

Cold start problem

The cold start problem appears when the recommender system cannot draw any inference from the existing data, which is insufficient. Cold start refers to a condition when the system cannot produce efficient recommendations for the cold (or new) users who have not rated any item or have rated a very few items. It generally arises when a new user enters the system or new items (or products) are inserted into the database. Some solutions to this problem are as follows: (a) Ask new users to explicitly mention their item preference. (b) Ask a new user to rate some items at the beginning. (c) Collect demographic information (or meta-data) from the user and recommend items accordingly.

Shilling attack problem

This problem arises when a malicious user fakes his identity and enters the system to give false item ratings [ 87 ]. Such a situation occurs when the malicious user wants to either increase or decrease some item’s popularity by causing a bias on selected target items. Shilling attacks greatly reduce the reliability of the system. One solution to this problem is to detect the attackers quickly and remove the fake ratings and fake user profiles from the system.

Synonymy problem

This problem arises when similar or related items have different entries or names, or when the same item is represented by two or more names in the system [ 78 ]. For example, babywear and baby cloth. Many recommender systems fail to distinguish these differences, hence reducing their recommendation accuracy. To alleviate this problem many methods are used such as demographic filtering, automatic term expansion and Singular Value Decomposition [ 76 ].

Latency problem

The latency problem is specific to collaborative filtering approaches and occurs when new items are frequently inserted into the database. This problem is characterized by the system’s failure to recommend new items. This happens because new items must be reviewed before they can be recommended in a collaborative filtering environment. Using content-based filtering may resolve this issue, but it may introduce overspecialization and decrease the computing time and system performance. To increase performance, the calculations can be done in an offline environment and clustering-based techniques can be used [ 76 ].

Sparsity problem

Data sparsity is a common problem in large scale data analysis, which arises when certain expected values are missing in the dataset. In the case of recommender systems, this situation occurs when the active users rate very few items. This reduces the recommendation accuracy. To alleviate this problem several techniques can be used such as demographic filtering, singular value decomposition and using model-based collaborative techniques.

Grey sheep problem

The grey sheep problem is specific to pure collaborative filtering approaches where the feedback given by one user do not match any user neighbourhood. In this situation, the system fails to accurately predict relevant items for that user. This problem can be resolved by using pure content-based approaches where predictions are made based on the user’s profile and item properties.

Scalability problem

Recommender systems, especially those employing collaborative filtering techniques, require large amounts of training data, which cause scalability problems. The scalability problem arises when the amount of data used as input to a recommender system increases quickly. In this era of big data, more and more items and users are rapidly getting added to the system and this problem is becoming common in recommender systems. Two common approaches used to solve the scalability problem is dimensionality reduction and using clustering-based techniques to find users in tiny clusters instead of the complete database.

Methodology

The purpose of this study is to understand the research trends in the field of recommender systems. The nature of research in recommender systems is such that it is difficult to confine each paper to a specific discipline. This can be further understood by the fact that research papers on recommender systems are scattered across various journals such as computer science, management, marketing, information technology and information science. Hence, this literature review is conducted over a wide range of electronic journals and research databases such as ACM Portal, IEEE/IEE Library, Google Scholars and Science Direct [ 88 ].

The search process of online research articles was performed based on 6 descriptors: “Recommender systems”, “Recommendation systems”, “Movie Recommend*”, “Music Recommend*”, “Personalized Recommend*”, “Hybrid Recommend*”. The following research papers described below were excluded from our research:

News articles.

Master’s dissertations.

Non-English papers.

Unpublished papers.

Research papers published before 2011.

We have screened a total of 350 articles based on their abstracts and content. However, only research papers that described how recommender systems can be applied were chosen. Finally, 60 papers were selected from top international journals indexed in Scopus or E-SCI in 2021. We now present the PRISMA flowchart of the inclusion and exclusion process in Fig.  5 .

PRISMA flowchart of the inclusion and exclusion process. Abstract and content not suitable to the study: * The use or application of the recommender system is not specified: **

Each paper was carefully reviewed and classified into 6 categories in the application fields and 3 categories in the techniques used to develop the system. The classification framework is presented in Fig.  6 .

Classification framework

The number of relevant articles come from Expert Systems with Applications (23%), followed by IEEE (17%), Knowledge-Based System (17%) and Others (43%). Table 3 depicts the article distribution by journal title and Table 4 depicts the sector-wise article distribution.

Both forward and backward searching techniques were implemented to establish that the review of 60 chosen articles can represent the domain literature. Hence, this paper can demonstrate its validity and reliability as a literature review.

Review on state-of-the-art recommender systems

This section presents a state-of-art literature review followed by a chronological review of the various existing recommender systems.

Literature review

In 2011, Castellano et al. [ 1 ] developed a “NEuro-fuzzy WEb Recommendation (NEWER)” system for exploiting the possibility of combining computational intelligence and user preference for suggesting interesting web pages to the user in a dynamic environment. It considered a set of fuzzy rules to express the correlations between user relevance and categories of pages. Crespo et al. [ 2 ] presented a recommender system for distance education over internet. It aims to recommend e-books to students using data from user interaction. The system was developed using a collaborative approach and focused on solving the data overload problem in big digital content. Lin et al. [ 3 ] have put forward a recommender system for automatic vending machines using Genetic algorithm (GA), k-means, Decision Tree (DT) and Bayesian Network (BN). It aimed at recommending localized products by developing a hybrid model combining statistical methods, classification methods, clustering methods, and meta-heuristic methods. Wang and Wu [ 4 ] have implemented a ubiquitous learning system for providing personalized learning assistance to the learners by combining the recommendation algorithm with a context-aware technique. It employed the Association Rule Mining (ARM) technique and aimed to increase the effectiveness of the learner’s learning. García-Crespo et al. [ 5 ] presented a “semantic hotel” recommender system by considering the experiences of consumers using a fuzzy logic approach. The system considered both hotel and customer characteristics. Dong et al. [ 6 ] proposed a structure for a service-concept recommender system using a semantic similarity model by integrating the techniques from the view of an ontology structure-oriented metric and a concept content-oriented metric. The system was able to deliver optimal performance when compared with similar recommender systems. Li et al. [ 7 ] developed a Fuzzy linguistic modelling-based recommender system for assisting users to find experts in knowledge management systems. The developed system was applied to the aircraft industry where it demonstrated efficient and feasible performance. Lorenzi et al. [ 8 ] presented an “assumption-based multiagent” system to make travel package recommendations using user preferences in the tourism industry. It performed different tasks like discovering, filtering, and integrating specific information for building a travel package following the user requirement. Huang et al. [ 9 ] proposed a context-aware recommender system through the extraction, evaluation and incorporation of contextual information gathered using the collaborative filtering and rough set model.

In 2012, Chen et al. [ 10 ] presented a diabetes medication recommender model by using “Semantic Web Rule Language (SWRL) and Java Expert System Shell (JESS)” for aggregating suitable prescriptions for the patients. It aimed at selecting the most suitable drugs from the list of specific drugs. Mohanraj et al. [ 11 ] developed the “Ontology-driven bee’s foraging approach (ODBFA)” to accurately predict the online navigations most likely to be visited by a user. The self-adaptive system is intended to capture the various requirements of the online user by using a scoring technique and by performing a similarity comparison. Hsu et al. [ 12 ] proposed a “personalized auxiliary material” recommender system by considering the specific course topics, individual learning styles, complexity of the auxiliary materials using an artificial bee colony algorithm. Gemmell et al. [ 13 ] demonstrated a solution for the problem of resource recommendation in social annotation systems. The model was developed using a linear-weighted hybrid method which was capable of providing recommendations under different constraints. Choi et al. [ 14 ] proposed one “Hybrid Online-Product rEcommendation (HOPE) system” by the integration of collaborative filtering through sequential pattern analysis-based recommendations and implicit ratings. Garibaldi et al. [ 15 ] put forward a technique for incorporating the variability in a fuzzy inference model by using non-stationary fuzzy sets for replicating the variabilities of a human. This model was applied to a decision problem for treatment recommendations of post-operative breast cancer.

In 2013, Salehi and Kmalabadi [ 16 ] proposed an e-learning material recommender system by “modelling of materials in a multidimensional space of material’s attribute”. It employed both content and collaborative filtering. Aher and Lobo [ 17 ] introduced a course recommender system using data mining techniques such as simple K-means clustering and Association Rule Mining (ARM) algorithm. The proposed e-learning system was successfully demonstrated for “MOOC (Massively Open Online Courses)”. Kardan and Ebrahimi [ 18 ] developed a hybrid recommender system for recommending posts in asynchronous discussion groups. The system was built combining both collaborative filtering and content-based filtering. It considered implicit user data to compute the user similarity with various groups, for recommending suitable posts and contents to its users. Chang et al. [ 19 ] adopted a cloud computing technology for building a TV program recommender system. The system designed for digital TV programs was implemented using Hadoop Fair Scheduler (HFC), K-means clustering and k-nearest neighbour (KNN) algorithms. It was successful in processing huge amounts of real-time user data. Lucas et al. [ 20 ] implemented a recommender model for assisting a tourism application by using associative classification and fuzzy logic to predict the context. Niu et al. [ 21 ] introduced “Affivir: An Affect-based Internet Video Recommendation System” which was developed by calculating user preferences and by using spectral clustering. This model recommended videos with similar effects, which was processed to get optimal results with dynamic adjustments of recommendation constraints.

In 2014, Liu et al. [ 22 ] implemented a new route recommendation model for offering personalized and real-time route recommendations for self-driven tourists to minimize the queuing time and traffic jams infamous tourist places. Recommendations were carried out by considering the preferences of users. Bakshi et al. [ 23 ] proposed an unsupervised learning-based recommender model for solving the scalability problem of recommender systems. The algorithm used transitive similarities along with Particle Swarm Optimization (PSO) technique for discovering the global neighbours. Kim and Shim [ 24 ] proposed a recommender system based on “latent Dirichlet allocation using probabilistic modelling for Twitter” that could recommend the top-K tweets for a user to read, and the top-K users to follow. The model parameters were learned from an inference technique by using the differential Expectation–Maximization (EM) algorithm. Wang et al. [ 25 ] developed a hybrid-movie recommender model by aggregating a genetic algorithm (GA) with improved K-means and Principal Component Analysis (PCA) technique. It was able to offer intelligent movie recommendations with personalized suggestions. Kolomvatsos et al. [ 26 ] proposed a recommender system by considering an optimal stopping theory for delivering books or music recommendations to the users. Gottschlich et al. [ 27 ] proposed a decision support system for stock investment recommendations. It computed the output by considering the overall crowd’s recommendations. Torshizi et al. [ 28 ] have introduced a hybrid recommender system to determine the severity level of a medical condition. It could recommend suitable therapies for patients suffering from Benign Prostatic Hyperplasia.

In 2015, Zahálka et al. [ 29 ] proposed a venue recommender: “City Melange”. It was an interactive content-based model which used the convolutional deep-net features of the visual domain and the linear Support Vector Machine (SVM) model to capture the semantic information and extract latent topics. Sankar et al. [ 30 ] have proposed a stock recommender system based on the stock holding portfolio of trusted mutual funds. The system employed the collaborative filtering approach along with social network analysis for offering a decision support system to build a trust-based recommendation model. Chen et al. [ 31 ] have put forward a novel movie recommender system by applying the “artificial immune network to collaborative filtering” technique. It computed the affinity of an antigen and the affinity between an antibody and antigen. Based on this computation a similarity estimation formula was introduced which was used for the movie recommendation process. Wu et al. [ 32 ] have examined the technique of data fusion for increasing the efficiency of item recommender systems. It employed a hybrid linear combination model and used a collaborative tagging system. Yeh and Cheng [ 33 ] have proposed a recommender system for tourist attractions by constructing the “elicitation mechanism using the Delphi panel method and matrix construction mechanism using the repertory grids”, which was developed by considering the user preference and expert knowledge.

In 2016, Liao et al. [ 34 ] proposed a recommender model for online customers using a rough set association rule. The model computed the probable behavioural variations of online consumers and provided product category recommendations for e-commerce platforms. Li et al. [ 35 ] have suggested a movie recommender system based on user feedback collected from microblogs and social networks. It employed the sentiment-aware association rule mining algorithm for recommendations using the prior information of frequent program patterns, program metadata similarity and program view logs. Wu et al. [ 36 ] have developed a recommender system for social media platforms by aggregating the technique of Social Matrix Factorization (SMF) and Collaborative Topic Regression (CTR). The model was able to compute the ratings of users to items for making recommendations. For improving the recommendation quality, it gathered information from multiple sources such as item properties, social networks, feedback, etc. Adeniyi et al. [ 37 ] put forward a study of automated web-usage data mining and developed a recommender system that was tested in both real-time and online for identifying the visitor’s or client’s clickstream data.

In 2017, Rawat and Kankanhalli [ 38 ] have proposed a viewpoint recommender system called “ClickSmart” for assisting mobile users to capture high-quality photographs at famous tourist places. Yang et al. [ 39 ] proposed a gradient boosting-based job recommendation system for satisfying the cost-sensitive requirements of the users. The hybrid algorithm aimed to reduce the rate of unnecessary job recommendations. Lee et al. [ 40 ] proposed a music streaming recommender system based on smartphone activity usage. The proposed system benefitted by using feature selection approaches with machine learning techniques such as Naive Bayes (NB), Support Vector Machine (SVM), Multi-layer Perception (MLP), Instance-based k -Nearest Neighbour (IBK), and Random Forest (RF) for performing the activity detection from the mobile signals. Wei et al. [ 41 ] have proposed a new stacked denoising autoencoder (SDAE) based recommender system for cold items. The algorithm employed deep learning and collaborative filtering method to predict the unknown ratings.

In 2018, Li et al. [ 42 ] have developed a recommendation algorithm using Weighted Linear Regression Models (WLRRS). The proposed system was put to experiment using the MovieLens dataset and it presented better classification and predictive accuracy. Mezei and Nikou [ 43 ] presented a mobile health and wellness recommender system based on fuzzy optimization. It could recommend a collection of actions to be taken by the user to improve the user’s health condition. Recommendations were made considering the user’s physical activities and preferences. Ayata et al. [ 44 ] proposed a music recommendation model based on the user emotions captured through wearable physiological sensors. The emotion detection algorithm employed different machine learning algorithms like SVM, RF, KNN and decision tree (DT) algorithms to predict the emotions from the changing electrical signals gathered from the wearable sensors. Zhao et al. [ 45 ] developed a multimodal learning-based, social-aware movie recommender system. The model was able to successfully resolve the sparsity problem of recommender systems. The algorithm developed a heterogeneous network by exploiting the movie-poster image and textual description of each movie based on the social relationships and user ratings.

In 2019, Hammou et al. [ 46 ] proposed a Big Data recommendation algorithm capable of handling large scale data. The system employed random forest and matrix factorization through a data partitioning scheme. It was then used for generating recommendations based on user rating and preference for each item. The proposed system outperformed existing systems in terms of accuracy and speed. Zhao et al. [ 47 ] have put forward a hybrid initialization method for social network recommender systems. The algorithm employed denoising autoencoder (DAE) neural network-based initialization method (ANNInit) and attribute mapping. Bhaskaran and Santhi [ 48 ] have developed a hybrid, trust-based e-learning recommender system using cloud computing. The proposed algorithm was capable of learning online user activities by using the Firefly Algorithm (FA) and K-means clustering. Afolabi and Toivanen [ 59 ] have suggested an integrated recommender model based on collaborative filtering. The proposed model “Connected Health for Effective Management of Chronic Diseases”, aimed for integrating recommender systems for better decision-making in the process of disease management. He et al. [ 60 ] proposed a movie recommender system called “HI2Rec” which explored the usage of collaborative filtering and heterogeneous information for making movie recommendations. The model used the knowledge representation learning approach to embed movie-related information gathered from different sources.

In 2020, Han et al. [ 49 ] have proposed one Internet of Things (IoT)-based cancer rehabilitation recommendation system using the Beetle Antennae Search (BAS) algorithm. It presented the patients with a solution for the problem of optimal nutrition program by considering the objective function as the recurrence time. Kang et al. [ 50 ] have presented a recommender system for personalized advertisements in Online Broadcasting based on a tree model. Recommendations were generated in real-time by considering the user preferences to minimize the overhead of preference prediction and using a HashMap along with the tree characteristics. Ullah et al. [ 51 ] have implemented an image-based service recommendation model for online shopping based random forest and Convolutional Neural Networks (CNN). The model used JPEG coefficients to achieve an accurate prediction rate. Cai et al. [ 52 ] proposed a new hybrid recommender model using a many-objective evolutionary algorithm (MaOEA). The proposed algorithm was successful in optimizing the novelty, diversity, and accuracy of recommendations. Esteban et al. [ 53 ] have implemented a hybrid multi-criteria recommendation system concerned with students’ academic performance, personal interests, and course selection. The system was developed using a Genetic Algorithm (GA) and aimed at helping university students. It combined both course information and student information for increasing system performance and the reliability of the recommendations. Mondal et al. [ 54 ] have built a multilayer, graph data model-based doctor recommendation system by exploiting the trust concept between a patient-doctor relationship. The proposed system showed good results in practical applications.

In 2021, Dhelim et al. [ 55 ] have developed a personality-based product recommending model using the techniques of meta path discovery and user interest mining. This model showed better results when compared to session-based and deep learning models. Bhalse et al. [ 56 ] proposed a web-based movie recommendation system based on collaborative filtering using Singular Value Decomposition (SVD), collaborative filtering and cosine similarity (CS) for addressing the sparsity problem of recommender systems. It suggested a recommendation list by considering the content information of movies. Similarly, to solve both sparsity and cold-start problems Ke et al. [ 57 ] proposed a dynamic goods recommendation system based on reinforcement learning. The proposed system was capable of learning from the reduced entropy loss error on real-time applications. Chen et al. [ 58 ] have presented a movie recommender model combining various techniques like user interest with category-level representation, neighbour-assisted representation, user interest with latent representation and item-level representation using Feed-forward Neural Network (FNN).

Comparative chronological review

A comparative chronological review to compare the total contributions on various recommender systems in the past 10 years is given in Fig.  7 .

Comparative chronological review of recommender systems under diverse applications

This review puts forward a comparison of the number of research works proposed in the domain of recommender systems from the year 2011 to 2021 using various deep learning and machine learning-based approaches. Research articles are categorized based on the recommender system classification framework as shown in Table 5 . The articles are ordered according to their year of publication. There are two key concepts: Application fields and techniques used. The application fields of recommender systems are divided into six different fields, viz. entertainment, health, tourism, web/e-commerce, education and social media/others.

Algorithmic categorization, simulation platforms and applications considered for various recommender systems

This section analyses different methods like deep learning, machine learning, clustering and meta-heuristic-based-approaches used in the development of recommender systems. The algorithmic categorization of different recommender systems is given in Fig.  8 .

Algorithmic categorization of different recommender systems

Categorization is done based on content-based, collaborative filtering-based, and optimization-based approaches. In [ 8 ], a content-based filtering technique was employed for increasing the ability to trust other agents and for improving the exchange of information by trust degree. In [ 16 ], it was applied to enhance the quality of recommendations using the account attributes of the material. It achieved better performance concerning with F1-score, recall and precision. In [ 18 ], this technique was able to capture the implicit user feedback, increasing the overall accuracy of the proposed model. The content-based filtering in [ 30 ] was able to increase the accuracy and performance of a stock recommender system by using the “trust factor” for making decisions.

Different collaborative filtering approaches are utilized in recent studies, which are categorized as follows:

Model-based techniques

Neuro-Fuzzy [ 1 ] based technique helps in discovering the association between user categories and item relevance. It is also simple to understand. K-Means Clustering [ 2 , 19 , 25 , 48 ] is efficient for large scale datasets. It is simple to implement and gives a fast convergence rate. It also offers automatic recovery from failures. The decision tree [ 2 , 44 ] technique is easy to interpret. It can be used for solving the classic regression and classification problems in recommender systems. Bayesian Network [ 3 ] is a probabilistic technique used to solve classification challenges. It is based on the theory of Bayes theorem and conditional probability. Association Rule Mining (ARM) techniques [ 4 , 17 , 35 ] extract rules for projecting the occurrence of an item by considering the existence of other items in a transaction. This method uses the association rules to create a more suitable representation of data and helps in increasing the model performance and storage efficiency. Fuzzy Logic [ 5 , 7 , 15 , 20 , 28 , 43 ] techniques use a set of flexible rules. It focuses on solving complex real-time problems having an inaccurate spectrum of data. This technique provides scalability and helps in increasing the overall model performance for recommender systems. The semantic similarity [ 6 ] technique is used for describing a topological similarity to define the distance among the concepts and terms through ontologies. It measures the similarity information for increasing the efficiency of recommender systems. Rough set [ 9 , 34 ] techniques use probability distributions for solving the challenges of existing recommender models. Semantic web rule language [ 10 ] can efficiently extract the dataset features and increase the model efficiency. Linear programming-based approaches [ 13 , 42 ] are employed for achieving quality decision making in recommender models. Sequential pattern analysis [ 14 ] is applied to find suitable patterns among data items. This helps in increasing model efficiency. The probabilistic model [ 24 ] is a famous tool to handle uncertainty in risk computations and performance assessment. It offers better decision-making capabilities. K-nearest neighbours (KNN) [ 19 , 37 , 44 ] technique provides faster computation time, simplicity and ease of interpretation. They are good for classification and regression-based problems and offers more accuracy. Spectral clustering [ 21 ] is also called graph clustering or similarity-based clustering, which mainly focuses on reducing the space dimensionality in identifying the dataset items. Stochastic learning algorithm [ 26 ] solves the real-time challenges of recommender systems. Linear SVM [ 29 , 44 ] efficiently solves the high dimensional problems related to recommender systems. It is a memory-efficient method and works well with a large number of samples having relative separation among the classes. This method has been shown to perform well even when new or unfamiliar data is added. Relational Functional Gradient Boosting [ 39 ] technique efficiently works on the relational dependency of data, which is useful for statical relational learning for collaborative-based recommender systems. Ensemble learning [ 40 ] combines the forecast of two or more models and aims to achieve better performance than any of the single contributing models. It also helps in reducing overfitting problems, which are common in recommender systems.

SDAE [ 41 ] is used for learning the non-linear transformations with different filters for finding suitable data. This aids in increasing the performance of recommender models. Multimodal network learning [ 45 ] is efficient for multi-modal data, representing a combined representation of diverse modalities. Random forest [ 46 , 51 ] is a commonly used approach in comparison with other classifiers. It has been shown to increase accuracy when handling big data. This technique is a collection of decision trees to minimize variance through training on diverse data samples. ANNInit [ 47 ] is a type of artificial neural network-based technique that has the capability of self-learning and generating efficient results. It is independent of the data type and can learn data patterns automatically. HashMap [ 50 ] gives faster access to elements owing to the hashing methodology, which decreases the data processing time and increases the performance of the system. CNN [ 51 ] technique can automatically fetch the significant features of a dataset without any supervision. It is a computationally efficient method and provides accurate recommendations. This technique is also simple and fast for implementation. Multilayer graph data model [ 54 ] is efficient for real-time applications and minimizes the access time through mapping the correlation as edges among nodes and provides superior performance. Singular Value Decomposition [ 56 ] can simplify the input data and increase the efficiency of recommendations by eliminating the noise present in data. Reinforcement learning [ 57 ] is efficient for practical scenarios of recommender systems having large data sizes. It is capable of boosting the model performance by increasing the model accuracy even for large scale datasets. FNN [ 58 ] is one of the artificial neural network techniques which can learn non-linear and complex relationships between items. It has demonstrated a good performance increase when employed in different recommender systems. Knowledge representation learning [ 60 ] systems aim to simplify the model development process by increasing the acquisition efficiency, inferential efficiency, inferential adequacy and representation adequacy. User-based approaches [ 2 , 55 , 59 ] specialize in detecting user-related meta-data which is employed to increase the overall model performance. This technique is more suitable for real-time applications where it can capture user feedback and use it to increase the user experience.

Optimization-based techniques

The Foraging Bees [ 11 ] technique enables both functional and combinational optimization for random searching in recommender models. Artificial bee colony [ 12 ] is a swarm-based meta-heuristic technique that provides features like faster convergence rate, the ability to handle the objective with stochastic nature, ease for incorporating with other algorithms, usage of fewer control parameters, strong robustness, high flexibility and simplicity. Particle Swarm Optimization [ 23 ] is a computation optimization technique that offers better computational efficiency, robustness in control parameters, and is easy and simple to implement in recommender systems. Portfolio optimization algorithm [ 27 ] is a subclass of optimization algorithms that find its application in stock investment recommender systems. It works well in real-time and helps in the diversification of the portfolio for maximum profit. The artificial immune system [ 31 ]a is computationally intelligent machine learning technique. This technique can learn new patterns in the data and optimize the overall system parameters. Expectation maximization (EM) [ 32 , 36 , 38 ] is an iterative algorithm that guarantees the likelihood of finding the maximum parameters when the input variables are unknown. Delphi panel and repertory grid [ 33 ] offers efficient decision making by solving the dimensionality problem and data sparsity issues of recommender systems. The Firefly algorithm (FA) [ 48 ] provides fast results and increases recommendation efficiency. It is capable of reducing the number of iterations required to solve specific recommender problems. It also provides both local and global sets of solutions. Beetle Antennae Search (BAS) [ 49 ] offers superior search accuracy and maintains less time complexity that promotes the performance of recommendations. Many-objective evolutionary algorithm (MaOEA) [ 52 ] is applicable for real-time, multi-objective, search-related recommender systems. The introduction of a local search operator increases the convergence rate and gets suitable results. Genetic Algorithm (GA) [ 2 , 22 , 25 , 53 ] based techniques are used to solve the multi-objective optimization problems of recommender systems. They employ probabilistic transition rules and have a simpler operation that provides better recommender performance.

Features and challenges

The features and challenges of the existing recommender models are given in Table 6 .

Simulation platforms

The various simulation platforms used for developing different recommender systems with different applications are given in Fig.  9 .

Simulation platforms used for developing different recommender systems

Here, the Java platform is used in 20% of the contributions, MATLAB is implemented in 7% of the contributions, different fold cross-validation are used in 8% of the contributions, 7% of the contributions are utilized by the python platform, 3% of the contributions employ R-programming and 1% of the contributions are developed by Tensorflow, Weka and Android environments respectively. Other simulation platforms like Facebook, web UI (User Interface), real-time environments, etc. are used in 50% of the contributions. Table 7 describes some simulation platforms commonly used for developing recommender systems.

Application focused and dataset description

This section provides an analysis of the different applications focused on a set of recent recommender systems and their dataset details.

Recent recommender systems were analysed and found that 11% of the contributions are focused on the domain of healthcare, 10% of the contributions are on movie recommender systems, 5% of the contributions come from music recommender systems, 6% of the contributions are focused on e-learning recommender systems, 8% of the contributions are used for online product recommender systems, 3% of the contributions are focused on book recommendations and 1% of the contributions are focused on Job and knowledge management recommender systems. 5% of the contributions concentrated on social network recommender systems, 10% of the contributions are focused on tourist and hotels recommender systems, 6% of the contributions are employed for stock recommender systems, and 3% of the contributions contributed for video recommender systems. The remaining 12% of contributions are miscellaneous recommender systems like Twitter, venue-based recommender systems, etc. Similarly, different datasets are gathered for recommender systems based on their application types. A detailed description is provided in Table 8 .

Performance analysis of state-of-art recommender systems

The performance evaluation metrics used for the analysis of different recommender systems is depicted in Table 9 . From the set of research works, 35% of the works use recall measure, 16% of the works employ Mean Absolute Error (MAE), 11% of the works take Root Mean Square Error (RMSE), 41% of the papers consider precision, 30% of the contributions analyse F1-measure, 31% of the works apply accuracy and 6% of the works employ coverage measure to validate the performance of the recommender systems. Moreover, some additional measures are also considered for validating the performance in a few applications.

Research gaps and challenges

In the recent decade, recommender systems have performed well in solving the problem of information overload and has become the more appropriate tool for multiple areas such as psychology, mathematics, computer science, etc. [ 80 ]. However, current recommender systems face a variety of challenges which are stated as follows, and discussed below:

Deployment challenges such as cold start, scalability, sparsity, etc. are already discussed in Sect. 3.

Challenges faced when employing different recommender algorithms for different applications.

Challenges in collecting implicit user data

Challenges in handling real-time user feedback.

Challenges faced in choosing the correct implementation techniques.

Challenges faced in measuring system performance.

Challenges in implementing recommender system for diverse applications.

Numerous recommender algorithms have been proposed on novel emerging dimensions which focus on addressing the existing limitations of recommender systems. A good recommender system must increase the recommendation quality based on user preferences. However, a specific recommender algorithm is not always guaranteed to perform equally for different applications. This encourages the possibility of employing different recommender algorithms for different applications, which brings along a lot of challenges. There is a need for more research to alleviate these challenges. Also, there is a large scope of research in recommender applications that incorporate information from different interactive online sites like Facebook, Twitter, shopping sites, etc. Some other areas for emerging research may be in the fields of knowledge-based recommender systems, methods for seamlessly processing implicit user data and handling real-time user feedback to recommend items in a dynamic environment.

Some of the other research areas like deep learning-based recommender systems, demographic filtering, group recommenders, cross-domain techniques for recommender systems, and dimensionality reduction techniques are also further required to be studied [ 83 ]. Deep learning-based recommender systems have recently gained much popularity. Future research areas in this field can integrate the well-performing deep learning models with new variants of hybrid meta-heuristic approaches.

During this review, it was observed that even though recent recommender systems have demonstrated good performance, there is no single standardized criteria or method which could be used to evaluate the performance of all recommender systems. System performance is generally measured by different evaluation matrices which makes it difficult to compare. The application of recommender systems in real-time applications is growing. User satisfaction and personalization play a very important role in the success of such recommender systems. There is a need for some new evaluation criteria which can evaluate the level of user satisfaction in real-time. New research should focus on capturing real-time user feedback and use the information to change the recommendation process accordingly. This will aid in increasing the quality of recommendations.

Conclusion and future scope

Recommender systems have attracted the attention of researchers and academicians. In this paper, we have identified and prudently reviewed research papers on recommender systems focusing on diverse applications, which were published between 2011 and 2021. This review has gathered diverse details like different application fields, techniques used, simulation tools used, diverse applications focused, performance metrics, datasets used, system features, and challenges of different recommender systems. Further, the research gaps and challenges were put forward to explore the future research perspective on recommender systems. Overall, this paper provides a comprehensive understanding of the trend of recommender systems-related research and to provides researchers with insight and future direction on recommender systems. The results of this study have several practical and significant implications:

Based on the recent-past publication rates, we feel that the research of recommender systems will significantly grow in the future.

A large number of research papers were identified in movie recommendations, whereas health, tourism and education-related recommender systems were identified in very few numbers. This is due to the availability of movie datasets in the public domain. Therefore, it is necessary to develop datasets in other fields also.

There is no standard measure to compute the performance of recommender systems. Among 60 papers, 21 used recall, 10 used MAE, 25 used precision, 18 used F1-measure, 19 used accuracy and only 7 used RMSE to calculate system performance. Very few systems were found to excel in two or more matrices.

Java and Python (with a combined contribution of 27%) are the most common programming languages used to develop recommender systems. This is due to the availability of a large number of standard java and python libraries which aid in the development process.

Recently a large number of hybrid and optimizations techniques are being proposed for recommender systems. The performance of a recommender system can be greatly improved by applying optimization techniques.

There is a large scope of research in using neural networks and deep learning-based methods for developing recommender systems. Systems developed using these methods are found to achieve high-performance accuracy.

This research will provide a guideline for future research in the domain of recommender systems. However, this research has some limitations. Firstly, due to the limited amount of manpower and time, we have only reviewed papers published in journals focusing on computer science, management and medicine. Secondly, we have reviewed only English papers. New research may extend this study to cover other journals and non-English papers. Finally, this review was conducted based on a search on only six descriptors: “Recommender systems”, “Recommendation systems”, “Movie Recommend*”, “Music Recommend*”, “Personalized Recommend*” and “Hybrid Recommend*”. Research papers that did not include these keywords were not considered. Future research can include adding some additional descriptors and keywords for searching. This will allow extending the research to cover more diverse articles on recommender systems.

Availability of data and materials

Not applicable.

Castellano G, Fanelli AM, Torsello MA. NEWER: A system for neuro-fuzzy web recommendation. Appl Soft Comput. 2011;11:793–806.

Article   Google Scholar  

Crespo RG, Martínez OS, Lovelle JMC, García-Bustelo BCP, Gayo JEL, Pablos PO. Recommendation system based on user interaction data applied to intelligent electronic books. Computers Hum Behavior. 2011;27:1445–9.

Lin FC, Yu HW, Hsu CH, Weng TC. Recommendation system for localized products in vending machines. Expert Syst Appl. 2011;38:9129–38.

Wang SL, Wu CY. Application of context-aware and personalized recommendation to implement an adaptive ubiquitous learning system. Expert Syst Appl. 2011;38:10831–8.

García-Crespo Á, López-Cuadrado JL, Colomo-Palacios R, González-Carrasco I, Ruiz-Mezcua B. Sem-Fit: A semantic based expert system to provide recommendations in the tourism domain. Expert Syst Appl. 2011;38:13310–9.

Dong H, Hussain FK, Chang E. A service concept recommendation system for enhancing the dependability of semantic service matchmakers in the service ecosystem environment. J Netw Comput Appl. 2011;34:619–31.

Li M, Liu L, Li CB. An approach to expert recommendation based on fuzzy linguistic method and fuzzy text classification in knowledge management systems. Expert Syst Appl. 2011;38:8586–96.

Lorenzi F, Bazzan ALC, Abel M, Ricci F. Improving recommendations through an assumption-based multiagent approach: An application in the tourism domain. Expert Syst Appl. 2011;38:14703–14.

Huang Z, Lu X, Duan H. Context-aware recommendation using rough set model and collaborative filtering. Artif Intell Rev. 2011;35:85–99.

Chen RC, Huang YH, Bau CT, Chen SM. A recommendation system based on domain ontology and SWRL for anti-diabetic drugs selection. Expert Syst Appl. 2012;39:3995–4006.

Mohanraj V, Chandrasekaran M, Senthilkumar J, Arumugam S, Suresh Y. Ontology driven bee’s foraging approach based self-adaptive online recommendation system. J Syst Softw. 2012;85:2439–50.

Hsu CC, Chen HC, Huang KK, Huang YM. A personalized auxiliary material recommendation system based on learning style on facebook applying an artificial bee colony algorithm. Comput Math Appl. 2012;64:1506–13.

Gemmell J, Schimoler T, Mobasher B, Burke R. Resource recommendation in social annotation systems: A linear-weighted hybrid approach. J Comput Syst Sci. 2012;78:1160–74.

Article   MathSciNet   Google Scholar  

Choi K, Yoo D, Kim G, Suh Y. A hybrid online-product recommendation system: Combining implicit rating-based collaborative filtering and sequential pattern analysis. Electron Commer Res Appl. 2012;11:309–17.

Garibaldi JM, Zhou SM, Wang XY, John RI, Ellis IO. Incorporation of expert variability into breast cancer treatment recommendation in designing clinical protocol guided fuzzy rule system models. J Biomed Inform. 2012;45:447–59.

Salehi M, Kmalabadi IN. A hybrid attribute–based recommender system for e–learning material recommendation. IERI Procedia. 2012;2:565–70.

Aher SB, Lobo LMRJ. Combination of machine learning algorithms for recommendation of courses in e-learning System based on historical data. Knowl-Based Syst. 2013;51:1–14.

Kardan AA, Ebrahimi M. A novel approach to hybrid recommendation systems based on association rules mining for content recommendation in asynchronous discussion groups. Inf Sci. 2013;219:93–110.

Chang JH, Lai CF, Wang MS, Wu TY. A cloud-based intelligent TV program recommendation system. Comput Electr Eng. 2013;39:2379–99.

Lucas JP, Luz N, Moreno MN, Anacleto R, Figueiredo AA, Martins C. A hybrid recommendation approach for a tourism system. Expert Syst Appl. 2013;40:3532–50.

Niu J, Zhu L, Zhao X, Li H. Affivir: An affect-based Internet video recommendation system. Neurocomputing. 2013;120:422–33.

Liu L, Xu J, Liao SS, Chen H. A real-time personalized route recommendation system for self-drive tourists based on vehicle to vehicle communication. Expert Syst Appl. 2014;41:3409–17.

Bakshi S, Jagadev AK, Dehuri S, Wang GN. Enhancing scalability and accuracy of recommendation systems using unsupervised learning and particle swarm optimization. Appl Soft Comput. 2014;15:21–9.

Kim Y, Shim K. TWILITE: A recommendation system for twitter using a probabilistic model based on latent Dirichlet allocation. Inf Syst. 2014;42:59–77.

Wang Z, Yu X, Feng N, Wang Z. An improved collaborative movie recommendation system using computational intelligence. J Vis Lang Comput. 2014;25:667–75.

Kolomvatsos K, Anagnostopoulos C, Hadjiefthymiades S. An efficient recommendation system based on the optimal stopping theory. Expert Syst Appl. 2014;41:6796–806.

Gottschlich J, Hinz O. A decision support system for stock investment recommendations using collective wisdom. Decis Support Syst. 2014;59:52–62.

Torshizi AD, Zarandi MHF, Torshizi GD, Eghbali K. A hybrid fuzzy-ontology based intelligent system to determine level of severity and treatment recommendation for benign prostatic hyperplasia. Comput Methods Programs Biomed. 2014;113:301–13.

Zahálka J, Rudinac S, Worring M. Interactive multimodal learning for venue recommendation. IEEE Trans Multimedia. 2015;17:2235–44.

Sankar CP, Vidyaraj R, Kumar KS. Trust based stock recommendation system – a social network analysis approach. Procedia Computer Sci. 2015;46:299–305.

Chen MH, Teng CH, Chang PC. Applying artificial immune systems to collaborative filtering for movie recommendation. Adv Eng Inform. 2015;29:830–9.

Wu H, Pei Y, Li B, Kang Z, Liu X, Li H. Item recommendation in collaborative tagging systems via heuristic data fusion. Knowl-Based Syst. 2015;75:124–40.

Yeh DY, Cheng CH. Recommendation system for popular tourist attractions in Taiwan using delphi panel and repertory grid techniques. Tour Manage. 2015;46:164–76.

Liao SH, Chang HK. A rough set-based association rule approach for a recommendation system for online consumers. Inf Process Manage. 2016;52:1142–60.

Li H, Cui J, Shen B, Ma J. An intelligent movie recommendation system through group-level sentiment analysis in microblogs. Neurocomputing. 2016;210:164–73.

Wu H, Yue K, Pei Y, Li B, Zhao Y, Dong F. Collaborative topic regression with social trust ensemble for recommendation in social media systems. Knowl-Based Syst. 2016;97:111–22.

Adeniyi DA, Wei Z, Yongquan Y. Automated web usage data mining and recommendation system using K-Nearest Neighbor (KNN) classification method. Appl Computing Inform. 2016;12:90–108.

Rawat YS, Kankanhalli MS. ClickSmart: A context-aware viewpoint recommendation system for mobile photography. IEEE Trans Circuits Syst Video Technol. 2017;27:149–58.

Yang S, Korayem M, Aljadda K, Grainger T, Natarajan S. Combining content-based and collaborative filtering for job recommendation system: A cost-sensitive Statistical Relational Learning approach. Knowl-Based Syst. 2017;136:37–45.

Lee WP, Chen CT, Huang JY, Liang JY. A smartphone-based activity-aware system for music streaming recommendation. Knowl-Based Syst. 2017;131:70–82.

Wei J, He J, Chen K, Zhou Y, Tang Z. Collaborative filtering and deep learning based recommendation system for cold start items. Expert Syst Appl. 2017;69:29–39.

Li C, Wang Z, Cao S, He L. WLRRS: A new recommendation system based on weighted linear regression models. Comput Electr Eng. 2018;66:40–7.

Mezei J, Nikou S. Fuzzy optimization to improve mobile health and wellness recommendation systems. Knowl-Based Syst. 2018;142:108–16.

Ayata D, Yaslan Y, Kamasak ME. Emotion based music recommendation system using wearable physiological sensors. IEEE Trans Consum Electron. 2018;64:196–203.

Zhao Z, Yang Q, Lu H, Weninger T. Social-aware movie recommendation via multimodal network learning. IEEE Trans Multimedia. 2018;20:430–40.

Hammou BA, Lahcen AA, Mouline S. An effective distributed predictive model with matrix factorization and random forest for big data recommendation systems. Expert Syst Appl. 2019;137:253–65.

Zhao J, Geng X, Zhou J, Sun Q, Xiao Y, Zhang Z, Fu Z. Attribute mapping and autoencoder neural network based matrix factorization initialization for recommendation systems. Knowl-Based Syst. 2019;166:132–9.

Bhaskaran S, Santhi B. An efficient personalized trust based hybrid recommendation (TBHR) strategy for e-learning system in cloud computing. Clust Comput. 2019;22:1137–49.

Han Y, Han Z, Wu J, Yu Y, Gao S, Hua D, Yang A. Artificial intelligence recommendation system of cancer rehabilitation scheme based on IoT technology. IEEE Access. 2020;8:44924–35.

Kang S, Jeong C, Chung K. Tree-based real-time advertisement recommendation system in online broadcasting. IEEE Access. 2020;8:192693–702.

Ullah F, Zhang B, Khan RU. Image-based service recommendation system: A JPEG-coefficient RFs approach. IEEE Access. 2020;8:3308–18.

Cai X, Hu Z, Zhao P, Zhang W, Chen J. A hybrid recommendation system with many-objective evolutionary algorithm. Expert Syst Appl. 2020. https://doi.org/10.1016/j.eswa.2020.113648 .

Esteban A, Zafra A, Romero C. Helping university students to choose elective courses by using a hybrid multi-criteria recommendation system with genetic optimization. Knowledge-Based Syst. 2020;194:105385.

Mondal S, Basu A, Mukherjee N. Building a trust-based doctor recommendation system on top of multilayer graph database. J Biomed Inform. 2020;110:103549.

Dhelim S, Ning H, Aung N, Huang R, Ma J. Personality-aware product recommendation system based on user interests mining and metapath discovery. IEEE Trans Comput Soc Syst. 2021;8:86–98.

Bhalse N, Thakur R. Algorithm for movie recommendation system using collaborative filtering. Materials Today: Proceedings. 2021. https://doi.org/10.1016/j.matpr.2021.01.235 .

Ke G, Du HL, Chen YC. Cross-platform dynamic goods recommendation system based on reinforcement learning and social networks. Appl Soft Computing. 2021;104:107213.

Chen X, Liu D, Xiong Z, Zha ZJ. Learning and fusing multiple user interest representations for micro-video and movie recommendations. IEEE Trans Multimedia. 2021;23:484–96.

Afolabi AO, Toivanen P. Integration of recommendation systems into connected health for effective management of chronic diseases. IEEE Access. 2019;7:49201–11.

He M, Wang B, Du X. HI2Rec: Exploring knowledge in heterogeneous information for movie recommendation. IEEE Access. 2019;7:30276–84.

Bobadilla J, Serradilla F, Hernando A. Collaborative filtering adapted to recommender systems of e-learning. Knowl-Based Syst. 2009;22:261–5.

Russell S, Yoon V. Applications of wavelet data reduction in a recommender system. Expert Syst Appl. 2008;34:2316–25.

Campos LM, Fernández-Luna JM, Huete JF. A collaborative recommender system based on probabilistic inference from fuzzy observations. Fuzzy Sets Syst. 2008;159:1554–76.

Funk M, Rozinat A, Karapanos E, Medeiros AKA, Koca A. In situ evaluation of recommender systems: Framework and instrumentation. Int J Hum Comput Stud. 2010;68:525–47.

Porcel C, Moreno JM, Herrera-Viedma E. A multi-disciplinar recommender system to advice research resources in University Digital Libraries. Expert Syst Appl. 2009;36:12520–8.

Bobadilla J, Serradilla F, Bernal J. A new collaborative filtering metric that improves the behavior of recommender systems. Knowl-Based Syst. 2010;23:520–8.

Ochi P, Rao S, Takayama L, Nass C. Predictors of user perceptions of web recommender systems: How the basis for generating experience and search product recommendations affects user responses. Int J Hum Comput Stud. 2010;68:472–82.

Olmo FH, Gaudioso E. Evaluation of recommender systems: A new approach. Expert Syst Appl. 2008;35:790–804.

Zhen L, Huang GQ, Jiang Z. An inner-enterprise knowledge recommender system. Expert Syst Appl. 2010;37:1703–12.

Göksedef M, Gündüz-Öğüdücü S. Combination of web page recommender systems. Expert Syst Appl. 2010;37(4):2911–22.

Shao B, Wang D, Li T, Ogihara M. Music recommendation based on acoustic features and user access patterns. IEEE Trans Audio Speech Lang Process. 2009;17:1602–11.

Shin C, Woo W. Socially aware tv program recommender for multiple viewers. IEEE Trans Consum Electron. 2009;55:927–32.

Lopez-Carmona MA, Marsa-Maestre I, Perez JRV, Alcazar BA. Anegsys: An automated negotiation based recommender system for local e-marketplaces. IEEE Lat Am Trans. 2007;5:409–16.

Yap G, Tan A, Pang H. Discovering and exploiting causal dependencies for robust mobile context-aware recommenders. IEEE Trans Knowl Data Eng. 2007;19:977–92.

Meo PD, Quattrone G, Terracina G, Ursino D. An XML-based multiagent system for supporting online recruitment services. IEEE Trans Syst Man Cybern. 2007;37:464–80.

Khusro S, Ali Z, Ullah I. Recommender systems: Issues, challenges, and research opportunities. Inform Sci Appl. 2016. https://doi.org/10.1007/978-981-10-0557-2_112 .

Blanco-Fernandez Y, Pazos-Arias JJ, Gil-Solla A, Ramos-Cabrer M, Lopez-Nores M. Providing entertainment by content-based filtering and semantic reasoning in intelligent recommender systems. IEEE Trans Consum Electron. 2008;54:727–35.

Isinkaye FO, Folajimi YO, Ojokoh BA. Recommendation systems: Principles, methods and evaluation. Egyptian Inform J. 2015;16:261–73.

Yoshii K, Goto M, Komatani K, Ogata T, Okuno HG. An efficient hybrid music recommender system using an incrementally trainable probabilistic generative model. IEEE Trans Audio Speech Lang Process. 2008;16:435–47.

Wei YZ, Moreau L, Jennings NR. Learning users’ interests by quality classification in market-based recommender systems. IEEE Trans Knowl Data Eng. 2005;17:1678–88.

Bjelica M. Towards TV recommender system: experiments with user modeling. IEEE Trans Consum Electron. 2010;56:1763–9.

Setten MV, Veenstra M, Nijholt A, Dijk BV. Goal-based structuring in recommender systems. Interact Comput. 2006;18:432–56.

Adomavicius G, Tuzhilin A. Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng. 2005;17:734–49.

Symeonidis P, Nanopoulos A, Manolopoulos Y. Providing justifications in recommender systems. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans. 2009;38:1262–72.

Zhan J, Hsieh C, Wang I, Hsu T, Liau C, Wang D. Privacy preserving collaborative recommender systems. IEEE Trans Syst Man Cybernet. 2010;40:472–6.

Burke R. Hybrid recommender systems: survey and experiments. User Model User-Adap Inter. 2002;12:331–70.

Article   MATH   Google Scholar  

Gunes I, Kaleli C, Bilge A, Polat H. Shilling attacks against recommender systems: a comprehensive survey. Artif Intell Rev. 2012;42:767–99.

Park DH, Kim HK, Choi IY, Kim JK. A literature review and classification of recommender systems research. Expert Syst Appl. 2012;39:10059–72.

Download references

Acknowledgements

We thank our colleagues from Assam Down Town University who provided insight and expertise that greatly assisted this research, although they may not agree with all the interpretations and conclusions of this paper.

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and affiliations.

Department of Computer Science & Engineering, Assam Down Town University, Panikhaiti, Guwahati, 781026, Assam, India

Deepjyoti Roy & Mala Dutta

You can also search for this author in PubMed   Google Scholar

Contributions

DR carried out the review study and analysis of the existing algorithms in the literature. MD has been involved in drafting the manuscript or revising it critically for important intellectual content. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Deepjyoti Roy .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Roy, D., Dutta, M. A systematic review and research perspective on recommender systems. J Big Data 9 , 59 (2022). https://doi.org/10.1186/s40537-022-00592-5

Download citation

Received : 04 October 2021

Accepted : 28 March 2022

Published : 03 May 2022

DOI : https://doi.org/10.1186/s40537-022-00592-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Recommender system
• Machine learning
• Content-based filtering
• Collaborative filtering
• Deep learning

Help | Advanced Search

Mathematics > Optimization and Control

Title: multi methods of matrix analysis use for control and optimization system in control engineering.

Abstract: Matrix analysis plays a crucial role in the field of control engineering, providing a powerful mathematical framework for the analysis and design of control systems. This research report explores various applications of matrix analysis in control engineering, focusing on its contributions to system modeling, stability analysis, controllablity, observability, and optimization. The report also discusses specific examples and case studies to illustrate the practical significance of matrix analysis in addressing real-world control engineering challenges Analyze controllability. Informally, a system is controllable if we can construct a set of inputs that will drive the system to any given state. Analyze observability. Informally, observability means that by controlling the inputs and watching the outputs of a system we can determine what the states were. Optimal Control is a control method that aims to find the optimal control input to achieve the best performance of the system under certain constraints. This performance index can be the system output, energy consumption, time, etc.

Submission history

Access paper:.

• Download PDF
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

•  We're Hiring!
•  Help Center

Power System analysis

• Most Cited Papers
• Most Downloaded Papers
• Newest Papers
• Save to Library
• Last »
• Fault location Follow Following
• Electrical Power Distribution Follow Following
• Power Quality Follow Following
• Processamento digital de sinais de áudio Follow Following
• Power System Follow Following
• Smart Grid and Intelligent Micro Grid Follow Following
• Electrical Follow Following
• Electrical Engineering Follow Following
• Power system distribution Follow Following
• ANSYS CFX Follow Following

Enter the email address you signed up with and we'll email you a reset link.

• Academia.edu Publishing
•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

Research Paper

System analysis research paper.

This sample System Analysis Research Paper is published for educational and informational purposes only. If you need help writing your assignment, please use our research paper writing service and buy a paper on any topic at affordable price. Also check our tips on how to write a research paper , see the lists of research paper topics , and browse research paper examples .

System analysis, system inquiry, or systems theory is the study of the interdependence of relationships. A system is composed of regularly interacting or interrelating parts that, when taken together, form a new whole with properties distinct from its constituent parts. Systems are seen to be structurally divisible but functionally indivisible wholes with emergent properties. Central to system analysis is the recognition that the structure of any system—the many interlocking, sometimes time-delayed, sometimes circular interrelationships among its components—is often just as important, if not more important, than the individual components themselves in determining the system’s behavior.

Systems are characterized by complexity, a set of boundaries, and the ability to regenerate. Complexity refers to a large number of densely connected parts and multiple levels of embeddedness and entanglement. A system is defined by a set of parametric conditions or boundaries that delimit it or set it apart from its environment. No system can be completely closed or else we could not perceive it; there are only varying degrees of closure set by boundaries. A system regenerates itself through the self-reproduction of its own elements and of the network of interactions that characterize them in a process known as autopoiesis. Thus an autopoietic system renews, repairs, and replicates or reproduces itself in a flow of matter and energy.

Systems can change through an evolutionary process with a tendency toward greater structural complexity and organizational simplicity, more efficient modes of operation, and greater dynamic harmony. Change is enacted through a process of feedback where information concerning the adequacy of the system, its operation, and its outputs are introduced into the system. Negative feedback signals that there is a discrepancy between what the system produces and what it should produce. It tells us that we should change something in the system so that we can reduce the deviation from the norms stated in the system’s output model. Positive feedback signals that the whole system should change, that we should increase the deviation from the present state and change the output model. Functionalism is based on this adaptation. To survive or maintain equilibrium with respect to its environment, any system must to some degree adapt to that environment, attain its goals, integrate its components, and maintain its latent pattern, a cultural template of some sort.

A system can be ordered as a hierarchy or a heterarchy. A hierarchy is a vertical arrangement of entities within systems and their subsystems. A heterarchy is an ordering of entities without a single peak or leading element, and which element is dominant at a given time depends on the total situation. Systems may be understood through holism, where attention is focused on the emergent properties of the whole rather than on the behavior of the isolated parts, or reductionism, where phenomena are understood by breaking them down into their smallest possible parts.

Several fields utilize system analysis. Cybernetics, chaos theory, and social dynamics, for example, are among the disciplines that apply system analysis. Some areas of education and environmental sustainability also utilize system analysis. The systems framework is also fundamental to organizational theory, as organizations are complex, dynamic, goal-oriented processes; in anthropological studies, notably those incorporating positive and negative feedback; and in cybernetics, catastrophe theory, chaos theory, and complexity theory, all of which have the common goal of explaining complex systems that consist of a large number of mutually interacting and interrelated parts. In biology the living systems theory of James Grier Miller is a general theory about how all living systems work, maintain themselves, develop, and change. Living systems can be as simple as a single cell or as complex as a supranational organization such as the European Union. In sociology the structural functionalism of Talcott Parsons argues that the largest system is “the action system” consisting of interrelated behaviors of individuals, embedded in a physical-organic environment with others, with each part in a social system arranged in a pattern of interpenetrating relationships influenced by a socializing culture that constitutes standards and channels for guiding actions. Societies (which are highly complex), like systems and organisms, have functional needs that must be met if the society is to survive. Parsons says that all societies have four basic needs: adaptation, goal attainment, integration, and pattern maintenance (i.e., inertia, latency, or self-maintenance).

The deterministic or restrictive nature of systems is addressed by aspects of structuralism. Structuralism rejects the concept of human freedom and choice and focuses instead on the way human behavior is determined by various structures. Thomas Kuhn, for example, notes how scientists operate under a standard praxis of “normal science,” deviating from a standard “paradigm” only in instances of irreconcilable anomalies. In political science the structural realism of Kenneth Waltz describes international politics as a systemic interaction of states within an anarchical environment. States first seek survival and are socialized by an anarchical environment to act and react based on threats to survival and to form self-help alliances with like units. The system effects described by Robert Jervis notes how political relations among states in a system, similar to biological interactions among cells and other scientific phenomena, can produce effects different from the sum of individual actions.

Bibliography:

• Banathy, Bela. 1996. Designing Social Systems in a Changing World. New York: Plenum.
• Bateson, Gregory. 1979. Mind and Nature: A Necessary Unity. New York: Dutton.
• Bausch, Kenneth C. 2001. The Emerging Consensus in Social Systems Theory. New York: Kluwer Academic/Plenum.
• Bertalanffy, Ludwig von. 1968. General System Theory: Foundations, Development, Applications. New York: George Braziller.
• Churchman, C. West. 1968. The Systems Approach. New York: Delacorte.
• International Society for System Sciences (ISSS). http://www.isss.org/
• Jantsch, Erich. 1980. The Self-Organizing Universe: Scientific and Human Implications of the Emerging Paradigm of Evolution. New York: Pergamon.
• Jervis, Robert. 1997. System Effects: Complexity in Political and Social Life. Princeton, NJ: Princeton University Press.
• Kahn, Herman. 1956. Techniques of System Analysis. Santa Monica, CA: Rand Corporation.
• Kuhn, Thomas. 1996. The Structure of Scientific Revolutions. 3rd ed. Chicago: University of Chicago Press.
• Miller, James Grier. 1978. Living Systems. New York: McGraw-Hill.
• Parsons, Talcott. [1937] 1967. The Structure of Action. New York: Free Press.
• Parsons, Talcott. 1977. Social Systems and the Evolution of Action Theory. New York: Free Press.
• Parsons, Talcott, and Neal J. Smelser. 1956. Economy and Society. Glencoe, IL: Free Press.
• System Dynamics Society. http://www.systemdynamics.org/
• Waltz, Kenneth. 1979. Theory of International Politics. Reading, MA: Addison-Wesley.
• Weinberg, Gerald M. 1975. An Introduction to General Systems Thinking. New York: Wiley-Interscience.
• How to Write a Research Paper
• Research Paper Topics
• Research Paper Examples

Free research papers are not written to satisfy your specific instructions. You can use our professional writing services to buy a custom research paper on any topic and get your high quality paper at affordable price.

ORDER HIGH QUALITY CUSTOM PAPER

Related Posts

Automated recognition of the cricket batting backlift technique in video footage using deep learning architectures

Tevin Moodley, Dustin van der Haar & Habib Noorbhai

Synchronization of passes in event and spatiotemporal soccer data

Henrik Biermann, Rumena Komitova, … Daniel Memmert

A Random Forest approach to identify metrics that best predict match outcome and player ranking in the esport Rocket League

Tim D. Smithies, Mark J. Campbell, … Adam J. Toth

Introduction

Association football, or simply football or soccer, is a widely popular and highly professionalised sport, in which two teams compete to score goals against each other. As each football team comprises up to 11 active players at all times and takes place on a very large pitch (also known as a soccer field), scoring goals tends to require a significant degree of strategic team-play. Under the rules codified in the Laws of the Game 1 , this competition has nurtured an evolution of nuanced strategies and tactics, culminating in modern professional football leagues. In today’s play, data-driven insights are a key driver in determining the optimal player setups for each game and developing counter-tactics to maximise the chances of success 2 .

When competing at the highest level the margins are incredibly tight, and it is increasingly important to be able to capitalise on any opportunity for creating an advantage on the pitch. To that end, top-tier clubs employ diverse teams of coaches, analysts and experts, tasked with studying and devising (counter-)tactics before each game. Several recent methods attempt to improve tactical coaching and player decision-making through artificial intelligence (AI) tools, using a wide variety of data types from videos to tracking sensors and applying diverse algorithms ranging from simple logistic regression to elaborate neural network architectures. Such methods have been employed to help predict shot events from videos 3 , forecast off-screen movement from spatio-temporal data 4 , determine whether a match is in-play or interrupted 5 , or identify player actions 6 .

The execution of agreed-upon plans by players on the pitch is highly dynamic and imperfect, depending on numerous factors including player fitness and fatigue, variations in player movement and positioning, weather, the state of the pitch, and the reaction of the opposing team. In contrast, set pieces provide an opportunity to exert more control on the outcome, as the brief interruption in play allows the players to reposition according to one of the practiced and pre-agreed patterns, and make a deliberate attempt towards the goal. Examples of such set pieces include free kicks, corner kicks, goal kicks, throw-ins, and penalties 2 .

Among set pieces, corner kicks are of particular importance, as an improvement in corner kick execution may substantially modify game outcomes, and they lend themselves to principled, tactical and detailed analysis. This is because corner kicks tend to occur frequently in football matches (with ~10 corners on average taking place in each match 7 ), they are taken from a fixed, rigid position, and they offer an immediate opportunity for scoring a goal—no other set piece simultaneously satisfies all of the above. In practice, corner kick routines are determined well ahead of each match, taking into account the strengths and weaknesses of the opposing team and their typical tactical deployment. It is for this reason that we focus on corner kick analysis in particular, and propose TacticAI, an AI football assistant for supporting the human expert with set piece analysis, and the development and improvement of corner kick routines.

TacticAI is rooted in learning efficient representations of corner kick tactics from raw, spatio-temporal player tracking data. It makes efficient use of this data by representing each corner kick situation as a graph—a natural representation for modelling relationships between players (Fig.  1 A, Table  2 ), and these player relationships may be of higher importance than the absolute distances between them on the pitch 8 . Such a graph input is a natural candidate for graph machine learning models 9 , which we employ within TacticAI to obtain high-dimensional latent player representations. In the Supplementary Discussion section, we carefully contrast TacticAI against prior art in the area.

A How corner kick situations are converted to a graph representation. Each player is treated as a node in a graph, with node, edge and graph features extracted as detailed in the main text. Then, a graph neural network operates over this graph by performing message passing; each node’s representation is updated using the messages sent to it from its neighbouring nodes. B How TacticAI processes a given corner kick. To ensure that TacticAI’s answers are robust in the face of horizontal or vertical reflections, all possible combinations of reflections are applied to the input corner, and these four views are then fed to the core TacticAI model, where they are able to interact with each other to compute the final player representations—each internal blue arrow corresponds to a single message passing layer from ( A ). Once player representations are computed, they can be used to predict the corner’s receiver, whether a shot has been taken, as well as assistive adjustments to player positions and velocities, which increase or decrease the probability of a shot being taken.

Uniquely, TacticAI takes advantage of geometric deep learning 10 to explicitly produce player representations that respect several symmetries of the football pitch (Fig.  1 B). As an illustrative example, we can usually safely assume that under a horizontal or vertical reflection of the pitch state, the game situation is equivalent. Geometric deep learning ensures that TacticAI’s player representations will be identically computed under such reflections, such that this symmetry does not have to be learnt from data. This proves to be a valuable addition, as high-quality tracking data is often limited—with only a few hundred matches played each year in every league. We provide an in-depth overview of how we employ geometric deep learning in TacticAI in the “Methods” section.

From these representations, TacticAI is then able to answer various predictive questions about the outcomes of a corner—for example, which player is most likely to make first contact with the ball, or whether a shot will take place. TacticAI can also be used as a retrieval system—for mining similar corner kick situations based on the similarity of player representations—and a generative recommendation system, suggesting adjustments to player positions and velocities to maximise or minimise the estimated shot probability. Through several experiments within a case study with domain expert coaches and analysts from Liverpool FC, the results of which we present in the next section, we obtain clear statistical evidence that TacticAI readily provides useful, realistic and accurate tactical suggestions.

To demonstrate the diverse qualities of our approach, we design TacticAI with three distinct predictive and generative components: receiver prediction, shot prediction, and tactic recommendation through guided generation, which also correspond to the benchmark tasks for quantitatively evaluating TacticAI. In addition to providing accurate quantitative insights for corner kick analysis with its predictive components, the interplay between TacticAI’s predictive and generative components allows coaches to sample alternative player setups for each routine of interest, and directly evaluate the possible outcomes of such alternatives.

We will first describe our quantitative analysis, which demonstrates that TacticAI’s predictive components are accurate at predicting corner kick receivers and shot situations on held-out test corners and that the proposed player adjustments do not strongly deviate from ground-truth situations. However, such an analysis only gives an indirect insight into how useful TacticAI would be once deployed. We tackle this question of utility head-on and conduct a comprehensive case study in collaboration with our partners at Liverpool FC—where we directly ask human expert raters to judge the utility of TacticAI’s predictions and player adjustments. The following sections expand on the specific results and analysis we have performed.

In what follows, we will describe TacticAI’s components at a minimal level necessary to understand our evaluation. We defer detailed descriptions of TacticAI’s components to the “Methods” section. Note that, all our error bars reported in this research are standard deviations.

Benchmarking TacticAI

We evaluate the three components of TacticAI on a relevant benchmark dataset of corner kicks. Our dataset consists of 7176 corner kicks from the 2020 to 2021 Premier League seasons, which we randomly shuffle and split into a training (80%) and a test set (20%). As previously mentioned, TacticAI operates on graphs. Accordingly, we represent each corner kick situation as a graph, where each node corresponds to a player. The features associated with each node encode the movements (velocities and positions) and simple profiles (heights and weights) of on-pitch players at the timestamp when the corresponding corner kick was being taken by the attacking kicker (see the “Methods” section), and no information of ball movement was encoded. The graphs are fully connected; that is, for every pair of players, we will include the edge connecting them in the graph. Each of these edges encodes a binary feature, indicating whether the two players are on opposing teams or not. For each task, we generated the relevant dataset of node/edge/graph features and corresponding labels (Tables  1 and 2 , see the “Methods” section). The components were then trained separately with their corresponding corner kick graphs. In particular, we only employ a minimal set of features to construct the corner kick graphs, without encoding the movements of the ball nor explicitly encoding the distances between players into the graphs. We used a consistent training-test split for all benchmark tasks, as this made it possible to benchmark not only the individual components but also their interactions.

Accurate receiver and shot prediction through geometric deep learning

One of TacticAI’s key predictive models forecasts the receiver out of the 22 on-pitch players. The receiver is defined as the first player touching the ball after the corner is taken. In our evaluation, all methods used the same set of features (see the “Receiver prediction” entry in Table  1 and the “Methods” section). We leveraged the receiver prediction task to benchmark several different TacticAI base models. Our best-performing model—achieving 0.782 ± 0.039 in top-3 test accuracy after 50,000 training steps—was a deep graph attention network 11 , 12 , leveraging geometric deep learning 10 through the use of D 2 group convolutions 13 . We supplement this result with a detailed ablation study, verifying that both our choice of base architecture and group convolution yielded significant improvements in the receiver prediction task (Supplementary Table  2 , see the subsection “Ablation study” in the “Methods” section). Considering that corner kick receiver prediction is a highly challenging task with many factors that are unseen by our model—including fatigue and fitness levels, and actual ball trajectory—we consider TacticAI’s top-3 accuracy to reflect a high level of predictive power, and keep the base TacticAI architecture fixed for subsequent studies. In addition to this quantitative evaluation with the evaluation dataset, we also evaluate the performance of TacticAI’s receiver prediction component in a case study with human raters. Please see the “Case study with expert raters” section for more details.

For shot prediction, we observe that reusing the base TacticAI architecture to directly predict shot events—i.e., directly modelling the probability \({\mathbb{P}}(\,{{\mbox{shot}}}| {{\mbox{corner}}}\,)\) —proved challenging, only yielding a test F 1 score of 0.52 ± 0.03, for a GATv2 base model. Note that here we use the F 1 score—the harmonic mean of precision and recall—as it is commonly used in binary classification problems over imbalanced datasets, such as shot prediction. However, given that we already have a potent receiver predictor, we decided to use its output to give us additional insight into whether or not a shot had been taken. Hence, we opted to decompose the probability of taking a shot as

where \({\mathbb{P}}(\,{{\mbox{receiver}}}| {{\mbox{corner}}}\,)\) are the probabilities computed by TacticAI’s receiver prediction system, and \({\mathbb{P}}(\,{{\mbox{shot}}}| {{\mbox{receiver}}},{{\mbox{corner}}}\,)\) models the conditional shot probability after a specific player makes first contact with the ball. This was implemented through providing an additional global feature to indicate the receiver in the corresponding corner kick (Table  1 ) while the architecture otherwise remained the same as that of receiver prediction (Supplementary Fig.  2 , see the “Methods” section). At training time, we feed the ground-truth receiver as input to the model—at inference time, we attempt every possible receiver, weighing their contributions using the probabilities given by TacticAI’s receiver predictor, as per Eq. ( 1 ). This two-phased approach yielded a final test F 1 score of 0.68 ± 0.04 for shot prediction, which encodes significantly more signal than the unconditional shot predictor, especially considering the many unobservables associated with predicting shot events. Just as for receiver prediction, this performance can be further improved using geometric deep learning; a conditional GATv2 shot predictor with D 2 group convolutions achieves an F 1 score of 0.71 ± 0.01.

Moreover, we also observe that, even just through predicting the receivers, without explicitly classifying any other salient features of corners, TacticAI learned generalisable representations of the data. Specifically, team setups with similar tactical patterns tend to cluster together in TacticAI’s latent space (Fig.  2 ). However, no clear clusters are observed in the raw input space (Supplementary Fig.  1 ). This indicates that TacticAI can be leveraged as a useful corner kick retrieval system, and we will present our evaluation of this hypothesis in the “Case study with expert raters” section.

We visualise the latent representations of attacking and defending teams in 1024 corner kicks using t -SNE. A latent team embedding in one corner kick sample is the mean of the latent player representations on the same attacking ( A – C ) or defending ( D ) team. Given the reference corner kick sample ( A ), we retrieve another corner kick sample ( B ) with respect to the closest distance of their representations in the latent space. We observe that ( A ) and ( B ) are both out-swing corner kicks and share similar patterns of their attacking tactics, which are highlighted with rectangles having the same colours, although they bear differences with respect to the absolute positions and velocities of the players. All the while, the latent representation of an in-swing attack ( C ) is distant from both ( A ) and ( B ) in the latent space. The red arrows are only used to demonstrate the difference between in- and out-swing corner kicks, not the actual ball trajectories.

Lastly, it is worth emphasising that the utility of the shot predictor likely does not come from forecasting whether a shot event will occur—a challenging problem with many imponderables—but from analysing the difference in predicted shot probability across multiple corners. Indeed, in the following section, we will show how TacticAI’s generative tactic refinements can directly influence the predicted shot probabilities, which will then corresponds to highly favourable evaluation by our expert raters in the “Case study with expert raters” section.

Controlled tactic refinement using class-conditional generative models

Equipped with components that are able to potently relate corner kicks with their various outcomes (e.g. receivers and shot events), we can explore the use of TacticAI to suggest adjustments of tactics, in order to amplify or reduce the likelihood of certain outcomes.

Specifically, we aim to produce adjustments to the movements of players on one of the two teams, including their positions and velocities, which would maximise or minimise the probability of a shot event, conditioned on the initial corner setup, consisting of the movements of players on both teams and their heights and weights. In particular, although in real-world scenarios both teams may react simultaneously to the movements of each other, in our study, we focus on moderate adjustments to player movements, which help to detect players that are not responding to a tactic properly. Due to this reason, we simplify the process of tactic refinement through generating the adjustments for only one team while keeping the other fixed. The way we train a model for this task is through an auto-encoding objective: we feed the ground-truth shot outcome (a binary indicator) as an additional graph-level feature to TacticAI’s model (Table  1 ), and then have it learn to reconstruct a probability distribution of the input player coordinates (Fig.  1 B, also see the “Methods” section). As a consequence, our tactic adjustment system does not depend on the previously discussed shot predictor—although we can use the shot predictor to evaluate whether the adjustments make a measurable difference in shot probability.

This autoencoder-based generative model is an individual component that separates from TacticAI’s predictive systems. All three systems share the encoder architecture (without sharing parameters), but use different decoders (see the “Methods” section). At inference time, we can instead feed in a desired shot outcome for the given corner setup, and then sample new positions and velocities for players on one team using this probability distribution. This setup, in principle, allows for flexible downstream use, as human coaches can optimise corner kick setups through generating adjustments conditioned on the specific outcomes of their interest—e.g., increasing shot probability for the attacking team, decreasing it for the defending team (Fig.  3 ) or amplifying the chance that a particular striker receives the ball.

TacticAI makes it possible for human coaches to redesign corner kick tactics in ways that help maximise the probability of a positive outcome for either the attacking or the defending team by identifying key players, as well as by providing temporally coordinated tactic recommendations that take all players into consideration. As demonstrated in the present example ( A ), for a corner kick in which there was a shot attempt in reality ( B ), TacticAI can generate a tactically-adjusted setting in which the shot probability has been reduced, by adjusting the positioning of the defenders ( D ). The suggested defender positions result in reduced receiver probability for attacking players 2–5 (see bottom row), while the receiver probability of Attacker 1, who is distant from the goalpost, has been increased ( C ). The model is capable of generating multiple such scenarios. Coaches can inspect the different options visually and additionally consult TacticAI’s quantitative analysis of the presented tactics.

We first evaluate the generated adjustments quantitatively, by verifying that they are indistinguishable from the original corner kick distribution using a classifier. To do this, we synthesised a dataset consisting of 200 corner kick samples and their corresponding conditionally generated adjustments. Specifically, for corners without a shot event, we generated adjustments for the attacking team by setting the shot event feature to 1, and vice-versa for the defending team when a shot event did happen. We found that the real and generated samples were not distinguishable by an MLP classifier, with an F 1 score of 0.53 ± 0.05, indicating random chance level accuracy. This result indicates that the adjustments produced by TacticAI are likely similar enough to real corner kicks that the MLP is unable to tell them apart. Note that, in spite of this similarity, TacticAI recommends player-level adjustments that are not negligible—in the following section we will illustrate several salient examples of this. To more realistically validate the practical indistinguishability of TacticAI’s adjustments from realistic corners, we also evaluated the realism of the adjustments in a case study with human experts, which we will present in the following section.

In addition, we leveraged our TacticAI shot predictor to estimate whether the proposed adjustments were effective. We did this by analysing 100 corner kick samples in which threatening shots occurred, and then, for each sample, generated one defensive refinement through setting the shot event feature to 0. We observed that the average shot probability significantly decreased, from 0.75 ± 0.14 for ground-truth corners to 0.69 ± 0.16 for adjustments ( z  = 2.62,  p  < 0.001). This observation was consistent when testing for attacking team refinements (shot probability increased from 0.18 ± 0.16 to 0.31 ± 0.26 ( z  = −4.46,  p  < 0.001)). Moving beyond this result, we also asked human raters to assess the utility of TacticAI’s proposed adjustments within our case study, which we detail next.

Case study with expert raters

Although quantitative evaluation with well-defined benchmark datasets was critical for the technical development of TacticAI, the ultimate test of TacticAI as a football tactic assistant is its practical downstream utility being recognised by professionals in the industry. To this end, we evaluated TacticAI through a case study with our partners at Liverpool FC (LFC). Specifically, we invited a group of five football experts: three data scientists, one video analyst, and one coaching assistant. Each of them completed four tasks in the case study, which evaluated the utility of TacticAI’s components from several perspectives; these include (1) the realism of TacticAI’s generated adjustments, (2) the plausibility of TacticAI’s receiver predictions, (3) effectiveness of TacticAI’s embeddings for retrieving similar corners, and (4) usefulness of TacticAI’s recommended adjustments. We provide an overview of our study’s results here and refer the interested reader to Supplementary Figs.  3 – 5 and the  Supplementary Methods for additional details.

We first simultaneously evaluated the realism of the adjusted corner kicks generated by TacticAI, and the plausibility of its receiver predictions. Going through a collection of 50 corner kick samples, we first asked the raters to classify whether a given sample was real or generated by TacticAI, and then they were asked to identify the most likely receivers in the corner kick sample (Supplementary Fig.  3 ).

On the task of classifying real and generated samples, first, we found that the raters’ average F 1 score of classifying the real vs. generated samples was only 0.60 ± 0.04, with individual F 1 scores ( \({F}_{1}^{A}=0.54,{F}_{1}^{B}=0.64,{F}_{1}^{C}=0.65,{F}_{1}^{D}=0.62,{F}_{1}^{E}=0.56\) ), indicating that the raters were, in many situations, unable to distinguish TacticAI’s adjustments from real corners.

The previous evaluation focused on analysing realism detection performance across raters. We also conduct a study that analyses realism detection across samples. Specifically, we assigned ratings for each sample—assigning +1 to a sample if it was identified as real by a human rater, and 0 otherwise—and computed the average rating for each sample across the five raters. Importantly, by studying the distribution of ratings, we found that there was no significant difference between the average ratings assigned to real and generated corners ( z  = −0.34,  p  > 0.05) (Fig.  4 A). Hence, the real and generated samples were assigned statistically indistinguishable average ratings by human raters.

In task 1, we tested the statistical difference between the real corner kick samples and the synthetic ones generated by TacticAI from two aspects: ( A.1 ) the distributions of their assigned ratings, and ( A.2 ) the corresponding histograms of the rating values. Analogously, in task 2 (receiver prediction), ( B.1 ) we track the distributions of the top-3 accuracy of receiver prediction using those samples, and ( B.2 ) the corresponding histogram of the mean rating per sample. No statistical difference in the mean was observed in either cases (( A.1 ) ( z  = −0.34,  p  > 0.05), and ( B.1 ) ( z  = 0.97,  p  > 0.05)). Additionally, we observed a statistically significant difference between the ratings of different raters on receiver prediction, with three clear clusters emerging ( C ). Specifically, Raters A and E had similar ratings ( z  = 0.66,  p  > 0.05), and Raters B and D also rated in similar ways ( z  = −1.84,  p  > 0.05), while Rater C responded differently from all other raters. This suggests a good level of variety of the human raters with respect to their perceptions of corner kicks. In task 3—identifying similar corners retrieved in terms of salient strategic setups—there were no significant differences among the distributions of the ratings by different raters ( D ), suggesting a high level of agreement on the usefulness of TacticAI’s capability of retrieving similar corners ( F 1,4  = 1.01,  p  > 0.1). Finally, in task 4, we compared the ratings of TacticAI’s strategic refinements across the human raters ( E ) and found that the raters also agreed on the general effectiveness of the refinements recommended by TacticAI ( F 1,4  = 0.45,  p  > 0.05). Note that the violin plots used in B.1 and C – E model a continuous probability distribution and hence assign nonzero probabilities to values outside of the allowed ranges. We only label y -axis ticks for the possible set of ratings.

For the task of identifying receivers, we rated TacticAI’s predictions with respect to a rater as +1 if at least one of the receivers identified by the rater appeared in TacticAI’s top-3 predictions, and 0 otherwise. The average top-3 accuracy among the human raters was 0.79 ± 0.18; specifically, 0.81 ± 0.17 for the real samples, and 0.77 ± 0.21 for the generated ones. These scores closely line up with the accuracy of TacticAI in predicting receivers for held-out test corners, validating our quantitative study. Further, after averaging the ratings for receiver prediction sample-wise, we found no statistically significant difference between the average ratings of predicting receivers over the real and generated samples ( z  = 0.97,  p  > 0.05) (Fig.  4 B). This indicates that TacticAI was equally performant in predicting the receivers of real corners and TacticAI-generated adjustments, and hence may be leveraged for this purpose even in simulated scenarios.

There is a notably high variance in the average receiver prediction rating of TacticAI. We hypothesise that this is due to the fact that different raters may choose to focus on different salient features when evaluating the likely receivers (or even the amount of likely receivers). We set out to validate this hypothesis by testing the pair-wise similarity of the predictions by the human raters through running a one-away analysis of variance (ANOVA), followed by a Tukey test. We found that the distributions of the five raters’ predictions were significantly different ( F 1,4  = 14.46,  p  < 0.001) forming three clusters (Fig.  4 C). This result indicates that different human raters—as suggested by their various titles at LFC—may often use very different leads when suggesting plausible receivers. The fact that TacticAI manages to retain a high top-3 accuracy in such a setting suggests that it was able to capture the salient patterns of corner kick strategies, which broadly align with human raters’ preferences. We will further test this hypothesis in the third task—identifying similar corners.

For the third task, we asked the human raters to judge 50 pairs of corners for their similarity. Each pair consisted of a reference corner and a retrieved corner, where the retrieved corner was chosen either as the nearest-neighbour of the reference in terms of their TacticAI latent space representations, or—as a feature-level heuristic—the cosine similarities of their raw features (Supplementary Fig.  4 ) in our corner kick dataset. We score the raters’ judgement of a pair as +1 if they considered the corners presented in the case to be usefully similar, otherwise, the pair is scored with 0. We first computed, for each rater, the recall with which they have judged a baseline- or TacticAI-retrieved pair as usefully similar—see description of Task 3 in the  Supplementary Methods . For TacticAI retrievals, the average recall across all raters was 0.59 ± 0.09, and for the baseline system, the recall was 0.36 ± 0.10. Secondly, we assess the statistical difference between the results of the two methods by averaging the ratings for each reference–retrieval pair, finding that the average rating of TacticAI retrievals is significantly higher than the average rating of baseline method retrievals ( z  = 2.34,  p  < 0.05). These two results suggest that TacticAI significantly outperforms the feature-space baseline as a method for mining similar corners. This indicates that TacticAI is able to extract salient features from corners that are not trivial to extract from the input data alone, reinforcing it as a potent tool for discovering opposing team tactics from available data. Finally, we observed that this task exhibited a high level of inter-rater agreement for TacticAI-retrieved pairs ( F 1,4  = 1.01,  p  > 0.1) (Fig.  4 D), suggesting that human raters were largely in agreement with respect to their assessment of TacticAI’s performance.

Finally, we evaluated TacticAI’s player adjustment recommendations for their practical utility. Specifically, each rater was given 50 tactical refinements together with the corresponding real corner kick setups—see Supplementary Fig.  5 , and the “Case study design” section in the  Supplementary Methods . The raters were then asked to rate each refinement as saliently improving the tactics (+1), saliently making them worse (−1), or offering no salient differences (0). We calculated the average rating assigned by each of the raters (giving us a value in the range [− 1, 1] for each rater). The average of these values across all five raters was 0.7 ± 0.1. Further, for 45 of the 50 situations (90%), the human raters found TacticAI’s suggestion to be favourable on average (by majority voting). Both of these results indicate that TacticAI’s recommendations are salient and useful to a downstream football club practitioner, and we set out to validate this with statistical tests.

We performed statistical significance testing of the observed positive ratings. First, for each of the 50 situations, we averaged its ratings across all five raters and then ran a t -test to assess whether the mean rating was significantly larger than zero. Indeed, the statistical test indicated that the tactical adjustments recommended by TacticAI were constructive overall ( \({t}_{49}^{{{{{{{{\rm{avg}}}}}}}}}=9.20,\, p \, < \, 0.001\) ). Secondly, we verified that each of the five raters individually found TacticAI’s recommendations to be constructive, running a t -test on each of their ratings individually. For all of the five raters, their average ratings were found to be above zero with statistical significance ( \({t}_{49}^{A}=5.84,\, {p}^{A} \, < \, 0.001;{t}_{49}^{B}=7.88,\; {p}^{B} \, < \, 0.001;{t}_{49}^{C}=7.00,\; {p}^{C} \, < \, 0.001;{t}_{49}^{D}=6.04,\; {p}^{D} \, < \, 0.001;{t}_{49}^{E}=7.30,\, {p}^{E} \, < \, 0.001\) ). In addition, their ratings also shared a high level of inter-agreement ( F 1,4  = 0.45,  p  > 0.05) (Fig.  4 E), suggesting a level of practical usefulness that is generally recognised by human experts, even though they represent different backgrounds.

Taking all of these results together, we find TacticAI to possess strong components for prediction, retrieval, and tactical adjustments on corner kicks. To illustrate the kinds of salient recommendations by TacticAI, in Fig.  5 we present four examples with a high degree of inter-rater agreement.

These examples are selected from our case study with human experts, to illustrate the breadth of tactical adjustments that TacticAI suggests to teams defending a corner. The density of the yellow circles coincides with the number of times that the corresponding change is recognised as constructive by human experts. Instead of optimising the movement of one specific player, TacticAI can recommend improvements for multiple players in one generation step through suggesting better positions to block the opposing players, or better orientations to track them more efficiently. Some specific comments from expert raters follow. In A , according to raters, TacticAI suggests more favourable positions for several defenders, and improved tracking runs for several others—further, the goalkeeper is positioned more deeply, which is also beneficial. In B , TacticAI suggests that the defenders furthest away from the corner make improved covering runs, which was unanimously deemed useful, with several other defenders also positioned more favourably. In C , TacticAI recommends improved covering runs for a central group of defenders in the penalty box, which was unanimously considered salient by our raters. And in D , TacticAI suggests substantially better tracking runs for two central defenders, along with a better positioning for two other defenders in the goal area.

We have demonstrated an AI assistant for football tactics and provided statistical evidence of its efficacy through a comprehensive case study with expert human raters from Liverpool FC. First, TacticAI is able to accurately predict the first receiver after a corner kick is taken as well as the probability of a shot as the direct result of the corner. Second, TacticAI has been shown to produce plausible tactical variations that improve outcomes in a salient way, while being indistinguishable from real scenarios by domain experts. And finally, the system’s latent player representations are a powerful means to retrieve similar set-piece tactics, allowing coaches to analyse relevant tactics and counter-tactics that have been successful in the past.

The broader scope of strategy modelling in football has previously been addressed from various individual angles, such as pass prediction 14 , 15 , 16 , shot prediction 3 or corner kick tactical classification 7 . However, to the best of our knowledge, our work stands out by combining and evaluating predictive and generative modelling of corner kicks for tactic development. It also stands out in its method of applying geometric deep learning, allowing for efficiently incorporating various symmetries of the football pitch for improved data efficiency. Our method incorporates minimal domain knowledge and does not rely on intricate feature engineering—though its factorised design naturally allows for more intricate feature engineering approaches when such features are available.

Our methodology requires the position and velocity estimates of all players at the time of execution of the corner and subsequent events. Here, we derive these from high-quality tracking and event data, with data availability from tracking providers limited to top leagues. Player tracking based on broadcast video would increase the reach and training data substantially, but would also likely result in noisier model inputs. While the attention mechanism of GATs would allow us to perform introspection of the most salient factors contributing to the model outcome, our method does not explicitly model exogenous (aleatoric) uncertainty, which would be valuable context for the football analyst.

While the empirical study of our method’s efficacy has been focused on corner kicks in association football, it readily generalises to other set pieces (such as throw-ins, which similarly benefit from similarity retrieval, pass and/or shot prediction) and other team sports with suspended play situations. The learned representations and overall framing of TacticAI also lay the ground for future research to integrate a natural language interface that enables domain-grounded conversations with the assistant, with the aim to retrieve particular situations of interest, make predictions for a given tactical variant, compare and contrast, and guide through an interactive process to derive tactical suggestions. It is thus our belief that TacticAI lays the groundwork for the next-generation AI assistant for football.

We devised TacticAI as a geometric deep learning pipeline, further expanded in this section. We process labelled spatio-temporal football data into graph representations, and train and evaluate on benchmarking tasks cast as classification or regression. These steps are presented in sequence, followed by details on the employed computational architecture.

Raw corner kick data

The raw dataset consisted of 9693 corner kicks collected from the 2020–21, 2021–22, and 2022–23 (up to January 2023) Premier League seasons. The dataset was provided by Liverpool FC and comprises four separate data sources, described below.

Our primary data source is spatio-temporal trajectory frames (tracking data), which tracked all on-pitch players and the ball, for each match, at 25 frames per second. In addition to player positions, their velocities are derived from position data through filtering. For each corner kick, we only used the frame in which the kick is being taken as input information.

Secondly, we also leverage event stream data, which annotated the events or actions (e.g., passes, shots and goals) that have occurred in the corresponding tracking frames.

Thirdly, the line-up data for the corresponding games, which recorded the players’ profiles, including their heights, weights and roles, is also used.

Lastly, we have access to miscellaneous game data, which contains the game days, stadium information, and pitch length and width in meters.

Graph representation and construction

We assumed that we were provided with an input graph \({{{{{{{\mathcal{G}}}}}}}}=({{{{{{{\mathcal{V}}}}}}}},\,{{{{{{{\mathcal{E}}}}}}}})\) with a set of nodes \({{{{{{{\mathcal{V}}}}}}}}\) and edges \({{{{{{{\mathcal{E}}}}}}}}\subseteq {{{{{{{\mathcal{V}}}}}}}}\times {{{{{{{\mathcal{V}}}}}}}}\) . Within the context of football games, we took \({{{{{{{\mathcal{V}}}}}}}}\) to be the set of 22 players currently on the pitch for both teams, and we set \({{{{{{{\mathcal{E}}}}}}}}={{{{{{{\mathcal{V}}}}}}}}\times {{{{{{{\mathcal{V}}}}}}}}\) ; that is, we assumed all pairs of players have the potential to interact. Further analyses, leveraging more specific choices of \({{{{{{{\mathcal{E}}}}}}}}\) , would be an interesting avenue for future work.

Additionally, we assume that the graph is appropriately featurised. Specifically, we provide a node feature matrix, \({{{{{{{\bf{X}}}}}}}}\in {{\mathbb{R}}}^{| {{{{{{{\mathcal{V}}}}}}}}| \times k}\) , an edge feature tensor, \({{{{{{{\bf{E}}}}}}}}\in {{\mathbb{R}}}^{| {{{{{{{\mathcal{V}}}}}}}}| \times | {{{{{{{\mathcal{V}}}}}}}}| \times l}\) , and a graph feature vector, \({{{{{{{\bf{g}}}}}}}}\in {{\mathbb{R}}}^{m}\) . The appropriate entries of these objects provide us with the input features for each node, edge, and graph. For example, \({{{{{{{{\bf{x}}}}}}}}}_{u}\in {{\mathbb{R}}}^{k}\) would provide attributes of an individual player \(u\in {{{{{{{\mathcal{V}}}}}}}}\) , such as position, height and weight, and \({{{{{{{{\bf{e}}}}}}}}}_{uv}\in {{\mathbb{R}}}^{l}\) would provide the attributes of a particular pair of players \((u,\, v)\in {{{{{{{\mathcal{E}}}}}}}}\) , such as their distance, and whether they belong to the same team. The graph feature vector, g , can be used to store global attributes of interest to the corner kick, such as the game time, current score, or ball position. For a simplified visualisation of how a graph neural network would process such an input, refer to Fig.  1 A.

To construct the input graphs, we first aligned the four data sources with respect to their game IDs and timestamps and filtered out 2517 invalid corner kicks, for which the alignment failed due to missing data, e.g., missing tracking frames or event labels. This filtering yielded 7176 valid corner kicks for training and evaluation. We summarised the exact information that was used to construct the input graphs in Table  2 . In particular, other than player heights (measured in centimeters (cm)) and weights (measured in kilograms (kg)), the players were anonymous in the model. For the cases in which the player profiles were missing, we set their heights and weights to 180 cm and 75 kg, respectively, as defaults. In total, we had 385 such occurrences out of a total of 213,246( = 22 × 9693) during data preprocessing. We downscaled the heights and weights by a factor of 100. Moreover, for each corner kick, we zero-centred the positions of on-pitch players and normalised them onto a 10 m × 10 m pitch, and their velocities were re-scaled accordingly. For the cases in which the pitch dimensions were missing, we used a standard pitch dimension of 110 m × 63 m as default.

We summarised the grouping of the features in Table  1 . The actual features used in different benchmark tasks may differ, and we will describe this in more detail in the next section. To focus on modelling the high-level tactics played by the attacking and defending teams, other than a binary indicator for ball possession—which is 1 for the corner kick taker and 0 for all other players—no information of ball movement, neither positions nor velocities, was used to construct the input graphs. Additionally, we do not have access to the player’s vertical movement, therefore only information on the two-dimensional movements of each player is provided in the data. We do however acknowledge that such information, when available, would be interesting to consider in a corner kick outcome predictor, considering the prevalence of aerial battles in corners.

Benchmark tasks construction

TacticAI consists of three predictive and generative models, which also correspond to three benchmark tasks implemented in this study. Specifically, (1) Receiver prediction, (2) Threatening shot prediction, and (3) Guided generation of team positions and velocities (Table  1 ). The graphs of all the benchmark tasks used the same feature space of nodes and edges, differing only in the global features.

For all three tasks, our models first transform the node features to a latent node feature matrix, \({{{{{{{\bf{H}}}}}}}}={f}_{{{{{{{{\mathcal{G}}}}}}}}}({{{{{{{\bf{X}}}}}}}},\, {{{{{{{\bf{E}}}}}}}},\, {{{{{{{\bf{g}}}}}}}})\) , from which we could answer queries: either about individual players—in which case we learned a relevant classifier or regressor over the h u vectors (the rows of H )—or about the occurrence of a global event (e.g. shot taken)—in which case we classified or regressed over the aggregated player vectors, ∑ u h u . In both cases, the classifiers were trained using stochastic gradient descent over an appropriately chosen loss function, such as categorical cross-entropy for classifiers, and mean squared error for regressors.

For different tasks, we extracted the corresponding ground-truth labels from either the event stream data or the tracking data. Specifically, (1) We modelled receiver prediction as a node classification task and labelled the first player to touch the ball after the corner was taken as the target node. This player could be either an attacking or defensive player. (2) Shot prediction was modelled as graph classification. In particular, we considered a next-ball-touch action by the attacking team as a shot if it was a direct corner, a goal, an aerial, hit on the goalposts, a shot attempt saved by the goalkeeper, or missing target. This yielded 1736 corners labelled as a shot being taken, and 5440 corners labelled as a shot not being taken. (3) For guided generation of player position and velocities, no additional label was needed, as this model relied on a self-supervised reconstruction objective.

The entire dataset was split into training and evaluation sets with an 80:20 ratio through random sampling, and the same splits were used for all tasks.

Graph neural networks

The central model of TacticAI is the graph neural network (GNN) 9 , which computes latent representations on a graph by repeatedly combining them within each node’s neighbourhood. Here we define a node’s neighbourhood, \({{{{{{{{\mathcal{N}}}}}}}}}_{u}\) , as the set of all first-order neighbours of node u , that is, \({{{{{{{{\mathcal{N}}}}}}}}}_{u}=\{v\,| \,(v,\, u)\in {{{{{{{\mathcal{E}}}}}}}}\}\) . A single GNN layer then transforms the node features by passing messages between neighbouring nodes 17 , following the notation of related work 10 , and the implementation of the CLRS-30 benchmark baselines 18 :

where \(\psi :{{\mathbb{R}}}^{k}\times {{\mathbb{R}}}^{k}\times {{\mathbb{R}}}^{l}\times {{\mathbb{R}}}^{m}\to {{\mathbb{R}}}^{{k}^{{\prime} }}\) and \(\phi :{{\mathbb{R}}}^{k}\times {{\mathbb{R}}}^{{k}^{{\prime} }}\to {{\mathbb{R}}}^{{k}^{{\prime} }}\) are two learnable functions (e.g. multilayer perceptrons), \({{{{{{{{\bf{h}}}}}}}}}_{u}^{(t)}\) are the features of node u after t GNN layers, and ⨁ is any permutation-invariant aggregator, such as sum, max, or average. By definition, we set \({{{{{{{{\bf{h}}}}}}}}}_{u}^{(0)}={{{{{{{{\bf{x}}}}}}}}}_{u}\) , and iterate Eq. ( 2 ) for T steps, where T is a hyperparameter. Then, we let \({{{{{{{\bf{H}}}}}}}}={f}_{{{{{{{{\mathcal{G}}}}}}}}}({{{{{{{\bf{X}}}}}}}},\, {{{{{{{\bf{E}}}}}}}},\, {{{{{{{\bf{g}}}}}}}})={{{{{{{{\bf{H}}}}}}}}}^{(T)}\) be the final node embeddings coming out of the GNN.

It is well known that Eq. ( 2 ) is remarkably general; it can be used to express popular models such as Transformers 19 as a special case, and it has been argued that all discrete deep learning models can be expressed in this form 20 , 21 . This makes GNNs a perfect framework for benchmarking various approaches to modelling player–player interactions in the context of football.

Different choices of ψ , ϕ and ⨁ yield different architectures. In our case, we utilise a message function that factorises into an attentional mechanism, \(a:{{\mathbb{R}}}^{k}\times {{\mathbb{R}}}^{k}\times {{\mathbb{R}}}^{l}\times {{\mathbb{R}}}^{m}\to {\mathbb{R}}\) :

yielding the graph attention network (GAT) architecture 12 . In our work, specifically, we use a two-layer multilayer perceptron for the attentional mechanism, as proposed by GATv2 11 :

where \({{{{{{{{\bf{W}}}}}}}}}_{1},\, {{{{{{{{\bf{W}}}}}}}}}_{2}\in {{\mathbb{R}}}^{k\times h}\) , \({{{{{{{{\bf{W}}}}}}}}}_{e}\in {{\mathbb{R}}}^{l\times h}\) , \({{{{{{{{\bf{W}}}}}}}}}_{g}\in {{\mathbb{R}}}^{m\times h}\) and \({{{{{{{\bf{a}}}}}}}}\in {{\mathbb{R}}}^{h}\) are the learnable parameters of the attentional mechanism, and LeakyReLU is the leaky rectified linear activation function. This mechanism computes coefficients of interaction (a single scalar value) for each pair of connected nodes ( u ,  v ), which are then normalised across all neighbours of u using the \({{{{{{{\rm{softmax}}}}}}}}\) function.

Through early-stage experimentation, we have ascertained that GATs are capable of matching the performance of more generic choices of ψ (such as the MPNN 17 ) while being more scalable. Hence, we focus our study on the GAT model in this work. More details can be found in the subsection “Ablation study” section.

Geometric deep learning

In spite of the power of Eq. ( 2 ), using it in its full generality is often prone to overfitting, given the large number of parameters contained in ψ and ϕ . This problem is exacerbated in the football analytics domain, where gold-standard data is generally very scarce—for example, in the English Premier League, only a few hundred games are played every season.

In order to tackle this issue, we can exploit the immense regularity of data arising from football games. Strategically equivalent game states are also called transpositions, and symmetries such as arriving at the same chess position through different move sequences have been exploited computationally since the 1960s 22 . Similarly, game rotations and reflections may yield equivalent strategic situations 23 . Using the blueprint of geometric deep learning (GDL) 10 , we can design specialised GNN architectures that exploit this regularity.

That is, geometric deep learning is a generic methodology for deriving mathematical constraints on neural networks, such that they will behave predictably when inputs are transformed in certain ways. In several important cases, these constraints can be directly resolved, directly informing neural network architecture design. For a comprehensive example of point clouds under 3D rotational symmetry, see Fuchs et al. 24 .

To elucidate several aspects of the GDL framework on a high level, let us assume that there exists a group of input data transformations (symmetries), \({\mathfrak{G}}\) under which the ground-truth label remains unchanged. Specifically, if we let y ( X ,  E ,  g ) be the label given to the graph featurised with X ,  E ,  g , then for every transformation \({\mathfrak{g}}\in {\mathfrak{G}}\) , the following property holds:

This condition is also referred to as \({\mathfrak{G}}\) -invariance. Here, by \({\mathfrak{g}}({{{{{{{\bf{X}}}}}}}})\) we denote the result of transforming X by \({\mathfrak{g}}\) —a concept also known as a group action. More generally, it is a function of the form \({\mathfrak{G}}\times {{{{{{{\mathcal{S}}}}}}}}\to {{{{{{{\mathcal{S}}}}}}}}\) for some state set \({{{{{{{\mathcal{S}}}}}}}}\) . Note that a single group element, \({\mathfrak{g}}\in {\mathfrak{G}}\) can easily produce different actions on different \({{{{{{{\mathcal{S}}}}}}}}\) —in this case, \({{{{{{{\mathcal{S}}}}}}}}\) could be \({{\mathbb{R}}}^{| {{{{{{{\mathcal{V}}}}}}}}| \times k}\) ( X ), \({{\mathbb{R}}}^{| {{{{{{{\mathcal{V}}}}}}}}| \times | {{{{{{{\mathcal{V}}}}}}}}| \times l}\) ( E ) and \({{\mathbb{R}}}^{m}\) ( g ).

It is worth noting that GNNs may also be derived using a GDL perspective if we set the symmetry group \({\mathfrak{G}}\) to \({S}_{| {{{{{{{\mathcal{V}}}}}}}}}|\) , the permutation group of \(| {{{{{{{\mathcal{V}}}}}}}}|\) objects. Owing to the design of Eq. ( 2 ), its outputs will not be dependent on the exact permutation of nodes in the input graph.

Frame averaging

A simple mechanism to enforce \({\mathfrak{G}}\) -invariance, given any predictor \({f}_{{{{{{{{\mathcal{G}}}}}}}}}({{{{{{{\bf{X}}}}}}}},\, {{{{{{{\bf{E}}}}}}}},\, {{{{{{{\bf{g}}}}}}}})\) , performs frame averaging across all \({\mathfrak{G}}\) -transformed inputs:

This ensures that all \({\mathfrak{G}}\) -transformed versions of a particular input (also known as that input’s orbit) will have exactly the same output, satisfying Eq. ( 5 ). A variant of this approach has also been applied in the AlphaGo architecture 25 to encode symmetries of a Go board.

In our specific implementation, we set \({\mathfrak{G}}={D}_{2}=\{{{{{{{{\rm{id}}}}}}}},\leftrightarrow,\updownarrow,\leftrightarrow \updownarrow \}\) , the dihedral group. Exploiting D 2 -invariance allows us to encode quadrant symmetries. Each element of the D 2 group encodes the presence of vertical or horizontal reflections of the input football pitch. Under these transformations, the pitch is assumed completely symmetric, and hence many predictions, such as which player receives the corner kick, or takes a shot from it, can be safely assumed unchanged. As an example of how to compute transformed features in Eq. ( 6 ), ↔( X ) horizontally reflects all positional features of players in X (e.g. the coordinates of the player), and negates the x -axis component of their velocity.

Group convolutions

While the frame averaging approach of Eq. ( 6 ) is a powerful way to restrict GNNs to respect input symmetries, it arguably misses an opportunity for the different \({\mathfrak{G}}\) -transformed views to interact while their computations are being performed. For small groups such as D 2 , a more fine-grained approach can be assumed, operating over a single GNN layer in Eq. ( 2 ), which we will write shortly as \({{{{{{{{\bf{H}}}}}}}}}^{(t)}={g}_{{{{{{{{\mathcal{G}}}}}}}}}({{{{{{{{\bf{H}}}}}}}}}^{(t-1)},\, {{{{{{{\bf{E}}}}}}}},\, {{{{{{{\bf{g}}}}}}}})\) . The condition that we need a symmetry-respecting GNN layer to satisfy is as follows, for all transformations \({\mathfrak{g}}\in {\mathfrak{G}}\) :

that is, it does not matter if we apply \({\mathfrak{g}}\) it to the input or the output of the function \({g}_{{{{{{{{\mathcal{G}}}}}}}}}\) —the final answer is the same. This condition is also referred to as \({\mathfrak{G}}\) -equivariance, and it has recently proved to be a potent paradigm for developing powerful GNNs over biochemical data 24 , 26 .

To satisfy D 2 -equivariance, we apply the group convolution approach 13 . Therein, views of the input are allowed to directly interact with their \({\mathfrak{G}}\) -transformed variants, in a manner very similar to grid convolutions (which is, indeed, a special case of group convolutions, setting \({\mathfrak{G}}\) to be the translation group). We use \({{{{{{{{\bf{H}}}}}}}}}_{{\mathfrak{g}}}^{(t)}\) to denote the \({\mathfrak{g}}\) -transformed view of the latent node features at layer t . Omitting E and g inputs for brevity, and using our previously designed layer \({g}_{{{{{{{{\mathcal{G}}}}}}}}}\) as a building block, we can perform a group convolution as follows:

Here, ∥ is the concatenation operation, joining the two node feature matrices column-wise; \({{\mathfrak{g}}}^{-1}\) is the inverse transformation to \({\mathfrak{g}}\) (which must exist as \({\mathfrak{G}}\) is a group); and \({{\mathfrak{g}}}^{-1}{\mathfrak{h}}\) is the composition of the two transformations.

Effectively, Eq. ( 8 ) implies our D 2 -equivariant GNN needs to maintain a node feature matrix \({{{{{{{{\bf{H}}}}}}}}}_{{\mathfrak{g}}}^{(t)}\) for every \({\mathfrak{G}}\) -transformation of the current input, and these views are recombined by invoking \({g}_{{{{{{{{\mathcal{G}}}}}}}}}\) on all pairs related together by applying a transformation \({\mathfrak{h}}\) . Note that all reflections are self-inverses, hence, in D 2 , \({\mathfrak{g}}={{\mathfrak{g}}}^{-1}\) .

It is worth noting that both the frame averaging in Eq. ( 6 ) and group convolution in Eq. ( 8 ) are similar in spirit to data augmentation. However, whereas standard data augmentation would only show one view at a time to the model, a frame averaging/group convolution architecture exhaustively generates all views and feeds them to the model all at once. Further, group convolutions allow these views to explicitly interact in a way that does not break symmetries. Here lies the key difference between the two approaches: frame averaging and group convolutions rigorously enforce the symmetries in \({\mathfrak{G}}\) , whereas data augmentation only provides implicit hints to the model about satisfying them. As a consequence of the exhaustive generation, Eqs. ( 6 ) and ( 8 ) are only feasible for small groups like D 2 . For larger groups, approaches like Steerable CNNs 27 may be employed.

Network architectures

While the three benchmark tasks we are performing have minor differences in the global features available to the model, the neural network models designed for them all have the same encoder–decoder architecture. The encoder has the same structure in all tasks, while the decoder model is tailored to produce appropriately shaped outputs for each benchmark task.

Given an input graph, TacticAI’s model first generates all relevant D 2 -transformed versions of it, by appropriately reflecting the player coordinates and velocities. We refer to the original input graph as the identity view, and the remaining three D 2 -transformed graphs as reflected views.

Once the views are prepared, we apply four group convolutional layers (Eq. ( 8 )) with a GATv2 base model (Eqs. ( 3 ) and ( 4 )) as the \({g}_{{{{{{{{\mathcal{G}}}}}}}}}\) function. Specifically, this means that, in Eqs. ( 3 ) and ( 4 ), every instance of \({{{{{{{{\bf{h}}}}}}}}}_{u}^{(t-1)}\) is replaced by the concatenation of \({({{{{{{{{\bf{h}}}}}}}}}_{{\mathfrak{h}}}^{(t-1)})}_{u}\parallel {({{{{{{{{\bf{h}}}}}}}}}_{{{\mathfrak{g}}}^{-1}{\mathfrak{h}}}^{(t-1)})}_{u}\) . Each GATv2 layer has eight attention heads and computes four latent features overall per player. Accordingly, once the four group convolutions are performed, we have a representation of \({{{{{{{\bf{H}}}}}}}}\in {{\mathbb{R}}}^{4\times 22\times 4}\) , where the first dimension corresponds to the four views ( \({{{{{{{{\bf{H}}}}}}}}}_{{{{{{{{\rm{id}}}}}}}}},\, {{{{{{{{\bf{H}}}}}}}}}_{\leftrightarrow },\, {{{{{{{{\bf{H}}}}}}}}}_{\updownarrow },\, {{{{{{{{\bf{H}}}}}}}}}_{\leftrightarrow \updownarrow }\in {{\mathbb{R}}}^{22\times 4}\) ), the second dimension corresponds to the players (eleven on each team), and the third corresponds to the 4-dimensional latent vector for each player node in this particular view. How this representation is used by the decoder depends on the specific downstream task, as we detail below.

For receiver prediction, which is a fully invariant function (i.e. reflections do not change the receiver), we perform simple frame averaging across all views, arriving at

and then learn a node-wise classifier over the rows of \({{{{{{{{\bf{H}}}}}}}}}^{{{{{{{{\rm{node}}}}}}}}}\in {{\mathbb{R}}}^{22\times 4}\) . We further decode H node into a logit vector \({{{{{{{\bf{O}}}}}}}}\in {{\mathbb{R}}}^{22}\) with a linear layer before computing the corresponding softmax cross entropy loss.

For shot prediction, which is once again fully invariant (i.e. reflections do not change the probability of a shot), we can further average the frame-averaged features across all players to get a global graph representation:

and then learn a binary classifier over \({{{{{{{{\bf{h}}}}}}}}}^{{{{{{{{\rm{graph}}}}}}}}}\in {{\mathbb{R}}}^{4}\) . Specifically, we decode the hidden vector into a single logit with a linear layer and compute the sigmoid binary cross-entropy loss with the corresponding label.

For guided generation (position/velocity adjustments), we generate the player positions and velocities with respect to a particular outcome of interest for the human coaches, predicted over the rows of the hidden feature matrix. For example, the model may adjust the defensive setup to decrease the shot probability by the attacking team. The model output is now equivariant rather than invariant—reflecting the pitch appropriately reflects the predicted positions and velocity vectors. As such, we cannot perform frame averaging, and take only the identity view’s features, \({{{{{{{{\bf{H}}}}}}}}}_{{{{{{{{\rm{id}}}}}}}}}\in {{\mathbb{R}}}^{22\times 4}\) . From this latent feature matrix, we can then learn a conditional distribution from each row, which models the positions or velocities of the corresponding player. To do this, we extend the backbone encoder with conditional variational autoencoder (CVAE 28 , 29 ). Specifically, for the u -th row of H id , h u , we first map its latent embedding to the parameters of a two-dimensional Gaussian distribution \({{{{{{{\mathcal{N}}}}}}}}({\mu }_{u}| {\sigma }_{u})\) , and then sample the coordinates and velocities from this distribution. At training time, we can efficiently propagate gradients through this sampling operation using the reparameterisation trick 28 : sample a random value \({\epsilon }_{u} \sim {{{{{{{\mathcal{N}}}}}}}}(0,1)\) for each player from the unit Gaussian distribution, and then treat μ u  +  σ u ϵ u as the sample for this player. In what follows, we omit edge features for brevity. For each corner kick sample X with the corresponding outcome o (e.g. a binary value indicating a shot event), we extend the standard VAE loss 28 , 29 to our case of outcome-conditional guided generation as

where h u is the player embedding corresponding to the u th row of H id , and \({\mathbb{KL}}\) is Kullback–Leibler (KL) divergence. Specifically, the first term is the generation loss between the real player input x u and the reconstructed sample decoded from h u with the decoder p ϕ . Using the KL term, the distribution of the latent embedding h u is regularised towards p ( h u ∣ o ), which is a multivariate Gaussian in our case.

A complete high-level summary of the generic encoder–decoder equivariant architecture employed by TacticAI can be summarised in Supplementary Fig.  2 . In the following section, we will provide empirical evidence for justifying these architectural decisions. This will be done through targeted ablation studies on our predictive benchmarks (receiver prediction and shot prediction).

Ablation study

We leveraged the receiver prediction task as a way to evaluate various base model architectures, and directly quantitatively assess the contributions of geometric deep learning in this context. We already see that the raw corner kick data can be better represented through geometric deep learning, yielding separable clusters in the latent space that could correspond to different attacking or defending tactics (Fig.  2 ). In addition, we hypothesise that these representations can also yield better performance on the task of receiver prediction. Accordingly, we ablate several design choices using deep learning on this task, as illustrated by the following four questions:

Does a factorised graph representation help? To assess this, we compare it against a convolutional neural network (CNN 30 ) baseline, which does not leverage a graph representation.

Does a graph structure help? To assess this, we compare against a Deep Sets 31 baseline, which only models each node in isolation without considering adjacency information—equivalently, setting each neighbourhood \({{{{{{{{\mathcal{N}}}}}}}}}_{u}\) to a singleton set { u }.

Are attentional GNNs a good strategy? To assess this, we compare against a message passing neural network 32 , MPNN baseline, which uses the fully potent GNN layer from Eq. ( 2 ) instead of the GATv2.

Does accounting for symmetries help? To assess this, we compare our geometric GATv2 baseline against one which does not utilise D 2 group convolutions but utilises D 2 frame averaging, and one which does not explicitly utilise any aspect of D 2 symmetries at all.

Each of these models has been trained for a fixed budget of 50,000 training steps. The test top- k receiver prediction accuracies of the trained models are provided in Supplementary Table  2 . As already discussed in the section “Results”, there is a clear advantage to using a full graph structure, as well as directly accounting for reflection symmetry. Further, the usage of the MPNN layer leads to slight overfitting compared to the GATv2, illustrating how attentional GNNs strike a good balance of expressivity and data efficiency for this task. Our analysis highlights the quantitative benefits of both graph representation learning and geometric deep learning for football analytics from tracking data. We also provide a brief ablation study for the shot prediction task in Supplementary Table  3 .

Training details

We train each of TacticAI’s models in isolation, using NVIDIA Tesla P100 GPUs. To minimise overfitting, each model’s learning objective is regularised with an L 2 norm penalty with respect to the network parameters. During training, we use the Adam stochastic gradient descent optimiser 33 over the regularised loss.

All models, including baselines, have been given an equal hyperparameter tuning budget, spanning the number of message passing steps ({1, 2, 4}), initial learning rate ({0.0001, 0.00005}), batch size ({128, 256}) and L 2 regularisation coefficient ({0.01, 0.005, 0.001, 0.0001, 0}). We summarise the chosen hyperparameters of each TacticAI model in Supplementary Table  1 .

Data availability

The data collected in the human experiments in this study have been deposited in the Zenodo database under accession code https://zenodo.org/records/10557063 , and the processed data which is used in the statistical analysis and to generate the relevant figures in the main text are available under the same accession code. The input and output data generated and/or analysed during the current study are protected and are not available due to data privacy laws and licensing restrictions. However, contact details of the input data providers are available from the corresponding authors on reasonable request.

Code availability

All the core models described in this research were built with the Graph Neural Network processors provided by the CLRS Algorithmic Reasoning Benchmark 18 , and their source code is available at https://github.com/google-deepmind/clrs . We are unable to release our code for this work as it was developed in a proprietary context; however, the corresponding authors are open to answer specific questions concerning re-implementations on request. For general data analysis, we used the following freely available packages: numpy v1.25.2 , pandas v1.5.3 , matplotlib v3.6.1 , seaborn v0.12.2 and scipy v1.9.3 . Specifically, the code of the statistical analysis conducted in this study is available at https://zenodo.org/records/10557063 .

The International Football Association Board (IFAB). Laws of the Game (The International Football Association Board, 2023).

Tuyls, K. et al. Game plan: what AI can do for football, and what football can do for AI. J. Artif. Intell. Res. 71 , 41–88 (2021).

Article   Google Scholar  

Goka, R., Moroto, Y., Maeda, K., Ogawa, T. & Haseyama, M. Prediction of shooting events in soccer videos using complete bipartite graphs and players’ spatial–temporal relations. Sensors 23 , 4506 (2023).

Article   ADS   PubMed   PubMed Central   Google Scholar  

Omidshafiei, S. et al. Multiagent off-screen behavior prediction in football. Sci. Rep. 12 , 8638 (2022).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Lang, S., Wild, R., Isenko, A. & Link, D. Predicting the in-game status in soccer with machine learning using spatiotemporal player tracking data. Sci. Rep. 12 , 16291 (2022).

Baccouche, M., Mamalet, F., Wolf, C., Garcia, C. & Baskurt, A. Action classification in soccer videos with long short-term memory recurrent neural networks. In International Conference on Artificial Neural Networks (eds Diamantaras, K., Duch, W. & Iliadis, L. S.) pages 154–159 (Springer, 2010).

Shaw, L. & Gopaladesikan, S. Routine inspection: a playbook for corner kicks. In Machine Learning and Data Mining for Sports Analytics: 7th International Workshop, MLSA 2020, Co-located with ECML/PKDD 2020 , Proceedings, Ghent, Belgium, September 14–18, 2020, Vol . 7 , 3–16 (Springer, 2020).

Araújo, D. & Davids, K. Team synergies in sport: theory and measures. Front. Psychol. 7 , 1449 (2016).

Article   PubMed   PubMed Central   Google Scholar  

Veličković, P. Everything is connected: graph neural networks. Curr. Opin. Struct. Biol. 79 , 102538 (2023).

Article   PubMed   Google Scholar  

Bronstein, M. M., Bruna, J., Cohen, T. & Veličković, P. Geometric deep learning: grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 (2021).

Brody, S., Alon, U. & Yahav, E. How attentive are graph attention networks? In International Conference on Learning Representations (ICLR, 2022). https://openreview.net/forum?id=F72ximsx7C1 .

Veličković, P. et al. Graph attention networks. In International Conference on Learning Representations (ICLR, 2018). https://openreview.net/forum?id=rJXMpikCZ .

Cohen, T. & Welling, M. Group equivariant convolutional networks. In International Conference on Machine Learning (eds Balcan, M. F. & Weinberger, K. Q.) 2990–2999 (PMLR, 2016).

Honda, Y., Kawakami, R., Yoshihashi, R., Kato, K. & Naemura, T. Pass receiver prediction in soccer using video and players’ trajectories. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 3502–3511 (2022). https://ieeexplore.ieee.org/document/9857310 .

Hubáček, O., Sourek, G. & Železný, F. Deep learning from spatial relations for soccer pass prediction. In MLSA@PKDD/ECML (eds Brefeld, U., Davis, J., Van Haaren, J. & Zimmermann, A.) Vol. 11330, (Lecture Notes in Computer Science, Springer, Cham, 2018).

Sanyal, S. Who will receive the ball? Predicting pass recipient in soccer videos. J Visual Commun. Image Represent. 78 , 103190 (2021).

Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O. & Dahl, G. E. Neural message passing for quantum chemistry. In International Conference on Machine Learning (Precup, D. & Teh, Y. W.) 1263–1272 (PMLR, 2017).

Veličković, P. et al. The CLRS algorithmic reasoning benchmark. In International Conference on Machine Learning (eds Chaudhuri, K. et al.) 22084–22102 (PMLR, 2022).

Vaswani, A. et al. Attention is all you need. In Advances in Neural Information Processing Systems (eds Guyon, I. et al.) Vol. 30 (Curran Associates, Inc., 2017).

Veličković, P. Message passing all the way up. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning (GTRL, 2022). https://openreview.net/forum?id=Bc8GiEZkTe5 .

Baranwal, A., Kimon, F. & Aukosh, J. Optimality of message-passing architectures for sparse graphs. In Thirty-seventh Conference on Neural Information Processing Systems (2023). https://papers.nips.cc/paper_files/paper/2023/hash/7e991aa4cd2fdf0014fba2f000f542d0-Abstract-Conference.html .

Greenblatt, R. D., Eastlake III, D. E. & Crocker, S. D. The Greenblatt chess program. In Proc. Fall Joint Computer Conference , 14–16 , 801–810 (Association for Computing Machinery, 1967). https://dl.acm.org/doi/10.1145/1465611.1465715 .

Schijf, M., Allis, L. V. & Uiterwijk, J. W. Proof-number search and transpositions. ICGA J. 17 , 63–74 (1994).

Fuchs, F., Worrall, D., Fischer, V. & Welling, M. SE(3)-transformers: 3D roto-translation equivariant attention networks. Adv. Neural Inf. Process. Syst. 33 , 1970–1981 (2020).

Google Scholar  

Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529 , 484–489 (2016).

Article   ADS   CAS   PubMed   Google Scholar  

Satorras, V. G., Hoogeboom, E. & Welling, M. E ( n ) equivariant graph neural networks. In International Conference on Machine Learning (eds Meila, M. & Zhang, T.) 9323–9332 (PMLR, 2021).

Cohen, T. S. & Welling, M. Steerable CNNs. In International Conference on Learning Representations (ICLR, 2017). https://openreview.net/forum?id=rJQKYt5ll .

Kingma, D. P. & Welling, M. Auto-encoding variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14–16, 2014, Conference Track Proceedings (ICLR, 2014). https://openreview.net/forum?id=33X9fd2-9FyZd .

Sohn, K., Lee, H. & Yan, X. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems (eds Cortes, C, Lawrence, N., Lee, D., Sugiyama, M. & Garnett, R.) Vol. 28 (Curran Associates, Inc., 2015).

Fernández, J. & Bornn, L. Soccermap: a deep learning architecture for visually-interpretable analysis in soccer. In Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14–18, 2020, Proceedings, Part V (eds Dong, Y., Ifrim, G., Mladenić, D., Saunders, C. & Van Hoecke, S.) 491–506 (Springer, 2021).

Zaheer, M. et al. Deep sets. In Advances in Neural Information Processing Systems Vol. 30 (eds Guyon, I., et al.) (Curran Associates, Inc., 2017).

Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O. & Dahl, G. E. Neural message passing for quantum chemistry. In Proc. 34th International Conference on Machine Learning , Vol. 70 of Proceedings of Machine Learning Research, 6–11 Aug 2017 (eds Precup, D. & Whye Teh, Y) 1263–1272 (PMLR, 2017).

Kingma, G. E. & Ba, J. Adam: a method for stochastic optimization. In ICLR (Poster) , (eds Bengio, Y. & LeCun, Y.) (International Conference of Learning Representations (ICLR), 2015). https://openreview.net/forum?id=8gmWwjFyLj .

Download references

Acknowledgements

We gratefully acknowledge the support of James French, Timothy Waskett, Hans Leitert and Benjamin Hervey for their extensive efforts in analysing TacticAI’s outputs. Further, we are thankful to Kevin McKee, Sherjil Ozair and Beatrice Bevilacqua for useful technical discussions, and Marc Lanctôt and Satinder Singh for reviewing the paper prior to submission.

Author information

These authors contributed equally: Zhe Wang, Petar Veličković, Daniel Hennes.

Authors and Affiliations

Google DeepMind, 6-8 Handyside Street, London, N1C 4UZ, UK

Zhe Wang, Petar Veličković, Daniel Hennes, Nenad Tomašev, Laurel Prince, Michael Kaisers, Yoram Bachrach, Romuald Elie, Li Kevin Wenliang, Federico Piccinini, Jerome Connor, Yi Yang, Adrià Recasens, Mina Khan, Nathalie Beauguerlange, Pablo Sprechmann, Pol Moreno, Nicolas Heess & Demis Hassabis

Liverpool FC, AXA Training Centre, Simonswood Lane, Kirkby, Liverpool, L33 5XB, UK

William Spearman

Liverpool FC, Kirkby, UK

University of Alberta, Amii, Edmonton, AB, T6G 2E8, Canada

Michael Bowling

Google DeepMind, London, UK

You can also search for this author in PubMed   Google Scholar

Contributions

Z.W., D. Hennes, L.P. and K.T. coordinated and organised the research effort leading to this paper. P.V. and Z.W. developed the core TacticAI models. Z.W., W.S. and I.G. prepared the Premier League corner kick dataset used for training and evaluating these models. P.V., Z.W., D. Hennes and N.T. designed the case study with human experts and Z.W. and P.V. performed the qualitative evaluation and statistical analysis of its outcomes. Z.W., P.V., D. Hennes, N.T., L.P., M. Kaisers, Y.B., R.E., L.K.W., F.P., W.S., I.G., N.H., M.B., D. Hassabis and K.T. contributed to writing the paper and providing feedback on the final manuscript. J.C., Y.Y., A.R., M. Khan, N.B., P.S. and P.M. contributed valuable technical and implementation discussions throughout the work’s development.

Corresponding authors

Correspondence to Zhe Wang , Petar Veličković or Karl Tuyls .

Ethics declarations

Competing interests.

The authors declare no competing interests but the following competing interests: TacticAI was developed during the course of the Authors’ employment at Google DeepMind and Liverpool Football Club, as applicable to each Author.

Peer review

Peer review information.

Nature Communications thanks Rui Luo and the other anonymous reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information, peer review file, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Wang, Z., Veličković, P., Hennes, D. et al. TacticAI: an AI assistant for football tactics. Nat Commun 15 , 1906 (2024). https://doi.org/10.1038/s41467-024-45965-x

Download citation

Received : 13 October 2023

Accepted : 08 February 2024

Published : 19 March 2024

DOI : https://doi.org/10.1038/s41467-024-45965-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Deep Learning-Based Recommender Systems Research Progress: A Bibliometric Analysis

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

8-hour time-restricted eating linked to a 91% higher risk of cardiovascular death

03/19/24 Editor’s note:  

• The research authors have shared their full poster presentation for updated details about their research abstract. Please see the digital file attached, under additional resources below, for these details.
• The most current statistics, reviewed and confirmed by the research authors, are in the poster (please see the digital file attached, under additional resources below) and the news release. 
• As with any new science development, patients should always consult with their doctor prior to making changes to their health regimens.

As noted in all American Heart Association scientific meetings news releases, research abstracts are considered preliminary until published in a peer-reviewed scientific journal.

Research Highlights :

• A study of over 20,000 adults found that those who followed an 8-hour time-restricted eating schedule, a type of intermittent fasting, had a 91% higher risk of death from cardiovascular disease.
• People with heart disease or cancer also had an increased risk of cardiovascular death.
• Compared with a standard schedule of eating across 12-16 hours per day, limiting food intake to less than 8 hours per day was not associated with living longer.

Embargoed until 3 p.m. CT/4 p.m. ET , Monday, March 18, 2024

CHICAGO, March 18, 2024 — An analysis of over 20,000 U.S. adults found that people who limited their eating across less than 8 hours per day, a time-restricted eating plan, were more likely to die from cardiovascular disease compared to people who ate across 12-16 hours per day, according to preliminary research presented at the American Heart Association’s  Epidemiology and Prevention│Lifestyle and Cardiometabolic Scientific Sessions 2024 , March 18- 21, in Chicago. The meeting offers the latest science on population-based health and wellness and implications for lifestyle.

Time-restricted eating, a type of intermittent fasting, involves limiting the hours for eating to a specific number of hours each day, which may range from a 4- to 12-hour time window in 24 hours. Many people who follow a time-restricted eating diet follow a 16:8 eating schedule, where they eat all their foods in an 8-hour window and fast for the remaining 16 hours each day, the researchers noted. Previous research has found that time-restricted eating improves several cardiometabolic health measures, such as blood pressure, blood glucose and cholesterol levels.

“Restricting daily eating time to a short period, such as 8 hours per day, has gained popularity in recent years as a way to lose weight and improve heart health,” said senior study author Victor Wenze Zhong, Ph.D., a professor and chair of the department of epidemiology and biostatistics at the Shanghai Jiao Tong University School of Medicine in Shanghai, China. “However, the long-term health effects of time-restricted eating, including risk of death from any cause or cardiovascular disease, are unknown.”

In this study, researchers investigated the potential long-term health impact of following an 8-hour time-restricted eating plan. They reviewed information about dietary patterns for participants in the annual 2003-2018 National Health and Nutrition Examination Surveys (NHANES) in comparison to data about people who died in the U.S., from 2003 through December 2019, from the Centers for Disease Control and Prevention’s National Death Index database.

The analysis found:

• People who followed a pattern of eating all of their food across less than 8 hours per day had a 91% higher risk of death due to cardiovascular disease.
• The increased risk of cardiovascular death was also seen in people living with heart disease or cancer.
• Among people with existing cardiovascular disease, an eating duration of no less than 8 but less than 10 hours per day was also associated with a 66% higher risk of death from heart disease or stroke.
• Time-restricted eating did not reduce the overall risk of death from any cause.
• An eating duration of more than 16 hours per day was associated with a lower risk of cancer mortality among people with cancer.

“We were surprised to find that people who followed an 8-hour, time-restricted eating schedule were more likely to die from cardiovascular disease. Even though this type of diet has been popular due to its potential short-term benefits, our research clearly shows that, compared with a typical eating time range of 12-16 hours per day, a shorter eating duration was not associated with living longer,” Zhong said.

“It’s crucial for patients, particularly those with existing heart conditions or cancer, to be aware of the association between an 8-hour eating window and increased risk of cardiovascular death. Our study’s findings encourage a more cautious, personalized approach to dietary recommendations, ensuring that they are aligned with an individual’s health status and the latest scientific evidence,” he continued. “Although the study identified an association between an 8-hour eating window and cardiovascular death, this does not mean that time-restricted eating caused cardiovascular death.” Study details and background:

• The study included approximately 20,000 adults in the U.S. with an average age of 49 years.
• Study participants were followed for a median length of 8 years and maximum length of 17 years.
• The study included data for NHANES participants who were at least 20 years old at enrollment, between 2003-2018, and had completed two 24-hour dietary recall questionnaires within the first year of enrollment.
• Approximately half of the participants self-identified as men, and half self-identified as women. 73.3% of the participants self-identified as non-Hispanic white adults, 11% self-identified as Hispanic adults, 8% self-identified as non-Hispanic Black adults and 6.9% of adults self-identified as another racial category, including mixed-race adults and adults of other non-Hispanic races.

The study’s limitations included its reliance on self-reported dietary information, which may be affected by participant’s memory or recall and may not accurately assess typical eating patterns. Factors that may also play a role in health, outside of daily duration of eating and cause of death, were not included in the analysis.

Future research may examine the biological mechanisms that underly the associations between a time-restricted eating schedule and adverse cardiovascular outcomes, and whether these findings are similar for people who live in other parts of the world, the authors noted.

“Overall, this study suggests that time-restricted eating may have short-term benefits but long-term adverse effects. When the study is presented in its entirety, it will be interesting and helpful to learn more of the details of the analysis,” said Christopher D. Gardner, Ph.D., FAHA, the Rehnborg Farquhar Professor of Medicine at Stanford University in Stanford, California, and chair of the writing committee for the Association’s 2023 scientific statement, Popular Dietary Patterns: Alignment with American Heart Association 2021 Dietary Guidance . 

“One of those details involves the nutrient quality of the diets typical of the different subsets of participants. Without this information, it cannot be determined if nutrient density might be an alternate explanation to the findings that currently focus on the window of time for eating. Second, it needs to be emphasized that categorization into the different windows of time-restricted eating was determined on the basis of just two days of dietary intake,” he said.

“It will also be critical to see a comparison of demographics and baseline characteristics across the groups that were classified into the different time-restricted eating windows – for example, was the group with the shortest time-restricted eating window unique compared to people who followed other eating schedules,  in terms of weight, stress, traditional cardiometabolic risk factors or other factors associated with adverse cardiovascular outcomes? This additional information will help to better understand the potential independent contribution of the short time-restricted eating pattern reported in this interesting and provocative abstract.”

Co-authors, their disclosures and funding sources are listed in the abstract.

Statements and conclusions of studies that are presented at the American Heart Association’s scientific meetings are solely those of the study authors and do not necessarily reflect the Association’s policy or position. The Association makes no representation or guarantee as to their accuracy or reliability. Abstracts presented at the Association’s scientific meetings are not peer-reviewed, rather, they are curated by independent review panels and are considered based on the potential to add to the diversity of scientific issues and views discussed at the meeting. The findings are considered preliminary until published as a full manuscript in a peer-reviewed scientific journal.

The Association receives funding primarily from individuals; foundations and corporations (including pharmaceutical, device manufacturers and other companies) also make donations and fund specific Association programs and events. The Association has strict policies to prevent these relationships from influencing the science content. Revenues from pharmaceutical and biotech companies, device manufacturers and health insurance providers and the Association’s overall financial information are here .

Additional Resources:

• Poster at EPI-Lifestyle 2024  (PDF): Association of 8-Hour Time-Restricted Eating with All-Cause and Cause-Specific Mortality
• Multimedia is available on the right column of the release link  https://newsroom.heart.org/news/8-hour-time-restricted-eating-linked-to-a-91-higher-risk-of-cardiovascular-death?preview=cdac59c3c907975eecaef517166f08f8
• After March 18, 2024, view  abstract P192  in the EPI│Lifestyle Scientific Sessions 2024  Online Program Planner .
• AHA news release:  Reducing total calories may be more effective for weight loss than intermittent fasting  (January 2023)
• AHA news release: 10 popular diets scored for heart-healthy elements; some need improvement (April 2023)
• AHA news release: New look at nutrition research identifies 10 features of a heart-healthy eating pattern (November 2021)
• AHA healthy eating tips: Eat Smart
• For more news from AHA EPI|Lifestyle Scientific Sessions 2024, follow us on X (formerlTwitter) @HyeartNews , #EPILifestyle24.​

The American Heart Association’s EPI|Lifestyle Scientific Sessions 2024 is the world’s premier meeting dedicated to the latest advances in population-based science. The 2024 meeting is in-person only, Monday through Thursday, March 18-21 at the Hilton Chicago. The primary goal of the meeting is to promote the development and application of translational and population science to prevent heart disease and stroke and foster cardiovascular health. The sessions focus on risk factors, obesity, nutrition, physical activity, genetics, metabolism, biomarkers, subclinical disease, clinical disease, healthy populations, global health and prevention-oriented clinical trials. The Councils on Epidemiology and Prevention and Lifestyle and Cardiometabolic Health (Lifestyle) jointly planned the EPI|Lifestyle Scientific Sessions 2024. Follow the conference on Twitter at #EPILifestyle24 .

About the American Heart Association

The American Heart Association is a relentless force for a world of longer, healthier lives. We are dedicated to ensuring equitable health in all communities. Through collaboration with numerous organizations, and powered by millions of volunteers, we fund innovative research, advocate for the public’s health and share lifesaving resources. The Dallas-based organization has been a leading source of health information for a century. During 2024 - our Centennial year - we celebrate our rich 100-year history and accomplishments. As we forge ahead into our second century of bold discovery and impact, our vision is to advance health and hope for everyone, everywhere. Connect with us on heart.org , Facebook , X or by calling 1-800-AHA-USA1.

For Media Inquiries and AHA Expert Perspective:

AHA Communications & Media Relations in Dallas: 214-706-1173; [email protected]

John Arnst: [email protected], 214-706-1060

For Public Inquiries: 1-800-AHA-USA1 (242-8721)

heart.org and stroke.org