research paper on number system

Help | Advanced Search

Computer Science > Neural and Evolutionary Computing

Title: number systems for deep neural network architectures: a survey.

Abstract: Deep neural networks (DNNs) have become an enabling component for a myriad of artificial intelligence applications. DNNs have shown sometimes superior performance, even compared to humans, in cases such as self-driving, health applications, etc. Because of their computational complexity, deploying DNNs in resource-constrained devices still faces many challenges related to computing complexity, energy efficiency, latency, and cost. To this end, several research directions are being pursued by both academia and industry to accelerate and efficiently implement DNNs. One important direction is determining the appropriate data representation for the massive amount of data involved in DNN processing. Using conventional number systems has been found to be sub-optimal for DNNs. Alternatively, a great body of research focuses on exploring suitable number systems. This article aims to provide a comprehensive survey and discussion about alternative number systems for more efficient representations of DNN data. Various number systems (conventional/unconventional) exploited for DNNs are discussed. The impact of these number systems on the performance and hardware design of DNNs is considered. In addition, this paper highlights the challenges associated with each number system and various solutions that are proposed for addressing them. The reader will be able to understand the importance of an efficient number system for DNN, learn about the widely used number systems for DNN, understand the trade-offs between various number systems, and consider various design aspects that affect the impact of number systems on DNN performance. In addition, the recent trends and related research opportunities will be highlighted

Submission history

Access paper:.

Download PDF
Other Formats

References & Citations

Google Scholar
Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

EDITORIAL article

Editorial: approximate number system and mathematics.

$\nJingguang Li$

1 College of Education, Dali University, Dali, China
2 State Key Laboratory of Cognitive Neuroscience and Learning, IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, China
3 Department of Psychology, Uppsala University, Uppsala, Sweden

Editorial on the Research Topic Approximate Number System and Mathematics

Humans process quantity information without the aid of language or symbols to guide a variety of everyday life decisions. The cognitive system that supports this intuitive skill is often referred to as the approximate number system (ANS). It has been argued that the ANS serves as the foundation of the formal symbolic number system—mathematics ( Dehaene, 1997 ). Abundant empirical evidence is supportive of this view: acuity of the ANS is positively correlated with symbolic math performance ( Chen and Li, 2014 ), training of the ANS may cause improvements in symbolic math performance ( Bugden et al., 2016 ), and the ANS and symbolic number processing may share a common neural underpinning ( Piazza et al., 2004 ). However, recently several theories and empirical data cast doubt on the role of the ANS in symbolic math processing ( Reynvoet and Sasanguie, 2016 ; Leibovich et al., 2017 ). This Research Topic aims to advance our understanding of the underlying mechanisms of the overlap between the ANS and mathematics.

The first portion of this Research Topic centers on the measurement issue of the ANS. Liu et al. demonstrated that regularity of visual features in the non-symbolic numerical task influenced processing of numerical information. For regular patterns of dot arrays, numerosity processing is inhibited; but for random patterns, numerosity information could be extracted independently of visual features. Thus, to measure ANS acuity, it is necessary to avoid regular dot patterns in the non-symbolic numerical task. van Hoogmoed and Kroesbergen suggested that convex hull, the smallest convex polygon that contains an array of dots, could be a plausible confounding factor in the non-symbolic numerical task. By using event-related potentials (ERP) from electroencephalography recordings, they found no signs of a distance effect for numerosity, but a distance effect for convex hull instead. Consequently, non-numerical visual features might at least partly influence performance in non-symbolic numerical tasks. Hence, it is unclear whether non-numerical visual processing or numerical processing in the non-symbolic numerical task contributes to the widely reported association between ANS acuity and math performance. Furthermore, their ERP data indicated that symbolic and non-symbolic numerosties where processed differentially, questioning if non-symbolic and symbolic numerosities share the same neural circuitry, as previously suggested (e.g., Dehaene, 1997 ). Braham et al. addressed this issue by using hierarchical linear modeling, which has the advantage of being able to isolate the numerical and non-numerical visual component in non-symbolic numerical task performance both within and between individuals. Critically, they found that only the numerical component contributed to adults' math ability. Finally, Guillaume and Van Rinsveld performed a meta-analysis regarding the variability of the Weber fraction in different versions of the non-symbolic number comparison paradigm. They found that different methods used for controlling for non-numerical information cause highly variable Weber fraction scores. Accordingly, they recommended not to compare Weber fraction scores from different tasks.

The second portion of this Research Topic focuses on the correlation between ANS acuity and math ability. Testing this correlation is the first step for further investigation of the causal relationship between ANS and math performance. Starr et al. suggested a new path underlying the association between ANS and math performance. They found that ANS manipulability (i.e., the ability to perform arithmetic operations on approximate numerical quantities) positively predicted math achievement in preschool children, and the predictive power of ANS manipulability was independent of the influence of ANS acuity. Wei et al. examined the relationship between number magnitude processing and symbolic approximate arithmetic performance (i.e., the ability to provide an approximate answer to an arithmetic question), which should arguably be largely uninfluenced by language. They found that both semantic and spatial number processing (indexed by the two-digit number comparison and number-line estimation task, respectively) are positively correlated to the symbolic approximate arithmetic performance, and these associations are moderated by the task difficulty of the symbolic approximate arithmetic task.

Two studies demonstrated that the correlation between ANS acuity and math performance is moderated by multiple factors. Cai et al. found that the correlation between ANS acuity and math performance varies across different grade levels (kindergarten vs. primary school), type of math tests, and type of ANS tests (non-symbolic estimation vs. number-line task). Using latent class modeling, Chew et al. identified four different magnitude ability profiles based on children's performance in the non-symbolic and symbolic numerical task. Further, they observed both stability and change in the four different profiles across a 1-year time period. Finally, profile membership was differentially related math performance at different ages.

Another two studies revealed that differences in math ability of different populations could be attributed to differences in ANS acuity. Lonnemann et al. found that Chinese children have better counting skills than their German peers. More importantly, the advantages in counting in Chinese children were accompanied by superior performance in a non-symbolic numerical comparison task. In addition, Oliveira et al. reported a case study on a girl with specific numerical processing impairment and a rare genetic disorder−22q11.2 deletion syndrome. The girl has normal general intelligence; however, she manifested severe deficits in single-digit calculation accompanied by poor performance in the non-symbolic numerical comparison task.

The third portion of this Research Topic examines whether training of the ANS leads to improvement in symbolic math performance. The training approach not only tests the causal relationship between ANS acuity and math performance, but also provides valuable insights for math education ( Bugden et al., 2016 ). Szkudlarek and Brannon found a transfer effect from ANS training to math performance. A group of preschool children trained for 1 month with a computer-based non-symbolic arithmetic training program. After controlling for confounding factors, children with low math abilities in the ANS-training group outperformed control-group children on informal symbolic math problems. In contrast, Kim et al. did not find a transfer effect in their training experiment with first-grade children. Although significant improvement in ANS acuity was observed following a 6-week training period, children showed no improvement in math performance. To resolve the discrepancies between the above two training studies, more replication studies with rigorous methodologies are needed ( Szucs and Myers, 2017 ).

The final portion of this Research Topic examines the distinction and mapping between the ANS and the symbolic numerical processing system by analyzing psychophysical features of different non-symbolic and symbolic numerical tasks. Krajcsi et al. made an extensive comparison of the several psychophysical properties of non-symbolic and symbolic number comparison, including error rates, reaction times, and diffusion-model drift rates. They found that the ratio-based ANS model only fits the non-symbolic number comparison data, but not the symbolic comparison data. Accordingly, the authors argued that different cognitive systems are in charge of symbolic and non-symbolic number processing. Chesney and Matthews found that different versions of non-symbolic numerosity tasks give rise to differences in performance. More specifically, while a free estimation task showed a classical pattern of scalar variability there was no evidence for this error pattern in a number-line and ratio estimation task. Furthermore, participants showed underestimation in the free estimation task but accurate estimation in the ratio task. They argued that these task constraints affect the ANS-math mapping process.

Taken together, this Research Topic combines diverse methodologies to advance our understanding of the relationship between the approximate number system and mathematics. According to the new data in this Research Topic, it might be too simple to conclude that the ANS and math are related or separated. Instead, it is worth asking how (i.e., the cognitive paths) and when (i.e., different developmental stages, task variants, and types of participants) the ANS is linked to math.

Author Contributions

JL wrote the first draft. JL, XZ, and ML contributed to the revision of the paper.

The organization of this Research Topic was supported by the National Natural Science Foundation of China (31500884) and the Innovation Team of Dali University (SKPY2019303) to JL and by a grant from The Swedish Foundation for Humanities and Social Sciences (P15-0430:1) to ML.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Bugden, S., DeWind, N. K., and Brannon, E. M. (2016). Using cognitive training studies to unravel the mechanisms by which the approximate number system supports symbolic math ability. Curr. Opin. Behav. Sci. 10, 73–80. doi: 10.1016/j.cobeha.2016.05.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Chen, Q., and Li, J. (2014). Association between individual differences in non-symbolic number acuity and math performance: a meta-analysis. Acta Psychol. 148, 163–172. doi: 10.1016/j.actpsy.2014.01.016

Dehaene, S. (1997). The Number Sense: How the Mind Creates Mathematics . New York, NY: Oxford University Press.

Google Scholar

Leibovich, T., Katzin, N., Harel, M., and Henik, A. (2017). From “sense of number” to “sense of magnitude”: the role of continuous magnitudes in numerical cognition. Behav. Brain Sci. 40:E164. doi: 10.1017/S0140525X16000960

CrossRef Full Text | Google Scholar

Piazza, M., Izard, V., Pinel, P., Le Bihan, D., and Dehaene, S. (2004). Tuning curves for approximate numerosity in the human intraparietal sulcus. Neuron 44, 547–555. doi: 10.1016/j.neuron.2004.10.014

Reynvoet, B., and Sasanguie, D. (2016). The symbol grounding problem revisited: a thorough evaluation of the ANS mapping account and the proposal of an alternative account based on symbol–symbol associations. Front. Psychol. 7:1581. doi: 10.3389/fpsyg.2016.01581

Szucs, D., and Myers, T. (2017). A critical analysis of design, facts, bias and inference in the approximate number system training literature: a systematic review. Trends Neurosci. Educ. 6, 187–203. doi: 10.1016/j.tine.2016.11.002

Keywords: approximate number system, number sense, non-symbolic number acuity, numerical cognition, mathematics

Citation: Li J, Zhou X and Lindskog M (2019) Editorial: Approximate Number System and Mathematics. Front. Psychol. 10:2084. doi: 10.3389/fpsyg.2019.02084

Received: 24 August 2019; Accepted: 27 August 2019; Published: 12 September 2019.

Edited and reviewed by: Bernhard Hommel , Leiden University, Netherlands

Copyright © 2019 Li, Zhou and Lindskog. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Jingguang Li, jingguang.li.k@gmail.com

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Automatic Number Plate Recognition

Ieee account.

Change Username/Password
Update Address

Purchase Details

Payment Options
Order History
View Purchased Documents

Profile Information

Communications Preferences
Profession and Education
Technical Interests
US & Canada: +1 800 678 4333
Worldwide: +1 732 981 0060
Contact & Support
About IEEE Xplore
Accessibility
Terms of Use
Nondiscrimination Policy
Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Numbered References

This section describes the numbered reference system and gives examples from one version of the system.* Ask your instructor what specific style to use.

Documentation styles for chemistry, computer science, mathematics, physics, and medical sciences are summarized in James D. Lester, Writing Research Papers, 5th ed., pp. 231-237, available in the Writing Center.

*This version is used, with slight variations, in many journals and in two books on scientific writing: Robert A. Day’s How to Write and Publish a Scientific Paper and F. Peter Woodford’s Scientific Writing for Graduate Students (both available in the Writing Center).

Create a reference list

List the works cited, with corresponding numbers, on a new page after the text, titled References . Although the sample list below is not arranged alphabetically, you should arrange your reference list in alphabetical order.

Check James D. Lester, Writing Research Papers, 5th ed., pp. 231-237 (available in the Writing Center) or the documentation manual for your field for arrangement and numbering of the list and for style and order of elements within each entry.

Use the table below for guidelines on how to format the entries in a numbered reference list.

Book (1): Single author

1. Zimmerman, B.K. 1984. Biofuture, confronting the genetic era. Plenum Press, New York.

Book (2): 2 authors

2. Nelkin, D., and M. Pollack. 1980. Problems and procedures in the regulation of technological risk. In R. Schwing and W. Albers (eds.), Societal risk management. p. 136-67. Plenum Press, New York.

Book (3): Editor in place of author

3. Milunsky, A., and G.J. Annas (eds.). 1976. Genetics and the law. Plenum Press, New York.

Book (4): 2nd or later edition

4. Burns, George W. 1980. The science of genetics: an introduction to heredity. 4th ed. Macmillan. New York.

Book (5): Volume in a multivolume work

5. Hotchkiss, R.D. 1980. Recombinant DNA research, vol. 5. NIH Publication No. 80-2130.

Book (6): Subtitled volume of multivolume work

6. Office of Technology Assessment, U.S. Congress. 1980. Energy for biological processes, vol. 2. Technical analysis. U.S. Government Printing Office, Washington, D.C.

Book (7): Article in edited collection

7. Nelkin, D., and M. Pollack. 1980. Problems and procedures in the regulation of technological risk. In R. Schwing and W. Albers (eds.), Societal risk management. p. 136-67. Plenum Press, New York.

Book (8): Corporate author

8. National Research Council. 1977. World food and nutrition study: the potential contributions of research. Author, Washington, D.C.

9. Office of Technology Assessment, U.S. Congress. 1980. Energy for biological processes, vol. 2. Technical analysis. U.S. Government Printing Office, Washington, D.C.

Journal Article (1): Single author

10. Myers, N. 1979. Conserving our global stock. Environment 21: 25-30.

Journal Article (2): More than 1 author

11. Israel, M.A., et al. 1979. Molecular cloning of polyoma virus DNA in escherichia coli: plasmid vector systems. Science 203: 883-887.

Journal Article (3): In edited collection

12. Nelkin, D., and M. Pollack. 1980. Problems and procedures in the regulation of technological risk. In R. Schwing and W. Albers (eds.), Societal risk management. p. 136-67. Plenum Press, New York.

Create in-text citations

When using numbered references, cite a source by using the number assigned to that source in the reference list.

Use the information below for guidelines on how to cite numbered references correctly in your text.

Number your citations

Depending upon the system used in your field, either:

Arrange the sources you cite alphabetically and then number them; or
Number the citations consecutively according to the first mention of each source in the text (using the same number for subsequent references to the same source).

Format your citations

Place the number in parentheses or in square brackets; or
Use a superscript (a number above the text line, as for a footnote).

Include a page number

Add a comma and the page number(s) of the source.

The method was described in 1979 (2, p. 885).

[The citation indicates that the method was described on page 885 of reference number 2 (Israel et al.) on the reference list in the Writing Center handout about numbered references entitled “The Reference List.” Notice that the period for the sentence comes after the closing parenthesis.]

Make the citation part of your sentence

Place the number directly after the author’s name or mention of the work :

The work of Nelkin and Pollack (6) supports this theory.

A 1979 study (4) showed. . . .

You can refer to a number of works within one pair of parentheses or brackets or in a series of superscript numbers:

Numerous studies (1, 3, 4, 8, 9) refer to . . .

Cite References in Your Paper

FRONT MATTER
TABLE OF CONTENTS

Number System: In-Text Citation

Generally, the number system is favored in fields where you typically report experimental field or lab work. Technical fields such as materials science, aerospace engineering, and chemistry tend to favor the number system.

When you use the number system, your responsibility is to indicate in your text—either in parentheses or brackets—a number that corresponds to a source on your references page. The first source you cite in your text receives the number 1, the second number 2, and so on. If you repeat a reference to a source later in the text, it retains its original number—thus, all references to source number 4 receive a 4 after them in parentheses or brackets. You delay the appropriate punctuation until after the parentheses or brackets:

If the load on the thrust bearing can be decreased by some means, the life of the turbodrill can be significantly increased (1).

Many authors prefer to identify the source at the beginning of the reference, perhaps including the author’s name directly in the text:

Gould et al. (5) found a clear relation between. . .

The number system is especially handy for citing equations, because you can simply insert the citation number logically as you introduce the equation to avoid confusion with any other numbers:

The line’s slope is used in the following equation (7) to calculate. . .

Other In-Text Citation Practices

Slight but important mechanical differences exist among in-text citation practices, in particular when you are trying to conform to a specific style, such as MLA (Modern Language Association) or APA (American Psychological Association). For example, MLA style requires you to provide the page number of your citation in-text, but not the year, while APA style asks you to place a comma between author and year. Please feel welcome to explore all of these nuances for yourself if you wish, and recognize that some professors will insist that you conform to a particular style. When professors do not dictate a particular style, they will usually simply expect you to use the author-year or number system with consistency throughout the paper.

Remember, too, that journals within you field have already made informed decisions about which in-text citation practices they use. To settle on citation particulars, many writers model a journal in their field—mandatory, of course, if you submit material to a journal hoping for publication.

Read up on the specifics of various citation styles, in particular MLA and APA, at the following pages:

"Research and Citation Resources" article from Purdue's Online Writing Lab (OWL)

"Citation Style for Research Papers" article from Long Island University

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

We're Hiring!
Help Center

Binary Number System

In today's world, computer plays a very significant role. It comes in different sizes, shapes and applications and had made our life simpler. The language used by the computers is in the form of binary numbers that is in 0 and 1 form .It is the lowest level that helps the machine to read. Computer usually works in binary but gives answer in decimals and that helps it to save the space. This is important as it simplifies the design of computer and related technologies. That's why it is considered as the perfect numbering system for computer. It is also considered easy and there is no comparison how much easier binary is than decimal. In this, we only need 2 digits, o and 1 while in decimal we need 10 digits that made the process much harder. It is a method of storing simple numbers such as 35 and 380 as pattern of 0's and 1's. Due to its digital nature, computers electronic can easily manipulate numbers stored in binary by treating as "on "and "off." Computers are having circuits that perform the arithmetical operations such as add, subtract, multiply, divide, and do many other things to numbers stored in binary.

Related Papers

MAILE MACHETHE

Giuliano Donzellini

The representation of numbers is essential for the digital logic design. In this chapter, positional number systems (decimal, binary, octal, hexadecimal), BCD and Gray codes are presented together with the rules for the conversion between numbers encoded in different bases and the representations of negative numbers. Then, the rules for the arithmetic operations and the circuits that execute them are presented. The addition of binary number is examined with particular attention, since it is the operation at the basis of all computational circuits. Alphanumeric codes and the concept of parity for error detection complete the chapter.

Aditya Garg

Journal of Education and Practice

Nsikak Udofia

chaganti pavan

The Mathematics Enthusiast

Daniel Lande

Alfie Parkin

After looking into coding permutations of something, I fell down the rabbit hole of Binary numbers and found an interesting relationship involving the number of 1's. In this article, I look at Binary numbers and the relationship between them and Pascal's Triangle. Through looking at graphs and the use of algebra, I write a formula for calculating the number of 1's in any Binary number using complex numbers and logarithms. All diagrams and plots are my own.

Mugunthan V

Vaibhav Katiyar

Evolution Of Number Systems Research Paper

Sample Evolution Of Number Systems Research Paper. Browse other research paper examples and check the list of research paper topics for more inspiration. If you need a religion research paper written according to all the academic standards, you can always turn to our experienced writers for help. This is how your paper can get an A! Feel free to contact our research paper writing service for professional assistance. We offer high-quality assignments for reasonable rates.

1. The Term ‘Number System’

A ‘number system’ consists of a set of mental entities and operations together with systems of symbols such as number words, tallies, or numerical sign systems and of symbolic activities such as counting, addition, or multiplication representing them. Number systems evolved in the course of the development of most civilizations. This evolution changed not only numbers and number representations but also the meaning of the term ‘number’ itself.

Academic Writing, Editing, Proofreading, And Problem Solving Services

Get 10% off with 24start discount code.

Symbolic activities such as counting, the use of tallies for representing quantities, or performing calculations by means of tokens or systems of written symbols, are usually considered practices dealing with ‘numbers.’ However, such symbolic activities historically precede an explicit number concept. Explicit concepts of number are the outgrowth of reﬂection upon such activities.

When ﬁrst explicit concepts of abstract ‘numbers’ emerged in history, they were restricted to the number system which is called today ‘natural numbers,’ that is, to the system ‘1, 2, 3, 4, …’ The subsequent history of the term number is characterized by a series of extensions of the number concept to cover ‘integers,’ ‘rational numbers,’ ‘real numbers,’ ‘complex numbers,’ ‘algebraic numbers,’ and other mathematical objects designated, with lesser or greater justiﬁcation, as a number system. These extensions usually included numbers of earlier stages as special kinds of numbers, but necessarily also dismissed certain properties which seemed to be essential in earlier stages. Consequently, any extension of number systems was accompanied by controversial discussions about the question whether the newly-considered mathematical objects were really numbers. These doubts are often reﬂected in the designation of such numbers as ‘irrational,’ ‘negative,’ ‘imaginary,’ etc. Finally, the variety of potential candidates for the term number became so great that the answer to the question of what is and what is not a ‘number’ turned into a merely conventional decision. The focus of attention shifted from the investigation of types of numbers as they were historically determined towards the study of their common structures.

2. Cultures Without Number Systems

Until recently, ‘prearithmetical’ cultures existed without any arithmetical techniques at all such as ﬁnger counting, counting notches, counting knots, or tokens. The best known examples are the Australian aborigines (Blake 1981, Dixon 1980) and certain South American natives (Closs 1986) whose languages did possess terms for quantities, yet of an exclusively qualitative nature, terms such as ‘many,’ ‘high,’ ‘big,’ or ‘wide.’ The quantitative aspects of an object of cognition were not yet distinguished from its speciﬁc physical appearance. Since from periods before the Late Neolithic no objects or signs have been identiﬁed that might have had some kind of arithmetical function, it is likely that up to that time human culture remained entirely prearithmetical. While artifacts with repeated signs and regular patterns from periods as early as the Late Paleolithic have occasionally been interpreted as representations of numbers (Marshack 1972), such an interpretation seems to be untenable since these sign repetitions lack the characteristic subdivision by counting levels which is present in all real counting systems (Damerow 1998).

3. Proto-Arithmetic

The occurrence of the simplest genuine arithmetical activities known from recent nonliterate cultures (see, e.g., Saxe 1982) date back to the Late Neolithic and the Early Bronze Age. These activities aiming at the identiﬁcation and control of quantities are based on structured and standardized systems of symbols representing objects. Their emergence as counting and tallying techniques may have been a consequence of sedentariness. Symbols are the most simple tools for the construction of one-to-one correspondences in counting and tallying that can be transmitted from generation to generation. The organization of agricultural cultivation, animal husbandry and household administration led to social conditions that apparently made symbolic techniques useful, and their systematic transmission and development possible. Such techniques are ‘proto-arithmetical’ insofar as the symbols represent objects and not ‘numbers,’ and consequently are not used for symbolic transformations which correspond to such arithmetical operations as addition and multiplication.

Early explorers and travelers encountering indigenous cultures using proto-arithmetical techniques often anachronistically interpreted their activities from a modern numerical perspective and believed the limitations of proto-arithmetic resulted from deﬁcient mental abilities of such peoples. It was only in the ﬁrst half of the twentieth century that anthropologists and psychologists challenged these beliefs (e.g., see Klein, Melanie (1882–1960); Boas, Franz (1858–1942)) and began seriously to study culturally speciﬁc mental constructions connected with proto-arithmetical activities (Levy-Bruhl 1923, Wertheimer 1925).

4. From Proto-Arithmetic To Systems Of Numerical Symbols

A further step in the evolution of number systems resulted from the rise of early civilizations and the invention of writing. This transition is particularly well documented in the case of the Ancient Near East. A system of clay tokens possessing proto-arithmetical functions has been identiﬁed which may have been widely used already during the period from the beginning of sedentariness in the areas surrounding the Mesopotamian lowland plain and in the Nile valley around 8000 BC until the emergence of cities around 4000 BC (Schmandt-Besserat 1992).

In the fourth millennium BC this proto-arithmetic system of clay tokens became the central tool for the control of a locally-centralized economic administration. In order to serve this purpose, the protoarithmetical capabilities of the system were exploited to the limits of their capacity. The safekeeping of tokens in closed and sealed spherical clay envelopes demonstrates that they were used for the encoding of important information. The tokens were complemented and later completely replaced by markings that were impressed on the surface of such clay envelopes or on the surface of sealed clay tablets. Thus, the system was transformed into a more suitable medium by substituting written signs for tokens. Around 3200 BC, the numerical notations were supplemented with pictograms, and the tablets achieved a more complex structure. They now displayed several quantitative notations arranged according to their function, as we ﬁnd in an administrative form. The earliest attestation of addition as an operation with symbols is provided by the notation of totals of the entries on such tablets, usually inscribed on the reverse of the tablet containing these entries.

Based on their origin, the oldest written numerical notation systems exhibited, for a short period, very unusual characteristics. Since they initially still represented units of counting and measurements, and since the numerical relations between these units were dependent on the counted or measured objects, the numerical signs had no uniquely determined values. Their values resulted from their metrological context, and changed with the respective areas of application without any apparent attempt to attain unambiguous numerical values for the signs. From the viewpoint of modern arithmetic, the signs thus represented in different contexts different numbers (Damerow 1995, Damerow and Englund 1987). Some 500 years later, however, the system had developed into a sort of numerical sign system as is known from other early civilizations. In particular, the signs had now attained ﬁxed numerical values.

5. Systems Of Numerical Symbols In Early Civilizations

Most, if not all, advanced civilizations, in particular the Egyptian empire, the Mesopotamian city-states, the Mediterranean cultures, the Chinese empire, the Central American cultures, and the Inca culture, have, independent from each other or by adaptation from other cultures, developed systems of numerical symbols that exhibit similar semiotic characteristics. The basic symbols of these systems were the same as at the proto-arithmetical level, signs for units and not for numbers, but since they represented now complex metrologies they had necessarily also to deal with fractions. Furthermore, part of the systems were now complex symbol transformations, that is, arithmetical techniques such as Egyptian calculation using unit fractions (Chace 1927), sexagesimal arithmetical techniques of the Babylonians (Neugebauer 1934), transformations of rod numerals on the Chinese counting board (Li and Du 1987), calendrical calculations in the pre-Columbian culture of the Maya (Closs 1986, Thompson 1960), or techniques of the use of knotted cords (quipu) as administrative tools by the Inca (Ascher and Ascher 1971–1972, Locke 1923).

The development of these numerical techniques was historically closely related with the administrative problems that had arisen through the concentration of economic goods and services in the governmental centers of early state organization. The dramatic rise in the quantities of products had to be controlled, and the immense variety of decision-making implications had to be executed administratively (Høyrup 1994, Nissen et al. 1993).

The rules dictating the use of numerical signs reﬂected this function; they corresponded to their meaning in the social context. The numerical sign systems thus exhibited a great variety of structures with no internal differentiation between rules representing a universal number concept and rules depending on the speciﬁc kind of symbolic representation and its function to control quantities in a speciﬁc social setting. It was only the cultural exchange between advanced civilizations with developed systems of numerical notations that created the preconditions for the differentiation between universal numbers and speciﬁc notations.

6. The Euclidean Conceptualization Of Number

The emergence of an abstract concept of number as a consequence of cultural exchange is particularly well known from the ancient Greek culture which played a crucial role in the historical process of transforming the heritage of early Near Eastern civilizations into the intellectual achievements of the hegemonic cultures of the Hellenistic and Roman world. The reﬂective restructuring of divergent bodies of knowledge brought about new kinds of general concepts and, in particular, an abstract concept of number.

The oldest known examples of general propositions about abstract numbers, for instance, the statement that the number of prime numbers is inﬁnite, are handed down to us through the deﬁnitions and theorems of Euclid’s Elements (Heath 1956). The relevance of these deﬁnitions and theorems was no longer based on their practical applicability, but only on their role within a closed system of mental operations. These operations reﬂected certain arithmetical activities in the medium of written language. In the case of the number concept these operations were the techniques of counting and tallying. Euclid deﬁned a number as a ‘multitude composed of units’ and hence restricted the abstract concept of number to natural numbers. Platonism, in particular, dogmatically excluded from theoretical arithmetic all numerical structures that did not come under the Euclidian deﬁnition. Fractions of a number, for instance, and even more so irrational numbers such as the square roots of natural numbers that are not squares were not accepted themselves as numbers.

In order to circumvent the problems of incommensurability, a theory of proportions was developed, not for numbers, but for entities designated as ‘magnitudes’ (Heath 1956). This theory served at the same time as a substitute for an expanded concept of number that could cover all the arithmetical activities and symbolic operations existing at that time.

At a practical level, however, operations with notations for other types of number systems had been developed and were used long before they were theoretically reﬂected in an extended number concept. Any exchange of goods by merchants or by administrators of a centralized economy implicitly involves fractions. The balancing of debits and credits in bookkeeping similarly involves negative numbers. Accordingly, arithmetical techniques for dealing implicitly or even explicitly with fractions or with negative and irrational numbers had thus been developed already in early civilizations long before they were reﬂected in an explicit number concept.

7. The Impact Of Algebra

The problems resulting from the restrictive Greek number concept were aggravated with the renaissance of ancient mathematics in the Early Modern Era. In particular, the development of algebraic notation techniques and of analytical methods applied to continuous processes in mechanics made the distinction between numbers and magnitudes obsolete. The use of variables in order to solve equations was indifferent to this conceptual distinction. Fractions, negative numbers, irrational numbers, inﬁnitesimals, and inﬁnity became more and more accepted as numbers. Although inﬁnitesimals were again expelled in the nineteenth century after their missing logical foundation resulted in serious contradictions in the conceptual system of mathematics, they were ﬁnally reintroduced in a new form by the nonstandard analysis of the twentieth century.

The introduction of algebraic notations thus contributed to the transformation of existing arithmetical techniques into conceptualized number systems, but it also initiated the creation of completely new ones. A new type of number system with no immediately obvious meaning emerged, reﬂecting no longer primary arithmetical techniques but rather abstract operations with variables and equations. A well-known example is provided by ‘imaginary’ square roots of negative numbers which occurred as solutions to higher degree equations. Such solutions, although calculated correctly according to rules for numbers, seemed to make no sense since squaring a number never results in a negative number. Nevertheless, it turned out to be possible to operate consistently with such imaginary numbers as if indeed they were, in fact, numbers. They were, thus, ﬁnally accepted as ‘complex numbers.’

The justiﬁcation of seemingly meaningless number systems was not the only problem posed by the development of mathematics in the nineteenth century. The term number became questionable for other reasons too. On the one hand, a productive new theory, even today called simply ‘theory of numbers,’ was created; it is, however, a discipline with a speciﬁc topic. The term number is in this theory restricted to (positive and negative) integers. On the other hand, a growing number of new types of mathematical systems emerged which, with greater or lesser emphasis, were called number systems, for example, ‘algebraic numbers,’ ‘transcendental numbers,’ ‘higher order complex numbers,’ ‘quaternions,’ ‘biquaternions,’ etc. In order to organize the hodgepodge of artiﬁcially-constructed number systems, a heuristic principle was formulated. The construction of number systems should follow the ‘principle of the permanence of formal laws’ of an arithmetica universalis (Hankel 1867), that is, they should be constructed in such a way that some of the rules of ordinary numbers are preserved. It was hoped, in particular, that ‘higher order complex numbers’ of a certain type might serve as the foundation for an inﬁnite number and virtually all possible number systems, an expectation which later turned out to be misguided.

8. Formalism And The Decline Of Number

The formalistic approach, which in the nineteenth century led to the ﬂourishing number systems and ascribed to them a key role in the foundation of mathematics, reduced their importance in the course of the structuralist rebuilding of mathematics in the twentieth century. In fact, the recombination of structures and operations of mathematical objects in order to create new ones with new properties made it more and more difficult to distinguish number systems from other systems of mathematical objects. The designation of an object as a number was increasingly seen as merely an historical convention. Number systems were downgraded to instances of structures abstracted from them, in particular, of algebraic structures resembling addition and multiplication and topological structures determining distance and continuity. Not numbers, but ‘sets’ equipped with such structures, which were designated artiﬁcially as ‘groups,’ ‘ﬁelds,’ ‘rings,’ ‘lattices,’ ‘compact spaces,’ etc., became the building blocks of modern mathematics.

Bibliography:

Ascher M, Ascher R 1971–1972 Numbers and relations from ancient Andean Quipus. Archive for History of Exact Sciences 8: 288–320
Blake B J 1981 Australian Aboriginal Languages. Angus and Robertson, London
Chace A B 1927 The Rhind Mathematical Papyrus. Mathematical Association of America, Oberlin, OH
Closs M P (ed.) 1986 Native American Mathematics. University of Texas Press, Austin, TX
Damerow P 1995 Abstraction and Representation: Essays on the Cultural Evolution of Thinking. Kluwer Academic Publishers, Dordrecht, The Netherlands
Damerow P 1998 Prehistory and cognitive development. In: Langer J, Killen M (eds.) Piaget, Evolution, and Development. Erlbaum, Mahwah, NJ
Damerow P, Englund R K 1987 Die Zahlzeichensysteme der Archaischen Texte aus Uruk. In: Green M W, Nissen H J (eds.) Zeichenliste der Archaischen Texte aus Uruk. Gebr. Mann, Berlin
Dixon R M W 1980 The Languages of Australia. Cambridge University Press, Cambridge, UK
Hankel H 1867 Theorie der complexen Zahlensysteme insbesondere der gemeinen imaginaren Zahlen und der Hamilton’schen Quaternionen nebst ihrer geometrischen Darstellung. Voss, Leipzig
Heath S T L (ed.) 1956 The Thirteen Books of Euclid’s Elements. Dover, New York
Høyrup J 1994 In Measure, Number, and Weight: Studies in Mathematics and Culture. State University of New York Press, Albany, NY
Levy-Bruhl L 1923 Primitive Mentality. Macmillan, New York
Li Y, Du S 1987 Chinese Mathematics: A Concise History. Clarendon, Oxford
Locke L L 1923 The Ancient Quipu or Peruvian Knot Record. The American Museum of Natural History, New York
Marshack A 1972 The Roots of Civilization: The Cognitive Beginnings of Man’s First Art, Symbol and Notation. Weidenfeld and Nicolson, London
Neugebauer O 1934 Vorgriechische Mathematik. Springer, Berlin
Nissen H J, Damerow P, Englund R K 1993 Archaic Bookkeeping: Early Writing and Techniques of Economic Administration in the Ancient Near East. Chicago University Press, Chicago, IL
Saxe G B 1982 Culture and the development of numerical cognition: Studies among the Oksapmin of Papua New Guinea. In: Brainerd C J (ed.) The Development of Logical and Mathematical Cognition. Springer, New York
Schmandt-Besserat D 1992 Before Writing. University of Texas Press, Austin, TX
Thompson J E S 1960 Maya Hieroglyphic Writing. University of Oklahoma Press, Norman, OK
Wertheimer M 1925 Uber das Denken der Natur olker: Zahlen und Zahlgebilde. In: Wertheimer, M (ed.) Drei Abhandlungen zur Gestalttheorie. Palm and Enke, Erlangen, Germany

ORDER HIGH QUALITY CUSTOM PAPER

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

View all journals
My Account Login
Explore content
About the journal
Publish with us
Sign up for alerts
Open access
Published: 19 March 2024

TacticAI: an AI assistant for football tactics

Zhe Wang ORCID: orcid.org/0000-0002-0748-5376 1 na1 ,
Petar Veličković ORCID: orcid.org/0000-0002-2820-4692 1 na1 ,
Daniel Hennes ORCID: orcid.org/0000-0002-3646-5286 1 na1 ,
Nenad Tomašev ORCID: orcid.org/0000-0003-1624-0220 1 ,
Laurel Prince 1 ,
Michael Kaisers 1 ,
Yoram Bachrach 1 ,
Romuald Elie 1 ,
Li Kevin Wenliang 1 ,
Federico Piccinini 1 ,
William Spearman 2 ,
Ian Graham 3 ,
Jerome Connor 1 ,
Yi Yang 1 ,
Adrià Recasens 1 ,
Mina Khan 1 ,
Nathalie Beauguerlange 1 ,
Pablo Sprechmann 1 ,
Pol Moreno 1 ,
Nicolas Heess ORCID: orcid.org/0000-0001-7876-9256 1 ,
Michael Bowling ORCID: orcid.org/0000-0003-2960-8418 4 ,
Demis Hassabis 1 &
Karl Tuyls ORCID: orcid.org/0000-0001-7929-1944 5

Nature Communications volume 15 , Article number: 1906 ( 2024 ) Cite this article

134 Altmetric

Metrics details

Computational science
Information technology

Identifying key patterns of tactics implemented by rival teams, and developing effective responses, lies at the heart of modern football. However, doing so algorithmically remains an open research challenge. To address this unmet need, we propose TacticAI, an AI football tactics assistant developed and evaluated in close collaboration with domain experts from Liverpool FC. We focus on analysing corner kicks, as they offer coaches the most direct opportunities for interventions and improvements. TacticAI incorporates both a predictive and a generative component, allowing the coaches to effectively sample and explore alternative player setups for each corner kick routine and to select those with the highest predicted likelihood of success. We validate TacticAI on a number of relevant benchmark tasks: predicting receivers and shot attempts and recommending player position adjustments. The utility of TacticAI is validated by a qualitative study conducted with football domain experts at Liverpool FC. We show that TacticAI’s model suggestions are not only indistinguishable from real tactics, but also favoured over existing tactics 90% of the time, and that TacticAI offers an effective corner kick retrieval system. TacticAI achieves these results despite the limited availability of gold-standard data, achieving data efficiency through geometric deep learning.

Automated recognition of the cricket batting backlift technique in video footage using deep learning architectures

Tevin Moodley, Dustin van der Haar & Habib Noorbhai

Synchronization of passes in event and spatiotemporal soccer data

Henrik Biermann, Rumena Komitova, … Daniel Memmert

A Random Forest approach to identify metrics that best predict match outcome and player ranking in the esport Rocket League

Tim D. Smithies, Mark J. Campbell, … Adam J. Toth

Introduction

Association football, or simply football or soccer, is a widely popular and highly professionalised sport, in which two teams compete to score goals against each other. As each football team comprises up to 11 active players at all times and takes place on a very large pitch (also known as a soccer field), scoring goals tends to require a significant degree of strategic team-play. Under the rules codified in the Laws of the Game 1 , this competition has nurtured an evolution of nuanced strategies and tactics, culminating in modern professional football leagues. In today’s play, data-driven insights are a key driver in determining the optimal player setups for each game and developing counter-tactics to maximise the chances of success 2 .

When competing at the highest level the margins are incredibly tight, and it is increasingly important to be able to capitalise on any opportunity for creating an advantage on the pitch. To that end, top-tier clubs employ diverse teams of coaches, analysts and experts, tasked with studying and devising (counter-)tactics before each game. Several recent methods attempt to improve tactical coaching and player decision-making through artificial intelligence (AI) tools, using a wide variety of data types from videos to tracking sensors and applying diverse algorithms ranging from simple logistic regression to elaborate neural network architectures. Such methods have been employed to help predict shot events from videos 3 , forecast off-screen movement from spatio-temporal data 4 , determine whether a match is in-play or interrupted 5 , or identify player actions 6 .

The execution of agreed-upon plans by players on the pitch is highly dynamic and imperfect, depending on numerous factors including player fitness and fatigue, variations in player movement and positioning, weather, the state of the pitch, and the reaction of the opposing team. In contrast, set pieces provide an opportunity to exert more control on the outcome, as the brief interruption in play allows the players to reposition according to one of the practiced and pre-agreed patterns, and make a deliberate attempt towards the goal. Examples of such set pieces include free kicks, corner kicks, goal kicks, throw-ins, and penalties 2 .

Among set pieces, corner kicks are of particular importance, as an improvement in corner kick execution may substantially modify game outcomes, and they lend themselves to principled, tactical and detailed analysis. This is because corner kicks tend to occur frequently in football matches (with ~10 corners on average taking place in each match 7 ), they are taken from a fixed, rigid position, and they offer an immediate opportunity for scoring a goal—no other set piece simultaneously satisfies all of the above. In practice, corner kick routines are determined well ahead of each match, taking into account the strengths and weaknesses of the opposing team and their typical tactical deployment. It is for this reason that we focus on corner kick analysis in particular, and propose TacticAI, an AI football assistant for supporting the human expert with set piece analysis, and the development and improvement of corner kick routines.

TacticAI is rooted in learning efficient representations of corner kick tactics from raw, spatio-temporal player tracking data. It makes efficient use of this data by representing each corner kick situation as a graph—a natural representation for modelling relationships between players (Fig. 1 A, Table 2 ), and these player relationships may be of higher importance than the absolute distances between them on the pitch 8 . Such a graph input is a natural candidate for graph machine learning models 9 , which we employ within TacticAI to obtain high-dimensional latent player representations. In the Supplementary Discussion section, we carefully contrast TacticAI against prior art in the area.

A How corner kick situations are converted to a graph representation. Each player is treated as a node in a graph, with node, edge and graph features extracted as detailed in the main text. Then, a graph neural network operates over this graph by performing message passing; each node’s representation is updated using the messages sent to it from its neighbouring nodes. B How TacticAI processes a given corner kick. To ensure that TacticAI’s answers are robust in the face of horizontal or vertical reflections, all possible combinations of reflections are applied to the input corner, and these four views are then fed to the core TacticAI model, where they are able to interact with each other to compute the final player representations—each internal blue arrow corresponds to a single message passing layer from ( A ). Once player representations are computed, they can be used to predict the corner’s receiver, whether a shot has been taken, as well as assistive adjustments to player positions and velocities, which increase or decrease the probability of a shot being taken.

Uniquely, TacticAI takes advantage of geometric deep learning 10 to explicitly produce player representations that respect several symmetries of the football pitch (Fig. 1 B). As an illustrative example, we can usually safely assume that under a horizontal or vertical reflection of the pitch state, the game situation is equivalent. Geometric deep learning ensures that TacticAI’s player representations will be identically computed under such reflections, such that this symmetry does not have to be learnt from data. This proves to be a valuable addition, as high-quality tracking data is often limited—with only a few hundred matches played each year in every league. We provide an in-depth overview of how we employ geometric deep learning in TacticAI in the “Methods” section.

From these representations, TacticAI is then able to answer various predictive questions about the outcomes of a corner—for example, which player is most likely to make first contact with the ball, or whether a shot will take place. TacticAI can also be used as a retrieval system—for mining similar corner kick situations based on the similarity of player representations—and a generative recommendation system, suggesting adjustments to player positions and velocities to maximise or minimise the estimated shot probability. Through several experiments within a case study with domain expert coaches and analysts from Liverpool FC, the results of which we present in the next section, we obtain clear statistical evidence that TacticAI readily provides useful, realistic and accurate tactical suggestions.

To demonstrate the diverse qualities of our approach, we design TacticAI with three distinct predictive and generative components: receiver prediction, shot prediction, and tactic recommendation through guided generation, which also correspond to the benchmark tasks for quantitatively evaluating TacticAI. In addition to providing accurate quantitative insights for corner kick analysis with its predictive components, the interplay between TacticAI’s predictive and generative components allows coaches to sample alternative player setups for each routine of interest, and directly evaluate the possible outcomes of such alternatives.

We will first describe our quantitative analysis, which demonstrates that TacticAI’s predictive components are accurate at predicting corner kick receivers and shot situations on held-out test corners and that the proposed player adjustments do not strongly deviate from ground-truth situations. However, such an analysis only gives an indirect insight into how useful TacticAI would be once deployed. We tackle this question of utility head-on and conduct a comprehensive case study in collaboration with our partners at Liverpool FC—where we directly ask human expert raters to judge the utility of TacticAI’s predictions and player adjustments. The following sections expand on the specific results and analysis we have performed.

In what follows, we will describe TacticAI’s components at a minimal level necessary to understand our evaluation. We defer detailed descriptions of TacticAI’s components to the “Methods” section. Note that, all our error bars reported in this research are standard deviations.

Benchmarking TacticAI

We evaluate the three components of TacticAI on a relevant benchmark dataset of corner kicks. Our dataset consists of 7176 corner kicks from the 2020 to 2021 Premier League seasons, which we randomly shuffle and split into a training (80%) and a test set (20%). As previously mentioned, TacticAI operates on graphs. Accordingly, we represent each corner kick situation as a graph, where each node corresponds to a player. The features associated with each node encode the movements (velocities and positions) and simple profiles (heights and weights) of on-pitch players at the timestamp when the corresponding corner kick was being taken by the attacking kicker (see the “Methods” section), and no information of ball movement was encoded. The graphs are fully connected; that is, for every pair of players, we will include the edge connecting them in the graph. Each of these edges encodes a binary feature, indicating whether the two players are on opposing teams or not. For each task, we generated the relevant dataset of node/edge/graph features and corresponding labels (Tables 1 and 2 , see the “Methods” section). The components were then trained separately with their corresponding corner kick graphs. In particular, we only employ a minimal set of features to construct the corner kick graphs, without encoding the movements of the ball nor explicitly encoding the distances between players into the graphs. We used a consistent training-test split for all benchmark tasks, as this made it possible to benchmark not only the individual components but also their interactions.

Accurate receiver and shot prediction through geometric deep learning

One of TacticAI’s key predictive models forecasts the receiver out of the 22 on-pitch players. The receiver is defined as the first player touching the ball after the corner is taken. In our evaluation, all methods used the same set of features (see the “Receiver prediction” entry in Table 1 and the “Methods” section). We leveraged the receiver prediction task to benchmark several different TacticAI base models. Our best-performing model—achieving 0.782 ± 0.039 in top-3 test accuracy after 50,000 training steps—was a deep graph attention network 11 , 12 , leveraging geometric deep learning 10 through the use of D 2 group convolutions 13 . We supplement this result with a detailed ablation study, verifying that both our choice of base architecture and group convolution yielded significant improvements in the receiver prediction task (Supplementary Table 2 , see the subsection “Ablation study” in the “Methods” section). Considering that corner kick receiver prediction is a highly challenging task with many factors that are unseen by our model—including fatigue and fitness levels, and actual ball trajectory—we consider TacticAI’s top-3 accuracy to reflect a high level of predictive power, and keep the base TacticAI architecture fixed for subsequent studies. In addition to this quantitative evaluation with the evaluation dataset, we also evaluate the performance of TacticAI’s receiver prediction component in a case study with human raters. Please see the “Case study with expert raters” section for more details.

For shot prediction, we observe that reusing the base TacticAI architecture to directly predict shot events—i.e., directly modelling the probability ${\mathbb{P}}(\,{{\mbox{shot}}}| {{\mbox{corner}}}\,)$ —proved challenging, only yielding a test F 1 score of 0.52 ± 0.03, for a GATv2 base model. Note that here we use the F 1 score—the harmonic mean of precision and recall—as it is commonly used in binary classification problems over imbalanced datasets, such as shot prediction. However, given that we already have a potent receiver predictor, we decided to use its output to give us additional insight into whether or not a shot had been taken. Hence, we opted to decompose the probability of taking a shot as

where ${\mathbb{P}}(\,{{\mbox{receiver}}}| {{\mbox{corner}}}\,)$ are the probabilities computed by TacticAI’s receiver prediction system, and ${\mathbb{P}}(\,{{\mbox{shot}}}| {{\mbox{receiver}}},{{\mbox{corner}}}\,)$ models the conditional shot probability after a specific player makes first contact with the ball. This was implemented through providing an additional global feature to indicate the receiver in the corresponding corner kick (Table 1 ) while the architecture otherwise remained the same as that of receiver prediction (Supplementary Fig. 2 , see the “Methods” section). At training time, we feed the ground-truth receiver as input to the model—at inference time, we attempt every possible receiver, weighing their contributions using the probabilities given by TacticAI’s receiver predictor, as per Eq. ( 1 ). This two-phased approach yielded a final test F 1 score of 0.68 ± 0.04 for shot prediction, which encodes significantly more signal than the unconditional shot predictor, especially considering the many unobservables associated with predicting shot events. Just as for receiver prediction, this performance can be further improved using geometric deep learning; a conditional GATv2 shot predictor with D 2 group convolutions achieves an F 1 score of 0.71 ± 0.01.

Moreover, we also observe that, even just through predicting the receivers, without explicitly classifying any other salient features of corners, TacticAI learned generalisable representations of the data. Specifically, team setups with similar tactical patterns tend to cluster together in TacticAI’s latent space (Fig. 2 ). However, no clear clusters are observed in the raw input space (Supplementary Fig. 1 ). This indicates that TacticAI can be leveraged as a useful corner kick retrieval system, and we will present our evaluation of this hypothesis in the “Case study with expert raters” section.

We visualise the latent representations of attacking and defending teams in 1024 corner kicks using t -SNE. A latent team embedding in one corner kick sample is the mean of the latent player representations on the same attacking ( A – C ) or defending ( D ) team. Given the reference corner kick sample ( A ), we retrieve another corner kick sample ( B ) with respect to the closest distance of their representations in the latent space. We observe that ( A ) and ( B ) are both out-swing corner kicks and share similar patterns of their attacking tactics, which are highlighted with rectangles having the same colours, although they bear differences with respect to the absolute positions and velocities of the players. All the while, the latent representation of an in-swing attack ( C ) is distant from both ( A ) and ( B ) in the latent space. The red arrows are only used to demonstrate the difference between in- and out-swing corner kicks, not the actual ball trajectories.

Lastly, it is worth emphasising that the utility of the shot predictor likely does not come from forecasting whether a shot event will occur—a challenging problem with many imponderables—but from analysing the difference in predicted shot probability across multiple corners. Indeed, in the following section, we will show how TacticAI’s generative tactic refinements can directly influence the predicted shot probabilities, which will then corresponds to highly favourable evaluation by our expert raters in the “Case study with expert raters” section.

Controlled tactic refinement using class-conditional generative models

Equipped with components that are able to potently relate corner kicks with their various outcomes (e.g. receivers and shot events), we can explore the use of TacticAI to suggest adjustments of tactics, in order to amplify or reduce the likelihood of certain outcomes.

Specifically, we aim to produce adjustments to the movements of players on one of the two teams, including their positions and velocities, which would maximise or minimise the probability of a shot event, conditioned on the initial corner setup, consisting of the movements of players on both teams and their heights and weights. In particular, although in real-world scenarios both teams may react simultaneously to the movements of each other, in our study, we focus on moderate adjustments to player movements, which help to detect players that are not responding to a tactic properly. Due to this reason, we simplify the process of tactic refinement through generating the adjustments for only one team while keeping the other fixed. The way we train a model for this task is through an auto-encoding objective: we feed the ground-truth shot outcome (a binary indicator) as an additional graph-level feature to TacticAI’s model (Table 1 ), and then have it learn to reconstruct a probability distribution of the input player coordinates (Fig. 1 B, also see the “Methods” section). As a consequence, our tactic adjustment system does not depend on the previously discussed shot predictor—although we can use the shot predictor to evaluate whether the adjustments make a measurable difference in shot probability.

This autoencoder-based generative model is an individual component that separates from TacticAI’s predictive systems. All three systems share the encoder architecture (without sharing parameters), but use different decoders (see the “Methods” section). At inference time, we can instead feed in a desired shot outcome for the given corner setup, and then sample new positions and velocities for players on one team using this probability distribution. This setup, in principle, allows for flexible downstream use, as human coaches can optimise corner kick setups through generating adjustments conditioned on the specific outcomes of their interest—e.g., increasing shot probability for the attacking team, decreasing it for the defending team (Fig. 3 ) or amplifying the chance that a particular striker receives the ball.

TacticAI makes it possible for human coaches to redesign corner kick tactics in ways that help maximise the probability of a positive outcome for either the attacking or the defending team by identifying key players, as well as by providing temporally coordinated tactic recommendations that take all players into consideration. As demonstrated in the present example ( A ), for a corner kick in which there was a shot attempt in reality ( B ), TacticAI can generate a tactically-adjusted setting in which the shot probability has been reduced, by adjusting the positioning of the defenders ( D ). The suggested defender positions result in reduced receiver probability for attacking players 2–5 (see bottom row), while the receiver probability of Attacker 1, who is distant from the goalpost, has been increased ( C ). The model is capable of generating multiple such scenarios. Coaches can inspect the different options visually and additionally consult TacticAI’s quantitative analysis of the presented tactics.

We first evaluate the generated adjustments quantitatively, by verifying that they are indistinguishable from the original corner kick distribution using a classifier. To do this, we synthesised a dataset consisting of 200 corner kick samples and their corresponding conditionally generated adjustments. Specifically, for corners without a shot event, we generated adjustments for the attacking team by setting the shot event feature to 1, and vice-versa for the defending team when a shot event did happen. We found that the real and generated samples were not distinguishable by an MLP classifier, with an F 1 score of 0.53 ± 0.05, indicating random chance level accuracy. This result indicates that the adjustments produced by TacticAI are likely similar enough to real corner kicks that the MLP is unable to tell them apart. Note that, in spite of this similarity, TacticAI recommends player-level adjustments that are not negligible—in the following section we will illustrate several salient examples of this. To more realistically validate the practical indistinguishability of TacticAI’s adjustments from realistic corners, we also evaluated the realism of the adjustments in a case study with human experts, which we will present in the following section.

In addition, we leveraged our TacticAI shot predictor to estimate whether the proposed adjustments were effective. We did this by analysing 100 corner kick samples in which threatening shots occurred, and then, for each sample, generated one defensive refinement through setting the shot event feature to 0. We observed that the average shot probability significantly decreased, from 0.75 ± 0.14 for ground-truth corners to 0.69 ± 0.16 for adjustments ( z = 2.62, p < 0.001). This observation was consistent when testing for attacking team refinements (shot probability increased from 0.18 ± 0.16 to 0.31 ± 0.26 ( z = −4.46, p < 0.001)). Moving beyond this result, we also asked human raters to assess the utility of TacticAI’s proposed adjustments within our case study, which we detail next.

Case study with expert raters

Although quantitative evaluation with well-defined benchmark datasets was critical for the technical development of TacticAI, the ultimate test of TacticAI as a football tactic assistant is its practical downstream utility being recognised by professionals in the industry. To this end, we evaluated TacticAI through a case study with our partners at Liverpool FC (LFC). Specifically, we invited a group of five football experts: three data scientists, one video analyst, and one coaching assistant. Each of them completed four tasks in the case study, which evaluated the utility of TacticAI’s components from several perspectives; these include (1) the realism of TacticAI’s generated adjustments, (2) the plausibility of TacticAI’s receiver predictions, (3) effectiveness of TacticAI’s embeddings for retrieving similar corners, and (4) usefulness of TacticAI’s recommended adjustments. We provide an overview of our study’s results here and refer the interested reader to Supplementary Figs. 3 – 5 and the Supplementary Methods for additional details.

We first simultaneously evaluated the realism of the adjusted corner kicks generated by TacticAI, and the plausibility of its receiver predictions. Going through a collection of 50 corner kick samples, we first asked the raters to classify whether a given sample was real or generated by TacticAI, and then they were asked to identify the most likely receivers in the corner kick sample (Supplementary Fig. 3 ).

On the task of classifying real and generated samples, first, we found that the raters’ average F 1 score of classifying the real vs. generated samples was only 0.60 ± 0.04, with individual F 1 scores ( ${F}_{1}^{A}=0.54,{F}_{1}^{B}=0.64,{F}_{1}^{C}=0.65,{F}_{1}^{D}=0.62,{F}_{1}^{E}=0.56$ ), indicating that the raters were, in many situations, unable to distinguish TacticAI’s adjustments from real corners.

The previous evaluation focused on analysing realism detection performance across raters. We also conduct a study that analyses realism detection across samples. Specifically, we assigned ratings for each sample—assigning +1 to a sample if it was identified as real by a human rater, and 0 otherwise—and computed the average rating for each sample across the five raters. Importantly, by studying the distribution of ratings, we found that there was no significant difference between the average ratings assigned to real and generated corners ( z = −0.34, p > 0.05) (Fig. 4 A). Hence, the real and generated samples were assigned statistically indistinguishable average ratings by human raters.

In task 1, we tested the statistical difference between the real corner kick samples and the synthetic ones generated by TacticAI from two aspects: ( A.1 ) the distributions of their assigned ratings, and ( A.2 ) the corresponding histograms of the rating values. Analogously, in task 2 (receiver prediction), ( B.1 ) we track the distributions of the top-3 accuracy of receiver prediction using those samples, and ( B.2 ) the corresponding histogram of the mean rating per sample. No statistical difference in the mean was observed in either cases (( A.1 ) ( z = −0.34, p > 0.05), and ( B.1 ) ( z = 0.97, p > 0.05)). Additionally, we observed a statistically significant difference between the ratings of different raters on receiver prediction, with three clear clusters emerging ( C ). Specifically, Raters A and E had similar ratings ( z = 0.66, p > 0.05), and Raters B and D also rated in similar ways ( z = −1.84, p > 0.05), while Rater C responded differently from all other raters. This suggests a good level of variety of the human raters with respect to their perceptions of corner kicks. In task 3—identifying similar corners retrieved in terms of salient strategic setups—there were no significant differences among the distributions of the ratings by different raters ( D ), suggesting a high level of agreement on the usefulness of TacticAI’s capability of retrieving similar corners ( F 1,4 = 1.01, p > 0.1). Finally, in task 4, we compared the ratings of TacticAI’s strategic refinements across the human raters ( E ) and found that the raters also agreed on the general effectiveness of the refinements recommended by TacticAI ( F 1,4 = 0.45, p > 0.05). Note that the violin plots used in B.1 and C – E model a continuous probability distribution and hence assign nonzero probabilities to values outside of the allowed ranges. We only label y -axis ticks for the possible set of ratings.

For the task of identifying receivers, we rated TacticAI’s predictions with respect to a rater as +1 if at least one of the receivers identified by the rater appeared in TacticAI’s top-3 predictions, and 0 otherwise. The average top-3 accuracy among the human raters was 0.79 ± 0.18; specifically, 0.81 ± 0.17 for the real samples, and 0.77 ± 0.21 for the generated ones. These scores closely line up with the accuracy of TacticAI in predicting receivers for held-out test corners, validating our quantitative study. Further, after averaging the ratings for receiver prediction sample-wise, we found no statistically significant difference between the average ratings of predicting receivers over the real and generated samples ( z = 0.97, p > 0.05) (Fig. 4 B). This indicates that TacticAI was equally performant in predicting the receivers of real corners and TacticAI-generated adjustments, and hence may be leveraged for this purpose even in simulated scenarios.

There is a notably high variance in the average receiver prediction rating of TacticAI. We hypothesise that this is due to the fact that different raters may choose to focus on different salient features when evaluating the likely receivers (or even the amount of likely receivers). We set out to validate this hypothesis by testing the pair-wise similarity of the predictions by the human raters through running a one-away analysis of variance (ANOVA), followed by a Tukey test. We found that the distributions of the five raters’ predictions were significantly different ( F 1,4 = 14.46, p < 0.001) forming three clusters (Fig. 4 C). This result indicates that different human raters—as suggested by their various titles at LFC—may often use very different leads when suggesting plausible receivers. The fact that TacticAI manages to retain a high top-3 accuracy in such a setting suggests that it was able to capture the salient patterns of corner kick strategies, which broadly align with human raters’ preferences. We will further test this hypothesis in the third task—identifying similar corners.

For the third task, we asked the human raters to judge 50 pairs of corners for their similarity. Each pair consisted of a reference corner and a retrieved corner, where the retrieved corner was chosen either as the nearest-neighbour of the reference in terms of their TacticAI latent space representations, or—as a feature-level heuristic—the cosine similarities of their raw features (Supplementary Fig. 4 ) in our corner kick dataset. We score the raters’ judgement of a pair as +1 if they considered the corners presented in the case to be usefully similar, otherwise, the pair is scored with 0. We first computed, for each rater, the recall with which they have judged a baseline- or TacticAI-retrieved pair as usefully similar—see description of Task 3 in the Supplementary Methods . For TacticAI retrievals, the average recall across all raters was 0.59 ± 0.09, and for the baseline system, the recall was 0.36 ± 0.10. Secondly, we assess the statistical difference between the results of the two methods by averaging the ratings for each reference–retrieval pair, finding that the average rating of TacticAI retrievals is significantly higher than the average rating of baseline method retrievals ( z = 2.34, p < 0.05). These two results suggest that TacticAI significantly outperforms the feature-space baseline as a method for mining similar corners. This indicates that TacticAI is able to extract salient features from corners that are not trivial to extract from the input data alone, reinforcing it as a potent tool for discovering opposing team tactics from available data. Finally, we observed that this task exhibited a high level of inter-rater agreement for TacticAI-retrieved pairs ( F 1,4 = 1.01, p > 0.1) (Fig. 4 D), suggesting that human raters were largely in agreement with respect to their assessment of TacticAI’s performance.

Finally, we evaluated TacticAI’s player adjustment recommendations for their practical utility. Specifically, each rater was given 50 tactical refinements together with the corresponding real corner kick setups—see Supplementary Fig. 5 , and the “Case study design” section in the Supplementary Methods . The raters were then asked to rate each refinement as saliently improving the tactics (+1), saliently making them worse (−1), or offering no salient differences (0). We calculated the average rating assigned by each of the raters (giving us a value in the range [− 1, 1] for each rater). The average of these values across all five raters was 0.7 ± 0.1. Further, for 45 of the 50 situations (90%), the human raters found TacticAI’s suggestion to be favourable on average (by majority voting). Both of these results indicate that TacticAI’s recommendations are salient and useful to a downstream football club practitioner, and we set out to validate this with statistical tests.

We performed statistical significance testing of the observed positive ratings. First, for each of the 50 situations, we averaged its ratings across all five raters and then ran a t -test to assess whether the mean rating was significantly larger than zero. Indeed, the statistical test indicated that the tactical adjustments recommended by TacticAI were constructive overall ( ${t}_{49}^{{{{{{{{\rm{avg}}}}}}}}}=9.20,\, p \, < \, 0.001$ ). Secondly, we verified that each of the five raters individually found TacticAI’s recommendations to be constructive, running a t -test on each of their ratings individually. For all of the five raters, their average ratings were found to be above zero with statistical significance ( ${t}_{49}^{A}=5.84,\, {p}^{A} \, < \, 0.001;{t}_{49}^{B}=7.88,\; {p}^{B} \, < \, 0.001;{t}_{49}^{C}=7.00,\; {p}^{C} \, < \, 0.001;{t}_{49}^{D}=6.04,\; {p}^{D} \, < \, 0.001;{t}_{49}^{E}=7.30,\, {p}^{E} \, < \, 0.001$ ). In addition, their ratings also shared a high level of inter-agreement ( F 1,4 = 0.45, p > 0.05) (Fig. 4 E), suggesting a level of practical usefulness that is generally recognised by human experts, even though they represent different backgrounds.

Taking all of these results together, we find TacticAI to possess strong components for prediction, retrieval, and tactical adjustments on corner kicks. To illustrate the kinds of salient recommendations by TacticAI, in Fig. 5 we present four examples with a high degree of inter-rater agreement.

These examples are selected from our case study with human experts, to illustrate the breadth of tactical adjustments that TacticAI suggests to teams defending a corner. The density of the yellow circles coincides with the number of times that the corresponding change is recognised as constructive by human experts. Instead of optimising the movement of one specific player, TacticAI can recommend improvements for multiple players in one generation step through suggesting better positions to block the opposing players, or better orientations to track them more efficiently. Some specific comments from expert raters follow. In A , according to raters, TacticAI suggests more favourable positions for several defenders, and improved tracking runs for several others—further, the goalkeeper is positioned more deeply, which is also beneficial. In B , TacticAI suggests that the defenders furthest away from the corner make improved covering runs, which was unanimously deemed useful, with several other defenders also positioned more favourably. In C , TacticAI recommends improved covering runs for a central group of defenders in the penalty box, which was unanimously considered salient by our raters. And in D , TacticAI suggests substantially better tracking runs for two central defenders, along with a better positioning for two other defenders in the goal area.

We have demonstrated an AI assistant for football tactics and provided statistical evidence of its efficacy through a comprehensive case study with expert human raters from Liverpool FC. First, TacticAI is able to accurately predict the first receiver after a corner kick is taken as well as the probability of a shot as the direct result of the corner. Second, TacticAI has been shown to produce plausible tactical variations that improve outcomes in a salient way, while being indistinguishable from real scenarios by domain experts. And finally, the system’s latent player representations are a powerful means to retrieve similar set-piece tactics, allowing coaches to analyse relevant tactics and counter-tactics that have been successful in the past.

The broader scope of strategy modelling in football has previously been addressed from various individual angles, such as pass prediction 14 , 15 , 16 , shot prediction 3 or corner kick tactical classification 7 . However, to the best of our knowledge, our work stands out by combining and evaluating predictive and generative modelling of corner kicks for tactic development. It also stands out in its method of applying geometric deep learning, allowing for efficiently incorporating various symmetries of the football pitch for improved data efficiency. Our method incorporates minimal domain knowledge and does not rely on intricate feature engineering—though its factorised design naturally allows for more intricate feature engineering approaches when such features are available.

Our methodology requires the position and velocity estimates of all players at the time of execution of the corner and subsequent events. Here, we derive these from high-quality tracking and event data, with data availability from tracking providers limited to top leagues. Player tracking based on broadcast video would increase the reach and training data substantially, but would also likely result in noisier model inputs. While the attention mechanism of GATs would allow us to perform introspection of the most salient factors contributing to the model outcome, our method does not explicitly model exogenous (aleatoric) uncertainty, which would be valuable context for the football analyst.

While the empirical study of our method’s efficacy has been focused on corner kicks in association football, it readily generalises to other set pieces (such as throw-ins, which similarly benefit from similarity retrieval, pass and/or shot prediction) and other team sports with suspended play situations. The learned representations and overall framing of TacticAI also lay the ground for future research to integrate a natural language interface that enables domain-grounded conversations with the assistant, with the aim to retrieve particular situations of interest, make predictions for a given tactical variant, compare and contrast, and guide through an interactive process to derive tactical suggestions. It is thus our belief that TacticAI lays the groundwork for the next-generation AI assistant for football.

We devised TacticAI as a geometric deep learning pipeline, further expanded in this section. We process labelled spatio-temporal football data into graph representations, and train and evaluate on benchmarking tasks cast as classification or regression. These steps are presented in sequence, followed by details on the employed computational architecture.

Raw corner kick data

The raw dataset consisted of 9693 corner kicks collected from the 2020–21, 2021–22, and 2022–23 (up to January 2023) Premier League seasons. The dataset was provided by Liverpool FC and comprises four separate data sources, described below.

Our primary data source is spatio-temporal trajectory frames (tracking data), which tracked all on-pitch players and the ball, for each match, at 25 frames per second. In addition to player positions, their velocities are derived from position data through filtering. For each corner kick, we only used the frame in which the kick is being taken as input information.

Secondly, we also leverage event stream data, which annotated the events or actions (e.g., passes, shots and goals) that have occurred in the corresponding tracking frames.

Thirdly, the line-up data for the corresponding games, which recorded the players’ profiles, including their heights, weights and roles, is also used.

Lastly, we have access to miscellaneous game data, which contains the game days, stadium information, and pitch length and width in meters.

Graph representation and construction

We assumed that we were provided with an input graph ${{{{{{{\mathcal{G}}}}}}}}=({{{{{{{\mathcal{V}}}}}}}},\,{{{{{{{\mathcal{E}}}}}}}})$ with a set of nodes ${{{{{{{\mathcal{V}}}}}}}}$ and edges ${{{{{{{\mathcal{E}}}}}}}}\subseteq {{{{{{{\mathcal{V}}}}}}}}\times {{{{{{{\mathcal{V}}}}}}}}$ . Within the context of football games, we took ${{{{{{{\mathcal{V}}}}}}}}$ to be the set of 22 players currently on the pitch for both teams, and we set ${{{{{{{\mathcal{E}}}}}}}}={{{{{{{\mathcal{V}}}}}}}}\times {{{{{{{\mathcal{V}}}}}}}}$ ; that is, we assumed all pairs of players have the potential to interact. Further analyses, leveraging more specific choices of ${{{{{{{\mathcal{E}}}}}}}}$ , would be an interesting avenue for future work.

Additionally, we assume that the graph is appropriately featurised. Specifically, we provide a node feature matrix, ${{{{{{{\bf{X}}}}}}}}\in {{\mathbb{R}}}^{| {{{{{{{\mathcal{V}}}}}}}}| \times k}$ , an edge feature tensor, ${{{{{{{\bf{E}}}}}}}}\in {{\mathbb{R}}}^{| {{{{{{{\mathcal{V}}}}}}}}| \times | {{{{{{{\mathcal{V}}}}}}}}| \times l}$ , and a graph feature vector, ${{{{{{{\bf{g}}}}}}}}\in {{\mathbb{R}}}^{m}$ . The appropriate entries of these objects provide us with the input features for each node, edge, and graph. For example, ${{{{{{{{\bf{x}}}}}}}}}_{u}\in {{\mathbb{R}}}^{k}$ would provide attributes of an individual player $u\in {{{{{{{\mathcal{V}}}}}}}}$ , such as position, height and weight, and ${{{{{{{{\bf{e}}}}}}}}}_{uv}\in {{\mathbb{R}}}^{l}$ would provide the attributes of a particular pair of players $(u,\, v)\in {{{{{{{\mathcal{E}}}}}}}}$ , such as their distance, and whether they belong to the same team. The graph feature vector, g , can be used to store global attributes of interest to the corner kick, such as the game time, current score, or ball position. For a simplified visualisation of how a graph neural network would process such an input, refer to Fig. 1 A.

To construct the input graphs, we first aligned the four data sources with respect to their game IDs and timestamps and filtered out 2517 invalid corner kicks, for which the alignment failed due to missing data, e.g., missing tracking frames or event labels. This filtering yielded 7176 valid corner kicks for training and evaluation. We summarised the exact information that was used to construct the input graphs in Table 2 . In particular, other than player heights (measured in centimeters (cm)) and weights (measured in kilograms (kg)), the players were anonymous in the model. For the cases in which the player profiles were missing, we set their heights and weights to 180 cm and 75 kg, respectively, as defaults. In total, we had 385 such occurrences out of a total of 213,246( = 22 × 9693) during data preprocessing. We downscaled the heights and weights by a factor of 100. Moreover, for each corner kick, we zero-centred the positions of on-pitch players and normalised them onto a 10 m × 10 m pitch, and their velocities were re-scaled accordingly. For the cases in which the pitch dimensions were missing, we used a standard pitch dimension of 110 m × 63 m as default.

We summarised the grouping of the features in Table 1 . The actual features used in different benchmark tasks may differ, and we will describe this in more detail in the next section. To focus on modelling the high-level tactics played by the attacking and defending teams, other than a binary indicator for ball possession—which is 1 for the corner kick taker and 0 for all other players—no information of ball movement, neither positions nor velocities, was used to construct the input graphs. Additionally, we do not have access to the player’s vertical movement, therefore only information on the two-dimensional movements of each player is provided in the data. We do however acknowledge that such information, when available, would be interesting to consider in a corner kick outcome predictor, considering the prevalence of aerial battles in corners.

Benchmark tasks construction

TacticAI consists of three predictive and generative models, which also correspond to three benchmark tasks implemented in this study. Specifically, (1) Receiver prediction, (2) Threatening shot prediction, and (3) Guided generation of team positions and velocities (Table 1 ). The graphs of all the benchmark tasks used the same feature space of nodes and edges, differing only in the global features.

For all three tasks, our models first transform the node features to a latent node feature matrix, ${{{{{{{\bf{H}}}}}}}}={f}_{{{{{{{{\mathcal{G}}}}}}}}}({{{{{{{\bf{X}}}}}}}},\, {{{{{{{\bf{E}}}}}}}},\, {{{{{{{\bf{g}}}}}}}})$ , from which we could answer queries: either about individual players—in which case we learned a relevant classifier or regressor over the h u vectors (the rows of H )—or about the occurrence of a global event (e.g. shot taken)—in which case we classified or regressed over the aggregated player vectors, ∑ u h u . In both cases, the classifiers were trained using stochastic gradient descent over an appropriately chosen loss function, such as categorical cross-entropy for classifiers, and mean squared error for regressors.

For different tasks, we extracted the corresponding ground-truth labels from either the event stream data or the tracking data. Specifically, (1) We modelled receiver prediction as a node classification task and labelled the first player to touch the ball after the corner was taken as the target node. This player could be either an attacking or defensive player. (2) Shot prediction was modelled as graph classification. In particular, we considered a next-ball-touch action by the attacking team as a shot if it was a direct corner, a goal, an aerial, hit on the goalposts, a shot attempt saved by the goalkeeper, or missing target. This yielded 1736 corners labelled as a shot being taken, and 5440 corners labelled as a shot not being taken. (3) For guided generation of player position and velocities, no additional label was needed, as this model relied on a self-supervised reconstruction objective.

The entire dataset was split into training and evaluation sets with an 80:20 ratio through random sampling, and the same splits were used for all tasks.

Graph neural networks

The central model of TacticAI is the graph neural network (GNN) 9 , which computes latent representations on a graph by repeatedly combining them within each node’s neighbourhood. Here we define a node’s neighbourhood, ${{{{{{{{\mathcal{N}}}}}}}}}_{u}$ , as the set of all first-order neighbours of node u , that is, ${{{{{{{{\mathcal{N}}}}}}}}}_{u}=\{v\,| \,(v,\, u)\in {{{{{{{\mathcal{E}}}}}}}}\}$ . A single GNN layer then transforms the node features by passing messages between neighbouring nodes 17 , following the notation of related work 10 , and the implementation of the CLRS-30 benchmark baselines 18 :

where $\psi :{{\mathbb{R}}}^{k}\times {{\mathbb{R}}}^{k}\times {{\mathbb{R}}}^{l}\times {{\mathbb{R}}}^{m}\to {{\mathbb{R}}}^{{k}^{{\prime} }}$ and $\phi :{{\mathbb{R}}}^{k}\times {{\mathbb{R}}}^{{k}^{{\prime} }}\to {{\mathbb{R}}}^{{k}^{{\prime} }}$ are two learnable functions (e.g. multilayer perceptrons), ${{{{{{{{\bf{h}}}}}}}}}_{u}^{(t)}$ are the features of node u after t GNN layers, and ⨁ is any permutation-invariant aggregator, such as sum, max, or average. By definition, we set ${{{{{{{{\bf{h}}}}}}}}}_{u}^{(0)}={{{{{{{{\bf{x}}}}}}}}}_{u}$ , and iterate Eq. ( 2 ) for T steps, where T is a hyperparameter. Then, we let ${{{{{{{\bf{H}}}}}}}}={f}_{{{{{{{{\mathcal{G}}}}}}}}}({{{{{{{\bf{X}}}}}}}},\, {{{{{{{\bf{E}}}}}}}},\, {{{{{{{\bf{g}}}}}}}})={{{{{{{{\bf{H}}}}}}}}}^{(T)}$ be the final node embeddings coming out of the GNN.

It is well known that Eq. ( 2 ) is remarkably general; it can be used to express popular models such as Transformers 19 as a special case, and it has been argued that all discrete deep learning models can be expressed in this form 20 , 21 . This makes GNNs a perfect framework for benchmarking various approaches to modelling player–player interactions in the context of football.

Different choices of ψ , ϕ and ⨁ yield different architectures. In our case, we utilise a message function that factorises into an attentional mechanism, $a:{{\mathbb{R}}}^{k}\times {{\mathbb{R}}}^{k}\times {{\mathbb{R}}}^{l}\times {{\mathbb{R}}}^{m}\to {\mathbb{R}}$ :

yielding the graph attention network (GAT) architecture 12 . In our work, specifically, we use a two-layer multilayer perceptron for the attentional mechanism, as proposed by GATv2 11 :

where ${{{{{{{{\bf{W}}}}}}}}}_{1},\, {{{{{{{{\bf{W}}}}}}}}}_{2}\in {{\mathbb{R}}}^{k\times h}$ , ${{{{{{{{\bf{W}}}}}}}}}_{e}\in {{\mathbb{R}}}^{l\times h}$ , ${{{{{{{{\bf{W}}}}}}}}}_{g}\in {{\mathbb{R}}}^{m\times h}$ and ${{{{{{{\bf{a}}}}}}}}\in {{\mathbb{R}}}^{h}$ are the learnable parameters of the attentional mechanism, and LeakyReLU is the leaky rectified linear activation function. This mechanism computes coefficients of interaction (a single scalar value) for each pair of connected nodes ( u , v ), which are then normalised across all neighbours of u using the ${{{{{{{\rm{softmax}}}}}}}}$ function.

Through early-stage experimentation, we have ascertained that GATs are capable of matching the performance of more generic choices of ψ (such as the MPNN 17 ) while being more scalable. Hence, we focus our study on the GAT model in this work. More details can be found in the subsection “Ablation study” section.

Geometric deep learning

In spite of the power of Eq. ( 2 ), using it in its full generality is often prone to overfitting, given the large number of parameters contained in ψ and ϕ . This problem is exacerbated in the football analytics domain, where gold-standard data is generally very scarce—for example, in the English Premier League, only a few hundred games are played every season.

In order to tackle this issue, we can exploit the immense regularity of data arising from football games. Strategically equivalent game states are also called transpositions, and symmetries such as arriving at the same chess position through different move sequences have been exploited computationally since the 1960s 22 . Similarly, game rotations and reflections may yield equivalent strategic situations 23 . Using the blueprint of geometric deep learning (GDL) 10 , we can design specialised GNN architectures that exploit this regularity.

That is, geometric deep learning is a generic methodology for deriving mathematical constraints on neural networks, such that they will behave predictably when inputs are transformed in certain ways. In several important cases, these constraints can be directly resolved, directly informing neural network architecture design. For a comprehensive example of point clouds under 3D rotational symmetry, see Fuchs et al. 24 .

To elucidate several aspects of the GDL framework on a high level, let us assume that there exists a group of input data transformations (symmetries), ${\mathfrak{G}}$ under which the ground-truth label remains unchanged. Specifically, if we let y ( X , E , g ) be the label given to the graph featurised with X , E , g , then for every transformation ${\mathfrak{g}}\in {\mathfrak{G}}$ , the following property holds:

This condition is also referred to as ${\mathfrak{G}}$ -invariance. Here, by ${\mathfrak{g}}({{{{{{{\bf{X}}}}}}}})$ we denote the result of transforming X by ${\mathfrak{g}}$ —a concept also known as a group action. More generally, it is a function of the form ${\mathfrak{G}}\times {{{{{{{\mathcal{S}}}}}}}}\to {{{{{{{\mathcal{S}}}}}}}}$ for some state set ${{{{{{{\mathcal{S}}}}}}}}$ . Note that a single group element, ${\mathfrak{g}}\in {\mathfrak{G}}$ can easily produce different actions on different ${{{{{{{\mathcal{S}}}}}}}}$ —in this case, ${{{{{{{\mathcal{S}}}}}}}}$ could be ${{\mathbb{R}}}^{| {{{{{{{\mathcal{V}}}}}}}}| \times k}$ ( X ), ${{\mathbb{R}}}^{| {{{{{{{\mathcal{V}}}}}}}}| \times | {{{{{{{\mathcal{V}}}}}}}}| \times l}$ ( E ) and ${{\mathbb{R}}}^{m}$ ( g ).

It is worth noting that GNNs may also be derived using a GDL perspective if we set the symmetry group ${\mathfrak{G}}$ to ${S}_{| {{{{{{{\mathcal{V}}}}}}}}}|$ , the permutation group of $| {{{{{{{\mathcal{V}}}}}}}}|$ objects. Owing to the design of Eq. ( 2 ), its outputs will not be dependent on the exact permutation of nodes in the input graph.

Frame averaging

A simple mechanism to enforce ${\mathfrak{G}}$ -invariance, given any predictor ${f}_{{{{{{{{\mathcal{G}}}}}}}}}({{{{{{{\bf{X}}}}}}}},\, {{{{{{{\bf{E}}}}}}}},\, {{{{{{{\bf{g}}}}}}}})$ , performs frame averaging across all ${\mathfrak{G}}$ -transformed inputs:

This ensures that all ${\mathfrak{G}}$ -transformed versions of a particular input (also known as that input’s orbit) will have exactly the same output, satisfying Eq. ( 5 ). A variant of this approach has also been applied in the AlphaGo architecture 25 to encode symmetries of a Go board.

In our specific implementation, we set ${\mathfrak{G}}={D}_{2}=\{{{{{{{{\rm{id}}}}}}}},\leftrightarrow,\updownarrow,\leftrightarrow \updownarrow \}$ , the dihedral group. Exploiting D 2 -invariance allows us to encode quadrant symmetries. Each element of the D 2 group encodes the presence of vertical or horizontal reflections of the input football pitch. Under these transformations, the pitch is assumed completely symmetric, and hence many predictions, such as which player receives the corner kick, or takes a shot from it, can be safely assumed unchanged. As an example of how to compute transformed features in Eq. ( 6 ), ↔( X ) horizontally reflects all positional features of players in X (e.g. the coordinates of the player), and negates the x -axis component of their velocity.

Group convolutions

While the frame averaging approach of Eq. ( 6 ) is a powerful way to restrict GNNs to respect input symmetries, it arguably misses an opportunity for the different ${\mathfrak{G}}$ -transformed views to interact while their computations are being performed. For small groups such as D 2 , a more fine-grained approach can be assumed, operating over a single GNN layer in Eq. ( 2 ), which we will write shortly as ${{{{{{{{\bf{H}}}}}}}}}^{(t)}={g}_{{{{{{{{\mathcal{G}}}}}}}}}({{{{{{{{\bf{H}}}}}}}}}^{(t-1)},\, {{{{{{{\bf{E}}}}}}}},\, {{{{{{{\bf{g}}}}}}}})$ . The condition that we need a symmetry-respecting GNN layer to satisfy is as follows, for all transformations ${\mathfrak{g}}\in {\mathfrak{G}}$ :

that is, it does not matter if we apply ${\mathfrak{g}}$ it to the input or the output of the function ${g}_{{{{{{{{\mathcal{G}}}}}}}}}$ —the final answer is the same. This condition is also referred to as ${\mathfrak{G}}$ -equivariance, and it has recently proved to be a potent paradigm for developing powerful GNNs over biochemical data 24 , 26 .

To satisfy D 2 -equivariance, we apply the group convolution approach 13 . Therein, views of the input are allowed to directly interact with their ${\mathfrak{G}}$ -transformed variants, in a manner very similar to grid convolutions (which is, indeed, a special case of group convolutions, setting ${\mathfrak{G}}$ to be the translation group). We use ${{{{{{{{\bf{H}}}}}}}}}_{{\mathfrak{g}}}^{(t)}$ to denote the ${\mathfrak{g}}$ -transformed view of the latent node features at layer t . Omitting E and g inputs for brevity, and using our previously designed layer ${g}_{{{{{{{{\mathcal{G}}}}}}}}}$ as a building block, we can perform a group convolution as follows:

Here, ∥ is the concatenation operation, joining the two node feature matrices column-wise; ${{\mathfrak{g}}}^{-1}$ is the inverse transformation to ${\mathfrak{g}}$ (which must exist as ${\mathfrak{G}}$ is a group); and ${{\mathfrak{g}}}^{-1}{\mathfrak{h}}$ is the composition of the two transformations.

Effectively, Eq. ( 8 ) implies our D 2 -equivariant GNN needs to maintain a node feature matrix ${{{{{{{{\bf{H}}}}}}}}}_{{\mathfrak{g}}}^{(t)}$ for every ${\mathfrak{G}}$ -transformation of the current input, and these views are recombined by invoking ${g}_{{{{{{{{\mathcal{G}}}}}}}}}$ on all pairs related together by applying a transformation ${\mathfrak{h}}$ . Note that all reflections are self-inverses, hence, in D 2 , ${\mathfrak{g}}={{\mathfrak{g}}}^{-1}$ .

It is worth noting that both the frame averaging in Eq. ( 6 ) and group convolution in Eq. ( 8 ) are similar in spirit to data augmentation. However, whereas standard data augmentation would only show one view at a time to the model, a frame averaging/group convolution architecture exhaustively generates all views and feeds them to the model all at once. Further, group convolutions allow these views to explicitly interact in a way that does not break symmetries. Here lies the key difference between the two approaches: frame averaging and group convolutions rigorously enforce the symmetries in ${\mathfrak{G}}$ , whereas data augmentation only provides implicit hints to the model about satisfying them. As a consequence of the exhaustive generation, Eqs. ( 6 ) and ( 8 ) are only feasible for small groups like D 2 . For larger groups, approaches like Steerable CNNs 27 may be employed.

Network architectures

While the three benchmark tasks we are performing have minor differences in the global features available to the model, the neural network models designed for them all have the same encoder–decoder architecture. The encoder has the same structure in all tasks, while the decoder model is tailored to produce appropriately shaped outputs for each benchmark task.

Given an input graph, TacticAI’s model first generates all relevant D 2 -transformed versions of it, by appropriately reflecting the player coordinates and velocities. We refer to the original input graph as the identity view, and the remaining three D 2 -transformed graphs as reflected views.

Once the views are prepared, we apply four group convolutional layers (Eq. ( 8 )) with a GATv2 base model (Eqs. ( 3 ) and ( 4 )) as the ${g}_{{{{{{{{\mathcal{G}}}}}}}}}$ function. Specifically, this means that, in Eqs. ( 3 ) and ( 4 ), every instance of ${{{{{{{{\bf{h}}}}}}}}}_{u}^{(t-1)}$ is replaced by the concatenation of ${({{{{{{{{\bf{h}}}}}}}}}_{{\mathfrak{h}}}^{(t-1)})}_{u}\parallel {({{{{{{{{\bf{h}}}}}}}}}_{{{\mathfrak{g}}}^{-1}{\mathfrak{h}}}^{(t-1)})}_{u}$ . Each GATv2 layer has eight attention heads and computes four latent features overall per player. Accordingly, once the four group convolutions are performed, we have a representation of ${{{{{{{\bf{H}}}}}}}}\in {{\mathbb{R}}}^{4\times 22\times 4}$ , where the first dimension corresponds to the four views ( ${{{{{{{{\bf{H}}}}}}}}}_{{{{{{{{\rm{id}}}}}}}}},\, {{{{{{{{\bf{H}}}}}}}}}_{\leftrightarrow },\, {{{{{{{{\bf{H}}}}}}}}}_{\updownarrow },\, {{{{{{{{\bf{H}}}}}}}}}_{\leftrightarrow \updownarrow }\in {{\mathbb{R}}}^{22\times 4}$ ), the second dimension corresponds to the players (eleven on each team), and the third corresponds to the 4-dimensional latent vector for each player node in this particular view. How this representation is used by the decoder depends on the specific downstream task, as we detail below.

For receiver prediction, which is a fully invariant function (i.e. reflections do not change the receiver), we perform simple frame averaging across all views, arriving at

and then learn a node-wise classifier over the rows of ${{{{{{{{\bf{H}}}}}}}}}^{{{{{{{{\rm{node}}}}}}}}}\in {{\mathbb{R}}}^{22\times 4}$ . We further decode H node into a logit vector ${{{{{{{\bf{O}}}}}}}}\in {{\mathbb{R}}}^{22}$ with a linear layer before computing the corresponding softmax cross entropy loss.

For shot prediction, which is once again fully invariant (i.e. reflections do not change the probability of a shot), we can further average the frame-averaged features across all players to get a global graph representation:

and then learn a binary classifier over ${{{{{{{{\bf{h}}}}}}}}}^{{{{{{{{\rm{graph}}}}}}}}}\in {{\mathbb{R}}}^{4}$ . Specifically, we decode the hidden vector into a single logit with a linear layer and compute the sigmoid binary cross-entropy loss with the corresponding label.

For guided generation (position/velocity adjustments), we generate the player positions and velocities with respect to a particular outcome of interest for the human coaches, predicted over the rows of the hidden feature matrix. For example, the model may adjust the defensive setup to decrease the shot probability by the attacking team. The model output is now equivariant rather than invariant—reflecting the pitch appropriately reflects the predicted positions and velocity vectors. As such, we cannot perform frame averaging, and take only the identity view’s features, ${{{{{{{{\bf{H}}}}}}}}}_{{{{{{{{\rm{id}}}}}}}}}\in {{\mathbb{R}}}^{22\times 4}$ . From this latent feature matrix, we can then learn a conditional distribution from each row, which models the positions or velocities of the corresponding player. To do this, we extend the backbone encoder with conditional variational autoencoder (CVAE 28 , 29 ). Specifically, for the u -th row of H id , h u , we first map its latent embedding to the parameters of a two-dimensional Gaussian distribution ${{{{{{{\mathcal{N}}}}}}}}({\mu }_{u}| {\sigma }_{u})$ , and then sample the coordinates and velocities from this distribution. At training time, we can efficiently propagate gradients through this sampling operation using the reparameterisation trick 28 : sample a random value ${\epsilon }_{u} \sim {{{{{{{\mathcal{N}}}}}}}}(0,1)$ for each player from the unit Gaussian distribution, and then treat μ u + σ u ϵ u as the sample for this player. In what follows, we omit edge features for brevity. For each corner kick sample X with the corresponding outcome o (e.g. a binary value indicating a shot event), we extend the standard VAE loss 28 , 29 to our case of outcome-conditional guided generation as

where h u is the player embedding corresponding to the u th row of H id , and ${\mathbb{KL}}$ is Kullback–Leibler (KL) divergence. Specifically, the first term is the generation loss between the real player input x u and the reconstructed sample decoded from h u with the decoder p ϕ . Using the KL term, the distribution of the latent embedding h u is regularised towards p ( h u ∣ o ), which is a multivariate Gaussian in our case.

A complete high-level summary of the generic encoder–decoder equivariant architecture employed by TacticAI can be summarised in Supplementary Fig. 2 . In the following section, we will provide empirical evidence for justifying these architectural decisions. This will be done through targeted ablation studies on our predictive benchmarks (receiver prediction and shot prediction).

Ablation study

We leveraged the receiver prediction task as a way to evaluate various base model architectures, and directly quantitatively assess the contributions of geometric deep learning in this context. We already see that the raw corner kick data can be better represented through geometric deep learning, yielding separable clusters in the latent space that could correspond to different attacking or defending tactics (Fig. 2 ). In addition, we hypothesise that these representations can also yield better performance on the task of receiver prediction. Accordingly, we ablate several design choices using deep learning on this task, as illustrated by the following four questions:

Does a factorised graph representation help? To assess this, we compare it against a convolutional neural network (CNN 30 ) baseline, which does not leverage a graph representation.

Does a graph structure help? To assess this, we compare against a Deep Sets 31 baseline, which only models each node in isolation without considering adjacency information—equivalently, setting each neighbourhood ${{{{{{{{\mathcal{N}}}}}}}}}_{u}$ to a singleton set { u }.

Are attentional GNNs a good strategy? To assess this, we compare against a message passing neural network 32 , MPNN baseline, which uses the fully potent GNN layer from Eq. ( 2 ) instead of the GATv2.

Does accounting for symmetries help? To assess this, we compare our geometric GATv2 baseline against one which does not utilise D 2 group convolutions but utilises D 2 frame averaging, and one which does not explicitly utilise any aspect of D 2 symmetries at all.

Each of these models has been trained for a fixed budget of 50,000 training steps. The test top- k receiver prediction accuracies of the trained models are provided in Supplementary Table 2 . As already discussed in the section “Results”, there is a clear advantage to using a full graph structure, as well as directly accounting for reflection symmetry. Further, the usage of the MPNN layer leads to slight overfitting compared to the GATv2, illustrating how attentional GNNs strike a good balance of expressivity and data efficiency for this task. Our analysis highlights the quantitative benefits of both graph representation learning and geometric deep learning for football analytics from tracking data. We also provide a brief ablation study for the shot prediction task in Supplementary Table 3 .

Training details

We train each of TacticAI’s models in isolation, using NVIDIA Tesla P100 GPUs. To minimise overfitting, each model’s learning objective is regularised with an L 2 norm penalty with respect to the network parameters. During training, we use the Adam stochastic gradient descent optimiser 33 over the regularised loss.

All models, including baselines, have been given an equal hyperparameter tuning budget, spanning the number of message passing steps ({1, 2, 4}), initial learning rate ({0.0001, 0.00005}), batch size ({128, 256}) and L 2 regularisation coefficient ({0.01, 0.005, 0.001, 0.0001, 0}). We summarise the chosen hyperparameters of each TacticAI model in Supplementary Table 1 .

Data availability

The data collected in the human experiments in this study have been deposited in the Zenodo database under accession code https://zenodo.org/records/10557063 , and the processed data which is used in the statistical analysis and to generate the relevant figures in the main text are available under the same accession code. The input and output data generated and/or analysed during the current study are protected and are not available due to data privacy laws and licensing restrictions. However, contact details of the input data providers are available from the corresponding authors on reasonable request.

Code availability

All the core models described in this research were built with the Graph Neural Network processors provided by the CLRS Algorithmic Reasoning Benchmark 18 , and their source code is available at https://github.com/google-deepmind/clrs . We are unable to release our code for this work as it was developed in a proprietary context; however, the corresponding authors are open to answer specific questions concerning re-implementations on request. For general data analysis, we used the following freely available packages: numpy v1.25.2 , pandas v1.5.3 , matplotlib v3.6.1 , seaborn v0.12.2 and scipy v1.9.3 . Specifically, the code of the statistical analysis conducted in this study is available at https://zenodo.org/records/10557063 .

The International Football Association Board (IFAB). Laws of the Game (The International Football Association Board, 2023).

Tuyls, K. et al. Game plan: what AI can do for football, and what football can do for AI. J. Artif. Intell. Res. 71 , 41–88 (2021).

Article Google Scholar

Goka, R., Moroto, Y., Maeda, K., Ogawa, T. & Haseyama, M. Prediction of shooting events in soccer videos using complete bipartite graphs and players’ spatial–temporal relations. Sensors 23 , 4506 (2023).

Article ADS PubMed PubMed Central Google Scholar

Omidshafiei, S. et al. Multiagent off-screen behavior prediction in football. Sci. Rep. 12 , 8638 (2022).

Article ADS CAS PubMed PubMed Central Google Scholar

Lang, S., Wild, R., Isenko, A. & Link, D. Predicting the in-game status in soccer with machine learning using spatiotemporal player tracking data. Sci. Rep. 12 , 16291 (2022).

Baccouche, M., Mamalet, F., Wolf, C., Garcia, C. & Baskurt, A. Action classification in soccer videos with long short-term memory recurrent neural networks. In International Conference on Artificial Neural Networks (eds Diamantaras, K., Duch, W. & Iliadis, L. S.) pages 154–159 (Springer, 2010).

Shaw, L. & Gopaladesikan, S. Routine inspection: a playbook for corner kicks. In Machine Learning and Data Mining for Sports Analytics: 7th International Workshop, MLSA 2020, Co-located with ECML/PKDD 2020 , Proceedings, Ghent, Belgium, September 14–18, 2020, Vol . 7 , 3–16 (Springer, 2020).

Araújo, D. & Davids, K. Team synergies in sport: theory and measures. Front. Psychol. 7 , 1449 (2016).

Article PubMed PubMed Central Google Scholar

Veličković, P. Everything is connected: graph neural networks. Curr. Opin. Struct. Biol. 79 , 102538 (2023).

Article PubMed Google Scholar

Bronstein, M. M., Bruna, J., Cohen, T. & Veličković, P. Geometric deep learning: grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 (2021).

Brody, S., Alon, U. & Yahav, E. How attentive are graph attention networks? In International Conference on Learning Representations (ICLR, 2022). https://openreview.net/forum?id=F72ximsx7C1 .

Veličković, P. et al. Graph attention networks. In International Conference on Learning Representations (ICLR, 2018). https://openreview.net/forum?id=rJXMpikCZ .

Cohen, T. & Welling, M. Group equivariant convolutional networks. In International Conference on Machine Learning (eds Balcan, M. F. & Weinberger, K. Q.) 2990–2999 (PMLR, 2016).

Honda, Y., Kawakami, R., Yoshihashi, R., Kato, K. & Naemura, T. Pass receiver prediction in soccer using video and players’ trajectories. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 3502–3511 (2022). https://ieeexplore.ieee.org/document/9857310 .

Hubáček, O., Sourek, G. & Železný, F. Deep learning from spatial relations for soccer pass prediction. In MLSA@PKDD/ECML (eds Brefeld, U., Davis, J., Van Haaren, J. & Zimmermann, A.) Vol. 11330, (Lecture Notes in Computer Science, Springer, Cham, 2018).

Sanyal, S. Who will receive the ball? Predicting pass recipient in soccer videos. J Visual Commun. Image Represent. 78 , 103190 (2021).

Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O. & Dahl, G. E. Neural message passing for quantum chemistry. In International Conference on Machine Learning (Precup, D. & Teh, Y. W.) 1263–1272 (PMLR, 2017).

Veličković, P. et al. The CLRS algorithmic reasoning benchmark. In International Conference on Machine Learning (eds Chaudhuri, K. et al.) 22084–22102 (PMLR, 2022).

Vaswani, A. et al. Attention is all you need. In Advances in Neural Information Processing Systems (eds Guyon, I. et al.) Vol. 30 (Curran Associates, Inc., 2017).

Veličković, P. Message passing all the way up. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning (GTRL, 2022). https://openreview.net/forum?id=Bc8GiEZkTe5 .

Baranwal, A., Kimon, F. & Aukosh, J. Optimality of message-passing architectures for sparse graphs. In Thirty-seventh Conference on Neural Information Processing Systems (2023). https://papers.nips.cc/paper_files/paper/2023/hash/7e991aa4cd2fdf0014fba2f000f542d0-Abstract-Conference.html .

Greenblatt, R. D., Eastlake III, D. E. & Crocker, S. D. The Greenblatt chess program. In Proc. Fall Joint Computer Conference , 14–16 , 801–810 (Association for Computing Machinery, 1967). https://dl.acm.org/doi/10.1145/1465611.1465715 .

Schijf, M., Allis, L. V. & Uiterwijk, J. W. Proof-number search and transpositions. ICGA J. 17 , 63–74 (1994).

Fuchs, F., Worrall, D., Fischer, V. & Welling, M. SE(3)-transformers: 3D roto-translation equivariant attention networks. Adv. Neural Inf. Process. Syst. 33 , 1970–1981 (2020).

Google Scholar

Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529 , 484–489 (2016).

Article ADS CAS PubMed Google Scholar

Satorras, V. G., Hoogeboom, E. & Welling, M. E ( n ) equivariant graph neural networks. In International Conference on Machine Learning (eds Meila, M. & Zhang, T.) 9323–9332 (PMLR, 2021).

Cohen, T. S. & Welling, M. Steerable CNNs. In International Conference on Learning Representations (ICLR, 2017). https://openreview.net/forum?id=rJQKYt5ll .

Kingma, D. P. & Welling, M. Auto-encoding variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14–16, 2014, Conference Track Proceedings (ICLR, 2014). https://openreview.net/forum?id=33X9fd2-9FyZd .

Sohn, K., Lee, H. & Yan, X. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems (eds Cortes, C, Lawrence, N., Lee, D., Sugiyama, M. & Garnett, R.) Vol. 28 (Curran Associates, Inc., 2015).

Fernández, J. & Bornn, L. Soccermap: a deep learning architecture for visually-interpretable analysis in soccer. In Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14–18, 2020, Proceedings, Part V (eds Dong, Y., Ifrim, G., Mladenić, D., Saunders, C. & Van Hoecke, S.) 491–506 (Springer, 2021).

Zaheer, M. et al. Deep sets. In Advances in Neural Information Processing Systems Vol. 30 (eds Guyon, I., et al.) (Curran Associates, Inc., 2017).

Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O. & Dahl, G. E. Neural message passing for quantum chemistry. In Proc. 34th International Conference on Machine Learning , Vol. 70 of Proceedings of Machine Learning Research, 6–11 Aug 2017 (eds Precup, D. & Whye Teh, Y) 1263–1272 (PMLR, 2017).

Kingma, G. E. & Ba, J. Adam: a method for stochastic optimization. In ICLR (Poster) , (eds Bengio, Y. & LeCun, Y.) (International Conference of Learning Representations (ICLR), 2015). https://openreview.net/forum?id=8gmWwjFyLj .

Download references

Acknowledgements

We gratefully acknowledge the support of James French, Timothy Waskett, Hans Leitert and Benjamin Hervey for their extensive efforts in analysing TacticAI’s outputs. Further, we are thankful to Kevin McKee, Sherjil Ozair and Beatrice Bevilacqua for useful technical discussions, and Marc Lanctôt and Satinder Singh for reviewing the paper prior to submission.

Author information

These authors contributed equally: Zhe Wang, Petar Veličković, Daniel Hennes.

Authors and Affiliations

Google DeepMind, 6-8 Handyside Street, London, N1C 4UZ, UK

Zhe Wang, Petar Veličković, Daniel Hennes, Nenad Tomašev, Laurel Prince, Michael Kaisers, Yoram Bachrach, Romuald Elie, Li Kevin Wenliang, Federico Piccinini, Jerome Connor, Yi Yang, Adrià Recasens, Mina Khan, Nathalie Beauguerlange, Pablo Sprechmann, Pol Moreno, Nicolas Heess & Demis Hassabis

Liverpool FC, AXA Training Centre, Simonswood Lane, Kirkby, Liverpool, L33 5XB, UK

William Spearman

Liverpool FC, Kirkby, UK

University of Alberta, Amii, Edmonton, AB, T6G 2E8, Canada

Michael Bowling

Google DeepMind, London, UK

You can also search for this author in PubMed Google Scholar

Contributions

Z.W., D. Hennes, L.P. and K.T. coordinated and organised the research effort leading to this paper. P.V. and Z.W. developed the core TacticAI models. Z.W., W.S. and I.G. prepared the Premier League corner kick dataset used for training and evaluating these models. P.V., Z.W., D. Hennes and N.T. designed the case study with human experts and Z.W. and P.V. performed the qualitative evaluation and statistical analysis of its outcomes. Z.W., P.V., D. Hennes, N.T., L.P., M. Kaisers, Y.B., R.E., L.K.W., F.P., W.S., I.G., N.H., M.B., D. Hassabis and K.T. contributed to writing the paper and providing feedback on the final manuscript. J.C., Y.Y., A.R., M. Khan, N.B., P.S. and P.M. contributed valuable technical and implementation discussions throughout the work’s development.

Corresponding authors

Correspondence to Zhe Wang , Petar Veličković or Karl Tuyls .

Ethics declarations

Competing interests.

The authors declare no competing interests but the following competing interests: TacticAI was developed during the course of the Authors’ employment at Google DeepMind and Liverpool Football Club, as applicable to each Author.

Peer review

Peer review information.

Nature Communications thanks Rui Luo and the other anonymous reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information, peer review file, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Wang, Z., Veličković, P., Hennes, D. et al. TacticAI: an AI assistant for football tactics. Nat Commun 15 , 1906 (2024). https://doi.org/10.1038/s41467-024-45965-x

Download citation

Received : 13 October 2023

Accepted : 08 February 2024

Published : 19 March 2024

DOI : https://doi.org/10.1038/s41467-024-45965-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

Explore articles by subject
Guide to authors
Editorial policies

This paper is in the following e-collection/theme issue:

Published on 18.3.2024 in Vol 26 (2024)

Outcomes and Costs of the Transition From a Paper-Based Immunization System to a Digital Immunization System in Vietnam: Mixed Methods Study

Authors of this article:

Original Paper

Thi Thanh Huyen Dang 1 , MD, PhD ;
Emily Carnahan 2 , MPH ;
Linh Nguyen 3 , MPH ;
Mercy Mvundura 2 , PhD ;
Sang Dao 3 , MPH ;
Thi Hong Duong 1 , MD, PhD ;
Trung Nguyen 1 , MPH ;
Doan Nguyen 1 , MD ;
Tu Nguyen 3 , MSc ;
Laurie Werner 2 , MPA ;
Tove K Ryman 4 , MPH, PhD ;
Nga Nguyen 3 , MD, PhD

1 National Expanded Program on Immunization, National Institute of Hygiene and Epidemiology, Hanoi, Vietnam

2 PATH, Seattle, WA, United States

3 PATH, Hanoi, Vietnam

4 Bill & Melinda Gates Foundation, Seattle, WA, United States

Corresponding Author:

Nga Nguyen, MD, PhD

1101, 11th floor, Hanoi Towers

49 Hai Ba Trung Street

Hanoi, 100000

Phone: 84 243936221 ext 130

Email: [email protected]

Background: The electronic National Immunization Information System (NIIS) was introduced nationwide in Vietnam in 2017. Health workers were expected to use the NIIS alongside the legacy paper-based system. Starting in 2018, Hanoi and Son La provinces transitioned to paperless reporting. Interventions to support this transition included data guidelines and training, internet-based data review meetings, and additional supportive supervision visits.

Objective: This study aims to assess (1) changes in NIIS data quality and use, (2) changes in immunization program outcomes, and (3) the economic costs of using the NIIS versus the traditional paper system.

Methods: This mixed methods study took place in Hanoi and Son La provinces. It aimed to analyses pre- and postintervention data from various sources including the NIIS; household and health facility surveys; and interviews to measure NIIS data quality, data use, and immunization program outcomes. Financial data were collected at the national, provincial, district, and health facility levels through record review and interviews. An activity-based costing approach was conducted from a health system perspective.

Results: NIIS data timeliness significantly improved from pre- to postintervention in both provinces. For example, the mean number of days from birth date to NIIS registration before and after intervention dropped from 18.6 (SD 65.5) to 5.7 (SD 31.4) days in Hanoi ( P <.001) and from 36.1 (SD 94.2) to 11.7 (40.1) days in Son La ( P <.001). Data from Son La showed that the completeness and accuracy improved, while Hanoi exhibited mixed results, possibly influenced by the COVID-19 pandemic. Data use improved; at postintervention, 100% (667/667) of facilities in both provinces used NIIS data for activities beyond monthly reporting compared with 34.8% (202/580) in Hanoi and 29.4% (55/187) in Son La at preintervention. Across nearly all antigens, the percentage of children who received the vaccine on time was higher in the postintervention cohort compared with the preintervention cohort. Up-front costs associated with developing and deploying the NIIS were estimated at US $0.48 per child in the study provinces. The commune health center level showed cost savings from changing from the paper system to the NIIS, mainly driven by human resource time savings. At the administrative level, incremental costs resulted from changing from the paper system to the NIIS, as some costs increased, such as labor costs for supportive supervision and additional capital costs for equipment associated with the NIIS.

Conclusions: The Hanoi and Son La provinces successfully transitioned to paperless reporting while maintaining or improving NIIS data quality and data use. However, improvements in data quality were not associated with improvements in the immunization program outcomes in both provinces. The COVID-19 pandemic likely had a negative influence on immunization program outcomes, particularly in Hanoi. These improvements entail up-front financial costs.

Introduction

Since 2017, the National Immunization Information System (NIIS) in Vietnam has been used nationwide by immunization facilities from the national, provincial, district, and commune levels to capture, store, and access immunization data [ 1 ]. The NIIS is a digital system that includes an immunization registry comprising individual-level, longitudinal information on vaccine doses administered and a logistics management system for vaccines and related supplies. As of June 2022, the NIIS has recorded data from approximately 31 million clients and has been used across 15,000 immunization facilities.

Although there is a growing body of evidence on the outcomes associated with digital systems to support vaccine service delivery in low- and middle-income countries [ 2 - 9 ], there are still many questions regarding how they affect data quality, use, and vaccination outcomes in practice. A challenge that has emerged in multiple country contexts is the parallel or dual reporting required when digital systems are introduced and health workers are expected to continue to use the legacy paper-based forms in addition to the new digital system. For example, Tanzanian facilities that had transitioned entirely to using an electronic immunization registry had higher odds of system use compared with those maintaining parallel electronic and paper-based systems [ 10 ].

Moreover, there is limited evidence on the costs of development and implementation of the system, recurrent costs of the system, and cost implications of eliminating a parallel paper system at the service delivery level [ 11 , 12 ]. The Vietnamese experience of introducing and scaling digital tools for immunization presents an opportunity to help fill these evidence gaps.

History of the NIIS

Before the introduction of the NIIS, a paper-based immunization registry and vaccine stock management system were used. Health workers captured data on paper forms, which were compiled in a monthly report. The paper-based system created a significant workload for health workers, and the immunization data were often delayed or incomplete, limiting the availability of reliable data at the higher levels of the health system [ 13 ].

From 2009 to 2012, the National Expanded Program on Immunization (NEPI), in collaboration with PATH and the World Health Organization, developed and piloted a logistics management tracking system for vaccines and related supplies (VaxTrak) and an immunization registry (ImmReg) at the commune and district levels. After the pilot phase, ImmReg expanded to all districts in the pilot province, and VaxTrak was scaled nationwide. An evaluation of ImmReg in 2015 showed that the system was highly accepted by health workers and improved vaccine coverage and timeliness [ 6 ]. In 2014 and 2015, NEPI and PATH integrated ImmReg and VaxTrak into a single comprehensive software system. From 2016 to 2018, the Vietnam General Department of Preventive Medicine and PATH developed the NIIS based on the pilot software and scaled it nationwide [ 1 ].

Transition to a Paperless System

Beginning in 2018, the government of Vietnam collaborated with PATH to provide technical support to strengthen NIIS implementation and transition to a paperless system in 2 provinces: Hanoi and Son La. This work was funded by the Bill & Melinda Gates Foundation and was implemented from 2018 to 2022.

We hypothesized that transitioning to a paperless system would improve data quality and data use in the intervention areas. We hypothesized that if data quality and data use improved, these changes could lead to improvements in immunization program outcomes. The objective of this study was to examine the short-term outcomes and costs associated with the NIIS and the transition to paperless reporting, focusing on three main categories:

What are the changes in data quality and data use because to the paperless transition interventions?
What are the changes in immunization program outcomes (timeliness, dropout rates, and coverage) because of the paperless transition interventions?
What are the incremental financial costs associated with developing, deploying, and maintaining the NIIS, including the economic cost implications of transitioning to paperless reporting?

This mixed methods study aimed to evaluate the short-term outcomes and costs associated with the transition to a paperless system. Mixed methods were used to quantify the observed changes in outcomes and costs and to qualitatively describe why and how changes occurred. Pre-post analyses were conducted to understand the short-term changes in data quality, data use, and immunization program outcomes. Financial data were extracted from the project and partners’ records, and interviews were conducted at various levels of the health system to inform the cost analysis. This study was conducted from July 2019 to November 2021.

This study was conducted in 2 intervention provinces, Hanoi and Son La. These provinces were selected for the transition to a paperless system because they have a variety of geographic, demographic, and health system characteristics that may influence digital readiness. Hanoi, the capital city, is primarily urban with a high population density, high immigration rate, good infrastructure, and many private-sector and fee-based facilities. In contrast, Son La is a mountainous border province primarily composed of rural districts and has low population density, limited resources, and fee-based facilities.

Each province provides immunizations in public district health centers, hospitals, and commune health centers (CHCs) as well as private fee-based immunization facilities (FIFs). Primary data collection for this study also occurred at the national level, for example, to capture financial costs from the project and partners related to the development of the NIIS.

The transition to a paperless system and overall immunization activities in Vietnam were impacted by the COVID-19 pandemic, starting in 2020. The government mandated social distancing lockdowns multiple times in each province, which meant that individuals were not allowed to leave their home without special authorization, and nonessential businesses were closed. Immunization services were disrupted and, in some cases, unavailable, as health care workers were occupied with the COVID-19 response.

Multimedia Appendix 1 includes an overview of the characteristics of the study provinces (Table S1 in Multimedia Appendix 1 ) and dates when COVID-19 social distancing was applied in each province (Table S2 in Multimedia Appendix 1 ).

Interventions

A technical working group composed of the Ministry of Health, NEPI, PATH, and Viettel (the NIIS developer) oversees the implementation of the NIIS. A readiness assessment was conducted from June to July 2019 to provide the technical working group with information about the progress, needs, and challenges of transitioning to a fully paperless immunization system [ 14 ]. NEPI and PATH designed interventions (summarized in Textbox 1 ) to support the transition to paperless reporting based on the readiness assessment results. Interventions included detailed implementation guidelines and standard operating procedures for the transition to paperless reporting, internet-based data review meetings, additional supportive supervision visits, and Zalo (Vietnam’s popular social media and chat app) groups for end users to exchange knowledge and experiences in NIIS use.

Key interventions

Guidelines and training on the shift to paperless reporting : implementation guidelines and standard operating procedures for the transition to paperless were implemented through a training of trainers and cascaded training approach for health workers.
Data quality and data use guidelines and training for health workers at the province, district, and commune health center levels.
Internet-based data review meetings at the district level where all communes share progress on paperless transition. Challenges identified through these meetings were used to prioritize areas for support. Initially, these meetings were held monthly but later shifted to quarterly.
Additional supportive supervision visits from the government and PATH at district and commune facilities to support data quality, data use, and the overall transition to paperless. During the COVID-19 pandemic, these shifted to internet-based supportive supervision visits. Internet-based supportive supervision guidelines and trainings were developed for districts and provinces.
Zalo groups for end users (at least 1 National Immunization Information System [NIIS] user per facility) in each district to exchange knowledge and experiences in NIIS use. Zalo is a popular social media and chat app in Vietnam.

Facilities began transitioning to paperless reporting in November 2019 in Son La and in January 2020 in Hanoi. All facilities in both provinces have retired from the paper-based immunization management logbook and have completely transitioned to paperless reporting using the NIIS as of January 2020. Although all sites are officially reporting using the NIIS, some paper-based systems are still used to comply with various inspections, payment procedures, or requirements from other ministries.

In addition to the key interventions ( Textbox 1 ), an e-learning system and e-immunization cards were piloted at a smaller scale. The e-learning system was developed and piloted in 6 districts (in the 2 provinces) to train managers at the national, regional, provincial, and district levels and facility health workers (CHCs, FIFs, and hospitals) on using the NIIS. The e-immunization card, a mobile phone app that allows parents or clients to access their individual demographic information and vaccination data, was also developed and launched in the 2 provinces.

Study Design and Data Collection

Various methods were used to collect data related to each of the study aims (data quality and use, immunization performance, and costing) in the 2 provinces. This section describes the study design, data sources, and data collection approach for each study objective. Table S3 in Multimedia Appendix 1 summarizes the data collection methods across the 3 study aims.

Data Quality and Use

A pre- or postintervention study design was used to assess changes in data quality and use. Data collection included self-administered facility assessments, household surveys, and facility surveys conducted at a sample of CHCs, FIFs, and hospitals; details of the methodology have been published elsewhere [ 14 ]. The same methods were used for pre- and postintervention data collection, with the addition of in-depth interviews with Expanded Program on Immunization (EPI) officers at the district and commune levels at postintervention.

The self-administered facility assessment was sent via email to all immunization facilities to collect basic information regarding infrastructure, capacity, and NIIS data use.

Facility surveys were conducted on a purposively selected sample of districts, communes, hospitals, and FIFs in each province. Purposive sampling was performed in consultation with NEPI and provincial Centers for Disease Control and Prevention (CDC) to select a mix of facility types (fee based, private, and public), geographies (urban, semiurban, rural, and mountainous), and experiences with NIIS. The study was designed to survey the same facilities at pre- and postintervention. At preintervention, 8 FIFs and 7 hospitals were included; at postintervention, 5 FIFs and 7 hospitals were included (the change was owing to 3 FIFs that had closed by the time of the postintervention survey). In each facility, 20 clients in the paper logbook were randomly selected. The facility surveys captured structured information about the facility, the use of the NIIS, and demographic and immunization information for the 20 sampled clients.

A household survey was conducted in a sample of households with children aged <2 years to capture demographic and immunization information about the children from their home-based immunization cards. The household survey was conducted in the same purposively selected communes as the facility surveys. Within each commune, villages or living quarters were selected for convenience, and all households with children in the defined age range were included.

The NEPI and PATH staff trained data collectors who used KoboToolbox platform for data entry for all data collection forms. More details on the sampling approach and a full list of facilities included in the different evaluations conducted as part of this study are included in Table S4 in Multimedia Appendix 1 .

Structured interviews were conducted to understand the factors related to data quality and data use. CDC staff led the interviews with immunization managers at the district level and health workers at the commune level. Interviews involving NEPI and CDC personnel were performed by the PATH staff. Interviews were conducted over the phone and in person at the interviewees’ workplaces, lasting around 35 minutes. The interviews were recorded with consent and transcribed.

Immunization Program Outcomes

A pre- or postintervention study design was used to assess changes in immunization program outcomes in the 2 provinces using NIIS data. NIIS data were exported for a pre- and postintervention cohort of children in each province. Due to the differing commencement dates of paperless reporting in 2 project provinces was different, Son La began in November 2019, while Hanoi followed in Jan 2020. As a result, the preintervention cohort group comprises children born between July 1 and September 30, 2018, while the postintervention group comprises those born between July 1 and September 30, 2020. Each child’s immunization information was analyzed for their first 12 months of life.

The costing study used a mixed methods approach including primary data collection using a microcosting approach and secondary data collection from financial record reviews. The costs of all activities were estimated from the perspectives of the implementing organizations (Hanoi and Son La provinces for NIIS systems use and Viettel, PATH, and NEPI) and hence take the health system perspective. No client costs were included.

We captured the costs for the different activities of implementing the NIIS from the software design and development activities to the deployment, accounting for the costs of the different partners engaged in this process. For the partner leading the software design, development, and deployment, we include the costs of the infrastructure, server, bandwidth, technical support, help desk, training, and maintenance and operations of the system. For the NEPI and the subnational levels, the costs included those for development of training materials, conducting staff training, data entry, meetings, and internet setup, at facilities where it was needed.

Data on the incremental financial costs for designing, developing, and deploying the NIIS software were obtained through the NEPI and partner organization expenditure records review. We also obtained information from each EPI administration and health facility in the sample on the expenditures for NIIS-related training and meetings and other deployment costs, including costs for data back entry and internet or phone setup at facilities.

Data on the recurrent financial costs of the NIIS were obtained via interviews with the head of each facility or person in charge of the facility finances in each study facility in the 2 provinces. The NEPI provided records on hardware inventory and repair requests, which were used to estimate replacement rates and maintenance costs for equipment.

Data Analysis

This section describes the variables and data analysis approach for each study objective.

For qualitative data analysis, we first transcribed all the interviews. Then, a member of the research team, who was trained on qualitative data methods, was assigned to code the transcripts using Microsoft Excel. This coding process followed a content analysis approach and involved a 3-level coding process: initially, open coding was applied to 5 transcripts to identify major themes; subsequently, the research team held discussions to reach a consensus on the major themes and any emergent themes; and finally, a final codebook was created before coding the remaining transcripts. This approach helped to gain a comprehensive understanding of health workers’ perspectives on improving data quality and data use from the NIIS as well as to identify the barriers and facilitators of immunization coverage.

In the analysis of quantitative data, we used Stata (version 14; StataCorp) as our statistical tool. For categorical variables, we used the chi-square test, and in cases where the expected cell counts were <5, Fisher exact test was used. For continuous variables, we first checked the variable distribution, and given the absence of a normal distribution, the Wilcoxon-Mann-Whitney U test was used to compare group differences. In addition, to investigate the relationship between immunization outcomes and its determinants, we conducted a multivariable logistic regression analysis. The threshold for statistical significance was set at P <.05.

The data use outcome of interest was the percentage of facilities using NIIS data to inform specified routine activities (eg, making monthly vaccination plans).

The data quality outcomes of interest were the quantitative measures of timeliness, completeness, and accuracy. Information from the household and health facility surveys was compared with the NIIS data to assess data quality. Refer to Table S5 in Multimedia Appendix 1 for the definitions of the data quality indicators.

Data exported from the NIIS were cleaned. Duplicate records, unreliable data (eg, vaccination date before birth date), and records with “lost to follow-up” status were excluded. The primary outcome of interest was on-time vaccination, determined by the recommended age for vaccine delivery according to the NEPI vaccination schedule [ 15 ]. The secondary immunization program outcomes of interest were dropout rates and full vaccination coverage. Multimedia Appendix 1 provides details on NIIS data cleaning and definitions for the primary and secondary outcomes of interest (Table S6 in Multimedia Appendix 1 ).

Quantitative data were analyzed using Stata (version 14). We computed the total incremental costs per health system level associated with the NIIS implementation. We also estimated the cost per child for the NIIS implementation activities. For this analysis, we allocated a proportion of the costs for NIIS implementation to the 2 provinces according to their annual birth cohort size relative to the national birth cohort. The annual birth cohort for Vietnam in 2019 was approximately 1.5 million [ 16 ], whereas the 2 study provinces (Hanoi and Son La) had an annual birth cohort of 162,000 and 25,000, respectively, based on data from the NIIS, representing approximately 12% (187,000/1,500,000) of the annual birth cohort of Vietnam. The NIIS implementation costs were spread over 5 birth cohorts in these cost-per-child calculations, as the system implementation was done over the 5-year period.

To estimate the economic costs associated with service delivery and reporting using either the paper-based system, electronic system, or both, we used an activity-based costing approach. Ingredients or components of the activities were quantified for each resource type, including human resource time use for different immunization activities where there would be a change when using the NIIS versus the paper system, capital costs for equipment and supplies, and recurrent costs for internet connectivity and equipment maintenance attributable to using the NIIS or the paper system. The quantification was done by conducting interviews at the study facilities using structured costing questionnaires. The unit cost of each resource was obtained from secondary data sources, and the total costs for each activity by resource type were estimated. To obtain the total costs, we aggregated the costs for each activity by resource type. We estimated the recurrent costs per facility and per child, with the latter based on only 1 birth cohort as these are annual recurrent costs.

Ethical Considerations

This study served as the end-line evaluation activity within the project’s work plan. The project was a collaborative effort between the NEPI and PATH. This evaluation constitutes 1 of the project activities outlined in the project documents submitted to the Vietnam Ministry of Health. As per the regulations set forth by the Vietnamese government, the project documents were reviewed and certified by units within the Ministry of Health and other relevant ministries before receiving approval from the Ministry of Health. The study was reviewed, considered as project evaluation, and approved by the Vietnam Ministry of Health (decision 1996/QĐ-BYT), and this study does not require ethics review in accordance with the circular 04/TT-BYT issued by the Vietnam Ministry of Health [ 17 ]. This circular [ 17 ] regulates the establishment, functions, tasks, and rights of research ethics committees, and it is specified that only research involving human subjects necessitates research ethics committee approval before implementation and supervision during the research process. In addition, the study was reviewed by PATH’s US-based Research Determination Committee, which concluded that the activity did not involve “human subjects” as defined in the US Government 45 Code of Federal Regulations 46.102(e) [ 18 ] and did not require US ethics review.

In addition, before conducting the interviews, comprehensive information was provided to all participants, encompassing the study’s objectives, participant rights, and strict confidentiality measures applied to protect their personal information. Informed consent was diligently obtained from each participant before starting the interviews. Their consent to participate in the study was obtained before proceeding with the interviews. Each qualitative interviewee received VN ₫150.000 (US $6.50) as payment for their time. In facilities selected for the costing study, health facility staff received VN ₫400.000 (US $17) for their time participating in structured costing interviews. Interviews were conducted in private settings to ensure confidentiality. All information was coded and only accessible to the study team, and data privacy was emphasized during training of the data collection team. The NEPI provided official permission for the use of the data extracted from the NIIS. All identifying data were coded, and names were eliminated before data analysis.

Data quality and use were measured through the NIIS data export, household surveys, and facility surveys and further explained through qualitative interviews.

Data Quality

The NIIS data quality evaluation considered the attributes of timeliness, completeness, and accuracy.

On the basis of the data exported from the NIIS, timeliness significantly improved from pre- to postintervention for all indicators across all health system levels in both provinces ( Table 1 ). Between pre- and postintervention, there was a significant decrease in the mean number of days from birth date to NIIS registration (Hanoi: 18.6, SD 65.5 to 5.7, SD: 31.4 days; P <.001 and Son La: 36.1, SD 94.2 to 11.7, SD 40.1; P <.001). Across all health system levels (CHCs, FIFs, and hospitals), there were significant decreases in the mean number of days from the injection date to when the injection was updated in the NIIS. Stock transactions (only assessed at the CHC level) also showed a significant decrease in the mean number of days from stock arrival date to NIIS voucher date (Hanoi: 10.5, SD 36.1 to 5.2, SD: 19.8 days; P <.001 and Son La 13.4, SD 38.1 to 6.5, SD 23.5; P <.001).

a N/A: not applicable.

b NIIS: National Immunization Information System.

c Dependent variables were not normally distributed; therefore, the Wilcoxon-Mann-Whitney U test was used.

Completeness

Completeness was assessed by comparing information from the household and facility surveys with the information from the NIIS ( Table 2 ). At the CHC level, completeness of registration, client information, and injection information captured in the NIIS significantly improved between pre- and postintervention in Hanoi and Son La. At the FIF level in Hanoi, there was a decline in the percentage of clients registered in the NIIS; however, among those registered, there was an increase in the completeness of client information. At the FIF level in Son La, there was increased completeness of registration, client information, and immunization information. At the hospital level, there was an improvement in the percentage of clients registered in the NIIS and a decline in the completeness of client information in both provinces, and the completeness of immunization information remained unchanged at 100% (Hanoi: preintervention n=63, postintervention n=153; Son La: preintervention n=41, postintervention n=74) in both provinces at pre- and postintervention.

c HH: household.

d FIF: fee-based immunization facility.

The accuracy of demographic and immunization information was assessed by comparing information on clients’ personal immunization cards (captured through the household survey) with their information entered in the NIIS. The percentage of clients with demographic information matched between the 2 sources significantly increased in Hanoi (199/217, 91.7% to 340/353, 96.3%; P =.02) and Son La (101/119, 84.9% to 107/107, 100%; P <.001) from pre- to postintervention ( Table 3 ). The percentage of injections with immunization information matched between the 2 sources also significantly increased in Son La (1037/1216, 85.3% to 1097/1097, 100%; P <.001) but significantly decreased in Hanoi (2147/2188, 98.1% to 3786/3981, 95.1%; P =.01).

The interviews indicated that respondents at all levels (national, provincial, district, and facility levels) and across both provinces had a strong understanding of data quality, defined as timeliness, completeness, and accuracy. Most respondents (13/16, 81%) participated in the intervention training on data quality and data use and indicated that it was useful for their work and that they had applied what they learned:

I have participated in the training course on data quality and data use last year. After the training, my knowledge and skill in data quality and data use are better, so I applied to my daily work, I usually check the input data to make them complete and accurate before entering into the system. [CHC staff]

All respondents rated their facility’s data quality in the NIIS as “good” or “very good” in terms of timeliness, completeness, and accuracy. All respondents mentioned human resources as the most important factor associated with data quality, including health workers’ knowledge and skills (in data entry, analysis, quality assessment, and use), understanding the importance of data quality, and bandwidth to support the immunization program when working across health areas.

Data use was measured through facility assessments asking health workers about the activities that the NIIS data were used to inform. In the preintervention survey, 34.8% (202/580) of the facilities in Hanoi and 29.4% (55/187) of the facilities in Son La indicated that they had used the NIIS data for additional activities beyond monthly reporting. In the postintervention survey, 100% (Hanoi: 468/468 and Son La: 199/199) of the facilities in both provinces indicated using the NIIS data for activities beyond monthly reporting. Table S7 in Multimedia Appendix 1 shows the frequency by activity. At postintervention, the most common uses of data among facilities were to inform monthly vaccination plans, campaign plans, or annual immunization plans. At the management level, the most common use of the NIIS data was to evaluate the performance of health facilities. From the qualitative interviews, the most frequently mentioned obstacle to using the data was the lack of health workers’ capacity for data analysis and use.

On-time vaccination was the primary immunization performance outcome of interest. Secondary outcomes were dropout rates and vaccination coverage (refer to the Tables S8-12 in Multimedia Appendix 1 ).

Study Population Characteristics

Immunization outcomes were assessed by comparing the NIIS data for a cohort of children pre- and postintervention in the 2 provinces. After data cleaning, 81,485 children were included in the sample. Their population characteristics are summarized in Table S8 in Multimedia Appendix 1 . In Hanoi, there were small but significant differences in the ethnicity and rural and urban location of children in the pre- and postintervention cohorts. In Son La, there was also a significant difference in the ethnicity of children in the pre- and postintervention cohorts. In both provinces, there were significant differences in the percentage of children vaccinated primarily from FIFs versus CHCs at pre- and postintervention.

On-Time Vaccination by Antigen

Across all antigens, apart from the measles-containing vaccine first dose in Hanoi, the percentage of children who received the vaccine on time increased from pre- to postintervention, and nearly all increases were statistically significant ( Table 4 ).

a BCG: bacillus Calmette-Guérin.

b Penta: pentavalent.

c MCV1: measles-containing vaccine first dose.

In the multiple logistic regression analysis, children at postintervention were approximately 1.18 times as likely in Hanoi and 1.69 times as likely in Son La to receive timely administration of pentavalent (Penta) 3 compared with those in the preintervention cohort ( P <.001; Table 5 ). In Hanoi, there was no significant difference in timely Penta 3 vaccination by gender or ethnicity, but children in urban areas were 1.6 times as likely to receive Penta 3 on time compared with those in rural areas. In Son La, there was also no difference by gender, but Thai children were 1.3 times as likely to receive Penta 3 on time compared with other ethnicities. In both provinces, children who were mostly vaccinated from FIFs were more likely to receive Penta 3 on time compared with children mostly vaccinated from CHCs.

A separate logistic regression analysis for on-time full vaccination is included in Table S12 in Multimedia Appendix 1

On-Time Full Immunization Coverage

Figure 1 shows the full immunization coverage over time for each birth cohort. In Hanoi, 87.8% (29,649/33,752) of the children in the preintervention cohort and 81.9% (29,167/35,611) of the children in the postintervention cohort ( P <.001) had reached full immunization before their first birthday. In contrast, there was an increase in the percentage of children reaching full immunization before their first birthday in Son La, from 63.3% (3947/6233) preintervention to 88.3% (5040/5705) postintervention ( P <.001). The results for immunization coverage by antigen for the pre- and postintervention birth cohorts in each province are included Table S9 in Multimedia Appendix 1 .

Up-Front Costs of NIIS Implementation

The up-front NIIS software design, development, and deployment took approximately 2300 person-months of labor from 2015 to 2020, and the partner costs for this labor were estimated at approximately US $1.75 million ( Table 6 ). Most of the software developer costs (US $1,233,152/US $1,745,712, 71%) pertained to the deployment of the system. In addition, there were up-front costs per facility for implementing the NIIS, including costs for back entry of data, internet setup (where needed), training of users, and meeting costs. At the national and provincial levels, the bulk of the up-front costs were spent on training and meetings. As trainings were paid for by the higher administrative levels, training costs are low or 0 at district levels and health facilities. At these levels, the larger cost share was for deploying the NIIS.

a NIIS: National Immunization Information System.

The up-front costs for the NIIS implementation allocated to the 2 study provinces were approximately US $419,000 ( Table 7 ). When these costs are allocated over 5 birth cohorts, the estimated cost per child for the NIIS implementation was estimated at US $0.48.

b NEPI: National Expanded Program on Immunization.

Recurrent Costs for the NIIS

The software developer estimated the annual recurrent costs for the system operation to be US $85,000. At health facilities, the average monthly economic cost for health worker labor for immunization-related activities done using the paper system were estimated to be US $146 ( Table 8 ). In comparison, the monthly cost for labor with the NIIS was US $67, which is less than half of the labor cost when using the paper system. The most significant savings in labor time costs, resulting from the transition from the paper system to the NIIS, occurred through reduced time spent on organizing immunization sessions, data management, and reporting. However, the NIIS also resulted in additional activities for staff, including checking of duplicates and introduction of e-immunization cards. The NIIS also added new facility costs, including recurrent costs for internet, printing, and SMS text messaging reminders and the capital costs of equipment, which amounts to an average cost of US $28 per facility. However, with the NIIS, there were savings in printing costs as registers and ledgers would not be printed, and the average costs of these were US $4.51 per month or US $58 per year. Overall, the total monthly costs per facility with NIIS (US $95) are lower than with the paper system (US $151).

Table 9 presents similar costs for the administrative levels. At most levels, except at provincial CDC, labor costs are lower with the NIIS than with the paper system. At the CDC, the labor costs for supportive supervision are the largest share of costs, and these costs increase when using the NIIS compared with when using the paper system. At the district health centers, there is a decline in labor costs for activities such as management and reporting when using the NIIS, and hence, labor costs are lower with the NIIS. As mentioned above, implementing the NIIS incurs additional costs, including recurrent costs for internet, printing, and equipment maintenance and capital costs for equipment, which makes the total monthly costs for NIIS more than the total monthly costs for the paper system. There are incremental costs resulting from the change from the paper system to the NIIS at the administrative levels.

b N/A: not applicable.

Although labor costs decrease at most health system levels with the transition to the NIIS, in practice, this may not translate into budget line savings as staff are retained and their time is reallocated. The financial costs associated with the paper system primarily entail printing registers, but these expenses are relatively minor in comparison with the those incurred with the electronic system. The estimated annual financial recurrent costs per child for the NIIS are US $3.17, and these account for the annual costs of the server (with the relevant portion allocated to the study provinces) and the costs for capital equipment and internet. There are incremental financial costs to the health system when moving from the legacy paper system to the electronic system.

Principal Findings

Health workers at the province, district, and CHC levels successfully transitioned from the legacy paper-based system to the NIIS for paperless reporting, proving that the transition was possible in 2 very different provincial contexts. Although the transition to paperless reporting results in lower labor costs at the facility, district, and national levels, it requires incremental financial recurrent costs (estimated at US $3.17 per child per year) to maintain the NIIS.

This study found improvements in data quality in Hanoi and Son La between pre- and postintervention. There were significant improvements in data timeliness at all levels of the health system and in both provinces. It is likely that the interventions contributed to this improvement because the guidelines, trainings, data review meetings, and supportive supervision visits emphasized the importance of timely data. Timeliness indicators performed better in Hanoi than in Son La, which may be because of the different service delivery practices; in Hanoi, vaccines are delivered on a weekly basic, whereas in Son La, they are delivered monthly. In addition, in Hanoi, more clients visit FIFs that deliver immunization daily.

In Son La, there were also improvements across nearly all the completeness and accuracy indicators. However, there were mixed results in terms of data completeness and accuracy in Hanoi, with some statistically significant declines in quality. This may be partly because Hanoi was more affected by the COVID-19 pandemic than Son La; there were 6 rounds of mandated social distancing across all 30 districts in Hanoi. When immunization services were available, facilities may have been understaffed if human resources were reassigned to the COVID-19 response and they may have faced higher demand from a backlog of children who could not be vaccinated during social distancing. Overburdened health workers may prioritize service delivery over data quality.

The observed differences in provincial results may also have been influenced by the way interventions were delivered in each province. The facilities in Son La received timely support through internet-based supervisory assistance. In Hanoi, they maintained in-person supportive supervision, which was less timely, and limited the number of facilities that could be visited. Facilities in Son La also received refresher trainings, which Hanoi did not, as Hanoi had an ongoing focus on COVID-19 vaccination.

There were large increases from pre- to postintervention in the percentage of facilities using the NIIS data to inform their activities. Results from the costing analysis showed that at the CHCs, health workers’ time spent on immunization program data entry and reporting declined as the transition to paperless resulted in efficiencies and reduced workload. It is possible that this saved time was dedicated to data use.

For most activities, a higher percentage of facilities in Son La reported using NIIS data compared with Hanoi. These results may have also been influenced by the COVID-19 pandemic or the differences in project interventions. In addition, Hanoi has a more mobile population, with clients frequently moving between facilities. This implies that vaccination plans informed by NIIS data may not be as accurate or comprehensive compared with a location such as Son La, where there is less population movement.

Although only 44.7% (84/188) of facilities in Son La and 27.7% (125/451) of facilities in Hanoi reported using the NIIS to evaluate data quality, previous studies have shown that data use can drive data quality improvements [ 19 ]. Through the qualitative interviews, health workers recommended building data quality indicators directly into the NIIS to help users understand their data quality easily.

The study also found that the majority of on-time vaccination rates for various antigens in the 2 project provinces improved compared to those in the preintervention group. With more timely data captured in the NIIS, health workers know which vaccines a child is due for and can send reminders or follow-up to deliver the vaccines on time. For the clients using the e-immunization card, it may have also contributed to timely vaccination by reminding clients of their vaccination schedule and due dates. However, the on-time vaccination rate for the first dose of measles in Hanoi decreased after the intervention compared to before, which may have been owing to the COVID-19 lockdowns from July to September 2021, when the postintervention cohort was aged 9 to 12 months. In Hanoi, many parents prefer waiting to receive the measles-rubella vaccine at 12 months, which is delivered at FIFs, versus the monovalent measles vaccine, which is part of the standard EPI schedule at 9 months.

In Son La, improvements in data quality and on-time vaccination were also associated with improvements in vaccination coverage. Full immunization at 12 months was 88.3% (5040/5705) in the postintervention cohort compared with 63.3% (3947/6233) in the preintervention cohort. A previous evaluation of ImmReg (the earlier version of the NIIS) in another province in Vietnam, Ben Tre, also found improvements in on-time vaccination and vaccination coverage when comparing pre- and postintroduction of the digital immunization registry, and these improvements were sustained or increased 1 year later [ 6 ]. However, in Hanoi, despite improvements in on-time vaccination, there were declines in vaccination coverage, again likely because of the COVID-19 pandemic. According to the 2021 annual EPI report for Vietnam, full immunization coverage decreased nationwide from 96.8% in 2020 to 87.1% in 2021 [ 20 ].

Introducing these new electronic systems, which have improved data quality and timeliness and could potentially improve immunization outcomes, comes at a cost, as up-front financial investments have to be made for software design, development, and deployment. The cost per child for NIIS development and deployment was estimated at US $0.48 in the 2 study provinces in Vietnam. This is much lower than that estimated in Tanzania and Zambia, the only other low- and middle-income countries with comparable cost estimates for the development of electronic immunization registries [ 11 , 12 ]. In our study, costs were annualized over 5 birth cohorts, whereas in Tanzania and Zambia, costs were annualized over 3 birth cohorts. Even if we were to use 3 birth cohorts for this analysis, our estimated costs for Vietnam would still be lower. The estimated lower costs per child for Vietnam could be in part because most CHCs had existing computers, and hence, there was no mass procurement of equipment required, which is not the case in many other countries. In Vietnam, the equipment and connectivity costs were shared across multiple health programs, further reducing the costs borne by the immunization program, unlike in Tanzania and Zambia, where immunization was the first health area to be digitalized at the health facility level. The availability of electricity at the health facility level in Vietnam also reduced costs, as there was no need for the procurement of alternative power sources such as solar chargers. In addition, software development in Vietnam was conducted by an in-country telecommunications partner that provided in-kind and pro bono services and support, some of which could not be quantified, resulting in lower costs than if done through an external organization.

Our study found that at the CHC level, there are cost savings for health workers’ time use and other resources such as printing with the NIIS compared with the paper system, and at the administrative level, there are incremental costs. The NIIS results in some savings in labor time, although some new activities are added for the staff. There are also recurring costs, such as ongoing technical support and maintenance provided by Viettel, the need for refresher training at the local level for new staff, and expenses associated with equipment maintenance and replacement as well as connectivity provided by local government. These recurrent costs are important to consider for sustainability.

Limitations

Our study has several limitations. First, we conducted the study in 2 provinces and had small sample sizes for the costing results, facility survey results for data quality, and other results based on provider interviews. The small sample size and purposive sampling limit the generalizability of our findings; however, 2 distinct provinces with a mix of facility types were intentionally sampled so that the results can be used to understand the costs and outcomes of the transition to paperless reporting in a range of settings. Second, the time frame for the evaluation was short, which limited the magnitude of the change that could be observed. Third, there were minor differences in the data collection approaches and responses between pre- and postintervention. For example, the urban sampling approach was adjusted for the feasibility of data collection, and the response rate for facility assessment was lower at postintervention. Fourth, for data collected through provider interviews, such as for the costing study, potential recall bias arises due to retrospective nature of the inquires, as interviewees were asked to recall cost from past periods. Future studies should consider including costing data collection at different phases of the project to reduce recall bias. This would also facilitate the availability of costing data available to inform decision-making at various implementation phases. Our study may also have underestimated the costs associated with the paper system, as we did not incorporate expenses related to printing paper home-based records and invitation letters. These materials are printed to assist the caregivers in monitoring their children’s vaccination history or reminding them when their child is due for vaccination. We also have not accounted for the saved labor costs for health care workers who would deliver the invitation letters to households. Another limitation is that the COVID-19 pandemic affected the originally planned timelines for data collection and the immunization service delivery. We did not try to account for the impact of the pandemic on findings regarding the data quality and data use presented in this study.

Conclusions

Health workers in the 2 provinces successfully transitioned to paperless reporting while maintaining or improving data quality. We recommend that other provinces in Vietnam transition to paperless reporting by introducing the guidelines and standard operating procedures used in Hanoi and Son La and providing ongoing support through trainings, data review meetings, supportive supervision, and peer networks. Future studies should monitor data quality and immunization outcomes in other provinces as well as the sustainability of the observed changes in Hanoi and Son La.

Introducing these new electronic systems comes with costs—both up-front and recurrent—but there are advantages, as seen in the improvements in data quality and on-time vaccination. In Vietnam, stakeholders should plan and budget for the sustainability of the system at each level of the health system, given the recurrent costs including repairing and replacement of equipment, connectivity, refresher training, software system, and supportive supervision. Other countries planning to implement similar interventions should plan to collect costing data throughout to inform decision-making and budgeting.

Acknowledgments

The authors thank the Bill & Melinda Gates Foundation for providing support for this study and the National Immunization Information System (NIIS) Technical Working Group members for their leadership in designing, developing, and deploying the NIIS. The authors thank the data collectors who participated in this study. Finally, the authors acknowledge the invaluable collaboration of the health workers, managers, and leaders in Hanoi and Son La provinces.

Data Availability

The data sets generated during and analyzed during this study are available from the corresponding author on reasonable request.

Conflicts of Interest

None declared.

Supplementary tables.

Dang H, Dao S, Carnahan E, Kawakyu N, Duong H, Nguyen T, et al. Determinants of scale-up from a small pilot to a national electronic immunization registry in Vietnam: qualitative evaluation. J Med Internet Res. Sep 22, 2020;22(9):e19923. [ FREE Full text ] [ CrossRef ] [ Medline ]
Yohana E, Mitiku S, Kayumba PC, Swalehe O. Electronic immunization registry in improving vaccine supply chain availability in Tanga City Council, Tanzania. Rwanda J Med Health Sci. Sep 10, 2021;4(2):223-236. [ CrossRef ]
Siddiqi DA, Abdullah S, Dharma VK, Shah MT, Akhter MA, Habib A, et al. Using a low-cost, real-time electronic immunization registry in Pakistan to demonstrate utility of data for immunization programs and evidence-based decision making to achieve SDG-3: insights from analysis of big data on vaccines. Int J Med Inform. May 2021;149:104413. [ CrossRef ] [ Medline ]
Uddin MJ, Shamsuzzaman M, Horng L, Labrique A, Vasudevan L, Zeller K, et al. Use of mobile phones for improving vaccination coverage among children living in rural hard-to-reach areas and urban streets of Bangladesh. Vaccine. Jan 04, 2016;34(2):276-283. [ FREE Full text ] [ CrossRef ] [ Medline ]
Chen L, Du X, Zhang L, van Velthoven MH, Wu Q, Yang R, et al. Effectiveness of a smartphone app on improving immunization of children in rural Sichuan Province, China: a cluster randomized controlled trial. BMC Public Health. Aug 31, 2016;16(1):909. [ FREE Full text ] [ CrossRef ] [ Medline ]
Nguyen NT, Vu HM, Dao SD, Tran HT, Nguyen TX. Digital immunization registry: evidence for the impact of mHealth on enhancing the immunization system and improving immunization coverage for children under one year old in Vietnam. Mhealth. 2017;3:26. [ FREE Full text ] [ CrossRef ] [ Medline ]
Gilbert SS, Thakare N, Ramanujapuram A, Akkihal A. Assessing stability and performance of a digitally enabled supply chain: retrospective of a pilot in Uttar Pradesh, India. Vaccine. Apr 19, 2017;35(17):2203-2208. [ FREE Full text ] [ CrossRef ] [ Medline ]
Gilbert SS, Bulula N, Yohana E, Thompson J, Beylerian E, Werner L, et al. The impact of an integrated electronic immunization registry and logistics management information system (EIR-eLMIS) on vaccine availability in three regions in Tanzania: a pre-post and time-series analysis. Vaccine. Jan 16, 2020;38(3):562-569. [ FREE Full text ] [ CrossRef ] [ Medline ]
Fritz J, Herrick T, Gilbert SS. Estimation of health impact from digitalizing last-mile logistics management information systems (LMIS) in Ethiopia, Tanzania, and Mozambique: a lives saved tool (LiST) model analysis. PLoS One. 2021;16(10):e0258354. [ FREE Full text ] [ CrossRef ] [ Medline ]
Carnahan E, Ferriss E, Beylerian E, Mwansa FD, Bulula N, Lyimo D, et al. Determinants of facility-level use of electronic immunization registries in Tanzania and Zambia: an observational analysis. Glob Health Sci Pract. Sep 30, 2020;8(3):488-504. [ FREE Full text ] [ CrossRef ] [ Medline ]
Mvundura M, Di Giorgio L, Lymo D, Mwansa FD, Ngwegwe B, Werner L. The costs of developing, deploying and maintaining electronic immunisation registries in Tanzania and Zambia. BMJ Glob Health. Nov 25, 2019;4(6):e001904. [ FREE Full text ] [ CrossRef ] [ Medline ]
Mvundura M, Di Giorgio L, Vodicka E, Kindoli R, Zulu C. Assessing the incremental costs and savings of introducing electronic immunization registries and stock management systems: evidence from the better immunization data initiative in Tanzania and Zambia. Pan Afr Med J. Feb 12, 2020;35(Suppl 1):11. [ FREE Full text ] [ CrossRef ] [ Medline ]
World Health Organization, PATH. Optimize: Vietnam report. PATH. 2013. URL: https://media.path.org/documents/TS_opt_vietnam_rpt.pdf [accessed 2024-01-29]
Duong H, Dao S, Dang H, Nguyen L, Ngo T, Nguyen T, et al. The transition to an entirely digital immunization registry in Ha Noi province and Son La Province, Vietnam: readiness assessment study. JMIR Form Res. Oct 25, 2021;5(10):e28096. [ FREE Full text ] [ CrossRef ] [ Medline ]
Minh An DT, Lee JK, Van Minh HV, Trang NT, Thu Huong NT, Nam YS, et al. Timely immunization completion among children in Vietnam from 2000 to 2011: a multilevel analysis of individual and contextual factors. Glob Health Action. Mar 01, 2016;9(1):29189. [ FREE Full text ] [ CrossRef ] [ Medline ]
Vietnam: country information. Gavi, The Vaccine Alliance. URL: https://www.gavi.org/programmes-impact/country-hub/west-pacific/vietnam [accessed 2022-09-28]
4/TT-BYT on establishment, functions, tasks and rights of research ethics committees, 05 March 2020. Vietnam Ministry of Health. URL: https://thuvienphapluat.vn/van-ban/Bo-may-hanh-chinh/Thong-tu-4-TT-BYT-2020-thanh-lap-chuc-nang-nhiem-vu-Hoi-dong-dao-duc-trong-nghien-cuu-y-sinh-hoc-440446.aspx [accessed 2024-01-25]
Home page. Code of Federal Regulations. URL: https://www.ecfr.gov/current/title-45/subtitle-A/subchapter-A/part-46/subpart-A/section-46.102 [accessed 2024-01-25]
Werner L, Seymour D, Puta C, Gilbert S. Three waves of data use among health workers: the experience of the better immunization data initiative in Tanzania and Zambia. Glob Health Sci Pract. Sep 2019;7(3):447-456. [ FREE Full text ] [ CrossRef ] [ Medline ]
National immunization program and vaccine delivery the national expanded program on immunization. The World Bank Document. URL: https://openknowledge.worldbank.org/server/api/core/bitstreams/022f4737-59de-4f41-960f-e82f05093402/content [accessed 2024-02-27]

Abbreviations

Edited by T Leung, T de Azevedo Cardoso; submitted 14.12.22; peer-reviewed by V Horner, N Mejia, M Muhonde; comments to author 07.06.23; revised version received 28.07.23; accepted 26.01.24; published 18.03.24.

©Thi Thanh Huyen Dang, Emily Carnahan, Linh Nguyen, Mercy Mvundura, Sang Dao, Thi Hong Duong, Trung Nguyen, Doan Nguyen, Tu Nguyen, Laurie Werner, Tove K Ryman, Nga Nguyen. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 18.03.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

IMAGES

Types of Number System: Definition, Conversion, Equations & Examples
What is Number System and Binary Number System. IGNOU
PPT ON NUMBER SYSTEM
NUMBER SYSTEM (PART 1)
Number System
The Real Number System Notes

VIDEO

types of number definition chart paper on # NUMBER SYSTEM
12th class EVS(Environment) Pre board paper 2024 |January#exam#Pseb Full solved paper Full solution
student of paper number from prantes 😂//viral//short//
Backing Paper number one quality available at Aarohi Sewing Enterprises
Number System pyqs 2023 CSAT paper part-1, 2024
6- Results section, 2nd part: Series of "How to write a scientific paper"

COMMENTS

Development of number sense and numeration: a continuum hypothesis
Consequently, this paper initially defines number sense as understood within the context of the study . Subsequently , an updated synthesis of the literature review done in the author's thesis ...
Automatic Number Plate Recognition:A Detailed Survey of Relevant
Conclusions and Future Research. This paper presents a detailed survey on ANPR algorithms proposed and experimented in recent relevant studies. ... Fujimoto K., Horino M. Development of vehicle-license number recognition system using real-time image processing and its application to travel-time measurement; Proceedings of the 41st IEEE ...
(PDF) Transformation of number system
This number system wo rks similar to our base 10 system, only we have used alphabets instead of num erals and we have increased. our digits up-to 16, every number sy stem has its own rules of ...
PDF Number Systems for Deep Neural Network Architectures: A Survey
hardware design of DNNs is considered. In addition, this paper highlights the challenges associated with each number system and various solutions that are proposed for addressing them. The reader will be able to understand the importance of an efficient number system for DNN, learn about the widely used
Number Systems for Deep Neural Network Architectures: A Survey
Download a PDF of the paper titled Number Systems for Deep Neural Network Architectures: A Survey, by Ghada Alsuhli and 5 other authors ... Using conventional number systems has been found to be sub-optimal for DNNs. Alternatively, a great body of research focuses on exploring suitable number systems. This article aims to provide a ...
(PDF) The Real Number System
This conception was crucial for Weierstrass, Dedekind and Cantor, the central (German) figures in the rigorization of the real number system. Each one of them presented a rigorous definition of ...
Emerging Residue Number System Technologies and Applications
Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications. ... The Residue Number System (RNS ...
PDF Editorial: Approximate Number System and Mathematics
The cognitive system that supports this intuitive skill is often referred to as the approximate number system (ANS). It has been argued that the ANS serves as the foundation of the formal symbolic number system—mathematics (Dehaene, 1997). Abundant empirical evidence is supportive of this view: acuity of the ANS is positively correlated with ...
Frontiers
The cognitive system that supports this intuitive skill is often referred to as the approximate number system (ANS). It has been argued that the ANS serves as the foundation of the formal symbolic number system—mathematics ( Dehaene, 1997 ). Abundant empirical evidence is supportive of this view: acuity of the ANS is positively correlated ...
PDF Number Systems and Number Representation
Learn the basics of number systems and number representation in this lecture from Princeton University's COS 217 course. Topics include binary, hexadecimal, and octal systems, unsigned and signed integers, and rational numbers. This lecture is from spring 2015 and may differ slightly from other versions of the course.
Sensors
Technologies and services towards smart-vehicles and Intelligent-Transportation-Systems (ITS), continues to revolutionize many aspects of human life. This paper presents a detailed survey of current techniques and advancements in Automatic-Number-Plate-Recognition (ANPR) systems, with a comprehensive performance comparison of various real-time tested and simulated algorithms, including those ...
PDF NUMBER THEORY IN CRYPTOGRAPHY
them. Number theory is crucial to their existence, and this paper will begin by providing the necessary background in this eld to be able to understand the material. Contents 1. Introduction1 2. Number Theory Background1 2.1. Basic Principles1 2.2. De nitions and Theorems to Know2 3. RSA Encryption4 3.1. Background4 3.2. Attacks6 4. Di e-Helman ...
Automatic Number Plate Recognition
The proposed system first detects the vehicle and then captures the vehicle image. Vehicle number plate region is extracted using the image segmentation and characters are recognized using optical character recognition technique. The system can handle noisy, low illuminated, cross angled, nonstandard font number plates.
Automatic Number Plate Recognition
In this given research paper, a full algorithm and network flow for ANPR and its efficient applications are shown. The concept of ANPR system is based on the matching of templates and exactness (result) of this system was established as 75-85% for Indian number plates.
Number Systems: A Path into Rigorous Mathematics
Abstract. Mathematics is known for its rigor and beauty. While not everyone is able to appreciate the beauty that mathematics possesses, the rigor of mathematics is easily discernible. The book ...
PDF The real number system is not dense rather than it maintains gap theory
The statement that the real number system is dense is wrong. That is density property of real numbers is wrong. If we take two consecutive real numbers a,b then we can get a third real no (a+b)/2=c in between a,b . But we will get two gaps in between a,c and c,b. So we can say real numbers maintains gap theory instead of density property.
Numbered References
This section describes the numbered reference system and gives examples from one version of the system.* Ask your instructor what specific style to use. Documentation styles for chemistry, computer science, mathematics, physics, and medical sciences are summarized in James D. Lester, Writing Research Papers, 5th ed., pp. 231-237, available in the Writing Center.
Number System: In-Text Citation
Number System: In-Text Citation. Generally, the number system is favored in fields where you typically report experimental field or lab work. Technical fields such as materials science, aerospace engineering, and chemistry tend to favor the number system. When you use the number system, your responsibility is to indicate in your text—either ...
(PDF) Binary Number System
Now, in order to convert binary number number of two's; the third, the number of fours; the to decimal, first of all, Write down the binary number fourth the number of eights, and so on. Examples of and list the powers of 2 from left to right. Then, binary numbers are: -100,101,110 and so on.
Evolution Of Number Systems Research Paper
1. The Term 'Number System'. A 'number system' consists of a set of mental entities and operations together with systems of symbols such as number words, tallies, or numerical sign systems and of symbolic activities such as counting, addition, or multiplication representing them. Number systems evolved in the course of the development ...
Developing understanding of number system structure from ...
In the conclusion of the paper, an overview of number systems through history is summarized (Part I, NORMA XXVII, 1/2022), with special reference to the importance of positional number system, i.e ...
TacticAI: an AI assistant for football tactics
In modern football games, data-driven analysis serves as a key driver in determining tactics. Wang, Veličković, Hennes et al. develop a geometric deep learning algorithm, named TacticAI, to ...
Global, regional, and national burden of disorders affecting the
As the leading cause of overall disease burden in the world, with increasing global DALY counts, effective prevention, treatment, and rehabilitation strategies for disorders affecting the nervous system are needed.
NVIDIA Blackwell Platform Arrives to Power a New Era of Computing
The GB200 NVL72 provides up to a 30x performance increase compared to the same number of NVIDIA H100 Tensor Core GPUs for LLM inference workloads, and reduces cost and energy consumption by up to 25x. The platform acts as a single GPU with 1.4 exaflops of AI performance and 30TB of fast memory, and is a building block for the newest DGX SuperPOD.
(PDF) Number Theory
Abstract and Figures. Number theory is a branch of mathematics that is primarily focused on the study of positive integers, or natural numbers, and their properties such as divisibility, prime ...
Journal of Medical Internet Research
Background: The electronic National Immunization Information System (NIIS) was introduced nationwide in Vietnam in 2017. Health workers were expected to use the NIIS alongside the legacy paper-based system. Starting in 2018, Hanoi and Son La provinces transitioned to paperless reporting. Interventions to support this transition included data guidelines and training, internet-based data review ...
135264 PDFs
The study of properties of integers and prime numbers. | Explore the latest full-text research PDFs, articles, conference papers, preprints and more on NUMBER THEORY. Find methods information ...

Computer Science > Neural and Evolutionary Computing

Submission history

References & Citations

BibTeX formatted citation

Bibliographic and Citation Tools

arXivLabs: experimental projects with community collaborators

EDITORIAL article

Author Contributions

Conflict of Interest Statement

Automatic Number Plate Recognition

Purchase Details

Profile Information

Numbered References

Create a reference list

Book (1): Single author

Book (2): 2 authors

Book (3): Editor in place of author

Book (4): 2nd or later edition

Book (5): Volume in a multivolume work

Book (6): Subtitled volume of multivolume work

Book (7): Article in edited collection

Book (8): Corporate author

Journal Article (1): Single author

Journal Article (2): More than 1 author

Journal Article (3): In edited collection

Create in-text citations

Number your citations

Format your citations

Include a page number

Make the citation part of your sentence

Cite References in Your Paper

Number System: In-Text Citation

Other In-Text Citation Practices

Binary Number System

Related Papers

RELATED PAPERS

Evolution Of Number Systems Research Paper

1. The Term ‘Number System’

Academic Writing, Editing, Proofreading, And Problem Solving Services

2. Cultures Without Number Systems

3. Proto-Arithmetic

4. From Proto-Arithmetic To Systems Of Numerical Symbols

5. Systems Of Numerical Symbols In Early Civilizations

6. The Euclidean Conceptualization Of Number

7. The Impact Of Algebra

8. Formalism And The Decline Of Number

ORDER HIGH QUALITY CUSTOM PAPER

TacticAI: an AI assistant for football tactics

Similar content being viewed by others

Automated recognition of the cricket batting backlift technique in video footage using deep learning architectures

Synchronization of passes in event and spatiotemporal soccer data

A Random Forest approach to identify metrics that best predict match outcome and player ranking in the esport Rocket League

Introduction

Benchmarking TacticAI

Accurate receiver and shot prediction through geometric deep learning

Controlled tactic refinement using class-conditional generative models

Case study with expert raters

Raw corner kick data

Graph representation and construction

Benchmark tasks construction

Graph neural networks

Geometric deep learning

Frame averaging

Group convolutions

Network architectures

Ablation study

Training details

Data availability

Code availability

Acknowledgements

Author information

Authors and Affiliations

Contributions

Corresponding authors

Ethics declarations

Peer review

Additional information

Supplementary information

About this article

Share this article