ensure reliability qualitative research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Family Med Prim Care
• v.4(3); Jul-Sep 2015

Validity, reliability, and generalizability in qualitative research

Lawrence leung.

1 Department of Family Medicine, Queen's University, Kingston, Ontario, Canada

2 Centre of Studies in Primary Care, Queen's University, Kingston, Ontario, Canada

In general practice, qualitative research contributes as significantly as quantitative research, in particular regarding psycho-social aspects of patient-care, health services provision, policy setting, and health administrations. In contrast to quantitative research, qualitative research as a whole has been constantly critiqued, if not disparaged, by the lack of consensus for assessing its quality and robustness. This article illustrates with five published studies how qualitative research can impact and reshape the discipline of primary care, spiraling out from clinic-based health screening to community-based disease monitoring, evaluation of out-of-hours triage services to provincial psychiatric care pathways model and finally, national legislation of core measures for children's healthcare insurance. Fundamental concepts of validity, reliability, and generalizability as applicable to qualitative research are then addressed with an update on the current views and controversies.

Nature of Qualitative Research versus Quantitative Research

The essence of qualitative research is to make sense of and recognize patterns among words in order to build up a meaningful picture without compromising its richness and dimensionality. Like quantitative research, the qualitative research aims to seek answers for questions of “how, where, when who and why” with a perspective to build a theory or refute an existing theory. Unlike quantitative research which deals primarily with numerical data and their statistical interpretations under a reductionist, logical and strictly objective paradigm, qualitative research handles nonnumerical information and their phenomenological interpretation, which inextricably tie in with human senses and subjectivity. While human emotions and perspectives from both subjects and researchers are considered undesirable biases confounding results in quantitative research, the same elements are considered essential and inevitable, if not treasurable, in qualitative research as they invariable add extra dimensions and colors to enrich the corpus of findings. However, the issue of subjectivity and contextual ramifications has fueled incessant controversies regarding yardsticks for quality and trustworthiness of qualitative research results for healthcare.

Impact of Qualitative Research upon Primary Care

In many ways, qualitative research contributes significantly, if not more so than quantitative research, to the field of primary care at various levels. Five qualitative studies are chosen to illustrate how various methodologies of qualitative research helped in advancing primary healthcare, from novel monitoring of chronic obstructive pulmonary disease (COPD) via mobile-health technology,[ 1 ] informed decision for colorectal cancer screening,[ 2 ] triaging out-of-hours GP services,[ 3 ] evaluating care pathways for community psychiatry[ 4 ] and finally prioritization of healthcare initiatives for legislation purposes at national levels.[ 5 ] With the recent advances of information technology and mobile connecting device, self-monitoring and management of chronic diseases via tele-health technology may seem beneficial to both the patient and healthcare provider. Recruiting COPD patients who were given tele-health devices that monitored lung functions, Williams et al. [ 1 ] conducted phone interviews and analyzed their transcripts via a grounded theory approach, identified themes which enabled them to conclude that such mobile-health setup and application helped to engage patients with better adherence to treatment and overall improvement in mood. Such positive findings were in contrast to previous studies, which opined that elderly patients were often challenged by operating computer tablets,[ 6 ] or, conversing with the tele-health software.[ 7 ] To explore the content of recommendations for colorectal cancer screening given out by family physicians, Wackerbarth, et al. [ 2 ] conducted semi-structure interviews with subsequent content analysis and found that most physicians delivered information to enrich patient knowledge with little regard to patients’ true understanding, ideas, and preferences in the matter. These findings suggested room for improvement for family physicians to better engage their patients in recommending preventative care. Faced with various models of out-of-hours triage services for GP consultations, Egbunike et al. [ 3 ] conducted thematic analysis on semi-structured telephone interviews with patients and doctors in various urban, rural and mixed settings. They found that the efficiency of triage services remained a prime concern from both users and providers, among issues of access to doctors and unfulfilled/mismatched expectations from users, which could arouse dissatisfaction and legal implications. In UK, a care pathways model for community psychiatry had been introduced but its benefits were unclear. Khandaker et al. [ 4 ] hence conducted a qualitative study using semi-structure interviews with medical staff and other stakeholders; adopting a grounded-theory approach, major themes emerged which included improved equality of access, more focused logistics, increased work throughput and better accountability for community psychiatry provided under the care pathway model. Finally, at the US national level, Mangione-Smith et al. [ 5 ] employed a modified Delphi method to gather consensus from a panel of nominators which were recognized experts and stakeholders in their disciplines, and identified a core set of quality measures for children's healthcare under the Medicaid and Children's Health Insurance Program. These core measures were made transparent for public opinion and later passed on for full legislation, hence illustrating the impact of qualitative research upon social welfare and policy improvement.

Overall Criteria for Quality in Qualitative Research

Given the diverse genera and forms of qualitative research, there is no consensus for assessing any piece of qualitative research work. Various approaches have been suggested, the two leading schools of thoughts being the school of Dixon-Woods et al. [ 8 ] which emphasizes on methodology, and that of Lincoln et al. [ 9 ] which stresses the rigor of interpretation of results. By identifying commonalities of qualitative research, Dixon-Woods produced a checklist of questions for assessing clarity and appropriateness of the research question; the description and appropriateness for sampling, data collection and data analysis; levels of support and evidence for claims; coherence between data, interpretation and conclusions, and finally level of contribution of the paper. These criteria foster the 10 questions for the Critical Appraisal Skills Program checklist for qualitative studies.[ 10 ] However, these methodology-weighted criteria may not do justice to qualitative studies that differ in epistemological and philosophical paradigms,[ 11 , 12 ] one classic example will be positivistic versus interpretivistic.[ 13 ] Equally, without a robust methodological layout, rigorous interpretation of results advocated by Lincoln et al. [ 9 ] will not be good either. Meyrick[ 14 ] argued from a different angle and proposed fulfillment of the dual core criteria of “transparency” and “systematicity” for good quality qualitative research. In brief, every step of the research logistics (from theory formation, design of study, sampling, data acquisition and analysis to results and conclusions) has to be validated if it is transparent or systematic enough. In this manner, both the research process and results can be assured of high rigor and robustness.[ 14 ] Finally, Kitto et al. [ 15 ] epitomized six criteria for assessing overall quality of qualitative research: (i) Clarification and justification, (ii) procedural rigor, (iii) sample representativeness, (iv) interpretative rigor, (v) reflexive and evaluative rigor and (vi) transferability/generalizability, which also double as evaluative landmarks for manuscript review to the Medical Journal of Australia. Same for quantitative research, quality for qualitative research can be assessed in terms of validity, reliability, and generalizability.

Validity in qualitative research means “appropriateness” of the tools, processes, and data. Whether the research question is valid for the desired outcome, the choice of methodology is appropriate for answering the research question, the design is valid for the methodology, the sampling and data analysis is appropriate, and finally the results and conclusions are valid for the sample and context. In assessing validity of qualitative research, the challenge can start from the ontology and epistemology of the issue being studied, e.g. the concept of “individual” is seen differently between humanistic and positive psychologists due to differing philosophical perspectives:[ 16 ] Where humanistic psychologists believe “individual” is a product of existential awareness and social interaction, positive psychologists think the “individual” exists side-by-side with formation of any human being. Set off in different pathways, qualitative research regarding the individual's wellbeing will be concluded with varying validity. Choice of methodology must enable detection of findings/phenomena in the appropriate context for it to be valid, with due regard to culturally and contextually variable. For sampling, procedures and methods must be appropriate for the research paradigm and be distinctive between systematic,[ 17 ] purposeful[ 18 ] or theoretical (adaptive) sampling[ 19 , 20 ] where the systematic sampling has no a priori theory, purposeful sampling often has a certain aim or framework and theoretical sampling is molded by the ongoing process of data collection and theory in evolution. For data extraction and analysis, several methods were adopted to enhance validity, including 1 st tier triangulation (of researchers) and 2 nd tier triangulation (of resources and theories),[ 17 , 21 ] well-documented audit trail of materials and processes,[ 22 , 23 , 24 ] multidimensional analysis as concept- or case-orientated[ 25 , 26 ] and respondent verification.[ 21 , 27 ]

Reliability

In quantitative research, reliability refers to exact replicability of the processes and the results. In qualitative research with diverse paradigms, such definition of reliability is challenging and epistemologically counter-intuitive. Hence, the essence of reliability for qualitative research lies with consistency.[ 24 , 28 ] A margin of variability for results is tolerated in qualitative research provided the methodology and epistemological logistics consistently yield data that are ontologically similar but may differ in richness and ambience within similar dimensions. Silverman[ 29 ] proposed five approaches in enhancing the reliability of process and results: Refutational analysis, constant data comparison, comprehensive data use, inclusive of the deviant case and use of tables. As data were extracted from the original sources, researchers must verify their accuracy in terms of form and context with constant comparison,[ 27 ] either alone or with peers (a form of triangulation).[ 30 ] The scope and analysis of data included should be as comprehensive and inclusive with reference to quantitative aspects if possible.[ 30 ] Adopting the Popperian dictum of falsifiability as essence of truth and science, attempted to refute the qualitative data and analytes should be performed to assess reliability.[ 31 ]

Generalizability

Most qualitative research studies, if not all, are meant to study a specific issue or phenomenon in a certain population or ethnic group, of a focused locality in a particular context, hence generalizability of qualitative research findings is usually not an expected attribute. However, with rising trend of knowledge synthesis from qualitative research via meta-synthesis, meta-narrative or meta-ethnography, evaluation of generalizability becomes pertinent. A pragmatic approach to assessing generalizability for qualitative studies is to adopt same criteria for validity: That is, use of systematic sampling, triangulation and constant comparison, proper audit and documentation, and multi-dimensional theory.[ 17 ] However, some researchers espouse the approach of analytical generalization[ 32 ] where one judges the extent to which the findings in one study can be generalized to another under similar theoretical, and the proximal similarity model, where generalizability of one study to another is judged by similarities between the time, place, people and other social contexts.[ 33 ] Thus said, Zimmer[ 34 ] questioned the suitability of meta-synthesis in view of the basic tenets of grounded theory,[ 35 ] phenomenology[ 36 ] and ethnography.[ 37 ] He concluded that any valid meta-synthesis must retain the other two goals of theory development and higher-level abstraction while in search of generalizability, and must be executed as a third level interpretation using Gadamer's concepts of the hermeneutic circle,[ 38 , 39 ] dialogic process[ 38 ] and fusion of horizons.[ 39 ] Finally, Toye et al. [ 40 ] reported the practicality of using “conceptual clarity” and “interpretative rigor” as intuitive criteria for assessing quality in meta-ethnography, which somehow echoed Rolfe's controversial aesthetic theory of research reports.[ 41 ]

Food for Thought

Despite various measures to enhance or ensure quality of qualitative studies, some researchers opined from a purist ontological and epistemological angle that qualitative research is not a unified, but ipso facto diverse field,[ 8 ] hence any attempt to synthesize or appraise different studies under one system is impossible and conceptually wrong. Barbour argued from a philosophical angle that these special measures or “technical fixes” (like purposive sampling, multiple-coding, triangulation, and respondent validation) can never confer the rigor as conceived.[ 11 ] In extremis, Rolfe et al. opined from the field of nursing research, that any set of formal criteria used to judge the quality of qualitative research are futile and without validity, and suggested that any qualitative report should be judged by the form it is written (aesthetic) and not by the contents (epistemic).[ 41 ] Rolfe's novel view is rebutted by Porter,[ 42 ] who argued via logical premises that two of Rolfe's fundamental statements were flawed: (i) “The content of research report is determined by their forms” may not be a fact, and (ii) that research appraisal being “subject to individual judgment based on insight and experience” will mean those without sufficient experience of performing research will be unable to judge adequately – hence an elitist's principle. From a realism standpoint, Porter then proposes multiple and open approaches for validity in qualitative research that incorporate parallel perspectives[ 43 , 44 ] and diversification of meanings.[ 44 ] Any work of qualitative research, when read by the readers, is always a two-way interactive process, such that validity and quality has to be judged by the receiving end too and not by the researcher end alone.

In summary, the three gold criteria of validity, reliability and generalizability apply in principle to assess quality for both quantitative and qualitative research, what differs will be the nature and type of processes that ontologically and epistemologically distinguish between the two.

Source of Support: Nil.

Conflict of Interest: None declared.

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 18, Issue 2
• Issues of validity and reliability in qualitative research
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Helen Noble 1 ,
• Joanna Smith 2
• 1 School of Nursing and Midwifery, Queens's University Belfast , Belfast , UK
• 2 School of Human and Health Sciences, University of Huddersfield , Huddersfield , UK
• Correspondence to Dr Helen Noble School of Nursing and Midwifery, Queens's University Belfast, Medical Biology Centre, 97 Lisburn Rd, Belfast BT9 7BL, UK; helen.noble{at}qub.ac.uk

https://doi.org/10.1136/eb-2015-102054

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Evaluating the quality of research is essential if findings are to be utilised in practice and incorporated into care delivery. In a previous article we explored ‘bias’ across research designs and outlined strategies to minimise bias. 1 The aim of this article is to further outline rigour, or the integrity in which a study is conducted, and ensure the credibility of findings in relation to qualitative research. Concepts such as reliability, validity and generalisability typically associated with quantitative research and alternative terminology will be compared in relation to their application to qualitative research. In addition, some of the strategies adopted by qualitative researchers to enhance the credibility of their research are outlined.

Are the terms reliability and validity relevant to ensuring credibility in qualitative research?

Although the tests and measures used to establish the validity and reliability of quantitative research cannot be applied to qualitative research, there are ongoing debates about whether terms such as validity, reliability and generalisability are appropriate to evaluate qualitative research. 2–4 In the broadest context these terms are applicable, with validity referring to the integrity and application of the methods undertaken and the precision in which the findings accurately reflect the data, while reliability describes consistency within the employed analytical procedures. 4 However, if qualitative methods are inherently different from quantitative methods in terms of philosophical positions and purpose, then alterative frameworks for establishing rigour are appropriate. 3 Lincoln and Guba 5 offer alternative criteria for demonstrating rigour within qualitative research namely truth value, consistency and neutrality and applicability. Table 1 outlines the differences in terminology and criteria used to evaluate qualitative research.

• View inline

Terminology and criteria used to evaluate the credibility of research findings

What strategies can qualitative researchers adopt to ensure the credibility of the study findings?

Unlike quantitative researchers, who apply statistical methods for establishing validity and reliability of research findings, qualitative researchers aim to design and incorporate methodological strategies to ensure the ‘trustworthiness’ of the findings. Such strategies include:

Accounting for personal biases which may have influenced findings; 6

Acknowledging biases in sampling and ongoing critical reflection of methods to ensure sufficient depth and relevance of data collection and analysis; 3

Meticulous record keeping, demonstrating a clear decision trail and ensuring interpretations of data are consistent and transparent; 3 , 4

Establishing a comparison case/seeking out similarities and differences across accounts to ensure different perspectives are represented; 6 , 7

Including rich and thick verbatim descriptions of participants’ accounts to support findings; 7

Demonstrating clarity in terms of thought processes during data analysis and subsequent interpretations 3 ;

Engaging with other researchers to reduce research bias; 3

Respondent validation: includes inviting participants to comment on the interview transcript and whether the final themes and concepts created adequately reflect the phenomena being investigated; 4

Data triangulation, 3 , 4 whereby different methods and perspectives help produce a more comprehensive set of findings. 8 , 9

Table 2 provides some specific examples of how some of these strategies were utilised to ensure rigour in a study that explored the impact of being a family carer to patients with stage 5 chronic kidney disease managed without dialysis. 10

Strategies for enhancing the credibility of qualitative research

In summary, it is imperative that all qualitative researchers incorporate strategies to enhance the credibility of a study during research design and implementation. Although there is no universally accepted terminology and criteria used to evaluate qualitative research, we have briefly outlined some of the strategies that can enhance the credibility of study findings.

• Sandelowski M
• Lincoln YS ,
• Barrett M ,
• Mayan M , et al
• Greenhalgh T
• Lingard L ,

Twitter Follow Joanna Smith at @josmith175 and Helen Noble at @helnoble

Competing interests None.

Read the full text or download the PDF:

Articles and blog posts

Validity and reliability in qualitative research.

What is Validity and Reliability in Qualitative research?

In Quantitative research, reliability refers to consistency of certain measurements, and validity – to whether these measurements “measure what they are supposed to measure”. Things are slightly different, however, in Qualitative research.

Reliability in qualitative studies is mostly a matter of “being thorough, careful and honest in carrying out the research” (Robson, 2002: 176). In qualitative interviews, this issue relates to a number of practical aspects of the process of interviewing, including the wording of interview questions, establishing rapport with the interviewees and considering ‘power relationship’ between the interviewer and the participant (e.g. Breakwell, 2000; Cohen et al., 2007; Silverman, 1993).

What seems more relevant when discussing qualitative studies is their validity , which very often is being addressed with regard to three common threats to validity in qualitative studies, namely researcher bias , reactivity and respondent bias (Lincoln and Guba, 1985).

Researcher bias refers to any kind of negative influence of the researcher’s knowledge, or assumptions, of the study, including the influence of his or her assumptions of the design, analysis or, even, sampling strategy. Reactivity , in turn, refers to a possible influence of the researcher himself/herself on the studied situation and people. Respondent bias refers to a situation where respondents do not provide honest responses for any reason, which may include them perceiving a given topic as a threat, or them being willing to ‘please’ the researcher with responses they believe are desirable.

Robson (2002) suggested a number of strategies aimed at addressing these threats to validity, being prolonged involvement , triangulation , peer debriefing , member checking ,  negative case analysis  and keeping an audit trail .

So, what exactly are these strategies and how can you apply them in your research?

Prolonged involvement refers to the length of time of the researcher’s involvement in the study, including involvement with the environment and the studied participants. It may be granted, for example, by the duration of the study, or by the researcher belonging to the studied community (e.g. a student investigating other students’ experiences). Being a member of this community, or even being a friend to your participants (see my blog post on the ethics of researching friends ), may be a great advantage and a factor that both increases the level of trust between you, the researcher, and the participants and the possible threats of reactivity and respondent bias. It may, however, pose a threat in the form of researcher bias that stems from your, and the participants’, possible assumptions of similarity and presuppositions about some shared experiences (thus, for example, they will not say something in the interview because they will assume that both of you know it anyway – this way, you may miss some valuable data for your study).

Triangulation may refer to triangulation of data through utilising different instruments of data collection, methodological triangulation through employing mixed methods approach and theory triangulation through comparing different theories and perspectives with your own developing “theory” or through drawing from a number of different fields of study.

Peer debriefing and support is really an element of your student experience at the university throughout the process of the study. Various opportunities to present and discuss your research at its different stages, either at internally organised events at your university (e.g. student presentations, workshops, etc.) or at external conferences (which I strongly suggest that you start attending) will provide you with valuable feedback, criticism and suggestions for improvement. These events are invaluable in helping you to asses the study from a more objective, and critical, perspective and to recognise and address its limitations. This input, thus, from other people helps to reduce the researcher bias.

Member checking , or testing the emerging findings with the research participants, in order to increase the validity of the findings, may take various forms in your study. It may involve, for example, regular contact with the participants throughout the period of the data collection and analysis and verifying certain interpretations and themes resulting from the analysis of the data (Curtin and Fossey, 2007). As a way of controlling the influence of your knowledge and assumptions on the emerging interpretations, if you are not clear about something a participant had said, or written, you may send him/her a request to verify either what he/she meant or the interpretation you made based on that. Secondly, it is common to have a follow-up, “validation interview” that is, in itself, a tool for validating your findings and verifying whether they could be applied to individual participants (Buchbinder, 2011), in order to determine outlying, or negative, cases and to re-evaluate your understanding of a given concept (see further below). Finally, member checking, in its most commonly adopted form, may be carried out by sending the interview transcripts to the participants and asking them to read them and provide any necessary comments or corrections (Carlson, 2010).

Negative case analysis is a process of analysing ‘cases’, or sets of data collected from a single participant, that do not match the patterns emerging from the rest of the data. Whenever an emerging explanation of a given phenomenon you are investigating does nto seem applicable to one, or a small number, of the participants, you should try to carry out a new line of analysis aimed at understanding the source of this discrepancy. Although you may be tempted to ignore these “cases” in fear of having to do extra work, it should become your habit to explore them in detail, as the strategy of negative case analysis, especially when combined with member checking, is a valuable way of reducing researcher bias.

Finally, the notion of keeping an audit trail refers to monitoring and keeping a record of all the research-related activities and data, including the raw interview and journal data, the audio-recordings, the researcher’s diary (see this post about recommended software for researcher’s diary ) and the coding book.

If you adopt the above strategies skilfully, you are likely to minimize threats to validity of your study. Don’t forget to look at the resources in the reference list, if you would like to read more on this topic!

Breakwell, G. M. (2000). Interviewing. In Breakwell, G.M., Hammond, S. & Fife-Shaw, C. (eds.) Research Methods in Psychology. 2nd Ed. London: Sage. Buchbinder, E. (2011). Beyond Checking: Experiences of the Validation Interview. Qualitative Social Work, 10 (1), 106-122. Carlson, J.A. (2010). Avoiding Traps in Member Checking. The Qualitative Report, 15 (5), 1102-1113. Cohen, L., Manion, L., & Morrison, K. (2007). Research Methods in Education. 6th Ed. London: Routledge. Curtin, M., & Fossey, E. (2007). Appraising the trustworthiness of qualitative studies: Guidelines for occupational therapists. Australian Occupational Therapy Journal, 54, 88-94. Lincoln, Y. S. & Guba, E. G. (1985). Naturalistic Inquiry. Newbury Park, CA: SAGE. Robson, C. (2002). Real world research: a resource for social scientists and practitioner-researchers. Oxford, UK: Blackwell Publishers.

Silverman, D. (1993) Interpreting Qualitative Data. London: Sage.

There is an argument for using your identity and biases to enrich the research (see my recent blog… researcheridentity.wordpress.com) providing that the researcher seeks to fully comprehend their place in the research and is fully open, honest and clear about that in the write up. I have come to see reliability and validity more as a defence of is the research rigorous, thorough and careful therefore is it morally, ethically and accurately defensible?

Hi Nathan, thank you for your comment. I agree that being explicit about your own status and everything that you bring into the study is important – it’s a very similar issue (although seemingly it’s a different topic) to what I discussed in the blog post about grounded theory where I talked about being explicit about the influence of our previous knowledge on the data. I have also experienced this dilemma of “what to do with” my status as simultaneously a “researcher” an “insider” a “friend” and a “fellow Polish migrant” when conducting my PhD study of Polish migrants’ English Language Identity, and came to similar conclusions as the ones you reach in your article – to acknowledge these “multiple identities” and make the best of them.

I have read your blog article and really liked it – would you mind if I shared it on my Facebook page, and linked to it from my blog section on this page?

Please do share my blog by all means; I’d be delighted. Are you on twitter? I’m @Nathan_AHT_EDD I strongly believe that we cannot escape our past, including our multiple/present habitus and identities when it comes to qualitative educational research. It is therefore, arguably, logical to ethically and sensibly embrace it/them to enrich the data. Identities cannot be taken on and off like a coat, they are, “lived as deeply committed personal projects” (Clegg, 2008: p.336) and so if we embrace them we bring a unique insight into the process and have a genuine investment to make the research meaningful and worthy of notice.

Hi Nathan, I don’t have twitter… I know – somehow I still haven’t had time to get to grips with it. I do have Facebook, feel free to find me there. I also started to follow your blog so that I am notified about your content. I agree with what you said here and in your posts, and I like the topic of your blog. This is definitely something that we should pay more attention to when doing research. It would be interesting to talk some time and exchange opinions, as our research interests seem very closely related. Have a good day !

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• The 4 Types of Reliability in Research | Definitions & Examples

The 4 Types of Reliability in Research | Definitions & Examples

Published on August 8, 2019 by Fiona Middleton . Revised on June 22, 2023.

Reliability tells you how consistently a method measures something. When you apply the same method to the same sample under the same conditions, you should get the same results. If not, the method of measurement may be unreliable or bias may have crept into your research.

There are four main types of reliability. Each can be estimated by comparing different sets of results produced by the same method.

Table of contents

Test-retest reliability, interrater reliability, parallel forms reliability, internal consistency, which type of reliability applies to my research, other interesting articles, frequently asked questions about types of reliability.

Test-retest reliability measures the consistency of results when you repeat the same test on the same sample at a different point in time. You use it when you are measuring something that you expect to stay constant in your sample.

Why it’s important

Many factors can influence your results at different points in time: for example, respondents might experience different moods, or external conditions might affect their ability to respond accurately.

Test-retest reliability can be used to assess how well a method resists these factors over time. The smaller the difference between the two sets of results, the higher the test-retest reliability.

How to measure it

To measure test-retest reliability, you conduct the same test on the same group of people at two different points in time. Then you calculate the correlation between the two sets of results.

Test-retest reliability example

You devise a questionnaire to measure the IQ of a group of participants (a property that is unlikely to change significantly over time).You administer the test two months apart to the same group of people, but the results are significantly different, so the test-retest reliability of the IQ questionnaire is low.

Improving test-retest reliability

• When designing tests or questionnaires , try to formulate questions, statements, and tasks in a way that won’t be influenced by the mood or concentration of participants.
• When planning your methods of data collection , try to minimize the influence of external factors, and make sure all samples are tested under the same conditions.
• Remember that changes or recall bias can be expected to occur in the participants over time, and take these into account.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Interrater reliability (also called interobserver reliability) measures the degree of agreement between different people observing or assessing the same thing. You use it when data is collected by researchers assigning ratings, scores or categories to one or more variables , and it can help mitigate observer bias .

People are subjective, so different observers’ perceptions of situations and phenomena naturally differ. Reliable research aims to minimize subjectivity as much as possible so that a different researcher could replicate the same results.

When designing the scale and criteria for data collection, it’s important to make sure that different people will rate the same variable consistently with minimal bias . This is especially important when there are multiple researchers involved in data collection or analysis.

To measure interrater reliability, different researchers conduct the same measurement or observation on the same sample. Then you calculate the correlation between their different sets of results. If all the researchers give similar ratings, the test has high interrater reliability.

Interrater reliability example

A team of researchers observe the progress of wound healing in patients. To record the stages of healing, rating scales are used, with a set of criteria to assess various aspects of wounds. The results of different researchers assessing the same set of patients are compared, and there is a strong correlation between all sets of results, so the test has high interrater reliability.

Improving interrater reliability

• Clearly define your variables and the methods that will be used to measure them.
• Develop detailed, objective criteria for how the variables will be rated, counted or categorized.
• If multiple researchers are involved, ensure that they all have exactly the same information and training.

Parallel forms reliability measures the correlation between two equivalent versions of a test. You use it when you have two different assessment tools or sets of questions designed to measure the same thing.

If you want to use multiple different versions of a test (for example, to avoid respondents repeating the same answers from memory), you first need to make sure that all the sets of questions or measurements give reliable results.

The most common way to measure parallel forms reliability is to produce a large set of questions to evaluate the same thing, then divide these randomly into two question sets.

The same group of respondents answers both sets, and you calculate the correlation between the results. High correlation between the two indicates high parallel forms reliability.

Parallel forms reliability example

A set of questions is formulated to measure financial risk aversion in a group of respondents. The questions are randomly divided into two sets, and the respondents are randomly divided into two groups. Both groups take both tests: group A takes test A first, and group B takes test B first. The results of the two tests are compared, and the results are almost identical, indicating high parallel forms reliability.

Improving parallel forms reliability

• Ensure that all questions or test items are based on the same theory and formulated to measure the same thing.

Internal consistency assesses the correlation between multiple items in a test that are intended to measure the same construct.

You can calculate internal consistency without repeating the test or involving other researchers, so it’s a good way of assessing reliability when you only have one data set.

When you devise a set of questions or ratings that will be combined into an overall score, you have to make sure that all of the items really do reflect the same thing. If responses to different items contradict one another, the test might be unreliable.

Two common methods are used to measure internal consistency.

• Average inter-item correlation : For a set of measures designed to assess the same construct, you calculate the correlation between the results of all possible pairs of items and then calculate the average.
• Split-half reliability : You randomly split a set of measures into two sets. After testing the entire set on the respondents, you calculate the correlation between the two sets of responses.

Internal consistency example

A group of respondents are presented with a set of statements designed to measure optimistic and pessimistic mindsets. They must rate their agreement with each statement on a scale from 1 to 5. If the test is internally consistent, an optimistic respondent should generally give high ratings to optimism indicators and low ratings to pessimism indicators. The correlation is calculated between all the responses to the “optimistic” statements, but the correlation is very weak. This suggests that the test has low internal consistency.

Improving internal consistency

• Take care when devising questions or measures: those intended to reflect the same concept should be based on the same theory and carefully formulated.

It’s important to consider reliability when planning your research design , collecting and analyzing your data, and writing up your research. The type of reliability you should calculate depends on the type of research  and your  methodology .

If possible and relevant, you should statistically calculate reliability and state this alongside your results .

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Quantitative research
• Ecological validity

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Reliability and validity are both about how well a method measures something:

• Reliability refers to the  consistency of a measure (whether the results can be reproduced under the same conditions).
• Validity   refers to the  accuracy of a measure (whether the results really do represent what they are supposed to measure).

If you are doing experimental research, you also have to consider the internal and external validity of your experiment.

You can use several tactics to minimize observer bias .

• Use masking (blinding) to hide the purpose of your study from all observers.
• Triangulate your data with different data collection methods or sources.
• Use multiple observers and ensure interrater reliability.
• Train your observers to make sure data is consistently recorded between them.
• Standardize your observation procedures to make sure they are structured and clear.

Reproducibility and replicability are related terms.

• A successful reproduction shows that the data analyses were conducted in a fair and honest manner.
• A successful replication shows that the reliability of the results is high.

Research bias affects the validity and reliability of your research findings , leading to false conclusions and a misinterpretation of the truth. This can have serious implications in areas like medical research where, for example, a new form of treatment may be evaluated.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Middleton, F. (2023, June 22). The 4 Types of Reliability in Research | Definitions & Examples. Scribbr. Retrieved April 10, 2024, from https://www.scribbr.com/methodology/types-of-reliability/

Is this article helpful?

Fiona Middleton

Other students also liked, reliability vs. validity in research | difference, types and examples, what is quantitative research | definition, uses & methods, data collection | definition, methods & examples, unlimited academic ai-proofreading.

✔ Document error-free in 5minutes ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

Chapter 6. Reflexivity

Introduction.

Related to epistemological issues of how we know anything about the social world, qualitative researchers understand that we the researchers can never be truly neutral or outside the study we are conducting. As observers, we see things that make sense to us and may entirely miss what is either too obvious to note or too different to comprehend. As interviewers, as much as we would like to ask questions neutrally and remain in the background, interviews are a form of conversation, and the persons we interview are responding to us . Therefore, it is important to reflect upon our social positions and the knowledges and expectations we bring to our work and to work through any blind spots that we may have. This chapter discusses the concept of reflexivity and its importance for conducting reliable qualitative research.

Reflexivity: What It Is and Why It Is Important

Remember our discussion in epistemology ? Qualitative researchers tend to question assertions of absolute fact or reality, unmediated through subject positions and subject knowledge. There are limits to what we know because we are part of the social worlds we inhabit. To use the terminology of standpoint theorists, we have a standpoint from which we observe the world just as much as anyone else. In this, we too are the blind men, and the world is our elephant. None of us are omniscient or neutral observers. Because of this epistemological standpoint, qualitative researchers value the ability to reflect upon and think hard about our own effects on our research. We call this reflexivity. Reflexivity “generally involves the self-examination of how research findings were produced, and, particularly, the role of the researcher in their construction” ( Heaton 2004:104 ).

There are many aspects of being reflexive. First, there is the simple fact that we are human beings with the limitations that come with that condition. We have likes and dislikes, biases, blind spots, preferences, and so on. If we do not take these into account, they can prevent us from being the best researcher we can be. Being reflective means, first and foremost, trying as best as possible to bracket out elements of our own character and understanding that get in the way. It is important to note that bias (in this context, at least) is not inherently wrong. It just is. Unavoidable. But by noting it, we can minimize its impact or, in some cases, help explain more clearly what it is we see or why it is that we are asking the questions we are asking. For example, I might want to communicate to my audience that I grew up poor and that I have a lot of sympathy and concern for first-generation college students as a result. This “bias” of mine motivates me to do the work I do, even as I try to ensure that it does not blind me to things I find out in the course of my research. [1]

A second aspect of being reflexive is being aware that you yourself are part of the research when you are conducting qualitative research. This is particularly true when conducting interviews, observing interactions, or participating in activities. You have a body, and it will be “read” by those in the field. You will be perceived as an insider or an outsider, as a friend or foe, as empathetic or hostile. Some of this will be wrong. People will prejudge you based on the color of your skin, your presented gender, the accent of your language. People will classify you based on the clothes you wear, and they will be more open to you if you remind them of a friendly aunt or uncle and more reserved if you remind them of someone they don’t like. This is all natural and inevitable. Your research will suffer if you do not take this into account, if you do not reflect upon how you are being read and how this might be influencing what people tell you or what they are willing to do in front of you. The flip side of this problem is that your particular body and presence will open some doors barred to other researchers. Finding sites and contexts where your presented self is a benefit rather than a burden is an important part of your individual research career. Be honest with yourself about this, and you will be more successful as a qualitative researcher. Learn to leverage yourself in your research.

The third aspect of being reflexive is related to how we communicate our work to others. Being honest with our position, as I am about my own social background and its potential impact on what I study or about how I leveraged my own position to get people to open up to me, helps our audiences evaluate what we have found. Maybe I haven’t entirely eliminated my biases or weaknesses, but by telling my audience who I am and where I potentially stand, they can take account of those biases and weaknesses in their reading of my findings. Letting them know that I wore pink when talking with older men because that made them more likely to be kind to me (a strategy acknowledged by Posselt [ 2016 ]) helps them understand the interview context. In other words, my research becomes more reliable when my own social position and the strategies I used are communicated.

Some people think being reflective is just another form of narcissistic navel-gazing. “The study is not about you!” they might cry. True, to some degree—but that also misses the point. All studies on the social world are inevitably about us as well because we are part of that social world. It is actually more dangerous to pretend that we are neutral observers, outside what we are observing. Pierre Bourdieu makes this point several times, and I think it is worth quoting him here: “The idea of a neutral science is fiction, an interested fiction which enables its authors to present a version of the dominant representation of the social world, naturalized and euphemized into a particularly misrecognizable and symbolically, therefore, particularly effective form, and to call it scientific” (quoted in Lemert 1981:278 ).

Bourdieu ( 1984 ) argues that reflective analysis is “not an epistemological scruple” but rather “an indispensable pre-condition of scientific knowledge of the object” ( 92 ). It would be narcissistic to present findings without reflection, as that would give much more weight to any findings or insights that emerge than is due.

The critics are right about one thing, however. Putting oneself at the center of the research is also inappropriate. [2] The focus should be on what is being researched, and the reflexivity is there to advance the study, not to push it aside. This issue has emerged at times when researchers from dominant social positions reflect upon their social locations vis-à-vis study participants from marginalized locations. A researcher who studies how low-income women of color experience unemployment might need to address her White, upper-class, fully employed social location, but not at the cost of crowding out the stories, lived experiences, and understandings of the women she has interviewed. This can sometimes be a delicate balance, and not everyone will agree that a person has walked it correctly.

Examples of Reflexivity in Practice

Most qualitative researchers include a positionality statement in any “methods section” of their publications. This allows readers to understand the location of the researcher, which is often helpful for gauging reliability . Many journals now require brief positionality statements as well. Here are a few examples of such statements.

The first is from an ethnographic study of elite golfers. Ceron-Anaya ( 2017 ) writes about his class, race, and gender and how these aspects of his identity and social location affected his interactions with research participants:

My own class origins, situated near the intersection between the middle and the lower-middle class, hindered cooperation in some cases. For example, the amiable interaction with one club member changed toward the end of the interview when he realized that I commonly moved about in the city by public transportation (which is a strong class indicator). He was not rude but stopped elaborating on the answers as he had been doing up to that point.…Bodily confidence is a privilege of the privileged. My subordinate position, vis-à-vis golfers, was ameliorated by my possession of cultural capital, objectified in my status of researcher/student in a western university. However, my cultural capital dwindled in its value at the invisible but firm boundary between the upper-middle and the upper class. The few contacts I made with members of the upper class produced no connections with other members of the same group, illustrating how the research process is also inserted in the symbolic and material dynamics that shape the field. ( 288 )

What did you learn from Ceron-Anaya’s reflection? If he hadn’t told you about his background, would this have made a difference in reading about elite golfers? Would the findings be different had Ceron-Anaya driven up to the club in a limousine? Is it helpful to know he came by bus?

The second example is from a study on first-generation college students. Hinz ( 2016 ) discusses both differences and similarities between herself and those she interviewed and how both could have affected the study:

I endeavored to avoid researcher bias by allowing the data to speak for itself, but my own habitus as a White, female, middle-class second-generation college student with a few years of association with Selective State [elite university] may have influenced my interpretation. Being a Selective State student at the time of the interviews provided a familiarity with the environment in which the participants were living, and an ease of communication facilitated by a shared institutional culture. And yet, not being a first-gen myself, it seemed as if I were standing on the periphery of their experience, looking in. ( 289–290 )

Note that Hinz cannot change who she is, nor should she. Being aware (reflective) that she may “stand on the periphery” of the experience of those she interviews has probably helped her listen more closely rather than assume she understands what is really going on. Do you find her more reliable given this?

These statements can be quite long, especially when found in methodological appendixes in books rather than short statements in articles. This last lengthy example comes from my own work. I try to place myself, explaining the motivations for the research I conducted at small liberal arts colleges:

I began this project out of a deep curiosity about how college graduates today were faring in an increasingly debt-ridden and unequal labor market. I was working at a small liberal arts college when I began thinking about this project and was poised to take a job at another one. During my interview for the new job, I was told that I was a good fit, because I had attended Barnard College, so I knew what the point of a liberal arts college was. I did. A small liberal arts college was a magical place. You could study anything you wanted, for no reason at all, simply for the love of it. And people would like you for it. You were surrounded by readers, by people who liked to dress up in costume and recite Shakespeare, by people who would talk deep into the night about the meaning of life or whether “beauty” existed out there, in nature, or was simply a projection of our own circumstances. My own experience at Barnard had been somewhat like that. I studied Ancient Greek and Latin, wrote an undergraduate thesis on the legal standing of Vestal Virgins in Ancient Rome, and took frequent subway rides to the Cloisters, the medieval annex of the Metropolitan Museum of Art, where I sketched the courtyard and stared at unicorn tapestries. But I also worked full-time, as a waitress at a series of hectic and demanding restaurants around the city, as a security guard for the dorm, as a babysitter for some pretty privileged professors who lived in doorman buildings along Riverside Park, and at the library (the best job by far). I also constantly worried I would not be able to finish my degree, as every year I was unsure how I would come up with the money to pay for costs of college above and beyond the tuition (which, happily, was covered by the college given my family’s low income). Indeed, the primary reason I studied the Classics was because all the books were freely available in the library. There are no modern textbooks—you just find a copy of the Iliad. There are a lot of those in a city like New York. Due to my fears, I pushed to graduate one year early, taking a degree in “Ancient Studies” instead of “Classics,” which could have led on to graduate training. From there, I went to law school, which seemed like a safe choice. I do not remember ever having a conversation with anyone about how to find a job or what kinds of job one could do with a degree in Ancient Studies. I had little to no social networks, as I had spent my time studying and working. And I was very lucky, because I graduated with almost zero debt. For years, until that job interview, I hadn’t really thought my Barnard experience had been that great or unusual. But now it was directly helping me get a job, about fifteen years after graduation. And it probably had made me a better person, whatever that means. Had I graduated with debt, however, I am not so sure that it would have been worth it. Was it, on balance, a real opportunity and benefit for poor students like me? Even now? I had a hunch of what I might find if I looked: small liberal arts colleges were unique places of opportunity for low-income first-generation working-class students who somehow managed to find and get in to one of them (no easy task). I thought that, because of their ethos, their smallness, the fact that one could not hide from professors, these colleges would do a fair job equalizing opportunities and experiences for all their students. I wanted to tell this story. But that is not the story that I found, or not entirely. While everyone benefits from the kind of education a small liberal arts college can offer, because students begin and continue so differently burdened and privileged, the advantages of the already-advantaged are amplified, potentially increasing rather than decreasing initial inequalities. That is not really a surprising story, but it is an important one to tell and to remember. Education doesn’t reduce inequality. Going to a good college doesn’t level the playing field for low-income, first-generation, working-class students. But perhaps it can help them write a book about that. ( Hurst 2019:259–261 )

What do you think? Did you learn something about the author that would help you, as a reader, understand the reasons and context for the study? Would you trust the researcher? If you said yes, why?

How to Do It

How does one become a reflective researcher? Practice! Nearly every great qualitative researcher maintains a reflexive journal (there are exceptions that prove the rule), a type of diary where they record their thinking on the research process itself. This might include writing about the research design (chapter 2), plotting out strategies for sample selection (chapter 6), or talking through what one believes can be known (chapter 3). During analysis, this journal is a place to record ideas and insights and pose questions for further reflection or follow-up studies. This journal should be highly personal. It is a place to record fears, concerns, and hopes as well. Why are you studying what you are studying? What is really motivating you? Being clear with yourself and being able to put it down in words are invaluable to the research process.

Today, there are many blogs out there on writing reflective journals, with helpful suggestions and examples. Although you may want to take a look at some of these, the form of your own journal will probably be unique. This is you, the researcher, on the page. Each of us looks different. Use the journal to interrogate your decisions and clarify your intent. If you find something during the study of note, you might want to ask yourself what led you to note that. Why do you think this “thing” is a “thing”? What about your own position, background, or researcher status that makes you take note? And asking yourself this question might lead you to think about what you did not notice. Other questions to ask yourself include the following: How do I know “that thing” I noted? So what? What does it mean? What are the implications? Who cares about this and why? Remember that doing qualitative research well is recursive , meaning that we may begin with a research design, but the steps of doing the research often loop back to the beginning. By keeping a reflective journal, you allow yourself to circle back to the beginning, to make changes to the study to keep it in line with what you are really interested in knowing.

One might also consider designing research that includes multiple investigators, particularly those who may not share your preconceptions about the study. For example, if you are studying conservative students on campus, and you yourself thoroughly identify as liberal, you might want to pair up with a researcher interested in the topic who grew up in a conservative household. If you are studying racial regimes, consider creating a racially diverse team of researchers. Or you might include in your research design a component of participatory research wherein members of the community of interest become coresearchers. Even if you can’t form a research team, you can reach out to others for feedback as you move along. Doing research can be a lonely enterprise, so finding people who will listen to you and nudge you to clarify your thinking where necessary or move you to consider an aspect you have missed is invaluable.

Finally, make it a regular part of your practice to write a paragraph reporting your perspectives, positions, values, and beliefs and how these may have influenced the research. This paragraph may be included in publications upon request.

Internal Validity

Being reflexive can help ensure that our studies are internally valid. All research must be valid to be helpful. We say a study’s findings are externally valid when they are equally true of other times, places, people. Quantitative researchers often spend a lot of time grappling with external validity , as they are often trying to demonstrate that their sample is representative of a larger population. Although we do not do that in qualitative research, we do sometimes make claims that the processes and mechanisms we uncover here, in this particular setting, are likely to be equally active in that setting over there, although there may be (will be!) contextual differences as well. Internal validity is more peculiar to qualitative research. Is your finding an accurate representation of what you are studying? Are you describing the people you are observing or interviewing as they really are? This is internal validity , and you should be able to see how this connects with the requirement of reflexivity. To the extent that you leave unexamined your own biases or preconceptions, you will fail at accurately representing those people and processes you study. Remember that “bias” here is not a moral failing in the way we commonly use bias in the nonresearch world but an inevitable product of our being social beings who inhabit social worlds, with all the various complexities surrounding that. Because of things that have happened to you, certain things (concepts, quotes, activities) might jump out at you as being particularly important. Being reflexive allows you to take a step back and grapple with the larger picture, reflecting on why you might be seeing X (which is present) but also missing Y (which is also present). It also allows you to consider what effect/impact your presence has on what you are observing or being told and to make any adjustments necessary to minimize your impact or, at the very least, to be aware of these effects and talk about them in any descriptions or presentations you make. There are other ways of ensuring internal validity (e.g., member checking , triangulation ), but being reflective is an essential component.

Advanced: Bourdieu on Reflexivity

One researcher who really tackled the issue of reflexivity was Pierre Bourdieu. [3] Known in the US primarily as a theorist, Bourdieu was a very capable and thorough researcher, who employed a variety of methods in his wide-ranging studies. Originally trained as an anthropologist, he became uncomfortable with the unreflective “outsider perspective” he was taught to follow. How was he supposed to observe and write about the various customs and rules of the people he was studying if he did not take into account his own supposedly neutral position in the observations? And even more interestingly, how could he write about customs and rules as if they were lifted from but outside of the understandings and practice of the people following them? When you say “God bless you” to someone who sneezes, are you really following a social custom that requires the prevention of illness through some performative verbal ritual of protection, or are you saying words out of reflex and habit? Bourdieu wondered what it meant that anthropologists were so ready to attribute meaning to actions that, to those performing them, were probably unconsidered. This caused him to ponder those deep epistemological questions about the possibilities of knowledge, particularly what we can know and truly understand about others. Throughout the following decades, as he developed his theories about the social world out of the deep and various studies he engaged in, he thought about the relationship between the researcher and the researched. He came to several conclusions about this relationship.

First, he argued that researchers needed to be reflective about their position vis-à-vis the object of study. The very fact that there is a subject and an object needs to be accounted for. Too often, he argued, the researcher forgets that part of the relationship, bracketing out the researcher entirely, as if what is being observed or studied exists entirely independently of the study. This can lead to false reports, as in the case where a blind man grasps the trunk of the elephant and claims the elephant is cylindrical, not having recognized how his own limitations of sight reduced the elephant to only one of its parts.

As mentioned previously, Bourdieu ( 1984 ) argued that “reflective analysis of the tools of analysis is not an epistemological scruple but an indispensable precondition of scientific knowledge of the object” ( 92 ). It is not that researchers are inherently biased—they are—but rather that the relationship between researcher and researched is an unnatural one that needs to be accounted for in the analysis. True and total objectivity is impossible, as researchers are human subjects themselves, called to research what interests them (or what interests their supervisors) and also inhabiting the social world. The solution to this problem is to be reflective and to account for these aspects in the analysis itself. Here is how Bourdieu explains this charge:

To adopt the viewpoint of REFLEXIVITY is not to renounce objectivity but to question the privilege of the knowing subject, which the antigenetic vision arbitrarily frees, as purely noetic, from the labor of objectification. To adopt this viewpoint is to strive to account for the empirical “subject” in the very terms of the objectivity constructed by the scientific subject (notably by situating it in a determined place in social space-time) and thereby to give oneself awareness and (possible) mastery of the constraints which may be exercised on the scientific subject via all the ties which attach it to the empirical “subject,” to its interests, motives, assumptions, beliefs, its doxa, and which it must break in order to constitute itself . ( 1996:207 ; emphases added)

Reflexivity, for Bourdieu, was a trained state of mind for the researcher, essential for proper knowledge production. Let’s use a story from Hans Christian Andersen to illustrate this point. If you remember this story from your childhood, it goes something like this: Two con artists show up in a town in which its chief monarch spends a lot of money on expensive clothes and splashy displays. They sense an opportunity to make some money out of this situation and pretend they are talented weavers from afar. They tell the vain emperor that they can make the most magnificent clothes anyone has ever seen (or not seen, as the case may be!). Because what they really do is “pretend” to weave and sew and hand the emperor thin air, which they then help him to put on in an elaborate joke. They tell him that only the very stupid and lowborn will be unable to see the magnificent clothes. Embarrassed that he can’t see them either, he pretends he can. Everyone pretends they can see clothes, when really the emperor walks around in his bare nakedness. As he parades through town, people redden and bow their heads, but no one says a thing. That is, until one child looks at the naked emperor and starts to laugh. His laughter breaks the spell, and everyone realizes the “naked truth.”

Now let us add a new thread to this story. The boy did not laugh. Years go by, and the emperor continues to wear his new clothes. At the start of every day, his aides carefully drape the “new clothes” around his naked body. Decades go by, and this is all “normal.” People don’t even see a naked emperor but a fully robed leader of the free world. A researcher, raised in this milieu, visits the palace to observe court habits. She observes the aides draping the emperor. She describes the care they take in doing so. She nowhere reports that the clothes are nonexistent because she herself has been trained to see them . She thus misses a very important fact—that there are no clothes at all! Note that it is not her individual “biases” that are getting in the way but her unreflective acceptance of the reality she inhabits that binds her to report things less accurately than she might.

In his later years, Bourdieu turned his attention to science itself and argued that the promise of modern science required reflectivity among scientists. We need to develop our reflexivity as we develop other muscles, through constant practice. Bourdieu ( 2004 ) urged researchers “to convert reflexivity into a disposition, constitutive of their scientific habitus, a reflexivity reflex , capable of acting not ex poste , on the opus operatum , but a priori , on the modus operandi ” ( 89 ). In other words, we need to build into our research design an appreciation of the relationship between researcher and researched.

To do science properly is to be reflective, to be aware of the social waters in which one swims and to turn one’s researching gaze on oneself and one’s researcher position as well as on the object of the research. Above all, doing science properly requires one to acknowledge science as a social process. We are not omniscient gods, lurking above the humans we observe and talk to. We are human too.

Further Readings

Barry, Christine A., Nicky Britten, Nick Barbar, Colin Bradley, and Fiona Stevenson. 1999. “Using Reflexivity to Optimize Teamwork in Qualitative Research.”  Qualitative Health Research  9(1):26–44. The coauthors explore what it means to be reflexive in a collaborative research project and use their own project investigating doctor-patient communication about prescribing as an example.

Hsiung, Ping-Chun. 2008. “Teaching Reflexivity in Qualitative Interviewing.” Teaching Sociology 36(3):211–226. As the title suggests, this article is about teaching reflexivity to those conducting interviews.

Kenway, Jane, and Julie McLeod. 2004. “Bourdieu’s Reflexive Sociology and ‘Spaces of Points of View’: Whose Reflexivity, Which Perspective?” British Journal of Sociology of Education 25(4):525–544. For a more nuanced understanding of Bourdieu’s meaning of reflexivity and how this contrasts with other understandings of the term in sociology.

Kleinsasser, Audrey M. 2000. “Researchers, Reflexivity, and Good Data: Writing to Unlearn.” Theory into Practice 39(3):155–162. Argues for the necessity of reflexivity for the production of “good data” in qualitative research.

Linabary, Jasmine R., and Stephanie A. Hamel. 2017. “Feminist Online Interviewing: Engaging Issues of Power, Resistance and Reflexivity in Practice.” Feminist Review 115:97–113. Proposes “reflexive email interviewing” as a promising method for feminist research.

Rabbidge, Michael. 2017. “Embracing Reflexivity: The Importance of Not Hiding the Mess.” TESOL Quarterly 51(4):961–971. The title here says it all.

Wacquant, Loïc J. D. 1989. “Towards a Reflexive Sociology: A Workshop with Pierre Bourdieu.” Sociological Theory 7(1):26–63. A careful examination of Bourdieu’s notion of reflexivity by one of his most earnest disciples.

• Someone might ask me if I have truly been able to “stand” in the shoes of more privileged students and if I might be overlooking similarities among college students because of my “biased” standpoint. These are questions I ask myself all the time. They have even motivated me to conduct my latest research on college students in general so that I might check my observations that working-class college students are uniquely burdened ( Hurst 2019 ). One of the things I did find was that middle-class students, relative to upper-class students, are also relatively disadvantaged and sometimes experience (feel) that disadvantage. ↵
• Unless, of course, one is engaged in autoethnography! Even in that case, however, the point of the study should probably be about a larger phenomenon or experience that can be understood more deeply through insights that emerge in the study of the particular self, not really a study about that self. ↵
• I mentioned Pierre Bourdieu earlier in the chapter. For those who want to know more about his work, I’ve included this advanced section. Undergraduates should feel free to skip over. ↵

The practice of being conscious of and reflective upon one’s own social location and presence when conducting research.  Because qualitative research often requires interaction with live humans, failing to take into account how one’s presence and prior expectations and social location affect the data collected and how analyzed may limit the reliability of the findings.  This remains true even when dealing with historical archives and other content.  Who we are matters when asking questions about how people experience the world because we, too, are a part of that world.

The branch of philosophy concerned with knowledge.  For researchers, it is important to recognize and adopt one of the many distinguishing epistemological perspectives as part of our understanding of what questions research can address or fully answer.  See, e.g., constructivism , subjectivism, and  objectivism .

A statement created by the researcher declaring their own social position (often in terms of race, class, gender) and social location (e.g., junior scholar or tenured professor) vis-à-vis the research subjects or focus of study, with the goal of explaining and thereby limiting any potential biases or impacts of such position on data analyses, findings, or other research results.  See also reflexivity .

Reliability is most often explained as consistency and stability in a research instrument, as in a weight scale, deemed reliable if predictable and accurate (e.g., when you put a five-pound bag of rice on the scale on Tuesday, it shows the same weight as when you put the same unopened bag on the scale Wednesday).  Qualitative researchers don’t measure things in the same way, but we still must ensure that our research is reliable, meaning that if others were to conduct the same interview using our interview guide, they would get similar answers.  This is one reason that reflexivity is so important to the reliability of qualitative research – we have to take steps to ensure that our own presence does not “tip the scales” in one direction or another or that, when it does, we can recognize that and make corrections.  Qualitative researchers use a variety of tools to help ensure reliability, from intercoder reliability to triangulation to reflexivity.

In mostly quantitative research, validity refers to “the extent to which an empirical measure adequately reflects the real meaning of the concept under consideration” ( Babbie 1990 ). For qualitative research purposes, practically speaking, a study or finding is valid when we are measuring or addressing what we think we are measuring or addressing.  We want our representations to be accurate, as they really are, and not an artifact of our imaginations or a result of unreflected bias in our thinking.

A method of ensuring trustworthiness where the researcher shares aspects of written analysis (codes, summaries, drafts) with participants before the final write-up of the study to elicit reactions and/or corrections.   Note that the researcher has the final authority on the interpretation of the data collected; this is not a way of substituting the researcher’s analytical responsibilities.  See also peer debriefing . 

The process of strengthening a study by employing multiple methods (most often, used in combining various qualitative methods of data collection and analysis).  This is sometimes referred to as data triangulation or methodological triangulation (in contrast to investigator triangulation or theory triangulation).  Contrast mixed methods .

Introduction to Qualitative Research Methods Copyright © 2023 by Allison Hurst is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License , except where otherwise noted.

Academia Info UK

• Dissertation

5 Ways To Ensure Reliability And Validity In Qualitative Research

• Research methods
• Research techniques
• Research statistical analysis techniques
• Research designing
• Mapping of the research’s methodology
• Analysis of results
• Collection of data (primary or secondary)

5 Ways to ensure Reliability and Validity in qualitative research

Reliability versus validity:, 2. use primary validation methods.

• Credibility: The credibility of qualitative research measures the correct anticipation of participants’ meaning.
• Authenticity: It aims to check if the voices of different people are considered or not.
• Critically: It highlights whether the researcher critically evaluates all aspects of the research or not.
• Integrity: It measures the originality of researcher.

3. Use the secondary validation method:

• Explicitness
• Sensitivity
• Thoroughness

The students should adopt transparency approaches:

• Qualitative research demands the researcher to cite reliable sources of data collection.
• It suggests not re-citing someone else’s work without giving him/her credit.
• It needs clarity in terms of thinking process, as well as drafting.
• It wants the researcher not to present false results. The duplicate research with the use of such fake data miss-guides the audience. The research aims to solve social issues. But fake representation of data will disturb society as well, and this should be avoided.

Standardise the qualitative research conditions

You may like these posts, post a comment.

• Privacy Policy

#buttons=(Accept !) #days=(20)

Contact form.

• Open access
• Published: 11 April 2024

Teachers and educators’ experiences and perceptions of artificial-powered interventions for autism groups

• Guang Li 1 ,
• Mohammad Amin Zarei 2 ,
• Goudarz Alibakhshi 2 &
• Akram Labbafi 3  

BMC Psychology volume  12 , Article number:  199 ( 2024 ) Cite this article

Metrics details

Artificial intelligence-powered interventions have emerged as promising tools to support autistic individuals. However, more research must examine how teachers and educators perceive and experience these AI systems when implemented.

The first objective was to investigate informants’ perceptions and experiences of AI-empowered interventions for children with autism. Mainly, it explores the informants’ perceived benefits and challenges of using AI-empowered interventions and their recommendations for avoiding the perceived challenges.

Methodology

A qualitative phenomenological approach was used. Twenty educators and parents with experience implementing AI interventions for autism were recruited through purposive sampling. Semi-structured and focus group interviews conducted, transcribed verbatim, and analyzed using thematic analysis.

The analysis identified four major themes: perceived benefits of AI interventions, implementation challenges, needed support, and recommendations for improvement. Benefits included increased engagement and personalized learning. Challenges included technology issues, training needs, and data privacy concerns.

Conclusions

AI-powered interventions show potential to improve autism support, but significant challenges must be addressed to ensure effective implementation from an educator’s perspective. The benefits of personalized learning and student engagement demonstrate the potential value of these technologies. However, with adequate training, technical support, and measures to ensure data privacy, many educators will likely find integrating AI systems into their daily practices easier.

Implications

To realize the full benefits of AI for autism, developers must work closely with educators to understand their needs, optimize implementation, and build trust through transparent privacy policies and procedures. With proper support, AI interventions can transform how autistic individuals are educated by tailoring instruction to each student’s unique profile and needs.

Peer Review reports

Introduction

Autism education has become an increasingly important area of focus in recent years due to the rising prevalence of autism spectrum conditions (ASC) among children. The estimated prevalence of ASC has increased from 1 in 10,000 in the 1960s to at least 1 in 100 today [ 1 , 2 , 3 ]. ASC is a neurodevelopmental condition characterized by impaired social interaction and communication abilities and stereotypical or obsessive behavior patterns. These impairments can significantly impact an individual’s social, educational, and employment experiences, leading to poor long-term outcomes and difficulties in social transactions, independent work, and job fulfillment [ 4 , 5 , 6 , 7 , 8 , 9 , 10 ].

The reported prevalence of autism spectrum disorders (ASDs) in developed countries is around 2% [ 11 ]. ASDs typically manifest within the first three years of life. They are characterized by challenges in social interaction, speech and language delays, avoidance of eye contact, difficulty adapting to changes in the environment, display of repetitive behaviors, and differences in learning profiles [ 11 , 12 , 13 ]. Those with ASDs, including children and adults, have a high frequency of anxiety and depression. Neurobiological research has revealed differences in brain development between children with ASDs and neurotypical children [ 14 ]. These excessive connections are thought to be due to reduced pruning of damaged neuronal connections during brain development, resulting in disordered neural patterning across the brain and dysregulation in cognitive function coordination between different brain regions [ 14 , 15 ].

The dominant perspective regarding AI technologies has revolved mainly around understanding these systems as a collection of processes and their corresponding responses, emphasizing autonomy, adaptability, and interactivity [ 16 , 17 , 18 , 19 , 20 , 21 ]. These characteristics are considered fundamental technological focuses that researchers argue should be integral to AI systems. Although autonomy, adaptability, and interactivity are significant, they may only cover some essential criteria for an adequate K-12 education. Specifically, these criteria are about skills taught by human educators, such as B. Self-efficacy, technical skills, and socialization skills. Samuel [ 22 ] emphasizes that AI technologies should replicate human actions and mimic expressions of “human intelligence, cognition, and logic.” This highlights the need to refine features that determine effective AI in education. The recent challenges in education due to the pandemic provide a unique opportunity to examine the demands on stakeholders, including educators, students, and parents [ 23 , 24 , 25 , 26 , 27 ].

The potential of artificial intelligence (AI) to drive developments in education is well-recognized [ 6 , 7 ]. Artificial intelligence is one of the technological advancements which can be used in education. AI encompasses a range of technologies that aim to simulate human intelligence, including machine learning, natural language processing, and computer vision [ 8 ]. These technologies have already been used in various applications, from speech recognition to image classification, and can potentially revolutionize how we think about education. In the context of autism, AI has the potential to provide personalized learning experiences that are tailored to the specific needs of each child [ 8 ]. For example, AI-powered systems can analyze a child’s behavior and responses to stimuli and use this information to adapt the learning materials and activities to suit their needs. Furthermore, AI can also be utilized to support communication and social interaction, which are areas of difficulty for many children with autism [ 9 ].

AI-powered interventions in the context of autism education refer to the utilization of artificial intelligence technologies to create tailored and interactive experiences for individuals on the autism spectrum. These interventions encompass a spectrum of applications, including educational tools, therapeutic programs, and support systems designed to address the unique learning and social communication needs of individuals with autism. AI technologies such as machine learning, natural language processing, and computer vision are employed to analyze and respond to the specific behaviors, preferences, and challenges exhibited by each individual [ 1 , 2 , 3 , 4 , 5 , 6 ]. The goal is to provide personalized and adaptive learning experiences, enhance social interaction skills, and offer targeted support for cognitive and emotional development. Examples of AI-powered interventions include virtual reality scenarios, interactive games, and educational software that can dynamically adjust content based on real-time feedback, creating a more individualized and effective educational approach for children with autism [ 2 , 3 , 4 , 5 ].

Moreover, there is a risk of bias and discrimination in AI-powered interventions for children with autism. For example, if the AI system is trained on data that is not representative of the diverse population of children with autism, it may not be effective for all individuals [ 10 ]. Moreover, there is a risk of perpetuating harmful stereotypes or reinforcing inappropriate behaviors if the AI system is not designed and programmed with ethical considerations (10). Third, there are concerns about data privacy and security when using AI in education for children with autism. For instance, if sensitive personal information is collected and stored by the AI system, there is a risk that it could be misused or accessed by unauthorized parties [ 16 ]. Therefore, it is essential to address these challenges and concerns to fully realize the potential of AI in education for children with autism. By doing so, we can create evidence-based and ethically sound interventions that support personalized learning and social communication skills while mitigating the risks associated with AI-powered education.

The potential of AI in autism education lies in its ability to offer personalized learning experiences, tailoring interventions to the unique needs of each child [ 8 ]. By analyzing a child’s behavior and responses, AI can adapt learning materials, potentially revolutionizing education for children with autism. However, this transformative potential is not without challenges. The risk of bias and discrimination looms large, as AI systems may not be effective if trained on non-representative data, perpetuating harmful stereotypes [ 10 ]. Ethical considerations become paramount, addressing concerns about data privacy and security, which, if overlooked, pose potential risks associated with unauthorized access and misuse of sensitive information [ 16 ]. Bridging the gap between the promise of AI in education and its responsible application is crucial. Therefore, this study aims to explore educators’ experiences and perceptions of AI-powered interventions for autism, shedding light on the nuanced landscape where technological advancements intersect with the delicate realm of autism education.

Research questions

In line with the research gap mentioned in the previous section, the following research questions are raised:

What are the benefits and challenges of using AI-powered interventions to support the learning and social communication skills of children with autism from teachers’ and educators’ perceptions?

How can AI-powered interventions be designed and implemented to ensure that they are culturally and linguistically appropriate for a diverse population of children with autism while also avoiding bias and discrimination in the learning materials and activities?

Review of literature

Theoretical background.

Machine learning is a component of artificial intelligence (AI) wherein models perform tasks autonomously without human intervention. Traditional machine learning models are trained using input data, enabling accurate outcome predictions. Deep learning, a subset of machine learning, employs extensive data to prepare models, achieving similarly high prediction accuracies. Both models are frequently utilized in diagnosing neurological disorders such as autism [ 28 , 29 ], ADHD [ 30 , 31 ], and depression [ 32 , 33 ]. Diagnostic inputs encompass images from computerized tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET) scans, or electroencephalogram (EEG) signals.

AI has been instrumental in social skills training for children with autism spectrum disorders (ASDs), aiding in recognizing and responding to social cues. Belpaeme et al. [ 34 ] utilized sensory features (facial expressions, body movements, and voice recordings) as inputs to a machine-learning model implemented in a robot for analyzing autistic children’s behavior and engagement levels during therapy. This study demonstrated the robot’s potential to adapt to interactants, influencing engagement. Another survey by Sanghvi et al. [ 35 ] employed postural expressions, specifically silhouette images of the upper body during chess playing, to analyze the engagement levels of autistic children. The integration of representative data with an affect recognition model suggested the potential for the robot to serve as a game-mate for autistic children in real-world scenarios. Kim et al. [ 36 ] employed audio recordings to assess the emotional states of autistic children, enhancing the robot’s ability to evaluate engagement and modify responses for a more interactive learning environment.

Various studies explored diverse input features such as facial expressions [ 37 ], body movements [ 38 ], and biosignals [ 39 ]. Esteban et al. [ 40 ] investigated facial expressions, direction of look, body posture, and voice tones as input features to a model within the NAO robot for assessing the social engagement of autistic children, showcasing the capability of robots to possess increased autonomy. Rudovic et al. [ 41 ] developed a personalized deep model using coordinated video recordings, audio recordings, and biosignals to assess engagement in autistic children, outperforming non-personalized machine learning solutions. Another study created a hybrid physical education teaching tool using speech recognition and artificial intelligence, achieving a recognition accuracy of over 90% for a voice interactive educational robot. Collectively, these studies affirm that AI holds promise in enhancing social interaction and supportive education for children with mental disorders.

Artificial intelligence and education

The use of AI technology in education has led to increased published studies on the subject, with a reported growing interest and impact of research on AI in education [ 42 ]. AI literacy, which refers to the capacity to comprehend the essential processes and concepts underpinning AI in various products and services, has been discussed in several studies [ 43 , 44 , 45 , 46 , 47 ]. Ng et al. [ 48 ] proposed a four-dimensional AI literacy framework covering knowing and understanding AI, using and applying AI, evaluating and creating AI, and AI ethics.

Recent review papers on AI in education have highlighted several major AI applications, such as intelligent tutoring systems, natural language processing, educational robots, educational data mining, discourse analysis, neural networks, affective computing, and recommender systems [ 22 , 23 , 33 – 34 ]. However, Chen et al. [ 49 ] identified some critical issues in their review paper on AI in education, including a lack of effort in integrating deep learning technologies into educational settings, insufficient use of advanced techniques, and a scarcity of studies that simultaneously employed AI technologies and delved extensively into educational theories. Furthermore, there needs to be more knowledge and discussion on the role of AI in early childhood education (ECE), an area often ignored in cutting-edge research.

Using AI to teach children with ASD

Autism Spectrum Disorder (ASD) is a complex neurodevelopmental disorder that affects communication, social interaction, and behavior (1). The disorder is characterized by various symptoms and severity levels, making it challenging to provide effective interventions for affected individuals [12]. Children with ASD often experience difficulties in learning and require specialized educational interventions to help them achieve their full potential [1]. In recent years, there has been growing interest in the potential of AI to improve the learning outcomes of children with autism [8). AI has the potential to provide personalized learning experiences that are tailored to the specific needs of each child with autism [ 9 ]. For example, AI-powered systems can analyze a child’s behavior and responses to stimuli and use this information to adapt the learning materials and activities to suit their needs [8].

AI can also be used to support communication and social interaction, which are areas of difficulty for many children with autism [10]. Chatbots and virtual assistants can provide a non-judgmental and non-threatening environment for children to practice their social skills while providing feedback and guidance [ 23 ]. These interventions can be particularly valuable for children who struggle with face-to-face interactions or feel uncomfortable in social situations [ 24 ]. Despite the potential benefits of using AI in education for children with autism, several challenges and concerns need to be addressed:

First, there is a lack of consensus on the most effective ways to use AI to support learning for autistic children [ 8 ]. While there have been some promising results from initial studies, more research is needed to determine the most effective methods for using AI to personalize learning and support social communication skills in this population [10]. Second, there is a risk of bias and discrimination in AI-powered interventions for children with autism. For example, if the AI system is trained on data that is not representative of the diverse population of children with autism, it may not be effective for all individuals [ 9 ]. Moreover, there is a risk of perpetuating harmful stereotypes or reinforcing inappropriate behaviors if the AI system is not designed and programmed with ethical considerations [ 23 ]. And, third, there are concerns about data privacy and security when using AI in education for children with autism. For instance, if sensitive personal information is collected and stored by the AI system, there is a risk that it could be misused or accessed by unauthorized parties [10].

Several research studies have investigated the use of AI in education for children with autism. For example, Goodwin and Stone [8] developed an AI-powered system called Maki, which uses natural language processing to provide personalized feedback on social communication skills. The system was effective in improving social communication skills in children with autism. Similarly, Alzoubi et al. [ 50 ] developed an AI-powered system that uses virtual reality to provide social skills training for children with autism. The system was found to be effective in improving social skills and reducing anxiety in children with autism.

Other research studies have explored the potential of AI to improve different aspects of learning for children with autism. For example, Zhang et al. [ 10 ] developed an AI-assisted system that uses computer vision and machine learning to provide personalized feedback on handwriting skills. The system was effective in improving handwriting skills in children with autism. Similarly, Wang et al. [ 51 ] developed an AI-powered system that uses game-based learning to enhance math skills in children with autism.

There have also been efforts to develop AI-powered systems that can assist teachers and parents in providing effective interventions for children with autism. The system effectively improved the quality of interventions offered by teachers and parents. However, there are also concerns about the potential negative impacts of AI on children with autism. For example, some studies have suggested that excessive use of AI-powered interventions could reduce face-to-face interactions and social skills development [9]. Additionally, there are concerns about the potential for AI-powered interventions to replace human teachers and therapists, which could have negative implications for the quality of care provided to children with autism [8].

To address these concerns and maximize the potential benefits of AI for children with autism, it is essential to prioritize ethical considerations and involve stakeholders in designing and implementing AI-powered interventions [ 23 ]. This includes ensuring that AI systems are developed and programmed to avoid bias and discrimination, protecting the privacy and security of personal data, and promoting transparency and accountability in using AI in education for children with autism [ 10 ].

Other studies have investigated using chatbots and virtual assistants to support social communication skills in children with autism. For example, Kocaballi et al. [ 52 ] developed a chatbot called Tess that provides social skills training and support for children with autism. The chatbot was effective in improving social communication skills in children with autism. Similarly, Tanaka et al. [ 53 ] developed a virtual assistant called Miko that uses artificial empathy to support social communication skills in children with autism.

Further studies highlighted the importance of ethical consideration while using AL in education for children with autism. For example, there is a risk of perpetuating harmful stereotypes or reinforcing inappropriate behaviors if the AI system is not designed and programmed with ethical considerations [ 23 ]. Moreover, there is a risk of bias and discrimination if the AI system is trained on data that is not representative of the diverse population of children with autism [9]. Therefore, it is essential to carefully consider the ethical implications of using AI in education for children with autism. In conclusion, utilizing AI in education can transform how we think about learning and support children with autism to achieve their full potential.

Research Methodology

The study used purposive sampling to select 20 informants who met specific criteria. These individuals were parents or educators of autistic children and had valuable experience using AI-powered interventions to improve their children’s learning and social communication skills. They were all Iranian living in Tehran, Iran. 30% ( n  = 6) were female and 70% ( n  = 14) were male.

The participants in the study encompassed an age range spanning from 29 to 58 years old. Educators teaching experience was above 8 years. Recruitment efforts were conducted through various channels and social media platforms to ensure a diverse and representative sample. Potential participants were fully informed about the study’s purpose, procedures, and possible benefits throughout the recruitment process. They were also told of their rights as participants and the assurance of confidentiality. To confirm their willingness to participate, informants were asked for written consent before formal inclusion in the study.

Data collection

The study used semi-structured interviews and focus groups to collect data from the informants. The researcher developed the interview questions (Appendix), and a panel of three qualitative researchers reviewed their relevance. Interviews were conducted individually, either in person or virtually, and lasted approximately 45–60 min each. Focus groups with 3–5 participants conducted almost or in person were also organized. The duration of the focus group discussions was between 60 and 90 min. During the data collection process, the interviews and focus group sessions were audio-recorded to capture participants’ responses and insights accurately. These recordings were later transcribed verbatim, allowing a comprehensive analysis of the data collected. Through semi-structured interviews and focus groups, the study aimed to obtain complete and detailed information about participants’ experiences and perspectives regarding using AI-assisted interventions to support the learning and social communication skills of children with autism. The semi-structured nature of the interviews allowed for flexibility in exploring different topics while ensuring a consistent data collection framework for all participants. Additionally, the dynamic and interactive nature of the focus groups encouraged group discussions and allowed participants to share and build on one another.

Data analysis

Following the data collection phase, the study thoroughly analyzed the information collected. The audio recordings of the interviews and focus group sessions were transcribed verbatim, resulting in a comprehensive text dataset that captured participants’ responses and insights. The analysis began with a thorough familiarization process in which researchers immersed themselves in the transcribed data to understand participants’ accounts deeply. This immersion allowed researchers to identify recurring themes, patterns, and noteworthy information in the data set. A systematic analysis approach was used to ensure reliability and validity. Data were coded using a combination of inductive and deductive methods. First, an open coding process was conducted in which researchers generated initial codes by closely examining the data and labeling meaningful segments. As the analysis progressed, these codes were refined, grouped, and organized into categories and subcategories, creating a coding framework. After coding, researchers conducted a thematic analysis by identifying overarching themes from the data. The topics represented vital concepts, ideas, and perspectives shared by participants regarding the use of AI-assisted interventions to support the learning and social communication skills of children with autism. Throughout the analysis, the researchers ensured the accuracy and trustworthiness of the findings by employing techniques such as member checking, where participants were allowed to review and validate the interpretations made from their data.

Ethical considerations

The study adhered to ethical guidelines for conducting research with human subjects. Informed consent was obtained from all participants. Participants’ privacy and confidentiality were protected throughout the research process. The study also obtained ethical clearance from a relevant research ethics committee.

The study’s findings were presented in a report summarizing the themes and sub-themes that emerged from the data analysis. The report also provides recommendations for designing and implementing culturally and linguistically appropriate AI-powered interventions for children with autism while avoiding bias and discrimination in the learning materials and activities. The report also includes direct participant quotes to illustrate their experiences and perceptions. The findings are presented based on the order of research questions,

Benefits and challenges of AI-powered interventions

Informants of the study mentioned three benefits and some challenges of AI-empowered intervention for children with autism. Each is explained and exemplified as follows.

Increased engagement and motivation among children with autism

AI-powered interventions can use technologies like robots, virtual reality, and interactive games to provide personalized and engaging experiences for children with autism. Informants believed that AI-powered interventions can effectively increase engagement and motivation among children with autism. For example, educator 1 stated, “Children with autism who interacted with a humanoid robot showed increased engagement and motivation compared to those who received traditional therapy.” Educator 5 said, “By leveraging AI technologies, interventions for children with autism can be tailored to their needs and preferences, providing a more personalized and engaging learning experience. This can lead to improved outcomes and better quality of life for children with autism and their families. This finding is also supported by parent one, who stated, “My son used to struggle with traditional teaching methods, but with AI-powered interventions, he is more engaged and motivated to learn. The technology provides him with immediate feedback, which helps him understand his mistakes and learn from them.”

Customized and individualized interventions that cater to the unique needs of each child

Informants argued that every child with autism is unique, with their own set of strengths and challenges. Therefore, interventions tailored to each child’s specific needs and preferences can be more effective in promoting their development and well-being. This finding echoes the direct quotation by educator 6 who stated, “One size does not fit all when it comes to autism interventions. Each child is unique and requires a personalized approach that takes into account their individual strengths, challenges, and interests.” (Educator 6). Similarly, parent 6 stated, “As a parent, I have learned that the key to helping my child with autism is to focus on his individual needs. By working with his teachers and therapists to develop a personalized intervention plan, we have seen significant progress in his development and well-being.”

Real-time feedback to both children and educators about progress and areas for improvement

Real-time feedback involves providing immediate and ongoing information about a child’s performance and progress in a given activity or intervention. This feedback can reinforce positive behaviors, correct errors, and identify areas where additional support or instruction may be needed. Real-time feedback can be especially beneficial for children with autism, who may benefit from more frequent and targeted feedback to support their learning and development. By providing timely and specific feedback, children with autism can better understand their strengths and areas for improvement, and educators can adjust their interventions and supports accordingly. As an example, one of the educators stated, “Real-time feedback is crucial in helping children with autism learn and grow. By providing immediate and targeted feedback, we can reinforce positive behaviors and help children build new skills.” (Educator 4). Another educator stated, “Real-time feedback is not just important for children but for educators as well. By receiving ongoing feedback about a child’s progress, we can make more informed decisions about the interventions and supports that are most effective for them.“(Educator 8).

The potential for AI-powered interventions to enhance the work of educators and provide them with additional tools and resources

AI-powered interventions have the potential to enhance the work of educators and provide them with additional tools and resources to support the learning and development of children with autism. AI technologies like machine learning algorithms and natural language processing can analyze and interpret data from various sources, including assessment results, behavioral observations, and social communication interactions. This can provide educators with valuable insights and information about each child’s strengths, challenges, and learning needs. Educator 10 stated, “AI-powered interventions can provide educators with powerful tools and resources for supporting autistic children. By analyzing data and providing real-time feedback, these interventions can help educators tailor their teaching strategies and supports to the unique needs of each child.” Educator 3 also stated,” AI-powered interventions have the potential to transform the way we support children with autism in the classroom. By providing educators with insights and information about each child’s learning needs, these interventions can help us deliver more effective and personalized instruction.”

Challenges of AI-powered interventions

The content of interviews with informants was analyzed, and five main themes were extracted. Each is explained and exemplified as follows.

Lack of personalization

Informants stated that while AI-powered interventions have the potential to be personalized, there is a risk that they may not account for the unique needs and preferences of each child. For example, educator 3 stated, “We need to remember that technology is a tool, not a replacement for human interaction.”

Limited access to technology

Not all families and schools can access the necessary technologies for AI-powered interventions. As a parent of a child with autism notes, “Technology can be expensive, and not all families can afford it.”

Difficulty in interpreting and responding to social cues

Children with autism may have trouble analyzing and reacting to social cues, making it challenging to interact with AI technologies. A clinical psychologist notes: “Children with autism may struggle to understand that a robot or virtual character is not a real person, which can limit the effectiveness of AI-powered interventions.”

Ethical concerns

Ethical concerns surrounding using AI technologies with children include privacy, data security, and the potential for misuse or unintended consequences. The Director of Education at one School for Children with Autism notes: “We need to be mindful of the potential risks and unintended consequences of using AI technologies with children with autism.”

Lack of human interaction

While AI-powered interventions can be engaging and interactive, they cannot replace the importance of human interaction in promoting social and emotional development in children with autism. As a parent of a child with autism notes: “Technology can be helpful, but it is important to balance it with real-life experiences and interactions.”

Concerns about the cost and affordability of these interventions

One concern related to using interventions for children with autism is their cost and affordability. Many interventions, such as behavioral and developmental therapies, assistive technologies, and specialized education programs, can be expensive and may not be covered by insurance or other funding sources. This can create barriers for families, particularly those with limited financial resources, in accessing the interventions their child needs to thrive. As Educator 9 stated, “The cost of interventions for children with autism can be a significant burden for families, particularly those with limited financial resources. We must ensure these interventions are accessible and affordable for all families.” Similarly, parent 5 stated, “As a parent of a child with autism, the cost of interventions has been a major concern for our family. Based on our financial limitations, we have had to decide which interventions to prioritize.”

Suggestions for improving the quality of AL-empowered interventions

Interviews with informants were thematically analyzed, and different themes were extracted. Each theme is explained and exemplified as follows.

Using culturally and linguistically appropriate interventions

Participants emphasized the importance of designing and implementing AI-powered interventions that are culturally and linguistically appropriate for a diverse population of children with autism. Some of the suggestions made by participants include:

Ensuring that the language and content of the interventions are culturally sensitive and relevant to the target population.

Incorporating diverse perspectives and experiences into the design and development process.

Providing interventions in multiple languages to accommodate diverse linguistic backgrounds.

Quotations from educators and parents support these suggestions. For instance, educator 1 stated, “Cultural sensitivity is important when designing interventions for children with autism, particularly for those from diverse backgrounds. We need to ensure that the interventions are culturally relevant and take into account the unique needs and experiences of each child.” Similarly, parent 6 stated, “As a parent of a child with autism who comes from a different cultural background, I appreciate interventions that take into account my child’s unique needs and experiences. It’s important to have interventions that are culturally sensitive and relevant.”

Avoiding bias and discrimination

Participants also emphasized the importance of avoiding bias and discrimination in AI-powered interventions’ learning materials and activities. Some of the suggestions made by participants include:

Conducting regular audits of the interventions to identify and address any potential biases or discriminatory content.

Incorporating diverse perspectives and experiences into the design and development process to avoid perpetuating stereotypes.

Providing training and education to educators and developers to ensure that they are aware of and can address potential biases and discrimination.

Quotations from informants support these strategies. As an example, educator 8 stated,

“We need to be careful to avoid stereotypes and biases in the interventions we design and implement. It’s important to be aware of potential biases and to work to address them.” Similarly, parent 7 stated, “To ensure that AI-powered interventions are effective and inclusive, we need to make sure that they are designed with diversity and inclusivity in mind. This means avoiding discrimination and bias in the materials and activities.”

Training educators

Participants discussed the role of educators in implementing AI-powered interventions to support the learning and social communication skills of children with autism. Some of the key findings include:

The importance of providing training and education to educators to ensure that they can effectively implement these interventions.

The need for educators to work collaboratively with parents and other professionals to ensure that the interventions are tailored to the unique needs of each child.

“Educators play a critical role in implementing AI-powered interventions. They need to be trained and educated on how to use these interventions effectively and how to tailor them to the unique needs of each child.” [Educator 3).

We regularly audit the interventions to identify and address potential biases or discriminatory content

Conducting regular audits of interventions for children with autism is an essential step in ensuring that these interventions are effective, evidence-based, and free from biases or discriminatory content. Regular audits help identify areas for improvement, ensure that interventions are aligned with current best practices and ethical guidelines, and promote accountability and transparency in developing and implementing these interventions. Here are two quotations that address the importance of conducting regular audits of interventions for children with autism. To exemplify this finding, the following quotations are presented:

“As educators and researchers, it is our responsibility to ensure that interventions for children with autism are evidence-based, effective, and free from biases or discriminatory content. Regular audits can help us identify and address any areas of concern and promote the highest standards of quality and ethical practice.” (Educator 4). “Regular audits are essential to ensuring that interventions for children with autism are meeting the needs of all children, regardless of their race, ethnicity, gender, or other factors. We must be vigilant in identifying and addressing any biases or discriminatory content that may be present, and work to create interventions that are inclusive and equitable for all children.” (Educator 9).

Involving families and communities in the design and implementation process ensures their voices and perspectives are heard and valued

Involving families and communities in the design and implementation process of interventions for children with autism is crucial to ensuring that their voices and perspectives are heard and valued. Families and communities can provide valuable insights and feedback on the needs and preferences of children with autism and the effectiveness and cultural relevance of interventions. Here are two quotations that address the importance of involving families and communities in the design and implementation process:

“Families and communities are essential partners in the design and implementation of interventions for children with autism. Their insights and feedback can help us create interventions that are effective, culturally relevant, and responsive to the needs of all children.” (Educator 10). “As a parent of a child with autism, I know firsthand the importance of involving families and communities in the design and implementation of interventions. By listening to our voices and perspectives, researchers and educators can create interventions that are more meaningful and effective for our children.” (Parent 8).

Discussion and implications

The present study aimed at exploring the teachers and educators’ experiences and perceptions of artificial intelligence powered interventions for Autism groups. A qualitative research study was employed and interviews were analyzed thematically and different themes were extracted. Participant believed that AI-powered interventions represent a groundbreaking frontier in reshaping the support systems for the learning and social communication skills of children with autism [ 54 ]. Participants also highlighted several noteworthy benefits, with a critical emphasis on the heightened engagement and motivation witnessed among children with autism when exposed to AI-powered interventions [ 1 , 2 , 54 ]. Recognizing the limitations of traditional teaching methods in meeting the distinctive learning needs of these children, AI interventions emerge as a promising avenue [ 1 , 2 ].

The first advantage underscored by participants is the adaptability of AI-powered interventions to provide personalized and individualized support, furnishing real-time feedback to children and educators regarding progress and areas for improvement [ 3 , 4 , 5 ]. This tailored approach aligns seamlessly with the diverse and unique challenges presented by children with autism. However, embracing AI-powered interventions is full of challenges, and participants articulated various concerns [ 55 , 56 ]. Technical glitches and difficulties were identified as potential disruptors of the learning process, prompting apprehensions about an overreliance on technology [ 55 , 56 ]. Moreover, the limited access to technology and resources in specific communities and regions raises concerns about the equitable distribution of intervention benefits [ 55 , 56 ]. Addressing these challenges is imperative to ensure that all children with autism, irrespective of geographical location or socioeconomic status, have equitable access to effective interventions.

The second theme, cultural and linguistic appropriateness, emerged as a primary consideration, with participants highlighting the importance of interventions tailored to the diverse backgrounds of children with autism [ 55 , 56 ]. This includes ensuring that the language and content of interventions are culturally sensitive and relevant, integrating diverse perspectives into the design process, and providing interventions in multiple languages ​​to accommodate linguistic diversity [ 7 , 8 , 9 ]. This finding is consistent with the findings of the previous research which highlighted that language differences can pose significant barriers to accessing autism interventions, highlighting the urgent need for interventions in the child’s native language [ 66 ].

As the third extracted theme “mitigating bias and discrimination in AI-powered interventions” extracted as another critical aspect, necessitating regular audits to identify and rectify potential biases [ 57 ]. The imperative of incorporating diverse perspectives into the design process and providing training to educators and developers to address biases and discrimination was highlighted as crucial [ 10 , 11 ]. This finding confirms the findings of the study that emphasizes the pivotal role of involving families and communities in designing and developing autism interventions to ensure cultural sensitivity and effectiveness [ 67 ].

Despite the above-mentioned potential of AI-powered interventions, the participants concurrently acknowledged the need for further research to evaluate the effectiveness of remote interventions and ensure their cultural and linguistic appropriateness [ 12 , 13 ]. Simultaneously, there are apprehensions and concerns with the potential for these interventions to exacerbate existing disparities in access to care if not implemented equitably. Moreover, challenges have been discerned alongside these benefits, prompting a comprehensive approach to ensure effectiveness, inclusivity, and accessibility [ 55 , 56 ]. Technical glitches, concerns about overreliance on technology, and limited access to resources pose hurdles that need addressing [ 55 , 56 ]. Policymakers must prioritize equitable access, focusing on both technological infrastructure and training programs for educators [ 55 , 56 ].

In addition, ensuring cultural and linguistic appropriateness emerges as a critical consideration in designing and implementing AI-powered interventions [ 55 , 56 ]. Culturally sensitive content, diverse perspectives in development, and multilingual offerings are underscored as essential [ 7 , 8 , 9 ]. Recognizing potential biases and discrimination, participants advocate for regular audits, diversity in development teams, and education on bias mitigation as integral components of ethical AI intervention deployment [ 10 , 11 , 57 ].

AI-powered interventions have emerged as a promising avenue to revolutionize the support for children with autism, offering transformative benefits while presenting challenges that demand careful consideration [ 54 ]. One pivotal advantage emphasized by participants is the heightened engagement and motivation observed among children with autism undergoing AI-powered interventions [ 54 ]. This is particularly noteworthy as traditional teaching methods often need to catch up in meeting the unique learning needs of these children. AI interventions, utilizing technologies such as robots, virtual reality, and interactive games, create personalized and engaging experiences, as reported by educators and parents.

It can also be concluded that transformative potential of AI-powered interventions underscores the need for collaborative efforts among educators, parents, and developers, ensuring effectiveness, inclusivity, and accessibility for all children [ 60 , 61 , 62 , 63 , 64 , 65 ]. The imperative of providing interventions in multiple languages and incorporating diverse perspectives into the design and development process is underscored [ 63 ]. Additionally, including culturally responsive teaching practices alongside AI interventions emerges as a strategy to enhance engagement and outcomes, particularly for children from diverse cultural backgrounds [ 68 ]. Ongoing research, collaborative endeavors, and an unwavering commitment to addressing challenges are imperative to maximize the benefits of AI-powered interventions for children with autism.

It can also be inferred that the collaborative involvement of families and communities is championed to enhance interventions’ impact and cultural sensitivity [ 12 , 13 , 67 ]. Balancing technology with human interaction is deemed crucial, emphasizing the irreplaceable role of personal connections in social and emotional development [ 39 , 41 ]. Moreover, the potential for AI-powered interventions to address access disparities, especially in remote or underserved areas, highlights the importance of further research and evaluation [ 58 , 59 ]. However, concerns persist about exacerbating existing disparities, demanding meticulous attention to cultural, linguistic, and regional nuances.

As another concluding remark, it can be inferred that AI-powered interventions have the potential to revolutionize the way we support the learning and social communication skills of children with autism. These interventions can provide customized and individualized interventions that cater to the unique needs of each child, providing real-time feedback to both children and educators about progress and areas for improvement. AI-powered interventions can also improve access to care for children with autism, particularly for those in remote or underserved areas. The findings suggest that to ensure that AI-powered interventions are culturally and linguistically appropriate for a diverse population of children with autism while also avoiding bias and discrimination in the learning materials and activities, it is essential to incorporate various perspectives and experiences into the design and development process, provide interventions in multiple languages, ensure that the language and content of the interventions are culturally sensitive and relevant, deliver training and education to educators and developers, conduct regular audits of the interventions, involve families and community members in the design and implementation process, and use culturally responsive teaching practices. These efforts can help to address the challenges and considerations of using AI-powered interventions and ensure that all children with autism have access to practical, inclusive, and culturally appropriate interventions.

However, several challenges and considerations need to be taken into account to ensure that these interventions are effective, inclusive, and accessible to all children with autism. These challenges include technical difficulties, overreliance on technology, limited access to technology and resources in specific communities and regions, and the need to design and implement culturally and linguistically appropriate interventions to avoid bias and discrimination.

Finally, one recurring theme is the importance of professional development for educators, which recognizes their critical role in successfully applying AI-powered interventions. Providing educators with technological expertise, cultural sensitivity, and ethical awareness is essential. Furthermore, legislators, educators, and parents must work together to prioritize the financial accessibility of interventions. The ramifications in this complex environment suggest a comprehensive and collaborative strategy. The key to success is overcoming obstacles, adopting technology responsibly, and giving accessibility and inclusivity top priority in intervention and education initiatives. Because technology constantly changes, we must remain committed to ongoing iteration and improvement. Community, parent, and educator feedback loops help us refine AI-powered interventions.

Limitations and suggestions for further studies

The current body of research on AI-powered interventions for children with autism, while promising, grapples with several limitations that warrant careful consideration. Firstly, the generalization of findings remains a challenge, as many studies tend to focus on specific demographic groups or particular manifestations of autism spectrum disorder (ASD). This limits the broader applicability of the insights gained, as the diversity within the autism spectrum may not be comprehensively represented. Additionally, a notable gap exists in understanding the long-term efficacy of AI interventions. While short-term outcomes are frequently explored, there is a scarcity of research delving into the sustained impact of these interventions on the developmental trajectories of children with autism. Longitudinal studies are crucial to elucidating AI-powered approaches’ durability and lasting benefits.

Moreover, the current literature may lack ethnic and cultural diversity, raising concerns about AI interventions’ universal applicability and artistic sensitivity. This underrepresentation hinders our understanding of how these technologies might function across diverse populations. Ethical considerations, although acknowledged, need to be thoroughly examined. Privacy, data security, and potential biases in algorithmic decision-making demand a more in-depth investigation to ensure responsible and equitable use of AI technologies in educational settings.

To address these limitations, future research should prioritize several vital areas. Long-term impact assessments are imperative to ascertain the sustained efficacy of AI interventions over time. Diverse and inclusive studies encompassing a range of ethnicities and cultural backgrounds are essential to validate the universal applicability of these technologies. Robust ethical frameworks should be developed to guide the implementation of AI interventions, addressing privacy, security, and bias concerns. Comparative studies, pitting AI interventions against traditional methods, will offer nuanced insights into their relative advantages and limitations. Family and community involvement in designing and implementing AI interventions should be explored further, recognizing the unique insights these stakeholders bring. Finally, comprehensive cost-benefit analyses are necessary to evaluate the economic aspects of AI interventions, ensuring their affordability and long-term viability in diverse educational settings. In navigating these avenues, researchers can contribute substantively to the responsible and inclusive integration of AI-powered interventions for children with autism.

Data availability

The data will be made available upon request from the corresponding author (Corresponding author: email: [email protected].

Lotter V. Epidemiology of autistic conditions in young children: 1. Prevalence. Social Psychiatry. 1966;1:124–35.

Article   Google Scholar  

Baird G, Simonoff E, Pickles A, Chandler S, Loucas T, Meldrum D, Charman T. Prevalence of disorders of the autism spectrum in a population cohort of children in South Thames: the special needs and Autism Project (SNAP). Lancet. 2006;368(9531):210–5.

Russell G, Rodgers LR, Ukoumunne OC, Ford T. Prevalence of parent-reported ASD and ADHD in the UK: findings from the Millennium Cohort Study. J Autism Dev Disord. 2014;44:31–40.

Legg H, Tickle A. UK parents’ experiences of their child receiving a diagnosis of autism spectrum disorder: a systematic review of the qualitative evidence. Autism. 2019;23:1897–910.

Howlin P, Moss P, Savage S, Rutter M. Social outcomes in mid- to later adulthood among individuals diagnosed with autism and average nonverbal IQ as children. J Am Acad Child Adolesc Psychiatry. 2013;52:572–81.

Williamson B. Big Data in Education: the Digital Future of Learning, Policy and Practice. Sage; 2017.

Luckin R. Machine learning and Human Intelligence: the Future of Education for the 21st Century. ERIC; 2018.

Goodwin MS, Stone JV. Artificial intelligence, autism and use of technology: a critical analysis. Psychol Cogn Sci Open J. 2018;4(1):12–7. https://doi.org/10.17140/PCSOJ-4-141 .

Lombardo MV, Lai MC, Baron-Cohen S. Big data approaches to decomposing heterogeneity across the autism spectrum. Mol Psychiatry. 2018;24(10):1435–50. https://doi.org/10.1038/s41380-018-0321-0 .

Zhang T, Wright C, Joshi G. Artificial intelligence in autism education: a systematic literature review. Res Dev Disabil. 2021;112., Article 103858. https://doi.org/10.1016/j.ridd.2021.103858 .

Kopp S, Gillberg C. Swedish child and adolescent psychiatric out-patients: a five-year cohort. Eur Child Adolesc Psychiatry. 2003;12:30–5. https://doi.org/10.1007/s00787-003-0309-3 .

Baygin M, Dogan S, Tuncer T, Barua PD, Faust O, Arunkumar N, Abdulhay EW, Palmer EE. Acharya U.R. Automated ASD detection using hybrid deep lightweight features extracted from EEG signals. Comput Biol Med. 2021;134:104548. https://doi.org/10.1016/j.compbiomed.2021.104548 .

Khodatars M, Shoeibi A, Sadeghi D, Ghaasemi N, Jafari M, Moridian P, Khadem A, Alizadehsani R, Zare A, Kong Y, et al. Deep learning for neuroimaging-based diagnosis and rehabilitation of Autism Spectrum disorder: a review. Comput Biol Med. 2021;139:104949. https://doi.org/10.1016/j.compbiomed.2021.104949 .

Tang G, Gudsnuk K, Kuo S-H, Cotrina ML, Rosoklija G, Sosunov A, Sonders MS, Kanter E, Castagna C, Yamamoto A, et al. Loss of mTOR-Dependent Macroautophagy causes autistic-like synaptic pruning deficits. Neuron. 2014;83:1131–43. https://doi.org/10.1016/j.neuron.2014.07.040 .

Article   PubMed   PubMed Central   Google Scholar  

Sharif H, Khan RA. A novel framework for automatic detection of Autism: A study on Corpus Callosum and Intracranial Brain Volume. arXiv. 2019.

Ahmad MA, Teredesai A, Eckert C. (2020). Fairness, accountability, transparency in AI at scale: Lessons from national programs. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (pp. 690–690). https://doi.org/10.1145/3351095.3375690 .

Baker T, Smith L, Anissa N. Educ-AI-Tion Rebooted? Exploring the future of Artificial Intelligence in Schools and Colleges. London: Nesta; 2019. Available online at. https://www.nesta.org.uk/report/education-rebooted .

Google Scholar  

Chai CS, Lin PY, Jong MSY, Dai Y, Chiu TK, Huang B. (2020b, August). Factors Influencing Students’ Behavioral Intention to Continue Artificial Intelligence Learning. In 2020 International Symposium on Educational Technology (ISET) (pp. 147 150). IEEE. https://doi.org/10.1109/ISET49818.2020.00040 .

Chai CS, Wang X, Xu C. (2020). An extended theory of planned behavior for the modelling of Saudi Arabi secondary school students’ intention to learn. Artificial Intelligence. Mathematics, 8 (11), 2089. https://doi.org/10.3390/math8112089 .

Dai Y, Chai CS, Lin PY, Jong MSY, Guo Y, Qin J. Promoting students’ well-being by developing their readiness for the artificial intelligence age. Sustainability. 2020;12(16):6597. https://doi.org/10.3390/su12166597 .

Dignum V. The role and challenges of education for responsible AI. Lond Rev Educ. 2021;19(1):1–11. https://doi.org/10.14324/LRE.19.1.01 .

Druga S, Vu ST, Likhith E, Qiu T. (2019). Inclusive AI literacy for kids around the world. In Proceedings of FabLearn 2019 (pp. 104–111). https://doi.org/10.1145/3311890.3311904 .

Fourtané S. (2020). Ethics of AI: Benefits and risks of artificial intelligence systems. interesting engineering. Retrieved from.

Samuel J. (2021). A call for proactive policies for informatics and artificial intelligence technologies. Scholars Strategy Network. Available online at: https://scholars.org/contribution/call-proactive-policies-informatics-and (accessed October 2, 2022).

Samuel J, Navazali GG, Rahman M, Esawi E, Samuel Y. COVID-19 public sentiment insights and machine learning for tweets classification. Information. 2020;11(6):314–30. https://doi.org/10.3390/info11060314 .

Samuel J, Kashyap R, Samuel Y, Pelaez A. Adaptive cognitive fit: artificial intelligence augmented management of information facets and representations. Int J Manage. 2022;65(4):102505. https://doi.org/10.1016/j.ijinfomgt.2022.102505 .

Williamson B, Eynon R. Historical threads, missing links, and future directions in AI in education. Learn Media Technol. 2020;45(3):223. https://doi.org/10.1080/17439884.2020.1798995 .

Pham T-H, Vicnesh J, Wei JKE, Oh SL, Arunkumar N, Abdulhay EW, Ciaccio EJ, Acharya UR. Autism Spectrum Disorder Diagnostic System using HOS bispectrum with EEG signals. Int J Environ Res Public Heal. 2020;17:971.

Oh SL, Jahmunah V, Arunkumar N, Abdulhay EW, Gururajan R, Adib N, Ciaccio EJ, Cheong KH, Acharya. U.R. A novel automated autism spectrum disorder detection system. Complex Intell Syst. 2021;7:2399–413.

Dubreuil-Vall L, Ruffini G, Camprodon JA. Deep learning convolutional neural networks discriminate adult ADHD from healthy individuals on the basis of event-related spectral EEG. Front Neurosci. 2020;14:1–12.

Tor HT, Ooi CP, Lim-Ashworth NS, Wei JKE, Jahmunah V, Oh SL, Acharya UR, Fung DSS. Automated detection of conduct disorder and attention deficit hyperactivity disorder using decomposition and nonlinear techniques with EEG signals. Comput Methods Programs Biomed. 2021;200:105941.

Acharya UR, Oh SL, Hagiwara Y, Tan JH, Adeli H, Subha DP. Automated EEG-based screening of depression using deep convolutional neural network. Comput Methods Programs Biomed. 2018;161:103–13.

He L, Cao C. Automated depression analysis using convolutional neural networks from speech. J Biomed Inf. 2018;83:103–11. [CrossRef]Detecting Depression Using a Framework Combining Deep Multimodal Neural Networks with a Purpose-Built Automated Evaluation. Psychol. Assess. 2019, 31, 1019.

Belpaeme T, Baxter PE, Read R, Wood R, Cuayáhuitl H, Kiefer B, Racioppa S, Kruijff-Korbayová I, Athanasopoulos G, Enescu V, et al. Multimodal Child-Robot Interaction: Building Social Bonds J Hum Robot Interact. 2013;1:33–53.

Sanghvi J, Castellano G, Leite I, Pereira A, McOwan PW, Paiva A. Automatic analysis of affective postures and body motion to detect engagement with a game companion. In Proceedings of the 6th International Conference on Human-Robot Interaction—HRI ’11, Lausanne, Switzerland, 6–9 March 2011; pp. 305–312.

Kim JC, Azzi P, Jeon M, Howard AM, Park CH. Audio-based emotion estimation for interactive robotic therapy for children with autism spectrum disorder. In Proceedings of the 2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI), Jeju, Korea, 28 June–1 July 2017; pp. 39–44.

Anzalone SM, Boucenna S, Ivaldi S, Chetouani M. Evaluating the Engagement with Social Robots. Int J Soc Robot. 2015;7:465–78. [CrossRef].

Colton MB, Ricks DJ, Goodrich MA, Dariush B, Fujimura K, Fujiki M. Toward therapist-in-the-loop assistive robotics for children with autism and specific language impairment. In Proceedings of the 23rd Convention of the Society for the Study of Artificial Intelligence and Simulation of Behaviour, AISB 2009, Edinburgh, UK, 6–9 April 2009.

Hernandez J, Riobo I, Rozga A, Abowd GD, Picard RW. Using electrodermal activity to recognize ease of engagement in children during social interactions. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Seattle, WA, USA, 13–17 September 2014; pp. 307–317.

Esteban PG, Baxter P, Belpaeme T, Billing E, Cai H, Cao H-L, Coeckelbergh M, Costescu C, David D, De Beir A, et al. How to build a Supervised Autonomous System for Robot-enhanced therapy for children with Autism Spectrum Disorder. Paladyn J Behav Robot. 2017;8:18–38.

Rudovic O, Lee J, Dai M, Schuller B, Picard RW. Personalized machine learning for robot perception of affect and engagement in autism therapy. Sci Robot. 2018;3:eaao6760.

Kandlhofer M, Steinbauer G, Hirschmugl-Gaisch S, Huber P. Artificial Intelligence and computer science in education: From kindergarten to university. In 2016 IEEE Frontiers in Education Conference (FIE) (pp. 1–9)., 2016. IEEE.

El Kebir M, Callahan BL, Roseberry K, Masterson T. Ethical considerations for artificial intelligence to support individuals with autism: a review. Autism. 2021;25(4):894–907.

Chen B, Liu Z, Sun C. Artificial intelligence in education: a review. Neurocomputing. 2020;396:3–14.

Burgsteiner H, Gula B. AI literacy: what (and how) teachers should teach about AI. KI–Künstliche Intelligenz. 2016;30(1):89–93.

Kandlhofer M, Steinert M, Wagner M. AI literacy in education: developing critical thinking skills through an AI literacy programme. Eur J Open Distance E-Learning. 2016;19(1):82–93.

Zawacki-Richter O, Marín VI, Bond M, Gouverneur F. Systematic review of research on artificial intelligence applications in higher education–where are the educators? Int J Educational Technol High Educ. 2019;16(1):39.

Ng R, Sui DZ, Chen B. AI literacy for education: a four-dimensional framework. J Educational Comput Res. 2021;59(3):450–66.

Chen B, Wang X, Liu Z. Artificial intelligence in education: recent advances and future directions. Comput Educ. 2022;180:104913.

Alzoubi R, Al-Qudah Z, Al-Sharrah H. Virtual reality-based social skills training for children with autism spectrum disorder. J Med Syst. 2021;45(5):1–10.

Wang Y, Xu H, Luo X, Xie Y, Liu J. The application of artificial intelligence in autism spectrum disorder: a systematic review. J Med Syst. 2021;45(9):1–15.

Kocaballi AB, Velusamy V, Quiroz JC. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. Can J Psychiatry. 2020;65(6):381–7.

Tanaka H, Negoro H, Iwasaka H. Miko, a conversational android for promoting social communication skills in children with autism spectrum disorder. Cyberpsychology Behav Social Netw. 2021;24(3):191–6.

Liang W-C, Chen C-Y, Lee CC. Technology in education for children with autism in the United States: a systematic review and meta-synthesis of qualitative studies. Volume 175. Computers & Education; 2021.

Ienca M, Wangmo T. Ethical considerations for AI in medical education. Med Teach. 2018. https://doi.org/10.1080/0142159X.2018.1539598 .

Keshavarzi S, Ghaznavi G. Artificial intelligence-enabled technologies in autism interventions: a scoping review. J Biomed Inform. 2021;113., Article 103628. https://doi.org/10.1016/j.jbi.2020.103628 .

Gillespie-Lynch K, Kapp SK, Shane-Simpson C, Smith DS, Hutman T. Intersectionality and the study of autism. J Autism Dev Disord. 2017;47(10):3256–60. https://doi.org/10.1007/s10803-017-3168-z .

Leslie DL, Viner RM, Lomax-Smith SC. Bias in machine learning, and how to address it. J Adolesc Health. 2019;65(5):503–5. https://doi.org/10.1016/j.jadohealth.2019.08.010 .

Selbst AD, Barocas S. The intuitive appeal of explainable machines. Fordham Law Rev. 2018;87(3):1085–139.

Whitney DG, von Ameringen M, van der Maesen GP, Skinner SR, Ethans KD. A scoping review of technology-based language interventions for children with autism spectrum disorder. Rev J Autism Dev Disorders. 2019;6(2):124–38. https://doi.org/10.1007/s40489-019-00153-0 .

Avanzo M, Porzio M, Lorenzon L, Milan L, Sghedoni R, Russo G, Mettivier G. Artificial intelligence applications in medical imaging: a review of the medical physics research in Italy. Physica Med. 2021;83:221–41.

Turner Brown C, Watson ZEP, Tolson JF, Kozlowski AM. Perspectives of parenting a child with autism: differences among hispanic, African-American, Asian and caucasian parents. J Family Social Work. 2017;20(1):24–43.

Bortoletto R, Colizzi M. Cannabis Use in Autism: reasons for concern about risk for psychosis. Healthc (Basel). 2022;10(8):1553. https://doi.org/10.3390/healthcare10081553 .

Grynszpan O, Weiss PL, Perez-Diaz F, Gal E. Innovative technology-based interventions for autism spectrum disorders: a meta-analysis. Autism. 2014;18(4):346–61.

Liu X, Wu Q, Zhao W, Luo X. Technology-facilitated diagnosis and treatment of individuals with Autism Spectrum disorder: an Engineering Perspective. Appl Sci. 2017;7(10):1051. https://doi.org/10.3390/app7101051 .

Guldberg K. Evidence-based practice in autism educational research: can we bridge the research and practice gap? Oxf Rev Educ. 2017;43(2):149–61.

Kang-Yi CD, Grinker RR, Beidas R, Agha A, Russell R, Shah SB, Shea K, Mandell DS. Influence of Community-Level Cultural beliefs about autism on families’ and professionals’ care for children. Transcult Psychiatry. 2018;55(5):623–47. https://doi.org/10.1177/1363461518779831 .

Article   PubMed Central   Google Scholar  

Bateman KJ, Wilson SE. Supporting diverse learners with Autism through a culturally responsive visual communication intervention. Intervention School Clin. 2021;56(5):301–7. https://doi.org/10.1177/1053451220963090 .

Download references

Acknowledgements

The authors would like to thank all participants who contributed to the study.

This work was supported by The General Project of Beijing Postdoctoral Research Foundation in 2023, “Research on the Representation of the Tacit Knowledge of High School History Teachers Based on Natural language processing”. (Project No.2023-zz-182)

Author information

Authors and affiliations.

School of History, Capital Normal University, Beijing, China

Allameh Tabataba’i university, Tehran, Iran

Mohammad Amin Zarei & Goudarz Alibakhshi

Maraghe Branch, PhD Candidate of English Language Teaching, Islamic Azad University, Teheran, Iran

Akram Labbafi

You can also search for this author in PubMed   Google Scholar

Contributions

GA and AL conceived and designed the concept and wrote the paper, and MAZ collected the data. AL and GL wrote the manuscript. GA, GL, and MAZ edited the manuscript.

Corresponding author

Correspondence to Mohammad Amin Zarei .

Ethics declarations

Ethical approval consent.

The Ethics Committee of Allameh Tabataba’i University approved the study. The ethics committee agreed that the study was conducted based on the principles and guidelines of the institute. The patients/participants provided written informed consent to participate in this study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Li, G., Zarei, M.A., Alibakhshi, G. et al. Teachers and educators’ experiences and perceptions of artificial-powered interventions for autism groups. BMC Psychol 12 , 199 (2024). https://doi.org/10.1186/s40359-024-01664-2

Download citation

Received : 21 July 2023

Accepted : 15 March 2024

Published : 11 April 2024

DOI : https://doi.org/10.1186/s40359-024-01664-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial intelligence
• AI-empowered interventions
• Children with autism

BMC Psychology

ISSN: 2050-7283

• General enquiries: [email protected]

This paper is in the following e-collection/theme issue:

Published on 11.4.2024 in Vol 26 (2024)

Evaluating the Digital Health Experience for Patients in Primary Care: Mixed Methods Study

Authors of this article:

Original Paper

• Melinda Ada Choy 1, 2 , BMed, MMed, DCH, MD   ; 
• Kathleen O'Brien 1 , BSc, GDipStats, MBBS, DCH   ; 
• Katelyn Barnes 1, 2 , BAPSC, MND, PhD   ; 
• Elizabeth Ann Sturgiss 3 , BMed, MPH, MForensMed, PhD   ; 
• Elizabeth Rieger 1 , BA, MClinPsych, PhD   ; 
• Kirsty Douglas 1, 2 , MBBS, DipRACOG, Grad Cert HE, MD  

1 School of Medicine and Psychology, College of Health and Medicine, The Australian National University, Canberra, Australia

2 Academic Unit of General Practice, Office of Professional Leadership and Education, ACT Health Directorate, Canberra, Australia

3 School of Primary and Allied Health Care, Monash University, Melbourne, Australia

Corresponding Author:

Melinda Ada Choy, BMed, MMed, DCH, MD

School of Medicine and Psychology

College of Health and Medicine

The Australian National University

Phone: 61 51244947

Email: [email protected]

Background: The digital health divide for socioeconomic disadvantage describes a pattern in which patients considered socioeconomically disadvantaged, who are already marginalized through reduced access to face-to-face health care, are additionally hindered through less access to patient-initiated digital health. A comprehensive understanding of how patients with socioeconomic disadvantage access and experience digital health is essential for improving the digital health divide. Primary care patients, especially those with chronic disease, have experience of the stages of initial help seeking and self-management of their health, which renders them a key demographic for research on patient-initiated digital health access.

Objective: This study aims to provide comprehensive primary mixed methods data on the patient experience of barriers to digital health access, with a focus on the digital health divide.

Methods: We applied an exploratory mixed methods design to ensure that our survey was primarily shaped by the experiences of our interviewees. First, we qualitatively explored the experience of digital health for 19 patients with socioeconomic disadvantage and chronic disease and second, we quantitatively measured some of these findings by designing and administering a survey to 487 Australian general practice patients from 24 general practices.

Results: In our qualitative first phase, the key barriers found to accessing digital health included (1) strong patient preference for human-based health services; (2) low trust in digital health services; (3) high financial costs of necessary tools, maintenance, and repairs; (4) poor publicly available internet access options; (5) reduced capacity to engage due to increased life pressures; and (6) low self-efficacy and confidence in using digital health. In our quantitative second phase, 31% (151/487) of the survey participants were found to have never used a form of digital health, while 10.7% (52/487) were low- to medium-frequency users and 48.5% (236/487) were high-frequency users. High-frequency users were more likely to be interested in digital health and had higher self-efficacy. Low-frequency users were more likely to report difficulty affording the financial costs needed for digital access.

Conclusions: While general digital interest, financial cost, and digital health literacy and empowerment are clear factors in digital health access in a broad primary care population, the digital health divide is also facilitated in part by a stepped series of complex and cumulative barriers. Genuinely improving digital health access for 1 cohort or even 1 person requires a series of multiple different interventions tailored to specific sequential barriers. Within primary care, patient-centered care that continues to recognize the complex individual needs of, and barriers facing, each patient should be part of addressing the digital health divide.

Introduction

The promise of ehealth.

The rapid growth of digital health, sped up by the COVID-19 pandemic and associated lockdowns, brings the promise of improved health care efficiency, empowerment of consumers, and health care equity [ 1 ]. Digital health is the use of information and communication technology to improve health [ 2 ]. eHealth, which is a type of digital health, refers to the use of internet-based technology for health care and can be used by systems, providers, and patients [ 2 ]. At the time of this study (before COVID-19), examples of eHealth used by patients in Australia included searching for web-based health information, booking appointments on the web, participating in online peer-support health forums, using mobile phone health apps (mobile health), emailing health care providers, and patient portals for electronic health records.

Digital health is expected to improve chronic disease management and has already shown great potential in improving chronic disease health outcomes [ 3 , 4 ]. Just under half of the Australian population (47.3%) has at least 1 chronic disease [ 5 ]. Rates of chronic disease and complications from chronic disease are overrepresented among those with socioeconomic disadvantage [ 6 ]. Therefore, patients with chronic disease and socioeconomic disadvantage have a greater need for the potential benefits of digital health, such as an improvement in their health outcomes. However, there is a risk that those who could benefit most from digital health services are the least likely to receive them, exemplifying the inverse care law in the digital age by Hart [ 7 ].

Our Current Understanding of the Digital Health Divide

While the rapid growth of digital health brings the promise of health care equity, it may also intensify existing inequities [ 8 ]. The digital health divide for socioeconomic disadvantage describes a pattern in which patients considered socioeconomically disadvantaged who are already marginalized through poor access to traditional health care are additionally hindered through poor access to digital health [ 9 ]. In Australia, only 67.4% of households in the lowest household income quintile have home internet access, compared to 86% of the general population and 96.9% of households in the highest household income quintile [ 10 ]. Survey-based studies have also shown that even with internet access, effective eHealth use is lower in populations considered disadvantaged, which speaks to broader barriers to digital health access [ 11 ].

The ongoing COVID-19 global pandemic has sped up digital health transitions with the rapid uptake of telephone and video consultations, e-prescription, and the ongoing rollout of e-mental health in Australia. These have supported the continuation of health care delivery while limiting physical contact and the pandemic spread; however, the early evidence shows that the digital health divide remains problematic. A rapid review identified challenges with reduced digital access and digital literacy among the older adults and racial and ethnic minority groups, which are both groups at greater health risk from COVID-19 infections [ 12 ]. An Australian population study showed that the rapid uptake of telehealth during peak pandemic was not uniform, with the older adults, very young, and those with limited English language proficiency having a lower uptake of general practitioner (GP) telehealth services [ 13 ].

To ensure that digital health improves health care outcome gaps, it is essential to better understand the nature and nuance of the digital health divide for socioeconomic disadvantage. The nature of the digital health divide for socioeconomic disadvantage has been explored primarily through quantitative survey data, some qualitative papers, a few mixed methods papers, and systematic reviews [ 11 , 14 - 16 ]. Identified barriers include a lack of physical hardware and adequate internet bandwidth, a reduced inclination to seek out digital health, and a low ability and confidence to use digital health effectively [ 16 ]. The few mixed methods studies that exist on the digital health divide generally triangulate quantitative and qualitative data on a specific disease type or population subgroup to draw a combined conclusion [ 17 , 18 ]. These studies have found digital health access to be associated with education, ethnicity, and gender as well as trust, complementary face-to-face services, and the desire for alternative sources of information [ 17 , 19 ].

What This Work Adds

This project sought to extend previous research by using an exploratory mixed methods design to ensure that the first step and driver of our survey of a larger population was primarily shaped by the experiences of our interviewees within primary care. This differs from the triangulation method, which places the qualitative and quantitative data as equal first contributors to the findings and does not allow one type of data to determine the direction of the other [ 18 ]. We qualitatively explored the experience of digital health for patients with socioeconomic disadvantage and chronic disease and then quantitatively measured some of the qualitative findings via a survey of the Australian general practice patient population. Our key objective was to provide comprehensive primary mixed methods data, describing the experience and extent of barriers to accessing digital health and its benefits, with a focus on the digital health divide. We completed this research in a primary care context to investigate a diverse community-based population with conceivable reasons to seek digital help in managing their health. Findings from this mixed methods study were intended to provide health care providers and policy makers with a more detailed understanding of how specific barriers affect different aspects or steps of accessing digital health. Ultimately, understanding digital health access can influence the future design and implementation of digital health services by more effectively avoiding certain barriers or building in enablers to achieve improved digital health access not only for everyone but also especially for those in need.

Study Design

We conducted a sequential exploratory mixed methods study to explore a complex phenomenon in depth and then measure its prevalence. We qualitatively explored the experience of digital health for patients with chronic disease and socioeconomic disadvantage in the first phase. Data from the first phase informed a quantitative survey of the phenomenon across a wider population in the second phase [ 18 ]. Both stages of research were conducted before the COVID-19 pandemic in Australia.

Recruitment

Qualitative phase participants.

The eligibility criteria for the qualitative phase were as follows: English-speaking adults aged ≥18 years with at least 1 self-reported chronic disease and 1 marker of socioeconomic disadvantage (indicated by ownership of a Health Care Card or receiving a disability pension, unemployment, or a user of public housing). A chronic disease was defined to potential participants as a diagnosed long-term health condition that had lasted at least 6 months (or is expected to last for at least 6 months; examples are listed in Multimedia Appendix 1 ). The markers of socioeconomic disadvantage we used to identify potential participants were based on criteria typically used by local general practices to determine which patients can have lower or no out-of-pocket expenses. Apart from unemployment, the 3 other criteria to identify socioeconomic disadvantage are means-tested government-allocated public social services [ 20 ]. Qualitative phase participants were recruited from May to July 2019 through 3 general practices and 1 service organization that serve populations considered socioeconomically disadvantaged across urban, regional, and rural regions in the Australian Capital Territory and South Eastern New South Wales. A total of 2 recruitment methods were used in consultation with and as per the choice of the participating organizations. Potential participants were either provided with an opportunity to engage with researchers (KB and MAC) in the general practice waiting room or identified by the practice or organization as suitable for an interview. Interested participants were given a detailed verbal and written description of the project in a private space before providing written consent to be interviewed. All interview participants received an Aus $50 (US $32.68) grocery shopping voucher in acknowledgment of their time.

Quantitative Phase Participants

Eligibility for the quantitative phase was English-speaking adults aged ≥18 years. The eligibility criteria for the quantitative phase were deliberately broader than those for the qualitative phase to achieve a larger sample size within the limitations of recruitment and with the intention that the factors of socioeconomic disadvantage and having a chronic disease could be compared to the digital health access of a more general population. The quantitative phase participants were recruited from November 2019 to February 2020. Study information and paper-based surveys were distributed and collected through 24 general practices across the Australian Capital Territory and South Eastern New South Wales regions, with an option for web-based completion.

Ethical Considerations

Qualitative and quantitative phase research protocols, including the participant information sheet, were approved by the Australian Capital Territory Health Human Research Ethics Committee (2019/ETH/00013) and the Australian National University Human Research Ethics Committee (2019/ETH00003). Qualitative phase participants were given a verbal and written explanation of the study, including how and when they could opt out, before they provided written consent. All interview participants received an Aus $50 (US $32.68) grocery shopping voucher in acknowledgment of their time. Quantitative participants were given a written explanation and their informed consent was implied by return of a completed survey. Participants in both phases of the study were told that all their data was deidentified. Consent was implied through the return of a completed survey.

Qualitative Data Collection and Analysis

Participants were purposively sampled to represent a range in age, gender, degree of socioeconomic disadvantage, and experience of digital health. The sampling and sample size were reviewed regularly by the research team as the interviews were being completed to identify potential thematic saturation.

The interview guide was developed by the research team based on a review of the literature and the patient dimensions of the framework of access by Levesque et al [ 21 ]. The framework by Levesque et al [ 21 ] is a conceptualization of health care access comprising 5 service and patient dimensions of accessibility and ability. The patient dimensions are as follows: (1) ability to perceive, (2) ability to seek, (3) ability to reach, (4) ability to pay, and (5) ability to engage [ 21 ]. The key interview topics included (1) digital health use and access, including facilitators and barriers; (2) attitudes toward digital health; and (3) self-perception of digital health skills and potential training. The interview guide was reviewed for face and content validity by the whole research team, a patient advocate, a digital inclusion charity representative, and the general practices where recruitment occurred. The questions and guide were iteratively refined by the research team to ensure relevance and support reaching data saturation. The interview guide has been provided as Multimedia Appendix 1 . The interviews, which took 45 minutes on average, were taped and transcribed. An interview summary sheet and reflective journal were completed by the interviewer after each interview to also capture nonverbal cues and tone.

Interview transcriptions were coded and processed by inductive thematic analysis. Data collection and analysis were completed in parallel to support the identification of data saturation. Data saturation was defined as no significant new information arising from new interviews and was identified by discussion with the research team [ 22 ]. The 2 interviewers (MAC and KB) independently coded the first 5 transcripts and reflected on them with another researcher (EAS) to ensure intercoder validity and reliability. The rest of the interviews were coded independently by the 2 interviewers, who regularly met to reflect on emerging themes and thematic saturation. Data saturation was initially indicated after 15 interviews and subsequently confirmed with a total of 19 interviews. Coding disagreements and theme development were discussed with at least 1 other researcher (EAS, ER, and KD). Thematic saturation and the final themes were agreed upon by the entire research team.

Quantitative Survey Development

The final themes derived in the qualitative phase of the project guided the specific quantitative phase research questions. The final themes were a list of ordered cumulative barriers experienced by participants in accessing digital health and its benefits ( Figure 1 ). The quantitative survey was designed to test the association between barriers to access and the frequency of use of digital health as a proxy measure for digital health access.

In the survey, the participants were asked about their demographic details, health and chronic diseases, knowledge, use and experience of digital health tools, internet access, perception of digital resource affordability, trust in digital health and traditional health services, perceived capability, health care empowerment, eHealth literacy, and relationship with their GP.

Existing scales and questions from the literature and standardized Australian-based surveys were used whenever possible. We used selected questions and scales from the Australian Bureau of Statistics standards, the eHealth Literacy Scale (eHEALS), the eHealth Literacy Questionnaire, and the Southgate Institute for Health Society and Equity [ 17 , 23 - 26 ]. We adapted other scales from the ICEpop Capability Measure for Adults, the Health Care Empowerment Inventory (HCEI), the Patient-Doctor Relationship Questionnaire, and the Chao continuity questionnaire [ 23 , 27 - 29 ]. Where an existing scale to measure a barrier or theme did not exist, the research team designed the questions based on the literature. Our questions around the frequency of digital health use were informed by multiple existing Australian-based surveys on general technology use [ 30 , 31 ]. Most of the questions used a Likert scale. Every choice regarding the design, adaptation, or copy of questions for the survey was influenced by the qualitative findings and decided on by full agreement among the 2 researchers who completed and coded the interviews. A complete copy of the survey is provided in Multimedia Appendix 2 .

Pilot-testing of the survey was completed with 5 patients, 2 experts on digital inclusion, and 3 local GPs for both the paper surveys and web-based surveys via Qualtrics Core XM (Qualtrics LLC). The resulting feedback on face and content validity, functionality of the survey logic, and feasibility of questionnaire completion was incorporated into the final version of the survey.

The survey was offered on paper with a participant information sheet, which gave the patients the option to complete the web-based survey. The survey was handed out to every patient on paper to avoid sampling bias through the exclusion of participants who could not complete the web-based survey [ 32 ].

Quantitative Data Treatment and Analysis

Data were exported from Qualtrics Core XM to an SPSS (version 26; IBM Corp) data set. Data cleaning and screening were undertaken (KB and KO).

Descriptive statistics (number and percentage) were used to summarize participant characteristics, preference measures, and frequency of eHealth use. Significance testing was conducted using chi-square tests, with a threshold of P <.05; effect sizes were measured by the φ coefficient for 2×2 comparisons and Cramer V statistic for all others. Where the cells sizes were too small, the categories were collapsed for the purposes of significance testing. The interpretation of effect sizes was as per the study by Cohen [ 33 ]. The analysis was conducted in SPSS and SAS (version 9.4; SAS Institute).

Participant Characteristics

Participants’ self-reported characteristics included gender, indigenous status, income category, highest level of education, marital status, and language spoken at home.

Age was derived from participant-reported year of birth and year of survey completion as of 2019 and stratified into age groups. The state or territory of residence was derived from the participant-reported postcode. The remoteness area was derived using the postcode reported by the participants and mapped to a modified concordance from the Australian Bureau of Statistics. Occupation-free text responses were coded using the Australian Bureau of Statistics Census statistics level 1 and 2 descriptors. The country of birth was mapped to Australia, other Organisation for Economic Cooperation and Development countries, and non–Organisation for Economic Cooperation and Development countries.

Frequency of eHealth Use

A summary measure of the frequency of eHealth use was derived from the questions on the use of different types of eHealth.

Specifically, respondents were asked if they had ever used any form of web-based health (“eHealth“) and, if so, to rate how often (never, at least once, every now and then, and most days) against 6 types of “eHealth” (searching for health information online, booking appointments online, emailing health care providers, using health-related mobile phone apps, accessing My Health Record, and accessing online health forums). The frequency of eHealth use was then classified as follows:

• High user: answered “most days” to at least 1 question on eHealth use OR answered “every now and then” to at least 2 questions on eHealth use
• Never user: answered “no” to having ever used any form of eHealth OR “never” to all 6 questions on eHealth use
• Low or medium user: all other respondents.

The frequency of eHealth use was reported as unweighted descriptive statistics (counts and percentages) against demographic characteristics and for the elements of each of the themes identified in phase 1.

Overview of Key Themes

Data were reported against the 6 themes from the phase 1 results of preference, trust, cost, structural access, capacity to engage, and self-efficacy. Where the components of trust, cost, capacity to engage, and self-efficacy had missing data (for less than half of the components only), mean imputation was used to minimize data loss. For each theme, the analysis excluded those for whom the frequency of eHealth use was unknown.

Preference measures (survey section D1 parts 1 to 3) asked participants to report against measures with a 4-point Likert scale (strongly disagree, disagree, agree, and strongly agree). Chi-square tests were conducted after the categories were condensed into 2 by combining strongly disagree and as well as combining strongly agree and agree.

Summary measures for trust were created in 4 domains: trust from the eHealth Literacy Questionnaire (survey section D1 parts 4 to 8), trust from Southgate—GPs, specialists, or allied health (survey section D2 parts 1 to 5), trust from Southgate—digital health (survey section D2 parts 6, 7, 9, and 10), and trust from Southgate—books or pamphlets (survey section D2 part 8). The data were grouped as low, moderate, and high trust based on the assigned scores from the component data. Chi-square tests were conducted comparing low-to-moderate trust against high trust for GP, specialists, or allied health and comparing low trust against moderate-to-high trust for book or pamphlet.

Summary measures for cost were created from survey item C10. To measure cost, participants were asked about whether they considered certain items or services to be affordable. These included cost items mentioned in the qualitative phase interviews relating to mobile phones (1 that connects to the internet, 1 with enough memory space to download apps, downloads or apps requiring payment, repairs, and maintenance costs), having an iPad or tablet with internet connectivity, a home computer or laptop (owning, repairs, and maintenance), home fixed internet access, and an adequate monthly data allowance. These 9 items were scored as “yes definitely”=1 or 0 otherwise. Chi-square tests were conducted with never and low or medium eHealth users combined.

Structural Access

Structural access included asking where the internet is used by participants (survey section C8) and factors relating to internet access (survey section C8 parts 1-3) reporting against a 4-point Likert scale (strongly disagree, disagree, agree, and strongly agree). Chi-square tests were conducted with strongly disagree, disagree, agree, or strongly agree, and never, low, or medium eHealth use combined.

Capacity to Engage

Summary measures for capacity to engage were created from survey section E1. To measure the capacity to engage, participants were asked about feeling “settled and secure,” “being independent,” and “achievement and progress” as an adaptation of the ICEpop Capability Measure for Adults [ 27 ], reporting against a 4-point Likert-like scale. Responses were scored from 1 (“I am unable to feel settled and secure in any areas of my life”) to 4 (“I am able to feel settled and secure in all areas of my life”).

The summary capacity measure was derived by the summation of responses across the 3 questions, which were classified into 4 groups, A to D, based on these scores. Where fewer than half of the responses were missing, mean imputation was used; otherwise, the record was excluded. Groups A and B were combined for significance testing.

Self-Efficacy

Summary measures for self-efficacy were adapted from the eHEALS (E3) and the HCEI (E2) [ 23 , 24 ].

Survey section E3—eHEALS—comprised 8 questions, with participants reporting against a 5-point Likert scale for each (strongly disagree, disagree, neither, agree, and strongly agree). These responses were assigned 1 to 5 points, respectively. The summary eHEALS measure was derived by the summation of responses across the 8 questions, which were classified into 5 groups, A to E, based on these scores. Where fewer than half of the responses were missing, mean imputation was used; otherwise, the record was excluded. Groups A to C and D to E were combined for significance testing.

Survey section E2—HCEI—comprised 5 questions, with participants reporting against a 5-point Likert scale for each (strongly disagree, disagree, neither, agree, and strongly agree). Strongly disagree and disagree and neither were combined, and similarly agree and strongly agree were combined for significance testing.

Qualitative Results

The demographic characteristics of the patients that we interviewed are presented in Table 1 .

The key barriers found to accessing digital health included (1) strong patient preference for human-based health services; (2) low trust in digital health services; (3) high financial costs of necessary tools, maintenance, and repairs; (4) poor publicly available internet access options; (5) reduced capacity to engage due to increased life pressures; and (6) low self-efficacy and confidence in using digital health.

Rather than being an equal list of factors, our interviewees described these barriers as a stepped series of cumulative hurdles, which is illustrated in Figure 1 . Initial issues of preference and trust were foundational to a person even when considering the option of digital health, while digital health confidence and literacy were barriers to full engagement with and optimal use of digital health. Alternatively, interviewees who did use digital health had been enabled by the same factors that were barriers to others.

a GP: general practitioner.

b Multiple answers per respondent.

Strong Patient Preference for Human-Based Health Services

Some patients expressed a strong preference for human-based health services rather than digital health services. In answer to a question about how digital health services could be improved, a patient said the following:

Well, having an option where you can actually bypass actually having to go through the app and actually talk directly to someone. [Participant #10]

For some patients, this preference for human-based health services appeared to be related to a lack of exposure to eHealth. These patients were not at all interested in or had never thought about digital health options. A participant responded the following to the interviewer’s questions:

Interviewer: So when...something feels not right, how do you find out what’s going on?

Respondent: I talk to Doctor XX.

Interviewer: Do you ever Google your symptoms or look online for information?

Respondent: No, I have never even thought of doing that actually. [Participant #11]

For other patients, their preference for human-based health care stemmed from negative experiences with technology. These patients reported actively disliking computers and technology in general and were generally frustrated with what they saw as the pitfalls of technology. A patient stated the following:

If computers and internet weren’t so frigging slow because everything is on like the slowest speed network ever and there’s ads blocking everything. Ads, (expletive) ads. [Participant #9]

A patient felt that he was pushed out of the workforce due his inability to keep up with technology-based changes and thus made a decision to never own a computer:

But, you know, in those days when I was a lot younger those sorts of things weren’t about and they’re just going ahead in leaps and bounds and that’s one of the reasons why I retired early. I retired at 63 because it was just moving too fast and it’s all computers and all those sorts of things and I just couldn’t keep up. [Participant #17]

Low Trust in Digital Health Services

Several patients described low trust levels for digital and internet-based technology in general. Their low trust was generally based on stories they had heard of other people’s negative experiences. A patient said the following:

I don’t trust the internet to be quite honest. You hear all these stories about people getting ripped off and I’ve worked too hard to get what I’ve got rather than let some clown get it on the internet for me. [Participant #11]

Some of this distrust was specific to eHealth. For example, some patients were highly suspicious of the government’s motives with regard to digital health and were concerned about the privacy of their health information, which made them hesitant about the concept of a universal electronic health record. In response to the interviewer’s question, a participant said the following:

Interviewer: Are there any other ways you think that eHealth might help you?

Respondent: I’m sorry but it just keeps coming back to me, Big Brother. [Participant #7]

Another participant said the following:

I just would run a mile from it because I just wouldn’t trust it. It wouldn’t be used to, as I said, for insurance or job information. [Participant #16]

High Financial Costs of the Necessary Tools, Maintenance, and Repairs

A wide variety of patients described affordability issues across several different aspects of the costs involved in digital health. They expressed difficulty in paying for the following items: a mobile phone that could connect to the internet, a mobile phone with enough memory space to download apps, mobile phone apps requiring extra payment without advertisements, mobile phone repair costs such as a broken screen, a computer or laptop, home internet access, and adequate monthly data allowance and speeds to functionally use the internet. Current popular payment systems, such as plans, were not feasible for some patients. A participant stated the following:

I don’t have a computer...I’m not in the income bracket to own a computer really. Like I could, if I got one on a plan kind of thing or if I saved up for x-amount of time. But then like if I was going on the plan I’d be paying interest for having it on like lay-buy kind of thing, paying it off, and if it ever got lost or stolen I would still have to repay that off, which is always a hassle. And yeah. Yeah, I’m like financially not in the state where I’m able to...own a computer right now as I’m kind of paying off a number of debts. [Participant #9]

Poor Publicly Available Internet Access Options

Some patients described struggling without home internet access. While they noted some cost-free public internet access points, such as libraries, hotel bars, and restaurants, they often found these to be inconvenient, lacking in privacy, and constituting low-quality options for digital health. A patient stated the following:

...it’s incredibly slow at the library. And I know why...a friend I went to school with used to belong to the council and the way they set it up, they just got the raw end of the stick and it is really, really slow. It’s bizarre but you can go to the X Hotel and it’s heaps quicker. [Participant #15]

In response to the interviewer's question, a participant said the following:

Interviewer: And do you feel comfortable doing private stuff on computers at the library...?

Respondent: Not really, no, but I don’t have any other choice, so, yeah. [Participant #9]

Reduced Capacity to Engage Due to Increased Life Pressures

When discussing why they were not using digital health or why they had stopped using digital health, patients often described significant competing priorities and life pressures that affected their capacity to engage. An unemployed patient mentioned that his time and energy on the internet were focused primarily on finding work and that he barely had time to focus on his health in general, let alone engage in digital health.

Other patients reported that they often felt that their ability to learn about and spend time on digital health was taken up by caring for sick family members, paying basic bills, or learning English. Some patients said that the time they would have spent learning digital skills when they were growing up had been lost to adverse life circumstances such as being in jail:

So we didn’t have computers in the house when I was growing up. And I didn’t know I’ve never...I’ve been in and out of jail for 28 odd years so it sort of takes away from learning from this cause it’s a whole different… it’s a whole different way of using a telephone from a prison. [Participant #11]

Low Self-Efficacy and Confidence in Starting the Digital Health Process

Some patients had a pervasive self-perception of being slow learners and being unable to use technology. Their stories of being unconfident learners seemed to stem from the fact that they had been told throughout their lives that they were intellectually behind. A patient said the following:

The computer people...wouldn’t take my calls because I’ve always been dumb with that sort of stuff. Like I only found out this later on in life, but I’m actually severely numerically dyslexic. Like I have to triple-check everything with numbers. [Participant #7]

Another patient stated the following:

I like went to two English classes like a normal English class with all the kids and then another English class with about seven kids in there because I just couldn’t I don’t know maybe because I spoke another language at home and they sort of like know I was a bit backward. [Participant #6]

These patients and others had multiple missing pieces of information that they felt made it harder to engage in digital health compared to “easier” human-based services. A patient said the following:

Yeah I’ve heard of booking online but I just I don’t know I find it easier just to ring up. And I’ll answer an email from a health care provider but I wouldn’t know where to start to look for their email address. [Participant #11]

In contrast, the patients who did connect with digital health described themselves as independent question askers and proactive people. Even when they did not know how to use a specific digital health tool, they were confident in attempting to and asking for help when they needed it. A patient said the following:

I’m a “I will find my way through this, no matter how long it takes me” kind of person. So maybe it’s more my personality...If I have to ask for help from somewhere, wherever it is, I will definitely do that. [Participant #3]

Quantitative Results

A total of 487 valid survey responses were received from participants across 24 general practices. The participant characteristics are presented in detail in Table S1 in Multimedia Appendix 3 .

The mean age of the participants was approximately 50 years (females 48.9, SD 19.4 years; males 52.8, SD 20.0 years), and 68.2% (332/487) of the participants identified as female. Overall, 34.3% (151/439) of respondents reported never using eHealth, and 53.8% (236/439) reported high eHealth use.

There were statistically significant ( P <.05) differences in the frequency of eHealth use in terms of age group, gender, state, remoteness, highest level of education, employment status, occupation group, marital status, and language spoken at home, with effect sizes being small to medium. Specifically, high eHealth characteristics were associated with younger age, being female, living in an urban area, and being employed.

Table 2 presents the frequency of eHealth use against 3 internet preference questions.

Preference for using the internet and technology in general and for health needs in particular were significantly related to the frequency of eHealth use ( P <.05 for each), with the effect sizes being small to medium.

a Excludes those for whom frequency of eHealth use is unknown.

b Chi-square tests conducted with strongly disagree and disagree combined, and agree and strongly agree combined.

Table 3 presents the frequency of eHealth use against 4 measures of trust.

The degree of trust was not statistically significantly different for the frequency of eHealth use for any of the domains.

b eHLQ: eHealth Literacy Questionnaire.

c Derived from survey question D1, parts 4 to 8. Mean imputation used where ≤2 responses were missing. If >2 responses were missing, the records were excluded.

d Derived from survey question D2, parts 1 to 5. Mean imputation used where ≤2 responses were missing. If >2 responses were missing, the records were excluded.

e Chi-square test conducted comparing low-to-moderate trust against high trust.

f Derived from survey question D2, parts 6, 7, 9, and 10. Mean imputation used where ≤2 responses were missing. If >2 responses were missing, the records were excluded.

g Derived from survey question D2 part 8.

h Chi-square test conducted comparing low trust against moderate-to-high trust.

Affordability of items and services was reported as No cost difficulty or Cost difficulty. eHealth frequency of use responses were available for 273 participants; among those with no cost difficulty , 1% (2/204) were never users, 14.2% (29/204) were low or medium users, and 84.8% (173/204) were high users of eHealth; among those with cost difficulty , 1% (1/69) were never users, 26% (18/69) were low or medium users, and 73% (50/69) were high users. There was a statistically significant difference in the presence of cost as a barrier between never and low or medium eHealth users compared to high users ( χ 2 1 =5.25; P =.02), although the effect size was small.

Table 4 presents the frequency of eHealth use for elements of structural access.

Quality of internet access and feeling limited in access to the internet were significantly associated with frequency of eHealth use ( P <.05), although the effect sizes were small.

b N/A: not applicable (cell sizes insufficient for chi-square test).

c Chi-square tests conducted with strongly disagree and disagree combined, agree and strongly agree combined, and never and low or medium categories combined.

Table 5 presents the frequency of eHealth use against respondents’ capacity to engage.

Capacity to engage was not significantly different for the frequency of eHealth use ( P =.54). 

b Derived from survey item E1. Where 1 response was missing, the mean imputation was used. If >1 response was missing, the record was excluded.

c Chi-square tests conducted with groups A and B combined.

Table 6 presents the frequency of eHealth use for elements of self-efficacy.

Statistically significant results were observed for the relationship between self-efficacy by eHEALS (moderate effect size) and frequency of eHealth use as well as for some of the questions from the HCEI (reliance on health professionals or others to access and explain information; small effect size; P <.05).

b eHEALS: eHealth Literacy Scale.

c eHEALS derived from item E3 (8 parts). Where ≤ 4 responses were missing, mean imputation was used. If >4 responses were missing, the records were excluded. Groups A to C as well as groups D to E were combined for the chi-square test.

d Strongly disagree, disagree, neither, and agree or strongly agree combined for significance testing.

Principal Findings

This paper reports on the findings of a sequential exploratory mixed methods study on the barriers to digital health access for a group of patients in Australian family medicine, with a particular focus on chronic disease and socioeconomic disadvantage.

In the qualitative first phase, the patients with socioeconomic disadvantage and chronic disease described 6 cumulative barriers, as demonstrated in Figure 1 . Many nonusers of digital health preferred human-based services and were not interested in technology, while others were highly suspicious of the technology in general. Some digitally interested patients could not afford quality hardware and internet connectivity, a barrier that was doubled by low quality and privacy when accessing publicly available internet connections. Furthermore, although some digitally interested patients had internet access, their urgent life circumstances left scarce opportunity to access digital health and develop digital health skills and confidence.

In our quantitative second phase, 31% (151/487) of the survey participants from Australian general practices were found to have never used a form of digital health. Survey participants were more likely to use digital health tools frequently when they also had a general digital interest and a digital health interest. Those who did not frequently access digital health were more likely to report difficulty affording the financial costs needed for digital access. The survey participants who frequently accessed digital health were more likely to have high eHealth literacy and high levels of patient empowerment.

Comparison With Prior Work

In terms of general digital health access, the finding that 31% (151/487) of the survey participants had never used one of the described forms of eHealth is in keeping with an Australian-based general digital participation study that found that approximately 9% of the participants were nonusers and 17% rarely engaged with the internet at all [ 34 ]. With regard to the digital health divide, another Australian-based digital health divide study found that increased age, living in a lower socioeconomic area, being Aboriginal or Torres Strait Islander, being male, and having no tertiary education were factors negatively associated with access to digital health services [ 17 ]. Their findings correspond to our findings that higher-frequency users of eHealth were associated with younger age, being female, living in an urban area, and being employed. Both studies reinforce the evidence of the digital health divide based on gender, age, and socioeconomic disadvantage in Australia.

With regard to digital health barriers, our findings provide expanded details on the range of digital health items and services that present a cost barrier to consumers. Affordability is a known factor in digital access and digital health access, and it is measured often by general self-report or relative expenditure on internet access to income [ 30 ]. Our study revealed the comprehensive list of relevant costs for patients. Our study also demonstrated factors of cost affordability beyond the dollar value of an item, as interviewees described the struggle of using slow public internet access without privacy features and the risks involved in buying a computer in installments. When we reflected on the complexity and detail of the cost barrier in our survey, participants demonstrated a clear association between cost and the frequency of digital health use. This suggests that a way to improve digital health access for some people is to improve the quality, security, and accessibility of public internet access options as well as to provide free or subsidized hardware, internet connection, and maintenance options for those in need, work that is being done by at least 1 digital inclusion charity in the United Kingdom [ 35 ].

Many studies recognize the factors of eHealth literacy and digital confidence for beneficial digital health access [ 36 ]. Our interviews demonstrated that some patients with socioeconomic disadvantage have low digital confidence, but that this is often underlined by a socially reinforced lifelong low self-confidence in their intellectual ability. In contrast, active users, regardless of other demographic factors, described themselves as innately proactive question askers. This was reinforced by our finding of a relationship between health care empowerment and the frequency of eHealth use. This suggests that while digital health education and eHealth literacy programs can improve access for some patients, broader and deeper long-term solutions addressing socioeconomic drivers of digital exclusion are needed to improve digital health access for some patients with socioeconomic disadvantage [ 8 ]. The deep permeation of socially enforced low self-confidence and lifelong poverty experienced by some interviewees demonstrate that the provision of free hardware and a class on digital health skills can be, for some, a superficial offering when the key underlying factor is persistent general socioeconomic inequality.

The digital health divide literature tends to identify the digital health divide, the factors and barriers that contribute to it, and the potential for it to widen if not specifically addressed [ 16 ]. Our findings have also identified the divide and the barriers, but what this study adds through our qualitative phase in particular is a description of the complex interaction of those barriers and the stepped nature of some of those barriers as part of the individual’s experience in trying to access digital health.

Strengths and Limitations

A key strength of this study is the use of a sequential exploratory mixed methods design. The initial qualitative phase guided a phenomenological exploration of digital health access experiences for patients with chronic disease and socioeconomic disadvantage. Our results in both study phases stem from the patients’ real-life experiences of digital health access. While some of our results echo the findings of other survey-based studies on general digital and digital health participation, our method revealed a greater depth and detail of some of these barriers, as demonstrated in how our findings compare to prior work.

As mentioned previously, the emphasis of this study on the qualitative first phase is a strength that helped describe the interactions between different barriers. The interviewees described their experiences as cumulative unequal stepped barriers rather than as producing a nonordered list of equal barriers. These findings expand on the known complexity of the issue of digital exclusion and add weight to the understanding that improving digital health access needs diverse, complex solutions [ 17 ]. There is no panacea for every individual’s digital health access, and thus, patient-centered digital health services, often guided by health professionals within the continuity of primary care, are also required to address the digital health divide [ 37 ].

While the sequential exploratory design is a strength of the study, it also created some limitations for the second quantitative phase. Our commitment to using the qualitative interview findings to inform the survey questions meant that we were unable to use previously validated scales for every question and that our results were less likely to lead to a normal distribution. This likely affected our ability to demonstrate significant associations for some barriers. We expect that further modeling is required to control for baseline characteristics and determine barrier patterns for different types of users.

One strength of this study is that the survey was administered to a broad population of Australian family medicine patients with diverse patterns of health via both paper-based and digital options. Many other digital health studies use solely digital surveys, which can affect the sample. However, we cannot draw conclusions from our survey about patients with chronic disease due to the limitations of the sample size for these subgroups.

Another sample-based limitation of this study was that our qualitative population did not include anyone aged from 18 to 24 years, despite multiple efforts to recruit. Future research will hopefully address this demographic more specifically.

While not strictly a limitation, we recognize that because this research was before COVID-19, it did not include questions about telehealth, which has become much more mainstream in recent years. The patients may also have changed their frequency of eHealth use because of COVID-19 and an increased reliance on digital services in general. Future work in this area or future versions of this survey should include telehealth and acknowledge the impact of COVID-19. However, the larger concept of the digital health divide exists before and after COVID-19, and in fact, our widespread increased reliance on digital services makes the digital divide an even more pressing issue [ 12 ].

Conclusions

The experience of digital health access across Australian primary care is highly variable and more difficult to access for those with socioeconomic disadvantage. While general digital interest, financial cost, and digital health literacy and empowerment are clear factors in digital health access in a broad primary care population, the digital health divide is also facilitated in part by a stepped series of complex and cumulative barriers.

Genuinely improving digital health access for 1 cohort or even 1 person requires a series of multiple different interventions tailored to specific sequential barriers. Given the rapid expansion of digital health during the global COVID-19 pandemic, attention to these issues is necessary if we are to avoid entrenching inequities in access to health care. Within primary care, patient-centered care that continues to recognize the complex individual needs of, and barriers facing, each patient should be a part of addressing the digital health divide.

Acknowledgments

The authors are thankful to the patients who shared their experiences with them via interview and survey completion. The authors are also very grateful to the general practices in the Australian Capital Territory and New South Wales who kindly gave their time and effort to help organize interviews, administer, and post surveys in the midst of the stress of day-to-day practice life and the bushfires of 2018-2019. The authors thank and acknowledge the creators of the eHealth Literacy Scale, the eHealth Literacy Questionnaire, the ICEpop Capability Measure for Adults, the Health Care Empowerment Inventory, the Patient-Doctor Relationship Questionnaire, the Chao continuity questionnaire, and the Southgate Institute for Health Society and Equity for their generosity in sharing their work with the authors [ 17 , 19 - 25 ]. This study would not have been possible without the support of the administrative team of the Academic Unit of General Practice. This project was funded by the Royal Australian College of General Practitioners (RACGP) through the RACGP Foundation IPN Medical Centres Grant, and the authors gratefully acknowledge their support.

Data Availability

The data sets generated during this study are not publicly available due to the nature of our original ethics approval but are available from the corresponding author on reasonable request.

Authors' Contributions

MAC acquired the funding, conceptualized the project, and organized interview recruitment. MAC and KB conducted interviews and analyzed the qualitative data. EAS, ER, and KD contributed to project planning, supervision and qualitative data analysis. MAC, KB and KO wrote the survey and planned quantitative data analysis. MAC and KB recruited practices for survey administration. KO and KB conducted the quantitative data analysis. MAC and KO, with KB drafted the paper. EAS, ER, and KD helped with reviewing and editing the paper.

Conflicts of Interest

None declared.

Phase 1 interview guide.

Phase 2 survey: eHealth and digital divide.

Phase 2 participant characteristics by frequency of eHealth use.

• Eysenbach G. What is e-health? J Med Internet Res. 2001;3(2):E20. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Iyawa GE, Herselman M, Botha A. Digital health innovation ecosystems: from systematic literature review to conceptual framework. Procedia Comput Sci. 2016;100:244-252. [ FREE Full text ] [ CrossRef ]
• Berrouiguet S, Baca-García E, Brandt S, Walter M, Courtet P. Fundamentals for future mobile-health (mHealth): a systematic review of mobile phone and web-based text messaging in mental health. J Med Internet Res. Jun 10, 2016;18(6):e135. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Shen H, van der Kleij RM, van der Boog PJ, Chang X, Chavannes NH. Electronic health self-management interventions for patients with chronic kidney disease: systematic review of quantitative and qualitative evidence. J Med Internet Res. Nov 05, 2019;21(11):e12384. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Australia's health 2018. Australian Institute of Health and Welfare. 2018. URL: https://www.aihw.gov.au/reports/australias-health/australias-health-2018/contents/table-of-contents [accessed 2024-04-04]
• Australian Institute of Health and Welfare. Chronic Diseases and Associated Risk Factors in Australia, 2006. Canberra, Australia. Australian Institute of Health and Welfare; 2006.
• Hart JT. The inverse care law. The Lancet. Feb 27, 1971;297(7696):405-412. [ CrossRef ]
• Davies AR, Honeyman M, Gann B. Addressing the digital inverse care law in the time of COVID-19: potential for digital technology to exacerbate or mitigate health inequalities. J Med Internet Res. Apr 07, 2021;23(4):e21726. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Choi NG, Dinitto DM. The digital divide among low-income homebound older adults: internet use patterns, eHealth literacy, and attitudes toward computer/internet use. J Med Internet Res. May 02, 2013;15(5):e93. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Household use of information technology. Australian Bureau of Statistics. 2018. URL: https://tinyurl.com/4efm6u92 [accessed 2024-03-24]
• Kontos E, Blake KD, Chou WY, Prestin A. Predictors of eHealth usage: insights on the digital divide from the health information national trends survey 2012. J Med Internet Res. Jul 16, 2014;16(7):e172. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Litchfield I, Shukla D, Greenfield S. Impact of COVID-19 on the digital divide: a rapid review. BMJ Open. Oct 12, 2021;11(10):e053440. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Butler DC, Joshy G, Douglas KA, Sayeed MS, Welsh J, Douglas A, et al. Changes in general practice use and costs with COVID-19 and telehealth initiatives: analysis of Australian whole-population linked data. Br J Gen Pract. Apr 27, 2023;73(730):e364-e373. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Arsenijevic J, Tummers L, Bosma N. Adherence to electronic health tools among vulnerable groups: systematic literature review and meta-analysis. J Med Internet Res. Feb 06, 2020;22(2):e11613. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kontos EZ, Bennett GG, Viswanath K. Barriers and facilitators to home computer and internet use among urban novice computer users of low socioeconomic position. J Med Internet Res. Oct 22, 2007;9(4):e31. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Latulippe K, Hamel C, Giroux D. Social health inequalities and eHealth: a literature review with qualitative synthesis of theoretical and empirical studies. J Med Internet Res. Apr 27, 2017;19(4):e136. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Foley K, Freeman T, Ward P, Lawler A, Osborne R, Fisher M. Exploring access to, use of and benefits from population-oriented digital health services in Australia. Health Promot Int. Aug 30, 2021;36(4):1105-1115. [ CrossRef ] [ Medline ]
• Cresswell JW, Plano Clark VL. Designing and Conducting Mixed Methods Research. Thousand Oaks, CA. SAGE Publications; 2007.
• Tappen RM, Cooley ME, Luckmann R, Panday S. Digital health information disparities in older adults: a mixed methods study. J Racial Ethn Health Disparities. Feb 2022;9(1):82-92. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Who can get a card. Services Australia. URL: https://www.servicesaustralia.gov.au/who-can-get-health-care-card?context=21981 [accessed 2023-11-03]
• Levesque JF, Harris MF, Russell G. Patient-centred access to health care: conceptualising access at the interface of health systems and populations. Int J Equity Health. Mar 11, 2013;12:18. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bryant A, Charmaz K. The SAGE Handbook of Grounded Theory, Paperback Edition. Thousand Oaks, CA. SAGE Publications; 2010.
• Johnson MO, Rose CD, Dilworth SE, Neilands TB. Advances in the conceptualization and measurement of health care empowerment: development and validation of the health care empowerment inventory. PLoS One. 2012;7(9):e45692. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Norman CD, Skinner HA. eHEALS: the eHealth literacy scale. J Med Internet Res. Nov 14, 2006;8(4):e27. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kayser L, Karnoe A, Furstrand D, Batterham R, Christensen KB, Elsworth G, et al. A multidimensional tool based on the eHealth literacy framework: development and initial validity testing of the eHealth Literacy Questionnaire (eHLQ). J Med Internet Res. Feb 12, 2018;20(2):e36. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Standards. Australian Bureau of Statistics. URL: https://www.abs.gov.au/statistics/standards [accessed 2024-04-04]
• Al-Janabi H, Flynn TN, Coast J. Development of a self-report measure of capability wellbeing for adults: the ICECAP-A. Qual Life Res. Feb 2012;21(1):167-176. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Van der Feltz-Cornelis CM, Van Oppen P, Van Marwijk HW, De Beurs E, Van Dyck R. A patient-doctor relationship questionnaire (PDRQ-9) in primary care: development and psychometric evaluation. Gen Hosp Psychiatry. 2004;26(2):115-120. [ CrossRef ] [ Medline ]
• Chao J. Continuity of care: incorporating patient perceptions. Fam Med. 1988;20(5):333-337. [ Medline ]
• Wilson CK, Thomas J, Barraket J. Measuring digital inequality in Australia: the Australian digital inclusion index. JTDE. Jun 30, 2019;7(2):102-120. [ CrossRef ]
• Digital participation: a view of Australia's online behaviours. Australia Post. Jul 2017. URL: https://auspost.com.au/content/dam/auspost_corp/media/documents/white-paper-digital-inclusion.pdf [accessed 2024-04-04]
• Poli A, Kelfve S, Motel-Klingebiel A. A research tool for measuring non-participation of older people in research on digital health. BMC Public Health. Nov 08, 2019;19(1):1487. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cohen J. Statistical Power Analysis for the Behavioral Sciences Second Edition. London, UK. Routledge; 1988.
• Borg K, Smith L. Digital inclusion and online behaviour: five typologies of Australian internet users. Behav Inf Technol. Feb 15, 2018;37(4):367-380. [ CrossRef ]
• Mathers A, Richardson J, Vincent S, Joseph C, Stone E. Good Things Foundation COVID-19 response report. Good Things Foundation. 2020. URL: https://tinyurl.com/2peu3kak [accessed 2024-04-04]
• Norman CD, Skinner HA. eHealth literacy: essential skills for consumer health in a networked world. J Med Internet Res. Jun 16, 2006;8(2):e9. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Neves AL, Burgers J. Digital technologies in primary care: implications for patient care and future research. Eur J Gen Pract. Dec 11, 2022;28(1):203-208. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by T Leung; submitted 03.07.23; peer-reviewed by T Freeman, H Shen; comments to author 16.08.23; revised version received 30.11.23; accepted 31.01.24; published 11.04.24.

©Melinda Ada Choy, Kathleen O'Brien, Katelyn Barnes, Elizabeth Ann Sturgiss, Elizabeth Rieger, Kirsty Douglas. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 11.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.